Differentiating DevOps Leaders from Laggards
August 02, 2018

Anand Akela
Tricentis

Results from new Forrester Consulting research reveal the 20 most important Agile and DevOps quality metrics that separate DevOps/Agile experts from their less advanced peers. The study was commissioned by Tricentis and led by Forrester VP and Principal Analyst Diego Lo Giudice.

The research examined 94 quality metrics that were classified into four categories. As a high-level summary, the research uncovered the following:

■ Understanding of business risk is the most important determining factor of DevOps and Agile maturity.

■ Experts focus primarily on contextual metrics (e.g., requirements coverage) while others focus on “counting” metrics (e.g., number of tests).

■ Experts are more likely to measure the user experience across an end-to-end transaction while others rely on application-specific or team-specific metrics.

Agile/DevOps Leaders vs Laggards

The study also found that companies with the most successful Agile and DevOps initiatives do a number of things differently:

■ They are transforming software testing into Continuous Testing by adopting five core practices, such as automating end-to-end functional testing and integrating testers into cross-functional teams.

■ They are almost twice as likely to consider automating the software quality process to be a “critical business differentiator” (53% vs. 27%).

■ They are significantly more likely (by 23 percentage points) to have high levels of automation for key testing and QA processes (test case design, functional test automation, test data management, etc.).

“Organizations adopt DevOps to accelerate the delivery of innovative software that creates positive user experiences,” explained Clark Golestani, Managing Director of C Sensei Group and former President, Emerging Businesses and Global CIO of Merck. “If you start accelerating delivery without also rethinking your approach to quality, you will end up rapidly delivering updates that drive your customers to competitors. Automated, continuous testing is the gate that allows you to protect your organization from significant risk as you innovate at the speed modern business demands.”

The Risk “Blind Spot”

Risk is a prevalent thread throughout the various parts of the report. Most firms (80%) believe they deliver within acceptable business risk, but fewer than a quarter state that their QA and testing processes completely cover business risk. Moreover, just 15% of respondents say that their test suites reliably provide a good indication of business risk.

Interestingly, although risk-related metrics do not rank high in overall popularity, Agile and DevOps experts measure them significantly more frequently than the non-experts do. In fact, Agile and DevOps experts consistently rank risk-related metrics among their top 3 most valuable metrics in different phases and categories.

Other notable findings include:

■ Only 26% of firms with active Agile/DevOps adoptions are currently practicing the five core Continuous Testing practices shown to separate Agile/DevOps leaders from laggards.

■ Many CXOs and high-level decision makers are vastly overestimating the maturity of their firms’ Agile and DevOps practices.

■ Firms that rely on manual testing are more than twice as likely to report that testing is a bottleneck (59%), compared to those with more automated quality processes (26%).

Anand Akela is VP of Product Marketing at Tricentis

The Latest

September 18, 2018

To celebrate IT Professionals Day 2018 (this year on September 18), the SolarWinds IT Pro Day 2018: A World Powered by Tech Pros survey explores a "Tech PROactive" world where technology professionals have the time, resources, and ability to use their technology prowess to do absolutely anything ...

September 17, 2018

The role of DevOps in capitalizing on the benefits of hybrid cloud has become increasingly important, with developers and IT operations now working together closer than ever to continuously plan, develop, deliver, integrate, test, and deploy new applications and services in the hybrid cloud ...

September 13, 2018

"Our research provides compelling evidence that smart investments in technology, process, and culture drive profit, quality, and customer outcomes that are important for organizations to stay competitive and relevant -- both today and as we look to the future," said Dr. Nicole Forsgren, co-founder and CEO of DevOps Research and Assessment (DORA), referring to the organization's latest report Accelerate: State of DevOps 2018: Strategies for a New Economy ...

September 12, 2018

This next blog examines the security component of step four of the Twelve-Factor methodology — backing services. Here follows some actionable advice from the WhiteHat Security Addendum Checklist, which developers and ops engineers can follow during the SaaS build and operations stages ...

September 10, 2018

When thinking about security automation, a common concern from security teams is that they don't have the coding capabilities needed to create, implement, and maintain it. So, what are teams to do when internal resources are tight and there isn't budget to hire an outside consultant or "unicorn?" ...

September 06, 2018

In evaluating 316 million incidents, it is clear that attacks against the application are growing in volume and sophistication, and as such, continue to be a major threat to business, according to Security Report for Web Applications (Q2 2018) from tCell ...

September 04, 2018

There's a welcome insight in the 2018 Accelerate State of DevOps Report from DORA, because for the first time it calls out database development as a key technical practice which can drive high performance in DevOps ...

August 29, 2018

While everyone is convinced about the benefits of containers, to really know if you're making progress, you need to measure container performance using KPIs.These KPIs should shed light on how a DevOps team is faring in terms of important parameters like speed, quality, availability, and efficiency. Let's look at the specific KPIs to track for each of these broad categories ...

August 27, 2018

Protego Labs recently discovered that 98 percent of functions in serverless applications are at risk, with 16 percent considered "serious" ...

August 23, 2018

After another record year of breaches, The 2018 DevSecOps Community Survey found that 3 in 10 respondents suspected or verified breaches stemming from vulnerabilities in open source components — a 55% increase over 2017, and 121% increase since 2014 ...

Share this