DEVOPSdigest

What makes an engineering team elite?

This is a question many technical leaders before me have been asked or asked themselves and one that I returned to often throughout my career. While I won't be the last to ask this question, our industry has come a long way in defining engineering excellence, providing parameters through which to define what constitutes as "elite."

But what exactly should you measure to benchmark elite engineering performance?

According to data sourced from nearly 2,000 engineering teams and 4.5 million branches, pull requests and merge requests in our engineering benchmarks report, elite teams — those within the top 10% — are great in 3 or more of these categories:

■ Cycle time under 42 hours

■ Coding time less than 30 minutes

■ Pickup time under 1 hour

■ Review time under 1 hour

■ Deploy time under 1 hour

■ Deployments on a daily basis

■ Fewer than 105 code changes in their pull requests (PRs)

■ Rework rate under 2%

■ Planning accuracy of 80% or more

■ Capacity accuracy above 85%

*Calculated using 75th percentile

Augmenting DORA metrics to measure excellence

You may have noticed these metrics aren't just the typical indicators of engineering performance: DORA metrics(link is external). These new metrics build on DORA to provide a more comprehensive view and fresher perspective on software delivery management by benchmarking metrics that translate into meaningful insights and create action.

We can measure all we want, but measurements are meaningless unless they ignite change in the behaviors of your engineering organization.

Yes, DORA metrics are a fantastic jumping-off point for benchmarking your engineering team. However, context is crucial when it comes to truly improving performance, and DORA metrics just don't provide enough information to chart a new course for your engineering ship on their own:

Three reasons to build on your DORA metrics

1. DORA metrics are lagging indicators

While understanding past performance through lagging indicators of success (like DORA metrics) is essential for engineering leaders, it isn't the most crucial aspect of performance to understand. Technical leaders must have a firm grip on leading indicators of success that actually foster proactivity and change in real-time, such as pull request size and coding time.

2. DORA metrics don't clearly connect to business impact

Even if we meet our goals according to the DORA metrics, we may still fail to deliver actual business value. How? Often, we simply aren't speaking the same language as our non-technical executive peers, setting us up for misalignment. Alternatively, metrics like planning accuracy allow engineering leaders to highlight their successes in a way that makes sense to the non-technical crowd. When this harmony is achieved, engineering success will align with business goals and customer needs, creating synergy between all stakeholders.

3. DORA metrics don't show us where to improve workflows

While DORA metrics measure progress in the software delivery lifecycle, they don't provide the context in workflows we need to adjust our performance proactively. Metrics like pull-request size and rework rate offer better context and real-time data around engineering improvement.

Adding elite engineering metrics to DORA metrics

As we look beyond DORA metrics, we can break down the benchmarks of an elite engineering team into three categories: delivery lifecycle, developer workflow and business alignment. These three categories work together to create a harmonious flow of proactive improvement and, ultimately, engineering excellence.

1. Delivery lifecycle benchmarks for elite engineering teams

There are five key metrics to look at when benchmarking the delivery lifecycle: cycle time, coding time, pickup time, review time and deploy time.

Cycle time — lead time for changes as a DORA metric — is one of my favorite engineering metrics because it gives such an overarching view of your delivery lifecycle. As you review the time it takes from first commit to release, you can optimize your processes to deliver new features to your customers faster by drilling down into coding time, pickup time, review time and deployment time.

BENCHMARK: Elite engineering teams see a cycle time of under 42 hours, with coding time under 30 minutes and pickup, review and deploy times under one hour.

2. Developer workflow benchmarks for elite engineering teams

Three key metrics help us measure developer workflow: Pull-request size, deploy frequency, and rework rate. Pull-request size (or merge requests) shows us the average code footprint of a change. Deploy frequency offers a deeper understanding of cycle time by measuring the frequency at which code is sent to production and enabling developers to complete a delivery end-to-end. Rework rate shows us the average code churn.

BENCHMARK: Elite engineering teams see fewer than 105 code changes in their pull requests, daily deployments, and a rework rate under 2%.

3. Business alignment benchmarks for elite engineering teams

Two metrics are king when it comes to business alignment benchmarks for elite engineering teams: planning accuracy and capacity accuracy. Planning accuracy demonstrates how often your team delivers on their promises. On the other hand, capacity accuracy illustrates your team's ability to deliver on unplanned work by measuring all completed work against planned work, delivering a ratio to hold your team accountable.

BENCHMARK: Elite engineering teams see planning accuracy of 80% or more and capacity accuracy above 85%.

Achieving elite status matters

I have come to understand that settling for "okay" isn't an option, especially in the current tech ecosystem. A recent report by StartupGenome(link is external) indicated that 90% of startups unfortunately do not make it. Of the remaining 10% that do survive, only 15% exit for $50 million or more. These statistics are not shared to discourage, but to set a realistic picture of the competitive environment we find ourselves in.

While DORA metrics provide helpful snapshots of our work, they can't be the sole measure of elite engineering teams. We must look deeper into our everyday processes to set engineering benchmarks that help us achieve our status as engineering elites. If you haven't considered moving beyond DORA metrics, I encourage you to be the visionary in your engineering organization and initiate a discussion around implementing elite engineering benchmarks in your organization today.