DEVOPSdigest

Rapid business change, fueled by software innovation, is transforming how software delivery organizations define, develop, test, and release business applications. For these software organizations to keep their competitive advantage in today's complex and volatile digital marketplace, they must become more agile, adaptive, and integrated into the business. They must get fully on board with the digital transformation of business practices. However, most current software delivery practices can't keep pace with the demands of the business.

Long software delivery cycles are a significant impediment to business technology innovation. Agile development teams have shortened development cycles, but Agile by itself is insufficient as it does not remove the cultural and technical barriers between development and operations. DevOps principles and practices developed in response to this problem, facilitates cooperation and coordination amongst teams to deliver software faster and with better quality.

Every enterprise IT organization currently has software delivery processes in place. This is known as its deployment pipeline. In fact, large enterprises have many deployment pipelines since they typically have hundreds of applications spread across many types of application infrastructures that are usually managed by different, geographically dispersed teams.

The whole goal of scaling "DevOps" for the enterprise is to prioritize and optimize these existing deployment pipelines, seek efficiencies, and remove waste to deliver better business outcomes. Creating new and optimizing existing deployment pipelines in large IT organizations is key to improving their efficiency and effectiveness in delivering software at the speed that the business requires.

So where do you start? Every enterprise IT organization is unique in that it will have different bottlenecks and constraints in its deployment pipelines. I recommend that you prioritize your deployment pipelines and conduct a value stream mapping(link is external) exercise to identify specific problem areas. Starting and Scaling DevOps in the Enterprise(link is external), by Gary Gruver is a great book and provides a good framework for getting started. With that being said, there are some common problem areas that typically produce the longest lead times in your software delivery process.

First, let's make sure we are on the same page with what lead time is. Lead time is one of two metrics commonly used to evaluate performance in an IT value stream or deployment pipeline. The other metric is task time. Whereas the lead time clock starts when the customer request is made and ends when it is fulfilled, the task time clock starts only when you begin work on the customer request; specifically it omits the time in the queue waiting to be processed. Requests are fulfilled for both internal and external customers. So, if the Dev team requests a new test environment, it may take only one day to actually create the environment, but the request might wait in the queue for four days. Therefore, the actual lead time would be five days and the processing time would be one day.

Here are the most common areas that generate the longest lead times:

1. Organization Bottlenecks

DevOps culture strives to break down the organizational silos and transition more to product teams. This is because the current silo'd organizational structure provides headwinds to the objective of short lead times and continuous flow. Organizational silos are artifacts of the industrial era designed specifically for "Batch and Queue" processing which drives up lead times with handoffs from one team or organization to another.

Handoffs drive lead times up by requiring additional communication and usually using different tools. Each handoff is potentially a queue in itself. Resolving ambiguities require additional communication between teams and can result in significant delays, high costs, and failed releases. Unfortunately, only so much can be successfully conveyed with words and diagrams. There is a great deal of tacit knowledge, which is difficult to convey by just writing it down or verbalizing it. You also get work bounce back or work bounced around between teams. Handoffs can become the "Hot Potato" that nobody wants. You need to strive to reduce the number of handoffs by either automating a significant portion of the work or re-organizing into product teams that are not dependent on other teams to deliver value to the customer.

Shared resources across projects and organizational silos is another bottleneck. Lead times can typically exceed 2x greater than task times if waiting on a shared resource while they are completing another task for another project or team. The more you can encapsulate the team to be enabled to continuously work on creating customer value, the faster the flow, the better the quality, and the lower the lead time.

2. Approval Processes

The approval process is another area that can generate long lead times and waste. The approval process within most large enterprises is slow and complex. It is often comprised of a set of manual stovepipe processes that use email and Microsoft office tools to track, manage, and, more often than not, wait on people for approval of a software change. Insufficient data leads to hasty or faulty approvals or bounce backs, further frustrating software delivery teams, reducing quality, and impeding deployments.

Change approval processes originally were developed to mitigate risk and provide oversight to ensure adherence to auditable standards for moving changes into production. Over time this process tends to get bureaucratic as board members struggle to understand the changes the teams are implementing. External change approval boards can increase lead times significantly, with negligible impact on stability.

Peer reviews, along with the DevOps practice of infrastructure as code and automating the deployment pipeline, enable better change control, a more rigorous approval process, and a dramatic improvement in speed. All the software artifacts and automated scripts are kept under version control in one SCM tool. With everything in version control, the SCM system contains all the documented changes and can provide a complete auditable record. Toolchain integration with the change management system will provide visibility of the state of the change, while the approval process is the automated test that validates whether that particular change passes the criteria to be deployed. Releasing into production might need approval from the business, but everything up to that point could be automated with dramatically reduced lead times.

Read The 5 Longest Lead Times in Software Delivery - Part 2, outlining the other three common areas that generate the longest lead times.