Financial services organizations are high value targets for cyber criminals all over the world. Because of this, it is imperative that the keys and certificates used by financial service DevOps teams are properly protected. If not, bad actors can easily exploit cryptographic assets and wreak havoc on sensitive corporate data, all while remaining undetected ...
Every enterprise IT organization is unique in that it will have different bottlenecks and constraints in its deployment pipelines. With that being said, there are some common problem areas that typically produce the longest lead times in your software delivery process. Here are three more most common areas that generate the longest lead times.
3. Environment Management and Provisioning
The effective and efficient management of dev, test and production environments is critical to a successful release deployment. The combination of increased business requests, the large number of applications, and multiple application infrastructures have exponentially increased the complexity of managing these environments. There is nothing more demoralizing to a dev team than having to wait to get an environment to test a new feature. Lack of environment availability and/or environment contention can create extremely long lead times, delay releases, and increase the cost of release deployments. Dev and Test environments also often are misconfigured or are so different from production environments that they end up with production problems despite having passed preproduction testing.
Creating these environments is a very repetitive task that should be documented, automated, and put under revision control. You need to implement a process to schedule, manage, track, and control all of the environments in your deployment pipeline. Automated and self-service environmental provisioning will streamline the process to reduce lead times. The environments you create need to be as "production-like" as possible. Your developers will also be far more productive and happy. As you automate the provisioning of your environments your MTTR (mean-time-to-repair) will go down significantly as you will be able to replace your environments on a moment's notice and begin to move towards an immutable infrastructure.
4. Manual Software Deployments
People should not move or deploy the "bits" as machines are far better and much more consistent at deploying applications than humans. You would be surprised at the number of organizations that still manually deploy their code. Automating manual deployment tasks is one of the first things you should look at. You can get a lot of quick wins with automation, and this approach can be delivered rapidly without major organizational changes. The initial effort to document and automate your deployment processes pays off once you start letting the machines perform the work. It is not uncommon for organizations to see deployment lead times reduced by over 90%.
Automate your code and configuration deployments with a single set of deployment processes across all environments. Ensure that these deploy from the same source. Deploying the same way across all of your environments is extremely efficient in both time and cost. By using the same process, it gets tested more often and any environmental issues will be easier to identify. All preproduction deployments should be rehearsals for the final deployment into production. The more automated this process is, the more repeatable and reliable it will be. When it's time to deploy to production, you will be ready. This translates into dramatically lower lead times, less downtime and keeps the business open so that it can make more money.
5. Manual Software Testing
Once the environment is ready and the code is deployed, it's time to test to ensure the code is working as expected and doesn't break anything else. The problem is that most organizations today manually test their code base. Manual software testing drives lead times up because the process is very slow, error prone and expensive to scale out across large organizations. As the velocity of software delivery increases, you have to exponentially increase the number of human resources to test the software changes. Furthermore, manual testing provides lower overall coverage. The time and expense of manual testing forces organizations into the "Batch and Queue" mode which slows the overall flow and dramatically increases lead times.
Automated testing is a prime area to focus on when you need to reduce lead times. Automated testing is less expensive, more reliable and repeatable, can provide broader coverage, and is a lot faster. There will be an initial cost of developing the automated test scripts, but a lot of that can be absorbed by shifting manual tester resources to "Test Development Engineers" to focus on automated API-based testing. Over time your manual testing costs and lead times will go down as your quality improves.
The velocity and complexity of software delivery continues to increase as businesses adapt to new economic conditions. Optimizing and automating your deployment pipelines will dramatically reduce your lead times and enable you to deliver software faster and with better quality. Delivering software faster means businesses can innovate and test out new ideas more quickly. The business can deliver features and bring on new revenue streams faster, making them agile enough to respond immediately to marketplace opportunity, events and trends.
Mark Levy is Director of Strategy, Software Delivery, at Micro Focus.