In my first blog in this series, I highlighted some of the main challenges teams face with trying to scale mainframe DevOps. To get past these hurdles, the key is to develop an incremental approach that enables teams to capture value along each step of the journey ...
In many ways running the gauntlet of audit is like playing out the "Do You Feel Lucky" scene from the movie Dirty Harry. Even if organizations have bridged the chasm between Dev and Ops, their go-fast efforts can be shot to pieces by those darned list-wielding and trigger-happy compliance police.
Maverick cops aside, compliance is more critical than ever, especially given the increase in cybersecurity threats and potential for data breaches. What's more, as physical and digital becomes inseparable, organizations have an added responsibility – keeping customers secure and safe as they design, build, test and release software.
So perhaps the old-school approach of loading up the audit list "Magnum-44" and firing off at the end of a development cycle is the best way to mitigate risk. Better that than have all this new-fangled continuous delivery and silo-busting talk (no separation of duties - shock-horror) threaten the foundations of governance and regulatory control – or worse, result in customer harm.
But it doesn't have to be this way. Audit needn't kill innovation, just as DevOps shouldn't cause undue consternation for auditors. Each can and should benefit the other; it just takes some work and plain old common sense.
Mutual Respect and Early Engagement
The earlier DevOps practitioners engage audit the better. Good auditors have broad perspectives regarding the world of business operations, which can be invaluable during application design, development and testing. Remember too, most auditors will be unfamiliar with modern DevOps practices, so education in context of how the new tools and automated processes benefit the auditing function is critical.
For example, take synthetic test data generation. All well and good for supporting our velocity goals, but also key evidence that a control strategy is in place to address a specific business risk – protecting customer information and sensitive data.
Many DevOps practices won't just be unfamiliar to auditors, they'll also be counterintuitive. Get ready for some friction (and more auditing) if a request for documented change control procedures is glibly rebuffed by developers stating that they don't exist, because – wait for it - they can deploy straight to production.
But there's no need to hit the panic button. The most important thing DevOps practitioners can do is to illustrate how internal procedures are still supporting controls' objectives – which can actually be easier when development and operations silos are removed. Like for example automatically initiating static-code analysis with a software build to ensure incorruptible code is moving through the pipeline – or development and IT operations working together during code reviews to ensure comprehensive application and infrastructure monitoring is established before production deployments.
Continuous Testing – the Auditing "Enforcer"
Impressing the value of DevOps practices (especially continuous testing) to auditors can also be beneficial. The communication and collaboration stimulated by automated testing can help auditors understand how business risks are being systemically managed.
For example, if security vulnerabilities are detected through an automated test during a software build, then the practice of failing the build becomes a key control surfaced by automation. Combine this with a rule that no production deployments can be made until the problem is fixed and the control becomes even better.
Of course it won't always be plain sailing. New automated methods of software validation may require more education, especially when they negate the need for the more traditional and familiar auditing controls. Be prepared to evidence how practices such as test-driven development (developers writing tests before coding) or requirements-based design (automatically generating test cases based on business requirements) are ensuring services deployed to production incur less risk.
It'll also be necessary to demonstrate how DevOps practices deliver controls that cater for production failures. For example, providing developers immediate access to application performance information specific to their code changes so they can quickly troubleshoot problems. Then by applying code fixes to the continuous delivery pipeline, tools such as release automation can demonstrate how resolutions are being applied faster or environments rolled back to a known good state.
From Box Ticking to Continuous Improvement
DevOps practices and auditing are not mutually exclusive; they're equally important in any business improvement program. By automating risk mitigation controls with DevOps and establishing bi-directional feedback loops, DevOps practitioners can quickly impress how new methods reduce business risk and support compliance goals – faster and with lower cost. Get this right and audit will view DevOps as less "tick in the box" and more as a value-added function.
In many ways audit is a real litmus-test for DevOps. It requires developers and IT operations purposefully collaborating and coordinating activities towards a top of mind boardroom issue – business governance. Sure, DevOps speed always grabs the headlines, but let's not forget the responsibility teams have in supporting many other corporate stakeholders and systems – including audit, risk management and compliance.
So the next time you get a visit from auditors – don't dread it or duck for cover – engage and demonstrate how DevOps supports shared business goals.
Pete Waterhouse is a Senior Strategist at CA Technologies.