DEVOPSdigest

Zero-day vulnerabilities create security holes that can and almost certainly will be exploited. They also could crash your system when you do an upgrade. These threats aren't new, but their threat profile has increased; some of these vulnerabilities persist for long periods of time. So pervasive are the concerns around these threats that companies like Google, with its Project Zero, employ a team of skilled security analysts solely focused on discovering zero-day vulnerabilities.

Flaws in the Status Quo

The traditional software development and QA testing processes fail to identify bugs and flaws that manifest in live usage. Static and dynamic testing, RASP, and vulnerability assessments all look for known problems or known fallible coding techniques which makes it difficult to identify zero-day vulnerabilities (which are, by definition, unknown.)

Even using blue-green or canary staging approaches, software bugs may not be seen and will propagate, meaning the code or application problems caused by these flaws are pushed live, because that code is not tested with production traffic. Rollbacks can be used, but they're expensive. In fact, the costs associated with correcting software defects in production are as much as 640 times more expensive than prior to release (Jones, Capers. Applied Software Management. Global Analysis of Productivity and Quality), 2008.

The most common means by which bad actors execute external attacks continue to be
application weaknesses and software vulnerabilities, according to Forrester(link is external). Since the existing testing methodologies have trouble finding these critical zero-day vulnerabilities, other approaches are being tried, including advanced log analysis and bug bounty programs. Log analysis does not find all potential zero-days vulnerabilities, as the exploit (if it is in the logs) may be discovered but too often after it has become a problem, even if artificial intelligence and machine learning are applied.

Bug bounty programs crowdsource bugs with the help of the security research and hacker communities, with the resulting information then sold to software vendors. It is an interesting idea, but this approach still ignores a key fact: it's significantly better to prevent those glitches from going live at all than it is to find them after the fact, when they've already had the potential to be exploited. Also, you don't know if a hacker is going to turn around and sell that exploit to a competitor or even a nation-state or hold for his or her own (nefarious) purposes. When it comes to the underworld of the internet, you have no assurance where this information is being sold or to whom.

Updating the Status Quo by Testing with Production Traffic

To observe how new releases will perform with live (not just simulated) customer traffic,
software and DevOps teams need a way to test accurately in real time. By evaluating release versions side-by- side in lockstep and actively looking for differences in responses, defects can be predicted and detected quickly. Software regressions are identified before customers are affected. It is crucial to fix the defects before they go into production to prevent the potential release of zero-day vulnerabilities.

But it is equally important to provide clear and reproducible bug reports to developers based on production traffic flows to identify where the problem is in the execution of the code. One can track IDs of the same transaction to see how different versions behave. The determination of faulty code is derived from observations of primary network communications and interactions rather than secondary data from logs or the interpretation or analysis of those logs.

If organizations will adopt proactive software testing, they will be able to find these flaws and bugs before they've been pushed live. By doing this, these flaws or bugs can be addressed first at the root, rather than having to do a rollback. As the adage goes, an ounce of prevention is worth a pound of cure.

Defeating Zero-Days

Zero-days can wreak havoc in both the short and long terms. Testing for vulnerabilities and errors before releases go out will help software and DevOps teams have the upper hand over cybercriminals. This is possible through testing with production traffic. This approach prevents costly rollbacks and clunky staging. It also has multiple benefits: it finds zero-day vulnerabilities, as well as the place in the code execution where the failure occurs and a reproducible roadmap the developer can use to fix the bug. Using production traffic in testing helps enterprises protect their data while releasing excellent new versions of their software.