App Developers Waste Time Debugging Errors Discovered in Production
November 28, 2016

Mohit Bhatnagar

Almost half (43 percent) of app developers spend between 10 and 25 percent of their time debugging application errors discovered in production, rather than developing new features, according to ClusterHQ's first Application Testing survey.

The top three challenges reported by survey respondents were:

■ An inability to fully recreate production environments in testing (33%)

■ Interdependence on external systems that makes integration testing difficult (27%)

■ Testing against unrealistic data before moving into production (26%)

With more organizations transitioning to microservices-based applications, it is apparent that legacy testing processes are no longer sufficient to meet the speed and agility requirements of modern businesses.

“Our Application Testing survey highlights that legacy software development practices, like relying on narrow subsets of synthetic data for testing, no longer cut it for teams focused on maximizing the amount of time they spend building features that users value,” said Mark Davis, CEO, ClusterHQ. “Forward looking software developer leaders understand that to deliver innovation to customers they must effectively manage the entire application lifecycle across a diverse range of infrastructures, a process that begins with identifying and eliminating bugs as early as possible so that teams can focus on adding end-user value.”

Errors found in production are costly

Respondents were asked to identity the environment in which they encounter bugs that are most costly to fix. Unsurprisingly, the majority (62%) selected production as the most expensive stage of app development to fix errors, followed by development (18%), staging (7%), QA (7%) and testing (6%).

Next, the survey took a deeper look at how often bugs are being encountered in production as a result of incomplete testing. Findings revealed:

■ Every day: 11%
■ Two to three times per week: 13%
■ Once per week: 12%
■ Every other week: 15%
■ Once a month: 25%
■ Don’t know: 21%
■ Never: 3%

A quarter of respondents report encountering bugs discovered in production one or more times per week. Developing new features should be priority number one in agile software development, however when asked what percentage of development time is spent debugging application errors discovered in production instead of working on new features, results showed:

■ Less than 10%: 34%
■ Between 10-25%: 43%
■ Between 25-50%: 18%
■ Between 50-75%: 4%
■ More than 75%: 1%

Causes of bugs in production

Testing is a crucial part of an application’s lifecycle, but it’s inherently challenging to ensure that tests done in development will mirror what happens in production. Survey takers were asked to select the top challenge associated with testing that causes bugs to appear in production. They responded:

■ Inability to fully recreate production environments in testing: 33%
■ Interdependence on external systems, making integration testing difficult: 27%
■ Testing against unrealistic data before moving into production: 26%
■ Difficulty sharing test data across different teams: 10%
■ Difficulty creating staging environments for testing: 4%

Recreating production environments was cited as the leading cause of bugs appearing in production, supporting the notion that testing on a laptop is like “flying” in a flight simulator – the experience can be very different when you actually get up in the air. This challenge was followed closely by interdependence on external systems that makes integration testing cumbersome, which leads into the third most cited challenge: testing against unrealistic data. At present, data is difficult to move between all the places that it is needed, including in test infrastructure. As a result, unrealistic, mock data sets are often used to test applications. However, these unrealistic data sets cannot prepare applications for all real world variables, and thus cause serious, expensive, and time consuming issues down the line.

Testing apps against realistic data

A resounding number of respondents (88%) would like the ability to test with realistic data during application development. However, this is not as easy as it sounds, and there are real challenges that prevent this from being the norm. When asked to identify the biggest challenge to testing against realistic data sets, respondents reported:

■ Keeping test data up-to-date: 23%
■ Moving data to all the places it is needed for testing: 19.5%
■ Keeping track of different versions of data to be used for different purposes: 19.5%
■ Managing access controls to data: 18%
■ Making copies of production data: 14%
■ Storage costs: 6%

The full survey polled 386 respondents. Not all respondents answered every question. Results shown are based on the percent of people who answered each individual question. In this survey, 41 percent identified as DevOps team members, followed by developers (37%), other (8%), operations (7%), QA (5%) and security (2%). Respondents work for organizations that range in size from 1 to 100 employees (49%), 101-500 employees (20%), 501-2500 employees (10%), 2501-5000 employees (5%), 5001-10,000 employees (3%) and 10,000+ employees (13%).

Mohit Bhatnagar is VP Product at ClusterHQ.

The Latest

August 15, 2018

Microservices are a hot topic in IT circles these days. The idea of a modular approach to system building – where you have numerous, smaller software services that talk to each other instead of monolithic components – has many benefits ...

August 13, 2018

Agile is expanding within the enterprise. Agile adoption is growing within organizations, both more broadly and deeply, according to the 12th annual State of Agile report from CollabNet VersionOne. A higher percentage of respondents this year report that "all or almost all" of their teams are agile, and that agile principles and practices are being adopted at higher levels in the organization ...

August 09, 2018

For the past 13 years, the Ponemon Institute has examined the cost associated with data breaches of less than 100,000 records, finding that the costs have steadily risen over the course of the study. The average cost of a data breach was $3.86 million in the 2018 study, compared to $3.50 million in 2014 – representing nearly 10 percent net increase over the past 5 years of the study ...

August 08, 2018

Hidden costs in data breaches – such as lost business, negative impact on reputation and employee time spent on recovery – are difficult and expensive to manage, according to the 2018 Cost of a Data Breach Study, sponsored by IBM Security and conducted by Ponemon Institute. The study found that the average cost of a data breach globally is $3.86 million ...

August 06, 2018

The previous chapter in this WhiteHat Security series discussed dependencies as the second step of the Twelve-Factor App. This next chapter examines the security component of step three of the Twelve-Factor methodology — storing configurations within the environment.

August 02, 2018

Results from new Forrester Consulting research reveal the 20 most important Agile and DevOps quality metrics that separate DevOps/Agile experts from their less advanced peers ...

July 31, 2018

Even organizations that understand the importance of cybersecurity in theory often stumble when it comes to marrying security initiatives with their development and operations processes. Most businesses agree that everyone should be responsible for security, but this principle is not being upheld on a day-to-day basis in many organizations. That’s bad news for everyone. Here are some best practices for implementing SecOps ...

July 30, 2018

While the technologies, processes, and cultural shifts of DevOps have improved the ability of software teams to deliver reliable work rapidly and effectively, security has not been a focal point in the transformation of cloud IT infrastructure. SecOps is a methodology that seeks to address this by operationalizing and hardening security throughout the software lifecycle ...

July 26, 2018

Organizations are shifting away from traditional, monolithic architectures, with three-quarters of survey respondents delivering at least some of their applications and more than one-third delivering most of their applications as microservices, according to the State of DevOps Observability Report from Scalyr ...

July 24, 2018

What top considerations must companies make to ensure – or at least help improve – Agile at scale? The following are key techniques and practices to help accelerate Agile delivery rollouts and scale Agile and DevOps in the Enterprise ...

Share this