App Developers Waste Time Debugging Errors Discovered in Production
November 28, 2016

Mohit Bhatnagar
ClusterHQ

Almost half (43 percent) of app developers spend between 10 and 25 percent of their time debugging application errors discovered in production, rather than developing new features, according to ClusterHQ's first Application Testing survey.

The top three challenges reported by survey respondents were:

■ An inability to fully recreate production environments in testing (33%)

■ Interdependence on external systems that makes integration testing difficult (27%)

■ Testing against unrealistic data before moving into production (26%)

With more organizations transitioning to microservices-based applications, it is apparent that legacy testing processes are no longer sufficient to meet the speed and agility requirements of modern businesses.

“Our Application Testing survey highlights that legacy software development practices, like relying on narrow subsets of synthetic data for testing, no longer cut it for teams focused on maximizing the amount of time they spend building features that users value,” said Mark Davis, CEO, ClusterHQ. “Forward looking software developer leaders understand that to deliver innovation to customers they must effectively manage the entire application lifecycle across a diverse range of infrastructures, a process that begins with identifying and eliminating bugs as early as possible so that teams can focus on adding end-user value.”

Errors found in production are costly

Respondents were asked to identity the environment in which they encounter bugs that are most costly to fix. Unsurprisingly, the majority (62%) selected production as the most expensive stage of app development to fix errors, followed by development (18%), staging (7%), QA (7%) and testing (6%).

Next, the survey took a deeper look at how often bugs are being encountered in production as a result of incomplete testing. Findings revealed:

■ Every day: 11%
■ Two to three times per week: 13%
■ Once per week: 12%
■ Every other week: 15%
■ Once a month: 25%
■ Don’t know: 21%
■ Never: 3%

A quarter of respondents report encountering bugs discovered in production one or more times per week. Developing new features should be priority number one in agile software development, however when asked what percentage of development time is spent debugging application errors discovered in production instead of working on new features, results showed:

■ Less than 10%: 34%
■ Between 10-25%: 43%
■ Between 25-50%: 18%
■ Between 50-75%: 4%
■ More than 75%: 1%

Causes of bugs in production

Testing is a crucial part of an application’s lifecycle, but it’s inherently challenging to ensure that tests done in development will mirror what happens in production. Survey takers were asked to select the top challenge associated with testing that causes bugs to appear in production. They responded:

■ Inability to fully recreate production environments in testing: 33%
■ Interdependence on external systems, making integration testing difficult: 27%
■ Testing against unrealistic data before moving into production: 26%
■ Difficulty sharing test data across different teams: 10%
■ Difficulty creating staging environments for testing: 4%

Recreating production environments was cited as the leading cause of bugs appearing in production, supporting the notion that testing on a laptop is like “flying” in a flight simulator – the experience can be very different when you actually get up in the air. This challenge was followed closely by interdependence on external systems that makes integration testing cumbersome, which leads into the third most cited challenge: testing against unrealistic data. At present, data is difficult to move between all the places that it is needed, including in test infrastructure. As a result, unrealistic, mock data sets are often used to test applications. However, these unrealistic data sets cannot prepare applications for all real world variables, and thus cause serious, expensive, and time consuming issues down the line.

Testing apps against realistic data

A resounding number of respondents (88%) would like the ability to test with realistic data during application development. However, this is not as easy as it sounds, and there are real challenges that prevent this from being the norm. When asked to identify the biggest challenge to testing against realistic data sets, respondents reported:

■ Keeping test data up-to-date: 23%
■ Moving data to all the places it is needed for testing: 19.5%
■ Keeping track of different versions of data to be used for different purposes: 19.5%
■ Managing access controls to data: 18%
■ Making copies of production data: 14%
■ Storage costs: 6%

The full survey polled 386 respondents. Not all respondents answered every question. Results shown are based on the percent of people who answered each individual question. In this survey, 41 percent identified as DevOps team members, followed by developers (37%), other (8%), operations (7%), QA (5%) and security (2%). Respondents work for organizations that range in size from 1 to 100 employees (49%), 101-500 employees (20%), 501-2500 employees (10%), 2501-5000 employees (5%), 5001-10,000 employees (3%) and 10,000+ employees (13%).

Mohit Bhatnagar is VP Product at ClusterHQ.

The Latest

June 20, 2018

The larger the company size, the higher the proportion of low IT performers, according to the State of DevOps: Market Segmentation Report from Puppet, based on the 2017 State of DevOps Survey data ...

June 18, 2018

An overwhelming 83 percent of respondents have concerns about deploying traditional firewalls in the cloud, according to Firewalls and the Cloud, a survey conducted by Barracuda Networks...

June 14, 2018

Despite the vast majority of cloud management decision-makers believing that DevOps and microservice enablement are important, very few believe that their organizations are capable of delivering them today — a gap that is costing the average enterprise $34 million per year, according to new report from the Ponemon Institute ...

June 12, 2018

Dev teams are doing their best to give the customers what they want, but oftentimes find themselves in between a rock and a hard place. Teams are struggling to get up to speed with new tools that are meant to make their lives easier and more realistic to hit deadlines. With spring cleaning season upon us, take time this season to tune up agile processes and continue the work of advancing the shift towards DevOps ...

June 11, 2018

The ability to create a culture of DevOps is critical to any organization's ability to deliver applications and services at a high rate of speed, but can we clearly and concisely answer the question: What exactly is DevOps? Despite the best intentions, some large companies are struggling to understand what DevOps actually is, and what it takes to fully implement its concepts and reap its benefits ...

June 07, 2018

The Twelve-Factor App is a methodology that offers a 12-step best practice approach for developers to apply when building software-as-a-service apps that are both scalable and maintainable in a DevOps world. As software continues to be written and deployed at a faster rate and in the cloud, development teams are finding there is more room for failure and vulnerabilities. This blog series will discuss how to build a Twelve-Factor app securely ...

June 05, 2018

Everyone understands the importance of code quality for applications, particularly when DevOps results in releases becoming faster and faster, reducing the room for error. The same issues increasingly apply to databases, which are a vital part of DevOps workflows. Fail to integrate the database into DevOps and you'll face bottlenecks that slow down your processes and undermine your efforts ...

June 04, 2018

DevOps and security traditionally have been siloed functions and security is often seen as a policing function by DevOps team members. However, more mature business leaders are trying to bridge the gap between the two functions to achieve business excellence. This theme was evident from our recent survey where 39% of respondents cited that DevOps and development teams care greatly about their cybersecurity posture, showing that the silo between security/IT and development teams is diminishing ...

May 31, 2018

DEVOPSdigest asked experts from across the IT industry for their opinions on the top tools to support DevSecOps. Part 5, the last installment, offers some final thoughts about "tools" that are not necessarily technology ...

May 29, 2018

DEVOPSdigest asked experts from across the IT industry for their opinions on the top tools to support DevSecOps. Part 4 covers code and data ...

Share this