Crisis Management: What Should You Do When Things Go Wrong?
June 12, 2017

Joyce Lin
Postman

What if you discover a fatal error or an exploit in your app? What if your app is down during a crucial time? As a developer, how you react to a crisis can mean the difference between minor blip and an embarrassing or costly company blunder.

Your organization is depending on you to save the day. Here's a crisis management plan to get things right when they go wrong:

1. Stay calm and think clearly

This is obvious, but easier said than done. Staying calm will allow you to think more clearly. Working long hours under stress frequently leads to subpar code, and may be the reason why your app is down in the first place.

Don't underestimate the value of taking a walk, grabbing a snack or something else that changes your outlook to see the issue in a different light. And don't worry about wasting time, your brain will still be working on the issue in the background.

2. Revert to working code

If the appropriate solution might take a while to implement, or you have no idea how long the fix will take, roll back to a previous version of the code as a temporary measure. This is the last time you know your service was working, and it's a stable build.

Reverting to working code can provide the extra time you need to thoughtfully address the issue. When you're no longer rushed, you can gather the information you need to more effectively resolve the problem.

3. Monitor and alert stakeholders

You're a responsible developer, so hopefully you've set up some monitoring and alerting for your app. Before you ever find yourself in crisis mode, make sure you set up the right triggers and intervals for monitoring.

Once you receive the initial alert of a failure, notify the appropriate engineers of the outage so they can get started on a fix right away. Also notify other internal stakeholders so they're aware that their services might be impacted. As an alternative to notifications, you can set up a status page to inform those who rely on your app of updates on performance and availability.

4. Debug the issue

Now that you're set up for success, it's time to dig into the issue – starting with your logs. Logs are only as helpful as you make them. You should be logging the right activities with descriptive log statements. In addition to your existing log statements, add temporary log statements to guide the debugging process.

If you're still uncertain about the cause of the outage, focus on isolating the issue. Some code bases are structured in a way that is easy to see where the code is failing, but some dependencies and abstractions make it tricky to pin down the root culprit. If you're at a loss, you can try the debugging variation of a binary search by dividing and conquering to pin down where the code is failing in the most efficient manner.

Finally, don't work in a silo. You might benefit from talking through the problem or getting another perspective. Some developers like to pair program, rely on another teammate for rubber duck debugging, or use a literal rubber duck to slow down and articulate code line by line.

5. Push fixes with continuous deployment

When you're making code changes under pressure, you might be hacking together a solution. As a result, you might also be cutting corners and incurring technical debt in exchange for a quick turnaround.

Running an automated test suite guarantees consistent code coverage, and rigorous regression testing hedges against unintentionally affecting another dependency. Automating the build, testing and deployment process ensures you deliver patches in the fastest and most efficient manner possible.

6. Communicate changes to the team

Once you've patched the fix, communicate the status and diffs to the team. Just because your world came to a standstill with the problem doesn't mean the same happened for the rest of your team. They are continuing to work on their own features and issues.

Communicate what is necessary to keep them up to speed on the changes, and how it might impact what they're working on. Some teams keep track of an activity feed to stay up-to-date with the latest code changes, while others prefer to set up an integration with their preferred messaging platforms.

Joyce Lin is a Developer Evangelist for Postman.

The Latest

June 23, 2017

Why Agile? DEVOPSdigest asked the experts for their opinions on what are the most important advantages of being Agile. Part 3 covers how Agile enables you to grow and adapt to change ...

June 21, 2017

Why Agile? DEVOPSdigest asked the experts for their opinions on what are the most important advantages of being Agile. Part 2 is all about speed ...

June 19, 2017

Earlier this year, DEVOPSdigest featured a list of expert opinions on the essential steps to become agile. Now that we have an idea on how to achieve agility, however, we have to consider why. What's the payoff? With this question in mind, DEVOPSdigest asked the experts — including analysts, consultants and vendors — for their opinions on what are the most important advantages of being Agile ...

June 15, 2017

In the development community, creating additional efficiency through improved collaboration has been prevalent for some time. But despite the head start on the rest of the corporate world when it comes to collaboration, many organizations function today as they did 15-20 years ago. Since time is money in the tech world, outdated collaboration is a huge missed opportunity ...

June 14, 2017

Given the efforts we put in these days to deconstructing monolithic applications, and using distributed microservices to make us more agile, the potential for app performance to take a nosedive because of unseen (and unanticipated) network congestion and outages is only getting greater. There is help at hand, though, in the form of new ways to program network awareness directly into your code ...

June 12, 2017

What if you discover a fatal error or an exploit in your app? What if your app is down during a crucial time? As a developer, how you react to a crisis can mean the difference between minor blip and an embarrassing or costly company blunder. Here's a crisis management plan to get things right when they go wrong ...

June 08, 2017

Recently, the results from SmartBear Software's annual survey, the State of Code Review 2017: Trends & Insights into Dev Collaboration were released. One point I found interesting is that it suggests only 66 percent of organizations can get releases out on time. Why are the other 34 percent struggling to get releases out the door? ...

June 06, 2017

Today's app development landscape is competitive and expensive. Thousands of apps are released each month, and user acquisition and retention are costing app developers millions. User abandonment is one of the main battles of every app developer — as every lost user means another wasted investment ...

June 05, 2017

Developers love using containers to build, run and ship applications in a flexible and simple way. However, the technology has received backlash for not being as secure as other (traditional) methods, such as Virtual Machines (VMs). Securing containers and securing VMs requires a completely different process. Below are four key differences between securing containers versus securing VMs ...

May 31, 2017

DevOps results in improvements in software delivery performance, according to a new first-of-its-kind study by CA Technologies to quantify the benefits for companies that combine DevOps methodologies with Cloud-based tools and delivery mechanisms ...

Share this