Crisis Management: What Should You Do When Things Go Wrong?
June 12, 2017

Joyce Lin
Postman

What if you discover a fatal error or an exploit in your app? What if your app is down during a crucial time? As a developer, how you react to a crisis can mean the difference between minor blip and an embarrassing or costly company blunder.

Your organization is depending on you to save the day. Here's a crisis management plan to get things right when they go wrong:

1. Stay calm and think clearly

This is obvious, but easier said than done. Staying calm will allow you to think more clearly. Working long hours under stress frequently leads to subpar code, and may be the reason why your app is down in the first place.

Don't underestimate the value of taking a walk, grabbing a snack or something else that changes your outlook to see the issue in a different light. And don't worry about wasting time, your brain will still be working on the issue in the background.

2. Revert to working code

If the appropriate solution might take a while to implement, or you have no idea how long the fix will take, roll back to a previous version of the code as a temporary measure. This is the last time you know your service was working, and it's a stable build.

Reverting to working code can provide the extra time you need to thoughtfully address the issue. When you're no longer rushed, you can gather the information you need to more effectively resolve the problem.

3. Monitor and alert stakeholders

You're a responsible developer, so hopefully you've set up some monitoring and alerting for your app. Before you ever find yourself in crisis mode, make sure you set up the right triggers and intervals for monitoring.

Once you receive the initial alert of a failure, notify the appropriate engineers of the outage so they can get started on a fix right away. Also notify other internal stakeholders so they're aware that their services might be impacted. As an alternative to notifications, you can set up a status page to inform those who rely on your app of updates on performance and availability.

4. Debug the issue

Now that you're set up for success, it's time to dig into the issue – starting with your logs. Logs are only as helpful as you make them. You should be logging the right activities with descriptive log statements. In addition to your existing log statements, add temporary log statements to guide the debugging process.

If you're still uncertain about the cause of the outage, focus on isolating the issue. Some code bases are structured in a way that is easy to see where the code is failing, but some dependencies and abstractions make it tricky to pin down the root culprit. If you're at a loss, you can try the debugging variation of a binary search by dividing and conquering to pin down where the code is failing in the most efficient manner.

Finally, don't work in a silo. You might benefit from talking through the problem or getting another perspective. Some developers like to pair program, rely on another teammate for rubber duck debugging, or use a literal rubber duck to slow down and articulate code line by line.

5. Push fixes with continuous deployment

When you're making code changes under pressure, you might be hacking together a solution. As a result, you might also be cutting corners and incurring technical debt in exchange for a quick turnaround.

Running an automated test suite guarantees consistent code coverage, and rigorous regression testing hedges against unintentionally affecting another dependency. Automating the build, testing and deployment process ensures you deliver patches in the fastest and most efficient manner possible.

6. Communicate changes to the team

Once you've patched the fix, communicate the status and diffs to the team. Just because your world came to a standstill with the problem doesn't mean the same happened for the rest of your team. They are continuing to work on their own features and issues.

Communicate what is necessary to keep them up to speed on the changes, and how it might impact what they're working on. Some teams keep track of an activity feed to stay up-to-date with the latest code changes, while others prefer to set up an integration with their preferred messaging platforms.

Joyce Lin is a Developer Evangelist for Postman
Share this

Industry News

February 18, 2025

Check Point® Software Technologies Ltd.(link is external) announced that its Check Point CloudGuard solution has been recognized as a Leader across three key GigaOm Radar reports: Application & API Security, Cloud Network Security, and Cloud Workload Security.

February 13, 2025

LaunchDarkly announced the private preview of Warehouse Native Experimentation, its Snowflake Native App, to offer Data Warehouse Native Experimentation.

February 13, 2025

SingleStore announced the launch of SingleStore Flow, a no-code solution designed to greatly simplify data migration and Change Data Capture (CDC).

February 13, 2025

ActiveState launched its Vulnerability Management as a Service (VMaas) offering to help organizations manage open source and accelerate secure software delivery.

February 12, 2025

Genkit for Node.js is now at version 1.0 and ready for production use.

February 12, 2025

JFrog signed a strategic collaboration agreement (SCA) with Amazon Web Services (AWS).

February 12, 2025

mabl launched of two new innovations, mabl Tools for Playwright and mabl GenAI Test Creation, expanding testing capabilities beyond the bounds of traditional QA teams.

February 11, 2025

Check Point® Software Technologies Ltd.(link is external) announced a strategic partnership with leading cloud security provider Wiz to address the growing challenges enterprises face securing hybrid cloud environments.

February 11, 2025

Jitterbit announced its latest AI-infused capabilities within the Harmony platform, advancing AI from low-code development to natural language processing (NLP).

February 11, 2025

Rancher Government Solutions (RGS) and Sequoia Holdings announced a strategic partnership to enhance software supply chain security, classified workload deployments, and Kubernetes management for the Department of Defense (DOD), Intelligence Community (IC), and federal civilian agencies.

February 10, 2025

Harness and Traceable have entered into a definitive merger agreement, creating an advanced AI-native DevSecOps platform.

February 10, 2025

Endor Labs announced a partnership with GitHub that makes it easier than ever for application security teams and developers to accurately identify and remediate the most serious security vulnerabilities—all without leaving GitHub.

February 06, 2025

GitHub announced a wave of new features and enhancements to GitHub Copilot to streamline coding tasks based on an organization’s specific ways of working.

February 06, 2025

Mirantis launched k0rdent, an open-source Distributed Container Management Environment (DCME) that provides a single control point for cloud native applications – on-premises, on public clouds, at the edge – on any infrastructure, anywhere.

February 06, 2025

Hitachi Vantara announced a new co-engineered solution with Cisco designed for Red Hat OpenShift, a hybrid cloud application platform powered by Kubernetes.