10 Ways to Make Your Outage Emergency Room Fun
November 03, 2017

Andrew Marshall
Cedexis

It's 3:47am. You and the rest of the Ops team have been summoned from your peaceful slumber to mitigate an application delivery outage. As you catch up on the frantic emails, Slack chats, and text messages from everyone on your international sales team (Every. Single. One.), your mind races as you switch to problem solving mode. It's time to start thinking about how to make this mitigation FUN!

1. Gently remind everyone that the SaaS-delivered automated Application Delivery strategy you proposed would have prevented this

No need to rub it in, but … if you had turned on a software-defined application delivery platform for your hybrid infrastructure, you'd all be sound asleep right now. Automated real-time delivery decisions and failover would be nice, right? Just sayin'.

2. Drop a generationally-divisive Giphy in the emergency Slack channel

Your coworkers' opinions on gaming consoles, Spiderman movies, and music are different than yours! The perfect animated GIF will remind them of your erudite tastes, while you have their attention. Extra credit if you can work in some low-key shade about not listening to your equally sophisticated opinions on optimized outage mitigation.

3. Hold an impromptu Cloud Health Dashboard page customization contest

Oh hey, look at that! Your cloud provider's health dashboard page says everything is fine…because it's powered by the services that went down. Help your team vent their creative energy (and frustration) with some fun customized MS Paint updates to the offending page. Bonus points for art that reminds everyone of the value of a multi-cloud strategy powered by a programmable application delivery platform.

4. Use health metrics to cure ills instead of just conducting another really good post-mortem

Tired of “learning lessons” from these emergency room drills? You depend on NGINX for your LLB, but don't have a way to use those LLB health metrics and data to automate global delivery. Disjointed delivery intelligence means you don't know how your apps will land with users. You need an end user-centric approach to app delivery that automates the best delivery path and ensures failover is in place at all times. Micro-outages often fly under your passive monitoring radar, but that doesn't mean your users don't notice them. An active, integrated app delivery approach re-routes automatically before you lose business. Post-mortems are fun…but so is making sure your apps survive the last mile. Arguably more so?

5. Open (and then close) some tickets for the sales team

“Sales needs to sell more stuff.” You'll feel better.

6. Build a countdown clock for when you can finally replace your old ADC hardware

Sure, your Mode1 ADC hardware was a sunk cost so you're stuck with them for a while. But you're one unnecessary emergency closer to having a fully software-defined application delivery platform for your hybrid cloud. And now you're even closer. And closer … Tick. Tock.

7. Leverage real user measurements (RUM) to foresee and fix problems before they happen

Probably best to do this during normal work hours. User experience data from around the world can detect degrading, sluggish resources in real-time, and user-centric app delivery logic powered by RUM can make quick re-routing decisions automatically. No more getting woken up after the application crashes. While you're wishing you had RUM on your side, you can look up some fun facts from the countries experiencing app outages. Did you know Luxembourgish is an official language?

8. Send an email

You too can be the weird colleague who sends emails with crazy, middle-of-the-night time stamps.

9. Do some virtual window shopping

Browse around to see what you could have purchased with the money you were just forced to spend on unplanned cloud instance provisioning in order to keep your app running. That desktop Nerf missile launcher (or 700 of them) would have been pretty nice.

10. Swear off the game of chance

You've just proven it's not that much fun after all. Don't just dump everything onto one cloud and call it done. Clouds go down for so many reasons. Use an application delivery platform control layer to build in the capability to auto-switch to an available resource, while you sleep soundly. Running on multi-cloud without an abstracted control layer removes most of the value of the cloud. Swear off the game of chance. Out loud. Right now.

Andrew Marshall is Director of Product Marketing at Cedexis

The Latest

August 15, 2018

Microservices are a hot topic in IT circles these days. The idea of a modular approach to system building – where you have numerous, smaller software services that talk to each other instead of monolithic components – has many benefits ...

August 13, 2018

Agile is expanding within the enterprise. Agile adoption is growing within organizations, both more broadly and deeply, according to the 12th annual State of Agile report from CollabNet VersionOne. A higher percentage of respondents this year report that "all or almost all" of their teams are agile, and that agile principles and practices are being adopted at higher levels in the organization ...

August 09, 2018

For the past 13 years, the Ponemon Institute has examined the cost associated with data breaches of less than 100,000 records, finding that the costs have steadily risen over the course of the study. The average cost of a data breach was $3.86 million in the 2018 study, compared to $3.50 million in 2014 – representing nearly 10 percent net increase over the past 5 years of the study ...

August 08, 2018

Hidden costs in data breaches – such as lost business, negative impact on reputation and employee time spent on recovery – are difficult and expensive to manage, according to the 2018 Cost of a Data Breach Study, sponsored by IBM Security and conducted by Ponemon Institute. The study found that the average cost of a data breach globally is $3.86 million ...

August 06, 2018

The previous chapter in this WhiteHat Security series discussed dependencies as the second step of the Twelve-Factor App. This next chapter examines the security component of step three of the Twelve-Factor methodology — storing configurations within the environment.

August 02, 2018

Results from new Forrester Consulting research reveal the 20 most important Agile and DevOps quality metrics that separate DevOps/Agile experts from their less advanced peers ...

July 31, 2018

Even organizations that understand the importance of cybersecurity in theory often stumble when it comes to marrying security initiatives with their development and operations processes. Most businesses agree that everyone should be responsible for security, but this principle is not being upheld on a day-to-day basis in many organizations. That’s bad news for everyone. Here are some best practices for implementing SecOps ...

July 30, 2018

While the technologies, processes, and cultural shifts of DevOps have improved the ability of software teams to deliver reliable work rapidly and effectively, security has not been a focal point in the transformation of cloud IT infrastructure. SecOps is a methodology that seeks to address this by operationalizing and hardening security throughout the software lifecycle ...

July 26, 2018

Organizations are shifting away from traditional, monolithic architectures, with three-quarters of survey respondents delivering at least some of their applications and more than one-third delivering most of their applications as microservices, according to the State of DevOps Observability Report from Scalyr ...

July 24, 2018

What top considerations must companies make to ensure – or at least help improve – Agile at scale? The following are key techniques and practices to help accelerate Agile delivery rollouts and scale Agile and DevOps in the Enterprise ...

Share this