A Smarter, Data Science-Driven Approach to Operations
December 04, 2015

Amit Sasturkar
OpsClarity

Data science and machine learning algorithms have become pervasive throughout the modern consumer world. There are many successful applications of machine learning in consumer products that we use on a daily basis, including:

■ Movie, music and product recommendations

■ Ad targeting

■ Web search

But, when we look at the Ops world, we find that there is no breakthrough product that incorporates these same machine learning innovations.

A relevant comparison is with page-level and host-level features (for example, page rank for URLs or host rank for hosts) used in search ranking. These features are typically a function of the WebMap (the massive graph where nodes are URLs, and edges are hyperlinks between URLs). The page rank algorithm allows the ranking of URLs in the WebMap based on the hyperlinks between them. It is a very effective way to get a reasonable estimate of the overall importance of a URL.

What if we used similar ideas to rank the hundreds or sometimes thousands of alerts that operations engineers receive, especially when they are managing hundreds of machines? What is the equivalent of the WebMap in the Ops world?

Another relevant example is provided by duplicate web page detection. These algorithms run as MapReduce jobs on massive Hadoop clusters (thousands of machines) and detect duplicate pages across tens of billions of web pages. When the mappers or reducers fail or when there are performance degradations, hundreds of alerts are generated, many of them for the same underlying root cause.

What if we applied the techniques of web page duplicate detection to eliminate the duplicate and unnecessary alerts received by Ops?

A third big challenge is personalization of content. Personalization is a well-studied problem in the consumer space, with user feedback — both implicit (clicks and actions) and explicit (reviews and ratings) — contributing critical inputs to the learning algorithms. Employing this type of machine learning means that the more time a user spends with a product, the better their user experience will be.

What if we incorporated feedback to learn Ops users’ preferences and continuously improve the accuracy of alert generation and alert ranking?

The answers to these questions will become evident as we bring the innovations in data science and machine learning that are commonplace in the consumer world to the Ops world. DevOps teams need, in effect, an “expert assistant” that can learn their application and system environment, detect and correlate failures, and make recommendations that drive increased focus and productivity — even as everything is continuously changing. It’s time for Ops to get smarter.

Amit Sasturkar is Co-Founder and CTO of OpsClarity.

The Latest

June 25, 2018

The previous chapter in this WhiteHat Security series discussed Codebase as the first step of the Twelve-Factor App and defined a security best practice approach for ensuring a secure source control system. Considering the importance of applying security in a modern DevOps world, this next chapter examines the security component of step two of the Twelve-Factor methodology. Here follows some actionable advice from the WhiteHat Security Addendum Checklist, which developers and ops engineers can follow during the SaaS build and operations stages ...

June 21, 2018

DevSecOps is quickly gaining support and traction, within and beyond information security teams. In fact, 70% of respondents believe their culture can embrace the change needed to fuse Security and DevOps, according to a new survey of 80 security professionals by Aqua Security ...

June 20, 2018

The larger the company size, the higher the proportion of low IT performers, according to the State of DevOps: Market Segmentation Report from Puppet, based on the 2017 State of DevOps Survey data ...

June 18, 2018

An overwhelming 83 percent of respondents have concerns about deploying traditional firewalls in the cloud, according to Firewalls and the Cloud, a survey conducted by Barracuda Networks...

June 14, 2018

Despite the vast majority of cloud management decision-makers believing that DevOps and microservice enablement are important, very few believe that their organizations are capable of delivering them today — a gap that is costing the average enterprise $34 million per year, according to new report from the Ponemon Institute ...

June 12, 2018

Dev teams are doing their best to give the customers what they want, but oftentimes find themselves in between a rock and a hard place. Teams are struggling to get up to speed with new tools that are meant to make their lives easier and more realistic to hit deadlines. With spring cleaning season upon us, take time this season to tune up agile processes and continue the work of advancing the shift towards DevOps ...

June 11, 2018

The ability to create a culture of DevOps is critical to any organization's ability to deliver applications and services at a high rate of speed, but can we clearly and concisely answer the question: What exactly is DevOps? Despite the best intentions, some large companies are struggling to understand what DevOps actually is, and what it takes to fully implement its concepts and reap its benefits ...

June 07, 2018

The Twelve-Factor App is a methodology that offers a 12-step best practice approach for developers to apply when building software-as-a-service apps that are both scalable and maintainable in a DevOps world. As software continues to be written and deployed at a faster rate and in the cloud, development teams are finding there is more room for failure and vulnerabilities. This blog series will discuss how to build a Twelve-Factor app securely ...

June 05, 2018

Everyone understands the importance of code quality for applications, particularly when DevOps results in releases becoming faster and faster, reducing the room for error. The same issues increasingly apply to databases, which are a vital part of DevOps workflows. Fail to integrate the database into DevOps and you'll face bottlenecks that slow down your processes and undermine your efforts ...

June 04, 2018

DevOps and security traditionally have been siloed functions and security is often seen as a policing function by DevOps team members. However, more mature business leaders are trying to bridge the gap between the two functions to achieve business excellence. This theme was evident from our recent survey where 39% of respondents cited that DevOps and development teams care greatly about their cybersecurity posture, showing that the silo between security/IT and development teams is diminishing ...

Share this