A Smarter, Data Science-Driven Approach to Operations
December 04, 2015

Amit Sasturkar
OpsClarity

Data science and machine learning algorithms have become pervasive throughout the modern consumer world. There are many successful applications of machine learning in consumer products that we use on a daily basis, including:

■ Movie, music and product recommendations

■ Ad targeting

■ Web search

But, when we look at the Ops world, we find that there is no breakthrough product that incorporates these same machine learning innovations.

A relevant comparison is with page-level and host-level features (for example, page rank for URLs or host rank for hosts) used in search ranking. These features are typically a function of the WebMap (the massive graph where nodes are URLs, and edges are hyperlinks between URLs). The page rank algorithm allows the ranking of URLs in the WebMap based on the hyperlinks between them. It is a very effective way to get a reasonable estimate of the overall importance of a URL.

What if we used similar ideas to rank the hundreds or sometimes thousands of alerts that operations engineers receive, especially when they are managing hundreds of machines? What is the equivalent of the WebMap in the Ops world?

Another relevant example is provided by duplicate web page detection. These algorithms run as MapReduce jobs on massive Hadoop clusters (thousands of machines) and detect duplicate pages across tens of billions of web pages. When the mappers or reducers fail or when there are performance degradations, hundreds of alerts are generated, many of them for the same underlying root cause.

What if we applied the techniques of web page duplicate detection to eliminate the duplicate and unnecessary alerts received by Ops?

A third big challenge is personalization of content. Personalization is a well-studied problem in the consumer space, with user feedback — both implicit (clicks and actions) and explicit (reviews and ratings) — contributing critical inputs to the learning algorithms. Employing this type of machine learning means that the more time a user spends with a product, the better their user experience will be.

What if we incorporated feedback to learn Ops users’ preferences and continuously improve the accuracy of alert generation and alert ranking?

The answers to these questions will become evident as we bring the innovations in data science and machine learning that are commonplace in the consumer world to the Ops world. DevOps teams need, in effect, an “expert assistant” that can learn their application and system environment, detect and correlate failures, and make recommendations that drive increased focus and productivity — even as everything is continuously changing. It’s time for Ops to get smarter.

Amit Sasturkar is Co-Founder and CTO of OpsClarity.

Share this

Industry News

April 03, 2025

StackGen has partnered with Google Cloud Platform (GCP) to bring its platform to the Google Cloud Marketplace.

April 03, 2025

Tricentis announced its spring release of new cloud capabilities for the company’s AI-powered, model-based test automation solution, Tricentis Tosca.

April 03, 2025

Lucid Software has acquired airfocus, an AI-powered product management and roadmapping platform designed to help teams prioritize and build the right products faster.

April 03, 2025

AutonomyAI announced its launch from stealth with $4 million in pre-seed funding.

April 02, 2025

Kong announced the launch of the latest version of Kong AI Gateway, which introduces new features to provide the AI security and governance guardrails needed to make GenAI and Agentic AI production-ready.

April 02, 2025

Traefik Labs announced significant enhancements to its AI Gateway platform along with new developer tools designed to streamline enterprise AI adoption and API development.

April 02, 2025

Zencoder released its next-generation AI coding and unit testing agents, designed to accelerate software development for professional engineers.

April 02, 2025

Windsurf (formerly Codeium) and Netlify announced a new technology partnership that brings seamless, one-click deployment directly into the developer's integrated development environment (IDE.)

April 02, 2025

Opsera raised $20M in Series B funding.

April 02, 2025

The Cloud Native Computing Foundation® (CNCF®), which builds sustainable ecosystems for cloud native software, is making significant updates to its certification offerings.

April 01, 2025

The Cloud Native Computing Foundation® (CNCF®), which builds sustainable ecosystems for cloud native software, announced the Golden Kubestronaut program, a distinguished recognition for professionals who have demonstrated the highest level of expertise in Kubernetes, cloud native technologies, and Linux administration.

April 01, 2025

Red Hat announced new capabilities and enhancements for Red Hat Developer Hub, Red Hat’s enterprise-grade internal developer portal based on the Backstage project.

April 01, 2025

Platform9 announced that Private Cloud Director Community Edition is generally available.

March 31, 2025

Sonatype expanded support for software development in Rust via the Cargo registry to the entire Sonatype product suite.

March 31, 2025

CloudBolt Software announced its acquisition of StormForge, a provider of machine learning-powered Kubernetes resource optimization.