Log Analytics is DEAD
December 03, 2015

Albert Mavashev

Log Analytics is DEAD. Did I really say that?? Yes I did. Log Analytics is a process of investigating logs and hoping to derive actionable information that might be useful to the business. Many log analytics tools are used to gain visibility into web traffic, security, application behavior, etc. But how valuable and practical is log analytics in reality?

One basic precondition for log analytics is that information to be delved into must be in log files and here lies the basic problem:

In order to derive useful analytics from logs one must have proper logging instrumentation and have it enabled everywhere, all the time.

Not only is this approach impractical and very expensive, except in a few limited cases, but it is also burdensome, imposing a significant performance overhead on the systems that produce these logs.

One must log gigabytes and gigabytes of data, store this data and then analyze it in order to detect a problem. I would call this a brute force approach. As most brute force approaches, it is expensive, slow and unwieldy. In many cases log analytics is used to catch occasional errors or exceptions. Do we really need to have all these logs to catch a few outliers?

Log analytics quickly turns into a Big Data problem – store and analyze everything, everywhere, all the time. Is that really needed? Maybe, or maybe not …

Simple Example

You deploy log analytics and it tells you've got 100 errors or exceptions in the past hour. Typically, you will want to investigate this and start with a specific exception.

Your next question would be “is what am I looking and noise or something that requires attention?” Then you will ask “what else happened” and “why?”. There is a series of questions you would ask might include the following:

■ What was my application doing?

■ What was the response time?

■ What was CPU, memory utilization?

■ What were the I/O rates and network utilization?

■ What was Java GC doing?

■ What other abnormal conditions occurred that I should be looking at?

There are so many variables. There are too many to look at and too much to analyze.

What do you do? Unfortunately this is where log analytics stops, you have to jump elsewhere. The path to root-cause becomes lengthy and painful. You may know that there is a problem, but why you have a problem in many cases is not clear.

We have all this data (big data) yet I don’t know what it means or where to look to find meaning. Of course one can say that you can parse out the log entries and extract metrics. Who will write the parsers? Who maintains the rules? Who writes complex regular expressions? What if the required metrics are not in the log files? In most cases they won’t be.

The biggest problem with log analytics is that what can be analyzed must be always logged. You need to know what information you need for root cause in advance. How often do you know what you need in advance? It is what you don’t know, have not thought about, did not instrument, did not log. It is unlikely you will have the information you will need.

Customers don’t want log analytics; customers want solutions to their problems. So what do I propose? I think log analytics is really morphing into a larger discipline.

The Post Log Analytics World

It is Application Analytics that combines logs, metrics, transactions, topology, changes, and more, along with machine learning techniques: where asking about quality of service, application performance, business and IT KPIs is a click away.

This approach must be combined with smart instrumentation, heuristics and even crowd-sourced knowledge that points to anomalies, suppresses noise and reveals important attributes without constantly collecting terabytes of data.

How do I understand what I don’t know or have not collected yet? How do I know what questions to ask?

Essentially Application Analytics is about managing risks lurking within application and IT infrastructures which are inherently complex and “broken”.

Log Analytics is dead, not because is not useful, but because it must quickly evolve into the next level.

Albert Mavashev is Chief Technology Officer at jKool.

The Latest

August 13, 2018

Agile is expanding within the enterprise. Agile adoption is growing within organizations, both more broadly and deeply, according to the 12th annual State of Agile report from CollabNet VersionOne. A higher percentage of respondents this year report that "all or almost all" of their teams are agile, and that agile principles and practices are being adopted at higher levels in the organization ...

August 09, 2018

For the past 13 years, the Ponemon Institute has examined the cost associated with data breaches of less than 100,000 records, finding that the costs have steadily risen over the course of the study. The average cost of a data breach was $3.86 million in the 2018 study, compared to $3.50 million in 2014 – representing nearly 10 percent net increase over the past 5 years of the study ...

August 08, 2018

Hidden costs in data breaches – such as lost business, negative impact on reputation and employee time spent on recovery – are difficult and expensive to manage, according to the 2018 Cost of a Data Breach Study, sponsored by IBM Security and conducted by Ponemon Institute. The study found that the average cost of a data breach globally is $3.86 million ...

August 06, 2018

The previous chapter in this WhiteHat Security series discussed dependencies as the second step of the Twelve-Factor App. This next chapter examines the security component of step three of the Twelve-Factor methodology — storing configurations within the environment.

August 02, 2018

Results from new Forrester Consulting research reveal the 20 most important Agile and DevOps quality metrics that separate DevOps/Agile experts from their less advanced peers ...

July 31, 2018

Even organizations that understand the importance of cybersecurity in theory often stumble when it comes to marrying security initiatives with their development and operations processes. Most businesses agree that everyone should be responsible for security, but this principle is not being upheld on a day-to-day basis in many organizations. That’s bad news for everyone. Here are some best practices for implementing SecOps ...

July 30, 2018

While the technologies, processes, and cultural shifts of DevOps have improved the ability of software teams to deliver reliable work rapidly and effectively, security has not been a focal point in the transformation of cloud IT infrastructure. SecOps is a methodology that seeks to address this by operationalizing and hardening security throughout the software lifecycle ...

July 26, 2018

Organizations are shifting away from traditional, monolithic architectures, with three-quarters of survey respondents delivering at least some of their applications and more than one-third delivering most of their applications as microservices, according to the State of DevOps Observability Report from Scalyr ...

July 24, 2018

What top considerations must companies make to ensure – or at least help improve – Agile at scale? The following are key techniques and practices to help accelerate Agile delivery rollouts and scale Agile and DevOps in the Enterprise ...

July 23, 2018

Digital transformation is an important part of most corporate agendas for 2018. Successful digital transformation, encompassing your current business, partners and both current and prospective customers, isn't always easy. However, adopting an enterprise-wide Agile methodology can help ease the burden and deliver discernible ROI much faster ...

Share this