What Is Log Analysis? (And How to Get Started)

This post’s mission is to answer what seems to be a simple question: “What is log analysis?” It doesn’t seem too hard to answer, right? Obviously from the name log analysis is the process of analyzing log entries. Case closed? Can we finish the post here?

Well, not so fast.

Just saying log analysis means analyzing logs is as obviously true as it is useless. It doesn’t take us long to come up with further questions.

• How is the analysis performed? Manually? Automatically?
• Do you have to do something special to your log files or prepare them in any way before they’re ready for such analysis?
• Why would anyone want to analyze log files in the first place? It sounds like the dullest thing ever. Are there benefits?

This post will answer the questions above and more. By the end of it, you’ll understand what log analysis is. You’ll also know why it’s important, what its benefits are, and how to get started.

Log Analysis: a Quick Definition

In the simplest possible terms, log analysis is the process of extracting knowledge from your log entries. That’s pretty much it, but let’s qualify this definition a bit more.

One important aspect of real log analysis is it must be automated. It’s common for even small to mid-sized organizations to produce gigabytes worth of log files every day. There’s simply no way for a human to process that amount of data, let alone analyze it in ways that lead to valuable and useful insights.

Why Would You Want to Analyze Your Logs?

In the previous section, I gave you a simple definition for log analysis. Don’t let that fool you, though: simple doesn’t always mean easy, and it doesn’t in this case. As you’ll soon see, log analysis is complex and involves a lot of components. Before we get there, though, let’s quickly discuss the motivations behind log analysis. Why should you and your team care about log analysis? What’s in it for you?

Here’s the thing: many organizations incur the error I like to call “cargo-cult logging.” In other words, they log “just because.” Sure, they leverage log entries as a troubleshooting mechanism, and a valuable one at that. But then they stop there. They have all this data sitting around and they don’t know what to do with it. That’s a sad example of waste. But what potential are they wasting, exactly?

In any organization, all sorts of processes generate logs. Operating systems, web servers, applications, and databases are just a few. Just by aggregating log entries from disparate sources into a centralized location, you already gain a lot of value. Since logs are everywhere, they offer a unique opportunity to observe all facets of an operation.

Through the examination of log entries, you can get a glimpse of every layer of an IT infrastructure. You can correlate events from disparate sources, discovering relationships that would’ve been otherwise undisclosed. For instance, by correlating data from web server logs and your e-commerce application logs, you could demonstrate poor performance correlates negatively with conversion rates.

By performing log analysis, you’re able to use your logs proactively, not reactively. You’re able to turn your logging strategy from a mere troubleshooting aid to a valuable tool in the decision-making process.

What Are the Components Involved in Log Analysis?

Do you remember the questions from the start of the post? Here they are again:

• How is the analysis performed? Manually? Automatically?
• Do you have to do something special to your log files or prepare them in any way before they’re ready for such analysis?
• Why would anyone want to analyze log files in the first place? It sounds like the dullest thing ever. Are there benefits?

As it turns out, we’ve already answered questions one and three! What about the second one? Do we need to prepare our logs in some way before we can analyze them? The answer is yes. We’ll now cover this in more detail, talking about the steps involved in the log analysis process.

Bringing It All Together

Before we can do any type of log analysis, we need to have…well, logs. As we’ve just seen, only when you have all of the logs generated by various processes across your organization centralized into a single location can you start reaping the benefits made possible by log analysis.

This “bringing it all together” process is often called log aggregation or log centralization. At the end of the day, it’s all the same: bringing all your log events into a single, centralized location where they can all be understood in context.

Making It All Understandable

Just collecting all your logs into a single location isn’t enough. Why? Well, sadly, logs from disparate sources will most likely have different ways of expressing information.

For instance, they might not share the same date formats. Speaking of which, you might also have to deal with different time zones. In that case, you’ll thank the people who designed the logging systems if they had the foresight to log using UTC or local time plus UTC offset.

The different logs also might not agree on the names for logging levels—one log’s “informational” might be another log’s “warning.” Some logs might be in plain text, while others leverage structured logging. You get the picture: logs from different sources might employ radically different ways of expressing and encoding information.

So, an essential step in the log analysis process is to iron out those differences. This involves not only data cleaning to fix eventual mistakes or absences, but also the normalization of different formats and parsing and interpreting the data into a neutral, universal format that can then be analyzed.

Making It All Searchable

Having a lot of useful information and not being able to find what you’re looking for is more than useless. It’s maddening, as thought experiments and fiction stories have shown.

That’s why searching is such an essential capability in the log analysis process. Not only does your log analysis tool of choice need to offer powerful and fast searching capabilities, but the log data itself also needs to be in a condition where it can easily be searched. This includes, among other things, being indexed just like database tables are.

Using a Log Analysis Tool

An essential component of a log analysis strategy is picking a log analysis tool. As we discussed earlier, log analysis is mostly an automatic process, since there’s no way a human being can handle the sheer amount of log data most companies routinely generate.

With a comprehensive log analysis and log management solution like SolarWinds^® Loggly^®, you can aggregate logs from different sources. You can then quickly search the vast amount of log data imported, allowing you not only to investigate and fix problems but also to identify trends that can help you stop issues before they become critical.

Log Analysis: Evolve Your Logging Approach

Most organizations leverage logging as a mere aid in troubleshooting. Though this capability is already very useful, your logs can help you in much more profound ways.

To unlock those capabilities, you need to understand and leverage log analysis. Only then will you be able to take your logging approach to the next level, transforming it into a real force for change inside your organization.

This post was written by Carlos Schults. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.

The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.