Unstructured Log Data Usually Leads to Constrained Log Analysis

It’s no secret that the more structured and detailed your log data is, the less stressful your troubleshooting will be during times of need. However, a significant amount of your log data is likely not organized into separate data elements and, even worse, was likely created without the expectation of others relying on it for critical troubleshooting. As such, it can be really hard for everybody but the original developer to decipher what specific logs mean, and it’s rare for one person to understand all the logs. This can make getting insight from your logs quite painful. It’s why Loggly invests so heavily in automatically detecting and applying structure to the most common types of logs to allow for focused filtering and advanced analysis.

Even with our unique approach of automatically applying structure to logs, we find that more than one-third of customer logs are not structured. Many times, the logs are coming from a legacy or third-party app over which you have no control or where refactoring the app would just be too much effort given your other priorities. Even so, you probably wish there were some way to make unstructured logs more useful and less painful during your next analysis.

At Loggly, we heard our customers talking about this very problem and have been hard at work solving it. However, creating parsing rules to allow a one-time “field extraction” didn’t seem like enough since there are plenty of times when different people would like to digest the same log in different ways.

Wouldn’t it be better if you had a way to add structure but keep the original event for others to create their own interpretations as well? Inject intelligence or context that makes it easier for others to analyze?

Derived Fields Bring the Full Capabilities of Loggly to Unstructured Logs

I’m really excited about derived fields, our newest functionality announced this morning and available in beta to our Pro and Enterprise customers. Derived fields enable users to specify custom parsing rules to inject intelligence into or fully structure any log or selected log parts during the ingestion process. Loggly adds derived fields to your original log events as metadata and automatically catalogs them through Loggly Dynamic Field Explorer^TM, providing access to one-click analysis.

Analyzing Unstructured Logs Used to Be a Lot of Work…

The traditional approach to analyzing unstructured logs is to create one-off regular expressions during search to perform custom or advanced analyses. Not only does this mean you have to wait for that custom parsing to complete (which could take minutes or hours depending on how much data you are parsing at once) but also that you can’t “save” the results of the parsing for your next query (thus even more waiting). You waste a lot of time and aren’t in a position to use your log data proactively—to solve problems as they’re emerging.

… But Derived Fields Are Different

Derived fields are another way that Loggly does the work of crunching through huge volumes of log data during ingestion and turning it into meaningful information and insights for interactive analysis. With derived fields you get:

• Ingestion-time processing: Derived fields are created at log ingestion time, not upon search. When users log into Loggly, they have immediate access to insights because the data has already been parsed. With many log management solutions, parsing rules run every time users generate a search and may take minutes or sometimes even hours to produce results.
• Automatic summarization, not one-time parsing: Once you create your derived fields, Loggly automatically applies those rules to all log data it receives. There’s no longer any need to re-create complex regular expressions for each query. When you have a question, you use Field Explorer to navigate by field names and values and to create sophisticated analyses without typing any commands.
• Navigable summaries as your entry point: While most traditional log management solutions rely on the search box as the primary interface, Loggly offers Field Explorer, which generates navigable log summaries as the entry point. With derived fields, you can use Field Explorer with any type of log data.
• Multiple overlapping definitions: In addition, logs mean different things to different people. Other log management solutions that parse data make permanent alterations (for example, splitting one field into multiple fields). Because Loggly derived fields are metadata attached to your log events, the original event data stays intact. You can create as many derived fields as you like from the same log events, delivering unique context that is all exposed to the user through Field Explorer.

Interactive Feedback Helps You Create Your Parsing Rules

Derived Fields give you the full power of regex. (BTW, keep your eye out for a post later this week from Liz Bennett; it will help you understand the differences between good regexes and bad ones.) But at the same time, our interface provides some dynamic help for people who aren’t regex power users:

• As you create your regex, you can see if your rule is a valid regular expression and whether it will actually match the log events you expect it to.
• After you complete the rule, you can see the output that the rule will actually produce once it’s enabled in production. You’ll be able to see what your log events will look like and which values will be visible to all of your users through Field Explorer. This is another way to make the call if your rule is the right one.

A Step Forward in Revealing What Matters

At Loggly, we’re taking a new approach to log analysis. As I have discussed in previous blog posts, we’re not trying to build a better metal detector (in the form of more complex tools and commands you need to learn) but rather to give you a treasure map that reveals what matters in your logs. Derived fields are another step in this “summarize first” strategy—a step that we know will help your team work more effectively no matter what types of log data you have.

The Key Benefits of Derived Fields

• Identify and resolve issues faster because it is much easier to spot the data that matters.
• Spend less time on training, less time with reference guides, and more time on understanding your data.
• Generate advanced analytics even with legacy applications that send out unstructured, text-based logs that you cannot update or have no plans to do so.
• Extend advanced log analysis to more members of your team.

See Derived Fields for Yourself

View my recorded webinar where I demonstrate these exciting new capabilities.

Get Derived Fields Now!

If you haven’t yet experienced Loggly, sign up for a 14-day free trial to see what it’s like to inject intelligence into your logs. Our free trials include full access to the Loggly Pro plan feature set.

The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.