Why JSON is the best application log format … and how to switch
Introduction
In recent years the JSON format has emerged as the de facto standard for message passing. It is both readable and reasonably compact, and it provides a standardized format for structuring data. Almost every programming language can parse it, so it’s simple to implement in languages which have not yet implemented built-in JSON functionality. JSON’s Unicode encoding makes it universally accessible, and its large and established user base provides an active community of helpful examples, patterns, and support.
In this blog post, we will examine the process of converting your log format into the JSON format by giving several examples in popular languages.
The biggest benefit of logging in JSON is that it’s a structured data format. This makes it possible for you to analyze your logs like big data. It’s not just readable text, but a database that can be queried. This allows summaries and analytics to take place and help you monitor your application and troubleshoot issues faster.
Audience and Environment
We’ll assume your app is currently logging in a non-JSON format. Because JSON is a universal format, it won’t matter if you’re on a Mac or PC, what language you’re using, what server you’re running, or what editing environment you prefer. We’ll give examples with popular environments such as Node.js (JavaScript), Sinatra (Ruby), Java (Log4j), and Laravel (PHP5), and look at Apache and Nginx if you need flexible parsing for your web server logs as well.
How to Switch Your App to Log in the JSON Format
In this section we will show you how to configure several common applications to log in JSON format.
JavaScript: Configuring Your Node.js App to Log in JSON with Winston
Node.js has brought JavaScript to the forefront of web development and proven its worth as more than just a front-end browser language. As JSON is a subset of JavaScript itself, using it within your Node.js app is a no-brainer. Using a logging library such as Winston makes it as simple as a configuration option. Here’s some example code to get you started:
example_file.js:
For further reading on using Winston and other Node.js logging tools with JSON logs and Loggly’s service, as well as more example code and output, check out these articles from our Ultimate Guide to Logging:
Ruby: Configuring Your Sinatra App to Log in the JSON Format
Sinatra is a framework which has inspired numerous other frameworks in many languages, including the Express JavaScript framework which forms the “E” in the MEAN stack (MongoDB/Express/AngularJS/Node.js). It is simple, clean, elegant, and used by major industry players such as Apple, LinkedIn, GitHub, Heroku, and the NSA (source: sinatrarb.com/wild.html). We can use the Logglier gem to log JSON in our Sinatra app: Example file: sinatra.rb (runs with `ruby ./sinatra.rb` with ruby 2.1.1 and the required gems installed globally)
Hitting the /test2 route on your localhost when running Sinatra will result in the following JSON message being sent to Loggly:
This example assumes you have the logglier gem installed globally on your system. Read more about configuration on our site.
Java: Configuring Your App to Log in the JSON Format with Log4j
Using Log4j, we can configure JSON output as described here: https://logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout. JSON can also be used for the configuration file instead of the default XML format, if preferred: https://logging.apache.org/log4j/2.x/manual/configuration.html#JSON. (Note that you’ll need to install the Jackson Data Processor to parse the JSON files if you use JSON for your config instead of the default XML.) The Log4j example file:
The Log4j configuration file (as XML):
For further information on logging with JSON and Loggly in Java, check out this article from our Ultimate Guide to Logging: https://www-staging.loggly.com/ultimate-guide/java-logging-basics/#json. (And thank you to Andre Newman for contributing the Java code and config!)
PHP: Configuring Your Laravel App to Log in the JSON Format
Laravel has gained popularity in the LAMP world in recent years and is becoming the framework of choice for many teams. Here is an article on logging JSON logs from Laravel to Loggly from 2014.
Web Server Logs
The default log format for Apache and Nginx logs are more compact than logs in JSON, but if you need the parsing flexibility afforded by logging in the JSON format for your web server logs take a look at these examples:
Switching Nginx to Log in the JSON Format
Take a look at configuring Nginx to output the JSON format.
Making Sure Your JSON is Valid
It’s a good idea to use a tool such as https://github.com/zaach/jsonlint to check your JSON data. You can test the output of your new logging format and make sure it’s real-and-proper JSON.
What To Do With The Logs
Now that the logs are in JSON format, we can do powerful things with them. Many languages allow the creation of objects from JSON, and popular NoSQL databases store the format natively, but the most relevant application for our purposes may be aggregating our JSON data in a service that gives us powerful reporting, searching, and insights to our data. You can sign up for a free account on Loggly.com and send your JSON logs (among other formats) to the servers to observe trends about your data and receive alerts when certain thresholds are hit.
Don’t forget to look at examples outside of your chosen language if you get stuck. You can often convert bits of existing code from another language and convert them into your own custom solution.
Navigating JSON Logs with Loggly Dynamic Field Explorer
Once you send your logs to us in JSON format, Loggly will automatically parse your JSON and visualize its structure in a tree-like format. This makes it much easier to see what’s in your logs, figure out where to look for a problem, or spot anomalies.
Conclusion
While tabs on a single line provide the most compact logging format, JSON provides the most flexible one. It is relatively compact (compared with, say, XML). Most importantly, it allows for very flexible machine-first parsing.
Happy logging!
Looking to switch to JSON, sign-up for a free Loggly Account, no credit card required.
The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.
David Liedle
