How to write effective logs for remote logging
In my last blog post, I listed many of the reasons that you should be using log management. In this post I will explain methods to log effectively. Logging effectively is not a precursor to using remote logging. Having logs is always better than not having logs. However, building log outputs with careful consideration will greatly improve your ability to gather information from them.
Debunking the Biggest Remote Logging Myth
The most common argument I hear when I propose the use of remote logging on a project is that a lot of logs will be produced, bandwidth will be used, and that this bandwidth is significant. But if you look more closely, the bandwidth is not significant. When you’re trying to read a continuous textual stream of logs, you might think that it seems like a lot of data. However, our Internet is packed full of streaming audio and video that dwarf the text streams generated by remote logging.
Remote Logging Bandwidth Analysis
Even though this is one of the most common points of resistance, I will try to keep my response brief. The following table compares data use of a few applications. Look at the table and identify what kind of network availability your platform will have. I’m no expert here, but I’d generally classify them as: Satellite, 3G/4G connections (together due to bandwidth caps), and ground connection. All bandwidth usage is a function of two variables: (1) How often does the app log (2) How large is the average log string.
B = f * (s*8) where f is the frequency of log sends per second and s is the size of the average log statement in bytes.
Logging:
Bandwidth used by a UI application: 0.2 * 30, B = 48bps
Bandwidth used by average server: 5 * 40, B = 1.6Kbps
Bandwidth used by full logcat: 25 * 95, B = 19Kbps
Bandwidth used by an IoT device: 1 * 30, B = 24bps
(Source: Mark-1 Eyeball)
Streaming:
Bandwidth used by sloppy audio stream: 32Kbps
Bandwidth used by quality audio stream: 96kbps
Bandwidth used by 720p video: 2Mbps
Bandwidth used by 1080p video: 4.5Mbps
(Source: Video Bandwidth)
I usually tell clients that the free Loggly tier will support up to 60 mobile applications in beta development. Logging 720p video content on the free Loggly tier will work for 13 ⅓ minutes per day before the cap is reached. The use of bandwidth is not significant. So I recommend that they log always, whenever possible.
How to Use Logs for Debugging
The purpose of your remote log is to tell a story. Because of Loggly search capabilities, each device is treated independently. You can browse a log that details the entire life span of a particular piece of equipment from boot up to shut down. If you’re writing an application that mostly sits still, let your logs sit still. Loggly doesn’t mind if your application is silent.
The purpose of a remote log is to present your application’s state. In order to write logs that convey this state effectively, consider building logs according to the following rules. These rules are by no means exhaustive.
Log Structure
Log everything as JSON. Include global state parameters in _every_ log message that’s your global state. With it you can do forensics for any event.
Insert Objects into Their Own Object Name Dict
This can be unintuitive, especially if you’re not writing Javascript, but it applies to all languages equally. You’re already uploading JSON to Loggly, right? So, in any language, write your logs like this:
Why do this? Now you can index your logs based on every reported crucialData item. Have an error in crucialData? Search your logs for the error, then show only the crucialDatas that were on or near that error.
Be Descriptive AND Be Verbose
Don’t abbreviate. Ever. The only code appropriate for abbreviation is mathematical notation. The abbreviation that is obvious now is only obvious because of your current state of mind and personal context. Write out full variable names, and sometimes what they mean, where they are, and what they are doing, and log them. The benefit of brief messaging in bandwidth usage pales compared to the painful consequence of processing the information. Remember: machines are basically free. Human thought is very expensive.
Log Deeply inside Failure-Likely Cases
Complicated code needs complicated logging—this is the only place you’re going to be able to debug it. And it will have bugs.
Log Sequence
Log transitions in sequence diagrams and flowcharts. If your app is moving from one state to another, create a message that notifies that the transition has occurred. 
State transitions are almost always good logging material
(Source: Paciello Group)
Don’t Log Nested Noops
Log constructively. Don’t write log statements for repetitive data, or uninformative data. Sometimes it is essential to log out a loop’s behavior. Unless a particular loop has special importance (or more likely a specially difficult to reproduce bug), only log the entrance and the result of the loop.
Consider:
Versus:
Don’t log in loops. Only log change.
(Source: Instructables)
Don’t Log Coordinate Systems (Except during Development)
Coordinates are messy and basically never informative. If you need to know the coordinates of an object or event, it isn’t because of the coordinates themselves, but only the event that depends on them. Log coordinates of an object or event that receives attention, that enters a state, or receives a message.
Log State Judiciously
Developers often log the global state of their application. A trivial example of an IoT device is a thermometer. For this device, this might be the device’s IP address, mac address, current temperature, current voltage, and current connections. This kind of logging is valuable, but remember that log statements like this are redundant. Don’t try to understand app state by reading changes in the state logs. Log events instead. Events demonstrate state transitions historically and temporally. Attach state to the event messages so that you can compare events that occur in different states.
Log Configuration Extensively
System settings, app versions, and hardware/firmware/driver versions are all absolutely critical to effective debugging. In general, I recommend attaching every configuration setting and version to every log statement. You won’t regret it. Now your logs can be compared across different configurations and platforms. This enables you to identify platform-specific bugs. You can also manage updates more effectively.
Summary
Building quality logging is about evaluating each log statement’s frequency versus the reward of making that information available times the risk of not making it available. Each time you write a method, decide if logging the entry of that message is useful and adequately infrequent. Sometimes you need to produce a log statement on every line of code if the risk of a problem occurring in that code, or the consequences of not finding an issue in the code, outweigh the frequency that the event occurs at all.
The purpose of your logs, just like the purpose of your code, should be to tell a story to another developer. Each event is a sentence in the story. With all of your running instances telling stories to you, you can much more easily determine what went wrong.
The more JSON-structured information you put in each log statement, the more power you have to understand them.
Read More: What Is Remote Logging? »
About Thomson
Having received his M.S. in Computer Science at Colorado State University, Thomson now provides on-demand consulting services with Cardinal Peak, LLC near Boulder, Colorado. When he’s not building UIs, analyzing data, or auditing security protocols, he spends time adventuring and sailing the seven seas with his wife and four children. Cardinal Peak clients include Samsung, Beyond Oblivion, Yonder, Time Warner Cable, Dish Network, Ball Aerospace, Cable Labs, Covidien, Comcast, and a variety of big new customers up and coming in Internet of Things (IoT). Follow him on Twitter: @thomcom.
The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.
Thomson Comer
