Five Things Every Developer Should Know About Syslog
If you are responsible for deploying and running applications, you probably have at least a passing familiarity with syslog. After all, it’s the most widely adopted standard for how log data is formatted and how it should be moved between systems.
Using a Syslog utility is also one of the ways that Loggly customers can send us logs without installing agents on their machines, so we want developers to clearly understand what it is and how it works. Here are the five key things that every developer should know about syslog.
Many of the examples I have shown below are for the rsyslog utility, but they all have equivalents in syslog-ng.
1. Syslog (the standard) is distinct from a Syslog logging utility.
It’s easy to confuse the Syslog protocol with the mechanisms and applications that handle syslog. The Syslog protocol, defined in the IETF’s RFC 5424, sets a standard format for log messages. However, there are also syslog utilities or services where log messages are sent for handling:
- rsyslog and syslog-ng are the most popular examples of utilities that handle syslog formatted messages. Ubuntu ships with rsyslog by default.
- There’s even a syslog utility for Windows: NXLOG
2. Syslog forwarding lets you move log data between syslog on different machines.
By default, a syslog utility is configured to save log events to some set of files on a local system. However, it can also be configured to forward log events to another machine. This feature is commonly used to send all log events to a centralized log storage location, from which they can be searched using tools like grep/awk/sed or better yet, sent to a log management service like Loggly.
It’s good to note that you can encrypt your log data when forwarding between different machines. Log events that contain sensitive information may be exposed if they’re forwarded over a wide-area network, so you should configure your Syslog utility to encrypt the connection forwarding sensitive log events. This can only be used for TCP forwarding connections.
3. Filtering gives you a way to apply actions to just some but not all of your log data.
If you want to limit what you want saved into log files, syslog filters are how you express what you want to include or exclude. For example, you might want your production servers to log only WARN priority messages and above. In this case, rsyslog can be configured to apply different actions on log events depending on a variety of properties like facility, priority, or whether the message matches a regular expression rule. Filters are also a way to adjust what is logged without making changes in the application itself.
The traditional syslog format allows filtering based on facility and priority, but rsyslog and syslog-ng allow you to define rules that vary actions based on a variety of properties.
Some further examples of what filtering can help you do:
- Write DEBUG level events to a local file and forward higher priority events to another machine.
- Write log events for business analytics into a file available to business analytics users, while general log events from the applications are written to a file available to the operations team.
- Forward only log events from a particular set of applications to Loggly.
David Lang has written a comprehensive article on filtering with Rsyslog that is well worth a read (PDF).
4. Key-value pairs in the syslog header give you a way to structure your log data.
The syslog header format defines fields for information like timestamp, hostname, application name, and process ID. The message itself, however, is free text. If there is particular part of the message you’re interested in, the application consuming the logs will need to extract it from the message.
But there’s also another way to include structured data with syslog. The syslog standard allows for a section in the message header, called Structured Data (SD) to include key-value pairs that can be easily parsed and interpreted. The SD field can be used to annotate the log event with additional information.
While Loggly is able to ingest data structured in JSON, we still leverage SD key-value pairs for the authentication token, which is included with each message that is passed into our system. Optionally, Loggly users may also include special key-value pairs in the SD header to tag log events.
5. Syslog utilities can read log events directly from your application log files.
Syslog utilities typically read log events from a special location or network socket. Some applications, however, insist on only logging to a file or files. In these situations, you can configure your syslog utility to fetch events from these files directly. The syslog utility will keep track of how far into a file it has fetched events and will monitor files for any new events as they are written.
Any filtering or forwarding rules can still be applied to events fetched from a file.
Get More Out of Syslog with Loggly
Because we don’t need our customers to install agents, Loggly is a natural extension from syslog aggregation. If you’re using syslog today, it’s really easy to send your data to Loggly and turn log management into a much less painful chore. Now that you’re a syslog master, there’s no reason to wait! You can click here to start a free trial. Send us those logs (via Syslog or any of the dozens of other ways we ingest them) and we will send you a killer shirt too!
Additional Reading
The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.
Ivan Tam
