Rsyslog: Manual Configuration and Troubleshooting

The BSD Syslog standard has been with us for a long time, and even with the advent of journald, it’s here to stay. Rsyslog logs messages to the network or to local disk with high performance. And, its client-server architecture and multithreaded architecture make it easy to scale your logging infrastructure. It’s one of the most robust implementations of syslog available on Linux.

Let’s look at how you can configure Rsyslog for your system and configure its syslog daemon to forward logs to another server.

What Is Rsyslog?

Rsyslog is an open-source high-performance logging utility. It offers many powerful features for log processing:

• Multithreaded log processing
• TCP over SSL and TLS
• Reliable Event Logging Protocol (RELP)
• Logging to SQL database including PostgreSQL, Oracle, and MySQL
• Flexible and configurable output formats
• Filtering on all aspects of log messages

Rsyslog logs are billed as the “rocket-fast system for log processing” because of their exceptional throughput capabilities. A well-configured system can process more than a million messages per second to a local destination. Even when configured to send messages over a network, its ability to scale provides excellent performance.

Rsyslog Installation

Depending on which Linux distribution you’re running, Rsyslog may already be installed and running.

Run this in a shell:

$ ps -ef |grep rsyslog

syslog    5069     1  0 13:22 ?        00:00:00 /usr/sbin/rsyslogd -n

egoebel+  5121  5592  0 13:23 pts/2    00:00:00 grep --color=auto rsys

The first line shows Rsyslog running on my system. The second shows the grep command I entered. So, if you see two lines, and one of them contains the rsyslogd program, it’s already installed and running on your system. If you only see one, you need to install Rsyslog on your system.

If you’re running on Ubuntu or one of its derivatives like Mint, you can either install from your distribution’s repository or add the latest from Rsyslog maintainers.

The first two lines add the new repository to your system. The last installs rsyslog.

$ sudo add-apt-repository ppa:adiscon/v8-stable

$sudo apt-get update

$sudo apt-get install rsyslog

For more details on installing Rsyslog, check out the official Rsyslog docs here.

If you’re using RedHat or CentOS, you can add the Rsyslog yum repository to your system and install the package.

$ cd /etc/yum.repos.d/

$ wget http://rpms.adiscon.com/v8-stable/rsyslog.repo # for CentOS 7

$ wget http://rpms.adiscon.com/v8-stable/rsyslog-rhel7.repo # for RHEL 7

$ yum install rsyslog

More information, including the GPG key for this repo, can be found in the Rsyslog documentation.

Rsyslog Configuration

If you installed Rsyslog or it was already there, then it’s running with a default configuration. Let’s start by looking at the configuration file.

Rsyslog Logs Input

The default configuration for Rsyslog is to receive messages via a UNIX domain socket. Your installation is very likely configured for it already. The main configuration file is located at /etc/rsyslog.conf.

Somewhere near the top of the file, you’ll see an entry like this:

module(load="imuxsock")

The modular Rsyslog architecture makes it easy to add extensions. This line tells it to load a module named imuxsock for receiving messages via dev/log. This is the default location for local programs using the syslog standard.

A few lines down, you should see:

#module(load="imudp")

#input(type="imudp" port="514")

A line that begins with # is a comment, so Rsyslog ignores these lines. But they show you how to set up an Rsyslog server to receive messages over UDP.

The first line loads the imudp module. The second line establishes where the module should listen for logging messages: over UDP port 514.

Verify that those two lines are commented out, then run this shell command and examine the output.

$ netstat -an |egrep "^udp"|grep 514

$

The lines are still commented out. So, there’s no service listening on UDP port 514 now—ensuring they were appropriately commented out.

Now, uncomment the line using your favorite text editor, then restart the service and check again. You’ll have to use the sudo command to change the file.

$ sudo systemctl restart rsyslog

$ netstat -an |egrep "^udp"|grep 514

udp        0      0 0.0.0.0:514             0.0.0.0:*                         

udp6       0      0 :::514                  :::*

The daemon is listening on UDP port 514 over both TCP/IP versions 4 and 6 now.

Rsyslog Rules

Toward the bottom of your config file, you should see a block like this:

#

# Include all config files in /etc/rsyslog.d/

#

$IncludeConfig /etc/rsyslog.d/*.conf

Rsyslog configurations can include other files. This makes the files easy to manage, especially if you’re responsible for managing a large network of systems. This directive tells rsyslogd to load all the files contained in /etc/rsyslog.d/.

Here are the contents of that directory on a standard installation:

$ ls /etc/rsyslog.d/

20-ufw.conf  50-default.conf  postfix.conf

Rsyslog uses standard file globbing to load the files, which ensures it evaluates a directory of files in alphabetical order. This is important to understand since the directives in one file may supersede a previous one.

Rsyslog Selectors and Actions

To send a log message to a location, you need to write a rule matching the message. Then you can send it somewhere.

Let’s write a rule for debug messages. Open /etc/syslog.d/50-Default.conf in your favorite text editor and add this line:

*.=debug                        /var/log/debug

This configuration line contains a selector and an action.

First, the selector is *.=debug. This expression contains two components: facility.priority. The facility indicates where the message is sent from. The priority indicates how important the message is. You’re wildcarding the facility with the asterisk and matching the priority with =debug with only debug messages. The equals operator indicates an exact match.

Next, the action tells rsyslogd where to put the messages that match the filter. In this case, we only need to supply a valid path and file name. Here, any debug messages will be sent to /var/log/debug.

Now it’s time to test this rule. We’ll need a program that sends log messages.

Testing Rsyslog Logs

You’re going to use the logger utility to test your Rsyslog configuration. This tool should already be installed on your system. If not, check your distribution’s documentation for instructions on how to add it.

With logger, you specify a message facility and priority with the -p option. This sends a message with the daemon facility and debug priority.

$ logger -p daemon.debug "This is a test."

Where will this message end up?

First, it arrives via the imuxsock input, since logger uses the default logging mechanism. Then, it matches the *.=debug selector since the level is debug (with the facility being daemon). Finally, it’s handled by the /var/log/debug action.

Let’s check for the message in /var/log/debug. You’ll see a message with your login name and the test log message.

Sep  19 16:32:40 hala egoebelbecker: This is a test.

Now, let’s try one more configuration change.

Conditional Selectors

Earlier you set up your rsyslogd to accept messages over UDP. Let’s test this setting with a filter that sends UDP messages to a specific log file.

If $inputname == "imudp" then {

Action (type="omfile" File="/var/log/udp.log")

}

Rsyslog supports conditional expressions in selectors. This selector uses if/then to evaluate a message property, inputname, which contains the name of the input module that received the message. If the name matches, it places it in a file named /var/log/udp.log. When you use the conditional syntax for a selector, the syntax for specifying actions is a bit more verbose because you’re introducing a powerful scripting language.

Let’s test this setting. Restart rsyslogd and send a message over UDP by adding a couple of command-line arguments to logger.

$ logger -p daemon.debug -n 127.0.0.1 -P 514 -d "Sending this over UDP!"

The -n and -p options specify the server name (or address) and service port, respectively. The -d options indicate that logger should use UDP.

Now check the log file.

Sep 19 21:19:28 hala egoebelbecker Sending this over UDP!

The message is there!

Master Rsyslog Logs

You’ve installed and configured your systems to use Rsyslog for system logging. You set up a rule to direct messages to different log files based on their priority. Then you added one that directed them based on how they arrived at the server. Don’t stop here: keep experimenting and see how you can use Rsyslog logging to improve your monitoring and debugging workflow.

If you log to Rsyslog, you can direct your log messages to SolarWinds^® Loggly^® too. Loggly provides you with proactive alerts and data visualizations. You build your own interactive dashboards and drill down into your logs using the Loggly Dynamic Field Explorer^™. Sign up for a free trial and add even more power to Rsyslog here.

This post was written by Eric Goebelbecker. Eric has worked in the financial markets in New York City for 25 years, developing infrastructure for market data and financial information exchange (FIX) protocol networks. He loves to talk about what makes teams effective (or not so effective!).