Docker Syslog Logging and Troubleshooting

When you’re troubleshooting an application, the first instinct is to read the logs to understand what’s happening. When working with containers, though, we need to consider the ephemeral nature of a container. By the time you find out there’s a problem with your application, the container and the logs might be gone. Fortunately, Docker has a native way of reading logs from a container.

However, we live in a world where we won’t have only one container running, and our systems have multiple dependencies. We need to have a centralized location for our logs to have a better context of the problem. Also, if the container has gone away, we can read the logs collected while it was running.

In this post, I’ll provide a quick overview of why having a syslog server might be a good option. Then, I’ll show you how you can integrate syslog and Docker to improve your troubleshooting strategy through logs.

Reading Logs From a Container

Docker comes with a native command, docker logs, to read logs from a container. The following command runs a container that simply prints a text indefinitely.

docker run -d busybox /bin/sh -c 'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done'

If you want to read the logs from that container, you need either the name or the ID of the container. Then, you can read the logs with the docker logs CONTAINER, and you can add the -f flag to see logs live. See below for an example of the commands you’d usually run:

$ docker ps

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES

340461436e07        busybox             "/bin/sh -c 'i=0; wh…"   7 seconds ago       Up 7 seconds                            fervent_shtern

$ docker logs 340461436e07 -f

0: Fri Sep  4 11:14:43 UTC 2020

1: Fri Sep  4 11:14:44 UTC 2020

2: Fri Sep  4 11:14:45 UTC 2020

3: Fri Sep  4 11:14:46 UTC 2020

4: Fri Sep  4 11:14:47 UTC 2020

That’s initially good because you can read the logs from a container. However, if the container goes away, you can’t read its logs anymore using the logs command. This command only works while the container is running. We need a better way to collect container logs, so we don’t lose them when the container stops—containers come and go all the time.

There are many ways you can process and retain Docker container logs. But this time, I’ll use a syslog server. You’ll notice, though, there’s going to be a missing piece once logs are there—I’ll come back to this later.

Why Syslog for Logging in Docker?

Using a syslog server might not be the first option you think of when you think about collecting container logs. However, it’s a cheap and straightforward strategy to help you get started. It’s likely you already have a logging architecture using syslog and the missing piece was how to collect logs from Docker.

For instance, you could use a central syslog server as an aggregation location and then process all logs for removal of sensitive or private information, aggregating data, or forwarding logs to other places. Additionally, for performance reasons at the networking level, you could decide to have a syslog server locally as a buffer to network traffic. Or, perhaps you’re even planning on removing syslog soon, but you need to process Docker logs in the meantime. Nonetheless, troubleshooting without proper logs is challenging, so syslog provides a quick and easy way to start.

Configuring a Syslog Server

I’m using an Ubuntu 18.04 server in AWS, which already comes with rsyslog (the syslog server) installed. Otherwise, you’ll install it with the sudo apt install -y rsyslog command if you’re using Ubuntu. Once you have rsyslog, it’s good practice to create a configuration file specifically for the Docker daemon. We’d like to have a dedicated location for these types of logs, including the Docker daemon and container logs.

First, create a configuration file to collect all logs from the Docker daemon. Create a file at “/etc/rsyslog.d/49-docker-daemon.conf” using sudo with the following content:

$template DockerLogs, "/var/log/docker/daemon.log"

if $programname startswith 'dockerd' then -?DockerLogs

& stop

Now, create a configuration file to collect logs from containers in a separate file. Create a file at “/etc/rsyslog.d/48-docker-containers.conf” using sudo with the following content:

$template DockerContainerLogs,"/var/log/docker/%hostname%_%syslogtag:R,ERE,1,ZERO:.*container_name/([^\[]+)--end%.log"

if $syslogtag contains 'container_name'  then -?DockerContainerLogs

& stop

And let’s restart the rsyslog server to confirm everything is working:

sudo service rsyslog restart

Configuring Docker for Sending Logs to Syslog

Once rsyslog is ready to process the Docker logs, let’s configure the Docker daemon to send the logs to syslog.

Modify (or create it, as I did) the Docker daemon configuration (“/etc/docker/daemon.json”) with sudo permissions, and add the following content:

{

  "log-driver": "syslog",

  "log-opts": {

    "tag": "container_name/{{.Name}}",

    "labels": "dev",

    "syslog-facility": "daemon"

  }

}

In this case, I’m including a tag with the value dev to identify the logs I’m collecting in this server. Also, I’m using the local syslog server (through sockets), but this could be a remote server as well. For instance, using the syslog-address property, you could have something like this:

"syslog-address": "tcp://${SYSLOG_SERVER_IP}:514"

You might use the local syslog server to avoid losing logs for network connectivity and to forward logs to a centralized server.

Now, let’s restart the Docker daemon:

sudo systemctl daemon-reload

sudo systemctl restart docker

To make sure everything is working, let’s create a new container:

docker run -d busybox /bin/sh -c 'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done'

This time, if you try to read the logs from the container using the log command, you’ll get an error:

$ sudo docker ps

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES

985d45dc3840        busybox             "/bin/sh -c 'i=0; wh…"   12 minutes ago      Up 12 minutes                           affectionate_newton

$ sudo docker logs 985d45dc3840

Error response from daemon: configured logging driver does not support reading

So, to read the logs from the container, you can go to the file generated by rsyslog:

$ ls -la /var/log/docker/

total 56

drwxr-xr-x  2 syslog syslog  4096 Sep  4 15:10 .

drwxrwxr-x 10 root   syslog  4096 Sep  4 15:05 ..

-rw-r-----  1 syslog adm      274 Sep  4 15:10 daemon.log

-rw-r-----  1 syslog adm    38772 Sep  4 15:11 ip-172-31-8-118_affectionate_newton.log

Both the daemon and container file logs are there. Let’s look at each one of them:

$ tail -f /var/log/docker/daemon.log

Sep  4 15:10:47 ip-172-31-8-118 dockerd[7298]: time="2020-09-04T15:10:47.664928943Z" level=error msg="Handler for GET /v1.40/containers/985d45dc38408ce0d4af08d5b71d64e90052c6d5f72ae3fa7a0d2497e96c9a32/logs returned error: configured logging driver does not support reading"

$ tail -f /var/log/docker/ip-172-31-8-118_affectionate_newton.log

Sep  4 15:12:49 ip-172-31-8-118 container_name/affectionate_newton[7298]: 895: Fri Sep  4 15:12:49 UTC 2020

Sep  4 15:12:50 ip-172-31-8-118 container_name/affectionate_newton[7298]: 896: Fri Sep  4 15:12:50 UTC 2020

Sep  4 15:12:51 ip-172-31-8-118 container_name/affectionate_newton[7298]: 897: Fri Sep  4 15:12:51 UTC 2020

Sep  4 15:12:52 ip-172-31-8-118 container_name/affectionate_newton[7298]: 898: Fri Sep  4 15:12:52 UTC 2020

Sep  4 15:12:53 ip-172-31-8-118 container_name/affectionate_newton[7298]: 899: Fri Sep  4 15:12:53 UTC 2020

^C

As you can see, the name of the files and the log format complies with the structure I defined in the rsyslog configuration.

And that’s how you can configure the Docker daemon to send logs to a syslog server.

There’s a Missing Piece

What’s the missing piece I previously mentioned? Well, once logs land at the syslog server, you need to do something with them. Even if logs are now centralized and it doesn’t matter if the container source of logging goes away, you still need to have a better way of reading logs. Going to a server and exploring logs manually with a grep or tail command won’t scale (in addition to being notoriously dynamic, containers and their logs tend to be quite prolific). What if you use syslog as an intermediary location? For instance, once you collect logs, you can send them to a system in which to run search queries, identify patterns, or create dashboards and alerts.

Using SolarWinds^® Loggly^®, there are several ways you can process logs. For instance, you could have an agent running in the syslog server or sidecar container forwarding the logs to Loggly. In Loggly, you can then analyze the logs you’re collecting from your systems, even if they’re not running in containers. Give it a try in Loggly today.

This post was written by Christian Meléndez. Christian is a technologist who started as a software developer and has more recently become a cloud architect focused on implementing continuous delivery pipelines with applications in several flavors, including .NET, Node.js, and Java, often using Docker containers.