Centralizing Apache Logs

Centralizing Apache Logs

By default, Apache stores all logs to the local disk. This works well for development environments and small deployments, but becomes unsustainable once you have more than one server. Not only is it frustrating having to open each log file on each server, but trying to trace requests across multiple servers can quickly become time-consuming.

Log centralization services prevent this by allowing you to store logs from your Apache servers in a single location. This makes it possible to view all of your web logs without having to open each log file individually. Many log centralization services can also automatically parse your logs, and provide a user interface that lets you scroll, search, and filter through your log data in near real time.

This section shows different methods of aggregating and centralizing logs from your Apache servers.

Syslog

Syslog is a logging service commonly found on Linux, Unix, and Mac systems. Syslog handles logs from a number of different sources including applications, system services, daemons, and hardware. Syslog is reliable, standardized, and can even forward your logs to another syslog server.

A common approach to reading Apache logs is to configure syslog with file monitoring. With file monitoring enabled, syslog periodically scans a file on the system for changes, then imports those changes into its own log file. The benefit is you get the complete original log message wrapped in the standard syslog message format without modifying the original file.

Configuring File Monitoring in Syslog

The most common way to enable file monitoring is by installing and configuring rsyslog. rsyslog is a complete syslog server with file monitoring built in. It’s easy to configure, fast, and supports log rotation, which is commonly found in Linux distributions such as Ubuntu.

The following rsyslog configuration monitors both the Apache access and error logs. You may need to replace the file names depending on your configuration.

module(load="imfile" PollingInterval="10")

# Apache access file:
input(type="imfile"
      File="/var/log/apache2/access.log"
      Tag="apache-access"
      Severity="info")

# Apache error file:
input(type="imfile"
      File="/var/log/apache2/error.log"
      Tag="apache-error"
      Severity="info")

Save this to your rsyslog configuration file, then restart the rsyslog service.

sudo service rsyslog restart

Some vendors have scripts or agents that will configure rsyslog to monitor these log files, making setup easier. For example, SolarWinds^® Loggly^® built a script that will automatically configure rsyslog to monitor your Apache logs.

Filtering Logs Before Centralization

In some situations, you may want to filter your logs before sending them to your centralization service. For example, you may only want to send error codes in order to use less storage on the remote system. With rsyslog, we can add a condition to our file monitoring rule that only allows events containing certain HTTP status codes.

This configuration example drops all messages where the status code is not 500 or 502. stop tells rsyslog to discard the message.

if $programname == 'apache2-access' and not ($msg contains '500' or $msg contains '502') then stop

Piping to Logger

Apache doesn’t just support logging to files. For example, you can also send logs directly to a syslog service using a custom logging pipeline. The most common method is to use the /usr/bin/logger command, which forwards logs over a syslog socket to the syslog service. This lets you bypass the file monitoring process, which could have performance advantages on slower storage devices. In addition, you no longer have to store a separate log file for Apache.

The downside to this approach is it removes the local backup provided by your Apache logs. If there’s a problem sending your logs from logger to syslog, you could lose messages. In addition, logger supports a maximum message size of 1024 bytes. However, you can increase the size of this by adding the --size parameter.

To set up a logging pipe, open your Apache configuration file and replace your logging configuration with the following.

ErrorLog  "| /usr/bin/logger -thttpd -plocal6.err"
CustomLog "| /usr/bin/logger -thttpd -plocal6.notice" extended_ncsa

Restart the Apache service to apply the changes.

sudo service apache2 restart

Now your logs will no longer be written to the access.log and error.log files, but will instead go straight to syslog. If you want to continue logging to file as well as syslog, you can use the following configuration instead. This uses the tee command to first pipe the log message to file, then pipes the output from that command to logger:

ErrorLog  "|/usr/bin/tee -a /var/log/www/error.log  | /usr/bin/logger -thttpd -plocal6.err"
CustomLog "|/usr/bin/tee -a /var/log/www/access.log | /usr/bin/logger -thttpd -plocal6.notice" extended_ncsa