Apache Logging Basics

Apache Logging Basics

The Apache log records events that were handled by the Apache web server including requests from other computers, responses sent by Apache, and actions internal to the Apache server. This section of the guide explains the basics of Apache logging including the types of logs generated by Apache, where they are stored, and how to interpret them. We’ll also cover advanced topics such as setting custom log formats and configuring modules to get richer data.

Types of Apache Logs

Apache generates two kinds of logs: access logs and error logs.

Access Log

The access log contains information about requests coming in to the web server. This information can include what pages people are viewing, the success status of requests, and how long the server took to respond. Here is an example of a typical access log entry:

10.185.248.71 - - [09/Jan/2015:19:12:06 +0000] 808840 "GET /inventoryService/inventory/purchaseItem?userId=20253471&itemId=23434300 HTTP/1.1" 500 17 "-" "Apache-HttpClient/4.2.6 (java 1.5)"

Error Log

The error log contains information about errors the web server encountered when processing requests, such as missing files. It also includes diagnostic information about the server itself. Here is an example error log:

[Thu Mar 13 19:04:13 2014] [error] [client 50.0.134.125] File does not exist: /var/www/favicon.ico

Log Locations

By default, Apache stores access and error logs in separate files on the server. The exact location depends on your operating system. You can find the location for your OS by clicking on the following links:

Debian/Ubuntu/Linux Mint
Redhat/Fedora/CentOS Configuration
OpenSUSE

Configuring Apache Logs

Apache has a highly configurable logging framework that lets you adjust logging behavior globally or for each virtual host. There are several directives you can use to change logging behavior. Some of the more common directives are the log level and log format directives, which we will explain in greater detail.

Log Level Directive

The LogLevel directive determines the minimum severity level of events that are logged to a specific destination. The severity level represents how important the event is and can range from “Emerg” to “Trace8”, with “Emerg” representing events that may lead to instability, and “Trace8” representing trace-level messages. For example, LogLevel crit will allow logs with “Crit”, “Alert”, and “Emerg” severity through, but will ignore all other levels.

Log Format

The LogFormat directive controls the layout and formatting of log events. Apache uses the Common Log Format (CLF) by default, but you can specify your own format string to change the fields included in each log.

You can also use the CustomLog directive to change the location of the log file. In Linux, Apache commonly writes logs to the /var/log/apache2 or /var/log/httpd directories depending on your OS and Virtual Host overrides. You can also define a LogFormat string after the filename, which will only apply the format string to this file.

For example, the following configuration writes logs to logs/access_log using the standard “common” format. The logs/ subdirectory is relative to the directory specified in the ServerRoot directive:

LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog "logs/access_log" common

CustomLog "logs/access_log" common

You can find a full list of fields in the Apache log documentation. We recommend using at least the following five fields, as they are important for monitoring server health and for troubleshooting issues:

1. %>s – The HTTP status code for the request. This shows the final request status after any internal redirection; for the original status, use %s.
2. %U – The URL path requested, excluding any additional URL parameters such as a query string.
3. %a – The IP address of the client making the request. This is useful for identifying traffic from a particular source.
4. %T – How long it took to process the request in seconds. This is useful for measuring the speed of your site. Use %D to make the same measurement in microseconds.
5. %{UNIQUE_ID}e – Also commonly known as the request ID, this logs a unique identifier with each request. This is useful for tracing a request from Apache to your web application server.

Assigning Nicknames

LogFormat strings can be assigned nicknames, which you can then use with a CustomLog directive to write logs using the specified format. This allows you to use the same log format for several log files without having to redefine the format each time. This is particularly useful when using different log files for multiple virtual hosts.

For example, let’s create an example format and name it “vhost_combined.” We’ll then create a CustomLog directive that writes logs to a file using the vhost_combined format.

LogFormat "%v:%p %h %l %u %t "%r" %>s %O "%{Referer}i" "%{User-Agent}i"" vhost_combined
CustomLog /var/log/apache2/vhost.log vhost_combined

Formatting as JSON

Storing your logs as plain text makes them easy to scan in case you ever need to read your log files. However, this makes it difficult to use tools such as log management solutions to read your logs, since these tools must know how your logs are formatted. Most log management solutions support the default Apache log format, but if not, you should consider using a structured format like JSON.

JSON (short for JavaScript Object Notation) is a flexible format for storing structured data. JSON stores a collection of nestable name/value pairs, letting you store almost any data type and structure. JSON is also effectively self-documenting, since the name of the key describes the data that it contains. JSON supports many basic data types including strings, numbers, booleans, arrays, and null values.

Here is an example of a LogFormat that stores logs in JSON format:

LogFormat "{ \"time\":\"%t\", \"remoteIP\":\"%a\", \"host\":\"%V\", \"request\":\"%U\", \"query\":\"%q\", \"method\":\"%m\", \"status\":\"%>s\", \"userAgent\":\"%{User-agent}i\", \"referer\":\"%{Referer}i\" }"

Default Virtual Host Overrides

Virtual Hosts (vhosts) are used to run more than one website on a single Apache server. You can define a separate logging configuration for each vhost, which is given priority over the global logging configuration. This lets you log each website to a separate directory, for example. Leaving these parameters out defaults to the global logging configuration.

For example, the following configuration is for a vhost at example.com. Logs are written to separate access.log and error.log files in the /var/www/example.com/logs directory:

<VirtualHost *:80>
     ServerName example.com
     ServerAdmin webmaster@example.com
     DocumentRoot /var/www/example.com
     LogLevel info ssl:warn
     ErrorLog /var/www/example.com/logs/error.log
     CustomLog /var/www/example.com/logs/access.log example
</VirtualHost>

The following sections show the default configuration file locations and directives used in different Linux distributions.

Debian/Ubuntu/Linux Mint

On Debian-based distributions, the default vhost configuration for unencrypted sites (port 80) can be found at /etc/apache2/sites-available/000-default.conf. The default vhost configuration for sites encrypted with SSL/TLS (port 443) is at /etc/apache2/sites-available/default-ssl.conf.

Table of Default Directives

* Conditioned environment variable. Provides support for multiple Apache server instances.

** Environment variable. Used to dynamically set the initial log path.

Red Hat/Fedora/CentOS

On Red Hat-based distributions, the main configuration file is located at /etc/httpd/conf/httpd.conf. You can place additional vhost config files in the /etc/httpd/conf.d directory, which is automatically read by the server on start.

Table of Default Directives

* Conditioned on loaded log_config_module.

** Conditioned on loaded logio_module.

OpenSUSE

In OpenSUSE, the default vhost config for unencrypted sites (port 80) is located at /etc/apache2/default-vhost.conf, while the default config for sites encrypted with SSL/TLS is located at /etc/apache2/default-vhost-ssl.conf.

Table of Default Directives

* Conditioned on loaded log_config_module.

** Conditioned on loaded logio_module.

*** Conditioned on loaded mod_ssl.

Log-Related Modules

The Apache web server offers a number of modules that either change the way Apache works or extend its capability. The following modules add or change the logging behavior in useful ways.

mod_log_config

This is the base logging module that Apache uses, and the one that we covered in this section of the guide.

mod_log_debug

This module provides additional options for logging debug messages. For example, you can log events to a specific URL path, log requests after they are processed, log messages from a specific client if they result in timeouts, and more. Note that this module is experimental and may not be included in your Apache distribution.

For example, the following configuration logs the IP address of a client whenever it requests a specific path:

<Location /path/to/some/specific/directory>
    LogMessage “/path/to/specific/directory has been requested by”
    ${REMOTE_ADDR}
</Location>

mod_log_forensic

This module enables logging before and after a request is processed. Each entry is assigned a unique ID, which can be used to trace events between the forensic log and normal log. However, the forensic logger does not support custom formats. This module also supersedes the mod_unique_id module.

Once the module is enabled, you can use the forensic keyword to specify which log file is the forensic log. You can also use the %{forensic-id} pattern in other LogFormat strings to add forensic data to normal logs. Here, we use /var/www/example.com/logs/forensic.log:

<VirtualHost *:80>
    ServerName example.com
    ServerAdmin webmaster@example.com
    DocumentRoot /var/www/example.com
    LogLevel info ssl:warn
    ErrorLog /var/www/example.com/logs/error.log
    CustomLog /var/www/example.com/logs/access.log example
    CustomLog /var/www/example.com/logs/forensic.log forensic
</VirtualHost>

Each line in the forensic log starts with either a “+” or “-” character. “+” indicates the entry log line for this particular request, and “-” indicates subsequent entries for the same request.

For example, the following entry log was generated by calling http://localhost using the default log format:

+3264:55958cb4:0|GET / HTTP/1.1|Host:localhost|Connection:keep-alive|
Cache-Control:max-age=0|
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8|
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36|
Accept-Encoding:gzip, deflate, sdch|
Accept-Language:en-US,en;q=0.8|
If-None-Match:"2cf6-519e8edfca429-gzip"|
If-Modified-Since:Thu, 02 Jul 2015 18%3a51%3a39 GMT

The ending log entry from the same request appears as:

-3264:55958cb4:0

If an ending entry does not appear in the log, the request did not complete.

mod_logio

This module adds the ability to log the number of bytes sent and received per request. This includes bytes received, sent, and transferred (combination of received and sent). The module also accurately accounts for changes in size due to SSL and TLS encryption. This module requires mod_log_config and is typically included in Apache by default.

This module adds the LogIOTrackTTFB On|Off directive, which toggles the ability to track the time to first byte (TTFB). TTFB is the amount of time from when a request is received to when the response is first sent to the client. You can then use %^FB to include the TTFB measurement in a LogFormat string.

mod_filter

This module provides context-sensitive filters to the output chain by registering any number of filter providers. mod_filter is not specific to logging, but allows for extracting specific requests based on the filter provider. The context containers include: main apache config, vhost config, within directory tags, and .htaccess files.

Employing this module allows for filtering requests containing items such as certain injection criteria, and which IP address it’s from.

This module is provided by default in many of the package distributions, but may require enabling. For the purposes of logging, the FilterTrace directive posts information to the error log. Directives include:

• AddOutputFilterByType – assigns an output filter to a particular media type
• FilterChain – configures a filter chain
• FilterDeclare – declares a smart filter
• FilterProtocol – causes the mod_filter to handle response headers correctly
• FilterProvider – registers filter providers
• FilterTrace – allows for debugging/diagnostic information to an error log prior to provider processing

The following example applies filtering in a vhost context container conditionally on filter and include modules:

<IfModule mod_filter.c>
    <IfModule mod_include.c>
		#Declare a resource type filter:
		FilterDeclare xss
		#Register a provider:
		FilterProvider xss INCLUDES %{REQUEST_FILENAME}="(/[<>]+)$"
		#FilterProvider ...
		#Build the chain:
		FilterChain xss
		#Declare a custom log:
		CustomLog /var/www/log/xss.log xss
		#Format the log entry:
		LogFormat "%h %u %t "%r" %>s "%{Referer}i" "%{User-Agent}i"" xss
	</IfModule>
</IfModule> 

mod_unique_id:

This module constructs an environment variable and a unique identifier for each request. It’s often included in package distributions but may require enabling. This unique identifier is written to the access log.

This module has been superseded by mod_log_forensic for forensic purposes, but is still supported for others.

Unlike the forensic identifier, the unique identifier is passed to the application handler via the environment variable UNIQUE_ID. This allows application developers to trace a request through the web server to the application server. It can be useful for debugging a request.

Apache spins off child processes to handle requests, and a child instance processes several requests at a time. As a result, it is sometimes desirable to use a unique identifier to identify a single request across multiple server instances and child processes.

Once enabled, the module provides an identifier by default to the application handler.

For example:

UNIQUE_ID: Vaf3en8AAQEAAAtoQlAAAAAA

The identifier is constructed from a 32-bit IP address, 32-bit process ID, 32-bit timestamp coupled to a 16-bit counter for tighter resolution than a single-second, 32-bit thread index. The timestamp component is UTC to prevent issues with daylight savings time adjustments. The application handler should treat the identifier as an opaque token only and not dissected into constituents.