Analyze Access Logs

This Solution describes how to analyze web access logs. You can graph and alert on error rates, view the most commonly requested pages, and more.

A Solution is a step-by-step guide for accomplishing a specific task, designed to make sense even if you're just getting started with Scalyr server monitoring. If you're new to Scalyr, you should read the short Getting Started guide. For help with other tasks, see the Solutions directory.

Prerequisites

1. The Scalyr Agent should be installed on the server(s) you want to monitor, and should be configured to upload your access logs. In the Scalyr Agent configuration file, each access log should be tagged with parser: "accessLog".

To verify:

  • In the navigation bar, click Overview.
  • Find each server in the list at the bottom of the page.
  • Verify that your web server's access log is listed next to each server.
  • Click the link for each log, and click on a log message. A box will appear showing details for that message. Verify that parsed fields such as "agent", "authUser", and "bytes" are listed.

If any server is not listed:

If any log file is not listed:

If the log view does not show parsed attributes:

  • Open the Scalyr Agent configuration file (agent.json) on the affected server, and find the entry that refers to your web access log. Check whether it specifies parser: "accessLog". If not, update the file accordingly. The relevant configuration entry should look something like this: { path: "/var/log/httpd/access*", attributes: {parser: "accessLog"} }

Steps

1. To see an overview of traffic to all of your web servers, click Dashboards in the navigation bar, and select the "Paths" dashboard.

This will display a list of each unique request page in your access logs, with the most frequently requested pages at the top. Click on any numeric value to see a graph of the data behind that number. For instance, the "2xx" column shows the number of requests for that page in the last hour which yielded a successful response (HTTP status 200 through 299). Click on the number to see a graph of successful responses per second.

2. To see graphs of traffic for an individual server, click Dashboards in the navigation bar, and select the "WebServer" dashboard.

This will display a set of graphs summarizing traffic to a particular server. Use the Host dropdown to view different servers.

3. To see the pages which most frequently trigger a server error (HTTP status 500 through 599):

  • In the navigation bar, click Search.
  • In the Expression box, enter dataset='accesslog' status >= 500
  • Click the Search button
  • Switch to the Facets tab
  • If your web servers haven't had any errors in the last few hours, congratulations! You'll see a "No matching events" message. Otherwise, you'll see a breakdown of all fields in your web access logs for failed request. Scroll down to the section for the uriPath field to see the most common error pages. If more than 10 pages have generated errors, click the see all values link to view more.

Further Reading

For additional search options, see Query Language. To read more about the log viewer, see Exploring Data.

For instructions on viewing which IP addresses generate the most traffic to your web servers, or related queries such as the most common user agent, see the Display Common IP Addresses solution.