Install Scalyr Agent (Docker)

This document describes the new recommended method for importing Docker logs to Scalyr using syslog. If you have used the previous method (logs collected via the Docker API), you should be warned that upgrading to the new method will change the file path and file name of the uploaded logs. You should plan your transition accordingly.

These instructions are for installing the Scalyr Agent in a Docker container. If you plan to run the Agent directly on Linux, see the Linux installation page. For Windows, see the Windows installation page.

Warning! If you are using Docker 1.13, please see the instructions below for fixing an issue that will prevent your container logs from being uploaded to Scalyr.

The Scalyr Agent is a daemon that uploads logs and system metrics to the Scalyr servers. This page provides streamlined instructions to get you up and running quickly in a Docker environment.

In our recommended architecture, you will run the Scalyr Agent in its own Docker container on each host running Docker in your system. This Scalyr Agent container will collect logs and metrics from all other containers running on the same host.

To start the Scalyr Agent container, please do the following:

1. Download the Scalyr Agent docker image

The official image is available from Docker Hub. Run:

docker pull scalyr/scalyr-docker-agent

The default :latest tag will always point to the latest released agent version.

2. Create a configuration file snippet to hold your api key. In this example, we will create /tmp/api_key.json:

(log in to see your API token)

3. Determine the path for the Docker API socket on your machine. Typically, the socket is located at either /run/docker.sock or /var/run/docker.sock. Be sure that you find the path to the true socket and not just a symlink to it. Use ls -l to verify. If both /run/docker.sock and /var/run/docker.sock are not symlinks, use /var/run/docker.sock. If neither file exists, try executing netstat --unix -l | grep docker.sock to locate the socket.

For the rest of the instructions, it is assumed that the socket is located at /var/run/docker.sock.

4. Launch the scalyr-agent container. Be sure to substitute /var/run/docker.sock with the path to your Docker socket.

docker run -d --name scalyr-docker-agent -v /tmp/api_key.json:/etc/scalyr-agent-2/agent.d/api_key.json -v /var/run/docker.sock:/var/scalyr/docker.sock -p 601:601 scalyr/scalyr-docker-agent

5. Configure other containers to send their logs to Scalyr via the Scalyr Agent container. When you start a container, you must supply these additional options to direct their logs to the Scalyr Agent container: --log-driver=syslog --log-opt syslog-address=tcp://127.0.0.1:601.

For example, to start a container running a "Hello World" example, you would execute:

docker run  --log-driver=syslog --log-opt syslog-address=tcp://127.0.0.1:601 -d ubuntu /bin/sh -c 'while true; do echo Hello, World!; sleep 1; done'

Note, if you are using Docker version 1.13 or after, you will have to also follow the instructions below for fixing an issue that will prevent your container logs from being uploaded.

That's It!

We hope that was easy. If you've had any trouble, please let us know. Otherwise, if this is your first time using Scalyr, this would be an excellent time to head on to the Getting Started guide.

You should also check out the Log Parsing page to set up a parser for your logs. Scalyr becomes an even more powerful tool for analysis and visualization when your logs are properly parsed.

For complete documentation on agent configuration options, see the agent reference.

For more details on creating custom images with your configuration files, modifying the configuration files in the agent, setting a server host name, assigning parsers, etc, read on...

Further Reading

Creating custom Scalyr Agent Docker images

As you will see below, you can turn on several useful features in the Scalyr Agent by modifying its configuration files. Modifying these files on one container is relatively simple in Docker, but can become burdensome when you wish to use the same configuration files on many Docker instances. To ease this burden, we have provided tools to allow you to easily create new Scalyr Agent Docker images that include your custom configuration files.

You can create a custom Docker image based on a running Scalyr Agent by executing the following commands:

mkdir /tmp/scalyr-docker-agent
cd /tmp/scalyr-docker-agent
docker exec -i scalyr-docker-agent scalyr-agent-2-config --docker-create-custom-dockerfile - | tar -xz
docker build -t customized-scalyr-docker-agent .

This will leave a new Docker image on your local Docker instance with the repository customized-scalyr-docker-agent. You can change this using the docker tag command. From there, you can use any of the standard methods for distributing your Docker image to the rest of your servers.

You can launch a new Scalyr Agent container using this image by executing:

docker run -d --name scalyr-docker-agent -v /var/run/docker.sock:/var/scalyr/docker.sock -p 601:601 customized-scalyr-docker-agent

Modifying configuration files

We have also created a tool to help you modify multiple configuration files on a running Scalyr Agent container at one time. It handles reading the configuration files from the container and then writing them back.

To read the current configuration files from the running Scalyr Agent, execute the following commands:

mkdir /tmp/scalyr-agent-config
cd /tmp/scalyr-agent-config
docker exec -i scalyr-docker-agent scalyr-agent-2-config --export-config - | tar -xz

After these commands finish, your current directory will have one file agent.json and a directory agent.d. The agent.json file is a copy of the running Scalyr Agent's /etc/scalyr-agent-2/agent.json configuration file. Likewise, the agent.d directory is a copy of the /etc/scalyr-agent-2/agent.d directory.

You can edit these files to make whatever changes you need. To write the changes back once you are finished, execute this command:

tar -zc agent.json agent.d/* | docker exec -i scalyr-docker-agent scalyr-agent-2-config --import-config -

There is no need to restart the Scalyr Agent after writing the configuration files. The running Scalyr Agent should notice the new configuration files and read them within 30 seconds.

Setting a server host name

By default, the Scalyr Agent's container id is used as the server host name for all logs it uploads. This is not ideal since this id can change over time, such as when you upgrade to a new version of the Scalyr Agent.

To avoid this issue, you may manually set a server host name for the container. To do this, follow the instructions above for modifying configuration files and create a new file agent.d/server_host.json with the following contents:

{
  "server_attributes": {
    "serverHost": "my-docker-host-01"
  }
}

Then, use the commands for writing the configuration back to the running Scalyr Agent, and your logs should begin appearing under the new server host name within 30 seconds.

You may also use the environment variable import feature to avoid having to create a unique server_host.json file for each server host name. This feature allows you to import variables from the environment and use them to define fields in a Scalyr configuration file.

For example, you could have defined your agent.d/server_host.json file with the following contents:

{
  "import_vars": [ "DOCKER_HOST_NAME" ],

  "server_attributes": {
    "serverHost": "$DOCKER_HOST_NAME"
  }
}

In this example, the environment variable DOCKER_HOST_NAME is used to define the server host name. This is particularly useful if you supply these environment variables using the -e Docker option when you launch the container. For example, you can follow the instructions above for creating a custom image with this configuration and then execute this command:

docker run -d -e DOCKER_HOST_NAME='my-host' --name scalyr-docker-agent -v /var/run/docker.sock:/var/scalyr/docker.sock -p 601:601 customized-scalyr-docker-agent

Customizing the log file names

By default, all logs sent to the Scalyr Agent container via syslog are uploaded with log files names like:

/var/log/scalyr-agent-2/containers/MyContainerName.log

You may customizes this file path based on your needs. To do this, you will need to edit the agent.d/docker.json file. Please follow the instructions above to export the configuration of your running Scalyr Agent.

You need to modify the agent.d/docker.json file to add in the line below that beings with docker_logfile_template:

{
  "monitors": [
    {
      "module": "scalyr_agent.builtin_monitors.syslog_monitor",
      "mode": "docker",
      "docker_logfile_template": "/var/mylogs/${CID}/${CNAME}.log"
    }, {
      "module": "scalyr_agent.builtin_monitors.docker_monitor",
      "log_mode": "syslog"
    }
  ]
}

Be sure to save your configuration file changes back to the running Scalyr Agent using the above instructions.

As you can see, you may use the variables $CID and $CNAME to define your file path. These are the id and name of the container generating the log file.

Note, the agent will not create directories. Make sure all directories in your path exist on the container.

Setting parsers, sampling and redaction rules

By default, all logs sent to the Scalyr Agent container are configured to use the dockerSyslog parser. You may override this by providing additional configuration.

To do so, you must create a log configuration stanza that matches the particular log file whose parser you wish to set. For example, follow the instructions above to export the Scalyr Agent's configuration and then create the following agent.d/container_logs.log file:

{
  "logs" [ {
        "path": "/var/log/scalyr-agent-2/containers/*Frontend.log",
        "attributes": {"parser": "frontendLog"}
      }, {
        "path": "/var/log/scalyr-agent-2/containers/*Backend.log",
        "attributes": {"parser": "backendLog"}
      } ]
}

After you finish your modifications, be sure to import them back to the running Scalyr Agent using the above instructions.

This would set frontendLog to be the parser for all logs uploaded by containers whose names end in Frontend. Similarly, it would set backendLog to be the parser for all logs with containers ending in Backend. As long as your container name defines the type of logs it generates, you will be able to set appropriate parsers.

You can use this technique to also specify sampling and redaction rules. See the documentation on sampling and redaction rules for more information.

Starting the Scalyr Agent container

As mentioned above, the typical command to start the Scalyr Agent command is below:

docker run -d --name scalyr-docker-agent -v /tmp/api_key.json:/etc/scalyr-agent-2/agent.d/api_key.json -v /var/run/docker.sock:/var/scalyr/docker.sock -p 601:601 scalyr/scalyr-docker-agent

For reference, here is a description of the options:

  • -d is to run in detached mode (needed when running a daemon process such as the scalyr-agent)
  • --name scalyr-docker-agent names this container 'scalyr-docker-agent'. It is useful to give a fixed name to the Scalyr Agent container.
  • -v /tmp/api_key.json:/etc/scalyr-agent-2/agent.d/api_key.json maps /tmp/api_key.json on the host, to /etc/scalyr-agent-2/agent.d/api_key.json on the container. This allows you to insert configuration snippets into the agent.d configuration directory to add in configuration read by the Scalyr Agent. You can avoid having to include this option if you follow the instructions to create a custom Docker image with your configuration files.
  • scalyr/scalyr-docker-agent:2.0.23 specifies the image to run.
  • -v /var/run/docker.sock:/var/scalyr/docker.sock maps the Docker API socket to a path in the container. The Scalyr Agent requires access to the Docker API socket to collect metrics and determine container names.
  • -p 601:601 maps the host's port 601 to the container's port 601. This is the default syslog port the Scalyr Agent listens on.

Stopping the Scalyr Agent container

Assuming you have a container named scalyr-docker-agent running, it can be stopped with the following command:

docker stop scalyr-docker-agent

Restarting the Scalyr Agent container

Assuming you have previously stopped a container named scalyr-docker-agent, you can restart it with the command

docker start scalyr-docker-agent

Checking the status of a running scalyr-agent container

To check the status of a currently running Scalyr Agent container, use the following command:

docker exec scalyr-docker-agent scalyr-agent-2 status

or

docker exec scalyr-docker-agent scalyr-agent-2 status -v

to get more verbose output.

Logs from other containers

In order to gather log files from other containers, you must share volumes containing those log files with the Scalyr Agent container. You can then configure the Scalyr Agent as normal to copy those files to the Scalyr servers.

There are numerous ways to share volumes from other containers. See the Docker documentation on managing data in containers

As a simple example, imagine you have an Apache container that was configured to log its access files to /var/log/httpd/access.log on the host. When running the Scalyr Agent container you would simply map that file to a file in the container with the -v command line parameter: -v /var/log/httpd/access.log:/var/log/httpd/access.log

The full command to run the container would be:

docker run -d --name scalyr-docker-agent -p 601:601 -v /tmp/api_key.json:/etc/scalyr-agent-2/agent.d/api_key.json -v /var/log/httpd/access.log:/var/log/httpd/access.log -v /var/run/docker.sock:/var/scalyr/docker.sock  scalyr/scalyr-docker-agent

You would then export your configuration files and create a agent.d/more_logs.json file with the following content:

{
  "logs": [
    {
      "path": "/var/log/httpd/access.log",
      "attributes": {parser: "accessLog"}
    }
  ]
}

And Scalyr will automatically start tracking this file after you re-import the configuration to the running Scalyr Agent.

If you have multiple containers that each have files you wish to upload, you should probably consider creating a Data Volume Container

This volume would be shared among all containers, with each container configured to output log data to a specific path.

e.g. assuming you had followed the instructions in the link above and created a data volume container called 'logdata'. You could then run the scalyr-agent container with the following command:

docker run -d --name scalyr-docker-agent -p 601:601 -v /tmp/api_key.json:/etc/scalyr-agent-2/agent.d/api_key.json --volumes-from logdata -v /var/run/docker.sock:/var/scalyr/docker.sock  scalyr/scalyr-docker-agent

And configure the Scalyr Agent to track any log files of interest that existed in logdata’s volumes.

Fixing issue with Docker 1.13

If you are using Docker version 1.13 or after, you will need to make a configuration change to allow for your container logs to be uploaded to Scalyr. This is because the 1.13 release changed the default values for some options that the Scalyr Docker Agent relies on. We will be releasing a new version of the Scalyr Docker Agent that will fix this issue, but for now, you will need to make the following configuration change.

Please follow the instructions above to modify your Scalyr Agent's configuration. You need to modify the agent.d/docker.json file to add in the line below that beings with docker_regex:

{
  "monitors": [
    {
      "module": "scalyr_agent.builtin_monitors.syslog_monitor",
      "mode": "docker",
      "docker_regex": "^.*([a-z0-9]{12})\\[\\d+\\]: "
    }, {
      "module": "scalyr_agent.builtin_monitors.docker_monitor",
      "log_mode": "syslog"
    }
  ]
}

Be sure to save your configuration file changes back to the running Scalyr Agent using the above instructions. Once that is done, you should see your container logs being upload to Scalyr.