Configuring Scalyr Agent (Docker)

When running in a Dockerized environment, the Scalyr Agent monitors standard output and error from other containers. This is achieved using the Docker Monitor Plugin.

Docker containers are dynamically assigned hash-based IDs upon start. Arbitrary dynamic IDs make it difficult to define rules that reliably match specific container logs. Managing these rules in agent.json or agent.d snippets becomes impractical.

A more practical approach is to maintain container-specific rules that are co-located with the scripts or commands that start up your respective containers. This is made possible through Docker labels.

Label-Based Log Config

The recommended approach for configuring container-specific settings is with Docker labels. Docker labels let you control how logs are imported from containers by the Scalyr Agent. For example, for a specific container log, you can set the parser or sampling rate (percentage of logs lines) to upload.

Here's how it works: the agent takes any container label that begins with com.scalyr.config.log.. It then maps the label to the corresponding log_config stanza option (minus the com.scalyr.config.log. prefix).

Note: This mapping is specific to that container — there is only one log file per Docker container (capturing all standard output and error). Therefore, you must apply labels to every container you want the settings to apply to. You can also place all configuration labels in a single text file (e.g. labels.txt) and then apply the file when running the docker container.

For example, if you add the following label to your container:

com.scalyr.config.log.parser=accessLog

then the Scalyr Agent automatically adds the following option to that container's log_config:

{ "parser": "accessLog" }

This feature is enabled by default. To turn it off, set the use_labels_for_log_config option to false in the docker_monitor configuration and the agent will not map container labels to log_config stanzas.

The following log_config fields can be configured via container labels and behave as described in the Scalyr help docs:

  • parser
  • attributes
  • sampling_rules
  • rename_logfile
  • redaction_rules

Note: Keys for docker labels cannot include underscores and should be replaced with hyphens. The agent automatically converts hyphens to underscores. E.g. the label: com.scalyr.config.log.rename-logfile converts to rename_logfile.

See sampling and redaction rules for more on these options, and see the next section for rules on mapping labels to log_config options.

Label mapping rules

The rules for mapping container labels to log_config options are as follows:

Values separated by a period are mapped to object keys, e.g., if a label on a given container is specified as:

com.scalyr.config.log.attributes.tier=prod

it will be mapped to the following object, which is then applied to the log_config stanza for that container:

{ "attributes": { "tier": "prod" } }

Arrays can be specified by using one or more digits as the key, e.g. if the labels are:

com.scalyr.config.log.sampling-rules.0.match-expression=INFO
com.scalyr.config.log.sampling-rules.0.sampling-rate=0.1
com.scalyr.config.log.sampling-rules.1.match-expression=FINE
com.scalyr.config.log.sampling-rules.1.sampling-rate=0

This will be mapped to the following structure:

{
  "sampling_rules":
  [
    { "match_expression": "INFO", "sampling_rate": 0.1 },
    { "match_expression": "FINE", "sampling_rate": 0 }
  ]
}

Note again the automatic conversion of hyphens in docker label keys to underscores.

Array keys are sorted by numeric order before processing and unique objects must have different array keys. Duplicate array keys will result in earlier values being overwritten. There is no guarantee about the order of processing for items with the same numeric array key.

Attaching labels as Attributes

You can optionally configure the Scalyr Agent to automatically attach any Docker labels you have set on your container to the log lines from that container. For example, suppose you add a Docker label called tier=prod to all of your production containers. The agent can automatically retrieve this label and attach it to all log lines for those containers (as well as other similar labels such as tier=staging).

This feature is turned off by default so you must turn it on by setting labels_as_attributes to true. Note: By default, any labels used for configuring logs are ignored by the labels_as_attributes feature and will not be attached as attributes.

Here are more config variables that control the attaching of labels to log lines:

  • labels_as_attributes - When true, upload labels as log attributes for the logs generated by the container. Labels must satisfy the include/exclude filters (see below) . Defaults to false.
  • label_include_globs - A list of glob strings for including labels to be attached as log attributes. Any label that matches any glob in this list becomes an attribute as long as it not excluded by label_exclude_globs. Defaults to ['*'] (everything).
  • label_exclude_globs - A list of glob strings for excluding labels from being attached as log attributes. Any label that matches any glob on this list will be excluded. Exclusion rules are applied after inclusion rules. Defaults to ['com.scalyr.config.*']
  • label_prefix - A string to add to the beginning of any label key before attaching it as a log attribute. E.g. if the value for label_prefix is docker_ then the container labels app and tier will be attached to log lines as attribute keys docker_app and docker_tier respectively. Defaults to empty string.

The above config variables are environment-aware, meaning you can set them by passing them in as environment variables at container launch time.

Here's an example docker run command that starts a Scalyr Agent container that will attach the labels tier, app and region, prefixed by dl_:

docker run -d --name scalyr-docker-agent \
-e SCALYR_API_KEY=<Your API Key> \
-e SCALYR_LABELS_AS_ATTRIBUTES=true \
-e SCALYR_LABEL_INCLUDE_GLOBS=tier,app,region \
-e SCALYR_LABEL_PREFIX=dl_ \
-v /var/run/docker.sock:/var/scalyr/docker.sock \
-v /var/lib/docker/containers:/var/lib/docker/containers \
scalyr/scalyr-agent-docker-json:latest

And here's an example command that starts a barebones alpine container that writes to standard out. This container defines 4 labels:

docker run -d -name barebones-container \
-l tier=production \
-l app=nginx \
-l region=us \
-l unimportant-label=some-value \
alpine \
/bin/sh -c 'while true; do echo Hello World!; sleep 1; done'

The result: all log rows from the alpine container will have the searchable fields dl_tier=production, dl_app=nginx and region=us when viewed in the Scalyr UI. Notice however that the label unimportant-label is ignored by the Scalyr Agent because it does not match any of label_include_globs. Thus, using labels is a powerful and convenient way to classify container logs in a multi-container environment.

Warning: These attributes contribute towards your total log volume, so it is wise to limit them to a small set of approved values to avoid paying for attributes you don't need.

Modifying configuration files

Many config variables are environment-aware. Therefore, you can pass them as environment variables in the docker run command. However, because environment variables cannot be changed for a running process you must restart the Scalyr Agent container to reflect any changes.

In case you need to modify config variables without restarting the Scalyr Docker Agent, we have created a tool that helps you export the agent.json and agent.d config files from a live Scalyr Agent container, and then writes these files back once you have modified them.

To extract the config files from the running Scalyr Agent, execute the following commands:

mkdir /tmp/scalyr-agent-config
cd /tmp/scalyr-agent-config
docker exec -i scalyr-docker-agent scalyr-agent-2-config --export-config - | tar -xz

After these commands finish, your current directory will contain one file agent.json and a directory agent.d. agent.json is a copy of the running Scalyr Agent's /etc/scalyr-agent-2/agent.json config file. Likewise, the agent.d directory is a copy of the running agent's /etc/scalyr-agent-2/agent.d directory.

Edit these files to suit your needs. To persist the changes when finished, execute this command:

tar -zc agent.json agent.d/* | docker exec -i scalyr-docker-agent scalyr-agent-2-config --import-config -

There is no need to restart the Scalyr Agent after writing back the config files. The running agent will notice the new config files and read them within 30 seconds.

Creating Custom Scalyr Agent Docker Images

Modifying config files for one container is relatively simple in Docker, but can become burdensome when you wish to reuse the same configuration across multiple instances. To ease this burden, we have provided tools for creating new Scalyr Agent Docker images with your custom config files.

Using a running Scalyr Agent as a "template", create a new Docker image by executing the following commands:

mkdir /tmp/scalyr-docker-agent
cd /tmp/scalyr-docker-agent
docker exec -i scalyr-docker-agent scalyr-agent-2-config --docker-create-custom-dockerfile - | tar -xz
docker build -t customized-scalyr-docker-agent .

This will leave a new Docker image on your local Docker instance with the repository customized-scalyr-docker-agent. You can change this using the docker tag command. Thereafter, use any standard method for distributing your Docker image to your other servers.

Then launch a new Scalyr Agent container that uses this image by executing:

docker run -d --name scalyr-docker-agent \
-v /var/run/docker.sock:/var/scalyr/docker.sock \
-v /var/lib/docker/containers:/var/lib/docker/containers \
customized-scalyr-docker-agent

Setting a Server Hostname

By default, the Scalyr Agent's container id is used as the server host name for all logs it uploads. This is not ideal as the id can change over time, such as when you upgrade to a new version of the Scalyr Agent.

To avoid this issue, you may manually set a server host name for the container. To do this, follow the instructions above for modifying configuration files and create a new file agent.d/server_host.json with the following contents:

{
  "server_attributes": {
    "serverHost": "my-docker-host-01"
  }
}

Then, use the commands for writing the configuration back to the running Scalyr Agent, and subsequent logs should begin appearing under the new server host name within 30 seconds.

You may also use the environment variable import feature to avoid having to create a unique server_host.json file for each server host name. This feature allows you to import variables from the environment and use them to define fields in a Scalyr configuration file.

For example, you could have defined your agent.d/server_host.json file with the following contents:

{
  "import_vars": [ "DOCKER_HOST_NAME" ],

  "server_attributes": {
    "serverHost": "$DOCKER_HOST_NAME"
  }
}

In this example, the environment variable DOCKER_HOST_NAME is used to define the server host name. This is particularly useful if you supply these environment variables using the -e Docker option when you launch the container. For example, you can follow instructions above for creating a custom image with this configuration and then execute this command:

docker run -d -e DOCKER_HOST_NAME='my-host' \
--name scalyr-docker-agent \
-v /var/run/docker.sock:/var/scalyr/docker.sock \
-p 601:601 customized-scalyr-docker-agent

Customizing Log File Names

By default, all docker logs gathered by the recommended json logging driver are uploaded with file names like:

/docker/MyContainerName.log

rather than the full path on disk which will be something like:

/var/lib/docker/containers/377c8ff5b9853c7a47d0d0cba605a4389283cd6f4278cd34d1a928c78cafe618/377c8ff5b9853c7a47d0d0cba605a4389283cd6f4278cd34d1a928c78cafe618-json.log

You can customize the name of this log file by applying a label to the container as follows:

com.scalyr.config.log.rename-logfile=/scalyr/my-custom-name.log

Thus, the command to start the container with this label would be:

docker run -d --name webserver -l com.scalyr.config.log.rename-logfile=/scalyr/my-custom-name.log ...

Alternatively, you can define all labels in a text file (one label per line) and specify that file at run time:

docker run -d --name webserver --label-file label-file.txt ...

After doing this, logs for this container will appear in the Scalyr UI under the logfile /scalyr/my-custom-name.log.

Setting parsers, Sampling and Redaction rules

By default, all logs sent to the Scalyr Agent container are configured to use specific parsers:

  • The default parser is docker when using the recommended json logging driver
  • The default parser is agentSyslogDocker when using the syslog logging driver

You may override the default parser. The recommended approach for configuring this container-specific setting is through Docker labels.

To do so, add the following label to the container whose parser you wish to set.

com.scalyr.config.log.attributes.parser=frontendLog

Thus, the command to start the container with this label would be:

docker run -d --name frontend -l com.scalyr.config.log.attributes.parser=frontendLog ...

Alternatively, you can define all labels in a text file (one label per line) and specify that file at run time:

docker run -d --name frontend --label-file label-file.txt ...

This sets the frontendLog as the parser for all containers which have this label defined.

You can use this technique to also specify sampling and redaction rules. See the documentation on sampling and redaction rules for more information.

  • When customizing log file names through labels, avoid defining log config stanzas in agent.json or agent.d snippets that match the same log path as the container with the label. Otherwise, the config stanzas in agent.json or agent.d will override the label-based settings, thereby causing the label-based settings to be ignored.

Starting the Scalyr Agent Container

The typical command to start the Scalyr Agent command is below:

docker run -d --name scalyr-docker-agent \
-e SCALYR_API_KEY=<Your API key> \
-v /var/run/docker.sock:/var/scalyr/docker.sock \
-v /var/lib/docker/containers:/var/lib/docker/containers \
scalyr/scalyr-docker-agent-json:latest

For reference, here is a description of the options:

  • -d is to run in detached mode (needed when running a daemon process such as the scalyr-agent)
  • --name scalyr-docker-agent names this container 'scalyr-docker-agent'. It is useful to give a fixed name to the Scalyr Agent container for easy reference.
  • -e SCALYR_API_KEY=<Your API key> sets the write API_KEY that allows the Scalyr Agent to upload logs. Alternatively, you could map a modular snippet into the agent.d directory using -v /tmp/api_key.json:/etc/scalyr-agent-2/agent.d/api_key.json. Finally, you could also creating a custom Docker image where your API key is materialized as a modular snippet or in the agent.json
  • scalyr/scalyr-docker-agent-json:latest specifies the image (and version) to run.
  • -v /var/run/docker.sock:/var/scalyr/docker.sock maps the Docker API socket to a path in the container. The Scalyr Agent requires access to the Docker API socket to collect metrics and determine container names.
  • -v /var/lib/docker/containers:/var/lib/docker/containers maps the Docker containers directory, which is how the Scalyr Agent locates JSON log files managed by Docker (if using the preferred JSON file logging driver). If using the Syslog logging driver, you would instead specify -p 601:601 to map the Syslog ports.

Stopping the Scalyr Agent container

You can stop and restart the container as a way to stop and restart the Scalyr Agent. This allows you to stop the agent temporarily, and then start it back up again with all the same settings (and same filesystem, with checkpoints for recently uploaded logs).

Assuming you have a container named scalyr-docker-agent running, you can stop it with the following command:

docker stop scalyr-docker-agent-json

Restarting the Scalyr Agent container

Assuming you have previously stopped a container named scalyr-docker-agent, you can restart it with the command

docker start scalyr-docker-agent-json

Checking the Status of a Running Scalyr Agent Container

To check the status of a currently running Scalyr Agent container, use the following command:

docker exec scalyr-docker-agent-json scalyr-agent-2 status

or

docker exec scalyr-docker-agent-json scalyr-agent-2 status -v

to get more verbose output.

Application Logs from Other Containers

Output to standard output and error are gathered by Docker and immediately seen by the Scalyr Agent. However, to gather other application log files, you must share volumes containing those log files with the Scalyr Agent container. You can then configure the Scalyr Agent as normal to upload those log files.

There are numerous ways to share volumes from other containers. See the Docker documentation on managing data in containers

As a simple example, imagine you have an Apache container configured to log all access at /var/log/httpd/access.log. When running the Scalyr Agent container you would simply map that file using the -v command line parameter:

-v /var/log/httpd/access.log:/var/log/httpd/access.log

The full command to run the container would be:

docker run -d --name scalyr-docker-agent \
-e SCALYR_API_KEY=<Your API key> \
-v /var/run/docker.sock:/var/scalyr/docker.sock \
-v /var/lib/docker/containers:/var/lib/docker/containers \
-v /var/log/httpd/access.log:/var/log/httpd/access.log \
scalyr/scalyr-agent-docker-json

Then export your configuration files and create a agent.d/more_logs.json file with the following content:

{
  "logs": [
    {
      "path": "/var/log/httpd/access.log",
      "attributes": {parser: "accessLog"}
    }
  ]
}

The Scalyr Agent will automatically start tracking and uploading this file after you re-import the configuration to the running agent.

If you have multiple containers, each with files you wish to upload, you should probably consider creating a Data Volume Container.

This volume would be shared among all containers, with each container configured to output log data to a unique path.

E.g. assuming you followed the instructions in the link above and created a data volume container called 'logdata'. You could then run the scalyr-agent container with the following command:

docker run -d --name scalyr-docker-agent \
-e SCALYR_API_KEY=<Your API key> \
-v /var/run/docker.sock:/var/scalyr/docker.sock \
-v /var/lib/docker/containers:/var/lib/docker/containers \
-v /var/log/httpd/access.log:/var/log/httpd/access.log \
--volumes-from logdata \
scalyr/scalyr-docker-agent-json

And then configure the agent to track any log files of interest that exist in logdata’s volumes.

Conclusion

You may also be interested in general instructions on setting up Log Parsers. With properly parsed logs, Scalyr becomes an even more powerful tool for analysis and visualization.

For complete documentation on agent configuration options, see the Agent Reference.