Install Scalyr Agent (Kubernetes)

This document describes the recommended method for importing logs from a Kubernetes cluster to Scalyr.

These instructions are for installing the Scalyr Agent as a DaemonSet on your Kubernetes cluster. If you plan to run the Agent directly on Linux, see the Linux installation page. For Windows, see the Windows installation page, and for running the Agent separately in a Docker container, see the Docker installation page.

The Scalyr Agent is a daemon that uploads logs and system metrics to the Scalyr servers. This page provides streamlined instructions to get you up and running quickly in a Kubernetes environment.

In our recommended architecture, you will run the Scalyr Agent as a DaemonSet on your cluster. The DaemonSet runs a Scalyr Agent pod on each node in your cluster that will collect logs and metrics from all other pods running on the same node. Note that this will run the agent on the master node; if you do not want to run the agent on the master, comment the following in https://github.com/scalyr/scalyr-agent-2/blob/master/k8s/scalyr-agent-2.yaml:

tolerations:
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"

To create and launch the Scalyr Agent DaemonSet, please do the following:

1. Configure a secret that contains your "Write Logs" Scalyr API key

kubectl create secret generic scalyr-api-key --from-literal=scalyr-api-key="<your scalyr api key>"

2. Launch the DaemonSet

kubectl create -f https://raw.githubusercontent.com/scalyr/scalyr-agent-2/master/k8s/scalyr-agent-2.yaml

That's It!

We hope that was easy. If you've had any trouble, please [let us know](support@scalyr.com). Otherwise, if this is your first time using Scalyr, this would be an excellent time to head on to the Getting Started guide.

You should also check out the Log Parsing page to set up a parser for your logs. Scalyr becomes an even more powerful tool for analysis and visualization when your logs are properly parsed.

For complete documentation on agent configuration options, see the agent reference.

For more details on creating custom images with your configuration files, modifying the configuration files in the agent, setting a server host name, assigning parsers, etc, read on...

Further Reading

Configuration Using Annotations

Starting in the 2.0.36 release of the Scalyr Agent, you can now use k8s pod annotations to control how the Scalyr Agent will import your Kubernetes logs. This greatly simplifies performing common tasks such as assigning parser names to a pod's logs as well as turning on sampling and redaction rules.

With this support, the Scalyr Agent automatically reads any pod annotation that begins with the prefix log.config.scalyr.com/ and maps it to the corresponding option in the log_config stanza for that pod's container (minus the prefix). These annotations can be added to a pod either via the kubectl annotate command or by modifying your Kubernetes YAML configuration files. See the Kubernetes annotation documentation for more information.

For example, if you add the following annotation to your pod:

log.config.scalyr.com/attributes.parser: accessLog

The Scalyr Agent will automatically use the following for that pod's log_config stanza:

{ "attributes": { "parser": "accessLog" } }

More details are given below on how the annotations are mapped to configuration options.

The following fields can be configured a log via pod annotations and behave the same as described in the Scalyr help docs:

  • parser
  • attributes
  • sampling_rules
  • rename_logfile
  • redaction_rules

The following configuration fields are support but behave differently when configured via k8s annotations:

  • exclude (see below)
  • lineGroupers (not supported at all)
  • path (the path is always fixed for k8s container logs)

Mapping Configuration Options

The Kubernetes monitor takes the string value of each annotation that begins with prefix log.config.scalyr.com/ and maps it to a dict, or array value according to the following format:

Values separated by a period are mapped to dict keys e.g. if one annotation on a given pod was specified as:

  log.config.scalyr.com/server_attributes.tier: prod

Then this would be mapped to the following dict, which would then be applied to the log config for all containers in that pod:

{ "server_attributes": { "tier": "prod" } }

Arrays can be specified by using one or more digits as the key, e.g. if the annotations were

  log.config.scalyr.com/sampling_rules.0.match_expression: INFO
  log.config.scalyr.com/sampling_rules.0.sampling_rate: 0.1
  log.config.scalyr.com/sampling_rules.1.match_expression: FINE
  log.config.scalyr.com/sampling_rules.1.sampling_rate: 0

This will be mapped to the following structure:

{ "sampling_rules":
  [
    { "match_expression": "INFO", "sampling_rate": 0.1 },
    { "match_expression": "FINE", "sampling_rate": 0 }
  ]
}

Array keys are sorted by numeric order before processing and unique objects need to have different digits as the array key. If a sub-key has an identical array key as a previously seen sub-key, then the previous value of the sub-key is overwritten

There is no guarantee about the order of processing for items with the same numeric array key, so if the config was specified as:

  log.config.scalyr.com/sampling_rules.0.match_expression: INFO
  log.config.scalyr.com/sampling_rules.0.match_expression: FINE

It is not defined or guaranteed what the actual value will be (INFO or FINE).

Excluding Logs

Containers and pods can be specifically included/excluded from having their logs collected and sent to Scalyr. Unlike the normal log_config exclude option which takes an array of log path exclusion globs, annotations simply support a Boolean true/false for a given container/pod. Both include and `exclude` are supported, with include always overriding exclude if both are set. e.g.

log.config.scalyr.com/exclude: true

has the same effect as

log.config.scalyr.com/include: false

By default the agent monitors the logs of all pods/containers, and you have to manually exclude pods/containers you don't want. You can also set a value in agent.json k8s_include_all_containers: false, in which case all containers are excluded by default and have to be manually included.

Applying Config Options to Specific Containers in a Pod

If a pod has multiple containers and you only want to apply log configuration options to a specific container you can do so by prefixing the option with the container name, e.g. if you had a pod with two containers nginx and helper1 and you wanted to exclude helper1 logs you could specify the following annotation:

log.config.scalyr.com/helper1.exclude: true

Config items specified without a container name are applied to all containers in the pod, but container specific settings will override pod-level options, e.g. in this example:

log.config.scalyr.com/exclude: true
log.config.scalyr.com/nginx.include: true

All containers in the pod would be excluded except for the nginx container, which is included.

This technique is applicable for all log config options, not just include/exclude. For example you could set the line sampling rules for all containers in a pod, but use a different set of line sampling rules for one specific container in the pod if needed.

Dynamic Updates

Currently all annotation config options except exclude: true`/`include: false can be dynamically updated using the kubectl annotate command.

For exclude: true`/`include: false once a pod/container has started being logged, then while the container is still running, there is currently no way to dynamically start/stop logging of that container using annotations without updating the config yaml, and applying the updated config to the cluster.

Creating Custom Scalyr Agent Configurations

The default Kubernetes configuration file for the Scalyr Agent does the following:

  • Creates a DaemonSet running the scalyr/scalyr-k8s-agent:latest container image (this image is available on Dockerhub). This image differs from our standard docker container scalyr-docker-agent in that it has been configured to read the raw container logs from the Kubernetes node rather than relying on syslog or the Docker API.
  • Maps /var/lib/docker/containers of the node to /var/lib/docker/containers on the scalyr-k8s-agent container. This gives the container access to the raw logs from other pods and containers running on that node.
  • Exposes your "Write Logs" Scalyr API key to the container in the environment variable SCALYR_API_KEY. This is required by the default scalyr configuration file of the scalyr-k8s-agent image.

You can see the full configuration file here.

Creating Custom Scalyr Agent Docker Images

You can turn on several useful features in the Scalyr Agent by modifying its configuration files. As discussed above, it is often easier to use the annotation-based configuration options. However, sometimes you may wish to include your configuration options by default in your own custom Docker image to ease deployment. If you wish to take this approach, we have provided tools to allow you to easily create new Scalyr Agent Docker images that include your custom configuration files, which you can then run on your Kubernetes cluster.

You can create a test a configuration on a single Docker container and once it is working to your satisfaction, you can use these tools to export the configuration along with a Dockerfile that will build a custom image that uses your custom configuration.

Assuming you have created and tested your configuration changes on a standalone docker container called 'scalyr-agent' (based off the scalyr/scalyr-k8s-agent:latest image) you can create a custom Docker image based on that configuration by executing the following commands on the currently running container:

mkdir /tmp/scalyr-agent
cd /tmp/scalyr-agent
docker exec -i scalyr-agent scalyr-agent-2-config --k8s-create-custom-dockerfile - | tar -xz
docker build -t customized-scalyr-k8s-agent .

This will leave a new Docker image on your local Docker instance with the repository tag customized-scalyr-k8s-agent. You can change the name using the docker tag command. From there, you can use any of the standard methods to make this container available to your Kubernetes cluster.

You can launch a DaemonSet that uses the new Scalyr Agent container by changing the image specified in your Kubernetes config file:

spec:
  template:
      spec:
        containers:
        - name: scalyr-agent
          image: customized-scalyr-k8s-agent

and then launching the Daemonset based on the new configuration file

kubectl create -f my-custom-configuration.yaml

Testing your Custom Configuration Locally

Before deploying a custom Scalyr Agent image to your cluster, it's a good idea to test it locally first to make sure there are no configuration errors or problems.

This can be done via Docker with the following command:

docker run -ti -e SCALYR_API_KEY="$SCALYR_API_KEY" \
  --name scalyr-k8s-agent \
  -v /var/lib/docker/containers:/var/lib/docker/containers \
  -v /var/run/docker.sock:/var/scalyr/docker.sock \
  customized-scalyr-k8s-agent /bin/bash

Which will launch a single container based on your customized image and drop you in to a bash shell.

Make sure to replace $SCALYR_API_KEY with your scalyr api key (or export it to an environment variable called $SCALYR_API_KEY before running the above command in order to expose your api key to the container.

Then from the shell prompt you can manually launch the Scalyr agent by executing:

scalyr-agent-2 start

If everything is configured correctly, you should see a message similar to 'Configuration and server connection verified, starting agent in background.'

Once you can confirm that the scalyr-agent runs correctly and is uploading logs to the Scalyr server, then you are ready to run that container on your Kubernetes cluster.

Modifying Configuration Files

If you want to modify the configuration of the Scalyr Agent on the fly, you need to create and enable a ConfigMap.

You can create a ConfigMap based on an existing Scalyr Agent's configuration. For example, if you had a test container called scalyr-k8s-agent running on Docker as a standalone container you can run the following commands to export its configuration files:

mkdir /tmp/scalyr-agent-config
cd /tmp/scalyr-agent-config
docker exec -i scalyr-k8s-agent scalyr-agent-2-config --export-config - | tar -xz

After these commands finish, your current directory will have one file agent.json and a directory agent.d. The agent.json file is a copy of the running Scalyr Agent's /etc/scalyr-agent-2/agent.json configuration file. Likewise, the agent.d directory is a copy of the /etc/scalyr-agent-2/agent.d directory.

Note: It's important to run this command on a container based off the scalyr /scalyr-k8s-agent rather than scalyr/scalyr-docker-agent in order to have the correct default configuration.

You can then edit those files to make whatever changes you need and write the changes back to the container for further testing, with this command:

tar -zc agent.json agent.d/* | docker exec -i scalyr-k8s-agent scalyr-agent-2-config --import-config -

There is no need to restart the Scalyr Agent after writing the configuration files. The running Scalyr Agent should notice the new configuration files and read them within 30 seconds.

Once you have configured the Agent to your needs, and confirmed the configuration changes are working as expected, export the configuration files once more, and in the exported directory (which will contain a single agent.json file and an agent.d/ directory) you can execute the following command to create a ConfigMap called scalyr-config

kubectl create configmap scalyr-config $(for i in $(find .); do echo "--from-file=\"$i\""; done)

This creates a single config map containing all files in the current directory and any subdirectories.

Once you have created a ConfigMap, you need to add it to the DaemonSet's configuration. This can be done as follows:

spec:
  template:
    spec:
      containers:
        volumeMounts:
        - name: scalyr-config
          mountPath: /etc/scalyr-agent-2
      volumes:
        - name: scalyr-config
          configMap:
            name: scalyr-config

This will write the contents of each of the files in the scalyr-config ConfigMap to a volume mount located in the scalyr-agent-2 configuration directory, overwriting any existing files.

Note: If you specify a ConfigMap as a volume mount in your DaemonSet configuration, but you have not yet created that ConfigMap, then Kubernetes will not start any pods that require the ConfigMap until it has been created.

Once you have configured the DaemonSet to use ConfigMap, then when you wish to update the configuration, and assuming once again you are in a directory containing an exported Scalyr Agent configuration (that is a directory with a single agent.json file and an agent.d/ directory) you can replace the contents of ConfigMap by executing:

kubectl create configmap scalyr-config $(for i in $(find .); do echo "--from-file=\"$i\""; done) -o yaml --dry-run | kubectl replace -f -

The changes will manifest on all nodes running the DaemonSet that are configured to use the ConfigMap, and once the configuration files have changed the Scalyr Agent will automatically detect these changes and start using the new configuration.

Stopping the Scalyr Agent DaemonSet

If you wish to stop running the Scalyr Agent DaemonSet entirely then assuming you are using the default configuration and your DaemonSet's name is 'scalyr-agent-2' you can run:

kubectl delete daemonset scalyr-agent-2

There is currently no convenient way to temporarily stop the Scalyr Agent already running on a specific node. If you wish to avoid running the Scalyr Agent on a specific node in your cluster, then see the section on Running the DaemonSet on only some nodes.

Checking the Status of a Node Running the Scalyr Agent

To check the status of a currently running Scalyr Agent container, first use kubectl to find the name of the pod you are interested in, e.g.:

kubectl get pods

And once you have the name of the pod that is running the Scalyr Agent use the following command:

kubectl exec scalyr-pod-name — scalyr-docker-agent scalyr-agent-2 status

For more verbose output, run:

kubectl exec scalyr-pod-name — scalyr-docker-agent scalyr-agent-2 status -v

Running the Daemonset on only some Nodes

If you would like to run the Scalyr Agent only on certain nodes of your cluster, you can do so by adding a nodeSelector or a nodeAffinity section to your config file. For example, if the nodes that you wanted to run the Scalyr Agent on had a label 'logging=scalyr' then you could add the following nodeSelector to your configuration file:

spec:
  nodeSelector:
    logging: scalyr

or if you wanted to use nodeAffinity:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: logging
            operator: In
            values: ["scalyr"]

You can read more about specifying node selectors and affinity here.