Configuring Scalyr Agent (Kubernetes)

You should hopefully have already seen how easy it is to install Scalyr in a Kubernetes cluster. You might find that this basic installation is all you need. This document goes into further detail on how you might refine configuration of Scalyr Agents running in your Kubernetes cluster.

The Scalyr Agent monitors Kubernetes clusters through the Kubernetes monitor Plugin.

The agent implements the Kubernetes-recommended cluster-level logging architecture. Specifically, a node-level logging agent runs on each node, implemented as a DaemonSet replica. Each agent captures standard output and standard error from other pods within the same node and uploads these logs to the Scalyr backend.

Configuration falls into two broad categories:

A) Configuring Pod Logs

Configuring the monitoring of individual Pod logs is achieved through Annotations.

This pertains to how logs from specific pods are handled by the Agent. The Agent examines annotations on each pod, and for any annotation that begins with the prefix log.config.scalyr.com/, it extracts the entries (minus the prefix) and maps them to the log_config stanza for that pod's containers.

The following log_config fields can be configured via pod annotations and behave in the same way as specified in the main Scalyr Agent reference:

  • parser
  • attributes
  • sampling_rules
  • rename_logfile
  • redaction_rules

However, the following log_config fields behave differently when configured via k8s annotations:

  • exclude (see "Excluding Logs" below)
  • lineGroupers (not supported at all)
  • path (the path is always fixed for k8s container logs)

B) Configuring the Scalyr Agent

In a Kubernetes environment, configuration of the Scalyr Agent is achieved using ConfigMaps. This relates to configuring behavior other than the handling of individual pod logs.

Examples are:

  • setting the write API key
  • setting the Kubernetes cluster name
  • configuring the Kubernetes API cache expiration interval

===================================================================================================

A) Configuring Pod Logs with Annotations

Starting in the 2.0.36 Scalyr Agent release, you can use k8s pod annotations to control how the agent imports Kubernetes logs. This greatly simplifies common tasks such as assigning parser names to a pod's logs as well as turning on sampling and redaction rules. Annotations are the Scalyr-recommended method of configuring pod logs and other resources not intrinsic to the Scalyr Agent.

Scalyr Agent automatically reads any pod annotation that begins with the prefix log.config.scalyr.com/ and maps it to the corresponding option in the log_config stanza for that pod's container (minus the log.config.scalyr.com/ prefix). These annotations can be added to a pod either via the kubectl annotate command or by modifying your Kubernetes YAML configuration files. See the Kubernetes annotation documentation for more information.

For example, if you add the following annotation to your pod:

log.config.scalyr.com/attributes.parser: accessLog

The Scalyr Agent will automatically insert the following fields into the log_config stanza for the pod:

{ "attributes": { "parser": "accessLog" } }

What you can do with Annotations

Here's what you can configure with annotations:

  • Setting a Parser
  • Excluding Logs
  • Sampling High-volume Logs
  • Attaching Attributes
  • Redaction Rules

Setting Parsers

You can add a custom parser to a pod in a Kubernetes cluster. Read more about applying parsers to the Scalyr Agent here.

Annotation

log.config.scalyr.com/attributes.parser: "custom-parser"

CLI

kubectl annotate pod <pod-name> --overwrite log.config.scalyr.com/attributes.parser=custom-parser

YAML

metadata:
    annotations:
      "log.config.scalyr.com/attributes.parser": "custom-parser"

Excluding Logs

Containers and pods can be specifically included/excluded from having their logs collected and sent to Scalyr. Unlike the normal log_config exclude option which takes an array of log path exclusion globs, Annotations simply support a Boolean true/false for a given container/pod. Both include and exclude are supported, with include always overriding exclude if both are set. e.g.

log.config.scalyr.com/exclude: true

has the same effect as:

log.config.scalyr.com/include: false

By default the agent monitors the logs of all pods/containers and you have to manually exclude pods/containers you don't want.

You can also set k8s_include_all_containers: false in the kubernetes_monitor config section of agent.d/docker.json, in which case all containers are excluded by default and have to be manually included.

Annotation

log.config.scalyr.com/include:  false
log.config.scalyr.com/exclude:  true

CLI

Command line configuration of log exclusion is not supported.

YAML

metadata:
   annotations:
      "log.config.scalyr.com/exclude": "true/false"

Sampling High-volume Logs

You can read more about log sampling here.

Annotation

log.config.scalyr.com/sampling_rules.0.match_expression: INFO
log.config.scalyr.com/sampling_rules.0.sampling_rate: 0.1
log.config.scalyr.com/sampling_rules.1.match_expression: FINE
log.config.scalyr.com/sampling_rules.1.sampling_rate: 0

CLI

kubectl annotate pod <pod name> --overwrite log.config.scalyr.com/sampling_rules.0.match_expression=INFO
kubectl annotate pod <pod name> --overwrite log.config.scalyr.com/sampling_rules.0.sampling_rate=0.1
kubectl annotate pod <pod name> --overwrite log.config.scalyr.com/sampling_rules.1.match_expression=FINE
kubectl annotate pod <pod name> --overwrite log.config.scalyr.com/sampling_rules.1.sampling_rate=0

YAML

metadata:
   annotations:
      "log.config.scalyr.com/sampling_rules.0.match_expression": "INFO",
      "log.config.scalyr.com/sampling_rules.0.sampling_rate": "0.1",
      "log.config.scalyr.com/sampling_rules.1.match_expression": "FINE",
      "log.config.scalyr.com/sampling_rules.1.sampling_rate": "0"

Attaching Attributes

You can specify additional fields in attributes. This allows you to distinguish between different pods. For instance, you might give the pod a field like service: "database" to identify the service that generates this log; you could then add $service = "database" to any query to restrict your search to these database logs.

Annotation

log.config.scalyr.com/attributes.key: value

CLI

kubectl annotate pod <pod name> --overwrite log.config.scalyr.com/attributes.key=value

YAML

metadata:
   annotations:
      "log.config.scalyr.com/attributes.key": "value"

Redaction Rules

You can read more about log redaction here.

The following is an example stanza for defining redaction rules. It replace terms like "userInfo=username password" with "userInfo=username".

Annotation

"log.config.scalyr.com/redaction_rules.match_expression": "userInfo=([^ ]+) [^ ]+"
"log.config.scalyr.com/redaction_rules.replacement": "userInfo=\\1"

CLI

kubectl annotate pod <pod name> --overwrite log.config.scalyr.com/redaction_rules.0.match_expression='userInfo=([^ ]+) [^ ]+'
kubectl annotate pod <pod name> --overwrite log.config.scalyr.com/redaction_rules.0.replacement='userInfo=\\1'

YAML

metadata:
   annotations:
      "log.config.scalyr.com/redaction_rules.0.match_expression": "userInfo=([^ ]+) [^ ]+"
      "log.config.scalyr.com/redaction_rules.0.replacement": "userInfo=\\1"

Renaming Logs

You may also wish to rename your log file(s) before they are uploaded to the Scalyr servers. For example, your pod might have a log file /docker/foo.log, but you might want to give it a more meaningful name for searching in Scalyr, e.g., /scalyr/access.log

You can read more about renaming log files here.

Annotation

log.config.scalyr.com/path=/docker/foo.log
log.config.scalyr.com/rename_logfile=/scalyr/access.log

CLI

kubectl annotate pod <pod-name> --overwrite log.config.scalyr.com/path=/docker/foo.log
kubectl annotate pod <pod-name> --overwrite log.config.scalyr.com/rename_logfile=/scalyr/access.log

YAML

metadata:
   annotations:
       "log.config.scalyr.com/path": "/docker/foo.log"
       "log.config.scalyr.com/rename_logfile": "/scalyr/access.log"

The following are not currently supported or have limited support in the Kubernetes Agent

  • lineGroupers
  • path
  • server_attributes

How Annotations are Mapped

The Kubernetes monitor takes the string value of each annotation that begins with prefix log.config.scalyr.com/ and maps it to a dict or array value according to the following rules:

Values separated by a period are mapped to dict keys. E.g., if one annotation on a given pod was specified as:

  log.config.scalyr.com/server_attributes.tier: prod

This is mapped to the following dict, which is then applied to the log config for all containers in that pod:

{ "server_attributes": { "tier": "prod" } }

Arrays can be specified by using one or more digits as the key. E.g. the following annotations:

  log.config.scalyr.com/sampling_rules.0.match_expression: INFO
  log.config.scalyr.com/sampling_rules.0.sampling_rate: 0.1
  log.config.scalyr.com/sampling_rules.1.match_expression: FINE
  log.config.scalyr.com/sampling_rules.1.sampling_rate: 0

Will be mapped to this structure:

{ "sampling_rules":
  [
    { "match_expression": "INFO", "sampling_rate": 0.1 },
    { "match_expression": "FINE", "sampling_rate": 0 }
  ]
}

Array keys are sorted by numeric order before processing and unique objects need to have different digits as the array key. Duplication of keys results in overwriting of the key value, with no guarantee of order of processing.

Therefore, in the following example of duplicate keys:

  log.config.scalyr.com/sampling_rules.0.match_expression: INFO
  log.config.scalyr.com/sampling_rules.0.match_expression: FINE

The resulting value is non-deterministic and could be either INFO or FINE.

Applying Config Options to Specific Containers in a Pod

If a pod has multiple containers and you only want to apply log configuration options to a specific container, do so by prefixing the option with the container name. For example, if you had a pod with two containers nginx and helper1 and you want to exclude helper1 logs you could specify the following annotation:

log.config.scalyr.com/helper1.exclude: true

Config items specified without a container name are applied to all containers in the pod, but container-specific settings will override pod-level options. Take the following example:

log.config.scalyr.com/exclude: true
log.config.scalyr.com/nginx.include: true

All containers in the pod would be excluded except for the nginx container, which is included.

This technique is applicable for all log config options, not just include / exclude. For example you could set the line sampling rules for all containers in a pod, but use a different set of line sampling rules for one specific container in the pod if needed.

Dynamic Annotations

Currently all annotation config options except exclude: true / include: false can be dynamically updated using the kubectl annotate command.

For exclude: true / include: false, once a pod/container has started being logged, for as long as the container is still running, there is currently no way to dynamically start/stop logging of that container using annotations. You must update the config YAML and apply the updated config to the cluster.

===================================================================================================

B) Configuring the K8s Agent with ConfigMaps

Understanding ConfigMaps

Now that we've seen how to configure the monitoring of pod logs with Annotations, we now examine how to configure the Scalyr Agent in a Kubernetes environment.

As we have seen in previous documents, the Scalyr Agent is primarily configured through the local agent.json file or agent.d snippets and can also fall-back to environment variables.

In a Kubernetes environment, it is impractical to login to each Scalyr Agent pod to modify config files. Instead, the main configuration method in Kubernetes is through ConfigMaps. Furthermore, it is more convenient to use the environment-variable fallback in Kubernetes.

ConfigMaps are key-value maps of arbitrary strings. When a pod is started, each key-value can be made available to the pod either as a system environment variable or else materialized as a file in the pod's local file system.

There are many ways to use ConfigMaps but here are two common scenarios:

  • A ConfigMap key-value pair of ABC=123 becomes an environment variable ABC with string value 123.
  • A ConfigMap key-value of abc.txt=123 materializes in the pod's local file system as the file abc.txt containing the text 123.

Having understood ConfigMaps, let's see how the Scalyr Agent may be configured:

Preferred approach: Environment variables through a ConfigMap

The Scalyr Agent reference describes how some config variables are environment-aware, meaning they fall back to a specifically named environment value if not defined in the agent.json file or agent.d snippets.

Configuring through environment variables is convenient in a Kubernetes environment. Customers can define all Scalyr Agent variables in a single Kubernetes ConfigMap that materializes as environment variables system-wide.

Please see the installation instructions for how to configure environment variables through a ConfigMap.

Fallback approach: agent.json & agent.d using a ConfigMap

Not all agent config variables are environment-aware. For those, you must modify agent.json and agent.d snippets. You can edit the config values in these files. Or if you need the flexibility, you could still map-in arbitrarily named environment variables through variable substitution (using import_vars).

Either way, you will need to modify the agent.json and/or agent.d snippets.

Again, it is impractical to have to login to each agent pod to modify config files. But again, ConfigMaps come to our rescue! You use ConfigMaps to store entire file contents of the agent.json and agent.d snippets. The ConfigMap then materializes as local files in each Scalyr Agent pod, thus overwriting the default copies of these files that are built into the image.

To modify Scalyr Agent config files on the fly, first extract a running agent's configuration into a ConfigMap. Then modify that ConfigMap and inject it back by saving the ConfigMap.

Here are the steps:

1. Export your Configuration Files

2. Make your Changes

3. Test your Configuration

4. Create your ConfigMap

5. Modify your DaemonSet

Exporting Your Configuration Files

Here are instructions on how to extract a running Scalyr Agent's configuration.

For example, you have a test container called scalyr-k8s-agent running on Docker as a standalone container. You can run the following commands to export its config files:

mkdir /tmp/scalyr-agent-config
cd /tmp/scalyr-agent-config
docker exec -i scalyr-k8s-agent scalyr-agent-2-config --export-config - | tar -xz

Upon completion, your current directory will have one agent.json file and a directory agent.d. The agent.json file is a copy of the running Scalyr Agent's /etc/scalyr-agent-2/agent.json config file. Likewise, the agent.d directory is a copy of the /etc/scalyr-agent-2/agent.d directory.

Note: To obtain the correct base configuration files, it's important to run this command against a container based off the scalyr/scalyr-k8s-agent image and not the scalyr/scalyr-docker-agent image.

Test Your Configuration

Once extracted, you can then edit those files to suit your needs, then persist the changes back to the container for further testing, with this command:

tar -zc agent.json agent.d/* | docker exec -i scalyr-k8s-agent scalyr-agent-2-config --import-config -

There is no need to restart the Scalyr Agent after saving back the configuration files. The running Scalyr Agent will notice the new config files within 30 seconds.

Create Your ConfigMap

After you have configured the Agent and confirmed the changes are working as expected, export the configuration files to your ConfigMap. Changes to this ConfigMap will reflect the global configuration of the DaemonSet pods.

You will make most of your changes in the agent.d directory. But if you need to modify the agent.json file, please see the next section.

1. Start off in the /agent.d directory (created when you exported your configuration).

cd /tmp/scalyr-agent-config/agent.d

2. Create the agent.d ConfigMap:

kubectl create configmap scalyr-config-agent-d $(for i in $(find . -type f); do echo "--from-file=\"$i\""; done) -o yaml

3. View the ConfigMap to make sure your changes were rendered correctly:

kubectl get configmap scalyr-config-agent-d -o yaml

Example: Your scalyr-config-agent-d ConfigMap should look like this:

data:
  api-key.json: |+
    {
      import_vars: [ "SCALYR_API_KEY" ],
      api_key: "$SCALYR_API_KEY"
    }

  docker.json: |+
    {
      "monitors":[
        {
          "module": "scalyr_agent.builtin_monitors.kubernetes_monitor",
        }
      ]
    }

  k8s_events.json: |+
    {
      "monitors":[
        {
          "module": "scalyr_agent.builtin_monitors.kubernetes_events_monitor"
        }
      ]
    }

  key.json: |+
    {
      import_vars: [ "SCALYR_API_KEY" ],
      api_key: "123412341234"

    }

  scalyr-server.json: |+
    {
      "import_vars": [
        { "var": "SCALYR_SERVER", "default": "https://agent.scalyr.com" }
      ],
      "scalyr_server": "$SCALYR_SERVER"
    }

Modify Your DaemonSet Configuration

Once you have created your ConfigMap, you must add it to the DaemonSet YAML as follows:

spec:
  template:
    spec:
      containers:
        volumeMounts:
        - name: scalyr-config-agent-d
          mountPath: /etc/scalyr-agent-2/agent.d
      volumes:
        - name: scalyr-config-agent-d
          configMap:
            name: scalyr-config-agent-d

This will write the contents of each file in the scalyr-config-agent-d ConfigMap to the volume mount located in the scalyr-agent-2 configuration directory.

Note: If you specify a ConfigMap as a volume mount in your DaemonSet configuration but fail to create the ConfigMap, Kubernetes will not start any pods that require that ConfigMap until it has been created.

Update Your ConfigMap

Once you have configured the DaemonSet to use a ConfigMap and you wish to update the configuration (assuming you're in a directory containing the files you want to modify), you can replace the contents of ConfigMap by executing:

kubectl create configmap <ConfigMap-Name> $(for i in $(find . -type f); do echo "--from-file=\"$i\""; done) -o yaml --dry-run | kubectl replace -f -

The changes will manifest on all nodes running the DaemonSet that are configured to use the ConfigMap, and once the configuration files have changed the Scalyr Agent automatically detects and uses the new configuration.

Updating agent.json

In some cases you may want to update your agent.json file. Since Kubernetes does not support recursive ConfigMaps, the process is to create a second ConfigMap and add it to your DaemonSet configuration.

1. Export your Configuration files

2. Make your changes

3. Test your Configuration

4. Create another ConfigMap with the following command:

kubectl create configmap scalyr-config-agent-json --from-file=/tmp/scalyr-agent-config/agent.json

5. Append the following to your DaemonSet configuration

spec:
  template:
    spec:
      containers:
        volumeMounts:
        - name: scalyr-config-agent-json
          mountPath: /etc/scalyr-agent-2
      volumes:
        - name: scalyr-config-agent-json
          configMap:
            name: scalyr-config-agent-json

6. Verify.

kubectl get configmap scalyr-config-agent-json -o yaml

Your scalyr-config-agent-json ConfigMap should look something like this:

data:
agent.json: |
  {
      // Note:  It is assumed that another file such as `agent.d/api-key.json`
      // will contain the api key for the user's Scalyr account.
      // No need for system and agent monitors.  The k8 plugin will gather
      // metrics on the container running the agent.
      implicit_metric_monitor: false,
      implicit_agent_process_metrics_monitor: false,
      "server_attributes": {
        "hello": "world"
      }
  }

Troubleshooting Tip: Comments and other syntax issues such as empty comments will affect the way this ConfigMap is written.

===================================================================================================

(C) Creating Custom Images

This section provides background and instructions for creating custom Scalyr Agent images.

Understanding the Scalyr Agent DaemonSet

The example Kubernetes DaemonSet configuration for the Scalyr Agent does the following:

  • Maps /var/lib/docker/containers of the node to /var/lib/docker/containers on the scalyr-k8s-agent container. This gives the container access to the raw logs from other pods and containers running on that node.
  • Maps /var/log/pods of the node to /var/log/pods on the scalyr-k8s-agent container. This gives the container access to the raw logs from other pods and containers when the underlying runtime is ContainerD (or another runtime compatible with the Container Runtime Interface (CRI)).
  • Maps /var/log/containers of the node to /var/log/containers on the scalyr-k8s-agent container. This gives the container access to the raw logs from other pods and containers when the underlying runtime is ContainerD (or another runtime compatible with the Container Runtime Interface (CRI)).
  • Imports the entire contents of the scalyr-config ConfigMap as environment variables. These include the SCALYR_API_KEY environment variable that the agent needs to start up, since the shipped agent.json file doens't contain an api_key entry.

The current recommended DaemonSet YAML is scalyr-agent-2-envfrom.yaml.

(Note: Prior to release 2.0.47, the example Kubernetes DaemonSet configuration only mapped 4 values from the scalyr-config ConfigMap into environment variables. You had to add new mappings for each new environment-aware variable. It also did not create mounts for the ContainerD support. That legacy example DaemonSet YAML was scalyr-agent-2.yaml.)

Creating a Custom Scalyr Agent Docker Image

As discussed above, the preferred configuration methods are Annotations (for pod logs) and Environment Variables through ConfigMaps (for agent config). We've also seen how non-environment-aware config variables can be modified on the fly by extracting agent.json and agent.d snippets from a running agent, then modified and re-injected. If you wish to persist the modified agent.json and agent.d snippets, you have to go one step further beyond merely exporting and modifying the agent.json and agent.d snippets. Instead, you also export a custom Dockerfile which you then use to build your custom Docker image. We provide tools to do this.

As before, start off by running the scalyr-k8s-agent image in a single Docker container and test it. Assuming you've created and tested your configuration changes in a container called scalyr-agent (based off the scalyr/scalyr-k8s-agent:latest image), you can create a custom image by executing the following commands:

mkdir /tmp/scalyr-agent
cd /tmp/scalyr-agent
docker exec -i scalyr-agent scalyr-agent-2-config --k8s-create-custom-dockerfile - | tar -xz
docker build -t customized-scalyr-k8s-agent .

This will leave a new Docker image on your local Docker instance with the repository tag customized-scalyr-k8s-agent. You can change the name using the docker tag command. Thereafter, use any of the standard methods to make this container available to your Kubernetes cluster.

For example, you can now reference the custom agent image in your DaemonSet YAML file:

spec:
  template:
      spec:
        containers:
        - name: scalyr-agent
          image: customized-scalyr-k8s-agent

And then launch the DaemonSet with the edited YAML:

kubectl create -f my-custom-configuration.yaml

Testing your Custom Configuration Locally

Before deploying a custom Scalyr Agent image to your cluster, it's a good idea to test it locally to make sure there are no configuration errors.

This can be done via Docker as follows:

docker run -ti -e SCALYR_API_KEY="$SCALYR_API_KEY" \
  --name scalyr-k8s-agent \
  -v /var/lib/docker/containers:/var/lib/docker/containers \
  -v /var/run/docker.sock:/var/scalyr/docker.sock \
  customized-scalyr-k8s-agent /bin/sh

Which will launch a single container based on your customized image and drop you in to a Bourne shell.

Before running the above command, make sure to replace $SCALYR_API_KEY with your Scalyr api key (or export it to an environment variable called $SCALYR_API_KEY) to expose your api key to the container.

From the container's shell prompt, manually launch the Scalyr Agent by executing:

scalyr-agent-2 start

If everything is configured correctly, you should see a message similar to:

Configuration and server connection verified, starting agent in background.

Note: Because you're starting a Kubernetes agent in a standalone Docker image, you may still see some warnings or errors related to Kubernetes. But the agent would still have started, and you would still see the above `starting agent in background` message, which you can also verify by viewing /var/log/scalyr-agent-2/agent.log.

Once you confirm that the Scalyr Agent is running correctly, you're ready to deploy the image to your Kubernetes cluster.

Conclusion

To recap, we've learned how to: (A) configure the monitoring of individual pod logs (B) configure the Scalyr Agent and (C) create custom images.

Even so, you might not need to perform any of the above — A simple install of the Scalyr Kubernetes Agent will instantly make your cluster observable and searchable in the Scalyr UI.

You may also be interested in general instructions on setting up Log Parsers. When logs are properly parsed, Scalyr becomes an even more powerful tool for analysis and visualization.

For complete documentation on agent configuration options, see the Scalyr Agent reference.