Install Scalyr Agent (Kubernetes)

These instructions are for installing the Scalyr Agent as a DaemonSet on your Kubernetes cluster. If you plan to run the Agent directly on Linux, see the Linux installation page. For Windows, see the Windows installation page, and for running the Agent separately in a Docker container, see the Docker installation page.

We're assuming you have a conceptual understanding of Kubernetes such as DaemonSets, Pods, Nodes, Config Maps etc.

The Scalyr Agent is a daemon that uploads logs and container metrics from a Kubernetes cluster to Scalyr servers. This is achieved with the Kubernetes monitor plugin. This page has streamlined instructions to get you up and running quickly in a Kubernetes environment.

We implement the Kubernetes-recommended node-level logging architecture, running the Scalyr Agent as a DaemonSet in your cluster. The DaemonSet runs a Scalyr Agent pod on each node. Each Agent pod collects logs from all other pods running on the same node. By default, the Scalyr Agent will collect pod logs, container metrics, and Kubernetes Events for all nodes.

  • Logs: Log collection is supported regardless of underlying runtime (Docker Engine, Containerd, CRI-O etc). As long as the container logs are accessible to the Scalyr Agent pod, the agent can monitor and upload them.
  • Metrics: Metrics are currently only supported for the Docker Engine runtime.
  • Kubernetes Events: Kubernetes Events represent state changes, errors or other Kubernetes system messages. The Scalyr Agent can monitor Kubernetes Events regardless of underlying runtime. For more details on Kubernetes Events, including how to disable event collection, see the Kubernetes Events monitor plugin.

Installation

Listed below are instructions for installing Scalyr Kubernetes Agent for the first time. These instructions are the recommended approach for agent release 2.0.47 and above. Instructions for prior versions can be viewed here.

1. Create a Secret that contains your Write Logs API key. You can find your API key here under "Logs Access Keys". Make sure the key has "write" access.

kubectl create secret generic scalyr-api-key --from-literal=scalyr-api-key="(log in to view API tokens)"

2. Create a ConfigMap that defines your cluster name (and any other environment-aware variables you may need). We recommend doing this in a YAML file but you can also create it from literals.

> curl https://raw.githubusercontent.com/scalyr/scalyr-agent-2/release/k8s/scalyr-agent-2-configmap.yaml
> cat scalyr-agent-2-configmap.yaml
apiVersion: v1
kind: ConfigMap
data:
  SCALYR_K8S_CLUSTER_NAME: <your_cluster_name>
  #
  # EU customers must uncomment this next line:
  # SCALYR_SERVER: "https://upload.eu.scalyr.com"
  #
  # Define any other environment variables for config variables that are "environment aware".
  # Most environment variables start with "SCALYR_"
  # See https://www.scalyr.com/help/scalyr-agent#environmentAware for more details.
  #
  # Example:
  # SCALYR_K8S_EVENTS_DISABLE: "true"
  #
metadata:
  name: scalyr-config
  namespace: default

> (Edit and save scalyr-agent-2-configmap.yaml)

> kubectl create -f scalyr-agent-2-configmap.yaml

3. Create a service account that grants the DaemonSet the required privileges to access the Kubernetes API.

kubectl create -f https://raw.githubusercontent.com/scalyr/scalyr-agent-2/release/k8s/scalyr-service-account.yaml

4. Install the new Scalyr k8s DaemonSet, directly injecting ConfigMap key-values as environment variables:

kubectl create -f https://raw.githubusercontent.com/scalyr/scalyr-agent-2/release/k8s/scalyr-agent-2-envfrom.yaml

THAT'S IT!

Your cluster logs should be searchable in Scalyr within seconds. For further configuration refinements, please see Configuring Scalyr Agent (Kubernetes).

Upgrading

To upgrade the Scalyr Kubernetes Agent:

1. Create a ConfigMap that defines your cluster name (and any other environment-aware variables you may need).

(See previous "Installation" section for defining a ConfigMap from literals or YAML file.)

2. Create a service account that gives the DaemonSet the required privileges to access the Kubernetes API.

kubectl create -f https://raw.githubusercontent.com/scalyr/scalyr-agent-2/release/k8s/scalyr-service-account.yaml

3. Delete your existing Scalyr k8s DaemonSet. Note: If you used a different yaml for deployment, use that.

kubectl delete -f https://raw.githubusercontent.com/scalyr/scalyr-agent-2/release/k8s/scalyr-agent-2-envfrom.yaml

4. Install the new Scalyr k8s DaemonSet.

kubectl create -f https://raw.githubusercontent.com/scalyr/scalyr-agent-2/release/k8s/scalyr-agent-2-envfrom.yaml

For further configuration refinements, please see Configuring Scalyr Agent (Kubernetes).

Useful tips

Checking the status of a Scalyr Agent container

To check the status of a currently running Scalyr Agent container, first use kubectl to find the name of the pod in which you are interested:

kubectl get pods

Once you have the name of the pod running the Scalyr Agent, use the following command:

kubectl exec <pod_name> -- scalyr-agent-2 status

For more verbose output, run:

kubectl exec <pod_name> -- scalyr-agent-2 status -v

Stopping the Scalyr Agent DaemonSet

If you wish to stop running the Scalyr Agent DaemonSet entirely, then assuming you are using the default configuration and your DaemonSet's name is 'scalyr-agent-2' run:

kubectl delete daemonset scalyr-agent-2

There is currently no convenient way to stop the Scalyr Agent already running on a specific node. But you can prevent the agent from starting up on specific nodes by following the instructions in the next section:

Disabling the agent on the Master Node

The Scalyr Agent DaemonSet runs an agent pod on each node. This also runs the agent on the master node. To disable the agent on the master node, comment the following in the DaemonSet scalyr-agent-2-envfrom.yaml file:

tolerations:
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"

Running the DaemonSet on select nodes

To run the Scalyr Agent on specific nodes of your cluster, add a nodeSelector or nodeAffinity section to your pod config file. For example, if your desired nodes are labeled logging=scalyr then adding the following nodeSelector will restrict Scalyr Agent pods to Nodes that have a logging=scalyr label:

spec:
  nodeSelector:
    logging: scalyr

or if you wanted to use nodeAffinity:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: logging
            operator: In
            values: ["scalyr"]

Read more about specifying node selectors and affinity.

Conclusion and Next Steps

We hope that was easy. If you've had any trouble, please let us know. Otherwise, if this is your first time using Scalyr, this would be an excellent time read the Getting Started guide.

Now that you have the Scalyr Kubernetes Agent up and running, please see Configuring Scalyr Agent (Kubernetes) for details on configuring the agent and logging of pod logs.