Scalyr Agent Classic

This page describes the "classic" Scalyr Agent. For new installations, we strongly encourage use of the current Scalyr Agent. If you are currently using the classic agent, we encourage you to upgrade to the current agent; see Migrating from Scalyr Agent Classic. If you're not sure which agent you're using: if ps aux | grep 'scalyr-agent-[2]' has any output, then you're running the current agent. If ps aux | grep 'scalyrRela[y]' has any output, you're running the classic agent.

The Scalyr Agent is a daemon which you can install on each of your servers. It uploads logs and system metrics to Scalyr.

Installation

For standard installation using yum or apt-get, go to the Agent Installation page. If you'd prefer to download the installer manually, jump down to the Package Download or Tarball Download sections.

Configuration

To configure the agent, edit the file /etc/scalyrAgent/agentConfig.json. After editing the configuration file, you may need to restart the agent to pick up the changes: sudo service scalyr-agent restart. (Some changes are picked up without a restart; for instance, you don't need to restart after editing the list of uploaded log files.)

For an exhaustive list of configuration options, review the file /etc/scalyrAgent/exampleConfig.json.

Customize Host Name

By default, Scalyr will identify your server by its hostname. If your hostname is something unhelpful like "ip-12-23-34-45", you can specify a different name by adding a serverHost field to the scalyr_client.global_attributes section of /etc/scalyrAgent/agentConfig.json:

"scalyr_client.global_attributes": {
  serverHost: "app-server-3",
  ...
},

Then execute sudo scalyr-agent restart to pick up the configuration change. All subsequent events/data from this server will be listed under the new name. Older data will still be listed under the old hostname; the old hostname will disappear from the Overview page after 24 hours or so.

Uploading Log Files

To upload log files, edit the "logcopier_config" section of /etc/scalyrAgent/agentConfig.json. It can contain any number of directory clauses, each describing a group of related log files. For instance:

directories: [
   {
     path: "/var/log/httpd/access*",
     file_attributes: {parser: "accessLog"}
   }
 ]

This will upload all files in the /var/log/httpd directory whose names begin with access. The files will be tagged with the field parser=accessLog. You should always specify a parser field; this is used to identify the log and associate it with parsing rules. The parser field should not contain spaces or punctuation, other than spaces, hyphens, or underscores.

You can specify additional fields in file_attributes. This allows you to distinguish between different log files from the same server. For instance, you might give the file a field like service: "memcache" to identify the service that generates this log; you could then add service = "memcache" to any query to restrict the query to memcache logs.

The path uses standard "glob" syntax: a * matches any text within a file or directory name, ** matches a chain of one or more directories, and ? matches a single character. If you need a more complex regular expression to match your log file names, you can use an alternate syntax using a filename_patterns field:

directories: [
   {
     path: "/var/log/httpd",
     filename_patterns: ["access_log_[0-9]+", "internal_log_[0-9]+"],
     file_attributes: {parser: "accessLog"}
   }
 ]

When using filename_patterns, the path field should specify a directory, not a file, and cannot use glob syntax. filename_patterns contains one or more regular expressions, which are matched against the name of each file in the directory.

Log Filtering

Sometimes, you might not want to upload all messages in a log. For instance, the log might be cluttered with noisy debugging messages. You can specify sampling rules in the agent configuration to either subsample certain types of message (uploading only a random sample), or exclude them entirely. Each filter rule contains a regular expression and a "fraction" field. The agent finds the first regular expression that matches each message, and uses the fraction to decide whether to upload that message. The following configuration will upload 10% of all messages containing the word "INFO", no messages containing the word "FINE", and (by default) all messages that don't contain either word:

directories: [
   {
     path: "/var/log/app/log*",
     file_attributes: {parser: "appLog"},
     sample: [
       { pattern: "INFO", fraction: 0.1 },
       { pattern: "FINE", fraction: 0 }
     ]
   }
 ]

Filtering occurs before the log leaves your server; filtered lines are not sent to Scalyr.

You can also instruct the agent to rewrite or delete certain portions of a log message. This can be used to redact passwords or other sensitive information. Redaction occurs before the log leaves your server; redacted text is not sent to Scalyr.

To use this feature, add a "replace" section to your upload rule. For example:

directories: [
   {
     path: "/var/log/app/log*",
     file_attributes: {parser: "appLog"},
     replace: [
       // delete all instances of password=...
       "password=[^& ]*",

       // Replace terms like "userInfo=username password" with "userInfo=username"
       {
         pattern: "userInfo=([^ ]+) [^ ]+",
         replacement: "userInfo=$1"
       }
     ]
   }
 ]

Gathering Process Metrics

The agent can measure CPU, memory, and other metrics for individual processes within a server. You must configure the agent with a list of processes to monitor. See the Application Dashboard page for instructions.

Agent Logs

For diagnostic purposes, you may sometimes need to look at the agent's own log output. Most information of interest is in a file named relay.log; you may also need to look at agent.log and/or tcollector.log. These files are in /var/log/scalyrAgent.

Upgrades

To upgrade your agent installation to the latest version, if you installed using yum or apt-get, simply execute

sudo yum update scalyr-agent

or

sudo apt-get install scalyr-agent

If you downloaded the RPM package, Debian package, or tarball directly, get the latest build (see Package Download) and run one of the following commands:

sudo rpm -U scalyr-agent-1.0.22-1.noarch.rpm # RPM
sudo dpkg -i scalyr-agent_1.0.22_all.deb # Debian package
./scalyrAgentInstaller.sh # tarball

The latest agent release is 1.0.22. To determine which version you're running:

scalyr-agent version # If installed from a package
~/scalyrAgent/agent.sh version # If installed from the tarball

Troubleshooting: no data from host

If you are not seeing the data you expect in Scalyr, the first step is to check whether the agent is uploading any data at all. Click the Overview link in the navigation bar, find the host in question, and click on the hostname. If the host does not appear, or linked page shows no matching events, then the agent on that host is not running or is not successfully uploading data.

To diagnose this, first check whether the agent is running: scalyr-agent status (or ~/scalyrAgent/agent.sh status if you installed from the tarball). You can also try ps aux | grep scalyrRelay for a more direct verification; the output should contain a command line along the lines of java -Xms40m -Xmx40m -jar scalyrRelay.jar.

If the agent is not running, try launching it with sudo scalyr-agent start (or ~/scalyrAgent/agent.sh start if you installed from the tarball). You should see some disgnostic messages appearing over the course of 10 or 20 seconds, followed by:

Congratulations! The Scalyr agent has started successfully and
is relaying system metrics to Scalyr. To view the results, go
to: https://www.scalyr.com/logSummary

[etc.]

If you instead see some sort of error message, continue reading.

If the agent is running but failing to upload data, open /var/log/scalyrAgent/relay.log. You might also need to look at /var/log/scalyrAgent/agent.log or /var/log/scalyrAgent/tcollector.log. If the agent fails to launch, look at the log output from the "start" command. Here are a series of possible error messages, with corresponding fixes:

Mon Feb 25 20:51:05 UTC 2013: user/error/badKnobFile (Error parsing [<configuration file "agentConfig.json" in filesystem "/usr/share/scalyrAgent/scalyrRelay">] as JSON)

com.scalyr.api.knobs.BadConfigurationFileException: File is not in valid JSON format (Unexpected character '/' (line 7, byte position 181))

Mon Feb 25 20:51:06 UTC 2013: user/error/badKnobFile (Knob: ignoring file [<configuration file "agentConfig.json" in filesystem "/usr/share/scalyrAgent/scalyrRelay">]: it does not contain valid JSON)

Any of these messages indicate a syntax error in /etc/scalyrAgent/agentConfig.json. If you search the output for BadConfigurationFileException, you should see a message identifying the line number where the error occurred. Fix the error and try sudo scalyr-agent restart (or ~/scalyrAgent/agent.sh restart for the tarball) to relaunch the agent.

UPLOAD FAILURE: Server response had bad status [error/client/badParam]; complete text: {"message":"Missing field 'token'","status":"error\/client\/badParam"}

Mon Feb 25 20:54:59 UTC 2013: server/error (Bad response from Scalyr (status [error/client/badParam], message [Missing field 'token']))

Either of these messages indicate that you have not specified your API key. Open /etc/scalyrAgent/agentConfig.json and search for scalyr_client.write_logs_key. You should see a line like this:

"scalyr_client.write_logs_key": "",

Paste in your API token:

(log in to see your API token)

Any of the following messages indicate that your API key is present, but incorrect:

UPLOAD FAILURE: Server response had bad status [error/client/badParam]; complete text: {"message":"authorization token [...Rmy8-] is invalid","status":"error\/client\/badParam"}

Mon Feb 25 21:01:10 UTC 2013: server/error (Bad response from Scalyr (status [error/client/badParam], message [authorization token [...Rmy8-] is invalid]))

Follow the instructions above for pasting in the correct key.

Troubleshooting: log not present

If the agent on a given host is uploading at least some data, but is not uploading a particular log file, open /etc/scalyrAgent/agentConfig.json on that host. Find the logcopier_config section, and verify that the directories list contains an appropriate entry for the log file in question. It should look something like this:

"logcopier_config": {
  directories: [
    ...,
    {
      path: "/var/log/tomcat6/access_log",
      file_attributes: {parser: "accessLog"}
    },
    ...
  ]
},

Verify that the path (and filename_patterns, if used) match the absolute path of your log file, and file_attributes has at least a "parser" field. Also verify that the agent has read access to the log file and its surrounding directories. To double-check the user under which the agent is running, execute ps aux | grep scalyrRelay and look for the user of the scalyrRelay process. To switch the agent to a new user, execute sudo scalyr-agent-config --run_as XXXX (where XXXX is the new username) and then restart the agent.

Internal Details

The agent has three parts:

  • The relay — a Java daemon that buffers log messages and uploads them to Scalyr over https. The daemon collects messages from any log files you specify in the agent configuration, and also accepts data via the Graphite and OpenTSDB network protocols.
  • tcollector — a set of Python scripts that gather system metrics. It is part of the OpenTSDB project, and sends metrics to the relay through the OpenTSDB protocol.
  • scripts — a set of bash scripts that coordinate the configuration, launching, and termination of the relay and tcollector.

The package installation comprises the following files:

  • tcollector — a set of Python scripts that gather system metrics. It is part of the OpenTSDB project, and sends metrics to the relay through the OpenTSDB protocol.
  • /etc/scalyrAgent/agentConfig.json — the agent configuration file
  • /etc/scalyrAgent/exampleConfig.json — a sample configuration file, for reference only
  • /var/log/scalyrAgent/agent.log — log output from the bash scripts
  • /var/log/scalyrAgent/relay.log — the primary log file; contains output from the Java relay daemon
  • /var/log/scalyrAgent/tcollector.log — log output from tcollector
  • /var/log/scalyrAgent/{agent,tcollector}.pid — the process IDs of the running agent (script) and tcollector processes
  • /var/lib/scalyrAgent/relay_checkpoints — records information about log messages that have already been uploaded, so that messages are not re-uploaded after an agent restart.
  • Command aliases — /usr/sbin/{scalyr-agent, scalyr-agent-config}. These are aliases to the shell scripts for launching and managing the agent.
  • Relay — /usr/share/scalyrAgent/scalyrRelay/*
  • Agent scripts — /usr/share/scalyrAgent/*
  • tcollector — /usr/share/scalyrAgent/tcollector/*
  • Startup scripts — /etc/init.d/scalyr-agent, and various scalyr-agent files in /etc/rcN.d When installed via the tarball, all of the agent files reside in the scalyrAgent directory, created in the directory where the installation was performed. The only exception are the startup scripts, which are created if you run sudo ./agent.sh install_rcinit. These reside in /etc/init.d/scalyr-agent, and various scalyr-agent files in /etc/rcN.d.

Package Download

If you prefer to download the agent package directly, rather than using our yum or apt-get repositories, you can do so. For RPM:

wget https://www.scalyr.com/scalyr-repo/stable/latest/scalyr-agent-1.0.22-1.noarch.rpm
sudo rpm -i scalyr-agent-1.0.22-1.noarch.rpm

For Debian / Ubuntu:

wget https://www.scalyr.com/scalyr-repo/stable/latest/scalyr-agent_1.0.22_all.deb
sudo dpkg -i scalyr-agent_1.0.22_all.deb

Tarball Download

If you'd prefer not to use our agent packages, you can install from simple tarball. Run the following commands, from a Unix account that has read access to your logs.

1. Download and run the installer:

wget https://www.scalyr.com/scalyr-repo/stable/latest/scalyrAgentInstaller.sh
bash ./scalyrAgentInstaller.sh --write_logs_key -

2. When prompted, enter your "Write Logs" API key:

(log in to view API tokens)

3. To configure log uploading, open ~/scalyrAgent/configs/agentConfig.json and find the logcopier_config section, and within that, the directories clause. Remove the example text and insert something like this:

directories: [
  {
    path: "/var/log/tomcat6/access_log",
    file_attributes: {parser: "accessLog"}
  }
]

Substitute the path for your web server's access log. You can upload any number of log files, of any type; give each file its own entry in the directories list. Specify parser: "accessLog" for standard web access logs; for other log types, choose a name that describes the type of data in the log, using only letters, digits, spaces, underscores, and hyphens.

4. Launch the agent:

./scalyrAgent/agent.sh start

If everything is set up correctly, you should see a message like this: "Congratulations! The Scalyr monitor has started successfully and is relaying system metrics to Scalyr."

5. Add an rc.init script to auto-launch the agent on host startup

sudo ./agent.sh install_rcinit

Note: when you install the agent using the tarball, some files wind up in different locations than when using a package. The configuration file is ~/scalyrAgent/configs/agentConfig.json, and logs are in ~/scalyrAgent/logs. Use these files, rather than the files mentioned elsewhere in the agent documentation.