Linux Process Metrics

This agent monitor plugin records CPU consumption, memory usage, and other metrics for a specified process. You can use this plugin to record resource usage for a web server, database, or other application.

An agent monitor plugin is a component of the Scalyr Agent. To use a plugin, simply add it to the monitors section of the Scalyr Agent configuration file (/etc/scalyr/agent.json). For more information, see Agent Plugins.

Sample Configuration

Here is a simple configuration fragment showing use of the linux_process_metrics plugin. This sample will record resource usage for any process whose command line contains a match for the regular expression java.*tomcat6:

monitors: [
  {
     module:      "scalyr_agent.builtin_monitors.linux_process_metrics",
     id:          "tomcat",
     commandline: "java.*tomcat6",
  }
]

Metrics Aggregation

If the regular expression you provided eg. java.*tomcat6 matches more than one process, only the first matched process will be used to report metrics by default. If you would like to aggregate all the matched processes, you can set the optional param aggregate_multiple_processes to true:

When you aggregate the processes, a single metric will be reported that will include all of the processes. For example, the app.cpu metric will be the sum of the amount of CPU used for all the processes. And app.mem.bytes will be the sum of the RAM used across all of the processes.

monitors: [
  {
     module:      "scalyr_agent.builtin_monitors.linux_process_metrics",
     id:          "tomcat",
     commandline: "java.*tomcat6",
     aggregate_multiple_processes: true
  }
]

Include child processes

If you would like to include any child processes that are spawned by the matched processes, you can set the optional param include_child_processes to true and any process whose parent is one of the matched process will be included in the aggregated metrics. This is done recursively, so any children of the child processes will also be included.

monitors: [
  {
     module:      "scalyr_agent.builtin_monitors.linux_process_metrics",
     id:          "tomcat",
     commandline: "java.*tomcat6",
     aggregate_multiple_processes: true,
     include_child_processes: true
  }
]

To record information for more than one type of process group, use several copies of the linux_process_metrics plugin in your configuration.

Viewing Data

After adding this plugin to the agent configuration file, wait one minute for data to begin recording. Then click the Dashboards menu and select Linux Process Metrics. (The dashboard will not be listed until the agent begins sending data.)

You'll have to edit the dashboard file for each id value you've used. From the dashboard page, click the Edit Dashboard link. Look for the following bit of code, near the top of the file:

  // On the next line, list each "id" that you've used in a linux_process_metrics
  // clause in the Scalyr Agent configuration file (agent.json).
  values: [ "agent" ]

Edit the values list according to the list of ids you've used. For instance, if you've used "tomcat" (as in the example above), the list would look like this:

  values: [ "agent", "tomcat" ]

The "agent" ID is used to report metrics for the Scalyr Agent itself.

You can now return to the dashboard. Use the dropdowns near the top of the page to select the host and process you'd like to view.

Configuration Reference

Option Usage
module Always scalyr_agent.builtin_monitors.linux_process_metrics
id Included in each log message generated by this monitor, as a field named instance. Allows you to distinguish between values recorded by different monitors.
commandline A regular expression which will match the command line of the process you're interested in, as shown in the output of ps aux. (If multiple processes match the same command line pattern, only one will be monitored.)
aggregate_multiple_processes If set to true, aggregates the metrics of all processes matched by the commandline regex. Default false
include_child_processes If set to true, includes processes whose parent process is one of the processes matched by the commandline regex in the metrics aggregation. This is done recursively, so any children of the child processes will also be included. Note that you need the aggregate_multiple_processes set to true for this feature. Default false

Log Reference

Each event recorded by this plugin will have the following fields:

Field Meaning
monitor Always linux_process_metrics
instance The id value from the monitor configuration, e.g. tomcat
app Same as instance; provided for compatibility with the original Scalyr Agent.
metric The name of a metric being measured, e.g. "app.cpu"
value The metric value

Some metrics have additional fields, as documented in the table below.

This table lists all metrics recorded by the Linux Process Metrics plugin.

Metric Description
app.cpu type=user User-mode CPU usage, in 1/100ths of a second.
app.cpu type=system System-mode CPU usage, in 1/100ths of a second,
app.uptime Process uptime, in milliseconds,
app.threads The number of threads being used by the process.
app.nice The nice value for the process.
app.mem.bytes type=vmsize Virtual memory usage, in bytes.
app.mem.bytes type=resident Resident memory usage, in bytes.
app.mem.bytes type=peak_vmsize Peak virtual memory usage, in bytes.
app.mem.bytes type=peak_resident Peak resident memory usage, in bytes.
app.disk.bytes type=read Total bytes read from disk.
app.disk.requests type=read Total disk read requests.
app.disk.bytes type=write Total bytes written to disk.
app.disk.requests type=write Total disk write requests.