Log Processing

Data Scrubbing

After the Scalyr Agent hands off each log entry to the cloud, but before you can search for it, that log entry has to be processed.

The first processing step is Data Scrubbing, where you can redact passwords or other sensitive information by rewriting or removing certain portions of a log entry.

Please note that this step occurs before parsing, so you are operating on a raw log line.

In the above example, we use a regex to match IP addresses and replace the digits with hash signs. The filter:

$serverHost=*

matches everything, but could easily be more fine-grained. To match, say, any log entry containing “password” on prod database servers, you might do something like this.

$serverHost contains “prod” $logFile contains “db_” password

Filter Ordering

You can make as many filters as you need, but please note that every data scrubbing filter will be applied to every log entry that comes in.

This may have unexpected consequences if multiple filters match the same entry, and you inadvertently remove in one filter the very text you need to match on in the second one. You can drag and drop the filters to reorder them and get around this problem.

Other Ways to Scrub Data

You can also remove unwanted text in the agent, before your logs even go across the wire to Scalyr.

If you prefer, it’s also possible to replace text in the parser.