Alert Using HipChat

This Solution describes how to send Scalyr alert notifications through HipChat. When a Scalyr alert is triggered or resolves, Scalyr will send a message to a HipChat room. You can use HipChat for some or all of your Scalyr alerts. You can also choose to have Scalyr send notifications to both HipChat and a list of e-mail addresses or other recipients.

A Solution is a step-by-step guide for accomplishing a specific task, designed to make sense even if you're just getting started with Scalyr server monitoring. If you're new to Scalyr, you should read the short Getting Started guide. For help with other tasks, see the Solutions directory.

We encourage you to use HipChat API v2, but we also support HipChat API. The steps to use either are quite similar, but the usage of the older HipChat API will be highlighted in this color. Our instructions are for HipChat API V2, but if you want to use Hipchat API v1, the differences will be called out.

Steps

1. You'll need a "Notification" API token from HipChat:

For HipChat API v1, follow the steps:

  • Go to https://www.hipchat.com/admin/api.
  • Log into your HipChat account
  • If you see a token listed with type "Notification", you can use it.
  • Otherwise, find the Create new token form. Select Type: Notification, enter a label such as "Scalyr Alerts", and click Create.
  • Make a note of the long sequence of letters and digits listed for your notification token.

2. Log into Scalyr, click the Alerts navigation link, and click Edit Alerts.

3. Edit the Alerts configuration file to specify HipChat as the alert recipient. To do this for all Scalyr alerts, create or edit an alertAddress field at the top level of the file:

{
  "alertAddress": "hipchat2:room=ID&token=XXX",

  alerts: [
    ...
  ]
}

Replace NAME with the name of a HipChat room, and XXX with the API token you generated in HipChat.

For HipChat API v1 users, replace all hipchat2: prefix with hipchat: e.g. "alertAddress": "hipchat:room=ID&token=XXX, foo@example.com, bar@example.com",

To send notifications to one or more e-mail addresses in addition to HipChat, list them all in alertAddress:

  "alertAddress": "hipchat2:room=ID&token=XXX, foo@example.com, bar@example.com",

If you only want to use HipChat for certain alerts, you can specify an alertAddress field for those alerts:

{
  alertAddress: "email@example.com",

  alerts: [
    // This alert will be sent to HipChat
    {
      trigger: "count:1m(error) > 10",
      "alertAddress": "hipchat2:room=ID&token=XXX",
    },

    // This alert will send notifications to email@example.com
    {
      trigger: "mean:10m($source='tsdb' $serverHost='server1' metric='proc.stat.cpu_rate' type='user') > 50"
    }
  ]
}

The HipChat endpoint defaults to http://api.hipchat.com, but if you want to send requests to custom endpoints, you can pass apiUrl option in the alertAddress eg:

"alertAddress": "hipchat2:room=ID&token=XXX&apiUrl=https://mydomain.api.hipchat.com"

``apiUrl`` option is not supported for HipChat API v1

To link a whole group of alerts to HipChat, specify an appropriate alertAddress for the group. See Specifying Alert Recipients.

HipChat notification settings

When Scalyr sends a message to HipChat, it can optionally instruct HipChat to add a "notification", such as a sound. You can control the exact form of this notification in your HipChat settings:

http://help.hipchat.com/knowledgebase/articles/64430-how-do-notifications-work

Scalyr sends two messages to HipChat each time an alert triggers: one when the alert first triggers, and one when the alert resolves. By default, Scalyr will tell HipChat to add a notification only for the trigger message. You can override this behavior by adding &notify=none or &notify=all to the hipchat: alert address. You can also add &notify=trigger to explicitly select the default behavior.

Using multiple HipChat rooms

You can send notifications for different alerts to different rooms in HipChat. Simply attach an appropriate alertAddress to each alert or alert group. You can use the same API token with different rooms.

Troubleshooting

If you don't see messages appearing in HipChat, you may have misconfigured the hipchat: notification address. For example, you may have an incorrect authorization token, in which case HipChat will refuse to accept notifications from Scalyr. To check for HipChat API errors, search your logs for (tag='hipchatApiError').

Further Reading

The Alerts reference documents the format of the Scalyr alerts file.

Scalyr sends notifications to HipChat at the same time it would send an e-mail notification. In particular, Grace Periods and Repeated Notifications both apply.