Home
On-call

Grafana

Grafana unified alerting delivers to BatonDeck through a Webhook contact point.

Grafana unified alerting delivers to BatonDeck through a Webhook contact point.

  • Auth: api_key — Grafana cannot compute the BatonDeck HMAC, but it can send a custom header.
  • Endpoint: POST https://<your-oncall-host>/i/<integrationId>
  • Header: x-batondeck-key: <your key>

1. Create the integration

create_integration {
  projectId, name: "prod-grafana", source: "grafana", boardId: "B-…",
  auth: { mode: "api_key", apiKey: "<a long random value you generate>" },
  routing: { mode: "oncall", onCallAgent: "claude-oncall" }
}

You generate the key (minimum 8 characters — generate far more, e.g. openssl rand -base64 32). BatonDeck stores only its SHA-256 hash, so there is no read-back: keep your copy. The response's endpointPath is what you paste into Grafana.

2. Add the contact point

Alerting → Contact points → Add contact point, integration type Webhook.

FieldValue
URLhttps://<your-oncall-host>/i/<integrationId>
HTTP MethodPOST
Custom / extra HTTP headersname x-batondeck-key, value your key

Then point a notification policy at the contact point so your alert rules route to it.

The one thing to get right: the key goes in a custom header named x-batondeck-key. Grafana's webhook contact point also offers Basic Auth and Authorization Header Scheme / Credentials, and both are tempting. Neither works: in api_key mode the function reads only x-batondeck-key, and the Authorization header is read only in jwt mode. Putting the key in Authorization produces api_key_mismatch in the delivery log against a config that looks completely correct. If your Grafana version has no custom-header field, front it with a proxy that adds the header.

3. What lands on the ticket

Grafana's webhook body is fixed — you do not template it. This is how each entry in its alerts[] array becomes an incident:

Ticket fieldComes from
Correlation keyalerts[].fingerprint (falls back to a hash of the alert's label set)
Firing / resolvedalerts[].status"resolved" resolves, anything else fires
Severityalerts[].labels.severity, defaulting to warning
Titlealerts[].annotations.summary, else alerts[].labels.alertname, else "Grafana alert"
Descriptionalerts[].annotations.description
Labels / annotationsthe alert's own maps, verbatim
LinksgeneratorURL, silenceURL, dashboardURL, panelURL — whichever are present
Occurred atstartsAt; for a resolved alert, endsAt when it is a real timestamp

Two consequences worth designing your alert rules around:

  • Give every rule a summary annotation. Without one, tickets are titled by raw alertname — readable to you, much less useful to an agent picking the ticket up cold.
  • Set a severity label. Everything without one is warning → priority normal. Use critical / error / warning / info, or map your own vocabulary with the integration's mapping.severityMap.

Grouped deliveries: one ticket per alert fingerprint

Grafana groups alerts by notification policy before it sends, so a single webhook call can carry many alerts. BatonDeck fans that out: each entry in alerts[] becomes its own delivery and its own incident ticket, keyed on its own fingerprint. Twelve alerts in one group is twelve tickets, twelve correlation keys, twelve independent lifecycles — not one ticket listing twelve problems.

This is deliberate. An agent works one problem at a time, and a grouped ticket cannot be claimed, resolved or closed per-problem. If you want fewer tickets, group less aggressively at the rule level — a single rule producing one alert per instance really is N problems.

The envelope's top-level status is ignored entirely; each alert's own status decides whether it fires or resolves. A mixed group therefore opens some tickets and resolves others in one call, which is correct.

When nothing arrives

Delivery log reasonWhat it means
No rows at allGrafana never reached the function — check the URL, and check the contact point is actually referenced by a notification policy. Grafana's Test button on the contact point is the quickest probe.
api_key_mismatchThe header name or value differs — almost always the Authorization trap above.
404 from the endpointWrong integration id, or the integration is disabled. Both answer identically on purpose.

Next: On-call ingestion overview · Alertmanager