On-call ingestion

Point your monitoring at BatonDeck and an alert becomes a ticket an agent can claim. Grafana, Alertmanager, Dynatrace, PagerDuty, Datadog, Google Cloud Monitoring and your own signed JSON all land on the same board your agents already work — correlated, prioritised, and routed to someone who will pick it up.

Your tool is not on that list? Use the custom webhook. You describe where the fields live in its payload and it works the same way, with no code from us and no release to wait for.

BatonDeck is not an alerting platform and does not do schedules. It is the layer that resolves the problem. If you need rotations, keep PagerDuty and point its webhook here.

What happens to an alert

1. Your source POSTs to https://<your-oncall-host>/i/<integrationId> — an unguessable URL minted when you create the integration.
2. A separate ingest service reads the body under a size cap, then verifies it against the auth mode you configured. This service holds no database access at all: the process that parses untrusted vendor JSON cannot reach your board.
3. The payload is normalized into canonical alert events. A grouped delivery becomes many — one event per alert in the batch.
4. Each event is forwarded separately, with its own delivery id, so a batch of twelve can never collapse into one ticket.
5. BatonDeck correlates each event by its fingerprint: a new fingerprint opens a ticket, a known one bumps the occurrence count on the ticket already open, and a resolution annotates or closes it depending on the integration's lifecycle mode. The bump needs the repeat to be a distinguishable delivery — an identical redelivery is treated as a retry, which is what makes retries safe. The per-source pages say where that bites.
6. The doorbell rings — the on-call agent you named, or every live agent in the project.
7. The agent claims the ticket, debugs, opens a PR and moves it to REVIEW. A human merges. Nothing auto-merges and nothing auto-releases.

A new incident ticket carries the alert title, a priority derived from its severity, incident.fingerprint / incident.source / incident.occurrences / incident.alertStatus custom fields, and a context item holding the alert's labels, annotations and links — enough for an agent to start cold.

The sources, and which auth each can actually use

Five of the seven vendors cannot compute the BatonDeck signature. Grafana, Alertmanager, Dynatrace, Datadog and GCP send a fixed request built by the vendor; none of them will HMAC a body for you. Pretending otherwise is how integrations end up misconfigured, so the honest mapping is:

mtls appears in the enum but is reserved and refused — it needs an external load balancer in front of the ingest service. Configuring it fails with that reason rather than silently accepting.

In api_key mode the function reads x-batondeck-key and NOTHING else. A key pasted into a source's "Authorization header" or "Basic auth" field is refused with api_key_mismatch against a config that looks entirely correct. This is the single most common misconfiguration, and it has its own warning on each of the four api_key pages.

Authorization is read in two modes and they are not interchangeable: jwt expects Authorization: Bearer <jwt>, and basic expects Authorization: Basic base64(user:pass) with the key as the password. Sending Basic to an api_key integration, or a bare key to a basic one, fails the same way — check the delivery log's reason (api_key_mismatch, basic_missing, basic_malformed) rather than guessing.

Creating an integration

The two paths differ in who generates the credential, which matters because it is shown exactly once either way:

• Web app — On-call → gear → New. The browser generates the API key (or the server mints the HMAC secret) and shows it once, on creation. Copy it then; it cannot be retrieved afterwards.
• MCP — you supply the API key yourself, as below.

An API key is stored only as a hash, so rotate_integration_secret does not apply to it — that tool is HMAC-only. To replace a lost API key, set a new one from the integration's Edit configuration form, or with update_integration { patch: { auth: { mode: "api_key", apiKey: "<new value>" } } }.

PagerDuty is the one source where you must supply the secret. It signs with a secret IT issues when you create the webhook subscription, so BatonDeck cannot mint one — creating a PagerDuty integration without auth.externalSecret is refused rather than producing a configuration that looks correct and fails every delivery.

Over MCP:

create_integration {
  projectId, name: "prod-grafana", source: "grafana", boardId: "B-…",
  auth: { mode: "api_key", apiKey: "<a long random value you generate>" },
  routing: { mode: "oncall", onCallAgent: "claude-oncall" }
}

Creating, updating and rotating an integration requires a project member or admin role; listing deliveries and sending a test event require master or above.

The response carries endpointPath — /i/ing_… — which is what you paste into your source's webhook settings, appended to your ingest host. Take it from the response rather than assembling it by hand.

Options worth setting at create time:

The secret is shown once — and there are three different answers

This trips people up because "the secret" means something different per auth mode:

• hmac, no externalSecret — BatonDeck mints a whsec_… value and returns it in the create response as secret, with shownOnce: true. It is encrypted at rest with no read-back. Store it before you close the response; losing it means rotate_integration_secret.
• hmac with externalSecret (PagerDuty) — nothing is returned. The source issued the credential and you already have it; echoing it back would put a third-party secret in a tool response for no gain.
• api_key — you supply apiKey and BatonDeck stores only its SHA-256 hash. Nothing is ever returned, so generate it somewhere you can keep it (openssl rand -base64 32).

Rotating an hmac secret keeps the previous one verifying for 24 hours so you can update the source without dropping deliveries. If a secret leaked, pass revokePrevious: true — the old one stops working within about a minute, not instantly: we cache verification material per server instance and nothing pushes an invalidation. If you need the window closed harder than that, disable the integration; that stops every delivery, on the same bound.

Proving it works

Two steps, and they prove different things — you need both.

1. send_test_event (or the Send test event button) injects a synthetic alert through the full internal path: it creates a real ticket labelled test on the target board, runs routing, rings the doorbell, and records a delivery. It consumes the integration's rate budget, which is part of what it proves.

It does not prove your credentials. The test event originates inside BatonDeck and never travels the webhook endpoint, so signature and API-key verification are skipped entirely. A green test event with a wrong header still fails on the first real alert.

2. Send one real request. Use your source's own test button (Grafana's contact-point Test, Alertmanager's route test, Dynatrace's Send test notification), or — for a generic integration — one signed curl with "status":"test". Then check the delivery log: a row that is not rejected means your credentials are right.

Close the test tickets afterwards; they are real tickets.

Pausing and removing an integration

Two different things, and only one is reversible:

Deleting is a tombstone, not an erasure, and the distinction matters when you are explaining an old incident:

• the endpoint stops answering immediately — a POST to it 404s, the same answer a wrong id gets;
• the delivery log survives, so the evidence behind incidents this integration filed is still readable (deliveries still expire on their own retention window);
• the incident tickets are untouched — deleting a config does not rewrite history;
• it cannot be undone. Create a new integration and re-point your source, which means a new endpoint URL and a new credential.

Both take the integration's version and refuse a mismatch with STALE, so a stale browser tab cannot act on a configuration someone else has changed.

Reading the delivery log

On-call → gear → the integration, or list_deliveries. Every attempt appears, refusals included. Refusals carry a reason, and each source page maps the reasons you are likely to see back to the setting that caused them. The short version:

A 404 from the endpoint means the integration id is wrong or the integration is disabled. Those answer identically on purpose: an unguessable URL should not become an oracle telling a scanner which ids exist.

Per-source guides

• Generic signed JSON — the full HMAC scheme, the request schema, bash and Node signing
• Grafana — webhook contact point
• Alertmanager — native webhook_config
• Dynatrace — problem notification with a custom payload template
• Datadog — monitor webhook with a `# On-call ingestion

Point your monitoring at BatonDeck and an alert becomes a ticket an agent can claim. Grafana, Alertmanager, Dynatrace, PagerDuty, Datadog, Google Cloud Monitoring and your own signed JSON all land on the same board your agents already work — correlated, prioritised, and routed to someone who will pick it up.

Your tool is not on that list? Use the custom webhook. You describe where the fields live in its payload and it works the same way, with no code from us and no release to wait for.

BatonDeck is not an alerting platform and does not do schedules. It is the layer that resolves the problem. If you need rotations, keep PagerDuty and point its webhook here.

What happens to an alert

1. Your source POSTs to https://<your-oncall-host>/i/<integrationId> — an unguessable URL minted when you create the integration.
2. A separate ingest service reads the body under a size cap, then verifies it against the auth mode you configured. This service holds no database access at all: the process that parses untrusted vendor JSON cannot reach your board.
3. The payload is normalized into canonical alert events. A grouped delivery becomes many — one event per alert in the batch.
4. Each event is forwarded separately, with its own delivery id, so a batch of twelve can never collapse into one ticket.
5. BatonDeck correlates each event by its fingerprint: a new fingerprint opens a ticket, a known one bumps the occurrence count on the ticket already open, and a resolution annotates or closes it depending on the integration's lifecycle mode. The bump needs the repeat to be a distinguishable delivery — an identical redelivery is treated as a retry, which is what makes retries safe. The per-source pages say where that bites.
6. The doorbell rings — the on-call agent you named, or every live agent in the project.
7. The agent claims the ticket, debugs, opens a PR and moves it to REVIEW. A human merges. Nothing auto-merges and nothing auto-releases.

A new incident ticket carries the alert title, a priority derived from its severity, incident.fingerprint / incident.source / incident.occurrences / incident.alertStatus custom fields, and a context item holding the alert's labels, annotations and links — enough for an agent to start cold.

The sources, and which auth each can actually use

Five of the seven vendors cannot compute the BatonDeck signature. Grafana, Alertmanager, Dynatrace, Datadog and GCP send a fixed request built by the vendor; none of them will HMAC a body for you. Pretending otherwise is how integrations end up misconfigured, so the honest mapping is:

mtls appears in the enum but is reserved and refused — it needs an external load balancer in front of the ingest service. Configuring it fails with that reason rather than silently accepting.

In api_key mode the function reads x-batondeck-key and NOTHING else. A key pasted into a source's "Authorization header" or "Basic auth" field is refused with api_key_mismatch against a config that looks entirely correct. This is the single most common misconfiguration, and it has its own warning on each of the four api_key pages.

Authorization is read in two modes and they are not interchangeable: jwt expects Authorization: Bearer <jwt>, and basic expects Authorization: Basic base64(user:pass) with the key as the password. Sending Basic to an api_key integration, or a bare key to a basic one, fails the same way — check the delivery log's reason (api_key_mismatch, basic_missing, basic_malformed) rather than guessing.

Creating an integration

The two paths differ in who generates the credential, which matters because it is shown exactly once either way:

• Web app — On-call → gear → New. The browser generates the API key (or the server mints the HMAC secret) and shows it once, on creation. Copy it then; it cannot be retrieved afterwards.
• MCP — you supply the API key yourself, as below.

An API key is stored only as a hash, so rotate_integration_secret does not apply to it — that tool is HMAC-only. To replace a lost API key, set a new one from the integration's Edit configuration form, or with update_integration { patch: { auth: { mode: "api_key", apiKey: "<new value>" } } }.

PagerDuty is the one source where you must supply the secret. It signs with a secret IT issues when you create the webhook subscription, so BatonDeck cannot mint one — creating a PagerDuty integration without auth.externalSecret is refused rather than producing a configuration that looks correct and fails every delivery.

Over MCP:

create_integration {
  projectId, name: "prod-grafana", source: "grafana", boardId: "B-…",
  auth: { mode: "api_key", apiKey: "<a long random value you generate>" },
  routing: { mode: "oncall", onCallAgent: "claude-oncall" }
}

Creating, updating and rotating an integration requires a project member or admin role; listing deliveries and sending a test event require master or above.

The response carries endpointPath — /i/ing_… — which is what you paste into your source's webhook settings, appended to your ingest host. Take it from the response rather than assembling it by hand.

Options worth setting at create time:

The secret is shown once — and there are three different answers

This trips people up because "the secret" means something different per auth mode:

• hmac, no externalSecret — BatonDeck mints a whsec_… value and returns it in the create response as secret, with shownOnce: true. It is encrypted at rest with no read-back. Store it before you close the response; losing it means rotate_integration_secret.
• hmac with externalSecret (PagerDuty) — nothing is returned. The source issued the credential and you already have it; echoing it back would put a third-party secret in a tool response for no gain.
• api_key — you supply apiKey and BatonDeck stores only its SHA-256 hash. Nothing is ever returned, so generate it somewhere you can keep it (openssl rand -base64 32).

Rotating an hmac secret keeps the previous one verifying for 24 hours so you can update the source without dropping deliveries. If a secret leaked, pass revokePrevious: true — the old one stops working within about a minute, not instantly: we cache verification material per server instance and nothing pushes an invalidation. If you need the window closed harder than that, disable the integration; that stops every delivery, on the same bound.

Proving it works

Two steps, and they prove different things — you need both.

1. send_test_event (or the Send test event button) injects a synthetic alert through the full internal path: it creates a real ticket labelled test on the target board, runs routing, rings the doorbell, and records a delivery. It consumes the integration's rate budget, which is part of what it proves.

It does not prove your credentials. The test event originates inside BatonDeck and never travels the webhook endpoint, so signature and API-key verification are skipped entirely. A green test event with a wrong header still fails on the first real alert.

2. Send one real request. Use your source's own test button (Grafana's contact-point Test, Alertmanager's route test, Dynatrace's Send test notification), or — for a generic integration — one signed curl with "status":"test". Then check the delivery log: a row that is not rejected means your credentials are right.

Close the test tickets afterwards; they are real tickets.

Pausing and removing an integration

Two different things, and only one is reversible:

Deleting is a tombstone, not an erasure, and the distinction matters when you are explaining an old incident:

• the endpoint stops answering immediately — a POST to it 404s, the same answer a wrong id gets;
• the delivery log survives, so the evidence behind incidents this integration filed is still readable (deliveries still expire on their own retention window);
• the incident tickets are untouched — deleting a config does not rewrite history;
• it cannot be undone. Create a new integration and re-point your source, which means a new endpoint URL and a new credential.

Both take the integration's version and refuse a mismatch with STALE, so a stale browser tab cannot act on a configuration someone else has changed.

Reading the delivery log

On-call → gear → the integration, or list_deliveries. Every attempt appears, refusals included. Refusals carry a reason, and each source page maps the reasons you are likely to see back to the setting that caused them. The short version:

A 404 from the endpoint means the integration id is wrong or the integration is disabled. Those answer identically on purpose: an unguessable URL should not become an oracle telling a scanner which ids exist.

Per-source guides

• Generic signed JSON — the full HMAC scheme, the request schema, bash and Node signing
• Grafana — webhook contact point
• Alertmanager — native webhook_config
• Dynatrace — problem notification with a custom payload template
• Datadog — monitor webhook with a -variable payload template
• PagerDuty — v3 webhook subscription and upstream routing
• Custom webhook — any other tool: describe where the fields live in its payload