Skip to main content
Incident response has two entry points: a PagerDuty lifecycle responder and a custom-webhook responder for alert sources that send an HTTP POST. Both keep investigation evidence in a report, support optional Slack follow-ups, and prepare a draft PR only when the evidence supports a contained fix for humans to review. Use PagerDuty lifecycle handling when acknowledgement and resolution must return to the incident’s existing response. Use the pinned webhook installation below when you need a separate investigation for each custom alert.

PagerDuty responder (1.12.0)

The Incident Response role in @auto/agents@1.12.0 uses PagerDuty as the incident lifecycle authority. Its catalog entry requires PagerDuty; the configuration references a connection named pagerduty. Have a human connect and allocate it to the project before enabling the responder. The standard slack connection is optional, and Datadog is optional supporting evidence, not an authority for acknowledgement or resolution. See connections.
Release 1.12.0 replaces the custom-webhook entry point with PagerDuty lifecycle triggers; it is not a drop-in upgrade for custom alert senders. Keep @auto/agents@1.11.0/incident-response.yaml pinned to retain the webhook workflow below. A merged release does not prove that the deployed registry or an existing installation has advanced; check the resolved managed-template version before migrating.
Once the required connection is available, select the PagerDuty entrypoint explicitly:
.auto/agents/incident-response.yaml
All three PagerDuty triggers route to provider.entity, whose incident key comes from the PagerDuty incident ID: The resolution trigger is:
These are provider events, not webhook.incident.opened payloads. The role reads incident.id, incident.status, incident.title, incident.url, and service.summary; it does not use the custom alert’s top-level title, severity, and link fields. Its PagerDuty triggers replace the incident-webhook trigger rather than reuse its endpoint or shared secret. See triggers.

Allowed actions and reports

The role declares this local tool:
The responder can add a PagerDuty note or acknowledge an incident. It must not resolve, assign, page, or escalate an incident; paging and escalation are unsupported, and a human denial ends that path. Receiving a resolution event is not permission to resolve an incident itself. The prompt calls for one bounded report containing the timeline, impact, evidence, confidence, and safest next steps. With Slack available, it keeps one incident thread for follow-ups. Missing provider access must be reported as missing evidence or an unavailable action. Test-fire output is labeled TEST, not treated as production. A draft fix PR is appropriate only for a small, reversible change supported by evidence; the responder must not merge or change production.

Custom webhook: how it works

The entry point is a custom webhook trigger: a trigger that declares endpoint: plus an auth: policy instead of a provider connection. At apply time, auto reserves a globally unique slug for the endpoint and returns its ingest URL in the apply receipt. The posted JSON body becomes the event payload verbatim — so the trigger’s templates reference the alert’s own fields ({{title}}, {{severity}}), with no payload. prefix and no provider schema in between. See cron and webhooks. The endpoint contract, in full:
  • URL: POST https://<your auto host>/api/v1/webhook-endpoints/{slug}/events.
  • Auth: this example uses bearer_token — the caller sends Authorization: Bearer <secret>, and the secret is resolved server-side from the project secret named by the trigger’s secretRef. hmac_sha256 (an x-auto-signature-256 header carrying a hex HMAC of the raw body, optionally sha256=-prefixed) and none are the alternatives.
  • Body: any JSON object. A top-level "event" string selects the event key — "incident.opened" routes as webhook.incident.opened; a body without one lands on the webhook.received fallback key.
  • Response: 202 with { eventRecordId, created: true, routerStatus } on first delivery; a repeated optional top-level dedupKey returns 200 with created: false and never re-routes.
Everything after ingest is ordinary routing: routing: { kind: spawn } starts one investigation session per alert.

Install the published webhook responder

Pin the webhook-based Incident Response role to @auto/agents@1.11.0:
.auto/agents/incident-response.yaml
Before the apply, create the shared secret the endpoint will verify: a project secret named incident-webhook-secret, set from the web app’s Secrets settings so the value never lands in the repository or a transcript. See secrets. Slack triage delivery is optional zero-configuration wiring on the standard slack connection and #incidents channel; without Slack, the run report is the complete triage. Merge for GitHub Sync to apply, take the endpoint’s ingest URL from the apply receipt (from a session, auto.webhooks.get returns it too), and point your alerting system at it.

The full custom-webhook configuration

.auto/fragments/environments/agent-runtime.yaml
.auto/agents/incident-response.yaml
The write surface is deliberately narrow: contents: write + pullRequests: write exist so the agent can push a fix branch and open a draft PR, but there is no merge capability on the mount, so merge_pull_request is invisible and uncallable no matter what the prompt says — see mounts. The prompt’s payload fields ({{title}}, {{severity}}, {{service}}, {{description}}, {{link}}) assume the alert shape below; match them to whatever your alerting system actually sends, since the body passes through verbatim.

Walkthrough

1

An alert fires

Your monitoring system POSTs to the ingest URL:
The bearer token is verified against the incident-webhook-secret project secret in constant time; "event": "incident.opened" maps the body to the webhook.incident.opened key; dedupKey makes alert-source retries idempotent. The response is 202 with the recorded event id.
2

One session per alert

The incident-webhook trigger matches and spawns an investigation session. The raw alert body is the event payload, rendered into the initialPrompt — no translation layer to maintain.
3

Evidence before hypothesis

The sandbox boots with acme/widgets@main mounted 100 commits deep. The agent reads the alert, scans the last day of commits touching the affected area, pulls logs or metrics if an observability tool is wired (see the variation below), and forms a hypothesis with explicit confidence — what supports it, what would refute it.
4

The triage lands, and the thread stays live

The run’s report is the triage: severity line, timeline, suspected cause with evidence, next steps, what was ruled out. With Slack connected, the agent posts one top-level message in #incidents, threads the full triage under it, and binds the thread so responder questions in it deliver straight back into this session (attributedSessions) — same evidence discipline, same context.
5

The fix on a platter

When the cause is a clear, contained change — a bad commit to revert, a config value to correct — the agent pushes a focused branch and opens a draft PR whose body states the hypothesis the fix encodes and how to verify it, then links it from the thread. It never merges, never pushes to main, and never declares the incident resolved; humans do that.

Variations

  • HMAC instead of bearer. For alert sources that sign payloads, switch the trigger to auth: { kind: hmac_sha256, secretRef: incident-webhook-secret }; the caller sends x-auto-signature-256: sha256=<hex HMAC-SHA256 of the raw body>.
  • Catch unshaped providers. Sources you can’t teach to send an event field land on webhook.received. Add a second trigger on event: webhook.received with fallback: true to the same endpoint and discriminate with where on the raw body — fallback triggers fire only when no non-fallback trigger on the same endpoint and event key matched.
  • Severity routing. The payload is yours, so where filters over it directly: send sev1 alerts to a paging flow and keep the agent on $.severity: { in: [sev2, sev3] } — one filter, no code.
  • Wire in observability. Add a remote MCP tool for your observability stack so “pull the logs before speculating” is real, for example Datadog:
    OAuth tokens for mcp_oauth tools are held and refreshed server-side; the sandbox never sees them. See tools.
  • Read-only responder. If you want triage without the fix path, set the mount to contents: read, pullRequests: read and remove create_pull_request/update_pull_request — the investigation and thread behavior are unchanged.
  • Smoke test. After apply, send the curl above with a test payload and confirm a session spawns and a triage thread appears in #incidents; reply in the thread and confirm the agent answers.