> ## Documentation Index
> Fetch the complete documentation index at: https://docs.auto.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Incident Response

> A first responder woken by an alerting webhook: it correlates the alert with recent changes, posts an evidence-based triage, answers follow-ups in the thread, and drafts a fix PR when the cause is clear.

This example turns any alerting system that can send an HTTP `POST` — PagerDuty, Datadog monitors, Sentry alert rules, a `curl` in a runbook — into a trigger for a first responder. The agent investigates the alert against the codebase and recent commits, delivers a structured triage (to Slack when connected), stays in the thread for responder questions, and — when the evidence points at a clear, contained code fix — serves it up as a **draft PR** for humans to review and merge.

Use it when alerts routinely burn twenty minutes of "who touched this last?" before anyone forms a hypothesis.

## How it works

```mermaid theme={null}
flowchart TD
    A["Alert source POSTs JSON to<br/>the endpoint's ingest URL"] -->|"Authorization: Bearer …"| B["webhook.incident.opened"]
    B -->|spawn| C["Investigation session"]
    C --> D["Correlate: alert payload ×<br/>recent commits × observability"]
    D --> E["Triage report<br/>(+ Slack thread when connected)"]
    D -->|"clear, contained cause"| F["Draft fix PR"]
    G["Responder replies in thread"] -->|"deliver: attributedSessions"| C
```

The entry point is a **custom webhook trigger**: a trigger that declares `endpoint:` plus an `auth:` policy instead of a provider connection. At apply time, auto reserves a globally unique slug for the endpoint and returns its ingest URL in the apply receipt. The posted JSON body becomes the event payload verbatim — so the trigger's templates reference the alert's own fields (`{{title}}`, `{{severity}}`), with no `payload.` prefix and no provider schema in between. See [cron and webhooks](/reference/events/cron-and-webhooks).

The endpoint contract, in full:

* **URL**: `POST https://<your auto host>/api/v1/webhook-endpoints/{slug}/events`.
* **Auth**: this example uses `bearer_token` — the caller sends `Authorization: Bearer <secret>`, and the secret is resolved server-side from the project secret named by the trigger's `secretRef`. `hmac_sha256` (an `x-auto-signature-256` header carrying a hex HMAC of the raw body, optionally `sha256=`-prefixed) and `none` are the alternatives.
* **Body**: any JSON object. A top-level `"event"` string selects the event key — `"incident.opened"` routes as `webhook.incident.opened`; a body without one lands on the `webhook.received` fallback key.
* **Response**: `202` with `{ eventRecordId, created: true, routerStatus }` on first delivery; a repeated optional top-level `dedupKey` returns `200` with `created: false` and never re-routes.

Everything after ingest is ordinary routing: `routing: { kind: spawn }` starts one investigation session per alert.

## Install from the consolidated agent package

The packaged Incident Response role is published in `@auto/agents`:

```yaml .auto/agents/incident-response.yaml theme={null}
name: incident-response
imports:
  - "@auto/agents@latest/incident-response.yaml"
variables:
  repoFullName: acme/widgets
```

**Before the apply**, create the shared secret the endpoint will verify: a project secret named `incident-webhook-secret`, set from the web app's Secrets settings so the value never lands in the repository or a transcript. See [secrets](/reference/secrets).

Slack triage delivery is optional zero-configuration wiring on the standard `slack` connection and `#incidents` channel; without Slack, the run report is the complete triage. Merge for [GitHub Sync](/concepts/github-sync) to apply, take the endpoint's ingest URL from the apply receipt (from a session, `auto.webhooks.get` returns it too), and point your alerting system at it.

## The full configuration

```text theme={null}
.auto/
  agents/incident-response.yaml
  assets/sentinel.png
  fragments/environments/agent-runtime.yaml
```

```yaml .auto/fragments/environments/agent-runtime.yaml theme={null}
harness: claude-code
environment:
  name: agent-runtime
  image:
    kind: preset
    name: node24
  resources:
    memoryMB: 8192
```

```yaml .auto/agents/incident-response.yaml theme={null}
name: incident-response
model:
  provider: anthropic
  id: claude-opus-4-8
identity:
  displayName: Incident Response
  username: incident-response
  avatar:
    asset: .auto/assets/sentinel.png
  description: First responder for production alerts - delivers evidence-based triage, optionally posts to Slack, and drafts a fix PR.
imports:
  - ../fragments/environments/agent-runtime.yaml
systemPrompt: |
  You are the incident response agent for acme/widgets. When an alert
  arrives, your job is fast, evidence-based triage — not heroics.

  Investigation protocol:
  - Read the alert payload carefully; identify the affected service and
    the symptom.
  - Correlate with recent change: inspect the last day of commits on main
    in the mounted checkout (git log) and look for changes touching the
    affected area.
  - When an observability tool is available, pull the relevant logs,
    monitors, or metrics for the alert window before speculating.
  - Form a hypothesis with explicit confidence: likely cause, supporting
    evidence, and what would confirm or refute it.

  Reporting protocol:
  - Deliver the triage as this run's report. Open with one line —
    "[severity] service: one-line symptom" — then the alert link and the
    full triage: timeline, suspected cause with evidence, suggested next
    steps, and what you ruled out.
  - Slack delivery is optional and uses the standard `slack` connection
    name and `#incidents` channel. When the chat tool is available, also
    post one top-level message with the opening line and alert link, thread
    the full triage beneath it, and call auto.chat.subscribe for follow-up
    questions. When the tool is unavailable, do not treat Slack delivery as
    a failure; the run report remains the complete triage.
  - Incident details can be sensitive. Do not open a GitHub issue by
    default.

  Fix protocol (serve the fix on a platter):
  - When the evidence points at a specific code change with a clear,
    contained fix — a bad commit to revert, a config value to correct, a
    small patch — prepare it: create a focused branch from main in the
    mounted checkout, implement the minimal fix, push the branch, and open
    a draft pull request with create_pull_request.
  - The PR body states the hypothesis the fix encodes with its evidence
    and says how to verify it; keep it about the code change and leave the
    detailed incident narrative in your triage report. Reference the PR in
    the report and, when available, the Slack thread.
  - Keep the fix minimal and reversible; run the repo's relevant checks
    when the environment allows and report what you ran. Never force a fix:
    when the cause is uncertain or the change would sprawl, the triage with
    suggested next steps is a complete deliverable on its own.

  Hard limits: your only writes are the draft fix PR. Do not merge the PR,
  push to main, restart services, mutate infrastructure, or declare an
  incident resolved — humans review the fix and decide that. If the
  evidence is thin, say so plainly rather than manufacturing a conclusion.
initialPrompt: |
  A production alert arrived.

  Alert:
  - Title: {{title}}
  - Severity: {{severity}}
  - Service: {{service}}
  - Description: {{description}}
  - Link: {{link}}

  Investigate following your responder instructions, then deliver your
  triage as this run's report. When the chat tool is available, also post
  the triage to Slack #incidents and subscribe to the thread for follow-ups.
  If the evidence points at a clear, contained code fix, also open a draft
  fix PR and reference it in the report and, when available, the Slack
  thread.
mounts:
  - kind: git
    repository: acme/widgets
    mountPath: /workspace/repo
    ref: main
    depth: 100
    auth:
      kind: githubApp
      capabilities:
        contents: write
        pullRequests: write
        issues: none
        checks: read
        actions: read
workingDirectory: /workspace/repo
tools:
  auto:
    kind: local
    implementation: auto
  chat:
    kind: local
    implementation: chat
    auth:
      kind: connection
      provider: slack
      connection: slack
      optional: true
  github:
    kind: github
    tools:
      - pull_request_read
      - create_pull_request
      - update_pull_request
triggers:
  - name: incident-webhook
    event: webhook.incident.opened
    endpoint: incident-webhook
    auth:
      kind: bearer_token
      secretRef: incident-webhook-secret
    routing:
      kind: spawn
  - name: mention
    event: chat.message.mentioned
    connection: slack
    optional: true
    where:
      $.chat.provider: slack
      $.auto.authored: false
      $.auto.attributions:
        exists: false
    message: |
      {{message.author.userName}} mentioned you on Slack:

      {{message.text}}

      Channel: {{chat.channelId}}
      Thread: {{chat.threadId}}

      Reply in that thread with chat.send. If the user provides alert details
      or clearly asks for an incident investigation, handle it. If required
      context is missing, ask for the alert details. Otherwise, briefly explain
      that you investigate production alerts, optionally post triage to
      #incidents, open a draft fix PR when the cause is clear, and answer
      follow-up questions in the incident thread.
    routing:
      kind: spawn
  - name: thread-reply
    events:
      - chat.message.mentioned
      - chat.message.subscribed
    connection: slack
    optional: true
    where:
      $.chat.provider: slack
      $.auto.authored: false
      $.auto.attributions:
        exists: true
    message: |
      {{message.author.userName}} replied in your incident thread:

      {{message.text}}

      Channel: {{chat.channelId}}
      Thread: {{chat.threadId}}

      Answer in that thread with chat.send, keeping the evidence discipline
      from your instructions.
    routing:
      kind: deliver
      routeBy:
        kind: attributedSessions
      onUnmatched: drop
```

The write surface is deliberately narrow: `contents: write` + `pullRequests: write` exist so the agent can push a fix branch and open a draft PR, but there is no merge capability on the mount, so `merge_pull_request` is invisible and uncallable no matter what the prompt says — see [mounts](/reference/mounts). The prompt's payload fields (`{{title}}`, `{{severity}}`, `{{service}}`, `{{description}}`, `{{link}}`) assume the alert shape below; match them to whatever your alerting system actually sends, since the body passes through verbatim.

## Walkthrough

<Steps>
  <Step title="An alert fires">
    Your monitoring system POSTs to the ingest URL:

    ```sh theme={null}
    curl -X POST https://<your auto host>/api/v1/webhook-endpoints/<slug>/events \
      -H "Authorization: Bearer $INCIDENT_WEBHOOK_SECRET" \
      -H "Content-Type: application/json" \
      -d '{
        "event": "incident.opened",
        "dedupKey": "alert-7f3a",
        "title": "p99 latency breach on widgets-api",
        "severity": "sev2",
        "service": "widgets-api",
        "description": "p99 above 2s for 10 minutes on /v1/orders",
        "link": "https://monitoring.example.com/alerts/7f3a"
      }'
    ```

    The bearer token is verified against the `incident-webhook-secret` project secret in constant time; `"event": "incident.opened"` maps the body to the `webhook.incident.opened` key; `dedupKey` makes alert-source retries idempotent. The response is `202` with the recorded event id.
  </Step>

  <Step title="One session per alert">
    The `incident-webhook` trigger matches and spawns an investigation session. The raw alert body is the event payload, rendered into the `initialPrompt` — no translation layer to maintain.
  </Step>

  <Step title="Evidence before hypothesis">
    The sandbox boots with `acme/widgets@main` mounted 100 commits deep. The agent reads the alert, scans the last day of commits touching the affected area, pulls logs or metrics if an observability tool is wired (see the variation below), and forms a hypothesis with explicit confidence — what supports it, what would refute it.
  </Step>

  <Step title="The triage lands, and the thread stays live">
    The run's report is the triage: severity line, timeline, suspected cause with evidence, next steps, what was ruled out. With Slack connected, the agent posts one top-level message in `#incidents`, threads the full triage under it, and binds the thread so responder questions in it deliver straight back into this session (`attributedSessions`) — same evidence discipline, same context.
  </Step>

  <Step title="The fix on a platter">
    When the cause is a clear, contained change — a bad commit to revert, a config value to correct — the agent pushes a focused branch and opens a **draft** PR whose body states the hypothesis the fix encodes and how to verify it, then links it from the thread. It never merges, never pushes to main, and never declares the incident resolved; humans do that.
  </Step>
</Steps>

## Variations

* **HMAC instead of bearer.** For alert sources that sign payloads, switch the trigger to `auth: { kind: hmac_sha256, secretRef: incident-webhook-secret }`; the caller sends `x-auto-signature-256: sha256=<hex HMAC-SHA256 of the raw body>`.
* **Catch unshaped providers.** Sources you can't teach to send an `event` field land on `webhook.received`. Add a second trigger on `event: webhook.received` with `fallback: true` to the same endpoint and discriminate with `where` on the raw body — fallback triggers fire only when no non-fallback trigger on the same endpoint and event key matched.
* **Severity routing.** The payload is yours, so `where` filters over it directly: send `sev1` alerts to a paging flow and keep the agent on `$.severity: { in: [sev2, sev3] }` — one filter, no code.
* **Wire in observability.** Add a remote MCP tool for your observability stack so "pull the logs before speculating" is real, for example Datadog:

  ```yaml theme={null}
  tools:
    datadog:
      kind: mcp_remote
      description: Datadog logs, monitors, metrics, and dashboards.
      url: https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=all
      transport: streamable_http
      auth:
        kind: mcp_oauth
        connection: datadog-acme
  ```

  OAuth tokens for `mcp_oauth` tools are held and refreshed server-side; the sandbox never sees them. See [tools](/reference/tools).
* **Read-only responder.** If you want triage without the fix path, set the mount to `contents: read`, `pullRequests: read` and remove `create_pull_request`/`update_pull_request` — the investigation and thread behavior are unchanged.
* **Smoke test.** After apply, send the `curl` above with a test payload and confirm a session spawns and a triage thread appears in `#incidents`; reply in the thread and confirm the agent answers.