Incident Response

This example turns any alerting system that can send an HTTP POST — PagerDuty, Datadog monitors, Sentry alert rules, a curl in a runbook — into a trigger for a first responder. The agent investigates the alert against the codebase and recent commits, delivers a structured triage (to Slack when connected), stays in the thread for responder questions, and — when the evidence points at a clear, contained code fix — serves it up as a draft PR for humans to review and merge. Use it when alerts routinely burn twenty minutes of “who touched this last?” before anyone forms a hypothesis.

How it works

The entry point is a custom webhook trigger: a trigger that declares endpoint: plus an auth: policy instead of a provider connection. At apply time, auto reserves a globally unique slug for the endpoint and returns its ingest URL in the apply receipt. The posted JSON body becomes the event payload verbatim — so the trigger’s templates reference the alert’s own fields ({{title}}, {{severity}}), with no payload. prefix and no provider schema in between. See cron and webhooks. The endpoint contract, in full:

URL: POST https://<your auto host>/api/v1/webhook-endpoints/{slug}/events.
Auth: this example uses bearer_token — the caller sends Authorization: Bearer <secret>, and the secret is resolved server-side from the project secret named by the trigger’s secretRef. hmac_sha256 (an x-auto-signature-256 header carrying a hex HMAC of the raw body, optionally sha256=-prefixed) and none are the alternatives.
Body: any JSON object. A top-level "event" string selects the event key — "incident.opened" routes as webhook.incident.opened; a body without one lands on the webhook.received fallback key.
Response: 202 with { eventRecordId, created: true, routerStatus } on first delivery; a repeated optional top-level dedupKey returns 200 with created: false and never re-routes.

Everything after ingest is ordinary routing: routing: { kind: spawn } starts one investigation session per alert.

Install from the consolidated agent package

The packaged Incident Response role is published in @auto/agents:

.auto/agents/incident-response.yaml

name: incident-response
imports:
  - "@auto/agents@latest/incident-response.yaml"
variables:
  repoFullName: acme/widgets

Before the apply, create the shared secret the endpoint will verify: a project secret named incident-webhook-secret, set from the web app’s Secrets settings so the value never lands in the repository or a transcript. See secrets. Slack triage delivery is optional zero-configuration wiring on the standard slack connection and #incidents channel; without Slack, the run report is the complete triage. Merge for GitHub Sync to apply, take the endpoint’s ingest URL from the apply receipt (from a session, auto.webhooks.get returns it too), and point your alerting system at it.

The full configuration

.auto/
  agents/incident-response.yaml
  assets/sentinel.png
  fragments/environments/agent-runtime.yaml

.auto/fragments/environments/agent-runtime.yaml

harness: claude-code
environment:
  name: agent-runtime
  image:
    kind: preset
    name: node24
  resources:
    memoryMB: 8192

.auto/agents/incident-response.yaml

name: incident-response
model:
  provider: anthropic
  id: claude-opus-4-8
identity:
  displayName: Incident Response
  username: incident-response
  avatar:
    asset: .auto/assets/sentinel.png
  description: First responder for production alerts - delivers evidence-based triage, optionally posts to Slack, and drafts a fix PR.
imports:
  - ../fragments/environments/agent-runtime.yaml
systemPrompt: |
  You are the incident response agent for acme/widgets. When an alert
  arrives, your job is fast, evidence-based triage — not heroics.

  Investigation protocol:
  - Read the alert payload carefully; identify the affected service and
    the symptom.
  - Correlate with recent change: inspect the last day of commits on main
    in the mounted checkout (git log) and look for changes touching the
    affected area.
  - When an observability tool is available, pull the relevant logs,
    monitors, or metrics for the alert window before speculating.
  - Form a hypothesis with explicit confidence: likely cause, supporting
    evidence, and what would confirm or refute it.

  Reporting protocol:
  - Deliver the triage as this run's report. Open with one line —
    "[severity] service: one-line symptom" — then the alert link and the
    full triage: timeline, suspected cause with evidence, suggested next
    steps, and what you ruled out.
  - Slack delivery is optional and uses the standard `slack` connection
    name and `#incidents` channel. When the chat tool is available, also
    post one top-level message with the opening line and alert link, thread
    the full triage beneath it, and call auto.chat.subscribe for follow-up
    questions. When the tool is unavailable, do not treat Slack delivery as
    a failure; the run report remains the complete triage.
  - Incident details can be sensitive. Do not open a GitHub issue by
    default.

  Fix protocol (serve the fix on a platter):
  - When the evidence points at a specific code change with a clear,
    contained fix — a bad commit to revert, a config value to correct, a
    small patch — prepare it: create a focused branch from main in the
    mounted checkout, implement the minimal fix, push the branch, and open
    a draft pull request with create_pull_request.
  - The PR body states the hypothesis the fix encodes with its evidence
    and says how to verify it; keep it about the code change and leave the
    detailed incident narrative in your triage report. Reference the PR in
    the report and, when available, the Slack thread.
  - Keep the fix minimal and reversible; run the repo's relevant checks
    when the environment allows and report what you ran. Never force a fix:
    when the cause is uncertain or the change would sprawl, the triage with
    suggested next steps is a complete deliverable on its own.

  Hard limits: your only writes are the draft fix PR. Do not merge the PR,
  push to main, restart services, mutate infrastructure, or declare an
  incident resolved — humans review the fix and decide that. If the
  evidence is thin, say so plainly rather than manufacturing a conclusion.
initialPrompt: |
  A production alert arrived.

  Alert:
  - Title: {{title}}
  - Severity: {{severity}}
  - Service: {{service}}
  - Description: {{description}}
  - Link: {{link}}

  Investigate following your responder instructions, then deliver your
  triage as this run's report. When the chat tool is available, also post
  the triage to Slack #incidents and subscribe to the thread for follow-ups.
  If the evidence points at a clear, contained code fix, also open a draft
  fix PR and reference it in the report and, when available, the Slack
  thread.
mounts:
  - kind: git
    repository: acme/widgets
    mountPath: /workspace/repo
    ref: main
    depth: 100
    auth:
      kind: githubApp
      capabilities:
        contents: write
        pullRequests: write
        issues: none
        checks: read
        actions: read
workingDirectory: /workspace/repo
tools:
  auto:
    kind: local
    implementation: auto
  chat:
    kind: local
    implementation: chat
    auth:
      kind: connection
      provider: slack
      connection: slack
      optional: true
  github:
    kind: github
    tools:
      - pull_request_read
      - create_pull_request
      - update_pull_request
triggers:
  - name: incident-webhook
    event: webhook.incident.opened
    endpoint: incident-webhook
    auth:
      kind: bearer_token
      secretRef: incident-webhook-secret
    routing:
      kind: spawn
  - name: mention
    event: chat.message.mentioned
    connection: slack
    optional: true
    where:
      $.chat.provider: slack
      $.auto.authored: false
      $.auto.attributions:
        exists: false
    message: |
      {{message.author.userName}} mentioned you on Slack:

      {{message.text}}

      Channel: {{chat.channelId}}
      Thread: {{chat.threadId}}

      Reply in that thread with chat.send. If the user provides alert details
      or clearly asks for an incident investigation, handle it. If required
      context is missing, ask for the alert details. Otherwise, briefly explain
      that you investigate production alerts, optionally post triage to
      #incidents, open a draft fix PR when the cause is clear, and answer
      follow-up questions in the incident thread.
    routing:
      kind: spawn
  - name: thread-reply
    events:
      - chat.message.mentioned
      - chat.message.subscribed
    connection: slack
    optional: true
    where:
      $.chat.provider: slack
      $.auto.authored: false
      $.auto.attributions:
        exists: true
    message: |
      {{message.author.userName}} replied in your incident thread:

      {{message.text}}

      Channel: {{chat.channelId}}
      Thread: {{chat.threadId}}

      Answer in that thread with chat.send, keeping the evidence discipline
      from your instructions.
    routing:
      kind: deliver
      routeBy:
        kind: attributedSessions
      onUnmatched: drop

The write surface is deliberately narrow: contents: write + pullRequests: write exist so the agent can push a fix branch and open a draft PR, but there is no merge capability on the mount, so merge_pull_request is invisible and uncallable no matter what the prompt says — see mounts. The prompt’s payload fields ({{title}}, {{severity}}, {{service}}, {{description}}, {{link}}) assume the alert shape below; match them to whatever your alerting system actually sends, since the body passes through verbatim.

Walkthrough

An alert fires

Your monitoring system POSTs to the ingest URL:

curl -X POST https://<your auto host>/api/v1/webhook-endpoints/<slug>/events \
  -H "Authorization: Bearer $INCIDENT_WEBHOOK_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "event": "incident.opened",
    "dedupKey": "alert-7f3a",
    "title": "p99 latency breach on widgets-api",
    "severity": "sev2",
    "service": "widgets-api",
    "description": "p99 above 2s for 10 minutes on /v1/orders",
    "link": "https://monitoring.example.com/alerts/7f3a"
  }'

The bearer token is verified against the incident-webhook-secret project secret in constant time; "event": "incident.opened" maps the body to the webhook.incident.opened key; dedupKey makes alert-source retries idempotent. The response is 202 with the recorded event id.

One session per alert

The incident-webhook trigger matches and spawns an investigation session. The raw alert body is the event payload, rendered into the initialPrompt — no translation layer to maintain.

Evidence before hypothesis

The sandbox boots with acme/widgets@main mounted 100 commits deep. The agent reads the alert, scans the last day of commits touching the affected area, pulls logs or metrics if an observability tool is wired (see the variation below), and forms a hypothesis with explicit confidence — what supports it, what would refute it.

The triage lands, and the thread stays live

The run’s report is the triage: severity line, timeline, suspected cause with evidence, next steps, what was ruled out. With Slack connected, the agent posts one top-level message in #incidents, threads the full triage under it, and binds the thread so responder questions in it deliver straight back into this session (attributedSessions) — same evidence discipline, same context.

The fix on a platter

When the cause is a clear, contained change — a bad commit to revert, a config value to correct — the agent pushes a focused branch and opens a draft PR whose body states the hypothesis the fix encodes and how to verify it, then links it from the thread. It never merges, never pushes to main, and never declares the incident resolved; humans do that.

Variations

HMAC instead of bearer. For alert sources that sign payloads, switch the trigger to auth: { kind: hmac_sha256, secretRef: incident-webhook-secret }; the caller sends x-auto-signature-256: sha256=<hex HMAC-SHA256 of the raw body>.
Catch unshaped providers. Sources you can’t teach to send an event field land on webhook.received. Add a second trigger on event: webhook.received with fallback: true to the same endpoint and discriminate with where on the raw body — fallback triggers fire only when no non-fallback trigger on the same endpoint and event key matched.
Severity routing. The payload is yours, so where filters over it directly: send sev1 alerts to a paging flow and keep the agent on $.severity: { in: [sev2, sev3] } — one filter, no code.

Wire in observability. Add a remote MCP tool for your observability stack so “pull the logs before speculating” is real, for example Datadog:

tools:
  datadog:
    kind: mcp_remote
    description: Datadog logs, monitors, metrics, and dashboards.
    url: https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=all
    transport: streamable_http
    auth:
      kind: mcp_oauth
      connection: datadog-acme

OAuth tokens for mcp_oauth tools are held and refreshed server-side; the sandbox never sees them. See tools.

Read-only responder. If you want triage without the fix path, set the mount to contents: read, pullRequests: read and remove create_pull_request/update_pull_request — the investigation and thread behavior are unchanged.
Smoke test. After apply, send the curl above with a test payload and confirm a session spawns and a triage thread appears in #incidents; reply in the thread and confirm the agent answers.

Get Started

Concepts

Examples

Reference

Incident Response

How it works

Install from the consolidated agent package

The full configuration

Walkthrough

Variations

​How it works

​Install from the consolidated agent package

​The full configuration

​Walkthrough

​Variations

How it works

Install from the consolidated agent package

The full configuration

Walkthrough

Variations