From f46dee7c8de1c63579c704bdf357e71f0fd46639 Mon Sep 17 00:00:00 2001 From: Evgeny Pavlov Date: Thu, 13 Aug 2026 12:14:45 -0700 Subject: [PATCH] Create Hackbot docs --- docs/README.md | 2 + docs/hackbot/README.md | 83 +++++++++++++++++++ docs/hackbot/actions.md | 155 +++++++++++++++++++++++++++++++++++ docs/hackbot/agents.md | 109 ++++++++++++++++++++++++ docs/hackbot/api.md | 137 +++++++++++++++++++++++++++++++ docs/hackbot/architecture.md | 119 +++++++++++++++++++++++++++ docs/hackbot/deployment.md | 138 +++++++++++++++++++++++++++++++ docs/hackbot/runtime.md | 118 ++++++++++++++++++++++++++ docs/hackbot/security.md | 109 ++++++++++++++++++++++++ docs/hackbot/tools.md | 133 ++++++++++++++++++++++++++++++ docs/hackbot/tracing.md | 5 +- docs/hackbot/triggers.md | 111 +++++++++++++++++++++++++ 12 files changed, 1217 insertions(+), 2 deletions(-) create mode 100644 docs/hackbot/README.md create mode 100644 docs/hackbot/actions.md create mode 100644 docs/hackbot/agents.md create mode 100644 docs/hackbot/api.md create mode 100644 docs/hackbot/architecture.md create mode 100644 docs/hackbot/deployment.md create mode 100644 docs/hackbot/runtime.md create mode 100644 docs/hackbot/security.md create mode 100644 docs/hackbot/tools.md create mode 100644 docs/hackbot/triggers.md diff --git a/docs/README.md b/docs/README.md index c28dab0c05..80931d0f47 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,3 +1,5 @@ +- [Hackbot platform](hackbot/README.md) — agent runtime, services, deployment + Detailed documentation per model - [Regressor model for predicting risky commits](models/regressor.md) diff --git a/docs/hackbot/README.md b/docs/hackbot/README.md new file mode 100644 index 0000000000..de6156ecb8 --- /dev/null +++ b/docs/hackbot/README.md @@ -0,0 +1,83 @@ +# Hackbot + +Hackbot is a platform for running **autonomous agents against Mozilla's engineering +systems**: Firefox source, Bugzilla, Phabricator, Taskcluster CI, TestRail. + +An agent is a container that gets started with a small set of inputs, investigates +something, and exits. It reports what it found, and — crucially — it does not change +the outside world while it runs. It _records_ what it wants to change; the platform +applies that afterwards. Everything else in the platform exists to make that shape +work: start the container, give it what it needs, collect what it produced, apply +what it proposed. + +## The life of a run + +``` + trigger control plane execution + ─────── ───────────── ───────── + + hackbot-ui ─┐ + pulse- ├──> hackbot-api ──> Cloud Run Job ──> ┌──────────┬─────────────┐ + listener ─┤ (validate, (one execution │ agent │ broker │ + Phabricator ─┘ record Run, per run) │ (no │ (holds keys,│ + webhook mint upload │ creds) │ if needed) │ + policy) └────┬─────┴─────────────┘ + ▲ │ + │ artifacts + + run finished summary.json + (Pub/Sub) │ + │ ▼ + └────────────────────── GCS results bucket + │ + ├─> finalize: read summary, list artifacts, + │ set terminal status + └─> apply recorded actions (Bugzilla comment, + Phabricator revision, Slack, ...) +``` + +1. **Trigger.** Someone or something calls `POST /agents/{name}/runs` with inputs. +2. **Dispatch.** hackbot-api validates the inputs, records a `Run`, mints a signed + upload policy scoped to that run, and starts one Cloud Run Job execution. +3. **Run.** The runtime inside the container prepares what the agent declared it needs + (source checkout, Firefox build paths, model credentials), calls the agent's + `main()`, and writes `summary.json` plus artifacts to the results bucket. +4. **Finalize.** A completion event brings the run to a terminal state and publishes + `run.completed`. +5. **Apply.** The actions the agent recorded become real API calls — automatically for + opted-in agents, on demand from the UI otherwise. + +## Where to read next + +| If you want to… | Read | +| ---------------------------------------------------------- | ---------------------------------- | +| Understand the components and why they're split that way | [architecture.md](architecture.md) | +| Write or modify an agent | [agents.md](agents.md) | +| Know what the runtime hands your agent | [runtime.md](runtime.md) | +| Find a tool your agent can call, or add one | [tools.md](tools.md) | +| Understand how agents change the world (record-then-apply) | [actions.md](actions.md) | +| Work on the control-plane service | [api.md](api.md) | +| Know how runs get started | [triggers.md](triggers.md) | +| Reason about credentials and trust boundaries | [security.md](security.md) | +| Deploy, configure, or run things locally | [deployment.md](deployment.md) | +| Look at traces of a run | [tracing.md](tracing.md) | + +## Code map + +| Path | What it is | +| --------------------------------------------------- | ------------------------------------------------------------------- | +| `libs/hackbot-runtime/` | The in-container runtime: agent contract, context, results, actions | +| `libs/agent-tools/` | The tools a model can call: declarations + per-framework adapters | +| `libs/phabricator-client/`, `libs/testrail-client/` | Shared API clients | +| `agents//` | One self-contained agent: logic, image, local compose | +| `services/hackbot-api/` | Control plane (FastAPI): runs, artifacts, actions, webhooks | +| `services/hackbot-ui/` | Web UI (Next.js): trigger, observe, review and apply actions | +| `services/hackbot-pulse-listener/` | Watches Taskcluster CI failures and dispatches repair runs | + +## Conventions in these docs + +These docs cover **design and integration** — the contracts between the parts and the +decisions worth knowing before changing something — and stop short of restating code. Each +fact lives in one file; the others link to it. + +Every PR that changes the Hackbot runtime, deployment, or agents should leave these docs +true. diff --git a/docs/hackbot/actions.md b/docs/hackbot/actions.md new file mode 100644 index 0000000000..b17d8ecc5d --- /dev/null +++ b/docs/hackbot/actions.md @@ -0,0 +1,155 @@ +# Actions: record now, apply later + +An agent never mutates Bugzilla, Phabricator, TestRail or Slack while it runs. It calls a +tool that **records what it intends to do**; hackbot-api performs it after the run has +finished and is known good. + +Why the indirection: + +- A run that fails halfway leaves **no half-applied side effects**. +- Every intent is **reviewable** — visible in `summary.json` and in the UI, with the + agent's own stated reasoning, before anything lands. +- Applying is **idempotent and retryable**, which matters because the event delivery that + drives it is at-least-once. +- Auto-apply is **per-agent opt-in**, so a new or unproven agent can run in + propose-only mode with no code change. + +## Recording (inside the run) + +The write-action tools are declared in `hackbot_runtime/actions/` — one module per domain, +using the same `@tool` decorator as [read tools](tools.md). The tool context is the +`ActionsRecorder`, and each handler does one thing: `recorder.record(...)` and return a +confirmation string. + +```python +recorder.record( + "phabricator.submit_patch", # . + {"bug_id": ..., "title": ...}, # params the apply step will need + reasoning="why the agent is doing this", # audit trail + attachments={"log": Path(...)}, # optional files, published as artifacts + ref="patch", # optional label, see cross-references below +) +``` + +The recorded list becomes `summary.json`'s `actions` array. Attachments are published +under the stable key `attachments//` and referenced by that key — the +local path disappears with the container. + +### The catalog + +Nine action types, each with a declaration the agent calls and a handler that applies it. +`actions/handlers/registry.py` is the authoritative type → handler map. + +As with read tools, nothing is exposed by default: an agent lists the dotted types it may +record in its `config.py` and passes them to `actions_server_for`, which builds a server +carrying only those. `bug-fix`, for instance, allows `phabricator.submit_patch` on a fresh +triage run but swaps it for `phabricator.update_patch` on a follow-up. + +| Action type | Records the intent to… | Params | +| --------------------------- | ------------------------------------------ | ---------------------------------- | +| `bugzilla.update_bug` | Change a bug's fields | `bug_id`, `changes` | +| `bugzilla.add_comment` | Comment on a bug | `bug_id`, `text`, `is_private` | +| `bugzilla.add_attachment` | Attach a file to a bug | `bug_id`, + a `file` attachment | +| `bugzilla.create_bug` | File a new bug | the new bug's fields | +| `phabricator.submit_patch` | Deliver a fix as a **new** revision | `bug_id`, `title`, `summary` | +| `phabricator.update_patch` | Add a new diff to an **existing** revision | `revision_id` | +| `phabricator.add_comment` | Reply on a revision without changing code | `revision_id`, `text` | +| `testrail.submit_test_plan` | Submit a generated test plan to TestRail | the validated feature + test cases | +| `slack.post_message` | Post a message to Slack | `channel`, `text` | + +All but `testrail.submit_test_plan` take a **`reasoning`** argument — a free-text audit trail +stored on the action and shown in the UI beside the proposed change. `phabricator.submit_patch` +is the only model-facing tool that exposes **`ref`** (see cross-references below). + +`testrail` and `slack` also provide `record_test_plan` / `record_message` helpers that agent +code calls directly rather than the model choosing to — for an action the agent always takes +once it has a result, not one the model decides on. + +`bugzilla.add_comment` appends a feedback-reaction footer to every recorded comment, and +`is_private=true` marks it security-group-only. + +Adding a type is a declaration in the domain module plus one line in the handler registry. +The dispatch loop never changes. + +### The two patch actions + +`submit_patch` and `update_patch` are deliberately separate, each taking only the +parameters its own case needs, so a model cannot create a revision when it meant to update +one by getting an optional argument wrong. That is the general pattern for write-actions: +prefer several narrow tools over one with mode flags. + +Neither takes a patch file. The agent's final working-tree state _is_ the diff — which is +why the runtime builds the Phabricator payload during `publish_changes`, while the checkout +still exists. + +### Hooks + +`ActionsRecorder` supports per-type hooks that run before an action is appended. A hook may +mutate the action (enrichment) or raise (validation gate — nothing is recorded and no +attachment is published). They live on the recorder rather than in the tool declarations so +the runtime can attach cross-cutting behaviour without every handler knowing about it. + +The Searchfox permalink expansion on `bugzilla.add_comment` is the working example: the +agent writes `{{searchfox.permalink}}/path/to/file.js#412`, the hook expands it at record +time, and the comment awaiting review already shows clickable URLs. + +## Applying (after the run) + +Triggered by the `run.completed` event, on a subscription filtered to **succeeded** runs. + +1. **Record rows.** Every entry in `summary.json`'s `actions` is upserted as a + `run_actions` row (`pending`), keyed `(run_id, idx)`. This happens for _all_ succeeded + runs, whether or not the agent auto-applies, so the UI can always show and apply them. +2. **Apply, if opted in.** With `auto_apply_actions=True` on the agent's registry entry, + pending rows are applied immediately. Otherwise they wait for a human to click apply. +3. **Dispatch.** Each row's `type` selects a handler from the registry. The handler gets + the params and an `ApplyContext` — which can `download_artifact(key)` without knowing + GCS is behind it, keeping the runtime library free of a storage dependency. +4. **Stamp.** The row records `applied` or `failed`, its result, and its error. Only a real + success sets `applied_at`. + +Rows are committed one action at a time, and an `applied` row is never reapplied — so a +retried event or a repeated manual apply-all is safe and resumes where it stopped. + +Actions from a run that is not `succeeded` are never applied. The subscription filters +them out and the applier checks again, because acting on a run that never reached a +verified-good state is not wanted even if it recorded something before erroring. + +## Cross-action references + +An action's result often isn't known until it's applied — a Phabricator revision has no URL +until it exists. So a later action can reference an earlier one's result by label: + +``` +submit_patch(..., ref="patch") +add_comment(text="Patch up for review: {{actions.patch.url}}") +``` + +`{{actions..}}` is substituted at apply time, recursively through params. +Resolution draws on rows already `applied` in earlier passes as well as this one, so a +later manual apply can still reference an earlier action's result. + +An unresolvable placeholder is **left as-is and logged**, rather than raising. The action +then fails with an error a human can read, instead of silently posting mangled text. + +## Bugzilla coalescing + +Same-bug field changes are merged with the closest comment into a single +`PUT /bug/{id}`, so Bugzilla applies them as one transaction — one bugmail, one history +entry, instead of a burst. Other comments on that bug still apply separately. A group is +applied at its last member's index, once every earlier dependency has resolved, and any +group whose rows carry a `ref` is excluded (nothing should reference a coalesced member's +result). + +## Where to look + +| Concern | File | +| -------------------------------- | ---------------------------------------------------------- | +| Recording mechanics, hooks | `libs/hackbot-runtime/hackbot_runtime/actions/recorder.py` | +| Action declarations (per domain) | `libs/hackbot-runtime/hackbot_runtime/actions/*.py` | +| Apply-side handlers | `libs/hackbot-runtime/hackbot_runtime/actions/handlers/` | +| Type → handler map | `.../actions/handlers/registry.py` | +| Orchestration, refs, coalescing | `services/hackbot-api/app/actions_applier.py` | + +Record side and apply side deliberately live in the **same library**, so the set of +actions an agent can request and the set the platform can apply cannot drift apart. diff --git a/docs/hackbot/agents.md b/docs/hackbot/agents.md new file mode 100644 index 0000000000..e6eb42ef6a --- /dev/null +++ b/docs/hackbot/agents.md @@ -0,0 +1,109 @@ +# Agents + +> For the hands-on recipe — copy the reference agent, folder layout, running it locally — +> see [`agents/README.md`](../../agents/README.md). This page covers the contract and how +> an agent integrates with the platform. + +## The contract + +An agent is a Python package started as `python -m hackbot_agents.`. It owes the +platform three things: + +```python +class AgentInputs(BaseSettings): # per-run inputs, read from env (bug_id <- BUG_ID) + bug_id: int + +async def main(ctx: HackbotContext) -> BugFixResult: + ... + +run_async(main) # runtime takes over: config, auth, summary, exit +``` + +- **Success** is returning a `HackbotAgentResult` subclass. It lands in + `summary.json`'s `findings`. +- **Failure** is raising. `AgentError` for an expected, explainable failure; anything else + for a crash. Either way the runtime writes `summary.json` with `status: "error"` and + exits non-zero. +- **`ctx`** is the agent's only window to the platform. + +Everything around `main()` — config discovery, credentials, tracing, publishing artifacts, +writing `summary.json`, the exit code — is the runtime's job: [runtime.md](runtime.md). + +## What an agent declares: `hackbot.toml` + +Only capabilities the platform must **prepare** on the agent's behalf. Everything is +optional; an agent that needs nothing prepared ships a file with only comments. + +```toml +[source] # shallow checkout, prepared on request +repo_url = "https://github.com/mozilla-firefox/firefox.git" +checkout_path = "/workspace/firefox" +ref = "..." # optional; SOURCE_REF overrides per run + +[firefox] # Firefox build paths from that checkout +enabled = true +objdir = "objdir-ff-asan" +``` + +Not here: per-run inputs, secrets, model choice, tool selection. Those are environment +or code. + +## The catalog + +| Agent | Does | Source | Firefox build | Auto-applies actions | +| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | :----: | :-----------: | :------------------: | +| `bug-fix` | Triage a Bugzilla bug and produce a candidate fix as a Phabricator revision. Also handles `@hackbot` follow-ups on an existing revision. | yes | yes | **yes** | +| `test-repair` | Classify a CI test failure as regression or intermittent, blame the culprit commit, propose a fix. | yes | yes | **yes** | +| `build-repair` | Analyze a Firefox build failure at a specific commit and produce a candidate fix. | yes | yes | no | +| `frontend-triage` | Read-only root-cause analysis and fix plan for a desktop frontend bug. | yes | no | no | +| `autowebcompat-repro` | Reproduce a web-compatibility report in headless Firefox via DevTools MCP. | no | no | no | +| `test-plan-generator` | Generate Firefox QA test cases, run them through DevTools MCP, report results. | no | no | no | + +Two shapes recur. **Source agents** (`bug-fix`, `test-repair`, `build-repair`) check out +Firefox, often build it, edit the tree, and let the runtime capture the diff. **Browser +agents** (`autowebcompat-repro`, `test-plan-generator`) need no checkout; they drive a +Firefox binary through the DevTools MCP server. + +Several agents run in **two stages** — a read-only analysis stage that reaches a verdict, +then a fix stage that only runs if the verdict warrants it. The two stages often use +different models. This keeps a "nothing to fix here" outcome cheap. + +Agents needing credentialed reads also ship a **broker sidecar** holding the API keys +([security.md](security.md)). + +## Registering an agent + +Two additions in `services/hackbot-api/`: + +1. **`app/schemas.py`** — a Pydantic input model. This _is_ the agent's public API: it is + what `GET /agents` publishes as a JSON schema, what `POST /agents/{name}/runs` + validates against, and what becomes the run's env overrides. +2. **`app/agents.py`** — one `AGENT_REGISTRY` entry: `name`, `description`, `job_name` + (the Cloud Run Job), `input_schema`, and optionally `auto_apply_actions=True`. + +Env vars are derived from the schema (`bug_id` → `BUG_ID`, lists and dicts JSON-encoded), +so there is no per-agent mapping code to write. `build_env` exists as an escape hatch for +an agent whose env genuinely doesn't map 1:1, and should stay unused. + +**Deploy-time constants are not inputs.** The broker's loopback URL, model defaults, and +similar belong in the Job's static env, not in the input schema. + +The UI's agent list (`services/hackbot-ui/lib/agents.ts`) is a separate list and needs the +new name too. + +## Conventions worth keeping + +**One folder per agent, self-contained.** Logic, `hackbot.toml`, `Dockerfile`, +`compose.yml`, prompts and rules all live under `agents//`. You should be able to +understand one agent without reading another. + +**Prompts and rules are files, not string literals.** `prompts/*.md` and `rules/*.md` are +read at startup. They are the part reviewers most often need to read. + +**Never create `hackbot_agents/__init__.py`.** `hackbot_agents` is a PEP 420 namespace +package; an `__init__.py` makes agents overwrite each other when installed side by side. + +**Reuse the shared pieces** rather than reimplementing them: `Reporter` for rendering the +streamed model messages into the run log, `actions_server_for` for recordable write +actions, [`agent-tools`](tools.md) for read tools. Assembling `ClaudeAgentOptions` and driving the +client loop stays in the agent — that is the part that should differ. diff --git a/docs/hackbot/api.md b/docs/hackbot/api.md new file mode 100644 index 0000000000..d49f8c634b --- /dev/null +++ b/docs/hackbot/api.md @@ -0,0 +1,137 @@ +# hackbot-api + +The control plane. FastAPI on Cloud Run, backed by Cloud SQL Postgres. It is the only +component that knows the agent catalog, and the only writer of run state. + +## Endpoints + +### Public — `X-API-Key` + +| Method | Path | Does | +| ------ | --------------------------------- | ------------------------------------------------------ | +| GET | `/agents` | The catalog, each with its input JSON schema | +| POST | `/agents/{agent}/runs` | Validate inputs, create a run, start an execution | +| GET | `/runs` | List runs; filter by `agent`, `status`, `requested_by` | +| GET | `/runs/{run_id}` | One run: status, inputs, summary, artifacts | +| GET | `/runs/{run_id}/artifacts/{path}` | A short-lived signed GCS download URL | +| GET | `/runs/{run_id}/actions` | Recorded actions and their apply state | +| POST | `/runs/{run_id}/actions/apply` | Apply all pending actions (idempotent) | +| GET | `/health` | Health check | + +`POST /agents/{agent}/runs` accepts an `X-On-Behalf-Of` header carrying the requesting +user's email, stored as `requested_by` — the caller is a trusted service (the UI), so this +is attribution, not authentication. + +Artifact downloads are restricted to artifacts already listed on the run, which both scopes +the download to that run's prefix and prevents probing unrelated objects. + +### Inbound webhooks — HMAC signature + +| POST | `/webhooks/phabricator` | `@hackbot` mention on a revision triggers a bug-fix run | + +Authenticated by Phabricator's own HMAC signature over the raw body, so it sits on its own +router without the API-key dependency. See [triggers.md](triggers.md). + +### Internal events — Google OIDC token + +| POST | `/internal/events/agent-run-finished` | An execution reached a terminal state | +| POST | `/internal/events/apply-run-actions` | Consumer of `run.completed` | + +Named for the domain outcome or the job they do, not the GCP mechanism that feeds them. + +## Creating a run + +``` +validate payload against the agent's input schema ── 422 on mismatch +mint a V4 signed POST policy scoped to runs// +insert Run(status=pending) +trigger a Cloud Run Job execution with env overrides on the `agent` container +store execution_name +``` + +Env overrides are the run id, the results bucket/prefix/policy, and the inputs mapped from +the schema. If the trigger fails the run is marked `failed` with the reason and the caller +gets a 502 — a run row always exists, so a failed dispatch is visible rather than lost. + +## Run states + +``` +pending ──> running ──> succeeded + └──> failed + └──> timed_out +``` + +Terminal status is decided in `finalize_run` from the **execution status** and the +**`summary.json`** together: + +| Execution | `summary.json` | Result | +| ------------- | ---------------- | ------------------------------------- | +| cancelled | any | `timed_out` | +| any | missing | `failed` | +| any | `status != "ok"` | `failed` | +| not succeeded | `status == "ok"` | `failed` (exited non-zero despite ok) | +| succeeded | `status == "ok"` | `succeeded` | + +Requiring both to agree is deliberate: an OOM kill after a clean `summary.json` write, or a +crash before writing one, both land as failures rather than false successes. + +## Completion detection + +`GET /runs/{run_id}` is a plain database read. Completion is detected **out of band**: + +``` +Cloud Run Job execution completes + → Cloud Logging sink on the `system_event` completion audit log + → Pub/Sub push + → POST /internal/events/agent-run-finished + → finalize_run +``` + +The completion log fires for success and failure alike, including OOM and crash. The route +only has to identify _which_ run finished — `finalize_run` re-queries the authoritative +execution status rather than trusting the payload. Correlation is on `execution_name`, with +a suffix match as a fallback so a v1/v2 resource-name prefix mismatch doesn't break it. + +`finalize_run` is idempotent via `finalized_at`, because push delivery is at-least-once. It +reads `summary.json`, lists the artifacts, sets the terminal state, and publishes +`run.completed`. + +## Events + +One topic per domain, `-events` — `agent-run-events` today. Events carry routing +keys as Pub/Sub **attributes** (`event_type`, `agent`, `status`) because subscription +filters can only match attributes, never the body; the JSON body carries the fuller +payload. + +Publishing is best-effort and never raises: the `Run` row is durably committed first, so a +lost publish means a delayed downstream reaction, not lost primary state. + +`run.completed` currently drives one consumer, `apply-run-actions`. Additional consumers +(notifications, outbound webhooks) get their **own route** named after their own job, and a +new event domain gets its own topic rather than overloading this one — keeping IAM, +retention and schema separable. + +## Data model + +**`runs`** — `run_id` (uuid, pk), `agent`, `status`, `inputs`, `requested_by`, +`execution_name`, `results_prefix`, `summary`, `artifacts`, `error`, `created_at`, +`updated_at`, `finalized_at`. Indexed on `agent`, `status`, `requested_by`, `created_at`; +listing orders by `created_at desc, run_id desc` so offset paging is stable when timestamps +collide. + +**`run_actions`** — one row per entry in a run's `summary.json` actions, unique on +`(run_id, idx)`: `type`, `params`, `ref`, `status` (`pending`/`applied`/`failed`), +`result`, `error`, `applied_at`. See [actions.md](actions.md). + +Schema changes go through Alembic (`services/hackbot-api/alembic/`). + +## Local development + +Commands and the full config reference are in [deployment.md](deployment.md). Two things +specific to this service: + +- **`WEBHOOK_SECRET` has no default**, so a missing one fails at startup rather than + silently accepting or rejecting deliveries. +- **Signing GCS URLs needs an impersonating credential** — + `gcloud auth application-default login --impersonate-service-account=`. See + [security.md](security.md) for why, and what the deployed service needs instead. diff --git a/docs/hackbot/architecture.md b/docs/hackbot/architecture.md new file mode 100644 index 0000000000..f95a50fab8 --- /dev/null +++ b/docs/hackbot/architecture.md @@ -0,0 +1,119 @@ +# Architecture + +## Components + +**`hackbot-api`** — the control plane. The only component that knows which agents exist, +what inputs they take, and what state their runs are in. Owns the Postgres database, +starts executions, and applies recorded actions. FastAPI on Cloud Run. + +**Agent images** (`agents//`) — one container image per agent, deployed as a Cloud +Run **Job**; a run is one execution of that Job. See [agents.md](agents.md). + +**`hackbot-runtime`** — the library inside the agent container. It owns everything that +is the same for every agent: loading config, preparing the source checkout, providing +model credentials, capturing source changes, collecting recorded actions, writing +`summary.json`, and tracing. Agent authors write logic, not plumbing. + +**`hackbot-ui`** — the human surface. Trigger runs, watch them, read findings, download +artifacts, review and apply recorded actions, retrigger failures. + +**`hackbot-pulse-listener`** — an always-on Cloud Run worker pool that watches +Taskcluster failures and dispatches build-repair / test-repair runs. It is a _client_ of +hackbot-api, not part of it. + +**`agent-tools`** — the tools an agent's model can call, declared once and adapted per +framework. Read tools (Bugzilla, Phabricator, Searchfox, Firefox build, VCS) live here; +write-actions live in the runtime. See [tools.md](tools.md). + +## The main design decisions + +### Agents are one-shot jobs, not services + +A run clones Firefox, maybe builds it, reasons for a while, and exits. That is a batch +workload with a wide latency spread (minutes to hours), so agents are Cloud Run **Jobs** +rather than request-serving services: no idle cost, per-execution isolation, generous +timeout (default 8h). A crashed or OOM-killed run is just a failed execution, and the +platform learns about it the same way it learns about a clean exit. + +The consequence to keep in mind: **nothing survives a run**. The checkout, the build, +the logs — all gone when the container exits. Anything worth keeping must be published +as an artifact. + +### `summary.json` is the whole agent→platform contract + +An agent reports by returning a result object or raising. The runtime turns that into +exactly one file: + +```json +{ + "status": "ok" | "error", + "error": null, + "findings": { ... }, // the agent's own result model + "actions": [ ... ] // what it wants the platform to do +} +``` + +plus an exit code. Everything downstream — the terminal status, the UI, the applier, the +notification emails — reads only this. The runtime writes it on **every** path, including +when the agent raises, so a failed run is still explainable. + +This is why the API can be indifferent to what an agent actually does. Adding an agent +does not touch the run lifecycle. + +### The agent container holds no credentials + +It is the least-trusted component in the system, so it gets no durable credential. Three +mechanisms, one principle — rationale and details in [security.md](security.md): + +- **Third-party API keys** (Bugzilla, Phabricator) live in a **broker sidecar**. +- **Model and tracing credentials** come from **Workload Identity Federation** — the + container exchanges its own Google identity for short-lived tokens. +- **Writing results** is a **signed GCS POST policy**, per run, scoped to that run's prefix. + +### Agents propose; the platform disposes + +An agent never posts a Bugzilla comment or creates a Phabricator revision while it runs. It +records the intent into `summary.json`; hackbot-api turns those records into real API calls +once the run reaches a verified-good terminal state. See [actions.md](actions.md) for what +that buys and how it works. + +### Configuration is split by who owns it + +| Where | What | Changes when | +| --------------------------------------------- | --------------------------------------------------------------- | ----------------------- | +| `agents//hackbot.toml` | Capabilities the agent needs prepared (`[source]`, `[firefox]`) | The agent changes | +| `AGENT_REGISTRY` + input schema (hackbot-api) | The agent's public input contract | The agent's API changes | +| Cloud Run Job env / Secret Manager | Deploy-time constants and secrets | The deployment changes | +| Per-execution env overrides | This run's inputs | Every run | + +Per-run inputs are derived from the Pydantic input schema automatically +(`bug_id` → `BUG_ID`), so registering an agent is one schema plus one registry entry — no +per-agent env-mapping code. + +### Tool declarations are framework-neutral + +Tools are declared as plain `@tool`-decorated handlers that import no agent framework; +adapters render them for a specific one (claude-agent-sdk today). One declaration therefore +backs a read tool, a recorded write-action, and an in-process or brokered MCP server alike. +See [tools.md](tools.md). + +### Local runs and deployed runs take the same path + +The runtime's only branch on environment is "is there an uploader configured?" — if not, +artifacts are written to disk under the same keys they'd have in GCS. So +`docker compose up ` exercises what production exercises, minus the upload. + +### Platform-specific glue is isolated at the edges + +Cloud Run and Eventarc appear in narrow, named places: the job trigger, the completion-log +parser, the push-auth check. The routes that consume them are named after the domain +outcome (`agent-run-finished`, `apply-run-actions`), not the mechanism, and the finalize +logic re-queries authoritative status rather than trusting the event payload. Moving or +adding an execution platform means adding a payload parser, not rewriting the lifecycle. + +## What is _not_ in this repository + +The deploy scripts for **hackbot-api**, the **agent Cloud Run Jobs**, and the **Pub/Sub +topics, subscriptions and Eventarc triggers** are managed outside this repo. Only +`services/hackbot-ui/deploy.sh` and `services/hackbot-pulse-listener/deploy.sh` live here. +See [deployment.md](deployment.md). diff --git a/docs/hackbot/deployment.md b/docs/hackbot/deployment.md new file mode 100644 index 0000000000..bbc6a5cfff --- /dev/null +++ b/docs/hackbot/deployment.md @@ -0,0 +1,138 @@ +# Deployment and configuration + +Everything runs on GCP in a Hackbot-only project. + +| Component | Runs as | Deployed by | +| ------------------------ | ----------------------------------------------------- | ------------------------------------------- | +| `hackbot-api` | Cloud Run **service** + Cloud SQL Postgres | Outside this repo | +| Agents | Cloud Run **Job**, one per agent | Outside this repo | +| Event wiring | Logging sink, Pub/Sub topic + subscriptions, Eventarc | Outside this repo | +| `hackbot-ui` | Cloud Run **service** | `services/hackbot-ui/deploy.sh` | +| `hackbot-pulse-listener` | Cloud Run **worker pool** (no HTTP) | `services/hackbot-pulse-listener/deploy.sh` | + +**Only the UI and listener have deploy scripts in this repository.** Provisioning for the +API, the agent Jobs, and the event plumbing lives elsewhere; code comments referring to a +`deploy-events.sh` refer to that external tooling. If you add a component here, add its +deploy script alongside it and list it above. + +Both scripts follow the same pattern: build, push to Artifact Registry, then deploy — +creating a dedicated least-privilege service account and granting it +`secretmanager.secretAccessor` on only the secrets it reads. Secret values are read from +env (so `source .env` works) and used only to seed a secret that does not exist yet — +existing secrets are never overwritten. Rotate with `gcloud secrets versions add`. + +## The agent Job shape + +An agent's Job manifest declares one or two containers per task, built as separate targets +of the same Dockerfile: + +- **`agent`** — `python -m hackbot_agents.`. No credentials. Receives the + per-execution env overrides. +- **`broker`** (when the agent needs credentialed reads) — + `python -m hackbot_agents..broker`. Holds the API keys, fully configured at deploy + time, and is never touched by per-execution overrides. + +The container name `agent` is what per-execution overrides target, so it must match. +`task_count=1`, timeout from `JOB_EXECUTION_TIMEOUT_SECONDS` (default 8h). `hackbot.toml` is +copied into the image's working directory, where the runtime discovers it. + +Registering the agent in `AGENT_REGISTRY` requires the Job to exist under the `job_name` +given there. + +## Environments + +`ENVIRONMENT` (`development` / `production`) selects the Sentry environment and, in the +listener, the Pulse queue name — both local and prod authenticate as the same Pulse user, so +the queue name must vary or the two consumers steal each other's messages. + +Weave tracing has its own per-environment projects (`hackbot-prod`, `hackbot-dev`, +`hackbot-test`) selected by `WEAVE_PROJECT` — see [tracing.md](tracing.md). + +## Configuration reference + +Each service parses its config once with `pydantic-settings` from env or `.env`. Nested +models bind from prefixed vars, splitting on the first underscore only +(`PHABRICATOR_API_KEY` → `phabricator.api_key`). + +### hackbot-api + +| Group | Vars | +| ----------- | ----------------------------------------------------------------------------------------- | +| GCP | `GCP_PROJECT`, `GCP_REGION`, `RESULTS_BUCKET` | +| Database | `CLOUD_SQL_INSTANCE`, `DB_USER`, `DB_PASS`, `DB_NAME` | +| Jobs | `JOB_EXECUTION_TIMEOUT_SECONDS`, `SIGNED_POLICY_MAX_BYTES`, `SIGNED_POLICY_GRACE_SECONDS` | +| Auth | `EXTERNAL_API_KEY`, `PUSH_AUTH_AUDIENCE`, `PUSH_AUTH_SERVICE_ACCOUNT` | +| Phabricator | `PHABRICATOR_URL`, `PHABRICATOR_API_KEY` | +| Webhook | `WEBHOOK_SECRET` (**required**), `WEBHOOK_BOT_PHID`, `WEBHOOK_MENTION_TOKEN` | +| Events | `RUN_EVENTS_TOPIC` | +| Misc | `HACKBOT_API_URL`, `PORT`, `ENVIRONMENT`, `SENTRY_DSN` | + +### Agent containers + +Set by the platform per execution: `RUN_ID`, `RESULTS_BUCKET`, `RESULTS_PREFIX`, +`RESULTS_POLICY_URL`, `RESULTS_POLICY_FIELDS`, plus one var per input-schema field +(`BUG_ID`, `FAILURE_TASKS`, …). + +Set at deploy time: `BROKER_URL`, `SOURCE_REPO`, the Anthropic federation ids +(`ANTHROPIC_FEDERATION_RULE_ID`, `ANTHROPIC_ORGANIZATION_ID`, +`ANTHROPIC_SERVICE_ACCOUNT_ID`, `ANTHROPIC_WORKSPACE_ID`), `WEAVE_PROJECT`. + +Local only: `ANTHROPIC_API_KEY`, `WANDB_API_KEY`, `ARTIFACTS_DIR`. + +### hackbot-ui + +`HACKBOT_API_URL`, `HACKBOT_API_KEY`, `BETTER_AUTH_URL`, `BETTER_AUTH_SECRET`, +`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`. Every instance must share +`BETTER_AUTH_SECRET` — that is the only shared session state. + +### hackbot-pulse-listener + +`PULSE_USER`, `PULSE_PASSWORD`, `HACKBOT_API_URL`, `HACKBOT_API_KEY`, `HACKBOT_UI_URL`, +`WATCHED_REPOS`, plus the filter tuning (`MAX_PUSH_AGE_HOURS`, +`TREEHERDER_CLASSIFICATION_WAIT_SECONDS`, `MAX_TEST_REPAIRS_PER_DAY`, …) and SendGrid +notification settings. `DRY_RUN=true` logs intended calls without POSTing. See its +[README](../../services/hackbot-pulse-listener/README.md). + +## Running locally + +The repo is a `uv` workspace (`agents/*`, `libs/*`, `services/*`), so everything shares one +lockfile. + +**An agent**, exactly as it ships, via its `compose.yml`: + +```bash +# .env at the repo root: ANTHROPIC_API_KEY, BUGZILLA_API_URL, BUGZILLA_API_KEY, ... +BUG_ID=1234567 docker compose up bug-fix-agent --build +``` + +Compose starts the broker alongside the agent, just like the Job does. With no upload +policy configured, `summary.json`, logs, attachments and the source patch are written to +`~/hackbot/artifacts/` on the host under the same keys they'd have in GCS. Apply the +agent's changes with `git am changes/changes.patch`. + +Add a new agent's `compose.yml` to the root `docker-compose.yml` `include:` list. + +**The services:** + +```bash +uv run --package hackbot-api uvicorn app.main:app --reload +uv run --package hackbot-pulse-listener python -m app +cd services/hackbot-ui && npm install && npm run dev +``` + +**Tests:** + +```bash +uv run pytest libs/hackbot-runtime/tests +uv run --package hackbot-api pytest services/hackbot-api/tests +uv run --package hackbot-pulse-listener pytest services/hackbot-pulse-listener/tests +``` + +## Observability + +- **Traces** — Weave, one project per environment. [tracing.md](tracing.md). +- **Errors** — Sentry, per service, tagged with environment and release. +- **Logs** — Cloud Logging. The same completion logs that drive finalization are the record + of every execution. +- **Run history** — the `runs` and `run_actions` tables are the system of record, queryable + through `GET /runs`. diff --git a/docs/hackbot/runtime.md b/docs/hackbot/runtime.md new file mode 100644 index 0000000000..83d8964c89 --- /dev/null +++ b/docs/hackbot/runtime.md @@ -0,0 +1,118 @@ +# The runtime (`hackbot-runtime`) + +The library that runs inside every agent container. It owns the parts of a run that are +identical for all agents, so an agent's code is only its own logic. + +## What it does around `main()` + +``` +run_async(main) + ├─ discover hackbot.toml (cwd, then above the entry point's module) + ├─ build HackbotContext from that config + environment + ├─ configure Anthropic + W&B credentials (Workload Identity Federation, or API keys) + ├─ start a Weave trace labelled with the agent's name + ├─ call main(ctx) ──> HackbotAgentResult, or an exception + ├─ publish logs/agent.log + ├─ publish changes/changes.patch + changes.json (+ phabricator_diff.json if needed) + ├─ publish summary.json { status, error, findings, actions } + └─ exit 0 / 1 +``` + +Publishing is best-effort per artifact and ordered so the most important thing — +`summary.json` — is written even if an earlier step fails. A run that produces no +`summary.json` is treated by the API as a failure, so this ordering matters. + +## `HackbotContext` + +The single object `main()` receives. Its platform fields come from the environment (set by +the orchestrator); its capabilities come from `hackbot.toml`. + +### Capabilities + +| Member | Gives you | +| ---------------------------------- | -------------------------------------------------------------- | +| `await prepare_repo(ref=, depth=)` | The source checkout, cloned or refreshed. Call once, up front. | +| `repo_path` | The prepared checkout (raises if `prepare_repo` hasn't run) | +| `firefox` | `FirefoxContext` — build paths derived from the checkout | +| `anthropic.api_key` | Model credentials, validated on access | + +`prepare_repo` resolves its ref from the argument, then `SOURCE_REF`, then +`[source].ref`. Preparing twice at conflicting refs raises rather than silently editing +the wrong tree — the checkout is a shared, single-use resource for the run. + +The checkout is **shallow by default** (`depth=1`, or `2` when pinned to a ref so the +commit's own diff is computable). An agent that needs real history passes `depth` +explicitly. `ensure_source_repo` is idempotent and recovers a checkout left broken by an +earlier failed run. + +`checkout_revision(ctx, revision_id, broker_url)` is the variant for follow-up runs: it +asks the broker for a Phabricator revision's base commit and raw diff, checks out the base, +and applies the diff **uncommitted** — so the run's change base stays at the revision's +base and the final submission is the complete updated revision. + +### Results and artifacts + +| Member | Does | +| ------------------------------- | -------------------------------------------------------- | +| `publish_file(key, path, type)` | Upload a file under `key` | +| `publish_json(key, payload)` | Upload JSON under `key` | +| `publish_changes()` | Collect and publish the agent's source diff | +| `log_path` | A writable path for the run log; published automatically | +| `actions` | The `ActionsRecorder` — see [actions.md](actions.md) | +| `run_artifacts_dir` | Local artifact dir, used when no uploader is configured | + +**The one publishing rule:** if `RESULTS_POLICY_URL` is set, the artifact is POSTed to GCS +under `key`; otherwise it is written to `artifacts_dir/run_id/key`. Same key either way, so +a downstream apply step resolves it identically against GCS or a local directory. This is +what makes local runs faithful. + +### Standard artifact keys + +| Key | Content | +| --------------------------------- | ---------------------------------------------------------- | +| `summary.json` | The run contract: status, error, findings, actions | +| `logs/agent.log` | The rendered agent transcript | +| `changes/changes.patch` | mbox patch, applied with `git am` | +| `changes/changes.json` | Base commit, repo URL, commits and files touched | +| `changes/phabricator_diff.json` | Prebuilt Phabricator submission payload (only when needed) | +| `attachments//` | Files attached to a recorded action | + +## Capturing source changes + +After the agent runs, its work may be committed locally, uncommitted, or untracked. +`changes.collect` captures all of it relative to the commit the checkout started from: +any uncommitted remainder is wrapped into one synthetic commit, then `git format-patch` +produces a single mbox that `git am` applies in one command, preserving each local commit's +message and author. Binary and untracked files included. + +The checkout is ephemeral, which is what makes mutating its index safe. + +When the agent recorded a Phabricator patch action, `publish_changes` _also_ builds the +Phabricator submission payload here — while the checkout still exists — using moz-phab's +own diff-building code as a library. The apply step downstream therefore never needs a +checkout of its own. It's best-effort: a failure to build it doesn't fail the run. + +## Model credentials + +Anthropic (model access) and W&B (tracing) are both configured before `main()` runs, so an +agent just reads `ctx.anthropic.api_key` and never touches the environment. Deployed, both +use Workload Identity Federation and the container holds no long-lived key; locally both +fall back to their API-key env var, with no special casing needed either side. Mechanism +and failure modes: [security.md](security.md). + +## Tools and actions + +The tools a model can call are **not** in this library — read tools live in `agent-tools` +([tools.md](tools.md)), recordable write-actions in `hackbot_runtime/actions/` +([actions.md](actions.md)). The runtime's part is `ctx.actions`, the recorder those +write-actions append to. + +## Other helpers + +- **`claude.Reporter`** — renders streamed claude-agent-sdk messages (turns, thinking, + tool calls, results, cost) to stdout and the run log. Every agent would otherwise + reimplement this. +- **`searchfox`** — expands `{{searchfox.permalink}}` placeholders in recorded comments + into revision-pinned Searchfox URLs at record time, and declines to link paths that + don't exist in the checkout. Registered as an action hook. +- **`errors.AgentError`** — the "expected, explainable failure" exception. diff --git a/docs/hackbot/security.md b/docs/hackbot/security.md new file mode 100644 index 0000000000..f05d1b8330 --- /dev/null +++ b/docs/hackbot/security.md @@ -0,0 +1,109 @@ +# Trust boundaries + +The central constraint: **an agent container is the least trusted component in the system.** +It runs model-directed code against untrusted input (bug reports, review comments, CI logs), +so it is given no durable credential and no ability to change anything directly. + +## What the agent container can and cannot do + +| Can | Cannot | +| --------------------------------------------------------- | ------------------------------------------ | +| Read Bugzilla / Phabricator via the broker's loopback URL | Hold or read a Bugzilla or Phabricator key | +| Call Anthropic with a short-lived federated token | Hold a long-lived Anthropic API key | +| Write objects under its own run's GCS prefix | Read or write another run's objects | +| Edit its own ephemeral checkout | Push anywhere, or land anything | +| _Record_ a Bugzilla comment or Phabricator revision | Actually post one | + +Each of those is a different mechanism. + +## The broker sidecar + +An agent that needs credentialed reads ships a second container from the same image. The +broker holds the third-party API keys (Secret Manager-backed, configured at deploy time) and +exposes them as capabilities over loopback (`BROKER_URL`, e.g. `http://127.0.0.1:8765`): + +- `/{bugzilla,phabricator}/mcp` — read-only MCP tool servers, live during the run. +- `GET /phabricator/revision/{id}/patch` — a revision's base commit and raw diff, so a + follow-up run can reproduce the revision's tree without a Conduit key. + +It exposes only what a run legitimately needs, and only reads — every write goes through the +recorded-actions path instead. **Per-execution env overrides target the `agent` container by +name**, which is what stops a run's inputs from reaching or altering the broker's +environment. + +Today `bug-fix`, `build-repair`, `frontend-triage` and `autowebcompat-repro` run a broker. +`test-repair` reaches an MCP server via an injected `BUGZILLA_MCP_URL` instead, and +`test-plan-generator` needs no credentialed reads at all. The invariant holds in every case: +**the key is never in the agent container.** + +## Workload Identity Federation + +Anthropic and W&B both accept a Google-signed OIDC identity token exchanged for a +short-lived access token, so neither needs a static key in the container. + +The runtime fetches the token from the GCP metadata server (`format=full`, so it carries the +`email` claim the federation rule matches on), writes it to a file in a 0700 directory with +an atomic replace, points the SDK at that file, and refreshes it every 30 minutes on a daemon +thread — tokens live ~1h and runs can be longer. + +Federation is enabled by the presence of `ANTHROPIC_FEDERATION_RULE_ID`. An +`ANTHROPIC_API_KEY` set _alongside_ it is refused with an error, because the key would take +precedence and silently shadow federation. + +## The signed upload policy + +The agent Job has **no GCP write identity**. Its only write capability is a V4 signed GCS +POST policy minted per run by hackbot-api and passed in as env: + +- `starts-with $key, runs//` — cannot write outside its own prefix. +- `content-length-range 0..5 GiB`. +- Expires at the job timeout plus a grace window. + +Downloads for humans work the same way in reverse: hackbot-api mints a short-lived signed +GET URL for one artifact, and only for an artifact already listed on the run. The bucket is +never public. + +On Cloud Run the API has no private key to sign with (the metadata server gives tokens +only), so it wraps its own credentials with `impersonated_credentials` targeting itself, +delegating `sign_bytes` to the IAM `signBlob` API. This is why its service account needs +`roles/iam.serviceAccountTokenCreator` **on itself**. + +## Authenticating callers of hackbot-api + +Three distinct schemes, one per class of caller: + +| Caller | Scheme | +| --------------------------- | -------------------------------------------------------------------------------------- | +| UI, pulse listener, scripts | `X-API-Key`, compared in constant time | +| Phabricator | HMAC-SHA256 over the raw body, constant-time compared | +| Eventarc / Pub/Sub push | Google-signed OIDC bearer token, verified for audience **and** issuing service account | + +The push-token check is not redundant with platform IAM. The service allows unauthenticated +invocations — that is how API-key callers reach it at all — so IAM on the subscription does +not protect these routes on its own. The token is verified in the route. + +## Authenticating humans (UI) + +Google OAuth, `@mozilla.com` only, enforced twice: once in the OAuth callback before a +session is issued, and again on every server-side proxy request. Sessions are stateless +signed+encrypted cookies. The middleware's cookie check is an optimistic guard, not the +authority. See [triggers.md](triggers.md). + +## Authorizing `@hackbot` mentions + +A webhook signature proves the delivery came from Phabricator; it says nothing about _who_ +commented. So the comment author must additionally be a member of the `bmo-editbugs-team` +project. [triggers.md](triggers.md) covers that check and the other guards on the path. + +## Recorded actions as a review gate + +The record-then-apply split ([actions.md](actions.md)) is a security property as much as a +correctness one: proposed effects are inspectable before they land, auto-apply is off by +default, and only a `succeeded` run's actions are ever applied. + +## Secrets + +Secrets live in Secret Manager and are mounted as env vars on the service or Job that needs +them. Each deployed service runs as its **own least-privilege service account** granted +`secretmanager.secretAccessor` on only the secrets it reads. Locally, secrets come from a +root `.env`, which is never committed. diff --git a/docs/hackbot/tools.md b/docs/hackbot/tools.md new file mode 100644 index 0000000000..54d2d52636 --- /dev/null +++ b/docs/hackbot/tools.md @@ -0,0 +1,133 @@ +# Agent tools (`agent-tools`) + +The tools an agent's model can call. A separate library from the runtime +(`libs/agent-tools/`) because nothing in it is Hackbot-specific: it declares tools and +adapts them to agent frameworks, and knows nothing about runs, artifacts or the platform. + +## Declaring a tool + +A tool is an `async` handler whose **first parameter is a context object**: + +```python +@tool +async def get_bugs(ctx: BugzillaContext, bug_ids: Annotated[list[int], Field(description=...)]) -> dict: + """Fetch one or more bugs by ID in a single bulk request.""" +``` + +The decorator derives everything the model sees from the function itself: + +| Model sees | Comes from | +| --------------- | ------------------------------------------ | +| tool name | the function name | +| namespace | the defining module's basename | +| description | the docstring | +| argument schema | the typed signature, minus the `ctx` param | + +So the docstring and the `Annotated[..., Field(description=...)]` hints **are** the prompt +for that tool. They are worth writing carefully — they are what the model reads to decide +whether and how to call it. + +Collect a module's tools with `tools_in(__name__)`, conventionally exported as `TOOLS`. + +## Framework-neutrality + +`agent_tools.registry` imports only pydantic — no agent framework. Adapters translate a +`ToolDefinition` into a specific framework's server: + +- **`agent_tools.claude_sdk.build_sdk_server(name, ctx, tools)`** — an in-process MCP + server for one domain. Tool names are bare function names. +- **`hackbot_runtime.actions.claude_sdk.actions_server_for(recorder, types)`** — the shared + `actions` server for write-actions. Namespace-prefixed (`bugzilla_update_bug`) because + one server hosts every write domain. + +`claude_sdk.py` is the only module in the library that imports `claude-agent-sdk`, behind +the `[claude-sdk]` extra. Adding LangChain support means adding one adapter, not touching +any handler — `ToolDefinition.args_model` is already a plain pydantic model usable as an +`args_schema`. + +Handlers raise **`ToolError`** for expected failures; the adapter renders it as the +framework's error signal, optionally with a structured payload the model can act on rather +than a bare message. + +## Read tools here, write-actions in the runtime + +**`agent-tools` declares read tools. `hackbot-runtime` declares write-actions.** A read +tool returns data during the run; a write-action records an intent for later +([actions.md](actions.md)). Both use the same `@tool` decorator and the same adapter — the +split is by effect, not by mechanism, and it is what keeps the "agents don't mutate the +world mid-run" invariant checkable by looking at which library a tool came from. + +## The catalog + +Every tool below is read-only. Nothing is exposed by default: an agent allowlists the +individual tools it wants in its `config.py`, by their MCP name +(`mcp____`, e.g. `mcp__bugzilla__get_bugs`). + +| Namespace | Tool | Returns | +| ------------- | ----------------------- | --------------------------------------------------------------- | +| `bugzilla` | `search_bugs` | Bugs matching raw REST query parameters | +| | `get_bugs` | One or more bugs by id, in a single bulk request | +| | `get_bug_comments` | Every comment on a bug | +| | `get_bug_attachments` | A bug's attachments | +| | `download_attachment` | One attachment decoded to a local file | +| `phabricator` | `get_revision` | A revision's title, summary, status, reviewers | +| | `get_revision_comments` | Every comment, oldest first, general and inline | +| | `get_revision_diff` | The raw unified diff (latest diff by default) | +| `searchfox` | `search_identifier` | Exact-identifier matches across the tree | +| | `search_text` | Full-text / regex matches | +| | `find_definition` | The source of a symbol's definition | +| | `get_function_at_line` | The innermost function enclosing a line | +| | `get_blame` | The changeset that last touched each line | +| | `get_file` | A file's full content, at HEAD or a revision | +| `mozilla_vcs` | `get_commit_info` | A changeset's author, date, description, parents, files | +| | `get_commit_diff` | A changeset's unified diff | +| | `file_history` | Recent changesets touching a file, newest first | +| `firefox` | `bootstrap_firefox` | `./mach bootstrap` — installs the build toolchain (slow) | +| | `build_firefox` | A build from the configured mozconfig; `target` builds one dir | +| | `evaluate_testcase` | Runs a testcase in Firefox under xvfb; crash output via grizzly | +| | `evaluate_js_shell` | Runs a JS testcase in the SpiderMonkey shell; crash output | + +## Contexts and extras + +Each namespace has a context object carrying what its tools need, and an optional +dependency extra so an agent installs only what it uses: + +| Namespace | Context | Extra | Backed by | +| ------------- | -------------------- | ----------------------- | --------------------------------- | +| `bugzilla` | `BugzillaContext` | `agent-tools[bugzilla]` | `bugsy` | +| `phabricator` | `PhabricatorContext` | — | injected `PhabricatorClient` | +| `searchfox` | `SearchfoxContext` | `[searchfox]` | `searchfox` client | +| `mozilla_vcs` | `MozillaVcsContext` | `[vcs]` | `hg.mozilla.org` over `httpx` | +| `firefox` | `FirefoxContext` | `[firefox]` | `grizzly-framework`, `prefpicker` | + +`FirefoxContext.from_source_repo(path, objdir=...)` derives every build path from the +prepared checkout, which is why `ctx.firefox` on the runtime context just works once +`[firefox]` is declared in `hackbot.toml`. + +The library's `__init__` imports no submodule, so pulling in one tool never drags in +another's optional dependencies. + +## Which namespaces need a broker + +Only `bugzilla` and `phabricator`: their servers normally run in the broker rather than +in-process, because the agent container holds no key for either. The declarations are +identical either way — only the process hosting them moves ([security.md](security.md)). +`searchfox` and `mozilla_vcs` query public services with no credentials, and `firefox` runs +locally against the checkout. + +## The firefox tools are the expensive ones + +`bootstrap_firefox` is ~10-15 min on a cold image; `build_firefox` is tens of minutes on a +full tree, much less incrementally. An agent that only needs to confirm a localized fix +compiles should pass `target` to build one directory instead of the whole tree. + +## Adding a tool + +1. Write the handler in the right domain module, `@tool`-decorated, `ctx` first, returning + plain data (a `str` is shown verbatim to the model; anything else is JSON-encoded). +2. Raise `ToolError` for expected failures. +3. Add the optional dependency to the namespace's extra if it needs one. +4. Allowlist it in the agents that should have it — no agent gains a tool implicitly. + +A new namespace is a new module plus a context dataclass; `tools_in(__name__)` and the +adapters need no changes. diff --git a/docs/hackbot/tracing.md b/docs/hackbot/tracing.md index e6f8610b02..a7ebe630a1 100644 --- a/docs/hackbot/tracing.md +++ b/docs/hackbot/tracing.md @@ -1,4 +1,4 @@ -## Tracing (Weave) +# Tracing (Weave) Dashboards: [prod](https://wandb.ai/moz-bugbug/hackbot-prod/weave/agents), [dev](https://wandb.ai/moz-bugbug/hackbot-dev), [test](https://wandb.ai/moz-bugbug/hackbot-test) @@ -30,4 +30,5 @@ environment: ``` **In deployment**, the agent container holds no long-lived key: it authenticates -via W&B [Identity Federation](https://docs.wandb.ai/platform/hosting/iam/identity_federation). +via W&B [Identity Federation](https://docs.wandb.ai/platform/hosting/iam/identity_federation), +the same mechanism the runtime uses for Anthropic — see [security.md](security.md). diff --git a/docs/hackbot/triggers.md b/docs/hackbot/triggers.md new file mode 100644 index 0000000000..ab8ebf495c --- /dev/null +++ b/docs/hackbot/triggers.md @@ -0,0 +1,111 @@ +# Triggers: how runs start + +Every path ends at the same call — `POST /agents/{agent}/runs`. Nothing bypasses the API to +start an execution directly, which is what keeps run state complete regardless of origin. + +``` +hackbot-ui ─────────────┐ +pulse-listener ─────────┼──> POST /agents/{agent}/runs ──> Cloud Run Job execution +Phabricator webhook ────┘ +``` + +## hackbot-ui — humans + +A Next.js app on Cloud Run. It can trigger any agent in its list, filter and page through +recent runs, poll a run to terminal state, render findings, download artifacts via +short-lived signed URLs, review recorded actions and apply them, and retrigger a failed run +with the same inputs. + +**The API key never reaches the browser.** Every call goes through a server-side route +handler under `app/api/*` that re-validates the session and injects `X-API-Key` +(`lib/hackbot.ts`). Retrigger reads the original inputs server-side, so the browser only +sends a run id. + +**Auth is Google OAuth via better-auth, `@mozilla.com` only, and completely stateless** — +no database. The session lives in a signed + encrypted (JWE) cookie; the only shared state +is `BETTER_AUTH_SECRET`. That makes it Cloud Run-friendly out of the box: sessions survive +scale-to-zero and work across any number of instances, as long as every instance shares the +secret. + +The domain restriction is enforced in two independent layers: the Google provider's +`mapProfileToUser` rejects a non-Mozilla identity during the OAuth callback, before a +session is issued, and `getAuthedEmail()` re-checks the domain on every proxy request. +`middleware.ts` is only an optimistic cookie-presence guard — it redirects to `/login`, or +returns 401 JSON for `/api/*`, but is not the authority. + +Runs are attributed to the signed-in user via `X-On-Behalf-Of`, including retriggers +(attributed to whoever clicked, not the original requester). + +Adding an agent to the UI means adding it to `lib/agents.ts`, the shared list behind both +the trigger form and the run filter. + +## hackbot-pulse-listener — Taskcluster CI + +An always-on Cloud Run **worker pool** (no HTTP port) that consumes `task-failed` messages +from `pulse.mozilla.org`, decides which failures are worth an agent, and dispatches +`build-repair` (failed build tasks) or `test-repair` (failed test tasks). When the run +finishes it polls the result and emails a report. + +**It holds no investigation logic.** Each agent resolves the push, the commit range and the +failing tests itself from the task id. The listener only decides _what to hand off_ — which +keeps the expensive reasoning in one place and lets the filter stay cheap. + +The filtering is the substance of this service, and +[`services/hackbot-pulse-listener/README.md`](../../services/hackbot-pulse-listener/README.md) +documents it properly. The shape: + +1. **Route** by watched project and task kind. +2. **Discard** what isn't this push's failure: action-task-scheduled tasks (backfills, + retriggers) and pushes older than `MAX_PUSH_AGE_HOURS`. +3. **Dedupe** per push per agent, in memory. +4. **Judge** whether the failure is new. Test failures go through a Treeherder + classification gate first (the cheap filter — most stop here), then an ancestor walk + comparing failing manifests within the same configuration; build failures compare task + labels. +5. **Budget** — at most `MAX_TEST_REPAIRS_PER_DAY` test-repair runs per rolling 24h, since + each one clones and builds Firefox. +6. **Dispatch and report** — trigger, poll to terminal, email. + +Three properties worth carrying in your head when changing it: + +- **Every check fails open.** An upstream error runs the agent rather than dropping a + possible regression. +- **Dedupe keys are claimed only when a run is actually triggered**, so a task rejected as + intermittent or inherited leaves the push open for the next one. +- **All state is in-memory** — dedupe caches, the daily budget, pending-run tracking. A + restart resets them. + +## Phabricator webhook — `@hackbot` mentions + +An `@hackbot` mention in a comment on a Differential revision triggers a `bug-fix` follow-up +run against that revision. + +The delivery is authenticated by **Phabricator's HMAC-SHA256 signature** over the raw body, +not the API key. The payload carries only PHIDs, so the receiver calls Conduit to fetch the +triggering transactions, find the mention, and resolve the revision to a revision id plus +Bugzilla bug id. A revision with no bug id is skipped — `bug-fix` needs one. + +Guards, each closing a specific failure mode: + +- **Loop prevention** — comments authored by the bot's own PHID are ignored. +- **Authorization** — the comment author must belong to the `bmo-editbugs-team` Phabricator + project. Membership is cached with a short TTL; an unknown author triggers one refresh so + new members take effect promptly, then a cooldown so unauthorized deliveries don't cause a + Conduit call each. +- **Dedupe** — retried deliveries are deduped by triggering transaction PHID, and a + transaction is marked seen **only after a successful trigger**. A transient Conduit + failure therefore 500s and gets reprocessed on retry rather than dropped as a duplicate. +- **Fresh transactions only** — a payload mixing new and already-seen PHIDs can't + re-trigger on an old one. + +One review can leave several inline comments, each its own transaction; all qualifying ones +are combined and passed to the agent as XML-tagged `` elements carrying the comment +id, type and diff id. The comment text is **passed through as data** — the agent's prompts +frame identity and scope, not the receiver. + +The receiver triggers over the **public API** (a loopback call while co-located) rather than +calling the database and job internals directly, so splitting it into its own service later +is a matter of repointing a URL. + +The follow-up run then uses `checkout_revision` to prepare its tree at the revision's base +commit with the revision's diff applied — see [runtime.md](runtime.md).