Skip to content

Create Hackbot docs #6589

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
evgenyrp wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Create Hackbot docs #6589

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
- [Hackbot platform](hackbot/README.md) — agent runtime, services, deployment

Detailed documentation per model

- [Regressor model for predicting risky commits](models/regressor.md)
83 changes: 83 additions & 0 deletions docs/hackbot/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Hackbot

Hackbot is a platform for running **autonomous agents against Mozilla's engineering
systems**: Firefox source, Bugzilla, Phabricator, Taskcluster CI, TestRail.

An agent is a container that gets started with a small set of inputs, investigates
something, and exits. It reports what it found, and — crucially — it does not change
the outside world while it runs. It _records_ what it wants to change; the platform
applies that afterwards. Everything else in the platform exists to make that shape
work: start the container, give it what it needs, collect what it produced, apply
what it proposed.

## The life of a run

```
trigger control plane execution
─────── ───────────── ─────────

hackbot-ui ─┐
pulse- ├──> hackbot-api ──> Cloud Run Job ──> ┌──────────┬─────────────┐
listener ─┤ (validate, (one execution │ agent │ broker │
Phabricator ─┘ record Run, per run) │ (no │ (holds keys,│
webhook mint upload │ creds) │ if needed) │
policy) └────┬─────┴─────────────┘
▲ │
│ artifacts +
run finished summary.json
(Pub/Sub) │
│ ▼
└────────────────────── GCS results bucket
│
├─> finalize: read summary, list artifacts,
│ set terminal status
└─> apply recorded actions (Bugzilla comment,
Phabricator revision, Slack, ...)
```

1. **Trigger.** Someone or something calls `POST /agents/{name}/runs` with inputs.
2. **Dispatch.** hackbot-api validates the inputs, records a `Run`, mints a signed
upload policy scoped to that run, and starts one Cloud Run Job execution.
3. **Run.** The runtime inside the container prepares what the agent declared it needs
(source checkout, Firefox build paths, model credentials), calls the agent's
`main()`, and writes `summary.json` plus artifacts to the results bucket.
4. **Finalize.** A completion event brings the run to a terminal state and publishes
`run.completed`.
5. **Apply.** The actions the agent recorded become real API calls — automatically for
opted-in agents, on demand from the UI otherwise.

## Where to read next

| If you want to… | Read |
| ---------------------------------------------------------- | ---------------------------------- |
| Understand the components and why they're split that way | [architecture.md](architecture.md) |
| Write or modify an agent | [agents.md](agents.md) |
| Know what the runtime hands your agent | [runtime.md](runtime.md) |
| Find a tool your agent can call, or add one | [tools.md](tools.md) |
| Understand how agents change the world (record-then-apply) | [actions.md](actions.md) |
| Work on the control-plane service | [api.md](api.md) |
| Know how runs get started | [triggers.md](triggers.md) |
| Reason about credentials and trust boundaries | [security.md](security.md) |
| Deploy, configure, or run things locally | [deployment.md](deployment.md) |
| Look at traces of a run | [tracing.md](tracing.md) |

## Code map

| Path | What it is |
| --------------------------------------------------- | ------------------------------------------------------------------- |
| `libs/hackbot-runtime/` | The in-container runtime: agent contract, context, results, actions |
| `libs/agent-tools/` | The tools a model can call: declarations + per-framework adapters |
| `libs/phabricator-client/`, `libs/testrail-client/` | Shared API clients |
| `agents/<name>/` | One self-contained agent: logic, image, local compose |
| `services/hackbot-api/` | Control plane (FastAPI): runs, artifacts, actions, webhooks |
| `services/hackbot-ui/` | Web UI (Next.js): trigger, observe, review and apply actions |
| `services/hackbot-pulse-listener/` | Watches Taskcluster CI failures and dispatches repair runs |

## Conventions in these docs

These docs cover **design and integration** — the contracts between the parts and the
decisions worth knowing before changing something — and stop short of restating code. Each
fact lives in one file; the others link to it.

Every PR that changes the Hackbot runtime, deployment, or agents should leave these docs
true.
155 changes: 155 additions & 0 deletions docs/hackbot/actions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
# Actions: record now, apply later

An agent never mutates Bugzilla, Phabricator, TestRail or Slack while it runs. It calls a
tool that **records what it intends to do**; hackbot-api performs it after the run has
finished and is known good.

Why the indirection:

- A run that fails halfway leaves **no half-applied side effects**.
- Every intent is **reviewable** — visible in `summary.json` and in the UI, with the
agent's own stated reasoning, before anything lands.
- Applying is **idempotent and retryable**, which matters because the event delivery that
drives it is at-least-once.
- Auto-apply is **per-agent opt-in**, so a new or unproven agent can run in
propose-only mode with no code change.

## Recording (inside the run)

The write-action tools are declared in `hackbot_runtime/actions/` — one module per domain,
using the same `@tool` decorator as [read tools](tools.md). The tool context is the
`ActionsRecorder`, and each handler does one thing: `recorder.record(...)` and return a
confirmation string.

```python
recorder.record(
"phabricator.submit_patch", # <domain>.<verb>
{"bug_id": ..., "title": ...}, # params the apply step will need
reasoning="why the agent is doing this", # audit trail
attachments={"log": Path(...)}, # optional files, published as artifacts
ref="patch", # optional label, see cross-references below
)
```

The recorded list becomes `summary.json`'s `actions` array. Attachments are published
under the stable key `attachments/<action_index>/<name>` and referenced by that key — the
local path disappears with the container.

### The catalog

Nine action types, each with a declaration the agent calls and a handler that applies it.
`actions/handlers/registry.py` is the authoritative type → handler map.

As with read tools, nothing is exposed by default: an agent lists the dotted types it may
record in its `config.py` and passes them to `actions_server_for`, which builds a server
carrying only those. `bug-fix`, for instance, allows `phabricator.submit_patch` on a fresh
triage run but swaps it for `phabricator.update_patch` on a follow-up.

| Action type | Records the intent to… | Params |
| --------------------------- | ------------------------------------------ | ---------------------------------- |
| `bugzilla.update_bug` | Change a bug's fields | `bug_id`, `changes` |
| `bugzilla.add_comment` | Comment on a bug | `bug_id`, `text`, `is_private` |
| `bugzilla.add_attachment` | Attach a file to a bug | `bug_id`, + a `file` attachment |
| `bugzilla.create_bug` | File a new bug | the new bug's fields |
| `phabricator.submit_patch` | Deliver a fix as a **new** revision | `bug_id`, `title`, `summary` |
| `phabricator.update_patch` | Add a new diff to an **existing** revision | `revision_id` |
| `phabricator.add_comment` | Reply on a revision without changing code | `revision_id`, `text` |
| `testrail.submit_test_plan` | Submit a generated test plan to TestRail | the validated feature + test cases |
| `slack.post_message` | Post a message to Slack | `channel`, `text` |

All but `testrail.submit_test_plan` take a **`reasoning`** argument — a free-text audit trail
stored on the action and shown in the UI beside the proposed change. `phabricator.submit_patch`
is the only model-facing tool that exposes **`ref`** (see cross-references below).

`testrail` and `slack` also provide `record_test_plan` / `record_message` helpers that agent
code calls directly rather than the model choosing to — for an action the agent always takes
once it has a result, not one the model decides on.

`bugzilla.add_comment` appends a feedback-reaction footer to every recorded comment, and
`is_private=true` marks it security-group-only.

Adding a type is a declaration in the domain module plus one line in the handler registry.
The dispatch loop never changes.

### The two patch actions

`submit_patch` and `update_patch` are deliberately separate, each taking only the
parameters its own case needs, so a model cannot create a revision when it meant to update
one by getting an optional argument wrong. That is the general pattern for write-actions:
prefer several narrow tools over one with mode flags.

Neither takes a patch file. The agent's final working-tree state _is_ the diff — which is
why the runtime builds the Phabricator payload during `publish_changes`, while the checkout
still exists.

### Hooks

`ActionsRecorder` supports per-type hooks that run before an action is appended. A hook may
mutate the action (enrichment) or raise (validation gate — nothing is recorded and no
attachment is published). They live on the recorder rather than in the tool declarations so
the runtime can attach cross-cutting behaviour without every handler knowing about it.

The Searchfox permalink expansion on `bugzilla.add_comment` is the working example: the
agent writes `{{searchfox.permalink}}/path/to/file.js#412`, the hook expands it at record
time, and the comment awaiting review already shows clickable URLs.

## Applying (after the run)

Triggered by the `run.completed` event, on a subscription filtered to **succeeded** runs.

1. **Record rows.** Every entry in `summary.json`'s `actions` is upserted as a
`run_actions` row (`pending`), keyed `(run_id, idx)`. This happens for _all_ succeeded
runs, whether or not the agent auto-applies, so the UI can always show and apply them.
2. **Apply, if opted in.** With `auto_apply_actions=True` on the agent's registry entry,
pending rows are applied immediately. Otherwise they wait for a human to click apply.
3. **Dispatch.** Each row's `type` selects a handler from the registry. The handler gets
the params and an `ApplyContext` — which can `download_artifact(key)` without knowing
GCS is behind it, keeping the runtime library free of a storage dependency.
4. **Stamp.** The row records `applied` or `failed`, its result, and its error. Only a real
success sets `applied_at`.

Rows are committed one action at a time, and an `applied` row is never reapplied — so a
retried event or a repeated manual apply-all is safe and resumes where it stopped.

Actions from a run that is not `succeeded` are never applied. The subscription filters
them out and the applier checks again, because acting on a run that never reached a
verified-good state is not wanted even if it recorded something before erroring.

## Cross-action references

An action's result often isn't known until it's applied — a Phabricator revision has no URL
until it exists. So a later action can reference an earlier one's result by label:

```
submit_patch(..., ref="patch")
add_comment(text="Patch up for review: {{actions.patch.url}}")
```

`{{actions.<ref>.<field>}}` is substituted at apply time, recursively through params.
Resolution draws on rows already `applied` in earlier passes as well as this one, so a
later manual apply can still reference an earlier action's result.

An unresolvable placeholder is **left as-is and logged**, rather than raising. The action
then fails with an error a human can read, instead of silently posting mangled text.

## Bugzilla coalescing

Same-bug field changes are merged with the closest comment into a single
`PUT /bug/{id}`, so Bugzilla applies them as one transaction — one bugmail, one history
entry, instead of a burst. Other comments on that bug still apply separately. A group is
applied at its last member's index, once every earlier dependency has resolved, and any
group whose rows carry a `ref` is excluded (nothing should reference a coalesced member's
result).

## Where to look

| Concern | File |
| -------------------------------- | ---------------------------------------------------------- |
| Recording mechanics, hooks | `libs/hackbot-runtime/hackbot_runtime/actions/recorder.py` |
| Action declarations (per domain) | `libs/hackbot-runtime/hackbot_runtime/actions/*.py` |
| Apply-side handlers | `libs/hackbot-runtime/hackbot_runtime/actions/handlers/` |
| Type → handler map | `.../actions/handlers/registry.py` |
| Orchestration, refs, coalescing | `services/hackbot-api/app/actions_applier.py` |

Record side and apply side deliberately live in the **same library**, so the set of
actions an agent can request and the set the platform can apply cannot drift apart.
Loading