Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions examples/agent_demo/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,16 @@ RUN pip install --no-cache-dir ".[mcp]" "fastapi>=0.115" "uvicorn>=0.30"
COPY examples/agent_demo ./examples/agent_demo
# The scenario the image serves (scenarios/<name>/), and its data,
# prepared at build time. The answers stay in the image for scoring; agents
# only ever get the workspace copy. A scenario with a "clean" step gives
# Claude Code a sandboxed shell, which on Linux needs bubblewrap and socat
# (not installed here): the hosted demo serves readmission.
ARG DEMO_SCENARIO=readmission
# only ever get the workspace copy. The insurance scenario has the agent
# clean raw exports with a script, so each agent gets a shell; Claude Code's
# sandbox (bubblewrap) can't run in a container, so, like Codex, it runs
# in the per-run user below (CLAUDE_SANDBOX=off).
ARG DEMO_SCENARIO=insurance
ENV DEMO_SCENARIO=$DEMO_SCENARIO
RUN python examples/agent_demo/prepare_data.py \
# The synthetic data's generator is public, so its seed is drawn here and
# kept nowhere (not in the image, its layers or its history): with a known
# seed an agent with a shell could rebuild the outcomes.
RUN DEMO_SEED=$(od -An -N8 -tu8 /dev/urandom | tr -d ' ') python examples/agent_demo/prepare_data.py \
&& rm -rf examples/agent_demo/runs \
&& chmod -R go-rwx examples/agent_demo/answers

Expand All @@ -57,6 +61,7 @@ ENV DEMO_HOSTED=1 \
DEMO_RUNS=/tmp/runs \
DEMO_RUN_UIDS=20000-20063 \
CODEX_SANDBOX=danger-full-access \
CLAUDE_SANDBOX=off \
TABCTX_BASE_URL=https://proxy.vectorinstitute.ai \
TABCTX_MCP_BIN=/usr/local/bin/tabctx-mcp \
DISABLE_AUTOUPDATER=1 \
Expand Down
62 changes: 43 additions & 19 deletions examples/agent_demo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

Two scenarios, each a question about a table, in [`scenarios/`](scenarios/):

- **readmission** (the default, and the hosted demo): which hospital
- **readmission** (the default): which hospital
patients will be back within 30 days? Public data, below.
- **insurance**: which policyholders will cancel at renewal, and what
- **insurance** (the hosted demo): which policyholders will cancel at renewal, and what
might keep them? The agent first cleans two messy raw exports with a
script it writes, then does the same steps. Synthetic data, see
[Insurance scenario](#insurance-scenario).
Expand Down Expand Up @@ -77,6 +77,11 @@ steps gives each agent its shell: Claude Code's Bash runs in its
sandbox, Codex in its own, and Gemini CLI and pi without one (see the
tools by agent, below).

The data comes from a fixed seed, so runs are repeatable. The generator
is public, so with that seed an agent with a shell could rebuild the
withheld outcomes: the hosted image draws a seed at build time
(`DEMO_SEED`) and keeps it nowhere.

```bash
export DEMO_SCENARIO=insurance
uv run --extra mcp python examples/agent_demo/prepare_data.py
Expand All @@ -97,20 +102,33 @@ uv run --extra mcp --with fastapi --with uvicorn python examples/agent_demo/ui/a
# open http://127.0.0.1:8765
```

Pick an agent, edit the task if you like, and click **Run agent**. Each
tabctx call appears as a card as soon as the agent makes it:

- **Inspect:** a map of the table's columns.
- **Evaluate:** both models against gradient boosting on held-out rows.
- **Predict:** the predicted split, and a risk table.
- **Explain:** which columns matter.
- **What-if:** the probability with and without a change, for one patient.

When the agent finishes, the page shows its score against the real
outcomes and its report. **Replay last run** plays a finished run back
without running anything. Hover over any mark for its numbers. The page
has light and dark themes, and listens on 127.0.0.1 only, since it
starts agents on this machine.
The page is built for whoever asked the question, not for whoever runs
the agent. It shows the question in plain English with its two files
(`LABELS["question"]` in the scenario's `task.py`). **Ask** starts the
agent; **watch a recorded run** plays a finished one back without
running anything. The answer comes back as a short brief: one section
per step of the task, each a plain question, a one-sentence answer
written from the run's own numbers, and one visual, in the scenario's
own column names (`LABELS["column_names"]`):

- **Is the data usable?** (a task with a "clean" step): five of the
messiest raw rows, underlined like misspellings, then shown as the
agent cleaned them, and the spellings merged into one value. The page
sets the raw file beside the clean one (`ui/cleaning.py`, using the
task's `RAW_COLUMNS` and `CLEAN_HISTORY`).
- **Can it be predicted?** The tabular model against gradient boosting,
as "N times in 100".
- **Who is most at risk?** The expected count, and the top ten.
- **What drives it?** What the predictions lean on.
- **What could change it?** Each what-if as a move from today's risk.
- **The memo,** its summary up front, and **was it right?**: its top ten
against the real outcomes.

**Show the work** (top right) opens every command and tabctx call beside
the answer, with the agent's full instructions (editable locally). The
agent and, when hosted, the gateway key are under the agent button.
Hover over any mark for its numbers. The page has light and dark themes,
and listens on 127.0.0.1 only, since it starts agents on this machine.

The UI reads the same `runs/<scenario>/<agent>/transcript.jsonl` files that
`run.sh` writes. `ui/transcripts.py` turns each agent's format into one
Expand All @@ -124,8 +142,11 @@ The image holds the page, Claude Code, Codex and pi, and `tabctx-mcp`, and
nothing secret.

- **Anyone with the link** can replay the showcase runs in
[`showcase/readmission/`](showcase/readmission/). These are three real runs, copied by
`make_showcase.py` with session details and local paths removed.
[`showcase/insurance/`](showcase/insurance/) (Claude Code's run). Showcase runs are
real runs, copied by
`make_showcase.py` with session details and local paths removed. For a
task with a cleaning step it also saves the page's view of what was
cleaned (`cleaning.json`), since the raw and clean files aren't copied.
- **To run an agent**, a visitor signs in with their own gateway key. The
app checks the key's project models (`/admin/me/models`) and gives each
agent the best model that project can use. Agents the project can't run
Expand Down Expand Up @@ -164,7 +185,10 @@ When the task has a "clean" step, the agent must run code:

- **Claude Code** also gets Bash, only inside its sandbox
(`allowUnsandboxedCommands: false`). It can't write outside its run
folder or read `answers/`.
folder or read `answers/`. The hosted demo can't run that sandbox
(bubblewrap needs privileges a container lacks), so there
`CLAUDE_SANDBOX=off` gives it Bash as the run's own Unix user, which
can't read `answers/` or other runs, as Codex does.
- **Gemini CLI** and **pi** get their shells with no sandbox, as you, so
they could read `answers/`. Treat their scores on such a task as
unchecked, or run them as another user.
Expand Down
9 changes: 8 additions & 1 deletion examples/agent_demo/make_showcase.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
"""Copy finished runs into showcase/, the runs anyone can replay in the
hosted demo: the transcript (without the session details an agent CLI
logs at start, with local paths made relative and the login name masked), the report, and the
score against the real outcomes (so the predictions file isn't needed).
score against the real outcomes (so the predictions file isn't needed),
and, for a task with a cleaning step, the page's view of what it cleaned.

uv run --extra mcp python examples/agent_demo/make_showcase.py claude codex pi

Expand All @@ -20,6 +21,9 @@
import scenario as scenarios
from score import score

sys.path.insert(0, str(Path(__file__).resolve().parent / "ui"))
from cleaning import SAVED, for_run # noqa: E402

HERE = Path(__file__).resolve().parent
# Events that record the machine, not the run (the session's tools, cwd,
# ids and account details), and pi's token-by-token streaming, which the
Expand Down Expand Up @@ -67,6 +71,9 @@ def main(agents: list[str]) -> None:
(out / "score.json").write_text(
json.dumps(score(run_dir, scenario), indent=2) + "\n"
)
view = for_run(run_dir, scenario)
if view: # the page's messy -> tidy view, without the raw files
(out / SAVED).write_text(json.dumps(view, indent=1) + "\n")
print(agent, len(kept), "events ->", out)


Expand Down
9 changes: 8 additions & 1 deletion examples/agent_demo/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
# The hosted demo (ui/, Dockerfile) sets: TABCTX_MCP_BIN (the tabctx-mcp
# command), RUN_DIR (this run's folder),
# ISOLATE_HOME=1 (the agent gets HOME inside RUN_DIR, so no state is shared
# between visitors) and CODEX_SANDBOX.
# between visitors), CODEX_SANDBOX and CLAUDE_SANDBOX=off.
set -euo pipefail

HERE=$(cd "$(dirname "$0")" && pwd)
Expand Down Expand Up @@ -86,6 +86,13 @@ claude)
# (which would also allow an unsandboxed retry).
tools+=,Bash
settings=(--settings "$(printf '{"sandbox": {"enabled": true, "autoAllowBashIfSandboxed": true, "allowUnsandboxedCommands": false}, "permissions": {"deny": ["Read(/%s/**)"]}}' "$SCENARIO_ANSWERS")")
if [[ ${CLAUDE_SANDBOX:-on} == off ]]; then
# The hosted demo: its sandbox (bubblewrap) can't run in a container,
# so Bash runs as this run's own Unix user instead, which can't read
# the answers or other runs (as Codex does there).
allowed+=,Bash
settings=(--settings "$(printf '{"permissions": {"deny": ["Read(/%s/**)"]}}' "$SCENARIO_ANSWERS")")
fi
fi
# ${a[@]+...}: an empty array is "unbound" to macOS's bash 3.2 under set -u.
claude -p "$PROMPT" ${model[@]+"${model[@]}"} ${settings[@]+"${settings[@]}"} --mcp-config "$config" --strict-mcp-config \
Expand Down
10 changes: 8 additions & 2 deletions examples/agent_demo/scenarios/insurance/prepare_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@

from __future__ import annotations

import os
from typing import TYPE_CHECKING

import numpy as np
Expand Down Expand Up @@ -317,8 +318,13 @@ def build(


def main(scenario: Scenario, **sizes: int) -> None:
"""Write the scenario's data; sizes are build()'s row counts."""
history, current, answers = build(**sizes)
"""Write the scenario's data; sizes are build()'s row counts.

DEMO_SEED replaces the fixed seed. The hosted image draws one at build
time and keeps it nowhere: this generator is public, so with the fixed
seed an agent with a shell could rebuild the withheld outcomes."""
seed = int(os.environ.get("DEMO_SEED") or SEED)
history, current, answers = build(seed, **sizes)
scenario.workspace.mkdir(parents=True, exist_ok=True)
scenario.answers.mkdir(parents=True, exist_ok=True)
history.to_csv(scenario.workspace / task.HISTORY, index=False)
Expand Down
41 changes: 41 additions & 0 deletions examples/agent_demo/scenarios/insurance/task.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,18 @@
POSITIVE: ("Cancelled", "LAPSED", "cxl"),
NEGATIVE: ("Renewed", "RENEWED", "Active"),
}
# The page's "messy -> tidy" view: which raw column became which clean one,
# and the clean history the agent writes (PROMPT.md names it).
RAW_COLUMNS = {
"Prov": "province",
"Product Type": "product_type",
"Annual Premium": "annual_premium",
"Pay Method": "pay_method",
"Advisor?": "has_advisor",
"Age": "age",
"Status at Renewal": "cancelled",
}
CLEAN_HISTORY = "renewals_2025.csv"
POLICY_PREFIX = "MM-" # real policies; the export also has test rows and a total


Expand All @@ -37,6 +49,13 @@ def positive_rate(history) -> float:
# For the page: what a row is, what the positive outcome is called, and
# PROMPT.md's steps as icons (a tool's icon, "clean", or "report").
LABELS = {
"org": "Maple Mutual",
"question": (
"Clean up these two exports, find who is likely to cancel at renewal "
"and why, and test what could keep them. Write it up for the head of "
"retention."
),
"audience": "the head of retention",
"entity": "policyholders",
"outcome": "cancelled",
"window": "at renewal",
Expand All @@ -50,4 +69,26 @@ def positive_rate(history) -> float:
["what_if", "What-if", "Test retention offers on the likeliest to cancel"],
["report", "Report", "Write report.md"],
],
# Column names as a retention team says them, and how a value reads
# ("{}" is the value).
"column_names": {
"policy_id": ["Policy"],
"years_as_customer": ["Years as a customer"],
"age": ["Age"],
"province": ["Province"],
"product_type": ["Product"],
"annual_premium": ["Annual premium", "${}"],
"premium_change_pct": ["Premium increase", "+{}%"],
"pay_method": ["Pays by"],
"pay_freq": ["Payment frequency"],
"missed_payments_12m": ["Missed payments"],
"has_advisor": ["Has an advisor"],
"online_account": ["Online account"],
"smoker": ["Smoker"],
"advisor_contacts_12m": ["Advisor contacts"],
"other_products": ["Other products held"],
"claims_12m": ["Recent claims"],
"dependents": ["Dependents"],
"cancelled": ["Cancelled?"],
},
}
29 changes: 29 additions & 0 deletions examples/agent_demo/scenarios/readmission/task.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,13 @@
# For the page: what a row is, what the positive outcome is called, and
# PROMPT.md's steps as icons (a tool's icon, or "report").
LABELS = {
"org": "Hospital readmissions",
"question": (
"Which of the patients we're discharging now are most likely to be "
"back within 30 days, why, and what could lower that? Write it up for "
"the care team."
),
"audience": "the care team",
"entity": "patients",
"outcome": "readmitted",
"window": "within 30 days",
Expand All @@ -29,4 +36,26 @@
["what_if", "What-if", "Test care changes on the highest-risk patients"],
["report", "Report", "Write report.md"],
],
# Column names as a care team says them.
"column_names": {
"patient_id": ["Patient"],
"time_in_hospital": ["Days in hospital"],
"num_lab_procedures": ["Lab tests"],
"num_procedures": ["Procedures"],
"num_medications": ["Medications"],
"number_outpatient": ["Outpatient visits (past year)"],
"number_emergency": ["Emergency visits (past year)"],
"number_inpatient": ["Hospital stays (past year)"],
"number_diagnoses": ["Diagnoses"],
"discharge_disposition": ["Discharged to"],
"admission_type": ["Admission type"],
"admission_source": ["Admitted from"],
"medical_specialty": ["Specialty"],
"diag_1": ["Main diagnosis"],
"A1Cresult": ["A1C result"],
"max_glu_serum": ["Glucose test"],
"diabetesMed": ["On diabetes medication"],
"change": ["Medication changed"],
"readmitted_30d": ["Back within 30 days?"],
},
}
Loading
Loading