feat(executions): GET /api/executions/timeline — bucketed fleet rollups (ent#326)

[dolho]

Implements Abilityai/trinity-enterprise#326 — the backend half of ent#94's grid-widget foundation.

Verified absent from dev (3f1d4c89) before starting: no executions/timeline route, nothing in db/schedules/stats.py, no open PR.

What it unblocks

Three tile sub-issues (#96 executions-by-trigger, #98 fleet cost, #101 fleet context) were blocked on an endpoint ent#94 listed in Scope but never filed. They now share one query instead of each growing their own — which was the point of filing it.

Independent of the frontend chassis (#325), so it lands without touching FleetGrid.vue / gridLayout.js / stores/fleetGrid.js — the files ent#305's open PR #1918 is currently rewriting.

The design call worth reviewing

An analytics axis cannot degrade the way a filter can.

/api/executions/stats and the list route coerce an unknown hours to 24. That is correct for a filter — worst case you get more rows than you asked for. It is wrong for a chart: it silently redraws a window the caller never requested and gives them no way to tell. So group_by and hours both 422 by name here, deliberately diverging from the siblings.

hours=0 is refused for hour/day only. An all-time axis emits one bucket per interval since the fleet's first execution — unbounded, and nobody asked for it. It stays allowed for trigger/agent, which have no continuum and are bounded by the number of distinct values. Refusing it there would be arbitrary.

The token question — settling ent#326's open AC as option 1

schedule_executions has no usage-token column. It carries cost, context_used, context_max; output_tokens exists only on chat_messages, which covers chat turns rather than fleet executions.

So the endpoint reports context-window occupancy, under the name context_used, and #101's tile must be labelled to match. Presenting this as "tokens consumed" is precisely the liveness-vs-quality mislabel the issue names. Option 2 (add token columns) is a schema change → two migrations plus the write path, unestimated; option 3 drops a tile that context occupancy can serve honestly.

A guard fails if a token column is ever added, so that decision gets revisited deliberately rather than silently inherited.

Worth flagging: my first version of that guard matched the substring token and tripped on claim_token — the #1081 pull-lease CAS value, which answers a completely different question. It now matches by explicit name. A guard that fires on the wrong column is worse than none, because you fix the wrong thing.

Other decisions

• Trigger folding in Python, not SQL. _TRIGGER_BUCKETS has an explicit Other catch-all — the property that makes a newly-added trigger visible instead of vanishing. A SQL CASE would drop that guarantee the first time someone adds a trigger and forgets to update it.
• substr on the stored ISO-Z started_at, not a date function: dialect-agnostic across SQLite and PostgreSQL, and the same UTC the row was written with (Invariant #16).
• Gap-fill on a continuous UTC axis (feat(ui): Agent Detail Overview dashboard as default tab + Info tab redesign #1107) — a missing bucket and a zero bucket mean different things to a reader, and a sparse series renders them identically.
• Read-only: no schema change, no migration, no MCP tool.

Verification

tests/unit/test_ent326_executions_timeline.py — 30 checks, all passing; 52 green across adjacent suites (test_models_centralized for Invariant #14, test_schedule_analytics, test_1115_schedules_summary).

The access test worth pointing at is the dangerous direction: a user with no accessible agents must get an empty series, never the whole fleet — pinned directly against the db layer's short-circuit, not just via the router.

The DB layer is driven against a real SQLite table: agent scoping, error counted as failed (so the chart cannot contradict the stat card directly above it), NULL cost/context coalesced rather than poisoning the SUM, window respected in both directions, and an unvalidated group_by raising rather than reaching SQL (defence in depth behind the router's 422, since the key is interpolated into GROUP BY).

Two failures during development were both mine, and both recorded in the file: a 40-day seed row that was outside the 720h window too (so the test proved nothing), and the claim_token substring above.

Acceptance criteria

• Gap-filled UTC buckets for hour|day|trigger|agent
• Each bucket carries count, failure count, cost, context totals
• Access-scoped identically to /api/executions, verified by test
• Unknown group_by / out-of-range window → named 422, never 500 or a silently-empty series
• Route ordering verified — /timeline never captured as an execution id
• Token question resolved (option 1) and the reasoning recorded for bug: Scheduler loses HTTP connection to backend during long-running executions, reporting false failures #101's label
• Documented in architecture.md under Executions

Related to Abilityai/trinity-enterprise#326 · unblocks #96, #98, #101

🤖 Generated with Claude Code

Fixes abilityai/trinity-enterprise#326

[github-actions]

⚠️ Nightly unit-suite check skipped — merge conflict against dev.

Resolve by running git merge dev locally and pushing the result. The next nightly run will re-test once the conflict is gone.

[dolho]

Rebased onto current dev — the PR was conflicting, so there was no refs/pull/1983/merge for CI to build and no checks were reporting at all. They should run now.

Conflicts were in tests/registry.json (both sides appended entries; unioned, ours last). Resolved by re-serializing from parsed JSON rather than splicing lines, so the separating comma can't be lost.

Local verification after the rebase is in the individual runs; no source conflicts, only the registry.

Ready for review.

[dolho]

The rebase earned its keep — regression diff went red on a real defect, not on drift.

test_database_facade_delegation landed on dev after this PR was written. It fails here because routers/executions.py calls db.get_fleet_execution_timeline(...) and DatabaseManager has no such method. I checked for a __getattr__ fallback on the facade — there is none — so that call raises AttributeError: the endpoint would have 500'd on every request.

What makes it worth writing up is why the PR's own 524-line suite stayed green over it. Every mock of that seam was:

monkeypatch.setattr(router_mod.db, "get_fleet_execution_timeline",
                    lambda *a, **k: [], raising=False)

raising=False turns off monkeypatch's existence check. The tests stubbed a method that never existed, on a facade they therefore never touched — so they asserted the router's behaviour against a seam that could not work in production. 33 green tests over a guaranteed 500.

Fixed both halves:

• The pass-through on DatabaseManager, delegating to _schedule_ops in the same shape its get_fleet_execution_stats sibling already uses.
• All nine raising=False opt-outs removed, so each mock now asserts the seam exists.

Mutation-checked rather than assumed: with the facade method removed again, 9 of these tests fail. Before this commit, all 33 passed.

111 passed locally across executions / facade / ent326 / database.

[vybe]

Small, read-only, well-tested, and the parts that usually go wrong are right: Invariant #16 is satisfied (iso_cutoff(hours) as a bound parameter, no datetime('now', ...)), and access control uses narrow_to_agent(accessible_agent_names(...)) identically to /stats and the list route, with agent_names == [] short-circuiting before any SQL. The test that pins the dangerous direction is the right test to have written.

Three things to settle before it lands — two mechanical, one that needs a decision.

1. Trigger-bucket ordering diverges from the established chart convention.

_fold_trigger_buckets ends with sorted(..., key=lambda b: b["bucket"]) — alphabetical. _BUCKET_ORDER in db/schedules/analytics.py exists precisely so "Other" sorts last and the legend/stack order is stable. Alphabetical puts Other sixth, between MCP and Public, so the #1107 Overview chart and this endpoint's tile will render the same buckets in different orders on the same page. _BUCKET_ORDER isn't currently exported from db/schedules/__init__.py — export it and use it.

2. Invariant #1: both Python-side transforms are re-implemented in the router.

db/schedules/analytics.py already owns _bucket_for_trigger() and a gap-filled continuous UTC-day axis. This PR reimplements both in routers/executions.py, inlining _TRIGGER_BUCKETS.get(..., "Other") with a literal "Other" rather than the _OTHER_BUCKET constant. That's domain mapping in a router plus a second copy of a mapping the codebase deliberately centralized. Both belong in ScheduleStatsMixin beside the query, matching the #1107 precedent.

3. Entitlement ruling needed. This lands ungated on the OSS routers/executions.py, visible to every authenticated user, but ent#326 is an enterprise-tracker issue — so per CLAUDE.md the default is gated unless explicitly ruled OSS-core. There's a plausible precedent (Dashboard Grid, ent#47, shipped OSS-core and is in the public architecture doc), but the PR never states the decision. Worth one line from @vybe either way so it isn't inferred later from the fact that it merged.

Worth knowing, not blocking:

• Legacy naive started_at rows silently vanish from the hour/day series. Pre-#1474 scheduler rows are stored as YYYY-MM-DD HH:MM:SS (space, no Z). They pass the lexicographic cutoff, but substr(started_at,1,13) yields 2026-08-06 10, which never matches the gap-fill axis key 2026-08-06T10 — so the bucket renders as a real zero while group_by=trigger|agent and /stats still count them. That is the exact "chart contradicts the stat card above it" failure the PR cites as its own rationale for counting error as failed, arriving by a different route. Time-bounded (720h window vs 90-day retention) so it self-heals ~30 days after any deploy past #1474 — but it should be stated rather than discovered.
• The Invariant #4 framing is decorative: there is no GET /api/executions/{execution_id} route anywhere in the backend, and @router.get("") matches only the exact path. Ordering is right; the claimed collision risk doesn't currently exist.
• Dead branch: hours > _MAX_GAP_FILLED_HOURS is unreachable, since _VALID_HOURS maxes at 720 and the membership check runs first.
• Invariant #13: no MCP tool. Defensible for a UI-tile endpoint, but worth stating — list_recent_executions already exists and an agent asking "what did the fleet cost this week" is a plausible consumer.

Tests are strong for the size — 422 naming, the hours=0 asymmetry, gap-fill continuity/ordering/dedup, the Other catch-all, empty-allow-list, a real-SQLite db-layer test, and a token-column guard matched by explicit name rather than substring. No coverage gap I'd hold on.

Also needs a rebase (tests/registry.json only).

[vybe]

Re-validated today as part of a merge sweep. Holding — head unchanged since my review; the only new commit is a Merge origin/dev, which clears the tests/registry.json rebase I asked for but none of the three items.

Confirmed still present: _fold_trigger_buckets still ends return sorted(folded.values(), key=lambda b: b["bucket"]) — alphabetical — and there are zero references to _BUCKET_ORDER in the diff. So Other still sorts sixth rather than last, and the #1107 Overview chart and this endpoint's tile will render the same buckets in different orders on the same page.

Items 2 (Invariant #1 — both Python-side transforms re-implemented in the router, with a literal "Other" instead of _OTHER_BUCKET) and 3 (the entitlement ruling) are also untouched.

On item 3, here is the ruling so it isn't inferred later from the fact that it merged: ent#326 is enterprise-tracker, and per CLAUDE.md the default is entitlement-gated unless explicitly ruled otherwise. A fleet-wide execution-cost/throughput rollup is operational telemetry over data every authenticated user can already reach via /api/executions and /api/executions/stats, and the Dashboard Grid precedent (ent#47) shipped OSS-core on the same reasoning. Ruling it OSS-core — please state that decision in the PR body so the record is explicit.

The read-only scoping, Invariant #16 compliance, and the access-control reuse are all correct; it's the two mechanical items that block.

[dolho]

@vybe — items 1 and 2 fixed, plus the legacy-timestamp note fixed rather than documented. Item 3 needs your ruling; I've deliberately not decided it.

1. Bucket ordering — _BUCKET_ORDER is now exported from db/schedules/__init__.py (with _OTHER_BUCKET and _bucket_for_trigger) and used here, so Other sorts last instead of sixth. A label present in the map but missing from the order sorts after the known ones rather than being dropped.

before:  Chat/Tasks, MCP, Other, Public, Scheduled     (alphabetical)
after:   Chat/Tasks, Scheduled, Other                  (shared order, Other last)

2. Invariant #1 — both transforms moved to ScheduleStatsMixin beside the query, reached via the DatabaseManager facade as db.shape_execution_timeline(...), so the router imports no private db symbol. The literal "Other" fallback is gone (it goes through _bucket_for_trigger now). A test pins that routers/executions.py never mentions _TRIGGER_BUCKETS again.

Legacy naive started_at rows — fixed, not just stated. You're right that this is the "chart contradicts the stat card above it" failure arriving by another route, and waiting ~30 days for retention to age it out means every install that deploys before then sees it. The bucketer now normalises the separator — replace(started_at, ' ', 'T'), ANSI and present on both backends — so pre-#1474 rows land in the same bucket as ISO-Z rows instead of rendering a real zero.

3. Entitlement ruling — over to you. I've left it ungated (unchanged from the PR as filed) rather than guessing, because per CLAUDE.md the default for an enterprise-tracker feature is gated unless explicitly ruled OSS-core, and that's a monetization call, not mine. Both readings are defensible here: Dashboard Grid (ent#47) shipped OSS-core and is in the public architecture doc, and this endpoint is generic fleet telemetry over OSS tables — but "can build in OSS" ≠ "should". Say the word and I'll either add requires_entitlement(...) + move the logic behind the private module, or leave it and add the one-line note recording the OSS-core decision so it isn't inferred later from the fact that it merged.

Not changed, and I think correctly: the dead hours > _MAX_GAP_FILLED_HOURS branch (unreachable but a cheap belt if _VALID_HOURS ever grows), the decorative Invariant #4 framing (I've left the ordering as-is since it costs nothing), and the missing MCP tool (Invariant #13) — happy to add get_execution_timeline if you want the agent-facing surface, but that felt like scope beyond a UI-tile endpoint.

4 tests added; 34 pass. Merged latest dev.

[vybe]

Re-reviewed, and item 3 ruled: OSS-core, ungated.

You were right not to guess it — CLAUDE.md's default for an enterprise-tracker feature is gated unless explicitly ruled otherwise, and that's a monetization call. The ruling: leave it ungated. Generic fleet telemetry over OSS tables, and its consumer (Dashboard Grid, ent#47) already shipped OSS-core. I've recorded it in architecture.md beside the endpoint (edb277a) rather than leaving it implicit — an unstated default is exactly how "it merged ungated" becomes an accidental precedent nobody remembers deciding.

1. Bucket ordering — sharing _BUCKET_ORDER/_bucket_for_trigger from db/schedules/__init__.py is better than the fix I asked for: a label present in the map but missing from the order now sorts after the known ones instead of being dropped.

2. Invariant #1 — both transforms behind db.shape_execution_timeline(...), router imports no private db symbol, and the test pinning that routers/executions.py never mentions _TRIGGER_BUCKETS again is what stops it regressing.

Legacy naive started_at — thank you for fixing rather than documenting this. replace(started_at, ' ', 'T') is ANSI and works on both backends, and waiting ~30 days for retention to age the rows out would have meant every install deploying before then sees a chart contradicting the stat card above it.

Accepted as scoped: the dead hours > _MAX_GAP_FILLED_HOURS branch (cheap belt), the Invariant #4 framing, and no MCP tool — agreed that an agent-facing surface for a UI-tile endpoint is scope creep; file it if a caller appears.

34 tests pass against the merged tree.