Skip to content

feat(alerts): on-call schedules, and escalation steps that page whoever is on call - #2306

Merged
Makisuo merged 71 commits into
mainfrom
feat/alert-oncall-schedules
Nov 7, 2026
Merged

feat(alerts): on-call schedules, and escalation steps that page whoever is on call#2306
Makisuo merged 71 commits into
mainfrom
feat/alert-oncall-schedules

Merge remote-tracking branch 'origin/main' into feat/alert-oncall-sch…

3ef14dc
Select commit
Loading
Failed to load commit list.
Maple Review Bot / Maple / review succeeded Nov 6, 2026 in 3m 27s

Confidence 10/10 · No issues found

🟢 Confidence 10/10 · safe to merge. Resolver, paging transaction and notifier were read and are pinned by new tests; the v2 schedule schemas and some wiring modules were only partly read.
no findings · tests covered · risk high · 4/4 new units observable

Adds on-call schedules with a layered rotation resolver, escalation steps that page a member or whoever is on call, an email/chat-DM notifier, and v2 REST plus MCP surfaces. Reading the resolver, the paging transaction and delivery found no defect. The open McpQueryError and VcsWebhookEnqueueFailure issues are untouched by this diff.

  • onCallAt/renderOnCall resolve layered rotations, weekly windows and overrides in a schedule's time zone
  • Escalation steps take user and schedule targets, queued as alert_member_notifications in the step's transaction
  • AlertMemberNotifier drains member pages to email, Slack DM and Discord DM, skipping those queued before an ack
  • New v2 /v2/alerts/oncall_schedules CRUD, overrides, oncall/shifts reads, and seven MCP tools

Before merge

Production impact

Open errors in the changed files (3)
Issue Service Occurrences File
@maple/mcp/errors/McpQueryError (4 issues) maple-chat 30 apps/ai/src/mcp/tools/registry.ts
@maple/api/routes/VcsWebhookEnqueueFailure (2 issues) maple-api 9 apps/api/src/runtime/http-graph.ts
@maple/mcp/errors/McpQueryError (4 issues) maple-ai 12 apps/ai/src/mcp/tools/registry.ts

After this merges, Maple checks whether they stop.

Production traffic of the changed files (last 7 days)
File Calls/day Busiest operations
packages/backend/src/services/alerts/AlertMemberNotifier.ts 116 AlertMemberNotifier.drain 116/day, 0.0% err
packages/backend/src/services/alerts/ChatAlertPoster.ts 24 AlertDelivery.chat 24/day, 0.0% err
packages/backend/src/services/alerts/EscalationChainsService.ts 15 EscalationChainsService.list 15/day, 0.0% err
Telemetry this change adds and removes (25)
  • ➕ span name McpTool.createOnCallOverride · apps/ai/src/mcp/tools/create-oncall-override.ts:40
  • ➕ span name McpTool.createOnCallSchedule · apps/ai/src/mcp/tools/create-oncall-schedule.ts:37
  • ➕ span name McpTool.deleteOnCallOverride · apps/ai/src/mcp/tools/delete-oncall-override.ts:23
  • ➕ span name McpTool.deleteOnCallSchedule · apps/ai/src/mcp/tools/delete-oncall-schedule.ts:21
  • ➕ span name McpTool.listOnCallSchedules · apps/ai/src/mcp/tools/list-oncall-schedules.ts:21
  • ➕ span name McpTool.updateOnCallSchedule · apps/ai/src/mcp/tools/update-oncall-schedule.ts:38
  • ➕ span name McpTool.whoIsOnCall · apps/ai/src/mcp/tools/who-is-on-call.ts:29
  • ➕ span name AlertIncidentPagesService.listPages · packages/backend/src/services/alerts/AlertIncidentPagesService.ts:77
  • ➕ attribute maple.escalation.chat_workspaces · packages/backend/src/services/alerts/AlertMemberNotifier.ts:346
  • ➕ span name AlertMemberNotifier.deliver · packages/backend/src/services/alerts/AlertMemberNotifier.ts:354
  • ➕ attribute maple.escalation.recipients_missing · packages/backend/src/services/alerts/AlertsService.ts:6133
  • ➕ attribute maple.escalation.oncall_gap · packages/backend/src/services/alerts/AlertsService.ts:6145
  • ➕ span name AlertDelivery.chatDirect · packages/backend/src/services/alerts/ChatAlertPoster.ts:262
  • ➕ span name OnCallSchedulesService.list · packages/backend/src/services/alerts/OnCallSchedulesService.ts:308
  • ➕ span name OnCallSchedulesService.get · packages/backend/src/services/alerts/OnCallSchedulesService.ts:358
  • ➕ span name OnCallSchedulesService.create · packages/backend/src/services/alerts/OnCallSchedulesService.ts:367
  • ➕ span name OnCallSchedulesService.update · packages/backend/src/services/alerts/OnCallSchedulesService.ts:419
  • ➕ span name OnCallSchedulesService.delete · packages/backend/src/services/alerts/OnCallSchedulesService.ts:458
  • ➕ span name OnCallSchedulesService.listOverrides · packages/backend/src/services/alerts/OnCallSchedulesService.ts:505
  • ➕ span name OnCallSchedulesService.createOverride · packages/backend/src/services/alerts/OnCallSchedulesService.ts:521
  • ➕ span name OnCallSchedulesService.deleteOverride · packages/backend/src/services/alerts/OnCallSchedulesService.ts:586
  • ➕ span name OnCallSchedulesService.onCallAt · packages/backend/src/services/alerts/OnCallSchedulesService.ts:616
  • ➕ span name OnCallSchedulesService.shifts · packages/backend/src/services/alerts/OnCallSchedulesService.ts:636
  • ➕ span name OnCallSchedulesService.memberNames · packages/backend/src/services/alerts/OnCallSchedulesService.ts:660
  • ➕ span name Discord.directMessage · packages/chat-platform/src/connectors/discord/outbound.ts:521
What was checked
  • renderOnCall merges adjacent equal pieces and clamps onCallAt bounds back to the real handoff (packages/alerting-core/src/oncall.ts)
  • Member rows are claimed by compare-and-set and settled only by the claimer, so a concurrent drain cannot double-page
  • Every schedule and override query filters orgId (OnCallSchedulesService.ts:261,285), and schedule delete answers 409 when a chain step targets it
Observability coverage: 4 of 4 changes observable
Change Kind Observable Evidence
v2 on-call schedule, override, oncall and shifts routes inbound HTTP entrypoint yes Registered on the existing v2 HttpApi group; api HTTP spans are auto-instrumented via apps/api/src/runtime/http-graph.ts
AlertMemberNotifier drain loop and per-page delivery background worker yes Effect.withSpan("AlertMemberNotifier.drain") (AlertMemberNotifier.ts:521) and Effect.fn("AlertMemberNotifier.deliver") per page with maple.escalation.* attributes (:354)
Slack and Discord direct message outbound chat client yes AlertDelivery.chatDirect span in ChatAlertPoster.deliver; existing AlertDelivery.chat spans are live in telemetry (~24/day)
Page email outbound email yes Nested in the AlertMemberNotifier.deliver span with maple.escalation.recipient_channel; no address recorded

3ef14dc · Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

View more details on Maple Review Bot