Repository navigation
feat(alerts): on-call schedules, and escalation steps that page whoever is on call - #2306
Merged
Merged
Conversation
An open incident can now be acknowledged ("someone is on it") and
unacknowledged. The ack is a sub-state of `open`, like the telemetry hold:
four nullable columns on alert_incidents (who, when, name, source), so every
open/resolved consumer keeps working. alert_incident_events keeps the history
(acknowledged, unacknowledged, ack_cleared) for the timeline and for on-call
escalation to read.
The lifecycle planner holds reminders and de-escalation notices while an
incident is acknowledged. An escalation to a higher severity still pages and
clears the ack. Resolve still sends the all-clear and clears it.
Surfaces: POST /v2/alerts/incidents/:id/acknowledge and /unacknowledge
(alerts:write, 409 on a resolved incident), GET /:id/events, an
acknowledged=true|false list filter, and the acknowledge_alert_incident MCP
tool (approval-gated in chat like the other mutating tools).
… in SQL The escalation cleared the ack from the tick's prefetch and wrote ack_cleared after the claimed update, outside it. A failed event insert could then lose the escalation page, and a human ack or unack during the evaluation got the wrong outcome. Every escalation now clears whatever ack stands when it commits, with a guarded update in the same claimed write, and writes ack_cleared only when one was cleared. The planner no longer carries the flag. Also: an API-key ack records no user (the key is not a person), the unacknowledge docs say an overdue reminder goes out on the next check, and the MCP incident list takes an acknowledged filter.
… page, list the newest events
# Conflicts: # apps/ai/src/mcp/tool-descriptors.generated.ts
# Conflicts: # packages/domain/src/mcp-outputs/catalog.ts
…, and send each step once
…lert-escalation-chains # Conflicts: # packages/backend/src/services/alerts/AlertsService.ts
Domain schemas for on-call schedules (layers, rotations, restrictions, overrides) and a pure resolver: onCallAt and renderOnCall with DST-aware local handoffs.
Same change as #2302, so CI's effect lint passes on this stack before it lands.
… and lose to a resolve or ack - the scheduler reads only runs with a decision to make, so standing acks cannot crowd out due steps - a step locks its incident and the drain drops a step whose incident closed or was acknowledged - a flap-suppressed re-open starts the chain past step 1 instead of dropping it - de-escalation stops the critical tier's chain; escalation refreshes a running chain's payload - v2 PATCH tiers without escalation_chain_id keep the stored chain; MCP "none" detaches every tier
Schedules and overrides as org-scoped tables, and OnCallSchedulesService: CRUD, overrides, who is on call at an instant and rendered shifts. Participants and override users must be workspace members; 50 schedules per org, 500 upcoming overrides per schedule, overrides at most 90 days and not over already.
/v2/alerts/oncall_schedules CRUD with onc_ ids, who is on call at an instant, rendered shifts and ovr_ overrides, audited. MCP tools list_oncall_schedules, who_is_on_call, create/update/delete_oncall_schedule and create/delete_oncall_override, naming people by display name.
…and formThresholds
Escalation steps can now target a member or an on-call schedule next to destinations. At step time the schedule's on-call person is resolved (a gap pages nobody and is logged and marked on the span), users are deduped, and one email row plus one row per linked chat account is queued in alert_member_notifications in the step's transaction. A new AlertMemberNotifier drains that outbox in the standard lane after the destination queue: lease claims, skip once the incident is acked or resolved, bounded retries for transient failures. Chat transports gain directMessage (Slack posts to the user id, Discord opens a DM channel, Hazel refuses). Chains are validated against members and schedules, a paged schedule cannot be deleted, schedule documents list their chains, and the incident timeline (v2 events, get_incident_timeline) shows who was paged on which channel without the address.
This comment has been minimized.
This comment has been minimized.
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 21 minutes. View limit details
Tip 🎯 Customize your risk assessments
✅ Automatic approvals, with your rules
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
This was referenced Nov 6, 2026
…hedules # Conflicts: # apps/ai/src/mcp/tool-descriptors.generated.ts
This comment has been minimized.
This comment has been minimized.
Makisuo
removed
the
preview:web
…-chains # Conflicts: # apps/api/src/routes/v2/alerts.http.test.ts # apps/api/src/routes/v2/v2-test-support.ts # apps/api/src/runtime/http-graph.ts # apps/api/src/runtime/service-graph.ts # apps/ios/Packages/MapleAPI/Sources/MapleAPI/openapi.json # packages/backend/src/services/alerts/AlertsService.ts # packages/domain/src/audit-actions.ts # packages/domain/src/http/index.ts # packages/domain/src/http/v2/api.ts # packages/domain/src/http/v2/public-id.ts
A due step first asks whether a silence covers the incident (one rule read, only when a step would go out) and defers under one, rechecking every minute. An incident that opens under a silence still starts its chain; step 1 waits. When the silence ends, the catch-up page re-keys the chain's due step to its own time, so a destination in both the tier and the step is paged once.
…hedules # Conflicts: # apps/api/src/routes/v2/v2-test-support.ts # apps/api/src/runtime/http-graph.ts # apps/api/src/runtime/service-graph.ts # apps/ios/Packages/MapleAPI/Sources/MapleAPI/openapi.json # packages/backend/src/services/alerts/AlertsService.ts # packages/backend/src/services/alerts/alert-escalation-runs.ts # packages/domain/src/audit-actions.ts # packages/domain/src/http/index.ts # packages/domain/src/http/v2/api.ts # packages/domain/src/http/v2/public-id.ts
This comment has been minimized.
This comment has been minimized.
…atch-up already paged Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hedules # Conflicts: # packages/backend/src/services/alerts/alert-escalation-runs.ts
This comment has been minimized.
This comment has been minimized.
…-chains # Conflicts: # apps/ai/src/mcp/dispatcher.ts # apps/ai/src/mcp/lib/params.ts # apps/ai/src/mcp/resources/instructions.ts # apps/ai/src/mcp/tool-descriptors.generated.ts # apps/ai/src/mcp/tools/runtime-requirements.ts # apps/ai/test/mcp/unimplemented-tool-services.ts # apps/landing/src/content/docs/reference/mcp.md # packages/domain/src/mcp-outputs/catalog.ts # packages/domain/src/mcp-outputs/index.ts
…hedules # Conflicts: # apps/ai/src/mcp/dispatcher.ts # apps/ai/src/mcp/lib/params.ts # apps/ai/src/mcp/resources/instructions.ts # apps/ai/src/mcp/tool-descriptors.generated.ts # apps/ai/src/mcp/tools/get-incident-timeline.ts # apps/ai/src/mcp/tools/runtime-requirements.ts # apps/ai/test/mcp/unimplemented-tool-services.ts # apps/landing/src/content/docs/reference/mcp.md # packages/domain/src/mcp-outputs/catalog.ts # packages/domain/src/mcp-outputs/index.ts
This comment has been minimized.
This comment has been minimized.
…-chains # Conflicts: # apps/ai/src/mcp/tool-descriptors.generated.ts # apps/ai/src/mcp/tools/get-alert-rule.ts # packages/backend/src/services/alerts/AlertRulesService.ts # packages/backend/src/services/alerts/AlertsService.ts
…hedules # Conflicts: # apps/ai/src/mcp/tool-descriptors.generated.ts # apps/ai/src/mcp/tools/get-incident-timeline.ts
This comment has been minimized.
This comment has been minimized.
…-chains # Conflicts: # apps/ai/src/mcp/tool-descriptors.generated.ts
…hedules # Conflicts: # apps/ai/src/mcp/tool-descriptors.generated.ts # apps/ai/src/mcp/tools/get-incident-timeline.ts
This comment has been minimized.
This comment has been minimized.
…edules # Conflicts: # apps/ai/src/mcp/dispatcher.ts # apps/ai/src/mcp/lib/escalation-chains.ts # apps/ai/src/mcp/lib/params.ts # apps/ai/src/mcp/resources/instructions.ts # apps/ai/src/mcp/tool-descriptors.generated.ts # apps/ai/src/mcp/tools/__tests__/alert-tools.test.ts # apps/ai/src/mcp/tools/__tests__/escalation-chain-tools.test.ts # apps/ai/src/mcp/tools/create-escalation-chain.ts # apps/ai/src/mcp/tools/list-escalation-chains.ts # apps/ai/src/mcp/tools/mutating.ts # apps/ai/src/mcp/tools/registry.ts # apps/ai/src/mcp/tools/runtime-requirements.ts # apps/ai/src/mcp/tools/update-escalation-chain.ts # apps/ai/test/mcp/unimplemented-tool-services.ts # apps/api/src/routes/v2/alerts.http.test.ts # apps/api/src/routes/v2/escalation-chains.http.ts # apps/api/src/routes/v2/v2-test-support.ts # apps/api/src/runtime/http-graph.ts # apps/api/src/runtime/service-graph.ts # apps/ios/Packages/MapleAPI/Sources/MapleAPI/openapi.json # apps/landing/src/content/docs/reference/mcp.md # apps/web/src/page/investigations/conversation/kit/tool-metadata.ts # packages/alerting-core/src/index.ts # packages/backend/src/services/alerts/AlertsService.escalation.test.ts # packages/backend/src/services/alerts/AlertsService.ts # packages/backend/src/services/alerts/EscalationChainsService.ts # packages/backend/src/services/alerts/alert-escalation-runs.ts # packages/backend/src/services/org/OrganizationService.ts # packages/db/src/tables/escalation-chains.ts # packages/db/src/tables/index.ts # packages/domain/src/audit-actions.ts # packages/domain/src/http/escalation-chains.ts # packages/domain/src/http/index.ts # packages/domain/src/http/v2/api.ts # packages/domain/src/http/v2/escalation-chains.ts # packages/domain/src/http/v2/index.ts # packages/domain/src/http/v2/openapi.test.ts # packages/domain/src/http/v2/public-id.ts # packages/domain/src/http/v2/resource-ids.ts # packages/domain/src/http/v2/v2-contract.test.ts # packages/domain/src/mcp-outputs/catalog.ts # packages/domain/src/mcp-outputs/escalation-chains.ts # packages/domain/src/mcp-outputs/index.ts # packages/primitives/src/index.ts
Maple review🟢 Confidence 10/10 · safe to merge. Resolver, paging transaction and notifier were read and are pinned by new tests; the v2 schedule schemas and some wiring modules were only partly read. Adds on-call schedules with a layered rotation resolver, escalation steps that page a member or whoever is on call, an email/chat-DM notifier, and v2 REST plus MCP surfaces. Reading the resolver, the paging transaction and delivery found no defect. The open
Before merge
Production impactOpen errors in the changed files (3)
After this merges, Maple checks whether they stop.
|
|
Merged with 1 step from "Before merge" still open. If it is done, tick it on the review comment; if not, now is the time.
|
Maple: after this shipped✅ Shipped clean. Nothing this change touches got worse in the hour after it deployed. Deployed to Errors in the changed files:
Maple looks once, an hour after the first deploy that carries the merge. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On-call schedules, and escalation steps that page "whoever is on call" by email and chat direct message.
Stack (merge in order): #2287 (ack) → #2304 → #2306 (this PR) → #2305 → #2307 (schedule UI).
What changed
oncall_schedules,oncall_overrides):onCallAt,renderOnCallin@maple/alerting-core):maple.escalation.oncall_gap; the run still advances.alert_member_notifications, drained byAlertMemberNotifierafter the destination drain):chat.postMessagewith the user id as channel;chat:writeis enough, so no new scope.pagedincident event ("Paged Ben Brook by Slack DM") in v2/incidents/{id}/eventsandget_incident_timeline, showing names only./v2/alerts/oncall_schedules(CRUD,GET /{id}/oncall?at=,GET /{id}/shifts?from=&to=, overrides), withonc_andovr_ids. Chain steps takeuser_idandschedule_idtargets. Deleting a schedule a chain targets is a 409.list_oncall_schedules,who_is_on_call,create_oncall_schedule,update_oncall_schedule,delete_oncall_schedule,create_oncall_overrideanddelete_oncall_override. The chain tools acceptuser_idsandschedule_ids.Verification
Resolver: 21 tests, including:
onCallAtagreeing every 37 minutes over 60 days.AlertsService.oncall-paging.test.ts:Suites:
Typecheck: backend, db, domain, chat-platform, api, ai, web, chat-bot and alerting are clean.
Lint:
oxlint -c .oxlintrc.effect.jsonis clean.Migrations: additive only (three new tables).
An independent review pass is in progress; its fixes will land as follow-up commits on this PR.
🤖 Generated with Claude Code
Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.