Skip to content

feat(alerts): maintenance windows - #2323

Merged
Makisuo merged 64 commits into
mainfrom
feat/alerts-maintenance-windows
Nov 7, 2026
Merged

Makisuo merged 64 commits into
mainfrom
feat/alerts-maintenance-windows

Conversation

@Makisuo

@Makisuo Makisuo commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Maintenance windows: scheduled, recurring silences that also keep planned maintenance out of the anomaly baselines.

What changed

  • Model. A window is a schedule plus silence matchers (rules, tags, services, environments) with an optional reason. The schedule is either one-off (once), or weekly on chosen days at a local HH:MM with a duration, in an IANA time zone. Occurrences are computed in that zone, so DST is handled: a start inside a spring-forward gap opens at the next valid time, and on a fall-back overlap the earlier instant wins.
  • Reuses silences. Occurrences that overlap the next 48 hours become ordinary alert_silences rows linked by maintenance_window_id. Silencing, the audit trail, acknowledgement and the page owed when an occurrence ends all go through the existing silence path.
    • A 15-minute maintenance_windows lane and every create or update keep the rows in sync. Each write locks the window and checks it is unchanged, and stale future rows are removed.
    • Editing re-syncs. A running occurrence takes the new matchers and end, or ends now if the new schedule drops it. Disabling or deleting ends a running occurrence now; re-enabling reopens it.
    • The plain silence API refuses to edit or end a window's silences (400 pointing at the window). Window silences don't count toward the 200 open-silence cap.
  • Anomaly baselines. Windows that name a service or environment and have exclude_from_baselines (the default) keep their hours out of the detector's 7-day seasonal baselines. A planned outage no longer teaches the baseline that the outage is normal. Exclusions come from the occurrences that actually ran (the flag is recorded on each silence row), so deleting or editing a window later doesn't rewrite history. Error-spike baselines are 7-day totals with no hourly breakdown, so they're left as they are.
  • Limits. 100 windows per org (ended one-off windows don't count). The lane pages through all windows with a cursor.
  • API. v2 /v2/alerts/maintenance_windows (list, retrieve, create, update, delete; mw_ ids, audit events). Silences gain maintenance_window_id. MCP: list_maintenance_windows, create_maintenance_window and delete_maintenance_window (needs confirm).
  • Migration. Adds alert_maintenance_windows, plus maintenance_window_id and exclude_from_baselines on alert_silences, with a unique index on (window, start).

The dashboard UI for windows comes next, on top of the silences UI.

Verification

  • Occurrence maths: 24 domain tests, covering week boundaries, multiple days, occurrences already running, and Berlin and New York DST gaps and overlaps.
  • Sync tests:
    • 48h materialization, idempotence and paging;
    • overlapping occurrences (a daily window lasting 25 hours);
    • edits that move, drop or shorten a running occurrence;
    • disable and re-enable mid-occurrence;
    • the race guard and stale-row cleanup.
  • Integration: a weekly window active now records the trigger as silenced, and the owed page goes out exactly once after the occurrence ends, including across a disable and re-enable.
  • Detector: a maintenance spike in the baseline hides a real anomaly without a window and breaches with one. Ended, flag-off and plain silences don't change baselines.
  • Suites:
    • backend alerts and org: 812
    • domain: 1330
    • db with parity: 71
    • alerting: 25
    • web-data: 304
    • v2 routes: 389
    • MCP: 1024
    • web alerts: 278
  • Typecheck clean in every touched package; both oxlint configs clean.
  • An independent review found eight sync and baseline bugs. Each was fixed with a regression test.

Stack: #2318 (anomalies page through alert rules) → #2319 (silences match anomaly labels) → #2320 (anomalies in the alerts hub) → this PR (maintenance windows)

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

The anomaly detector opened anomaly incidents that nobody could route, ack,
silence or escalate. A new `anomaly` alert signal type reads the detector's
open anomalies the way uptime and SLO rules read their sources, so every
anomaly becomes an alert incident with the normal lifecycle.

- One undeletable, unrouted default anomaly rule per org (backfilled on the
  anomaly tick) plus optional rules scoped by service, environment and signal.
- An anomaly goes to the most specific rule that covers it and has a tier at
  or below its severity; it hands over when a narrower rule qualifies.
- Incidents are keyed by the anomaly incident id, link `anomaly_incident_id`,
  reuse the anomaly's issue and triage (no second issue or triage), and store
  a `group_label` such as "checkout (production)" for every surface.
- Notifications show the anomaly sentence instead of the severity code.
- v2 API and MCP expose `anomaly_signals`, `anomaly_incident_id` and
  `group_label`.
…ncidents

# Conflicts:
#	apps/api/src/routes/v2/alert-incidents.http.ts
#	packages/backend/src/services/alerts/AlertReadModelsService.ts
#	packages/backend/src/services/alerts/AlertsService.ts
#	packages/db/src/tables/alerts.ts
#	packages/domain/src/http/alerts.ts
#	packages/domain/src/http/v2/alert-incidents.ts
#	packages/web-data/src/collections/alerts.ts
…nment

An anomaly rule's group key is the anomaly incident id and the default rule
has no scope, so a silence on a service never muted anomalies in it. Incidents
now record labels (service, environment); silence matching prefers them over
the rule's scope and group key, at open, on reminders, on the owed page after
a silence ends and on resolves.
- The rule editor has an Anomaly mode: scope by service, environment and
  signal, and route the warning and critical tiers. The default anomaly rule
  keeps its scope and may stay unrouted.
- Anomaly incidents show the detector's evidence (observed vs baseline chart,
  signal, deviation, error-spike fingerprints, issue and triage) on the
  incident page, with no threshold numbers.
- The inbox has an Anomalies view (open, resolved, all); the rules list marks
  the default anomaly rule. /anomalies redirects there; /anomalies/:id stays.
- MCP says an anomaly rule has no query window to preview instead of
  reporting a preview that returned no series.
- iOS shows an anomaly incident's value as Warning or Critical.
- Anomaly rules reject excludeServiceNames rather than dropping it.
- The default-rule guard reads the default rule under the write lock for
  every rule write.
- Anomaly rules never take the orphan path, so a rule disabled mid-tick
  cannot send its incidents to a warehouse probe.
A maintenance window is a schedule (once, or weekly on chosen days at a local
time in an IANA time zone) plus silence matchers. Occurrences in the next 48
hours become ordinary alert silences linked to the window, so silencing, the
audit trail and the page owed after a window ends reuse the silence path.
Editing re-materializes; deleting or disabling ends a running occurrence.

Windows that name a service or environment also keep their hours out of the
anomaly detector's seasonal baselines, so planned maintenance does not teach
the baseline that an outage is normal.

v2 /v2/alerts/maintenance_windows and MCP list/create/delete tools.
…ncidents

# Conflicts:
#	apps/ai/src/mcp/tool-descriptors.generated.ts
…t-labels

# Conflicts:
#	apps/ai/src/mcp/tool-descriptors.generated.ts
# Conflicts:
#	apps/web/src/page/alerts/hub/update.ts
#	apps/web/src/page/alerts/incident/command.ts
#	apps/web/src/page/alerts/incident/message.ts
#	apps/web/src/page/alerts/incident/model.ts
#	apps/web/src/page/alerts/incident/update.ts
#	apps/web/src/page/alerts/incident/view.ts
#	apps/web/src/page/anomalies/detail/view-body.ts
#	apps/web/src/page/anomalies/list/view-list.ts
…ndows

# Conflicts:
#	apps/ai/src/mcp/dispatcher.ts
#	apps/ai/src/mcp/lib/params.ts
#	apps/ai/src/mcp/resources/instructions.ts
#	apps/ai/src/mcp/tool-descriptors.generated.ts
#	apps/ai/src/mcp/tools/mutating.ts
#	apps/ai/src/mcp/tools/registry.ts
#	apps/ai/src/mcp/tools/runtime-requirements.ts
#	apps/ai/test/mcp/unimplemented-tool-services.ts
#	apps/landing/src/content/docs/reference/mcp.md
#	packages/domain/src/mcp-outputs/catalog.ts
#	packages/domain/src/mcp-outputs/index.ts
@coderabbitai

coderabbitai Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 8 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c9e23306-14be-4f01-81c2-c86927c11d18
Commits

Reviewing files that changed from the base of the PR and between 018c522 and 1a16bad.

Files ignored due to path filters: 2

  • apps/ai/src/mcp/tool-descriptors.generated.ts is excluded by !**/*.generated.*
  • packages/domain/src/generated/anticipated-error-identifiers.ts is excluded by !**/generated/**
Files selected for processing: 72

  • apps/ai/src/mcp/dispatcher.ts
  • apps/ai/src/mcp/lib/alert-silences.ts
  • apps/ai/src/mcp/lib/maintenance-windows.ts
  • apps/ai/src/mcp/lib/params.ts
  • apps/ai/src/mcp/resources/instructions.ts
  • apps/ai/src/mcp/tools/__tests__/alert-silence-tools.test.ts
  • apps/ai/src/mcp/tools/__tests__/maintenance-window-tools.test.ts
  • apps/ai/src/mcp/tools/create-maintenance-window.ts
  • apps/ai/src/mcp/tools/delete-maintenance-window.ts
  • apps/ai/src/mcp/tools/list-maintenance-windows.ts
  • apps/ai/src/mcp/tools/mutating.ts
  • apps/ai/src/mcp/tools/registry.ts
  • apps/ai/src/mcp/tools/runtime-requirements.ts
  • apps/ai/test/mcp/unimplemented-tool-services.ts
  • apps/alerting/src/schedule.ts
  • apps/alerting/src/scheduled.test.ts
  • apps/alerting/src/scheduled.ts
  • apps/api/src/routes/v2/alert-maintenance-windows.http.ts
  • apps/api/src/routes/v2/alert-silences.http.ts
  • apps/api/src/routes/v2/alerts.http.test.ts
  • apps/api/src/routes/v2/v2-test-support.ts
  • apps/api/src/runtime/http-graph.ts
  • apps/api/src/runtime/service-graph.ts
  • apps/ios/Packages/MapleAPI/Sources/MapleAPI/openapi.json
  • apps/landing/src/content/docs/reference/mcp.md
  • apps/web/src/page/alerts/hub/hub-silences.scene.test.ts
  • apps/web/src/page/alerts/hub/view-silences.ts
  • apps/web/src/page/alerts/incident/incident-anomaly.scene.test.ts
  • apps/web/src/page/alerts/incident/incident.scene.test.ts
  • apps/web/src/page/alerts/rule/rule.scene.test.ts
  • apps/web/src/page/alerts/shared/silences.test.ts
  • apps/web/src/page/alerts/silence-dialog/dialog.test.ts
  • packages/backend/src/services/alerts/AlertMaintenanceWindowsService.sync.test.ts
  • packages/backend/src/services/alerts/AlertMaintenanceWindowsService.test.ts
  • packages/backend/src/services/alerts/AlertMaintenanceWindowsService.ts
  • packages/backend/src/services/alerts/AlertSilencesService.ts
  • packages/backend/src/services/alerts/AlertsService.maintenance-windows.test.ts
  • packages/backend/src/services/alerts/AnomalyDetectionService.maintenance.test.ts
  • packages/backend/src/services/alerts/AnomalyDetectionService.ts
  • packages/backend/src/services/alerts/alert-maintenance-window-store.ts
  • packages/backend/src/services/alerts/alert-maintenance-window-sync.ts
  • packages/backend/src/services/alerts/alert-silence-store.ts
  • packages/backend/src/services/alerts/alerts-service-test-support.ts
  • packages/backend/src/services/alerts/anomaly/detection.test.ts
  • packages/backend/src/services/alerts/anomaly/detection.ts
  • packages/backend/src/services/org/OrganizationService.ts
  • packages/db/drizzle/20261106195741_alert_maintenance_windows/migration.sql
  • packages/db/drizzle/20261106195741_alert_maintenance_windows/snapshot.json
  • packages/db/drizzle/20261106203630_merge_maintenance_windows_escalation_lease/migration.sql
  • packages/db/drizzle/20261106203630_merge_maintenance_windows_escalation_lease/snapshot.json
  • packages/db/drizzle/20261106235808_merge_maintenance_windows_escalation_chains/migration.sql
  • packages/db/drizzle/20261106235808_merge_maintenance_windows_escalation_chains/snapshot.json
  • packages/db/drizzle/20261107000733_merge_maintenance_windows_oncall/migration.sql
  • packages/db/drizzle/20261107000733_merge_maintenance_windows_oncall/snapshot.json
  • packages/db/src/tables/alerts.ts
  • packages/db/src/tables/index.ts
  • packages/domain/src/audit-actions.ts
  • packages/domain/src/http/alert-maintenance-windows.test.ts
  • packages/domain/src/http/alert-maintenance-windows.ts
  • packages/domain/src/http/alert-silences.ts
  • packages/domain/src/http/index.ts
  • packages/domain/src/http/v2/alert-maintenance-windows.ts
  • packages/domain/src/http/v2/alert-silences.ts
  • packages/domain/src/http/v2/api.ts
  • packages/domain/src/http/v2/index.ts
  • packages/domain/src/http/v2/openapi.test.ts
  • packages/domain/src/http/v2/public-id.ts
  • packages/domain/src/http/v2/resource-ids.ts
  • packages/domain/src/mcp-outputs/catalog.ts
  • packages/domain/src/mcp-outputs/index.ts
  • packages/domain/src/mcp-outputs/maintenance-windows.ts
  • packages/primitives/src/index.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@Makisuo
Makisuo deployed to pr-preview November 6, 2026 21:30 — with GitHub Actions Active
@maple-review-bot

This comment has been minimized.

@Makisuo
Makisuo deployed to pr-preview November 6, 2026 21:39 — with GitHub Actions Active
@Makisuo

Makisuo commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

F4 / F5: fixed in 99835dc, with tests in AlertMaintenanceWindowsService.sync.test.ts.

  • F4: re-enabling a one-off window whose end has passed is refused ("A one-off maintenance window must end in the future"). Weekly windows and one-offs still running re-enable as before.
  • F5: a silence whose window was deleted now refuses edits with "This silence came from a maintenance window that was deleted, so it cannot be edited" instead of naming the window's UUID.

Also from the live preview re-test (89665ce): the hub's Silences tab no longer offers "End now" on a window's silence, which the server always refused. It shows a "Maintenance window" badge instead; scene test added.

@maple-review-bot

This comment has been minimized.

@Makisuo
Makisuo deployed to pr-preview November 6, 2026 21:52 — with GitHub Actions Active
@Makisuo
Makisuo deployed to pr-preview November 6, 2026 22:00 — with GitHub Actions Active
@Makisuo
Makisuo changed the base branch from feat/alerts-unified-ui to main November 6, 2026 23:56
Main now has the anomaly stack (#2318-#2320) and escalation chains (#2304).
This branch's own changes are replayed onto main; registrations keep both
the escalation-chain and maintenance-window tools, and a merge migration
joins the maintenance-window and escalation-chain migrations.
@maple-review-bot

This comment has been minimized.

@Makisuo

Makisuo commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator Author

F6, F7 and F8 (all notes) are queued as one small follow-up PR after this merges: a revision-based unchanged guard instead of the millisecond updatedAt, recounting the window cap inside the create transaction under the org lock, and importing the domain's weekday literals in the MCP tool.

…ce-windows

# Conflicts:
#	apps/ai/src/mcp/dispatcher.ts
#	apps/ai/src/mcp/lib/params.ts
#	apps/ai/src/mcp/resources/instructions.ts
#	apps/ai/src/mcp/tools/runtime-requirements.ts
#	apps/ai/test/mcp/unimplemented-tool-services.ts
#	apps/landing/src/content/docs/reference/mcp.md
#	packages/domain/src/http/v2/openapi.test.ts
#	packages/domain/src/mcp-outputs/catalog.ts
#	packages/domain/src/mcp-outputs/index.ts
@maple-review-bot

maple-review-bot Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge. Two child reviewers returned unusable output, so I read the sync, service, schema, route, MCP and test hunks myself; three earlier notes remain unfixed.
3 notes · tests covered · risk medium · 4/4 new units observable

Adds maintenance windows end to end: schedule model, a 15-minute lane writing occurrences as alert_silences, v2 CRUD, three MCP tools, anomaly baseline exclusion. No new defect; the three earlier notes remain unfixed, and the McpQueryError/VcsWebhookEnqueueFailure stacks this touches are untouched.

  • materialize writes each enabled window's next 48 hours as window-owned alert_silences
  • Silence API refuses to edit or end a window's silences; they skip the 200 open-silence cap
  • Occurrences that ran with exclude_from_baselines drop their hours from golden-signal baselines
  • v2 /v2/alerts/maintenance_windows CRUD with mw_ ids and audit events, plus three MCP tools

Before merge

Still open from earlier reviews

Production impact

Open errors in the changed files (3)
Issue Service Occurrences File
@maple/mcp/errors/McpQueryError (4 issues) maple-chat 30 apps/ai/src/mcp/tools/registry.ts
@maple/api/routes/VcsWebhookEnqueueFailure (2 issues) maple-api 9 apps/api/src/runtime/http-graph.ts
@maple/mcp/errors/McpQueryError (4 issues) maple-ai 12 apps/ai/src/mcp/tools/registry.ts

After this merges, Maple checks whether they stop.

1a16bad · What was checked, coverage and telemetry details are on the Maple / review check. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo deployed to pr-preview November 7, 2026 00:10 — with GitHub Actions Active
@Makisuo
Makisuo deployed to pr-preview November 7, 2026 00:16 — with GitHub Actions Active
@Makisuo Makisuo removed the preview Deploy a full PR preview stack (Cloudflare + AWS ingest); removing it tears it down label Nov 7, 2026
@Makisuo
Makisuo deployed to pr-preview November 7, 2026 00:21 — with GitHub Actions Active
@Makisuo
Makisuo merged commit ec5b479 into main Nov 7, 2026
42 checks passed
@Makisuo
Makisuo deleted the feat/alerts-maintenance-windows branch November 7, 2026 00:21
@maple-review-bot

Copy link
Copy Markdown

Merged with 5 steps from "Before merge" still open. If they are done, tick them on the review comment; if not, now is the time.

  • Manual · Ship migration 20261106195741_alert_maintenance_windows (and its three merge migrations) in the same prd deploy that starts the 15-minute lane
  • Migration · Check it is safe on production data and with the code still deployed
  • Migration · Check it is safe on production data and with the code still deployed
  • Migration · Check it is safe on production data and with the code still deployed
  • Migration · Check it is safe on production data and with the code still deployed

@maple-review-bot

Copy link
Copy Markdown

Maple: after this shipped

✅ Shipped clean. Nothing this change touches got worse in the hour after it deployed.

Deployed to maple-api (production) at 2026-11-07 00:45 UTC as e6907cf, the first version after the merge (the merge commit itself never reported). Compared the 60 minutes before and after.

Errors in the changed files:

  • ➖ @maple/mcp/errors/McpQueryError: 0/h → 0/h
  • ➖ @maple/api/routes/VcsWebhookEnqueueFailure: 0/h → 0/h
  • ➖ @maple/mcp/errors/McpQueryError: 0/h → 0/h
  • ➖ @maple/mcp/errors/McpQueryError: 0/h → 0/h
  • ➖ @maple/mcp/errors/McpQueryError: 0/h → 0/h
  • ➖ @maple/api/routes/VcsWebhookEnqueueFailure: 0/h → 0/h
  • ➖ @maple/mcp/errors/McpQueryError: 0/h → 0/h
  • ➖ @maple/mcp/errors/McpQueryError: 0/h → 0/h
  • ➖ @maple/mcp/errors/McpQueryError: 0/h → 0/h
  • ➖ @maple/mcp/errors/McpQueryError: 0/h → 0/h

Maple looks once, an hour after the first deploy that carries the merge.

This branch was successfully deployed

1 active deployment
pr-preview — 1a16bad2 Deployed Nov 7, 2026 by Makisuo via deploy-pr-preview #10070
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant