Skip to content

release: Misty 1.0.1 — discovery heartbeat reliability - #379

Merged
qiuethan merged 23 commits into
mainfrom
staging
Nov 5, 2026
Merged

qiuethan merged 23 commits into
mainfrom
staging

Conversation

@qiuethan

@qiuethan qiuethan commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

What this changes

Promote staging to production as Misty 1.0.1, including the discovery heartbeat reliability fix in #378. This is also the initial production activation of the release version guard and promotion/history-sync workflow.

Release contents

The only application behavior change relative to current main is discovery monitoring. Other changes are release tooling, documentation, and meeting example-environment guidance. The release intentionally spans zones because it promotes the reviewed staging branch as a whole.

Incident context

Two production doc-discover invocations exited 2 with item_budget and monitor_failed=true while Drive listing and catalog discovery requests succeeded. The item budget is an expected checkpointed stop. Resource metrics did not show memory or disk exhaustion. Existing logs did not identify the underlying transport/provider error.

The fix tolerates one transient heartbeat failure and makes recurring failures diagnosable. It does not claim that the original network cause has been proven.

Validation

  • fix(discovery): retry transient heartbeat failures #378 PR CI passed, including database suites, lint/format checks, and Docker build.
  • Local offline documentation-system and targeted discovery suites passed with imports verified against the intended checkout.
  • Release version 1.0.1 passes promotion validation against current main; no existing release tags reserve it.
  • Promotion/history-sync and release-version regression suites passed locally.
  • Final promotion candidate: fea8ac323c7a4d311556670d6e423428940bb411. Its merge-queue, staging-push, and promotion-PR CI passed, including the database suites and required main source guard. The release remains approved.
  • Staging catalog deployment 8b768f7d-c7a5-4258-ab94-d286c2b4bffc reached SUCCESS at candidate 880e52f39aaba6cf54c6d1898fc027005faecc76; /health/ready returned HTTP 200.
  • Staging discovery deployment ae7bc677-16cd-47f4-8098-e53f4a04e7b7 at the same candidate reached SUCCESS. Run 58dfa633-7563-4505-bd6c-7b5899c69ed8 completed at 2026-11-05 05:44 UTC with outcome=completed, exit_code=0, monitor_failed=false, and no queued listings or pending items.
  • The preceding staging attempt on deployment fab8375b-e059-46a3-8d9b-2d28b89b232c stopped on a separate catalog HTTP transport failure with monitor_failed=false. The successful follow-up resumed its preserved checkpoint. This evidence does not establish the cause of that separate transport failure.
  • Prerequisites fix: validate reused release sync snapshots #377 and chore: sync production history into staging #380 merged after approval and green queue checks. The final candidate contains current main and the history merge preserves the preceding staging tree exactly. Relative to the verified staging application revision, only the release-automation implementation and tests differ.
  • Final production preflight found the meeting health endpoint returning 200, no established inbound connections, and no session or hand-off events since 06:00 UTC.

Deployment and recovery

Merged with a merge commit after required approval and green checks. Main release commit: ffb766e2318cbad688952a603b9a84deefc9928e (VERSION 1.0.1), merged at 2026-11-05 06:53 UTC. This completed the documented initial activation; subsequent promotions can be opened by the production promotion workflow.

No database migration, new runtime setting, key rotation, or service dependency ordering was required. The main push successfully ran the release App workflow and merged history-sync #381. Staging commit 887ddaa31228323bb0c1f28927405f3df77ab729 contains the exact production commit and preserves the approved candidate's tree. The promotion workflow and main CI both passed.

Railway services tracking main deploy through their configured watch paths. Verify the resulting production source SHA and deployment ID for doc-discover, its successful actual invocation with monitor_failed=false, and affected service readiness. Preserve the checkpoint volume. A failed rollout must be repaired or reverted through staging; do not reset discovery state.

Production verification

All deployments below reached SUCCESS at source revision ffb766e2318cbad688952a603b9a84deefc9928e. HTTP health/readiness checks returned 200 for all seven Python services. The Discord bot registered its global commands and logged ready on the new deployment. Railway reported no current environment issues or pending work at the final check.

Service Deployment
doc-discover 1e8df981-72da-45fe-b677-09ee39a4b94f
documentation-system bb0aec57-b429-4217-9937-73c11ddb5a40
doc-source-refresh 4d66a339-ee84-4ae2-bdf8-4e8730b70a04
team-tracking b29133f8-08a1-404c-94a5-c78ab6946479
team-groups-sync 12920e28-b85a-483a-bc34-4d07c21bce32
connectors 8678dc2e-bc94-4628-ad78-c3e182a19741
gateway 0cfab517-c3d5-4cf0-9f38-aa5bb69de904
verification bf415d95-aa09-4b18-bdc8-6fbf092c3d73
meeting 96a0bf97-c67a-4b47-8d61-7c76fcab8660
llm ddd39f8d-8cd0-492b-80eb-4195eee6ab82
discord-bot 693d91da-f51d-4bb2-bcd0-3c0a41e0345d
  • Discovery run 27c4848b-ef70-4f01-bdb5-9889df185134 finished at 06:59 UTC with outcome=page_budget, exit_code=0, and monitor_failed=false; it preserved the existing scan/checkpoint.
  • The next scheduled discovery run, 54d96931-73d7-46c7-bfda-aa2bc713365b, finished at 07:01 UTC with outcome=completed_with_gaps, exit_code=0, monitor_failed=false, and no queued listings or pending items. Clean full-scan acceptance remains distinct from successful bounded invocation delivery.
  • The team-groups-sync deployment is ready for its normal schedule; no extra group-writing run was triggered for this release.

User documentation and publication

The heartbeat fix changes operator behavior and needs no member-facing wiki changes. The included documentation PRs define the separate wiki/release-publication process. This deployment request does not publish a GitHub Release or new wiki content.

Remaining uncertainty

Two actual production discovery invocations acknowledged their heartbeats successfully. The original intermittent transport/provider cause remains unidentified; successful runs establish deployment health but cannot prove that external delivery errors will never recur. Persistent failures still fail the job and now carry sanitized diagnostics.

Production preflight also found a pre-existing failure in the separate doc-source-refresh runner at 06:02 UTC: it completed its drain but failed monitor delivery, exited 2, and retained guard f8dc85de-7565-4538-8cf8-077e217b31bc. That runner uses a separate monitor implementation and is not changed by this discovery fix. Retain this baseline when assessing the rollout and follow the documented exact-owner recovery procedure after confirming the old runner and its API work have stopped.

Recovery completed during this rollout: the old runner and API deployments were both REMOVED, the database still held that exact finished owner, and the authorized owner-specific DELETE returned 200 with the matching run ID. Read-back found no guard. The subsequent scheduled refresh run d3fdb9b6-ecf9-488d-9476-65fe8f39326e completed at 07:02 UTC with exit_code=0, monitor_failed=false, drained=true, and guard_released=true (one document refreshed, no counted failures).

@github-project-automation github-project-automation Bot moved this to Backlog in Misty Nov 5, 2026
@github-actions github-actions Bot added size/xl >= 500 lines changed zone: discord-bot Owned by the discord-bot zone (docs/CODE-OWNERSHIP.md) zone: services/connectors Owned by the services/connectors zone (docs/CODE-OWNERSHIP.md) labels Nov 5, 2026
@github-actions github-actions Bot added zone: services/documentation-system Owned by the services/documentation-system zone (docs/CODE-OWNERSHIP.md) zone: services/llm Owned by the services/llm zone (docs/CODE-OWNERSHIP.md) zone: services/meeting Owned by the services/meeting zone (docs/CODE-OWNERSHIP.md) zone: services/team-tracking Owned by the services/team-tracking zone (docs/CODE-OWNERSHIP.md) zone: services/verification Owned by the services/verification zone (docs/CODE-OWNERSHIP.md) zone: docs Owned by the docs zone (docs/CODE-OWNERSHIP.md) zone: scripts Owned by the scripts zone (docs/CODE-OWNERSHIP.md) zone: .github Owned by the .github zone (docs/CODE-OWNERSHIP.md) zone: root Owned by the root zone (docs/CODE-OWNERSHIP.md) zone: services/gateway Owned by the services/gateway zone (docs/CODE-OWNERSHIP.md) labels Nov 5, 2026
@github-project-automation github-project-automation Bot moved this from Backlog to In review in Misty Nov 5, 2026
@railway-app
railway-app Bot temporarily deployed to Misty / staging November 5, 2026 06:49 Inactive
@qiuethan
qiuethan merged commit ffb766e into main Nov 5, 2026
53 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in Misty Nov 5, 2026

This branch was previously deployed

1 inactive deployment
Misty / staging — fea8ac32 Deployed Nov 5, 2026 by railway-app[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xl >= 500 lines changed zone: discord-bot Owned by the discord-bot zone (docs/CODE-OWNERSHIP.md) zone: docs Owned by the docs zone (docs/CODE-OWNERSHIP.md) zone: .github Owned by the .github zone (docs/CODE-OWNERSHIP.md) zone: root Owned by the root zone (docs/CODE-OWNERSHIP.md) zone: scripts Owned by the scripts zone (docs/CODE-OWNERSHIP.md) zone: services/connectors Owned by the services/connectors zone (docs/CODE-OWNERSHIP.md) zone: services/documentation-system Owned by the services/documentation-system zone (docs/CODE-OWNERSHIP.md) zone: services/gateway Owned by the services/gateway zone (docs/CODE-OWNERSHIP.md) zone: services/llm Owned by the services/llm zone (docs/CODE-OWNERSHIP.md) zone: services/meeting Owned by the services/meeting zone (docs/CODE-OWNERSHIP.md) zone: services/team-tracking Owned by the services/team-tracking zone (docs/CODE-OWNERSHIP.md) zone: services/verification Owned by the services/verification zone (docs/CODE-OWNERSHIP.md)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants