Skip to content

Demo-readiness docs, plus the before twin's consumer shutdown fix that CI surfaced - #14

Merged
erikshafer merged 3 commits into
mainfrom
chore/demo-readiness-docs
Sep 6, 2026
Merged

erikshafer merged 3 commits into
mainfrom
chore/demo-readiness-docs

Conversation

@erikshafer

@erikshafer erikshafer commented Sep 6, 2026 •

Copy link
Copy Markdown
Owner

Three commits. The first is the doc-only pass from the 2026-09-05 handoff. The second and third fix the CI flake this PR's own first run surfaced; the third supersedes the second's teardown shape after CI rejected it. Erik reads before merge; judgment calls flagged.

Commits 2 and 3: before twin consumer shutdown (the CI flake)

CI on commit 1 failed with [Test Class Cleanup Failure (BeforeTwinCoupons)] System.ObjectDisposedException while all 54 + 15 tests passed. Never seen on main, six local runs clean. A throwaway probe (boot the before twin against the compose broker with a pre-loaded queue, dispose it mid-drain) reproduced the exact exception plus two more faults in the same path:

  1. StopAsync runs twice, concurrently, on the same hosted service under Alba. WebApplicationFactory.DisposeAsync stops a host whose own Run() is already stopping (both call stacks captured). The old StopAsync disposed the channel on each call; the second call hit RabbitMQ.Client's disposed-object check. That is the CI exception.
  2. A delivery in flight at shutdown was cancelled through the host's stopping token mid-database-call and nacked without requeue: a graceful shutdown dropped a baker task.
  3. The channel was disposed under a running handler, whose ack then landed on a closed connection.

Commit 2 fixed those but added graceful channel and connection closes. CI rejected it with TaskCanceledException after a 30 s stall, reproduced locally by running two contract suites at once. The stack pointed at RabbitMQ.Client's Connection.CloseAsync. Two facts from the client's 7.2.2 source: a graceful connection close floors any timeout below 30 s up to 30 s and throws on expiry, and a connection whose main loop never finishes stalls on its channel 0 for 5 s plus 20 s, which is the stall I had measured. The graceful channel close was what left the loop unfinished under load.

Final shape (NotifyBakerConsumer.cs, RabbitMqConnection.cs; no other files):

  • StopAsync is idempotent: one shared shutdown task behind a lock.
  • The handler no longer takes the stopping token, so in-flight work finishes and acknowledges.
  • Shutdown cancels the consumer and waits for the delivery in flight under a 5 s cap (warning and abort on expiry), then aborts the channel via DisposeAsync. No graceful channel close.
  • The connection is aborted with the client's bounded, non-throwing AbortAsync(5 s) before it is disposed.

Verification: eight stressed runs (two suites concurrently, four rounds) with zero cleanup failures, zero cap warnings, zero dropped deliveries, each about 15 s wall clock, against 43 s and 73 s with failures for commit 2 under the same load; three unstressed whole-solution runs green. The probe was deleted, not committed. The after twin needs nothing: Wolverine owns its shutdown.

Counts re-run by the inventory's method: before twin 2,140 raw / 1,775 non-blank (was 2,061 / 1,705; two files edited, none created). After twin unchanged. docs/file-inventory.md, the README table, and the CLAUDE.md numbers are updated; the build log has both rounds in full.

Judgment call 3: the fix adds 79 lines to the before twin, all shutdown hygiene. That is honest weight a layered codebase carries for owning a raw consumer, and the slide numbers move with it. The alternative was a test-side workaround, which I did not want to do to the proof.

Commit 1: CLAUDE.md and README refresh

  • Mode paragraph under the intro: demo readiness since 2026-09-05, no new slices, no bumps (security patches excepted), sessions are for dry-run support, fixes, and doc accuracy.
  • Slice slate reads as built; the "pre-agreed fallbacks" are marked as schedule insurance never needed.
  • The talk-content rule about the inventory states the final counts once; workflow steps and the OpenSpec bullet no longer assume a slice is being built.
  • docs/critter-stack-audit.md added to "Where detail lives".
  • Judgment call 1: the enterprise-parody display-name open item is kept and annotated as the only one left. Decide it or delete the bullet.
  • README: "The three features" is now "Three features and one message" with a fourth table row for the RabbitMQ extension (judgment call 2: revert the heading if you want the exact talk phrase, keep the row); inventory table gains the slice 004 row; repo tour lists tests/LayerCake.Slices.Tests and the audit doc; zero em dashes.

The two manual checks from the handoff (in the build log)

  • CI on main after PR Slice 004: NotifyBaker over RabbitMQ on both twins #13: green, 1m23s.
  • Demo page against the broker path: src/frontend/index.html opened from disk in the installed Chrome and driven through its own buttons by a Playwright script (the Claude-in-Chrome extension could not inject into any page this session). Both twins: reachable, BDAY10 valid, publish 201, order priced 82.00 / 8.20 / 73.80 on both, baker row highlighted within about 250 ms on both, no page errors; both layercake-*-baker-tasks queues at 1 consumer, 0 ready, 2 published, 2 acked afterwards.
  • The compose volume carries two "Demo Walk ..." cakes and two extra orders from that walk (the probe's own rows were deleted). docker compose down -v before a dry run restores the clean seed.

Noted, not changed

  • .github/workflows/ci.yml comment still says only "No PostgreSQL service"; the suite now also starts RabbitMQ via Testcontainers.
  • Deck checklist line: pre-pull postgres:17 and rabbitmq:4 on the demo machine.

CLAUDE.md: mode paragraph (no new slices, no bumps, what sessions are for now),
slice slate marked built, schedule fallbacks marked history, file-inventory rule
states the final counts, critter-stack-audit.md listed, display-name open item
annotated as the only one left.

README: fourth features-table row for the RabbitMQ extension under a heading a
skimmer sees, slice 004 row and current whole-twin counts in the inventory table,
repo tour lists tests/LayerCake.Slices.Tests and docs/critter-stack-audit.md.

Build log: this pass, plus the two manual checks since slice 004 (CI green on
main; demo page clicked through against the broker path on both twins).
…ounded

CI on the docs PR failed with an ObjectDisposedException in a test class
cleanup while every test passed. Root cause: under Alba, WebApplicationFactory
stops a host whose own Run() is already stopping, so NotifyBakerConsumer.StopAsync
ran twice concurrently and the second call disposed a channel the first was
already disposing. A probe that disposed the host mid-drain also showed a
graceful shutdown dropping an in-flight baker task (cancelled through the
stopping token, nacked without requeue) and acking on a closed connection.

NotifyBakerConsumer: StopAsync runs one shared shutdown task; shutdown cancels
the consumer, waits for the delivery in flight, closes the channel, all under a
5 s cap that falls back to abort with a warning; the handler no longer takes the
stopping token. RabbitMqConnection: close (bounded) before dispose, since
disposing an open connection stalled 25 s on the client's abort path.

Suite green six times in a row on the final code. Counts re-run: before twin
2,142 raw / 1,776 non-blank; inventory, README, CLAUDE.md, build log updated.
@erikshafer erikshafer changed the title Demo-readiness docs: CLAUDE.md and README read as built, not building Demo-readiness docs, plus the before twin's consumer shutdown fix that CI surfaced Sep 6, 2026
CI rejected the first shutdown fix with a TaskCanceledException from
RabbitMQ.Client's Connection.CloseAsync after a 30 s stall. In client 7.2.2 a
graceful connection close floors its timeout at 30 s and throws on expiry, and
the graceful channel close added in the first round left the connection's main
loop unfinished under load, which is the documented cause of that stall.

Consumer: still idempotent StopAsync, no stopping token in the handler, cancel
the consumer and await the in-flight delivery under a 5 s cap; then abort the
channel through DisposeAsync instead of closing it. Connection: AbortAsync(5 s),
bounded and non-throwing, before DisposeAsync.

Eight stressed runs (two suites at once) with zero failures at about 15 s each,
against 43 s and 73 s with failures for the rejected version. Counts re-run:
before twin 2,140 raw / 1,775 non-blank; inventory, README, CLAUDE.md, build log updated.
@erikshafer erikshafer self-assigned this Sep 6, 2026
@erikshafer
erikshafer merged commit 0ee8388 into main Sep 6, 2026
1 check passed
@erikshafer
erikshafer deleted the chore/demo-readiness-docs branch September 6, 2026 03:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant