Skip to content

feat(kernel): complete P02b-4 transactional job runtime - #32

Open
cemililik wants to merge 284 commits into
mainfrom
development
Open

cemililik wants to merge 284 commits into
mainfrom
development

Conversation

@cemililik

@cemililik cemililik commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Implement P02b-4's nine approved steps: tenant-scoped transactional jobs and external prepare → execute → finalize. LearnStack's durable record governs business completion; stock Hangfire carries opaque wakeups.
  • Add confined storage, fenced completion and atomic terminal MUST audit, tenant capacity, independent worker pools, owned lifecycle/cleanup, and durable recurrence with recipient-bound fan-out.
  • Enforce the nineteen job catalogue obligations and complete bounded CI partitioning. Production business/recurrence definitions remain empty; public API and Phase 02b packets 5–8 are unchanged.

Approach

The existing transaction owner commits business state, produced work, audit and job completion together. External effects run outside business transactions. Positive outcome/quiescence receipts, immutable failure policy and stored-row witnesses govern capacity release, retry and finalization; unknown effects keep their reservations.

All nine runtime steps and all three ADR-0063 implementation steps completed two independent review rounds and verified corrections. Subsequent verified corrections also completed two fresh independent reviews. The delivery record owns source-specific evidence and the accepted availability limits.

CI uses one base, six Architecture, twelve ordinary Docker and one native execution VM. Each execution retains its twenty-minute ceiling; both fail-closed aggregates retain five minutes. The seven required names and branch protection are unchanged. Only aggregates receive Actions read permission; they independently verify current-attempt jobs, immutable artifacts, complete case/result unions and the common base witness.

Tests

Verified review corrections are committed and both fresh independent review rounds are complete. Final-source hosted A/B qualification is running; do not merge yet. Current source is 12c79f38, with assignment A running. The following receipts describe the previous source. Attempt A's case-level artifacts now return HTTP 404; its historical successful jobs remain visible, and the independently retained local verification does not substitute for remotely retrievable proof. Attempt B's archives remain available. The final correction source must qualify two complete, distinct layouts with both evidence sets retained.

Historical source 1dbde1a3 previously completed both layouts.

Assignment Actual run / attempt Status
A 39188256776 / attempt 1 27/27 historical jobs pass; case archives currently unavailable (404)
B 39188256776 / attempt 2, Re-run all jobs 27/27 jobs pass; 5,479 cases independently verified, zero skips and natural cleanup

Both attempts were independently reconstructed when their archives were available. Current API retrieval can reconstruct only attempt B; attempt A is no longer independently reproducible from the remote archives. The historical local verification covered exact case unions, attempt artifacts, common manifest/toolchain, natural ownership/cleanup, zero skips and source/merge binding; this is not a claim that attempt A artifacts remain available. Source/workflow SHA is d90fd29f302c3aba73db080a4c411711e34f987d, bound to PR head 1dbde1a30eabd400f68f4940053bdce8b665601d.

Attempt Backend aggregate artifact / SHA-256 Integration aggregate artifact / SHA-256
A 12159885492 / bd1b30154f8e5025ef3c4b9d3a2503c566f137d817a05d61d46848e2bba8604f 12159366005 / 60deffdf0964092cdbbfc4b0b28a6de1365a0b5272ef55e2fabe26c51f686fcf
B 12159753776 / d92adc59bbe9ebc057e31ee098af827a15541f5c1cec27545f32cd4b12048667 12160345701 / a46f815cfa57f19fe48160891338ced15bba795125fba04ecb097d4538d67bf1

The complete source, plan, manifest and actual toolchain match across attempts. Architecture changes owner for 218/254 selector groups and ordinary Docker for 992/1,079; actual co-membership differs in both populations and SharedSchema. Maximum execution-job wall times are 859 s (A) and 825 s (B), within the unchanged 1,200 s ceiling. All four actual aggregate jobs meet 300 s; pre-upload evidence processing takes 10.833/16.741 s (A backend/integration) and 17.601/27.103 s (B). Queue time is recorded separately and supplies no execution-budget allowance.

Fresh discovery owns 5,479 cases: 3,043 non-Docker and 2,436 Docker, including 31 native cases. Every discovered case executes exactly once in each assignment, including all theory rows. Both complete distributions, natural resource cleanup, zero skips and all seven required checks qualify. Attempt 2 uses Re-run all jobs; no failed-only rerun, timeout increase, omitted case or forced cleanup supplies passing evidence.

Focused correction validation:

  • Warning-as-error Release build, 35 actual PostgreSQL public-site/native-owner cases, seven corpus/Docker-trait guards, and isolated public-pool proofs pass without failures/skips. Omitting only child-factory disposal makes the owned-completion proof fail; exact restoration, a fresh clean build and the isolated case pass again. This deliberate failure is a falsifiability control, not a green suite.
  • Execution-only fixtures observe actual entry or committed handoff before a route-correct stock hint; exact generation snapshots also join the optional hint's actual native return. Public-pool proof observes its real used lifetime and tagged sessions without disposing the source itself. Bridge/repair/fairness/future-due/no-admission subjects preserve their natural paths and held-owner assertions.
  • Matching production-source validation records 2,452 Unit cases, 26 actual PostgreSQL acquisition/role/bootstrap/heartbeat cases and both exact credential/bootstrap Architecture guards passing. Restoring the old synchronous driver call makes both direct-source and factory acquisition controls fail; restoring the correction passes all three source/factory cases.
  • Recorded protocol controls include 49 selector/result, 40 launcher and 18 native-provisioning cases. The prior complete Architecture epoch passes 410 cases; frontend validation records 1,625 cases with typecheck/lint and no skips/todos. These local/predecessor receipts are supplementary to the independently qualified final-source attempts above.

Qualification corrections address verified acquisition and test-proof synchronization gaps. Historical failures, immutable receipts and diagnostic limits remain in the hosted qualification record. No missing historical assertion or internal failure cause is inferred from a later passing run.

The initial correction-source run 39231264902 failed six recurrence cases because a test helper still searched for the old six-column observation projection. That failure is not qualified evidence. The helper now isolates only the unique terminal projection while retaining every production CTE/binding and all planted hash/date/history controls. All 110 recurrence-store cases pass locally; two additional fresh sequential focused reviews approve without findings. Production SQL is unchanged by that follow-up.

Current correction validation and scope:

  • Unresolved capacity recovery is observable in jobs health without releasing reservations or changing admission. Actual PostgreSQL coverage includes peer parking and original-owner late positive cleanup; all 339 distinct affected cases have a latest applicable passing result. Intermediate fixture failures remain recorded as failures.
  • Checkpoint tests use the real production admission handler; tenant-specific attempt observation and positive runtime-budget controls replace copied/implicit fixture behavior.
  • Published manifests are checked against reviewed checkout manifests; binary/notice/source co-tampering and PAX commit substitution fail. Fork API shape handling preserves run/head/merge/workflow bindings and unexpected failures produce closed refusal codes.
  • Release build: zero warnings/errors. Unit 2,455, Architecture 410, HTTP 177, Contract 4 and frontend 1,625 pass with zero skips. All 51 plan/aggregate controls, 23 distribution controls, 40 native-owner, 18 provisioning, five native-environment and seven ADR parser controls pass.
  • Live documentation now assigns authority to trusted durable rows and JobExecutor; the correction record records verified fixes, rejected proposals and both review rounds. No Accepted ADR body, historical migration or privilege grant changes.

Risk

  • Tenant/organization isolation and the four-role privilege boundary change under ADR-0047. Guarded credentials, column grants, RLS and closed construction/transaction capabilities remain mandatory.
  • ADR-0059 can keep a whole pool drained until actual work/cleanup finishes. ADR-0060 deliberately terminates the API for its bounded unsafe-startup/unwind class; process death never releases uncertain external reservations.
  • ADR-0062's collector provisioning is confined to the dedicated disposable native CI VM. Local/shared hosts are excluded. No dashboard, operator recovery route, privileged maintenance executor or mapped readiness endpoint ships; their named phase owners remain explicit.

Migration / Rollback

Apply the complete migration chain before API startup. The migration role installs the pinned vendor schema and exact grants; runtime DDL is disabled. Validate guarded dispatcher/application connection configuration and topology bounds before activation.

Stop workers before a reviewed rollback. Real-PostgreSQL proofs cover exact inventory restoration; incompatible retained history refuses migration rather than inventing failure policy. Do not manually reset unresolved external-effect/capacity state.

Related

Prepared with Codex.

Close both independent review rounds and local unit verification for
the CI-exposed timing race. Preserve the failed-job disclosure and
require replacement final-head CI before merge.
Record the verified squash merge and final PR evidence so current status
carriers agree on the completed phase and P02b-0 entry handoff. Preserve
published development history while incorporating the merged main base.
Separate delivered transport and public-site behavior from the pending
durable-event and identity contracts before producer decisions begin.
Record the verified code, Hub and CI baseline so later packets inherit
explicit constraints instead of stale implementation claims.
Record the coupled producer choices and bounded nesting replacement
before runtime implementation. Keep the gates Proposed for maintainer
approval, with a four-step plan and explicit P02b-3 seed handoff.

ADR: 0057
Separate bounded supersessions from additive seam and audit amendments
before approval. Define the transaction snapshot and JSON admission
boundaries, and make the G1-dependent subscriber handoff explicit.

ADR: 0057
Record maintainer approval before runtime changes and distinguish the
bounded supersessions from additive seams and audit clarifications.
Register producer proofs without claiming implemented delivery.

ADR: 0057
Keep transaction announcements and producer ancestry scoped to the actual
owner so absorbed refusals cannot commit successful production. Reuse
one audit lifecycle for requests and deferred direct handlers without
letting an independent DI scope inherit another scope's reconciliation.

ADR: 0040, 0044, 0057
Prevent raw callers and stale transaction ancestors from granting producer
ownership. Exercise a deeper refusal so an implementation that marks only
one parent cannot silently pass the rollback proof.

ADR: 0040, 0057
Keep the lifecycle carriers aligned with the verified coordinator ownership
before immediate outbox production is introduced.

ADR: ADR-0057
Keep the packet entry consistent with its completed Step 1 review record.

ADR: ADR-0057
Make successful enqueue an immediate atomic write with trusted execution
metadata and explicit identifier-only routing. Refuse stale, foreign or
unowned frames before persistence and retain production receipts across
absorbed failures without changing event-free execution.

ADR: 0057
Distinguish the phase-entry inventory from current producer delivery and
record the first independent review round without rewriting entry evidence.

ADR: 0057
A persisted event with no subscriber cannot reveal accidental delivery.
Observe the resolved transport with a positive control and close both
independent outbox review rounds against the passing database proof.

ADR: 0057
Collect facts only after their source save succeeds and retain their
invocation origin until physical-owner completion. Translate completed
tenant provisioning into one pending outbox row on the same commit as
both business roots and MUST audit, with no subscriber delivery yet.

ADR: 0057, 0040, 0042, 0044
Record two fresh independent reviews and their focused passing repeats
before starting the producer enforcement step.

ADR: 0057
Keep producer guarantees measurable across complete assembly censuses and
transitive helper paths, with deliberate dirty and clean controls. Align the
catalogue and authoring guidance with delivered producer capabilities while
retaining dispatcher and consumer ownership in their named packets.

ADR: 0057
Module: Tenancy, Customization, Audit
Expand the producer and domain-handler walks to derived implementations
so normal class dispatch cannot bypass their capability checks. Plant
dirty and clean class paths and verify the interface-only regression
fails. Reconcile the lease migration's existing-row premise.

ADR: 0057
Record complete producer delivery with exact local evidence, verified
review fixes and preserved decision history. Keep PR approval separate
and name the remaining dispatcher and consumer decision passes.

ADR: 0057
Keep private browser diagnostics contained while distinguishing launch,
hydration and document-fallback failures in the existing CI proof. The
first PR run failed without identifying the browser operation; preserve
all assertions and deadlines while making the next result actionable.

ADR: 0055
Align current-state references with delivered producer completion and
keep durable ordering, dispatch and inbox obligations with their owning
packets. Process-local serialization is not a durable ordering proof.

ADR: 0057
Remove the remaining durable-order equivalence claim found during the
independent CI follow-up review. Partition serialization is process-local;
P02b-2 G3 still owns ordering across commits and retries.

ADR: 0057
Keep XML guidance consistent with the corrected architecture and glossary.
Process-local invocation serialization does not decide durable ordering
across concurrent commits or retry, which remains P02b-2 G3.

ADR: 0057
Always run browser cleanup without allowing its failure to replace the
original diagnostic phase. A cleanup-only failure remains classified as
shutdown, and neither path exposes private exception text.

ADR: 0055
Retain the first failed browser receipt and distinguish the subsequent
passing integration evidence from an established cause. Record bounded
diagnostics, verified review corrections and their independent reviews;
the PR checks own final-head CI approval.

ADR: 0057
Exercise actual outbox receipts and require positive intent and owner
write observations so missing production work cannot pass these tests.

ADR: 0057
Keep the review follow-up traceable to verified regressions, controlled
mutations and independent reviews without claiming new runtime behavior.

ADR: 0057
A live transaction means frame completion refused before COMMIT. Route
that failure through guarded rollback so durable audit records the known
failure and cleanup cannot replace its original exception. Prove actual
business/outbox rollback and raw event-save admission in both hosts.

ADR: 0033, 0040, 0057
Keep shared transaction state inside the sequential-execution census,
with exact exceptions for independent singleton flights. Distinguish
private browser proof failures without exposing their source. Reconcile
uncached settings ownership and correct the historical clause locators
without changing accepted decisions.

ADR: 0033, 0040, 0041, 0057
A closed phase token alone cannot prove that an error was sanitized.
Use the same private brand at the outer proof boundary, retaining
sanitized first-error identity while refusing forged raw messages.

ADR: 0055
Record the independently verified pre-commit audit correction, bounded
proof changes and both completed review rounds against actual local
regression counters. Preserve historical evidence and keep exact-head CI
ownership separate from local results.

ADR: 0033, 0040, 0041, 0057
Record both fresh independent Step 1 reviews and the committed
cross-partition correction. Keep hosted fit and the remaining
execution and aggregate obligations explicit.

ADR: 0063
Extend the existing owned process lifecycle with complete selector
rediscovery and exact shard result reconciliation. Confine non-Docker
jobs to their process subtree, preserve native-only collector setup,
and record actual toolchain, phase and resource observations.

Handle only verified proc task-exit races narrowly and retain closed
stage diagnostics for other membership failures. Keep hosted fit and
workflow attestation pending the final implementation step.

ADR: 0062, 0063
Record independent approval and exact local control results without
claiming Linux containment or final-source hosted qualification.

ADR: 0062, 0063
Observe failed units while waiting for readiness instead of spending
the full execution allocation. Retain a bounded closed startup cause
and a thirty-second handshake cap within the existing job budget.
Prove no workload starts and owned cleanup still runs after refusal.

ADR: 0062, 0063
Record both independent Step 2 reviews and the verified startup fix
before beginning workflow and aggregate enforcement.

ADR: 0062, 0063
Keep every discovered case and mandatory build gate behind the existing
required results while bounding each execution and aggregate. Verify
current-attempt job outcomes and immutable archive identities before
recomputing exact result unions; retire unsharded execution paths.

ADR: 0062, 0063
Bind both required results to the same current-attempt immutable base
witness so contradictory trait partitions cannot omit cases. Treat
GitHub's run head as the event snapshot rather than its mutable PR
projection, and classify only the five new CI paths as non-runtime.

ADR: 0062, 0063
Keep the validated weekly image release in each producer receipt and
remove a child-start race from the interruption proof. Adapt the
read-only observation control to the approved aggregate topology while
preserving native-only observation and all production time budgets.

ADR: 0062, 0063
Keep only finite facade source codes and bounded line numbers for
validated failed cases, without exporting raw stacks or weakening
completion evidence. Record completed source reviews and the remaining
hosted failures separately from local passing controls.

ADR: 0062, 0063
Keep the independently verified diagnostic and completed source reviews
separate from the remaining real Linux failures and final A/B proof.

ADR: 0062, 0063
Locate repeated hosted failures through existing test-owned closed
stages and bounded timing/pool facts while keeping raw diagnostics
private. Preserve every test, ownership rule and execution budget.

ADR: 0062, 0063
Use the real compiled record output rather than an internal property
that generated ToString omits. Keep unexpected record members unknown
and preserve closed failure-only observation.

ADR: 0062, 0063
Record the actual-record correction and fresh second approval alongside
complete predecessor results. Keep both failing cases and final-source
A/B qualification open rather than treating diagnostics as a fix.

ADR: 0062, 0063
The hosted pause proof cannot distinguish an unentered heartbeat from a
provider call or cleanup that has not returned. Retain bounded in-memory
phase and count observations without granting them lifecycle authority,
and export only closed failure facts through the existing CI diagnostic.
Include the verified capacity-control source in the same finite map.

ADR: 0047, 0059, 0063
Keep both independent approvals and their actual control evidence distinct
from the still-unresolved hosted failures and final-source qualification.

ADR: 0063
Keep execution-focused fixtures independent of the bridge repair grace
when a native hint skips the ACK row lock. Preserve body failures during
explicit cleanup and observe physical opening separately so hosted
failures retain their original bounded diagnostics.

ADR: 0047, 0059, 0063
Apply the verified handoff prerequisite to execution-only capacity and
cleanup scenarios so their waits cannot accidentally test the separate
repair grace. Preserve the original assertions, bounds and bridge tests.

ADR: 0047, 0063
Record both independent rounds for the native diagnostic and adjacent
execution qualification corrections without attributing historical
hosted failures or closing the final-source A/B gate.

ADR: 0063
Use the guarded driver's asynchronous DNS and socket path behind the
stock synchronous factory. This avoids its verified positive-fraction
socket timeout truncation without abandoning actual acquisition or
weakening the physical role guard. Keep initializer awaits context-free
and prove path selection, actual completion and role refusal in Postgres.

ADR: 0047, 0058, 0059, 0063
Cover bootstrap's remaining synchronous acquisition at the same guarded
source entry as the vendor factory. Preserve authority before lazy source
resolution and join the real asynchronous acquisition without duplicating
adapter logic. Exercise both callers with the held initializer and prove
the old common driver call fails both planted controls.

ADR: 0047, 0058, 0059, 0063
Record both independent correction reviews and keep source-specific failed
receipts distinct from live final-source qualification. Point the primary
status carriers at the open PR for subsequent immutable A/B evidence so a
historical failed run does not become the current source's status.

ADR: 0063
Execution and cleanup subjects must not depend on an initial stock hint
acquiring an ACK-locked row before the accepted repair grace expires.
Qualify actual entry or committed handoff with one route-correct stock hint,
while preserving natural bridge, fairness and no-admission proofs.

ADR: 0047, 0059, 0063
Retain both independent approvals and source-specific validation before
final hosted qualification resumes on the reviewed execution fixtures.

ADR: 0063
Observe the owned application pool rather than immediate database-wide
session counts, and join every qualification hint before exact native
work snapshots. Preserve held callbacks and reject omitted factory cleanup.

ADR: 0047, 0059, 0063
Retain both independent reviews and actual restoration controls before
resuming final-source hosted qualification.

ADR: 0063
Expose unresolved capacity recovery without releasing its reservations,
exercise the production admission owner in checkpoint proofs, and anchor
published distribution checks to the reviewed checkout. Preserve fork CI
identity and closed refusal diagnostics; reconcile live job documentation.

ADR: 0047, 0058, 0063
Remove remaining live payload and activator authority descriptions found
by the first independent correction review. Record both reviews and their
verification without changing accepted decisions or historical evidence.

ADR: 0047, 0063
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant