aarch64: fix/562-761-aarch64-testing-profile breaks the strict boot-test gate (0/25 vs main 15/15)

[ryanbreen]

Signature

docker/qemu/run-aarch64-boot-test-strict.sh 25 (soft-float, --features boot_tests, -cpu cortex-a72) is 0/25 (0%) on
fix/562-761-aarch64-testing-profile at 88199ba4 (round-6 head
95de851e plus a docs-only commit; kernel/src identical to 95de851e).

Controlled comparison, same host, same script, back to back, same day
(2026-09-04):

kernel	result
origin/main @ 78179c56, --features boot_tests	15/15 (100%)
this branch @ 95de851e/88199ba4, same features	0/25 (0%)

Both kernels were built from a from-scratch userspace/programs/build.sh --arch aarch64 + scripts/create_ext2_disk.sh --arch aarch64 in their own
worktree, so this is not a stale-fixture artifact.

Failure classes observed (branch only; 0/15 main-baseline boots show any of these six)

Of 25 branch boots:

• 12/25 -- [TEST:filesystem:block_wedge_oracle:FAIL:...], three
  distinct messages: "second lock was not sleep-permitted" (5),
  "parked lock was not sleep-permitted" (6), "requester never parked on
  the real wait queue" (1).
• 9/25 -- a CPU exception, and it is the same one every time:
  [INSTRUCTION_ABORT] FAR=0x40004530 ELR=0x40004530 ESR=0x8200000e IFSC=0xe TTBR0=0x40200000 from_el0=1. Bit-identical FAR/ELR/ESR/TTBR0
  across all 9 occurrences.
• 1/25 -- [TEST:process:census_widen_oracle:FAIL:census widening mutation oracle failed].
• 1/25 -- "Futex handoff oracle marker missing or failed".
• 1/25 -- "Userspace not detected" (109/109 boot_tests suite completed
  with BOOT_TESTS:PASS, but the poll window closed before userspace
  liveness printed).
• 1/25 -- "Exec smoke did not complete".

0 of these six match the pre-adjudicated aarch64 gate-noise set
(#555 softirq, #576 EL1 NULL-PC, #626, #586 starved wake-loss, #609
network kthread). #576's own signature is ELR=0, FAR=0; this branch's
instruction abort is ELR=FAR=0x40004530 in 9/9 occurrences (grep -c
against each of the 9 CPU exception-classified serials in
S-battery-all25/) -- a fixed, different address from #576's, so a
distinct fault.

#599 ("run-aarch64-boot-test-strict.sh is red on main (15-20/20)") was
filed 2026-08-18 against an older main revision and describes a different
shape (a silent wedge after [EXEC_SMOKE:LAUNCH], no fault marker). Main
has since recovered: the 15/15 clean run above is current-day evidence that
whatever #599 was measuring no longer reproduces at HEAD. This branch's
0/25 is not a restatement of #599.

Why this is the branch's regression, not host noise

The block_wedge_oracle and instruction-abort classes are content-specific
(a named test failing with a fixed message; a fault at a byte-identical
address 9/9 times), not the kind of thing host CPU contention produces.
fix/562-761-aarch64-testing-profile's own change is exactly the kind of
thing that could break block_wedge_oracle's timing assumptions: it moves
CPU 0's boot sequence off a preemption-pinned context and onto an ordinary,
preemptible kernel thread (docs/planning/green-program/aarch64-testing/TESTING-PROFILE-REVIVAL-2026-09-04.md,
"One relocation, widened at round 3"). block_wedge_oracle exercises the
same virtio-blk wait/wake path that pin removal changes the scheduling
context of.

Evidence

The full run log (25 boots) is
docs/planning/green-program/aarch64-testing/serials/prove/S-battery-run-log.txt;
the main-baseline run log (15 boots) is
docs/planning/green-program/aarch64-testing/serials/prove/S-main-baseline-run-log.txt;
all 25 branch serials are the 25 files under
docs/planning/green-program/aarch64-testing/serials/prove/S-battery-all25/.

Scope

Filed during an independent PROVE pass of
fix/562-761-aarch64-testing-profile round 6
(docs/planning/green-program/aarch64-testing/TESTING-PROFILE-PROVE-2026-09-04.md).
Not fixed here -- the PROVE task is scoped to running the batteries and
reporting; RCA needs the project's GDB-based kernel-debugging workflow
against context_switch.rs/block_mmio.rs, which is Tier-2 territory.
This is a blocking finding for that branch: the strict gate cannot be
used to accept the merge while it is 0/25.

[ryanbreen]

Prod-profile gate (run-aarch64-prod-profile-boot-test.sh) also degrades on this branch

Same round-6 PROVE pass, same host, 25 sequential invocations against the
shipped no---features production kernel (claim-lint:ok: the table right
below gives the 8/8 vs 13/25 counts this paragraph is introducing):

kernel	result
origin/main @ 78179c56	8/8 (100%)
this branch @ 95de851e/88199ba4	13/25 (52%)

12/25 branch failures, all "the boot progressed but a downstream marker
never appeared inside the script's 120 s poll window", none a crash marker:

• 10/12 -- FAIL: seam-absent timeout marker count must be exactly one
  (the FUTEX_HANDOFF_ORACLE_DRIVER:seam_absent:probe=-110 marker missing
  in 9 of those 10, and present but reading probe=-3 instead of -110
  in the 10th, boot 8 of P-battery-all25/).
• 1/12 -- FAIL: TTY oracle marker missing.
• 1/12 -- FAIL: Poll TCP oracle marker missing.

0/8 main-baseline boots show any of these three. This is a softer, timeout-
shaped symptom (the boot is not crashing, just not keeping pace with markers
the script expects on a fixed clock) rather than the S-battery's hard
failures, but it is the same branch, the same day, the same host, and main
does not reproduce it either -- consistent with, not proven identical to,
the same root cause named above (CPU 0's boot sequence moving off a
preemption pin and onto a schedulable, contended context).

Evidence: docs/planning/green-program/aarch64-testing/serials/prove/P-battery-run-log.txt
(25 boots), P-main-baseline-run-log.txt (8 boots), and the per-boot
serials under P-battery-all25/ and P-main-baseline/.

[ryanbreen]

Round 7: both classes repaired at the producer, plus two more that were hiding behind them

Branch head 7d0ac41f (6 commits on e3dc4040). Full write-up:
docs/planning/green-program/aarch64-testing/786-RCA-2026-09-04.md.

1. INSTRUCTION_ABORT (9/25) — a stale next_cr3 applied to a userspace ERET

0x40004530 is init::main+0x24 in the userspace ELF: the return address of
init's first svc. ESR 0x8200000e is a level-2 permission fault — the page
is mapped under the kernel root's identity map but is not EL0-executable. Init
returned from its first syscall onto the kernel page table.

Two per-CPU words decide which root a return to EL0 runs on, and the syscall
return corridor in syscall_entry.S reads both: next_cr3 (offset 64) first,
installing it when set; saved_process_cr3 (offset 80) as the fallback. Every
idle dispatch runs setup_idle_return_locked, which stores
next_cr3 = kernel_root — and nothing on the idle return corridor consumes that
store, so it sits armed.
claim-lint:ok: 7 of 7 #64-relative reads in boot.S are frame-relative, per
the round-6 RCA on this issue.

launch_init_from_elf then installed init's root with a raw msr ttbr0_el1 and
reconciled neither word. The ERET to EL0 worked, because the register was
right. Init's first svc entry stub saved the real TTBR0 into offset 80, the
return stub read the still-armed offset 64, installed the kernel root, and
ERETed — abort at init's own return address. Init is the one thread that reaches
EL0 without passing through dispatch_thread_locked, which is why no other
userspace thread hit it.
claim-lint:ok: 9 of 25 round-6 boots, bit-identical.

Fix: ttbr0.rs grows adopt_process_ttbr0() — the barrier/msr/TLBI
sequence, then set_saved_process_cr3(value) and set_next_cr3(0). Every
reachable process-root install goes through it now, not just init's: exec (which
published the saved root but left next_cr3 armed), both signal-delivery paths,
and the four ensure_current_address_space copies, one of which had no TLB
invalidation at all.
claim-lint:ok: 7 of 7 censused install functions, listed in
docs/planning/green-program/aarch64-testing/786-RCA-2026-09-04.md.

Not fixed, disclosed: kernel/src/syscall/time.rs::ensure_current_address_space
is a fifth copy of the same shape. syscall/time.rs is on CLAUDE.md's Tier-1
prohibited list, so it needs explicit approval. The new census ratchet prints it
on each run rather than hiding it (1 of the 7 censused functions):

TTBR0 installs still unreconciled behind the Tier-1 rule: ["kernel/src/syscall/time.rs::ensure_current_address_space"]

2. block_wedge_oracle (12/25) — a contended try_lock became a permanent verdict

idle_identity_must_not_sleep() refused on the None arm of
is_current_idle_thread(), which returns None only when
try_lock_scheduler() fails. On SMP that is ordinary momentary contention and
says nothing about who is running. Round 6's own probe builds measured it:
trylock_fail=1 idle_refused=0 cur_id=39|53 in 4 of 4 legible lines — the
identity a few instructions later was a kernel thread, not idle.

Fix: the arm returns false and counts into a new
IDLE_IDENTITY_UNREADABLE, kept separate from IDLE_SLEEP_REFUSED because an
unreadable identity is not an observation of the idle task reaching for a
blocking primitive. This does not let the idle identity block:
Scheduler::refuse_idle_block reads the identity exactly, inside the scheduler
lock, and all 6 blocking primitives that publish the caller's blocked state call
it — including block_current_for_io_publish, behind the waitqueue path this
predicate also guards. The lock-free predicate is an advisory fast path.
claim-lint:ok: 6 of 6 self.refuse_idle_block() call sites.

3. The boot-continuation strand — surfaced by fixing the first two

With boots running past the abort and the wedge, 2 of the next 3 failed on:

[SCHED_STRAND_FIRST:tid=11:shape=running:priv=kernel:state=1:dwell_ms=2027]

tid 11 is kboot. spawn_as_current's add_thread_as_current overwrites
cpu_state[cpu].current_thread and never touches the identity it displaces, so
kboot stayed Running forever on no CPU, in no queue, in no handoff slot —
the strand census's own definition — and, never Terminated, its kernel stack
could never be reclaimed. That same stale identity is why the abort reported
against the wrong thread: both preserved abort serials print
[INSTRUCTION_ABORT] deferred_tid=11 queued=1.
claim-lint:ok: 2 of 3 boots,
serials/r7/strict-prefix-boot-continuation-strand.txt.

spawn_as_current_retiring_outgoing retires the displaced identity in the same
scheduler critical section (skipping the idle identity, still its CPU's
fallback), and kthread::publish_current_kthread_exit does the rest of the
kthread ending. Safe while still on the retired stack: reclamation waits on
retirement_grace_target, whose fence requires a second scheduler entry on every
online CPU including this one, and this CPU is masked inside the handoff until
the ERET.
claim-lint:ok: retirement_grace_target / note_scheduling_epoch in
kernel/src/task/scheduler.rs.

4. census_widen_oracle (1/25) — a field that could not see the CPU it was about

The oracle requires cpu_silence_ms >= 1 and arms arm_target=4, a CPU past
online_cpu_count (4 online: 0..3). collect_strand_census sampled scheduler
silence over 0..online_cpu_count, so CPU 4's silence could never be measured;
the field reported the four busy CPUs, which is 0 whenever all four enter the
scheduler inside one tick. Observed across this session: 0, 1, 2, 2, 2, 3, 4, 6,
7 ms — a coin flip. The scan now covers every CPU slot and skips only a CPU that
is both past the online count and holding no queued work. The reading became
cpu_silence_ms=1192 matching queued_nondispatch_ms=1192, worst_silence_cpu=4.
This code is on main too, so the repair is not branch-local.
claim-lint:ok: 1 of 3 boots failed on it; 9 readings enumerated above.

tests/strand_handoff_structure.rs had pinned the old bound. That pin was the
defect, so it is retargeted, not removed.

R146 smoke at 7d0ac41f, 3 boots per profile

Profile	Verdicts
strict (boot_tests)	[OK] Boot 1: SUCCESS / [OK] Boot 2: SUCCESS / [OK] Boot 3: SUCCESS — PASS: 3/3 boots succeeded
production (no features)	PASS: production profile reached bsshd with the futex oracle seam absent ×3
testing (--features testing)	[PASS] marker=1 loaded=1 instruction_abort=0 wedge_fail=0 strand_first=0 panic=0 ×3

Builds: 0 warnings beyond the toolchain's core v0.0.0 notice on all three
aarch64 profiles and on x86 testing,external_test_bins (beast, exit 0, 0
warnings); check-kernel-no-neon.sh PASS: 0 FP/SIMD load/store instructions on
each artifact. 26 of 26 tests/*_structure.rs suites pass.
claim-lint:ok: 3 of 3 boots per profile; 26 of 26 suites.

What the smoke does not close

15 strict boots were run at this head across four invocations; 5 failed, and the
2 that remain after the §3 and §4 repairs are timeouts of the gate's fixed 18 s
poll window on boots that were progressing normally, not kernel failures:

• the exec-smoke cutoff's guest clock advanced 6.4 s inside a 20 s wall
  window (0.32× real time), against 44.9 s in 45 s (1.0×) for an
  unconstrained boot of the same kernel;
• the other reached [TESTS_COMPLETE:109/109], [BOOT_TESTS:PASS],
  [init] Breenix init starting (PID 1) and [spawn] path='/bin/heartbeat'
  before the window closed.

The strict gate runs -cpu cortex-a72, which forces TCG on this ARM host, so
guest speed tracks host load — load average was 4.2 during the failures and 2.6
during the 3/3 smoke. Round 6 recorded both classes at branch HEAD as well
(Exec smoke did not complete 1/25, Userspace not detected 1/25), so they
predate this round. The init service sequence needs ~9 s of guest time before
exec_smoke against an 18 s wall budget.

grep -c "INSTRUCTION_ABORT" and grep -c "block_wedge_oracle:FAIL" both return
0 on each of the 5 preserved failing serials. Serials under
docs/planning/green-program/aarch64-testing/serials/r7/.

The production-profile degradation in the comment above was not separately
re-measured at volume: 3 of 3 invocations passed here, which is too small a
sample to speak to the 13/25 figure.

[ryanbreen]

Status update (coordinator ruling R151, 2026-09-04): fixed on the parked branch fix/562-761-aarch64-testing-profile at 7d0ac41f -- round-7 RCA + fix for both classes this issue tracks: the INSTRUCTION_ABORT stale-next_cr3 TTBR0-shadow desync, and the block_wedge_oracle false refusal on a contended try_lock read of the idle identity. Full RCA is the comment above on this issue; the branch also carries docs/planning/green-program/aarch64-testing/786-RCA-2026-09-04.md. That branch is parked, not merged, its fix content landing through 2b3fb187 (see the status comments on #562/#761).

The fix is being re-sliced onto main rather than landing as one branch:

• Slice 1 (in flight): the TTBR0 half. Branch fix/ttbr0-shadow-reconciliation ports adopt_process_ttbr0() -- reconciling next_cr3/saved_process_cr3 at every process-root install -- to 9 of the 10 censused install sites; the 10th, kernel/src/syscall/time.rs::ensure_current_address_space, is Tier-1-prohibited under CLAUDE.md and is disclosed rather than fixed. Detail: docs/planning/green-program/aarch64-testing/TTBR0-SHADOW-SLICE-2026-09-04.md on that branch. That doc's own measurement (section 3) found the trigger latent on unmodified main today -- next_cr3 read 0x0 at init's ERET in 5/5 probed boots -- so this slice is defensive on main, not a red-to-green fix there; its proof of consequence is the parked branch's own A/B/A mutation battery (13/26 -> 0/24 -> 4/8 on reversion), cited by path in that same doc.
  claim-lint:ok: 9 of 10 sites, 5/5 probe boots, 13/26-0/24-4/8 A/B/A -- all cited above by path, per TTBR0-SHADOW-SLICE-2026-09-04.md.
• The idle-refusal half waits. The block_wedge_oracle fix (is_current_idle_thread returning false instead of refusing, on the arm where try_lock_scheduler() comes back empty) is not extracted into a slice yet. It sits behind aarch64 production profile: init service sequence goes silent on fix/562-761 (3/8 vs main 8/8) #789, which found that same function's lock acquisition is also the mechanism of a separate, worse defect -- an unmasked SCHEDULER lock taken from thread context that self-deadlocks under an interrupt. Both get fixed together, at the root, when aarch64 production profile: init service sequence goes silent on fix/562-761 (3/8 vs main 8/8) #789's fix lands.
  claim-lint:ok: see aarch64 production profile: init service sequence goes silent on fix/562-761 (3/8 vs main 8/8) #789 (RCA comment posted on that issue).

[ryanbreen]

Slice 1 landed: PR #795, merge commit bdb5be9071c838481890f4333860f7b4ff27a406.

This ports the adopt_process_ttbr0 discipline and routes 9 of the 10
process-root install decision sites through it, plus the graphics.rs TLBI
fix and the process_memory.rs shadow reconciliation. Full detail, review
rounds and evidence paths: docs/planning/green-program/aarch64-testing/TTBR0-SHADOW-SLICE-2026-09-04.md.

This is slice 1 of the re-slicing; #786 stays OPEN until the idle-refusal
slice lands. Slice 1b (the Tier-1 kernel/src/syscall/time.rs::ensure_current_address_space
site) follows.