x86 scheduler: a thread already is_blocked() is picked and dispatched, then mandatorily switched away — 4188 of 4847 kernel-context restores on the #772 A0 arm

[ryanbreen]

Split out of #772 by ruling R116. #772's own oracle was retired by R113, and
candidate A's refusal was withdrawn by R115; this is the question the #772
diag battery actually surfaced, and it deserves its own issue and its own
oracle rather than riding on #772's.

The observation

On the x86 dispatch path, a thread that is blocked in a syscall is picked by
Scheduler::schedule(), fully dispatched — context restore plus a CR3 write —
and then switched away again at the next preemption on the mandatory arm,
because its state is still is_blocked().

Counter census, re-derived in this slot from the committed readouts under
docs/planning/green-program/sockets/serials/772-diag/ (39 of 40 boots carry
counters; sums over each arm's boots):

arm	kernel-context restores	of which end in a MANDATORY switch-away	identical-frame share that is mandatory
A0 (refusal off, logs on)	4847	4188 = 86.4%	2634 / 3109 = 84.7%
A1 (refusal on, logs on)	4231	4007 = 94.7%	2518 / 2579 = 97.6%
B0 (refusal off, logs off)	2099	1929 = 91.9%	1410 / 1540 = 91.6%
B1 (refusal on, logs off)	2260	2135 = 94.5%	1586 / 1678 = 94.5%

The partition has no remainder in 4 of 4 arms:
DISPATCH_SAVE_REASON_KERNEL_BLOCKED_PREEMPT + _MANDATORY equals
DISPATCH_KERNEL_RESTORE_TOTAL exactly on every arm (4847, 4231, 2099, 2260),
so every blocked-in-syscall dispatch ends in exactly one blocked-in-syscall
save and the mandatory/preempt split covers all of them.

Reproduce, with no host access:

python3 - <<'PY'
import json, glob, collections
per = collections.defaultdict(collections.Counter)
for f in sorted(glob.glob('docs/planning/green-program/sockets/serials/772-diag/*/boot_*/census.json')):
    arm = f.split('/')[-3]
    for k, v in (json.load(open(f)).get('counters') or {}).items():
        per[arm][k] += v
for arm in ('A0', 'A1', 'B0', 'B1'):
    p = per[arm]
    print(arm, p['DISPATCH_KERNEL_RESTORE_TOTAL_CPU0'],
          p['DISPATCH_SAVE_REASON_KERNEL_BLOCKED_MANDATORY_CPU0'],
          p['DISPATCH_SAVE_REASON_KERNEL_BLOCKED_PREEMPT_CPU0'])
PY

The mechanism, as far as it is established

docs/planning/green-program/sockets/772-DIAG-2026-09-03.md symbolised the two
addresses the restores land on, against the binaries the battery itself built:
enable_and_hlt's ret after sti; hlt (582 of 769 restores in the A0
specimen boot) and the jmp after call interrupts::enable inside
without_interrupts (141 of 769). Both are park points of the
blocked-in-syscall wait loops — e.g. sys_waitpid's
(kernel/src/syscall/handlers.rs:3459-3498): check signals, yield_current(),
halt, re-read, loop. A thread dispatched there re-checks its condition, finds it
still unmet, halts again, and is switched away with its state still blocked.

So the dispatch buys a save, a restore and a CR3 write to execute one turn of a
poll loop that the wake would have driven anyway.

Where the pick is

Scheduler::schedule()'s selection loop (kernel/src/task/scheduler.rs:1875
onward, read in this slot): the local-queue arm at :1877-1893 pops a
candidate, reads (thread.state == ThreadState::Terminated, thread.owner_pid)
and skips only on terminated; the work-steal arm below it (:1895 onward)
applies the same predicate plus its aarch64-only affinity filters. Nothing in
either arm consults state.is_blocked().

The requeue block just above (:1820-1868) is the other half: it computes
is_blocked and refuses to enqueue a blocked thread (will_add = !is_terminated && !is_blocked && !in_queue), and it documents the duplicate-entry case that
can leave a thread already sitting in per_cpu_queues.

#647 has since landed (closed 2026-08-27): block_current/block_current_in_syscall
now perform the departure unconditionally — for q in self.per_cpu_queues.iter_mut() { q.retain(|&id| id != current_id) } at kernel/src/task/scheduler.rs:2803-2805 —
so "the primitive published Blocked and left the thread queued" is no longer an
available explanation. Whatever puts a blocked thread in front of the picker
today is something else: a thread enqueued Ready and then blocked before it was
popped, a wake that enqueues while the wait loop re-parks, or the duplicate-entry
case the requeue block names. That is part of what the first step has to answer.

What is NOT established

The counters above measure the state at the switch-away, not at the pick.
A thread that was Ready when picked, ran, blocked itself and was then switched
away lands in exactly the same counter. The halt-point symbolisation and the
identical-frame census make the "already blocked at pick" reading the
parsimonious one — the wait loops do not leave the blocked state to poll — but
this battery has no counter at the pick, so the reading is inferred rather than
measured.

First step, before any fix: a counter at the selection loop that counts,
per boot, how many picks return a thread whose state.is_blocked() is already
true (and, ideally, whether it came from the local queue or a steal). That is a
few lines beside the terminated read at :1881-1886, it is on the scheduler
lock rather than the interrupt-return hot path, and it makes the claim in this
issue's title either true or false rather than plausible.

Disclosed risk on any fix

Skipping blocked threads at the pick is not free. The post-halt re-check inside
each wait loop is a poll, and that poll has historically masked lost wakes: the
#568 comment at kernel/src/task/scheduler.rs:1790-1817 documents a wake that
was published Ready, had its timer-heap entry consumed, and was never enqueued —
recoverable only because something ran the thread again. A change that stops
dispatching blocked threads removes exactly that safety net, so it is a
soak-gated change (100+ boots), not a single-green-run change.

Related

• x86: a dispatched thread with a published wake returns the CPU without consuming it, 5-6 times per specimen #772 — the issue this was split out of. Its identical-RIP/RSP proxy is
  retired as an oracle (R113); the counters, the per-CPU dispatch mark, the
  reason tags and the census tooling land as instrumentation (R115).
• x86: a timer wake dispatches only after a full round robin - measured sleep overrun p90 2592 ms, max 10318 ms over 324 trials #766 — the timer-wake round-robin latency this dispatch cost sits inside.
• x86: context-switch path logs three formatted serial records per blocked-in-syscall dispatch with interrupts masked (CLAUDE.md interrupt-path violation; amplifier under #766) #775 — the three formatted serial records this path writes with interrupts
  masked. Silencing them halves the per-boot restore volume (484.7 to 209.9
  restores/boot) without changing the ratio, so it reduces the cost of this
  phenomenon without addressing it.
• Scheduler::block_current publishes Blocked but leaves the thread on the ready queue — all 14 callers open-code the rest #647 — closed; its landed resolution is why the obvious explanation above is
  ruled out rather than assumed.

[ryanbreen]

R113 round-2 direct oracle: the ZERO_ITER x KERNEL_BLOCKED_MANDATORY count, and why it still does not answer this issue's question

Branch exp/772-r113-oracle (measure 772-R113-measure-r2-2026-09-04.md,
oracle 772-R113-ORACLE-2026-09-04.md).

Restricted to KERNEL_BLOCKED_MANDATORY — the reason this issue's own
observation table keys on — the direct oracle's ZERO_ITER count is 500 of
865 identical-frame saves on the logs-on arm (L1, testing,external_test_bins,
3 boots) and 271 of 451 on the logs-off arm (L0, +quiet_dispatch_log, 3
boots): about 167 and 90 per boot.

That number does not settle this issue's question, and the reason is
structural rather than a matter of more boots. This issue's own "What is
NOT established" section says it plainly: "the counters above measure the
state at the switch-away, not at the pick," and asks for "a counter at
the selection loop that counts, per boot, how many picks return a thread
whose state.is_blocked() is already true" as the first step before any
fix. R113's ZERO_ITER/REVISIT split is a different measurement, taken at a
different point: note_dispatch_save
(kernel/src/interrupts/context_switch.rs:600 onward) classifies each
identical-frame observation at the save that follows a dispatch, by
comparing the park counter's value against the stamp set_dispatch_mark
recorded at the previous save — it has no visibility into
Scheduler::schedule()'s selection loop and cannot distinguish "picked
already-blocked" from "picked Ready, ran, blocked itself, then was switched
away" (this issue's own caveat about its switch-away counters, unchanged by
R113).

So the 500/865 and 271/451 figures narrow the ZERO_ITER share within the
reason this issue is about, but the counter this issue's "first step" asks
for — at the pick, not the save — is still unbuilt. That remains the
next instrument for this issue, not something R113 substitutes for.

This is a measurement note, not a landing and not a claim that this issue's
first-step counter is unnecessary.