Process-isolated ltsc2025 containers: LSM service start deadlocks in `CEventDispatcher::WaitForSessionState` on an event that is never signalled

[Kataglyphis]

Describe the bug

Process-isolated servercore:ltsc2025 containers on this Windows 11 26200 host
deadlock for ~141 s at service start. LSM blocks in
CEventDispatcher::WaitForSessionState on an auto-reset event that is never
set. Two dumps 30 s apart hold a byte-identical stack at 0.000 s thread CPU —
captured, not inferred.

The same build step, unchanged, across the transition:

when	identical RUN step
2026-08-30 17:29	3.441 s
2026-08-31 04:32	4.521 s
2026-09-01 11:36	144.7 s
2026-09-01 19:38	143.7 s
2026-09-02 00:24	143.9 s

• Requires ltsc2025 and process isolation and the servercore service
  set. Same host, same day: ltsc2022 process-isolated 1.8 s, ltsc2025 under
  Hyper-V 3.5 s, nanoserver ~2 s.
• A second, older defect on this host loses the container exit
  notification, so teardown runs to its timeout. On record here since
  2026-08-04 and filed as
  hcsshim#2855 on 2026-08-06.
  It escalated from filesystem-heavy containers to every container at the
  same time the start deadlock appeared. With the stock 30 s timeout the
  teardown is truncated and the scratch layer is corrupted
  (hcsshim::ExportLayer 0x3); with it raised, a trivial step costs 441 s.
• Full user-mode dumps of the deadlock available on request. Single host — no
  second machine here.

Happy to supply more. The hang reproduces on demand in about three minutes,
so I can run whatever would help inside it — further dumps, ETW traces, kernel
debugger output, extra experiments — and report back quickly. Just say what you
need.

---

A full debugging record follows — the wait stack and how it was captured,
the waited object and its holders, four measured configurations, ten falsified
hypotheses, the timeline of both symptoms, and scripts to reproduce the
diagnosis. It is long on purpose: it is meant to save you the experiments I
have already run. The ask itself is the final section, What would help.

Expected behavior

docker run --isolation=process servercore:ltsc2025 cmd /c echo hi starts,
prints and exits within a few seconds — as it did on this host until
2026-08-31, and as the identical image still does under Hyper-V isolation
today.

To Reproduce

docker run --rm --isolation=process mcr.microsoft.com/windows/servercore:ltsc2025 cmd /c echo hi

The command itself runs correctly and its output appears; the start costs
~141 s and the container then does not exit on its own.

Configuration:

• Edition: Windows 11 Pro 25H2, build 10.0.26200.9278, AMD Ryzen 9 9950X
• Base image: mcr.microsoft.com/windows/servercore:ltsc2025, digest sha256:eeaa17aefe5d949f03b1db17182f5855cf40e757533468cf5b50e07c7c385ada
• Image user-mode build: 10.0.26100.9278 — read from the dump: the silo's lsm.dll, ntdll.dll and KERNELBASE.dll
• Isolation: process
• Container engine: reproduces through both containerd 2.3.3 + BuildKit 0.32.2 and dockerd 29.7.2

Supported per the
version-compatibility documentation:
the Windows 11 client table marks Windows Server 2025 base images as supported
in process isolation from Windows 11 24H2 onwards.

This host runs a locally patched containerd-shim-runhcs-v1 making hcsshim's
hard-coded 30 s teardown timeout configurable
(microsoft/hcsshim#2855), set
to 5 min. It affects only the teardown wait. The ~141 s startup stall, the wait
stack and every control below are independent of it, and the stall reproduces
through stock dockerd with no patched shim involved.

Additional context

Symptom

• SCM logs 7022 The LSM service hung on starting at 140.07 s. LSM = Local
  Session Manager.
• LSM stays START_PENDING for the container's life, reporting
  NOT_STOPPABLE, IGNORES_SHUTDOWN. Of 121 services it is the only one neither
  RUNNING nor STOPPED.
• Boot continues once SCM times LSM out.
• The command itself then runs correctly; its output appears in docker logs.

Two independent measurements bracket the stall, and they agree:

SCM 7022 timeout, in-container event log	140.07 s
step start → container's first output (3 runs)	143.7 / 143.9 / 144.7 s
same step, same command, before the regression	4.521 s

144.7 − 140.07 ≈ 4.6 s, which is the healthy 4.5 s. The LSM timeout accounts
for the entire regression — nothing else in the boot got slower. "~141 s"
below refers to that timeout.

Wait stack

Two full user-mode dumps of the silo's svchost.exe -k DcomLaunch -p, 30 s
apart, byte-identical, waiting thread at 0.000 s user time:

ntdll!NtWaitForSingleObject+0x14
KERNELBASE!WaitForSingleObjectEx+0xaf
lsm!CEventDispatcher::CEventListEntry::WaitForSessionEvent+0x1e
lsm!CEventDispatcher::WaitForSessionState+0x285
lsm!CService::Start+0x8e6
lsm!ServiceMain+0xbd
svchost!ServiceStarter+0x3cd
sechost!ScSvcctrlThreadA+0x27

Reproduces on every container instance examined.

The waited object

From a separate live attach (handle data is absent from the dumps), read from
R10 of the thread whose stack shows lsm!CService::Start — the x64 syscall
stub does mov r10,rcx, so R10 still holds argument 1:

process           svchost.exe   (silo, pid 15900, thread 6)
waited handle     0x338
kernel object     0xffffe0831a1b57e0
  Type            Event
  Event Type      Auto Reset
  Event is        Waiting        <- for the container's whole life
  HandleCount     2
  PointerCount    65537

NtQuerySystemInformation(SystemExtendedHandleInformation) matched on that
object pointer attributes exactly one handle to any process: LSM itself. The
enumeration does see silo processes — it found LSM's handle that way.

Limits of that measurement: HandleCount is 2 and the enumeration accounts for
one. The second handle was not identified (LSM holding it twice is the simple
explanation, unproven). So no other process held a handle at snapshot time;
this does not prove nothing else can signal the event.

If you reproduce this: do not read the first KERNELBASE!WaitForSingleObjectEx
in the process — every service process has one, the SCM dispatcher's idle wait
in sechost!ScDispatcherLoop, and it names a different object. kb arg columns
are home-space reconstructions and are unreliable here. Bind the frame to the
thread, then read R10.

Controls

Same machine, same runtime, same day, same RUN echo step:

Configuration	Startup stall	Total step
servercore:ltsc2025, process isolation	~141 s	441.3 s
servercore:ltsc2022, process isolation	none	1.8 s
servercore:ltsc2025, Hyper-V isolation	none	3.5 s
nanoserver:ltsc2025, process isolation	none	~2 s

Startup stall is the defect; the 441.3 s total also contains the 300 s teardown
timeout, which only the failing row reaches.

The stall requires ltsc2025 and process isolation and the servercore
service set together. ltsc2022 — older user-mode on the identical 26200 kernel —
is unaffected, so process isolation as such is not broken here. Hyper-V gives
the image its own matching kernel. nanoserver has no LSM.

Hyper-V is a control, not a workaround: BuildKit 0.32 has no per-step isolation
option, and Hyper-V containers here are limited to a small fixed CPU count.

Falsified

Each with an explicit experiment:

• Host session broker — host LSM measured idle and ~20 s into a stall does
  nothing in between; the only differing frame is a COM GC worker, and no thread
  enters ContainerSessionServer::AskForSession. The container-side stack has
  zero RPC frames (rpcrt4, ContainerSessionClient, AskForSession). The
  container's LSM is blocked in its own dispatcher, not on the host.
• Something upstream in the silo boot — smss, csrss, wininit and
  services are PPL and refuse a read-only attach (Win32 error 0n5). Their
  descendants are reachable and idle-healthy during the stall: lsass in
  lsasrv!ServiceDispatcherThread, RPCSS in rpcss!CTime::Sleep,
  fontdrvhost in fontdrvhost!ServerRequestLoop — frames reached only after
  their parents completed. With 120 of 121 services settled, the silo boots
  normally around LSM.
• Windows Defender — reproduces with RTP off; platform 4.18.26070.9
  unchanged on disk since 2026-08-05; engine version constant across the working
  and failing periods.
• Container networking — CNI ADD succeeds with matching IP and gateway;
  --network none stalls identically.
• Windows Update inside the container — image committed with wuauserv and
  UsoSvc at Start=4 stalls identically.
• Disabling LSM — image committed with LSM Start=4 (verified with sc qc)
  removes the startup stall but keeps the teardown symptom.
• Disk — 515 GB free; disk event 51 appears only on successful
  teardowns.
• VBS/HVCI, hypervisor scheduler type, boot type — identical at the last
  working boot and every failing one.
• New KB or driver — the Setup event log has zero entries between
  2026-08-28 and 2026-09-01, and every driver binary predates the last healthy
  build. The one change to this build chain in that period — a base-image
  digest and three toolchain pins on 2026-08-26 — is bounded out under Onset.
• GPU — discrete GPU disabled (CM_PROB_DISABLED) throughout.

Onset

The two symptoms have different histories.

The lost exit notification is not new. It has been recorded on this host
since 2026-08-04, root-caused to host-side silo teardown on 2026-08-06 against
a measured 117 s worst case, and mitigated since then by a locally patched shim
raising tearDownTimeout from 30 s — filed upstream the same day as
hcsshim#2855. Through 2026-08-30 it tripped only on filesystem-heavy containers
(OpenCV, ONNX Runtime GenAI, TVM, GStreamer); trivial containers did not. The
mitigation was not continuous: Stevedore/containerd updates silently restore
the stock binary, and it was re-deployed on 2026-08-09, 2026-08-21 and
2026-08-28.

The service-start deadlock is new. No LSM stall is recorded on this host
before 2026-09-01; the dumps above are from 2026-09-02.

What changed around 2026-08-31. The teardown timeout began firing on every
container including RUN echo — a change of scope, not a first occurrence —
and container start acquired the ~141 s floor. The identical build step
measured 3.441 s (2026-08-30 17:29) and 4.521 s (2026-08-31 04:32), then
144.7 s (2026-09-01 11:36), 143.7 s and 143.9 s.

Bounds, and what is ruled out. A reboot at 2026-08-31 15:12:59 sits inside
the transition, but nothing was measured between 04:46:06 and 15:19:18 that
day, so the artifacts bound the change to that ~10.5 h window rather than
isolating the reboot as its trigger. The Setup event log has zero entries
between 2026-08-28 and 2026-09-01.

The last change to this build chain was 2026-08-26: the base image digest moved
d5bbb830 → eeaa17ae (the digest under Configuration above) along with
three toolchain pins. That is not the trigger — the chain rebuilt and ran on
eeaa17ae through 2026-08-31 03:23, and the 4.521 s healthy measurement above
is from that image at 04:32, five days after the bump.

What changed remains unidentified.

Teardown half

After the entrypoint exits — output present in docker logs, docker top
showing the process gone — the container stays Running until force-stopped;
the runtime waits for a shutdown notification that never arrives. docker stop
force-terminates in 91 s. Same shape as #547.

Shared root cause is unproven: disabling LSM does not fix the teardown. Both are
silo lifecycle transitions that fail to signal.

This half drives the wall-clock cost, since a build runtime pays it per step.
Measured on the same RUN echo: 2841.2 s at a 45 min timeout, 441.3 s at
5 min. hcsshim#2855 makes that wait configurable; it does not address why the
notification is lost.

Related reports

• Windows-Containers#547
  — same shape of teardown symptom, but not an unresolved case. Its reporter
  closed it on 2024-12-20 after it stopped reproducing on Docker Desktop 4.37.0;
  a Microsoft engineer could not reproduce it on 10.0.26100.2605; it had been
  narrowed to the IIS image. This report is base servercore, no IIS,
  current runtimes, with a wait stack.
• hcsshim#2566 — open and
  unanswered. Different symptom (PrepareLayer fails 0xc0370112; Hyper-V
  fails there too, unlike here), same pairing: Windows 11 25H2 / 26200 host,
  ltsc2025 / 26100 image, with "LTSC 2022 works fine" as the reporter's
  workaround — the same discriminator, on another machine.

Neither repository has an existing issue mentioning LSM or WaitForSessionState.

Artifacts and limitations

Observed on one host only; no second Windows machine is available here. The
evidence is a captured deadlock rather than a timing anomaly: two dumps 30 s
apart, byte-identical stack, 0.000 s thread CPU.

Available on request: two full user-mode dumps of the silo's
svchost.exe -k DcomLaunch taken 30 s apart during the hang (~30 MB each), and
the cdb transcripts behind every number above.

If reproduction fails on a build-matched host, please ask for the dumps rather
than closing.

Reproducing the diagnosis

Open source; needs an elevated shell and the WinDbg package (cdb.exe ships in
it):

• capture-lsm-waitstack.ps1
  — dumps the silo's svchosts twice, 30 s apart.
• find-lsm-event-holder.ps1
  — starts its own container, attaches non-invasively during the hang, resolves
  the waited object and its holders.

Two traps: silo processes report empty Win32_Process.ExecutablePath and
.CommandLine even to an elevated caller, so the silo must be found through the
process tree (fresh wininit.exe → its services.exe → their svchosts); and
dumps written by an elevated collector are SYSTEM-owned, so grant read access
before analysing them unelevated.

What would help

1. Symbolicate the wait. With private symbols for lsm.dll
10.0.26100.9278, lsm!CEventDispatcher::WaitForSessionState+0x285 should
   name the awaited session-state transition.
2. Name the signaller. What sets that event inside a silo, and when does it
   not run? No other process held a handle at snapshot time, which points at
   another thread in the same svchost or the kernel's session path — an
   inference from one snapshot, not a measurement.
3. Confirm whether the build pairing below is expected to work.

In process isolation the user-mode binaries come from the image and the kernel
from the host, so the deadlock sits across:

	build
blocked lsm.dll (from the image)	10.0.26100.9278
ntdll.dll / KERNELBASE.dll, same silo	10.0.26100.9278
host kernel and session components	10.0.26200.9278

A 26100 user-mode session stack on a 26200 kernel — same revision, different
base build. Documented as supported, and this exact pairing ran at ~4.5 s per
container here until 2026-08-31. Offered as a lead because the two
configurations that remove the
stall are exactly the two that remove this pairing: ltsc2022 replaces the
user-mode side, Hyper-V supplies a matching kernel. If recent servicing changed
a session-notification contract on the 26200 side, a 26100 lsm.dll waiting on
a state now reached differently would present exactly this way.

Untested here — it needs private symbols or a second machine.

[github-actions]

Thank you for creating an Issue. Please note that GitHub is not an official channel for Microsoft support requests. To create an official support request, please open a ticket here. Microsoft and the GitHub Community strive to provide a best effort in answering questions and supporting Issues on GitHub.