test/dst-rt: pin libumem 4.3.2 and run s3-umem under UMEM_DEBUG=default

test/dst-rt/FINDINGS.md

@@ -122,7 +122,9 @@ tripping the buffered-region writes.  `s1-preload-directio` PASSES.
 
 ### F1-002 (tool note, not a libdb bug): libumem `UMEM_DEBUG=default` aborts on the dev box
 
-**Status:** tool limitation, documented; the harness works around it.
+**Status:** fixed upstream in libumem v4.3.2 (`60a6452`; fix `490c943`,
+regression test `4abe0f6`). `install.sh` pins it, and the s3-umem arm uses
+`default` again. See "Fixed in libumem 4.3.2" after F1-003.
 
 On the dev box (Debian 12, x86_64, libumem built from the pinned commit),
 `UMEM_DEBUG=default` (which includes the `audit` per-buffer stack-capture
@@ -151,6 +153,8 @@ fill-pattern violation.  rtcheck verifies the data afterwards.
 
 ### F1-003 (tool, not a libdb bug): libumem `UMEM_DEBUG=default` HANGS on the GitHub Ubuntu 26.04 runner
 
+**Status:** fixed upstream in libumem v4.3.2, as F1-002.
+
 Where F1-002 saw `default` abort on the dev box, the hosted runner (kernel
 7.0.0-1017-azure, 4 vCPU, libumem fe9ff961 with its rseq fast path) **hangs**:
 a trivial one-`malloc` program under `LD_PRELOAD=libumem_malloc.so
@@ -175,9 +179,47 @@ in the workload is a FAIL verdict, never a silently cancelled job.  FAINT is
 bounded the same way (`S3_FAINT_TIMEOUT`=120 s); it was **not** the hang (it
 finishes in < 1 s on the runner, 4 injection positions, 0 crashes).
 
-Report upstream to libumem: `UMEM_DEBUG=default` (the `audit` component) both
-aborts (dev box, Debian 12 kernel 6.1) and hangs (Ubuntu 26.04 kernel 7.0) on a
-trivial program.
+Reported upstream to libumem on 2026-11-04: `UMEM_DEBUG=default` (the `audit`
+component) both aborts (dev box, Debian 12 kernel 6.1) and hangs (Ubuntu 26.04
+kernel 7.0) on a trivial program.
+
+### Fixed in libumem 4.3.2
+
+**Cause (libumem's diagnosis, matching ours).** With `audit` on, libumem
+records a stack for every vmem segment from inside `vmem_hash_insert()`, while
+it holds vmem locks. The first stack walk on each thread asked glibc
+`pthread_getattr_np()` for the stack bounds, and that call allocates. Under
+`LD_PRELOAD` the allocation is libumem's own, so the allocator re-entered
+itself. It then either failed the `VMC_POPULATOR` assertion (F1-002) or
+deadlocked its update thread against the main thread (F1-003), depending on
+rseq and timing.
+
+**Fix.** libumem 4.3.2 reads the stack bounds from `/proc/self/maps` with plain
+`open`/`read` into a stack buffer, so nothing on that path allocates. The fix
+is `490c943`, its regression test is `test/security/test_audit_reentry.sh`
+(`4abe0f6`), and the release is `v4.3.2` (`60a6452`). The ABI is unchanged.
+
+**Verified on the dev box** (Debian 12, x86_64, 32 vCPU, 2026-11-05). Each
+library was first shown live, by catching a planted 64-into-16 overflow under
+`guards,contents`. The programs are the one-`malloc` reproducer and a
+first-`malloc`-on-a-new-thread variant, 25 runs per cell:
+
+| `UMEM_DEBUG` / `UMEM_OPTIONS` | `fe9ff96` (old pin) | `60a6452` (v4.3.2) |
+|---|---|---|
+| `audit` | 25/25 abort | 25/25 exit 0 |
+| `audit` + `norseq` | 25/25 hang | 25/25 exit 0 |
+| `default` | 25/25 abort | 25/25 exit 0 |
+| `default` + `norseq` | 25/25 hang | 25/25 exit 0 |
+
+The s3 scenario against libdb `master` with 4.3.2 and `UMEM_DEBUG=default`:
+s3-umem is clean (no heap error, data verified) and s3-umem-sabotage catches
+the planted overflow, on seeds 1-5. `guards,contents` gives the same verdicts.
+
+**Harness change.** `install.sh` pins `LIBUMEM_PIN` to `60a6452`, and s3 uses
+`UMEM_DEBUG=default` again, so the audit allocation and free stacks are back in
+its reports. Every libumem run stays under `timeout -k`, because the SIGKILL
+fallback is what ends a hung `LD_PRELOAD` process. `S3_UMEM_MODE=guards,contents`
+still selects the old mode.
 
 # Part 2 — multi-node HA and XA (F2, scenarios S4–S8)
 

test/dst-rt/README.md

@@ -59,10 +59,10 @@ Gotchas the install encodes (so you do not rediscover them):
   `fault_inject32.so` (needs the 32-bit multilib). We build the 64-bit objects,
   stub the 32-bit blob, and hand-link; `map_c.o` must be compiled with `gcc`
   (as C++ the `_map_init`/`map` symbols `faint.c` references get mangled).
-- **libumem `UMEM_DEBUG=default` aborts** any program at startup on the dev box
-  (the `audit` component); S3 falls back to `UMEM_DEBUG=guards,contents`, the
-  detection half, which runs clean and still catches overflow + UAF. See
-  FINDINGS F1-002.
+- **libumem must be 4.3.2 or later.** Before it, `UMEM_DEBUG=default` (its
+  `audit` component) aborted any program at startup on the dev box and hung on
+  the Ubuntu 26.04 runner, so S3 used `UMEM_DEBUG=guards,contents`. install.sh
+  pins 4.3.2, and S3 uses `default` again. See FINDINGS F1-002 and F1-003.
 
 ## The oracle (`rtcheck`) and the ACK log
 
@@ -183,4 +183,5 @@ Real bugs a scenario finds are issue-style write-ups in
 [`FINDINGS.md`](FINDINGS.md), each with a reproducer. Engine fixes go on their
 own branches from `origin/master`, never on `test/dst-rt`. F1-001 (libdb cannot
 open under a *blanket* O_DIRECT; the native direct path is clean) and F1-002
-(libumem `audit` mode aborts on the dev box) are recorded there.
+(libumem `audit` mode aborted on the dev box; fixed in libumem 4.3.2) are
+recorded there.

test/dst-rt/scenarios/s3_alloc_failure.sh

@@ -114,25 +114,16 @@ else
 	echo "    SKIP: libfiu unavailable"; hi_emit s3-fiu-malloc skip
 fi
 
-# ---- s3-umem: guard redzones + contents (fill-pattern) over the workload --
+# ---- s3-umem: guard redzones, fill pattern and audit stacks over the workload
 echo "== s3-umem =="
 if fi_have umem; then
-	# UMEM_DEBUG mode: the brief asks for `default` (guards + audit).  On some
-	# hosts the `audit` component (per-buffer stack capture) aborts even a
-	# trivial program at startup, with no detection report -- a libumem/host
-	# limitation, NOT a libdb heap bug (a 1-malloc program aborts too).  Probe
-	# `default` with a trivial program; if it aborts spuriously, fall back to
-	# `guards,contents`, which is the DETECTION half of default (redzone +
-	# fill-pattern): it still catches an overflow and a use-after-free (the
-	# sabotage arm proves this), just without the audit backtraces.
-	# UMEM_DEBUG mode: guards,contents, the detection half of `default`
-	# (redzones + fill pattern).  It catches the planted overflow (the
-	# sabotage arm below) and runs the libdb workload clean.  `default` adds
-	# the audit component, which is not usable here: on the dev box it aborts
-	# a one-malloc program (F1-002), and on the GitHub Ubuntu 26.04 runner it
-	# HANGS intermittently (F1-003) -- a probe passed and the workload then
-	# hung, so no probe can vouch for it.  S3_UMEM_MODE overrides.
-	UMODE=${S3_UMEM_MODE:-guards,contents}
+	# UMEM_DEBUG mode: `default` (redzones, fill pattern, and audit
+	# allocation/free stacks).  Before libumem 4.3.2 the audit component
+	# re-entered the allocator under LD_PRELOAD, so `default` aborted on the
+	# dev box (F1-002) and hung on the Ubuntu 26.04 runner (F1-003), and this
+	# arm used guards,contents.  install.sh pins 4.3.2, which fixes that.
+	# S3_UMEM_MODE overrides; guards,contents still works.
+	UMODE=${S3_UMEM_MODE:-default}
 
 	_home="$RUN/umem/home"; mkdir -p "$_home"
 	# Every libumem run is bounded: on the Ubuntu 26.04 runner this arm hung
@@ -184,7 +175,7 @@ CEOF
 	"${CC:-cc}" -O0 "$RUN/sab_oob.c" -o "$RUN/sab_oob" 2>/dev/null
 	# Use the SAME mode the s3-umem arm settled on, so the teeth match the
 	# detection actually in force.
-	UMODE=${UMODE:-guards,contents}
+	UMODE=${UMODE:-default}
 	sout=$(timeout -k 5 30 env LD_PRELOAD="$UMEM_SO" UMEM_DEBUG="$UMODE" \
 	    UMEM_OPTIONS=abort "$RUN/sab_oob" 2>&1)
 	src2=$?

test/dst-rt/tools/install.sh

@@ -36,7 +36,7 @@ SKIP_APT=${SKIP_APT:-0}
 LIBDIRECTIO_REPO=https://github.com/gburd/libdirectio
 LIBDIRECTIO_PIN=67e57c3064f2d56273376d41bb759320c9e93110
 LIBUMEM_REPO=https://codeberg.org/gregburd/libumem
-LIBUMEM_PIN=fe9ff96114fbcbf63f89fefec6bc9585efb445f0
+LIBUMEM_PIN=60a64527e8710d955062d0c9f5fd44ce335fa7c1	# tag v4.3.2: audit re-entrancy fix (F1-002/F1-003)
 FAINT_REPO=https://github.com/misc0110/faint
 FAINT_PIN=b24cf2d573d245a3eb53fe47ffe3296355c09676
 # libfiu: apt where packaged (Debian 12: 1.1); Ubuntu 26.04 dropped it, so