diff --git a/test/dst-rt/FINDINGS.md b/test/dst-rt/FINDINGS.md index 22d3a6cb58..5801ec09df 100644 --- a/test/dst-rt/FINDINGS.md +++ b/test/dst-rt/FINDINGS.md @@ -122,7 +122,9 @@ tripping the buffered-region writes. `s1-preload-directio` PASSES. ### F1-002 (tool note, not a libdb bug): libumem `UMEM_DEBUG=default` aborts on the dev box -**Status:** tool limitation, documented; the harness works around it. +**Status:** fixed upstream in libumem v4.3.2 (`60a6452`; fix `490c943`, +regression test `4abe0f6`). `install.sh` pins it, and the s3-umem arm uses +`default` again. See "Fixed in libumem 4.3.2" after F1-003. On the dev box (Debian 12, x86_64, libumem built from the pinned commit), `UMEM_DEBUG=default` (which includes the `audit` per-buffer stack-capture @@ -151,6 +153,8 @@ fill-pattern violation. rtcheck verifies the data afterwards. ### F1-003 (tool, not a libdb bug): libumem `UMEM_DEBUG=default` HANGS on the GitHub Ubuntu 26.04 runner +**Status:** fixed upstream in libumem v4.3.2, as F1-002. + Where F1-002 saw `default` abort on the dev box, the hosted runner (kernel 7.0.0-1017-azure, 4 vCPU, libumem fe9ff961 with its rseq fast path) **hangs**: a trivial one-`malloc` program under `LD_PRELOAD=libumem_malloc.so @@ -175,9 +179,47 @@ in the workload is a FAIL verdict, never a silently cancelled job. FAINT is bounded the same way (`S3_FAINT_TIMEOUT`=120 s); it was **not** the hang (it finishes in < 1 s on the runner, 4 injection positions, 0 crashes). -Report upstream to libumem: `UMEM_DEBUG=default` (the `audit` component) both -aborts (dev box, Debian 12 kernel 6.1) and hangs (Ubuntu 26.04 kernel 7.0) on a -trivial program. +Reported upstream to libumem on 2026-11-04: `UMEM_DEBUG=default` (the `audit` +component) both aborts (dev box, Debian 12 kernel 6.1) and hangs (Ubuntu 26.04 +kernel 7.0) on a trivial program. + +### Fixed in libumem 4.3.2 + +**Cause (libumem's diagnosis, matching ours).** With `audit` on, libumem +records a stack for every vmem segment from inside `vmem_hash_insert()`, while +it holds vmem locks. The first stack walk on each thread asked glibc +`pthread_getattr_np()` for the stack bounds, and that call allocates. Under +`LD_PRELOAD` the allocation is libumem's own, so the allocator re-entered +itself. It then either failed the `VMC_POPULATOR` assertion (F1-002) or +deadlocked its update thread against the main thread (F1-003), depending on +rseq and timing. + +**Fix.** libumem 4.3.2 reads the stack bounds from `/proc/self/maps` with plain +`open`/`read` into a stack buffer, so nothing on that path allocates. The fix +is `490c943`, its regression test is `test/security/test_audit_reentry.sh` +(`4abe0f6`), and the release is `v4.3.2` (`60a6452`). The ABI is unchanged. + +**Verified on the dev box** (Debian 12, x86_64, 32 vCPU, 2026-11-05). Each +library was first shown live, by catching a planted 64-into-16 overflow under +`guards,contents`. The programs are the one-`malloc` reproducer and a +first-`malloc`-on-a-new-thread variant, 25 runs per cell: + +| `UMEM_DEBUG` / `UMEM_OPTIONS` | `fe9ff96` (old pin) | `60a6452` (v4.3.2) | +|---|---|---| +| `audit` | 25/25 abort | 25/25 exit 0 | +| `audit` + `norseq` | 25/25 hang | 25/25 exit 0 | +| `default` | 25/25 abort | 25/25 exit 0 | +| `default` + `norseq` | 25/25 hang | 25/25 exit 0 | + +The s3 scenario against libdb `master` with 4.3.2 and `UMEM_DEBUG=default`: +s3-umem is clean (no heap error, data verified) and s3-umem-sabotage catches +the planted overflow, on seeds 1-5. `guards,contents` gives the same verdicts. + +**Harness change.** `install.sh` pins `LIBUMEM_PIN` to `60a6452`, and s3 uses +`UMEM_DEBUG=default` again, so the audit allocation and free stacks are back in +its reports. Every libumem run stays under `timeout -k`, because the SIGKILL +fallback is what ends a hung `LD_PRELOAD` process. `S3_UMEM_MODE=guards,contents` +still selects the old mode. # Part 2 — multi-node HA and XA (F2, scenarios S4–S8) diff --git a/test/dst-rt/README.md b/test/dst-rt/README.md index 815cd67162..af75b0baa4 100644 --- a/test/dst-rt/README.md +++ b/test/dst-rt/README.md @@ -59,10 +59,10 @@ Gotchas the install encodes (so you do not rediscover them): `fault_inject32.so` (needs the 32-bit multilib). We build the 64-bit objects, stub the 32-bit blob, and hand-link; `map_c.o` must be compiled with `gcc` (as C++ the `_map_init`/`map` symbols `faint.c` references get mangled). -- **libumem `UMEM_DEBUG=default` aborts** any program at startup on the dev box - (the `audit` component); S3 falls back to `UMEM_DEBUG=guards,contents`, the - detection half, which runs clean and still catches overflow + UAF. See - FINDINGS F1-002. +- **libumem must be 4.3.2 or later.** Before it, `UMEM_DEBUG=default` (its + `audit` component) aborted any program at startup on the dev box and hung on + the Ubuntu 26.04 runner, so S3 used `UMEM_DEBUG=guards,contents`. install.sh + pins 4.3.2, and S3 uses `default` again. See FINDINGS F1-002 and F1-003. ## The oracle (`rtcheck`) and the ACK log @@ -183,4 +183,5 @@ Real bugs a scenario finds are issue-style write-ups in [`FINDINGS.md`](FINDINGS.md), each with a reproducer. Engine fixes go on their own branches from `origin/master`, never on `test/dst-rt`. F1-001 (libdb cannot open under a *blanket* O_DIRECT; the native direct path is clean) and F1-002 -(libumem `audit` mode aborts on the dev box) are recorded there. +(libumem `audit` mode aborted on the dev box; fixed in libumem 4.3.2) are +recorded there. diff --git a/test/dst-rt/scenarios/s3_alloc_failure.sh b/test/dst-rt/scenarios/s3_alloc_failure.sh index 71092f8181..5a761ae087 100755 --- a/test/dst-rt/scenarios/s3_alloc_failure.sh +++ b/test/dst-rt/scenarios/s3_alloc_failure.sh @@ -114,25 +114,16 @@ else echo " SKIP: libfiu unavailable"; hi_emit s3-fiu-malloc skip fi -# ---- s3-umem: guard redzones + contents (fill-pattern) over the workload -- +# ---- s3-umem: guard redzones, fill pattern and audit stacks over the workload echo "== s3-umem ==" if fi_have umem; then - # UMEM_DEBUG mode: the brief asks for `default` (guards + audit). On some - # hosts the `audit` component (per-buffer stack capture) aborts even a - # trivial program at startup, with no detection report -- a libumem/host - # limitation, NOT a libdb heap bug (a 1-malloc program aborts too). Probe - # `default` with a trivial program; if it aborts spuriously, fall back to - # `guards,contents`, which is the DETECTION half of default (redzone + - # fill-pattern): it still catches an overflow and a use-after-free (the - # sabotage arm proves this), just without the audit backtraces. - # UMEM_DEBUG mode: guards,contents, the detection half of `default` - # (redzones + fill pattern). It catches the planted overflow (the - # sabotage arm below) and runs the libdb workload clean. `default` adds - # the audit component, which is not usable here: on the dev box it aborts - # a one-malloc program (F1-002), and on the GitHub Ubuntu 26.04 runner it - # HANGS intermittently (F1-003) -- a probe passed and the workload then - # hung, so no probe can vouch for it. S3_UMEM_MODE overrides. - UMODE=${S3_UMEM_MODE:-guards,contents} + # UMEM_DEBUG mode: `default` (redzones, fill pattern, and audit + # allocation/free stacks). Before libumem 4.3.2 the audit component + # re-entered the allocator under LD_PRELOAD, so `default` aborted on the + # dev box (F1-002) and hung on the Ubuntu 26.04 runner (F1-003), and this + # arm used guards,contents. install.sh pins 4.3.2, which fixes that. + # S3_UMEM_MODE overrides; guards,contents still works. + UMODE=${S3_UMEM_MODE:-default} _home="$RUN/umem/home"; mkdir -p "$_home" # Every libumem run is bounded: on the Ubuntu 26.04 runner this arm hung @@ -184,7 +175,7 @@ CEOF "${CC:-cc}" -O0 "$RUN/sab_oob.c" -o "$RUN/sab_oob" 2>/dev/null # Use the SAME mode the s3-umem arm settled on, so the teeth match the # detection actually in force. - UMODE=${UMODE:-guards,contents} + UMODE=${UMODE:-default} sout=$(timeout -k 5 30 env LD_PRELOAD="$UMEM_SO" UMEM_DEBUG="$UMODE" \ UMEM_OPTIONS=abort "$RUN/sab_oob" 2>&1) src2=$? diff --git a/test/dst-rt/tools/install.sh b/test/dst-rt/tools/install.sh index b93242a532..ec471b7762 100755 --- a/test/dst-rt/tools/install.sh +++ b/test/dst-rt/tools/install.sh @@ -36,7 +36,7 @@ SKIP_APT=${SKIP_APT:-0} LIBDIRECTIO_REPO=https://github.com/gburd/libdirectio LIBDIRECTIO_PIN=67e57c3064f2d56273376d41bb759320c9e93110 LIBUMEM_REPO=https://codeberg.org/gregburd/libumem -LIBUMEM_PIN=fe9ff96114fbcbf63f89fefec6bc9585efb445f0 +LIBUMEM_PIN=60a64527e8710d955062d0c9f5fd44ce335fa7c1 # tag v4.3.2: audit re-entrancy fix (F1-002/F1-003) FAINT_REPO=https://github.com/misc0110/faint FAINT_PIN=b24cf2d573d245a3eb53fe47ffe3296355c09676 # libfiu: apt where packaged (Debian 12: 1.1); Ubuntu 26.04 dropped it, so