diff --git a/docs_src/_data/releases.toml b/docs_src/_data/releases.toml index 39abd53305..221d248d23 100644 --- a/docs_src/_data/releases.toml +++ b/docs_src/_data/releases.toml @@ -15,6 +15,7 @@ historical = ["v5.3.28", "v4.6.21-SSI"] [notes] "v2026.10.8" = "Regular release: faster transactional writes — the deadlock detector skips its graph when no deadlock is possible, and inserts stop locking idle cursor queues (btree inserts 1.43× at 4 threads, recno read-modify-write 1.88×); typed data and constraints (--enable-typed), Rust and Zig bindings. Formats, soname and ABI unchanged" +"v2026.10.7.2" = "LTS patch release of the 2026.10.7 line, fixes only, formats unchanged: DB_TXN_SERIALIZABLE could commit a write skew (three missed conflicts, now caught), recovery of many prepared transactions ran out of lockers with lockers free on other stripes, and an equal-key put over a deleted key or an all-deleted duplicate set kept the deleted key's bytes under a non-identity bt_compare" "v2026.10.7.1" = "LTS patch release of the 2026.10.7 line, fixes only, formats unchanged: a reverse cursor walk (DB_PREV) could silently skip live records next to deleted ones (a strict-aliasing miscompile, seen with clang on aarch64), DB_TRUNCATE with a subdatabase name destroyed the whole file (now refused with EINVAL), and a lock-order inversion when discarding a written file from the cache, which a debug build reported as a panic" "v2026.10.7" = "First LTS release: the 2026.10.x line freezes the region, database and log formats, the soname and the ABI. On macOS arm64 the hybrid mutex is selected again (region format changes there only; see the 2026.10.7 upgrade note); every other platform is byte-identical to 2026.10.6" "v2026.10.6" = "BREAKING: the environment region format changes (recover, remove the regions, restart). RFC 0015 scalability work and the AArch64 hybrid mutex, which fixes the FreeBSD/Linux arm64 multi-process limitation (B3)" diff --git a/docs_src/guides/upgrading/_meta.toml b/docs_src/guides/upgrading/_meta.toml index 63d178b73b..51974fcefd 100644 --- a/docs_src/guides/upgrading/_meta.toml +++ b/docs_src/guides/upgrading/_meta.toml @@ -10,6 +10,7 @@ order = [ "introduction", "upgrade_process", "upgrade_2026_10_8", + "upgrade_2026_10_7_2", "upgrade_2026_10_7_1", "upgrade_2026_10_7", "upgrade_2026_10_6", diff --git a/docs_src/guides/upgrading/upgrade_2026_10_7_2.md b/docs_src/guides/upgrading/upgrade_2026_10_7_2.md new file mode 100644 index 0000000000..b0cc02f26a --- /dev/null +++ b/docs_src/guides/upgrading/upgrade_2026_10_7_2.md @@ -0,0 +1,42 @@ +--- +title: "Upgrading to libdb 2026.10.7.2" +api-name: "Upgrading to libdb 2026.10.7.2" +--- +## Upgrading to libdb 2026.10.7.2 + +libdb 2026.10.7.2 is the second patch release of the 2026.10.7 long-term support (LTS) line. It contains bug fixes only. + +**No upgrade procedure is needed.** The shared-memory region format, the database and log formats, the soname and the public ABI are the same as 2026.10.7 and 2026.10.7.1 on every platform. An environment created by either opens with 2026.10.7.2 and the other way round, and processes from all three releases can share one environment. Install it in place of 2026.10.7.1. + +### Build signatures + +The signature depends on the platform and the configure options. On each +platform below, 2026.10.7, 2026.10.7.1 and 2026.10.7.2 built the same way print +the same value. + +| Platform (default `configure`) | 2026.10.7.1 | 2026.10.7.2 | +| --- | --- | --- | +| Linux x86_64 (Debian 12, gcc) | `0x841f4ea8` | `0x841f4ea8` | +| FreeBSD aarch64 (14.5, clang) | `0x56a06689` | `0x56a06689` | +| FreeBSD x86_64, macOS arm64, Windows | unchanged | unchanged | + +### What it fixes + +**`DB_TXN_SERIALIZABLE` committed write skews.** Serializable snapshot isolation (SSI) is meant to abort one of two transactions that each read what the other writes. It missed three cases, so both transactions committed a history that no serial order produces: + +- On a B-tree with more than one level, a transaction that write-locked a page dropped its own read marker on that page, even when it had read a different record there. A later writer of that page then saw no reader. +- A writer that had to wait behind another writer on the same page, or that re-acquired or upgraded a lock it already held, never checked the page's read markers. +- A reader that read a page after a writer had already locked it and checked the page was never seen by that writer, because a read marker never blocks and so never prompts a second check. + +Each is fixed: a transaction keeps its own markers, every way a write lock is granted checks the markers, and a reader now records the conflict with the writers already holding the page. Measured with the Adya cycle checker (`test/isolation/test_ssi_cycle`) at several records per leaf, 40 seeds each, 2026.10.7.1 committed a non-serializable history in 4 of 40 runs at 2 threads, 23 of 40 at 16 threads (B-tree) and 16 of 40 at 16 threads (Hash). 2026.10.7.2 commits none, at the same throughput. If you use `DB_TXN_SERIALIZABLE`, upgrade. + +**Recovery of many prepared transactions ran out of lockers.** Recovery creates one locker per prepared transaction. Since 2026.10.5 each of the 64 locker stripes has its own free list, and a stripe that ran dry could not borrow an idle locker from another stripe once the total reached `lk_max_lockers`. Recovery then failed with `BDB2055 Lock table is out of available locker entries` and panicked, even with lockers free elsewhere. A dry stripe now takes a free locker from another stripe before it gives up. The TCL test `recd015` (10,000 prepared transactions) failed on 2026.10.5 through 2026.10.7.1 and passes again. + +**An equal-key put over a deleted key kept the deleted key's bytes.** In a B-tree whose `bt_compare` treats different byte strings as equal (for example a case-insensitive comparison, `"abc"` equal to `"ABC"`), a put of `"ABC"` over a deleted `"abc"` that a cursor still referenced brought the pair back under the old bytes, `"abc"`. The stored key depended on whether an unrelated cursor happened to be open. The same happened to a duplicate set whose members were all deleted while a cursor referenced one of them. In SQL this showed up as a `COLLATE NOCASE` index or a WITHOUT ROWID primary key keeping the old spelling after an `UPDATE`. The put now stores the new key's bytes. For a duplicate set the set's one shared key is rewritten in place, so every member and every cursor sees the new spelling, and `DB->verify` passes. One case is unchanged: a duplicate set whose shared key is long enough to be stored on an overflow page keeps the old bytes. The set stays valid; see `test/KNOWN-ISSUES.md`. + +Each fix has a regression test that fails on 2026.10.7.1 and passes on 2026.10.7.2: `test/isolation/ssi_multiread_skew.c`, `test/isolation/test_ssi_cycle.c` and `test/isolation/ssi_livelock.c` (the isolation tier), `test/db/run_lock_stripe_exhaustion.sh`, `test/db/run_r1_revive.sh`, `test/db/run_r2_dup_revive.sh` and `test/db/run_r2_fuzz.sh`. + +### Also in this release + +- `run_qam_extent_vrfy` allows 600 seconds instead of 180. GitHub's Ubuntu 26.04 runner image needs about 181 seconds for its 775 extent-file syncs. +- A recno duplicate tree is descended with the recno internal-page macro, not the btree one. diff --git a/test/audit/QUAL-2026.10.7.2-freebsd-aarch64.txt b/test/audit/QUAL-2026.10.7.2-freebsd-aarch64.txt new file mode 100644 index 0000000000..8f0bfc383c --- /dev/null +++ b/test/audit/QUAL-2026.10.7.2-freebsd-aarch64.txt @@ -0,0 +1,50 @@ +# v2026.10.7.2 qualification: freebsd-aarch64 (FreeBSD 14.5-STABLE arm64) +base: defea6df Merge pull request #221 from berkeleydb/backport/2026.10.7/fixes-1 +cand: 1943cc9b release: libdb 2026.10.7.2 (LTS patch) + +## L1 region signature (default configure) + base: 0x56a06689 + cand: 0x56a06689 + L1 PASS + +## L2 soname, exported symbols, .text, handle sizes + base soname: libdb-2026.0.so + base exported symbols: 1848 + base .text: 1061308 + cand soname: libdb-2026.0.so + cand exported symbols: 1848 + cand .text: 1064460 + symbols removed: 0 added: 0 + handle_sizes (cand): run_handle_sizes.sh: PASS + +## L3 fixed bugs: base / release (each test from the candidate tree, run against both libraries) + run_r1_revive: FAIL / PASS + run_r2_dup_revive: FAIL / PASS + run_r2_fuzz: PASS / PASS + run_lock_stripe_exhaustion: FAIL / PASS + isolation tier (incl. ssi_cycle_btree, ssi_multiread_skew, ssi_livelock): base rc=1 / release rc=0 + base failing cells: + cand failing cells: + +## gate 1 test/db tier (release, diagnostic) + cand rel: rc=1 pass=35 skip=4 fail=1 + run_qam_extent_vrfy.sh: FAIL (rc=124) + cand diag: rc=1 pass=35 skip=4 fail=1 + run_qam_extent_vrfy.sh: FAIL (rc=124) + +## gate 8 soak (SSI changes SIREAD marker lifetime) + soak rc=0: 5 workload(s) run, 0 unexpected outcome(s) + +## done 2026-11-04T00:16:30Z + +## gate 1 rerun, run_qam_extent_vrfy TIMEOUT=1200 (see QUAL-2026.10.7.2.md) + cand rel (qam TIMEOUT=1200): rc=0 pass=37 skip=4 fail=0 + run_u9_serializable.sh: SKIP no javac on this host + run_u8_backup_config.sh: SKIP no javac on this host + run_mvcc_mtx_doublefree.sh: SKIP failchk reclaimed no mutexes (0 BDB2017 lines), so the F6 scenario was never constructed -- this run proves nothing either way + run_meson_autoconf_parity.sh: SKIP no meson on this host + cand diag (qam TIMEOUT=1200): rc=0 pass=37 skip=4 fail=0 + run_u9_serializable.sh: SKIP no javac on this host + run_u8_backup_config.sh: SKIP no javac on this host + run_mvcc_mtx_doublefree.sh: SKIP failchk reclaimed no mutexes (0 BDB2017 lines), so the F6 scenario was never constructed -- this run proves nothing either way + run_meson_autoconf_parity.sh: SKIP no meson on this host diff --git a/test/audit/QUAL-2026.10.7.2-freebsd-x86_64.txt b/test/audit/QUAL-2026.10.7.2-freebsd-x86_64.txt new file mode 100644 index 0000000000..b3d0393e09 --- /dev/null +++ b/test/audit/QUAL-2026.10.7.2-freebsd-x86_64.txt @@ -0,0 +1,49 @@ +# v2026.10.7.2 qualification: freebsd-x86_64 (FreeBSD 14.5-STABLE amd64) +base: defea6df Merge pull request #221 from berkeleydb/backport/2026.10.7/fixes-1 +cand: 1943cc9b release: libdb 2026.10.7.2 (LTS patch) + +## L1 region signature (default configure) + base: 0x56a06689 + cand: 0x56a06689 + L1 PASS + +## L2 soname, exported symbols, .text, handle sizes + base soname: libdb-2026.0.so + base exported symbols: 1848 + base .text: 1156684 + cand soname: libdb-2026.0.so + cand exported symbols: 1848 + cand .text: 1160828 + symbols removed: 0 added: 0 + handle_sizes (cand): run_handle_sizes.sh: PASS + +## L3 fixed bugs: base / release (each test from the candidate tree, run against both libraries) + run_r1_revive: FAIL / PASS + run_r2_dup_revive: FAIL / PASS + run_r2_fuzz: PASS / PASS + run_lock_stripe_exhaustion: FAIL / PASS + isolation tier (incl. ssi_cycle_btree, ssi_multiread_skew, ssi_livelock): base rc=1 / release rc=0 + base failing cells: + cand failing cells: + +## gate 1 test/db tier (release, diagnostic) + cand rel: rc=1 pass=35 skip=4 fail=1 + run_qam_extent_vrfy.sh: FAIL (rc=124) + cand diag: rc=1 pass=35 skip=4 fail=0 + +## gate 8 soak (SSI changes SIREAD marker lifetime) + soak rc=0: 5 workload(s) run, 0 unexpected outcome(s) + +## done 2026-11-04T00:10:25Z + +## gate 1 rerun, run_qam_extent_vrfy TIMEOUT=1200 (see QUAL-2026.10.7.2.md) + cand rel (qam TIMEOUT=1200): rc=0 pass=37 skip=4 fail=0 + run_u9_serializable.sh: SKIP no javac on this host + run_u8_backup_config.sh: SKIP no javac on this host + run_mvcc_mtx_doublefree.sh: SKIP failchk reclaimed no mutexes (0 BDB2017 lines), so the F6 scenario was never constructed -- this run proves nothing either way + run_meson_autoconf_parity.sh: SKIP no meson on this host + cand diag (qam TIMEOUT=1200): rc=0 pass=37 skip=4 fail=0 + run_u9_serializable.sh: SKIP no javac on this host + run_u8_backup_config.sh: SKIP no javac on this host + run_mvcc_mtx_doublefree.sh: SKIP failchk reclaimed no mutexes (0 BDB2017 lines), so the F6 scenario was never constructed -- this run proves nothing either way + run_meson_autoconf_parity.sh: SKIP no meson on this host diff --git a/test/audit/QUAL-2026.10.7.2-linux-x86_64.txt b/test/audit/QUAL-2026.10.7.2-linux-x86_64.txt new file mode 100644 index 0000000000..4db5912dda --- /dev/null +++ b/test/audit/QUAL-2026.10.7.2-linux-x86_64.txt @@ -0,0 +1,36 @@ +# v2026.10.7.2 qualification: linux-x86_64 (Linux 6.1.0-54-cloud-amd64 x86_64) +base: defea6df Merge pull request #221 from berkeleydb/backport/2026.10.7/fixes-1 +cand: 1943cc9b release: libdb 2026.10.7.2 (LTS patch) + +## L1 region signature (default configure) + base: 0x841f4ea8 + cand: 0x841f4ea8 + L1 PASS + +## L2 soname, exported symbols, .text, handle sizes + base soname: libdb-2026.0.so + base exported symbols: 1846 + base .text: 1302044 + cand soname: libdb-2026.0.so + cand exported symbols: 1846 + cand .text: 1306636 + symbols removed: 0 added: 0 + handle_sizes (cand): run_handle_sizes.sh: PASS + +## L3 fixed bugs: base / release (each test from the candidate tree, run against both libraries) + run_r1_revive: FAIL / PASS + run_r2_dup_revive: FAIL / PASS + run_r2_fuzz: PASS / PASS + run_lock_stripe_exhaustion: FAIL / PASS + isolation tier (incl. ssi_cycle_btree, ssi_multiread_skew, ssi_livelock): base rc=1 / release rc=0 + base failing cells: + cand failing cells: + +## gate 1 test/db tier (release, diagnostic) + cand rel: rc=0 pass=38 skip=3 fail=0 + cand diag: rc=0 pass=38 skip=3 fail=0 + +## gate 8 soak (SSI changes SIREAD marker lifetime) + soak rc=0: 5 workload(s) run, 0 unexpected outcome(s) + +## done 2026-11-03T23:55:33Z diff --git a/test/audit/QUAL-2026.10.7.2.md b/test/audit/QUAL-2026.10.7.2.md new file mode 100644 index 0000000000..d69c0ebab3 --- /dev/null +++ b/test/audit/QUAL-2026.10.7.2.md @@ -0,0 +1,114 @@ +# v2026.10.7.2 qualification (LTS patch release, 2026-11-03) + +The second patch release of the `2026.10.7` LTS line, qualified against the +gates for an LTS patch release in [`QUALIFICATION-MATRIX.md`](QUALIFICATION-MATRIX.md) +(gates 1–5 and 7, plus L1–L4). Gate 8, the soak, also ran, because the SSI fix +changes how long SIREAD markers live. The candidate was compared with the base +tag `v2026.10.7.1` (`defea6df0`), both built the same way on the same host. + +**Candidate:** `release/2026.10.7.2`, head `1943cc9b3` (PR #260). + +**Hosts:** EC2, on-demand: + +| Host | OS | vCPU | Compiler | +|---|---|---|---| +| Linux x86_64 | Debian 12, c6i.8xlarge | 32 | gcc | +| FreeBSD x86_64 | FreeBSD 14.5, c6i.4xlarge | 16 | clang | +| FreeBSD aarch64 | FreeBSD 14.5, c7g.4xlarge | 16 | clang | + +The raw per-host summaries are kept alongside this file: + +- `QUAL-2026.10.7.2-linux-x86_64.txt` +- `QUAL-2026.10.7.2-freebsd-x86_64.txt` +- `QUAL-2026.10.7.2-freebsd-aarch64.txt` + +## Result: GO + +| Gate | Linux x86_64 | FreeBSD x86_64 | FreeBSD aarch64 | +|---|---|---|---| +| **L1** region signature, base = release | `0x841f4ea8` = `0x841f4ea8` | `0x56a06689` = `0x56a06689` | `0x56a06689` = `0x56a06689` | +| **L2** handle sizes | PASS | PASS | PASS | +| **L2** soname, exported symbols | `libdb-2026.0.so`; 1846 → 1846, 0 removed, 0 added | `libdb-2026.0.so`; 1848 → 1848, 0 / 0 | `libdb-2026.0.so`; 1848 → 1848, 0 / 0 | +| **L2** `.text` (base → release) | 1,302,044 → 1,306,636 | 1,156,684 → 1,160,828 | 1,061,308 → 1,064,460 | +| **L3** `r1_revive`: base / release | **FAIL / PASS** | **FAIL / PASS** | **FAIL / PASS** | +| **L3** `r2_dup_revive`: base / release | **FAIL / PASS** | **FAIL / PASS** | **FAIL / PASS** | +| **L3** `r2_fuzz`: base / release | PASS / PASS | PASS / PASS | PASS / PASS | +| **L3** `lock_stripe_exhaustion`: base / release | **FAIL / PASS** | **FAIL / PASS** | **FAIL / PASS** | +| **L3** isolation tier: base / release | **rc=1 / rc=0** | **rc=1 / rc=0** | **rc=1 / rc=0** | +| **L3** `test_ssi_cycle` btree 2 threads, non-serializable / 40: base / release | **7 / 0** | **5 / 0** | **9 / 0** | +| **L3** `test_ssi_cycle` btree 16 threads | **21 / 0** | **20 / 0** | **27 / 0** | +| **L3** `test_ssi_cycle` hash 16 threads | **21 / 0** | **22 / 0** | **16 / 0** | +| `ssi_livelock`: SSI commits as % of SI, base / release | 99% / 98% | 73–102% / 92% | 90% / 91% | +| **1** `test/db` tier, release / release diagnostic | 38 pass, 3 skip, 0 fail / 38, 3, 0 | 37 pass, 4 skip, 0 fail / 37, 4, 0 † | 37 pass, 4 skip, 0 fail / 37, 4, 0 † | +| **8** soak | 5 workloads, 0 unexpected | 5, 0 | 5, 0 | +| **2** `s_validate` (22 checkers), `s_execbits`, `s_chk_copyright`, `s_chk_licence`, `s_message_id` | rc=0 | (bash-only; Linux covers) | (bash-only; Linux covers) | +| **4** CI region-signature and ABI gates (PR #260) | success | | | +| **5** CI at job level (PR #260, head `1943cc9b3`) | 64 success, 5 skipped, **0 failed** | | | +| **L4** every LTS commit traces to `master` | 17 cherry-picks, all with a `-x` trailer naming a `master` commit; 2 LTS-only (MANIFEST, release) | | | + +The skips are environmental: no JDK (`u8_backup_config`, `u9_serializable`), +no meson on FreeBSD (`meson_autoconf_parity`), and `mvcc_mtx_doublefree` when +failchk reclaims no mutex, so its scenario is never built. + +**† `run_qam_extent_vrfy` on FreeBSD.** The first tier run on both FreeBSD hosts +timed it out (`rc=124` at its 600 s limit). It is I/O-bound by design: about 775 +syncs, one per extent file. Run alone on the same hosts it takes 602–640 s, and +the **base tag takes the same time** (x86_64: base 636 s, release 640 s), so the +release did not slow it down. These hosts use their EBS root (UFS with soft +updates), not the instance store the 2026.10.7.1 hosts used. The tier was rerun +with `TIMEOUT=1200` for that one runner, and those are the counts above. The +limit on the line stays at 600 s; GitHub's runners need about 181 s. + +**L3, the SSI fix.** The isolation tier fails on the base on every host with +`ssi_multiread_skew` committing the cross-leaf write skew in 40/40 attempts, and +passes on the release. The cycle checker (`CYC_VALPAD=200`, several records per +leaf, 40 seeds, 2 s each) found non-serializable histories on the base on every +host and configuration, and none on the release. No release seed tripped the +throughput floor or the watchdog. + +**`ssi_livelock` on FreeBSD prints `BDB3010`.** On both FreeBSD hosts the +release run printed `BDB3010 error getting a page for writing` 30–89 times in +3 s. A probe that counts return codes showed that every transaction that printed +it returned `DB_SNAPSHOT_CONFLICT` or `DB_SNAPSHOT_UNSAFE` and aborted normally, +and none returned another code. `__memp_dirty` suppresses that message for a +deadlock but not for an SSI abort, and the SSI fix raises the abort rate under +contention. The message is noise, not a failure. The fix is #261, for the next +patch release. + +**Gate 6** (the metal benchmark) was not run. The SSI change adds a scan of the +page's write holders when a SIREAD marker is granted, which only applies to +`DB_TXN_SERIALIZABLE` transactions. `ssi_livelock` and the cycle checker show +the same commit throughput as the base. The other fixes are on rare paths: a put +over a deleted key, and locker allocation when a stripe runs dry. + +## L4: every commit since v2026.10.7.1 + +17 commits are cherry-picks with `-x` trailers, and 2 are LTS-only: + +- `84717f412`, LTS-only: this line's `test/MANIFEST`. The cherry-pick of + `b803b8cd1` took master's copy, which listed eight runners only master has. +- `1943cc9b3`, LTS-only: the release commit. + +All 17 `-x` trailers resolve to commits on `master` +(`git merge-base --is-ancestor origin/master`, checked after the +master PRs merged): + +| master PR | commits | +|---|---| +| #242 (CI 26.04) | 81e44c28c | +| R1 | 50ef9c73f | +| #244 (R2) | 76169312b, 9d6d00599, b9064d512, ca66e9f2a, 2e6ae3f7e, 07a240a25, 2d24d212a, 17116871e, d816880e0, ab11f6ff2 | +| #255 (recd015 lockers) | 6ff1561b1 | +| #257 (SSI) | cced5c481, 697c965eb, 2097393af, 12ed935bc | + +## What the release fixes + +| PR (LTS / master) | Fix | Who sees it | +|---|---|---| +| #258 / #257 | `DB_TXN_SERIALIZABLE` committed write skews (three missed conflicts) | every SSI user | +| #256 / #255 | recovery of many prepared transactions ran out of lockers with lockers free on other stripes | 2026.10.5 and later | +| #239 / R1 | an equal-key put over a deleted key kept the deleted key's bytes under a non-identity `bt_compare` | custom comparators, e.g. `COLLATE NOCASE` in SQL | +| #245 / #244 | the same, for a duplicate set whose members were all deleted | custom comparators with `DB_DUP`/`DB_DUPSORT` | + +Also backported: #248 (the `run_qam_extent_vrfy` timeout for the Ubuntu 26.04 +runner). diff --git a/test/audit/QUALIFICATION-MATRIX.md b/test/audit/QUALIFICATION-MATRIX.md index fa062de072..a2eacc38a1 100644 --- a/test/audit/QUALIFICATION-MATRIX.md +++ b/test/audit/QUALIFICATION-MATRIX.md @@ -64,8 +64,9 @@ Gate 6 (the cross-release metal benchmark) is skipped unless a fix touches a hot path. Gate 8 (the soak) is skipped unless a fix touches resource lifetime. The release notes say which gates ran. -The first LTS patch release, `v2026.10.7.1`, is recorded in -[`QUAL-2026.10.7.1.md`](QUAL-2026.10.7.1.md). +The LTS patch releases are recorded in +[`QUAL-2026.10.7.1.md`](QUAL-2026.10.7.1.md) and +[`QUAL-2026.10.7.2.md`](QUAL-2026.10.7.2.md). ## Additional gates for the LTS declaration