Skip to content

btree: bound the compressed decoder's varint reads before reading (over-read fix) - #264

Merged
gburd merged 2 commits into
masterfrom
fix/bt-decompress-bounds
Nov 4, 2026
Merged

gburd merged 2 commits into
masterfrom
fix/bt-decompress-bounds

Conversation

@gburd

@gburd gburd commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

What

__bam_defdecompress (the default compressed-btree key/data decoder,
src/btree/bt_compress.c) read each varint's leading byte — and the magic byte
*s at offset 0 — before checking the compressed buffer was long enough:

s = compressed->data;
if (*s == CMP_INT_SPARE_VAL) ...              /* *s, no size check at all */
size = __db_decompress_count_int(s);          /* reads s[0] */
if (size > compressed->size) return (EINVAL); /* the check is AFTER the read */

A compressed btree's pages come from disk, so a corrupt or truncated compressed
item is a trust boundary (db_verify, a hostile file, DB->get over a
damaged page). A short item over-reads up to one byte past compressed->size
at each of the three varint sites; a zero-length item reads *s with no check
at all — a heap over-read on hostile input.

Fix

Bound the current offset (size, which tracks s's displacement) strictly
inside compressed->size before every __db_decompress_count_int, and require
compressed->size >= 1 before *s. The __db_decompress_int32 reads were
already covered by the following size > compressed->size check, so only the
lookahead byte and the magic byte needed guarding. One hunk; no format,
log-record, ABI or region change (region sig 0x841f4ea8 unchanged).

How it was found and proved

Found by CBMC: test/cbmc/harness_bt_decompress.c (RFC 0025 B1, on
test/cbmc-expand / #263) proves the decoder's memory safety over every bounded
input. The unmodified engine code produces a counterexample (the over-read);
the -DB1_FIXED variant — this fix — verifies.

Test (fails before, passes after)

test/db/run_bt_decompress_bounds.sh + bt_decompress_bounds.c: the compressed
item is placed flush against a PROT_NONE guard page, so the over-read faults
on any build, not only under ASan.

  • master: case A (size 0) and case B (size 1) both die on SIGSEGV (the
    over-read); VERDICT ... FAIL.
  • this branch: both return EINVAL; VERDICT ... PASS.

Also confirmed under ASan: master aborts with a heap-buffer-overflow READ in
__db_decompress_count_int (db_compint.c:306) ← __bam_defdecompress
(bt_compress.c:548). SKIPs cleanly on a build without --enable-compression
(the decoder is not compiled in then). MANIFEST entry + executable mode added.

LTS

lts/2026.10.7 has the identical code. Backport:
backport/2026.10.7/bt-decompress-bounds (cherry-pick -x), L1/L3/L4 gates
pass (region sig byte-identical, base FAILs / backport PASSes).

…er-read fix)

__bam_defdecompress (the default compressed-btree key/data decoder) read each
varint's leading byte -- via __db_decompress_count_int -- and the magic byte
*s at offset 0 BEFORE checking the compressed buffer was long enough:

    s = compressed->data;
    if (*s == CMP_INT_SPARE_VAL) ...              (*s, no size check at all)
    size = __db_decompress_count_int(s);          (reads s[0])
    if (size > compressed->size) return (EINVAL); (the check is AFTER the read)

A compressed btree's pages come from disk, so a corrupt or truncated
compressed item is a trust boundary (db_verify, a hostile file, DB->get over a
damaged page). A short item over-reads up to one byte past compressed->size at
each of the three varint sites, and a zero-length item reads *s with no check
at all -- a heap over-read on hostile input.

Fix: before every __db_decompress_count_int, require the current offset
(tracked in `size`, which matches s's displacement) to be strictly inside
compressed->size; and require compressed->size >= 1 before reading *s. Every
__db_decompress_int32 read was already covered by the following
size > compressed->size check, so only the lookahead and the magic byte needed
guarding.

Proved over every bounded input by test/cbmc/harness_bt_decompress.c
(-DB1_FIXED verifies; the unmodified code's over-read is the CBMC
counterexample). Regression test test/db/bt_decompress_bounds.c places a
tightly-sized compressed item flush against a PROT_NONE guard page, so the
over-read faults on ANY build, not only under AddressSanitizer: it FAILs on
master (SIGSEGV in both the size-0 and the truncated-varint case) and PASSes
with this fix (clean EINVAL). SKIPs on a build without --enable-compression,
since the decoder is not compiled in then. MANIFEST entry + executable mode
added.

No format, log-record, ABI or region change.
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Coccinelle convention checks

No new violations. ✅

Resolved since baseline (3) -- update dist/cocci/baseline.txt to lock these in.
rule_mutex_unbalanced|MUTEX_UNBALANCED|src/btree/bt_curadj.c|return (ret);
rule_mutex_unbalanced|MUTEX_UNBALANCED|src/crypto/mersenne/mt19937db.c|return (ret);
rule_mutex_unbalanced|MUTEX_UNBALANCED|src/mp/mp_register.c|return (ret);

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

ABI diff vs v2026.10.8 (libabigail — authoritative)

abidiff exit bitmask: 0

Removed exported symbols (nm -D, _NNNN version suffix normalized)

None.


Advisory: libabigail/nm is the authoritative binary-ABI check; Coccinelle is complementary source-level early warning. See dist/cocci/README.md.

…nt reads

The previous commit stopped the decoder reading a varint past the end of
the item, but the values it decodes are from disk too.  With prefix 4 and
suffix 0xFFFFFFFE, prefix + suffix wraps to 2, "size += suffix" wraps back
under compressed->size, every check passes, and memcpy copies about 4 GB.
That happens on both the normal and the duplicate (0xFC) path.

__bamc_start_decompress had the same hole one level up: it decoded the
first record's length and copied that many bytes out of the page with no
check.  A compressed btree with one corrupted length byte crashed a plain
DB->get.

Bound prefix against the previous key or data, suffix and the data length
against what is left of the item, and prefix + suffix against UINT32_MAX,
before any of them is summed.  Check the first record's length the way
__bam_compress_salvage already does.

bt_decompress_bounds gains three cases: the wrap on each path (C, D), and
a real compressed database with its first length corrupted on disk (E,
public API only).  On the dev box (Debian 12, --enable-compression
--enable-diagnostic): master fails A-E with SIGSEGV; the previous commit
alone fails C, D and E; this commit passes all five.
@gburd

gburd commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Lead review: the first commit (2e99a1f) stops the varint lookahead over-read, but not the bigger hole. The decoded prefix, suffix and data length are from disk and can be any u32. With prefix 4 and suffix 0xFFFFFFFE, prefix + suffix wraps to 2, size += suffix wraps back under compressed->size, every check passes, and memcpy copies about 4 GB, on both paths. __bamc_start_decompress also copied the first record's decoded length out of the page unchecked, so one corrupted length byte crashes a plain DB->get.

e067d65 bounds those lengths before any sum, checks the first-record length the way __bam_compress_salvage already does, and adds cases C, D (the wrap on each path) and E (a real compressed database with one byte corrupted on disk, public API only) to bt_decompress_bounds.

Dev box, Debian 12, --enable-compression --enable-diagnostic:

code A B C D E
master SIGSEGV SIGSEGV SIGSEGV SIGSEGV SIGSEGV
2e99a1f only pass pass SIGSEGV SIGSEGV SIGSEGV
+ e067d65 pass pass pass pass pass

The CBMC harness missed the wrap because its varint model returned a value of at most about 2^17. It now returns any u32, and the first commit alone fails the proof (new b1-wrap arm on #263).

@gburd
gburd marked this pull request as ready for review November 4, 2026 21:08
@gburd
gburd merged commit c7b9778 into master Nov 4, 2026
109 of 115 checks passed
@gburd
gburd deleted the fix/bt-decompress-bounds branch November 4, 2026 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant