Fix RocksDB @expiresAt eviction sweeps

[kylebernhardy]

Summary

• restore exact RocksDB secondary-index enumeration and bounded composite cursor scans so native @expiresAt sweeps work
• preserve the source/cache contract: source-owned TTL comes from context.expiresAt (or table fallback), never from a returned @expiresAt field
• keep explicit write/context expiration ahead of record-field and table defaults while making indexed and full-scan expiration semantics agree
• batch non-blob evictions with bounded concurrency, clean dangling and legacy index entries transactionally, and preserve concurrent writes with version guards
• cancel and drain active sweeps before table/database teardown, including close, drop, and engine recreation paths

Fixes #1481. The broader native index-store work remains tracked in HarperFast/rocksdb-js#312.

Rollout behavior

This intentionally activates physical reclamation for expired RocksDB rows that reads already treat as absent. The prior Rocks sweep was effectively inert, so an upgraded node may have an existing backlog. The sweep drains that backlog without an audit or replication event, matching eviction semantics; it yields every 10 candidates and limits transaction commits to four in flight, but it does not impose the previously reviewed 10k/min ceiling. This behavior change needs explicit release-note and human rollout sign-off.

Verification

• npm run build
• npm run typecheck
• npm run test:types
• npm run lint:required
• npm run format:check
• RocksDB focused suite: 53 passing
• LMDB focused suite: 3 passing
• coverage includes explicit/context, record-field, and table-default precedence; source-context ownership; indexed/full-scan consistency; legacy index cleanup; exact/similar index values; pagination/counts; composite primary keys; 505-row continuation; conflict guards; commit-gated blob deletion; concurrent refresh; cancellation; close/drop races; tableless restore locking; and multi-root environment cleanup
• final Gemini CLI review: no actionable findings
• final Claude + Harper-domain review: merge recommendation; remaining first-sweep upgrade impact is release-note/human-signoff material

Comment generated by kAIle (GPT-5.6)

[gemini-code-assist]

Code Review

This pull request optimizes the record expiration sweep by introducing RocksDB index value enumeration and batching the eviction process in groups of 100 records. It also adds a fallback mechanism to clean up dangling index entries when a record is already gone. The review feedback correctly identifies a redundant database read in this fallback path when primaryStore.ifVersion is undefined, suggesting a simplification to avoid unnecessary I/O overhead.

[claude]

Reviewed; no blockers found. Prior blocker (abortRequested not cleared in failSchemaQuiesceFinalization) is fixed; all inline review threads are resolved.

[github-actions]

Patch cherry-pick: cancelled

The patch label was removed; cherry-pick branch cherry-pick/v5.1/pr-2155 was deleted.

[kriszyp]

Patch? Really?
And why the mixed authors?

[kylebernhardy]

Good catch. I removed the patch label; this should remain a main/v5.2+ fix, so the v5.1 cherry-pick was not appropriate.

Update: at Kyle’s request, I rewrote the five PR commits so both author and committer use Kyle Bernhardy <kyle@harperdb.io>, which GitHub maps to kylebernhardy. The commit contents are unchanged.

Comment generated by kAIle (GPT-5.6)

[cb1kenobi]

About a year ago, I tried to add support for getValues() and we ended up abandoning it in rocksdb-js on June 25, 2025 due to "key sort order" challenges.

I created an IndexStore that extends a Store and would handle the encoding of the keys: https://github.com/HarperFast/rocksdb-js/blob/index-store/src/index-store.ts. I don't remember the specifics, but I remember naively creating an array as the key won't work. If you wanted "hello", it gave you "hello world" too. I tried adding markers around the values, but even those could cause matches. I don't think I tested fixed width keys, but that might yield some success.

We ended up dropping dupSort, getValues(), and getValuesCount() from rocksdb-js.

It's a year later now, the dust has settled, things have stabilized, LLMs have gotten smarter, and maybe this is solvable. I think the next step for this PR is to add way more tests, specifically with similar values.

[kriszyp]

I guess additional concerns I have: I think http://github.com/HarperFast/rocksdb-js/issues/312 is the real solution here. This is would be kind of a temporary solution for... who? I thought the only place we had ever used this was for the EKV project that didn't go through and this received little attention after that. But sure, I guess there is an immediate need for this, we can keep going with this. I think the ticket is marked P1. We haven't been automatically prioritizing P1 tickets into 5.2 patch (while we still have open P0s, or at least P1s with active customer connections).
_{🤖 Reviewed with Claude}

[kriszyp]

but engineering-metrics should not be presented as blocked on it.

Ok, so you cool with this being milestone 5.3, then? (I assigned that).

[kylebernhardy]

Yes, v5.3 timing is acceptable. Engineering Metrics is no longer blocked on this change, and keeping it in v5.3 gives the remaining transaction-lifecycle correction and rollout behavior the right review space. Thanks for moving the milestone.

Comment generated by kAIle (Codex GPT-5.6)