fix(scorch): verify merge result integrity before introduction

[flash7777]

Summary

Add a post-merge verification step that reads back the newly created segment before introducing it into the index snapshot. If any posting list entry is unreadable (e.g. corrupt varint encoding), the merge result is discarded and the source segments are preserved.

This prevents corrupt segments from entering the index, where they would cause permanent read errors and eventually block the merger goroutine entirely.

Background

We observed sporadic corruption (~1% of merges) in merge result segments when merging segments that contain document drops (updates/deletes). The corruption manifests as ReadUvarint overruns in freq/norm data — the segment file has the correct size but contains unreadable varint sequences.

Key observations:

• Only occurs when drop bitmaps are non-empty (document updates)
• Affects high-frequency keyword terms (e.g. MimeType, RootID)
• Drop bitmaps were cloned before passing to MergeUsing — no external mutation; the bug is in the merge write path itself
• Without this guard, one corrupt merge result permanently blocks the merger (it retries the same segment combination indefinitely)
• The corrupt segment replaces valid source segments, causing silent data loss for affected terms

What this PR does

After segPlugin.MergeUsing() + segPlugin.OpenUsing(), iterate every posting of every term of every field in the new segment. If any Iterator.Next() returns an error, discard the segment file and return an error from the merge task. The source segments remain in the index unchanged.

Applied in both merge paths:

1. planMergeAtSnapshot (file merge, merger goroutine)
2. mergeAndPersistInMemorySegments (in-memory merge, persister goroutine)

Performance

The verification reads the merge result sequentially once. For typical merge tasks (2-4 segments, a few thousand documents), this takes 1-10ms against 100-1000ms for the merge itself (<10% overhead).

Reproduction

1. Create an index with ~10,000+ documents sharing a high-frequency keyword term (e.g. MimeType="application/pdf")
2. Reindex all documents with force (update = drop old + write new)
3. Let the merger run — ~1% of merge tasks produce corrupt segments
4. Without this patch: merger blocks permanently on the corrupt segment
5. With this patch: corrupt result discarded, merger continues

Environment: bleve v2.6.0, zapx v17, Go 1.24, 68k documents.

Related

• Rare crash in scorch #1306 (merger panic on corrupt postings)