feat(variant): retain source ID/QUAL/FILTER/INFO/FORMAT on the summary VCF (#104)

[TimD1-bot]

Note

Authorship: the content below was drafted by Claude Opus 5 (an AI coding agent) and
filed via gh under @TimD1-bot, a bot account operated by @TimD1. It reflects the
agent's analysis, not a statement authored by @TimD1.

Closes #104. Step 4 of 8 toward #48.

Rebased onto dev now that #229 has merged (d4d7149); this PR no longer stacks on anything and
is a single commit. It was previously written against the fprintf writer and has been
re-expressed against #229's htslib writer — see Rework onto #229 below.

summary.vcf is synthesized from the internal ctgVariants arrays, not copied from the input, so
its site-level columns were hardcoded placeholders: ID=., QUAL=., FILTER=PASS, INFO=.. This
sources them from the input instead.

Source record retention

parse_variants retains, per callset, the columns the writer cannot derive: ID, QUAL, the
FILTER column, the preserved INFO fields, and the sample's preserved FORMAT keys and values.
They live in a new srcRecords (src/variant.h:19) indexed by rec_idx, so the two halves of a
split CPX and the two co-located entries of a het-alt record share one copy. The store is shared by
shared_ptr from variantData through the per-haplotype containers to the merged ones the writer
reads, and shrink() releases the growth slack once its input is parsed.

Columns are held as the VCF text htslib rendered them (vcf_format) and re-typed against the
output header's own declarations when written, so each direction goes through htslib rather
than hand-rolled formatting. Holding one string per column per record costs far less than the
tagged per-field values a typed store would need.

Rework onto #229

#229 replaced the fprintf writer with bcf1_t/bcf_write, which deleted print_var_info,
print_var_sample, and print_var_empty — the three functions this PR originally modified. The
retention half is unchanged; the writing half is new:

Preserved column	Now written by
ID	bcf_update_id in set_var_record
QUAL	rec->qual, or bcf_float_set_missing for .
FILTER	bcf_add_filter per name in the retained column
INFO	bcf_update_info_{flag,int32,float,string}, typed by the output header
FORMAT	bcf_update_format_* via sample_fields::src_keys/src_vals
declarations	summary_vcf_header appends the retained ##FILTER/##INFO/##FORMAT lines

Writing through htslib makes declaration mandatory rather than cosmetic: htslib rejects any
value whose field the output header does not declare. Header propagation, previously a
self-description nicety, is now load-bearing.

The owning-callset rule, and the 1-sample → 2-sample boundary

Each record's columns come from the callset that already supplies its POS/REF/ALT: the query
wherever it calls, the truth only on a pure FN. Accepted loss: on a matched record the truth's
INFO/FORMAT is dropped.

A source record has one sample; the summary VCF has two. A carried FORMAT field therefore has to
be re-keyed across that boundary rather than written through — writing a source sample's values
straight into a two-sample buffer smears them, giving one sample a neighbour's value and shifting
the rest. var_sample_fields(..., owns_record) attaches the fields to the owner alone, and
set_source_formats reuses #229's pad_per_allele to give the non-owner missing values.
WriteSummaryVcf.MultiValuedFormatFieldStaysWithinItsSample pins this with a Number=2 field.

Number=A/R/G fields are skipped here

Their values index the source ALT list while output records carry normalized, split alleles, so
passing them through verbatim would be silently wrong for the emitted allele. They are omitted with
a WARN naming each one (INFO/AF, FORMAT/AD, FORMAT/PL) and left undeclared. The next step adds
correct subsetting and removes the WARN.

FILTER on evaluated records

Input FILTER values are preserved verbatim, evaluated records included. A GA4GH consumer reads a
non-PASS FILTER on an evaluated record as a filtered call and demotes it — filtered TPs become
FNs, filtered FPs become Ns. Rewriting it to PASS would misreport the input, so this is documented
on write_summary_vcf rather than designed around.

Behaviour changes worth flagging

• ##FILTER propagation beyond the issue's ##INFO/##FORMAT list. Without it a preserved
  non-PASS FILTER names an ID the header never declares, which htslib now rejects outright.
• Retention is unconditional. The original gated it on g.write; refactor(defs): split gt_t into parse-time and evaluation genotype enums #224/Write the summary VCF with htslib instead of fprintf #229-era dev removed
  that flag, so there is no longer an "output off" mode to skip retention in. The unit test that
  covered the gate went with it.

Memory

Re-measured against this base. No WGS input exists locally, so a 5M-record chr20 VCF was
synthesized with a caller-shaped layout (ID=., QUAL, PASS, an INFO Flag, GT:GQ:DP:AF:AD)
and run against a 100 kb BED so alignment work stays near zero and the delta is retention alone.
Peak RSS, dev (d4d7149) vs this branch:

records	base peak	this branch	delta
5M (rep 1)	143 MB	2103 MB	+1960 MB
5M (rep 2)	182 MB	2118 MB	+1936 MB

Steady state is 6 std::string headers per record — 144 B/record, with typical field values inside
libc++'s 22-byte SSO — so 5M records carry ~720 MB; the rest is std::vector growth slack and the
shrink() transient, released before alignment begins. At ~3% of the 64 GB --max-ram default this
sits well inside what is already budgeted. The ratio to the alignment working set is not measured
here, since the measurement deliberately suppresses alignment; the figures are absolute.

Deferred: the bcf_dup retention rework

The comment below planned, once #229 landed, to replace srcRecords with bcf_dup(rec) at parse
time plus bcf_hdr_merge() for the output header, cutting ~370 lines to ~80. Not done here, for two
reasons I would rather have on the record than buried:

• bcf_hdr_merge merges every declaration, including the Number=A/R/G fields this PR
  deliberately leaves undeclared and whose absence the tests assert. Selective declaration has to
  stay, which removes part of the claimed simplification.
• Reading INFO back out of a dup'd record requires unpacking it at write time, and those decode
  caches accumulate across the run. That is plausibly worse than the measured profile above and is
  unmeasured.

Worth doing as its own change with its own measurement, not folded into a conflict resolution. The
invariant that comment flagged — the 1-sample → 2-sample re-keying — is implemented and tested here
regardless.

Tests

• test_summary-vcf_preserves-source-fields over a query declaring Number=1, fixed Number=2,
  and Flag fields (all preserved) alongside Number=A/R/G ones (all dropped). Pins the full
  record for the merged 1|1, both CPX halves, both het-alt entries, and the truth-owned FN, plus
  the propagated header lines and the WARN.
• Eight existing summary-VCF assertions gained a real QUAL (. → 50), keeping Write the summary VCF with htslib instead of fprintf #229's float
  rendering (1.000000 → 1).
• 13 unit tests over retention, 6 over the writer.

Verification

• pytest: 132 passed (includes 768 gtest unit tests), from a clean build.
• make at -Wall -Wextra -Werror=missing-field-initializers: no warnings.
• doxygen: 0 warnings.
• bcftools 1.21 on the fixture output: view produces zero bytes on stderr, and
  bcftools query reads every preserved field back as its declared type — Integer DP, Number=2
SB, Flag SOMATIC, String TRUTHSET, and per-sample SDP/SAC with the non-owner missing.

[TimD1-bot]

Claude Opus 5 🤖: Planned rework, filed as #229.

Once the writer emits bcf1_t records via bcf_write, retention here becomes bcf_dup(rec) at parse time plus bcf_hdr_merge() for the output header. That removes srcRecords (8 parallel string vectors), its six src_* accessors, and retain_header_lines — roughly 370 lines down to ~80.

One invariant this PR will need: after n_sample = 2, every carried FORMAT field must be re-keyed from the source's single sample to two, with the non-owning sample set missing. Leaving 1-sample data in place silently emits garbage (a prototype produced AD=17,7,97).

This PR is behind 2 since #103 and #227 merged, so it needs a rebase regardless; the intent is to rebase onto #229's branch rather than onto dev. No action needed until #229 lands.

[TimD1-bot]

Claude Opus 5 🤖: #229 landed (merged to dev as d4d7149), so this is now rebased onto dev
rather than onto its branch — the branch was deleted on merge, and dev's squash is content-identical
to it, so the PR is a single commit against dev and CI runs.

Reworked onto the htslib writer. #229 deleted print_var_info, print_var_sample, and
print_var_empty, which were the three functions this PR modified. Retention is unchanged; writing
is now bcf_update_id, rec->qual, bcf_add_filter, bcf_update_info_* and bcf_update_format_*,
each typed against the output header's own declaration of the field. One consequence worth naming:
declaration is no longer cosmetic — htslib rejects a value whose field the header does not declare,
so summary_vcf_header propagating the input's ##FILTER/##INFO/##FORMAT lines is now
load-bearing rather than a self-description nicety.

The invariant you flagged is implemented and pinned. var_sample_fields(..., owns_record)
attaches carried FORMAT fields to the owning sample alone, and set_source_formats re-keys them
across the 1-sample → 2-sample boundary using #229's pad_per_allele, giving the non-owner missing
values. WriteSummaryVcf.MultiValuedFormatFieldStaysWithinItsSample pins it with a Number=2 field,
and bcftools query on the integration fixture reads back QUERY=12,17 / TRUTH=. — no smearing.

bcf_dup + bcf_hdr_merge deferred, deliberately. Two things I hit that the plan did not
anticipate:

• bcf_hdr_merge merges every declaration, including the Number=A/R/G fields this PR
  deliberately leaves undeclared and whose absence the tests assert. Selective declaration has to
  stay either way, which removes part of the 370→80 saving.
• Reading INFO back out of a dup'd record needs it unpacked at write time, and those decode caches
  accumulate over the run. Against a measured 144 B/record for the current store (~2.1 GB peak at 5M
  records, ~3% of the 64 GB --max-ram default), that is plausibly worse and is unmeasured.

It deserves its own change with its own measurement rather than being folded into a conflict
resolution. Happy to file it if you want it tracked.