Skip to content

refactor(phase): write the summary VCF with htslib instead of fprintf (#229) - #239

Merged
TimD1 merged 2 commits into
devfrom
229_td_htslib-summary-vcf
Aug 8, 2026
Merged

refactor(phase): write the summary VCF with htslib instead of fprintf (#229)#239
TimD1 merged 2 commits into
devfrom
229_td_htslib-summary-vcf

Conversation

@TimD1-bot

Copy link
Copy Markdown
Collaborator

Note

Authorship: the content below was drafted by Claude Opus 5 (an AI coding agent) and
filed via gh under @TimD1-bot, a bot account operated by @TimD1. It reflects the
agent's analysis, not a statement authored by @TimD1.

Fixes #229. This is the writer swap only — no feature change — so that the open summary-VCF PRs
(#209, #210, #211, #212) can rebase onto it and delete their hand-rolled equivalents.

Problem

vcfdist read VCFs with htslib but wrote the summary VCF with fprintf. Every output-side VCF
semantic therefore had to be re-implemented on strings: field separators, missing values,
per-allele list rendering, and the FORMAT column itself were all assembled by hand in
print_var_info / print_var_empty / print_var_sample, and the header was 30 fprintf calls.

Change

write_summary_vcf now opens the output with hts_open, writes a header built by
summary_vcf_header(), and emits each record as a bcf1_t through bcf_write. The three
print_var_* methods are replaced by:

  • set_var_record() — sets CHROM/POS/ID/REF/ALT/QUAL/FILTER on a cleared record. The INS/DEL
    left-anchoring and its contig-start guard are unchanged; POS moves into rec->pos as a 0-based
    coordinate rather than being rendered 1-based by hand.
  • var_sample_fields() / empty_sample_fields() — return one sample's FORMAT values as a
    sample_fields struct, with htslib's missing sentinels in place of the "." strings.
  • set_record_samples() — writes both samples' values into the record, one
    bcf_update_format_* call per field, in FORMAT declaration order.

Two encoding details are worth calling out, since neither has a string analogue:

  • GT goes through bcf_update_genotypes, not a string field. A sample with no call at the
    locus is one bcf_gt_missing allele, which renders as . exactly as before.
  • Per-allele lists are padded to a shared length. htslib stores the same value count for
    every sample, so a haploid sample beside a diploid one is padded with the end-of-vector
    marker (bcf_int32_vector_end / bcf_float_set_vector_end), which htslib renders as the
    shorter list. Padding with the missing sentinel instead would print a spurious trailing
    ., and padding GT with bcf_int32_missing would print a negative allele index.

PASS is appended to the header explicitly rather than relying on bcf_hdr_init emitting it —
appending an ID the header already holds is a no-op, so the output carries exactly one
##FILTER line either way, and the record's bcf_add_filter lookup can no longer depend on
htslib's version-specific init behaviour.

Output changes

bcf_write is not byte-identical to fprintf, as expected. Two differences, both cosmetic:

  • BC renders compactly. It is declared Type=Float, and htslib prints a float in its
    shortest round-tripping form: 1 and 0.8, not 1.000000 and 0.800000.
  • ##FILTER=<ID=PASS,...> precedes ##fileDate, because bcf_hdr_init emits it directly
    after ##fileformat. Header lines after ##fileformat are unordered in VCF.

Nothing else moved. Over the 72,585 records of the committed chr20 fixture, a field-by-field
diff against dev's output shows BC as the only differing field, and every one of its
values is unchanged at float32 precision (8 distinct renderings, e.g. 1.000000,.1,.).
The exact-string assertions in tests/integration/test-integration.yml and the FMT_BC
assertions in tests/unit/src/test_phase.cpp are regenerated to match; no other assertion
needed to change, which is the evidence that the record shape is preserved.

QQ keeps its current values: the string writer printed it with %d despite the field being
declared Type=Float, so the truncation is preserved here rather than silently widened by the
switch. Correcting it is a behaviour change and belongs in its own PR.

Testing

  • New WriteSummaryVcf.RecordsReadBackThroughHtslib reads the written file back with
    bcf_read, over a two-variant fixture of mixed ploidy so the padded per-allele encoding is
    exercised. A record whose values disagree with their header declaration is rejected on the way
    in, which no assertion over the rendered text would catch — this is the property the swap buys.
  • 749 unit tests and 111 pytest items pass; -Wall -Wextra is clean and doxygen reports no
    warnings.
  • bcftools view -h and bcftools query read the output without complaint.
  • Peak RSS on the chr20 fixture moves 147.3 MB → 152.1 MB (+3.3%), from htslib's writer buffers;
    runtime is unchanged within noise (2.26 s → 2.33 s user). No per-record state is retained —
    one bcf1_t is reused across the whole run — so the memory question the follow-on PRs raise
    does not start here.

TimD1 added 2 commits August 8, 2026 00:24
…#229)

The summary VCF was emitted with fprintf, so every VCF semantic on the
output side had to be re-implemented on strings. Build each record as a
bcf1_t and write it with bcf_write instead, so htslib owns the encoding.

write_summary_vcf() now opens the file with hts_open and writes a header
built by summary_vcf_header(); print_var_info/print_var_empty/
print_var_sample become set_var_record(), which sets the fixed fields,
and var_sample_fields()/empty_sample_fields(), which return one sample's
FORMAT values for set_record_samples() to write. Per-allele values carry
htslib's missing sentinels, and a sample of lower ploidy is padded with
the end-of-vector marker rather than a shorter rendered list.

The output changes in one visible way: BC is a Float, so htslib renders
it compactly ("1" and "0.8", not "1.000000" and "0.800000"). Over the
72,585 records of the committed chr20 fixture that is the only field
that differs; every other field is byte-identical and every BC value is
unchanged at float32 precision. The header also declares PASS before
fileDate, since bcf_hdr_init emits it first. Test assertions over the
rendered floats are regenerated accordingly.
The build copies htslib's headers into src/, so a bare "htslib/vcf.h" only
resolves for a file in that directory; from tests/unit/src it built locally
only because an unrelated Homebrew prefix happens to sit on the default
include path. Include src/variant.h instead, as test_variant.cpp does.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants