epic: Allele centric mapping, storage, and service - #791
Draft
bencap wants to merge 109 commits into
Draft
Conversation
- add shared seqrepo and seqrepo data proxy providers in services - reuse shared seqrepo provider in deps to remove duplicate config - align vrs sequence proxy behavior with dcd-mapping expectations
- add shared helpers to translate hgvs, normalize alleles, and compute ids - clear cached merkle digests before identification so mutated alleles do not keep stale ga4gh identifiers - document the invariant that all allele identification must route through the helper to prevent digest correctness regressions
Introduce a reusable declarative mixin implementing transaction-time SCD Type 2 versioning for rows that change over time, used by either versioned entities or link/association rows. - A row is live while valid_to is NULL; current/as_of express the half-open [valid_from, valid_to) point-in-time predicate so call sites never hand-roll it - supersede_with (single row) and supersede_live_where (bulk) stamp the retired valid_to and successor valid_from with one timestamp for a gap-free handoff regardless of transaction boundaries - retire/retire_live_where are withdrawal primitives; retire cascades to live child links named in __retire_cascade__ - bulk supersede refuses to run on cascade-bearing classes since it cannot fire the cascade - consumers must add a partial unique index over their natural key WHERE valid_to IS NULL as a backstop against duplicate live rows Cover the mixin with unit tests against purpose-built models, using explicit timestamps far from the transaction clock to prove the gap-free handoff and verify the partial unique index backstop.
Add a lib module with reusable helpers for parsing and rewriting HGVS strings ahead of VRS translation. - extract_accession returns the reference accession (substring before the first colon), tolerant of whitespace and missing separators - split_cis_phased_hgvs expands a bracketed multivariant expression into fully-qualified components carrying the original accession and coordinate prefix, since ga4gh's AlleleTranslator only yields a single Allele per call and each component must translate alone - join_cis_phased_hgvs is the inverse, recombining components into one bracketed block and returning None when they do not share a single accession and coordinate prefix Cover the helpers with unit tests including the split/join round trip and the mixed-accession and mixed-prefix rejection cases.
Add VRS translation support for cis-phased multivariant HGVS, building on the new hgvs split helpers. - translate_hgvs_to_variation translates each component HGVS to an Allele independently, returning a bare Allele for a single component and wrapping two or more in a CisPhasedBlock, mirroring dcd_mapping's vrs_map._construct_vrs_allele; the reverse-translation job emits bracketed genomic forms the AlleleTranslator cannot translate directly - identify_variation generalizes identify_allele to blocks, clearing every member and location digest plus the block's own before identifying so a stale member digest never propagates into the block id; the block digest is order-independent so a set dedups to one row Cover both with unit tests, including the order-independent block digest and the stale-digest clearing.
get_hgvs_from_post_mapped can now join the members of a multi-variant block (Haplotype/CisPhasedBlock) into a single bracketed cis-phased expression via the new combine_cis flag, replacing the previous behavior of returning None for any multi-variant block. - combine_cis defaults off because some consumers cannot yet handle a bracketed expression — notably ClinGen submission, which has no single CAID for a multi-variant cis block (#764) - the CSV export fallback opts in, so g./p. HGVS columns are populated from cis-phased post-mapped output instead of left empty - drop the stale commented-out error branches and the resolved TODO
Introduce parallel mapping tables for the Better Reverse Translation epic (#746). The existing mapped_variants table is left untouched (frozen serving) while the new schema is built out separately. - add alleles, mapping_records, and mapping_record_alleles tables with their ORM models; alleles are content-addressed by vrs_digest and shared across mapping records via the link table - mapping_record_alleles.is_authoritative distinguishes the assay's actual measurement from translator-derived links, since the same VRS allele can be authoritative for one record and derived for another - put mapping_records and mapping_record_alleles on valid-time versioning (ValidTime mixin): a re-map retires the prior live row by closing valid_to instead of deleting, so history is retained and point-in-time queries are a single predicate; partial unique indexes promote "one live row per key" to the database - derive Allele.transcript and MappingRecord.transcript as hybrid properties from the HGVS columns rather than storing them, so they cannot drift; drop the stored alleles.transcript column - add the cross_level_translation annotation type, written once per variant to record whether filling unmapped levels succeeded, was skipped, or failed - annotate Variant and TargetGeneMapping with mapping_records relationships and typed primary keys
Add type stubs for the VRS and hgvs APIs used by the reverse translation work so callers can be type-checked without casts. - ga4gh.core: type ga4gh_identify and re-export PrevVrsVersion - ga4gh.vrs: stub the data proxy, AlleleTranslator, and normalize; translate_from returns Any so callers annotate the concrete VRS subtype they expect without a redundant cast - hgvs.assemblymapper: stub AssemblyMapper with the g_to_t/c_to_g/c_to_p conversions used for cross-level translation
Wire the variant-annotation package into the server extra as a local editable (PEP 660) dependency so the API can drive the reverse translation pipeline. - declare variant-annotation as a develop path dependency in pyproject.toml and the server extra; lock pulls in its transitive deps (variant-translation, biopython, openpyxl, et-xmlfile) - mount ../variant-annotation into the dev and worker containers and prepend it to PYTHONPATH so the editable install resolves - pass --no-directory to poetry install in the Dockerfile so the build does not try to install the not-yet-copied editable sibling - ignore_missing_imports for variant_annotation.* in mypy: its editable import hook is unfollowable and it ships no py.typed, so treat it as untyped rather than maintaining drift-prone local stubs - add a Makefile with dev/test/lint/format targets
Rework the variant mapping job to persist its results into the new MappingRecord / Allele / MappingRecordAllele schema instead of MappedVariant, building on the closure tables for the Better Reverse Translation epic. - create one MappingRecord per variant (including failed variants, with null VRS data); successfully post-mapped variants additionally get-or-create an authoritative Allele and link it via an is_authoritative MappingRecordAllele - supersede a variant's prior live record through ValidTime supersede_with — retiring it and its allele links and inserting the new record under one timestamp — instead of flipping a current flag, so the old/new handoff is gap-free - require a TargetGeneMapping for every mapped score, raising on a miss rather than tolerating a null link, since dcd-mapping guarantees one - combine cis-phased members when deriving hgvs_assay_level - update the mapping job tests to assert against MappingRecord and the authoritative-allele link, including the gap-free supersession and cascade-retire of the prior link TODO#765: mapping is not idempotent, so each run always creates a new authoritative link.
Add a seqrepo_data_proxy to the worker context in both the standalone context and the on_job_start hook, so jobs performing VRS translation have a SeqRepo-backed data proxy available alongside the cdot HGVS data provider. Mirror it in the mock worker context used by tests.
Add a reverse_translate_variants_for_score_set worker job that builds the cross-level HGVS equivalence class for every mapped variant in a score set, persisting the candidates as non-authoritative alleles. - for each current authoritative MappingRecord, collapse the assay-level HGVS to its protein consequence and expand to all coding/genomic candidates via a single batched construct_equivalent_variants call from the variant-annotation library, run off-loop in a thread - resolve each record's coding (NM_) transcript from its target gene's cdna alignment, falling back to a batched UTA NP_→NM_ lookup for protein-level mappings; records with no coding transcript are skipped - translate each candidate to VRS and write it as a get-or-created Allele linked non-authoritatively to the record, deduping by vrs_digest and never relinking the record's authoritative allele - supersede the prior live derived links with the new set in one gap-free operation via ValidTime.supersede_live_where - record per-variant cross_level_translation annotation status (success / failed / skipped), retaining per-candidate translation errors as metadata; the job fails only when every variant fails - add WorkerCoordinateTranslator and NullTranscriptSource adapters for the library's translation ports, deferring AssemblyMapper init so its network calls do not fire under mocked unit tests - get_or_create_allele helper, job registration, and a pipeline dependency placing reverse translation after mapping and before CAR submission TODO#765: a re-run supersedes the whole derived set wholesale because re-mapping re-mints the records; idempotent records would let unchanged derived links stay live.
…n date Previously the cdna-transcript lookup was keyed by target_gene_id alone, which meant a re-mapped score set could bind a stale NM_ transcript from an earlier run rather than the one the current run emitted. - Key cdna_transcript_by_run on (target_gene_id, mapped_date); within a key the highest-id row wins, so a same-run replacement takes precedence. - Carry mapped_date through the MappingRecord query so each record anchors to its own run's cdna row. - Add _TranscriptResolutionSkipReason to distinguish recoverable skips (protein-coding target, transcript unresolved) from correct skips (non-coding/regulatory target, no protein consequence). Emit skip_category in annotation_metadata. - Add target_gene_id to _TranscriptResolution so skip classification can look up the target's TargetCategory. - Add Mapped[] type annotations to TargetGene.id and .category columns. - New tests: genomic-accession coding target RT, latest-cdna-row-within- run selection, stale-cdna-row isolation, skip classification (coding recoverable vs. regulatory correct), and cdna TGM persistence in the mapping job. Refs mavedb-api#763
Forward translation emits predicted protein consequences in parentheses (e.g. p.(Ala222Val)). The parens denote inference, not a distinct variant form, so normalize to the bare p. form before storing. Strings without prediction parens are returned unchanged. Includes parametrized unit tests.
…st collisions - Re-identify each component via normalize_and_identify after translate_from so the stored vrs_digest reflects current content, not a stale value cached by the reused AlleleTranslator's Merkle tree. Without this, distinct biological variants at the same position (e.g. A>C and A>T) share one digest and are merged by the digest-keyed get_or_create_allele. - Use ga4gh_identify with in_place="always" in identify_allele and identify_variation so an allele that already carries a translator-stamped id has it recomputed, not retained. - Coerce ReferenceLengthExpression -> LiteralSequenceExpression in normalize_and_identify, mirroring dcd_mapping's _rle_to_lse, so RT-built alleles hash identically to the mapper's authoritative alleles and dedup correctly across sources. - Add regression tests: stale-id overwrite, distinct-alt digest isolation, cis-phased ordering canonicalization, and RLE->LSE coercion.
…tighten comments - Emit the protein consequence (result.hgvs_p) as a protein-level member of the equivalence set alongside the coding/genomic candidates. Prediction parens (p.(Ala222Val)) are stripped via strip_protein_prediction_parens before translation and storage. Protein-assay inputs are excluded (the protein is already the authoritative allele). - Trim all inline and docstring comments to essential why; remove redundant prose throughout reverse_translation.py. - Add tests: protein allele persistence, skip-classification for all three skip categories (no_assay_level_hgvs, transcript_unresolved, no_coding_transcript).
…rom failures Previously, variant mapping success was determined solely by the presence of pre/post-mapped alleles. This conflated genuinely failed mappings with benign absences (intronic variants, no-protein-consequence variants) that legitimately produce no allele. - Add `MappingOutcome` enum to `lib/mapping/schema.py` mirroring `dcd_mapping.schemas.MappingOutcome`, with an `is_benign_absence` helper to classify INTRONIC and NO_PROTEIN_CONSEQUENCE outcomes - Rewrite the per-record outcome logic in `map_variants_for_score_set` to branch on the typed outcome rather than allele presence: MAPPED -> SUCCESS, benign absence -> SKIPPED, FAILED -> FAILED - Replace the `successful_mapped_variants` scalar with a typed `Counter[MappingOutcome]` that feeds `mapped_count`, `failed_count`, and `skipped_count` tallies in logs and final state decision - Derive `mapping_state` from genuine failures only; all-benign result sets are treated as complete, not failed - Raise `NonexistentMappingResultsError` when a score annotation has no `outcome` field (malformed/older payload) - Expand test coverage for intronic and no-protein-consequence scenarios, verifying correct status and failure-category assignment
… UTA-backed source WtCodonMode.ALL reads the reference codon via TranscriptSource.codon_at, which requires a real UTA connection. The previous NullTranscriptSource always returned None, silently breaking WT-codon resolution. - Extract `uta_transcript_source()` context manager into `translation_ports.py`; removes the ad-hoc UTA connection setup that was duplicated in the job and the now-deleted NullTranscriptSource - Scope the live UTA client around the full `run_in_executor` call so the connection outlives the synchronous executor block - Remove NullTranscriptSource; callers that only need transcript_for_protein (already resolved by the job) also benefit from the real client without extra cost - Add a TODO noting that non-substitution consequences (del/ins/delins/ fs/ext) are miscounted as FAILED rather than SKIPPED (#767)
…ndle null gracefully - `translate_hgvs_to_variation` now attaches an `Expression` to each allele produced from a cis-phased or single HGVS string, mirroring the dcd_mapping authoritative-allele convention so `post_mapped` is self-describing without a separate round-trip. - `hgvs_from_vrs_allele` returns `None` instead of crashing when `expressions` is null or empty (valid for cis-phased block members), and `get_hgvs_from_post_mapped` propagates that as a `None` result. - `post_mapped` is now serialized with `exclude_none=True`, matching the mapper's output format. - Minor formatting clean-ups in reverse_translation.py (no logic change).
… model - `submit_score_set_mappings_to_car` now operates on Allele rows (authoritative + RT-derived) rather than MappedVariant, deduplicating by allele_id so each VRS allele is registered exactly once regardless of how many variants share it. Adds `force_reregister` param and per-allele outcome counters. - `submit_score_set_mappings_to_ldh` queries MappingRecord + Allele for pre/post-mapped data instead of the deprecated MappedVariant join. - `construct_ldh_submission_entity` signature updated to accept MappingRecord and Allele separately, since those fields now live on different models. - `warm_clingen_cache` switched to the shared `get_alleles_for_score_set` helper to keep allele scope consistent across all three jobs. - Extracts `get_alleles_for_score_set` and `ScoreSetAlleleRow` into `lib/clingen/alleles.py` as the single canonical query for both CAR and cache jobs.
get_hgvs_from_post_mapped can now join the members of a multi-variant block (Haplotype/CisPhasedBlock) into a single bracketed cis-phased expression via the new combine_cis flag, replacing the previous behavior of returning None for any multi-variant block. - combine_cis defaults off because some consumers cannot yet handle a bracketed expression — notably ClinGen submission, which has no single CAID for a multi-variant cis block (#764) - the CSV export fallback opts in, so g./p. HGVS columns are populated from cis-phased post-mapped output instead of left empty - drop the stale commented-out error branches and the resolved TODO
…-keyed refresh Replace MappedVariant-based gnomAD linkage with valid-time GnomadAlleleLink rows keyed on the deduplicated Allele. Linking covers every current allele of a score set (authoritative and RT-derived), so protein/coding score sets — whose genomic allele is RT-derived — are no longer dropped; per-variant annotation status flows through the _annotate_gnomad choke point against authoritative links only (interim bandaid, the AnnotationEvent migration seam). Refresh / idempotency model: - One live link per allele (unique index on allele_id); a gnomAD version bump supersedes rather than accumulating one live link per version. - Supersede only on change — an unchanged re-run leaves the live link untouched, so the valid-time history records no spurious boundary. - Version-keyed skip avoids re-fetching alleles already current at the version; a force param bypasses the skip (re-ingestion / heal) without churning unchanged links. - Per-variant status is a per-run audit event: created / preexisting / skipped. Normalize CAIDs across the Athena join: the gnomAD Hail dump drops leading zeros (CA025094 -> CA25094), so exact-string matching silently missed zero-padded CAIDs. Extract group_alleles_for_annotation as the shared allele-grouping primitive for allele-subject annotation jobs (adopted by gnomAD; VEP/ClinVar to follow). Refs #742. Partially addresses #722 (CAID-completeness tracking there stays open).
Migrate the VEP functional-consequence job (Step 2 of the annotation infrastructure migration) off MappedVariant onto the deduplicated allele model. - New ValidTime vep_allele_consequences table: a single allele-keyed row collapses record + link (the consequence is a scalar with no shared external entity, unlike gnomAD/ClinVar). One live consequence per allele via a partial unique index. - Job runs over the score set's full allele set (authoritative + RT-derived) via get_alleles_for_score_set + the shared grouping primitive; VAS still fans only to authoritative variants (the bandaid seam). - Version-key the refresh on the Ensembl release (/info/software): skip alleles already current, abort if the release can't be fetched. Supersede is value-keyed, not version-keyed — an unchanged consequence at a new release bumps source_version/access_date in place to avoid churning history. force bypasses the skip (e.g. after editing VEP_CONSEQUENCES). - A no-result is treated as a non-answer, never a negative: held consequences are not retired on an empty/failed VEP fetch. - Annotation links stay one-directional to Allele (no reverse back-ref). Adds lib + job tests covering linkage, RT-derived scope, version skip, in-place bump, supersede-on-change, no-result handling, and release-fetch failure.
Step 3 of the #742 external-annotation migration. ClinVar linkage moves off MappedVariant onto the deduplicated allele model. - Rename clinical_controls -> clinvar_controls (ORM model + table + unique constraint). Internal only: the ClinicalControl* view models, the /clinical-controls serving endpoints, and the frozen mapped_variants_clinical_controls association are unchanged, since the view-model name is both the OpenAPI schema name and the record_type discriminator consumed by the UI. - New clinvar_allele_links ValidTime table + ClinvarAlleleLink model. Multi-live: partial unique index (allele_id, clinvar_control_id) WHERE valid_to IS NULL, so an allele accumulates one live link per release. - Refactor refresh_clinvar_controls onto get_alleles_for_score_set + group_alleles_for_annotation (payload = CAID, full allele scope). Links are get-or-create; a same-version re-resolution to a different control supersedes newest-wins (gap-free retire+insert) rather than leaving two live links. VAS writes funnel through the _annotate_clinvar choke point, fanned only to authoritative_variant_ids and version-scoped. - Additively capture ClinVar's VariationID: nullable clinvar_variation_id column populated forward from the variant_summary TSV (parse degrades to None on archival schemas lacking the column). Unserved; the dedicated clinvar_variants remodel is deferred to the read-cutover. Tests rewritten for the allele model, covering the multi-live link writes, the version-scoped supersede guard, and the authoritative-only VAS fan-out.
Steps 4 & 5 of the #742 migration. Both jobs are redundant under the allele model: Allele.hgvs_g/c/p are populated by the mapping job, and the reverse-translation equivalence space (genomic/coding/protein alleles linked per MappingRecord) replaces the ClinGen PA<->CA translation table. - Remove populate_hgvs_for_score_set and populate_variant_translations_for_score_set from the pipeline DAG (both were leaf nodes — no dependency edges to repair), the worker registry (BACKGROUND_FUNCTIONS + STANDALONE_JOB_DEFINITIONS), and the external_services package exports. - Delete the two job modules and the now-orphaned ClinGen HGVS helpers (extract_hgvs_from_ca_allele_data / extract_hgvs_from_pa_allele_data), used only by the HGVS job. - Keep lib/variant_translations.py and the variant_translations table/model, marked FROZEN (serving-only) — they back old-model serving and are dropped at read-cutover. - Delete the obsolete job tests and their conftest fixtures.
get_allele_translations(db, allele_id, *, as_of=None) returns an allele's full cross-layer equivalence set (genomic/coding/protein) by traversing the MappingRecordAllele link graph: allele -> its live links -> mapping record(s) -> all co-linked alleles. The relation is co-membership in a MappingRecord's allele set, not a shared identifier — ClinGen's CAID spans only the nucleotide layers (the protein allele carries a distinct PA), so the link graph is the only thing tying all layers together. This replaces what the retired variant_translations PA<->CA table provided. Forward-compatible with temporal reads: the same half-open valid-time predicate is applied at both the anchor and fan-out hops, so passing as_of reconstructs the equivalence set as of any instant. Defaults to the currently-live set.
…bulary Introduces the AnnotationEvent log: a single append-only event table whose subjects are Variant and Allele, selected by annotation_type via a polymorphic CHECK. "Current" is derived (DISTINCT ON … id DESC), never stored. Adds the shared Disposition (present/absent/not_applicable/failed) and EventReason vocabulary, the v_current_annotation_events view, and the supporting migrations.
Add GET /alleles/{identifier}, the allele-grain sibling of GET
/variants/{urn}: anchor identity + the full cross-layer equivalence class
(each member labelled relative to the focus — projection / candidate /
convergent) + digest-keyed annotations. One path overloaded across a VRS
digest, a nucleotide CAID, or a protein PAID via a custom Starlette
convertor. Public and measurement-agnostic; no Cat-VRS (that stays rooted
at the measured allele on the variant view).
Unify the molecular core with variant detail: extract a shared
AlleleIdentity (lib + view model) used by both endpoints' `alleles` map,
and mark the anchored allele with `isFocus` — replacing the
`derivation: "authoritative"` value on variant detail (there is no
authoritative *variant* at the allele grain).
Factor the nucleotide-level set into SequenceLevel.NUCLEOTIDE_LEVELS,
shared by cat_vrs and allele_detail.
…_sets router score_sets.py had its own private `enqueue_pipeline_entrypoint` helper for building `Pipeline`/`JobRun` records and enqueuing the `start_pipeline` ARQ job, duplicated at both of its call sites. Move that logic into `mavedb.lib.workflow.kickoff.enqueue_pipeline_for_score_set` so the `run_pipeline` script and other future callers share the same enqueue contract instead of reimplementing it. - Restore the discard-pipeline-on-enqueue-failure safety net that the router's original helper had but the extracted version initially dropped, so a failed ARQ enqueue never leaves an orphaned pipeline behind in the database. - Add unit and integration tests for `enqueue_pipeline_for_score_set`, covering correlation-id precedence, param merging, ARQ dedup handling, and pipeline cleanup on enqueue failure.
…lines - Add run_score_set_pipelines.py, which selects unmapped or unenriched score sets and enqueues the appropriate pipeline for each, ordered by gene to maximize ClinGen CAR cache reuse - Refactor run_pipeline.py to share pipeline creation/enqueueing logic via the new enqueue_pipeline_for_score_set helper instead of duplicating PipelineFactory/redis wiring inline
…ost-mapped vrs get_hgvs_from_post_mapped only matched a bare "Allele" type, so post-mapped payloads wrapped in a VariationDescriptor fell through to the unhandled branch and returned None instead of their hgvs.
…variants Historical score sets have no mapping_records/alleles data yet, since that substrate is populated only by the live mapping job going forward. This reconstructs it deterministically from existing mapped_variants rows so the new serving/read layer (v_variant_annotations, published_variants MV) resolves for old data too, without redoing the reverse-translation fan-out (deferred to a decoupled enrichment pass). - add migrate_mapped_variants_to_allele_substrate manual migration, with verify/rollback/rebuild-annotations subcommands and idempotent re-run support - freeze the legacy MV-index migration's column signature as a literal instead of importing it from the model, since a later migration rewrites that model - add EventReason.MIGRATED for backfill-reconstructed annotation events, since a migration can't know the original creation history, only present/absent/failed
…ords The read layer (v_variant_annotations, published_variants MV, statistics router) still joined the legacy mapped_variants table, so it never resolved the mapping_records/alleles substrate the mapping job now writes to. This completes the read-side cutover onto the new tables. - rewrite v_variant_annotations to pivot the record's live allele links into the flat hgvs_g/hgvs_c/hgvs_p triple, and pull clingen_allele_id and VEP consequence from the authoritative allele, via correlated scalar subqueries - rebuild published_variants_materialized_view on mapping_record_id / current_mapping_record, keeping every record version (not just the live one) so history-aware counts still work - update statistics router and tests to the renamed columns - freeze all historical migrations' view/MV SQL as inline literals instead of importing from the model, so replaying an old revision builds the shape that existed at that point in history, not whatever the model looks like after later migrations rewrite it - add view-level tests for the flat-triple reconstruction, protein-only assays, and unmapped variants under the new substrate
link_gnomad_variants_to_alleles queried alleles once per gnomAD variant row inside the loop. The predicate wraps clingen_allele_id in regexp_replace to bridge zero-padded (MaveDB) and stripped (gnomAD dump, #722) CAIDs, which is non-sargable and forces a sequential scan of the alleles table on every execution. Over a linking run this becomes N full scans, saturating database IO (observed as sustained IO:DataFileRead above max vCPUs on the enlarged post-backfill alleles table). Resolve all rows in a single IN scan before the loop and group the result in Python by normalize_caid, so the loop reads from an in-memory map. N scans collapse to one per run, with no schema change. Grouping reuses normalize_caid on both the input and result sides, so the SQL and Python normalization forms are now exercised together. - add test_links_a_batch_of_rows_in_one_pass covering multi-row batching: distinct CAIDs, zero-padding match (#722), and shared-CAID fan-out preserved across a single call
The --select unenriched path enqueues annotate_score_set, whose reverse-translation step needs a transcript. It previously selected any mapped-but-not-enriched score set, so ones with no usable transcript were re-selected on every run and burned worker cycles skipping every variant as transcript_unresolved. - split _needs_enrichment into _mapped_without_enrichment and a new _has_usable_transcript predicate (cdna TargetGeneMapping reference_accession, or a live NP_/XP_ RefSeq protein MappingRecord), mirroring the two transcript sources in reverse_translation.py - require a usable transcript for unenriched eligibility - report the count of mapped score sets skipped for lack of a transcript so they are visibly dropped rather than silently lost
- add preMapped/postMapped VRS pair to VariantDetail and switch
GET /variants/{urn} to exclude_none=False so absent fields
serialize as null instead of being dropped
- add streaming NDJSON GET /score-sets/{urn}/variant-details,
replacing the retired GET /score-sets/{urn}/mapped-variants
- replace MappedVariant-keyed gnomAD lookups with a substrate-based
get_gnomad_variants_with_variant_urns, pairing each gnomAD variant
with the score-set variant URNs it links to
(GnomADVariantWithVariantLinks)
- add as_of time-travel support to the CSV variants export and the
gnomad-variants endpoint, echoed via X-As-Of response headers
- extract mapped_hgvs_by_level/get_protein_hgvs_by_record helpers
out of score_set_variants for reuse by the CSV export
- replace the MappedVariant-keyed CSV joins with a substrate query (_csv_substrate_query) over the live mapping record, its authoritative allele, and the projection-group nucleotide sibling - add _gnomad_by_allele/_clinvar_by_allele batch fetches keyed by allele id, replacing the old mapped_variant_id-keyed lookups - reuse mapped_hgvs_by_level/get_protein_hgvs_by_record from score_set_variants so the CSV resolves post-mapped HGVS per level the same way the lean whole-set view does, instead of falling back to a per-row VRS parse - extract column planning (_plan_csv_columns) and header assembly (_csv_header_columns) out of get_score_set_variants_as_csv - thread as_of through the CSV export to time-travel the annotation layer (post-mapped HGVS, VEP, gnomAD, ClinVar) independently of the immutable scores/counts namespaces - add seed_csv_substrate test helper for the substrate-based fixtures
The mapped-variant endpoints removed earlier on this branch are still
live on main, so merging is what actually retires them for public API
consumers. Soften that landing per-route, based on whether the
replacement is wire-compatible:
- GET /score-sets/{urn}/mapped-variants now returns 410 Gone pointing
at variant-details, since the streaming NDJSON shape isn't
compatible with the old JSON array and a redirect would mislead
callers
- /mapped-variants/{urn} and its va/* and vrs/{identifier} siblings
301-redirect onto their /variants equivalents, since that move was a
straight relocation with an unchanged response shape
- POST /variants/clingen-allele-id-lookups stays fully removed with no
stub; it was effectively an internal-only route
- add a reusable 410 entry to routers/shared.py's BASE_RESPONSES
The include_nucleotide_siblings flag never actually gated siblings — reverse translation cross-links every record to its full synonymous nt set, so a sibling's record already links the anchor allele. Remove the flag (api + query param) and make the protein-apex fold-in unconditional for a CA query, so one behavior serves both the variant page and search: a CA reaches its protein consequence even when that protein assay hasn't been reverse-translated (its record links only the protein node). Resolve the apex once in _resolve_protein_apex, and instrument it: warn on a divergent apex (>1 PAID), and publish apex PAID/unregistered counts and the protein-consequence / apex-only-unresolved rates to the request log. Update tests to the one-behavior model.
Even with higher timeout settings, some VEP jobs were still hitting against the timeout window. Given the new checkpoint features, increasing the coarse timeout is lower risk and, in the aggregate, will help more jobs run to completion.
MaveDB's Cat-VRS relation vocabulary is member->defining framed and cannot be expressed as a subclass of the spec's own Relation enum, so interop was previously implicit (matching code strings only). Codes that are an exact match for a published Cat-VRS Relation term now carry an explicit ConceptMapping to it, so downstream consumers can resolve equivalence without guessing from naming. - extract lib/term_systems.py: a shared (system URI, prefix) -> Coding builder, replacing the single ad hoc _RELATION_SYSTEM constant so id/system pairs can't drift apart - add _SPEC_TERM_SYSTEM / _SPEC_EQUIVALENT tables in lib/cat_vrs.py mapping MaveDB relation codes to their Cat-VRS spec equivalents where one exists - add drift-guard tests asserting every MaveDB relation is either mapped or a declared gap, and that newly published spec terms get triaged
…coding "Sibling" and "synonymous cousin" described two unrelated relationships with confusingly similar names: a projection_group partner (the c<->g pair of one measured change) versus a convergent but distinct nt allele that happens to encode the same protein consequence. Rename both throughout code and identifiers to the terms the domain docs already use elsewhere: - projection_group partner: sibling_* -> projection_* (variables, columns aliases, function params) in score_set_variants.py, score_sets.py, and lib/allele_detail.py - distinct nt allele sharing a protein consequence: cousin -> convergent encoding, including is_convergent_cousin -> is_convergent_encoding in lib/cat_vrs.py and its call sites in variant_detail.py - update the worker-job error strings that referenced "nucleotide siblings" to match Behavior is unchanged; this is a pure rename.
No identifier or behavior changes. Rewords the sibling/cousin terminology in prose (see the companion refactor commit for code identifiers), cuts repetitive design-history narration, and fixes one inverted claim in lib/allele_measurements.py's module docstring: a protein assay's consequence is reachable through the shared protein node precisely when it has *not* yet been reverse-translated, not "even when" it has.
…pping-and-storage
…pping-and-storage
vrs_digest asserts that an allele's identifier is the GA4GH digest of its own post_mapped content, but nothing enforced it: writers adopted the id minted by the producer, which for mapped variants is computed before normalization. Because vrs_digest is a UNIQUE dedup key on an immutable, ValidTime-less table, a wrong value there cannot be corrected in place afterwards. Add canonical_variation_document() to re-derive identity and a single canonical serialization for any variation arriving from outside, and route both the live mapping ingest and the mapped_variants backfill migration through it so identity is recomputed rather than inherited. Add the audit_allele_identifiers script to measure drift before and after the backfill and to repair safely-correctable rows.
The API contribution stamped date=datetime.today(), which is not provenance: it records when a view was serialized, not when anything was contributed. What this contribution asserts is which software produced the annotation, and that is the agent's version. The wall-clock stamp also made the object unique per call, so identical provenance never deduplicated and va.ndjson changed bytes on every run over unchanged data, defeating checksum manifests and incremental sync for consumers of the public dump.
functional_consequence has been the only stored fact about a VEP result: no record of which transcript_consequences entry it came from, or whether it was a transcript-specific match at all versus VEP's cross-transcript most_severe headline, which regularly describes a different overlapping isoform than the allele's own transcript (#772). Add four nullable columns to vep_allele_consequences: consequence_terms (every term from the matched transcript entry, severity-ordered), consequence_source (transcript / most_severe / reference_identical, mirrored from variant_annotation.lib.vep.ConsequenceSource by VepConsequenceSource), matched_transcript (set only when the source is transcript), and resolver_version (variant_annotation.lib.vep.RESOLVER_VERSION at write time). resolver_version pairs with the existing source_version as the second axis the current-release skip keys on, so a resolution-rule fix re-queries every allele instead of looking current forever under an unchanged Ensembl release. All nullable with no backfill: a legacy NULL never matches the current resolver_version, so it is re-queried once and filled in place.
…rovenance lib/vep.py's local VEP_CONSEQUENCES ranking, get_functional_consequence, and run_variant_recoder duplicated variant_annotation.lib.vep's resolution rule under a different, staler ranking. Drop them in favor of resolve_consequences over the shared library, so this job and the offline CLI pipeline resolve a consequence identically by construction, and write the new provenance columns (consequence_terms, consequence_source, matched_transcript, resolver_version) in link_vep_consequences_to_alleles alongside the existing headline-term write. Add VepLinkVerdict.RETAINED_ON_ABSENCE: when VEP returns nothing for an allele that already has a live consequence, the prior value is kept rather than overwritten with a null, since a settled negative from one run should not erase a previously confirmed positive. The worker job now builds a transcript-aware VepInput per allele (only coding and genomic alleles are submitted; a protein allele's consequence is carried by the coding alleles its reverse translation enumerates from, so it is never VEP queried directly), wires in UTA's coding_interval_reference for best-effort reference-identical detection, and gates the current-release skip on resolver_version in addition to source_version so a resolution-rule change re-queries every allele instead of looking current forever. Drops the protein/Recoder end-to-end test now that protein alleles are never submitted.
…anscript-matched populate_vep_for_score_set was disabled (#772) because its consequences came from VEP's top-level most_severe_consequence, the worst call across every transcript overlapping a position rather than the one the variant was mapped to. Now that the worker job resolves against the allele's own transcript via the shared resolution kernel, re-add the job to the pipeline definition. Depend on reverse_translate_variants_for_score_set rather than the old submit_score_set_mappings_to_car: VEP needs the coding/genomic HGVS reverse translation produces, not confirmation that mappings were submitted to CAR.
…skip, not a failure (#767) Split the reverse-translation error loop on the library's typed reason: NOT_TRANSLATABLE maps to (not_applicable, not_translatable) and is counted as skipped; every other error keeps the failed path. Adds the NOT_TRANSLATABLE event reason. This keeps benign "nothing to translate" variants out of the FAILED tally, which the retroactive backfill (#747) depends on for a trustworthy failure count.
…ss retired MappedVariant surfaces
Add a shared deprecation helper and route every retired surface through it, so the
/mapped-variants 301 redirects, the /score-sets/{urn}/mapped-variants 410 tombstone, and the
deprecated CSV query parameters all emit consistent Deprecation / Link / Warning headers and
record each hit under one queryable deprecation_marker. The redirects name their 1:1 successor
as rel=successor-version; the tombstone (no wire-compatible successor) points at variant-details
via Warning. Sunset stays unset until a removal date is ratified. The usage marker makes 'who is
still on a deprecated surface' answerable before that date is chosen.
…e for its deprecation window
mapped/{urn}.mapped-variants.json is the last dump artifact sourced from the frozen MappedVariant
table; vrs/{urn}.vrs.ndjson supersedes it with a re-shaped (correct) representation. Gate its
emission on EXPORT_LEGACY_MAPPED_VARIANTS (default on) so it drops cleanly at the chosen version
boundary -- one operational change, not a code edit -- and removes with the table drop. Record the
deprecation and the re-shape rationale in the dump changelog.
…concile gate backfill_campaign.py adds two read-only commands for the MappedVariant -> allele-substrate backfill. 'coverage' reports each published score set's furthest lifecycle state (none -> partial_reshape -> reshaped -> rt -> enriched) with a roll-up. 'reconcile' is the gate that blocks dropping the frozen mapped_variants tables: it asserts measured-level coverage parity, zero per-variant gnomAD/ClinVar/VEP annotation regression, and no query reader of the legacy tables outside the migration/export allowlist, exiting non-zero unless the drop is safe.
…pping run (#763) Reverse translation resolved a variant's coding transcript from cdna TargetGeneMappings keyed by (target_gene_id, mapped_date). mapped_date is day-granular, so two mapping runs of one score set on the same calendar day were indistinguishable: a later same-day remap that emitted no cdna row would bind the earlier run's stale transcript instead of correctly skipping. Record the producing JobRun on each TargetGeneMapping (nullable FK, ON DELETE SET NULL) and resolve a record's transcript by its own run via (job_run_id, target_gene_id), so a run with no cdna row resolves to nothing and the variant is skipped as transcript-unresolved. Rows with no job_run_id (pre-column, or reshaped from legacy) fall back to the day-granular key. The resolution is contained in a single _cdna_transcript_resolver factory. Adds same-day two-run and mapped_date-fallback regression tests.
The dev dependency allowed ruff ~0.15.0 while the pre-commit hook pins v0.6.8, and the two format some idioms differently (dict-argument call hugging, chained-call breaks). Since the hook is what gates commits, running a newer local ruff rewrote code the hook then reverted, thrashing diffs. Pin the dev dependency to 0.6.8 so poetry run ruff and the hook agree; bump both together in future.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allele-centric data model: deduplicated VRS alleles, valid-time versioning, and annotation events
Replaces MappedVariant as the source of truth for mapping/annotation with a deduplicated, content-addressed Allele model, and moves every downstream annotation job (VEP, ClinVar, gnomAD, ClinGen CAR/LDH) onto it.