From bfabd4ffdc1c380b9d591d9488fc52b524c7b1b7 Mon Sep 17 00:00:00 2001 From: callumalpass Date: Tue, 10 Nov 2026 00:39:10 +1100 Subject: [PATCH] Infer a missing seed origin and merge equal-length lists (05A) Connect's engine (mdbase-connect #1123, #1228) upgrades an edited seed type whose lock records no origin, as collections set up by the legacy installer have, instead of leaving it stuck. Make that the rule: such a seed merges against the nearest listed baseline that merges cleanly into a type still implementing the desired contracts; one that already implements them is kept without a reason. Type documents compare by frontmatter values and body, not bytes, and equal-length arrays merge item by item. The reference model follows, and a conformance case covers the inferred merge. --- 05a-data-contracts.md | 48 ++++++++++++++------ CHANGELOG.md | 12 +++-- scripts/type_pack_model.py | 64 ++++++++++++++++++++++++--- tests/v0.3/type-packs/type-packs.yaml | 61 ++++++++++++++++++++++++- 4 files changed, 160 insertions(+), 25 deletions(-) diff --git a/05a-data-contracts.md b/05a-data-contracts.md index ae20025..a5325c9 100644 --- a/05a-data-contracts.md +++ b/05a-data-contracts.md @@ -395,30 +395,52 @@ The lock records it (see Pack Identity And Portable Provenance). When the target exists, is not an intentionally preserved seed target, and the resource declares `upgrade_from`, the engine plans exactly one of these, in order: -1. The live bytes equal the desired document: `preserve`. -2. The live bytes equal a baseline's document: `update`, writing the desired - document byte-for-byte. Byte equality proves the origin, whatever the lock +Two type documents are **equal** when their frontmatter values are equal and +their Markdown bodies are identical; YAML formatting and key order do not +matter, because other writers may re-serialize a type file. + +1. The live type equals the desired document: `preserve`. +2. The live type equals a baseline's document: `update`, writing the desired + document byte-for-byte. Equality proves the origin, whatever the lock records. 3. The origin is the desired document: `preserve`. The seed was already upgraded and has been edited since. 4. The origin is a baseline: `update` by a three-way merge of that baseline, the live type, and the desired type. -5. Otherwise, because the origin is unknown or is not a listed baseline: - `preserve`, with a `reason` stating that no upgrade baseline applies. The - type is left as it is; it does not make the pack conflict. - -An engine MUST NOT choose a merge baseline any other way. In particular, it MUST -NOT merge against a baseline the seed is not known to descend from, because the -differences between that baseline and the seed's actual origin would be applied -as if they were the user's edits. The assessment reports, for every seed -`update`, the baseline used as `upgrade_baseline: { digest, version? }`. +5. Otherwise the origin is unknown (a lock written before origins were + recorded, or none) or is not a listed baseline, and the engine **infers** a + merge base. It orders the listed baselines by how many frontmatter leaf + values differ from the live type's (a key present on one side only counts + once; equal-length arrays compare item by item; any other differing value + counts once), fewest first and in listed order on a tie. It takes the first + baseline whose three-way merge succeeds and whose merged type validates and + implements every exact contract version the desired type does, judged over + the pack's own contract resources: `update` with that merge, or `preserve` + when the merge changes nothing. +6. Otherwise, when the desired type declares `implements` and the live type + already validates and implements every exact contract version the desired + type does: `preserve`. The seed already fits the pack. +7. Otherwise: `preserve`, with a `reason` stating that no upgrade baseline + applies. The type is left as it is; it does not make the pack conflict. + +In cases 1, 2, 3, and 5 the lock's origin becomes the desired document; in the +others it is carried forward. An engine MUST NOT choose a merge baseline any +other way. An inferred base (case 5) may not be the seed's true origin, so the +differences between them could be applied as if they were the user's edits; the +conservative merge bounds this, since a seed whose `version` or other setting +differs from both sides is a competing change and fails. The assessment +reports, for every seed `update`, the baseline used as +`upgrade_baseline: { digest, version? }`, so the change is reviewed like any +other. A three-way merge is conservative. The type kind and name MUST match. Unchanged publisher settings retain live customizations; unchanged live settings accept publisher changes. Object settings merge recursively. Contract implementations merge by contract ID only when unambiguous, retaining customized field mappings and bindings while updating exact version references. Competing changes (including -ambiguous implementation lists) MUST fail closed. Other arrays are indivisible. +ambiguous implementation lists) MUST fail closed. Other arrays merge item by +item when the baseline, live, and desired arrays have the same length, and are +otherwise indivisible. Missing values and explicit null are distinct. Top-level setting removal requires separate manual review. The live Markdown body and unrelated YAML nodes MUST be preserved. Reformatting a changed YAML node is permitted. diff --git a/CHANGELOG.md b/CHANGELOG.md index 1355ff6..8c7b914 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -195,10 +195,14 @@ Bases, seed type upgrades, and saved-view identification. baselines, each a digest-pinned starter the publisher previously shipped (`{ digest, document, version? }`), so a collection can upgrade from any supported starter, not only the latest. The engine chooses the merge - baseline from the seed's origin and never guesses: an unedited seed equal to - any baseline is replaced with the exact desired bytes; an edited seed merges - against the baseline it descends from; a seed whose origin is unknown or not - listed is preserved with a reason rather than merged or made to conflict. + baseline from the seed's origin: an unedited seed equal to any baseline is + replaced with the exact desired bytes; an edited seed merges against the + baseline it descends from. A seed whose origin is unknown or not listed + merges against the nearest listed baseline that merges cleanly into a type + still implementing the desired contracts; failing that, it is kept, with a + reason unless it already implements them, rather than made to conflict. + Type documents compare by frontmatter values and body, not bytes, and + equal-length arrays merge item by item. Competing changes still fail closed, publication is atomic, and the assessment reports the baseline used. Engines that do not support the member must reject the manifest. Records are never migrated this way. diff --git a/scripts/type_pack_model.py b/scripts/type_pack_model.py index f87a1d1..c8daafc 100644 --- a/scripts/type_pack_model.py +++ b/scripts/type_pack_model.py @@ -4,7 +4,9 @@ conformance fixtures exercise: manifest validation of `upgrade_from`, the lock's seed `origin_digest`, the ordered choice of an upgrade baseline, and a structural three-way merge of type frontmatter that keeps the live body. -Contract-implementation merging and other engine detail are out of scope. +Contract validation, contract-implementation merging and other engine detail +are out of scope: an inferred merge "still implements the desired contracts" +when its `implements` entries cover the desired type's. """ from __future__ import annotations @@ -195,27 +197,47 @@ def _plan(self, resource: Resource, entry: dict[str, Any] | None) -> Planned: if entry is None: return Planned(resource.target, "create", resource.document, resource.digest) return Planned(resource.target, "preserve", None, previous_origin) - if live == resource.document: + if same_type_document(live, resource.document): return Planned(resource.target, "preserve", None, resource.digest) if not resource.declares_upgrade: return Planned(resource.target, "preserve", None, previous_origin) for baseline in resource.baselines: - if live == baseline.document: + if same_type_document(live, baseline.document): return Planned(resource.target, "update", resource.document, resource.digest, baseline=baseline) if previous_origin == resource.digest: return Planned(resource.target, "preserve", None, previous_origin) baseline = next((item for item in resource.baselines if item.digest == previous_origin), None) if baseline is None: - return Planned( - resource.target, "preserve", None, previous_origin, - reason=f"{resource.target}: no upgrade baseline applies to this type's origin", - ) + return self._infer(resource, live, previous_origin) try: merged = merge_documents(baseline.document, live, resource.document) except MergeConflict as conflict: return Planned(resource.target, "conflict", None, previous_origin, reason=f"{resource.target}: {conflict}") return Planned(resource.target, "update", merged, resource.digest, baseline=baseline) + def _infer(self, resource: Resource, live: bytes, previous_origin: str | None) -> Planned: + """No known origin: merge against the nearest listed baseline that merges cleanly.""" + live_frontmatter, _ = split_document(live) + candidates = sorted( + resource.baselines, + key=lambda baseline: differences(live_frontmatter, split_document(baseline.document)[0]), + ) + for baseline in candidates: + try: + merged = merge_documents(baseline.document, live, resource.document) + except MergeConflict: + continue + if implements_as_desired(merged, resource.document): + if same_type_document(merged, live): + return Planned(resource.target, "preserve", None, resource.digest) + return Planned(resource.target, "update", merged, resource.digest, baseline=baseline) + if "implements" in split_document(resource.document)[0] and implements_as_desired(live, resource.document): + return Planned(resource.target, "preserve", None, previous_origin) + return Planned( + resource.target, "preserve", None, previous_origin, + reason=f"{resource.target}: no upgrade baseline applies to this type's origin", + ) + def _plan_managed(self, resource: Resource, entry: dict[str, Any] | None, live: bytes | None) -> Planned: if live is None: return Planned(resource.target, "create", resource.document, None) @@ -226,6 +248,32 @@ def _plan_managed(self, resource: Resource, entry: dict[str, Any] | None, live: return Planned(resource.target, "conflict", None, None, reason=f"{resource.target} changed") +def same_type_document(a: bytes, b: bytes) -> bool: + """Equal as type documents: the same frontmatter values and the same body.""" + if a == b: + return True + try: + return split_document(a) == split_document(b) + except ValueError: + return False + + +def differences(a: Any, b: Any) -> int: + """How many frontmatter leaves differ, for choosing an inferred merge base.""" + if isinstance(a, dict) and isinstance(b, dict): + return sum(differences(a[key], b[key]) if key in a and key in b else 1 for key in {*a, *b}) + if isinstance(a, list) and isinstance(b, list) and len(a) == len(b): + return sum(differences(x, y) for x, y in zip(a, b)) + return int(a != b) + + +def implements_as_desired(candidate: bytes, desired: bytes) -> bool: + def implemented(document: bytes) -> set[tuple[str, str]]: + entries = split_document(document)[0].get("implements") or [] + return {(str(entry.get("contract")), str(entry.get("version"))) for entry in entries if isinstance(entry, dict)} + return implemented(desired) <= implemented(candidate) + + def merge_documents(base: bytes, live: bytes, desired: bytes) -> bytes: base_frontmatter, _ = split_document(base) live_frontmatter, live_body = split_document(live) @@ -257,4 +305,6 @@ def merge_values(base: Any, live: Any, desired: Any, path: str) -> Any: if value is not MISSING: merged[key] = value return merged + if all(isinstance(value, list) for value in (base, live, desired)) and len(base) == len(live) == len(desired): + return [merge_values(b, l, d, f"{path}/{index}") for index, (b, l, d) in enumerate(zip(base, live, desired))] raise MergeConflict(f"competing changes at {path or '/'}") diff --git a/tests/v0.3/type-packs/type-packs.yaml b/tests/v0.3/type-packs/type-packs.yaml index b94c93c..1ee73b1 100644 --- a/tests/v0.3/type-packs/type-packs.yaml +++ b/tests/v0.3/type-packs/type-packs.yaml @@ -338,7 +338,7 @@ groups: --- tests: - id: seed-upgrade-unknown-origin-preserved - name: "a user-owned type at a seed target is never merged into a starter" + name: "a user-owned type at a seed target that merges cleanly with no starter is preserved" operation: apply_type_pack input: pack: "examples/v0.3/seed-upgrades/v3/mdbase-pack.yaml" @@ -360,6 +360,65 @@ groups: covers: - type_packs.seed_type_upgrade + - name: "an edited seed with no recorded origin merges against the nearest baseline" + setup: + config: | + spec_version: "0.3.0" + settings: + validation: error + files: + _types/note.md: | + --- + kind: mdbase.type + name: note + version: 1 + description: A note. + match: + where: + type: note + schema: + dialect: json-schema-2020-12 + value: + type: object + required: [title] + properties: + title: { type: string } + mood: { type: string } + additionalProperties: true + --- + # Note + + My notes, as I keep them. + tests: + - id: seed-upgrade-unknown-origin-inferred + name: "the v1 baseline is nearest, merges cleanly, and keeps the user's edits" + operation: apply_type_pack + input: + pack: "examples/v0.3/seed-upgrades/v3/mdbase-pack.yaml" + history: + - apply: "examples/v0.3/seed-upgrades/v1/mdbase-pack.yaml" + expect: + valid: true + runs: + - valid: true + status: upgrade + actions: [update] + resources: + - target: _types/note.md + action: update + upgrade_baseline_version: 1 + target_frontmatter: + _types/note.md: + /version: 3 + /schema/value/properties/created: { type: string, format: date-time } + /schema/value/properties/mood: { type: string } + target_body_contains: + _types/note.md: ["My notes, as I keep them."] + lock_origin: + _types/note.md: "examples/v0.3/seed-upgrades/v3/_types/note.md" + covers: + - type_packs.seed_type_upgrade + - name: "upgrade baselines are validated" setup: config: |