Skip to content

task: Benchmark production prepare and frozen resource closure #430

Open

Description

@zengchuang-hw

Current state: Apps prepare archive is merged in #424. Apps execution binding has merged in #427.
Decision needed: Confirm the Benchmark production implementer and first supported CPU task/worker for this slice.
Next action: Confirm the first task and worker. The D023 CPU fixture is a candidate. Verify the real prepare output with the Apps consumer before integrating with #429.

Goal and acceptance

Provide the Benchmark prepare(PrepareRequest) and pure prepared encoding/validation ports needed for #371 step 2. Follow #387 and D023-1.

  • Return a fixed PreparedEvaluation, the canonical frozen protocol bytes, and a read-only resource entry keyed by role.
  • Freeze task/grader revision, candidate contract, worker recipe and digest, and the required resource pins. Benchmark computes metadata from the exact supplied read-only bytes or verifies existing fixed metadata. Apps independently verifies byte size and SHA-256 when committing the archive.
  • Cross-test production output with apps.bridge.verify_protocol. Relocating an outer reference on PreparedEvaluation while keeping the protocol and resource bytes fixed preserves prepared_id. Changing an ArtifactRef location inside the canonical ProtocolV1 changes protocol.sha256 and therefore prepared_id, while preserving protocol_digest because the semantic projection strips that declared location. Ref-shaped task parameters stay in the semantic projection. Verify all cases.
  • Verify T36 with real preparation and cross-process readback. A required resource unavailable during Benchmark preparation yields ArtifactUnavailable at the prepare boundary. If it becomes unavailable at Apps archive time, the outcome is ArtifactUnavailable at prepared_archive. In both cases no closure or PreparedRef is published.

Production offline assess belongs to #371 step 3. #414 tracks the legacy run/evaluate split.

Activity

  1. added
    kind:taskBounded delivery task with a delegated slice
    status:proposedWaiting for the decision named in the status block
    on Nov 13, 2026
  2. amy-why-3459 commented on Nov 13, 2026

    @amy-why-3459
    Collaborator

    Thanks for opening this. Some notes from reviewing the consumer side in #427 (approved on 6778eb19), whose verify_protocol is what Benchmark's output must satisfy:

    1. Keep the identity projection identical on both sides. On the Apps side, the protocol digest is computed over ProtocolV1 without display_notes. location is stripped only from the schema-declared ArtifactRef paths (baseline, reference, quality_reference, resources[].artifact), and ref-shaped values in task_parameters are kept. prepared.resources must equal the role-sorted closure, and roles must be unique; identical duplicates are rejected too since 6778eb19. If Benchmark's pure encoding re-implements this projection, the two will drift. Please either share one function or add a cross-test where Benchmark-produced bytes pass apps.bridge.verify_protocol unchanged.
    2. Relocation test. For "semantic identity under location-only relocation", a direct check would be: relocate every resource location and assert the digest is unchanged; change any non-location field and assert it changes.
    3. Missing required resource. Please name the expected error code and stage, and state that nothing is committed to the Apps archive (no partial closure).
    4. Purity vs. hashing. If prepare is pure and Apps supplies the final location, who reads resource bytes to compute sha256/size_bytes? Is it Benchmark through the read-only port, or Apps at commit time? Pinning this down avoids a gap where neither side verifies them.
    5. First task/worker. For the decision, one option is to start with the task behind the existing D023 golden / t108-v4 fixtures. Then the Apps tests in Archive complete Apps preparations #424/Bind saved Apps requests to frozen worker specs #427 can swap the injected port for the real one without new fixtures. That's a suggestion; the owner should decide.

    Linking the implementing PR here once it exists would help track T36 end to end with #429.

  3. amy-why-3459 commented on Nov 13, 2026

    @amy-why-3459
    Collaborator

    Thanks, the updated acceptance covers the earlier points. It now has a cross-test against apps.bridge.verify_protocol, relocation tests in both directions, a split of byte metadata ownership (Benchmark computes or verifies it, Apps re-verifies it at commit), and ArtifactUnavailable at both the prepare and prepared_archive boundaries with no published closure.

    One wording point in the new second bullet, to avoid a contradiction at implementation time: a location-only change also changes the frozen protocol bytes, and therefore the protocol ArtifactRef's sha256/size_bytes. As written, "changes to bytes inside the frozen protocol … update the byte digest and prepared ID" could be read as making the prepared ID change on relocation too. Could you state which identity each change moves? My reading of #427:

    • byte digest (protocol.sha256): changes on any byte change, including relocation;
    • semantic protocol_digest and prepared ID: unchanged on location-only changes, changed by any non-location field change, including inside a resource description.

    If the prepared ID is meant to follow the bytes rather than the semantic digest, that would conflict with the first half of the same bullet, so it's worth confirming either way.

  4. amy-why-3459 commented on Nov 13, 2026

    @amy-why-3459
    Collaborator

    Thanks for the clarification in the second acceptance bullet. It resolves the question from my last comment, and it corrects my earlier reading. I had assumed prepared_id stays fixed on any location-only change. The new text says it changes when the location sits inside the canonical ProtocolV1. That matches the contract on main: in docs/contracts/schema-and-semantics.md, the prepared_id row (L53), the D023-1 projection boundary (L62), and V31 strip only outer, schema-declared ArtifactRef.location and keep protocol.sha256 in the projection. So the three identities now move as follows:

    • outer ref relocation on PreparedEvaluation (protocol/resource bytes fixed): prepared_id unchanged;
    • location change inside ProtocolV1: protocol.sha256 and prepared_id change, protocol_digest unchanged;
    • any non-location field change: all three change.

    One small test suggestion: please cover the outer case for resources[].artifact as well as protocol. That way the "preserves prepared_id" assertion isn't checked only on the protocol ref.

    The status-line update (#427 merged) is noted. No further concerns on the description.

  5. zengchuang-hw commented on Nov 13, 2026

    @zengchuang-hw
    ContributorAuthor

    已按 D023 明确两层 relocation。只移动 PreparedEvaluation 外层引用且协议和资源原始字节不变,prepared_id 保持不变。若改的是 ProtocolV1 内的资源 locator,协议语义投影会剥离该字段,因此 protocol_digest 保持不变,但协议原始字节的 protocol.sha256 会变化;这个字节摘要参与 prepared_id,因此 prepared_id 也会变化。验收会覆盖这两种情况。

  6. Dong1017 commented on Nov 14, 2026

    @Dong1017
    Collaborator

    Decision — first CPU task + Benchmark implementer (Dong-bot as @Dong1017 delegate)

    Recorded 2026-11-15 (Asia/Shanghai). Authority: Dong-bot decision authority for @Dong1017 accept/decide. Thread basis: Amy acceptance polish complete (6856558760); Amy’s first-task option (6852857965); #362 S3 row still Unassigned.

    Accepted — first supported CPU task / worker

    D023 golden / t108-v4 fixture line is the first supported production CPU task/worker for this prepare slice. Same task drives #434 assess and pairs with #429’s D023 evaluation-mode ExecutionSpec (null step timeout variant already posted on #429). Apps consumers in #424/#427 can swap fixture ports for the real prepare output without inventing a new fixture family.

    Decision — Benchmark production implementer

    Not assigned in this comment. #362 still records S3 Benchmark production-slice ownership as unassigned; Draft #338 is explicitly not confirmation of full S3 ownership. Inventing a name here would contradict that record.

    Next action required before coding starts: name a delivery assignee for this bounded prepare / encode / validate slice (may be a delegated PR author without conferring full S3 module ownership), and optionally update the #362 coordination table. Until that name is recorded, keep this issue status:proposed.

    Acceptance text for the prepare port itself is otherwise clear (cross-test apps.bridge.verify_protocol, two-layer relocation, ArtifactUnavailable at prepare vs prepared_archive).

  7. zengchuang-hw commented on Nov 15, 2026

    @zengchuang-hw
    ContributorAuthor

    @Dong1017 The paired Apps consumers have now merged (#424, #427 and #444). #429 has an accepted Harness owner, and D023 / t108-v4 already gives #430 and #434 one common CPU contract. Please record a delivery assignee (one for the pair, or one per slice) on #430 / #434. A scoped implementation delegation is enough to start the two Benchmark PRs without assigning the entire S3 module. Once the production ports are available, I'll wire the Apps services and run the shared T36/T37/T39/T40 acceptance.

  8. Dong1017 commented on Nov 15, 2026

    @Dong1017
    Collaborator

    Decision — Benchmark delivery assignee for #430 / #434 (Dong-bot as @Dong1017 delegate)

    Recorded 2026-11-15 (Asia/Shanghai). Authority: store preferences.md Dong-bot decision authority for @Dong1017 accept/decide. Reply to @zengchuang-hw: Apps consumers (#424 / #427 / #444) and #429 Harness owner are in; please name a delivery assignee for the pair.

    Accepted — scoped delivery assignee (one for the pair)

    Delivery assignee for production Benchmark prepare (#430) and assess (#434): @Dong1017.

    Scope of this assignment:

    • Bounded PR delivery of the production ports against the already-accepted first CPU task (D023 / t108-v4) and the accepted task: Benchmark production offline assess for Apps #434 public port boundary (6885787031).
    • Accountable implementation + delivery review for these two slices. PR authorship may be further delegated; accountability stays with this assignee.
    • Enough to open the two Benchmark PRs and let Apps wire T36 / T37 / T39 / T40 once the ports land.

    Explicit non-assignment

    This is scoped implementation delegation only. It does not:

    Status

    Flipping status:proposed → status:accepted for this bounded #430 prepare delivery. Cross-post on #434. Optional follow-up: note the same scoped delivery line on #362 without rewriting the S3 owner cell.

  9. added
    status:acceptedThe decision owner accepted it in a recorded comment
    and removed
    status:proposedWaiting for the decision named in the status block
    on Nov 15, 2026
  10. self-assigned this
    on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind:taskBounded delivery task with a delegated slicestatus:acceptedThe decision owner accepted it in a recorded comment

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions