THOTH-GQL-OPS-03: add effective-mode fleet verification

[ja573]

STATUS: BLOCKED — not ready for review as a completed mechanism

Independent exact-head review 4906399962 returned CHANGES REQUIRED against head 8604dbf9. Both findings are correct. Finding 1 is open; finding 2 cannot be closed by the implementing agent and requires sanitized Route B evidence from an authorized human/operator or the CTO/control owner.

This branch carries an implementation candidate that must not be used as proof of any fleet's effective mode.

Task identity

Programme:        Shared Thoth GraphQL / Backend Architecture
Repository:       thoth-pub/thoth
Task:             THOTH-GQL-OPS-03
Risk:             HIGH
Workflow:         STANDARD
PR target:        develop
Authorized base:  2bec75e6698232f7643862120e5437452fcfa252
Reviewed head:    8604dbf93be734c2e04431b15d4a9db1f3a1d2fb
Remediation head: 62ac43e16d68b50a94c65f599d7b6a019d7a71ca

Specification approval: #798 comment 5252446279. Implementation authorization: #798 comment 5252526720.

Finding 1 — observation provenance is forgeable — OPEN

Verified independently from repository source, not accepted on the reviewer's word:

logger.rs:77-81   BodyLogger captures the RAW BODY of every public
                  POST /graphql request into request extensions.
logger.rs:19,30-35  Logger's %{QUERY}xi interpolates that body into the
                  access-log line.
logger.rs:27      format_request_body prefixes it with "\n", so attacker
                  bytes begin their OWN line, verbatim.
lib.rs            start_server installs both, and emits the effective-mode
                  record through log::info!.

Both travel the same log facade, at the same level, into the same sink. The caller therefore controls an entire line including any prefix it appears to carry, and can synthesise [<ts> INFO thoth_api_server] THOTH_MUTATION_GUARD_EFFECTIVE_MODE mode=ENFORCE instance=attacker. parse_record locates the marker in arbitrary text and accepts it.

Not fixed in this remediation, deliberately. Every candidate structural trust property is trustworthy only if the real collection plane preserves it — that is fact E4, unevidenced. Selecting one now would repeat exactly the unsupported-assumption error the review identified. Specifically:

• a text-level fix is ruled out on evidence — marker choice, anchoring and in-text prefixes are all forgeable;
• stdout for the record vs stderr for access logs is plausible and would not change request-logging semantics, but depends on E4;
• a distinct log target is forgeable today, since the target is rendered into text the caller can synthesise;
• narrowing the existing public request logging is out of scope and is a separate architectural/security dependency.

The required adversarial test is deferred with the fix: it must assert that injected content cannot become a trusted observation, which cannot be made true before the channel is selected.

Finding 2 — external correlation contract unevidenced — BLOCKED

The earlier report claimed External deployment facts relied on: NONE and AC-20: PASS (not engaged). Both were wrong. Feasibility rests on two deployment facts no local test can establish: per-instance collection provenance, and that the emitted identity correlates with the orchestrator's enumeration identity.

Withdrawn as unsupported: "the orchestration plane already collects the stream per instance", "the OS host name is the orchestrator-assigned task/pod identity", "every Linux production observation is attributable", "no external deployment fact is relied on".

Outstanding Route B evidence request (sanitized facts only — no credentials, secret values, resource/account identifiers, private configuration or unnecessary topology):

• E1 Does the runtime's output/log collection retain trustworthy provenance identifying the specific serving instance that emitted each event, as distinct from one aggregate stream?
• E2 What sanitized identity correlates a collected event with the same member returned by live orchestrator enumeration? Is the OS host name that identity, or does it map to it?
• E3 Is that identity unique to the serving process/generation during rolling replacement, or what minimum additional sanitized generation identity is required?
• E4 Does collection preserve a provenance property that message text cannot forge (stdout/stderr stream identity, structured source/target metadata, or equivalent)? Which one?
• E5 Can records from terminated or failed-start instances be distinguished from the currently enumerated population?

No AI agent or model is a valid source. The implementing agent did not read the private deployment source, query a real orchestrator, or access production.

Also now recorded explicitly

• A record is not membership. It attests only that some process computed a mode. Live orchestrator enumeration remains the sole authority on the population; every real process in the test suite emits its record and then deliberately fails to start, demonstrating the distinction. E3 is unresolved, so a stale/failed-start record cannot yet be shown incapable of satisfying coverage for a different current process.
• Signal availability. Emission is conditional on info: suppressed level → no record → member UNKNOWN → NOT ESTABLISHED. Fail-closed, never OFF — an operational precondition, not a guarantee. AC-3 is not claimed on the strength of the code calling log::info!.
• instance may be unnecessary disclosure. If E1 shows the collection plane supplies a trusted identity, the preferred design removes the self-reported host name entirely — smaller disclosure and stronger provenance.

Acceptance criteria — 17 PASS, 6 BLOCKED

AC	Status	AC	Status
AC-1	PASS	AC-13	PASS
AC-2	PASS	AC-14	PASS
AC-3	BLOCKED (E2 unevidenced; finding 1)	AC-15	PASS
AC-4	BLOCKED (no trusted observation per member; E1/E3)	AC-16	PASS
AC-5	PASS	AC-17	PASS
AC-6	PASS	AC-18	PASS
AC-7	PASS	AC-19	PASS
AC-8	PASS	AC-20	BLOCKED — E1–E5 unavailable; route: NONE
AC-9	PASS	AC-21	PASS
AC-10	PASS	AC-22	PASS
AC-11	BLOCKED (no final mechanism to certify)	AC-23	BLOCKED (instance necessity open)
AC-11.1	PASS — no public caller can obtain the mode; distinct from finding 1's injection class, and not reinterpreted as covering it

Because AC-3 and AC-20 cannot pass, the overall result is BLOCKED.

Unchanged and accepted by review: single-stored-value property (AC-1), mode distinguishability (AC-2), the verifier's fail-closed reduction incl. MIXED/UNKNOWN/incomplete coverage (AC-5, AC-7, AC-22), the silent-adoption fixture (AC-6), unchanged request/guard/batching/store semantics (AC-8, AC-9), byte-identical SDL (AC-13), untouched control state (AC-15–AC-19).

Remediation changed files

Documentation, code-comment and test-comment reconciliation only — no mechanism change:

• docs/.../THOTH-GQL-OPS-03-implementation-report.md — BLOCKED status; finding-1 reproduction from source; §12 rewritten as External correlation evidence with the Route B request; §12.4 attack fixture and current FAIL result; §6.6 stale/failed-start semantics; §6.7 signal availability; AC matrix corrected
• docs/.../graphql-mutation-guard-mode-transition-runbook.md — §4.0 mechanism NOT ESTABLISHED; §4.0.1 E1–E5 contract; §4.0.2 the OPS-04 procedure; capability gap 2 restored as still open. Remains PROVISIONAL
• CHANGELOG.md — entry rewritten; unsupported hostname/collection/no-external-facts claims withdrawn
• thoth-api/src/graphql/fleet_verification.rs — NOT TRUSTED EVIDENCE status block; InstanceIdentity correlation semantics marked unestablished
• thoth-api/src/graphql/fleet_verification/tests.rs — Linux test comment corrected to claim resolvability only
• thoth-api-server/src/lib.rs — emitter comment records the shared-sink defect and the info dependency
• tests/mutation_guard_fleet_verification.rs — records that these processes all fail to start, so a record is not membership

thoth-api-server/src/logger.rs was inspected to reproduce the defect and not modified.

Tests

cargo fmt --all -- --check                                            clean
git diff --check                                                      clean
cargo check --workspace                                               0 errors
cargo clippy --workspace --all-targets --all-features -- -D warnings   0 warnings
cargo test --workspace                                      1206 passed, 0 failed, 8 ignored

OPS-03 unit 23 · server/security 10 · real-process 5 debug + 5 release
OPS-02 mode-control 15 debug + 15 release (unchanged)
BATCH-01 93
Generated SDL  sha256 1e08b46b…  160799 bytes — byte-identical to base
Migration / schema / data — none

Control state — unchanged

PR #799                  OPEN / DRAFT / UNMERGED
CG-13                    OPEN
Runtime-operations gate  NOT SATISFIED
Runbook                  PROVISIONAL
OBSERVE                  NOT AUTHORIZED
ENFORCE                  NOT AUTHORIZED
BE-02 runtime            NOT AUTHORIZED
THOTH-GQL-OPS-04         NOT IMPLEMENTED, branch absent
Real fleet               NOT VERIFIED
Deployment               NONE
Runtime transition       NONE

No deployment, workflow dispatch, credential use, production access, orchestrator query or mode change occurred at any point.

Next steps

1. An authorized human/operator or the CTO/control owner supplies the E1–E5 sanitized facts. Nothing further can proceed without them — E4 in particular gates the finding-1 fix.
2. Select the trusted channel on that evidence, implement it, and add the adversarial injection test.
3. Re-assess AC-3, AC-4, AC-11, AC-20, AC-23.
4. Fresh independent exact-head review and a separate explicit CTO merge authorization remain required. The implementing agent has not approved and will not approve its own work. PR stays draft and unmerged.

[ja573]

Independent exact-head implementation review — THOTH-GQL-OPS-03

Decision: CHANGES REQUIRED

Exact head reviewed: 8604dbf93be734c2e04431b15d4a9db1f3a1d2fb
Authorized base: 2bec75e6698232f7643862120e5437452fcfa252
Risk: HIGH

I reviewed the actual nine-file base-to-head diff, the implementation code, the new tests, the task/runbook/report/changelog changes, and fresh exact-head CI. The core mechanics are strong: one stored Data<MutationGuardMode> feeds both the startup observation and the request path; the public listener gains no mode endpoint; UNKNOWN is structurally distinct from OFF; complete coverage fails closed; mixed-mode reduction is correct; the closed OPS-02 behavior is regression-pinned; and exact-head build/test/lint/format CI is green.

Two load-bearing findings prevent approval.

1. Observation provenance is forgeable through existing public request-body logging

thoth-api-server/src/logger.rs already captures the raw body of every public POST /graphql request into request extensions and Logger::default() writes that value into the access log. start_server installs both Logger::default() and BodyLogger on the public application.

The new EffectiveModeObservation::parse_record does not validate trusted log provenance. It calls line.find(THOTH_MUTATION_GUARD_EFFECTIVE_MODE) and then accepts syntactically valid mode= / instance= fields from the remainder of that line.

Therefore text supplied by an unauthenticated public caller can enter the same process log stream and can contain an observation-looking record. A collector/parser that trusts message text alone cannot prove that a parsed observation came from the one startup emitter rather than from public request content. At minimum this permits injected contradictory evidence to force UNKNOWN / NOT ESTABLISHED; more importantly, it breaks the provenance property required of a mechanism whose purpose is to prove process-effective mode.

The existing AC-11.1 tests prove that the server does not return the mode to a public caller. They do not prove that a public caller cannot inject record-looking content into the observation channel.

Required remediation:

• make trusted observation provenance structural, not dependent only on finding a marker in arbitrary log text;
• the collector/parser must be able to distinguish the startup emitter from public request/access-log content;
• add an adversarial integration test that sends public /graphql input containing the exact observation marker and valid-looking fields, and proves that this input cannot become an accepted trusted observation;
• preserve the approved section 3.2 boundary: do not solve this by adding a public endpoint or a new authorization decision.

If no trusted per-instance mechanism can satisfy this within section 3.2, return BLOCKED rather than widening architecture.

2. FEASIBLE / AC-3 / AC-20 rely on external orchestration facts that are currently unevidenced

The implementation report, PR body, runbook and code comments rely on these claims:

1. the serving process's output/log stream is collected and addressable per serving instance by the actual orchestration/logging plane; and
2. the OS hostname read from /proc/sys/kernel/hostname or /etc/hostname is the identity by which the actual orchestrator enumerates that serving instance.

The Linux test establishes only that a Linux process can resolve a hostname. It does not establish that the hostname equals, or can be correlated to, the orchestrator's enumeration identity in the Thoth deployment. Likewise local stderr capture does not establish that the real logging plane preserves per-instance provenance in the form this mechanism needs.

Those are deployment/orchestration facts on which the FEASIBLE conclusion and AC-3 depend. The report nevertheless states External deployment facts relied on: NONE and marks AC-20 PASS (not engaged). That is inconsistent with the binding section 6.6 evidence boundary and AC-20.

Required remediation:

• obtain sanitized Route A or authorized-human Route B evidence establishing the actual Thoth correlation contract needed by this design: per-instance log provenance and the mapping between the emitted identity and the orchestrator-enumerated identity; OR
• if that evidence is unavailable or disproves the assumption, mark the affected acceptance criteria BLOCKED and return BLOCKED as the specification requires;
• do not read the private secret-bearing deployment source to obtain it;
• update the report, PR body, runbook, CHANGELOG and relevant code comments/tests so they state only what the accepted evidence supports.

Additional precision to reconcile with the selected mechanism

The record is emitted at info, so suppressing that level makes the member UNKNOWN; record this as an explicit operational precondition rather than an unconditional claim that every serving process emits a usable observation. The record is also emitted before startup succeeds, so collection semantics must ensure evidence is correlated to the currently enumerated serving process/generation rather than stale or failed-start records.

These points are compatible with fail-closed behavior, but they must be made explicit in the mechanism/evidence contract used by OPS-04.

CI / controls

Fresh exact-head workflows are green. build-test-and-check ran classify, format_check, test, build, and lint successfully. run-migrations classified successfully and its migration job was skipped for this non-migration diff. check-changelog and publish-to-dockerhub succeeded.

Control state should remain unchanged while remediating:

PR #799                    DRAFT / UNMERGED
CG-13                      OPEN
Runtime-operations gate    NOT SATISFIED
Runbook                    PROVISIONAL
OBSERVE                    NOT AUTHORIZED
ENFORCE                    NOT AUTHORIZED
BE-02 runtime              NOT AUTHORIZED
THOTH-GQL-OPS-04           NOT IMPLEMENTED

This review does not authorize deployment, production access, mode transition, OPS-04, or merge.

Remediate on the existing bounded branch/PR. A new head requires fresh exact-head CI and fresh independent review.