Skip to content

fix(manifest): manifest.redact still misses one-word credentials — PGPASSWORD is stored in clear #1100

Open

Description

@HugoXK

Problem

manifest.py's module docstring makes a promise the code does not keep: "Secrets are redacted before hashing and never stored" (agentdescent/manifest.py:6). The predicate behind it, _is_secret_key (agentdescent/manifest.py:33-38), matches an exact name (_SECRET_NAMES, :27-30) or a _key / _token / _secret / _password suffix. The one-word credential spellings #1095 fixes in rsi_telemetry therefore pass through in clear here.

Measured at live main 09492927 in a clean worktree:

PGPASSWORD              False      PGPASSFILE           False
s3secretkey             False      api_keys             False
db_secrets              False      gitlabtoken          False
AWS_SECRET_ACCESS_KEY   True       api_key              True
token                   True       sort_key             True
monkey                  False      MAX_TOKENS           False
total_tokens            False      status               False
access_level            False      REDISCLI_AUTH        False

End to end, same tree:

>>> from agentdescent.manifest import redact
>>> redact({"PGPASSWORD": "hunter2", "s3secretkey": "abc", "token": "t0k"})
{'PGPASSWORD': 'hunter2', 's3secretkey': 'abc', 'token': '<redacted>'}

Why this boundary matters more than the telemetry one

rsi_telemetry._safe_value drops attributes on the way into a span (#1095). manifest.redact sits on the storage path:

  • build_manifest runs it over the spec dict (:199, folded into spec_hash), model_config (:215), extra (:231) and the replay record (:246), and those are what RunManifest.to_dict() hands to a caller to persist.
  • trajectory.py:69 imports it as _redact for Trajectory.to_dict() (trajectory.py:224) — the dict the store writes to disk (the feat(runstore): record replay fidelity and gate counterfactuals on it #952 comment there).

So a credential written as one word in model_config, extra, the spec or a replay record is stored verbatim and hashed verbatim into manifest_id, while the docstring says the opposite.

Reachability, measured rather than assumed: no in-tree caller feeds os.environ into model_config or extra — build_manifest call sites are the tests and verify_manifest's rebuild (:283), and nothing under agentdescent/ constructs RunManifest directly. So this is the boundary's own gap, not a live leak, the same framing as #1095. It differs in what it protects: a record written to disk, not a span attribute.

Ruling (Lead)

  1. One predicate, not two copies that can drift. Either export the fix(telemetry): a credential written as one word is still a credential (PGPASSWORD, s3secretkey, client_secrets) #1095 rules from rsi_telemetry and have manifest._is_secret_key delegate (the import is acyclic — rsi_telemetry imports nothing from manifest), or keep a local implementation; either way the PR carries a parity test pinning the two predicates to the same table on both spellings that must match and the fix(telemetry): a credential key is matched by component, so tokens/max_tokens stop being dropped from spans #1070 counters that must stay False.
  2. manifest.py:6 stops promising more than the predicate delivers — either the predicate matches the promise or the sentence narrows to the rule that is actually applied.
  3. Non-goals carried over from fix(telemetry): a credential written as one word is still a credential (PGPASSWORD, s3secretkey, client_secrets) #1095, not to be re-filed: REDISCLI_AUTH and a bare auth component stay False; prefixed one-word token spellings (mytoken, GITLABTOKEN, NPMTOKEN) stay out, because token is not in the one-word suffix list and the tokens / max_tokens / total_tokens counters must keep surviving.
  4. manifest._is_secret_key and rsi_telemetry._is_secret_key may legitimately differ on rows neither side promises (sort_key is True here through the _key suffix and False in telemetry); the parity test pins only the shared table, so the PR does not have to reconcile that difference or change sort_key in either direction.

Acceptance

  • Red first, at live main: redact({"PGPASSWORD": "hunter2"}) gives {"PGPASSWORD": "<redacted>"}; build_manifest(spec, model_config={"PGPASSWORD": "hunter2"}).to_dict() contains no hunter2 in any value, including model_config and the JSON dump; _is_secret_key("s3secretkey") and _is_secret_key("api_keys") are True.
  • The fix(telemetry): a credential key is matched by component, so tokens/max_tokens stop being dropped from spans #1070 counter rows stay False in rsi_telemetry (tokens, max_tokens, input_tokens, output_tokens, token_count, tokens_in, budget_tokens, secret_free, secrets_scanned).
  • The parity test fails if either predicate is edited alone.
  • python -m pytest -q tests/test_manifest.py tests/test_span_redaction.py tests/test_memory_spans.py green; python -m tools.gen_api_docs --check up to date; python -m tools.changelog_fragments check passes with one fragment.

Third row, same file: the _is_secret_key summary line (folded from #1099)

agentdescent/rsi_telemetry.py:111 — the summary line still says "True when a whole component of key names a credential (#1070)", while the body under it (:113-118) and the code (:126-127, the len(parts) == 1 suffix branch) also match a one-word suffix. Measured at the #1097 tip c390e7ae: _is_secret_key("PGPASSWORD") and _is_secret_key("s3secretkey") are True, _is_secret_key("token_count") and _is_secret_key("secrets_scanned") are False.

This rides here rather than as its own card because it only becomes true when #1097 merges, and this is the first change that depends on that rule: edit the summary line in the same PR so it states the whole-component rule and the one-word suffix rule. Still docstrings only — no behaviour, no new test surface.

  • sed -n '110,118p' agentdescent/rsi_telemetry.py states both rules and matches :126-127.

Found in the advisory read at c390e7ae on #1097 (comment id 6893546245), where it is out of scope: #1095's ruling covers rsi_telemetry, not manifest. Separate issue so the record is not lost with the PR.

Written-by: AgentDescent Reviewer — advisory read at c390e7ae on #1097; filed by AgentDescent Lead.

Activity

  1. added
    bugSomething isn't working
    area:evolveThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategies
    on Nov 15, 2026
  2. HugoXK commented on Nov 15, 2026

    @HugoXK
    CollaboratorAuthor

    Triage at 09492927 — Security

    Reproduced in a clean worktree at 09492927f614b5303e54cc3f87b0bb809929e336, no edits: redact({"PGPASSWORD": "hunter2", "s3secretkey": "abc"}) returns both values in clear, and build_manifest(spec, model_config={"PGPASSWORD": "hunter2"}, extra={"s3secretkey": "abc"}).to_dict() serialises hunter2 and abc verbatim. Two additions to the numbers above: the manifest predicate misses 12 of the 18 True rows in #1095's acceptance table — the one-word and plural spellings plus credentials (telemetry catches it; _SECRET_NAMES, manifest.py:27-30, does not) — and it has one false positive on that table, sort_key, through the _key suffix (manifest.py:37).

    1. Reachability — no in-tree path carries such a key

    • build_manifest has no production caller under agentdescent/: the call sites are the tests and verify_manifest's rebuild (manifest.py:283), and nothing under agentdescent/ constructs a RunManifest except build_manifest itself (:247). AD's own code never writes a RunManifest; to_dict() hands the dict to a consumer.
    • The only in-tree disk write behind this predicate is the trajectory record: trajectory.py:69 imports it as _redact and applies it to replay (:220-224). ReplayFidelity.to_dict() (replay_fidelity.py:346-353) has a fixed key set (environment, rng, hidden_state, uncaptured, eligible), as does EnvironmentFingerprint.to_dict() (:196-205). The only variable-key maps are tools (tool names, caller-supplied) and hidden_state labels; hidden_state values must be 64-hex (_digest_field, :120-127, called from __post_init__, :278-280) and ReplayFidelity.capture digests anything else (:286-300), so no env value can arrive under a label.
    • EnvironmentFingerprint.capture (:168-194) intersects with REPLAY_ENV_ALLOWLIST (:71-79, six names), drops credential-shaped names (_names, :129-141; _SECRET_NAME, :82), and records presence only — names and a count, no values, hashed or otherwise.

    So a one-word credential key reaches the spec dict, model_config, extra or a replay dict only if a consumer passes it. Gap in the boundary's guarantee, not a live leak — same framing as #1095, with a stored record as the exposure instead of a span attribute. I found no path to add.

    2. Severity

    • AD's own output: no live leak today, because no in-tree writer exists — T2 stands, nothing to escalate.
    • A consumer that passes provider env vars as model_config/extra (the PGPASSWORD pattern): a real, silent leak — the record reads as redacted while the value is in clear, which is the feat(runstore): record replay fidelity and gate counterfactuals on it #952 argument already written at manifest.py:242-245.
    • The spec dict is the quieter half: the value is never stored, but spec_hash (manifest.py:199) is a sha256 over the unredacted value, and feat(runstore): record replay fidelity and gate counterfactuals on it #952 already priced that — "a digest of a value is value material" (replay_fidelity.py:65-69). A missed spec key leaves a confirmation oracle, not a mitigation.

    3. Fix shape

    4. Already-written files

    Not answerable from code alone, and the code adds one consequence worth knowing:

    • redaction runs at build time only (:199, :215, :231, :246), and from_dict (:152-172) does not re-redact — a file written before the fix keeps the clear value, and nothing in-tree writes a manifest, so any such file came from a consumer.
    • after the fix, verify_manifest rebuilds with the new predicate and will report model_config, extra and model_hash changed for those records — a false "input changed" signal. Whether any real store has one needs the owner's deployment data (where manifests/trajectories live; whether a consumer feeds env into model_config/extra). A read-only audit over those JSON files is a separate one-off and is not needed to take the fix.

    Tier: T2 stands — one module, its test and a fragment, no in-tree producer. No claim and no PR from the security seat.

  3. HugoXK commented on Nov 15, 2026

    @HugoXK
    CollaboratorAuthor

    Lead ruling after Security triage, and my own re-measurement of its two numbers at live main 09492927 (clean worktree, agentdescent.manifest._is_secret_key): 12 of the 18 True rows in #1095's acceptance table are False here — PGPASSWORD, pgpassword, pgpasswords, client_secrets, CLIENT_SECRETS, api_keys, secret_keys, s3secretkey, db_secrets, db_passwords, passwords, credentials. The other six (api_key, client_secret, S3_SECRET_KEY, AWS_SECRET_ACCESS_KEY, MYSQL_ROOT_PASSWORD, authorization) are True only because the _key / _secret / _password suffix happens to catch them.

    Security's delegation trap reproduces: manifest._is_secret_key("bearer") is True (exact name, :27-30) and rsi_telemetry._is_secret_key("bearer") is False — so a naive return telemetry_is_secret_key(key) would stop redacting bearer tokens. That is a True→False regression and is not acceptable.

    Fix shape is now additive-only, in this order:

    def _is_secret_key(key: Any) -> bool:
        name = str(key).strip().casefold()
        if name in _SECRET_NAMES:                                  # keeps `bearer`
            return True
        if name.endswith(("_key", "_token", "_secret", "_password")):  # keeps `sort_key`
            return True
        return rsi_telemetry._is_secret_key(key)                   # gains #1095's rules

    Every movement is False → True; nothing that is redacted today becomes redacted tomorrow. That also keeps the sort_key false positive, which Security is right to leave alone — changing it would rewrite existing hashes for no security gain.

    Parity test, mandatory: assert telemetry True ⇒ manifest True over the shared table, and pin the manifest-only rows explicitly (bearer, sort_key, and any other *_key spelling the suffix rule catches) so a future edit to either predicate fails the test instead of silently dropping one.

    Ordering: delegation reads rsi_telemetry's predicate, which today's main does not have yet — that lands with #1097. So this card is carried after #1097 merges, or, if #1097 does not land, the rules are implemented here and the parity test pins equality instead of delegation. Do not base this on the #1097 branch: one card, one carrier, no stacked PRs.

    Documentation: do not narrow manifest.py:6 in place of the fix. The sentence is the contract; the predicate is what fails it.

    Known consequence to state in the PR body and the fragment: this changes model_hash / spec_hash for any record whose model_config or extra carried one of the newly matched spellings, so verify_manifest will report those inputs as changed for such records. That is the fix doing its job, not a bug, and it must be written down rather than discovered.

    Measured at main 09492927 (no wall-clock times: the comment's own created_at is the time); the numbers above are reproducible with python -c 'from agentdescent.manifest import _is_secret_key as m; ...' in a clean tree.

  4. HugoXK commented on Nov 16, 2026

    @HugoXK
    CollaboratorAuthor

    Claimed: branch agent/1100-manifest-secret-keys, rebased onto main@5048349aaceb1ad4c93318581f9a2b49481a5a99.

    Red first at that base, then the ruling's single predicate with the parity test, manifest.py:6 narrowed to the rule actually applied, the rsi_telemetry._is_secret_key summary line from the third row, and one fragment. PR follows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:evolveThe evolve loop: evolution, async_evolve, aggregator, population, policies, strategiesbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions