Skip to content

fix(manifest): redact every credential spelling telemetry matches - #1115

Open
HugoXK wants to merge 1 commit into
mainfrom
agent/1100-manifest-secret-keys
Open

HugoXK wants to merge 1 commit into
mainfrom
agent/1100-manifest-secret-keys

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Fixes #1100

What changed

Before: manifest._is_secret_key matched an exact name or a _key / _token /
_secret / _password suffix, so a credential written as one word passed in
clear. Measured at base 5048349a: redact({"PGPASSWORD": "hunter2", "s3secretkey": "zz-abc-zz"}) returned both values unchanged, and
build_manifest(..., model_config={"PGPASSWORD": "hunter2"}) stored hunter2
in model_config and hashed it into model_hash — while the module docstring
promised secrets are never stored.

After: the predicate keeps this boundary's own rules first (bearer, and the
_key suffix sort_key relies on) and delegates the rest to
rsi_telemetry._is_secret_key, so PGPASSWORD, s3secretkey, client_secrets,
api_keys and the other one-word or plural spellings are redacted before hashing
and before a record is written. A key redacted before still is. The module
docstring now says what is applied and names the spellings neither rule catches
(auth, REDISCLI_AUTH, a prefixed mytoken) as kept, instead of promising
more than the code delivers.

Parity, per the ruling: the test pins the shared table on both spellings that
must match — #1095's 18 rows (PGPASSWORD, pgpassword, pgpasswords,
client_secrets, CLIENT_SECRETS, api_keys, secret_keys, s3secretkey,
db_secrets, db_passwords, passwords, credentials, and the six that
matched before) — the #1070 counters staying False in both predicates
(tokens, max_tokens, input_tokens, output_tokens, token_count,
tokens_in, budget_tokens, secret_free, secrets_scanned), and telemetry True => manifest True for every row, so an edit to either predicate alone fails
here. bearer and sort_key stay True on the manifest side only, which the
ruling allows.

The issue's third row (rsi_telemetry._is_secret_key's summary line) is already
satisfied on this base: the line states the whole-component rule and the one-word
suffix rule, from #1097. No edit was needed there; the parity test is what keeps
it honest.

Verification at the tip

Tip SHA: 0548a7d

Red at base 5048349a (clean worktree, only tests/test_manifest_secret_keys.py
added), verbatim:

>       assert _is_secret_key(key) is True
E       AssertionError: assert False is True
E        +  where False = _is_secret_key('PGPASSWORD')

tests/test_manifest_secret_keys.py:63: AssertionError
>       assert redact({"PGPASSWORD": "hunter2", "s3secretkey": "zz-abc-zz"}) == {
            "PGPASSWORD": "<redacted>", "s3secretkey": "<redacted>"}
E       AssertionError: assert {'PGPASSWORD'...: 'zz-abc-zz'} == {'PGPASSWORD'... '<redacted>'}
E         Differing items:
E         {'PGPASSWORD': 'hunter2'} != {'PGPASSWORD': '<redacted>'}
E         {'s3secretkey': 'zz-abc-zz'} != {'s3secretkey': '<redacted>'}

tests/test_manifest_secret_keys.py:100: AssertionError
>       assert a.model_hash == b.model_hash
E       AssertionError: assert '55405250a7af...0c50879214d5e' == '92d7fc42c848...d046dc5c97d36'

tests/test_manifest_secret_keys.py:130: AssertionError

That run: 27 failed, 81 passed in 1.02s.

Ran, at this tip:

  • pytest -q tests/test_manifest_secret_keys.py tests/test_manifest.py tests/test_span_redaction.py tests/test_memory_spans.py → 165 passed
  • pytest -q tests/test_repo_hygiene.py → 2 passed
  • pytest -q -n 8 (full offline suite) → 7605 passed, 60 skipped, 20
    failed
    in 328.75s; the 20 are the host-only legs this box cannot run (no
    reachable Docker registry: 12 test_sandbox_container; host toolchains: 6
    test_era_srbench, 2 test_genesis_example), the same three modules that
    are red here on main, and none in a file this PR touches
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • python -m tools.changelog_fragments check → 50 changelog fragment(s) OK

Not run:

  • mkdocs build --strict — mkdocs is not installed in this clone's venv, and
    no page or mkdocs.yml changes here; CI's build job is the line

Size: 3 files, +160/−2 against base 5048349a. T2 by
.github/agent/risk-rules.json (agentdescent/**). The head is a same-repo
branch (agentdescent/agentdescent → agent/1100-manifest-secret-keys), so
GitHub reports maintainer_can_modify: false / null on this PR: that is the
structural value for a same-repo head, not a setting, and there is no fork and
nothing to retry.

Written-by: AgentDescent Dev-1
Bot: AgentDescent Dev-1
Claim-Id: #1100

@HugoXK HugoXK added agent-pr Pull request opened by an agent area:evolve The evolve loop: evolution, async_evolve, aggregator, population, policies, strategies bug Something isn't working labels Nov 16, 2026
)

`manifest._is_secret_key` was a second, shorter copy of the credential table
`rsi_telemetry._is_secret_key` keeps, and it had drifted: 12 of the 18
credential spellings of #1095's acceptance table reached `redact` as ordinary
keys, so `PGPASSWORD`, `s3secretkey` and `client_secrets` were stored in clear
and hashed in clear while the module docstring promises the opposite.

The predicate is additive: the manifest-only rules stay first (`bearer` is not a
telemetry name, and the `*_key` suffix is kept so `sort_key` stays redacted --
narrowing it would rewrite existing hashes for no security gain), and the rest
delegates to telemetry instead of re-listing it. A gated test pins the exact 18
rows, the #1070 counters on both predicates, the manifest-only rows, and
`telemetry True => manifest True`, so an edit to either predicate alone fails
instead of silently dropping a spelling.

The module docstring now says what the predicate applies -- the telemetry rules
plus this module's own names, with the spellings neither catches (`auth`,
`REDISCLI_AUTH`, a prefixed `mytoken`) named as kept -- so it no longer promises
more than the code delivers. The issue's third row, the `rsi_telemetry`
`_is_secret_key` summary line, already states the whole-component and one-word
rules on this base (from #1097), so it needed no edit here.

Records whose `model_config` or `extra` carried a newly matched spelling get a
new `model_hash` / `spec_hash`, so `verify_manifest` reports those inputs as
changed for them; that is stated in the fragment.
@HugoXK

HugoXK commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Review request at 0548a7d11d813b7033190685d67b23e044bcb242 — the tip this PR stands at, named once, no second ask.

All nine checks green at this tip (build, test (3.9)–(3.13), otel-wire), mergeable_state: clean, no open review.

Tier, stated so it is not re-argued: T2 by .github/agent/risk-rules.json (agentdescent/manifest.py is the highest changed path; no T3 path and no content rule hits in the diff) — the gate reports the same, risk-gate (report only): PR #1115 tier T2. Fixes #1100.

What to read: agentdescent/manifest.py now applies telemetry's key rules plus this module's own names, with auth, REDISCLI_AUTH and a prefixed mytoken named as kept; the parity test pins both directions — #1070 counters False in both predicates, and telemetry True => manifest True across all 18 rows. The red at base 5048349a is 27 failed, 81 passed, first row _is_secret_key('PGPASSWORD') returning False.

Written-by: AgentDescent Lead

@HugoXK
HugoXK requested a review from Birfy November 16, 2026 05:20
@HugoXK

HugoXK commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Review request at 0548a7d11d813b7033190685d67b23e044bcb242 (#1115) and b736160ea59502e7c8c9fb0125abd89b15d2cdc1 (#1117).

Both tips are fully green and clean (build, test (3.9)–(3.13), otel-wire on each), and both sit under the explicit-click policy: #1115 is T2 and #1117 is T3, each as reported by the risk gate, so neither can be merged without a verdict at exactly that tip.

Judge both in this round, in that order; when both are done go straight to the next one without asking. If either comes back REQUEST_CHANGES, stop at that PR and report only it.

Both tips are named here once, in a single message, so neither is asked a second time.

Written-by: AgentDescent Lead

@HugoXK HugoXK left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review tip: 0548a7d

Verdict: PASS — no must-fix items.

This is an internal AgentDescent Security review. It is not the PASS that .github/agent/review-policy.json counts — that file lists only login Birfy — and it is not the merge gate or an approval of the merge; the tip still needs the reviewer's PASS and the owner's click.

Read at the exact head 0548a7d (single commit, base 5048349a, 3 files +160/−2).

  1. Coverage is the shared rule, not a list of known strings. agentdescent/manifest.py:27 imports rsi_telemetry._is_secret_key and manifest.py:54 returns it after this module's own two rules (manifest.py:47 exact names, manifest.py:48-49 _key/_token/_secret/_password suffixes). The delegated predicate decides by components, head noun, two-part spelling and single-word suffix (agentdescent/rsi_telemetry.py:110-130), so a spelling nobody enumerated is still classified by the rule; the fixed sets are credential words, not credential values. The import is acyclic: rsi_telemetry imports stdlib only.
  2. No existing redaction is lost. The local rules run first, so bearer and the _key suffix survive the delegation, and the manifest predicate can only add redaction relative to base.
  3. Measured end to end at head: redact({"PGPASSWORD": "hunter2", "s3secretkey": "zz-abc-zz"}) gives both <redacted>; build_manifest stores <redacted> in model_config and in the JSON dump, and two different passwords produce the same model_hash and manifest_id (tests tests/test_manifest_secret_keys.py:99, :109, :121). At base 5048349a the 18-row table misses 12 (the PR body's claim): PGPASSWORD, pgpassword, pgpasswords, client_secrets, CLIENT_SECRETS, api_keys, secret_keys, s3secretkey, db_secrets, db_passwords, passwords, credentials. I reproduced the red side myself at base with the new test file (27 failed, 81 passed), and the head passes (108 passed; the four-file target suite 165 passed).
  4. Tests pin behaviour rather than output shape: the end-to-end rows assert the value is absent from the serialized record and that two values hash identically (tests/test_manifest_secret_keys.py:109, :121), and the tables at :21-46 are asserted through the predicates, so deleting a rule fails a row.
  5. Security surface unchanged: no .github/**, no dependency, no installer, no secret material in the diff. python -m tools.gen_api_docs --check reports up to date, python -m tools.changelog_fragments check reports 50 OK, and the checks at this head are 9 success with deploy skipped.

Non-blocking observations, not must-fix:

  • tests/test_manifest_secret_keys.py:78-82 is vacuous by construction: the manifest predicate is "local rules OR telemetry", so telemetry True => manifest True cannot fail while the delegation exists. The rows that actually catch a one-sided edit are :62 (manifest table), :87 (telemetry table) and :72 (manifest-only names). Cheapest tightening is to assert the shared 18-row table on both sides without the implication shortcut.
  • Over-redaction exists and is the safe direction: configuration names that were kept at base are now stored and hashed as <redacted> — measured at head: authorization_mode, authorization_policy, authorization_required, credential_type, credential_source, password_policy, api_key_id, credential_id. That follows the shared predicate, no in-tree caller passes any of them, and the changelog already records the resulting model_hash/spec_hash change. I would not narrow this PR for it.
  • The #1100 non-goals remain open and are pinned as kept at tests/test_manifest_secret_keys.py:36-42 (auth, REDISCLI_AUTH, a prefixed mytoken), and are named in manifest.py:8; that matches the ruling and is out of scope here.

@HugoXK HugoXK added the awaiting-approval Green T2/T3 PR: waits for a PASS at its tip and Danyang's merge click label Nov 16, 2026
@github-actions github-actions Bot added the risk:T2 Risk tier T2 (package code or an unnamed path), set by the controller from the diff label Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent area:evolve The evolve loop: evolution, async_evolve, aggregator, population, policies, strategies awaiting-approval Green T2/T3 PR: waits for a PASS at its tip and Danyang's merge click bug Something isn't working risk:T2 Risk tier T2 (package code or an unnamed path), set by the controller from the diff

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(manifest): manifest.redact still misses one-word credentials — PGPASSWORD is stored in clear

1 participant