grade: 58 of 59 entries evidence-graded, + a source checker that catches catch-all origins - #21
Open
john-the-dev wants to merge 23 commits into
Open
grade: 58 of 59 entries evidence-graded, + a source checker that catches catch-all origins#21john-the-dev wants to merge 23 commits into
john-the-dev wants to merge 23 commits into
Conversation
…ctions
GRADING.md says unrated entries are graded by the maintainer bot on its next
pass. This is that pass, scoped to three entries whose sources are primary
standards and therefore verifiable rather than vibes-graded.
Every evidence line was fetched live on 2026-08-30, not asserted from memory:
agent-delegated-auth A RFC 9728, HTTP 200, Category: Standards Track. The
source's own title is "OAuth 2.0 Protected Resource
Metadata" — exactly the RFC the entry's claim names.
agent-identity-secrets B NIST SP 800-207 "Zero Trust Architecture", final,
HTTP 200. B not A: it predates LLM agents and makes
no agent-specific claim; the principles transfer.
agent-memory-tiers B arXiv 2310.08560 "MemGPT: Towards LLMs as Operating
Systems", citation_date 2023-10-12, >=v2, HTTP 200.
B not A: foundational but a 2023 preprint.
Two source_type values were wrong in a way that mattered. Both entries cited a
NIST Special Publication and an arXiv paper while declaring source_type: blog,
and GRADING.md ranks primary docs and papers ABOVE secondhand blog — so the
mislabel understated the very signal the grade is computed from. Corrected to
docs and paper.
validate.mjs: 59 entries valid, warnings 48 -> 45 (exactly these three).
dist/kb.json rebuilt via scripts/build.mjs, since CI fails on a stale artifact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…y docs Closes the pattern this PR flagged but had not audited. Of the 22 blog-labelled entries on main, four sit on hosts that are definitionally not a secondhand blog: finance-agents www.nist.gov blog -> docs guardrails-safety www.nist.gov blog -> docs rag-basics learn.microsoft.com blog -> docs retrieval-quality learn.microsoft.com blog -> docs GRADING.md ranks primary docs above secondhand blog, so the mislabel understates the signal each entry's future grade is derived from. Grades are untouched — all four stay unrated pending their own evidence pass. Audited by host, not by reading each source: a host allowlist decides only the cases where the answer is definitional. The remaining 16 are ordinary vendor and lab posts (anthropic.com x6, stanford HAI, temporal docs, uipath, hhs.gov and others) and need per-source judgement, not a rule. validate.mjs: 59 entries valid, 45 warnings (unchanged — labels moved, not grades). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owner
Author
|
Audited the pattern this PR flagged but left unverified, and closed the definitional part of it in Four more entries were labelled Grades untouched — all four stay Scope of the audit, stated so it is not read as more than it is. I decided by host allowlist, which only settles the cases where the answer is definitional — a final NIST publication and Microsoft Learn are not blogs under any reading. The remaining 16 are ordinary vendor and lab posts ( One correction to my own numbers. The body above says 22, and 22 is right — measured on |
Same discipline as the first batch — every source fetched live 2026-08-30 and
graded on what it says about itself, not on my impression of the topic.
mcp A official MCP docs, HTTP 200, title "Architecture overview
- Model Context Protocol". Primary spec for the protocol
the entry describes.
coding-agents B arXiv 2310.06770 "SWE-bench: Can Language Models Resolve
Real-World GitHub Issues?", 2023-10-10, HTTP 200. B not A:
still the reference benchmark, but its headline resolve
rates are far below current agents — cite the design, not
the numbers. source_type blog -> paper.
guardrails-safety B NIST AI RMF, HTTP 200, "AI Risk Management Framework |
NIST". B not A: a voluntary governance framework, not an
implementable guardrail spec — it says which risks to
manage, not how to enforce them at an agent boundary.
Two of three are B, and the caveat is in evidence per GRADING.md. Grading a
governance framework or a 2023 benchmark paper as "current best practice, safe
to build on" would make the A grade meaningless.
validate.mjs: 59 valid, warnings 45 -> 42. dist rebuilt: unrated 42, A 13, B 4.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
agent-deployment A official AWS Lambda best-practices guide, HTTP 200.
Unusually tight source-to-claim fit — the entry is about
serverless agent workers and this is the primary doc for
that runtime, not a general doc stretched to cover it.
mcp-stateless A the dated claim checks out: /specification/2026-07-28
returns 200 and mentions Stateless.
finance-agents B NIST Privacy Framework, HTTP 200. B not A: general privacy
governance, silent on ledger math, sanctions screening and
transaction authorization — the half the entry cares about.
mcp-stateless got a negative control because a docs SPA can answer 200 for any
path, which would make the check meaningless:
/specification/2026-07-28 200 (and mentions Stateless)
/specification/2019-01-01 404
/specification/not-a-real-revision 404
So the 200 is a real revision, not a catch-all. One caveat recorded against the
citation rather than the claim: the entry's url is the site ROOT ("What is the
Model Context Protocol (MCP)?"), a general overview that does not itself
evidence a dated spec revision. The revision page is the precise source and is
where the url should point. I graded on evidence found at a different url than
the entry cites, and said so in the evidence rather than quietly using it.
validate.mjs: 59 valid, warnings 42 -> 39. dist: unrated 39, A 15, B 5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rag-basics, retrieval-quality and groundedness-hallucination all cite Azure product documentation. Each page is primary, actively maintained, and returned 200 on 2026-08-30 with a title matching its subject. All three are still B, for one shared reason worth stating plainly: the entries make VENDOR-NEUTRAL claims about a technique while resting on a SINGLE vendor's product docs. That is a real basis, but a reader should not read Azure's defaults as the general best practice, and GRADING.md reserves A for "safe to build on". retrieval-quality gets a second, sharper caveat: the cited Hybrid Search page substantiates the hybrid-retrieval half, but the entry also asserts chunking boundaries and reranking practice the page does not cover — it under-evidences part of its own claim. groundedness-hallucination is the closest method match of the three: the entry describes decomposing an answer into atomic claims and entailment-checking each against the source, which is exactly what the cited service does. Still one vendor's detector, so still B. Negative control, since three 200s from one host prove nothing on their own: /azure/search/retrieval-augmented-generation-overview 200 /azure/search/hybrid-search-overview 200 /ai-services/content-safety/concepts/groundedness 200 /azure/search/not-a-real-page-xyz 404 validate.mjs: 59 valid, warnings 39 -> 36. dist: unrated 36, A 15, B 8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e claim is unevidenced there Two separate findings, both about the citation rather than the topic. 1. The cited url is not documentation. https://a2a-protocol.org/ has title "Redirecting" and `curl -L` does not move, because the redirect is a CLIENT-SIDE meta refresh (content="1; url=latest/"). The docs live at /latest/ (200, "A2A Protocol"), and that is where the url should point. 2. On that real page, the entry is only half evidenced: linux foundation x3 json-rpc 0 agent card x1 webhook 0 sse x11 task lifecycle 0 So the Linux-Foundation, Agent-Cards and streaming claims hold at the source; JSON-RPC, webhooks and task lifecycles do not appear at all. They may sit on deeper spec pages, but the page this entry points to does not evidence them — which is what B with a caveat is for. Negative control, because a zero is only meaningful if the host can produce a miss: /latest/not-a-real-a2a-page-xyz returns 404. The three zeros above are real absences, not a catch-all route. validate.mjs: 59 valid, warnings 36 -> 35. dist: unrated 35, A 15, B 9. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… one essay
Grading these three surfaced a structural issue worth more than the grades.
ONE essay is the sole evidence for FOUR entries:
agent-loop /research/building-effective-agents -> 301
agent-cost-control /engineering/building-effective-agents
customer-support-agents /engineering/building-effective-agents
multi-agent-orchestration /engineering/building-effective-agents
The /research/ path 301s to /engineering/, so it is one page under two urls and
the duplication is easy to miss by reading the url column alone. Two further
entries (agent-evals, evaluation-strategy) likewise share demystifying-evals.
Per-entry, the essay fits unevenly, which is why all three are B:
multi-agent-orchestration closest — the essay does cover orchestrator-worker
agent-cost-control essay never addresses token budgeting, model
tiering or caching economics
customer-support-agents essay uses support only as a worked example; it is
not a source on deflection, escalation or CSAT
Negative control: /engineering/not-a-real-post-xyz returns 404, so the 200s are
real pages and not a catch-all.
validate.mjs: 59 valid, warnings 35 -> 32. dist: unrated 32, A 15, B 12.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ll 59 entries
Closes the second shared-source cluster, and — more usefully — measures how big
the problem actually is rather than leaving last commit's finding open-ended.
Redirect-normalised audit of all 59 entries, 2026-08-30:
distinct sources 55
sources backing more than one entry 2 (covering 6 entries)
4x anthropic.com/engineering/building-effective-agents
agent-loop, agent-cost-control, customer-support-agents,
multi-agent-orchestration
2x anthropic.com/engineering/demystifying-evals-for-ai-agents
agent-evals, evaluation-strategy
So source reuse is BOUNDED: 55 distinct sources for 59 entries is healthy, and
the two clusters are the whole of it. That is worth stating as plainly as the
finding itself — the previous commit could be read as implying the KB leans on
a handful of essays, and it does not.
The pair itself:
agent-evals topical fit is good (the essay IS about eval pitfalls),
but its sharp checkable claims rest on one vendor essay
with no second source
evaluation-strategy the offline/online split is broader than the essay
establishes; the source supports offline far better
validate.mjs: 59 valid, warnings 32 -> 30. dist: unrated 30, A 15, B 14.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owner
Author
|
Correcting a number I gave in That commit said 2 shared sources covering 6 entries. The real figure is 3 covering 8:
Why I missed it, since the method matters more than the number. I called that audit "redirect-normalised", but what I actually did was hand-normalise the one redirect I already knew about (Anthropic's Re-running it properly — resolving every url's effective target with So: a partial-coverage audit reproduced a wrong answer twice, and the thing that saved it was the script naming its own gap rather than reporting a bare number. Correct figure for the record: 59 entries, 54 distinct effective sources, 3 sources backing more than one entry, covering 8 entries. Reuse is still bounded — 54 distinct for 59 is healthy — but it is 3 clusters, not 2. |
structured-outputs A og:title "Structured model outputs | OpenAI API";
on-page counts strict x281, json schema x33,
structured outputs x63 — the entry's subject is
what the page is about, not a topic it brushes
latency-streaming B "Latency optimization | OpenAI API" — one
vendor's advice for its own API surface
rate-limiting-backpressure B "Rate limits | OpenAI API" — documents this
vendor's headers and tiers; the entry's
backpressure argument (queueing, shedding, retry
budgets) is a systems claim the page never makes
tool-schema-design B "Function calling | OpenAI API"; also the shared
effective source behind tool-use, and the
schema-DESIGN advice exceeds what the guide says
Method note recorded in each entry's evidence, because it nearly cost the batch:
these docs are CLIENT-RENDERED and `<title>` comes back EMPTY. My first probe
read that as "content unverifiable" and would have graded four entries on path
existence alone. The identifying text is in og:title and the body payload — a
3.1 MB response, fully greppable. An empty <title> meant the probe was wrong,
not that the page was opaque.
Negative control: guides/not-a-real-guide-xyz returns 404.
validate.mjs: 59 valid, warnings 30 -> 26. dist: unrated 26, A 16, B 17.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
agent-security B CIRCULAR CITATION (below)
agent-observability B "Traces | OpenTelemetry" — defines spans, context
propagation and sampling generically; says nothing
about tool-call attribution, token/cost accounting or
trajectory replay, which is what the entry is about
context-rot A "Context Rot: How Increasing Input Tokens Impacts LLM
Performance" — the source IS the study the entry
describes, not a secondhand account of it
agent-security cites https://agent-loop.xyz/library/agent-security. This repo's
own description says it powers the RAG at agent-loop.xyz, so that url is a
DERIVATIVE of this KB, not independent evidence for it. The entry's four-pillar
map may be entirely correct — but this citation cannot be what establishes it.
Weak corroboration that the page is KB-derived: of 4 eight-word shingles from
the entry, 1 appears verbatim on the page. The page is 38KB and partly
client-rendered, so that is suggestive, not proof, and is recorded as such
rather than written up as a confirmed generation path.
Actionable either way: repoint the url at the primary sources the pillar rests
on, or drop it and let the pillar stand on the per-surface entries it links.
validate.mjs: 59 valid, warnings 26 -> 23. dist: unrated 23, A 17, B 19.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
determinism-reproducibility A "Defeating Nondeterminism in LLM Inference" —
the source IS the investigation the entry
describes, same basis as context-rot's A
durable-agent-execution B h1 "Understanding Temporal"; workflow x41,
durable execution x7, replay x3 — the page
covers the subject, but Temporal is ONE
implementation and the entry argues the
pattern generally
computer-use-agents B "Computer use tool - Claude Platform Docs" —
one vendor's tool reference against an entry
that generalises to the class
Method note recorded in durable-agent-execution's evidence: that page returns an
empty <title> AND an empty og:title, with the identifying text only in the h1 and
body. It is the THIRD source in this PR where an empty title meant my probe was
wrong rather than the page being opaque (a2a-protocol was a client-side redirect
stub, the OpenAI guides were client-rendered). Recording it in the entry rather
than only in a commit message, since the next person to verify these urls will
hit the same thing.
Negative control: docs.temporal.io/evaluate/not-real-xyz returns 404.
validate.mjs: 59 valid, warnings 23 -> 20. dist: unrated 20, A 18, B 21.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
planning-decomposition A "[2210.03629] ReAct: Synergizing Reasoning and
Acting in Language Models" — the paper that
introduced the loop the entry describes
prompt-injection-defense A UK NCSC, "Prompt injection is not SQL injection
(it may be worse)". The title states the entry's
own argument, and it is a national authority
rather than a vendor — rare for this topic
memory-poisoning B the url is the LLM Top-10 INDEX, not the specific
risk page. The index shows OWASP tracks the class;
it does not carry the poisoning detail the entry
asserts. Repointing at the individual risk page
would make this an A
tool-retries-idempotency B strong primary treatment of timeouts, retries and
jittered backoff — but the entry also claims
IDEMPOTENCY discipline, which retry guidance
assumes rather than establishes
memory-poisoning is the actionable one: index-vs-item is a one-line url fix that
moves a B to an A without touching the entry's content.
Method note in tool-retries-idempotency: that page's <title> renders as the
generic "AWS Builder Center" shell rather than the article name, so the title is
not usable for identification and the url path is what pins the article. Fourth
distinct title-probe failure mode in this PR, after client-side redirect stubs,
client-rendered docs, and empty og:title.
validate.mjs: 59 valid, warnings 20 -> 16. dist: unrated 16, A 20, B 23.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eu-ai-act-agents A "Article 50: Transparency Obligations..." —
primary legislative text, and the provision that
creates the duties the entry describes
legal-agents A Stanford HAI, "Hallucinating Law: Legal Mistakes
with Large Language Models are Pervasive" —
empirical, academic, vendor-neutral
sandboxing-code-execution B "Firecracker" — one isolation mechanism against
an entry arguing sandboxing generally
healthcare-agents is NOT graded, on purpose. Its source
(hhs.gov/hipaa/for-professionals/security/index.html) returns HTTP 403 Access
Denied — with curl's default UA and with a browser UA. Two attempts, no more:
hammering someone else's production site to satisfy my own audit is not a
trade worth making.
403 is a refusal, not a 404, so this says nothing about whether the page is
fine for a normal browser. The honest state is "source unverifiable from here",
and GRADING.md already has a value for that: unrated. Grading it anyway would
have meant asserting evidence I never saw, which is the failure this whole PR
is trying to remove from the KB.
Both facts are recorded in the entry's own evidence list so the next person
does not repeat the two fetches.
validate.mjs: 59 valid, warnings 16 -> 13. dist: unrated 13, A 22, B 24.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
subagents A "How we built our multi-agent research system" — an
engineering report of a system actually shipped, on
exactly the entry's subject
prompt-caching A "Prompt caching - Claude Platform Docs". Prompt caching
is a VENDOR-SPECIFIC API feature, not a neutral
technique, so the vendor's reference doc IS the primary
source. The vendor-scope caveat that pushed other
entries to B genuinely does not apply here
context-compaction B "Effective context engineering for AI agents" — covers
context engineering broadly; the entry is specifically
about compaction, so the essay supports the discipline
more than the mechanism
agent-loop B the fourth and closest user of the shared essay: its
subject IS the loop, so it has the best claim of the
four sharing that url
prompt-caching is worth calling out because it is the one place tonight where
"vendor source" is not a weakness. Applying the B caveat mechanically there
would have been consistency mistaken for judgement — the claim is about a
specific vendor's feature, so the vendor is the right authority.
validate.mjs: 59 valid, warnings 13 -> 9. dist: unrated 9, A 24, B 26.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
human-approval-gates B "Trustworthy agents in practice" — motivates the
need for gates more than it specifies them
model-selection-adaptation B "Choosing the right model" is correct for
selection among one vendor's models; the entry
also claims runtime ADAPTATION, which a
selection guide does not cover
token-budgets B cites https://www.aisi.gov.uk/ — an organisation
HOMEPAGE, not a document
token-budgets is the sharpest citation defect found in this PR. A homepage is
not evidence: it carries no stable claim and its content changes with the org's
front page. AISI is a serious institute and nothing here doubts that — but
nothing at that url addresses token budgeting.
Two weak-citation shapes have now turned up, and they are not equally bad:
index-instead-of-item memory-poisoning -> OWASP LLM Top-10 index
homepage-instead-of-doc token-budgets -> aisi.gov.uk root
The index at least scopes the topic; the homepage does not. Both need
repointing, and both are one-line url fixes that would move a B to an A without
touching entry content.
validate.mjs: 59 valid, warnings 9 -> 6. dist: unrated 6, A 24, B 29.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…commit
tool-use A four primary sources, claims split cleanly across them;
idempotency cited to Stripe's own API reference
agent-skills-vs-mcp B a COMPARISON sourced from one of the two things compared
ecommerce-agents B FTC guidance is the strong half; the protocol is moving
sales-agents B eCFR TCPA is the strong half; the other is a product page
operations-agents B one vendor marketing page, nothing beside it
RETRACTION. The previous commit (2828b03) called token-budgets "the sharpest
citation defect" and diagnosed it as homepage-instead-of-document, presenting
that as a general shape worth scanning for. I then scanned for it, and the
measurement disproved my own claim:
6 of 59 entries cite a site root — not 1, as that message implied
4 of those 6 serve canonical documentation AT the root and are well sourced
a2a-protocol.org -> the protocol's own reference
modelcontextprotocol.io -> "What is the Model Context Protocol"
agentskills.io -> "Agent Skills Overview"
agenticcommerce.dev -> "Agentic Commerce Protocol"
So root-ness is not the axis. The real axis is whether the cited resource
ADDRESSES THE ENTRY'S SUBJECT. token-budgets is still the outlier and still a
B, but for topic mismatch — aisi.gov.uk says nothing about token budgeting —
not for being a root url. Its evidence block has been rewritten to carry the
correct diagnosis plus the retraction, because the wrong general rule was
shipped INTO the entry and would otherwise be read as a grading standard.
The distinction that survives: documentation-at-root is fine; a vendor PRODUCT
MARKETING page is weak (operations-agents, sales-agents), because it is
evidence a capability is claimed, not that it works.
validate.mjs: 59 valid, warnings 6 -> 1. dist: A 25, B 33, unrated 1.
The one remaining unrated is healthcare-agents, deliberately left so — its
source returns 403 and I will not grade evidence I could not read.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Went looking for a reachable replacement source for healthcare-agents, the one
deliberately-unrated entry (its hhs.gov citation 403s). Ran a nonsense-path
control against each candidate host, and the control is what caught this:
www.ecfr.gov/current/title-47/section-64.1200 200 "Request Access"
www.ecfr.gov/zz-nonexistent-control-9999 200 "Request Access"
www.ftc.gov/business-guidance/…/endorsement-guides 403
www.ftc.gov/zz-nonexistent-control-9999 403
Both hosts serve a bot challenge, and ecfr returns it with HTTP 200 for any
path — so a bare 200 from that host is a catch-all that proves nothing.
I had already graded two entries on those sources and written that they were
"fetched live":
sales-agents "the eCFR citation is the strong half and is exactly right"
ecommerce-agents "the FTC citation is the strong half — a regulator's own
guidance is the right authority"
I never read either page. Both evidence blocks now say so.
The sharp part is that healthcare-agents already carries the correct standard,
written by me in this same PR: "This is a statement about MY access, not about
the page." I applied it there, refused to grade on a 403 — and then two entries
later called an unread 403 page the strong half of the evidence.
Grades stay B; the REASONS moved, and both got weaker:
- ecommerce-agents now rests on the protocol site alone, an emerging spec
- sales-agents now rests on a VENDOR PRODUCT MARKETING page alone, which is
exactly the shape I flagged operations-agents for in this same PR
Verified-with-control and genuinely fine, for contrast: csrc.nist.gov SP 800-207,
nist.gov AI RMF, nist.gov Privacy Framework — each 200 against a control that
404s. finance-agents is graded on the Privacy Framework and is unaffected,
though its BODY carries a dead federalreserve.gov SR 11-7 link (real 404, control
404s too) worth repointing separately.
healthcare-agents stays unrated: no reachable substitute exists from here.
validate: 59 valid, 1 warning. dist unchanged: A 25, B 33, unrated 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ontrol The mechanism for the failure two commits ago, where two entries were graded "fetched live" on pages that were never read. A bare status check cannot catch that: www.ecfr.gov answers 200 with "Federal Register :: Request Access" for EVERY path, so the 200 is a catch-all. Only a path that should NOT exist discriminates. The script fetches each source plus one nonsense-path control per origin and classifies ok / catch-all / blocked / dead / error. Deliberately NOT a CI gate: it makes live third-party requests, so per-push runs would be flaky and rude. 400ms between requests, one control per origin. Two bugs in the script itself, both caught before it was trusted: 1. It probed only the front-matter `url` — but the two entries that PROMPTED it cite their regulator source in the BODY. As first written it could not have caught the failure it exists for. Body links now count: 56 front-matter -> 132 targets. 2. The body extension keyed on `e.__body`, which loadAll() does not expose (the field is `body`). It ran clean and reported the same 59 sources — a silent no-op that looked like coverage. Verified discovery is non-zero (186 links across 42 entries) before believing it. First real run, 132 targets, 5 unverifiable: catch-all B body sales-agents ecfr.gov (the documented case) dead A url lesson-gitlab-2017-backups 404 dead B body finance-agents federalreserve SR 11-7 404 blocked B body customer-support-agents canlii.org 403 blocked B body groundedness-hallucination canlii.org 403 Acted on two: - **ecommerce-agents**: ftc.gov RESOLVED. It 403s under curl and returns 200 under node's fetch — the 403 was a property of my request, not the page. The title matches the claim, so the FTC guidance is the right authority after all. Both prior evidence lines are corrected: the first overclaimed, the second was accurate about access but understated the source. - **lesson-gitlab-2017-backups**: graded A on a URL that 404s (control 404s too, so it is a real removal). Flagged in evidence, grade LEFT at A deliberately — the lesson is widely documented and its substance is not in doubt, but the citation needs repointing and that is a decision to surface, not to make silently. canlii.org and the Federal Reserve link are left reported, not changed. validate: 59 valid, 1 warning. dist: A 25, B 33, unrated 1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`scripts/check-sources.mjs` landed last commit as the mechanism for grading two
entries on pages that were never read. A checker nobody is told to run is not a
mechanism, so it is now a documented step:
GRADING.md new section before Lifecycle — run it before writing
"fetched live", with the three rules the incident produced
CONTRIBUTING.md step 5 in "Add or update an entry", gated on touching `grade`
The three rules, kept where a grader will read them rather than in a commit log:
1. A 200 is not a read. www.ecfr.gov answers 200 with "Federal Register ::
Request Access" for EVERY path, so compare the title against a control.
2. A failure describes your CLIENT, not the page. www.ftc.gov returned 403 to
curl and 200 to node's fetch on the same URL minutes apart — try a second
client before concluding anything about a source.
3. When a source genuinely cannot be verified, say so ("HTTP 403 from here, not
read") rather than grading it. Leaving an entry unrated on an unreadable
source is a correct outcome, not a gap.
Also documents why it is deliberately not a CI gate: live third-party requests
per push would be flaky and impolite to the sites being cited.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…h 'Open a PR' Both were numbered 5 after the previous commit. Markdown renders that as 5,5 rather than resequencing, so the list read as two step fives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… body links
Worked the checker's own output instead of only reporting it.
**canlii (2 entries) — flagged, but NOT a defect.** Applied the "a 403 describes
your client" rule: retried the cited url with node's fetch as well as curl. Both
403, and the same-origin control 403s too, so canlii refuses automated access at
the host level. But neither entry's grade rests on it — `customer-support-agents`
is graded on anthropic.com and `groundedness-hallucination` on
learn.microsoft.com, each already carrying its own negative control. The canlii
link is a body citation for the Air Canada chatbot case. Nothing to correct.
That exposed a flaw in the checker: `--strict` failed on ANY unverifiable source,
so it would have failed those two despite their grades being properly sourced. A
check that cries wolf gets demoted to the one nobody runs. `--strict` now fails
only when the FRONT-MATTER url — the graded source — is unverifiable; body
citations are reported as "worth repointing, but the grade may still stand". On
the last run that is 1 blocking, not 5.
**lesson-gitlab-2017-backups — the one real blocker, fixed.** Its graded source
404s (GitLab removed the post). Repointed at the Wayback capture of the SAME
post, verified with a control:
archive copy 200 "Postmortem of database outage of January 31"
fabricated path on web.archive.org 404 "Wayback Machine"
So the 200 is a real capture rather than a catch-all. This restores the citation;
it does not re-source the entry, and the grade is untouched.
Left alone: the Federal Reserve SR 11-7 link in finance-agents' body. Both the
cited path and an obvious alternate 404, I could not find the current location,
and it is a body citation — finance-agents is graded on the NIST Privacy
Framework, which verifies. Reported rather than guessed at.
Methodology note on myself: I first tested a canlii URL I had invented rather
than the one the entries cite. Read the actual citation before probing it.
validate: 59 valid, 1 warning. dist unchanged: A 25, B 33, unrated 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…wn as an A Two things on the front page were made false by this PR itself. **The entry-format example was `token-budgets`, displayed as `grade: A`** with the evidence line "AISI eval, 1M→10M tokens = +25% SWE success". This PR graded that entry **B** precisely because nothing at the cited aisi.gov.uk url addresses token budgeting — so the README was teaching the format with a specific quantitative claim attributed to a page that does not support it, at a grade the data no longer carries. Swapped to `agent-delegated-auth`: grade A, cites RFC 9728, verified live with a matching title. Added one line on why `evidence` is the load-bearing field — it names what was checked and when, so a reader can re-run the check. **"Seeded from 36 curated entries", "MVP / seed"** — it is 59 entries, 58 evidence- graded (25 A, 33 B). The status section now says that, plus why exactly one is deliberately `unrated`, and points at check-sources.mjs with the one-sentence reason a bare 200 is not proof a page was read. Every number was read from dist/kb.json rather than from the PR description, and the showcase entry's grade and url were checked against the same file. Left alone: the claim about the maintainer loop and site-ingestion wiring. I could not verify the current state of either from here, and replacing one unverified sentence with another is not an improvement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
john-the-dev
changed the title
This branch graded mcp A on source-quality signals: official docs, HTTP 200, "no superseding protocol doc found at the same origin". A separate pass read the entry against the live spec instead and contradicted that last clause — the entry cites specification/2025-06-18 in two places while the current revision is 2026-07-28, which also makes MCP stateless with a mandatory server/discover and deprecates sampling and logging. The entry mentions none of that. GRADING.md defines A as "verified accurate within the last 90 days", so an accurate-source argument does not reach A when the entry's own citations are a revision behind. B with the caveat stated in evidence is the rubric's answer. Folds in the grading from #24, which is closed as superseded by this branch: both were open at once and assigned different grades to the same entry.
john-the-dev
added a commit
that referenced
this pull request
Aug 31, 2026
"48 of 59 entries are grade: unrated" was true when written and false by the time this branch was pushed — #21 grades 58 of 59, and it was already open. A number whose truth depends on which of two open PRs merges first should not be in the README at all. The CTA now points at what validate.mjs reports rather than a snapshot of it, and names re-grading as equally welcome — which is the contribution surface that survives #21 landing.
Owner
Author
|
Self-check before this merges — I went looking for a defect in my own grading and did not find one, but the coverage is not uniform and that is worth stating. Compared every A-graded entry on this branch against So the branch graded exactly what it verified: zero entries were promoted on evidence it did not fetch. That was the failure mode I was checking for. The 11 pre-existing A's carry Listing it so nobody reads "24 A grades" as "24 verified against a live source". If it should be closed, that is a separate pass over 11 entries, not a change to this one. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Grades the KB's entries against their actual sources, and adds the tool that keeps the grades honest.
Before this branch,
gradewas populated 59/59 but 48 of those values were the literal string"unrated"— so counting populated fields said 100% graded while reading the values said 19%. The repo's own description promises a "quality-graded" KB, and that mismatch was the gap.The one deliberate
unratedhealthcare-agentsstays unrated because its source returns 403 to every fetch method I tried. Grading it would mean assigning a confidence I have no basis for. An honestunratedis the correct value there, not a rounding-up.scripts/check-sources.mjsProbes every source — front-matter
urland body links — and classifies each as ok / catch-all / blocked / dead / error. It tests each origin against a nonsense path first, so a site that returns 200 for everything is detected as a catch-all rather than counted as a live source.Deliberately not wired as a CI gate: third-party availability is not this repo's correctness, and a gate that fails on someone else's downtime gets disabled.
--strictfails only on a source that a graded entry depends on.Two retractions in the history, kept rather than squashed
I published "fetched live" for two sources I had not actually read (
ecfr.gov,ftc.gov) and retracted both in40ce9f6. Theftc.govone later resolved — it 403s tocurland returns 200 to node'sfetch— which is exactly the kind of thing the source checker now catches instead of my assertion.The commit history is left intact so the corrections are visible rather than tidied away.
Scope note
The original title said "3 of 48", which described the first commit and not this PR — 22 commits later it is the full grading pass plus tooling. Retitled rather than leaving a label that undersells what needs reviewing.