Skip to content

grade(mcp): first evidence-graded review — B, citations one spec revision behind - #24

Closed
john-the-dev wants to merge 1 commit into
mainfrom
grade/mcp-entry
Closed

grade(mcp): first evidence-graded review — B, citations one spec revision behind#24
john-the-dev wants to merge 1 commit into
mainfrom
grade/mcp-entry

Conversation

@john-the-dev

Copy link
Copy Markdown
Owner

Taking my own suggestion from #22"One entry, graded with dated evidence, is a complete and useful PR." This is that PR, so the invitation has a worked example attached rather than just a claim.

What I verified

Against the primary source the entry itself cites (modelcontextprotocol.io/docs/learn/architecture, checked 2026-08-31):

claim holds?
three participants: host / client / server
two layers: data + transport
data layer is JSON-RPC 2.0
transports: stdio + Streamable HTTP (optional SSE)

Nothing in the entry is wrong.

Why B and not A

The citations have aged past the content:

  • the entry links specification/2025-06-18 in two places; the live spec is 2026-07-28
  • its last_verified read 2026-07-29 — one day after that revision — so the date asserted a freshness the body did not have
  • 2026-07-28 makes MCP stateless with a mandatory server/discover, and deprecates sampling and logging. This entry frames the data layer around lifecycle messages and mentions neither server/discover nor elicitation

Per GRADING.md that is B — "accurate-but-aging, note the caveat in evidence", not C. C is partially superseded; here the structure is current and only the references trail it.

Deliberately not in this PR

  • Not refreshing the citations. Updating the 2025-06-18 links and the lifecycle framing is a content change; this is a grading change. Mixing them would make the grade unreviewable.
  • Not absorbing the stateless revision. entries/mcp-stateless.md already covers it as its own entry.

Result

validate.mjs: ✅ 59 entries valid.  warnings 48 -> 47

47 remain. The evidence items are written as dated, checkable statements per the rubric's own examples, so a later reader can re-run them rather than trust them.

First evidence-graded review of entries/mcp.md, following the invitation in #22.

Verified against the primary source the entry itself cites
(modelcontextprotocol.io/docs/learn/architecture, checked 2026-08-31): three
participants, two layers, JSON-RPC 2.0 data layer, stdio + Streamable HTTP.
All four hold, so nothing here is wrong.

B rather than A, because the citations have aged past the content:

- the entry links specification/2025-06-18 twice; the live spec is 2026-07-28
- its own last_verified said 2026-07-29 — one day AFTER that revision — so the
  date asserted a freshness the body did not have
- 2026-07-28 makes MCP stateless with a mandatory server/discover and deprecates
  sampling and logging; this entry frames the data layer around lifecycle
  messages and mentions neither server/discover nor elicitation

That is the rubric's "accurate-but-aging, note the caveat in evidence", not
"partially superseded" — the structure is current, the references are not. The
revision itself is already covered by the sibling entry mcp-stateless, so this
entry does not need to absorb it; it needs its citations refreshed, which is a
separate change from grading it.

validate.mjs: 59 entries valid, warnings 48 -> 47.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
john-the-dev added a commit that referenced this pull request Aug 31, 2026
This branch graded mcp A on source-quality signals: official docs, HTTP 200,
"no superseding protocol doc found at the same origin". A separate pass read the
entry against the live spec instead and contradicted that last clause — the entry
cites specification/2025-06-18 in two places while the current revision is
2026-07-28, which also makes MCP stateless with a mandatory server/discover and
deprecates sampling and logging. The entry mentions none of that.

GRADING.md defines A as "verified accurate within the last 90 days", so an
accurate-source argument does not reach A when the entry's own citations are a
revision behind. B with the caveat stated in evidence is the rubric's answer.

Folds in the grading from #24, which is closed as superseded by this branch:
both were open at once and assigned different grades to the same entry.
@john-the-dev

Copy link
Copy Markdown
Owner Author

Closing as superseded by #21 — and the overlap was my error, not a coincidence.

#21 (opened last night) grades 58 of 59 entries. I opened this PR and #22 this morning against a remembered figure of "48 unrated", which #21 had already obsoleted. Two of my own open PRs ended up assigning different grades to the same entry: A here on #21, B here.

The disagreement is real and this PR was right. #21 graded mcp A on source-quality signals — official docs, HTTP 200, "no superseding protocol doc found at the same origin". This PR read the entry against the live spec and contradicted that last clause: it cites specification/2025-06-18 in two places while the current revision is 2026-07-28, which makes MCP stateless with a mandatory server/discover and deprecates sampling and logging. GRADING.md defines A as verified accurate within the last 90 days, so an accurate-source argument does not reach A when the entry's own citations are a revision behind.

So the grading here is folded into #21 verbatim (9829363), not discarded:

before  Grades: {"A":25,"B":33,"unrated":1}
after   Grades: {"A":24,"B":34,"unrated":1}

This PR's CI was also red for a reason worth naming — dist/kb.json was stale against the entry edit. Rebuilt as part of the fold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant