test(tooling): gate the whole scripts/ tree against a cp1252 console abort (BACKLOG #1030) - #387
Open
wshallwshall wants to merge 10 commits into
Open
test(tooling): gate the whole scripts/ tree against a cp1252 console abort (BACKLOG #1030)#387wshallwshall wants to merge 10 commits into
wshallwshall wants to merge 10 commits into
Conversation
…abort (BACKLOG #1030)
Enforcement was per-file and hand-placed: test_cli.py asserts one STRING is encodable,
test_announce_hook.py asserts one FILE is ASCII, test_session_mail.py names five mail scripts in a
literal list. None generalises, so a glyph reaching print() from any other script was caught only by
a human reading the diff -- and the class recurred at least three times, the third TONIGHT, when a
scanner I had just written died mid-scan on U+2194 and printed a partial list that read as complete.
THE GATE IS ON REACHING AN UNGUARDED STREAM, NOT ON BARE ENCODABILITY, and that is the whole design
decision the item flagged as its real difficulty. sys.stdout carries errors='surrogateescape', which
round-trips only lone surrogates in DC80-DCFF; every other unencodable codepoint still raises.
sys.stderr carries backslashreplace and never raises. That asymmetry, not a strict/non-strict split,
is why the same text survives on stderr and aborts on stdout.
So a file may carry non-cp1252 characters IF IT HARDENS ITS OWN STDOUT. That is not an exemption
list: it is a property of the file, checked mechanically, and it is the REMEDY rather than a promise
about one. scripts/docs/backlog_status_check.py is why the distinction is load-bearing -- its
argparse description quotes the machine-parsed banner alphabet CLAUDE.md section 11 protects, and
remediation text that cannot show an author the character it wants added is not actionable. A gate
that could not express that would fire on correct code and be switched off.
That script is therefore FIXED rather than scrubbed: a stdout reconfigure at the CLI entry point,
scoped so importers are untouched. Measured before: `--help` raised UnicodeEncodeError on U+2705.
After: exit 0, the gate still reports 517 items, and parse_items still imports clean.
THE THREE PROPERTIES THE ITEM NAMES, each kept and each tested:
* PRINTS AND PINS WHAT IT SCANNED -- a walk that collapses to zero files would otherwise report
clean forever. Asserted, not merely emitted.
* READS THE WHOLE FILE, never line by line. splitlines() CONSUMES U+2028/U+2029, so a
line-oriented scan is structurally blind to them; a test demonstrates that mechanism rather than
asserting it.
* NEVER SILENTLY DROPS A FILE -- an undecodable file is a FAILURE, not a skip.
MEASURED, NOT INHERITED: the item counted 43 characters across four scripts/ files on 2026-08-05;
today it is ONE file, and it is the one that must keep them. Three were cleaned in between. The
control set is built from characters with recorded failures behind them, via chr() so this test
stays cp1252-clean itself -- verified: 0 non-cp1252 characters in it.
Scope stated in the file: scripts/**/*.py only. The engine already hardens both streams at
__main__.py; .ps1 has no equivalent reconfigure so generalising there is a different decision; docs/
is out because BACKLOG.md is a sanctioned holdout for that same alphabet.
…ed count (BACKLOG #1030)
Docstring only; no behaviour change. 14 tests unchanged and passing.
The gate's design -- harden the stream, never scrub the file -- was argued from principle. This
records the concrete case that lives in this repository, because the next reader to see a lone
variation selector in a source file will think it is lint.
backlog_status_check.py carries one non-cp1252 character beyond the banner alphabet: a lone U+FE0F
inside the banner regex as [<class>]\uFE0F?\s. That is an OPTIONAL VS-16, letting a banner be written
with or without the selector -- the handling CLAUDE.md section 11 mandates for any regex touching
that alphabet. Invisible at the point of use.
Delete it and the `?` binds to the CHARACTER CLASS. The pattern STILL COMPILES, so nothing at author
time objects. It matches an indented continuation line, group("emoji") returns None, and the dispatch
evaluates `None in _CLOSED` where _CLOSED is a str -- TypeError on any run touching the real ledger.
Verified by rebuilding the module with the character stripped and running parse_items against
docs/BACKLOG.md.
Two failure modes, and the second is the dangerous one: LOUDLY today, since every gate calling
parse_items dies; SILENTLY later, when the first banner authored WITH a selector stops matching --
and no banner carries one today, so nothing would catch that regression on the day it arrives.
NO COUNT IS PINNED, DELIBERATELY, and the omission is the point. An earlier draft of this note said
the scrub "silently inverts the parser while every test stays green". That was FALSE -- it crashes --
and it was falsifiable in one command, which would have discredited a correct surrounding argument.
The qualifying-line count is also ref-relative and grows with every filed item: measured 74 on
origin/main and 78 on an unpushed branch in the same hour. A figure would be stale the moment it was
written, and re-reading it would reproduce it, which reads as verification. Same hazard the
conftest banner refuses for the same reason.
wshallwshall
enabled auto-merge (squash)
August 14, 2026 12:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gates every script under
scripts/against aborting a stock Windows cp1252 console.Filed as its own PR rather than widened onto #385: zero file overlap with the #1249/#1230/#1235 work, so there is no serialisation need and #385's scope stays honest.
The contract is NOT "no non-cp1252 characters"
It is: carry them and HARDEN the stream, or do not carry them. That distinction is the whole item.
scripts/docs/backlog_status_check.pymust carry them -- its argparse description quotes the machine-parsed banner alphabet CLAUDE.md section 11 protects, and remediation text that cannot show an author the character it wants added is not actionable. So it hardens instead:Measured, not theorised: on a stock cp1252 console,
--helpraisedUnicodeEncodeErroron U+2705 before this line.errors="replace"is deliberate and is NOT tolerance for mangled text. The codec was the defect and is fixed here to UTF-8; replacement is the backstop for a stream that cannot be reconfigured at all, so one exotic codepoint can never again truncate a gate's output mid-sentence. Scoped to the CLI entry point -- importers keep their own stdout untouched.Why the gate itself is well-built
Those two properties are the difference between a gate and a green tick. An undecodable file is exactly the case a naive scanner drops and then reports clean.
Verification, re-run by the Lander
A gate whose own test could abort the console it defends would be the joke version of this item, so that self-check matters.
No banner flip
Zero files under
docs/; authoring ledger content is not the builder's. The author also declined to pin a qualifying-line count, recording why beside it: the figure is ref-relative and grew 74 to 78 within one hour, so a pinned number would be stale on arrival and would re-read as verification.