Measure and chart semantic strings and paths per release - #257
Merged
Merged
Conversation
Spec for #252. Three history files and three charts drawn by one renderer, with the quantities chart required to come out byte-identical through the refactor. Records why the strings cost pairs are measured against hand-written validation rather than a bare string: the bare counterpart of Uuid.Create is an assignment, so pairing against it would report a large ratio that only restates that validation is not free. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eleven tasks. The first makes the renderer subject-aware and is gated on the quantities chart coming out byte-identical, which is the only real regression test this pipeline has. Corrects two API names in the spec found while writing the plan: RemoveExtension rather than WithoutExtension, and AsAbsolute measured from a relative path because the absolute one returns this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The headline set, the grid width and the two title strings move onto a Subject record looked up by --subject. Ingest is untouched: it never read either, and the results directory is already per-subject because the filter selected it. The quantities chart renders byte-identical, which is the whole test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Seven rungs from the reflection machinery alone up to the mod-97 check, using shipped identifier types so the numbers describe types a caller actually holds. The one fixture is the no-validation floor, which nothing shipped occupies. Both failure paths are measured, so the cost of Create over TryCreate at a boundary that sees bad input is a number. The two rows come out nearly identical because SemanticString.TryFromString throws and catches an ArgumentException internally on every rejection, so TryCreate is exception-free in the caller's control flow only, not in the caller's cost -- documented in the class remarks. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Task 2's measurements disproved the plan's assertion that a throwing rejection costs far more than a non-throwing one. SemanticString's TryFromString is try/Create/catch, so TryCreate throws and catches internally on every rejection and both paths pay a full exception. The failure mode to watch for is the opposite of what was written: a cheap failure row means the specimen is being accepted.
Equality, ordering, hashing and the conversion back out, plus the two members that read as ordinary calls and are really full creations against the target type. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Task 3 measured CompareTo at 0.5714 ns and BenchmarkDotNet reported ZeroMeasurement: the JIT hoists it, so a panel would chart the harness's resolution rather than the library. Same effect the quantities suite already documents for relationship operators on a double. HashCode measures cleanly at 43.8881 ns and takes the eighth panel. Both benchmarks stay in the class; only what is drawn changes.
Not against a bare string: the bare counterpart of Uuid.Create is an assignment, and a ratio against no work at all would only restate that validation is not free. Paired against the check a caller would have written anyway, the ratio separates the validation both sides pay from the reflection only one side does. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The baseline was string.CompareOrdinal while SemanticString.CompareTo forwards to string.CompareTo, which is culture-sensitive. The pair was measuring ordinal collation against culture collation and reporting the difference as the wrapper's cost. The correction surfaced a property of the library worth recording: equality on a semantic string is ordinal and ordering is not, so two values can compare equal under == and sort by a different rule.
…irs against SemanticString.CompareTo forwards to String.CompareTo(String), which is culture-sensitive, but the baseline used string.CompareOrdinal. That compared ordinal collation against culture collation rather than isolating the wrapper's cost. The baseline now calls WeakString.CompareTo(WeakString) directly, the same call the semantic side ultimately makes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Absolute specimens are built per platform because IsAbsolutePath asks Path.IsPathFullyQualified, whose answer differs between Windows and Linux; a hardcoded Windows root would throw on every CI run and pass locally. The cached and uncached file name properties are measured side by side, because both read like field access and one is a full creation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CA1802 was correct: RelativeFile holds a compile-time literal with nothing computed at runtime, so const is what it should have been. The prior suppression bought nothing over taking the one-word fix, and grouping it with AbsoluteFile/AbsoluteDirectory (which must stay static readonly, since they call Path.Combine per platform) was stylistic rather than required. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A straight comparison, unlike the string one: Path is a real API doing the real work. What the semantic side adds is a validated wrapper around every result, so the ratio is what a caller pays for a path that cannot be passed where a different kind belongs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Strings along validation weight, which is the axis there is when the cost is concentrated at creation. Paths the same shape, with the cached and uncached file name panels adjacent because both read like field access and only one is. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…eport Adds two remarks paragraphs: the semantic side costs about the same across all four categories, and Create's much larger ratio is a property of its baseline being unusually cheap, not of creation being unusually expensive. Also notes the separator asymmetry between BareCreate and the validator it stands in for directly on BareCreate's summary, where a reader of the class can see it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The subjects documentation still said the eighth panel was an ordering, which it stopped being when CompareTo turned out to be hoisted and HashCode took its place. Says hashing now, and says why ordering is measured but not drawn.
The Subjects documentation described panel 8 as an ordering benchmark after the ZeroMeasurement swap replaced CompareTo with HashCode in the panel list; the prose was never updated to match. Corrects the bottom row description and adds a paragraph on why CompareTo is measured but not drawn, matching the precedent set by the quantities paragraph. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
One job rather than a matrix, so the reference workload is read once and stamped on every entry: that is what makes a strings point and a quantities point from the same run comparable, and it keeps the results to one push. The dispatch gains a subjects input, which is how a long backfill gets split rather than by raising the timeout. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every command inside a `while ... done <<< "$SUBJECTS"` loop inherits the here-string as its own stdin. BenchmarkDotNet prompts interactively when a filter selects nothing, which the backfill's version-skip path deliberately triggers for older packages -- so a drained stdin would silently truncate a loop to one subject while the step still exits zero. Redirecting each invocation from /dev/null closes that off. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The plan predicted the mod-97 check would be the heaviest validator. It is a tie with the format regular expression, and more to the point the whole spread across four structurally different validators is about 340 ns on a floor of 1,650 ns: the reflection machinery is the bill and the validator is a minor term. The seeding expectation is rewritten to match, and now names the real failure mode -- a validator row sitting at the unvalidated floor, which would mean its validator never ran.
Measured on one machine with one reference reading, recorded as local-seed the way the quantities history was. baselineNs is what lets these sit alongside the CI points that follow. The creation-benchmark gate check found the ladder flatter than the class docs predicted: Mod97 and FormatRegex tie within noise, so the "heaviest shipped validator" claim on Mod97 was wrong. Corrected the doc comment to state what was measured instead of what was assumed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Measured as published packages by today's benchmarks, which is the better comparison than checking out each tag: every version is timed by identical code rather than by whatever each tag shipped. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The README gains a chart per library, with the way to read a chart stated once above all three rather than three times. The strings section says plainly what the chart shows: a quantity's wrapper is free and a semantic string's is not, and what that buys. CLAUDE.md gains the benchmark project, which it never listed, and the three things about the pipeline that are not guessable from the code. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CLAUDE.md said "three things" while listing four; folds the 3.3.1 build-failure point into the bullet list as a fourth bullet instead of a trailing paragraph the intro line didn't count. Semantics.Benchmarks/README.md had the quantities class table and its two subsections nested under the shared "Measuring a published release" heading rather than under "Quantities", because the shared sections sit between the two quantities blocks in reading order. Moved the block so the quantities material is contiguous. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Corrects seven documentation issues found by a final whole-branch review of the strings/paths benchmarks work: a stale "ordering" panel reference, an allocation-exactness claim that does not hold for the paths chart (whose inputs are built per platform), a workflow comment that contradicted the strings/paths filters' whole-class design, an unexplained gap between the validators/benchmarks the prose names and what the charts draw, a wrong opening sentence in the benchmarks README, a workflow comment claiming a not-yet-true cross-chart baseline guarantee, and two disproven predictions left standing in the design spec. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
|
||
| /// <summary>A fully qualified file path, four segments below the root.</summary> | ||
| internal static readonly string AbsoluteFile = | ||
| Path.Combine(Root, "semantics", "src", "Semantics.Paths", "FilePath.cs"); |
|
|
||
| /// <summary>The directory that file sits in, used as the base for both conversions.</summary> | ||
| internal static readonly string AbsoluteDirectory = | ||
| Path.Combine(Root, "semantics", "src"); |
Main grew a third chart section in the same functions this branch made subject-aware, so the two refactors had to be reconciled rather than picked between. The cost pairs move from a standalone array onto the Subject record: quantities keeps the four main added, strings and paths declare none, and a subject with no pairs draws two sections and reserves no height for a third. The quantities chart still renders byte-identically, now against main's three-section version, which is the same gate this branch has held at every step. Main also fixed the CS8618 that made pre-4.0.0 packages unbuildable, so 3.3.1 is measurable again. Both new histories are backfilled to it and all three subjects now start there. The documentation said that version was permanently out of reach; it says instead what the episode actually teaches, which is that one project shared by three subjects means a compile error for one costs the other two their history. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
This was referenced Sep 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Written by Claude (Claude Code) on behalf of Matt Edmondson.
Closes #252.
Semantics.Quantitieshas had a measured, charted performance history since #242.Semantics.StringsandSemantics.Pathshad none. This gives them the same treatment, and the pipeline they share is now subject-aware rather than single-subject.What the numbers say
The quantities suite's headline finding was that the wrapper is free. The strings finding is the opposite, and that is the point of measuring it.
TryCreateis not the cheap option its name suggests.SemanticString.TryFromStringis implemented astry { Create(...) } catch (ArgumentException) { return false; }, so it throws and catches internally on every rejection. Against a hand-written regex check that returnsfalse, the ratio is 165x. At a boundary that rejects often, this is the most actionable number here.The validation ladder is nearly flat. Four structurally different validators cost 0.93, 1.06, 1.26 and 1.27 microseconds on top of a ~1,650 ns reflection floor. A Luhn pass, a character-set regular expression, a mod-97 pass and a format regular expression all land within about 340 ns of each other: the validator is a minor term and the reflection machinery is the bill. The plan predicted mod-97 would dominate; it ties with the format regex.
Equality is ordinal and ordering is not. Record
==routes throughEqualityComparer<string>.Default;CompareToforwards tostring.CompareTo(string), which is culture-sensitive. Two values can compare equal under==and sort by a different rule. Reported, not changed.FileNamerebuilds on every read. 2,528 ns and 944 B, against 2.1 ns and no allocation for the cachedFileNameWithoutExtension. A 1,181x gap between two properties that look identical at a call site.Cost-pair ratios, semantic against the code a caller would otherwise write:
The path
Createratio is large because its baseline is cheap (a single boolean check), not because creation is unusual: the semantic side costs about the same in all four path categories. Its baseline also deliberately omits the separator concatenationIsAbsolutePathAttributeperforms, because a baseline here means the code a caller writes rather than the library's internals. Both points are stated beside the table.Backfill coverage
Measured as published packages, so every version is timed by identical benchmark code.
No versions skipped. An earlier revision of this branch stopped both new subjects at 4.0.0, because
AbstractionCostBenchmarks.csheld aLength<T>field with no initializer and would not compileagainst a package where
Length<T>was still a reference type. Since all three subjects share onebenchmark project, that quantities-only file took the strings and paths backfills down with it.
Main has since fixed the field, so 3.3.1 is reachable and both new histories start there.
The coupling is worth keeping in mind even though this instance is fixed: a compile error introduced
for one subject costs the other two their history, and it surfaces as a build failure in a file the
person running the backfill was not touching. Both
Semantics.Benchmarks/README.mdandCLAUDE.mdsay so.
What changed
Semantics.BenchmarksgainsStrings/andPaths/folders: creation, operation and cost-pair classes for each. Specimens are shipped types (Uuid,Ulid,CreditCardNumber,Iban, the path types) rather than fixtures, except one unvalidated fixture for the no-validation rung, which nothing shipped occupies.scripts/benchmark-history.cstakes--subjectonrenderand looks the chart up from a dictionary.ingestis untouched: it never read the headline set. The quantities chart renders byte-identically, which was the acceptance test for the refactor and is re-verified at every step of this branch..github/workflows/benchmark-history.ymlloops a subject table, gains asubjectsdispatch input for splitting long backfills, and raises its timeout to 360 minutes.README.md,Semantics.Benchmarks/README.mdandCLAUDE.md, which did not previously mention the benchmark project at all.Merged with main
Main grew a third chart section (the quantity wrapper's cost over the bare storage type) inside the
same
DrawandSectionfunctions this branch made subject-aware, so the two refactors werereconciled rather than one picked over the other. The cost pairs moved from a standalone array onto
the
Subjectrecord: quantities keeps the four main added, strings and paths declare none, and asubject with no pairs draws two sections and reserves no height for a third.
The quantities chart still renders byte-identically, now against main's three-section version. That
check has gated every renderer change on this branch.
Charting cost pairs for strings and paths is deliberately not done here. The classes exist and their
ratios are in the table above, but the release workflow does not run them, so no history carries the
pair to divide. Worth its own issue now that main has set the precedent for quantities: the strings
Reject ratio is the number most worth watching per release.
Notes for review
Two benchmarks are measured and stored but not drawn, deliberately:
CompareTo(0.57 ns, hoisted by the JIT, so a panel would chart the harness's resolution) andToStringImplicit.ingestrecords every row and onlyrenderselects, so either can be promoted later without re-running any history.The paths chart's allocation row is the one place the usual "allocation is exact on any machine" rule does not hold: path specimens are built per platform, because whether a path is absolute is a question the operating system answers differently. All committed paths points are Windows-measured; the first CI point will be Linux. Stated beside the chart.
Neither workflow path has ever run on a runner. That predates this change, but this change triples what a first run must get right. Suggest cutting the first release after merge deliberately and watching the job, and dispatching the backfill with
subjects: stringsalone before trusting the full default.Three follow-ups worth their own issues: a
verify-chartsdrift guard (re-render from committedhistory, fail on drift), which would protect the committed SVGs the way
verify-generatedprotectsgenerated sources; cost-section panels for the strings and paths charts, now that main has set the
precedent for quantities; and the shared-project build coupling described above, which is mitigated
by a comment rather than by structure.
🤖 Generated with Claude Code