Collapse templated overloads in std.utf

[LightBender]

Rationale

This PR is part of a series of PR's requested by @WalterBright to explore different methods of reducing template usage and how much we can expect to reduce template usage by in Phobos 3.

The purpose of this PR is to reduce the number of instantiated templates in object files that are generated by std.utf. Given the pervasive usage of std.utf throughout the ecosystem reducing template usage here can significantly reduce the template bloat. I used a test harness app to produce these stats but I did not include the harness with this PR.

Pre-review checklist

• I have performed a self-review of my code.
• If my PR fixes a bug or introduces a new feature, I have added thorough tests.
• If my changes are non-trivial and do not concern a reported issue, I have added a changelog entry.

LLM/AI disclosure

This PR was written entirely by an LLM/AI.

Before / After Metrics

Primary goal: fewer template instantiations for string overloads of stride, strideBack, codeLength, isValidUTF, and validate.
Secondary goal: smaller generated object size.

Object size

	bytes
old	627766
new	600890
delta	-26876 (−4.3%)

Object size is the better bloat signal. The trampolines remain templates (one instantiation per string type), but the large UTF-8 / UTF-16 bodies are now shared non-template const(C)[] implementations.

-vtemplates=list-instances

	old	new	delta
log lines	3600	3499	−101
summary records	115	121	+6
total instantiations	3485	3378	−107
distinct instantiations (sum)	809	767	−42
instance lines	3485	3378	−107

Focused APIs (DMD summary totals / distinct)

name	old total	new total	Δ total	old distinct	new distinct	Δ distinct
stride	24	21	−3	18	18	0
strideBack	37	34	−3	32	32	0
codeLength	60	60	0	33	27	−6
isValidUTF	9	9	0	9	9	0
isValidUTFImpl	0	9	+9	0	3	+3
validate	9	9	0	9	9	0
validateImpl	0	9	+9	0	3	+3
decode	19	9	−10	14	5	−9
decodeImpl	28	19	−9	18	13	−5
decodeFront	22	22	0	16	16	0
decodeBack	18	18	0	12	12	0
byUTF	94	94	0	51	51	0

Interpretation

• Trampolines vs shared bodies. stride / strideBack still instantiate once per string type (the public trampoline). Distinct counts therefore barely move. The win is that string, char[], and const(char)[] (and the wchar analogs, including static arrays via cast(const(C)[])) share one strideUTF8 / strideUTF16 / strideBackUTF8 / strideBackUTF16 body. That shows up in object size, not in trampoline instantiation counts.
• codeLength. Matching strings now hit the identity codeLength(C)(const(C)[] input) (3 encodings) instead of the transcoding range overload. Distinct codeLength instantiations drop by 6; total call sites stay 60 because the driver still asks for every encoding combination.
• isValidUTF / validate. Public S-templates remain (9 string types). The heavy work moves to isValidUTFImpl / validateImpl with 3 encodings. Extra summary records (+6 overall) are these new impl templates; they replace duplicated string-typed bodies.
• decode / decodeImpl source is unchanged. Instantiations still drop because validateImpl / isValidUTFImpl now pass const(C)[], so decode is instantiated for three encodings rather than each string qualifier. byUTF is unchanged, as intended.
• Do not fold the trampoline into the large body. A previous byUTF experiment that merged two entry points into one function with a static if at the front increased object size. Tiny trampolines around a shared implementation is the pattern that actually shrinks generated code.

Tests

std/utf.d unittests pass in both modes:

dmd -main -unittest -version=StdUnittest -I. -oftestutf.exe std/utf.d && testutf.exe
dmd -main -unittest -version=StdUnittest -version=NoAutodecodeStrings -I. -oftestutf_na.exe std/utf.d && testutf_na.exe

Both printed 1 modules passed unittests.

Implementation notes (for reviewers)

• Public overloads stay S-templated. A public non-template const(dchar)[] next to const(char)[] clashes with autodecode (is(S : const dchar[])).
• Trampolines use cast(const(char)[]) / wchar / dchar, not is(immutable S == immutable C[], C), so static arrays such as char[4] still compile (buf.stride in encode unittests).
• Range overloads exclude strings (!is(S : const char[]) and analogs) so they do not duplicate the string body.
• UTF-16 private impls are @safe pure nothrow @nogc. Non-template helpers do not infer nothrow; without it, byUTF Result.back failed under -version=NoAutodecodeStrings.
• UTF-8 private impls stay @safe pure (they can throw UTFException).
• dchar stride / strideBack remain the original S-templates.

[Herringway]

Why object file size, specifically? That doesn't necessarily translate to a smaller final binary...

[LightBender]

@Herringway It acts as a confirmation that we're achieving the desired results. Hence it's a secondary metric.

[Herringway]

@Herringway It acts as a confirmation that we're achieving the desired results. Hence it's a secondary metric.

I've seen many quantifiable problems attributed to template bloat, but object file size is not one of them. Compilation times and final binary sizes being two big ones that come to mind. Two big metrics that we certainly don't want to backslide on, as well.

[LightBender]

Object file size isn't the problem we're trying to solve, template bloat is, but template shows up in object file size so it's a usable metric to checking the results of our effort.

[Herringway]

Object file size isn't the problem we're trying to solve, template bloat is, but template shows up in object file size so it's a usable metric to checking the results of our effort.

But it's not an important metric. Why did you choose the only metric that ISN'T important? We would all like smaller binaries and shorter compile times. Please measure those instead so that we can see the real benefits of these changes! I've seen too many big changes happen over the years that cause regressions in these areas, and I would appreciate the assurance that it is not regressing further here.

[schveiguy]

This PR is trading off performance for code size.

The index-less version needs to do less work. Yet, for arrays you are making it do the same work as the indexed version by passing 0 to the other versions. These tradeoffs might not be desirable. And it's not clear why you don't do the same for the non-array versions if that is the desired outcome. IMO, runtime performance trumps template bloat.

There is a missed opportunity here, in that the compiler already does a lot of free implicit conversions without any templates at all. Why don't we have a stride(const(char)[] str) and stride(const(char)[] str, size_t index)? This would capture so many cases, and if properly set up, might even take precedence over templates!

As a trivial matter, you shouldn't need casts here to call the non-templated versions, there is a danger here of using the blunt cast which might not work right for all types.

I'm going to be blunt, this kind of work is wasting everyone's time. I'm not going to review any more of these vibeslop PRs. If this were my repository, and you were sending me these kinds of PRs you would get a ban.

[LightBender]

@Herringway This is for you.

Compile time

Wall-clock dmd -c without -vtemplates (that flag writes a large log and is not a fair compile-time signal). One warmup compile discarded; five timed repeats. Times are milliseconds on this Windows host with DMD 2.113.0. Compile time of std.utf only is tested to be consistent with other metrics.

Test Driver

	old (ms)	new (ms)	delta (ms)
min	115	112	−3
median	118	114	−4
mean	119	115	−4
max	129	121	−8

• old samples (ms): 129, 121, 115, 115, 118
• new samples (ms): 116, 114, 114, 121, 112

Module (dmd -c std/utf.d)

	old (ms)	new (ms)	delta (ms)
min	66	67	+1
median	69	68	−1
mean	68	67	−1
max	69	68	−1

• old samples (ms): 69, 68, 69, 69, 66
• new samples (ms): 68, 67, 68, 68, 67

Unittests (dmd -c -main -unittest -version=StdUnittest std/utf.d)

	old (ms)	new (ms)	delta (ms)
min	451	446	−5
median	456	451	−5
mean	457	453	−4
max	465	463	−2

• old samples (ms): 456, 451, 455, 465, 460
• new samples (ms): 446, 449, 451, 459, 463

Compile time is noise at this scale: a few milliseconds on a ~450 ms unittest compile, and the driver/module deltas sit inside sample spread. But I find it fascinating that nearly every time moved down.

[Herringway]

These numbers suggest that this approach doesn't have any particularly meaningful impact. Perhaps it would be worth looking into @schveiguy's suggestions.

[crazymonkyyy]

wiggles around some template constraints
adds strideBackUTF16

this is not a good use of your ai tokens; phoboes style is bad at templates and I bet you told it to "maintain style" or gave it the style guide

Theres allot of simplicity on the table; like my 5 line nullable vs the 1000 lines; you should be getting more then 4%

[rikkimax]

this is not a good use of your ai tokens; phoboes style is bad at templates and I bet you told it to "maintain style" or gave it the style guide

The LLM's I have used, have all done the style automatically of the targeted codebase. No instruction.