uucore: find translations for locales like ru_RU and zh_CN

[Hekzory]

Closes #12305.

With LANG=ru_RU.UTF-8, uucore looks for ru-RU.ftl and nothing else, but the translations in coreutils-l10n are named ru.ftl, de.ftl, zh-Hans.ftl, es-ES.ftl and so on. So a system locale only gets its translation when the names happen to match (fr_FR, pt_BR, es_ES, nb_NO). ru_RU, de_DE, pl_PL, ja_JP, zh_CN and most others stay in English. I ran into this while translating uutils into Russian on Weblate: the strings only showed up with LANG=ru. #12305 is the same problem for Chinese.

Now the locale is matched against the files in the locales directory: the exact name first, then the language with its script, then the language alone, then another region of the same language (es_MX gets es-ES). For Chinese the script comes from the region (CN/SG are Hans, TW/HK/MO are Hant), so Simplified and Traditional never stand in for each other. If nothing matches, English is used as before. Against the coreutils-l10n files for ls, 5 of the 89 glibc UTF-8 locales whose language has a translation get it today, and all 89 with this.

There's no new dependency, and it doesn't change which variable is read, so it doesn't get in the way of #14870. #14327 tried this with fluent-langneg and CLDR data and was closed; #13331 has a similar same-language fallback as part of its LANGUAGE support.

Added a unit test for the lookup order and a short note in docs/src/l10n.md.

[codspeed]

Merging this PR will degrade performance by 13.59%

⚠️ 119 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• unexpand_large_file[10]
• unexpand_many_lines[100000]
• du_all_wide_tree[(5000, 500)]
• factor_64bit_semiprime
• sort_mixed_utf8_locale
• sort_reverse_utf8_locale
• sort_unique_utf8_locale
• sort_very_long_lines_utf8_locale
• dd_copy_partial
• cksum_default
• …and 109 more

ℹ️ 1 benchmark has no flame graph

The flamegraph could not be generated because too much time was spent in runtime internals, making the visualization not useful for performance analysis.

• sort_dictionary_order[500000]

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 6 regressed benchmarks
✅ 393 untouched benchmarks
⏩ 70 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	factor_three_39_bit_primes	321.7 ms	484.5 ms	-33.6%
❌	Simulation	factor_five_38_bit_primes	1.5 s	2.1 s	-25.33%
❌	Simulation	expand_many_lines[100000]	115.9 ms	135.1 ms	-14.21%
❌ ⚠️	Simulation	unexpand_large_file[10]	105.3 ms	120.5 ms	-12.67%
❌ ⚠️	Simulation	unexpand_many_lines[100000]	50.6 ms	57.9 ms	-12.58%
❌	Simulation	expand_custom_tabstops[50000]	31.6 ms	35.5 ms	-10.98%
⚡ ⚠️	Simulation	du_all_wide_tree[(5000, 500)]	35.7 ms	34.2 ms	+4.32%
⚡	Memory	du_all_wide_tree[(5000, 500)]	1,041.8 KB	1,010.5 KB	+3.1%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing Hekzory:locale-language-fallback (0ffdcfe) with main (1747ba1)²}

Footnotes

1. 70 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on main (1f1095b) during the generation of this report, so 1747ba1 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩