Add i18n tools to support Crowdin translation sync - #125
Merged
Conversation
A CLI and config-file replacement for the translation scripts copied between the micro:bit apps and this repo. It tidies and compiles react-intl catalogs and talks to Crowdin one file at a time: - tidy: sort, prune, and check translations keep their placeholders, now by parsing the ICU message rather than a regex that skipped plurals - compile: formatjs AST output with English backfill and package catalogs merged, byte-for-byte what `formatjs compile --ast` produced - download: per-file, per-language translation builds, no project zip - upload: replace English sources after showing the change, optionally keeping translations for corrections - status: per-language progress for the repo's files - new-strings: ml-trainer's word counter for translation estimates Translated catalogs hold only real translations; English is backfilled at compile time so diffs show translation changes alone. Ships TypeScript run directly by Node 24's type stripping, hence the erasable-syntax-only tsconfig and explicit .ts imports.
Replaces bin/i18n-packages.cjs, tidy-lang.cjs and update-translations.cjs with an i18n.config.mjs driving the shared tool. CI now checks the catalogs are tidy. The translated catalogs are tidied into their new sparse form: only the strings each locale actually has, without descriptions, and en-US holding only what differs from en (nothing, for these packages). Apps backfill from English when they compile.
Uploading on merge to main gets new strings to translators without a manual step; the weekly download opens or refreshes a pull request so translations are in the packages before the next release. Both need the org Crowdin token granted to this repo, and the download needs the repository setting allowing Actions to create pull requests.
Crowdin's lol locale replaces every string with a marker, so comparing its arguments and tags against English only ever reports noise.
A translation missing {name} or <link>, or with an apostrophe escaping
one, shows users literal markup. English is the safer rendering, so the
download leaves such strings out and names them for fixing in Crowdin
rather than failing.
data-microbit-org's playground survey has a Welsh translation but no file in Crowdin, so a catalog can now list its own languages, or none to mark it as compiled-only.
pxt-microbit-ml keeps its Node project in simx/ but its translated files across the repo, so the CLI runs from simx with --config ../i18n.config.mjs and paths must be relative to the config, not the working directory.
data-microbit-org's playground survey is in Crowdin with translation disabled, which the option covers as well as a file that is not there.
Keep the push-to-main trigger out until a dry run from the Actions tab has shown the upload does the right thing; the trigger to restore is noted in the workflow.
{lang} now expands to Crowdin's casing (ui.pt-BR.json) so file names match
the ids the apps and Crowdin already use, with {lang:lower} for the stubs
and pyright repos, which keep lowercase. Compile still reads lowercase
package catalogs so apps work against ui packages published before the
rename.
ui.pt-br.json becomes ui.pt-BR.json and so on, matching the ids the apps pass around and Crowdin's own. Apps compiling against a version of these packages published with lowercase names keep working: the tool falls back to them.
Finding a file used to list the whole branch recursively, thousands of entries once the website content is counted. Now each directory in the path is resolved with a name-filtered listing of its parent, and the file with one of its directory, cached per run. The client retries three times on 429 and 5xx responses, and directory builds are polled with a delay doubling from one second to ten.
All eight repos share one Crowdin token; starting them ten minutes apart keeps them off it at the same moment.
Deploying ui with
|
| Latest commit: |
0e68095
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://e1ded263.ui-2wg.pages.dev |
| Branch Preview URL: | https://i18n-tools.ui-2wg.pages.dev |
Running the TypeScript sources directly only worked because the workspace and npm-link consumers reach them through symlinks, which Node resolves to real paths. An installed copy fails at the first import with ERR_UNSUPPORTED_NODE_MODULES_TYPE_STRIPPING. tsc emits dist/ on prepare, so npm install in this repo and npm publish both build it, with .ts import specifiers rewritten to .js so the sources still run under vitest as they are. bin and exports point at dist.
download and status reported a catalog not yet in Crowdin by aborting the whole run, so the carousel, uploaded to Crowdin only this week, stopped ui-patterns downloading. Each is now one counted failure. upload treated crowdinFile as a bare name, but this repo's are ui/ui.en.json and so on within packages/. It now takes the full Crowdin path, creating any missing directories, which the dry run names. Also: a catalog's own languages list can add languages the config does not have, not only narrow it, and --language accepts the union; the directory build poll throws on timeout instead of downloading an unfinished build; new-strings reads the base file relative to the config directory; the CLI rejects flags that do not belong to the command. The README also documents the microbit-i18n GitHub App the download workflows use.
Pushes and pull requests made with GITHUB_TOKEN never start workflows, so the weekly sync pull request got no CI. A token from the microbit-i18n GitHub App, narrowed to contents and pull requests, does. Files are staged before the change check so a newly added language's file counts as a change; it did not before.
Contributor
Author
|
Merging to help test this but more validation will be done before it's used in anger. |
Contributor
Author
|
(The token/app aren't yet configured so nothing automatic is going to happen as yet) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This should significantly improve diffs for new strings by only updating the English source file while keeping the English backfill at build/dev time and stop us pulling the whole zip from Crowdin (which is slow to build, N x M languages and files, including many that aren't ready to be shipped) in order to sync a few files.
Used in this repo too.
Fixes #124