[jwrosewell]

Problem

Java's String.toLowerCase() and toUpperCase() with no argument use the default locale of the JVM. Under a Turkish or Azeri default locale a capital I lowers to a dotless i (ı), and a small i uppers to a dotted capital İ, so a name that holds one no longer matches its lower case form. On rc at ceab0888, 74 lines under src/main call one of them with no argument (76 calls in 21 files), among them:

• JsonBuilderElement.java lowers every element and property name it writes, so IsMobile became ısmobile, a key the client-side script never reads.
• DeviceDetectionCloudEngine.java line 381 and its five neighbours, the IP Intelligence and Location cloud engines and CloudAspectEngineBase lower the Property name to find its value in the cloud's lower case response, so DeviceId looked for deviceıd and found no value.
• PipelineBuilder.java lowers builder and method names from a configuration file to match them, and EvidenceKeyFilterShareUsage.java lowers Evidence keys before it compares them with the keys that are never shared.

The specification asks for lower case keys compared without regard to case. json-builder.md lines 59 to 63 at c8128ca4 says "These keys MUST be written in lower case. The key of a top-level entry is the Element Data's Data Key, and the key of a sub-entry is the Property name", and access-to-results.md lines 5 to 11 says "Where string keys are used ... Lower-case SHOULD be used. String comparisons MUST be case insensitive." Lowering in the root locale gives the same key on every JVM, whatever its default locale.

#83 fixed the four calls in EvidenceKeyFilterShareUsage.java, and its review asked for the rest to follow. This draft is that follow-up.

What changed and why

• Every case change under src/main passes Locale.ROOT, 76 calls in 21 files (d3990945), with import java.util.Locale where a file did not import it. Nothing else in those lines changed. EvidenceKeyFilterShareUsage.java gets the same four changes and the same import as Compare usage sharing keys in the root locale #83, line for line. Compare usage sharing keys in the root locale #83 has since merged into rc (b76596b), so that file merges as no change, and what this draft adds to rc is 72 calls in 20 files.
• A check that keeps it so. ci/check-root-locale.ps1 reads every Java file git tracks under a src/main folder and fails on any toLowerCase() or toUpperCase() with no argument, naming each line. .github/workflows/root-locale.yml runs its Pester tests and then the check on every pull request and on each push to main and rc, as the environment variable names check does. Tests under src/test are left out, because a test may set a locale on purpose.
• The numbers the examples write for a machine to read use the root locale too (d6263a26, from the review of 16:23 UTC on 14 November). CompareOnPrem.java line 424 at b33daa0f wrote a distance into a row of compare-output.csv with String.format("%.3f", ...), and MetricsOnPrem.java line 739 ended a comma separated row with String.format("%.2f", ...). Under a default locale with a decimal comma, such as German, 12.346 became 12,346 and the row gained a column. ShareUsageExample made up client IP addresses with String.format("%d.%d.%d.%d", ...), so under a locale whose digits are not 0 to 9, such as Arabic in Saudi Arabia, the address was Evidence no engine can read. Each now formats in Locale.ROOT, and the two CSV numbers move into package-private methods, kilometres and twoDecimalPlaces, so a test can reach them. The examples are not published packages, so this releases nothing.

Every published package whose main code changed is released by the version per package change (#63), and every one of them is already in Monday's plan.

Tests

Tests first, in 31153475.

• ci/check-root-locale.Tests.ps1 has 18 Pester tests. They check that the pattern finds a bare call and leaves Locale.ROOT, another locale, equalsIgnoreCase and Character.toLowerCase alone, that each line number is given, that only src/main Java files are read, and that the script run on a throwaway repository passes with a named locale and fails, naming the line, without one.
• JsonBuilderTests.JsonBuilder_TurkishLocale_NamesLowerCasedInRootLocale sets the default locale to tr-TR and builds the JSON for an element IpData with a Property IsMobile, and expects ipdata and ismobile.
• MissingPropertyHandlingTests.PropertyWithCapitalI_TurkishLocale_HasValue sets the default locale to tr-TR and has the Device Detection cloud engine read DeviceId from a response that gives deviceid, and expects the value.

Each ran on ubuntu-latest on a throwaway branch holding the commit and a temporary workflow that ran the Pester tests, the check, and mvn -Pdev -pl pipeline/pipeline.jsonbuilder,device-detection/device-detection.cloud -am test with only the two Java test classes, on Java 17 as the full build's Ubuntu_Java_17 entry does.

• Red, run 43582636250 on 3115347 (tmp/java-locale-red). The check named 74 lines and failed with "74 case changes name no locale". JsonBuilder_TurkishLocale_NamesLowerCasedInRootLocale failed with {"ıpdata": {"ısmobile": "True"}} ==> expected: <true> but was: <false>, and PropertyWithCapitalI_TurkishLocale_HasValue failed, 1 of the 3 in its class. The 18 Pester tests passed.
• Green, run 43582645800 on d399094 (tmp/java-locale-green). The check printed "Every case change in 525 main source files names its locale", the 18 Pester tests passed, JsonBuilderTests passed 16 of 16 and MissingPropertyHandlingTests 3 of 3.
• An earlier red, run 43582121683, ran on the runner's default JDK, 25.0.5, where the build stopped in PMD before any Java test ran, so the throwaway workflow was moved to Java 17.
• The example numbers. Tests first, in 576615ee. CsvNumbersTest.csvNumbers_GermanDefaultLocale_FullStop sets the default locale to German and expects kilometres(12.3456) as 12.346 and twoDecimalPlaces with a full stop too. RandomIpTest.randomIp_ArabicDefaultLocale_AsciiDigits sets the default locale to ar-SA and expects 20 made-up addresses written with the digits 0 to 9. Both ran on ubuntu-latest on Java 17 through a temporary workflow, building each examples folder against the packages on Maven Central. The red, run 43666180249 (tmp/java-exlocale-red2), ran the tests against the fix's methods with the default locale, as the methods do not exist before the fix. CsvNumbersTest failed with "expected:<12[.]346> but was:<12[,]346>" and RandomIpTest with the address ٢٣٦.١٦٦.٢٢٤.١٥٠. The green, run 43666180345 (tmp/java-exlocale-green2), ran d6263a2 and both passed.
• The check also runs on this pull request, run 43583321941, and passed, and the CI script tests ran its Pester file on Ubuntu and Windows (run 43583321945).

Full build

Run 43666245300 built bb1d8da, which is d6263a2 with rc 32b3784 merged in, on nine configurations, Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, and Windows with Java 8, 11, 17 and 21, and every job passed. JsonBuilderTests passed 16 of 16, MissingPropertyHandlingTests 3 of 3, and CsvNumbersTest and RandomIpTest 1 of 1 each in all nine jobs. Each of its ten Setting 'PullRequestSha' lines names bb1d8da, and its Configure job printed "Left out 6 macOS entries".

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

Base branch

rc at 32b3784, merged into the branch at bb1d8da. rc has since taken #67, #129, #133, #134, #144 and #145, and a trial merge of the head with rc at 2eebbff, which holds #145, is clean.

What remains

• The new check makes a pull request fail when it adds a case change with no locale under src/main. None of the other drafts open into rc on 14 November at 16:05 UTC adds one, by a search of each one's diff.
• Formatting numbers and dates with the default locale was checked after the review, in the package code at rc b33daa0f. It needs no change here, and the one case whose output a machine reads is fixed in #139.
  • 22 String.format calls name no locale. 21 of them build log, warning and exception messages for a person to read. The 22nd, "%04x" in ShareUsageElement.java line 648, writes an escape into the usage sharing record, and the Formatter documentation says of x that "No localization is applied".
  • DateSent in the usage sharing record is written by a SimpleDateFormat made with no locale (ShareUsageBase.java line 66). #139 writes DateSent with a DateTimeFormatter in Locale.ROOT and in UTC, so it is left to that draft.
  • The data update's If-Modified-Since goes through HttpURLConnection.setIfModifiedSince, which takes a number and leaves the date to the JDK (DataUpdateServiceDefault.java line 833).
  • No NumberFormat, DecimalFormat, MessageFormat or DateTimeFormatter in the package code is made without a locale.
  • In the examples, the two numbers written into CSV files and the made-up client IP addresses are for a machine, and are now written in the root locale (above). The examples' other numbers and dates are printed for the person running them, so they keep that person's locale.