Skip to content

Unit tests built once into one binary; customs@v0.3.1 - #141

Merged
winding-lines merged 1 commit into
mainfrom
customs-build-once
Nov 16, 2026
Merged

winding-lines merged 1 commit into
mainfrom
customs-build-once

Conversation

@winding-lines

Copy link
Copy Markdown
Contributor

What and why

customs 0.3.1 adds a new check, mojo-build-per-test (magmalake/customs#7). On main it reports test: 23 test files, each built into a binary of its own, so the code they share was compiled 23 times.

This follows census's scripts/test.sh:

  • New scripts/test.sh. It generates a main under build/multicall-gen/tests/ (which customs skips) and builds 21 test files into build/tests/multicall-tests, through scripts/mojo-build.sh so the build cache still works.
  • Tests registered by name. The generated main imports each def test_* by name rather than using __functions_in_module(), which is incomplete in an imported module.
  • Each file still runs as its own process, through a link build/tests/<test> to the one binary. After each file the script checks that the count of tests that ran matches its def test_*.
  • pixi run test-units runs all 21. The task names CI and people call still work: test-codec and the others run one file from the same binary, and test is unchanged for CI.
  • scripts/build-tests.sh builds the unit-test binary in parallel with test-libc-abi, test-head-loop and build/dista.
  • The customs pin moves to v0.3.1, so the new check guards this.

Built on their own, and why

  • tests/test_libc_abi.mojo: it is meant to fail at build time when dista's libc declarations clash with tcp's. In one binary, a clash would break every test, or another file's declaration could hide it.
  • tests/test_head_loop.mojo: its main ignores SIGPIPE with signal(13, SIG_IGN), which passes an integer as the handler, while dista.signals declares signal with a function. Mojo allows one signature per C symbol in a program, and the multicall build failed with "existing function with conflicting signature". It stays as it was.
  • test_parquet_tasks and test_iceberg_scan: they need the data environment and have their own tasks. Unchanged.

With these, test builds 2 test files on their own (plus the one binary), under customs' threshold of 3, so no warn is needed.

Tests run and passed per file, before and after

The same tests ran and passed in every file, 181 in all. (The commit message says 175; that is a miscount, the table is right.)

Test file def test_ Linux before Linux after macOS before macOS after How it runs now
tests/test_actors.mojo 16 16 run, 16 passed 16 run, 16 passed 16 run, 16 passed 16 run, 16 passed multicall
tests/test_admission.mojo 5 5 run, 5 passed 5 run, 5 passed 5 run, 5 passed 5 run, 5 passed multicall
tests/test_arity.mojo 4 4 run, 4 passed 4 run, 4 passed 4 run, 4 passed 4 run, 4 passed multicall
tests/test_auth.mojo 8 8 run, 8 passed 8 run, 8 passed 8 run, 8 passed 8 run, 8 passed multicall
tests/test_batches.mojo 6 6 run, 6 passed 6 run, 6 passed 6 run, 6 passed 6 run, 6 passed multicall
tests/test_cancel.mojo 4 4 run, 4 passed 4 run, 4 passed 4 run, 4 passed 4 run, 4 passed multicall
tests/test_codec.mojo 10 10 run, 10 passed 10 run, 10 passed 10 run, 10 passed 10 run, 10 passed multicall
tests/test_executors.mojo 7 7 run, 7 passed 7 run, 7 passed 7 run, 7 passed 7 run, 7 passed multicall
tests/test_fetch.mojo 6 6 run, 6 passed 6 run, 6 passed 6 run, 6 passed 6 run, 6 passed multicall
tests/test_frame.mojo 15 15 run, 15 passed 15 run, 15 passed 15 run, 15 passed 15 run, 15 passed multicall
tests/test_free.mojo 7 7 run, 7 passed 7 run, 7 passed 7 run, 7 passed 7 run, 7 passed multicall
tests/test_head_drivers.mojo 7 7 run, 7 passed 7 run, 7 passed 7 run, 7 passed 7 run, 7 passed multicall
tests/test_head_loop.mojo 4 4 run, 4 passed 4 run, 4 passed 4 run, 4 passed 4 run, 4 passed own build
tests/test_head_queue.mojo 2 2 run, 2 passed 2 run, 2 passed 2 run, 2 passed 2 run, 2 passed multicall
tests/test_libc_abi.mojo 1 1 run, 1 passed 1 run, 1 passed 1 run, 1 passed 1 run, 1 passed own build
tests/test_runtime.mojo 23 23 run, 23 passed 23 run, 23 passed 23 run, 23 passed 23 run, 23 passed multicall
tests/test_scale.mojo 12 12 run, 12 passed 12 run, 12 passed 12 run, 12 passed 12 run, 12 passed multicall
tests/test_shuffle.mojo 9 9 run, 9 passed 9 run, 9 passed 9 run, 9 passed 9 run, 9 passed multicall
tests/test_sizing.mojo 17 17 run, 17 passed 17 run, 17 passed 17 run, 17 passed 17 run, 17 passed multicall
tests/test_slots.mojo 8 8 run, 8 passed 8 run, 8 passed 8 run, 8 passed 8 run, 8 passed multicall
tests/test_spill.mojo 2 2 run, 2 passed 2 run, 2 passed 2 run, 2 passed 2 run, 2 passed multicall
tests/test_take.mojo 5 5 run, 5 passed 5 run, 5 passed 5 run, 5 passed 5 run, 5 passed multicall
tests/test_timeline.mojo 3 3 run, 3 passed 3 run, 3 passed 3 run, 3 passed 3 run, 3 passed multicall
total 181 181 passed 181 passed 181 passed 181 passed
  • Linux: sai, 16-core x86-64, pixi run test (default environment).
  • macOS: arm64, pixi run test-<file> for each of the 23 before, and test-units, test-libc-abi and test-head-loop after.
  • The shell tests' PASS lines are the same before and after: 13.

Timings (empty Mojo compile cache and empty build/ each time)

Before After
sai, all of pixi run test 530 s wall, 35m53s CPU 515 s wall, 16m09s CPU
sai, build-tests + the 23 unit-test tasks only 183 s 163 s
macOS arm64, the 23 unit-test tasks one after another, without build-tests 238 s 85 s

On sai, build-tests already built the 23 binaries 9 at a time, so wall time hardly changes there, but CPU time halves. A runner with fewer cores, such as GitHub's 4-core one, should gain more, as the macOS row shows; this PR's CI step time is the check. The one binary builds in 116 s on sai and 39 s on macOS.

Not checked

  • I did not run the nightly environment locally; CI runs it on main and the nightly.
  • I did not run the data environment, which this PR doesn't touch.

No changelog entry: this is test infrastructure, and nothing changes for a user.

🤖 Generated with Claude Code

customs 0.3.1's mojo-build-per-test (magmalake/customs#7) reported the
`test` task: 23 test files, each built into a binary of its own. The code
they share (dista, its dependencies) compiled 23 times.

scripts/test.sh (census's approach, morca-dev/census scripts/test.sh) builds
21 of them into build/tests/multicall-tests, with tests registered by name
rather than found with __functions_in_module(), which is incomplete in an
imported module. Each file still runs as its own process, through a link
build/tests/<test>, and the number of tests it ran is checked against the
`def test_*` found in it. `pixi run test-units` runs them all, and
`test-codec` and the others run one file from the same binary.
scripts/build-tests.sh builds the binary alongside the other two.

Built on their own, as before:
- tests/test_libc_abi.mojo: it must fail to build when dista's libc
  declarations clash with tcp's. In one binary a clash would break every
  test, or another file's declaration would hide it.
- tests/test_head_loop.mojo: its main calls `signal(13, SIG_IGN)` with an
  integer handler, and dista.signals declares `signal` with a function.
  Mojo allows one signature per C symbol in a program, so the multicall
  build failed with "existing function with conflicting signature".
- the data environment's test_parquet_tasks and test_iceberg_scan, which
  have their own tasks.

Counts, per file, before and after on Linux x86-64 (sai) and macOS arm64:
the same tests ran and passed in every file (175 tests).

Co-Authored-By: Claude Opus <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017TycsxF3PkSX49fnv7cSZd
@winding-lines

Copy link
Copy Markdown
Contributor Author

CI on this PR is green. In its logs, the unit tests ran 23 files, 181 tests run and 181 passed, in both stable / ubuntu-latest and nightly / ubuntu-latest, matching the table. The one binary built in 105 s on the runner.

The step times are not a clean comparison: this PR's runs restored main's compile cache, which holds per-file entries and not the new binary. The Tests step took 452 s (stable) and 264 s (nightly) here. On main it took 225 s / 256 s with a warm cache and 769 s / 790 s cold.

🤖 Generated with Claude Code

@winding-lines
winding-lines merged commit 2328e65 into main Nov 16, 2026
13 checks passed
@winding-lines
winding-lines deleted the customs-build-once branch November 16, 2026 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant