Repository navigation
Description
Follow-up to #117. The image's ~10 min is almost all two mojo build runs, so this looks at where the compile time goes and what would cut it. It covers how Modular uses Bazel, how the stdlib is shipped, and how big each part of the code is. Measured on an arm64 Mac with Mojo 1.2.0, mostly one cold run each, on an older census checkout; expect ±1–2 s of noise.
Findings
-
We already consume precompiled packages, except flare and json. (verified)
- Every tin arrives as a pixi-build source dependency. The pixi-build-mojo backend runs
mojo precompile, producinglib/mojo/<name>.mojoc: orca 4.6 MB, iceberg 1.6 MB, std 3.5 MB. scripts/multicall.shadds only-I bin -I pipelines -I src, which is the census's own code.- flare_mlake (107k lines) and json_mlake (22k) ship as source. Their recipes copy the source in because a
.mojocis locked to one compiler version.
- Every tin arrives as a pixi-build source dependency. The pixi-build-mojo backend runs
-
A
.mojocholds MLIR bytecode of the parsed and type-checked IR, with no object code and nothing specialized. (verified)- It saves parsing and type-checking only.
- Specialization of generics, the passes after it, and LLVM optimisation and codegen run again in every consumer, once per program.
- Sources:
mojo precompile --help; modular'smojo/docs/compiler/MojoCompilerWalkthrough.md("Mojo Packages and Precompiled Files"); the changelog (.mojocarrived in 1.0.0b2, and 1.1.0 dropped.mojopkg).
-
The stdlib and the MAX kernels ship the same way:
std.mojoc, plusnn8.0 MB,linalg5.4 MB,layout1.3 MB. Both of our binaries pull in nn and linalg through encoder.mojo (the console viaquerymodel). (verified; the cost of this is not isolated) -
Bazel at Modular (github.com/modular/modular, modular/rules_mojo): (verified)
mojo_libraryis onemojo precompileper library, producing a.mojoc.mojo_binaryis onemojo build --emit objectover the main file plus every transitive.mojoc, then a link. There are no per-library object files, and binaries share no codegen.- So Bazel caches what conda's
.mojocfiles already give us. - Modular's speed comes from a BuildBuddy remote cache and remote execution (
--jobs=1000) spread over thousands of small targets, not from making one big binary faster.
-
Where one program's time goes (single programs, cold,
-g1): (measured)Program Wall time Peak RSS mojo_stats48 s 4.8 GB gharchive57 s 5.4 GB mojo_embed59 s 5.2 GB console60 s 5.1 GB - user/real ≈ 1.8: the compiler pipeline itself is limited to ~2 cores; it isn't a
-jsetting. --mlir-timing: importing (parsing and loading.mojoc) ≈ 9%, specialization and lowering ≈ 10 s, LLVM ≈ 60%.
- user/real ≈ 1.8: the compiler pipeline itself is limited to ~2 cores; it isn't a
-
flare precompiled vs source:
consolebuilds in 57.5 s against 60.0 s, about 5% faster. (measured) -
The compile cache is
<env>/share/max/cache(orMODULAR_CACHE_DIR). It helps only when a program is unchanged. (measured)Change since the cached build Wall time None (identical rebuild) 7 s (48 s cold) Comment appended 7 s Lines shifted in a src/file17 s One string literal changed 54 s (full cost) In CI it can never hit:
-D ORCA_COMMIT=<sha>changes every program on every commit, and the cache is lost after each Docker build. -
Optimisation and debug levels (
mojo_stats, cold): (measured)Flags Wall time Peak RSS Binary -O3 -g1(today)48 s 4.8 GB 5.5 MB -g041 s (−15%) 3.4 GB 5.2 MB -O1 -g155 s 5.3 GB 5.6 MB -O0 -g1214 s 10.9 GB 35.7 MB A lower
-Ois slower, so that option is out.
Size (.mojo lines, tests excluded)
| Code | Lines | How the census gets it |
|---|---|---|
| Mojo stdlib (modular HEAD) | 122,497 | std.mojoc |
| MAX kernels, total | 595,030 | .mojoc |
| … nn / linalg / layout (via encoder) | 224,619 / 151,991 / 29,972 | .mojoc |
| census (src, bin, pipelines) | 21,708 | source |
| morca src | 66,321 | .mojoc |
| dista src | 10,579 | .mojoc |
| other tins (iceberg 24.2k, parquet 11.3k, postgres 8.4k, thrift 7.2k, …) | 79,378 | .mojoc |
| flare_mlake | 107,186 | source |
| json_mlake | 21,726 | source |
| Ours, total | ~307k |
Line counts are a weak proxy for cost. What costs time is the number of specialized instances reachable from each main, and their LLVM size.
Options, ranked
- Build the programs in parallel (separately, or in 4–6 groups instead of 2 multicall binaries) on a bigger runner. Inferred; high benefit; costs money.
- Each program alone is ~50–60 s and ~5 GB. The combined pipelines binary (~12 programs) takes 390–545 s, about the sum of its parts, while staying on ~2 cores.
- A 16-core / 64 GB runner could plausibly cut wall time 3–4×. sai (16 threads, 62 GB) is that machine.
- Costs: a bigger image (shared code duplicated, to be measured), and a rethink of the multicall design.
-g0for CI-only images (PR builds that are never deployed): 15% faster, 30% less memory. Keep-g1for images that get deployed. Verified; trivial.- Persist the compile cache, and move
ORCA_COMMIT/ORCA_SOURCEfrom-Dto runtime (ENValready sets them). It then hits on reruns, commits that touch no Mojo code, and the second platform of the same commit. It does nothing for an ordinary code commit. Low risk. - Precompile flare and json: ~5%, mainly for consistency. Do it at flare_mlake's next release.
- Bazel with rules_mojo: no gain on its own for these binaries; it pays only with remote execution fanning out many separate binaries, which is option 1 done another way. High cost and risk.
- A lower opt level: slower (measured). Dropped.
Recommended next step
-g0for non-deploy image builds.ORCA_COMMIT/ORCA_SOURCEfrom-Dto runtime.- Prototype one job on sai (or a larger runner) that builds the programs as separate or grouped parallel compiles. Measure wall time, image size and peak memory.
- Precompile flare and json at flare_mlake's next release.
Decisions for the user: whether to split the multicall binaries (option 1), and whether builds go on sai or on paid larger runners.
🤖 Generated with Claude Code
Activity
winding-lines commentedon Nov 12, 2026
Measured on sai (Threadripper 1900X, 8 cores / 16 threads, 62 GB), one job, census main b3eba2d (morca-mojo 0.25.0), amd64, inside the census's own
envstage, flags as scripts/build.sh passes them (-g1 --target-cpu x86-64-v3 -D MORCA_SOURCE=… -D MORCA_COMMIT=…), an empty Mojo compile cache (newMODULAR_CACHE_DIR) for everymojo build. Run: https://github.com/morca-dev/morca/actions/runs/43259456514 (workflow in morca-dev/morca#405).The 13 programs of
multicall-pipelines(morca_maintain + the 12 pipelines):multicall-pipelinesxargs -P 4xargs -P 8Each alone (sequential): morca_maintain 31 s, ballast 44 s, the other eleven 126–179 s each (gharchive 173, mojo_embed 179), 335–450 CPU-s and 3.1–3.9 GB each.
Sizes, MB (objcopied = the Dockerfile's
objcopy --compress-debug-sections=zlib; gzip =gzip -c | wc -cof that):The hypothesis doesn't hold: total CPU is not about the same. Built alone, the programs take 5× the CPU (4287 vs 859 CPU-s), because each one compiles the shared library code (morca, iceberg, parquet, …) again. On this machine a typical pipeline alone takes 126–179 s, not 48–60 s; the Mac figure came from a much faster machine. So even 8 at a time (505 s, 18 GB) is slower than the one combined binary (368 s, 6.5 GB), and the image would hold 5.5× the bytes. sai's load average was about 3.7 from other jobs when the runs started; that applies to all four rows.
winding-lines commentedon Nov 12, 2026
Mojo's compile cache: how it works, what defeats it, and the disk it needs
This follows the split measurement above. Mojo's compiler has been open source since 2026-08. I read it at
modular/modular@c0bfec57andmojo/v1.2.0, and tested the cache on a Mac with Mojo 1.2.0.How the cache works: it is whole-program, with two entries per
mojo build.-g,-j) plus the target.compileOptimizedLLVMModuleToObjectand never looked up. GPU kernels are the only thing cached piece by piece.What changes the key (orca_maintain, 11 s cold):
-strip-file-prefix=<checkout>/-D, even one the program never reads-g0vs-g1;--target-cpu;-jWhat this means for us:
-D MORCA_COMMITalone guarantees a full rebuild on every commit, so morca #402 is required.-g0for PR builds means a main build can never reuse its PR's cache entries. Once the cache is persisted, it's better to use the same-glevel everywhere.-strip-file-prefix=/src/project/($PWD/on a Mac).src/and 44% change the lock.Disk (against 500 GB): the Mojo cache is about 250–400 MB per build per arch, so a 10–15 GB cap per arch with a pruning cron. The Docker layers are about 10 GB per env revision, so keep the last 2–3 per arch, 30–40 GB. Under 70 GB in total.
Ranked:
-D, one-glevel,-strip-file-prefix. Hours of work.MODULAR_CACHE_DIRoutside the env, as a BuildKit cache mount on a self-hosted builder. About a day.Upstream proposals for Modular's repo, small to large:
enableLLVMPerFunctionSplittingas a flag.lowerLLVMModuleToObjects. Estimated 3–6 weeks, maybe 30–50% faster on census-only commits.winding-lines commentedon Nov 12, 2026
Compile cache: where it stands (step 1 of the plan, plus the follow-up the user asked for).
Main hits fully. Since #162 (cache mount carried between runs) and #163 (no
-D MORCA_COMMIT/MORCA_SOURCE, morca-mojo 0.25.1), a commit that changes no compiled source reuses the whole compile. Draft #165, on main plus logging, first run (restoring main's cache) and three reruns:-DThe 4th amd64 run was on a different CPU (EPYC 9V74, the others 7763) and still hit everything.
What it wasn't:
pixi install, on colima: 147 s, then 22 s, all hits.With the
-Ddefines put back (same value on a rerun), #165 attempts 5–7:Minimal repro, outside the census: doesn't reproduce.
cache-reprobranch, about 20 lines, one-D COMMITit reads.-g1 --target-cpu x86-64-v3 -strip-file-prefix=$PWD/on Mojo 1.2.0, on a GitHub runner (EPYC 7763) and on sai (Threadripper 1900X, Zen 1).-Dvalue, even of a define nothing reads, changes both keys, which is expected.So no report for Modular: what's left needs the census's two-entry compile with
-Don amd64, and main no longer passes-D. Parked.One thing to know about ci.yml: actions/cache doesn't save when the primary key was an exact hit. So a rerun's new entries are kept only if the run's own first attempt didn't already save that key. That's fine for one save per commit.
🤖 Generated with Claude Code
winding-lines commentedon Nov 12, 2026
The minimal repro's files (the
cache-reprobranch is deleted):repro.mojo:pixi.toml:run.sh NAME [flags](prints the cache lookups):