Skip to content

Image build: ~10 min of mojo build, where the time goes and what would cut it #153

Open

Description

@winding-lines

Follow-up to #117. The image's ~10 min is almost all two mojo build runs, so this looks at where the compile time goes and what would cut it. It covers how Modular uses Bazel, how the stdlib is shipped, and how big each part of the code is. Measured on an arm64 Mac with Mojo 1.2.0, mostly one cold run each, on an older census checkout; expect ±1–2 s of noise.

Findings

  1. We already consume precompiled packages, except flare and json. (verified)

    • Every tin arrives as a pixi-build source dependency. The pixi-build-mojo backend runs mojo precompile, producing lib/mojo/<name>.mojoc: orca 4.6 MB, iceberg 1.6 MB, std 3.5 MB.
    • scripts/multicall.sh adds only -I bin -I pipelines -I src, which is the census's own code.
    • flare_mlake (107k lines) and json_mlake (22k) ship as source. Their recipes copy the source in because a .mojoc is locked to one compiler version.
  2. A .mojoc holds MLIR bytecode of the parsed and type-checked IR, with no object code and nothing specialized. (verified)

    • It saves parsing and type-checking only.
    • Specialization of generics, the passes after it, and LLVM optimisation and codegen run again in every consumer, once per program.
    • Sources: mojo precompile --help; modular's mojo/docs/compiler/MojoCompilerWalkthrough.md ("Mojo Packages and Precompiled Files"); the changelog (.mojoc arrived in 1.0.0b2, and 1.1.0 dropped .mojopkg).
  3. The stdlib and the MAX kernels ship the same way: std.mojoc, plus nn 8.0 MB, linalg 5.4 MB, layout 1.3 MB. Both of our binaries pull in nn and linalg through encoder.mojo (the console via querymodel). (verified; the cost of this is not isolated)

  4. Bazel at Modular (github.com/modular/modular, modular/rules_mojo): (verified)

    • mojo_library is one mojo precompile per library, producing a .mojoc.
    • mojo_binary is one mojo build --emit object over the main file plus every transitive .mojoc, then a link. There are no per-library object files, and binaries share no codegen.
    • So Bazel caches what conda's .mojoc files already give us.
    • Modular's speed comes from a BuildBuddy remote cache and remote execution (--jobs=1000) spread over thousands of small targets, not from making one big binary faster.
  5. Where one program's time goes (single programs, cold, -g1): (measured)

    Program Wall time Peak RSS
    mojo_stats 48 s 4.8 GB
    gharchive 57 s 5.4 GB
    mojo_embed 59 s 5.2 GB
    console 60 s 5.1 GB
    • user/real ≈ 1.8: the compiler pipeline itself is limited to ~2 cores; it isn't a -j setting.
    • --mlir-timing: importing (parsing and loading .mojoc) ≈ 9%, specialization and lowering ≈ 10 s, LLVM ≈ 60%.
  6. flare precompiled vs source: console builds in 57.5 s against 60.0 s, about 5% faster. (measured)

  7. The compile cache is <env>/share/max/cache (or MODULAR_CACHE_DIR). It helps only when a program is unchanged. (measured)

    Change since the cached build Wall time
    None (identical rebuild) 7 s (48 s cold)
    Comment appended 7 s
    Lines shifted in a src/ file 17 s
    One string literal changed 54 s (full cost)

    In CI it can never hit: -D ORCA_COMMIT=<sha> changes every program on every commit, and the cache is lost after each Docker build.

  8. Optimisation and debug levels (mojo_stats, cold): (measured)

    Flags Wall time Peak RSS Binary
    -O3 -g1 (today) 48 s 4.8 GB 5.5 MB
    -g0 41 s (−15%) 3.4 GB 5.2 MB
    -O1 -g1 55 s 5.3 GB 5.6 MB
    -O0 -g1 214 s 10.9 GB 35.7 MB

    A lower -O is slower, so that option is out.

Size (.mojo lines, tests excluded)

Code Lines How the census gets it
Mojo stdlib (modular HEAD) 122,497 std.mojoc
MAX kernels, total 595,030 .mojoc
… nn / linalg / layout (via encoder) 224,619 / 151,991 / 29,972 .mojoc
census (src, bin, pipelines) 21,708 source
morca src 66,321 .mojoc
dista src 10,579 .mojoc
other tins (iceberg 24.2k, parquet 11.3k, postgres 8.4k, thrift 7.2k, …) 79,378 .mojoc
flare_mlake 107,186 source
json_mlake 21,726 source
Ours, total ~307k

Line counts are a weak proxy for cost. What costs time is the number of specialized instances reachable from each main, and their LLVM size.

Options, ranked

  1. Build the programs in parallel (separately, or in 4–6 groups instead of 2 multicall binaries) on a bigger runner. Inferred; high benefit; costs money.
    • Each program alone is ~50–60 s and ~5 GB. The combined pipelines binary (~12 programs) takes 390–545 s, about the sum of its parts, while staying on ~2 cores.
    • A 16-core / 64 GB runner could plausibly cut wall time 3–4×. sai (16 threads, 62 GB) is that machine.
    • Costs: a bigger image (shared code duplicated, to be measured), and a rethink of the multicall design.
  2. -g0 for CI-only images (PR builds that are never deployed): 15% faster, 30% less memory. Keep -g1 for images that get deployed. Verified; trivial.
  3. Persist the compile cache, and move ORCA_COMMIT/ORCA_SOURCE from -D to runtime (ENV already sets them). It then hits on reruns, commits that touch no Mojo code, and the second platform of the same commit. It does nothing for an ordinary code commit. Low risk.
  4. Precompile flare and json: ~5%, mainly for consistency. Do it at flare_mlake's next release.
  5. Bazel with rules_mojo: no gain on its own for these binaries; it pays only with remote execution fanning out many separate binaries, which is option 1 done another way. High cost and risk.
  6. A lower opt level: slower (measured). Dropped.

Recommended next step

  1. -g0 for non-deploy image builds.
  2. ORCA_COMMIT/ORCA_SOURCE from -D to runtime.
  3. Prototype one job on sai (or a larger runner) that builds the programs as separate or grouped parallel compiles. Measure wall time, image size and peak memory.
  4. Precompile flare and json at flare_mlake's next release.

Decisions for the user: whether to split the multicall binaries (option 1), and whether builds go on sai or on paid larger runners.

🤖 Generated with Claude Code

https://claude.ai/code/session_017TycszSbyw92Efu4ZeCYL6

Activity

  1. winding-lines commented on Nov 12, 2026

    @winding-lines
    ContributorAuthor

    Measured on sai (Threadripper 1900X, 8 cores / 16 threads, 62 GB), one job, census main b3eba2d (morca-mojo 0.25.0), amd64, inside the census's own env stage, flags as scripts/build.sh passes them (-g1 --target-cpu x86-64-v3 -D MORCA_SOURCE=… -D MORCA_COMMIT=…), an empty Mojo compile cache (new MODULAR_CACHE_DIR) for every mojo build. Run: https://github.com/morca-dev/morca/actions/runs/43259456514 (workflow in morca-dev/morca#405).

    The 13 programs of multicall-pipelines (morca_maintain + the 12 pipelines):

    build wall s user+sys s CPU / wall largest process RSS GB peak memory GB (cgroup anon)
    combined multicall-pipelines 368 859 2.3 6.7 6.5
    each alone, one at a time 1667 4287 2.6 3.9 3.8
    each alone, xargs -P 4 591 4669 7.9 3.9 9.9
    each alone, xargs -P 8 505 5293 10.5 3.9 18.0

    Each alone (sequential): morca_maintain 31 s, ballast 44 s, the other eleven 126–179 s each (gharchive 173, mojo_embed 179), 335–450 CPU-s and 3.1–3.9 GB each.

    Sizes, MB (objcopied = the Dockerfile's objcopy --compress-debug-sections=zlib; gzip = gzip -c | wc -c of that):

    raw objcopied gzip
    combined 66.4 21.6 10.2
    sum of the 13 separate 370.7 114.1 59.0

    The hypothesis doesn't hold: total CPU is not about the same. Built alone, the programs take 5× the CPU (4287 vs 859 CPU-s), because each one compiles the shared library code (morca, iceberg, parquet, …) again. On this machine a typical pipeline alone takes 126–179 s, not 48–60 s; the Mac figure came from a much faster machine. So even 8 at a time (505 s, 18 GB) is slower than the one combined binary (368 s, 6.5 GB), and the image would hold 5.5× the bytes. sai's load average was about 3.7 from other jobs when the runs started; that applies to all four rows.

  2. winding-lines commented on Nov 12, 2026

    @winding-lines
    ContributorAuthor

    Mojo's compile cache: how it works, what defeats it, and the disk it needs

    This follows the split measurement above. Mojo's compiler has been open source since 2026-08. I read it at modular/modular@c0bfec57 and mojo/v1.2.0, and tested the cache on a Mac with Mojo 1.2.0.

    How the cache works: it is whole-program, with two entries per mojo build.

    • Entry 1: the elaborated module, keyed on the module's bytecode plus the pass pipeline.
    • Entry 2: one object file, keyed on the elaborated module plus the options (opt level, -g, -j) plus the target.
    • Nothing is cached per function or per library. A per-split key is even built in compileOptimizedLLVMModuleToObject and never looked up. GPU kernels are the only thing cached piece by piece.
    • Pruning only removes other compiler versions. There is no size cap or LRU.

    What changes the key (orca_maintain, 11 s cold):

    change result time
    identical rebuild; mtime touched; env vars; a comment at the end both entries hit 1.3–1.6 s
    another checkout path entry 1 misses 4.4 s
    another path with -strip-file-prefix=<checkout>/ both hit 1.3 s
    lines shifted entry 1 misses 4.6–6.2 s
    one string literal changed both miss 12–13 s
    any -D, even one the program never reads both miss 11 s
    -g0 vs -g1; --target-cpu; -j entry 2 misses or both miss 8–11 s

    What this means for us:

    • The defines: -D MORCA_COMMIT alone guarantees a full rebuild on every commit, so morca #402 is required.
    • PR builds vs main: Build pull requests' images with -g0 #155's -g0 for PR builds means a main build can never reuse its PR's cache entries. Once the cache is persisted, it's better to use the same -g level everywhere.
    • Paths: add -strip-file-prefix=/src/project/ ($PWD/ on a Mac).
    • How often it would hit: of the last 120 merges to main, about 22% change no shipped Mojo (a full hit once the defines are gone). 8% change one program, 25% change src/ and 44% change the lock.

    Disk (against 500 GB): the Mojo cache is about 250–400 MB per build per arch, so a 10–15 GB cap per arch with a pruning cron. The Docker layers are about 10 GB per env revision, so keep the last 2–3 per arch, 30–40 GB. Under 70 GB in total.

    Ranked:

    1. Make builds hermetic: no -D, one -g level, -strip-file-prefix. Hours of work.
    2. Persist MODULAR_CACHE_DIR outside the env, as a BuildKit cache mount on a self-hosted builder. About a day.
    3. Prune the Docker layers to about 40 GB.
    4. Bazel: no gain beyond item 2, since rules_mojo also runs one action per binary. Splitting: no (see the measurement above).

    Upstream proposals for Modular's repo, small to large:

    • U1: keep debug locations in entry 2's key. Today a hit can reuse stale line tables.
    • U2: make entry 1 relocatable by default.
    • U3: a size cap and LRU for the cache.
    • U4: expose enableLLVMPerFunctionSplitting as a flag.
    • U5: a per-split object cache in lowerLLVMModuleToObjects. Estimated 3–6 weeks, maybe 30–50% faster on census-only commits.
    • U6: memoizing specializations across builds. Long term.
  3. winding-lines commented on Nov 12, 2026

    @winding-lines
    ContributorAuthor

    Compile cache: where it stands (step 1 of the plan, plus the follow-up the user asked for).

    Main hits fully. Since #162 (cache mount carried between runs) and #163 (no -D MORCA_COMMIT/MORCA_SOURCE, morca-mojo 0.25.1), a commit that changes no compiled source reuses the whole compile. Draft #165, on main plus logging, first run (restoring main's cache) and three reruns:

    multicall-apps multicall-pipelines image job
    before, a rerun with -D 436–453 s 379–389 s ~12–13 min
    amd64, 4 runs 72–101 s 58–78 s 280 s
    arm64, 4 runs 61–64 s 48–49 s 286 s

    The 4th amd64 run was on a different CPU (EPYC 9V74, the others 7763) and still hit everything.

    What it wasn't:

    • Source file times: touching every source file locally still hits.
    • Package file times: touching every installed package still hits.
    • Parallel builds: both binaries at once into one cache still hits (141 s, then 24 s).
    • The installed packages: byte-identical between CI runs (381 files).
    • Fresh containers or installs: two fresh containers, each with its own pixi install, on colima: 147 s, then 22 s, all hits.

    With the -D defines put back (same value on a rerun), #165 attempts 5–7:

    • arm64 reruns hit everything.
    • amd64 reruns hit the first of each binary's two transform-cache entries and miss the second.
    • The second entry's key was different on the first run, where the first entry was computed, than on both reruns, where the first entry came from the cache. Those three runs were on EPYC 9V45, 7763 and 9V45, so it doesn't follow the CPU.

    Minimal repro, outside the census: doesn't reproduce.

    • The program: tools/cache-repro on the cache-repro branch, about 20 lines, one -D COMMIT it reads.
    • The build: -g1 --target-cpu x86-64-v3 -strip-file-prefix=$PWD/ on Mojo 1.2.0, on a GitHub runner (EPYC 7763) and on sai (Threadripper 1900X, Zen 1).
    • The result: identical cache keys on both, with and without the define. On one machine the same build twice hits. A new -D value, even of a define nothing reads, changes both keys, which is expected.

    So no report for Modular: what's left needs the census's two-entry compile with -D on amd64, and main no longer passes -D. Parked.

    One thing to know about ci.yml: actions/cache doesn't save when the primary key was an exact hit. So a rerun's new entries are kept only if the run's own first attempt didn't already save that key. That's fine for one save per commit.

    🤖 Generated with Claude Code

  4. winding-lines commented on Nov 12, 2026

    @winding-lines
    ContributorAuthor

    The minimal repro's files (the cache-repro branch is deleted):

    repro.mojo:

    """Minimal program for Mojo's compile cache: one -D it reads (COMMIT), the
    rest plain code so the object is not trivial."""
    
    from std.sys import get_defined_string
    
    
    def fib(n: Int) -> Int:
        if n < 2:
            return n
        return fib(n - 1) + fib(n - 2)
    
    
    def main():
        comptime commit = get_defined_string["COMMIT", "none"]()
        var total = 0
        for i in range(25):
            total += fib(i)
        print("commit", commit, "total", total)

    pixi.toml:

    [workspace]
    name = "mojo-cache-repro"
    channels = ["https://conda.modular.com/max", "conda-forge"]
    platforms = ["linux-64", "linux-aarch64", "osx-arm64"]
    
    [dependencies]
    mojo-compiler = "==1.2.0"
    mojo = "==1.2.0"

    run.sh NAME [flags] (prints the cache lookups):

    #!/bin/sh
    # run.sh NAME [mojo build flags...]: builds repro.mojo with the shared cache
    # in ./cache, logging cache lookups, and prints NAME, the time and the lookups.
    name=$1; shift
    mkdir -p cache out
    t=$(date +%s%N)
    MODULAR_CACHE_DIR=$PWD/cache MODULAR_ENABLE_CACHE_LOGGING=1 pixi run mojo build -g1 --target-cpu x86-64-v3 -strip-file-prefix=$PWD/ "$@" repro.mojo -o out/$name > log-$name.txt 2>&1
    code=$?
    ms=$(( ($(date +%s%N) - t) / 1000000 ))
    look=$(grep -oE 'findSyncImpl: (cache hit|miss)[^/]*/[^ ]*' log-$name.txt | sed -E 's#findSyncImpl: (cache hit|miss).*/([A-Za-z0-9]{10})[A-Za-z0-9]*$#\1:\2#' | tr '\n' ' ')
    echo "$name exit=$code ${ms}ms cpu=$(grep -m1 'model name' /proc/cpuinfo | sed 's/.*: //') | $look"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions