lfm.zig

A high-performance Apple Silicon LLM inference runtime written in Zig.

lfm.zig explores how far CPU-only inference can be pushed on modern Macs. It runs GGUF weights directly from a read-only mmap, keeps quantized weights packed throughout inference, and drives the compute-heavy paths with ARM64 sdot and Apple's Accelerate framework. On the reference M5 Max system, the current runtime reaches 220.6 prefill tokens/s and 95.67 decode tokens/s with the 5.2 GB Q4_K_M model described in Reference performance.

The current release supports Liquid AI's dense LFM2.5-2.6B and sparse MoE LFM2.5-8B-A1B. Both use the reusable runtime pieces: a bounds-checked GGUF loader, tokenizer, packed quantized kernels, persistent CPU worker pool, attention and convolution caches, sampling, and an embeddable Agent API.

The runtime includes a command-line interface for generation and chat, plus a Zig library for streaming tokens, cloning agent state, and running tool-calling loops.

Current support

Supported today	Not supported today
Apple Silicon macOS	Intel Macs, Linux, and Windows
LFM2.5-2.6B and LFM2.5-8B-A1B GGUF files with the expected tensor layouts	Other model architectures
Q4_K and Q6_K quantized weights	Arbitrary GGUF quantization formats
CPU inference through ARM64 and Accelerate	Metal, MLX, MPS, CUDA, or Vulkan
Pinned Zig stable (0.16.0), with no third-party Zig packages	Other Zig versions

Model support is strict by design: the loader detects the dense or MoE architecture and validates its dimensions, layer schedule, and all 266 or 256 required tensor names and shapes before inference. Adding another model family requires an explicit model configuration and forward path while reusing the platform, storage, kernel, execution, and session layers already in place.

The project is independent and is not affiliated with or endorsed by Liquid AI. See ARCHITECTURE.md for a detailed implementation tour.

Quick start

1. Install the pinned Zig toolchain

./scripts/fetch-zig.sh

The zigw wrapper prefers a matching .zig-toolchain/zig, falls back to a matching system installation, and requires the version declared in .zigversion.

2. Download the model

The official LFM2.5-2.6B Q4_K_M model is approximately 1.67 GB:

mkdir -p "$HOME/.cache/models"
curl -fL \
  "https://huggingface.co/LiquidAI/LFM2.5-2.6B-GGUF/resolve/main/LFM2.5-2.6B-Q4_K_M.gguf" \
  -o "$HOME/.cache/models/LFM2.5-2.6B-Q4_K_M.gguf"

export LFM_WEIGHTS_FILE="$HOME/.cache/models/LFM2.5-2.6B-Q4_K_M.gguf"

For the larger sparse model, download LFM2.5-8B-A1B-Q4_K_M.gguf and point LFM_WEIGHTS_FILE at it. The compatible Heretic Q4_K_M variant has also passed the model-backed smoke test. The official weights are distributed under the LFM Open License 1.0.

3. Build and run

./zigw build -Doptimize=ReleaseFast
./zig-out/bin/lfm ask "What is 21 + 21?"

The release binary is written to zig-out/bin/lfm.

Command-line interface

lfm has three modes:

Command	Purpose
lfm chat	Interactive, cached, multi-turn chat
lfm ask <question>...	One-shot ChatML question and answer
lfm generate <prompt>...	Raw text continuation

# Interactive chat
./zig-out/bin/lfm chat

# One-shot question
./zig-out/bin/lfm ask "Explain rotary embeddings briefly."

# Deterministic raw completion
./zig-out/bin/lfm generate --greedy --max-gen 64 \
  "The capital of Denmark is"

All three commands accept the same options:

Option	Effect
--greedy	Use deterministic argmax decoding
--max-gen N	Generate at most N tokens per turn; default: 2048
--max-think N	Force </think> after N reasoning tokens
--no-think	Disable the reasoning block
--hide-think	Generate reasoning without printing it
--num-threads N	Cap the number of active CPU worker partitions

Interactive chat additionally supports:

Input	Effect
/clear	Reset the conversation
/exit	End the session
@path	Insert the contents of a file into the prompt

Commands can also run directly through the build system:

./zigw build -Doptimize=ReleaseFast run -- ask "What is 21 + 21?"

Using lfm.zig as a library

build.zig exports the lfm module and installs a static library. The main types are model.Model, which owns the mapped model, and agent.Agent, which owns conversation, sampling, cache, and convolution state.

Add the dependency

Until a stable package release is published, pin the repository as an SSH submodule:

git submodule add git@github.com:CerebralCoding/lfm_zig.git vendor/lfm_zig

Add the path dependency to your build.zig.zon:

.dependencies = .{
    .lfm_zig = .{ .path = "vendor/lfm_zig" },
},

Then import its module from your build.zig:

const lfm_dep = b.dependency("lfm_zig", .{
    .target = target,
    .optimize = optimize,
});
exe.root_module.addImport("lfm", lfm_dep.module("lfm"));

The API follows semantic versioning, but releases before 1.0.0 may make breaking API changes between minor versions. Pin an exact commit or release when using the library from another project.

Generate a response

const std = @import("std");
const lfm = @import("lfm");

pub fn answer(allocator: std.mem.Allocator) !void {
    var model = try lfm.model.Model.load(
        allocator,
        "LFM2.5-8B-A1B-Q4_K_M.gguf",
    );
    defer model.deinit();

    var agent = try lfm.agent.Agent.init(allocator, &model);
    defer agent.deinit();

    try agent.appendUser("What is the capital of Denmark?");
    var turn = try agent.assistantTurn(.{});
    defer turn.deinit();

    std.debug.print("{s}\n", .{turn.text});
}

Streaming and generation settings

Generation accepts a context plus function pointer and invokes it for every decoded token:

const Printer = struct {
    fn token(_: ?*anyopaque, _: u32, text: []const u8) void {
        std.debug.print("{s}", .{text});
    }
};

var turn = try agent.generate(.{ .call = Printer.token });
defer turn.deinit();

Agent configuration is explicit and mutable:

agent.useGreedy();
agent.setTemperature(0.2);
agent.setTopK(80);
agent.setRepeatPenalty(1.05);
agent.setMaxGen(2048);
agent.setMaxContext(32_768);
agent.setMaxThink(512);

The prompt and state APIs cover the common embedding workflows:

API	Purpose
append	Append raw text, adding BOS when the transcript is empty
appendUser	Append a complete ChatML user turn
appendSystem	Append a system turn and registered tool declarations
appendTokens	Append pre-tokenized IDs
prefill	Warm the newly appended prefix without generating
clone	Fork the transcript, sampler, KV cache, and convolution state

Each returned Turn owns its generated token IDs, decoded text, timing and token statistics, and stop reason (eos, max_new, or tool_call). Release it with Turn.deinit.

Tool calling

Tools are ordinary Zig callbacks described by a small, programmatically built JSON schema. Register tools before calling appendSystem, then use assistantTurnWithTools to let the agent generate calls, dispatch callbacks, append results, and continue until completion.

var schema = lfm.tool.Schema.init(allocator);
defer schema.deinit();
_ = try schema.req("a", .int, "First addend");
_ = try schema.req("b", .int, "Second addend");

const Add = struct {
    fn call(
        _: ?*anyopaque,
        alloc: std.mem.Allocator,
        request: *const lfm.tool.ToolCall,
    ) ![]u8 {
        const a = request.parseArg(i64, "a") orelse 0;
        const b = request.parseArg(i64, "b") orelse 0;
        return std.fmt.allocPrint(alloc, "{d}", .{a + b});
    }
};

const add = try lfm.tool.Tool.init(
    allocator,
    "add",
    "Add two integers.",
    &schema,
    .{ .context = null, .call = Add.call },
);
try agent.addTool(add); // ownership moves to the agent

See examples/tool_call_demo.zig for the minimal complete flow and examples/tool_suite_demo.zig for multiple tools, callback context, observer hooks, and error handling.

Examples

Example	What it demonstrates
examples/chat.sh	Interactive chat and one-shot prompts
examples/stream.sh	A streaming-style prompt loop
examples/throughput.sh	Repeated runs with aggregate throughput
examples/agent_library.zig	Library integration and agent settings
examples/tool_call_demo.zig	Minimal tool schema and call loop
examples/tool_suite_demo.zig	Multiple tools, context, and error handling
examples/tool_suite.sh	Shell wrapper for the multi-tool example

./examples/chat.sh --chat
./examples/chat.sh "Summarize Apple's Apple Silicon launch strategy"
./examples/stream.sh "How does vectorization help CPU inference?"
./examples/throughput.sh --threads 4 --runs 5 "$LFM_WEIGHTS_FILE"
./zigw build -Doptimize=ReleaseFast run-tool-suite-example
./examples/tool_suite.sh --model "$LFM_WEIGHTS_FILE"

Implementation

The current LFM2.5 inference paths are specialized for these models and platform:

• GGUF weights remain in a read-only private mmap and are demand-loaded by macOS.
• Q4_K and Q6_K weights are fused with per-256-element Q8 activation blocks.
• ARM64 sdot performs quantized integer inner products.
• Four-row quantized tiles reuse unpacked weights across prompt tokens.
• Prompt projections are batched and mixture-of-experts prompt work is grouped by expert.
• A persistent worker pool dynamically assigns contiguous row chunks.
• Accelerate BLAS handles dense F32 matrix operations; vDSP and vForce handle suitable vector operations.
• Cached grouped-query attention carries KV state across tokens.
• RMSNorm and softmax use F64 reductions for numerical stability.

There is no GPU fallback or alternative backend.

Testing and benchmarking

Fast tests do not require model weights:

./zigw build test
./zigw build test -Doptimize=ReleaseFast

The end-to-end smoke test loads LFM_WEIGHTS_FILE and checks a deterministic continuation:

./zigw build e2e -Doptimize=ReleaseFast

Use benchmark.sh for a combined throughput and correctness run:

MAX_NEW=128 ./benchmark.sh "$LFM_WEIGHTS_FILE"
MAX_NEW=128 ./benchmark.sh "$LFM_WEIGHTS_FILE" 4

Benchmark output reports prefill throughput, decode throughput, and the generated continuation used for the correctness check. To collect a macOS sample(1) profile instead:

./profile_macos.sh "$LFM_WEIGHTS_FILE"

Reference performance

The thermally controlled comparison recorded on 2026-07-26 used this machine:

Component	Specification
Machine identifier	Mac17,6
SoC	Apple M5 Max
CPU topology	18 physical/logical cores: 6 Super + 12 Performance
Memory	64 GiB
Operating system	macOS 26.5.2, build 25F84
Power	AC power; Low Power Mode disabled
Zig	0.16.0
Build	Native target, ReleaseFast, all 18 CPU workers
Temperature monitor	macmon 0.7.2

Both binaries used the official LFM2.5-8B-A1B-Q4_K_M.gguf, exactly 5,155,564,768 bytes with SHA-256 4923ec14f06b968b74d663e5949867d2d9c3bf13a20b8be1a9f9af39989b2bb0. The baseline was main at cf61ab092339. The optimized binary was commit 5943349, which is now part of main.

The controlled comparison ran benchmark.sh with MAX_NEW=256, using its exact 233-token long Copenhagen prompt. macmon measured CPU temperature immediately before each run:

Build	Starting CPU temperature	Prefill	Decode
Baseline (cf61ab0)	41.92 °C	215.4 tok/s	85.04 tok/s
Current (5943349)	42.14 °C	220.6 tok/s	95.67 tok/s

The current implementation improves prefill throughput by 2.4% and decode throughput by 12.5% in the matched run. Both continuations passed the benchmark's Copenhagen sanity check.

Absolute throughput varies with temperature, background load, prompt length, and growing attention context.

Dense-model decode optimization

On 2026-08-11, three MAX_NEW=256 runs of the same 233-token benchmark prompt with the official LFM2.5-2.6B-Q4_K_M.gguf measured the worker-dispatch change:

Build	Average prefill	Average decode
Before generation-based worker dispatch	201.0 tok/s	68.33 tok/s
Current default scheduling	199.8 tok/s	90.33 tok/s

This is a 32.2% decode-throughput improvement with effectively unchanged prefill throughput. Prompt batches still use all 18 logical CPUs on the test machine; memory-bound decode automatically uses 12 worker partitions. Passing --num-threads continues to override the automatic choice.

Memory use

The mapped Q4_K_M weights occupy approximately 1.67 GB for LFM2.5-2.6B or 5.2 GB for LFM2.5-8B-A1B. Runtime memory also includes the KV cache, convolution state, tokenizer tables, a small set of pre-decoded F32 tensors, and transient forward-pass arenas. Total use primarily depends on the selected model and context length.

Origins and license

lfm.zig is an independent Zig implementation inspired by maximecb/bebelm. While it shares the same high-level goal, it has been reimplemented and optimized specifically for Apple Silicon, with a focus on CPU inference using Apple-native technologies such as ARM64 SIMD and Accelerate.

The source code is available under the MIT License. Model weights remain separate artifacts governed by their respective licenses.

See CHANGELOG.md for release history and SECURITY.md for vulnerability reporting.