Local llama.cpp runners for chat and embeddings, managed with Nix and Make.
flake.nix reproducible dev tools: llama.cpp, curl, jq, sqlite, make
Makefile project commands: download models, run servers, smoke tests
models/ local GGUF files; ignored by git
- Nix with flakes enabled
- macOS/Linux supported by nixpkgs
llama-cpp
On Apple Silicon, the flake enables llama.cpp Metal support.
nix develop
make helpDownload default models:
make modelsDefaults:
- Chat:
Qwen/Qwen2.5-0.5B-Instruct-GGUF, Q4_K_M - Embeddings:
nomic-ai/nomic-embed-text-v1.5-GGUF, Q4_K_M
The model files are stored in ./models/ and are not committed.
Terminal 1:
nix develop
make serve-chatTerminal 2:
nix develop
make test-chatOr open a tiny terminal chat client:
make chatThe chat client streams answers by default using OpenAI-compatible SSE. To disable streaming:
node scripts/chat.mjs --no-streamCommands inside the chat client:
/reset clear conversation history
/exit quit
Endpoint:
POST http://127.0.0.1:8080/v1/chat/completions
Terminal 1:
nix develop
make serve-embedTerminal 2:
nix develop
make test-embedOr open a tiny terminal embedding client:
make embedSingle input mode:
node scripts/embed.mjs --input "The player can double jump after collecting boots."The client prints the returned model, vector dimensions, first few values, and Euclidean norm.
Endpoint:
POST http://127.0.0.1:8081/v1/embeddings
Use Make variables:
make model-chat \
CHAT_MODEL_NAME=my-chat.gguf \
CHAT_MODEL_URL=https://huggingface.co/org/repo/resolve/main/my-chat.gguf?download=true
make serve-chat CHAT_MODEL=./models/my-chat.ggufYou can also let llama-server pull a GGUF directly from Hugging Face:
make serve-chat CHAT_HF_REPO=ggml-org/Qwen3.6-27B-GGUF:Q8_0Qwen3.6-27B shortcut:
make serve-qwen36-27bQwen3.6-27B has a multimodal projector file (mmproj-...gguf) in the GGUF repo.
For text chat, the shortcut passes --no-mmproj so llama.cpp only downloads the
main model. If you need multimodal support, run serve-chat directly without
that flag or override CHAT_EXTRA_ARGS.
Note: CHAT_MODEL_URL must point at a .gguf file. The upstream
Qwen/Qwen3.6-27B repo contains the base model weights, not a llama.cpp GGUF;
use a GGUF repo such as ggml-org/Qwen3.6-27B-GGUF instead.
Embedding example:
make model-embed \
EMBED_MODEL_NAME=my-embed.gguf \
EMBED_MODEL_URL=https://huggingface.co/org/repo/resolve/main/my-embed.gguf?download=true
make serve-embed EMBED_MODEL=./models/my-embed.gguf EMBED_POOLING=meanmake clean fully removes disposable build.nosync/, tools/ and
gams_modules/, including unmarked/stale downloads. Node.js 24 and Git are
required. All trees are preflighted before deletion; tracked paths, non-directory
roots and symlinks (including nested/dangling links) are refused. Never store
authored files or irreplaceable data there. make test-clean checks isolated fixtures.
models/, custom model paths and llama.cpp external caches are excluded.
The existing explicit make clean-models command remains separate and unchanged;
it deletes MODELS_DIR. Neither command stops running servers.
serve-chatdepends on$(CHAT_MODEL), so missing local models are downloaded automatically.- If
CHAT_HF_REPOis set,serve-chatuses llama.cpp-hfinstead ofCHAT_MODEL/CHAT_MODEL_URL. serve-embeddepends on$(EMBED_MODEL), so missing models are downloaded automatically.- Keep index/query embedding model, pooling, and preprocessing consistent.
- For larger models, adjust
CHAT_CTX,CHAT_GPU_LAYERS, and model choice.