Skip to content

About

Local llama.cpp chat and embedding servers with Nix-managed tools and Make commands.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

6 Commits

Folders and files

Repository files navigation

gams-models

Local llama.cpp runners for chat and embeddings, managed with Nix and Make.

Layout

flake.nix   reproducible dev tools: llama.cpp, curl, jq, sqlite, make
Makefile    project commands: download models, run servers, smoke tests
models/     local GGUF files; ignored by git

Requirements

  • Nix with flakes enabled
  • macOS/Linux supported by nixpkgs llama-cpp

On Apple Silicon, the flake enables llama.cpp Metal support.

Start

nix develop
make help

Download default models:

make models

Defaults:

  • Chat: Qwen/Qwen2.5-0.5B-Instruct-GGUF, Q4_K_M
  • Embeddings: nomic-ai/nomic-embed-text-v1.5-GGUF, Q4_K_M

The model files are stored in ./models/ and are not committed.

Chat server

Terminal 1:

nix develop
make serve-chat

Terminal 2:

nix develop
make test-chat

Or open a tiny terminal chat client:

make chat

The chat client streams answers by default using OpenAI-compatible SSE. To disable streaming:

node scripts/chat.mjs --no-stream

Commands inside the chat client:

/reset   clear conversation history
/exit    quit

Endpoint:

POST http://127.0.0.1:8080/v1/chat/completions

Embedding server

Terminal 1:

nix develop
make serve-embed

Terminal 2:

nix develop
make test-embed

Or open a tiny terminal embedding client:

make embed

Single input mode:

node scripts/embed.mjs --input "The player can double jump after collecting boots."

The client prints the returned model, vector dimensions, first few values, and Euclidean norm.

Endpoint:

POST http://127.0.0.1:8081/v1/embeddings

Override models

Use Make variables:

make model-chat \
  CHAT_MODEL_NAME=my-chat.gguf \
  CHAT_MODEL_URL=https://huggingface.co/org/repo/resolve/main/my-chat.gguf?download=true

make serve-chat CHAT_MODEL=./models/my-chat.gguf

You can also let llama-server pull a GGUF directly from Hugging Face:

make serve-chat CHAT_HF_REPO=ggml-org/Qwen3.6-27B-GGUF:Q8_0

Qwen3.6-27B shortcut:

make serve-qwen36-27b

Qwen3.6-27B has a multimodal projector file (mmproj-...gguf) in the GGUF repo. For text chat, the shortcut passes --no-mmproj so llama.cpp only downloads the main model. If you need multimodal support, run serve-chat directly without that flag or override CHAT_EXTRA_ARGS.

Note: CHAT_MODEL_URL must point at a .gguf file. The upstream Qwen/Qwen3.6-27B repo contains the base model weights, not a llama.cpp GGUF; use a GGUF repo such as ggml-org/Qwen3.6-27B-GGUF instead.

Embedding example:

make model-embed \
  EMBED_MODEL_NAME=my-embed.gguf \
  EMBED_MODEL_URL=https://huggingface.co/org/repo/resolve/main/my-embed.gguf?download=true

make serve-embed EMBED_MODEL=./models/my-embed.gguf EMBED_POOLING=mean

Cleanup

make clean fully removes disposable build.nosync/, tools/ and gams_modules/, including unmarked/stale downloads. Node.js 24 and Git are required. All trees are preflighted before deletion; tracked paths, non-directory roots and symlinks (including nested/dangling links) are refused. Never store authored files or irreplaceable data there. make test-clean checks isolated fixtures.

models/, custom model paths and llama.cpp external caches are excluded. The existing explicit make clean-models command remains separate and unchanged; it deletes MODELS_DIR. Neither command stops running servers.

Notes

  • serve-chat depends on $(CHAT_MODEL), so missing local models are downloaded automatically.
  • If CHAT_HF_REPO is set, serve-chat uses llama.cpp -hf instead of CHAT_MODEL/CHAT_MODEL_URL.
  • serve-embed depends on $(EMBED_MODEL), so missing models are downloaded automatically.
  • Keep index/query embedding model, pooling, and preprocessing consistent.
  • For larger models, adjust CHAT_CTX, CHAT_GPU_LAYERS, and model choice.

About

Local llama.cpp chat and embedding servers with Nix-managed tools and Make commands.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages