Helper module/class for LLM-based apps

New Feature Summary

Now that we have more than a couple of LLM-based apps, and they share a lot of internal logic, I think we should consider refactor some of the LLM_based implementation to the SDK level. For example;

model loading (via litellm)
prompt sending and collecting responses
common LLM configuration (temp, prompt levels - system/user, ...)
common implementation of "long" configuration passing (much longer than usual runtime params that are passable via query string)

Alternatives

No response

Additional context

No response

@kelleyl

@kelleyl Please add more details from our conversation if there's anything missing.

After reviewing the three existing LLM-based apps (smolvlm2-captioner, llava-captioner, chapter-llama) (did I miss any?) and thinking through what the SDK should provide, here's a high-level plan.

Guiding principles

The SDK should handle common mechanics, not prescribe what the LLM does. The task space is wide (text-to-text, image-to-text, audio-to-text, structured data to text, etc.), so the base class must stay flexible.
All configuration should be done via runtime parameters for full transparency. No more YAML config files hiding prompts behind a filename. Every parameter value should be recorded in view metadata via sign_view().
Current LLM apps will likely need to be rewritten to conform to the new pattern. That's fine since we only have a small number of them and they're all still experimental.

LLM base class (tentatively `ClamsLLMApp`)

A new base class inheriting from ClamsApp, following the same pattern as universal parameters. Just as ClamsApp auto-injects pretty, runningTime, hwFetch, tfSamplingMode into every app's metadata, the LLM base class would auto-inject LLM-specific parameters into every LLM app's metadata. These would include things like (all names tentative):

prompt (string): the user prompt
systemPrompt (string): the system prompt
temperature, maxNewTokens, topP, topK, etc.: generation parameters

The exact parameter list is TBD, but the pattern is: declare them once in the SDK, auto-inject into metadata, and intercept/consume them in the SDK before (or alongside) _annotate().

This also ties into the simplification discussed in clamsproject/app-smolvlm2-captioner#8, which eliminates promptMap/systemPromptMap in favor of single prompt/systemPrompt per run.

Model lifecycle helpers

The SDK would provide helpers for:

Loading a model (HF transformers, or remote API) with device/quantization handling
Caching the loaded model across requests (don't reload the same config)
GPU memory cleanup between requests

The exact API is TBD. The challenge is that model loading is quite different across architectures (vision-language models, text-only LLMs, audio models, LoRA adapters), so this may need to be a set of utilities rather than a single rigid interface.

Inference helpers

A common inference method (tentatively generate()) that handles the tokenize-generate-decode cycle with generation params drawn from the auto-injected runtime parameters. App developers would call something like self.generate(prompt, ...) and get text back, without manually managing the generation parameter plumbing.

Again, the exact API is TBD since vision models need image inputs alongside text, audio models need audio inputs, etc.

Inference/annotation separation should be structural

The base class's API shape should enforce separation of inference from annotation creation (MMIF translation). For example, if self.generate(prompts, images, **params) --> List[str] returns just text, then app developers naturally write:

texts = self.generate(prompts, images=batch_images)  # e.g., OCR over multiple images

# MMIF translation block, clean separation
for src, text in zip(sources, texts):
    td = new_view.new_textdocument(text=text)
    new_view.new_annotation(AnnotationTypes.Alignment,
                            properties={'source': src, 'target': td.long_id})

The benefit is concrete: existing apps can incrementally adopt the base class by replacing only the generate() call, without touching annotation logic.

Multimodal input shape

Vision/audio/text-only LLMs need different inputs. A flexible shape:

def generate(
    self,
    prompts: Union[str, List[str]],
    images: Optional[List[Image]] = None,
    audio: Optional[List[AudioInput]] = None,
    **generation_params,
) -> Union[str, List[str]]:
    ...

Modalities are kwargs; the base class dispatches to the appropriate processor logic per backend. Captioner/OCR apps pass images=. Text-only apps pass nothing extra. Audio (ASR) pass audio=. This avoids prescribing a single multimodal "frame" while keeping the surface uniform.

Long parameter passing via JSON envelope

The "long configuration passing" item from the original issue description is being addressed in this comment on #197. This is a prerequisite for making prompt and systemPrompt proper runtime parameters instead of hiding them in YAML config files.

What this means for existing LLM apps

The existing apps (smolvlm2-captioner, llava-captioner, chapter-llama) share a lot of duplicated logic, but the duplication is mostly in areas that are hard to generalize (model-specific loading, task-specific prompt construction, task-specific output parsing). The SDK update would standardize the parts that are genuinely common:

Parameter declaration and handling (no more per-app prompt map parsing)
Generation parameter plumbing
Config transparency (everything in view metadata)
Model lifecycle management

The apps would need to be rewritten to use the new base class, but the result should be significantly simpler since the boilerplate moves into the SDK.

A few more design questions came up around prompting and reasoning models during the meeting today. Notes below.

Base class naming

ClamsLLMApp is probably the wrong name. The SDK isn't standardizing "being an LLM". It's standardizing a promptable interaction: a prompt plus optional content goes in, generated text comes out, with a managed parameter surface. That pattern is identical whether the backend is a text LLM, a VLM, another type of multimodal model, or a remote API.

Owen suggested ClamsPromptableApp. It names the interaction scope rather than the model architecture, and it signals the input/prompt surface that the SDK actually manages.

The only shortcoming I can think of now is discoverability, but that's solvable in documentation. With the addition of this new class, we only have two super classes for app implementation anyway.

An alternative I considered is ClamsGenerativeApp, but I have never been a fan of the term "GenAI", and it only emphasizes the output side, failing to signal the input-side interaction.

Static CoT / multi-turn prompts

Two cases need to be separated:

Inline CoT: "Think step by step. First identify X, then Y, then conclude Z." needs zero SDK support. It is just slightly more elaborate prompt text. There should be no "CoT mode."
Static multi-turn / few-shot scaffolds: a fixed sequence of system/user/assistant turns, e.g. few-shot exemplars or a scripted reasoning scaffold. These are not expressible as a single string and can be genuinely useful. The clean way to support this, I think, is to let the prompt parameter accept a structured message array, which generate() normalizes into the backend's chat format. Alternatively we can introduce a dedicated CoTPrompt type for this use case. Either way, this complexity will be a good stress test for the new envelope design.

Reasoning traces for reasoning models

It's my understanding that trace/answer isolation is not generalizable. Different models use different <think>-like markers, some APIs expose a separate response field, and o1-style models hide the marker entirely. The SDK cannot guess delimiters, so the isolation logic must stay in app code (model-specific regex or API field extraction).

What the SDK can do:

the discipline: when available, let devs preserve reasoning traces for transparency/reproducibility.
a storage convention: where a trace goes in MMIF once the app has isolated it. I'm thinking of a separate TextDocument (with mime=json) in a separate view for clean isolation.
optionally a helper, e.g. store_response(new_view, source, answer, trace=None), that creates the TextDocument + Alignment and puts the trace in the canonical place

So, where does the trace live in MMIF? That needs more discussion, I think. For the first release with "LLM helper", I'd be fine to just leave a placeholder for future implementation. Generally, I'm leaning toward the separate-TextDocument approach (traces can be very large; consumers shouldn't pay for them when they only want the answer), but this is a real tradeoff worth discussing.

Design principle for `generate()` API

This is independent of today's meeting, but something I want to be explicit about.

Scope

The original proposal mentions a generate() helper but is not explicit about its scope and role. The base class's API shape should enforce separation of inference from annotation creation. If self.generate(prompts, images, **params) → List[str] returns just text, then app developers naturally write:

texts = self.generate(prompts, images=batch_images)
for src, text in zip(sources, texts):
    td = new_view.new_textdocument(text=text)
    new_view.new_annotation(AnnotationTypes.Alignment,
                            properties={'source': src, 'target': td.id})

This keeps the new_annotation calls out of the generate call. It also lets future versions of an app incrementally adopt the base class by replacing only the generate() call, without touching annotation creation logic.

Patterns

Vision/audio/text-only LLMs need different inputs. A flexible shape:

def generate(
    self,
    prompts: Union[str, List[str]],
    images: Optional[List[Image]] = None,
    audio: Optional[List[AudioInput]] = None,
    **generation_params,
) -> Union[str, List[str]]:
    ...

Modalities are kwargs; the base class dispatches to the appropriate processor logic per backend. Captioner/OCR apps pass images=. Chapter-llama-style apps pass nothing extra. Audio captioners pass audio=. This avoids prescribing a single multimodal "frame" while keeping the surface uniform.

#291 addresses most of this. What's done:

ClamsPromptableApp base class with the auto-injected parameter surface (prompt, systemPrompt, promptMode, maxNewTokens, temperature, topP, topK, parallelPrompts) and an abstract generate(prompts, images, audios, ...) contract that enforces inference/annotation separation via the return shape (List[str]; List of string because the LLM outputs are expected to be simple string, MMIF I/O should still live in _annotate).
ClamsHFPromptableApp HF specialization with class-attribute model declaration, concrete batched generate(), default build_gen_kwargs().
clams.backends.hf module: load_hf_model (from_pretrained flow) and load_hf_pipeline (pipeline flow), with revision pinning, device auto-detect, and a move_to_device=False opt-out for library-style wrappers.
analyzer_versions: Dict[str, str] on AppMetadata + auto-derived model runtime parameter; raw org/repo lands in view.metadata.parameters, resolved org/repo@<rev> in view.metadata.appConfiguration for self-contained reproducibility.
build_conversation() covers string / multi-turn list / pre-built message-dict forms via promptMode={turn-taking, user-only}; response_to_grounded_textdocument() handles the standard TextDocument + Alignment + origins/origination write-out.
Naming: ClamsPromptableApp (per the 2026-05-18 meeting decision).

Left for follow-ups:

Other backends (litellm / OpenAI / Anthropic / vLLM): these should subclass ClamsPromptableApp and implement generate() themselves. A ClamsLiteLLMApp-style sibling is not in this PR.
Reasoning-trace storage convention: response_to_grounded_textdocument(reasoning_trace=...) exists as a hook but raises NotImplementedError when trying to pass non-null reasoning traces. The MMIF placement (separate TextDocument with mime=json in a separate view, or otherwise) is still TBD.
Inter-request GPU cleanup: per-instance cache holds models for the app's lifetime; no SDK-provided proactive torch.cuda.empty_cache() helper.
Quantization / LoRA: indirectly available via MODEL_KWARGS pass-through; no first-class API.

PR #294 and #299 follow up on this work and resolve the reasoning-trace item from above (and tie off a couple of other loose ends).

split_tagged_reasoning_trace(text, ...) method provides a basic implementation for inline "tag"-based reasoning trace parsing (e.g., <think>[trace]</think>[answer] into ([answer], [trace])). Channel-delimited reasoning (gpt-oss Harmony, Gemma's thought channel) is still not supported.
- assume_open param handles templates that prefill the opening tag into the prompt (e.g., Qwen3.5), where only </think> reaches the output; with it, a missing close tag reads as unterminated (truncated) reasoning, i.e., empty answer with the whole output as the trace.
response_to_grounded_textdocument(reasoning_trace=...) can now handle the trace, putting it into the modelReasoningTrace property of the same TextDocument that carries the answer (as @text). This should ease the parsing complexity for downstream consumers that only care about the answer part, regardless of whether a trace is present.
model_load_kwargs now supports per-variant from_pretrained kwargs, letting an analyzer_versions family mix members with different pre-quantization.
build_template_kwargs offers more control over the chat template (notably enable_thinking, tools, documents).
useReasoning (SDK-level runtime param, bool with default False): common auto-injected promptable param, extending the helper class/modules for wrapping promptable LLMs #291 surface. Apps that want a reasoning toggle can bind it to enable_thinking in build_template_kwargs.

Left for follow-ups:

channel-based trace formats (gpt, gemma, ...)
naming of modelReasoningTrace prop: also see proposing subtypes of TextDocument clams-vocabulary#11.
guard for the "thining loop"
Unchanged from above: other backends (litellm / OpenAI / Anthropic / vLLM subclassing ClamsPromptableApp) and inter-request GPU cleanup.

Revisiting this with a llama.cpp backend in mind. From recent meetings with the LoC folks, I learned that their pipeline doesn't use any of our torch-based LLM/VLM apps as-is. They run every app in its own python environment (no Docker, because of low-level hardware access on Mac and permission issues on enterprise systems), and they built their own "ollama captioner" from our SDK, swapping the torch/transformers stack for Ollama, because they work on generally old hardware with mixed architectures and low VRAM. That's a strong signal that the torch-only promptable app layer we built here doesn't reach a good portion of the users we'd like to serve.

There's also a hardware cliff on our side. PyTorch 2.15 (current PyPI release) dropped Maxwell, Pascal and Volta, and 2.14.x on the cu126 index is the last prebuilt release for them (https://dev-discuss.pytorch.org/t/notice-cuda-12-6-wheels-will-no-longer-be-published-from-pytorch-2-15-drops-maxwell-pascal-volta/3432). So our *-hf5 images built from now on won't run on the TITAN X/Xp machines we have.

I ran some probes on our own hardware:

On a Maxwell GPU, Qwen3-0.6B, 20 chyron note-3 --> note-4 items: transformers fp32 2.41 s/item vs. llama.cpp f16 0.49 s/item on the GPU, 9.49 vs. 2.62 s/item on CPU, with identical outputs (same for Qwen3-1.7B).
On Pascal GPUs, through Ollama's bundled llama-server: Gemma 4 12B at 51 tok/s on one card, Gemma 4 26B-A4B at 69 tok/s and the dense 31B at 23 tok/s across four cards.

With torch dropping support, these machines probably won't do a good job for fine-tuning or autopsy on the latest models. However, my conclusion is that they are still useful as inference-only machines (though not with torch, given the dropped support; just compiling torch from source is already a big headache). So I'd like to propose a llama.cpp backend for ClamsPromptableApp.

proposal

There are two ways to put llama.cpp under a CLAMS app;

binding: llama-cpp-python in the app process.
server: the app starts its own llama-server (the HTTP server shipped with llama.cpp) as a child process on localhost, and talks to it through the OpenAI-compatible chat API.

I think the server way is the right direction, mainly because;

multimodal: llama-server takes a mmproj file and handles multi-image prompts as upstream implements them. The llama-cpp-python API needs a python implementation written per model family to package multimodal prompts, and generally lags behind upstream.
parallelPrompts maps directly to the server's parallel slots (continuous batching). I learned the binding has no easy batched generation.
isolation: a CUDA OOM or segfault kills the server, not the gunicorn worker, so the app can still write an error view.
it's upstream itself; new architectures arrive without waiting for a binding release.
the client side doesn't have to be llama.cpp-specific. With the OpenAI-compatible API, the same client works for vLLM or SGLang launchers later, so the class can be something like ClamsOpenAIPromptableApp with a pluggable launcher, instead of ClamsLlamaCppPromptableApp.

To be clear, this is NOT "talk to an Ollama (or any) server running somewhere". Model selection must stay inside the app enclosure; the app picks the GGUF, the app starts the server, and the server dies with the app. Ollama is only where the speed numbers above came from (it runs the same llama-server underneath).

Model identity fits the current analyzer_versions scheme (#251), with the quantization variant added to the key;

analyzer_versions = {
    "ggml-org/gemma-4-E4B-it-GGUF:Q8_0": "<rev>",
}
# refined `model` value in appConfiguration --> "ggml-org/gemma-4-E4B-it-GGUF:Q8_0@<rev>"

A Hub revision is a git commit, so it pins both the model file and the mmproj file in the repo. For our own fine-tunes (e.g., the chyron KIE converter), we'd convert with the convert_hf_to_gguf.py matching the pinned llama.cpp and upload to a clamsproject/*-GGUF repo.

The tricky part: two servers in one container

With this, a production container runs two servers (gunicorn + llama-server), and that needs care;

lifecycle: pick a free port, wait for /health, kill the child on exit/signal, forward its log to the app logger.
worker count: number_of_workers() in clams/restify/__init__.py decides gunicorn workers by VRAM of GPU 0 // est_gpu_mem_typ. If each worker starts its own llama-server, we get N copies of the model. Options are one worker + server slots for concurrency, or one worker per GPU (worker i pinned to GPU i on multi-GPU machines). Either way, the restifier needs to know about the launcher.
GPU memory: the launcher can't free memory held by other processes (e.g., an Ollama service on the host keeping a model loaded). It should check free memory per GPU before starting and fail with a clear message, instead of letting llama.cpp silently offload layers to CPU.
_profile_cuda_memory uses torch, which won't be in the image and can't see a child process anyway. The nvidia-smi fallback should be the path for this backend.
where the llama-server binary comes from. I'd rather not have CLAMS build and maintain llama.cpp binaries; pinning an upstream build (release binaries or the ghcr.io/ggml-org/llama.cpp:server-cuda images) is preferred, but I haven't checked which GPU archs those CUDA builds include (Maxwell sm_52 and Pascal sm_61 are the question for me; we also need to check the Mac backend support). If they don't cover old GPUs, we can compile our own, but then version control and pinning get messy.

Implemented in #321 as ClamsGGUFPromptableApp (experimental). Where the decisions are documented:

HF vs GGUF, side by side: table; batch size; output differences.
Declaring a GGUF app; no parallelPrompts.
Server settings (LLAMA_ARG_*, no GPU memory check, no hardware knobs in the SDK).
Container image and the Pascal kernel compile.
Worker count and recycling (RESTIFIER_PRODUCTION_*).

Navigation Menu

Repository navigation

Helper module/class for LLM-based apps #263

New Feature Summary

Related

Alternatives

Additional context

Guiding principles

LLM base class (tentatively `ClamsLLMApp`)

Model lifecycle helpers

Inference helpers

Inference/annotation separation should be structural

Multimodal input shape

Long parameter passing via JSON envelope

What this means for existing LLM apps

Base class naming

Static CoT / multi-turn prompts

Reasoning traces for reasoning models

Design principle for `generate()` API

Scope

Patterns

proposal

The tricky part: two servers in one container

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Helper module/class for LLM-based apps #263

Description

New Feature Summary

Related

Alternatives

Additional context

Activity

keighrim commented on Nov 19, 2025

keighrim commented on Mar 23, 2026

Guiding principles

LLM base class (tentatively ClamsLLMApp)

Model lifecycle helpers

Inference helpers

Inference/annotation separation should be structural

Multimodal input shape

Long parameter passing via JSON envelope

What this means for existing LLM apps

keighrim commented on May 18, 2026

Base class naming

Static CoT / multi-turn prompts

Reasoning traces for reasoning models

Design principle for generate() API

Scope

Patterns

keighrim commented on May 29, 2026

keighrim commented on Jul 8, 2026

keighrim commented on Nov 6, 2026

proposal

The tricky part: two servers in one container

keighrim commented on Nov 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

LLM base class (tentatively `ClamsLLMApp`)

Design principle for `generate()` API