Repository navigation
Model-footprint analysis: 'will it fit on the device?' in seconds, for GGUF / safetensors / ONNX #5
Copy link
Copy link
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Idea
A model-footprint analysis CLI: given a model file, answer "will this fit on the target device?" in seconds — before anyone invests a day in converting it.
The concrete pain: there is no quick way to check whether a model fits on an embedded device with limited memory and compute (usable RAM ≈ 2.1 GB) before committing to a full conversion run. The tool should:
- accept GGUF, safetensors and ONNX files and analyze them from headers/metadata only (no payload load — a multi-GB file is answered in seconds);
- report the actual resident size of the weights — as stored, and (where the format carries encodings) as they would be held in memory after resolution, since a Q4 weight that has to dequantize costs ~8× its file size;
- print a verdict against the device budget (fits / does not fit, with margin and actionable suggestions such as KV-cache quantization or a smaller context).
What already exists in SKaiNET core
Most of the machinery landed with the SKEEP-003 memory-architecture arc (SKaiNET-developers/SKaiNET#932):
- Header-only planning for GGUF —
StreamingGGUFReader.planInput()builds aPlanInputfrom the tensor table + architecture metadata without reading payloads;MemoryPlans.plan()prices weights, KV cache and the forward slab against aBudgetand renders a fit table with suggestions. - Per-device policy —
PlannerProfile(e.g.MOBILE_2GB: reserve, mapped weights, auto KV quantization, strict mode) andDeviceFit/DeviceMemoryfor checking both RAM pools. - Resolved cost, not file cost —
WeightFormResolver(what the file holds × profile × kernel capabilities) andAllocationResolver.explain()(P7: AllocationResolver — one resolver decides domain and scope SKaiNET#1143): the plan prices what will actually be held, and can say per tensor where it lands and why, pre-load. - A working CLI skeleton —
skainet-apps/skainet-planalready does all of the above for GGUF and prints the plan.
Tracked as a coding issue in core: SKaiNET-developers/SKaiNET#1169 (safetensors + ONNX front ends, the ~2.1 GB budget flag and verdict ergonomics).
Research questions
- ONNX weight enumeration without materialization — initializers live inside the protobuf graph; what is the cheapest reliable way to enumerate names/dims/dtypes (and external-data references) without loading tensor bytes, and how do quantized ONNX conventions (QDQ, MatMulInteger) map onto a resident-bytes estimate?
- Runtime overhead model per format/runtime — file weights are a lower bound; what per-runtime headroom model (allocator overhead, graph buffers, KV cache for decoder models without GGUF-style metadata) makes the 2.1 GB verdict trustworthy rather than optimistic? Can we calibrate it against measured RSS on the device (plan-vs-actual, as SKEEP-003 M1 did for GGUF decode)?
- Verdict quality — false "fits" is the expensive failure. What safety margin and phrasing keep the tool honest when geometry metadata is missing (safetensors/ONNX), where only weights + headroom can be priced?
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity
[-]Model-footprint analysis: 'will it fit on the box?' in seconds, for GGUF / safetensors / ONNX[/-][+]Model-footprint analysis: 'will it fit on the device?' in seconds, for GGUF / safetensors / ONNX[/+]michalharakal commentedon Aug 26, 2026
The CLI implementation is parked on
feature/model-footprint-cli— a standalone Gradle project (model-footprint/) over the core APIs, build-verified against mavenLocal (tests green; the installed binary answers a synthetic safetensors with a correct fits-verdict at--profile edge --budget 2.1G). It waits here for the first stable core release carrying the capability PR (SKaiNET-developers/SKaiNET#1170); the README has the local-build recipe until then.