CloakNotes

A private workspace for turning audio into notes. Record or drop in a file, and it transcribes and summarises entirely on-device — no account, no cloud, no network call after install.

Windows desktop app. Flutter shell, Python inference engine, no bridge library between them.

┌──────────────────────────────┐        ┌─────────────────────────────┐
│  Flutter shell (Dart)        │  NDJSON│  Python engine              │
│                              │ ─────► │                             │
│  sidebar · layout · theming  │ stdout │  faster-whisper  (transcribe)│
│  recording · live metering   │◄───── │  llama.cpp      (summarise)  │
│  notes · history             │ stdin  │  sounddevice   (capture)    │
│                              │  stop  │                             │
└──────────────────────────────┘        └─────────────────────────────┘
                    no FFI, no bindings, no shared memory

---

Why it is built this way

The obvious way to ship an ML desktop app is to bind the runtime into the UI — whisper.cpp through FFI, ort through a plugin, the model loaded straight into the render process. That buys speed and costs a lot: the UI is now compilable only against a specific runtime's ABI, a model load takes the window down with it, and a crash in the decoder takes the window down with it too.

CloakNotes puts a process boundary in the middle instead.

The engine is a child process that speaks newline-delimited JSON on stdout and accepts a one-word stop on stdin. The shell spawns it, reads a line at a time, and maps each line to a state change. That buys four things:

• The UI cannot be taken down by the model. A decoder segfault is a dead child process and an error snackbar. The window survives.
• Swapping the engine is a file edit. Porting to C++ or ONNX means writing something that emits the same eleven event kinds. Nothing in app/lib/ knows what a GGUF file is.
• It is testable. Widget tests inject a stub engine and never spawn Python. All 34 of them run in about five seconds.
• It is debuggable. The event stream is a text protocol. paste this into a terminal and you can see exactly what the app sees is literally true — see Protocol.

The cost is process startup (~1.5 s for the interpreter) and one serialisation per event. For an operation that takes 40 s, that is free.

The engine has no idea Flutter exists. src/pipeline.py, transcribe.py, summarize.py and library.py import no UI toolkit — verified, and enforced by the fact that nothing in the Python tree references tkinter at all. The Dart side imports no inference library; the only .gguf strings in app/lib/ are a filename list used to check whether a model is installed.

---

The protocol

Eleven event kinds, all NDJSON, one object per line:

kind	Carries	When
input	device, rate	the capture stream opened, at the negotiated rate
recording	seconds	once a second, counting down
level	rms	~15×/s, the live meter (already dB-mapped)
captured	seconds, peak, rate	the take ended; peak is what it actually heard
stage	index, state, seconds	per pipeline stage: running → done
result	transcript, notes, path	both stages finished
done	elapsed, peak	telemetry for the history entry
devices	array of inputs	--devices only, not part of a run
log	message	Python stderr, surfaced rather than swallowed
error	message	user-facing; already plain English
cancelled / finished	—	terminal

You can drive the engine by hand and see the whole thing:

.\.venv312\Scripts\python.exe src\stream_cli.py --devices
.\.venv312\Scripts\python.exe src\stream_cli.py --record 5 --device 1

Two decisions worth explaining

Levels are mapped in decibels, not by a gain multiplier. The meter was fed a fixed ×8. That cannot work across hardware: a laptop array and a headset sit orders of magnitude apart in sensitivity, so one gain shows a flat line on one and pins the meter on the other. Measured in a quiet room here, the raw RMS is 0.0008 — ×8 is 0.006, which every meter renders as silence. It now maps −72…−12 dBFS, which is roughly where speech sits relative to full scale on any input.

The sample rate is negotiated, never assumed. 16 kHz is what Whisper wants, and hard-coding it is why recording "didn't work" on real hardware: 6 of the 16 inputs on this machine refuse to open at 16 kHz, including the Realtek analogue mic jack. The engine now tries 16 kHz and falls back to the device's own rate. Whisper resamples internally, so 48 kHz is fine. That one change turned the Realtek jack from impossible into working.

---

Setup

Requires Python 3.12 (the faster-whisper / ctranslate2 wheels have no 3.14 build) and Flutter 3.41+.

git clone https://github.com/toe-dot-tech/cloaknotes.git
cd cloaknotes

py -3.12 -m venv .venv312
.\.venv312\Scripts\python.exe -m pip install -r requirements.txt

# Then fetch a summariser model into models/ (see below)

Models

Weights are not in the repo — 1.1 GB of GGUF does not belong in git history. Fetch one into models/:

File	Size
tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf	638 MB
qwen2.5-0.5b-instruct-q4_k_m.gguf	469 MB

Whisper (base, int8) downloads from HuggingFace on first run and caches under ~/.cache/huggingface. That is the only network access the app ever makes, and it happens once, during install.

llama-cpp-python is pinned to 0.2.11, which cannot load the Qwen 2.5 architecture (unknown model architecture: 'qwen2'). TinyLlama is the one that works today. See Limitations.

Build and run

powershell -ExecutionPolicy Bypass -File .\tools\flutterw.ps1 build windows --release
# -> app\build\windows\x64\runner\cloaknotes.exe

The wrapper exists because Flutter's Windows toolchain needs the MSVC environment that a normal shell does not have. It locates vcvars64.bat and puts Ninja on PATH. tools/FLUTTER_TOOLCHAIN.md documents the four Flutter SDK patches this setup needs and why each one exists.

Stop the app before rebuilding — Windows will not overwrite a running .exe (LNK1104).

---

Tests

.\tools\flutterw.ps1 test                                # 34 Flutter tests, ~5s
.\.venv312\Scripts\python.exe tests\test_summarize.py    # 15 Python tests, instant

The Python suite covers the summariser's text post-processing — pure functions, no model, no audio, no GPU, running in about a millisecond. Every case in it corresponds to a defect that actually shipped; the comments name the failure rather than restating the assert.

The Flutter suite is mostly widget tests against the real shell at the shipping window size (1228 × 816 — 80% of 1920px at 125% DPI, not maximised). They cover the recording source swap, the microphone picker, note titling, snackbar behaviour and the responsive breakpoints.

Two things learned the hard way and worth knowing before adding tests:

• Never pumpAndSettle. The CTA progress sweep and the level meter animate forever by design, so it never settles. Use the settle() helper — eight frames of 80 ms. One pump(d) advances the clock but draws a single frame, and AnimatedSwitcher drops its outgoing child several frames after the fade reports done.
• Assert on keys, not types. find.byType(MicPicker) always succeeds — the widget stays in the tree returning a SizedBox.shrink() when it has nothing to show. Icons.mic_rounded is ambiguous; the record button uses it too.

---

Design

tokens.json is the single source of truth for colour and spacing, and tools/gen_tokens.py generates app/lib/tokens.g.dart from it. Change the JSON, regenerate, and both the Python and Dart sides follow. There are no hex literals in a widget file.

Every text pair clears WCAG AA (4.5:1), and the ratios are measured rather than eyeballed:

Pair	Light	Dark
ink on surface	18.11:1	17.79:1
ink_muted on surface	6.13:1	8.37:1
ink_dim on canvas	4.70:1	5.14:1
accent_fg on accent_fill	4.60:1	4.60:1

The UI does not name models. No "Whisper", no "TinyLlama", no "GGUF", no sample rate, no "RMS" appears anywhere a person can read it. The pipeline is shown as three anonymous stages with a progress bar. This is a deliberate product decision — the people who want this app do not know or care which decoder produced the text, and surfacing it makes the app feel like a toolchain rather than a product. It also constrains the wire protocol: error messages are written to be user-facing, so raw driver text is translated before it reaches the screen ("Cannot be used for recording", never "Invalid sample rate").

Responsive behaviour is breakpoint-driven, not breakpoint-guessed — the source column is clamp(210, width × 0.32, 380), and the two-column audio layout collapses to a stack below 620px while the notes split view collapses below 900px.

---

Measured

On a Lenovo Yoga L13, CPU only, no GPU:

Path	Peak RSS	Time
Whisper + LLM both resident	909 MB	—
Whisper + summarise, file in	780 MB	42 s for 18 s of audio
Summarise alone	718 MB	17 s

Numbers are peak working set of the whole process, read via K32GetProcessMemoryInfo (dependency-free, no psutil). They come back on the done event and are stored per run in the history.

---

Limitations

Stated plainly, because the interesting part of this repo is knowing where the edges are.

• Note quality is the weak link. A 1.1B model produces topic lists rather than structured notes, and still occasionally invents details. This is a model ceiling, not a plumbing bug — the fix is a 1.5B+ instruct model, not code.
• Qwen 2.5 does not load with the pinned llama-cpp-python==0.2.11. Needs 0.3.x. Until then TinyLlama is the only working option.
• Notes are not editable. They are generated and copied, not edited and re-saved. This is the single biggest gap versus a real notes app.
• Windows only. The device-probing path assumes WASAPI/sounddevice semantics, and the Python interpreter is located as .venv312\Scripts. The Dart shell itself is portable; the engine wrapper is not.
• The app finds its engine by walking up the directory tree looking for src/stream_cli.py. Move the .exe away from the repo and it cannot find Python. Fine for development, not for distribution.
• A sub-1B model hallucinating is a real risk in a tool whose entire promise is that your meeting notes are trustworthy. The transcript is shown beside the notes precisely so a wrong claim is checkable, but there is no verification layer yet.

---

Layout

app/lib/            Flutter shell — no inference library is imported here
  engine.dart         the process boundary: spawn, decode NDJSON, stop/kill
  library.dart        history store, write serialisation, title inference
  layout.dart         breakpoints and column maths
  state.dart          app state
  widgets/            sidebar, controls, mic picker, level meter, output panel
  screens/            audio, notes, coming-soon
app/test/           34 widget + unit tests
src/                Python engine — no UI toolkit is imported here
  stream_cli.py       the NDJSON endpoint; --devices, --record, --audio
  pipeline.py         run orchestration, capture, dB metering, stage events
  transcribe.py       faster-whisper wrapper, language aware
  summarize.py        llama.cpp wrapper + text post-processing
  library.py          history store (Dart is the primary reader)
  gui.py, workspace.py, design.py   the original Tkinter shell, kept for reference
tools/              token generator, build wrapper, palette sampler, SDK patches
tests/              15 Python tests
tokens.json         design tokens — generate tokens.g.dart from this

The Tkinter shell is retained deliberately. It is ~1,500 lines that were written and thrown away by the port, and it is the clearest record of what the design system looked like before Flutter. It is not on the shipping path.

---

Notable details

A few decisions that are not obvious from the code, recorded so they do not get "cleaned up" later:

• PopupMenuPosition.over is broken inside a padded subtree. It positions against the button's local rect, which put the microphone menu at x = −94 — entirely off-screen. The picker hand-places a showMenu against the overlay rect instead, and flips upward when there is no room below.
• Windows device names collide when truncated. "Microphone Array (Intel Smart Sound Technology…)" and "Microphone Array 2 ()" are different inputs, so stripping the parenthetical collapsed them onto one string. The rows ellipsise instead.
• Devices are probed by opening them. Windows lists monitor endpoints and analogue inputs as recordable inputs. Offering them leads to a dead end, so --devices actually opens each one and reports usable.
• Library writes are serialised through a queue. Every save is a read → modify → write cycle on one JSON file, and the result and done handlers fire without awaiting. Started together, the second read happens before the first write lands, so the second write-back drops the entry the first one added — finished runs left no history at all, intermittently, which is what made it so hard to see.
• A silent recording is reported, not transcribed. Below −120 dBFS the take is genuinely empty, and running the model on it spends minutes to return an empty note, which looks like the app worked.