imumd — IMU Movement Discovery

Unsupervised discovery of movement types from body-worn 6-axis IMU data (3-axis accelerometer + 3-axis gyroscope), and real-time detection of the discovered gestures from a live sensor stream.

Given unlabeled IMU recordings, imumd blindly clusters recurring movements into a vocabulary, surfaces one-off movements as salient, and — once a vocabulary is learned — tags those gestures live and forwards them as OSC (e.g. to SuperCollider) for sound or visuals. The target application is dance, captured with a Movesense sensor.

New here? The User Manual is the task-oriented guide — concepts, the two vocabulary modes (blind vs. labeled), sensor topologies, end-to-end workflows, and a full CLI/API reference. This README is the quickstart; the full research log (every design decision + real-data results) is in docs/RESEARCH.md.

Install

Python ≥ 3.10. Install from source into a virtual environment:

python3 -m venv .venv
.venv/bin/pip install -e ".[neural,osc]"

Optional extras (install only what you need):

Extra	Pulls in	For
neural	torch	the autoencoder representation (the live default)
osc	python-osc	the live OSC receiver/forwarder
umap	umap-learn	UMAP reduce-then-cluster
datasets	pooch	fetching public activity datasets (PAMAP2)
dev	pytest	running the test suite

The classical (non-neural) core needs none of these — just the base numpy/scipy/scikit-learn dependencies. Everything torch-related is imported lazily, so the classical path runs without torch installed.

Quickstart

1. Blind discovery on a recording

from imumd.movesense import load_movesense_wav
from imumd.pipeline import discover

rec = load_movesense_wav("MovesenseDataMix.wav", target_fs=50.0)  # 6-ch WAV -> Recording @ 50 Hz
d = discover(rec, min_cluster_size=10)
print(f"{d.clusters.n_clusters} movement types; "
      f"{int((d.clusters.labels == -1).sum())} salient (out-of-vocabulary) windows")

discover() chains: sliding windows → features → standardize → cluster (HDBSCAN) → salience. Swap the representation with feature_fn= (e.g. a trained autoencoder encoder); everything else is unchanged.

2. Real-time gesture detection (Python API)

from imumd.realtime import fit_vocabulary, save_vocabulary, load_vocabulary, RealtimeDetector

# fit a vocabulary offline (pool several takes of the same gestures)
vocab = fit_vocabulary([take1, take2, take3], representation="autoencoder",
                       min_cluster_size=15, threshold_scale=1.5)
save_vocabulary(vocab, "gestures.vocab")     # persist it...
vocab = load_vocabulary("gestures.vocab")    # ...reload instantly later, no retraining

det = RealtimeDetector(vocab, hop_seconds=0.5, smooth=3)
for sample in stream:                        # sample: a (6,) row [ax, ay, az, gx, gy, gz]
    for d in det.push(sample):               # emits a Detection every hop
        print("novel" if d.is_novel else f"gesture {d.label}", f"conf={d.confidence:.2f}")

fit_vocabulary freezes discovery into a Vocabulary (scaler + per-gesture centroids + novelty thresholds). RealtimeDetector streams a rolling 1 s window and returns the nearest gesture, or None (novel) when nothing matches.

3. Live from a Movesense → Holonist OSC stream

examples/live_movesense.py wraps the detector for a live Holonist feed and forwards each detection out as OSC /gesture <label> <confidence> (label -1 = novel). It defaults to the hardware-validated autoencoder at min_cluster_size=15 (catches all 7 gestures in the reference set; see §7.8).

# First launch: pool your takes, fit the AE@15 vocabulary, and CACHE it.
.venv/bin/python examples/live_movesense.py \
    --vocab-wav take1.wav take2.wav take3.wav take4.wav take5.wav \
    --vocab-cache gestures.vocab --sensor 1 --listen-port 8000 --out-port 57120

# Every launch after: no --vocab-wav, no retraining — the cache loads in ~milliseconds.
.venv/bin/python examples/live_movesense.py \
    --vocab-cache gestures.vocab --sensor 1 --listen-port 8000 --out-port 57120

Point Holonist at --listen-port; forward /gesture to --out-port (57120 = SuperCollider's default). Delete the cache file to refit (e.g. after adding a new take). For the deterministic, torch-free path, add --representation classical (recovers 6 of the 7 gestures; min_cluster_size then defaults to 10).

Multiple sensors are detected independently and tagged in the output (/gesture [sensor, label, conf]):

# broadcast one vocabulary to sensors 1–3
.venv/bin/python examples/live_movesense.py --vocab-cache g.vocab --sensors 1 2 3 --listen-port 8000

Per-sensor vocabularies — when sensors sit on different body parts with different movement repertoires, give each its own vocabulary. First fit and cache one vocabulary per body part from that part's own recordings (pool a couple of takes per part — a single short take is often too sparse to cluster at mcs=15):

from imumd.movesense import load_movesense_wav
from imumd.realtime import fit_vocabulary, save_vocabulary

parts = {
    "wrist": ["wrist_01.wav", "wrist_02.wav"],
    "ankle": ["ankle_01.wav", "ankle_02.wav"],
}
for name, wavs in parts.items():
    recs = [load_movesense_wav(p, target_fs=50.0) for p in wavs]
    vocab = fit_vocabulary(recs, representation="autoencoder",
                           min_cluster_size=15, threshold_scale=1.5)
    save_vocabulary(vocab, f"{name}.vocab")

Then map each cache to its sensor number (--sensor-vocab is repeatable and loads the caches instantly — no refitting):

.venv/bin/python examples/live_movesense.py \
    --sensor-vocab 1=wrist.vocab --sensor-vocab 2=ankle.vocab --listen-port 8000

Sensor 1 is then detected against the wrist vocabulary and sensor 2 against the ankle vocabulary, each tagged in the /gesture [sensor, label, conf] output.

Whole-body (fused) detection — instead of tagging each sensor independently, fuse all sensors into one whole-body gesture stream. Fit a single vocabulary from an interleaved 6N-channel WAV (N = sensor count) via load_multisensor_wav(path, n_sensors=2) + fit_fused_vocabulary(ms, n_sensors=2), or from a stand-in stack_sensors([rec1, rec2, ...]) of aligned single-sensor takes:

from imumd.movesense import load_multisensor_wav
from imumd.fused import fit_fused_vocabulary, save_vocabulary

# Fit from an interleaved 6N-channel WAV
ms = load_multisensor_wav("session.wav", n_sensors=2, target_fs=50.0)
vocab = fit_fused_vocabulary(ms, n_sensors=2, min_cluster_size=15, threshold_scale=1.5)
save_vocabulary(vocab, "body.vocab")

Then run live over N sensors with one /gesture/body [label, conf] output (no sensor arg — the gesture is inherently whole-body):

.venv/bin/python examples/live_movesense.py \
    --fused --sensors 1 2 --fused-vocab-wav session.wav --listen-port 8000 --out-port 57120

The --fused-vocab-wav is an interleaved WAV; --sensors specifies which Holonist sensors to concatenate (N is len(sensors), any 2–6). Optionally cache the fitted vocabulary with --vocab-cache body.vocab to skip refitting on next launch. Validated on stand-in multi-sensor data; hardware pending.

--fused fits with reduce-then-cluster by default: features are PCA-reduced to --reduce-dim dims (16 for --fused; unset/off for non-fused runs) before clustering, and the same PCA is applied live to each window (scaler → reducer → nearest-centroid). Pass --reduce-dim explicitly to override either default (any positive int, or non-fused runs to opt in). On real 2-sensor whole-body data, reduce-then-cluster meaningfully improved separability of subtle whole-body gestures over the un-reduced fit — see docs/RESEARCH.md §7.11.

Labeled (supervised) gestures — when a performer records one file per named gesture, skip blind clustering entirely and build a vocabulary straight from those labels: --labeled NAME=PATH (repeatable) fits one centroid per named gesture instead of discovering clusters. It composes with --fused/--sensors (one interleaved 6N-channel WAV per gesture, fused into a whole-body centroid) and --reduce-dim, and detections print the gesture NAME instead of a bare integer:

.venv/bin/python examples/live_movesense.py \
    --labeled wave=wave.wav --labeled clap=clap.wav --sensor 1 --listen-port 8000

# labeled + fused: each named recording is an interleaved 6N-channel WAV
.venv/bin/python examples/live_movesense.py \
    --labeled wave=wave_body.wav --labeled clap=clap_body.wav \
    --fused --sensors 1 2 --listen-port 8000

Because the labels carry information blind HDBSCAN discards, labeled mode distinguishes gestures the blind pipeline merges — on real 2-sensor whole-body data it separated two gestures (gA, gC) that §7.11's blind reduce-then-cluster fit left confused. See docs/RESEARCH.md §7.12.

Representations

Representation	Default mcs	Notes
classical	10	Engineered features. Deterministic, torch-free, instant.
autoencoder	15	1D-CNN embedding. The live default — hardware-validated to catch all 7 reference gestures; needs the neural extra and a few seconds' one-time training (cache it).

Which representation wins is task-dependent — the full comparison, 3-seed stability check, and min_cluster_size sweep are documented in docs/RESEARCH.md §7.8.

Data format

• Movesense WAV — a single 6-channel WAV in canonical order [ax, ay, az, gx, gy, gz], or six per-axis WAVs. load_movesense_wav / load_movesense_session repair dropouts and anti-alias downsample to 50 Hz.
• Holonist OSC — per-axis messages /m/{n}/acc/{x,y,z} and /m/{n}/gyro/{x,y,z} (n = sensor 1–6, one float each).

Tests

.venv/bin/pytest -q      # 126 passed

Torch-dependent tests self-skip when torch is not installed.