feat(storage): precompute waveform peaks on the validator node

[rickyrombo]

What

An opt-in stage in mediorum that decodes audio the node already stores and reduces it to a 750-bucket amplitude envelope — per-bucket RMS, normalized to peak, quantized to uint8, 750 bytes a track. Clients render a waveform from that without downloading or decoding the audio.

The idea comes from a standalone prototype that pulled mp3s over the network from a single node via signed /tracks/cidstream/{cid} requests. Running it on the node removes that egress entirely.

Why mediorum and not the ETL

The ETL is a chain indexer with no blob access, so a detached process would have to fetch audio over the network — the exact cost being removed. Mediorum already owns the blobs, already runs ffmpeg, and already has a worker-pool/backlog/retry pattern to follow.

Why local-only and not replicated

Mediorum rows replicate solely by riding the core chain as MediorumOperation txs — syncCoreMediorumOps walking committed blocks is the only path from one node to another; there is no peer-to-peer gossip. The table allowlist those txs validate against (opvalidation.mediorumTableTypes) is consulted by isValidMediorumOperationTx, which runs in FinalizeBlock, where result codes fold into CometBFT's LastResultsHash.

Registering waveforms as a crudr model would therefore turn a derived rendering hint into consensus-affecting state requiring a coordinated network upgrade, and commit ~1KB per track to the chain permanently.

So each node computes its own. Every input — bucket count, sample rate, frame size — is a compile-time constant rather than config, so two nodes produce byte-identical output and a client re-fetching from a different node after a reroute sees the same waveform. Replication remains available as a purely additive follow-up if it ever earns the cost; the table is CID-keyed either way.

Where waveforms come from

Four paths, none of which pays to read back a blob it just had:

1. Transcode hook. The finished 320 is still on local disk, so this costs no bucket read, and covers every new upload on the transcoding node. It sits after the mirror bookkeeping: it decodes a whole track and runs inline on a path HTTP handlers call synchronously, so it must not delay the node claiming to be a transcoded mirror. Failure is advisory — recorded against the waveforms row, never reaching onError, because a rendering hint must not be able to fail an upload.
2. Replication handoff. pullFileFromHostValidated already buffers a pulled blob to disk to verify its cid, so it hands that copy to a waveform job once the bucket write commits — after commit, so the peer is never told we hold a blob we have not finished storing, and on the worker pool, so the peer is not held open for a decode. Without this a replicated blob waits for a sweep and pays a bucket read when one arrives.
3. Backfill sweep. Walks uploads newest-first behind a keyset cursor, so the recent slice lands first rather than after a full history walk.
4. Retry sweep. Re-attempts failed rows, reading the waveforms table only, so it is indexed and proportional to the result set rather than to catalog size.

Throughput is set by the workers, not by a timer: a sweep refills the queue and stops when it is full, so OPENAUDIO_WAVEFORM_WORKERS is the knob that matters. The retry and linking sweeps run whether or not backfill is enabled — only the history walk is gated — because those are what repair a transient failure from the live hook, and a live-only node would otherwise never retry one.

The walk restarts after an interval rather than finishing permanently. Uploads replicate from peers carrying timestamps the descending walk has already passed, so "the end of history" is only ever true for now.

Linking a row to its upload

Rows carry upload_id, and discovery correlates on it. That makes an unlinked row worse than untidy: it is invisible to discovery, so its upload looks permanently outstanding and every re-walk recomputes a waveform that is already correct — and on a live-only node, where discovery never runs, nothing ever repairs it.

The paths differ in what they know:

• The transcode hook passes upload.ID directly for both the 320 and the preview.
• The pull request carries the upload id the sender already had. The receiver could resolve it locally instead, but only from the uploads row the sender publishes through consensus — which arrives on a block commit, while the pull request arrived over HTTP. Resolving locally is kept as the fallback for peers that predate the field.
• resolveWaveformUploadID covers what is left: the 320 by idx_uploads_transcode_cid_320, a preview through audio_previews to its source 320.
• linkOrphanWaveforms is the backstop, repairing the link in place rather than recomputing, on every sweep regardless of the backfill flag.

Previews are also recorded before their blob is handed to peers. That does not close the window on its own — the row still has to clear consensus — but publishing a blob before anything can account for it is a gap worth not opening.

Serving

GET /waveform/:cid returns {cid, version, buckets, sample_rate, duration_ms, peaks}, peaks base64-encoded because Go marshals []byte that way — divide each byte by 255 for a 0..1 envelope, pair with duration_ms to render.

On a miss it redirects to a peer that has it, mirroring serveBlob. The search cannot reuse hostHasBlob: holding the blob says nothing about holding the waveform, since waveforms are not replicated and a peer may have the feature switched off. Peers are probed for the waveform itself via HEAD. The probe sets localOnly, and a node seeing localOnly answers 404 rather than forwarding — without that a probe would recurse across the network. Bounded to three probes, since this sits inline in a user request.

Serving never triggers analysis. Enqueueing from an unauthenticated GET would bypass the archive-tier guard the sweeps apply, letting anyone trigger a cold-storage retrieval on a StoreAll node with no rate limiting in front of it. Pointing at a node that already holds the answer is a better response than promising to compute one.

Auth is requireHealthy + ensureNotDelisted, deliberately without requireRegisteredSignature. A 750-byte envelope is not decodable audio, and clients draw the waveform before playback begins, so requiring a stream signature would defeat the purpose. Calling that out as an explicit product decision. Delisting still applies — a waveform is derived from the audio and disappears with it.

Previews

A preview is its own blob with its own cid, and its waveform cannot be sliced out of the track's: it is peak-normalized independently, and thirty seconds of a long track occupies too few buckets to stretch across a player. Discovery enumerates the selected preview alongside the 320. Serving needed nothing — the route is cid-addressed, so a preview waveform is servable the moment a row exists.

Placement context is carried per blob rather than per upload. The 320 is replicated with the upload's placement hosts and the preview with none, and bucketForCID reads any non-empty placement as "force primary" — so handing a preview the upload's hosts would judge every preview primary-tier and read it out of cold storage with the archive flag off.

The preview is queued where it is produced, with its file handed over, rather than from the transcode hook afterwards. Nil placement means rank alone decides its tier, and a preview cid ranks independently of the track's and of who created it — so on a StoreAll node most previews route to that node's own archive tier, where a job arriving without a local file is refused by the archive guard. On a 70-peer network at a replication factor of 4 that is roughly sixteen previews in seventeen, which would leave the recommended configuration producing almost no preview waveforms at all. Queuing it at the point of production also covers the bare-CID /generate_preview endpoint, which has no upload row to hook into.

Versioning

Every row is stamped with a version, and discovery counts only rows at the current version as present. A version change therefore makes stale rows look absent and the ordinary walk recomputes them — no separate sweep, no table rewrite. The cursor records the version it walked under, and a mismatch restarts it immediately.

The version is derived rather than hand-maintained: a fingerprint over the algorithm version plus every parameter that affects output. A hand-bumped constant invites changing waveformBuckets and silently leaving a network full of waveforms computed under the old settings.

Two consequences:

• It is a fingerprint, not a sequence — it does not increase, so comparisons are equality, never ordering.
• It is opaque alone, so the console reports algorithm_version, buckets, and sample_rate beside it.

Writes are on conflict (cid) do update, so a recompute replaces the row in place. A CID keeps serving its existing waveform until the new one lands, and a version change never opens a coverage gap.

Console

A section on the Storage page, beside Repair & Cleanup, since a backfill pass is the same shape as a repair run. It renders only when the feature is enabled, matching how Archive Storage appears only once an archive bucket is configured — the feature is off by default and an empty section on every node is noise.

Two run tiles (Analysis, Backfill) over stat tiles. Analysis carries the settings and which of the nested switches are on; Backfill carries the pass, its progress and its counters, with progress estimated from where the cursor sits between the oldest and newest audio upload and labelled rough, since uploads are not spread evenly across that span.

The stat tiles count uploads, not blobs. An upload yields a 320 and sometimes a preview, so a blob count and an upload count are different units and cannot be compared with each other — and mixing them is how a figure ends up in two tiles at once. Counted per upload, each analyzable upload lands in exactly one bucket and the tiles sum to the analyzable catalog: Analyzed, Never Analyzed, Partial, Not Local, Skipped Archive (only where archive storage exists), Unavailable, Failed, To Recompute. Where an upload's blobs disagree — a finished 320 beside a failed preview — the worse state wins, since that is the one an operator would act on.

Unlinked Rows sits alongside them and is shown only when non-zero: the rollup is keyed by upload and structurally cannot see rows that resolved none, so it is the one figure that says the rest is describing a subset.

A final row counts requests served, missed and redirected — nothing else reports whether the waveforms are being consumed, and the redirect count is the closest thing to a measure of how evenly they are spread across the network.

Notes for review

• Why this doesn't extend the existing audio analyzer. analyzeAudio runs inline in the transcode worker on a 1-minute deadline and its errgroup cancels siblings on first error, so a waveform failure added there would kill BPM/key and write an error into the chain-submitted uploads row. Its WAV is also truncated to 120s — right for BPM/key, wrong for a full-track envelope. And its selector filters audio_analysis_status != 'done', which the existing catalog already satisfies, so it could never backfill.
• All the reported figures come from one sampled pass. Sharing a snapshot is what lets them reconcile; a query per tile describes a different moment per tile. The pass is a sample rather than a per-request count because its anti-join half is proportional to the catalog and cannot be indexed away — the preview key is extracted dynamically from jsonb. The console reports the age.
• Expected blob count mirrors waveformTargets exactly, including its requirement that a selected preview actually resolve to a blob. Deriving it from selected_preview alone lets an upload expect a row that can never be written, which leaves it short of its own expected count forever.
• waveforms.upload_id is nullable and not a key. An upload yields two analyzable blobs once previews count, and legacy Qm content has no upload row at all, so the relationship is not 1:1. Legacy content is excluded from linking for that reason — a null there is the right answer, and retrying it would never converge.
• audio_previews.source_c_id, not source_cid. gorm derives the column from AudioPreview.SourceCID and splits the acronym. Getting it wrong does not fail loudly: the query errors, the error is not ErrNoRows, and an empty return reads as "legacy content with no upload" — indistinguishable from success. The lookup is logged at Warn for that reason, and a test resolves a real preview.
• Read failures split three ways. Both-buckets-NotFound → not_local; any other bucket error → unavailable on a short backoff. Collapsing them would let a transient archive outage stamp a 24h backoff across a large slice of the catalog in a single sweep. Neither increments error_count, which is what lets a job migrate to whichever node actually holds the file.
• The archive guard sits at the read, not at enqueue. A cid's tier is not fixed — rendezvous rank shifts when the validator set changes — so a row queued while primary-tier can be read after it has become archive-tier. Checking immediately before the first bucket call makes the flag authoritative however the job arrived. It is skipped whenever the job carries the file itself — the replication handoff, the preview — because the guard exists to prevent cold-storage retrievals and there is no retrieval when the bytes are already on local disk. The 320 hook never reaches the guard for the same reason, reading its own output off disk.
• Temp-file ownership transfers with the job. enqueueWaveformJob reports the handoff definitively — a non-blocking send either placed the job or did not — so a flag marks the transfer and the deferred cleanup still covers a copy error, a cid mismatch, a failed write, and a full queue.
• Duration comes from the decoded sample count, not uploads.ff_probe — that column is probed from the original upload rather than the 320, and is null on older rows.
• Short-read guard. computeWaveform verifies it fed ffmpeg the whole source. A bucket read ending early without erroring would otherwise produce a plausible waveform for a fraction of the track and mark it done.
• Bounded memory. The accumulator merges adjacent frames and doubles the frame size on overflow, so an arbitrarily long track costs a fixed ~768KB. Sums and counts are carried separately and sqrt applied once at the bucket level, since RMS-of-RMS is wrong the moment a partial trailing frame exists.
• ffmpeg is pinned to one thread, matching the transcoder's own cap, since backfill runs alongside transcoding and PCM decode is single-threaded in practice anyway.

Config — all default false

Var	Default	Effect
OPENAUDIO_WAVEFORM_ENABLED	false	Master switch. Off ⇒ no routine, no live hook, no writes.
OPENAUDIO_WAVEFORM_BACKFILL_ENABLED	false	Historical walk. Off ⇒ live and replication paths only; retries and linking still run.
OPENAUDIO_WAVEFORM_ARCHIVE_ENABLED	false	Include archive-tier CIDs in the walk. Off ⇒ archive_skipped.
OPENAUDIO_WAVEFORM_WORKERS	2	Worker count, and the throughput knob.

They nest, so an operator can walk the cost curve one step at a time: nothing, then new uploads, then recent history, then the cold tier. A node that sets nothing behaves exactly as it does today.

Testing

54 tests — 41 across waveform_test.go, waveform_link_test.go and waveform_rollup_test.go, plus 13 covering the console section:

• Algorithm — silence dividing by zero, sub-second clips producing empty buckets (NaN), RMS-of-RMS after a frame merge, energy preserved across merges, and the merge path checked against a naive sample-exact reference implementation.
• Decode — a real ffmpeg decode of a track longer than 120s, a truncated source detected, non-audio rejected.
• Serving — stored peaks returned, HEAD without a body, localOnly refusing to forward, a cross-node 302, and a miss that provably enqueues nothing.
• Versioning and sweeps — the fingerprint changing for each parameter independently, a first run not mistaken for a version change, a stale cursor re-walking and finding a late-arriving upload, a version change re-enqueueing stale rows, the cursor stopping at a full queue rather than skipping past it, and retries running with backfill disabled.
• Linking — the pull request carrying the sender's upload id, a pull with no local uploads row still linking, a preview resolving through audio_previews, orphans repaired in place, and legacy content left alone.
• Analysis targets — an original pulled from a peer replicating without being decoded, and an older peer's 320 still analyzed via the resolve fallback.
• Archive tier — the same cid refused without a local file and analyzed with one, and the preview hook queuing a job whose handed-over file still exists.
• Rollup — each upload counted exactly once as it moves between states, worst-state-wins across disagreeing blobs, an unproducible preview not counted as expected work, uploads with nothing analyzable excluded, and unlinked rows counted apart.
• Ownership — the worker releasing a handed-over file even when analysis fails, and a full queue leaving no orphan.
• Console — hidden when disabled, live-only distinguished from backfilling, run progress, pending re-backfill, every stat tile, the sample age, Skipped Archive appearing only with archive storage, and Unlinked Rows only when non-zero.

Full ./pkg/mediorum/... and ./pkg/core/console/... suites pass on freshly created databases; go build ./..., go vet, and gofmt clean.

Verified end-to-end on a 4-node devnet: the live path produced a waveform immediately after transcode with duration_ms matching a 200s track; a mirror produced byte-identical peaks; nodes with the feature off returned 404 with an empty table; select count(*) from ops where "table"='waveforms' was 0; a delisted CID returned 403.

Also verified against a live archive node running this branch. A 180s synthetic track whose amplitude steps through 1.0 / 0.5 / 0.25 / 0.75 / 0.1 / 0.6 came back as 255 / 127 / 64 / 191 / 25 / 153 — the ±1s being uint8 quantization — with duration_ms 180000, confirming a full-length decode rather than a 120s truncation.

Deferred

Legacy Qm CID discovery, and storing a peak envelope alongside RMS — wavesurfer's default rendering is per-bucket peak, so RMS reads flatter by comparison. The bulk repair path is also left on the sweeps deliberately: it streams straight into the bucket with no local copy, and it walks millions of blobs, so a decode per pull would slow the durability sweep for little gain. The version column makes each a cheap follow-up.

🤖 Generated with Claude Code