Repository navigation
Description
Because
app-spoken-lid (https://github.com/clamsproject/app-spoken-lid, experimental, not on appdir yet) runs a language identification model on fixed windows of an audio track, and the windowing is the main thing the app adds on top of its model. We're looking at keeping each backend as a separate app (Whisper, AmberNet, VoxLingua106, and maybe Meta's MMS next), since their dependencies can't share an environment. Without a shared helper, each app needs its own copy of the windowing code.
The current copies have already a problem. https://github.com/clamsproject/app-spoken-lid has main plus ambernet and speechbrain branches, and each one has its own redundant chunk_audio;
main (Whisper) |
ambernet |
speechbrain |
|
|---|---|---|---|
| end of the last window | actual end | actual end | (idx + 1) * chunk, past the end of the audio |
| windows shorter than 1 s | kept | skipped | kept |
| time computation | sample offsets | sample offsets | float seconds, then int(s * 1000) |
We have no audio helper in the SDK at the moment. LID app loads the whole waveform with librosa, or whatever else apps picked (for example, app-tonedetection has its own loading code based on pydub).
proposal
iter_windows(start_ms, end_ms, window_ms, stride_ms=None)intimeunit_helper: yields(start_ms, end_ms)pairs of integers.stride_msdefaults towindow_ms, which gives non-overlapping windows. This is plain arithmetic, likevdh.sample_timepoints, so it can live intimeunit_helperwith no media dependency.iter_audio_windows(document, window_ms, stride_ms=None, sample_rate=16000, ...)in a newmmif.utils.audio_document_helper: decodes the audio, resamples as needed (parameterized), and yields(samples, start_ms, end_ms).
The audio helper is an optional shared utility, not a required processing path for every wrapper app. Some underlying analyzers accept media file paths and provide their own audio decoding, resampling, or chunking; those wrappers can continue using that processing instead of routing audio through iter_audio_windows. Apps that need SDK-controlled decoding, windows, or timeline handling can use the helper.
For iter_audio_windows;
- it should take both
AudioDocumentandVideoDocument, since apps also read the audio track of a video. So it can't be built onvdh.open_container, which rejects non-video documents, readsstreams.video[0], and writes video properties to the document. - it decodes with PyAV, as we did for
vdhin questionable TP alignment of TD annotation #379. This dropslibrosafrom the apps. Decoded samples may not exactly matchlibrosa's, so model scores can shift a little. - it streams, so a long broadcast isn't held in memory as a whole waveform.
- window times come from the audio stream's own timeline. A stream can start at a non-zero offset (same family of problem as questionable TP alignment of TD annotation #379), and LID
TimeFrames need to line up with annotations from the video. - the minimum length of the last window is a parameter, so every backend handles the short tail the same way.
Other things to decide;
- ADH and VDH will share PyAV as a dependency, so the current dependency check and extras should be updated to support using PyAV without requiring the full
cvextra. - a name for the file-level failure cases (no audio stream, truncated file). I think these should raise
RuntimeErrorand warn the same wayvdhdoes.
I'm not proposing an image version of iter_windows for video. iter_windows itself will work for video, but no app needs fixed-size image windows right now (SWT samples timepoints, and captioners work on input TimeFrames). A ClamsLIDApp base class also seems too much, since apps only share the generator and a few lines of output code.
Done when
- decide on names of the helper functions
iter_windows,iter_strided_windows, ...iter_audio_windows,iter_strided_audio_windows,running_resampler, ...)
-
iter_windowsis added, with integer ms in and out -
iter_audio_windowsworks both onAudioDocumentandVideoDocument(audio track) - window times are correct for a stream with a non-zero start offset
- audio apps can install PyAV without
opencv-pythonopencv-pythonis not a PyAV dependency; it remains inmmif-python[cv]only for deprecated VDH APIs that still usecv2, pending their removal in 2.0 (per issue questionable TP alignment of TD annotation #379).
- a missing audio stream and a truncated file both give an actionable error
-
app-spoken-lidreimplemented to use the helper, and its output is compared with thelibrosaversion on real audio (@Brendayy )
Additional context
No response
Activity