Skip to content

add audio_document_helper for windowed audio reads #416

Open

Description

@keighrim

Because

app-spoken-lid (https://github.com/clamsproject/app-spoken-lid, experimental, not on appdir yet) runs a language identification model on fixed windows of an audio track, and the windowing is the main thing the app adds on top of its model. We're looking at keeping each backend as a separate app (Whisper, AmberNet, VoxLingua106, and maybe Meta's MMS next), since their dependencies can't share an environment. Without a shared helper, each app needs its own copy of the windowing code.

The current copies have already a problem. https://github.com/clamsproject/app-spoken-lid has main plus ambernet and speechbrain branches, and each one has its own redundant chunk_audio;

main (Whisper) ambernet speechbrain
end of the last window actual end actual end (idx + 1) * chunk, past the end of the audio
windows shorter than 1 s kept skipped kept
time computation sample offsets sample offsets float seconds, then int(s * 1000)

We have no audio helper in the SDK at the moment. LID app loads the whole waveform with librosa, or whatever else apps picked (for example, app-tonedetection has its own loading code based on pydub).

proposal

  1. iter_windows(start_ms, end_ms, window_ms, stride_ms=None) in timeunit_helper: yields (start_ms, end_ms) pairs of integers. stride_ms defaults to window_ms, which gives non-overlapping windows. This is plain arithmetic, like vdh.sample_timepoints, so it can live in timeunit_helper with no media dependency.
  2. iter_audio_windows(document, window_ms, stride_ms=None, sample_rate=16000, ...) in a new mmif.utils.audio_document_helper: decodes the audio, resamples as needed (parameterized), and yields (samples, start_ms, end_ms).

The audio helper is an optional shared utility, not a required processing path for every wrapper app. Some underlying analyzers accept media file paths and provide their own audio decoding, resampling, or chunking; those wrappers can continue using that processing instead of routing audio through iter_audio_windows. Apps that need SDK-controlled decoding, windows, or timeline handling can use the helper.

For iter_audio_windows;

  • it should take both AudioDocument and VideoDocument, since apps also read the audio track of a video. So it can't be built on vdh.open_container, which rejects non-video documents, reads streams.video[0], and writes video properties to the document.
  • it decodes with PyAV, as we did for vdh in questionable TP alignment of TD annotation #379. This drops librosa from the apps. Decoded samples may not exactly match librosa's, so model scores can shift a little.
  • it streams, so a long broadcast isn't held in memory as a whole waveform.
  • window times come from the audio stream's own timeline. A stream can start at a non-zero offset (same family of problem as questionable TP alignment of TD annotation #379), and LID TimeFrames need to line up with annotations from the video.
  • the minimum length of the last window is a parameter, so every backend handles the short tail the same way.

Other things to decide;

  • ADH and VDH will share PyAV as a dependency, so the current dependency check and extras should be updated to support using PyAV without requiring the full cv extra.
  • a name for the file-level failure cases (no audio stream, truncated file). I think these should raise RuntimeError and warn the same way vdh does.

I'm not proposing an image version of iter_windows for video. iter_windows itself will work for video, but no app needs fixed-size image windows right now (SWT samples timepoints, and captioners work on input TimeFrames). A ClamsLIDApp base class also seems too much, since apps only share the generator and a few lines of output code.

Done when

  • decide on names of the helper functions
    • iter_windows, iter_strided_windows, ...
    • iter_audio_windows, iter_strided_audio_windows, running_resampler, ...)
  • iter_windows is added, with integer ms in and out
  • iter_audio_windows works both on AudioDocument and VideoDocument (audio track)
  • window times are correct for a stream with a non-zero start offset
  • audio apps can install PyAV without opencv-python
  • a missing audio stream and a truncated file both give an actionable error
  • app-spoken-lid reimplemented to use the helper, and its output is compared with the librosa version on real audio (@Brendayy )

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

  • Status
    Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions