BigBlueButton STT Agent for LiveKit

This application provides Speech-to-Text (STT) for BigBlueButton meetings using LiveKit as their audio bridge.

Supported STT engines:

• Gladia — via the official LiveKit Gladia plugin (default)
• OpenAI — via a direct REST client against /v1/audio/transcriptions; supports the official OpenAI API and any OpenAI-compatible endpoint
• Voxtral Realtime — Mistral Voxtral Mini Realtime served by a self-hosted vLLM instance via its realtime WebSocket API

Getting Started

Environment prerequisites

• Python 3.11+
• A LiveKit instance
• A Gladia API key, an OpenAI API key, or a vLLM server hosting Voxtral (depending on your chosen STT provider)
• uv:
  • See installation instructions: https://docs.astral.sh/uv/getting-started/installation/

Installing

1. Clone the repository:

   git clone git@github.com:bigbluebutton/bbb-livekit-stt.git
   cd bbb-livekit-stt

2. Install the dependencies:

   uv sync

3. Configure environment variables:

   Copy the example .env file:

   cp .env.example .env

   Now, edit the .env file and fill at least the following environment vars:

   LIVEKIT_URL=...
   LIVEKIT_API_KEY=...
   LIVEKIT_API_SECRET=...
   
   # For Gladia (default provider):
   GLADIA_API_KEY=...
   
   # For OpenAI (set STT_PROVIDER=openai):
   # STT_PROVIDER=openai
   # OPENAI_API_KEY=...
   

   Feel free to check .env.example for any other configurations of interest.

   All options ingested by the Gladia STT plugin are exposed via env vars, except its energy filter (see .env.example). The OpenAI provider takes OPENAI_API_KEY, OPENAI_STT_MODEL and OPENAI_BASE_URL (see below).

Running

The agent is run using the command-line interface provided by the livekit-agents library. The necessary environment variables will be picked up automatically.

Once started, the worker will connect to your LiveKit server and wait to be assigned to rooms. By default, the LiveKit server will dispatch a job to the worker for every new room created. The agent will then join the room, start listening to audio tracks, and generate transcription events when required.

Development

For development, use the dev command.

uv run python3 main.py dev

Production

For production, use the start command.

uv run python3 main.py start

Docker

Build the image:

docker build . -t bbb-livekit-stt

Run:

docker run --network host --rm -it --env-file .env bbb-livekit-stt

The container's log level is controlled by the LOG_LEVEL env var (TRACE, DEBUG, INFO, WARN, ERROR or CRITICAL; defaults to INFO). To run the agent with debug logging:

docker run --network host --rm -it --env-file .env -e LOG_LEVEL=DEBUG bbb-livekit-stt

The image's default command names every process in the container bbb-livekit-stt, job processes included; a command that runs python3 directly does not. process-exporter can group them by that name:

process_names:
  - name: bbb-livekit-stt
    comm:
      - bbb-livekit-stt

Pre-built images are available via GitHub Container Registry as well.

Metrics

The agent exposes Prometheus metrics through the LiveKit worker's built-in endpoint. Metrics are opt-in: set BBB_STT_PROMETHEUS_PORT to enable them.

BBB_STT_PROMETHEUS_PORT=8082
PROMETHEUS_MULTIPROC_DIR=/tmp/prometheus-multiproc

Scrape http://<host>:8082/metrics.

PROMETHEUS_MULTIPROC_DIR is required. LiveKit runs each room job in its own process, and every bbb_stt_* metric reaches the endpoint only through this shared directory. Set it in the environment or in .env; the Docker image sets it whenever the port is set.

With Docker:

docker run --network host --rm -it --env-file .env \
  -e BBB_STT_PROMETHEUS_PORT=8082 bbb-livekit-stt

The endpoint also serves LiveKit's own lk_agents_* metrics (active jobs, worker load, child process count).

Metric	Type	Labels	Meaning
bbb_stt_active_sessions	Gauge	provider, language	Active transcription sessions, one per subscribed microphone track. language is the locale's ISO 639-1 code, auto, or other
bbb_stt_active_agents	Gauge	provider	Running agents, one per room job
bbb_stt_session_starts_total	Counter	provider	Sessions successfully started
bbb_stt_session_start_failures_total	Counter	provider, reason	Sessions that could not be established
bbb_stt_session_ends_total	Counter	provider, reason	Sessions that ended: stopped, provider_closed, audio_ended or error
bbb_stt_agent_start_failures_total	Counter	provider, reason	Agents that could not be established
bbb_stt_transcripts_total	Counter	provider, event_type	Transcripts that passed filtering and were sent to BBB
bbb_stt_transcripts_discarded_total	Counter	provider, event_type, reason	Transcripts dropped before publication
bbb_stt_transcript_confidence	Histogram	provider, event_type	Confidence of published transcripts (Gladia only)
bbb_stt_utterance_duration_seconds	Histogram	provider, event_type	Duration of transcribed utterances
bbb_stt_provider_audio_seconds_total	Counter	provider	Audio seconds sent to the provider, as counted by the Gladia plugin (Gladia only)
bbb_stt_language_detection_mismatch_total	Counter	provider	Transcripts whose language differs from the requested one
bbb_stt_transcript_publish_failures_total	Counter	provider	Transcripts lost because publishing to Redis failed

Caveats worth knowing:

• No metric carries a meeting or user label, to keep cardinality bounded. The logs carry per-meeting detail.
• bbb_stt_session_start_failures_total{reason="no_audio_track"} is usually benign. BBB routinely sends the speech-locale change before the participant publishes a microphone track, and the session starts moments later. Alert on its rate relative to bbb_stt_session_starts_total, not on the raw count.
• bbb_stt_transcripts_total counts transcripts that were sent, not confirmed. A transcript is counted even when the Redis publish fails, so the confidence histogram has no gaps during a Redis outage. Subtract bbb_stt_transcript_publish_failures_total for what BBB received.
• bbb_stt_provider_audio_seconds_total undercounts. The Gladia plugin reports audio every 5 s or so. Audio sent after the last report can be lost when a session ends: under 5 s normally, more after a slow Gladia connection.
• A killed job keeps counting for up to a minute. The kernel kills a job for memory, and LiveKit for missed pings or an overrun shutdown. The worker drops the gauges of exited processes every 60 s. The sessions such a job was running get no bbb_stt_session_ends_total sample.
• The multiprocess directory grows with every job. Each job leaves a counter and a histogram file there until the worker restarts, so scrapes slow down over a long-running worker's life.
• A locale change restarts the participant's session. It appears as a stop and a start: bbb_stt_session_starts_total advances and the session gauge moves between language labels.
• A provider can reject a session after accepting it. Gladia checks the language only once its stream runs. A locale it cannot honour starts a session that ends with bbb_stt_session_ends_total{reason="error"}.
• Voxtral gives up on a server that keeps refusing sessions. A refusal is a WebSocket that opens, then closes or sends another event than session.created. A refusal 10 minutes after an earlier one, with no session started since, ends the session with reason="error".
• Voxtral and OpenAI ride out outages. Voxtral retries forever when a connection fails to open, as on a 502. It does the same when the WebSocket opens but no event arrives within 10 s, or when a started session drops. OpenAI logs a failed segment and moves on.

OpenAI STT provider

Set STT_PROVIDER=openai to use OpenAI STT instead of Gladia.

Official OpenAI API:

STT_PROVIDER=openai
OPENAI_API_KEY=your-key
# OPENAI_STT_MODEL=gpt-4o-transcribe  # default; use "whisper-1" for classic Whisper

OpenAI-compatible endpoint (e.g. a self-hosted Whisper server):

STT_PROVIDER=openai
OPENAI_API_KEY=any-value
OPENAI_BASE_URL=http://your-server:8000
OPENAI_STT_MODEL=your-model-name

Some caveats apply to this provider:

• It is not a streaming provider. Audio is segmented locally by silence detection and each segment is posted to /v1/audio/transcriptions, so transcripts arrive once an utterance ends rather than continuously.
• It does not support real-time translation. Only the original transcript language is returned, matching the user's BBB speech locale.
• The auto speech locale is not usable with it. The agent requests response_format=json, whose payload carries only the transcribed text, so there is no detected language to map back to a BBB locale and the transcript is discarded with a warning. Set an explicit locale in BBB when using OpenAI STT.

Voxtral Realtime STT provider

Set STT_PROVIDER=voxtral-realtime to use a self-hosted Voxtral Mini Realtime model served by vLLM. The agent streams each participant's audio over vLLM's realtime WebSocket API, gated by a local Silero VAD, and emits live interim captions from the model's incremental deltas.

STT_PROVIDER=voxtral-realtime
VOXTRAL_BASE_URL=https://your-vllm-server:8000/v1   # required
VOXTRAL_API_KEY=your-key                            # if your server enforces one
# VOXTRAL_MODEL=mistralai/Voxtral-Mini-4B-Realtime-2602  # default

VAD and segmentation tuning options (silence duration, pre-roll, split overlap, max segment length) are documented in .env.example.

Some caveats apply to this provider:

• It does not support real-time translation. Only the original transcript language is returned, matching the user's BBB speech locale.
• The auto speech locale is not usable with it, and the participant's chosen locale never reaches the model either. vLLM's realtime API carries no language in either direction: session.update accepts only model, and transcription.delta / transcription.done report only text. The limitation is in the model's streaming prompt format, not just the API — a target language is encoded as a lang:<code> prefix only in the offline transcription format, never in the streaming one. Voxtral Realtime therefore always detects the language itself, transcripts are labelled with the locale the participant selected, and under auto there is no language at all, so every transcript — interim and final — is discarded with a warning. Set an explicit locale in BBB when using Voxtral Realtime STT.

Development

Testing

Run the unit tests:

uv run pytest tests/ --ignore=tests/integration

Run with coverage:

uv run pytest tests/ --ignore=tests/integration --cov --cov-report=term-missing

Integration tests require a real API key and make live requests to the STT service.

For Gladia, set GLADIA_API_KEY and run:

GLADIA_API_KEY=your-key uv run pytest tests/integration -m integration

For OpenAI, set OPENAI_API_KEY and run:

OPENAI_API_KEY=your-key uv run pytest tests/integration -m integration

Linting

This project uses ruff for linting and formatting. To check for issues:

uv run ruff check .

To automatically fix fixable issues:

uv run ruff check --fix .

To format the code:

uv run ruff format .