Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Voice Support Simulator

A spoken-conversation trainer for bank support calls. You play the customer; the system plays the agent for a lost card, a failed transfer or a locked account. It transcribes what you say, streams a reply, speaks it back, and times every stage of every turn.

The subject is the turn loop and its latency, not the model: capture → transcription → bounded prompt window → language model → speech, with time-to-first-token measured against a budget, plus a post-call review that reads the agent's own words for compliance breaches. It runs with no API key, no microphone and no speaker — each stage resolves to a provider that can actually run where you started it, and the UI says which it picked rather than failing at the first call.

Run it

docker compose up -d --build     # then open http://localhost:8110

No key required: without GROQ_API_KEY the model resolves to the scripted local agent, and the container seeds a real opening exchange through the engine on first load so the page is not empty. docker compose down -v when you are done. Locally instead, on Python 3.12 (.python-version):

python3 -m venv project_venv && source project_venv/bin/activate
pip install -r requirements/base.txt         # pipeline + UI, no audio
streamlit run streamlit_app.py               # http://localhost:8501
python scripts/bench_latency.py --turns 12   # p50/p95/max time-to-first-token

The CLI is the microphone path, so it needs the audio extras:

pip install -r requirements.txt              # + PortAudio, webrtcvad, Kokoro (torch)
python main.py --persona card_lost --turns 3 # full-duplex, real barge-in

Those extras pull in torch, transformers and spaCy — only worth it where there is hardware to drive. On the base install main.py exits with that install hint instead of a traceback. With a microphone the UI starts in voice mode, without one in text mode; VOICE_IO forces either. --persona takes card_lost, transfer_failed or account_locked, and the CLI is the only entry point that writes logs/latency_log.csv.

Below is a seeded call against the offline scripted agent — no network, no audio hardware, and every number on screen measured rather than mocked.

A lost-card call in progress, with per-turn latency

With no credentials the sidebar reports transcription unavailable, the model scripted and synthesis silent, each labelled ready or stand-in in words: three status lines, not three exceptions. Also captured, a fresh call after switching scenario (04-scenario.png), the review panel (03-review.png) and the page at phone width (05-mobile.png).

One turn, and where the milliseconds accrue

sequenceDiagram
    participant C as Caller
    participant S as Transcriber
    participant E as ConversationEngine
    participant W as ConversationState
    participant M as LanguageModel
    participant P as split_sentences
    participant T as Synthesizer

    C->>S: speaks, then WAV bytes
    Note over C,S: VADStream ends the turn after 600 ms<br/>of silence (asr_module.max_silence_ms)
    S-->>E: transcript, recorded as asr_ms
    E->>W: add_turn("user", text)
    W-->>E: system prompt plus bounded history
    Note over W: two bounds, 8 turns and 6000 chars,<br/>so the prompt stops growing mid-call
    E->>M: stream(messages)
    M-->>E: first fragment
    Note over E: first_token_ms starts here — the only<br/>number checked against TURN_BUDGET_MS
    E->>P: fragments, as they arrive
    P-->>T: first complete sentence
    T-->>C: audio starts
    Note over M,T: the model is still writing while<br/>the first sentence is already playing
    M-->>E: remaining fragments, remaining sentences
    E->>W: add_turn("assistant", reply)
Loading

Synthesis starts at the first sentence boundary, not the end of the reply. split_sentences hands complete sentences to the synthesiser while the model is still writing, so the speaker — playing at a fixed rate — becomes the bottleneck instead of the provider, and test_synthesis_starts_before_the_model_has_finished_writing pins it: the second sentence must be produced after the first has been spoken.

The prompt window is bounded by characters as well as turns. ConversationState holds eight turns and 6,000 characters, evicting oldest-first and never dropping the most recent. Eight turns of "yes" is nothing; eight turns of a customer reading out a statement is a prompt re-sent, and re-paid for, on every turn after it — so without the second bound the agent gets slower the longer someone talks to it. The transcript keeps everything; only what is sent is trimmed.

The per-turn latency breakdown

TurnMetrics records asr_ms, first_token_ms, llm_ms, tts_ms and total_ms per turn. The stages overlap, so those are deliberately not additive, and only first_token_ms is compared to TURN_BUDGET_MS (default 1200) — a long answer that starts immediately feels fast, a short one that starts late does not. Going over flags the turn in the UI and counts it in the summary. Nothing raises, since discarding a slow answer is worse than delivering it late, but a regression is visible rather than merely felt.

Where a turn can be when the caller cuts in

stateDiagram-v2
    [*] --> Idle
    Idle --> Recording: start_recording()
    Recording --> Transcribing: stop_recording()
    Transcribing --> Idle: no audio, or no speech detected
    Transcribing --> Thinking: transcript in hand
    Thinking --> Speaking: token stream open, sentences pulled
    Speaking --> Speaking: next sentence, model still writing
    Speaking --> Idle: reply complete
    Thinking --> Interrupted: interrupt(), or a new recording starts
    Speaking --> Interrupted: interrupt(), or a new recording starts
    Interrupted --> Idle: rest of the reply dropped
Loading

SimpleVoiceHandler emits exactly those states as status strings and checks _should_stop_speaking on every sentence, so an interrupt lands at the next boundary rather than at the end of the reply. But Interrupted is reachable mid-turn only in the CLI: VoiceClient runs a VAD monitor during playback and sets the flag from the microphone, whereas a Streamlit rerun is synchronous, so nothing in the browser can call interrupt() while a turn is in flight.

Swapping a stage is an environment variable

Three protocols in src/providers/base.py (Transcriber, LanguageModel, SpeechSynthesizer), three registries in registry.py, six adapters in builtin.py:

Stage Hosted Stand-in
Transcription groq — whisper-large-v3-turbo unavailable — says it cannot hear
Language model groq — gpt-oss-20b scripted — fixed agent dialogue
Speech kokoro — 24 kHz, local silent — a valid WAV, no speaker

Resolution is explicit argument → environment variable → auto; under auto each candidate is asked whether it can run here and the best available priority wins. A provider named explicitly is built even when unavailable, so a misconfiguration surfaces its own error instead of being silently swapped. create_provider_set() degrades per stage, not per session: a stage that will not start records a warning, falls back to its stand-in, and the UI renders the warning. Adding a provider is a class and one decorator:

import os

from src.providers.registry import llm_registry


@llm_registry.register("my-vendor", available=lambda: bool(os.getenv("MY_VENDOR_KEY")), priority=20)
def _my_vendor():
    return MyVendorModel()  # needs .name, .complete(), .stream(), .status()

Then LLM_PROVIDER=my-vendor, or leave it on auto — engine, UI, CLI and benchmark are unchanged, because they only ever saw the protocol. The payoff is that degradation is a state, not an error path. Each stand-in keeps the downstream code on its normal path: the silent synthesiser emits a real WAV precisely so WAV parsing, duration arithmetic and export stay exercised. And the offline model is scripted, not generated — a support trainer whose offline mode invents plausible financial procedure is worse than one admitting it reads from a script, because the trainee cannot tell which half is real.

What the benchmark measures, and what it does not

scripts/bench_latency.py runs a fixed script of customer lines through ConversationEngine. Against the scripted provider, one run in a python:3.12-slim container limited to three CPUs:

provider          scripted
turns             12 in 4.01s
prompt window     820 chars held after the run

                    p50        p95       max
time to first token    12.7ms     15.4ms     15.4ms
whole turn            328.8ms    446.7ms    446.7ms

budget            1200ms — 0/12 turns over

Read that as the loop's own overhead and nothing else. The scripted provider emits word by word behind a deliberate time.sleep(0.012) (src/local_backend.py), so roughly 12 of those 13 milliseconds are the stand-in's own delay; the remainder is prompt assembly, the window and the streaming machinery, and the whole-turn figure is that same 12 ms times the word count of a canned reply. p95 is nearest-rank, shared with the in-app call summary (conversation.percentile). Both move with the machine and the interpreter, so run the command rather than trust the block. Hosted latency was not measured at all, because that needs a key and a paid account — nothing here is a claim about Groq's response times, and no number in this repo should be read as one.

The review is split by what is knowable

src/analysis/call_review.py does two different things and refuses to blur them. The compliance scan is deterministic — five regex rules over the agent's own utterances: full card number, PIN, password, CVV, guaranteed refund. Whether an agent asked for a PIN is a matter of fact, and a regex settles it every time, offline, for free; a model would settle it most of the time, which is the wrong reliability for the check that matters. Each rule needs a request verb, so "you'll set a new password" is not a breach while "tell me your password" is, and test_the_shipped_scripts_break_no_rules holds it to zero hits across every line the shipped scripts can produce. The qualitative grade is judgement, so it goes to the model: asked for JSON, parsed defensively, prose around it tolerated, score clamped, malformed output and exceptions both falling back. With no model reachable it drops to the four keyword criteria in src/feedback.py, labelled source="heuristic" — which the UI shows rather than passing a pattern match off as an assessment. A compliance breach caps the score regardless of the model's view.

Settings

All optional, read from the environment or a .env file. .env.example documents all seventeen; these are the ones that change behaviour.

Variable Default Effect
GROQ_API_KEY empty Real value ⇒ hosted ASR and LLM available. Empty or a placeholder ⇒ offline providers
LLM_PROVIDER / STT_PROVIDER / TTS_PROVIDER auto Force one adapter from the table above, or let auto probe
VOICE_IO auto text skips ASR and TTS, voice forces the audio path, auto looks for a microphone
TURN_BUDGET_MS 1200 Turns slower than this to their first token are flagged. Nothing fails
SEED_DEMO_CALL 0 1 plays a real opening exchange through the engine on first load

The rest tune the hosted models, the Kokoro voice, the say fallback and the CLI cost estimate.

Tests, and the image

pip install -r requirements/dev.txt
python -m pytest tests/     # 299 passed
ruff check . && ruff format --check .

No test touches a microphone, a speaker or the network. tests/conftest.py strips GROQ_API_KEY for every test, so a developer with a real key exported cannot make a live call by accident, and it stands in for groq and webrtcvad when they are not installed, so the suite runs on the base install — in a container, or on a CI runner with no sound card. tests/test_streamlit_render.py drives the real page through Streamlit's AppTest: empty state, seeded call, a typed turn, the review and the status labels.

The Dockerfile is multi-stage and non-root (assistant, uid 1000), on python:3.12-slim, with builder/runtime/test targets and a healthcheck on /_stcore/health via urllib so no curl package is needed. The runtime installs requirements/base.txt only: the audio extras drive hardware a container does not have, so torch, transformers, spaCy, PortAudio, ALSA headers, webrtcvad and the C toolchain are absent — which is why webrtcvad is lazy behind audio_devices.get_webrtcvad(). Measured on this pass: runtime image 851 MB, test image 902 MB, healthcheck healthy and / → 200 as uid 1000 with no key, and the suite inside the test image with --network none → 299 passed:

docker build --target test -t ai-voice-assistant:test . && docker run --rm --network none ai-voice-assistant:test

Host port 8110 only.

What this does not do

  • Text mode is not voice mode. It exercises everything after transcription and says nothing about accents, background noise or interruptions — barge-in is real only in the CLI, per the state diagram.
  • The offline agent is a script — four fixed steps per scenario, then a close. The scenario is chosen by keyword-matching the persona's system prompt, so rewording one past those hints silently selects another script; test_the_persona_prompt_selects_its_script pins the shipped personas.
  • The compliance scan is a fixed rule set, not a regulator's — five breaches, precise but not exhaustive. Feedback scoring grades the model's replies, not a trainee's.
  • The CSV log is CLI-only. Neither Streamlit path writes logs/latency_log.csv; the UI holds its metrics in session state for the page's lifetime and exports them on request.
  • Hosted paths are untested against the real service. The Groq SDK is on 1.x and its call signatures were checked, but no request has been made with a key; the audio extras were bumped without being installed, since that needs real devices.
  • Transitive dependencies float. requirements/ pins the direct dependencies exactly and lets pip resolve the rest — easier to maintain than a full freeze, but not a lockfile.
  • VoiceClient still owns its own turn assembly rather than running on ConversationEngine, though it shares split_sentences and ConversationState with it. That is the one duplication left.

License

MIT — see LICENSE.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages