Repository navigation
Conversation
ADR-0015 (proposed) records the voice design from the discussion in #132: the cloud only hears and speaks, one resident Agent session thinks and answers in the Character's voice and persona. CONTEXT.md gains voice command, resident session and spoken reply, and the Adapter becomes two-way. ADR-0007 notes that K0 long would become push-to-talk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
longzhi
marked this pull request as draft
October 12, 2026 15:07
5 tasks
…-assistant-7f44bc
Claude Code waits on Anthropic's terms for subscription login; ACP and Channels are recorded as rejected. Records the 2026-10-13 probe of a resident thread driven over stdio with a client-defined speak tool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No firmware of ours has used the box's microphone. The I2C0 scans found no ES7210 at 0x40-0x43, so the mic is most likely on the ES8311's own ADC, which our ES8311 sequence (esp_codec_dev's WORK_MODE_BOTH) already powers, with samples on GPIO47 as in xiaozhi 1.9.4's atk-dnesp32s3-box config. - core: device.mic.record (cable only) reports chip IDs at 0x18 and 0x40-0x43, sets the ES8311 mic gain (register 0x16, as es8311_set_mic_gain), records, then sends per-100 ms RMS/peak levels for both slots and the samples as base64 lines, one per poll, with a CRC. - device: I2S0's receiver shares the transmitter's BCLK/WS (signal loopback) and reads GPIO47 into PSRAM only while a capture runs; 24 kHz 16-bit stereo, since the ES8311 has one LRCK for ADC and DAC. - tools/mic-record.py writes a WAV and says whether a slot is live. - docs/firmware-bringup.md: facts table with sources, how to run and read it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vibebuddyd can now keep one long-running Codex thread for voice commands (ADR-0015), behind VIBEBUDDY_RESIDENT_SESSION=1. It spawns the user's own codex app-server lazily, checks the existing login, starts the thread once (read-only, approvals on request, speak and list_activities as dynamic tools) and resumes it on every later start. Approvals and questions are declined; turns run one at a time and later ones queue. The resident thread's own Codex hooks are dropped by thread id or working directory, so it never becomes an Activity. A new vibebuddy CLI sends a typed voice command with 'vibebuddy say' and prints the thread's codex:// link with 'vibebuddy resident'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first mic build boot-looped on a stack guard panic while drawing the look at boot. esp-hal's main stack is the internal DRAM left between the end of .bss and the reclaimed heap; the static 24 KB RX ring and the task's 2.4 KB local buffer took it from 47,784 to 21,032 bytes. The ring (12 KB, 4 descriptors) is now allocated from internal heap only while a capture runs and freed after, and the task copies straight from the ring into the PSRAM capture, so its pool is 184 bytes. The stack is 47,576 bytes, 208 below main. A lesson records how to check this. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The resident session's codex app-server now runs with the user's plugins, apps, MCP servers and Codex's own tool features (shell, exec, sub-agents, web search, browser, computer use and more) switched off through -c overrides. MCP servers are turned off one by one, named by 'codex mcp list --json', since an override can't empty mcp_servers. The user's AGENTS.md and hooks stay. A thread that can't be resumed is replaced, and the log now names both the old and the new thread id. The docs gain a privacy note on list_activities and say the box should speak a reply as soon as speak arrives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…voice-assistant-7f44bc
On the box the first capture ended in MIC ERROR STALLED with nothing to say why. The 12 KB ring of four descriptors had to be relinked by the mic task before the DMA reached its end, and the task only runs when the main loop yields; once the DMA hits the end of the chain it stops for good. The DMA now writes the whole recording into one PSRAM buffer (esp-hal's DmaRxBuf handles the cache), so nothing depends on the main loop. Its descriptors (~1 KB for 5 s) come from the internal heap for the capture. Each capture also prints MIC DEBUG lines at 100 ms and at the end: descriptors filled, I2S0 and GDMA channel 1 registers, and how often GPIO47/21/13 changed level, so the next flash says where it stops. Main stack: 47,472 bytes (main: 47,784). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the box all 71 descriptors filled and the capture still came back MIC ERROR SHORT. The I2S receiver signals an EOF every 4095 samples (esp-hal leaves RXEOF_NUM at its 12-bit maximum, 8190 bytes), and each EOF closes the descriptor it lands in, so about every third 4080-byte descriptor holds 30 bytes and the buffer held about two thirds of the samples. GDMA IN_INT_RAW 0x13 (suc_eof set) fits this. The buffer is now 1.5x the capture plus two descriptors, and the capture ends once the handed-back descriptors hold enough bytes. MIC DEBUG adds the byte count and the first six descriptor lengths, so the next flash shows whether the 4080 4080 30 pattern is the cause. The BCLK/WS level counts are gone: those pins are outputs with the input path off, so the input register reads 0 however fast they run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the box the descriptors came back as 4080, 14, 4080, 14: the I2S EOF cuts one every 4094 bytes, so a capture needs about twice its size in descriptors. 2.5 times leaves room. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- docs/hardware.md: box 30:ED:A0:A4:0D:08 recorded 5 s on 2026-10-13. The mic is the ES8311's ADC, samples on GPIO47, left slot; no ES7210; about -31 dBFS for a close voice at 30 dB gain. - device mic.rs and docs: RXEOF_NUM (0xFFE from esp-hal) counts bytes on the ESP32-S3, as ESP-IDF uses it, hence descriptors of 4080, 14, ...; the buffer stays at 2.5x, since lining EOFs up would need a raw register write racing esp-hal's that hasn't been tried on a box. - tools/mic-record.py judges slots from the samples with DC removed and the first 100 ms dropped, needs a window above -70 dBFS to call one LIVE, and bases the overall verdict on the left slot. - LESSONS.md: count received data by descriptor lengths, and make a bring-up diagnostic report its own state from the first build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-assistant-7f44bc # Conflicts: # daemon/src/main.rs
main took 0015 (Noise) and 0016 (resumable transfers) meanwhile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e-agent-a94da637ae9c47c0b # Conflicts: # firmware-rs/core/src/lib.rs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The computer side of docs/protocol.md "Voice commands (experimental)": - VOICE START/DATA/END/CANCEL lines are parsed and each DATA line is decoded on its own from the IMA ADPCM state it carries, with a decoder checked nibble for nibble against firmware-core's. Lost lines become silence and are logged; DATA lines stay out of the log. - voice.available goes out on every build report (after hello on each new link, and after a reboot) when the resident session is on and a speech recognizer is configured; never over the Relay. - Speech recognition sits behind a Recognizer trait with two backends: the user's own OpenAI-compatible endpoint, which wins, and Cloudflare Workers AI's Whisper for development. Settings come from the environment, speech.json in the config directory (kept out of config.json so an older App's PUT can't wipe the key), or a dev env file. The language is detected unless set. - On VOICE END: WAV -> recognizer -> voice.heard -> resident session -> voice.reply the moment speak is called -> voice.done, cut to fit the line; errors not_heard, unavailable and busy, one turn at a time. Resident::say_streaming hands replies over as they come; say and POST /v1/resident/say still collect them. An ignored test feeds a real recording through the real recognizer and resident session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…voice-assistant-7f44bc
While the computer offers voice (voice.available), a long K0 records until it is let go, up to 30 s, and streams the audio as VOICE START/DATA/END/CANCEL lines (docs/protocol.md). Without voice, K0 long still gives up the phase. - core/voice_command.rs: the turn (24 kHz IMA ADPCM, 1152 samples a line, each line carrying the coder state it starts from; under 0.5 s is cancelled) and the overlay: LISTENING with a level meter and seconds left, THINKING, then what was heard and the replies, wrapped; errors; closes 15 s after voice.done, after a quiet minute, or on K2. Needs-input, failed and a phase ending close it and cancel a recording, as with the menu. - adpcm.rs: an Encoder matching the decoder, with round-trip tests. - buttons.rs: K0LongEnd, K0 let go after a long press. - device/mic.rs: streaming through a circular DMA ring in PSRAM (128 descriptors, owner checks off, so it never stops), drained by the mic task every 10 ms into a heap queue; MIC STREAM reports rate, drops and backlog. Nothing new in static RAM: main stack 42,128 bytes vs 42,656 on main. - VOICE DATA goes over native USB and the local network only (Board:: write_stream): the UART's 11.5 KB/s can't carry 16 KB/s and blocks. - Microphone gain 42 dB while recording a voice command. - Comma and apostrophe glyphs; the font still has no CJK. - voice_preview example and docs images; docs updated (protocol, device menu, pomodoro, firmware bring-up, ADR-0007 amendment). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…voice-assistant-7f44bc # Conflicts: # docs/protocol.md
The box sends no VOICE DATA over the CH343 bridge, so voice.available is no longer sent there; the log says why voice stayed off (bridge or Relay). A turn that starts while another is still being answered gets voice.error busy right after VOICE START, as docs/protocol.md now allows, so the box stops recording; the rest of that turn is dropped. The voice tests now make their VOICE lines with firmware-core's own VoiceCommand, so they check the daemon against what the box writes: 1152-sample lines, an even END count that includes lost lines, and gaps filled to keep later audio in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each spoken reply is synthesized by the voice service
(POST {base}/v0/voice/speak, staging outside releases,
VIBEBUDDY_TTS_BASE_URL to set or switch off) behind a Synthesizer
trait, and its MP3 is forwarded to the box as it arrives, in pieces of
at most 768 base64 characters, then an end; a failed or cut-short
stream ends early with "error":"synthesis". Replies are spoken in order
on their own task so the text never waits for audio, and voice.done
follows the last reply's end. Replies over 1000 characters are split at
sentences. A dev-only table gives each Character an Edge voice, with a
Chinese voice for Chinese text.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The screen only had the 5x7 ASCII font, so voice transcripts, spoken replies and Chinese session titles showed as placeholders. font.rs adds 3,678 glyphs from Fusion Pixel Font 12 px (OFL 1.1): the 3,500 characters of level one of the 2013 Table of General Standard Chinese Characters, CJK punctuation and the full-width forms. Ark Pixel alone lacks over a hundred level-one characters at 12 px; Fusion fills them from Cubic 11. tools/make-cjk-font.py builds the table from pinned downloads (the font and Unihan's kTGH field): a sorted u16 index and one packed 12x12 bitmap per glyph, 73,562 bytes, included as read-only data so it stays in flash. The main stack is unchanged at 42,128 bytes. Canvas::draw_str draws UTF-8 with ASCII in the old font and Chinese centred on the same line, cuts at whole characters, and draws a dotted box for anything the table lacks. font::wrap breaks between Chinese characters, keeps ASCII words whole and hangs closing punctuation. Task card titles, projects, the title line and the voice overlay use it; text with Chinese goes to small type in the overlay. English screens are unchanged: compare-display.sh finds all 826 frames identical, and the voice and menu previews match byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…voice-assistant-7f44bc
voice.audio chunks for the turn the box recorded are base64-decoded and
handed to a ReplyPlayer (core spoken_reply.rs), which the audio task pulls
from whenever no announcement is queued: same I2S stream, same volume.
Each reply is buffered on the heap and decoded frame by frame with
nanomp3-core; the 22 KB decoder is built zeroed on the heap when a reply
starts and freed when nothing is left. A reply starts with 300 ms
buffered, goes quiet (not stutter) when it runs dry, and replies play in
order; a gap or out-of-order chunk drops that reply. An end with
"error":"synthesis" plays what came and logs VOICE AUDIO CUT SHORT.
Announcements and the Pomodoro chime win: the reply stops and the rest of
its turn is ignored. So do mute, a new K0 push-to-talk, K2 and the
overlay closing; the overlay stays up while a reply is still speaking.
nanomp3-core is vendored with l3_reorder marked #[inline(never)]: inlined,
LSR leaves an address the esp Xtensa backend can't select ("Cannot
select: i32 = Constant<4608>") at opt-level s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…voice-assistant-7f44bc
The daemon queued a whole reply's audio at once and the link's 64-frame queue dropped most of it, so the box stopped mid-sentence: wait for room instead. THINKING now gains dots and counts seconds, since recognition and the resident session take ten seconds or more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the box a reply stopped after its first word: one voice.audio line arrived as invalid JSON and the gap dropped the rest of the reply. UART and USB fed one line parser, so a stray byte on the bridge could split a line coming over USB; UART lines are now held until whole. A missing chunk is skipped instead of ending the reply: the MP3 decoder finds the next frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Codex App takes a thread's writer lock when the user opens it, and the daemon then started a second resident thread. Replace the thread only when its rollout is gone; otherwise fail the command and try again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ChatGPT App's own app-server takes the writer lock of every thread under ~/.codex it lists and keeps it, so `thread/resume` of the resident thread failed with "already has an active writer" while the App ran. The app-server now runs with CODEX_HOME=<state dir>/resident-codex (0700), which nothing else lists: - Login: resident-codex/auth.json is a symlink to the user's auth.json, re-made on every start. Codex 0.162.0 reads it with a plain open, reloads before refreshing, and saves by truncating in place, so one file stays shared. cli_auth_credentials_store="file" is pinned; a keychain login can't be shared (its entry is keyed by the Codex home's path). - Memories, read-only: features.memories stays off, since it also starts the memory pipeline, which rewrites the memories folder of the home it runs in. The daemon puts the user's memory_summary.md in the developer instructions and offers a read-only search_memories tool over MEMORY.md, raw_memories.md and rollout_summaries/. - The old thread is not found in the new home, so one new thread starts (logged), as before for a lost rollout. - `vibebuddy resident [n]` prints the last n exchanges, read with thread/turns/list (GET /v1/resident/history), instead of a codex:// link the Codex App can no longer open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The resident session runs on gpt-6-luna at low reasoning effort, starts with the daemon instead of on the first command, and speech synthesis reuses one HTTPS agent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Voice commands (#132): the design, and the resident session behind them, usable from the CLI before the box can hear. Draft until accounts and the microphone are settled.
Docs
codex app-server, which answers with a spoken reply in the picked Character's voice and persona. Records the rejected options (dictation, a session per command, the top task card, a cloud LLM brain, Muse's cloud, Claude Code Channels, ACP, recognition on the computer, a separate Agent voice).Resident session (experimental, off unless
VIBEBUDDY_RESIDENT_SESSION=1)daemon/src/app_server.rs,daemon/src/resident.rs: JSON-RPC client forcodex app-serverover stdio; one thread for good (thread/startonce,thread/resumeafter, a new thread with a warning if resume fails); read-only sandbox, every approval declined; client-defined toolsspeakandlist_activities; the user's MCP servers, plugins and built-in tools turned off with-coverrides.POST /v1/resident/say,GET /v1/resident, and a newvibebuddyCLI:vibebuddy say "<text>",vibebuddy resident(printscodex://threads/<id>).docs/resident-session.md, including a privacy note:list_activitiessends task card titles and project names to the user's Codex model provider.Tested:
cargo test --workspaceand clippy pass; end to end on a Mac with codex-cli 0.162.0 against an isolated daemon (memory kept across an app-server restart, file writes refused, MCP tools unavailable).🤖 Generated with Claude Code