Skip to content

Add stateful Gemini Live copilot - #44

Draft
0xtigerclaw wants to merge 3 commits into
TechyCSR:mainfrom
0xtigerclaw:agent/gemini-live-copilot
Draft

Add stateful Gemini Live copilot#44
0xtigerclaw wants to merge 3 commits into
TechyCSR:mainfrom
0xtigerclaw:agent/gemini-live-copilot

Conversation

@0xtigerclaw

Copy link
Copy Markdown

Summary

This PR adds a stateful Gemini Live copilot path for real-time screen and audio assistance, plus an optional browser-native response rail that users explicitly activate for the current tab.

What changed

  • adds Vertex AI Gemini Live audio sessions with input and output transcription
  • captures macOS system audio through the bundled helper and streams normalized PCM to Gemini Live
  • keeps an active-task anchor across changing screen frames and short conversational detours
  • adds continuous live-screen sampling with duplicate-frame suppression and intent-aware responses
  • adds a localhost-only browser bridge with strict extension-origin, channel, and command allowlists
  • adds an activeTab Chrome extension rail without persistent access to browsing history or every website
  • makes operational Electron overlays non-activating on macOS to avoid stealing focus from the active application
  • updates onboarding and settings for Google Cloud ADC and removes the shipped legacy Whisper runtime/dependencies
  • improves response readability, streaming status, window controls, and exit behavior

Why

The existing one-shot screenshot and speech workflows lose context when the screen or conversation changes quickly. They also introduce enough latency that the result is difficult to use during an interactive practice session or an AI-permitted call. The new flow maintains a bounded active-task state and streams results as the model processes current audio and screen context.

User impact

Users configure a Vertex AI project with GOOGLE_CLOUD_PROJECT and authenticate locally using:

gcloud auth application-default login

Live Screen and Call Copilot remain explicit user actions. The browser rail is injected only when the extension toolbar button is clicked for the current tab.

Security and privacy

  • no API keys, OAuth tokens, personal project IDs, or signing identities are committed
  • the browser bridge binds only to 127.0.0.1
  • webpage origins are rejected; only Chrome extension origins can connect
  • bridge events and commands are allowlisted
  • the extension uses activeTab instead of broad persistent site access
  • call recording and AI assistance should only be used with consent and where permitted

Validation

  • JavaScript syntax checks across changed runtime, UI, and extension files
  • shell and manifest validation
  • credential and machine-specific identifier scan
  • browser bridge integration test: extension origin accepted and webpage origin rejected
  • packaged ASAR inspection: no Whisper runtime/dependencies or local coaching materials
  • successful macOS production builds for x64 and arm64, including DMG and ZIP targets
  • conflict-free merge-tree check against current upstream main

Current limitations

  • Gemini Live system-audio capture currently targets macOS
  • local builds can be signed when a developer identity is present, but this PR does not add notarization credentials
  • the browser rail must be activated again after navigating to a new page

@vercel

vercel Bot commented Aug 10, 2026

Copy link
Copy Markdown

Someone is attempting to deploy a commit to the csrsoftwares' projects Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant