Add clean and highlighted-subtitle video export

README.md

@@ -16,7 +16,8 @@ heard again, it resumes. The prompt overlay is never burned into the recording.
 - Pause prompt movement when speech leaves the script without pausing recording.
 - Choose current-word, current-line, or highlight-free prompt scrolling.
 - Place the prompt eye-line in the upper, middle, or lower part of the preview.
-- Review and save every finished take as an MP4 with H.264 video and AAC audio.
+- Review and export either a clean take or an H.264/AAC MP4 with a polished
+  single-line lower third that highlights each spoken word in a chosen font.
 - Install the app as a PWA or packaged Microsoft Store desktop app on Windows.
 
 ## Run locally
@@ -77,7 +78,8 @@ to exercise the same loopback WebSocket used by Studio.
 The complete MVP is targeted at the packaged Windows desktop app and current
 Edge or Chrome on Windows. The app records native MP4 when available; otherwise
 the local API converts the browser's WebM take to MP4 with its bundled FFmpeg
-binary. Speech-following streams PCM only to the PrompterPro API at `127.0.0.1`,
+binary. Optional highlighted subtitles are rendered by that same local binary.
+Speech-following streams PCM only to the PrompterPro API at `127.0.0.1`,
 where sherpa-onnx performs recognition. If the local model is not installed,
 recording still works and the prompt can be moved with the on-screen controls
 or arrow keys.
@@ -97,8 +99,8 @@ Set `SHERPA_ONNX_MODEL_DIR` to use the same model from another location, or
 
 Scripts are stored locally in browser storage. Camera and microphone media stay
 on this computer. Microphone samples and transcripts used for prompt following
-travel only over the loopback interface to the local sherpa-onnx engine. A WebM
-take may pass through the same local API for MP4 conversion and is removed from
-its temporary folder after export. Only the optional AI generation form is sent
-to OpenAI; camera, microphone, recordings, transcripts, and the script library
-are not.
+travel only over the loopback interface to the local sherpa-onnx engine. A take
+may pass through the same local API for MP4 conversion or subtitle rendering and
+is removed from its temporary folder after export. Only the optional AI
+generation form is sent to OpenAI; camera, microphone, recordings, transcripts,
+and the script library are not.

docs/ARCHITECTURE.md

@@ -27,6 +27,13 @@ flowchart LR
   PCM -->|"Loopback WebSocket"| STT["Local sherpa-onnx"]
   STT --> ALIGN["Bounded fuzzy alignment"]
   ALIGN --> OVERLAY["Prompt overlay"]
+  ALIGN --> WORDS["Timed spoken words"]
+  WORDS --> ASS["Local ASS lower third"]
+  FILE --> STYLE{"Subtitle export?"}
+  STYLE -->|"No"| DOWNLOAD["Clean download"]
+  STYLE -->|"Yes"| ASS
+  ASS --> RENDER["Local FFmpeg subtitle render"]
+  RENDER --> DOWNLOAD
   UI --> API["Local Node API"]
   API --> OAI["OpenAI Responses API"]
   OAI --> API
@@ -50,10 +57,14 @@ original `MediaStream`, not a canvas composition, so the overlay is not recorded
   stable application data directory so random loopback ports do not split it
   across desktop sessions.
 - `src/components/Studio.tsx`: media device and recorder state machine.
+- `src/lib/videoExport.ts`: spoken-word timing backfill, lower-third paging,
+  preview selection, and the local binary export envelope.
 - `src/components/TeleprompterOverlay.tsx`: prompt rendering, eye-line, and
   current-word scrolling.
 - `server/index.ts`: input validation, local MP4 conversion, and server-side
   OpenAI call.
+- `server/subtitleExport.ts`: strict caption-envelope parsing and deterministic
+  ASS generation with active-word styling.
 - `scripts/build-server.mjs`: bundles the local API's JavaScript dependencies
   while leaving only FFmpeg and the native speech loader external.
 - `desktop/main.mjs`: hardened Electron window, media permissions and lifecycle
@@ -75,6 +86,13 @@ available, it records a WebM/Opus take and posts it only to the loopback API.
 The API converts it to H.264/AAC MP4 with the bundled FFmpeg executable, returns
 the file for review/save, and removes its temporary working directory.
 
+Speech-confirmed cursor advances are timestamped during recording. Review keeps
+the clean recording as the source of truth and can preview those words as a
+single lower-third line. A subtitle export sends the clean take, selected font,
+and validated word timings only to the loopback API. The API builds a temporary
+ASS track, re-encodes the video with libass so the active word is enlarged and
+accented, returns an H.264/AAC MP4, and removes all temporary inputs.
+
 ## Alignment approach
 
 The browser uses an `AudioWorklet` to copy mono PCM from the recording's
@@ -151,6 +169,7 @@ before promising equivalent background behavior or codec support.
 - Recognition speed and quality depend on the Windows computer's CPU and
   microphone.
 - Long recordings consume memory until stopped and downloaded.
-- MP4 fallback conversion briefly uses additional local disk, CPU, and memory.
+- MP4 fallback conversion and optional subtitle rendering briefly use additional
+  local disk, CPU, and memory.
 - Scripts do not yet sync across devices or users.
 - There is no account, cloud library, or server-side recording store.

docs/PRODUCT.md

@@ -19,7 +19,9 @@ fixed scroll speed.
 7. If speech does not match nearby script words, prompt movement pauses and the
    recording continues.
 8. Resume from nearby script text and prompt movement catches up.
-9. Stop, review, and save the finished MP4.
+9. Stop and review the finished take.
+10. Save a clean video, or choose a font and export a subtitled MP4 whose
+    lower-third line highlights the word currently being spoken.
 
 ## Information architecture
 
@@ -43,7 +45,7 @@ recording and lets Studio devote most of the screen to eye-line and readability.
 | Recording / following | Green status and moving current phrase | Stop, move prompt manually |
 | Recording / off script | Amber status; prompt stays put | Keep speaking or return to nearby text |
 | Processing | Recording finalization | Wait briefly |
-| Review | MP4 playback, save, new take | Save MP4 or record again |
+| Review | Playback plus clean/subtitled style and font controls | Export video or record again |
 | Error | Specific recovery message | Retry devices or use manual prompt |
 
 ## Voice-following behavior
@@ -66,8 +68,8 @@ recording and lets Studio devote most of the screen to eye-line and readability.
 - Large controls with visible keyboard focus.
 - `Space` starts/stops recording outside editable fields.
 - Arrow keys move the prompt in Studio.
-- Prompt font size, vertical eye-line, mirror mode, and word/line highlighting
-  are adjustable.
+- Prompt font size, vertical eye-line, mirror mode, word/line highlighting, and
+  exported subtitle font are adjustable.
 - Reduced-motion preferences disable smooth scrolling and decorative animation.
 - The presenter must explicitly grant camera and microphone permissions.
 
@@ -86,6 +88,8 @@ recording and lets Studio devote most of the screen to eye-line and readability.
 
 - A user can complete the full journey without a developer tool.
 - A saved MP4 contains synchronized H.264 video and AAC microphone audio.
+- Subtitle export shows one lower-third line and highlights the active spoken
+  word without adding the teleprompter overlay to the clean source.
 - Prompt movement responds to matching speech and stops for unmatched speech.
 - Recording duration continues to increase while prompt status is off-script.
 - Script CRUD persists after a refresh.

server/index.ts

@@ -1,4 +1,5 @@
 import { execFile } from "node:child_process";
+import { existsSync } from "node:fs";
 import { mkdtemp, rm, writeFile } from "node:fs/promises";
 import { createServer } from "node:http";
 import { createRequire } from "node:module";
@@ -13,6 +14,10 @@ import {
   attachLocalSpeechServer,
   getSpeechModelStatus,
 } from "./localSpeech.js";
+import {
+  buildAssSubtitles,
+  parseCaptionExportBody,
+} from "./subtitleExport.js";
 
 const currentFile = fileURLToPath(import.meta.url);
 const serverDirectory = path.dirname(currentFile);
@@ -27,7 +32,27 @@ const production =
   process.env.PROMPTER_PRODUCTION === "1";
 const execFileAsync = promisify(execFile);
 const require = createRequire(import.meta.url);
-const ffmpegPath = require("ffmpeg-static") as string | null;
+const importedFfmpegPath = require("ffmpeg-static") as string | null;
+
+function resolveFfmpegPath(): string | null {
+  const candidates = [
+    importedFfmpegPath,
+    importedFfmpegPath?.replace("app.asar", "app.asar.unpacked"),
+    path.join(rootDirectory, "node_modules", "ffmpeg-static", "ffmpeg.exe"),
+  ];
+  return (
+    candidates.find((candidate) => candidate && existsSync(candidate)) ?? null
+  );
+}
+
+function assVideoFilter(): string {
+  // The bundled Windows FFmpeg build uses fontconfig's system-font provider.
+  // Keeping the ASS path relative also avoids drive-letter escaping being
+  // misread by libass as another filter option.
+  return "ass=captions.ass";
+}
+
+const ffmpegPath = resolveFfmpegPath();
 
 app.disable("x-powered-by");
 
@@ -116,6 +141,116 @@ app.post(
   },
 );
 
+app.post(
+  "/api/recordings/subtitles",
+  express.raw({
+    type: "application/x-prompter-export",
+    limit: "1gb",
+  }),
+  async (request, response) => {
+    if (!ffmpegPath) {
+      response.status(503).json({
+        error: "Subtitle export is not available on this computer.",
+      });
+      return;
+    }
+    if (!Buffer.isBuffer(request.body) || request.body.length === 0) {
+      response.status(400).json({ error: "No recorded video was supplied." });
+      return;
+    }
+
+    let parsedExport;
+    try {
+      parsedExport = parseCaptionExportBody(request.body);
+    } catch (error) {
+      response.status(400).json({
+        error:
+          error instanceof Error
+            ? error.message
+            : "The subtitle export options are invalid.",
+      });
+      return;
+    }
+
+    const workingDirectory = await mkdtemp(
+      path.join(tmpdir(), "prompter-subtitle-export-"),
+    );
+    const inputPath = path.join(workingDirectory, "take.recording");
+    const subtitlePath = path.join(workingDirectory, "captions.ass");
+    const outputPath = path.join(workingDirectory, "take-subtitled.mp4");
+
+    try {
+      await Promise.all([
+        writeFile(inputPath, parsedExport.recording),
+        writeFile(
+          subtitlePath,
+          buildAssSubtitles(
+            parsedExport.request.words,
+            parsedExport.request.fontFamily,
+          ),
+          "utf8",
+        ),
+      ]);
+      await execFileAsync(
+        ffmpegPath,
+        [
+          "-hide_banner",
+          "-loglevel",
+          "error",
+          "-y",
+          "-i",
+          inputPath,
+          "-map",
+          "0:v:0",
+          "-map",
+          "0:a:0",
+          "-vf",
+          assVideoFilter(),
+          "-c:v",
+          "libx264",
+          "-preset",
+          "veryfast",
+          "-crf",
+          "20",
+          "-pix_fmt",
+          "yuv420p",
+          "-c:a",
+          "aac",
+          "-b:a",
+          "192k",
+          "-movflags",
+          "+faststart",
+          outputPath,
+        ],
+        {
+          cwd: workingDirectory,
+          maxBuffer: 16 * 1024 * 1024,
+          windowsHide: true,
+        },
+      );
+
+      response.type("video/mp4");
+      response.setHeader("Cache-Control", "no-store");
+      response.sendFile(outputPath, (sendError) => {
+        void rm(workingDirectory, { recursive: true, force: true });
+        if (sendError && !response.headersSent) {
+          response.status(500).json({
+            error: "The subtitled MP4 could not be returned.",
+          });
+        }
+      });
+    } catch (error) {
+      await rm(workingDirectory, { recursive: true, force: true });
+      const message =
+        error instanceof Error ? error.message : "Unknown subtitle export error";
+      console.error("Subtitle export failed:", message);
+      response.status(500).json({
+        error: "Subtitle export failed. Your clean recording is still safe.",
+      });
+    }
+  },
+);
+
 app.use(express.json({ limit: "32kb" }));
 
 interface GenerateBody {