diff --git a/README.md b/README.md index 9822569..6b3e2cd 100644 --- a/README.md +++ b/README.md @@ -89,6 +89,21 @@ npx github:microsoft/ResearchStudio For usage, see [Idea](ResearchStudio-Idea/#usage) and [Reel](ResearchStudio-Reel/#usage). We recommend using these skills with model versions $\ge$ `claude-opus-4.6` or `gpt-5.5`. +The PowerPoint renderer is independently maintained at +[`ai-nuts/pptx2video`](https://github.com/ai-nuts/pptx2video). Install its +skill and Python 3.11+ CLI runtime separately from ResearchStudio: + +```bash +npx skills add ai-nuts/pptx2video --skill pptx2video +python -m pip install \ + 'pptx2video[svg] @ git+https://github.com/ai-nuts/pptx2video.git@v0.5.0' +python -m playwright install chromium +pptx2video doctor --svg +``` + +Then invoke `/pptx2video` in a supporting agent host, or call the public +`pptx2video render` CLI directly. + ## License [MIT](LICENSE) diff --git a/ResearchStudio-Reel/skills/paper2reel/SKILL.md b/ResearchStudio-Reel/skills/paper2reel/SKILL.md index 849fbd1..b08b749 100644 --- a/ResearchStudio-Reel/skills/paper2reel/SKILL.md +++ b/ResearchStudio-Reel/skills/paper2reel/SKILL.md @@ -228,7 +228,7 @@ double-render text when the user turns on CC. `timeline.json` must already contain the explicit section mapping. If the deck uses slide ids instead of canonical poster ids, build the timeline with -`paper2video/scripts/build_timeline.py --section-map ...` first. +`python -m pptx2video.build_timeline --section-map ...` first. ## Required Hard Gate diff --git a/ResearchStudio-Reel/skills/paper2video/README.md b/ResearchStudio-Reel/skills/paper2video/README.md index 07aef7f..659bb43 100644 --- a/ResearchStudio-Reel/skills/paper2video/README.md +++ b/ResearchStudio-Reel/skills/paper2video/README.md @@ -22,11 +22,12 @@ The production route should use the shared `paper2assets` package whenever it ex ## Output -Written back into the same v2 bundle root, next to `manifest.json` and `assets/`: +Route A writes back into the same v2 bundle root, next to `manifest.json` and +`assets/`: | File | What it is | |---|---| -| `video.mp4` | Final H.264/AAC video with burned-in subtitles and a translucent caption background | +| `video.mp4` | Final H.264/AAC video with burned-in subtitles in an appended black bottom band | | `video_no_subtitles.mp4` | Required raw playback copy for `paper2reel`, so the reel CC toggle does not double-subtitle the video | | `video.pptx` | Editable deck used to render the video | | `assets/audio/*.{mp3,json}` | Per-section narration, script JSON, word timings, and TTS manifests | @@ -36,9 +37,33 @@ Written back into the same v2 bundle root, next to `manifest.json` and `assets/` | `assets/meta/` | Timeline, duration reports, visual cues, anchor contracts, and QA reports | The top level holds only deliverables plus `manifest.json`; everything else lives under `assets/`. +Route B delegates an existing or edited PPTX to the standalone CLI and must use +a fresh output path that does not already exist. ## Usage +Install the two external skills in order, then install and verify the public +`pptx2video` CLI runtime. The runtime requires Python 3.11 or newer: + +```bash +npx skills add hugohe3/ppt-master --skill ppt-master +npx skills add ai-nuts/pptx2video --skill pptx2video +python -m pip install \ + 'pptx2video[svg] @ git+https://github.com/ai-nuts/pptx2video.git@v0.5.0' +python -m playwright install chromium +pptx2video --version +pptx2video doctor --svg +``` + +For Route B, keep that destination separate from any existing Route A or +paper2assets bundle: + +```bash +PPTX2VIDEO_OUT=/absolute/path/to/new-video-bundle +test ! -e "$PPTX2VIDEO_OUT" +pptx2video render "$PPTX2VIDEO_OUT" --resolution 1080p +``` + From a Claude Code session: ```text @@ -47,6 +72,9 @@ From a Claude Code session: # or start from a raw PDF; the skill resolves the same bundle root first > /paper2video ./my_paper.pdf + +# render an existing or edited deck through the installed standalone skill +> /pptx2video ./edited.pptx ``` The final package is not complete until `video.mp4`, `video_no_subtitles.mp4`, `video.pptx`, `assets/meta/timeline.json`, and the video QA report are all present. @@ -58,8 +86,9 @@ The final package is not complete until `video.mp4`, `video_no_subtitles.mp4`, ` 3. **Delegate deck generation to ppt-master** — the skill must run the full ppt-master workflow, not a hand-written shortcut deck. 4. **Generate audio** with the shared `paper2poster/scripts/generate_audio.py` synthesizer, preserving one MP3 per script section. 5. **Build visual cue contracts** so important slide regions are anchored to narration chunks. -6. **Render and subtitle** with `render_video.py` and `add_subtitles.py`; the public MP4 may use burned, soft, bottom-bar, or disabled captions while the raw compatibility render remains available to downstream Paper2Reel. -7. **Build timeline metadata** so `paper2reel` can map poster sections, slide thumbnails, subtitles, and video seek times. +6. **Delegate rendering and subtitles** to `python -m pptx2video.render_video` and `python -m pptx2video.add_subtitles`; the public MP4 may use burned, soft, bottom-bar, or disabled captions while the raw compatibility render remains available to downstream Paper2Reel. +7. **Delegate timeline metadata** to `python -m pptx2video.build_timeline` so `paper2reel` can map poster sections, slide thumbnails, subtitles, and video seek times. +8. **Delegate strict media QA** to `python -m pptx2video.check_video_package`. ## Visual attention cues @@ -71,30 +100,36 @@ Visual cues are generated from the deck, the script, word-boundary timings, and Duration is controlled before audio is synthesized. `assets_to_script.py` / `notes_to_script.py` estimate section lengths against the target, `plan_tts_rate.py` checks measured MP4 duration, and large mismatches must be fixed by rewriting the narration rather than by clipping audio or truncating the final video. -## Scripts +## Paper2Video-owned scripts ``` scripts/ ├── assets_to_script.py # paper2assets narration -> video script + duration plan ├── notes_to_script.py # ppt-master notes -> video script -├── generate_edge_audio.py # Edge TTS helper with word timings ├── generate_cue_requirements.py # script -> visual anchor contract for ppt-master ├── generate_visual_cues.py # deck/script/timings -> positioned highlight cues -├── inject_pptx_anchors.py # preserve cue anchors inside editable PPTX -├── render_video.py # slides + audio + cues -> raw no-subtitle MP4 -├── add_subtitles.py # raw MP4 + captions -> final subtitled MP4 -├── build_timeline.py # section timeline for paper2reel -└── check_video_package.py # strict package and media QA gate +└── inject_pptx_anchors.py # preserve cue anchors inside editable PPTX ``` +The generic runtime is not owned or vendored by this skill. Install +`pptx2video`, then invoke `python -m pptx2video.generate_edge_audio`, +`python -m pptx2video.render_video`, +`python -m pptx2video.add_subtitles`, +`python -m pptx2video.build_timeline`, and +`python -m pptx2video.check_video_package`. + ## Requirements -- Python >= 3.10 -- ppt-master skill for the deck and speaker notes +- Python >= 3.11 +- Installed `ppt-master` and `pptx2video` skills +- Compatible `pptx2video` 0.5.x CLI runtime from `ai-nuts/pptx2video`, installed + with the `svg` extra +- Playwright Chromium installed with `python -m playwright install chromium` +- A passing `pptx2video doctor --svg` check - LibreOffice, Poppler, FFmpeg / FFprobe -- Playwright + Chromium for SVG slide-frame rendering +- Playwright + Chrome/Chromium for SVG slide-frame rendering - Edge TTS by default; Azure TTS is optional ## More detail -[`SKILL.md`](SKILL.md) is the authoritative, agent-facing spec: the full v2 output contract, both supported routes, duration-control loop, visual-cue contract, subtitle requirements, and strict QA gates. The [`references/`](references/) folder documents render details, script JSON, and visual cue semantics. +[`SKILL.md`](SKILL.md) is the authoritative, agent-facing spec: the full v2 output contract, both supported routes, duration-control loop, visual-cue contract, subtitle requirements, and strict QA gates. The [`references/`](references/) folder documents the narration schema, Paper2Video cue generation, and the minimal standalone-runtime bridge. diff --git a/ResearchStudio-Reel/skills/paper2video/SKILL.md b/ResearchStudio-Reel/skills/paper2video/SKILL.md index 0305ab5..36334c4 100644 --- a/ResearchStudio-Reel/skills/paper2video/SKILL.md +++ b/ResearchStudio-Reel/skills/paper2video/SKILL.md @@ -5,10 +5,10 @@ description: > narrated MP4 video. Prefer the shared paper2assets package when present so paper2poster, paper2blog, paper2slides, and paper2video use the same section order and narration. Preserve the advanced deck route by delegating slide - authoring to the external `hugohe3/ppt-master` project, then synthesize audio - with `skills/paper2poster/scripts/generate_audio.py`, render with - `skills/paper2video/scripts/render_video.py`, and burn final subtitles with - `skills/paper2video/scripts/add_subtitles.py`. + authoring to the installed `ppt-master` skill, then synthesize audio + with `skills/paper2poster/scripts/generate_audio.py`, then delegate generic + rendering, subtitles, timeline assembly, and strict media QA to the installed + `pptx2video` package. --- # paper2video - paper/assets/deck -> narrated MP4 @@ -23,9 +23,9 @@ paper.pdf -> assets/meta/sections.json + assets/meta/narration.json -> deck source (ppt-master / paper2slides / existing PPTX) -> assets/audio/*.mp3 from skills/paper2poster/scripts/generate_audio.py - -> raw MP4 from skills/paper2video/scripts/render_video.py - -> timeline.json from skills/paper2video/scripts/build_timeline.py - -> video.mp4 with burned-in subtitles from add_subtitles.py + -> raw MP4 from python -m pptx2video.render_video + -> timeline.json from python -m pptx2video.build_timeline + -> video.mp4 from python -m pptx2video.add_subtitles ``` The important branch-level contract is this: when a `paper2assets` package is @@ -35,7 +35,7 @@ paper2blog, paper2slides, and paper2video aligned. ## Paths -Run commands from the AutoResearch repo root unless noted. +Run Paper2Video workflow commands from `ResearchStudio-Reel/` unless noted. ```bash PAPER2POSTER=skills/paper2poster @@ -43,17 +43,34 @@ PAPER2VIDEO=skills/paper2video PAPER2ASSETS=skills/paper2assets ``` -`ppt-master` is an external dependency, not a path that every agent has: +Install both external skills before starting Paper2Video. Do not clone or +vendor either dependency inside ResearchStudio: ```bash -git clone https://github.com/hugohe3/ppt-master /path/to/ppt-master -PPT_MASTER_DIR=/path/to/ppt-master +npx skills add hugohe3/ppt-master --skill ppt-master +npx skills add ai-nuts/pptx2video --skill pptx2video ``` -Do not hard-code `~/.claude/skills/...`. Users may run this repo from Codex, -Claude Code, a shell, or another agent. +Install and verify the compatible public `pptx2video` 0.5.x CLI runtime before +either route. Use a Python 3.11 or newer environment: -## Output Contract +```bash +python -m pip install \ + 'pptx2video[svg] @ git+https://github.com/ai-nuts/pptx2video.git@v0.5.0' +python -m playwright install chromium +pptx2video --version +pptx2video doctor --svg +``` + +Do not hard-code a host-specific skill directory. Users may run this repository +from Codex, Claude Code, a shell, or another agent. Existing or edited decks can +also invoke the installed skill directly with `/pptx2video`. + +Paper2Video owns paper-specific orchestration and visual-cue preparation. The +installed package owns the generic audio timing, video composition, subtitle, +timeline, and media-QA modules used below. + +## Route A Output Contract Follow the shared paper2assets v2 layout. The paper2video bundle top level holds only deliverable files plus `manifest.json`; all audio, captions, slide decks, @@ -61,7 +78,7 @@ clips, rendered frames, reports, and timeline/cue metadata live under `assets/`: ```text / - video.mp4 # required, burned-in subtitles with translucent caption box + video.mp4 # required, burned-in subtitles in an appended black bottom band video_no_subtitles.mp4 # required, raw/pre-subtitle playback copy for paper2reel video.pptx # required, for follow-up editing manifest.json @@ -73,7 +90,8 @@ clips, rendered frames, reports, and timeline/cue metadata live under `assets/`: meta/ # duration reports, timeline, visual cues, QA reports ``` -Initialize it before running the route: +Initialize it before running Route A. Route B must skip this scaffolding and +pass a destination that does not yet exist to the standalone CLI: **Pick `` (resolve BEFORE any file writes).** The bundle directory is shared across every paper2* skill — when paper2assets, paper2poster, paper2blog, and paper2video target the same root, the video's slides/audio/clips sit next to the poster's HTML, the blog's `.docx`, and the shared narration script in one self-contained package. Resolve deterministically: @@ -104,7 +122,8 @@ VIDEO_META=$VIDEO_ASSETS/meta mkdir -p "$VIDEO_AUDIO" "$VIDEO_CAPTIONS" "$VIDEO_SLIDES" "$VIDEO_CLIPS" "$VIDEO_META/reports" ``` -The MP4 produced directly by `render_video.py` is the raw no-subtitle render. +The MP4 produced directly by `python -m pptx2video.render_video` is the raw +no-subtitle render. Keep an audit copy under `$VIDEO_CLIPS/video_raw.mp4`, and also copy it to `$VIDEO_OUT/video_no_subtitles.mp4` as a required deliverable. The default playback deliverable with burned-in subtitles is `$VIDEO_OUT/video.mp4`. @@ -190,7 +209,7 @@ python skills/paper2video/scripts/plan_tts_rate.py \ --target-minutes 3 \ --out "$VIDEO_AUDIO/tts_rate_plan.json" -python skills/paper2video/scripts/generate_edge_audio.py \ +python -m pptx2video.generate_edge_audio \ "$VIDEO_AUDIO/script.json" \ --outdir "$VIDEO_AUDIO" \ --rate-plan "$VIDEO_AUDIO/tts_rate_plan.json" \ @@ -220,8 +239,8 @@ default high-quality route is `ppt-master`. skill workflow. - Do not replace ppt-master with handwritten SVG, a local simplified generator, or a copied example deck whose content does not come from the paper. -- Before running ppt-master, read the external - `$PPT_MASTER_DIR/skills/ppt-master/SKILL.md` and follow its gates: source +- Before running ppt-master, invoke the installed `ppt-master` skill and follow + its gates: source conversion, project init/import, Strategist Eight Confirmations, optional image acquisition, sequential page-by-page SVG authoring by the main agent, `svg_quality_checker.py`, `total_md_split.py`, `finalize_svg.py`, and @@ -359,7 +378,7 @@ python skills/paper2video/scripts/generate_visual_cues.py \ 6. Render the video, pinning audio order to the script JSON: ```bash -python skills/paper2video/scripts/render_video.py "$VIDEO_OUT" \ +python -m pptx2video.render_video "$VIDEO_OUT" \ --pptx \ --audio-dir "$VIDEO_AUDIO" \ --script-json "$VIDEO_AUDIO/script.json" \ @@ -375,22 +394,24 @@ python skills/paper2video/scripts/render_video.py "$VIDEO_OUT" \ 7. Burn final subtitles and place final video/deck in the normalized output. ```bash -python skills/paper2video/scripts/add_subtitles.py "$VIDEO_OUT" \ +python -m pptx2video.add_subtitles "$VIDEO_OUT" \ --mp4 "$VIDEO_CLIPS/video_raw.mp4" \ --audio-dir "$VIDEO_AUDIO" \ --script-json "$VIDEO_AUDIO/script.json" \ --srt-out "$VIDEO_CAPTIONS/video.srt" \ --vtt-out "$VIDEO_CAPTIONS/video.vtt" \ --out "$VIDEO_OUT/video.mp4" +``` -The default burned-in subtitle render uses a translucent dark caption box so -narration text stays separate from dense PPT content. Use `--no-subtitle-box` -only for an explicitly approved legacy/plain-caption render. Use -`--subtitle-bar` to scale the complete slide above a solid black caption band -when captions must not overlap any PPT content. +The default burned-in subtitle render scales the complete slide above an +appended solid black caption band and burns white captions inside that reserved +band. Use `--subtitle-overlay` only when captions may cover slide pixels; in +overlay mode, `--subtitle-box` keeps the translucent dark caption background +and `--no-subtitle-box` requests plain text. Use `--no-subtitles` when the user disables captions. It still writes SRT/VTT for timeline and QA, but stream-copies only video/audio into `video.mp4`. +```bash cp "$VIDEO_CLIPS/video_raw.mp4" "$VIDEO_OUT/video_no_subtitles.mp4" cp "$VIDEO_SLIDES/slides.pptx" cp "$VIDEO_OUT/video.pptx" @@ -409,7 +430,7 @@ paper2reel must consume this file instead of guessing section start/end times from the final MP4. ```bash -python skills/paper2video/scripts/build_timeline.py \ +python -m pptx2video.build_timeline \ --script-json "$VIDEO_AUDIO/script.json" \ --duration-report "$VIDEO_META/video_duration_report.json" \ --visual-cue-plan "$VIDEO_META/visual_cue_plan.json" \ @@ -436,92 +457,44 @@ Prefer the grouped form when poster sections overlap: } ``` -### Route B - existing ppt-master deck video +### Route B - existing or edited PPTX via standalone pptx2video -Use this when ppt-master has already produced a complete project with -`notes/`, `svg_output/`, and `exports/*.pptx`. - -```text -/ - notes/.md - svg_output/.svg - exports/.pptx -``` - -If `notes/*.md` is missing but `notes/total.md` exists, run the external -ppt-master splitter: +Use the independently maintained `pptx2video` package for a complete native +PPTX. Do not copy its runtime into this skill and do not call +`skills/paper2video/scripts/` for this route. Install it, verify its native +dependencies, and render through the public CLI: ```bash -python "$PPT_MASTER_DIR/scripts/total_md_split.py" +PPTX2VIDEO_OUT=/absolute/path/to/new-video-bundle +python -m pip install \ + 'pptx2video[svg] @ git+https://github.com/ai-nuts/pptx2video.git@v0.5.0' +python -m playwright install chromium +pptx2video --version +pptx2video doctor --svg +test ! -e "$PPTX2VIDEO_OUT" +pptx2video render "$PPTX2VIDEO_OUT" --resolution 1080p ``` -Build TTS script JSON from ppt-master notes: +Always choose a fresh `$PPTX2VIDEO_OUT`; do not reuse the Route A `$VIDEO_OUT` +or an existing paper2assets root. The CLI writes the complete editable video +bundle, word-aligned captions, protocol reports, timeline, and strict QA +evidence. It returns success only after QA passes with zero errors and zero +warnings. Read [references/pptx2video.md](references/pptx2video.md) for the +minimal bridge contract. The standalone package owns all detailed PPTX +authoring, Animation Pane, geometry conflict, and Identity Map rules. -```bash -python skills/paper2video/scripts/notes_to_script.py \ - --voice alloy \ - --target-minutes 3 \ - --out /audio/script.json -``` +## Route A Final QA Gate -Generate audio: - -```bash -python skills/paper2poster/scripts/generate_audio.py \ - /audio/script.json \ - --outdir /audio -``` - -Render: - -```bash -python skills/paper2video/scripts/render_video.py \ - --pptx /exports/.pptx \ - --audio-dir /audio \ - --script-json /audio/script.json \ - --attention-mode highlight \ - --highlight-style spotlight_laser \ - --visual-cues /visual_cues.json \ - --target-minutes 3 \ - --duration-report-out "$VIDEO_META/video_duration_report.json" \ - --out "$VIDEO_CLIPS/video_raw.mp4" \ - --frames-out "$VIDEO_SLIDES/frames" -``` - -Burn final subtitles and copy the deck into the v2 assets bundle: - -```bash -python skills/paper2video/scripts/add_subtitles.py \ - --mp4 "$VIDEO_CLIPS/video_raw.mp4" \ - --audio-dir /audio \ - --script-json /audio/script.json \ - --srt-out "$VIDEO_CAPTIONS/video.srt" \ - --vtt-out "$VIDEO_CAPTIONS/video.vtt" \ - --out "$VIDEO_OUT/video.mp4" - -The default burned-in subtitle render uses a translucent dark caption box so -narration text stays separate from dense PPT content. Use `--no-subtitle-box` -only for an explicitly approved legacy/plain-caption render. Use -`--subtitle-bar` to scale the complete slide above a solid black caption band -when captions must not overlap any PPT content. -Use `--no-subtitles` when the user disables captions. It still writes SRT/VTT -for timeline and QA, but stream-copies only video/audio into `video.mp4`. - -cp "$VIDEO_CLIPS/video_raw.mp4" "$VIDEO_OUT/video_no_subtitles.mp4" -cp /exports/.pptx "$VIDEO_SLIDES/slides.pptx" -``` - -## Final QA Gate - -Run the final hard QA gate for either route. This is not a smoke test; it checks -the exact slide frames archived by `render_video.py --frames-out`, probes +Run the final hard QA gate for Route A. This is not a smoke test; it checks +the exact slide frames archived by +`python -m pptx2video.render_video --frames-out`, probes audio/video streams, checks PPTX geometry for text overflow/overlap and undersized visuals, checks rendered-frame blank space/sparsity, verifies the final MP4 duration, and rejects unsafe TTS rate plans when duration control is requested. ```bash -python skills/paper2video/scripts/check_video_package.py "$VIDEO_OUT" \ +python -m pptx2video.check_video_package "$VIDEO_OUT" \ --pptx \ --script-json "$VIDEO_AUDIO/script.json" \ --audio-dir "$VIDEO_AUDIO" \ @@ -548,7 +521,7 @@ For stricter semantic-anchor enforcement, also require the anchor contract and PPTX-backed anchors: ```bash -python skills/paper2video/scripts/check_video_package.py "$VIDEO_OUT" \ +python -m pptx2video.check_video_package "$VIDEO_OUT" \ --pptx \ --script-json "$VIDEO_AUDIO/script.json" \ --audio-dir "$VIDEO_AUDIO" \ @@ -630,68 +603,16 @@ When strict visual-attention alignment is required and Edge TTS is acceptable, use the bundled Edge helper because it can write word-boundary timings: ```bash -python skills/paper2video/scripts/generate_edge_audio.py \ +python -m pptx2video.generate_edge_audio \ /audio/script.json \ --outdir /audio \ --timings-out /audio/word_timings.json ``` Those timings let `generate_visual_cues.py --require-timestamps` and -`check_video_package.py --require-word-timings` reject highlight plans that only +`python -m pptx2video.check_video_package --require-word-timings` reject highlight plans that only use proportional/estimated timing. -## Rendering Details - -`render_video.py` does: - -1. Prefer `svg_final/*.svg` -> PNG frames via Playwright/Chrome. If no SVG deck - exists, or `--frame-source pptx` is explicitly set, use the legacy PPTX -> - PDF -> PNG path via LibreOffice and `pdftoppm`. -2. Copy the exact MP4 frames to `--frames-out` when provided; final QA should - point `--frames-dir` at that same directory. -3. MP3 duration probing via `ffprobe` or the ffmpeg fallback. -4. One MP4 segment per slide. -5. ffmpeg concat into a final H.264/AAC MP4. - -Audio ordering: - -- Preferred: `--script-json `. -- Auto-detected fallback: `/script.json`, then `/assets/meta/narration.json`, then `/narration.json` for legacy bundles. -- Secondary fallback: `/manifest.json`. -- Last resort: sorted `*.mp3` filenames. - -This matters for `paper2assets`, whose ids are semantic (`problem`, `method`, -`key-result`) rather than numeric (`01-intro`, `02-method`). - -ffmpeg selection: - -- First honor `PAPER2VIDEO_FFMPEG` / `PAPER2VIDEO_FFPROBE` if set. -- Then prefer `imageio_ffmpeg`'s bundled static ffmpeg when installed. -- Then fall back to system `ffmpeg` / `ffprobe`. - -This mirrors the ACL26 video prototype: its subtitle burn step used -`imageio_ffmpeg`'s ffmpeg 7.x because the system ffmpeg on this machine is -2.4.x and too old for reliable subtitles/audio filtering. - -Useful flags: - -| Flag | Purpose | -|---|---| -| `--resolution 720p|1080p|1440p|4k` | Output frame size (default 1080p) | -| `--frame-source auto|svg|pptx` | Slide raster source; `auto` prefers `svg_final` | -| `--svg-dir DIR` | Explicit SVG deck directory | -| `--frames-out DIR` | Persist the exact frames used by the MP4 for QA/review | -| `--fps N` | Frame rate (default 30) | -| `--pad-tail SECONDS` | Trailing silence after each slide (default 0.3) | -| `--start-pad SECONDS` | Leading silence before slide 1 (default 0.5) | -| `--target-minutes N` | Write a final duration report against an N-minute target | -| `--attention-mode none|highlight|cursor|both` | Burn positioned attention cues into slide segments (default `highlight`) | -| `--highlight-style box|spotlight|cursor|box_cursor|spotlight_cursor|laser|box_laser|spotlight_laser` | Presentation style for highlight cues; default `spotlight_laser` | -| `--visual-cues path.json` | Normalized per-slide highlight/cursor cue file | -| `--allow-missing-visual-cues` | Degraded/debug only; final output should not use it | -| `--frames-only` | Stop after slide-frame export | -| `--audio-only-check` | Verify frame/audio count and order | - ## Duration Control Use the duration target at script-generation time, before TTS: @@ -724,7 +645,7 @@ shorter section list; existing PPTX decks should keep the default `keep` mode. `--allow-extractive-duration-draft` is for experiments only and must not be used for final deliverables. -Then pass the same target to `render_video.py`. Rendering writes +Then pass the same target to `python -m pptx2video.render_video`. Rendering writes `_duration_report.json` with the real MP4 duration after audio exists. Use that measured duration to produce a conservative TTS rate plan: @@ -818,7 +739,7 @@ python skills/paper2video/scripts/generate_visual_cues.py \ Then pass positioned cues at render time: ```bash -python skills/paper2video/scripts/render_video.py \ +python -m pptx2video.render_video \ --pptx /exports/.pptx \ --audio-dir /audio \ --script-json /audio/script.json \ @@ -830,7 +751,7 @@ python skills/paper2video/scripts/render_video.py \ ``` `highlight` is the default final-delivery mode. If `--attention-mode` is not -`none`, `render_video.py` now requires `--visual-cues`; missing cues are a +`none`, `python -m pptx2video.render_video` requires `--visual-cues`; missing cues are a blocking error unless the agent explicitly passes the degraded/debug-only `--allow-missing-visual-cues`. @@ -872,7 +793,7 @@ each contracted narration chunk must match its exact `anchor_id`, and rendered from `pptx`, `pptx_cluster`, or a semantic fallback. The candidate review page shows the narration chunk, word-timing match, selected semantic target, final geometry target, and top rejected semantic/geometry alternatives. -`check_video_package.py --strict` reports geometry source counts and timing +`python -m pptx2video.check_video_package --strict` reports geometry source counts and timing source counts; it fails if `geometry_box` is malformed, does not match the rendered cue `box`, lies outside the normalized slide canvas, or if required word-timing alignment is low-confidence. @@ -923,7 +844,7 @@ Strict gate repair loop: mode fails, then exits non-zero. That is intentional: diagnostics exist for repair, while downstream rendering is still blocked until the gate passes. -After a highlighted render, run `build_timeline.py` with the same +After a highlighted render, run `python -m pptx2video.build_timeline` with the same `visual_cue_plan.json`, `visual_cues.json`, `duration_report.json`, script, and subtitle VTT. This is what binds every spoken chunk to its audio time, subtitle cue, and visual target. Do not let downstream tools cut video by ad-hoc @@ -939,18 +860,18 @@ the viewer can show duplicate subtitles when CC is enabled. ## Subtitles -`add_subtitles.py` can use either notes files or script JSON: +`python -m pptx2video.add_subtitles` can use either notes files or script JSON: - With `--script-json`, subtitle order and fallback text come from the JSON. - Without it, the script preserves legacy ppt-master behavior: sorted `notes/*.md` paired with sorted `audio/*.mp3`. -Default mode burns subtitles into the video pixels with a translucent dark -caption box. Pass `--soft` to mux a toggleable `mov_text` track instead. Pass -`--srt-only` to produce just the SRT. Pass `--no-subtitle-box` only for a -user-approved legacy/plain-caption render. Pass `--subtitle-bar` to preserve -the complete slide above a solid black bottom band and burn white captions -inside that reserved band; this avoids covering dense PPT content. +Default mode preserves the complete slide above an appended solid black bottom +band and burns white captions inside that reserved band. Pass +`--subtitle-overlay` to keep the original frame size and place captions over +slide pixels; overlay mode uses a translucent dark box by default, and +`--no-subtitle-box` removes that background. Pass `--soft` to mux a toggleable +`mov_text` track instead. Pass `--srt-only` to produce just the SRT. Pass `--no-subtitles` to keep the public MP4 caption-free while still writing the SRT/VTT timing sidecars required by the internal timeline and QA. @@ -960,19 +881,22 @@ Before calling the video done: - PPTX slide count equals the number of selected script sections. - Every selected section has `audio/.mp3`. -- `render_video.py --audio-only-check` passes. +- `python -m pptx2video.render_video --audio-only-check` passes. - `ffprobe` reports a positive duration for the final MP4. -- `check_video_package.py --strict` passes and writes `video_qa_report.json`. -- If visual attention is enabled, `check_video_package.py --strict-attention` +- `python -m pptx2video.check_video_package --strict` passes and writes + `video_qa_report.json`. +- If visual attention is enabled, + `python -m pptx2video.check_video_package --strict-attention` passes with `--require-visual-cues --require-cue-plan --require-timeline --require-word-timings`. - `timeline.json` exists and every chunk has the expected audio window, subtitle cues, and accepted visual cue before paper2reel consumes it. -- If subtitles are requested, `add_subtitles.py` uses the same `--start-pad`, - `--pad-tail`, and `--script-json` as `render_video.py`. +- If subtitles are requested, `python -m pptx2video.add_subtitles` uses the same + `--start-pad`, `--pad-tail`, and `--script-json` as + `python -m pptx2video.render_video`. ## References - `references/script_json_schema.md` - narration JSON shape and TTS gotchas. -- `references/render_video.md` - compositor internals and ffmpeg debugging. -- `references/visual_cues.md` - visual cue JSON schema and examples. +- `references/visual_cues.md` - paper-specific cue planning and anchor bridge. +- `references/pptx2video.md` - minimal bridge to the standalone PPTX renderer. diff --git a/ResearchStudio-Reel/skills/paper2video/references/pptx2video.md b/ResearchStudio-Reel/skills/paper2video/references/pptx2video.md new file mode 100644 index 0000000..7ba938b --- /dev/null +++ b/ResearchStudio-Reel/skills/paper2video/references/pptx2video.md @@ -0,0 +1,38 @@ +# Standalone pptx2video bridge + +Paper2Video delegates an existing or edited native PPTX to the independently +maintained [`ai-nuts/pptx2video`](https://github.com/ai-nuts/pptx2video) +skill and CLI. This skill does not copy or vendor that package's runtime. + +Install `ppt-master` first, then install the standalone `pptx2video` skill and +its compatible 0.5.x public CLI runtime: + +```bash +npx skills add hugohe3/ppt-master --skill ppt-master +npx skills add ai-nuts/pptx2video --skill pptx2video +python -m pip install \ + 'pptx2video[svg] @ git+https://github.com/ai-nuts/pptx2video.git@v0.5.0' +python -m playwright install chromium +pptx2video --version +``` + +The skill can be invoked directly as `/pptx2video`. Paper2Video uses the same +installed public CLI and has no repository-relative source path. + +Verify Python, native rendering, and SVG browser dependencies: + +```bash +pptx2video doctor --svg +``` + +Render into a new output directory: + +```bash +pptx2video render --resolution 1080p +``` + +The output directory must not already exist. Treat a zero exit status as valid +only because the standalone CLI itself requires strict QA with zero errors and +zero warnings. For authoring protocol, conflict resolution, and advanced flags, +use the installed `/pptx2video` skill and its references in the standalone +repository. diff --git a/ResearchStudio-Reel/skills/paper2video/references/render_video.md b/ResearchStudio-Reel/skills/paper2video/references/render_video.md deleted file mode 100644 index b8df951..0000000 --- a/ResearchStudio-Reel/skills/paper2video/references/render_video.md +++ /dev/null @@ -1,251 +0,0 @@ -# `render_video.py` — internals and debugging - -## Stage diagram - -``` -svg_final/*.svg ──Playwright/Chrome──▶ slide-01.png ... slide-NN.png - │ - └─ fallback: PPTX ──libreoffice──▶ PDF ──pdftoppm──▶ slide-01.png ... - │ - │ pair by script JSON order - │ (fallback: manifest/sorted names) - ▼ - audio/.mp3 ─ffprobe─▶ duration ───┘ - │ - visual_cues.json ──────────────────────┤ optional box/cursor attention overlays - │ - ▼ - ffmpeg per-slide segment - (libx264 + aac, padded with 0.3s silence) - │ - ▼ - ffmpeg concat demuxer (stream copy) - │ - ▼ - /exports/.mp4 -``` - -## Why these specific tool choices - -- **Browser SVG rasterization first**: ppt-master authors and previews the deck - as SVG, and `svg_final` contains the expanded icon/vector content used for - export. Rendering those SVGs with Chrome avoids LibreOffice reflowing text or - geometry into frames that do not match the deck the user inspected. -- **LibreOffice for PPTX→PDF as fallback**: still available via - `--frame-source pptx` or when no SVG deck exists, but it is no longer the - preferred path for ppt-master projects. -- **pdftoppm over `convert`**: ImageMagick's `convert` works but defaults to - Ghostscript under the hood, which is roughly 3× slower per page and - occasionally rasterizes embedded fonts as pixelated bitmaps at low DPI. -- **ffmpeg concat *demuxer* over concat *filter***: every per-slide segment - is encoded with identical codecs and dimensions, so the demuxer can - stream-copy them — no re-encoding pass. Concat filter would re-encode - everything (slow, lossy). -- **imageio-ffmpeg before system ffmpeg**: this repo may run on machines where - PATH points to an old ffmpeg (ACL26 had 2.4.x). When `imageio_ffmpeg` is - installed, the script prefers its modern static binary; set - `PAPER2VIDEO_FFMPEG` / `PAPER2VIDEO_FFPROBE` to override explicitly. -- **`-pix_fmt yuv420p`**: required for QuickTime/Safari/older Chrome - playback. Without it you get a video that plays everywhere except the - most popular consumer environment. - -## Per-slide segment recipe - -``` -ffmpeg -y \ - -loop 1 -framerate 30 -i slide-05.png \ - -i 05-results.mp3 \ - -vf "scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2:color=black,format=yuv420p" \ - -af "aresample=44100,aformat=channel_layouts=stereo:sample_rates=44100,apad" \ - -c:v libx264 -preset medium -crf 20 \ - -c:a aac -b:a 192k \ - -strict -2 \ - -pix_fmt yuv420p -r 30 -t \ - -movflags +faststart \ - seg_0005.mp4 -``` - -Notes: -- `-loop 1` on the image keeps the same frame on screen for the whole segment. -- Bare `apad` pads silence onto the end of the audio so the cut isn't - jarring. We avoid `apad=pad_dur=N` because older ffmpeg builds don't support - `pad_dur`; the segment-level `-t ` still clips the padded - stream to the desired length. We don't use `-shortest` because that ends the - segment when audio ends, which can clip the final word in some MP3 encodings. -- `-strict -2` keeps older ffmpeg builds working with their experimental AAC - encoder. Modern ffmpeg accepts it harmlessly. -- `-t` pins total duration so an over-padded audio doesn't extend the - segment past the user's pad-tail expectation. - -## Concat list format - -The list file given to ffmpeg's concat demuxer: - -``` -file '/abs/path/to/seg_0001.mp4' -file '/abs/path/to/seg_0002.mp4' -file '/abs/path/to/seg_0003.mp4' -``` - -Paths are absolute and quoted (via `shlex.quote`) so spaces/Unicode in the -project path don't break the parser. The demuxer requires consistent codecs -and resolution — that invariant is why we encode each segment with the same -parameters. - -## Audio ordering - -Call `render_video.py` with `--script-json ` whenever possible. -The compositor then orders `audio/.mp3` by the script's `sections` array, -which is required for paper2assets ids such as `problem`, `method`, and -`key-result`. - -If `--script-json` is omitted, the script auto-detects -`/script.json`, then `/narration.json`, then -`/manifest.json`. Only when none of those exist does it fall back to -alphabetical `*.mp3` order, which is safe for numeric ppt-master stems like -`01-intro.mp3` but unsafe for semantic ids. - -## Duration reports - -`render_video.py --target-minutes N` does not change narration length at render -time. It writes a final duration report after real audio durations are known: - -```json -{ - "schema_version": "paper2video_duration_report.v1", - "status": "within_tolerance", - "target_seconds": 180.0, - "actual_seconds": 198.2, - "target_delta_seconds": 18.2, - "slides": [ - {"index": 1, "audio": "01_intro.mp3", "audio_seconds": 18.4, "visual_cues": 2} - ] -} -``` - -Use `notes_to_script.py --target-minutes` or -`assets_to_script.py --target-minutes` before TTS to actually shape narration. -The render-time report is the real post-audio check. - -## Final package QA - -After rendering, run `check_video_package.py --strict`. Prefer passing -`--frames-dir /assets/slides/frames`, the directory written by -`render_video.py --frames-out`, so QA checks the exact frames used in the MP4. -If no frames directory is provided, the checker falls back to rendering the -PPTX through LibreOffice for legacy packages. It probes audio/MP4 streams, -verifies slide and script counts, checks PPTX text-box overflow/overlap risks, -flags undersized visuals, and runs pixel-level checks for blank/sparse rendered -slides. - -For videos with attention overlays, also pass `--strict-attention`, -`--require-visual-cues`, `--require-cue-plan`, and `--require-word-timings`. -That gate fails when cue coverage is too low, cue chunks are skipped, timings -fall back to proportional estimates, accepted cues are below confidence -threshold, accepted targets point at low-value slide chrome such as captions -or headers, or the rendered `geometry_box` no longer matches the cue `box`. - -## Attention overlays - -`--attention-mode highlight --visual-cues visual_cues.json` adds -per-slide ffmpeg overlays after slide scaling/padding and before final -`format=yuv420p`. Coordinates in `visual_cues.json` are normalized to the final -frame size, so a cue works across 720p/1080p/4K renders. - -The current renderer supports: - -- `highlight`: applies `type: "highlight"` cues. Box cues render as the - selected normalized region; point-only legacy cues render as the older soft - dot fallback. -- `cursor`: mouse-pointer overlay centered at a normalized point. -- `both`: applies both `highlight` and `cursor` cues when the cue file contains - both types. Use this only when cue targets are precise enough. - -`--highlight-style` controls how highlight cues look: - -| Style | Use | -|---|---| -| `box` | Low-opacity slate fill and border around the selected box | -| `cursor` | Mouse pointer only at the cue point | -| `box_cursor` | Box plus mouse pointer, useful for geometry review | -| `spotlight` | Feathered dim-out around the selected box | -| `spotlight_cursor` | Feathered dim-out plus mouse pointer | -| `laser` | Red laser-pointer dot only at the cue point | -| `box_laser` | Focus box plus red laser-pointer dot | -| `spotlight_laser` | Default. Feathered dim-out plus red laser-pointer dot | - -The default `spotlight_laser` style is the production path because it combines -semantic box geometry with a tolerant spotlight and a presenter-like laser dot. -Spotlight styles generate one full-frame transparent alpha mask per cue, keep -the accepted box at original brightness, and dim the outside area with a -continuous feathered falloff. At 1080p the default feather is about 56 px. They -are visually more tolerant than a hard box, but can make long full-video -renders slower than plain `box`. - -Cursor styles use a generated transparent pointer overlay. Within each slide, -the pointer eases from one cue point to the next shortly before the next cue -starts, so it behaves like a presenter moving a mouse rather than jumping -instantly between regions. - -Laser styles use the same eased cue-to-cue movement, but render a small red -dot with a soft halo instead of the mouse pointer. - -See `visual_cues.md` for the JSON shape. - -## Common failures and fixes - -### `[concat @ 0x...] DTS X < Y out of order` - -Usually means one of the segments has a different stream layout. Re-check -that every segment was produced by `encode_segment` (not pre-existing files -from a previous run with different settings). The script wipes -`/.video_work/` on each invocation precisely to avoid this — if -you passed `--keep-temp` from a previous run, clear it manually. - -### Output MP4 plays video but no audio - -Usually a missing AAC encoder. Run `ffmpeg -encoders | grep aac`. If -nothing prints, your ffmpeg was built without AAC support — install a -mainstream package, not a stripped-down one. - -### Output is tiny (e.g. 50KB) and `ffprobe` shows zero duration - -Concat list points at empty/corrupt segments. Re-run with `--keep-temp` and -inspect `/.video_work/segments/`. Each `seg_NNNN.mp4` should be -several MB and play standalone in `ffplay`/VLC. - -### LibreOffice "source file could not be loaded" - -`.pptx` path has unicode or relative components. The script resolves to -absolute paths but if you assemble the command yourself, prefer -`Path(...).resolve()` first. - -### LibreOffice silently produces no PDF - -A logged-in GUI session is holding a lock on `~/.config/libreoffice`. The -script sidesteps this by passing `-env:UserInstallation=file://` — -each render gets a fresh user profile. If you see this happen, you're -running an older or locally-modified copy of `render_video.py`. - -## Performance notes - -| Stage | Cost driver | Typical 20-slide deck | -|-------|-------------|----------------------| -| SVG → PNG | Browser screenshots at output resolution | 3–10 s | -| PPTX → PDF fallback | One LibreOffice startup | 5–15 s | -| PDF → PNG fallback | DPI × page count | 5–10 s @ 150 DPI | -| Encode segments | `crf 20 medium` × audio length | ~0.5× real-time × total audio | -| Concat | Stream copy | <2 s | - -A 5-minute narrated deck on a modern laptop takes about 3 minutes to render -end-to-end. Most of that time is in the per-segment x264 pass — drop to -`-preset veryfast` (edit `encode_segment`) if you want to halve render time -at the cost of ~15% larger output files. - -## Extending - -If you want to add slide-to-slide crossfade transitions, encode each -segment with a 0.5 s overlap region and use ffmpeg's `xfade` filter at -concat time instead of the demuxer. That requires a re-encoding concat -pass (slow), which is why the default keeps hard cuts. Ship the simple -version first and only add transitions when a user asks. diff --git a/ResearchStudio-Reel/skills/paper2video/references/script_json_schema.md b/ResearchStudio-Reel/skills/paper2video/references/script_json_schema.md index 53bec86..5079516 100644 --- a/ResearchStudio-Reel/skills/paper2video/references/script_json_schema.md +++ b/ResearchStudio-Reel/skills/paper2video/references/script_json_schema.md @@ -28,7 +28,8 @@ also produced from paper2assets by - **sections[*].id** — also the output filename (`audio/.mp3`). For ppt-master projects, this is usually the slide stem (`01-intro`). For paper2assets, this is the semantic section id (`problem`, `key-result`). - In both cases, pass the JSON to `render_video.py --script-json` so frame and + In both cases, pass the JSON to + `python -m pptx2video.render_video --script-json` so frame and audio pairing follows `sections` order instead of alphabetical filename order. - **sections[*].heading** — only used in `manifest.json`. Cosmetic. @@ -80,7 +81,8 @@ It also writes a sidecar `duration_plan.json`: ``` This sidecar is metadata only. `generate_audio.py` should consume `script.json`, -not `duration_plan.json`. After TTS, `render_video.py --target-minutes` writes a +not `duration_plan.json`. After TTS, +`python -m pptx2video.render_video --target-minutes` writes a real `duration_report.json` based on probed MP3/MP4 duration. `notes_to_script.py --min-chars` filters them out automatically. Just remember the resulting MP4 will also skip those slides — if a divider @@ -127,6 +129,7 @@ audio/ └── script.json ``` -`render_video.py` then reads the PPTX (4 slides) and audio dir (4 MP3s) and +`python -m pptx2video.render_video` then reads the PPTX (4 slides) and audio dir +(4 MP3s) and muxes them in sorted order. PNGs are named `slide-01.png … slide-04.png` inside the temp dir; matching is positional, not by stem. diff --git a/ResearchStudio-Reel/skills/paper2video/references/visual_cues.md b/ResearchStudio-Reel/skills/paper2video/references/visual_cues.md index 3991e7e..1e851ba 100644 --- a/ResearchStudio-Reel/skills/paper2video/references/visual_cues.md +++ b/ResearchStudio-Reel/skills/paper2video/references/visual_cues.md @@ -1,171 +1,19 @@ -# Visual cues JSON - semantic highlight boxes and cursor overlays +# Paper2Video visual cue bridge -`render_video.py` can burn simple attention cues into each static slide segment. -Use this when a narrated video feels too inert but you do not want to animate -the slide deck itself. +Paper2Video owns semantic cue planning for a research-paper narrative. The +installed `pptx2video` runtime owns the cue JSON schema and renders the accepted +boxes and points. Read the standalone skill's `references/visual_cues.md` when +changing that public schema or renderer behavior. -## CLI - -```bash -python skills/paper2video/scripts/render_video.py \ - --pptx /exports/.pptx \ - --audio-dir /audio \ - --script-json /audio/script.json \ - --attention-mode highlight \ - --highlight-style spotlight_laser \ - --visual-cues /visual_cues.json \ - --out /exports/.mp4 -``` - -`--attention-mode` controls which cues are applied: - -| Mode | Behavior | -|---|---| -| `none` | Ignore cues; use only for an approved no-highlight render | -| `highlight` | Apply only `type: "highlight"` cues; default final-delivery mode | -| `cursor` | Apply only `type: "cursor"` cues | -| `both` | Apply both cue types when the cue file contains both | - -## Schema - -Production highlight cues are box-first. Coordinates are normalized to the final video -frame: - -- `[0, 0]` is top-left. -- `[1, 1]` is bottom-right. -- A highlight `box` is `[x, y, w, h]` and is the geometry actually rendered. - The renderer expands it by about one border width, then draws a low-opacity - slate fill and soft border around that target. -- A highlight should also include `point` as the center for compatibility and - audit tooling. Point-only highlight cues remain valid as degraded/debug - fallbacks, but strict QA expects boxes. -- Automatic cues should include `semantic_*` and `geometry_*` fields. The - semantic fields explain what narration target was selected; the geometry - fields explain which PPTX or PPTX-cluster box was used for the visible - highlight. When PPTX geometry cannot be matched with enough confidence, - `geometry_matched` is false and the cue falls back to the semantic box. -- Timing should come from `edge_word_alignment`, which aligns each narration - chunk back to the real word-boundary timeline. `duration_proportional` is - scaffolding/debug timing only and should fail strict final QA when word - timings are required. -- A cursor `point` is `[x, y]`. - -```json -{ - "schema_version": "paper2video_visual_cues.v3", - "cue_shape": "semantic_box", - "slides": [ - { - "id": "07_fineweb_accuracy_lift", - "cues": [ - { - "start": 3.2, - "end": 8.5, - "type": "highlight", - "box": [0.12, 0.28, 0.14, 0.21], - "point": [0.19, 0.39], - "target": "cue_s07_c1_accuracy_lift", - "target_role": "result", - "semantic_target": "svg:g:cues07-result-card", - "semantic_source": "svg", - "semantic_box": [0.11, 0.27, 0.16, 0.22], - "geometry_target": "TextBox 18", - "geometry_source": "pptx", - "geometry_box": [0.12, 0.28, 0.14, 0.21], - "geometry_matched": true, - "geometry_match_score": 5.72, - "confidence": 0.82, - "color": "#64748B", - "opacity": 0.18, - "size": 56 - }, - { - "start": 9.0, - "duration": 4.0, - "type": "cursor", - "point": [0.52, 0.62], - "color": "#ff6b00", - "opacity": 0.95, - "size": 32 - } - ] - } - ] -} -``` - -## Slide matching - -Prefer `id`, matching the narration/audio stem: - -```json -{"id": "07_fineweb_accuracy_lift", "cues": []} -``` - -Use `index` only when ids are unavailable: - -```json -{"index": 7, "cues": []} -``` - -## Timing - -Cue times are relative to the start of that slide's own video segment, not the -global MP4 timeline. They should align with the narration text for that slide. - -Accepted timing forms: - -```json -{"start": 3.2, "end": 8.5} -{"at": 9.0, "duration": 4.0} -``` - -Times are clipped to the slide segment duration, including `--pad-tail`. - -## Rendering - -`highlight` cues are box-first, and the default presentation style is -`spotlight_laser`: a feathered spotlight around the accepted box plus a small -red laser-pointer dot at the cue center. Existing point-only cue files remain -valid and render as cursor/point fallbacks, but final strict attention QA -requires highlight boxes. - -`render_video.py --highlight-style` controls the presentation of accepted -highlight cues: - -| Style | Behavior | -|---|---| -| `box` | Subtle filled frame around the selected box | -| `cursor` | Mouse pointer only at the cue point | -| `box_cursor` | Box plus mouse pointer for debugging or reviewer comparisons | -| `spotlight` | Feathered dim-out around the selected box | -| `spotlight_cursor` | Feathered dim-out plus mouse pointer | -| `laser` | Red laser-pointer dot only at the cue point | -| `spotlight_laser` | Default delivery style: feathered dim-out plus red laser-pointer dot | - -The spotlight styles generate a full-frame transparent alpha mask for each cue: -the accepted box remains at original brightness while the surrounding slide -fades out with a continuous feather. At 1080p the default feather is about -56 px. They are visually tolerant, but full-video encoding can be slower -because each cue adds an extra overlay mask. - -Cursor styles render a generated transparent mouse pointer. The renderer keeps -the same cue point semantics, but eases the visible pointer between consecutive -cue points on a slide shortly before each next cue starts. - -Laser styles use the same cue point semantics and eased movement, but render a -small red dot with a soft halo instead of the mouse pointer. - -## Semantic vs geometry review +## Semantic and geometry review `generate_visual_cues.py` keeps semantic matching separate from rendered geometry. A cue may select an SVG semantic group, then render a matched PPTX -box. It may also promote a line-level text target to a nearby module/group when -that parent is still bounded enough for presentation-style highlighting. -Connected PPTX clusters are filtered so a large union box does not cross -unrelated regions. +box. It may promote a line-level text target to a nearby module when that +parent remains suitably bounded. Connected PPTX clusters are filtered so a +large union box does not cross unrelated regions. -Always write the review artifacts during automatic cue generation: +Always write review artifacts during automatic cue generation: ```bash python skills/paper2video/scripts/generate_visual_cues.py \ @@ -183,17 +31,13 @@ python skills/paper2video/scripts/generate_visual_cues.py \ --candidate-review-out /cue_candidate_review.html ``` -Use `cue_candidate_review.html` when a rendered frame looks wrong. It shows -the chunk text, word-timing match, selected semantic target, final geometry -target, promotion reason, and top semantic/geometry candidates that were -accepted or rejected. +Use `cue_candidate_review.html` to inspect the narration chunk, word-timing +match, semantic target, final geometry target, promotion reason, and rejected +candidates. -## Current limits +## Anchor contract -This implementation renders stable, deterministic overlays with ffmpeg -filters. It should not guess highlights from layout alone. For automatic cue -generation, first create a visual-anchor contract and ask ppt-master to create -semantic anchors: +Create the contract before asking ppt-master to author the deck: ```bash python skills/paper2video/scripts/generate_cue_requirements.py \ @@ -203,34 +47,13 @@ python skills/paper2video/scripts/generate_cue_requirements.py \ --markdown-out /cue_requirements.md ``` -Write anchors into both the final SVG and the exported PPTX when possible. -SVG/HTML anchors are still valuable semantic labels, especially when the slide -source groups related content more cleanly than PPTX. The cue generator now -prefers PPTX geometry for the rendered box when it can match the semantic SVG -target to a PPTX element or a small connected PPTX element cluster. PPTX shape -name, alt-text title, or alt-text description keeps that geometry auditable. -SVG/HTML can carry the same anchor in `id`, `data-cue-label`, ``, or -`<desc>`: +Write each stable `cue_` anchor into both SVG and PPTX metadata. Anchor a chart +row, formula block, diagram panel, card, or figure subregion. Do not anchor +headers, captions, logos, QR tiles, page numbers, or decorative backgrounds. +When `--anchor-contract` is present, require exact `anchor_id` matching instead +of falling back to layout guesses. -```xml -<g id="cue_s08_c2_multi_head_attention" - data-cue-label="multi-head attention split eight heads concatenate projections"> - <title>cue_s08_c2_multi_head_attention - Multi-head attention diagram: split Q/K/V projections into eight heads, then concatenate. - ... - -``` - -Anchor rules: - -- Prefer stable PPTX/SVG ids beginning with `cue_`. -- Include narration keywords in ``, `<desc>`, or `data-cue-label`. -- Anchor specific visual content: chart row, formula block, diagram panel, - card, or figure subregion. -- Do not anchor headers, captions, logos, QR tiles, page numbers, or background - chrome. -- When `--anchor-contract` is provided, exact `anchor_id` matching is required; - the matcher should not silently fall back to layout guessing for that chunk. +For final highlighted video, require PPTX-backed anchors: ```bash python skills/paper2video/scripts/generate_visual_cues.py <project_path> \ @@ -252,13 +75,6 @@ python skills/paper2video/scripts/generate_visual_cues.py <project_path> \ --repair-md-out <project_path>/cue_repair_requests.md ``` -`geometry_resolution.json` summarizes how many cues rendered from direct PPTX -boxes, PPTX connected clusters, or semantic fallbacks. Review this file with -`cue_audit.html` when a frame looks too broad or too narrow. Use -`--no-prefer-pptx-geometry` only for debugging an SVG-vs-PPTX geometry -regression; final highlighted videos should prefer PPTX geometry unless the -gate reports low confidence and requests repair. - -If strict mode fails, the script still writes cue audits and repair requests, -then exits non-zero. Fix the deck or narration and rerun before rendering a -highlighted video. +Review `geometry_resolution.json` and `cue_audit.html` when a box is too broad +or too narrow. Use `--no-prefer-pptx-geometry` only to debug SVG/PPTX geometry. +If strict mode fails, repair the deck or narration and rerun before rendering. diff --git a/ResearchStudio-Reel/skills/paper2video/scripts/add_subtitles.py b/ResearchStudio-Reel/skills/paper2video/scripts/add_subtitles.py deleted file mode 100755 index 3afbbb2..0000000 --- a/ResearchStudio-Reel/skills/paper2video/scripts/add_subtitles.py +++ /dev/null @@ -1,1193 +0,0 @@ -#!/usr/bin/env python3 -""" -add_subtitles.py — generate subtitles from per-slide narration notes and -either burn them into the video pixels (hardsub, default), mux them as a -soft mov_text track (`--soft`), or keep the public video caption-free while -still writing timing sidecars (`--no-subtitles`). - -Pipeline position (optional post-step for paper2video): - Inputs: - <project_path> : ppt-master project root (must contain notes/ + audio/) - --mp4 : the MP4 produced by render_video.py - --out : destination for the subtitled MP4 - --srt-out : (optional) where to also save the standalone .srt - --vtt-out : (optional) where to also save the standalone .vtt - --start-pad : MUST match render_video.py's --start-pad - --pad-tail : MUST match render_video.py's --pad-tail - - Steps: - 1. Read subtitle text from --script-json sections order when provided - (falling back to notes/<id>.md if present), otherwise pair sorted - notes/*.md with sorted audio/*.mp3 for legacy ppt-master projects. - 2. Probe each audio/<id>.mp3 to get the spoken duration of that slide. - 3. Split each slide's text into sentence-level cues (~80 chars each). - 4. Distribute the slide's audio duration across its cues proportional - to character count, snapping cue boundaries inside [start, end]. - 5. Walk slides in sorted order, advancing the clock by - pad + audio_duration + tail to mirror render_video.py's layout. - 6. Probe the bottom band of each slide for luminance; pick black text - on light slides and white text on dark ones. Add the opposite color - as an outline so the text is legible even when the picker is on - a borderline slide. - 7. Write the cues to <project>/exports/<stem>.srt and .vtt (always — - useful as YouTube/archive sidecars and timeline/visualization input). - 8. Default (hardsub): convert cues to an ASS file with a translucent - dark caption box and burn it into the video with ffmpeg's `ass=` - filter. The video is re-encoded (libx264 CRF 20 by default), - audio is stream-copied. Output: - <project>/exports/<stem>_subbed.mp4 - The subtitles are now part of every frame — no player toggle, no - font-rendering surprise on phones that don't honor mov_text. - 9. `--soft` mode: skip burn-in, stream-copy video+audio, and mux the - SRT as a `mov_text` track. Toggleable in players that honor it, - invisible on players that don't. Output filename unchanged. - 10. `--no-subtitles` mode: still write SRT/VTT for timeline and QA, - but stream-copy only the input video/audio streams to the output. - Existing subtitle/data streams are explicitly excluded. - -Why hardsub by default (this turn's contract — "make the script part of the -video file"): - Soft mov_text is a polite default for English-speaking desktop viewers, - but in practice: mobile share targets (WeChat, Twitter, Slack previews), - inline-played embeds on news sites, and most VR / signage stacks ignore - the soft track entirely. Burning the cues into pixels guarantees the - captions are seen by every viewer on every player. The cost is a full - re-encode of the video stream (~2–3× the previous render time) and a - decision baked into the file. That tradeoff is what the user is asking - for when they say "part of the video file". - -Why ASS instead of just feeding SRT to the `subtitles=` filter: - libass's SRT parser silently drops some `<font color>` corners, and we - rely on per-slide colors to keep cues legible. Converting our cues to - a tiny ASS file with explicit `{\\c&Hbbggrr&\\3c&Hbbggrr&}` overrides - per dialogue line gives us deterministic per-cue color + outline. The - SRT is still emitted as a sidecar. - -Per-slide text color (white vs. black) for readability: - Different slides may have different background colors (white hero pages, - dark navy cover pages, photographic backgrounds, etc.). To keep the - subtitle text legible on every slide we sample one frame per slide at - the slide's midpoint timestamp, measure the mean luminance of the - bottom band where the cues actually render, and pick the higher- - contrast color (`#FFFFFF` on dark backgrounds, `#000000` on light ones). - -Timing assumptions: - render_video.py lays out the timeline as: - [start_pad of silence] + [audio_1 + pad_tail] + [audio_2 + pad_tail] + ... - This script reproduces that math exactly. If you change start_pad or - pad_tail when calling render_video.py, pass the SAME values here or the - subtitles will drift relative to the audio. -""" - -from __future__ import annotations - -import argparse -import os -import re -import shutil -import subprocess -import sys -import tempfile -from dataclasses import dataclass -from pathlib import Path -import json - - -# --------------------------------------------------------------------------- -# Tool discovery — mirrors render_video.py so we accept the same fallback -# --------------------------------------------------------------------------- - -def _which(name: str) -> str | None: - return shutil.which(name) - - -def _imageio_ffmpeg_binary() -> str | None: - try: - import imageio_ffmpeg # type: ignore - return imageio_ffmpeg.get_ffmpeg_exe() - except Exception: - return None - - -def find_ffmpeg_pair() -> tuple[str, str]: - """Return (ffmpeg, ffprobe) paths or exit with guidance. - - Same fallback policy as render_video.py: prefer explicit env binaries, then - the static `imageio_ffmpeg` binary, then system binaries. The static binary - lacks ffprobe, so durations are probed by parsing `ffmpeg -i` stderr. - """ - env_ffmpeg = os.getenv("PAPER2VIDEO_FFMPEG") or os.getenv("FFMPEG_BINARY") - if env_ffmpeg: - env_path = Path(env_ffmpeg).expanduser() - if env_path.is_file(): - env_ffprobe = os.getenv("PAPER2VIDEO_FFPROBE") - if env_ffprobe and Path(env_ffprobe).expanduser().is_file(): - return str(env_path), str(Path(env_ffprobe).expanduser()) - return str(env_path), str(env_path) - - fallback = _imageio_ffmpeg_binary() - if fallback: - return fallback, fallback - - ffmpeg = _which("ffmpeg") - ffprobe = _which("ffprobe") - if ffmpeg and ffprobe: - return ffmpeg, ffprobe - - sys.exit( - "[add_subtitles] ffmpeg/ffprobe not found and imageio_ffmpeg is not installed.\n" - "Pick one:\n" - " • System install: sudo apt-get install -y ffmpeg\n" - " • Python fallback: pip install imageio-ffmpeg\n" - ) - - -def probe_duration(audio: Path, ffprobe: str, ffmpeg: str) -> float: - """Return audio duration in seconds; falls back to parsing ffmpeg stderr.""" - if ffprobe != ffmpeg: - out = subprocess.run( - [ffprobe, "-v", "error", "-show_entries", "format=duration", - "-of", "default=noprint_wrappers=1:nokey=1", str(audio)], - capture_output=True, text=True, - ) - if out.returncode == 0 and out.stdout.strip(): - return float(out.stdout.strip()) - - out = subprocess.run([ffmpeg, "-i", str(audio)], capture_output=True, text=True) - m = re.search(r"Duration:\s+(\d+):(\d+):([\d.]+)", out.stderr) - if not m: - sys.exit(f"[add_subtitles] could not probe duration for {audio.name}") - return int(m.group(1)) * 3600 + int(m.group(2)) * 60 + float(m.group(3)) - - -# --------------------------------------------------------------------------- -# Text → cue chunks -# --------------------------------------------------------------------------- - -# Abbreviations whose trailing period is NOT a sentence boundary. Conservative -# list — better to under-split than to break in the middle of "e.g." or -# "Ph.D.". Add to taste; the test is "would a reader pause here? if no, list it". -ABBREVS = ( - "e.g.", "i.e.", "etc.", "vs.", "Dr.", "Mr.", "Mrs.", "Ms.", "Prof.", - "St.", "Ph.D.", "U.S.", "U.K.", "Fig.", "Eq.", "No.", "Ref.", "Sec.", -) - -# Markdown line-level prefixes we should peel off before treating the body -# as spoken prose. ppt-master's per-slide notes are usually already plain -# text (total_md_split.py drops the H1 heading), but a stray bullet or a -# blockquote can sneak through. -_BULLET_RE = re.compile(r"^\s*[-*+]\s+") -_NUMLIST_RE = re.compile(r"^\s*\d+[.)]\s+") -_QUOTE_RE = re.compile(r"^\s*>\s?") -_HEADING_RE = re.compile(r"^\s{0,3}#{1,6}\s+") -_HR_RE = re.compile(r"^\s*[-*]{3,}\s*$") - - -def strip_markdown(text: str) -> str: - """Cheap markdown → plain prose. Mirrors notes_to_script.strip_markdown - closely enough for cue splitting; we don't need perfect fidelity here. - """ - out = [] - in_code = False - for raw in text.splitlines(): - line = raw.rstrip() - if line.lstrip().startswith("```"): - in_code = not in_code - continue - if in_code or _HR_RE.match(line): - continue - line = _HEADING_RE.sub("", line) - line = _BULLET_RE.sub("", line) - line = _NUMLIST_RE.sub("", line) - line = _QUOTE_RE.sub("", line) - # Inline emphasis/code/links — strip syntax, keep text. - line = re.sub(r"`([^`]+)`", r"\1", line) - line = re.sub(r"\*\*([^*]+)\*\*", r"\1", line) - line = re.sub(r"__([^_]+)__", r"\1", line) - line = re.sub(r"\*([^*]+)\*", r"\1", line) - line = re.sub(r"_([^_]+)_", r"\1", line) - line = re.sub(r"!\[([^\]]*)\]\([^)]*\)", r"\1", line) - line = re.sub(r"\[([^\]]+)\]\([^)]*\)", r"\1", line) - line = re.sub(r"<[^>]+>", "", line) - out.append(line) - # Collapse to single space-separated paragraph; subtitles don't honor - # line breaks in source notes. - return re.sub(r"\s+", " ", " ".join(out)).strip() - - -def _split_sentences(text: str) -> list[str]: - """Split on `. `, `? `, `! ` when the next char looks like a sentence start. - - Conservative: refuses to split right after a known abbreviation (e.g., - "e.g.", "Dr.") and refuses to split when the period sits between digits - (decimals, version numbers, "IPC 100." mid-list). - """ - # First, mask the periods inside known abbreviations with U+FFFD so the - # splitter can't trip on them. We restore them after splitting. - masked = text - for a in ABBREVS: - masked = masked.replace(a, a.replace(".", "�")) - - # Also mask decimal points (digit.digit) so "1.5" doesn't split. - masked = re.sub(r"(\d)\.(\d)", r"\1�\2", masked) - - parts = re.split(r"(?<=[.!?])\s+(?=[\"'(A-Z0-9])", masked) - parts = [p.replace("�", ".") for p in parts] - return [p.strip() for p in parts if p.strip()] - - -def _chunk_long_sentence(sentence: str, max_chars: int) -> list[str]: - """Break a too-long sentence at clause boundaries (commas, semicolons, - em-dashes / en-dashes / hyphen-surrounded-by-spaces). Falls back to - word-level chunking if no clause breaks help. - """ - if len(sentence) <= max_chars: - return [sentence] - - # Clause splitter — keep the punctuation glued to the left side. - pieces = re.split(r"(?<=[,;])\s+|\s+[—–-]\s+", sentence) - pieces = [p.strip() for p in pieces if p and p.strip()] - - # Reassemble greedily up to max_chars. - chunks: list[str] = [] - buf = "" - for p in pieces: - candidate = (buf + " " + p).strip() if buf else p - if len(candidate) <= max_chars: - buf = candidate - else: - if buf: - chunks.append(buf) - buf = p - if buf: - chunks.append(buf) - - # Last-resort word chunking for any chunk still way over budget. - final: list[str] = [] - for c in chunks: - if len(c) <= int(max_chars * 1.4): - final.append(c) - continue - words = c.split() - cur: list[str] = [] - for w in words: - cur.append(w) - if len(" ".join(cur)) >= max_chars: - final.append(" ".join(cur)) - cur = [] - if cur: - final.append(" ".join(cur)) - return final - - -def split_into_cues(text: str, max_chars: int) -> list[str]: - """Produce a list of subtitle-friendly chunks for one slide.""" - text = strip_markdown(text) - if not text: - return [] - sentences = _split_sentences(text) - cues: list[str] = [] - for s in sentences: - cues.extend(_chunk_long_sentence(s, max_chars)) - return cues - - -# --------------------------------------------------------------------------- -# Per-slide subtitle text color (white on dark / black on light) -# --------------------------------------------------------------------------- - -# Subtitle text colors. Only white and black per the user's contract — no -# off-grays. A high-contrast pure color reads cleanly over both photographic -# and flat-fill backgrounds and degrades well on older players. -COLOR_WHITE = "#FFFFFF" -COLOR_BLACK = "#000000" - -# Fraction of the frame height where mov_text actually renders. Players vary -# (QuickTime ~88%, VLC ~85%, mpv configurable) but the bottom 18%–8% band is -# almost always where the text lands. Sampling that strip — not the full -# frame — is what makes the picker robust against e.g. a dark navy header -# bar on an otherwise-white slide. -SUBTITLE_BAND_TOP = 0.78 -SUBTITLE_BAND_BOTTOM = 0.96 -# Side margin: skip the outer 5% to ignore page-number chips / decorative -# stripes that don't sit under the actual text. -SUBTITLE_BAND_SIDE = 0.05 - -# Above this mean-luma the band reads as "light" → use black text. -# 128 is the obvious midpoint but in practice 140 reads better because -# the eye weighs near-white pixels more heavily than near-black ones. -LUMA_THRESHOLD = 140.0 - - -def extract_frame_at(mp4: Path, timestamp: float, out_png: Path, ffmpeg: str) -> bool: - """Grab one frame from `mp4` at `timestamp` seconds. Returns True on success. - - Uses `-ss` BEFORE `-i` for fast-seek (input-side seek) — accurate enough - for picking a subtitle color, and avoids decoding from t=0 for every - slide which would make this O(slides²) on a 10-minute video. - """ - out_png.parent.mkdir(parents=True, exist_ok=True) - cmd = [ - ffmpeg, "-y", - "-ss", f"{max(timestamp, 0):.3f}", - "-i", str(mp4), - "-frames:v", "1", - "-q:v", "2", - str(out_png), - ] - proc = subprocess.run(cmd, capture_output=True, text=True) - return proc.returncode == 0 and out_png.is_file() and out_png.stat().st_size > 0 - - -def pick_color_for_frame(png_path: Path) -> tuple[str, float]: - """Return (color_hex, mean_luma) for the subtitle band of `png_path`. - - Mean luma is BT.601-weighted (0.299 R + 0.587 G + 0.114 B), matching how - the human eye perceives brightness — pure green looks much brighter than - pure blue at the same intensity. The threshold compares against this - perceptual luma, not raw RGB averages. - """ - try: - from PIL import Image # type: ignore - except ImportError: - sys.exit( - "[add_subtitles] Pillow is required for per-slide color picking.\n" - " Install with: pip install Pillow\n" - " Or pass --color white|black to skip auto-pick." - ) - - with Image.open(png_path) as img: - img = img.convert("RGB") - w, h = img.size - left = int(w * SUBTITLE_BAND_SIDE) - right = int(w * (1.0 - SUBTITLE_BAND_SIDE)) - top = int(h * SUBTITLE_BAND_TOP) - bottom = int(h * SUBTITLE_BAND_BOTTOM) - band = img.crop((left, top, right, bottom)) - - # Downsample first — averaging 1.7M pixels per slide adds up across - # a long deck. A 64×16 strip preserves enough fidelity for a binary - # white-vs-black decision and is ~1000× cheaper. - band = band.resize((64, 16), Image.BILINEAR) - pixels = list(band.getdata()) - - total_luma = 0.0 - for r, g, b in pixels: - total_luma += 0.299 * r + 0.587 * g + 0.114 * b - mean_luma = total_luma / len(pixels) - - color = COLOR_BLACK if mean_luma >= LUMA_THRESHOLD else COLOR_WHITE - return color, mean_luma - - -def pick_slide_colors(mp4: Path, slide_midpoints: list[float], - ffmpeg: str, default: str | None) -> list[str]: - """Return one color hex per slide, in order. - - `default` short-circuits the probe: if the user passed `--color white` - or `--color black`, we honor it and skip frame extraction entirely. - """ - if default in (COLOR_WHITE, COLOR_BLACK): - return [default] * len(slide_midpoints) - - colors: list[str] = [] - with tempfile.TemporaryDirectory(prefix="subtitle_probe_") as td: - td_path = Path(td) - for i, t in enumerate(slide_midpoints, start=1): - frame = td_path / f"probe_{i:03d}.png" - ok = extract_frame_at(mp4, t, frame, ffmpeg) - if not ok: - # If the seek failed (rare — usually only on the very last - # second of the video), fall back to white. Better to be a - # bit harder to read on one slide than to crash the run. - print(f"[add_subtitles] warn: could not sample frame at t={t:.2f}s " - f"for slide {i}; defaulting to white text.", file=sys.stderr) - colors.append(COLOR_WHITE) - continue - color, luma = pick_color_for_frame(frame) - colors.append(color) - label = "BLACK" if color == COLOR_BLACK else "WHITE" - print(f"[add_subtitles] slide {i:02d} t={t:6.2f}s " - f"band luma={luma:5.1f} → {label}") - return colors - - -# --------------------------------------------------------------------------- -# Cue timing -# --------------------------------------------------------------------------- - -@dataclass -class Cue: - index: int - start: float - end: float - text: str - color: str = COLOR_WHITE # one of COLOR_WHITE / COLOR_BLACK - - -def allocate_slide_cues(cues: list[str], audio_duration: float, - slide_start: float, - min_cue_dur: float, min_gap: float) -> list[tuple[float, float, str]]: - """Distribute one slide's audio_duration over its cue chunks. - - Cue durations are proportional to character length — long sentences get - more screen time. Each cue gets at least `min_cue_dur` seconds so flashes - don't blink past the viewer. After clamping, durations are rescaled to fit - exactly into [slide_start, slide_start + audio_duration]. A tiny `min_gap` - is reserved between adjacent cues so players don't merge them visually. - """ - if not cues: - return [] - if audio_duration <= 0: - return [] - - char_counts = [max(len(c), 1) for c in cues] - total = sum(char_counts) - raw = [audio_duration * n / total for n in char_counts] - - # Floor each cue to min_cue_dur. If the sum exceeds audio_duration we - # have to give up the floor for some cues — accept that, otherwise the - # last cue would spill past the slide boundary. - floored = [max(d, min_cue_dur) for d in raw] - if sum(floored) > audio_duration: - # Too many cues to fit min duration — fall back to even split. - per = max(audio_duration / len(cues), 0.4) - floored = [per] * len(cues) - - scale = audio_duration / sum(floored) if sum(floored) > 0 else 1.0 - if scale < 1.0: - floored = [d * scale for d in floored] - - out: list[tuple[float, float, str]] = [] - t = slide_start - for c, d in zip(cues, floored): - start = t - end = t + max(d - min_gap, 0.4) - out.append((start, end, c)) - t = start + d - # Make sure the last cue ends no later than the slide audio ends. - slide_end = slide_start + audio_duration - last_s, last_e, last_t = out[-1] - if last_e > slide_end: - out[-1] = (last_s, slide_end, last_t) - return out - - -# --------------------------------------------------------------------------- -# SRT formatting -# --------------------------------------------------------------------------- - -def fmt_ts(seconds: float) -> str: - """SRT timestamp: HH:MM:SS,mmm (note the comma decimal separator).""" - if seconds < 0: - seconds = 0.0 - h = int(seconds // 3600) - m = int((seconds % 3600) // 60) - s = seconds % 60 - return f"{h:02d}:{m:02d}:{s:06.3f}".replace(".", ",") - - -def write_srt(cues: list[Cue], path: Path) -> None: - """Write cues to an SRT file with per-cue `<font color="...">` tags. - - The font tag is the de-facto SRT styling convention. ffmpeg's mov_text - encoder parses it and stores the color in the tx3g style atom, so the - soft-track mux carries the color through to playback. Players that - don't understand the tag (very old hardware decoders) just render the - inner text as plain — the cue stays legible, only the styling is lost. - """ - lines: list[str] = [] - for c in cues: - lines.append(str(c.index)) - lines.append(f"{fmt_ts(c.start)} --> {fmt_ts(c.end)}") - # Soft-wrap long cues at ~42 chars on a clause boundary for two-line - # display in players that respect the SRT newline. - wrapped = _soft_wrap(c.text, target=42) - lines.append(f'<font color="{c.color}">{wrapped}</font>') - lines.append("") - path.write_text("\n".join(lines), encoding="utf-8") - - -def fmt_vtt_ts(seconds: float) -> str: - """WebVTT timestamp: HH:MM:SS.mmm.""" - if seconds < 0: - seconds = 0.0 - h = int(seconds // 3600) - m = int((seconds % 3600) // 60) - s = seconds % 60 - return f"{h:02d}:{m:02d}:{s:06.3f}" - - -def write_vtt(cues: list[Cue], path: Path) -> None: - lines: list[str] = ["WEBVTT", ""] - for c in cues: - lines.append(f"{fmt_vtt_ts(c.start)} --> {fmt_vtt_ts(c.end)}") - lines.append(_soft_wrap(c.text, target=42)) - lines.append("") - path.write_text("\n".join(lines), encoding="utf-8") - - -def _soft_wrap(text: str, target: int) -> str: - """Insert at most one newline near `target` chars at a word boundary. - - Most players render at most two lines of SRT cleanly. Anything longer - overflows or shrinks the font. We aim for two short lines, not three. - """ - if len(text) <= target: - return text - # Find the space closest to `target` without exceeding ~1.4× target. - cap = int(target * 1.4) - if len(text) <= cap: - # short enough to leave on one line - return text - # break at the space nearest to `target` - split_at = text.rfind(" ", 0, target + 10) - if split_at < target // 2: - split_at = text.find(" ", target) - if split_at <= 0: - return text - return text[:split_at].rstrip() + "\n" + text[split_at + 1:].lstrip() - - -# --------------------------------------------------------------------------- -# Mux SRT into MP4 as mov_text soft track -# --------------------------------------------------------------------------- - -def mux_subtitles(mp4: Path, srt: Path, out: Path, language: str, - title: str, ffmpeg: str) -> None: - """Stream-copy video+audio, encode SRT as mov_text into the output MP4.""" - out.parent.mkdir(parents=True, exist_ok=True) - cmd = [ - ffmpeg, "-y", - "-i", str(mp4), - "-f", "srt", "-i", str(srt), - "-map", "0:v:0", - "-map", "0:a:0?", - "-map", "1:0", - "-c:v", "copy", - "-c:a", "copy", - "-c:s", "mov_text", - "-metadata:s:s:0", f"language={language}", - "-metadata:s:s:0", f"title={title}", - "-disposition:s:0", "default", - "-movflags", "+faststart", - str(out), - ] - proc = subprocess.run(cmd, capture_output=True, text=True) - if proc.returncode != 0: - sys.exit(f"[add_subtitles] ffmpeg mux failed:\n{proc.stderr}") - - -# --------------------------------------------------------------------------- -# Hardsub — burn cues into the video pixels via libass -# --------------------------------------------------------------------------- - -def _hex_to_ass_color(hex_rgb: str, *, alpha: int = 0) -> str: - """SRT/CSS hex `#RRGGBB` → ASS `&HAABBGGRR&`. - - ASS color literals are little-endian BGR with a one-byte alpha prefix. - Alpha is inverse opacity: 00 is fully opaque and FF is fully transparent. - Picking the wrong byte order is the single most common ASS bug — a - "white" sub that renders red is the R and B bytes swapped. - """ - h = hex_rgb.lstrip("#") - r, g, b = h[0:2], h[2:4], h[4:6] - alpha = max(0, min(255, int(alpha))) - return f"&H{alpha:02X}{b}{g}{r}".upper() + "&" - - -def _opacity_to_ass_alpha(opacity: float) -> int: - """Convert CSS-like opacity to ASS inverse alpha.""" - opacity = max(0.0, min(1.0, float(opacity))) - return int(round((1.0 - opacity) * 255)) - - -def _ass_escape(text: str) -> str: - """Escape characters that have special meaning in ASS dialogue lines. - - ASS uses `{...}` for overrides and `\\N` for hard line breaks. We turn - our SRT-style newlines into `\\N` and neutralize literal braces. - """ - return (text.replace("\\", "\\\\") - .replace("{", "\\{") - .replace("}", "\\}") - .replace("\n", "\\N")) - - -def _ass_timestamp(seconds: float) -> str: - """ASS timestamp: H:MM:SS.cc (centiseconds, single-digit hour).""" - if seconds < 0: - seconds = 0.0 - h = int(seconds // 3600) - m = int((seconds % 3600) // 60) - s = seconds - h * 3600 - m * 60 - return f"{h}:{m:02d}:{s:05.2f}" - - -def write_ass(cues: list[Cue], path: Path, *, - video_w: int, video_h: int, - font_name: str, font_size: int, - outline_width: float, shadow_depth: float, - subtitle_box: bool, subtitle_bar: bool, - subtitle_bar_height: int, box_opacity: float, - box_padding: float) -> None: - """Emit an ASS subtitle file with per-event color overrides. - - The style block sets sensible defaults (font, size, outline/background). - By default paper2video uses a translucent dark box so burned-in subtitles - do not visually merge with PPT text. Bottom-bar mode reserves a solid black - band below a proportionally scaled slide; legacy no-box mode keeps the - per-event outline fallback for users who explicitly want plain captions. - - PlayResX/PlayResY MUST match the output frame size or libass renders - at the wrong scale (text either tiny or oversized). We pass video_w/h - in from the probed input MP4. - """ - box_alpha = _opacity_to_ass_alpha(box_opacity) - box_color = _hex_to_ass_color("#101820", alpha=box_alpha) - if subtitle_bar: - border_style = 1 - style_outline = 0.0 - style_shadow = 0.0 - style_primary = _hex_to_ass_color(COLOR_WHITE) - style_outline_color = _hex_to_ass_color(COLOR_BLACK) - style_back_color = _hex_to_ass_color(COLOR_BLACK) - margin_v = max(20, int(round(subtitle_bar_height * 0.34))) - elif subtitle_box: - border_style = 3 - style_outline = max(0.0, float(box_padding)) - style_shadow = 0.0 - style_primary = _hex_to_ass_color(COLOR_WHITE) - style_outline_color = box_color - style_back_color = box_color - margin_v = 60 - else: - border_style = 1 - style_outline = outline_width - style_shadow = shadow_depth - style_primary = _hex_to_ass_color(COLOR_WHITE) - style_outline_color = _hex_to_ass_color(COLOR_BLACK) - style_back_color = _hex_to_ass_color(COLOR_BLACK, alpha=128) - margin_v = 60 - - header = ( - "[Script Info]\n" - "ScriptType: v4.00+\n" - f"PlayResX: {video_w}\n" - f"PlayResY: {video_h}\n" - "ScaledBorderAndShadow: yes\n" - "WrapStyle: 0\n" - "\n" - "[V4+ Styles]\n" - "Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, " - "OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, " - "ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, " - "Alignment, MarginL, MarginR, MarginV, Encoding\n" - # Bold=0; Alignment=2 = bottom-center. Bottom-bar mode centers captions - # within its reserved band; overlay modes retain the historical 60px - # bottom margin so they do not collide with player chrome. - f"Style: Default,{font_name},{font_size},{style_primary},&H000000FF&," - f"{style_outline_color},{style_back_color},0,0,0,0,100,100,0,0," - f"{border_style},{style_outline},{style_shadow},2,40,40,{margin_v},1\n" - "\n" - "[Events]\n" - "Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, " - "Effect, Text\n" - ) - - body_lines: list[str] = [] - for c in cues: - if subtitle_box or subtitle_bar: - # Dark-box and black-bar modes intentionally keep cue text white. - # Their background is the contrast mechanism; using auto-picked - # black text would make the final captions hard to read. - override = f"{{\\c{_hex_to_ass_color(COLOR_WHITE)}}}" - else: - primary = _hex_to_ass_color(c.color) - # Outline is the opposite color — a black-text cue gets a white halo - # and vice versa. That gives the cue a defensive readable edge if - # the auto-pick lands on a slide whose actual background luma is - # close to the threshold. - opposite = COLOR_WHITE if c.color == COLOR_BLACK else COLOR_BLACK - outline = _hex_to_ass_color(opposite) - override = f"{{\\c{primary}\\3c{outline}}}" - text = override + _ass_escape(c.text) - line = ( - f"Dialogue: 0,{_ass_timestamp(c.start)},{_ass_timestamp(c.end)}," - f"Default,,0,0,0,,{text}" - ) - body_lines.append(line) - - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(header + "\n".join(body_lines) + "\n", encoding="utf-8") - - -def probe_video_dimensions(mp4: Path, ffmpeg: str) -> tuple[int, int]: - """Return (width, height) from the input MP4 by parsing ffmpeg's stderr.""" - out = subprocess.run([ffmpeg, "-i", str(mp4)], capture_output=True, text=True) - # Stream line looks like: ... Video: h264 (High) ... 1920x1080 [SAR ...] - m = re.search(r"Video:.*?(\d{2,5})x(\d{2,5})", out.stderr) - if not m: - # Sensible default — render_video.py's 1080p preset. Better to log - # and continue than to crash on a corner-case ffmpeg build whose - # stderr formatting differs. - print(f"[add_subtitles] warn: could not parse video dimensions from {mp4.name}; " - f"falling back to 1920x1080.", file=sys.stderr) - return 1920, 1080 - return int(m.group(1)), int(m.group(2)) - - -def subtitle_bar_geometry( - video_w: int, video_h: int, bar_height: int, -) -> tuple[int, int, int, int]: - """Return an even, aspect-preserving slide frame above a bottom bar. - - The final frame keeps the input resolution. The full slide is scaled - proportionally into the space above the bar, centered horizontally, and - never cropped or covered by captions. - """ - - if video_w < 2 or video_h < 2: - raise ValueError("video dimensions must be positive") - if bar_height <= 0 or bar_height >= video_h: - raise ValueError("subtitle bar height must fit inside the video frame") - content_h = max(2, video_h - bar_height) - content_h -= content_h % 2 - content_w = max(2, int(video_w * (content_h / video_h))) - content_w -= content_w % 2 - content_w = min(video_w - (video_w % 2), content_w) - x = max(0, (video_w - content_w) // 2) - return content_w, content_h, x, video_h - content_h - - -def burn_subtitles(mp4: Path, ass: Path, out: Path, ffmpeg: str, *, - crf: int, preset: str, - video_w: int, video_h: int, - subtitle_bar_height: int = 0) -> None: - """Re-encode the video with the ASS file rendered onto every frame. - - The `ass=` filter needs an OS path; on Linux/macOS we pass it verbatim, - on Windows ffmpeg requires backslash-escaped drive letters (`C\\:`). We - use the resolved absolute path so the filter doesn't depend on CWD. - - Bottom-bar mode first scales the complete slide proportionally into the - upper region and pads the remaining frame black. Audio is stream-copied — - there's no reason to re-encode the AAC track a second time and it would - only add drift. - """ - out.parent.mkdir(parents=True, exist_ok=True) - ass_abs = str(ass.resolve()) - # Filter-graph path escaping: colons and backslashes are filter syntax. - # Forward slashes are safe on Linux/macOS; on Windows we'd need extra - # escaping. Our pipeline doesn't target Windows for rendering, so the - # POSIX path passes through cleanly. - filters: list[str] = [] - if subtitle_bar_height: - content_w, content_h, x, _ = subtitle_bar_geometry( - video_w, video_h, subtitle_bar_height, - ) - filters.extend([ - f"scale={content_w}:{content_h}:flags=lanczos", - f"pad={video_w}:{video_h}:{x}:0:color=black", - ]) - filters.append(f"ass={ass_abs}") - cmd = [ - ffmpeg, "-y", - "-i", str(mp4), - "-vf", ",".join(filters), - "-c:v", "libx264", "-preset", preset, "-crf", str(crf), - "-pix_fmt", "yuv420p", - "-c:a", "copy", - "-movflags", "+faststart", - str(out), - ] - proc = subprocess.run(cmd, capture_output=True, text=True) - if proc.returncode != 0: - sys.exit(f"[add_subtitles] ffmpeg burn-in failed:\n{proc.stderr}") - - -def copy_without_subtitles(mp4: Path, out: Path, ffmpeg: str) -> None: - """Stream-copy the primary video/audio streams and exclude captions. - - The source is normally render_video.py's raw MP4. Explicit stream maps and - ``-sn`` keep this delivery correct even if a retried job accidentally - supplies an MP4 that already contains a soft subtitle track. - """ - - out.parent.mkdir(parents=True, exist_ok=True) - same_path = mp4.resolve() == out.resolve() - target = ( - out.with_name(f".{out.stem}.no-subtitles.tmp{out.suffix}") - if same_path - else out - ) - cmd = [ - ffmpeg, "-y", - "-i", str(mp4), - "-map", "0:v:0", - "-map", "0:a?", - "-c", "copy", - "-sn", "-dn", - str(target), - ] - proc = subprocess.run(cmd, capture_output=True, text=True) - if proc.returncode != 0: - target.unlink(missing_ok=True) - sys.exit(f"[add_subtitles] ffmpeg no-subtitles copy failed:\n{proc.stderr}") - if same_path: - target.replace(out) - - -# --------------------------------------------------------------------------- -# Project file gathering -# --------------------------------------------------------------------------- - -def collect_notes(project_path: Path) -> list[Path]: - notes_dir = project_path / "notes" - if not notes_dir.is_dir(): - sys.exit(f"[add_subtitles] notes/ not found under {project_path}") - md = sorted(p for p in notes_dir.glob("*.md") if p.name != "total.md") - if not md: - sys.exit(f"[add_subtitles] notes/ has no per-slide *.md files (only total.md?). " - f"Run ppt-master's total_md_split.py first.") - return md - - -def _load_script_sections(script_json: Path) -> list[dict]: - try: - payload = json.loads(script_json.read_text(encoding="utf-8")) - except json.JSONDecodeError as exc: - sys.exit(f"[add_subtitles] invalid script JSON {script_json}: {exc}") - - sections = payload.get("sections") or [] - if not isinstance(sections, list) or not sections: - sys.exit(f"[add_subtitles] script JSON has no sections array: {script_json}") - - out: list[dict] = [] - for idx, sec in enumerate(sections, start=1): - if not isinstance(sec, dict) or not sec.get("id"): - sys.exit(f"[add_subtitles] script section {idx} is missing an id in {script_json}") - out.append(sec) - return out - - -def autodetect_script_json(project_path: Path, audio_dir: Path) -> Path | None: - for candidate in ( - audio_dir / "script.json", - project_path / "assets" / "meta" / "narration.json", - project_path / "narration.json", - ): - if candidate.is_file(): - return candidate - return None - - -def collect_audio(audio_dir: Path) -> list[Path]: - if not audio_dir.is_dir(): - sys.exit(f"[add_subtitles] audio dir not found: {audio_dir}") - mp3s = sorted(audio_dir.glob("*.mp3")) - if not mp3s: - sys.exit(f"[add_subtitles] no .mp3 files in {audio_dir}") - return mp3s - - -def collect_timed_inputs( - project_path: Path, - audio_dir: Path, - script_json: Path | None, -) -> list[tuple[str, str, Path]]: - """Return (id, subtitle_text, mp3_path) in the timeline order. - - Without a script JSON this preserves the original ppt-master behavior: - pair sorted notes/*.md with sorted audio/*.mp3. With a script JSON, use - `sections` order and fall back to each section's `text` when notes/<id>.md - is not present. That lets paper2assets narration.json drive subtitles - without requiring synthetic notes files. - """ - if script_json is None: - script_json = autodetect_script_json(project_path, audio_dir) - - if script_json is None: - notes = collect_notes(project_path) - mp3s = collect_audio(audio_dir) - if len(notes) != len(mp3s): - sys.exit( - f"[add_subtitles] notes/audio count mismatch: " - f"{len(notes)} notes vs {len(mp3s)} mp3s.\n" - f" Notes: {[p.name for p in notes]}\n" - f" Audio: {[p.name for p in mp3s]}\n" - f" Subtitle timing requires 1-to-1 alignment. Regenerate the missing side." - ) - rows: list[tuple[str, str, Path]] = [] - for note_md, mp3 in zip(notes, mp3s): - if note_md.stem != mp3.stem: - print(f"[add_subtitles] warn: notes stem '{note_md.stem}' != audio stem '{mp3.stem}' " - f"— pairing by sort order. Check for renamed/missing files if subtitles drift.", - file=sys.stderr) - rows.append((note_md.stem, note_md.read_text(encoding="utf-8"), mp3)) - print("[add_subtitles] subtitle order from sorted notes/audio filenames") - return rows - - script_json = script_json.resolve() - notes_dir = project_path / "notes" - rows = [] - missing_audio = [] - for sec in _load_script_sections(script_json): - sid = str(sec["id"]) - mp3 = audio_dir / f"{sid}.mp3" - if not mp3.is_file(): - missing_audio.append(mp3.name) - continue - note_md = notes_dir / f"{sid}.md" - if note_md.is_file(): - text = note_md.read_text(encoding="utf-8") - else: - text = str(sec.get("text") or "") - rows.append((sid, text, mp3)) - - if missing_audio: - sys.exit( - f"[add_subtitles] script/audio mismatch using {script_json}:\n" - f" missing mp3s under {audio_dir}: {missing_audio}" - ) - if not rows: - sys.exit(f"[add_subtitles] no usable sections in {script_json}") - print(f"[add_subtitles] subtitle order from {script_json}") - return rows - - -def autodetect_mp4(exports_dir: Path) -> Path: - """Pick the newest *.mp4 in exports/ that isn't already a _subbed.mp4.""" - if not exports_dir.is_dir(): - sys.exit(f"[add_subtitles] exports/ not found: {exports_dir}") - candidates = [p for p in exports_dir.glob("*.mp4") if not p.stem.endswith("_subbed")] - if not candidates: - sys.exit(f"[add_subtitles] no MP4 in {exports_dir} — run render_video.py first or pass --mp4") - # Newest by mtime - return max(candidates, key=lambda p: p.stat().st_mtime) - - -# --------------------------------------------------------------------------- -# Main -# --------------------------------------------------------------------------- - -def main() -> int: - ap = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0]) - ap.add_argument("project_path", help="ppt-master project root (contains notes/, audio/, exports/)") - ap.add_argument("--mp4", default=None, - help="Input MP4 (default: newest non-_subbed.mp4 in <project>/exports)") - ap.add_argument("--audio-dir", default=None, - help="Per-slide audio directory (default: <project>/audio)") - ap.add_argument("--script-json", default=None, - help="Narration script JSON whose sections order defines subtitle/audio order. " - "Defaults to <audio-dir>/script.json, then <project>/assets/meta/narration.json, then <project>/narration.json, " - "then sorted notes/audio filenames.") - ap.add_argument("--out", default=None, - help="Output MP4 path (default: <project>/exports/<mp4_stem>_subbed.mp4)") - ap.add_argument("--srt-out", default=None, - help="Standalone SRT path (default: <project>/exports/<mp4_stem>.srt)") - ap.add_argument("--vtt-out", default=None, - help="Standalone WebVTT path (default: <project>/exports/<mp4_stem>.vtt)") - ap.add_argument("--start-pad", type=float, default=0.5, - help="Lead-in silence used by render_video.py (default: 0.5s — MUST match)") - ap.add_argument("--pad-tail", type=float, default=0.3, - help="Per-slide trailing silence used by render_video.py (default: 0.3s — MUST match)") - ap.add_argument("--max-chars-per-cue", type=int, default=85, - help="Soft cap on cue length before clause-level chunking (default: 85)") - ap.add_argument("--min-cue-duration", type=float, default=1.2, - help="Floor on per-cue screen time (default: 1.2s)") - ap.add_argument("--min-cue-gap", type=float, default=0.08, - help="Visible gap between adjacent cues (default: 0.08s)") - ap.add_argument("--language", default="eng", - help="ISO 639-2 language tag for the mov_text stream (default: eng)") - ap.add_argument("--track-title", default="Narration", - help="Display name for the subtitle track (default: Narration)") - ap.add_argument("--color", choices=("auto", "white", "black"), default="auto", - help="Subtitle text color (default: auto — sample each slide's " - "subtitle band and pick white-on-dark or black-on-light).") - ap.add_argument("--soft", action="store_true", - help="Soft-mux as mov_text instead of burning into pixels. " - "Use this if you want a toggleable caption track and don't " - "mind that some players (mobile previews, embeds) ignore it.") - ap.add_argument("--no-subtitles", action="store_true", - help="Keep the output MP4 caption-free while still writing SRT/VTT " - "for the internal timeline and QA. Video/audio are stream-copied.") - ap.add_argument("--font", default="DejaVu Sans", - help="Font for burned-in subtitles (default: DejaVu Sans — present " - "on most Linux installs; pick a system font you actually have).") - ap.add_argument("--font-size", type=int, default=44, - help="Burn-in font size in ASS units (default: 44 — readable at 1080p).") - ap.add_argument("--outline-width", type=float, default=2.0, - help="Stroke width around burn-in text (default: 2.0).") - ap.add_argument("--shadow-depth", type=float, default=0.5, - help="Drop-shadow depth for burn-in text (default: 0.5).") - ap.add_argument("--subtitle-box", dest="subtitle_box", action="store_true", default=True, - help="Burn subtitles with a translucent dark background box (default).") - ap.add_argument("--no-subtitle-box", dest="subtitle_box", action="store_false", - help="Legacy mode: burn plain text with outline/shadow and no background box.") - ap.add_argument("--subtitle-bar", action="store_true", - help="Burn white captions into a solid black bottom band. The complete " - "slide is scaled proportionally above the band so PPT content is " - "never covered or cropped.") - ap.add_argument("--subtitle-bar-height", type=float, default=0.16, - help="Bottom band height as a fraction of the output frame, 0.10..0.30 " - "(default: 0.16). Used only with --subtitle-bar.") - ap.add_argument("--subtitle-box-opacity", type=float, default=0.62, - help="Opacity for the subtitle background box, 0..1 (default: 0.62).") - ap.add_argument("--subtitle-box-padding", type=float, default=10.0, - help="ASS opaque-box padding around subtitle text (default: 10).") - ap.add_argument("--crf", type=int, default=20, - help="x264 CRF for the burn-in re-encode (default: 20 — visually " - "lossless for screen content).") - ap.add_argument("--preset", default="medium", - help="x264 preset for the burn-in re-encode (default: medium).") - ap.add_argument("--srt-only", action="store_true", - help="Write the .srt and exit (skip muxing/burning).") - args = ap.parse_args() - delivery_modes = sum(bool(value) for value in ( - args.soft, args.subtitle_bar, args.no_subtitles, - )) - if delivery_modes > 1: - ap.error("--soft, --subtitle-bar, and --no-subtitles are mutually exclusive") - if not 0.10 <= args.subtitle_bar_height <= 0.30: - ap.error("--subtitle-bar-height must be between 0.10 and 0.30") - - project_path = Path(args.project_path).resolve() - if not project_path.is_dir(): - sys.exit(f"[add_subtitles] project path not found: {project_path}") - - exports_dir = project_path / "exports" - mp4_path = Path(args.mp4).resolve() if args.mp4 else autodetect_mp4(exports_dir) - if not mp4_path.is_file(): - sys.exit(f"[add_subtitles] MP4 not found: {mp4_path}") - - audio_dir = Path(args.audio_dir).resolve() if args.audio_dir else project_path / "audio" - script_json = Path(args.script_json).resolve() if args.script_json else None - timed_inputs = collect_timed_inputs(project_path, audio_dir, script_json) - - ffmpeg, ffprobe = find_ffmpeg_pair() - - # Walk slides, advancing the clock just like render_video.py does. - # We collect per-slide cue groups together with each slide's midpoint - # timestamp, then probe the slide colors in one pass and apply them - # back to the cues. Two-pass keeps the audio probing and frame probing - # cleanly separated, and means a `--color white|black` short-circuit - # avoids touching ffmpeg's frame extractor at all. - t = max(args.start_pad, 0.0) - next_index = 1 - pending: list[tuple[int, list[Cue], float]] = [] # (slide_idx, cues, midpoint) - - for slide_idx, (sid, text, mp3) in enumerate(timed_inputs, start=1): - duration = probe_duration(mp3, ffprobe, ffmpeg) - cues = split_into_cues(text, args.max_chars_per_cue) - - if not cues: - # Skip silent / empty notes but still advance the clock. - t += duration + args.pad_tail - continue - - timed = allocate_slide_cues( - cues, duration, slide_start=t, - min_cue_dur=args.min_cue_duration, - min_gap=args.min_cue_gap, - ) - slide_cues: list[Cue] = [] - for start, end, txt in timed: - slide_cues.append(Cue(index=next_index, start=start, end=end, text=txt)) - next_index += 1 - - midpoint = t + duration / 2.0 - pending.append((slide_idx, slide_cues, midpoint)) - - # Advance past this slide's audio + the trailing silence pad. - t += duration + args.pad_tail - - if not pending: - sys.exit("[add_subtitles] no cues generated — every note file was empty.") - - # Per-slide color decision. `--color white|black` forces a single color - # across the whole deck and skips the probe pass entirely. - forced = ( - COLOR_WHITE - if args.subtitle_bar - else {"white": COLOR_WHITE, "black": COLOR_BLACK, "auto": None}[args.color] - ) - midpoints = [m for _, _, m in pending] - colors = pick_slide_colors(mp4_path, midpoints, ffmpeg, default=forced) - - all_cues: list[Cue] = [] - for (slide_idx, slide_cues, _), color in zip(pending, colors): - for c in slide_cues: - c.color = color - all_cues.append(c) - - if not all_cues: - sys.exit("[add_subtitles] no cues generated — every note file was empty.") - - srt_path = (Path(args.srt_out).resolve() if args.srt_out - else exports_dir / f"{mp4_path.stem}.srt") - srt_path.parent.mkdir(parents=True, exist_ok=True) - write_srt(all_cues, srt_path) - print(f"[add_subtitles] wrote {len(all_cues)} cues to {srt_path}") - vtt_path = (Path(args.vtt_out).resolve() if args.vtt_out - else exports_dir / f"{mp4_path.stem}.vtt") - vtt_path.parent.mkdir(parents=True, exist_ok=True) - write_vtt(all_cues, vtt_path) - print(f"[add_subtitles] wrote {len(all_cues)} cues to {vtt_path}") - - if args.srt_only: - return 0 - - out_path = (Path(args.out).resolve() if args.out - else exports_dir / f"{mp4_path.stem}_subbed.mp4") - - if args.no_subtitles: - copy_without_subtitles(mp4_path, out_path, ffmpeg) - print(f"[add_subtitles] copied video/audio without subtitle streams to {out_path}") - print("[add_subtitles] SRT/VTT remain available for timeline and QA.") - return 0 - - if args.soft: - # Soft mov_text track — stream-copy video+audio, no re-encode. - mux_subtitles(mp4_path, srt_path, out_path, - language=args.language, title=args.track_title, ffmpeg=ffmpeg) - print(f"[add_subtitles] muxed soft subtitle track into {out_path}") - print(f"[add_subtitles] toggle in VLC: Subtitle → Sub Track → {args.track_title}") - return 0 - - # Default: burn cues into the video pixels via libass. This is what the - # user means by "part of the video file" — there is no toggle, every - # player on every device shows the captions because they are pixels now. - video_w, video_h = probe_video_dimensions(mp4_path, ffmpeg) - subtitle_bar_height = ( - max(2, int(round(video_h * args.subtitle_bar_height))) - if args.subtitle_bar - else 0 - ) - ass_path = exports_dir / f"{mp4_path.stem}.ass" - write_ass( - all_cues, ass_path, - video_w=video_w, video_h=video_h, - font_name=args.font, font_size=args.font_size, - outline_width=args.outline_width, shadow_depth=args.shadow_depth, - subtitle_box=args.subtitle_box and not args.subtitle_bar, - subtitle_bar=args.subtitle_bar, - subtitle_bar_height=subtitle_bar_height, - box_opacity=args.subtitle_box_opacity, - box_padding=args.subtitle_box_padding, - ) - print(f"[add_subtitles] wrote ASS at {video_w}×{video_h} → {ass_path}") - print(f"[add_subtitles] burning subtitles into pixels (libx264 crf={args.crf} preset={args.preset})…") - burn_subtitles( - mp4_path, ass_path, out_path, ffmpeg, - crf=args.crf, preset=args.preset, - video_w=video_w, video_h=video_h, - subtitle_bar_height=subtitle_bar_height, - ) - print(f"[add_subtitles] burned subtitles into {out_path}") - print(f"[add_subtitles] captions are part of every frame — no player toggle required.") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/ResearchStudio-Reel/skills/paper2video/scripts/build_timeline.py b/ResearchStudio-Reel/skills/paper2video/scripts/build_timeline.py deleted file mode 100644 index 20e471f..0000000 --- a/ResearchStudio-Reel/skills/paper2video/scripts/build_timeline.py +++ /dev/null @@ -1,489 +0,0 @@ -#!/usr/bin/env python3 -"""Build the unified paper2video timeline contract. - -The timeline is the sidecar that ties narration text, audio windows, subtitle -cues, and visual-highlight targets to the same chunk ids. It does not render or -modify video; it only normalizes sidecar files produced by the existing -paper2video pipeline into one auditable contract. -""" - -from __future__ import annotations - -import argparse -import json -import re -import sys -from collections import OrderedDict -from datetime import datetime, timezone -from pathlib import Path -from typing import Any - -SCHEMA_VERSION = "paper2video_timeline.v1" - - -def utc_now() -> str: - return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z") - - -def read_json(path: Path) -> Any: - try: - return json.loads(path.read_text(encoding="utf-8")) - except FileNotFoundError: - sys.exit(f"[build_timeline] file not found: {path}") - except json.JSONDecodeError as exc: - sys.exit(f"[build_timeline] invalid JSON {path}: {exc}") - - -def write_json(path: Path, payload: Any) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") - - -def safe_id(raw: object, fallback: str) -> str: - text = str(raw or "").strip() - if not text: - text = fallback - text = re.sub(r"[^A-Za-z0-9_.-]+", "-", text).strip("-") - return text or fallback - - -def load_script_sections(path: Path) -> list[dict[str, Any]]: - payload = read_json(path) - sections = payload.get("sections") if isinstance(payload, dict) else None - if not isinstance(sections, list) or not sections: - sys.exit(f"[build_timeline] script JSON has no sections array: {path}") - out: list[dict[str, Any]] = [] - for idx, sec in enumerate(sections, start=1): - if not isinstance(sec, dict): - sys.exit(f"[build_timeline] script section {idx} is not an object") - sid = safe_id(sec.get("id"), f"slide-{idx:02d}") - out.append({ - "index": idx, - "id": sid, - "heading": str(sec.get("heading") or sec.get("title") or sid), - "text": str(sec.get("text") or sec.get("script") or "").strip(), - }) - return out - - -def load_section_mapping(path: Path | None) -> tuple[dict[str, str], "OrderedDict[str, list[str]]"]: - """Return primary slide-key -> section id plus explicit section groups. - - Accepted JSON shapes: - - 1. {"problem": [3, 4], "method": ["05_architecture"]} - 2. {"03_sequence_evolution": "problem", "4": "problem"} - - Shape 1 may express overlapping poster sections, for example both - "problem" and "motivation" can contain slide 3. Shape 2 is a single-owner - mapping and is used only to give each slide/chunk a primary section id. - """ - if path is None: - return {}, OrderedDict() - payload = read_json(path) - if not isinstance(payload, dict): - sys.exit(f"[build_timeline] section map must be a JSON object: {path}") - primary: dict[str, str] = {} - groups: "OrderedDict[str, list[str]]" = OrderedDict() - for key, value in payload.items(): - if isinstance(value, list): - section_id = safe_id(key, "section") - groups.setdefault(section_id, []) - for item in value: - groups[section_id].append(str(item)) - primary.setdefault(str(item), section_id) - try: - primary.setdefault(str(int(item)), section_id) - except (TypeError, ValueError): - pass - else: - section_id = safe_id(value, "section") - primary[str(key)] = section_id - try: - primary[str(int(key))] = section_id - except (TypeError, ValueError): - pass - return primary, groups - - -def load_by_slide(path: Path | None, field: str) -> dict[int, dict[str, Any]]: - if path is None: - return {} - payload = read_json(path) - slides = payload.get("slides") if isinstance(payload, dict) else None - if not isinstance(slides, list): - sys.exit(f"[build_timeline] {field} must contain a slides array: {path}") - out: dict[int, dict[str, Any]] = {} - for slide in slides: - if not isinstance(slide, dict): - continue - try: - idx = int(slide.get("index")) - except (TypeError, ValueError): - continue - out[idx] = slide - return out - - -def parse_vtt_time(raw: str) -> float: - match = re.match(r"(?:(\d+):)?(\d+):(\d+(?:\.\d+)?)", raw.strip()) - if not match: - raise ValueError(raw) - hours = int(match.group(1) or 0) - minutes = int(match.group(2)) - seconds = float(match.group(3)) - return hours * 3600 + minutes * 60 + seconds - - -def load_vtt(path: Path | None) -> list[dict[str, Any]]: - if path is None: - return [] - if not path.is_file(): - sys.exit(f"[build_timeline] captions VTT not found: {path}") - text = path.read_text(encoding="utf-8").replace("\r\n", "\n") - cues: list[dict[str, Any]] = [] - for block in [part.strip() for part in text.split("\n\n") if part.strip()]: - if block == "WEBVTT" or block.startswith("WEBVTT\n"): - continue - lines = block.splitlines() - time_idx = next((i for i, line in enumerate(lines) if "-->" in line), None) - if time_idx is None: - continue - left, right = lines[time_idx].split("-->", 1) - try: - start = parse_vtt_time(left.strip().split()[0]) - end = parse_vtt_time(right.strip().split()[0]) - except ValueError: - continue - body = "\n".join(lines[time_idx + 1:]).strip() - if body and end > start: - cues.append({"start": round(start, 3), "end": round(end, 3), "text": body}) - return cues - - -def overlap_captions(captions: list[dict[str, Any]], start: float, end: float) -> list[dict[str, Any]]: - out: list[dict[str, Any]] = [] - for cue in captions: - cue_start = float(cue["start"]) - cue_end = float(cue["end"]) - if cue_end <= start or cue_start >= end: - continue - out.append({ - "start": round(max(cue_start, start), 3), - "end": round(min(cue_end, end), 3), - "chunk_start": round(max(cue_start, start) - start, 3), - "chunk_end": round(min(cue_end, end) - start, 3), - "text": cue["text"], - }) - return out - - -def cue_lookup(visual_slide: dict[str, Any] | None) -> dict[tuple[float, float, str], dict[str, Any]]: - out: dict[tuple[float, float, str], dict[str, Any]] = {} - if not visual_slide: - return out - for cue in visual_slide.get("cues") or []: - if not isinstance(cue, dict): - continue - try: - start = round(float(cue.get("start")), 3) - end = round(float(cue.get("end")), 3) - except (TypeError, ValueError): - continue - target = str(cue.get("target") or "") - out[(start, end, target)] = cue - return out - - -def slide_lookup_keys(slide: dict[str, Any]) -> list[str]: - idx = int(slide["index"]) - return [str(slide.get("id") or ""), str(idx), f"{idx:02d}"] - - -def rebuild_sections_from_groups( - groups: "OrderedDict[str, list[str]]", - slides_out: list[dict[str, Any]], -) -> "OrderedDict[str, dict[str, Any]]": - lookup: dict[str, dict[str, Any]] = {} - for slide in slides_out: - for key in slide_lookup_keys(slide): - if key: - lookup[key] = slide - - rebuilt: "OrderedDict[str, dict[str, Any]]" = OrderedDict() - for section_id, keys in groups.items(): - selected: list[dict[str, Any]] = [] - seen: set[int] = set() - for raw_key in keys: - key = str(raw_key) - slide = lookup.get(key) - if slide is None: - try: - slide = lookup.get(str(int(key))) - except ValueError: - slide = None - if slide is None: - sys.exit(f"[build_timeline] section map references unknown slide {raw_key!r} for section {section_id!r}") - idx = int(slide["index"]) - if idx in seen: - continue - seen.add(idx) - selected.append(slide) - if not selected: - continue - rebuilt[section_id] = { - "id": section_id, - "slide_indices": [int(slide["index"]) for slide in selected], - "slide_ids": [str(slide["id"]) for slide in selected], - "chunk_ids": [chunk["id"] for slide in selected for chunk in slide.get("chunks", [])], - "start": round(min(float(slide["segment"]["start"]) for slide in selected), 3), - "end": round(max(float(slide["segment"]["end"]) for slide in selected), 3), - } - rebuilt[section_id]["seconds"] = round(float(rebuilt[section_id]["end"]) - float(rebuilt[section_id]["start"]), 3) - return rebuilt - - -def build_timeline( - *, - script_json: Path, - duration_report: Path, - visual_cue_plan: Path | None, - visual_cues: Path | None, - captions_vtt: Path | None, - audio_dir: Path | None, - video: Path | None, - section_map_path: Path | None, -) -> dict[str, Any]: - script_sections = load_script_sections(script_json) - report = read_json(duration_report) - report_slides = report.get("slides") if isinstance(report, dict) else None - if not isinstance(report_slides, list) or not report_slides: - sys.exit(f"[build_timeline] duration report has no slides array: {duration_report}") - if len(report_slides) != len(script_sections): - sys.exit( - "[build_timeline] script/duration slide count mismatch: " - f"{len(script_sections)} script sections vs {len(report_slides)} report slides" - ) - - cue_plan_by_slide = load_by_slide(visual_cue_plan, "visual cue plan") - visual_cues_by_slide = load_by_slide(visual_cues, "visual cues") - captions = load_vtt(captions_vtt) - section_map, explicit_section_groups = load_section_mapping(section_map_path) - - start_pad = float(report.get("start_pad") or 0.0) - pad_tail = float(report.get("pad_tail") or 0.0) - cursor = start_pad - slides_out: list[dict[str, Any]] = [] - flat_chunks: list[dict[str, Any]] = [] - sections: "OrderedDict[str, dict[str, Any]]" = OrderedDict() - - for idx, (script_sec, slide_report) in enumerate(zip(script_sections, report_slides), start=1): - slide_index = int(slide_report.get("index") or idx) - if slide_index != idx: - sys.exit(f"[build_timeline] non-sequential slide index in duration report: {slide_index} at position {idx}") - slide_id = script_sec["id"] - section_id = ( - section_map.get(slide_id) - or section_map.get(str(slide_index)) - or section_map.get(f"{slide_index:02d}") - or slide_id - ) - audio_seconds = float(slide_report.get("audio_seconds") or 0.0) - segment_seconds = float(slide_report.get("segment_seconds") or audio_seconds + pad_tail) - segment_start = round(cursor, 3) - segment_end = round(cursor + segment_seconds, 3) - audio_start = segment_start - audio_end = round(segment_start + audio_seconds, 3) - cursor += segment_seconds - - plan_slide = cue_plan_by_slide.get(slide_index, {}) - chunks = plan_slide.get("chunks") if isinstance(plan_slide, dict) else None - if not isinstance(chunks, list) or not chunks: - chunks = [{ - "chunk_index": 1, - "text": script_sec["text"], - "start": 0.0, - "end": audio_seconds, - "seconds": audio_seconds, - "timing_source": "slide_audio", - "accepted": False, - "reason": "visual_cue_plan_missing", - }] - - visual_lookup = cue_lookup(visual_cues_by_slide.get(slide_index)) - chunk_entries: list[dict[str, Any]] = [] - for chunk in chunks: - if not isinstance(chunk, dict): - continue - chunk_num = int(chunk.get("chunk_index") or len(chunk_entries) + 1) - local_start = round(float(chunk.get("start") or 0.0), 3) - local_end = round(float(chunk.get("end") or local_start), 3) - local_start = max(0.0, min(local_start, segment_seconds)) - local_end = max(local_start, min(local_end, segment_seconds)) - global_start = round(segment_start + local_start, 3) - global_end = round(segment_start + local_end, 3) - target = str(chunk.get("target") or "") - visual_cue = visual_lookup.get((round(local_start, 3), round(local_end, 3), target), {}) - visual_payload = None - if chunk.get("accepted") or visual_cue: - visual_payload = { - "accepted": bool(chunk.get("accepted")), - "anchor_id": chunk.get("anchor_id") or visual_cue.get("anchor_id"), - "anchor_matched": bool(chunk.get("anchor_matched") or visual_cue.get("anchor_matched")), - "target": target or visual_cue.get("target"), - "target_role": chunk.get("target_role") or visual_cue.get("target_role"), - "target_source": chunk.get("target_source") or visual_cue.get("target_source"), - "semantic_target": chunk.get("semantic_target") or visual_cue.get("semantic_target") or target or visual_cue.get("target"), - "semantic_original_target": chunk.get("semantic_original_target") or visual_cue.get("semantic_original_target"), - "semantic_promoted": bool(chunk.get("semantic_promoted") or visual_cue.get("semantic_promoted")), - "semantic_promotion": chunk.get("semantic_promotion") or visual_cue.get("semantic_promotion"), - "semantic_role": chunk.get("semantic_role") or visual_cue.get("semantic_role") or chunk.get("target_role") or visual_cue.get("target_role"), - "semantic_source": chunk.get("semantic_source") or visual_cue.get("semantic_source") or chunk.get("target_source") or visual_cue.get("target_source"), - "semantic_box": chunk.get("semantic_box") or visual_cue.get("semantic_box"), - "geometry_target": chunk.get("geometry_target") or visual_cue.get("geometry_target"), - "geometry_role": chunk.get("geometry_role") or visual_cue.get("geometry_role"), - "geometry_source": chunk.get("geometry_source") or visual_cue.get("geometry_source"), - "geometry_box": chunk.get("geometry_box") or visual_cue.get("geometry_box") or visual_cue.get("box"), - "geometry_matched": bool(chunk.get("geometry_matched") or visual_cue.get("geometry_matched")), - "geometry_match_score": chunk.get("geometry_match_score") or visual_cue.get("geometry_match_score"), - "geometry_match_iou": chunk.get("geometry_match_iou") or visual_cue.get("geometry_match_iou"), - "geometry_semantic_coverage": chunk.get("geometry_semantic_coverage") or visual_cue.get("geometry_semantic_coverage"), - "geometry_coverage": chunk.get("geometry_coverage") or visual_cue.get("geometry_coverage"), - "geometry_match_reason": chunk.get("geometry_match_reason") or visual_cue.get("geometry_match_reason"), - "point": chunk.get("point") or visual_cue.get("point"), - "region_box": chunk.get("geometry_box") or visual_cue.get("geometry_box") or chunk.get("region_box") or visual_cue.get("box"), - "confidence": chunk.get("confidence") or visual_cue.get("confidence"), - "reason": chunk.get("reason"), - "timing": chunk.get("timing"), - "cue": visual_cue or None, - } - chunk_id = f"{section_id}.s{slide_index:02d}.c{chunk_num:02d}" - subtitle_cues = overlap_captions(captions, global_start, global_end) - chunk_entry = { - "id": chunk_id, - "section_id": section_id, - "slide_id": slide_id, - "slide_index": slide_index, - "chunk_index": chunk_num, - "text": str(chunk.get("text") or "").strip(), - "timing_source": chunk.get("timing_source") or plan_slide.get("timing_source") or "unknown", - "local_start": local_start, - "local_end": local_end, - "start": global_start, - "end": global_end, - "seconds": round(global_end - global_start, 3), - "audio": { - "file": str((audio_dir / str(slide_report.get("audio") or f"{slide_id}.mp3")).resolve()) if audio_dir else str(slide_report.get("audio") or ""), - "slide_start": audio_start, - "slide_end": audio_end, - }, - "subtitles": subtitle_cues, - "visual_cue": visual_payload, - } - chunk_entries.append(chunk_entry) - flat_chunks.append(chunk_entry) - - slide_entry = { - "index": slide_index, - "id": slide_id, - "heading": script_sec["heading"], - "section_id": section_id, - "audio": { - "file": str((audio_dir / str(slide_report.get("audio") or f"{slide_id}.mp3")).resolve()) if audio_dir else str(slide_report.get("audio") or ""), - "seconds": round(audio_seconds, 3), - "start": audio_start, - "end": audio_end, - }, - "segment": { - "start": segment_start, - "end": segment_end, - "seconds": round(segment_seconds, 3), - "pad_tail": round(max(0.0, segment_seconds - audio_seconds), 3), - }, - "text": script_sec["text"], - "chunks": chunk_entries, - } - slides_out.append(slide_entry) - - sec = sections.setdefault(section_id, { - "id": section_id, - "slide_indices": [], - "slide_ids": [], - "chunk_ids": [], - "start": segment_start, - "end": segment_end, - }) - sec["slide_indices"].append(slide_index) - sec["slide_ids"].append(slide_id) - sec["chunk_ids"].extend([chunk["id"] for chunk in chunk_entries]) - sec["start"] = min(float(sec["start"]), segment_start) - sec["end"] = max(float(sec["end"]), segment_end) - - for sec in sections.values(): - sec["seconds"] = round(float(sec["end"]) - float(sec["start"]), 3) - sec["start"] = round(float(sec["start"]), 3) - sec["end"] = round(float(sec["end"]), 3) - - if explicit_section_groups: - sections = rebuild_sections_from_groups(explicit_section_groups, slides_out) - - actual_seconds = report.get("actual_seconds") - if actual_seconds is None: - actual_seconds = round(cursor, 3) - return { - "schema_version": SCHEMA_VERSION, - "created_at": utc_now(), - "resources": { - "script_json": str(script_json), - "duration_report": str(duration_report), - "visual_cue_plan": str(visual_cue_plan) if visual_cue_plan else None, - "visual_cues": str(visual_cues) if visual_cues else None, - "captions_vtt": str(captions_vtt) if captions_vtt else None, - "audio_dir": str(audio_dir) if audio_dir else None, - "video": str(video) if video else str(report.get("output") or ""), - "section_map": str(section_map_path) if section_map_path else None, - }, - "start_pad": round(start_pad, 3), - "pad_tail": round(pad_tail, 3), - "actual_seconds": round(float(actual_seconds), 3), - "sections": list(sections.values()), - "slides": slides_out, - "chunks": flat_chunks, - } - - -def main() -> int: - parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0]) - parser.add_argument("--script-json", required=True, type=Path) - parser.add_argument("--duration-report", required=True, type=Path) - parser.add_argument("--visual-cue-plan", type=Path) - parser.add_argument("--visual-cues", type=Path) - parser.add_argument("--captions-vtt", type=Path) - parser.add_argument("--audio-dir", type=Path) - parser.add_argument("--video", type=Path) - parser.add_argument("--section-map", type=Path, - help="Optional JSON that maps timeline slides to canonical poster/blog sections.") - parser.add_argument("--out", required=True, type=Path) - args = parser.parse_args() - - timeline = build_timeline( - script_json=args.script_json.resolve(), - duration_report=args.duration_report.resolve(), - visual_cue_plan=args.visual_cue_plan.resolve() if args.visual_cue_plan else None, - visual_cues=args.visual_cues.resolve() if args.visual_cues else None, - captions_vtt=args.captions_vtt.resolve() if args.captions_vtt else None, - audio_dir=args.audio_dir.resolve() if args.audio_dir else None, - video=args.video.resolve() if args.video else None, - section_map_path=args.section_map.resolve() if args.section_map else None, - ) - write_json(args.out.resolve(), timeline) - print( - f"[build_timeline] wrote {args.out.resolve()} " - f"({len(timeline['sections'])} sections, {len(timeline['slides'])} slides, " - f"{len(timeline['chunks'])} chunks)" - ) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/ResearchStudio-Reel/skills/paper2video/scripts/check_video_package.py b/ResearchStudio-Reel/skills/paper2video/scripts/check_video_package.py deleted file mode 100755 index 8778932..0000000 --- a/ResearchStudio-Reel/skills/paper2video/scripts/check_video_package.py +++ /dev/null @@ -1,1856 +0,0 @@ -#!/usr/bin/env python3 -"""Quality gate for a paper2video package. - -This checker is intentionally deterministic: it reads PPTX geometry, audio/video -metadata, visual-cue JSON, and the exported slide frames used by render_video.py. -It is not a replacement for human taste, but it catches the boring red-line -failures before a video is shown to users: missing audio, slide/audio drift, -blank frames, severe text-box overflow, obvious text/image overlap, cue timing -errors, and broken MP4 streams. -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import math -import os -import re -import shutil -import subprocess -import sys -import tempfile -import zipfile -from collections import Counter -from dataclasses import dataclass, field -from datetime import datetime, timezone -from pathlib import Path -from typing import Any -from xml.etree import ElementTree as ET - -try: - from PIL import Image, ImageStat -except Exception: # pragma: no cover - optional dependency - Image = None - ImageStat = None - -try: - import numpy as np -except Exception: # pragma: no cover - optional dependency - np = None - - -EMU_PER_PT = 12700 -EMU_PER_IN = 914400 -NS = { - "p": "http://schemas.openxmlformats.org/presentationml/2006/main", - "a": "http://schemas.openxmlformats.org/drawingml/2006/main", - "r": "http://schemas.openxmlformats.org/officeDocument/2006/relationships", -} -SCHEMA_VERSION = "paper2video_qa.v1" -NON_BLOCKING_WARNING_CODES = frozenset({"audio_extra_files"}) - - -@dataclass -class Finding: - severity: str - code: str - message: str - location: str | None = None - data: dict[str, Any] = field(default_factory=dict) - - -@dataclass -class Box: - slide_index: int - kind: str - name: str - x: float - y: float - w: float - h: float - text: str = "" - - @property - def area(self) -> float: - return max(0.0, self.w) * max(0.0, self.h) - - -def utc_now() -> str: - return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z") - - -def add(findings: list[Finding], severity: str, code: str, message: str, *, location: str | None = None, **data: Any) -> None: - findings.append(Finding(severity=severity, code=code, message=message, location=location, data=data)) - - -def findings_pass_gate( - findings: list[Finding], - *, - fail_on_warning: bool, -) -> bool: - """Return whether findings pass the final package gate. - - Unreferenced MP3s remain visible in the report but do not invalidate the - delivered timeline or MP4. Every other warning remains blocking when - strict/fail-on-warning policy is active, and errors always block. - """ - if any(finding.severity == "error" for finding in findings): - return False - if not fail_on_warning: - return True - return not any( - finding.severity == "warning" - and finding.code not in NON_BLOCKING_WARNING_CODES - for finding in findings - ) - - -def read_json(path: Path) -> Any: - try: - return json.loads(path.read_text(encoding="utf-8")) - except FileNotFoundError: - raise SystemExit(f"[check_video_package] file not found: {path}") - except json.JSONDecodeError as exc: - raise SystemExit(f"[check_video_package] invalid JSON {path}: {exc}") - - -def which(name: str) -> str | None: - return shutil.which(name) - - -def imageio_ffmpeg_binary() -> str | None: - try: - import imageio_ffmpeg # type: ignore - - return imageio_ffmpeg.get_ffmpeg_exe() - except Exception: - return None - - -def find_ffmpeg_pair() -> tuple[str | None, str | None]: - env_ffmpeg = os.getenv("PAPER2VIDEO_FFMPEG") or os.getenv("FFMPEG_BINARY") - if env_ffmpeg and Path(env_ffmpeg).expanduser().is_file(): - env_ffprobe = os.getenv("PAPER2VIDEO_FFPROBE") - if env_ffprobe and Path(env_ffprobe).expanduser().is_file(): - return str(Path(env_ffmpeg).expanduser()), str(Path(env_ffprobe).expanduser()) - return str(Path(env_ffmpeg).expanduser()), str(Path(env_ffmpeg).expanduser()) - - fallback = imageio_ffmpeg_binary() - if fallback: - return fallback, fallback - - return which("ffmpeg"), which("ffprobe") - - -def find_libreoffice() -> str | None: - return which("libreoffice") or which("soffice") - - -def find_pdftoppm() -> str | None: - found = which("pdftoppm") - if found: - return found - sibling = Path(sys.executable).resolve().parent / "pdftoppm" - if sibling.is_file(): - return str(sibling) - return None - - -def render_frames_from_pptx(pptx: Path, work_dir: Path, findings: list[Finding]) -> Path | None: - """Export PPTX to slide PNGs for visual QA. - - This is a fallback for legacy packages that did not archive the actual - render_video.py frames under assets/slides/frames. - """ - libreoffice = find_libreoffice() - pdftoppm = find_pdftoppm() - if not libreoffice: - add(findings, "error", "libreoffice_missing", "LibreOffice/soffice is required to render PPTX frames for QA.") - return None - if not pdftoppm: - add(findings, "error", "pdftoppm_missing", "pdftoppm is required to rasterize the QA PDF into frames.") - return None - if not pptx.is_file(): - add(findings, "error", "pptx_missing_for_frame_render", "Cannot render frames because PPTX is missing.", location=str(pptx)) - return None - - qa_root = work_dir / ".video_qa" - pdf_dir = qa_root / "pdf" - frames_dir = qa_root / "frames" - if frames_dir.exists(): - shutil.rmtree(frames_dir) - if pdf_dir.exists(): - shutil.rmtree(pdf_dir) - pdf_dir.mkdir(parents=True, exist_ok=True) - frames_dir.mkdir(parents=True, exist_ok=True) - - with tempfile.TemporaryDirectory(prefix="lo_profile_") as profile_dir: - cmd = [ - libreoffice, - f"-env:UserInstallation=file://{profile_dir}", - "--headless", "--norestore", "--nologo", - "--convert-to", "pdf", - "--outdir", str(pdf_dir), - str(pptx), - ] - try: - proc = subprocess.run(cmd, capture_output=True, text=True, timeout=600) - except subprocess.TimeoutExpired: - add(findings, "error", "pptx_frame_render_timeout", "LibreOffice timed out while exporting PPTX for QA frames.", location=str(pptx)) - return None - if proc.returncode != 0: - add( - findings, - "error", - "pptx_frame_render_failed", - "LibreOffice failed while exporting PPTX for QA frames.", - location=str(pptx), - stdout=proc.stdout[-1000:], - stderr=proc.stderr[-1000:], - ) - return None - - pdf = pdf_dir / f"{pptx.stem}.pdf" - if not pdf.exists(): - add(findings, "error", "pptx_frame_pdf_missing", "LibreOffice did not produce the expected QA PDF.", location=str(pdf), stdout=proc.stdout[-1000:]) - return None - - cmd = [pdftoppm, "-png", "-r", "120", str(pdf), str(frames_dir / "slide")] - try: - proc = subprocess.run(cmd, capture_output=True, text=True, timeout=600) - except subprocess.TimeoutExpired: - add(findings, "error", "pptx_frame_raster_timeout", "pdftoppm timed out while rasterizing QA frames.", location=str(pdf)) - return None - if proc.returncode != 0: - add(findings, "error", "pptx_frame_raster_failed", "pdftoppm failed while rasterizing QA frames.", location=str(pdf), stderr=proc.stderr[-1000:]) - return None - if not list(frames_dir.glob("slide-*.png")): - add(findings, "error", "pptx_frame_output_empty", "Frame rasterization produced no PNG files.", location=str(frames_dir)) - return None - return frames_dir - - -def probe_duration(path: Path, ffprobe: str | None, ffmpeg: str | None) -> float | None: - if ffprobe and ffmpeg and ffprobe != ffmpeg: - cmd = [ffprobe, "-v", "error", "-show_entries", "format=duration", "-of", "default=noprint_wrappers=1:nokey=1", str(path)] - proc = subprocess.run(cmd, capture_output=True, text=True) - if proc.returncode == 0 and proc.stdout.strip(): - try: - return float(proc.stdout.strip()) - except ValueError: - pass - if ffmpeg: - proc = subprocess.run([ffmpeg, "-i", str(path)], capture_output=True, text=True) - match = re.search(r"Duration:\s+(\d+):(\d+):([\d.]+)", proc.stderr) - if match: - return int(match.group(1)) * 3600 + int(match.group(2)) * 60 + float(match.group(3)) - return None - - -def probe_video_streams(path: Path, ffprobe: str | None, ffmpeg: str | None) -> dict[str, Any]: - out: dict[str, Any] = {"duration": None, "has_video": False, "has_audio": False} - if ffprobe and ffmpeg and ffprobe != ffmpeg: - cmd = [ - ffprobe, - "-v", - "error", - "-show_entries", - "format=duration:stream=codec_type", - "-of", - "json", - str(path), - ] - proc = subprocess.run(cmd, capture_output=True, text=True) - if proc.returncode == 0 and proc.stdout.strip(): - payload = json.loads(proc.stdout) - try: - out["duration"] = float(payload.get("format", {}).get("duration")) - except (TypeError, ValueError): - out["duration"] = None - stream_types = {s.get("codec_type") for s in payload.get("streams", []) if isinstance(s, dict)} - out["has_video"] = "video" in stream_types - out["has_audio"] = "audio" in stream_types - return out - - if ffmpeg: - proc = subprocess.run([ffmpeg, "-i", str(path)], capture_output=True, text=True) - stderr = proc.stderr - dur = re.search(r"Duration:\s+(\d+):(\d+):([\d.]+)", stderr) - if dur: - out["duration"] = int(dur.group(1)) * 3600 + int(dur.group(2)) * 60 + float(dur.group(3)) - out["has_video"] = "Video:" in stderr - out["has_audio"] = "Audio:" in stderr - return out - - -def natural_key(path: Path) -> list[Any]: - return [int(p) if p.isdigit() else p.lower() for p in re.split(r"(\d+)", path.stem)] - - -def load_script(script_json: Path) -> list[dict[str, str]]: - payload = read_json(script_json) - sections = payload.get("sections") if isinstance(payload, dict) else None - if not isinstance(sections, list) or not sections: - raise SystemExit(f"[check_video_package] script JSON has no non-empty sections array: {script_json}") - out: list[dict[str, str]] = [] - for i, sec in enumerate(sections, start=1): - if not isinstance(sec, dict) or not sec.get("id"): - raise SystemExit(f"[check_video_package] script section {i} is missing id") - out.append({"id": str(sec["id"]), "heading": str(sec.get("heading") or ""), "text": str(sec.get("text") or "")}) - return out - - -def list_slide_xml_names(zf: zipfile.ZipFile) -> list[str]: - try: - presentation = ET.fromstring(zf.read("ppt/presentation.xml")) - rels_root = ET.fromstring(zf.read("ppt/_rels/presentation.xml.rels")) - rels = { - rel.attrib.get("Id"): rel.attrib.get("Target") - for rel in rels_root - if rel.tag.rsplit("}", 1)[-1] == "Relationship" - } - ordered = [] - for sld_id in presentation.findall("p:sldIdLst/p:sldId", NS): - rid = sld_id.attrib.get(f"{{{NS['r']}}}id") - target = rels.get(rid) - if not target: - continue - target = target.lstrip("/") - if not target.startswith("ppt/"): - target = f"ppt/{target}" - if target in zf.namelist(): - ordered.append(target) - if ordered: - return ordered - except Exception: - pass - - names = [n for n in zf.namelist() if re.match(r"ppt/slides/slide\d+\.xml$", n)] - return sorted(names, key=lambda n: int(re.search(r"slide(\d+)\.xml$", n).group(1))) # type: ignore[union-attr] - - -def read_slide_size(zf: zipfile.ZipFile) -> tuple[int, int]: - try: - root = ET.fromstring(zf.read("ppt/presentation.xml")) - sld_sz = root.find("p:sldSz", NS) - if sld_sz is not None: - return int(sld_sz.attrib.get("cx", "12192000")), int(sld_sz.attrib.get("cy", "6858000")) - except Exception: - pass - return 12192000, 6858000 - - -def local_name(tag: str) -> str: - return tag.rsplit("}", 1)[-1] - - -def shape_name(el: ET.Element) -> str: - nv = el.find(".//p:cNvPr", NS) - if nv is not None: - return nv.attrib.get("name") or nv.attrib.get("id") or local_name(el.tag) - return local_name(el.tag) - - -def get_xfrm(el: ET.Element) -> tuple[float, float, float, float] | None: - xfrm = el.find(".//a:xfrm", NS) - if xfrm is None: - return None - off = xfrm.find("a:off", NS) - ext = xfrm.find("a:ext", NS) - if off is None or ext is None: - return None - try: - x = float(off.attrib.get("x", "0")) - y = float(off.attrib.get("y", "0")) - w = float(ext.attrib.get("cx", "0")) - h = float(ext.attrib.get("cy", "0")) - except ValueError: - return None - if w <= 0 or h <= 0: - return None - return x, y, w, h - - -def text_of(el: ET.Element) -> str: - return "\n".join(t.text or "" for t in el.findall(".//a:t", NS)).strip() - - -def first_font_pt(el: ET.Element) -> float: - sizes: list[float] = [] - for rpr in el.findall(".//a:rPr", NS): - raw = rpr.attrib.get("sz") - if raw and raw.isdigit(): - sizes.append(int(raw) / 100.0) - if sizes: - return max(6.0, min(max(sizes), 60.0)) - return 18.0 - - -def body_margins_pt(el: ET.Element) -> tuple[float, float, float, float]: - body_pr = el.find(".//a:bodyPr", NS) - defaults = {"lIns": 91440, "rIns": 91440, "tIns": 45720, "bIns": 45720} - vals = [] - for key in ("lIns", "rIns", "tIns", "bIns"): - raw = body_pr.attrib.get(key) if body_pr is not None else None - try: - vals.append(float(raw if raw is not None else defaults[key]) / EMU_PER_PT) - except ValueError: - vals.append(defaults[key] / EMU_PER_PT) - return tuple(vals) # type: ignore[return-value] - - -def char_units(ch: str) -> float: - if ch.isspace(): - return 0.34 - code = ord(ch) - if 0x4E00 <= code <= 0x9FFF or 0x3040 <= code <= 0x30FF or 0xAC00 <= code <= 0xD7AF: - return 1.0 - if ch in "il.,:;|'!": - return 0.28 - if ch in "MW@#%&": - return 0.85 - return 0.55 - - -def estimate_text_lines(text: str, width_pt: float, font_pt: float) -> int: - if not text.strip(): - return 0 - max_units = max(width_pt / max(font_pt, 1.0), 1.0) - lines = 0 - for para in re.split(r"\n+", text): - units = sum(char_units(ch) for ch in para.strip()) - lines += max(1, math.ceil(units / max_units)) - return lines - - -def box_intersection(a: Box, b: Box) -> float: - x1 = max(a.x, b.x) - y1 = max(a.y, b.y) - x2 = min(a.x + a.w, b.x + b.w) - y2 = min(a.y + a.h, b.y + b.h) - return max(0.0, x2 - x1) * max(0.0, y2 - y1) - - -def parse_pptx(pptx: Path, findings: list[Finding]) -> dict[str, Any]: - if not pptx.is_file(): - add(findings, "error", "pptx_missing", "PPTX file is missing.", location=str(pptx)) - return {"slide_count": 0, "slides": []} - - slides: list[dict[str, Any]] = [] - try: - with zipfile.ZipFile(pptx) as zf: - slide_names = list_slide_xml_names(zf) - slide_w, slide_h = read_slide_size(zf) - slide_area = max(float(slide_w) * float(slide_h), 1.0) - for idx, name in enumerate(slide_names, start=1): - root = ET.fromstring(zf.read(name)) - boxes: list[Box] = [] - for el in root.findall(".//p:sp", NS) + root.findall(".//p:pic", NS) + root.findall(".//p:graphicFrame", NS): - xfrm = get_xfrm(el) - if xfrm is None: - continue - kind = local_name(el.tag) - text = text_of(el) if kind == "sp" else "" - if kind == "sp" and text: - kind = "text" - elif kind == "pic": - kind = "image" - elif kind == "graphicFrame": - kind = "graphic" - boxes.append(Box(idx, kind, shape_name(el), *xfrm, text=text)) - - if text: - l_pt, r_pt, t_pt, b_pt = body_margins_pt(el) - font_pt = first_font_pt(el) - usable_w = max((xfrm[2] / EMU_PER_PT) - l_pt - r_pt, 1.0) - usable_h = max((xfrm[3] / EMU_PER_PT) - t_pt - b_pt, 1.0) - lines = estimate_text_lines(text, usable_w, font_pt) - needed_h = lines * font_pt * 1.16 - ratio = needed_h / usable_h if usable_h else 99 - if ratio > 1.22: - # Native SVG->PPTX conversion often represents - # intentionally single-line labels/cards as short - # text boxes whose text may visually extend within - # the designed row. Keep the XML estimate in the - # report for agent review, but only make it a hard - # finding when the estimate is extreme enough that - # it is likely to survive into the rendered frame. - compact_single_line = usable_h <= font_pt * 1.75 and len(text) <= 140 - sev = "error" if ratio > 4.0 or (ratio > 3.8 and not compact_single_line) else "info" - add( - findings, - sev, - "ppt_text_overflow_risk", - "Text likely exceeds its PPTX text box.", - location=f"slide {idx}: {shape_name(el)}", - needed_height_pt=round(needed_h, 1), - box_height_pt=round(usable_h, 1), - overflow_ratio=round(ratio, 2), - font_pt=round(font_pt, 1), - text_preview=re.sub(r"\s+", " ", text)[:180], - ) - - text_boxes = [b for b in boxes if b.kind == "text" and b.text.strip()] - image_boxes = [b for b in boxes if b.kind in {"image", "graphic"}] - visual_metrics = { - "visual_count": len(image_boxes), - "max_visual_axis_fill": 0.0, - "max_visual_area_ratio": 0.0, - "total_visual_area_ratio": 0.0, - "small_visual_count": 0, - } - if image_boxes: - axis_fills = [max(b.w / slide_w, b.h / slide_h) for b in image_boxes] - area_ratios = [b.area / slide_area for b in image_boxes] - visual_metrics = { - "visual_count": len(image_boxes), - "max_visual_axis_fill": round(max(axis_fills), 4), - "max_visual_area_ratio": round(max(area_ratios), 4), - "total_visual_area_ratio": round(sum(area_ratios), 4), - "small_visual_count": sum(1 for fill, area in zip(axis_fills, area_ratios) if fill < 0.24 and area < 0.035), - } - # Ignore purely textual/title slides, but when a slide does - # contain non-zero-area visuals they must be large enough to - # be useful in a 1080p video frame. SVG->PPTX conversion can - # leave zero-size graphicFrame placeholders; report those as - # info instead of failing strict QA on an invisible object. - meaningful_images = [ - b for b in image_boxes - if (b.area / slide_area) >= 0.003 or max(b.w / slide_w, b.h / slide_h) >= 0.08 - ] - small_meaningful_count = sum( - 1 - for b in meaningful_images - if max(b.w / slide_w, b.h / slide_h) < 0.24 and (b.area / slide_area) < 0.035 - ) - if meaningful_images and small_meaningful_count == len(meaningful_images): - add( - findings, - "warning", - "ppt_visuals_too_small", - "All picture/graphic elements on this slide are small; viewers may not be able to read them in the video.", - location=f"slide {idx}", - **visual_metrics, - meaningful_visual_count=len(meaningful_images), - ) - if ( - len(image_boxes) >= 1 - and visual_metrics["max_visual_axis_fill"] < 0.30 - and visual_metrics["total_visual_area_ratio"] < 0.075 - and len(text_boxes) <= 4 - ): - add( - findings, - "warning", - "slide_visual_story_too_small", - "Slide has a visual, but the visual story occupies too little of the slide.", - location=f"slide {idx}", - **visual_metrics, - ) - for a_i, a in enumerate(text_boxes): - for b in text_boxes[a_i + 1 :]: - inter = box_intersection(a, b) - if inter <= 0: - continue - overlap = inter / max(min(a.area, b.area), 1.0) - # Adjacent PPT text lines often have bounding boxes - # that overlap a little because each line carries font - # ascent/descent slack. Treat only heavy overlap as a - # red-line risk; lighter cases are usually normal line - # stacking, not visible collision. - if overlap > 0.55: - long_pair = len(a.text.strip()) > 80 or len(b.text.strip()) > 80 - add( - findings, - "warning" if long_pair else "info", - "ppt_text_text_overlap", - "Two text boxes substantially overlap.", - location=f"slide {idx}", - first=a.name, - second=b.name, - overlap_ratio=round(overlap, 3), - first_preview=re.sub(r"\s+", " ", a.text)[:120], - second_preview=re.sub(r"\s+", " ", b.text)[:120], - ) - for text_box in text_boxes: - for obj in image_boxes: - inter = box_intersection(text_box, obj) - if inter <= 0: - continue - overlap = inter / max(min(text_box.area, obj.area), 1.0) - if overlap > 0.16: - obj_area_ratio = obj.area / slide_area - obj_axis_fill = max(obj.w / slide_w, obj.h / slide_h) - full_bleed_or_background = obj_area_ratio > 0.30 or obj_axis_fill > 0.75 - small_visual_covered = obj_area_ratio < 0.08 and obj_axis_fill < 0.35 and overlap > 0.45 - add( - findings, - "warning" if small_visual_covered and not full_bleed_or_background else "info", - "ppt_text_visual_overlap", - "Text overlaps a picture or graphic frame; verify this is intentional.", - location=f"slide {idx}", - text_box=text_box.name, - visual=obj.name, - overlap_ratio=round(overlap, 3), - visual_area_ratio=round(obj_area_ratio, 4), - visual_axis_fill=round(obj_axis_fill, 4), - ) - slides.append({ - "index": idx, - "name": name, - "box_count": len(boxes), - "text_box_count": len(text_boxes), - "image_box_count": len(image_boxes), - "visual_metrics": visual_metrics, - }) - return {"slide_count": len(slide_names), "slide_size": [slide_w, slide_h], "slides": slides} - except zipfile.BadZipFile: - add(findings, "error", "pptx_invalid_zip", "PPTX is not a readable zip archive.", location=str(pptx)) - except ET.ParseError as exc: - add(findings, "error", "pptx_xml_parse_failed", f"Could not parse PPTX XML: {exc}", location=str(pptx)) - return {"slide_count": 0, "slides": slides} - - -def frame_foreground_metrics(rgb_image: Any) -> dict[str, Any] | None: - if np is None: - return None - arr = np.asarray(rgb_image.convert("RGB")).astype(np.int16) - height, width = arr.shape[:2] - border = max(4, min(width, height) // 40) - samples = np.concatenate( - [ - arr[:border, :, :].reshape(-1, 3), - arr[-border:, :, :].reshape(-1, 3), - arr[:, :border, :].reshape(-1, 3), - arr[:, -border:, :].reshape(-1, 3), - ], - axis=0, - ) - bg = np.median(samples, axis=0) - diff = np.max(np.abs(arr - bg), axis=2) - # Also treat dark ink on a white page as foreground even when border - # sampling is imperfect because of colored title bars. - dark_ink = np.max(arr, axis=2) < 238 - mask = (diff > 18) | dark_ink - # Ignore a tiny outer rim; PDF rasterization can add antialiasing noise. - rim = max(2, min(width, height) // 200) - mask[:rim, :] = False - mask[-rim:, :] = False - mask[:, :rim] = False - mask[:, -rim:] = False - - foreground = int(mask.sum()) - total = int(mask.size) - ratio = foreground / max(total, 1) - if foreground == 0: - return { - "foreground_ratio": 0.0, - "content_bbox": None, - "content_bbox_area_ratio": 0.0, - "bottom_blank_ratio": 1.0, - "edge_touch_ratio": 0.0, - } - ys, xs = np.where(mask) - x0, x1 = int(xs.min()), int(xs.max()) - y0, y1 = int(ys.min()), int(ys.max()) - bbox_area = ((x1 - x0 + 1) * (y1 - y0 + 1)) / max(total, 1) - edge_band = max(6, min(width, height) // 80) - edge_mask = np.zeros_like(mask) - edge_mask[:edge_band, :] = True - edge_mask[-edge_band:, :] = True - edge_mask[:, :edge_band] = True - edge_mask[:, -edge_band:] = True - edge_touch = int((mask & edge_mask).sum()) / max(foreground, 1) - return { - "foreground_ratio": round(ratio, 5), - "content_bbox": [round(x0 / width, 4), round(y0 / height, 4), round((x1 - x0 + 1) / width, 4), round((y1 - y0 + 1) / height, 4)], - "content_bbox_area_ratio": round(bbox_area, 5), - "bottom_blank_ratio": round(max(0, height - y1 - 1) / height, 5), - "edge_touch_ratio": round(edge_touch, 5), - } - - -def check_frames( - frames_dir: Path | None, - expected_count: int, - findings: list[Finding], - *, - min_foreground_ratio: float = 0.006, - sparse_foreground_ratio: float = 0.015, - min_bbox_area_ratio: float = 0.055, - sparse_bbox_area_ratio: float = 0.12, -) -> dict[str, Any]: - if frames_dir is None: - return {"checked": False, "frames": []} - if not frames_dir.is_dir(): - add(findings, "warning", "frames_dir_missing", "Rendered frames directory is missing.", location=str(frames_dir)) - return {"checked": False, "frames": []} - frames = sorted([p for p in frames_dir.iterdir() if p.suffix.lower() in {".png", ".jpg", ".jpeg"}], key=natural_key) - if expected_count and len(frames) != expected_count: - add(findings, "error", "frame_count_mismatch", "Rendered frame count does not match PPTX slide count.", location=str(frames_dir), expected=expected_count, actual=len(frames)) - details = [] - if Image is None: - add(findings, "warning", "pil_unavailable", "Pillow is not available; frame pixel checks were skipped.") - return {"checked": False, "frames": [str(p) for p in frames]} - if np is None: - add(findings, "warning", "numpy_unavailable", "NumPy is not available; rendered-frame foreground checks were skipped.") - first_size: tuple[int, int] | None = None - for idx, frame in enumerate(frames, start=1): - try: - with Image.open(frame) as im: - rgb = im.convert("RGB") - stat = ImageStat.Stat(rgb) - mean = sum(stat.mean) / 3.0 - var = sum(stat.var) / 3.0 - size = rgb.size - if first_size is None: - first_size = size - elif size != first_size: - add(findings, "error", "frame_size_mismatch", "Rendered frames have inconsistent dimensions.", location=str(frame), expected=first_size, actual=size) - if var < 4.0 or mean < 3.0 or mean > 252.0: - add(findings, "error", "blank_or_near_blank_frame", "Rendered slide frame appears blank or nearly blank.", location=str(frame), mean=round(mean, 2), variance=round(var, 2)) - metrics = frame_foreground_metrics(rgb) if np is not None else None - if metrics: - fg = float(metrics["foreground_ratio"]) - bbox_area = float(metrics["content_bbox_area_ratio"]) - bottom_blank = float(metrics["bottom_blank_ratio"]) - edge_touch = float(metrics["edge_touch_ratio"]) - if fg < min_foreground_ratio or bbox_area < min_bbox_area_ratio: - sev = "warning" if idx == 1 else "error" - add( - findings, - sev, - "rendered_slide_too_sparse", - "Rendered slide has very little visible foreground content.", - location=str(frame), - foreground_ratio=fg, - content_bbox_area_ratio=bbox_area, - content_bbox=metrics["content_bbox"], - ) - elif fg < sparse_foreground_ratio or bbox_area < sparse_bbox_area_ratio: - add( - findings, - "warning", - "rendered_slide_sparse", - "Rendered slide content occupies a small part of the frame; check for over-shrunk images or excessive blank space.", - location=str(frame), - foreground_ratio=fg, - content_bbox_area_ratio=bbox_area, - content_bbox=metrics["content_bbox"], - ) - if idx not in {1, len(frames)} and bottom_blank > 0.42 and bbox_area < 0.45: - add( - findings, - "warning", - "rendered_slide_bottom_blank", - "Rendered slide leaves a large blank band at the bottom.", - location=str(frame), - bottom_blank_ratio=bottom_blank, - content_bbox=metrics["content_bbox"], - ) - if edge_touch > 0.08: - add( - findings, - "warning", - "rendered_slide_edge_touch", - "Visible content touches the slide edge; check for cropped text/images.", - location=str(frame), - edge_touch_ratio=edge_touch, - ) - details.append({"index": idx, "path": str(frame), "width": size[0], "height": size[1], "mean": round(mean, 2), "variance": round(var, 2), "foreground": metrics}) - except Exception as exc: - add(findings, "error", "frame_read_failed", f"Could not read rendered frame: {exc}", location=str(frame)) - return {"checked": True, "frames": details} - - -def check_audio(audio_dir: Path | None, sections: list[dict[str, str]], ffprobe: str | None, ffmpeg: str | None, findings: list[Finding]) -> dict[str, Any]: - if audio_dir is None: - return {"checked": False, "files": []} - if not audio_dir.is_dir(): - add(findings, "error", "audio_dir_missing", "Audio directory is missing.", location=str(audio_dir)) - return {"checked": False, "files": []} - details = [] - for sec in sections: - path = audio_dir / f"{sec['id']}.mp3" - if not path.is_file(): - add(findings, "error", "audio_missing", "Expected MP3 for script section is missing.", location=str(path), section_id=sec["id"]) - continue - duration = probe_duration(path, ffprobe, ffmpeg) - if duration is None or duration <= 0: - add(findings, "error", "audio_duration_invalid", "Could not probe a positive MP3 duration.", location=str(path), section_id=sec["id"]) - details.append({"id": sec["id"], "path": str(path), "duration": duration}) - extra = sorted(p.name for p in audio_dir.glob("*.mp3") if p.stem not in {s["id"] for s in sections}) - if extra: - add(findings, "warning", "audio_extra_files", "Audio directory contains MP3s not referenced by script.json.", location=str(audio_dir), files=extra) - return {"checked": True, "files": details, "total_duration": sum(float(d["duration"] or 0) for d in details)} - - -def normalized_point_error(point: Any) -> str | None: - if not isinstance(point, list) or len(point) != 2: - return "point must be normalized [x, y] coordinates" - if not all(isinstance(v, (int, float)) for v in point): - return "point values must be numeric" - x, y = [float(v) for v in point] - if x < 0 or x > 1 or y < 0 or y > 1: - return "point is outside the slide canvas" - return None - - -def normalized_box_error(box: Any) -> str | None: - if not isinstance(box, list) or len(box) != 4: - return "box must be normalized [x, y, w, h] coordinates" - if not all(isinstance(v, (int, float)) for v in box): - return "box values must be numeric" - x, y, w, h = [float(v) for v in box] - if w <= 0 or h <= 0: - return "box width/height must be positive" - if x >= 1 or y >= 1 or x + w <= 0 or y + h <= 0: - return "box is completely outside the slide canvas" - if x < -0.0001 or y < -0.0001 or x + w > 1.0001 or y + h > 1.0001: - return "box extends outside the slide canvas" - return None - - -def highlight_module_box_error(box: Any, role: str | None = None) -> str | None: - """Reject word-sized boxes for presentation-style highlights.""" - if normalized_box_error(box): - return None - x, y, w, h = [float(v) for v in box] - area = w * h - role_l = str(role or "").lower() - if area < 0.012 and role_l not in {"qr"}: - return "highlight box is too small for presentation use; target a module, card, row, figure part, or bullet group instead of a word/short phrase" - return None - - -def _accepted_plan_chunks_by_slide(cue_plan_path: Path | None) -> dict[int, list[dict[str, Any]]]: - if cue_plan_path is None or not cue_plan_path.is_file(): - return {} - payload = read_json(cue_plan_path) - out: dict[int, list[dict[str, Any]]] = {} - for slide in payload.get("slides") or []: - if not isinstance(slide, dict): - continue - try: - slide_index = int(slide.get("index")) - except (TypeError, ValueError): - continue - chunks = [ - chunk - for chunk in (slide.get("chunks") or []) - if isinstance(chunk, dict) and chunk.get("accepted") - ] - out[slide_index] = chunks - return out - - -def _cue_matches_plan_chunk(cue: dict[str, Any], chunk: dict[str, Any], *, time_tolerance: float = 0.08) -> bool: - target = str(cue.get("target") or "") - if target and str(chunk.get("target") or "") != target: - return False - try: - cue_start = float(cue.get("start", 0) or 0) - cue_end = float(cue.get("end", 0) or 0) - chunk_start = float(chunk.get("start", 0) or 0) - chunk_end = float(chunk.get("end", 0) or 0) - except (TypeError, ValueError): - return False - return abs(cue_start - chunk_start) <= time_tolerance and abs(cue_end - chunk_end) <= time_tolerance - - -def check_visual_cues( - path: Path | None, - sections: list[dict[str, str]], - audio_report: dict[str, Any], - pad_tail: float, - findings: list[Finding], - *, - required: bool = False, - cue_plan_path: Path | None = None, - strict_attention: bool = False, - min_slide_coverage: float = 0.85, -) -> dict[str, Any]: - if path is None: - if required: - add(findings, "error", "visual_cues_required", "Visual-cue JSON is required for attention/highlight QA.") - return {"checked": False} - if not path.is_file(): - add(findings, "error", "visual_cues_missing", "Visual cues JSON is missing.", location=str(path)) - return {"checked": False} - payload = read_json(path) - slides = payload.get("slides") if isinstance(payload, dict) else None - if not isinstance(slides, list): - add(findings, "error", "visual_cues_schema", "Visual cues JSON must contain a slides array.", location=str(path)) - return {"checked": False} - duration_by_id = {item["id"]: float(item["duration"] or 0) for item in audio_report.get("files", []) if item.get("id")} - section_ids = {sec["id"] for sec in sections} - section_by_index = {idx: sec["id"] for idx, sec in enumerate(sections, start=1)} - accepted_plan_by_slide = _accepted_plan_chunks_by_slide(cue_plan_path) - cue_count = 0 - box_cue_count = 0 - geometry_source_counts: Counter[str] = Counter() - geometry_matched_count = 0 - geometry_field_errors = 0 - slides_with_cues: set[str] = set() - empty_slides = 0 - cues_missing_plan_match = 0 - for slide in slides: - if not isinstance(slide, dict): - add(findings, "error", "visual_cues_slide_schema", "Each visual-cue slide entry must be an object.", location=str(path)) - continue - sid = str(slide.get("id") or "") - slide_index = None - if slide.get("index") is not None: - try: - slide_index = int(slide["index"]) - except (TypeError, ValueError): - add(findings, "error", "visual_cues_bad_index", "Visual cue slide index must be an integer.", location=str(path), index=slide.get("index")) - if not sid and slide_index is not None: - sid = section_by_index.get(slide_index, "") - if sid and sid not in section_ids: - add(findings, "warning", "visual_cues_unknown_section", "Visual cue slide id is not in script.json.", location=str(path), section_id=sid) - cues = slide.get("cues") or [] - if not isinstance(cues, list): - add(findings, "error", "visual_cues_cues_schema", "Visual cue slide entry has non-list cues.", location=str(path), section_id=sid) - continue - if not cues: - empty_slides += 1 - severity = "error" if required else "warning" - add(findings, severity, "visual_cues_empty_slide", "A slide has no visual attention cues.", location=str(path), section_id=sid) - elif sid: - slides_with_cues.add(sid) - max_duration = duration_by_id.get(sid, 0) + pad_tail - for cue in cues: - cue_count += 1 - if not isinstance(cue, dict): - add(findings, "error", "visual_cue_schema", "Each cue must be an object.", location=str(path), section_id=sid) - continue - cue_type = str(cue.get("type") or "highlight") - try: - start = float(cue.get("start", cue.get("at", 0)) or 0) - end = float(cue.get("end", start + float(cue.get("duration", 0) or 0)) or 0) - except (TypeError, ValueError): - add(findings, "error", "visual_cue_bad_time", "Cue timing fields must be numeric.", location=str(path), section_id=sid, cue=cue) - continue - if start < -0.01 or end <= start: - add(findings, "error", "visual_cue_bad_time", "Cue has invalid start/end timing.", location=str(path), section_id=sid, cue=cue) - if max_duration > 0 and end > max_duration + 0.15: - add(findings, "error", "visual_cue_time_overflow", "Cue extends beyond the matching audio segment.", location=str(path), section_id=sid, end=end, segment_duration=max_duration) - point = cue.get("point") - box = cue.get("box") - if point is None and box is None: - add(findings, "error", "visual_cue_missing_geometry", "Cue must include a normalized box or point.", location=str(path), section_id=sid, cue=cue) - if point is not None: - point_error = normalized_point_error(point) - if point_error: - add(findings, "error", "visual_cue_bad_point", point_error, location=str(path), section_id=sid, cue=cue) - if box is not None: - box_error = normalized_box_error(box) - if box_error: - add(findings, "error", "visual_cue_bad_box", box_error, location=str(path), section_id=sid, cue=cue) - else: - box_cue_count += 1 - module_error = highlight_module_box_error(box, str(cue.get("target_role") or "")) - if module_error: - severity = "error" if strict_attention else "warning" - add(findings, severity, "visual_cue_micro_box", module_error, location=str(path), section_id=sid, cue=cue) - elif required and cue_type == "highlight": - add(findings, "error", "visual_cue_highlight_missing_box", "Strict highlight cues must include the target region box.", location=str(path), section_id=sid, cue=cue) - geometry_source = str(cue.get("geometry_source") or cue.get("target_source") or "").strip() - if geometry_source: - geometry_source_counts[geometry_source] += 1 - if cue.get("geometry_matched"): - geometry_matched_count += 1 - if cue_type == "highlight" and strict_attention: - geometry_box = cue.get("geometry_box") - semantic_box = cue.get("semantic_box") - if geometry_box is not None: - geometry_error = normalized_box_error(geometry_box) - if geometry_error: - geometry_field_errors += 1 - add(findings, "error", "visual_cue_geometry_box_bad", geometry_error, location=str(path), section_id=sid, cue=cue) - elif box is not None: - try: - if any(abs(float(a) - float(b)) > 0.002 for a, b in zip(geometry_box, box)): - geometry_field_errors += 1 - add(findings, "error", "visual_cue_geometry_box_mismatch", "geometry_box must match the rendered cue box.", location=str(path), section_id=sid, cue=cue) - except (TypeError, ValueError): - geometry_field_errors += 1 - if semantic_box is not None: - semantic_error = normalized_box_error(semantic_box) - if semantic_error: - geometry_field_errors += 1 - add(findings, "error", "visual_cue_semantic_box_bad", semantic_error, location=str(path), section_id=sid, cue=cue) - if geometry_source and geometry_source not in {"svg", "pptx", "pptx_cluster"}: - geometry_field_errors += 1 - add(findings, "error", "visual_cue_geometry_source_unknown", "geometry_source must be svg, pptx, or pptx_cluster.", location=str(path), section_id=sid, cue=cue) - if not str(cue.get("target") or "").strip(): - severity = "error" if strict_attention else "warning" - add(findings, severity, "visual_cue_target_missing", "Cue is missing a semantic target id.", location=str(path), section_id=sid, cue=cue) - if accepted_plan_by_slide and slide_index is not None: - candidates = accepted_plan_by_slide.get(slide_index, []) - if not any(_cue_matches_plan_chunk(cue, chunk) for chunk in candidates): - cues_missing_plan_match += 1 - severity = "error" if strict_attention else "warning" - add(findings, severity, "visual_cue_not_in_plan", "Cue does not match an accepted chunk in visual_cue_plan.json.", location=str(path), section_id=sid, cue=cue) - expected_sections = len(sections) - coverage = len(slides_with_cues) / expected_sections if expected_sections else 0.0 - if required and coverage + 1e-9 < min_slide_coverage: - add( - findings, - "error", - "visual_cue_coverage_low", - "Visual cues do not cover enough script sections for reliable highlight playback.", - location=str(path), - coverage=round(coverage, 3), - required=min_slide_coverage, - covered_sections=sorted(slides_with_cues), - expected_sections=expected_sections, - ) - if required and cue_count == 0: - add(findings, "error", "visual_cue_count_zero", "No visual attention cues were produced.", location=str(path)) - return { - "checked": True, - "cue_count": cue_count, - "box_cue_count": box_cue_count, - "slide_entries": len(slides), - "slides_with_cues": len(slides_with_cues), - "empty_slide_entries": empty_slides, - "cues_missing_plan_match": cues_missing_plan_match, - "geometry_source_counts": dict(sorted(geometry_source_counts.items())), - "geometry_matched_count": geometry_matched_count, - "geometry_field_errors": geometry_field_errors, - "coverage": round(coverage, 4), - } - - -def check_cue_plan( - path: Path | None, - findings: list[Finding], - *, - required: bool = False, - strict_attention: bool = False, - require_word_timings: bool = False, - min_acceptance_rate: float = 0.85, -) -> dict[str, Any]: - if path is None: - if required: - add(findings, "error", "cue_plan_required", "Cue-plan JSON is required for attention/highlight QA.") - return {"checked": False} - if not path.is_file(): - add(findings, "error", "cue_plan_missing", "Visual cue plan JSON is missing.", location=str(path)) - return {"checked": False} - payload = read_json(path) - slides = payload.get("slides") if isinstance(payload, dict) else None - if not isinstance(slides, list): - add(findings, "error", "cue_plan_schema", "Cue plan must contain a slides array.", location=str(path)) - return {"checked": False} - - for err in payload.get("errors") or []: - add(findings, "error", "cue_plan_error", "Cue planner reported an error.", location=str(path), detail=str(err)) - for warn in payload.get("warnings") or []: - add(findings, "warning", "cue_plan_warning", "Cue planner reported a warning.", location=str(path), detail=str(warn)) - - accepted = skipped = low_confidence = risky_targets = estimated_timing = 0 - missing_targets = missing_region_boxes = bad_region_boxes = bad_points = bad_timing = 0 - timing_source_counts: Counter[str] = Counter() - low_timing_alignment = 0 - min_confidence = float(payload.get("min_confidence") or 0) - for slide in slides: - if not isinstance(slide, dict): - add(findings, "error", "cue_plan_slide_schema", "Cue plan slide entry must be an object.", location=str(path)) - continue - timing_source = str(slide.get("timing_source") or "unknown") - timing_source_counts[timing_source] += 1 - if timing_source == "duration_proportional": - estimated_timing += 1 - chunks = slide.get("chunks") or [] - if not isinstance(chunks, list): - add(findings, "error", "cue_plan_chunks_schema", "Cue plan slide chunks must be an array.", location=str(path), section_id=slide.get("id")) - continue - for chunk in chunks: - if not isinstance(chunk, dict): - add(findings, "error", "cue_plan_chunk_schema", "Cue plan chunk must be an object.", location=str(path), section_id=slide.get("id")) - continue - try: - start = float(chunk.get("start")) - end = float(chunk.get("end")) - except (TypeError, ValueError): - bad_timing += 1 - add(findings, "error", "cue_plan_bad_time", "Cue-plan chunk start/end must be numeric.", location=str(path), section_id=slide.get("id"), chunk_index=chunk.get("chunk_index")) - start = end = None - if start is not None and end is not None and end <= start: - bad_timing += 1 - add(findings, "error", "cue_plan_bad_time", "Cue-plan chunk end must be after start.", location=str(path), section_id=slide.get("id"), chunk_index=chunk.get("chunk_index"), start=start, end=end) - timing = chunk.get("timing") if isinstance(chunk.get("timing"), dict) else {} - if require_word_timings and timing_source.startswith("edge_word_"): - score = timing.get("score") - if isinstance(score, (int, float)) and float(score) < 0.58: - low_timing_alignment += 1 - add(findings, "error", "cue_plan_low_timing_alignment", "Cue timing text alignment score is too low.", location=str(path), section_id=slide.get("id"), chunk_index=chunk.get("chunk_index"), score=score, timing=timing) - if chunk.get("accepted"): - accepted += 1 - conf = float(chunk.get("confidence") or 0) - if conf < min_confidence: - low_confidence += 1 - add(findings, "error", "cue_plan_low_confidence", "Accepted cue is below the plan confidence threshold.", location=str(path), section_id=slide.get("id"), confidence=conf, target=chunk.get("target")) - if str(chunk.get("target_role") or "") in {"header", "caption", "chrome", "footer", "background"}: - risky_targets += 1 - severity = "error" if strict_attention else "warning" - add(findings, severity, "cue_plan_risky_target", "Accepted cue points at slide chrome/header/caption and needs review.", location=str(path), section_id=slide.get("id"), target=chunk.get("target"), role=chunk.get("target_role")) - if not str(chunk.get("target") or "").strip(): - missing_targets += 1 - severity = "error" if strict_attention else "warning" - add(findings, severity, "cue_plan_target_missing", "Accepted cue is missing a semantic target id.", location=str(path), section_id=slide.get("id"), chunk_index=chunk.get("chunk_index")) - box = chunk.get("region_box") - if box is None: - missing_region_boxes += 1 - severity = "error" if strict_attention else "warning" - add(findings, severity, "cue_plan_region_box_missing", "Accepted cue is missing the target region box.", location=str(path), section_id=slide.get("id"), chunk_index=chunk.get("chunk_index"), target=chunk.get("target")) - else: - box_error = normalized_box_error(box) - if box_error: - bad_region_boxes += 1 - add(findings, "error", "cue_plan_region_box_bad", box_error, location=str(path), section_id=slide.get("id"), chunk_index=chunk.get("chunk_index"), target=chunk.get("target"), region_box=box) - else: - module_error = highlight_module_box_error(box, str(chunk.get("target_role") or "")) - if module_error: - bad_region_boxes += 1 - severity = "error" if strict_attention else "warning" - add(findings, severity, "cue_plan_micro_box", module_error, location=str(path), section_id=slide.get("id"), chunk_index=chunk.get("chunk_index"), target=chunk.get("target"), region_box=box) - point = chunk.get("point") - if point is not None: - point_error = normalized_point_error(point) - if point_error: - bad_points += 1 - add(findings, "error", "cue_plan_point_bad", point_error, location=str(path), section_id=slide.get("id"), chunk_index=chunk.get("chunk_index"), target=chunk.get("target"), point=point) - else: - skipped += 1 - if estimated_timing: - severity = "error" if require_word_timings else "warning" - add(findings, severity, "cue_plan_estimated_timing", "Cue plan uses proportional timing instead of word-boundary timing.", location=str(path), slide_count=estimated_timing) - total_chunks = accepted + skipped - acceptance_rate = accepted / total_chunks if total_chunks else 0.0 - if required and total_chunks == 0: - add(findings, "error", "cue_plan_empty", "Cue plan contains no cue chunks.", location=str(path)) - if strict_attention and skipped: - add(findings, "error", "cue_plan_skipped_chunks", "Cue planner skipped one or more narration chunks; highlight timing cannot be trusted.", location=str(path), skipped_chunks=skipped) - if strict_attention and acceptance_rate + 1e-9 < min_acceptance_rate: - add( - findings, - "error", - "cue_plan_acceptance_low", - "Cue-plan acceptance rate is below the strict attention threshold.", - location=str(path), - accepted_chunks=accepted, - skipped_chunks=skipped, - acceptance_rate=round(acceptance_rate, 3), - required=min_acceptance_rate, - ) - return { - "checked": True, - "accepted_chunks": accepted, - "skipped_chunks": skipped, - "low_confidence": low_confidence, - "risky_targets": risky_targets, - "missing_targets": missing_targets, - "missing_region_boxes": missing_region_boxes, - "bad_region_boxes": bad_region_boxes, - "bad_points": bad_points, - "bad_timing_chunks": bad_timing, - "timing_source_counts": dict(sorted(timing_source_counts.items())), - "low_timing_alignment": low_timing_alignment, - "estimated_timing_slides": estimated_timing, - "acceptance_rate": round(acceptance_rate, 4), - } - - -def _contract_chunks(payload: dict[str, Any]) -> dict[tuple[int, int], dict[str, Any]]: - out: dict[tuple[int, int], dict[str, Any]] = {} - for slide in payload.get("slides") or []: - if not isinstance(slide, dict): - continue - try: - slide_index = int(slide.get("index")) - except (TypeError, ValueError): - continue - for chunk in slide.get("chunks") or []: - if not isinstance(chunk, dict): - continue - try: - chunk_index = int(chunk.get("chunk_index")) - except (TypeError, ValueError): - continue - out[(slide_index, chunk_index)] = chunk - return out - - -def _cue_plan_chunks(path: Path | None) -> dict[tuple[int, int], dict[str, Any]]: - if path is None or not path.is_file(): - return {} - payload = read_json(path) - out: dict[tuple[int, int], dict[str, Any]] = {} - for slide in payload.get("slides") or []: - if not isinstance(slide, dict): - continue - try: - slide_index = int(slide.get("index")) - except (TypeError, ValueError): - continue - for chunk in slide.get("chunks") or []: - if not isinstance(chunk, dict): - continue - try: - chunk_index = int(chunk.get("chunk_index")) - except (TypeError, ValueError): - continue - out[(slide_index, chunk_index)] = chunk - return out - - -def check_anchor_contract( - path: Path | None, - cue_plan_path: Path | None, - findings: list[Finding], - *, - required: bool = False, - strict_attention: bool = False, - require_pptx_anchors: bool = False, -) -> dict[str, Any]: - if path is None: - if required: - add(findings, "error", "anchor_contract_required", "Visual anchor contract is required for strict attention QA.") - return {"checked": False} - if not path.is_file(): - add(findings, "error", "anchor_contract_missing", "Visual anchor contract is missing.", location=str(path)) - return {"checked": False} - payload = read_json(path) - if not isinstance(payload, dict): - add(findings, "error", "anchor_contract_schema", "Visual anchor contract must be a JSON object.", location=str(path)) - return {"checked": False} - if payload.get("schema_version") not in {"paper2video_visual_anchor_contract.v1", "paper2video_cue_requirements.v1"}: - add(findings, "error", "anchor_contract_schema_version", "Visual anchor contract has an unsupported schema_version.", location=str(path), schema_version=payload.get("schema_version")) - - contract = _contract_chunks(payload) - required_chunks = {key: chunk for key, chunk in contract.items() if chunk.get("required", True)} - missing_anchor_ids = 0 - for (slide_index, chunk_index), chunk in required_chunks.items(): - if not str(chunk.get("anchor_id") or "").strip(): - missing_anchor_ids += 1 - add(findings, "error", "anchor_contract_chunk_missing_anchor", "Required contract chunk is missing anchor_id.", location=str(path), slide_index=slide_index, chunk_index=chunk_index) - - plan = _cue_plan_chunks(cue_plan_path) - matched = unmatched = missing_in_plan = non_pptx_matches = 0 - if cue_plan_path is not None and cue_plan_path.is_file(): - for key, chunk in required_chunks.items(): - expected = str(chunk.get("anchor_id") or "").strip() - if not expected: - continue - planned = plan.get(key) - if not planned: - missing_in_plan += 1 - add(findings, "error" if strict_attention else "warning", "anchor_contract_chunk_missing_in_plan", "Anchor contract chunk is missing from cue plan.", location=str(path), slide_index=key[0], chunk_index=key[1], anchor_id=expected) - continue - actual = str(planned.get("anchor_id") or "").strip() - if actual != expected: - add(findings, "error" if strict_attention else "warning", "anchor_contract_id_mismatch", "Cue plan anchor_id does not match the visual anchor contract.", location=str(path), slide_index=key[0], chunk_index=key[1], expected_anchor=expected, actual_anchor=actual) - if planned.get("anchor_matched") and planned.get("accepted"): - matched += 1 - if require_pptx_anchors and str(planned.get("target_source") or "") != "pptx": - non_pptx_matches += 1 - add(findings, "error", "anchor_contract_non_pptx_match", "Anchor matched outside PPTX, but PPTX anchors were explicitly required for this QA run.", location=str(path), slide_index=key[0], chunk_index=key[1], anchor_id=expected, target_source=planned.get("target_source")) - else: - unmatched += 1 - add(findings, "error" if strict_attention else "warning", "anchor_contract_unmatched", "Required visual anchor was not matched by the cue plan.", location=str(path), slide_index=key[0], chunk_index=key[1], anchor_id=expected, reason=planned.get("reason")) - elif required or strict_attention: - add(findings, "error", "anchor_contract_cue_plan_missing", "Cue plan is required to validate the visual anchor contract.", location=str(path)) - - return { - "checked": True, - "required_chunks": len(required_chunks), - "missing_anchor_ids": missing_anchor_ids, - "matched_chunks": matched, - "unmatched_chunks": unmatched, - "missing_in_plan": missing_in_plan, - "non_pptx_matches": non_pptx_matches, - } - - -def check_timeline( - path: Path | None, - findings: list[Finding], - *, - required: bool = False, - strict_attention: bool = False, -) -> dict[str, Any]: - if path is None: - if required: - add(findings, "error", "timeline_required", "timeline.json is required to bind audio, subtitles, and visual cues.") - return {"checked": False} - if not path.is_file(): - add(findings, "error", "timeline_missing", "timeline.json is missing.", location=str(path)) - return {"checked": False} - payload = read_json(path) - if not isinstance(payload, dict): - add(findings, "error", "timeline_schema", "timeline.json must be a JSON object.", location=str(path)) - return {"checked": False} - if payload.get("schema_version") != "paper2video_timeline.v1": - add(findings, "error", "timeline_schema_version", "timeline.json has an unsupported schema_version.", location=str(path), schema_version=payload.get("schema_version")) - slides = payload.get("slides") or [] - chunks = payload.get("chunks") or [] - sections = payload.get("sections") or [] - if not isinstance(slides, list) or not isinstance(chunks, list) or not isinstance(sections, list): - add(findings, "error", "timeline_schema", "timeline.json must contain slides, sections, and chunks arrays.", location=str(path)) - return {"checked": True, "slide_count": 0, "section_count": 0, "chunk_count": 0} - - slide_windows: dict[int, tuple[float, float]] = {} - for slide in slides: - if not isinstance(slide, dict): - continue - try: - idx = int(slide.get("index")) - segment = slide.get("segment") or {} - slide_windows[idx] = (float(segment.get("start")), float(segment.get("end"))) - except (TypeError, ValueError): - continue - - chunk_ids: set[str] = set() - subtitle_chunks = 0 - visual_chunks = 0 - bad_time_chunks = 0 - missing_subtitles = 0 - missing_visuals = 0 - visual_geometry_errors = 0 - for chunk in chunks: - if not isinstance(chunk, dict): - add(findings, "error", "timeline_chunk_schema", "Timeline chunk entry must be an object.", location=str(path)) - continue - cid = str(chunk.get("id") or "") - if not cid: - add(findings, "error", "timeline_chunk_id_missing", "Timeline chunk is missing a stable id.", location=str(path), chunk=chunk) - elif cid in chunk_ids: - add(findings, "error", "timeline_chunk_id_duplicate", "Timeline chunk id is duplicated.", location=str(path), chunk_id=cid) - chunk_ids.add(cid) - try: - start = float(chunk.get("start")) - end = float(chunk.get("end")) - except (TypeError, ValueError): - add(findings, "error", "timeline_chunk_time_bad", "Timeline chunk start/end must be numeric.", location=str(path), chunk_id=cid) - bad_time_chunks += 1 - continue - if end <= start: - add(findings, "error", "timeline_chunk_time_bad", "Timeline chunk end must be after start.", location=str(path), chunk_id=cid, start=start, end=end) - bad_time_chunks += 1 - try: - slide_index = int(chunk.get("slide_index")) - except (TypeError, ValueError): - slide_index = 0 - if slide_index in slide_windows: - slide_start, slide_end = slide_windows[slide_index] - if start < slide_start - 0.05 or end > slide_end + 0.05: - add( - findings, - "error", - "timeline_chunk_outside_slide_window", - "Timeline chunk timing falls outside its slide segment window.", - location=str(path), - chunk_id=cid, - slide_index=slide_index, - chunk_window=[round(start, 3), round(end, 3)], - slide_window=[round(slide_start, 3), round(slide_end, 3)], - ) - bad_time_chunks += 1 - subtitles = chunk.get("subtitles") or [] - if subtitles: - subtitle_chunks += 1 - else: - missing_subtitles += 1 - add(findings, "warning", "timeline_chunk_no_subtitles", "Timeline chunk has no subtitle cues.", location=str(path), chunk_id=cid) - visual = chunk.get("visual_cue") - if isinstance(visual, dict) and visual.get("accepted") and (visual.get("region_box") or visual.get("point")): - visual_chunks += 1 - if visual.get("region_box") is not None: - box_error = normalized_box_error(visual.get("region_box")) - if box_error: - visual_geometry_errors += 1 - add(findings, "error", "timeline_visual_box_bad", box_error, location=str(path), chunk_id=cid, region_box=visual.get("region_box")) - if visual.get("point") is not None: - point_error = normalized_point_error(visual.get("point")) - if point_error: - visual_geometry_errors += 1 - add(findings, "error", "timeline_visual_point_bad", point_error, location=str(path), chunk_id=cid, point=visual.get("point")) - if strict_attention and not str(visual.get("target") or "").strip(): - add(findings, "error", "timeline_visual_target_missing", "Timeline visual cue is missing a semantic target id.", location=str(path), chunk_id=cid) - else: - missing_visuals += 1 - severity = "error" if strict_attention else "warning" - add(findings, severity, "timeline_chunk_no_visual_cue", "Timeline chunk has no accepted visual cue.", location=str(path), chunk_id=cid) - - slide_indices: set[int] = set() - for slide in slides: - if not isinstance(slide, dict): - add(findings, "error", "timeline_slide_schema", "Timeline slide entry must be an object.", location=str(path)) - continue - try: - idx = int(slide.get("index")) - except (TypeError, ValueError): - add(findings, "error", "timeline_slide_index_bad", "Timeline slide index must be an integer.", location=str(path), slide=slide) - continue - if idx in slide_indices: - add(findings, "error", "timeline_slide_index_duplicate", "Timeline slide index is duplicated.", location=str(path), slide_index=idx) - slide_indices.add(idx) - - section_windows = [] - for section in sections: - if not isinstance(section, dict): - add(findings, "error", "timeline_section_schema", "Timeline section entry must be an object.", location=str(path)) - continue - sid = str(section.get("id") or "") - if not sid: - add(findings, "error", "timeline_section_id_missing", "Timeline section is missing id.", location=str(path)) - continue - try: - start = float(section.get("start")) - end = float(section.get("end")) - except (TypeError, ValueError): - add(findings, "error", "timeline_section_time_bad", "Timeline section start/end must be numeric.", location=str(path), section_id=sid) - continue - if end <= start: - add(findings, "error", "timeline_section_time_bad", "Timeline section end must be after start.", location=str(path), section_id=sid, start=start, end=end) - section_slide_indices: set[int] = set() - for idx in section.get("slide_indices") or []: - try: - section_slide_indices.add(int(idx)) - except (TypeError, ValueError): - add(findings, "error", "timeline_section_slide_index_bad", "Timeline section slide_indices must be integers.", location=str(path), section_id=sid, slide_index=idx) - if sid != "title": - section_windows.append((start, end, sid, section_slide_indices)) - for cid in section.get("chunk_ids") or []: - if str(cid) not in chunk_ids: - add(findings, "error", "timeline_section_unknown_chunk", "Timeline section references a missing chunk id.", location=str(path), section_id=sid, chunk_id=cid) - - section_windows.sort() - for (prev_start, prev_end, prev_id, prev_slides), (start, end, sid, slides_for_section) in zip(section_windows, section_windows[1:]): - if start < prev_end - 0.05: - shared_slides = sorted(prev_slides & slides_for_section) - if shared_slides: - continue - add( - findings, - "warning", - "timeline_section_overlap", - "Non-title timeline sections overlap; visualization clips may play unexpected content.", - location=str(path), - first=prev_id, - second=sid, - first_window=[round(prev_start, 3), round(prev_end, 3)], - second_window=[round(start, 3), round(end, 3)], - ) - - return { - "checked": True, - "section_count": len(sections), - "slide_count": len(slides), - "chunk_count": len(chunks), - "chunks_with_subtitles": subtitle_chunks, - "chunks_with_visual_cues": visual_chunks, - "missing_subtitle_chunks": missing_subtitles, - "missing_visual_cue_chunks": missing_visuals, - "visual_geometry_errors": visual_geometry_errors, - "bad_time_chunks": bad_time_chunks, - } - - -def check_rate_plan( - path: Path | None, - findings: list[Finding], - *, - required: bool = False, - max_adjust_percent: float = 8.0, -) -> dict[str, Any]: - if path is None: - if required: - add(findings, "error", "tts_rate_plan_required", "TTS rate plan is required for duration-controlled video.") - return {"checked": False} - if not path.is_file(): - add(findings, "error", "tts_rate_plan_missing", "TTS rate plan is missing.", location=str(path)) - return {"checked": False} - payload = read_json(path) - if not isinstance(payload, dict): - add(findings, "error", "tts_rate_plan_schema", "TTS rate plan must be a JSON object.", location=str(path)) - return {"checked": False} - if payload.get("schema_version") != "paper2video_tts_rate_plan.v1": - add(findings, "error", "tts_rate_plan_schema_version", "TTS rate plan has an unsupported schema_version.", location=str(path), schema_version=payload.get("schema_version")) - status = str(payload.get("status") or "") - safe = bool(payload.get("safe")) - try: - recommended = abs(float(payload.get("recommended_adjust_percent") or 0.0)) - required_adjust = abs(float(payload.get("required_adjust_percent") or 0.0)) - except (TypeError, ValueError): - add(findings, "error", "tts_rate_plan_bad_adjustment", "TTS rate adjustment fields must be numeric.", location=str(path)) - recommended = required_adjust = 0.0 - if status == "needs_script_rewrite": - add(findings, "error", "tts_rate_requires_script_rewrite", "Duration mismatch is too large for safe TTS rate adjustment; rewrite script first.", location=str(path), required_adjust_percent=round(required_adjust, 3)) - elif not safe: - add(findings, "error", "tts_rate_plan_unsafe", "TTS rate plan is marked unsafe.", location=str(path), status=status) - if recommended > max_adjust_percent + 1e-9: - add(findings, "error", "tts_rate_adjustment_too_large", "Recommended TTS rate adjustment is too large for natural speech.", location=str(path), recommended_adjust_percent=round(recommended, 3), max_adjust_percent=max_adjust_percent) - elif recommended > 6.0: - add(findings, "warning", "tts_rate_adjustment_borderline", "Recommended TTS rate adjustment is audible; prefer script rewrite if quality matters.", location=str(path), recommended_adjust_percent=round(recommended, 3)) - return { - "checked": True, - "status": status, - "safe": safe, - "recommended_edge_rate": payload.get("recommended_edge_rate"), - "recommended_adjust_percent": payload.get("recommended_adjust_percent"), - "required_adjust_percent": payload.get("required_adjust_percent"), - "current_delta_seconds": payload.get("current_delta_seconds"), - } - - -def check_video(path: Path | None, target_minutes: float | None, tolerance_seconds: float, ffprobe: str | None, ffmpeg: str | None, findings: list[Finding]) -> dict[str, Any]: - if path is None: - return {"checked": False} - if not path.is_file(): - add(findings, "error", "video_missing", "Final MP4 is missing.", location=str(path)) - return {"checked": False} - streams = probe_video_streams(path, ffprobe, ffmpeg) - if not streams.get("has_video"): - add(findings, "error", "video_stream_missing", "MP4 has no video stream.", location=str(path)) - if not streams.get("has_audio"): - add(findings, "error", "audio_stream_missing", "MP4 has no audio stream.", location=str(path)) - duration = streams.get("duration") - if not isinstance(duration, (int, float)) or duration <= 0: - add(findings, "error", "video_duration_invalid", "Could not probe a positive MP4 duration.", location=str(path)) - elif target_minutes is not None: - target_seconds = target_minutes * 60 - delta = abs(duration - target_seconds) - if delta > tolerance_seconds: - add(findings, "error", "video_duration_out_of_tolerance", "Final MP4 duration is outside the requested tolerance.", location=str(path), duration=round(duration, 2), target_seconds=round(target_seconds, 2), tolerance_seconds=tolerance_seconds) - return {"checked": True, **streams} - - -def sha256_file(path: Path) -> str: - h = hashlib.sha256() - with path.open("rb") as f: - for chunk in iter(lambda: f.read(1024 * 1024), b""): - h.update(chunk) - return h.hexdigest() - - -def count_subtitle_cues(path: Path) -> int: - text = path.read_text(encoding="utf-8", errors="replace") - return len(re.findall(r"\d{2}:\d{2}:\d{2}[,.]\d{3}\s*-->\s*\d{2}:\d{2}:\d{2}[,.]\d{3}", text)) - - -def check_subtitle_delivery( - *, - raw_mp4: Path | None, - final_mp4: Path | None, - subtitle_file: Path | None, - required: bool, - ffprobe: str | None, - ffmpeg: str | None, - findings: list[Finding], -) -> dict[str, Any]: - if not required: - return {"checked": False} - report: dict[str, Any] = { - "checked": True, - "raw_mp4": str(raw_mp4) if raw_mp4 else None, - "final_mp4": str(final_mp4) if final_mp4 else None, - "subtitle_file": str(subtitle_file) if subtitle_file else None, - } - if raw_mp4 is None: - add(findings, "error", "subtitle_raw_mp4_required", "--raw-mp4 is required when --require-subtitles is set.") - elif not raw_mp4.is_file(): - add(findings, "error", "subtitle_raw_mp4_missing", "Raw pre-subtitle MP4 is missing.", location=str(raw_mp4)) - if final_mp4 is None: - add(findings, "error", "subtitle_final_mp4_required", "--mp4 final video is required when --require-subtitles is set.") - elif not final_mp4.is_file(): - add(findings, "error", "subtitle_final_mp4_missing", "Final subtitled MP4 is missing.", location=str(final_mp4)) - if subtitle_file is None: - add(findings, "error", "subtitle_sidecar_required", "--subtitle-file is required when --require-subtitles is set.") - elif not subtitle_file.is_file(): - add(findings, "error", "subtitle_sidecar_missing", "Subtitle sidecar file is missing.", location=str(subtitle_file)) - else: - cue_count = count_subtitle_cues(subtitle_file) - report["cue_count"] = cue_count - if cue_count <= 0: - add(findings, "error", "subtitle_sidecar_empty", "Subtitle sidecar has no timestamped cues.", location=str(subtitle_file)) - - if raw_mp4 and final_mp4 and raw_mp4.exists() and final_mp4.exists(): - same_path = raw_mp4.resolve() == final_mp4.resolve() - report["same_path"] = same_path - if same_path: - add(findings, "error", "subtitle_final_is_raw", "Final MP4 points at the raw render; add_subtitles.py was not applied.", location=str(final_mp4)) - elif raw_mp4.stat().st_size == final_mp4.stat().st_size and sha256_file(raw_mp4) == sha256_file(final_mp4): - add(findings, "error", "subtitle_final_identical_to_raw", "Final MP4 is byte-identical to the raw render; subtitles were likely skipped.", location=str(final_mp4)) - raw_streams = probe_video_streams(raw_mp4, ffprobe, ffmpeg) - final_streams = probe_video_streams(final_mp4, ffprobe, ffmpeg) - raw_duration = raw_streams.get("duration") - final_duration = final_streams.get("duration") - report["raw_duration"] = raw_duration - report["final_duration"] = final_duration - if isinstance(raw_duration, (int, float)) and isinstance(final_duration, (int, float)): - delta = abs(float(final_duration) - float(raw_duration)) - report["duration_delta"] = round(delta, 3) - if delta > 2.0: - add(findings, "warning", "subtitle_duration_drift", "Final subtitled MP4 duration differs from raw render by more than 2 seconds.", location=str(final_mp4), raw_duration=round(float(raw_duration), 3), final_duration=round(float(final_duration), 3)) - return report - - -def write_report(path: Path, report: dict[str, Any]) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") - - -def rel_to(path: Path, base: Path) -> str: - try: - return path.resolve().relative_to(base.resolve()).as_posix() - except ValueError: - return path.resolve().as_posix() - - -def maybe_write_manifest(project_dir: Path, args: argparse.Namespace, report_path: Path, passed: bool) -> None: - files: dict[str, str] = { - "assets_dir": "assets", - "audio_dir": "assets/audio", - "captions_dir": "assets/captions", - "slides_dir": "assets/slides", - "clips_dir": "assets/clips", - "meta_dir": "assets/meta", - "qa_report": rel_to(report_path, project_dir), - } - if args.mp4: - files["video"] = rel_to(args.mp4.resolve(), project_dir) - if args.raw_mp4: - files["video_no_subtitles"] = rel_to(args.raw_mp4.resolve(), project_dir) - if args.pptx: - files["slides_pptx"] = rel_to(args.pptx.resolve(), project_dir) - if args.script_json: - files["script_json"] = rel_to(args.script_json.resolve(), project_dir) - if args.subtitle_file: - files["captions_vtt"] = rel_to(args.subtitle_file.resolve(), project_dir) - if args.timeline: - files["timeline"] = rel_to(args.timeline.resolve(), project_dir) - if args.visual_cues: - files["visual_cues"] = rel_to(args.visual_cues.resolve(), project_dir) - if args.cue_plan: - files["visual_cue_plan"] = rel_to(args.cue_plan.resolve(), project_dir) - - manifest = { - "schema_version": "paper2video.v1", - "layout": "v2-assets", - "created_at": utc_now(), - "files": files, - "qa": { - "check": "check_video_package", - "passed": passed, - "report": rel_to(report_path, project_dir), - }, - } - write_report(project_dir / "manifest.json", manifest) - - -def default_report_path(project_dir: Path) -> Path: - return project_dir / "assets" / "meta" / "reports" / "video_qa_report.json" - - -def main() -> None: - parser = argparse.ArgumentParser(description="Run deterministic QA gates for a paper2video package.") - parser.add_argument("project_dir", type=Path) - parser.add_argument("--pptx", type=Path, required=True) - parser.add_argument("--script-json", type=Path, required=True) - parser.add_argument("--audio-dir", type=Path) - parser.add_argument("--frames-dir", type=Path, help="Rendered slide PNG/JPG directory. Defaults to <project>/assets/slides/frames when present.") - parser.add_argument("--visual-cues", type=Path) - parser.add_argument("--cue-plan", type=Path, help="visual_cue_plan.json written by generate_visual_cues.py.") - parser.add_argument("--anchor-contract", type=Path, help="visual_anchor_contract.json written by generate_cue_requirements.py.") - parser.add_argument("--timeline", type=Path, help="timeline.json written by build_timeline.py.") - parser.add_argument("--rate-plan", type=Path, help="tts_rate_plan.json written by plan_tts_rate.py.") - parser.add_argument("--mp4", type=Path) - parser.add_argument("--raw-mp4", type=Path, help="Raw MP4 before add_subtitles.py; used to verify final subtitle delivery.") - parser.add_argument("--subtitle-file", type=Path, help="SRT/VTT sidecar written by add_subtitles.py.") - parser.add_argument("--target-minutes", type=float) - parser.add_argument("--duration-tolerance-seconds", type=float, default=30.0) - parser.add_argument("--pad-tail", type=float, default=0.3) - parser.add_argument("--no-render-frames", action="store_true", help="Do not auto-render PPTX frames for visual QA when --frames-dir is omitted.") - parser.add_argument("--require-audio", action="store_true", help="Fail when --audio-dir is omitted.") - parser.add_argument("--require-mp4", action="store_true", help="Fail when --mp4 is omitted.") - parser.add_argument("--require-visual-cues", action="store_true", help="Fail when --visual-cues is omitted or coverage is low.") - parser.add_argument("--require-cue-plan", action="store_true", help="Fail when --cue-plan is omitted.") - parser.add_argument("--require-anchor-contract", action="store_true", help="Fail when visual anchor contract is omitted.") - parser.add_argument("--require-timeline", action="store_true", help="Fail when timeline.json is omitted or invalid.") - parser.add_argument("--require-rate-plan", action="store_true", help="Fail when tts_rate_plan.json is omitted for duration-controlled video.") - parser.add_argument("--require-subtitles", action="store_true", help="Fail unless subtitle sidecar exists and final MP4 differs from the raw pre-subtitle render.") - parser.add_argument("--require-word-timings", action="store_true", help="Fail if cue timings are proportional estimates rather than word-boundary timings.") - parser.add_argument("--strict-attention", action="store_true", help="Promote cue-plan semantic-alignment risks to hard failures.") - parser.add_argument("--allow-missing-attention", action="store_true", help="Degraded/debug only: allow --strict without visual cues/cue plan/timeline gates.") - parser.add_argument("--require-pptx-anchors", action="store_true", help="Require strict visual anchors to resolve to PPTX geometry.") - parser.add_argument("--min-cue-coverage", type=float, default=0.85, help="Minimum section coverage required when --require-visual-cues is used.") - parser.add_argument("--min-cue-acceptance", type=float, default=0.85, help="Minimum cue-plan acceptance rate required by --strict-attention.") - parser.add_argument("--max-tts-rate-adjust-percent", type=float, default=8.0, help="Hard maximum absolute TTS rate adjustment allowed by the final QA gate.") - parser.add_argument("--strict", action="store_true", help="Final-package hard gate: require audio, MP4, rendered frames, and fail on warnings.") - parser.add_argument("--out", type=Path, default=None) - parser.add_argument("--fail-on-warning", action="store_true") - args = parser.parse_args() - - findings: list[Finding] = [] - project_dir = args.project_dir.resolve() - sections = load_script(args.script_json.resolve()) - ffmpeg, ffprobe = find_ffmpeg_pair() - - pptx_report = parse_pptx(args.pptx.resolve(), findings) - slide_count = int(pptx_report.get("slide_count") or 0) - if slide_count and len(sections) != slide_count: - add(findings, "error", "script_slide_count_mismatch", "script.json section count does not match PPTX slide count.", expected=slide_count, actual=len(sections)) - - require_audio = args.require_audio or args.strict - require_mp4 = args.require_mp4 or args.strict - if require_audio and args.audio_dir is None: - add(findings, "error", "audio_dir_required", "Final video QA requires --audio-dir.") - if require_mp4 and args.mp4 is None: - add(findings, "error", "mp4_required", "Final video QA requires --mp4.") - - audio_report = check_audio(args.audio_dir.resolve() if args.audio_dir else None, sections, ffprobe, ffmpeg, findings) - - frames_dir = args.frames_dir.resolve() if args.frames_dir else None - if frames_dir is None: - bundled_frames = project_dir / "assets" / "slides" / "frames" - if bundled_frames.is_dir(): - frames_dir = bundled_frames.resolve() - if frames_dir is None and not args.no_render_frames: - frames_dir = render_frames_from_pptx(args.pptx.resolve(), project_dir, findings) - if args.strict and frames_dir is None: - add(findings, "error", "frames_required", "Strict video QA requires rendered slide frames.") - frames_report = check_frames(frames_dir, slide_count, findings) - - strict_attention_required = args.strict_attention or (args.strict and not args.allow_missing_attention) - cues_report = check_visual_cues( - args.visual_cues.resolve() if args.visual_cues else None, - sections, - audio_report, - args.pad_tail, - findings, - required=args.require_visual_cues or strict_attention_required, - cue_plan_path=args.cue_plan.resolve() if args.cue_plan else None, - strict_attention=strict_attention_required, - min_slide_coverage=args.min_cue_coverage, - ) - cue_plan_report = check_cue_plan( - args.cue_plan.resolve() if args.cue_plan else None, - findings, - required=args.require_cue_plan or strict_attention_required, - strict_attention=strict_attention_required, - require_word_timings=args.require_word_timings or strict_attention_required, - min_acceptance_rate=args.min_cue_acceptance, - ) - anchor_contract_report = check_anchor_contract( - args.anchor_contract.resolve() if args.anchor_contract else None, - args.cue_plan.resolve() if args.cue_plan else None, - findings, - required=args.require_anchor_contract, - strict_attention=strict_attention_required, - require_pptx_anchors=args.require_pptx_anchors, - ) - timeline_report = check_timeline( - args.timeline.resolve() if args.timeline else None, - findings, - required=args.require_timeline or strict_attention_required, - strict_attention=strict_attention_required, - ) - rate_plan_report = check_rate_plan( - args.rate_plan.resolve() if args.rate_plan else None, - findings, - required=args.require_rate_plan, - max_adjust_percent=args.max_tts_rate_adjust_percent, - ) - video_report = check_video(args.mp4.resolve() if args.mp4 else None, args.target_minutes, args.duration_tolerance_seconds, ffprobe, ffmpeg, findings) - subtitle_report = check_subtitle_delivery( - raw_mp4=args.raw_mp4.resolve() if args.raw_mp4 else None, - final_mp4=args.mp4.resolve() if args.mp4 else None, - subtitle_file=args.subtitle_file.resolve() if args.subtitle_file else None, - required=args.require_subtitles, - ffprobe=ffprobe, - ffmpeg=ffmpeg, - findings=findings, - ) - - counts = { - "error": sum(1 for f in findings if f.severity == "error"), - "warning": sum(1 for f in findings if f.severity == "warning"), - "info": sum(1 for f in findings if f.severity == "info"), - } - fail_on_warning = args.fail_on_warning or args.strict - passed = findings_pass_gate(findings, fail_on_warning=fail_on_warning) - report = { - "schema_version": SCHEMA_VERSION, - "created_at": utc_now(), - "project_dir": str(project_dir), - "inputs": { - "pptx": str(args.pptx.resolve()), - "script_json": str(args.script_json.resolve()), - "audio_dir": str(args.audio_dir.resolve()) if args.audio_dir else None, - "frames_dir": str(frames_dir) if frames_dir else None, - "visual_cues": str(args.visual_cues.resolve()) if args.visual_cues else None, - "cue_plan": str(args.cue_plan.resolve()) if args.cue_plan else None, - "anchor_contract": str(args.anchor_contract.resolve()) if args.anchor_contract else None, - "timeline": str(args.timeline.resolve()) if args.timeline else None, - "rate_plan": str(args.rate_plan.resolve()) if args.rate_plan else None, - "mp4": str(args.mp4.resolve()) if args.mp4 else None, - "raw_mp4": str(args.raw_mp4.resolve()) if args.raw_mp4 else None, - "subtitle_file": str(args.subtitle_file.resolve()) if args.subtitle_file else None, - }, - "options": { - "strict": args.strict, - "fail_on_warning": fail_on_warning, - "require_audio": require_audio, - "require_mp4": require_mp4, - "require_visual_cues": args.require_visual_cues, - "require_cue_plan": args.require_cue_plan, - "require_anchor_contract": args.require_anchor_contract, - "require_timeline": args.require_timeline, - "require_rate_plan": args.require_rate_plan, - "require_subtitles": args.require_subtitles, - "require_word_timings": args.require_word_timings, - "strict_attention": strict_attention_required, - "allow_missing_attention": args.allow_missing_attention, - "require_pptx_anchors": args.require_pptx_anchors, - "min_cue_coverage": args.min_cue_coverage, - "min_cue_acceptance": args.min_cue_acceptance, - "max_tts_rate_adjust_percent": args.max_tts_rate_adjust_percent, - }, - "passed": passed, - "counts": counts, - "script": {"section_count": len(sections), "section_ids": [s["id"] for s in sections]}, - "pptx": pptx_report, - "audio": audio_report, - "frames": frames_report, - "visual_cues": cues_report, - "cue_plan": cue_plan_report, - "anchor_contract": anchor_contract_report, - "timeline": timeline_report, - "tts_rate_plan": rate_plan_report, - "video": video_report, - "subtitles": subtitle_report, - "findings": [f.__dict__ for f in findings], - } - out_path = args.out or default_report_path(project_dir) - write_report(out_path, report) - maybe_write_manifest(project_dir, args, out_path, passed) - - status = "PASS" if passed else "FAIL" - print(f"[check_video_package] {status}: {counts['error']} error(s), {counts['warning']} warning(s)") - print(f"[check_video_package] report: {out_path}") - if not passed: - for finding in findings[:20]: - loc = f" ({finding.location})" if finding.location else "" - print(f" - {finding.severity.upper()} {finding.code}{loc}: {finding.message}") - raise SystemExit(1) - - -if __name__ == "__main__": - main() diff --git a/ResearchStudio-Reel/skills/paper2video/scripts/generate_edge_audio.py b/ResearchStudio-Reel/skills/paper2video/scripts/generate_edge_audio.py deleted file mode 100644 index bd2e7d6..0000000 --- a/ResearchStudio-Reel/skills/paper2video/scripts/generate_edge_audio.py +++ /dev/null @@ -1,221 +0,0 @@ -#!/usr/bin/env python3 -""" -Generate MP3 narration from a paper2video script JSON using edge-tts. - -The output contract intentionally matches paper2poster's generate_audio.py: -one <section.id>.mp3 per section plus a manifest.json under --outdir. -""" - -from __future__ import annotations - -import argparse -import asyncio -import json -import sys -from pathlib import Path - -try: - import edge_tts -except ImportError as exc: # pragma: no cover - depends on local env - raise SystemExit( - "[generate_edge_audio] edge_tts is not installed in this Python env. " - "Install edge-tts or use skills/paper2poster/scripts/generate_audio.py." - ) from exc - - -DEFAULT_VOICE = "en-US-AriaNeural" -TIMINGS_SCHEMA_VERSION = "paper2video_edge_word_boundaries.v1" - - -def load_rate_plan(path: Path, *, allow_unsafe: bool) -> str: - try: - payload = json.loads(path.read_text(encoding="utf-8")) - except FileNotFoundError: - sys.exit(f"[generate_edge_audio] rate plan not found: {path}") - except json.JSONDecodeError as exc: - sys.exit(f"[generate_edge_audio] invalid rate plan {path}: {exc}") - if payload.get("schema_version") != "paper2video_tts_rate_plan.v1": - sys.exit(f"[generate_edge_audio] unsupported rate plan schema: {payload.get('schema_version')}") - status = str(payload.get("status") or "") - safe = bool(payload.get("safe")) - if not safe and not allow_unsafe: - sys.exit( - "[generate_edge_audio] rate plan is not safe for automatic TTS regeneration " - f"(status={status}). Rewrite the narration script first, or pass " - "--allow-unsafe-rate-plan only for an explicit experiment." - ) - if status == "needs_script_rewrite" and not allow_unsafe: - sys.exit("[generate_edge_audio] rate plan requires script rewrite; refusing to hide it with TTS rate.") - rate = str(payload.get("recommended_edge_rate") or "+0%") - if not rate.endswith("%") or not (rate.startswith("+") or rate.startswith("-")): - sys.exit(f"[generate_edge_audio] invalid recommended_edge_rate in {path}: {rate!r}") - return rate - - -async def synthesize_section(text: str, *, voice: str, rate: str, pitch: str, out_path: Path) -> None: - communicate = edge_tts.Communicate(text=text, voice=voice, rate=rate, pitch=pitch) - await communicate.save(str(out_path)) - - -def edge_ticks_to_seconds(raw: object) -> float: - try: - value = float(raw) - except (TypeError, ValueError): - return 0.0 - # edge-tts WordBoundary offsets are 100ns ticks. - return value / 10_000_000.0 - - -async def synthesize_section_with_timings( - text: str, - *, - voice: str, - rate: str, - pitch: str, - out_path: Path, -) -> list[dict]: - communicate = edge_tts.Communicate( - text=text, - voice=voice, - rate=rate, - pitch=pitch, - boundary="WordBoundary", - ) - words: list[dict] = [] - with out_path.open("wb") as fh: - async for chunk in communicate.stream(): - kind = chunk.get("type") - if kind == "audio": - data = chunk.get("data") - if data: - fh.write(data) - elif kind == "WordBoundary": - start = edge_ticks_to_seconds(chunk.get("offset")) - duration = edge_ticks_to_seconds(chunk.get("duration")) - words.append({ - "text": str(chunk.get("text") or ""), - "start": round(start, 3), - "end": round(start + max(duration, 0.0), 3), - "duration": round(max(duration, 0.0), 3), - }) - return words - - -async def synthesize_all( - sections: list[dict], - *, - voice: str, - rate: str, - pitch: str, - outdir: Path, - collect_timings: bool, -) -> tuple[list[dict], list[dict]]: - manifest = [] - timing_sections = [] - for sec in sections: - sid = str(sec.get("id") or "").strip() - text = str(sec.get("text") or "").strip() - if not sid: - raise ValueError("every script section must have an id") - if not text: - raise ValueError(f"section {sid} has empty text") - out_path = outdir / f"{sid}.mp3" - print(f"[edge-tts] {sid} ({len(text)} chars, voice={voice}, rate={rate}) -> {out_path}") - words: list[dict] = [] - if collect_timings: - words = await synthesize_section_with_timings( - text, - voice=voice, - rate=rate, - pitch=pitch, - out_path=out_path, - ) - else: - await synthesize_section(text, voice=voice, rate=rate, pitch=pitch, out_path=out_path) - manifest.append({ - "id": sid, - "heading": sec.get("heading", sid), - "file": out_path.name, - "bytes": out_path.stat().st_size, - "provider": "edge-tts", - "voice": voice, - "rate": rate, - "pitch": pitch, - "word_boundaries": len(words), - }) - if collect_timings: - timing_sections.append({ - "id": sid, - "heading": sec.get("heading", sid), - "file": out_path.name, - "words": words, - }) - return manifest, timing_sections - - -def main() -> int: - ap = argparse.ArgumentParser(description="Generate paper2video narration audio with edge-tts.") - ap.add_argument("script", help="Path to script JSON") - ap.add_argument("--outdir", required=True, help="Directory to write <id>.mp3 files") - ap.add_argument("--voice", default=None, - help=f"Edge voice name (default: script.edge_voice or {DEFAULT_VOICE})") - ap.add_argument("--rate", default=None, help="Edge rate adjustment, e.g. +0%%, -8%%, +10%%") - ap.add_argument("--rate-plan", default=None, - help="Optional plan_tts_rate.py JSON. Uses recommended_edge_rate and refuses unsafe plans.") - ap.add_argument("--allow-unsafe-rate-plan", action="store_true", - help="Allow a rate plan whose status says the script should be rewritten. Experimental only.") - ap.add_argument("--pitch", default="+0Hz", help="Edge pitch adjustment, e.g. +0Hz") - ap.add_argument("--timings-out", default=None, - help="Optional JSON path for Edge WordBoundary timings used by visual cue alignment.") - args = ap.parse_args() - - script_path = Path(args.script).resolve() - payload = json.loads(script_path.read_text(encoding="utf-8")) - sections = payload.get("sections") or [] - if not isinstance(sections, list) or not sections: - sys.exit("[generate_edge_audio] script JSON has no sections array") - - voice = args.voice or payload.get("edge_voice") or DEFAULT_VOICE - if args.rate_plan: - rate = load_rate_plan(Path(args.rate_plan).resolve(), allow_unsafe=args.allow_unsafe_rate_plan) - if args.rate and args.rate != rate: - print(f"[generate_edge_audio] --rate-plan overrides --rate {args.rate} -> {rate}") - else: - rate = args.rate or "+0%" - outdir = Path(args.outdir).resolve() - outdir.mkdir(parents=True, exist_ok=True) - - try: - manifest, timing_sections = asyncio.run( - synthesize_all( - sections, - voice=voice, - rate=rate, - pitch=args.pitch, - outdir=outdir, - collect_timings=args.timings_out is not None, - ) - ) - except Exception as exc: - sys.exit(f"[generate_edge_audio] {exc}") - - (outdir / "manifest.json").write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") - if args.timings_out: - timings_path = Path(args.timings_out).resolve() - timings_path.parent.mkdir(parents=True, exist_ok=True) - timings_payload = { - "schema_version": TIMINGS_SCHEMA_VERSION, - "provider": "edge-tts", - "voice": voice, - "rate": rate, - "pitch": args.pitch, - "sections": timing_sections, - } - timings_path.write_text(json.dumps(timings_payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") - print(f"[edge-tts] wrote word-boundary timings to {timings_path}") - print(f"\n[edge-tts] wrote {len(manifest)} clips to {outdir}") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/ResearchStudio-Reel/skills/paper2video/scripts/generate_visual_cues.py b/ResearchStudio-Reel/skills/paper2video/scripts/generate_visual_cues.py index d4e86d2..1e3d1ff 100644 --- a/ResearchStudio-Reel/skills/paper2video/scripts/generate_visual_cues.py +++ b/ResearchStudio-Reel/skills/paper2video/scripts/generate_visual_cues.py @@ -16,7 +16,6 @@ import math import os import re -import shutil import subprocess import sys import xml.etree.ElementTree as ET @@ -24,6 +23,8 @@ from pathlib import Path from typing import Iterable +from pptx2video.ffmpeg import find_ffmpeg_pair + try: from extract_pptx_elements import extract_pptx_elements except ImportError: # pragma: no cover - same-dir import when used as a script @@ -591,6 +592,118 @@ def geometry_text_overlap(chunk: str, semantic_region: Region, geometry_region: return len((semantic_tokens | chunk_tokens) & geometry_tokens) +# A title or tagline can be exported as several one-line text boxes in one +# group. Expand its cue to the complete wrapped run before the renderer tightens +# the box to painted ink. Multi-line paragraph boxes remain separate. +GROUP_WRAPPED_LINES = os.environ.get( + "VIDEO_CUE_GROUP_WRAPPED_LINES", "1" +).strip().lower() not in ("0", "off", "false", "no") +WRAP_X_EPS = float(os.environ.get("VIDEO_CUE_WRAP_X_EPS", "0.02")) +WRAP_MIN_ASPECT = float(os.environ.get("VIDEO_CUE_WRAP_MIN_ASPECT", "2.5")) +WRAP_GAP_MAX_FRAC = float(os.environ.get("VIDEO_CUE_WRAP_GAP_FRAC", "0.15")) +WRAP_H_LO, WRAP_H_HI = 0.6, 1.6 + + +def _region_is_single_line(region: Region) -> bool: + _, _, width, height = region.box + return height > 0 and (width / height) >= WRAP_MIN_ASPECT + + +def _wrapped_adjacent(upper: Region, lower: Region) -> bool: + """Return whether ``lower`` is a wrapped continuation of ``upper``.""" + upper_x, upper_y, _, upper_height = upper.box + lower_x, lower_y, _, lower_height = lower.box + if abs(upper_x - lower_x) > WRAP_X_EPS: + return False + if ( + upper_height <= 0 + or lower_height <= 0 + or not (WRAP_H_LO <= lower_height / upper_height <= WRAP_H_HI) + ): + return False + if not (_region_is_single_line(upper) and _region_is_single_line(lower)): + return False + gap = lower_y - (upper_y + upper_height) + minimum_height = min(upper_height, lower_height) + return -0.5 * minimum_height <= gap <= WRAP_GAP_MAX_FRAC * minimum_height + + +def _wrapped_run(target: Region, siblings: list[Region]) -> list[Region]: + ordered = sorted(siblings, key=lambda region: region.box[1]) + try: + index = ordered.index(target) + except ValueError: + return [target] + run = [target] + cursor = index + while cursor + 1 < len(ordered) and _wrapped_adjacent( + ordered[cursor], ordered[cursor + 1] + ): + run.append(ordered[cursor + 1]) + cursor += 1 + cursor = index + while cursor - 1 >= 0 and _wrapped_adjacent( + ordered[cursor - 1], ordered[cursor] + ): + run.insert(0, ordered[cursor - 1]) + cursor -= 1 + return run + + +def union_wrapped_line_cues( + cue_entries: list[dict], + regions: list[Region], + *, + evidence_entries: list[tuple[dict, dict, dict, dict]] | None = None, +) -> None: + """Expand PPTX text cues and their evidence to complete wrapped runs.""" + if not GROUP_WRAPPED_LINES: + return + by_id = {region.region_id: region for region in regions if region.source == "pptx"} + evidence_by_cue = { + id(cue): (plan, audit, geometry) + for cue, plan, audit, geometry in (evidence_entries or []) + } + for cue in cue_entries: + if cue.get("geometry_source") != "pptx": + continue + target = by_id.get(cue.get("geometry_target")) + if ( + target is None + or target.shape_type != "TEXT_BOX" + or not target.parent_id + or not _region_is_single_line(target) + ): + continue + siblings = [ + region + for region in by_id.values() + if region.parent_id == target.parent_id and region.shape_type == "TEXT_BOX" + ] + if len(siblings) < 2: + continue + run = _wrapped_run(target, siblings) + if len(run) < 2: + continue + box = union_region_boxes(run) + if not box: + continue + rounded_box = round_list(box) + rounded_point = round_list(point_from_box(box)) + grouped_ids = [region.region_id for region in run] + cue["box"] = rounded_box + cue["point"] = rounded_point + cue["geometry_box"] = rounded_box + cue["grouped_wrapped_lines"] = grouped_ids + for entry in evidence_by_cue.get(id(cue), ()): + entry["geometry_box"] = rounded_box + entry["grouped_wrapped_lines"] = grouped_ids + if "point" in entry: + entry["point"] = rounded_point + if "region_box" in entry: + entry["region_box"] = rounded_box + + def geometry_match_score(chunk: str, semantic_region: Region, geometry_region: Region) -> tuple[float, list[str], float, float, float]: semantic_box = semantic_region.box geometry_box = geometry_region.box @@ -1706,42 +1819,11 @@ def load_anchor_contract(path: Path | None) -> dict[int, dict[int, dict]]: return out -def find_tool(name: str) -> str | None: - return shutil.which(name) - - -def imageio_ffmpeg_binary() -> str | None: - try: - import imageio_ffmpeg # type: ignore - - return imageio_ffmpeg.get_ffmpeg_exe() - except Exception: - return None - - -def find_ffmpeg_pair() -> tuple[str | None, str | None]: - env_ffmpeg = os.getenv("PAPER2VIDEO_FFMPEG") or os.getenv("FFMPEG_BINARY") - if env_ffmpeg and Path(env_ffmpeg).expanduser().is_file(): - env_ffprobe = os.getenv("PAPER2VIDEO_FFPROBE") - if env_ffprobe and Path(env_ffprobe).expanduser().is_file(): - return str(Path(env_ffmpeg).expanduser()), str(Path(env_ffprobe).expanduser()) - return str(Path(env_ffmpeg).expanduser()), str(Path(env_ffmpeg).expanduser()) - - fallback = imageio_ffmpeg_binary() - if fallback: - return fallback, fallback - - ffmpeg = find_tool("ffmpeg") - ffprobe = find_tool("ffprobe") - if ffmpeg and ffprobe: - return ffmpeg, ffprobe - if ffmpeg: - return ffmpeg, ffmpeg - return None, None - - def probe_duration(audio: Path) -> float: - ffmpeg, ffprobe = find_ffmpeg_pair() + ffmpeg, ffprobe = find_ffmpeg_pair( + required=False, + component="generate_visual_cues", + ) if ffprobe and ffmpeg and ffprobe != ffmpeg: out = subprocess.run( [ffprobe, "-v", "error", "-show_entries", "format=duration", @@ -1912,6 +1994,7 @@ def generate(project: Path, *, svg_dir: Path, sections: list[Section], audit_entries = [] plan_entries = [] geometry_entries = [] + cue_evidence_entries = [] slide_contract = (anchor_contract or {}).get(sec.index, {}) if anchor_contract and not slide_contract: msg = f"slide {sec.index} has no anchor contract entries" @@ -2093,7 +2176,19 @@ def generate(project: Path, *, svg_dir: Path, sections: list[Section], "end": end, "seconds": round(end - start, 3), }) + if cue is not None: + cue_evidence_entries.append( + (cue, plan_entry, audit_entries[-1], geometry_entries[-1]) + ) + # Treat same-group single-line title fragments as one visual unit. The + # standalone renderer then tightens transparent text to the painted ink + # while preserving filled cards. + union_wrapped_line_cues( + cue_entries, + regions, + evidence_entries=cue_evidence_entries, + ) cues_payload["slides"].append({ "index": sec.index, "id": sec.sid, diff --git a/ResearchStudio-Reel/skills/paper2video/scripts/plan_tts_rate.py b/ResearchStudio-Reel/skills/paper2video/scripts/plan_tts_rate.py index 73b1d89..a50d0d2 100644 --- a/ResearchStudio-Reel/skills/paper2video/scripts/plan_tts_rate.py +++ b/ResearchStudio-Reel/skills/paper2video/scripts/plan_tts_rate.py @@ -11,15 +11,15 @@ import argparse import json -import os import re -import shutil import subprocess import sys from datetime import datetime, timezone from pathlib import Path from typing import Any +from pptx2video.ffmpeg import find_ffmpeg_pair + SCHEMA_VERSION = "paper2video_tts_rate_plan.v1" @@ -41,37 +41,6 @@ def write_json(path: Path, payload: Any) -> None: path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") -def which(name: str) -> str | None: - return shutil.which(name) - - -def imageio_ffmpeg_binary() -> str | None: - try: - import imageio_ffmpeg # type: ignore - return imageio_ffmpeg.get_ffmpeg_exe() - except Exception: - return None - - -def find_ffmpeg_pair() -> tuple[str | None, str | None]: - env_ffmpeg = os.getenv("PAPER2VIDEO_FFMPEG") or os.getenv("FFMPEG_BINARY") - if env_ffmpeg and Path(env_ffmpeg).expanduser().is_file(): - env_ffprobe = os.getenv("PAPER2VIDEO_FFPROBE") - if env_ffprobe and Path(env_ffprobe).expanduser().is_file(): - return str(Path(env_ffmpeg).expanduser()), str(Path(env_ffprobe).expanduser()) - return str(Path(env_ffmpeg).expanduser()), str(Path(env_ffmpeg).expanduser()) - - fallback = imageio_ffmpeg_binary() - if fallback: - return fallback, fallback - - ffmpeg = which("ffmpeg") - ffprobe = which("ffprobe") - if ffmpeg and ffprobe: - return ffmpeg, ffprobe - return ffmpeg, ffprobe - - def probe_duration(path: Path, ffmpeg: str | None, ffprobe: str | None) -> float: if ffprobe: proc = subprocess.run( @@ -107,7 +76,7 @@ def load_script_ids(path: Path) -> list[str]: def speech_seconds_from_audio(script_json: Path, audio_dir: Path) -> tuple[float, list[dict[str, Any]]]: - ffmpeg, ffprobe = find_ffmpeg_pair() + ffmpeg, ffprobe = find_ffmpeg_pair(required=False, component="plan_tts_rate") details = [] total = 0.0 for sid in load_script_ids(script_json): diff --git a/ResearchStudio-Reel/skills/paper2video/scripts/render_video.py b/ResearchStudio-Reel/skills/paper2video/scripts/render_video.py deleted file mode 100755 index c38b5be..0000000 --- a/ResearchStudio-Reel/skills/paper2video/scripts/render_video.py +++ /dev/null @@ -1,2072 +0,0 @@ -#!/usr/bin/env python3 -""" -render_video.py — composite a presentation video (MP4) from a ppt-master deck -and per-slide narration MP3s. - -Pipeline position (Stage 3 of paper2video): - Inputs: - --pptx : the canonical PPTX written by ppt-master - --audio-dir : a directory of per-slide MP3s (paper2poster's TTS output) - --out : MP4 destination path - - Steps: - 1. Prefer ppt-master's final SVG frames (svg_final/*.svg) → PNG/slide - via a browser renderer; fall back to PPTX → PDF → PNG only when SVG - frames are unavailable or explicitly disabled. - 2. Pair each slide PNG with its matching MP3 (by script order) - 3. Probe each MP3's duration with ffprobe - 4. Build a per-slide concat segment, optionally pad trailing silence - 5. Concat into a single H.264 / AAC MP4 with ffmpeg's concat demuxer - 6. Verify the output plays and report duration - -Why prefer svg_final over PPTX → LibreOffice → PDF: - ppt-master authors and previews slides as SVG before exporting the PPTX. - LibreOffice can reflow text and vector geometry differently from - PowerPoint/Keynote, producing video frames that no longer match the deck the - user inspected. The final SVGs are the same 16:9 visual source used before - PPTX export, including expanded icon paths, so they are the safest source - for the video raster frames. The PPTX remains a required deliverable. - -ffmpeg fallback: - If system ffmpeg/ffprobe aren't on PATH, we fall back to imageio_ffmpeg's - bundled static binary. install with `pip install imageio-ffmpeg`. We don't - silently use moviepy or ffmpeg-python — they wrap the same binary, just - with more layers to debug when something goes wrong. -""" - -from __future__ import annotations - -import argparse -import json -import math -import os -import re -import shlex -import shutil -import struct -import subprocess -import sys -import tempfile -import zlib -from dataclasses import dataclass -from datetime import datetime, timezone -from pathlib import Path - -RESOLUTIONS = { - "720p": (1280, 720), - "1080p": (1920, 1080), - "1440p": (2560, 1440), - "4k": (3840, 2160), -} - -DURATION_REPORT_SCHEMA_VERSION = "paper2video_duration_report.v1" -HIGHLIGHT_BORDER_ALPHA = 0.68 -HIGHLIGHT_BOX_EXPAND_MULTIPLIER = 1.0 -SPOTLIGHT_DIM_COLOR = "0x000000" -SPOTLIGHT_BORDER_ALPHA = 0.34 -SPOTLIGHT_MAX_ALPHA = 0.24 -SPOTLIGHT_FEATHER_RATIO = 0.052 -SPOTLIGHT_MIN_FEATHER_PX = 56 -SPOTLIGHT_FEATHER_THICKNESS_MULTIPLIER = 8 -SPOTLIGHT_INNER_PAD_MULTIPLIER = 1.0 -CURSOR_MOVE_SECONDS = 0.55 -CURSOR_POINTER_FILL = "0x1E293B" -CURSOR_POINTER_BORDER = "0xF8FAFC" -CURSOR_POINTER_SHADOW = "0x000000" -CURSOR_POINTER_FILL_ALPHA = 0.94 -CURSOR_POINTER_BORDER_ALPHA = 0.96 -CURSOR_POINTER_SHADOW_ALPHA = 0.26 -CURSOR_OVERLAY_TIP_OFFSET = 3 -LASER_DOT_FILL = (239, 68, 68) -LASER_DOT_HALO = (248, 113, 113) -LASER_DOT_CORE_ALPHA = 0.96 -LASER_DOT_HALO_ALPHA = 0.34 -LASER_DOT_SIZE_MULTIPLIER = 0.55 -LASER_DOT_MIN_DIAMETER = 28 -LASER_DOT_MAX_DIAMETER = 48 -VALID_HIGHLIGHT_STYLES = { - "box", - "spotlight", - "cursor", - "box_cursor", - "spotlight_cursor", - "laser", - "box_laser", - "spotlight_laser", -} -CURSOR_STYLES = {"cursor", "box_cursor", "spotlight_cursor"} -LASER_STYLES = {"laser", "box_laser", "spotlight_laser"} -SPOTLIGHT_STYLES = {"spotlight", "spotlight_cursor", "spotlight_laser"} - - -# --------------------------------------------------------------------------- -# Tool discovery -# --------------------------------------------------------------------------- - -def _which(name: str) -> str | None: - return shutil.which(name) - - -def _imageio_ffmpeg_binary() -> str | None: - """Fall back to the imageio_ffmpeg static binary if it's installed. - - Some environments don't have system ffmpeg but do have the pip package, - which ships a portable binary. We use it for both ffmpeg and ffprobe - (the package ships ffmpeg only, so probing happens through ffmpeg's - own `-i` output as a last resort). - """ - try: - import imageio_ffmpeg # type: ignore - return imageio_ffmpeg.get_ffmpeg_exe() - except Exception: - return None - - -def find_libreoffice() -> str: - for cand in ("libreoffice", "soffice"): - path = _which(cand) - if path: - return path - sys.exit( - "[render_video] LibreOffice not found on PATH. Install with:\n" - " Ubuntu/Debian: sudo apt-get install -y libreoffice\n" - " macOS: brew install --cask libreoffice\n" - ) - - -def find_pdftoppm() -> str: - path = _which("pdftoppm") - if path: - return path - sys.exit( - "[render_video] pdftoppm not found (part of poppler-utils). Install with:\n" - " Ubuntu/Debian: sudo apt-get install -y poppler-utils\n" - " macOS: brew install poppler\n" - ) - - -def find_chrome() -> str | None: - """Return a local Chromium/Chrome executable for SVG screenshots.""" - for env_name in ("PAPER2VIDEO_CHROME", "CHROME", "CHROMIUM"): - raw = os.getenv(env_name) - if raw and Path(raw).expanduser().is_file(): - return str(Path(raw).expanduser()) - - for candidate in ( - "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", - "/Applications/Chromium.app/Contents/MacOS/Chromium", - "/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge", - ): - if Path(candidate).is_file(): - return candidate - - for name in ("google-chrome", "google-chrome-stable", "chromium", "chromium-browser", "msedge"): - path = _which(name) - if path: - return path - return None - - -def find_ffmpeg_pair() -> tuple[str, str]: - """Return (ffmpeg, ffprobe) paths or exit with guidance.""" - env_ffmpeg = os.getenv("PAPER2VIDEO_FFMPEG") or os.getenv("FFMPEG_BINARY") - if env_ffmpeg: - env_path = Path(env_ffmpeg).expanduser() - if env_path.is_file(): - env_ffprobe = os.getenv("PAPER2VIDEO_FFPROBE") - if env_ffprobe and Path(env_ffprobe).expanduser().is_file(): - return str(env_path), str(Path(env_ffprobe).expanduser()) - return str(env_path), str(env_path) - - fallback = _imageio_ffmpeg_binary() - if fallback: - # Prefer the bundled static ffmpeg when available. In the ACL26 - # environment, system ffmpeg is 2.4.x and lacks newer filters/codecs, - # while imageio-ffmpeg provides a modern static build. - return fallback, fallback - - ffmpeg = _which("ffmpeg") - ffprobe = _which("ffprobe") - if ffmpeg and ffprobe: - return ffmpeg, ffprobe - - sys.exit( - "[render_video] ffmpeg/ffprobe not found and imageio_ffmpeg is not installed.\n" - "Pick one:\n" - " • System install: sudo apt-get install -y ffmpeg\n" - " • Python fallback: pip install imageio-ffmpeg\n" - ) - - -# --------------------------------------------------------------------------- -# Stage A — deck source → per-slide PNG -# --------------------------------------------------------------------------- - -def natural_key(path: Path) -> list[object]: - return [int(part) if part.isdigit() else part.lower() for part in re.split(r"(\d+)", path.name)] - - -def discover_svg_dir(project_path: Path, explicit_svg_dir: Path | None, frame_source: str) -> Path | None: - if explicit_svg_dir is not None: - if not explicit_svg_dir.is_dir(): - sys.exit(f"[render_video] --svg-dir not found: {explicit_svg_dir}") - return explicit_svg_dir - - for name in ("svg_final", "svg_output"): - candidate = project_path / name - if candidate.is_dir() and list(candidate.glob("*.svg")): - return candidate - - if frame_source == "svg": - sys.exit( - "[render_video] --frame-source svg requested, but no SVG deck was found. " - "Expected <project>/svg_final/*.svg or <project>/svg_output/*.svg." - ) - return None - - -def collect_svgs(svg_dir: Path) -> list[Path]: - svgs = sorted(svg_dir.glob("*.svg"), key=natural_key) - if not svgs: - sys.exit(f"[render_video] SVG deck has no .svg files: {svg_dir}") - return svgs - - -def _resolve_svg_asset_href(raw: str, *, svg_path: Path, project_path: Path) -> str: - raw = raw.strip() - if not raw or raw.startswith(("#", "data:", "http://", "https://", "file:")): - return raw - - asset_part, sep, fragment = raw.partition("#") - asset_path = Path(asset_part) - candidates = [svg_path.parent / asset_path, project_path / asset_path] - for candidate in candidates: - if candidate.is_file(): - uri = candidate.resolve().as_uri() - return f"{uri}{sep}{fragment}" if sep else uri - return raw - - -def _inline_svg_html(svg_path: Path, project_path: Path) -> str: - text = svg_path.read_text(encoding="utf-8") - text = re.sub(r"^\s*<\?xml[^>]*>\s*", "", text) - - def replace_href(match: re.Match[str]) -> str: - attr, quote, href = match.group(1), match.group(2), match.group(3) - resolved = _resolve_svg_asset_href(href, svg_path=svg_path, project_path=project_path) - return f"{attr}={quote}{resolved}{quote}" - - text = re.sub(r"((?:xlink:)?href)=(['\"])([^'\"]+)\2", replace_href, text) - base_uri = svg_path.parent.resolve().as_uri() + "/" - return ( - "<!doctype html><html><head><meta charset=\"utf-8\">" - f"<base href=\"{base_uri}\">" - "<style>" - "html,body{margin:0;width:100%;height:100%;overflow:hidden;background:white;}" - "body>svg{width:100vw!important;height:100vh!important;display:block;}" - "</style></head><body>" - f"{text}" - "</body></html>" - ) - - -def render_svg_frames( - svgs: list[Path], - out_dir: Path, - *, - project_path: Path, - width: int, - height: int, - browser_executable: str | None = None, -) -> list[Path]: - """Render final ppt-master SVGs to PNG frames with a browser renderer.""" - try: - from playwright.sync_api import sync_playwright # type: ignore - except Exception: - sys.exit( - "[render_video] SVG frame rendering requires Playwright for Python. " - "Install it in this environment or rerun with --frame-source pptx for " - "the legacy LibreOffice path." - ) - - out_dir.mkdir(parents=True, exist_ok=True) - html_dir = out_dir.parent / "svg_html" - if html_dir.exists(): - shutil.rmtree(html_dir) - html_dir.mkdir(parents=True, exist_ok=True) - - browser_path = browser_executable or find_chrome() - launch_kwargs: dict[str, object] = {"headless": True} - if browser_path: - launch_kwargs["executable_path"] = browser_path - - frames: list[Path] = [] - try: - with sync_playwright() as p: - browser = p.chromium.launch(**launch_kwargs) - page = browser.new_page( - viewport={"width": width, "height": height}, - device_scale_factor=1, - ) - for idx, svg_path in enumerate(svgs, start=1): - html_path = html_dir / f"slide-{idx:02d}.html" - html_path.write_text(_inline_svg_html(svg_path, project_path), encoding="utf-8") - page.goto(html_path.resolve().as_uri(), wait_until="networkidle", timeout=60000) - frame_path = out_dir / f"slide-{idx:02d}.png" - page.screenshot(path=str(frame_path), full_page=False, omit_background=False) - frames.append(frame_path) - browser.close() - except Exception as exc: - sys.exit(f"[render_video] browser SVG render failed: {exc}") - - if len(frames) != len(svgs): - sys.exit(f"[render_video] expected {len(svgs)} SVG frame(s), rendered {len(frames)}") - return frames - - -def copy_frames(frames: list[Path], frames_out: Path) -> list[Path]: - frames_out = frames_out.resolve() - if frames and frames[0].parent.resolve() == frames_out: - return frames - if frames_out.exists(): - shutil.rmtree(frames_out) - frames_out.mkdir(parents=True, exist_ok=True) - copied: list[Path] = [] - for idx, frame in enumerate(frames, start=1): - dest = frames_out / f"slide-{idx:02d}{frame.suffix.lower() or '.png'}" - shutil.copy2(frame, dest) - copied.append(dest) - return copied - -def pptx_to_pdf(pptx_path: Path, out_dir: Path, libreoffice: str) -> Path: - """Convert PPTX to PDF in `out_dir`. Return the PDF path.""" - out_dir.mkdir(parents=True, exist_ok=True) - # Use a throwaway user profile so a logged-in LibreOffice GUI doesn't - # hold a lock on the default ~/.config/libreoffice profile. - with tempfile.TemporaryDirectory(prefix="lo_profile_") as profile_dir: - cmd = [ - libreoffice, - f"-env:UserInstallation=file://{profile_dir}", - "--headless", "--norestore", "--nologo", - "--convert-to", "pdf", - "--outdir", str(out_dir), - str(pptx_path), - ] - proc = subprocess.run(cmd, capture_output=True, text=True, timeout=600) - if proc.returncode != 0: - sys.exit(f"[render_video] LibreOffice failed (exit {proc.returncode}):\n" - f"stdout: {proc.stdout}\nstderr: {proc.stderr}") - - pdf = out_dir / (pptx_path.stem + ".pdf") - if not pdf.exists(): - sys.exit(f"[render_video] expected PDF not produced: {pdf}\n" - f"LibreOffice stdout:\n{proc.stdout}") - return pdf - - -def pdf_to_pngs(pdf_path: Path, out_dir: Path, dpi: int, pdftoppm: str) -> list[Path]: - """Rasterize a PDF to one PNG per page. - - We use the `-png` switch so output is RGB without alpha, and pad the page - number so sorted order matches slide order even with 100+ slides. - """ - out_dir.mkdir(parents=True, exist_ok=True) - prefix = out_dir / "slide" - cmd = [ - pdftoppm, "-png", "-r", str(dpi), - # Wide enough for any sane deck; pdftoppm default is 6 already. - str(pdf_path), str(prefix), - ] - proc = subprocess.run(cmd, capture_output=True, text=True, timeout=600) - if proc.returncode != 0: - sys.exit(f"[render_video] pdftoppm failed:\n{proc.stderr}") - - pngs = sorted(out_dir.glob("slide-*.png")) - if not pngs: - sys.exit(f"[render_video] pdftoppm produced no PNGs in {out_dir}") - return pngs - - -# --------------------------------------------------------------------------- -# Stage B — pair frames with audio -# --------------------------------------------------------------------------- - -@dataclass -class SlidePair: - index: int # 1-based slide number - frame: Path - audio: Path - duration: float # seconds (from probe) - - -@dataclass -class VisualCue: - cue_type: str - start: float - end: float - box: tuple[float, float, float, float] | None = None - point: tuple[float, float] | None = None - color: str = "#64748B" - opacity: float = 0.18 - border: int = 5 - size: int | None = None - style: str = "spotlight_laser" - - -def _load_script_order(script_json: Path) -> list[str]: - try: - payload = json.loads(script_json.read_text(encoding="utf-8")) - except json.JSONDecodeError as exc: - sys.exit(f"[render_video] invalid script JSON {script_json}: {exc}") - - sections = payload.get("sections") or [] - if not isinstance(sections, list): - sys.exit(f"[render_video] script JSON has no sections array: {script_json}") - - ids: list[str] = [] - for idx, sec in enumerate(sections, start=1): - if not isinstance(sec, dict) or not sec.get("id"): - sys.exit(f"[render_video] script section {idx} is missing an id in {script_json}") - ids.append(str(sec["id"])) - if not ids: - sys.exit(f"[render_video] script JSON has an empty sections array: {script_json}") - return ids - - -def _load_manifest_order(manifest_json: Path) -> list[str]: - try: - payload = json.loads(manifest_json.read_text(encoding="utf-8")) - except (FileNotFoundError, json.JSONDecodeError): - return [] - - # paper2poster's generate_audio.py writes a plain list of - # {"id": ..., "file": ...} entries. Be liberal in case a later manifest - # wraps that list under a root key. - entries = payload.get("sections") if isinstance(payload, dict) else payload - if not isinstance(entries, list): - return [] - - files: list[str] = [] - for item in entries: - if not isinstance(item, dict): - continue - if item.get("file"): - files.append(str(item["file"])) - elif item.get("id"): - files.append(f"{item['id']}.mp3") - return files - - -def autodetect_script_json(project_path: Path, audio_dir: Path) -> Path | None: - for candidate in ( - audio_dir / "script.json", - project_path / "assets" / "meta" / "narration.json", - project_path / "narration.json", - ): - if candidate.is_file(): - return candidate - return None - - -def collect_audio( - audio_dir: Path, - *, - script_json: Path | None = None, - project_path: Path | None = None, -) -> list[Path]: - if not audio_dir.is_dir(): - sys.exit(f"[render_video] audio dir not found: {audio_dir}") - - if script_json is None and project_path is not None: - script_json = autodetect_script_json(project_path, audio_dir) - - if script_json is not None: - script_json = script_json.resolve() - ids = _load_script_order(script_json) - ordered = [audio_dir / f"{sid}.mp3" for sid in ids] - missing = [p.name for p in ordered if not p.is_file()] - if missing: - sys.exit( - f"[render_video] script/audio mismatch using {script_json}:\n" - f" missing mp3s under {audio_dir}: {missing}" - ) - print(f"[render_video] audio order from {script_json}") - return ordered - - manifest_order = _load_manifest_order(audio_dir / "manifest.json") - if manifest_order: - ordered = [audio_dir / name for name in manifest_order] - if all(p.is_file() for p in ordered): - print(f"[render_video] audio order from {audio_dir / 'manifest.json'}") - return ordered - - mp3s = sorted(audio_dir.glob("*.mp3")) - if not mp3s: - sys.exit(f"[render_video] no .mp3 files in {audio_dir}") - print("[render_video] audio order from sorted *.mp3 filenames") - return mp3s - - -def probe_duration(audio: Path, ffprobe: str, ffmpeg: str) -> float: - """Return duration in seconds. - - Prefers ffprobe; falls back to parsing ffmpeg's stderr when only the - imageio_ffmpeg static binary is available (it doesn't ship ffprobe). - """ - if ffprobe != ffmpeg: # we have a real ffprobe - cmd = [ffprobe, "-v", "error", "-show_entries", "format=duration", - "-of", "default=noprint_wrappers=1:nokey=1", str(audio)] - out = subprocess.run(cmd, capture_output=True, text=True) - if out.returncode == 0 and out.stdout.strip(): - return float(out.stdout.strip()) - - # Fallback: ffmpeg -i prints "Duration: HH:MM:SS.xx" - out = subprocess.run([ffmpeg, "-i", str(audio)], capture_output=True, text=True) - m = re.search(r"Duration:\s+(\d+):(\d+):([\d.]+)", out.stderr) - if not m: - sys.exit(f"[render_video] could not probe duration for {audio.name}") - h, mm, ss = m.group(1), m.group(2), m.group(3) - return int(h) * 3600 + int(mm) * 60 + float(ss) - - -def pair_slides(frames: list[Path], audio_files: list[Path], - ffprobe: str, ffmpeg: str) -> list[SlidePair]: - """Match frames to audio in sorted order and probe durations. - - We match by index, not by filename, because LibreOffice writes - slide-01.png … slide-NN.png while paper2poster writes audio named after - the original slide stems. The only thing we need is that *count matches - and order matches*. The preferred order source is a script JSON - (`--script-json`, `audio/script.json`, `assets/meta/narration.json`, or `narration.json`); otherwise - we fall back to manifest order or sorted filenames. For ppt-master decks, - sorted filenames still work when notes use numeric prefixes. - - Order is guaranteed because: - • LibreOffice walks the PPTX slides in order - • paper2poster writes one MP3 per script section in array order - • notes_to_script.py walks notes/*.md sorted by filename, which is the - same sort ppt-master uses for SVGs. - """ - if len(frames) != len(audio_files): - sys.exit( - f"[render_video] slide/audio count mismatch: " - f"{len(frames)} frames vs {len(audio_files)} mp3s.\n" - f" Frames found: {[p.name for p in frames]}\n" - f" Audio found: {[p.name for p in audio_files]}\n" - f" Most likely a notes/<slide>.md was missing during Stage 2 — " - f"regenerate audio/script.json and re-run paper2poster's " - f"generate_audio.py before muxing." - ) - pairs = [] - for i, (frame, audio) in enumerate(zip(frames, audio_files), start=1): - dur = probe_duration(audio, ffprobe, ffmpeg) - pairs.append(SlidePair(index=i, frame=frame, audio=audio, duration=dur)) - return pairs - - -# --------------------------------------------------------------------------- -# Optional attention overlays — semantic highlight boxes / cursor markers -# --------------------------------------------------------------------------- - -def _utc_now() -> str: - return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z") - - -def _as_float_list(value: object, *, length: int, field: str) -> list[float]: - if not isinstance(value, list) or len(value) != length: - raise ValueError(f"{field} must be a list of {length} numbers") - out: list[float] = [] - for item in value: - if not isinstance(item, (int, float)): - raise ValueError(f"{field} must be a list of {length} numbers") - out.append(float(item)) - return out - - -def _clamp01(value: float) -> float: - return max(0.0, min(1.0, value)) - - -def _normalize_color(raw: object, fallback: str) -> str: - color = str(raw or fallback).strip() - if re.match(r"^#[0-9A-Fa-f]{6}$", color): - return "0x" + color[1:] - if re.match(r"^0x[0-9A-Fa-f]{6}$", color): - return color - if re.match(r"^[A-Za-z]+$", color): - return color - return "0x" + fallback.lstrip("#") - - -def _color_alpha(color: str, alpha: float) -> str: - return f"{color}@{max(0.0, min(alpha, 1.0)):.3f}" - - -def _cue_enabled_for_mode(cue_type: str, attention_mode: str) -> bool: - if attention_mode == "none": - return False - if attention_mode == "both": - return cue_type in {"highlight", "cursor"} - return cue_type == attention_mode - - -def _cue_time(raw: dict, pair: SlidePair, pad_tail: float) -> tuple[float, float]: - if "at" in raw and "start" not in raw: - start = float(raw["at"]) - else: - start = float(raw.get("start", 0.0)) - - if raw.get("end") is not None: - end = float(raw["end"]) - else: - end = start + float(raw.get("duration", 3.0)) - - segment_end = max(pair.duration + pad_tail, 0.1) - start = max(0.0, min(start, segment_end)) - end = max(start + 0.05, min(end, segment_end)) - return start, end - - -def load_visual_cues( - visual_cues_path: Path | None, - pairs: list[SlidePair], - *, - attention_mode: str, - highlight_style: str, - pad_tail: float, - allow_missing_visual_cues: bool = False, -) -> dict[int, list[VisualCue]]: - """Load cue JSON keyed by 1-based slide index. - - Accepted shape: - {"slides": [{"id": "audio_stem", "index": 1, "cues": [...]}]} - Each cue uses normalized coordinates: box=[x,y,w,h], point=[x,y]. - Highlight cues render as translucent boxes when `box` is available. - Legacy point-only cues are still accepted as soft dots. - """ - if attention_mode == "none": - return {} - if visual_cues_path is None: - message = ( - f"[render_video] attention mode '{attention_mode}' requires --visual-cues. " - "Use --attention-mode none for a user-approved no-highlight render, or pass " - "--allow-missing-visual-cues for a degraded/debug run." - ) - if not allow_missing_visual_cues: - sys.exit(message) - print(message + " Continuing without overlays.") - return {} - - try: - payload = json.loads(visual_cues_path.read_text(encoding="utf-8")) - except FileNotFoundError: - sys.exit(f"[render_video] visual cues file not found: {visual_cues_path}") - except json.JSONDecodeError as exc: - sys.exit(f"[render_video] invalid visual cues JSON {visual_cues_path}: {exc}") - - slides = payload.get("slides") if isinstance(payload, dict) else payload - if not isinstance(slides, list): - sys.exit(f"[render_video] visual cues JSON must contain a slides array: {visual_cues_path}") - - by_id = {pair.audio.stem: pair for pair in pairs} - by_index = {pair.index: pair for pair in pairs} - out: dict[int, list[VisualCue]] = {} - loaded = 0 - - for slide in slides: - if not isinstance(slide, dict): - sys.exit("[render_video] each visual cue slide entry must be an object") - pair: SlidePair | None = None - if slide.get("id") is not None: - pair = by_id.get(str(slide["id"])) - if pair is None and slide.get("index") is not None: - pair = by_index.get(int(slide["index"])) - if pair is None: - sys.exit(f"[render_video] visual cue slide does not match any audio stem/index: {slide}") - - raw_cues = slide.get("cues") or [] - if not isinstance(raw_cues, list): - sys.exit(f"[render_video] visual cues for slide {pair.index} must be an array") - - for raw in raw_cues: - if not isinstance(raw, dict): - sys.exit(f"[render_video] visual cue for slide {pair.index} must be an object") - cue_type = str(raw.get("type") or ("highlight" if attention_mode != "cursor" else "cursor")).strip() - if cue_type not in {"highlight", "cursor"}: - sys.exit(f"[render_video] unsupported visual cue type: {cue_type}") - if not _cue_enabled_for_mode(cue_type, attention_mode): - continue - - start, end = _cue_time(raw, pair, pad_tail) - color = _normalize_color(raw.get("color"), "#64748B") - opacity = float(raw.get("opacity", 0.18 if cue_type == "highlight" else 0.95)) - opacity = max(0.05, min(opacity, 1.0)) - border = max(1, int(raw.get("border", 5))) - try: - size = int(raw["size"]) if raw.get("size") is not None else None - except (TypeError, ValueError): - sys.exit(f"[render_video] visual cue size must be an integer on slide {pair.index}") - if size is not None: - size = max(1, size) - - if cue_type == "highlight": - box = None - if raw.get("box") is not None: - box_vals = _as_float_list(raw.get("box"), length=4, field="box") - x, y, w, h = box_vals - x = _clamp01(x) - y = _clamp01(y) - w = max(0.001, min(float(w), 1.0 - x)) - h = max(0.001, min(float(h), 1.0 - y)) - box = (x, y, w, h) - if raw.get("point") is not None: - point_vals = _as_float_list(raw.get("point"), length=2, field="point") - point = (_clamp01(point_vals[0]), _clamp01(point_vals[1])) - else: - point = (_clamp01(x + w / 2.0), _clamp01(y + h / 2.0)) - elif raw.get("point") is not None: - point_vals = _as_float_list(raw.get("point"), length=2, field="point") - point = (_clamp01(point_vals[0]), _clamp01(point_vals[1])) - else: - sys.exit(f"[render_video] highlight cue on slide {pair.index} needs either point or box") - style = str(raw.get("style") or highlight_style).strip() - cue = VisualCue(cue_type=cue_type, start=start, end=end, box=box, point=point, - color=color, opacity=opacity, border=border, size=size, style=style) - else: - point_vals = _as_float_list(raw.get("point"), length=2, field="point") - point = (_clamp01(point_vals[0]), _clamp01(point_vals[1])) - cue = VisualCue(cue_type=cue_type, start=start, end=end, point=point, - color=color, opacity=opacity, border=border, size=size, style="cursor") - - out.setdefault(pair.index, []).append(cue) - loaded += 1 - - print(f"[render_video] visual cues from {visual_cues_path}: {loaded} cue(s) enabled") - return out - - -def _enable_expr(cue: VisualCue) -> str: - return f"enable='between(t,{cue.start:.3f},{cue.end:.3f})'" - - -def _circle_drawbox_filters( - *, - cx: int, - cy: int, - radius: int, - width: int, - height: int, - color: str, - enable: str, -) -> list[str]: - """Approximate a circular dot with thin drawbox bands. - - Staying inside the simple `-vf` chain keeps this compatible with older - ffmpeg builds that lack richer shape/alpha filters. - """ - radius = max(4, radius) - band_count = max(21, min(51, radius // 2 + 1)) - if band_count % 2 == 0: - band_count += 1 - - filters: list[str] = [] - step = (2.0 * radius) / band_count - for band in range(band_count): - y0f = cy - radius + band * step - y1f = cy - radius + (band + 1) * step - ym = (y0f + y1f) / 2.0 - cy - half_w = math.sqrt(max(radius * radius - ym * ym, 0.0)) - - x0 = int(round(cx - half_w)) - x1 = int(round(cx + half_w)) - y0 = int(round(y0f)) - y1 = int(round(y1f)) - - x0_clip = max(0, x0) - y0_clip = max(0, y0) - x1_clip = min(width, x1) - y1_clip = min(height, y1) - w = x1_clip - x0_clip - h = y1_clip - y0_clip - if w <= 0 or h <= 0: - continue - filters.append(f"drawbox=x={x0_clip}:y={y0_clip}:w={w}:h={h}:color={color}:t=fill:{enable}") - return filters - - -def _box_pixels( - box: tuple[float, float, float, float], - *, - width: int, - height: int, - pad: int = 0, -) -> tuple[int, int, int, int]: - x = int(round(box[0] * width)) - pad - y = int(round(box[1] * height)) - pad - w = int(round(box[2] * width)) + pad * 2 - h = int(round(box[3] * height)) + pad * 2 - x = max(0, min(width - 1, x)) - y = max(0, min(height - 1, y)) - w = max(1, min(width - x, w)) - h = max(1, min(height - y, h)) - return x, y, w, h - - -def _box_draw_filters( - *, - x: int, - y: int, - w: int, - h: int, - color: str, - border_color: str, - thickness: int, - enable: str, -) -> list[str]: - return [ - f"drawbox=x={x}:y={y}:w={w}:h={h}:color={color}:t=fill:{enable}", - f"drawbox=x={x}:y={y}:w={w}:h={h}:color={border_color}:t={thickness}:{enable}", - ] - - -def _cursor_filters( - *, - point: tuple[float, float], - width: int, - height: int, - color: str, - border_color: str, - size: int | None, - enable: str, -) -> list[str]: - del color, border_color - x, y = _pointer_tip_pixels(point, width=width, height=height, size=size) - return _pointer_shape_filters( - x_expr=f"{x:.3f}", - y_expr=f"{y:.3f}", - width=width, - height=height, - size=size, - enable=enable, - ) - - -def _expr_add(expr: str, offset: int) -> str: - if offset == 0: - return f"({expr})" - op = "+" if offset > 0 else "-" - return f"({expr}){op}{abs(offset)}" - - -def _pointer_dimensions(*, width: int, height: int, size: int | None) -> tuple[int, int]: - base = size or max(34, min(width, height) // 24) - pointer_h = max(30, min(54, int(round(base * 0.72)))) - pointer_w = max(18, int(round(pointer_h * 0.62))) - return pointer_w, pointer_h - - -def _pointer_tip_pixels( - point: tuple[float, float], - *, - width: int, - height: int, - size: int | None, -) -> tuple[float, float]: - pointer_w, pointer_h = _pointer_dimensions(width=width, height=height, size=size) - x = point[0] * width - y = point[1] * height - x = max(2.0, min(width - pointer_w - 4.0, x)) - y = max(2.0, min(height - pointer_h - 4.0, y)) - return x, y - - -def _laser_dimensions(*, width: int, height: int, size: int | None) -> tuple[int, int]: - base = size or max(44, min(width, height) // 22) - diameter = max( - LASER_DOT_MIN_DIAMETER, - min(LASER_DOT_MAX_DIAMETER, int(round(base * LASER_DOT_SIZE_MULTIPLIER))), - ) - return diameter, diameter - - -def _ease_expr(start_px: float, end_px: float, start: float, end: float) -> str: - duration = max(0.001, end - start) - delta = end_px - start_px - return f"({start_px:.3f}+({delta:.3f})*(1-cos(PI*(t-{start:.3f})/{duration:.3f}))/2)" - - -def _pointer_shape_filters( - *, - x_expr: str, - y_expr: str, - width: int, - height: int, - size: int | None, - enable: str, -) -> list[str]: - pointer_w, pointer_h = _pointer_dimensions(width=width, height=height, size=size) - band_h = max(2, pointer_h // 12) - - def draw_triangle(*, dx: int, dy: int, inset: int, color: str) -> list[str]: - parts: list[str] = [] - for top in range(0, pointer_h, band_h): - frac = min(1.0, (top + band_h) / pointer_h) - band_w = max(2, int(round(pointer_w * frac))) - x = _expr_add(x_expr, dx + inset) - y = _expr_add(y_expr, dy + top + inset) - w = max(1, band_w - inset * 2) - h = max(1, min(band_h, pointer_h - top) - inset) - if w <= 0 or h <= 0: - continue - parts.append(f"drawbox=x={x}:y={y}:w={w}:h={h}:color={color}:t=fill:{enable}") - return parts - - filters: list[str] = [] - filters.extend( - draw_triangle( - dx=2, - dy=2, - inset=0, - color=_color_alpha(CURSOR_POINTER_SHADOW, CURSOR_POINTER_SHADOW_ALPHA), - ) - ) - filters.extend( - draw_triangle( - dx=0, - dy=0, - inset=0, - color=_color_alpha(CURSOR_POINTER_BORDER, CURSOR_POINTER_BORDER_ALPHA), - ) - ) - filters.extend( - draw_triangle( - dx=0, - dy=0, - inset=2, - color=_color_alpha(CURSOR_POINTER_FILL, CURSOR_POINTER_FILL_ALPHA), - ) - ) - - stem_x = int(round(pointer_w * 0.39)) - stem_y = int(round(pointer_h * 0.56)) - stem_w = max(5, int(round(pointer_w * 0.28))) - stem_h = max(9, int(round(pointer_h * 0.34))) - filters.append( - f"drawbox=x={_expr_add(x_expr, stem_x + 2)}:y={_expr_add(y_expr, stem_y + 2)}:" - f"w={stem_w}:h={stem_h}:" - f"color={_color_alpha(CURSOR_POINTER_SHADOW, CURSOR_POINTER_SHADOW_ALPHA)}:" - f"t=fill:{enable}" - ) - filters.append( - f"drawbox=x={_expr_add(x_expr, stem_x)}:y={_expr_add(y_expr, stem_y)}:" - f"w={stem_w}:h={stem_h}:" - f"color={_color_alpha(CURSOR_POINTER_BORDER, CURSOR_POINTER_BORDER_ALPHA)}:" - f"t=fill:{enable}" - ) - filters.append( - f"drawbox=x={_expr_add(x_expr, stem_x + 2)}:y={_expr_add(y_expr, stem_y + 2)}:" - f"w={max(1, stem_w - 4)}:h={max(1, stem_h - 4)}:" - f"color={_color_alpha(CURSOR_POINTER_FILL, CURSOR_POINTER_FILL_ALPHA)}:" - f"t=fill:{enable}" - ) - return filters - - -def _cursor_enabled_cues(cues: list[VisualCue]) -> list[VisualCue]: - out: list[VisualCue] = [] - for cue in cues: - if cue.point is None: - continue - if cue.cue_type == "cursor": - out.append(cue) - continue - if cue.cue_type == "highlight" and cue.style in CURSOR_STYLES: - out.append(cue) - return sorted(out, key=lambda item: (item.start, item.end)) - - -def _laser_enabled_cues(cues: list[VisualCue]) -> list[VisualCue]: - out: list[VisualCue] = [] - for cue in cues: - if cue.point is None: - continue - if cue.cue_type == "highlight" and cue.style in LASER_STYLES: - out.append(cue) - return sorted(out, key=lambda item: (item.start, item.end)) - - -def _spotlight_enabled_cues(cues: list[VisualCue]) -> list[VisualCue]: - out: list[VisualCue] = [] - for cue in cues: - if cue.cue_type != "highlight" or cue.box is None: - continue - if cue.style in SPOTLIGHT_STYLES: - out.append(cue) - return sorted(out, key=lambda item: (item.start, item.end)) - - -def _cursor_path_filters(cues: list[VisualCue], *, width: int, height: int) -> list[str]: - points = [cue for cue in sorted(cues, key=lambda item: (item.start, item.end)) if cue.point is not None] - filters: list[str] = [] - for index, cue in enumerate(points): - next_cue = points[index + 1] if index + 1 < len(points) else None - start = cue.start - end = cue.end - if end <= start: - continue - - x0, y0 = _pointer_tip_pixels(cue.point, width=width, height=height, size=cue.size) - stationary_end = end - if next_cue is not None and next_cue.start > start and next_cue.point is not None: - move_end = next_cue.start - move_start = max(start, move_end - CURSOR_MOVE_SECONDS) - stationary_end = min(end, move_start) - else: - move_start = move_end = 0.0 - - if stationary_end > start + 0.01: - filters.extend( - _pointer_shape_filters( - x_expr=f"{x0:.3f}", - y_expr=f"{y0:.3f}", - width=width, - height=height, - size=cue.size, - enable=f"enable='between(t,{start:.3f},{stationary_end:.3f})'", - ) - ) - - if next_cue is not None and next_cue.point is not None and move_end > move_start + 0.01: - x1, y1 = _pointer_tip_pixels(next_cue.point, width=width, height=height, size=next_cue.size) - filters.extend( - _pointer_shape_filters( - x_expr=_ease_expr(x0, x1, move_start, move_end), - y_expr=_ease_expr(y0, y1, move_start, move_end), - width=width, - height=height, - size=cue.size, - enable=f"enable='between(t,{move_start:.3f},{move_end:.3f})'", - ) - ) - return filters - - -def _laser_dot_filters( - *, - x_expr: str, - y_expr: str, - width: int, - height: int, - size: int | None, - enable: str, -) -> list[str]: - dot_w, dot_h = _laser_dimensions(width=width, height=height, size=size) - radius = max(8, min(dot_w, dot_h) // 2) - core_radius = max(4, int(round(radius * 0.30))) - band_h = max(2, radius // 8) - filters: list[str] = [] - - for radius_px, color in ( - (radius, _color_alpha("0xF87171", LASER_DOT_HALO_ALPHA)), - (max(core_radius + 3, int(round(radius * 0.48))), _color_alpha("0xEF4444", 0.72)), - (core_radius, _color_alpha("0xEF4444", LASER_DOT_CORE_ALPHA)), - ): - for y_offset in range(-radius_px, radius_px + 1, band_h): - y_mid = y_offset + band_h / 2.0 - half_w = int(round(math.sqrt(max(0.0, radius_px * radius_px - y_mid * y_mid)))) - if half_w <= 0: - continue - filters.append( - f"drawbox=x={_expr_add(x_expr, -half_w)}:" - f"y={_expr_add(y_expr, y_offset)}:" - f"w={half_w * 2}:h={band_h}:color={color}:t=fill:{enable}" - ) - return filters - - -def _laser_path_filters(cues: list[VisualCue], *, width: int, height: int) -> list[str]: - points = [cue for cue in sorted(cues, key=lambda item: (item.start, item.end)) if cue.point is not None] - filters: list[str] = [] - for index, cue in enumerate(points): - next_cue = points[index + 1] if index + 1 < len(points) else None - start = cue.start - end = cue.end - if end <= start: - continue - - x0 = cue.point[0] * width - y0 = cue.point[1] * height - stationary_end = end - if next_cue is not None and next_cue.start > start and next_cue.point is not None: - move_end = next_cue.start - move_start = max(start, move_end - CURSOR_MOVE_SECONDS) - stationary_end = min(end, move_start) - else: - move_start = move_end = 0.0 - - if stationary_end > start + 0.01: - filters.extend( - _laser_dot_filters( - x_expr=f"{x0:.3f}", - y_expr=f"{y0:.3f}", - width=width, - height=height, - size=cue.size, - enable=f"enable='between(t,{start:.3f},{stationary_end:.3f})'", - ) - ) - - if next_cue is not None and next_cue.point is not None and move_end > move_start + 0.01: - x1 = next_cue.point[0] * width - y1 = next_cue.point[1] * height - filters.extend( - _laser_dot_filters( - x_expr=_ease_expr(x0, x1, move_start, move_end), - y_expr=_ease_expr(y0, y1, move_start, move_end), - width=width, - height=height, - size=cue.size, - enable=f"enable='between(t,{move_start:.3f},{move_end:.3f})'", - ) - ) - return filters - - -def _blend_pixel(dst: tuple[int, int, int, int], src: tuple[int, int, int, int]) -> tuple[int, int, int, int]: - sr, sg, sb, sa = src - if sa <= 0: - return dst - dr, dg, db, da = dst - src_a = sa / 255.0 - dst_a = da / 255.0 - out_a = src_a + dst_a * (1.0 - src_a) - if out_a <= 0: - return 0, 0, 0, 0 - out_r = (sr * src_a + dr * dst_a * (1.0 - src_a)) / out_a - out_g = (sg * src_a + dg * dst_a * (1.0 - src_a)) / out_a - out_b = (sb * src_a + db * dst_a * (1.0 - src_a)) / out_a - return int(round(out_r)), int(round(out_g)), int(round(out_b)), int(round(out_a * 255)) - - -def _point_in_polygon(x: float, y: float, polygon: list[tuple[float, float]]) -> bool: - inside = False - j = len(polygon) - 1 - for i, (xi, yi) in enumerate(polygon): - xj, yj = polygon[j] - if (yi > y) != (yj > y): - cross_x = (xj - xi) * (y - yi) / ((yj - yi) or 1e-9) + xi - if x < cross_x: - inside = not inside - j = i - return inside - - -def _scale_polygon( - polygon: list[tuple[float, float]], - *, - scale: float, - origin: tuple[float, float], -) -> list[tuple[float, float]]: - ox, oy = origin - return [(ox + (x - ox) * scale, oy + (y - oy) * scale) for x, y in polygon] - - -def _png_chunk(kind: bytes, data: bytes) -> bytes: - body = kind + data - return struct.pack(">I", len(data)) + body + struct.pack(">I", zlib.crc32(body) & 0xFFFFFFFF) - - -def _write_png_rgba_raw(path: Path, width: int, height: int, raw_scanlines: bytes | bytearray) -> None: - png = bytearray(b"\x89PNG\r\n\x1a\n") - png.extend(_png_chunk(b"IHDR", struct.pack(">IIBBBBB", width, height, 8, 6, 0, 0, 0))) - png.extend(_png_chunk(b"IDAT", zlib.compress(bytes(raw_scanlines), level=9))) - png.extend(_png_chunk(b"IEND", b"")) - path.write_bytes(bytes(png)) - - -def _write_png_rgba(path: Path, width: int, height: int, pixels: list[tuple[int, int, int, int]]) -> None: - raw = bytearray() - for y in range(height): - raw.append(0) - row = pixels[y * width:(y + 1) * width] - for r, g, b, a in row: - raw.extend((r, g, b, a)) - _write_png_rgba_raw(path, width, height, raw) - - -def _smoothstep(value: float) -> float: - value = max(0.0, min(1.0, value)) - return value * value * (3.0 - 2.0 * value) - - -def _write_spotlight_mask_png( - path: Path, - *, - box: tuple[float, float, float, float], - width: int, - height: int, - thickness: int, -) -> None: - """Write a full-frame RGBA dimming mask with a smooth transparent window.""" - inner_pad = max(4, int(round(thickness * SPOTLIGHT_INNER_PAD_MULTIPLIER))) - x, y, w, h = _box_pixels(box, width=width, height=height, pad=inner_pad) - left = float(x) - top = float(y) - right = float(x + w) - bottom = float(y + h) - feather = max( - SPOTLIGHT_MIN_FEATHER_PX, - int(round(min(width, height) * SPOTLIGHT_FEATHER_RATIO)), - thickness * SPOTLIGHT_FEATHER_THICKNESS_MULTIPLIER, - ) - max_alpha = int(round(255 * SPOTLIGHT_MAX_ALPHA)) - - raw = bytearray() - for py in range(height): - raw.append(0) - cy = py + 0.5 - if cy < top: - dy = top - cy - elif cy > bottom: - dy = cy - bottom - else: - dy = 0.0 - for px in range(width): - cx = px + 0.5 - if cx < left: - dx = left - cx - elif cx > right: - dx = cx - right - else: - dx = 0.0 - if dx == 0.0 and dy == 0.0: - alpha = 0 - else: - distance = math.hypot(dx, dy) - alpha = int(round(max_alpha * _smoothstep(distance / feather))) - raw.extend((0, 0, 0, alpha)) - _write_png_rgba_raw(path, width, height, raw) - - -def _write_cursor_png(path: Path, *, width: int, height: int) -> tuple[int, int]: - """Write a small transparent mouse pointer PNG using only stdlib code.""" - tip = (CURSOR_OVERLAY_TIP_OFFSET, CURSOR_OVERLAY_TIP_OFFSET) - pointer = [ - tip, - (width * 0.78, height * 0.58), - (width * 0.54, height * 0.60), - (width * 0.68, height * 0.88), - (width * 0.52, height * 0.94), - (width * 0.39, height * 0.66), - (width * 0.21, height * 0.82), - ] - fill = _scale_polygon(pointer, scale=0.80, origin=tip) - layers = [ - ([(x + 2.0, y + 2.0) for x, y in pointer], (0, 0, 0, int(255 * CURSOR_POINTER_SHADOW_ALPHA))), - (pointer, (248, 250, 252, int(255 * CURSOR_POINTER_BORDER_ALPHA))), - (fill, (30, 41, 59, int(255 * CURSOR_POINTER_FILL_ALPHA))), - ] - - pixels: list[tuple[int, int, int, int]] = [] - samples = (0.25, 0.5, 0.75) - for py in range(height): - for px in range(width): - pixel = (0, 0, 0, 0) - for polygon, color in layers: - hits = 0 - for sy in samples: - for sx in samples: - if _point_in_polygon(px + sx, py + sy, polygon): - hits += 1 - if hits: - alpha = int(round(color[3] * hits / (len(samples) ** 2))) - pixel = _blend_pixel(pixel, (color[0], color[1], color[2], alpha)) - pixels.append(pixel) - _write_png_rgba(path, width, height, pixels) - return CURSOR_OVERLAY_TIP_OFFSET, CURSOR_OVERLAY_TIP_OFFSET - - -def _write_laser_png(path: Path, *, width: int, height: int) -> tuple[int, int]: - """Write a small transparent laser-pointer dot with a soft red halo.""" - center_x = (width - 1) / 2.0 - center_y = (height - 1) / 2.0 - halo_radius = max(1.0, min(width, height) * 0.46) - ring_radius = max(1.0, min(width, height) * 0.22) - core_radius = max(4.0, min(width, height) * 0.135) - - pixels: list[tuple[int, int, int, int]] = [] - for py in range(height): - for px in range(width): - dx = px + 0.5 - center_x - dy = py + 0.5 - center_y - distance = math.hypot(dx, dy) - pixel = (0, 0, 0, 0) - if distance <= halo_radius: - if distance <= core_radius: - alpha = int(round(255 * LASER_DOT_CORE_ALPHA)) - pixel = (*LASER_DOT_FILL, alpha) - elif distance <= ring_radius: - t = (distance - core_radius) / max(0.001, ring_radius - core_radius) - alpha = int(round(255 * (0.82 - 0.40 * _smoothstep(t)))) - pixel = (*LASER_DOT_FILL, alpha) - else: - t = (distance - ring_radius) / max(0.001, halo_radius - ring_radius) - alpha = int(round(255 * LASER_DOT_HALO_ALPHA * (1.0 - _smoothstep(t)))) - pixel = (*LASER_DOT_HALO, alpha) - pixels.append(pixel) - _write_png_rgba(path, width, height, pixels) - return int(round(center_x)), int(round(center_y)) - - -def _cursor_overlay_intervals( - cues: list[VisualCue], - *, - width: int, - height: int, - cursor_width: int, - cursor_height: int, - tip_x: int, - tip_y: int, -) -> tuple[list[tuple[float, float, str, str]], float, float]: - points = [cue for cue in cues if cue.point is not None] - if not points: - return [], 0.0, 0.0 - - def overlay_xy(cue: VisualCue) -> tuple[float, float]: - px = cue.point[0] * width - tip_x - py = cue.point[1] * height - tip_y - px = max(0.0, min(width - cursor_width, px)) - py = max(0.0, min(height - cursor_height, py)) - return px, py - - intervals: list[tuple[float, float, str, str]] = [] - for index, cue in enumerate(points): - next_cue = points[index + 1] if index + 1 < len(points) else None - x0, y0 = overlay_xy(cue) - start = cue.start - end = cue.end - if next_cue is not None and next_cue.start > start: - move_end = next_cue.start - move_start = max(start, move_end - CURSOR_MOVE_SECONDS) - if move_start > start + 0.01: - intervals.append((start, move_start, f"{x0:.3f}", f"{y0:.3f}")) - x1, y1 = overlay_xy(next_cue) - if move_end > move_start + 0.01: - intervals.append((move_start, move_end, _ease_expr(x0, x1, move_start, move_end), _ease_expr(y0, y1, move_start, move_end))) - elif end > start + 0.01: - intervals.append((start, end, f"{x0:.3f}", f"{y0:.3f}")) - - return intervals, points[0].start, max(point.end for point in points) - - -def _piecewise_overlay_expr(intervals: list[tuple[float, float, str, str]], *, axis: int) -> str: - if not intervals: - return "0" - expr = intervals[-1][2 + axis] - for start, end, x_expr, y_expr in reversed(intervals[:-1]): - value = x_expr if axis == 0 else y_expr - expr = f"if(between(t,{start:.3f},{end:.3f}),{value},{expr})" - return expr - - -def _outside_box_filters( - *, - x: int, - y: int, - w: int, - h: int, - width: int, - height: int, - color: str, - enable: str, -) -> list[str]: - filters: list[str] = [] - if y > 0: - filters.append(f"drawbox=x=0:y=0:w={width}:h={y}:color={color}:t=fill:{enable}") - bottom = y + h - if bottom < height: - filters.append(f"drawbox=x=0:y={bottom}:w={width}:h={height - bottom}:color={color}:t=fill:{enable}") - if x > 0 and h > 0: - filters.append(f"drawbox=x=0:y={y}:w={x}:h={h}:color={color}:t=fill:{enable}") - right = x + w - if right < width and h > 0: - filters.append(f"drawbox=x={right}:y={y}:w={width - right}:h={h}:color={color}:t=fill:{enable}") - return filters - - -def _spotlight_filters( - *, - box: tuple[float, float, float, float], - width: int, - height: int, - accent_color: str, - thickness: int, - enable: str, -) -> list[str]: - filters: list[str] = [] - # Dim only outside the selected target. This keeps the target at original - # slide brightness instead of washing the whole slide gray. - fade_layers = ( - (max(56, thickness * 11), 0.028), - (max(28, thickness * 6), 0.040), - (max(6, thickness * 2), 0.052), - ) - for pad, alpha in fade_layers: - x, y, w, h = _box_pixels(box, width=width, height=height, pad=pad) - filters.extend( - _outside_box_filters( - x=x, - y=y, - w=w, - h=h, - width=width, - height=height, - color=_color_alpha(SPOTLIGHT_DIM_COLOR, alpha), - enable=enable, - ) - ) - x, y, w, h = _box_pixels(box, width=width, height=height, pad=max(2, thickness)) - filters.append( - f"drawbox=x={x}:y={y}:w={w}:h={h}:" - f"color={_color_alpha(accent_color, SPOTLIGHT_BORDER_ALPHA)}:" - f"t={max(2, thickness - 1)}:{enable}" - ) - return filters - - -def _attention_filters( - cues: list[VisualCue], - *, - width: int, - height: int, - include_cursor: bool = True, - include_laser: bool = True, - include_spotlight: bool = True, -) -> list[str]: - filters: list[str] = [] - cursor_cues: list[VisualCue] = [] - laser_cues: list[VisualCue] = [] - for cue in cues: - color = _color_alpha(cue.color, cue.opacity) - border_color = _color_alpha(cue.color, HIGHLIGHT_BORDER_ALPHA) - enable = _enable_expr(cue) - - if cue.cue_type == "highlight" and cue.box is not None: - thickness = max(3, cue.border or min(width, height) // 180) - pad = max(1, int(round(thickness * HIGHLIGHT_BOX_EXPAND_MULTIPLIER))) - style = cue.style if cue.style in VALID_HIGHLIGHT_STYLES else "box" - x, y, w, h = _box_pixels(cue.box, width=width, height=height, pad=pad) - if style in {"box", "box_cursor", "box_laser"}: - filters.extend( - _box_draw_filters( - x=x, y=y, w=w, h=h, - color=color, - border_color=border_color, - thickness=thickness, - enable=enable, - ) - ) - elif include_spotlight and style in SPOTLIGHT_STYLES: - filters.extend( - _spotlight_filters( - box=cue.box, - width=width, - height=height, - accent_color=cue.color, - thickness=thickness, - enable=enable, - ) - ) - if include_cursor and style in CURSOR_STYLES and cue.point is not None: - cursor_cues.append(cue) - if include_laser and style in LASER_STYLES and cue.point is not None: - laser_cues.append(cue) - continue - - if cue.cue_type == "highlight" and cue.point is not None: - style = cue.style if cue.style in VALID_HIGHLIGHT_STYLES else "box" - if style in CURSOR_STYLES: - if include_cursor: - cursor_cues.append(cue) - continue - if style in LASER_STYLES: - if include_laser: - laser_cues.append(cue) - continue - radius = cue.size or max(42, min(width, height) // 18) - cx = int(round(cue.point[0] * width)) - cy = int(round(cue.point[1] * height)) - filters.extend( - _circle_drawbox_filters( - cx=cx, cy=cy, radius=radius, - width=width, height=height, - color=color, enable=enable, - ) - ) - continue - - if cue.cue_type == "cursor" and cue.point is not None: - if include_cursor: - cursor_cues.append(cue) - if cursor_cues: - filters.extend(_cursor_path_filters(cursor_cues, width=width, height=height)) - if laser_cues: - filters.extend(_laser_path_filters(laser_cues, width=width, height=height)) - return filters - - -# --------------------------------------------------------------------------- -# Stage C — encode each slide as an MP4 segment, then concat -# --------------------------------------------------------------------------- - -def encode_segment(pair: SlidePair, out_seg: Path, *, - width: int, height: int, fps: int, pad_tail: float, - ffmpeg: str, visual_cues: list[VisualCue] | None = None) -> None: - """Render one PNG + one MP3 → an MP4 segment of length audio + pad_tail. - - Image is scaled to fit `width`x`height` while preserving aspect ratio, - then padded with black to exact size — matches how a presentation app - letterboxes a slide on a 16:9 screen if the deck is 4:3, etc. - """ - total_dur = pair.duration + pad_tail - vf_filters = [ - f"scale={width}:{height}:force_original_aspect_ratio=decrease", - f"pad={width}:{height}:(ow-iw)/2:(oh-ih)/2:color=black", - ] - if visual_cues: - cursor_cues = _cursor_enabled_cues(visual_cues) - laser_cues = _laser_enabled_cues(visual_cues) - spotlight_cues = _spotlight_enabled_cues(visual_cues) - vf_filters.extend( - _attention_filters( - visual_cues, - width=width, - height=height, - include_cursor=not cursor_cues, - include_laser=not laser_cues, - include_spotlight=not spotlight_cues, - ) - ) - else: - cursor_cues = [] - laser_cues = [] - spotlight_cues = [] - - # apad pads the audio with silence so we don't depend on -shortest and risk - # the video ending mid-word. We intentionally use bare `apad` rather than - # `apad=pad_dur=...`: older ffmpeg builds (for example 2.4.x) don't support - # pad_dur. The segment-level `-t total_dur` below still clips the padded - # stream to exactly audio_duration + pad_tail. - # - # The aresample+aformat prefix is critical: TTS MP3s arrive with varying - # sample rates / channel layouts, and feeding them straight into AAC + apad - # produces segment-level AAC streams whose internal frame parameters drift - # between slides. The concat demuxer then stream-copies those mismatched - # streams into a single MP4 whose audio track decodes to garbage (silent / - # clicky playback). Pre-normalizing to 44.1 kHz stereo and pinning - # -profile:a aac_low makes every segment's AAC bytestream identical-shaped. - af = ( - f"aresample=44100," - f"aformat=channel_layouts=stereo:sample_rates=44100," - f"apad" - ) - - if spotlight_cues or cursor_cues or laser_cues: - with tempfile.TemporaryDirectory(prefix="paper2video_attention_") as td: - input_args = [ - "-loop", "1", "-framerate", str(fps), "-i", str(pair.frame), - "-i", str(pair.audio), - ] - next_input = 2 - spotlight_inputs: list[tuple[int, VisualCue]] = [] - for index, cue in enumerate(spotlight_cues): - if cue.box is None: - continue - thickness = max(3, cue.border or min(width, height) // 180) - mask_path = Path(td) / f"spotlight_{index:02d}.png" - _write_spotlight_mask_png( - mask_path, - box=cue.box, - width=width, - height=height, - thickness=thickness, - ) - input_args.extend(["-loop", "1", "-framerate", str(fps), "-i", str(mask_path)]) - spotlight_inputs.append((next_input, cue)) - next_input += 1 - - cursor_input: int | None = None - intervals: list[tuple[float, float, str, str]] = [] - first_start = 0.0 - last_end = 0.0 - if cursor_cues: - cursor_size = max((cue.size or 0) for cue in cursor_cues) or None - pointer_w, pointer_h = _pointer_dimensions(width=width, height=height, size=cursor_size) - cursor_w = pointer_w + 10 - cursor_h = pointer_h + 10 - cursor_path = Path(td) / "cursor.png" - tip_x, tip_y = _write_cursor_png(cursor_path, width=cursor_w, height=cursor_h) - intervals, first_start, last_end = _cursor_overlay_intervals( - cursor_cues, - width=width, - height=height, - cursor_width=cursor_w, - cursor_height=cursor_h, - tip_x=tip_x, - tip_y=tip_y, - ) - if intervals: - input_args.extend(["-loop", "1", "-framerate", str(fps), "-i", str(cursor_path)]) - cursor_input = next_input - next_input += 1 - - laser_input: int | None = None - laser_intervals: list[tuple[float, float, str, str]] = [] - laser_first_start = 0.0 - laser_last_end = 0.0 - if laser_cues: - laser_size = max((cue.size or 0) for cue in laser_cues) or None - laser_w, laser_h = _laser_dimensions(width=width, height=height, size=laser_size) - laser_path = Path(td) / "laser.png" - laser_tip_x, laser_tip_y = _write_laser_png(laser_path, width=laser_w, height=laser_h) - laser_intervals, laser_first_start, laser_last_end = _cursor_overlay_intervals( - laser_cues, - width=width, - height=height, - cursor_width=laser_w, - cursor_height=laser_h, - tip_x=laser_tip_x, - tip_y=laser_tip_y, - ) - if laser_intervals: - input_args.extend(["-loop", "1", "-framerate", str(fps), "-i", str(laser_path)]) - laser_input = next_input - next_input += 1 - - filter_parts = [f"[0:v]{','.join(vf_filters)}[base0]"] - current_label = "base0" - for index, (input_index, cue) in enumerate(spotlight_inputs): - mask_label = f"spotmask{index}" - next_label = f"spotbase{index}" - filter_parts.append(f"[{input_index}:v]format=rgba[{mask_label}]") - filter_parts.append( - f"[{current_label}][{mask_label}]overlay=x=0:y=0:" - f"enable='between(t,{cue.start:.3f},{cue.end:.3f})'[{next_label}]" - ) - current_label = next_label - - if cursor_input is not None: - x_expr = _piecewise_overlay_expr(intervals, axis=0) - y_expr = _piecewise_overlay_expr(intervals, axis=1) - filter_parts.append(f"[{cursor_input}:v]format=rgba[cursor]") - filter_parts.append( - f"[{current_label}][cursor]overlay=x='{x_expr}':y='{y_expr}':" - f"enable='between(t,{first_start:.3f},{last_end:.3f})'[withcursor]" - ) - current_label = "withcursor" - - if laser_input is not None: - x_expr = _piecewise_overlay_expr(laser_intervals, axis=0) - y_expr = _piecewise_overlay_expr(laser_intervals, axis=1) - filter_parts.append(f"[{laser_input}:v]format=rgba[laser]") - filter_parts.append( - f"[{current_label}][laser]overlay=x='{x_expr}':y='{y_expr}':" - f"enable='between(t,{laser_first_start:.3f},{laser_last_end:.3f})'[withlaser]" - ) - current_label = "withlaser" - - filter_parts.append(f"[{current_label}]format=yuv420p[v]") - cmd = [ - ffmpeg, "-y", - *input_args, - "-filter_complex", ";".join(filter_parts), - "-map", "[v]", - "-map", "1:a", - "-af", af, - "-c:v", "libx264", "-preset", "medium", "-crf", "20", - "-c:a", "aac", "-b:a", "192k", "-ar", "44100", "-ac", "2", - "-strict", "-2", - "-profile:a", "aac_low", - "-pix_fmt", "yuv420p", - "-r", str(fps), - "-t", f"{total_dur:.3f}", - "-movflags", "+faststart", - str(out_seg), - ] - proc = subprocess.run(cmd, capture_output=True, text=True) - else: - vf_filters.append("format=yuv420p") - cmd = [ - ffmpeg, "-y", - "-loop", "1", "-framerate", str(fps), "-i", str(pair.frame), - "-i", str(pair.audio), - "-vf", ",".join(vf_filters), - "-af", af, - "-c:v", "libx264", "-preset", "medium", "-crf", "20", - "-c:a", "aac", "-b:a", "192k", "-ar", "44100", "-ac", "2", - "-strict", "-2", - "-profile:a", "aac_low", - "-pix_fmt", "yuv420p", - "-r", str(fps), - "-t", f"{total_dur:.3f}", - "-movflags", "+faststart", - str(out_seg), - ] - proc = subprocess.run(cmd, capture_output=True, text=True) - if proc.returncode != 0: - sys.exit(f"[render_video] ffmpeg failed on slide {pair.index}:\n{proc.stderr}") - - -def concat_segments(segments: list[Path], out_path: Path, ffmpeg: str, - start_pad: float, fps: int, width: int, height: int) -> None: - """Concatenate per-slide segments into the final MP4. - - We use the concat *demuxer* (file-list approach) rather than the concat - filter because all our segments share codecs/dimensions — the demuxer is - a stream copy, much faster than re-encoding and bit-exact for video. - """ - out_path.parent.mkdir(parents=True, exist_ok=True) - with tempfile.TemporaryDirectory(prefix="concat_list_") as td: - list_path = Path(td) / "list.txt" - - # Optional leading silence before the first slide. - if start_pad > 0: - black_seg = Path(td) / "black.mp4" - blackcmd = [ - ffmpeg, "-y", - "-f", "lavfi", "-i", f"color=black:s={width}x{height}:r={fps}", - "-f", "lavfi", "-i", "anullsrc=channel_layout=stereo:sample_rate=44100", - "-t", f"{start_pad:.3f}", - "-c:v", "libx264", "-preset", "veryfast", "-crf", "20", - "-c:a", "aac", "-b:a", "192k", "-strict", "-2", - "-pix_fmt", "yuv420p", - "-shortest", str(black_seg), - ] - proc = subprocess.run(blackcmd, capture_output=True, text=True) - if proc.returncode != 0: - sys.exit(f"[render_video] ffmpeg failed on lead-in:\n{proc.stderr}") - segments = [black_seg] + segments - - list_lines = [f"file {shlex.quote(str(s.resolve()))}" for s in segments] - list_path.write_text("\n".join(list_lines) + "\n", encoding="utf-8") - - # All segments share codecs/dimensions AND identical AAC frame layout - # (encode_segment normalizes via aresample+aformat+aac_low), so the - # concat demuxer can stream-copy both video and audio — bit-exact and - # much faster than re-encoding. If you change encode_segment in a way - # that lets segment audio shape drift again, switch this back to a - # full re-encode or you'll get silent/corrupt audio in the output. - cmd = [ - ffmpeg, "-y", - "-f", "concat", "-safe", "0", "-i", str(list_path), - "-c", "copy", - "-movflags", "+faststart", - str(out_path), - ] - proc = subprocess.run(cmd, capture_output=True, text=True) - if proc.returncode != 0: - sys.exit(f"[render_video] ffmpeg concat failed:\n{proc.stderr}") - - -def verify_output(mp4: Path, ffprobe: str, ffmpeg: str) -> float: - """Confirm the file plays and return its duration.""" - if ffprobe != ffmpeg: - out = subprocess.run( - [ffprobe, "-v", "error", "-show_entries", "format=duration", - "-of", "default=noprint_wrappers=1:nokey=1", str(mp4)], - capture_output=True, text=True, - ) - if out.returncode == 0 and out.stdout.strip(): - return float(out.stdout.strip()) - - out = subprocess.run([ffmpeg, "-i", str(mp4)], capture_output=True, text=True) - m = re.search(r"Duration:\s+(\d+):(\d+):([\d.]+)", out.stderr) - if not m: - sys.exit(f"[render_video] could not verify {mp4} — ffmpeg/ffprobe gave no duration.") - return int(m.group(1)) * 3600 + int(m.group(2)) * 60 + float(m.group(3)) - - -def write_duration_report( - report_path: Path, - *, - out_path: Path, - pairs: list[SlidePair], - frame_source: str, - frames_dir: Path, - svg_dir: Path | None, - actual_seconds: float, - expected_seconds: float, - target_minutes: float | None, - tolerance_seconds: float, - start_pad: float, - pad_tail: float, - fps: int, - width: int, - height: int, - visual_cue_map: dict[int, list[VisualCue]], -) -> dict: - target_seconds = target_minutes * 60.0 if target_minutes is not None else None - if target_seconds is None: - status = "no_target" - delta = None - else: - delta = actual_seconds - target_seconds - if abs(delta) <= tolerance_seconds: - status = "within_tolerance" - elif delta > 0: - status = "above_target" - else: - status = "below_target" - - report = { - "schema_version": DURATION_REPORT_SCHEMA_VERSION, - "created_at": _utc_now(), - "output": str(out_path), - "status": status, - "target_minutes": target_minutes, - "target_seconds": round(target_seconds, 3) if target_seconds is not None else None, - "tolerance_seconds": tolerance_seconds, - "actual_seconds": round(actual_seconds, 3), - "expected_seconds": round(expected_seconds, 3), - "target_delta_seconds": round(delta, 3) if delta is not None else None, - "render_delta_seconds": round(actual_seconds - expected_seconds, 3), - "start_pad": start_pad, - "pad_tail": pad_tail, - "fps": fps, - "resolution": {"width": width, "height": height}, - "frame_source": frame_source, - "frames_dir": str(frames_dir), - "svg_dir": str(svg_dir) if svg_dir is not None else None, - "slides": [ - { - "index": pair.index, - "audio": pair.audio.name, - "audio_seconds": round(pair.duration, 3), - "segment_seconds": round(pair.duration + pad_tail, 3), - "visual_cues": len(visual_cue_map.get(pair.index, [])), - } - for pair in pairs - ], - } - report_path.parent.mkdir(parents=True, exist_ok=True) - report_path.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") - return report - - -# --------------------------------------------------------------------------- -# CLI -# --------------------------------------------------------------------------- - -def main() -> int: - ap = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0]) - ap.add_argument("project_path", help="ppt-master project root") - ap.add_argument("--pptx", required=True, help="Path to the exported PPTX") - ap.add_argument("--audio-dir", default=None, - help="Directory of per-slide MP3s (default: <project>/audio)") - ap.add_argument("--script-json", default=None, - help="Narration script JSON whose sections order defines audio order. " - "Defaults to <audio-dir>/script.json, then <project>/assets/meta/narration.json, then <project>/narration.json, " - "then manifest/sorted filenames.") - ap.add_argument("--out", default=None, - help="Output MP4 path (default: <project>/exports/<pptx_stem>.mp4)") - ap.add_argument("--resolution", choices=tuple(RESOLUTIONS), default="1080p", - help="Output frame size preset (default: 1080p)") - ap.add_argument("--dpi", type=int, default=None, - help="Legacy PPTX/PDF PNG render DPI; default chosen to match the resolution preset") - ap.add_argument("--frame-source", choices=("auto", "svg", "pptx"), default="auto", - help="Slide raster source. auto prefers <project>/svg_final, then svg_output, " - "and falls back to PPTX/PDF only when no SVG deck exists.") - ap.add_argument("--svg-dir", default=None, - help="Explicit SVG frame directory. Defaults to <project>/svg_final, then svg_output.") - ap.add_argument("--browser-executable", default=None, - help="Chrome/Chromium executable for SVG frame screenshots. Defaults to " - "PAPER2VIDEO_CHROME/CHROME/CHROMIUM or a common system install.") - ap.add_argument("--frames-out", default=None, - help="Copy the exact rendered frames used for the MP4 to this directory, " - "for example $VIDEO_OUT/assets/slides/frames.") - ap.add_argument("--fps", type=int, default=30, help="Output frame rate (default: 30)") - ap.add_argument("--pad-tail", type=float, default=0.3, - help="Silence appended after each slide's narration (default: 0.3s)") - ap.add_argument("--start-pad", type=float, default=0.5, - help="Black-screen silence before slide 1 (default: 0.5s)") - ap.add_argument("--target-minutes", type=float, default=None, - help="Target final video duration in minutes. render_video.py reports whether " - "the actual MP4 lands within tolerance; use notes_to_script.py or " - "assets_to_script.py with the same target to shape narration before TTS.") - ap.add_argument("--duration-tolerance-seconds", type=float, default=30.0, - help="Allowed final video duration error when --target-minutes is set (default: 30s).") - ap.add_argument("--duration-report-out", default=None, - help="Output duration report JSON (default: <out_stem>_duration_report.json " - "when --target-minutes is set).") - ap.add_argument("--attention-mode", choices=("none", "highlight", "cursor", "both"), default="highlight", - help="Attention overlay mode (default: highlight). Requires --visual-cues " - "for positioned highlights/cursors.") - ap.add_argument("--highlight-style", choices=tuple(sorted(VALID_HIGHLIGHT_STYLES)), default="spotlight_laser", - help="How highlight cues should render: box, spotlight, cursor, box+cursor, " - "spotlight+cursor, laser dot, box+laser, or spotlight+laser.") - ap.add_argument("--visual-cues", default=None, - help="JSON file describing per-slide highlight/cursor cues in normalized coordinates.") - ap.add_argument("--allow-missing-visual-cues", action="store_true", - help="Degraded/debug only: allow highlight/cursor/both without --visual-cues.") - ap.add_argument("--frames-only", action="store_true", - help="Stop after PNG export — useful for previewing slide rendering") - ap.add_argument("--audio-only-check", action="store_true", - help="Verify each slide has matching audio, then exit (no rendering)") - ap.add_argument("--keep-temp", action="store_true", - help="Keep the temp working dir (slides/segments) for debugging") - args = ap.parse_args() - - project_path = Path(args.project_path).resolve() - pptx_path = Path(args.pptx).resolve() - if not pptx_path.is_file(): - sys.exit(f"[render_video] PPTX not found: {pptx_path}") - if args.target_minutes is not None and args.target_minutes <= 0: - sys.exit("[render_video] --target-minutes must be positive") - if args.duration_tolerance_seconds < 0: - sys.exit("[render_video] --duration-tolerance-seconds must be non-negative") - - audio_dir = Path(args.audio_dir).resolve() if args.audio_dir else project_path / "audio" - script_json = Path(args.script_json).resolve() if args.script_json else None - out_path = Path(args.out).resolve() if args.out else project_path / "exports" / f"{pptx_path.stem}.mp4" - visual_cues_path = Path(args.visual_cues).resolve() if args.visual_cues else None - explicit_svg_dir = Path(args.svg_dir).resolve() if args.svg_dir else None - frames_out = Path(args.frames_out).resolve() if args.frames_out else None - browser_executable = Path(args.browser_executable).expanduser() if args.browser_executable else None - if browser_executable is not None and not browser_executable.is_file(): - sys.exit(f"[render_video] --browser-executable not found: {browser_executable}") - - width, height = RESOLUTIONS[args.resolution] - dpi = args.dpi or {"720p": 110, "1080p": 150, "1440p": 200, "4k": 300}[args.resolution] - - ffmpeg, ffprobe = find_ffmpeg_pair() - - # Working area - work_root = project_path / ".video_work" - if work_root.exists() and not args.keep_temp: - shutil.rmtree(work_root) - work_root.mkdir(parents=True, exist_ok=True) - - png_dir = work_root / "frames" - svg_dir = discover_svg_dir(project_path, explicit_svg_dir, args.frame_source) - use_svg = args.frame_source in {"auto", "svg"} and svg_dir is not None - frame_source_used = "svg" if use_svg else "pptx" - - if use_svg: - svgs = collect_svgs(svg_dir) - if svg_dir.name == "svg_output": - print( - "[render_video] WARNING: using svg_output; prefer svg_final because " - "svg_output may still contain unexpanded icon placeholders." - ) - print(f"[render_video] Stage A: SVG → PNG ({width}x{height}) from {svg_dir}") - frames = render_svg_frames( - svgs, - png_dir, - project_path=project_path, - width=width, - height=height, - browser_executable=str(browser_executable) if browser_executable else None, - ) - else: - print(f"[render_video] Stage A: PPTX → PDF → PNG (DPI={dpi}, {width}x{height})") - pdf_dir = work_root / "pdf" - libreoffice = find_libreoffice() - pdftoppm = find_pdftoppm() - pdf = pptx_to_pdf(pptx_path, pdf_dir, libreoffice) - frames = pdf_to_pngs(pdf, png_dir, dpi, pdftoppm) - - if frames_out is not None: - frames = copy_frames(frames, frames_out) - png_dir = frames_out - print(f"[render_video] {len(frames)} frame(s) under {png_dir}") - - if args.frames_only: - print(f"[render_video] --frames-only: stopping. Frames: {png_dir}") - return 0 - - print(f"[render_video] Stage B: pair frames with {audio_dir}/*.mp3") - audio_files = collect_audio(audio_dir, script_json=script_json, project_path=project_path) - pairs = pair_slides(frames, audio_files, ffprobe, ffmpeg) - total_audio = sum(p.duration for p in pairs) - print(f"[render_video] {len(pairs)} slide(s), audio total {total_audio:.1f}s") - if args.audio_only_check: - print("[render_video] --audio-only-check passed.") - return 0 - - visual_cue_map = load_visual_cues( - visual_cues_path, - pairs, - attention_mode=args.attention_mode, - highlight_style=args.highlight_style, - pad_tail=args.pad_tail, - allow_missing_visual_cues=args.allow_missing_visual_cues, - ) - - print(f"[render_video] Stage C: encode {len(pairs)} segment(s) and concat") - seg_dir = work_root / "segments" - seg_dir.mkdir(exist_ok=True) - segments: list[Path] = [] - for pair in pairs: - seg_path = seg_dir / f"seg_{pair.index:04d}.mp4" - encode_segment(pair, seg_path, - width=width, height=height, fps=args.fps, - pad_tail=args.pad_tail, ffmpeg=ffmpeg, - visual_cues=visual_cue_map.get(pair.index)) - segments.append(seg_path) - - concat_segments(segments, out_path, ffmpeg, - start_pad=args.start_pad, fps=args.fps, - width=width, height=height) - - duration = verify_output(out_path, ffprobe, ffmpeg) - expected = total_audio + args.start_pad + args.pad_tail * len(pairs) - drift = duration - expected - - print() - print(f"[render_video] DONE → {out_path}") - print(f" duration: {duration:.1f}s (expected ≈ {expected:.1f}s, drift {drift:+.1f}s)") - print(f" slides: {len(pairs)} resolution: {width}x{height}@{args.fps}fps") - print(f" frames: {frame_source_used} → {png_dir}") - - report_path: Path | None = None - if args.duration_report_out: - report_path = Path(args.duration_report_out).resolve() - elif args.target_minutes is not None: - report_path = out_path.with_name(f"{out_path.stem}_duration_report.json") - - if report_path is not None: - report = write_duration_report( - report_path, - out_path=out_path, - pairs=pairs, - frame_source=frame_source_used, - frames_dir=png_dir, - svg_dir=svg_dir if use_svg else None, - actual_seconds=duration, - expected_seconds=expected, - target_minutes=args.target_minutes, - tolerance_seconds=args.duration_tolerance_seconds, - start_pad=args.start_pad, - pad_tail=args.pad_tail, - fps=args.fps, - width=width, - height=height, - visual_cue_map=visual_cue_map, - ) - if args.target_minutes is not None: - print( - f" target: {args.target_minutes:.2f} min " - f"({report['status']}, delta {report['target_delta_seconds']:+.1f}s)" - ) - print(f" duration report: {report_path}") - - if not args.keep_temp: - shutil.rmtree(work_root, ignore_errors=True) - else: - print(f" work dir kept at: {work_root}") - - return 0 - - -if __name__ == "__main__": - sys.exit(main())