feat(subtitles): AI subtitle auto-sync — semantic timing correction with drift calibration and OCR fallback

[theNuvioGuy]

Background, in full honesty: I originally built this feature for NuvioTV and proposed it there — the team declined, as they don't want to take on features like this right now. The idea clearly resonates with users though: when I posted it on Reddit it drew a lot of demand — https://www.reddit.com/r/Nuvio/comments/1vaibjj/auto_subtitles_sync_with_ai_new_tool_i_developed/ — so I'm bringing it to ARVIO, which already has all the AI infrastructure it needs.

Motivation

Addon subtitles are frequently out of sync: cut for a different rip, a different frame rate, or a different edit. Today the practical fallback is AI translation of an embedded track — which works, but has two real costs this feature avoids:

1. Cost / token usage. AI translation calls the LLM continuously for the entire runtime — every subtitle window of a 2-hour movie goes through the model. Auto-sync sends a handful of tiny requests total (≈5 short dialogue lines each, roughly 3–8 calls per movie regardless of runtime), because the model is only asked to match a few lines, never to translate content. The result is then cached per stream, so future playbacks of the same file re-apply the sync with zero LLM calls. In practice this is orders of magnitude cheaper.

2. Quality in gendered languages. For languages with grammatical gender and gendered address (Hebrew, Arabic, and many others), machine translation from English is structurally handicapped: English "you said" doesn't say whether the speaker addresses a man or a woman, so the model must guess verb forms and pronouns — and it guesses wrong constantly. A human-authored subtitle in the target language already has all of this right. The only thing wrong with it is timing — and timing is exactly what this feature fixes. Auto-sync makes the good subtitles usable instead of generating mediocre ones.

The algorithm

Anchor 1 — flat offset (foreground, seconds):

• Gather ≥5 timestamped dialogue lines from a built-in (embedded) subtitle track — always correctly timed to the video. Buffered (upcoming) cues are read directly from the player, so this completes in seconds without waiting for dialogue to be spoken.
• Ask the LLM (Groq / Gemini, temperature 0, JSON mode) to semantically match those lines against a ±12-line window of the addon file. Each confident pair yields an offset builtInTime − addonTime.
• Pool offsets across up to 3 attempts (the window doubles when nothing matches — large real offsets can sit outside the default window) and reduce with a robust mean (median-referenced outlier rejection, >450 ms dropped). Music/SFX cues (♪, [Music]…) are filtered — they rarely exist in addon files.

Anchors 2–4 — drift calibration (background, best-effort):

• An invisible, text-only second ExoPlayer (no video/audio renderers → runs at 4× speed) seeks to up to 3 positions spread across the remaining runtime and repeats the measurement there.
• A robust OLS line delay(position) = intercept + rate·position is fit across all anchors — correcting frame-rate-style drift (e.g. 25 ↔ 23.976 fps ≈ +2.6 s/min). Safeguards, each of which exists because it prevented a real failure during testing:
  • Anchor reliability gate: a background anchor needs ≥2 pooled offsets agreeing within 450 ms, or it's discarded and retried at a nudged position.
  • Protected anchor 1: the outlier-rejection step may never drop anchor 1 (the user-verified foreground measurement) — two noisy background anchors must not redefine an already-correct sync.
  • Noise floor: a fitted rate whose total effect across the measured span is within the per-anchor noise bound is unmeasurable slope, not drift — it snaps to a flat fit.
  • Sanity bounds: rates beyond any plausible frame-rate mismatch, or fits with large residuals, are rejected outright — the flat delay stays.
• Non-linear rescue: when anchors measure consistently but no line fits them — the signature of stepwise timing (ad-break cuts / a different edit) — a background loop re-measures a flat offset just ahead of the live position whenever playback moves ≥10 minutes from the last measurement (by watching or by jumping, in either direction). This is what makes changing offsets within one file work.

Applying — renderer-side, zero interruption:

• Corrections apply inside the text renderer as delay(position) = base + rate·(position − anchor), with the anchor self-captured on the render thread (render positions live in ExoPlayer's private offset timebase — delta-only math is the only safe form). No MediaItem rebuild, no buffering hiccup, refits are instant and seek-proof.
• OCR fallback: when the file's only built-in track is image-based (PGS/DVB), rendered cue bitmaps are OCR'd with ML Kit's bundled on-device Latin recognizer and used as the reference — in both the foreground gather and the drift calibrator.

Code changes — deliberately minimal, built on what's already here

The diff is one commit touching a small set of files, because ARVIO already had almost every building block:

Reused as-is	Where
Groq/Gemini clients, API key storage, model picker, backoff, JSON handling	SubtitleTranslationService — auto-sync adds one method (matchSubtitleLines)
Buffered/realtime cue extraction from the selected text track	AiSubtitleRenderersFactory reflection walkers + onPlayerCues
Renderer-side timing hook (manual delay slider)	SubtitleOffsetRenderer — extended with the base+rate transform
Subtitle download / parse / timestamp transform / local serving	SubtitleSyncMatcher, localizeSubtitle
Per-stream persistence	the existing match cache (CachedSubMatch gains a rate field)
Settings UI, status pill/toasts, key entry	existing components, one new toggle

New files are only the isolated logic: SubtitleAutoSync.kt (pure math — unit-tested, no Android deps), SubtitleDriftCalibrator.kt (the invisible second player), SubtitleCueOcr.kt (~30 lines of ML Kit). The single new dependency is com.google.mlkit:text-recognition (bundled on-device model, no Play Services).

Nothing about existing behavior changes when the feature is off; the auto-sync toggle is mutually exclusive with the auto match-scan since they'd fight over the reference track.

Credits & testing

The idea and design direction are mine; the implementation was done with Claude Fable 5. I tested it end-to-end on genuinely problematic, old sources — The Office episodes (PAL-style drift) and A Bronx Tale among them — and it works nicely, including files where the subtitle offset changes partway through. Unit tests cover the math (robust mean, window selection, drift fit guards, noise floor, timestamp transforms).

[theNuvioGuy]

@silentbil Please see :)

[silentbil]

Thanks for the work @theNuvioGuy
i will take a look

[ProdigyV21]

Great work and thanks for the pr. Will leave this to @silentbil since he did all the current AI infrastructure.

[theNuvioGuy]

Update: Unified Auto Sync + adaptive drift resync

Added commit 7e699ab9 on top of the initial AI auto-sync implementation.

What changed

Combined "Find Best Match" and "AI Auto-Sync" into one feature:

• Single Auto Sync Subtitles master toggle (default ON), with a Use AI for Auto Sync sub-option.
• Master ON + AI OFF → auto-runs the timing-based match scan (as before).
• Master ON + AI ON → runs the full AI pipeline (anchor 1 → drift-rate anchor pass), falling back to the scan automatically when there's no built-in reference track or no API key configured.
• Subtitle picker now shows mode-aware actions ("Auto Sync — Without AI" / "Auto Sync — With AI" when the master is off, or a single "Auto Sync" matching the configured mode when on). Every manual action fully resets prior sync state before restarting.

Adaptive drift resync (replaces the old fixed 10-minute interval):

• The periodic resync loop now seeds its correction and cadence from already-measured anchors instead of waiting blind, then adapts the next check interval from the measured local drift velocity (tighter when drift is fast, backing off toward 10 min when it's stable).
• Fixed a false-negative where the robust line fit could silently drop one disagreeing anchor as an "outlier" and mask genuinely non-linear timing — an end-of-pass validation now catches this and correctly triggers the recovery path below.

New: background rematch on non-linear drift. When AI auto-sync detects non-linear timing (evidence of a wrong-cut subtitle for this rip), it now searches in the background for a better-cut addon subtitle and swaps to it automatically if one scores well — instead of just tracking the mismatch forever. Falls back to the existing resync loop if nothing better is found. Reference material gathered along the way is reused across steps to avoid redundant downloads/connections.

Playback fixes:

• Subtitles are hidden during the initial reference-track gather (viewer no longer sees the raw built-in track mid-sync).
• Fixed a bug where a large renderer-side sync correction could blank subtitles for several seconds after every seek on badly-drifted sources — large corrections now bake into the served file instead, keeping seeks instant.

Note for reviewers

This build currently shows extra toast/status messages describing background progress (e.g. "Auto-sync refinement 1/3", "Non-linear drift found — searching for a better subtitle…", "Better-match search — scoring N subtitles…"). These are intentionally left in for now so we can verify the background pipeline is actually progressing through each stage during testing. They will be removed/quieted before this ships for a smooth, mostly-silent UX — only the final outcome toast should remain.

[ProdigyV21]

Thanks for the substantial work here. This is a genuinely useful feature and the overall approach looks promising. The PR merges cleanly, the Sideload build compiles, and the new unit tests pass. I found two things that should be fixed before release:

1. A cancelled background calibration can repopulate the retained reference pool after a new stream has already cleared it. If the user switches sources/videos during calibration, old subtitle cues can leak into the next stream. Please add a per-stream generation/session ID and only record a gather when it still belongs to the active stream, or cancel-and-join before clearing.

2. subtitleAutoSyncEnabled is exported/restored by cloud sync but is missing from globalMergeKeys. Please add it beside subtitleAiAutoSync so a stale device cannot overwrite the newer setting.

The bundled ML Kit ARM64 library is 16 KB compatible. It does increase the app by roughly 19 MB of ARM native/model files, which is worth being aware of. Also, as noted in the PR, please quiet the detailed background progress messages before shipping.

One separate issue: the Play flavor currently fails in SeekPreviewFrameProvider.kt:544, but that file is unchanged by this PR and the failure comes from current main, so it should be handled separately.

Once the two PR-specific issues are addressed and source-switch/seek/PGS/debrid/HLS cases have been tested on a real TV, this should be in good shape to merge.

[ProdigyV21]

@theNuvioGuy An update/release is planned tomorrow. Would be nice to have this included in it.

[silentbil]

@theNuvioGuy An update/release is planned tomorrow. Would be nice to have this included in it.

Im working with him offline to create a better UX
he is out on vacation,
lets skip this pr to next one

[ProdigyV21]

Alright