PrecisionEdit

An embeddable SDK for scope-aware video editing: click an object, and edit only that object. The SDK is the product. Studio is the reference editor that shows what can be built with it.

Status: private development. See docs/BUILD_STATUS.md for exactly what works and what was measured, and docs/ROADMAP.md for what does not yet.

What works today

• Open a local video or the sample scene. Play, seek, and step frame by frame at the clip's frame rate, read from the container's own timestamps.
• Click an object, then drag it where you want it. Resize and rotate it with the handles or the keyboard. Every change is a preview first: Apply makes it one undoable step, Cancel or Escape leaves no trace. Remove an object, or drag a colour swatch onto it to recolour it (shading kept). All of this runs in your browser at no cost, and each preview says so before you apply it.
• Removing or moving an object rebuilds what was behind it, once, in the background: real background from other frames where the object moved away, texture invented from nearby where it never did (the app says how much), following pans, tilts and zooms, and matched to every frame's lighting. Checked on real AI-generated clips (tests/footage).
• Click an object to select it; Alt-click to exclude. Refine the mask with a brush and an eraser. Inspect it as film, outline or matte.
• Track the selection through the clip. Frames where the object is lost or the match is weak are flagged, and edits are withheld there until you correct them with a click.
• Adjust, blur and transform a selection or the whole frame over a time range. Pixels outside the selection are left byte-identical; this is tested.
• Move an object for real: its original area is rebuilt from other frames (clean plate, static camera), from its surroundings (inpainting, smooth backgrounds), or from an AI result, then the object is placed at its new position. Copy, which leaves the original visible, is still available.
• AI edit (Google Gemini Omni Flash, or an offline mock): describe the change, see the plan (range, data flow, documented cost estimate, what is and is not guaranteed), approve it, run it, then review the original, the AI result, the composite, and what the model changed outside the selection. The result is composited through your selection's per-frame mask, so pixels outside it keep their original values whatever the model returned.
• Crop, undo/redo for everything (selections, tracks, brush strokes, AI edits), compare with the original.
• Export to WebM (VP9, Opus) in the browser with sequential decoding: 1080p long-GOP is about 3.9× faster than before, with identical frames.

What does not work yet

• Real Gemini edits have not been run. Paid operations are switched off until you approve a budget (see below). The AI pipeline is tested end to end with a mock that imitates a model's output.
• Segmentation is still classical (colour region, labelled non-AI); SAM 2.1 is not integrated. The tracker is classical too: good on distinct objects, weaker on textured or deforming ones.
• Inpainting smears texture; a clean plate needs a static camera (the app checks and says so).
• Gemini Omni Flash edits at most 10 s per request, outputs 16:9 or 9:16 only, and has no mask input.
• No React SDK package yet; export is WebM only.

AI server

The Studio talks to a local server that holds the provider key; the browser never sees it.

# .env.local (git-ignored) at the repository root
GEMINI_API_KEY=...
# Paid operations stay refused unless both are set:
# PRECISIONEDIT_PAID_OPS=on
# PRECISIONEDIT_PAID_BUDGET_USD=2.00

pnpm dev:all        # Studio + AI server (or: pnpm dev and pnpm dev:ai)

"Check connection" in the AI panel lists the models your key can use (a free request). Nothing is sent to Google until you approve and run an edit. See SECURITY.md.

Quickstart

Requirements: Node ≥ 22, pnpm 10, ffmpeg and ffprobe on PATH (for the test fixtures and the export tests), and Google Chrome (for the browser tests).

pnpm install
pnpm fixtures      # generate the licence-free sample video
pnpm dev           # http://localhost:5173

Command	What it does
pnpm test	Unit tests (Vitest, Node)
pnpm typecheck	Strict TypeScript across packages
pnpm build	Build @precisionedit/core and the Studio
pnpm test:e2e	Browser workflow tests in Chrome (Playwright)
pnpm bench	CPU pixel-path timings
pnpm dev:all	Studio and the local AI server
PRECISIONEDIT_LIVE_FREE=1 pnpm test apps/ai-server/test/live-free.test.ts	Free live check of the configured Gemini key (no edit, no charge)

Using the core directly

import { SelectionStore, ColorRegionSegmenter, applyAdjustments, IDENTITY_ADJUSTMENTS } from "@precisionedit/core";

const store = new SelectionStore(new ColorRegionSegmenter());
const id = store.create("Car");
await store.addPrompt(id, { x: 412, y: 230, label: "positive" }, { time: 1.5, image: ctx.getImageData(0, 0, w, h) });
const mask = store.get(id)!.mask!;
const out = applyAdjustments(frame, { ...IDENTITY_ADJUSTMENTS, brightness: 0.2 }, mask);

Layout

packages/core: framework-independent editing core. apps/studio: React reference editor. tests/browser: Playwright. docs/: architecture, risks, quality gates, roadmap, and build status.

Licence

Apache-2.0. See LICENSE. See SECURITY.md for data flow and reporting.