Skip to content

feat: prototype editable cursor choreography for AI Edition - #116

Draft
YoneRai12 wants to merge 1 commit into
getopenscreen:mainfrom
YoneRai12:agent/ai-cursor-choreography-concept
Draft

feat: prototype editable cursor choreography for AI Edition#116
YoneRai12 wants to merge 1 commit into
getopenscreen:mainfrom
YoneRai12:agent/ai-cursor-choreography-concept

Conversation

@YoneRai12

@YoneRai12 YoneRai12 commented Jul 18, 2026

Copy link
Copy Markdown

What

This Draft explores editable cursor choreography as an AI Edition idea/prototype.

  • Add clip- and asset-owned cursor motion regions with normalized start/end/control points, virtual timeline time, and source time.
  • Preserve recorded cursor motion by default (recorded, 1x) and create an editable range from the playhead to the next click, split at detected rests and clicks.
  • Offer recorded, straight, arc, wave, loop, and overshoot presets, with 1x-4x speed, easing, and cycle controls.
  • Show original and edited paths in the preview, expose draggable control points, and add independently selectable motion segments to the timeline and inspector.
  • Use the same pure sampling functions for preview and MP4 export.
  • Restore automatic zoom placement for newly added recordings and restore global/per-region Auto-Focus controls. The timeline now shows visible labels for Auto Zoom, Auto-Focus, and Cursor Motion.

Why

The AI Edition transition retained some zoom primitives but removed the previous automatic placement flow and the global cursor-follow mode from the primary V4 workflow. It also had no first-class way to author cursor movement without changing the underlying recording.

This prototype tests whether cursor movement can become a non-destructive, per-clip timeline effect while keeping untouched footage exactly as recorded.

Prototype UX

  1. Leave the project unchanged to keep recorded cursor motion.
  2. Place the playhead inside a clip and choose Cursor Motion to build segments through the next click.
  3. Select an individual rest/click-split segment in the cursor motion lane.
  4. Choose a preset, speed, easing, and cycle count in the inspector.
  5. Drag control points directly over the preview while comparing the original and edited paths.
  6. Use the labeled Auto Zoom and Auto-Focus buttons for the restored automatic zoom and continuous cursor-follow workflows.

Architecture

  • cursorMotionRegions is backward-compatible with a default empty array. Old assets default to autoZoomState: processed; newly imported assets are marked pending so automatic zoom runs once without modifying old documents on load.
  • Every cursor motion region owns a clipId and assetId. It stores both virtual and source boundaries, and regions that cross or disagree with their render segment are rejected.
  • Stop/click splitting, speed clamping, preset sampling, crop projection, and endpoint preservation are pure functions with focused tests.
  • Preview and export resolve the active clip's telemetry only. Virtual time drives timeline effects while source time drives cursor interpolation.
  • Control-point dragging updates the in-memory document live and persists once on pointer release.
  • Render-plan segments carry asset-specific cursor samples and cursor motion regions into FrameRenderer; source changes clear the previous asset's telemetry and focus state.

Validation

  • npx tsc --noEmit - passed
  • Focused Vitest suite - 12 files, 123 tests passed
  • npm run test:browser - 2 files, 6 tests passed
  • npm run i18n:check - passed; all 12 translations match English across 7 namespaces
  • Biome check for all changed TypeScript/TSX/JSON files - passed
  • npm run test - 98/99 files and 1,010/1,012 tests passed. The two failures are existing Windows timeouts in electron/media/ffmpegEncodeSession.test.ts (cancel()/post-cancel writes); the implementation and test files are unchanged from base commit 4c2f842 and also fail in isolation.
  • Windows Electron smoke - verified automatically placed zoom regions, the Auto-Focus state change, visible toolbar controls, and Cursor Motion button feedback. The active recording had no click telemetry because the Windows cursor sampler helper was unavailable in this dev checkout, so a full control-point drag on a newly created region was not completed manually.

Known limitations and open questions

  • This is deliberately a Draft idea/prototype, not a finished feature, and needs refinement in interaction design, visual density, stop-detection thresholds, and subjective speed feel.
  • Cursor choreography is wired into the MP4 render-plan path; the separate GIF exporter does not yet consume cursor motion regions.
  • Real multi-monitor/DPI and multi-asset behavior is covered by owner/time/crop tests but still needs broader manual footage coverage.
  • The local Windows smoke environment lacked the native cursor sampler helper, so click-enabled end-to-end authoring remains an explicit manual validation gap.
  • Maintainers are welcome to reuse or evolve the concept, including adopting only the data model or sampling approach and re-implementing the UI.

Type of change

  • Bug fix
  • Feature
  • Enhancement
  • Documentation
  • Refactor / maintenance
  • Performance
  • Security

Release impact

  • Patch
  • Minor
  • Major / breaking change
  • No release note needed (Draft prototype)

Desktop impact

  • Windows
  • macOS
  • Linux
  • Installer / packaging
  • Not platform-specific

Screenshots / video

No attachment yet. The Windows Electron smoke confirmed the toolbar and timeline behavior; a polished demo should follow after the native cursor sampler is available in the test checkout.

@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1051a856-7894-4e09-8971-e3a2f5a81788

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@EtienneLescot
EtienneLescot force-pushed the feat/ai-edition branch 2 times, most recently from b43adda to 92fbe07 Compare July 26, 2026 15:25
@EtienneLescot

Copy link
Copy Markdown
Collaborator

main has just been rebased onto the 1.8.0 line — a ~390-commit merge landing the v4 editor, the native Rust compositor and export path, and the Linux capture stack. Every open PR is out of date; sorry for the churn. Here's where this one stands:

The base branch feat/ai-edition is superseded — that work is now on main (v4 editor, native compositor, whisper.cpp STT). Most of this 493-file draft is commits that have already landed, which is why the diff looks the size it does.

Suggestion: close it and reopen the cursor-choreography idea as a focused draft against main.

@EtienneLescot

Copy link
Copy Markdown
Collaborator

Same as #113 — ignore my close suggestion, keeping this open.

The AI Edition work landed on main, so the feat/ai-edition base is moot, but the choreography idea itself is still worth having on the table.

@EtienneLescot

Copy link
Copy Markdown
Collaborator

I had this backwards — apologies, twice over on the same topic.

I told you this one was superseded and pointed the porting effort at #113. It's the other way round: this is the portable one. Your choreography here is wired into V4Timeline.tsx, FloatingInspector.tsx, Preview.tsx, PreviewCanvas.tsx and VirtualPreview.tsx — all of which are alive in main. #113's UI targets VideoEditor.tsx, VideoPlayback.tsx, TimelineEditor.tsx and CursorMotionEditorOverlay.tsx, every one of them deleted in the 1.8.0 merge.

What's not portable here is the branch history, not the work: the base is 428 commits behind, and rebasing conflicts on 56 files at the first commit.

So the same route as the Fedora PR — leave the history alone, take the content. It's five files:

file what it needs
src/lib/cursor/cursorMotion.ts + test clean addition (+561) — though #113 carries a later revision of this module (581 lines), worth diffing before picking
src/components/ai-edition/CursorPreviewLayer.tsx clean addition (+723)
v4/V4Timeline.tsx, v4/FloatingInspector.tsx real merge — your versions come from an older v4 and main has moved

The compositor side is already done and waiting on feat/cursor-motion-contract: the scene contract carries motion regions and the Rust sampler renders them, for preview and export both. Details on #113.

Lifted verbatim from getopenscreen#116, which cannot be rebased: its base is 428 commits
behind and the first commit alone conflicts on 56 files. The module itself
needs none of that history -- it lands on main untouched, with its 17 tests
passing and no new type errors.

This is the pure model: presets, easing, speed, anchor kinds, rest and click
detection, and the sampling that turns a region into a position. No rendering,
no editor, no dependency on anything the 1.8.0 merge removed.

Taken from getopenscreen#116 rather than getopenscreen#113. The two carry different lineages of this
module, not an old and a new one: getopenscreen#116's knows about source time and crop
projection, which is what its editor and preview call into. getopenscreen#113's is the
shape the deleted web editor wanted.

Not wired to anything yet. The timeline lane and inspector controls need an
`AxcutCursorMotionRegion` on the document schema first, and getopenscreen#116's preview
layer is built on Pixi, which main dropped when the preview moved to the
native compositor -- that part needs rebuilding on the native overlay rather
than porting. The compositor already samples these presets for preview and
export (see `feat/cursor-motion-contract`), so the rendering half is done.

Co-authored-by: Etienne Lescot <etiennelescot@gmail.com>
Refs getopenscreen#116, getopenscreen#113
@EtienneLescot

Copy link
Copy Markdown
Collaborator

Your motion model is now on main's terms, on contrib/cursor-choreography — committed with you as author, so it stays your contribution wherever it lands.

cursorMotion.ts and its 330-line test file transplant onto current main untouched: 17 tests pass, no type errors, no lint. None of the 428-commit history was needed for it.

Two corrections to what I told you earlier, both from actually compiling things rather than eyeballing them:

I said to prefer #113's version of this module. Wrong — they're different lineages, not old and new. Yours knows about source time and crop projection, which is exactly what your editor and preview call into; #113's is shaped for the web editor that got deleted. Yours is the one that survives.

CursorPreviewLayer.tsx can't come across as-is. It's built on Pixi, and main dropped Pixi when the preview moved to the native compositor. That's not a port, it's a rewrite against the native overlay — so I left it out rather than hand you something that doesn't build.

Where that leaves the feature:

  • Model — done, on the branch above
  • Rendering — done, feat/cursor-motion-contract samples your presets in Rust, for preview and export both
  • Timeline lane + inspector — your wiring is independent of Pixi and should port, but it needs an AxcutCursorMotionRegion on the document schema first
  • Control-point dragging in the preview — needs rebuilding on NativeCompositorOverlay

So the two ends exist and the editor is the gap. Yours to take whenever you want it — or to leave, and it's still in the history either way.

@EtienneLescot
EtienneLescot force-pushed the agent/ai-cursor-choreography-concept branch from 707fccd to 00d7018 Compare August 1, 2026 13:29
@EtienneLescot
EtienneLescot changed the base branch from feat/ai-edition to main August 1, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants