DexVision

DexVision is a staged robot-learning and computer-vision project for building reusable manipulation skills with a simulated Shadow Hand. The active pipeline uses deterministic MuJoCo experts to generate replayable demonstrations, then trains and qualifies compact policies against frozen state and visual tests. The earlier OpenCV/MediaPipe hand-control prototype and its Level 2 dataset are retained as historical evidence, but they are not a data or control source for Level 4 onward.

Roadmap

Level 1  legacy camera hand-control prototype                         complete
Level 2  recording, replay, quality, datasets, retargeting benchmarks complete
Level 3  learning feasibility on the existing Level 2 datasets       complete
Level 4  comprehensive multi-session skill dataset                   complete
Level 5  full-scale skill learning and qualification                   planned
Level 6  robustness, reproducibility, results, and portfolio polish    planned
Level 7  language-guided composition of typed skills                   future

Level 3 established that the reproducible learning loop works but that no Level 2-trained policy passes the frozen closed-loop gates. Level 4 is complete: the immutable v1 release contains 1,112 active episodes and 2,633 visual frames, with all 74 coverage cells qualified, frozen whole-session splits, deterministic expert generation, causal phases, explicit failures and scripted corrections. Both Level 2 and Level 4 passed independent Git LFS retrieval and restoration from a clean clone. See release instructions and current status. Level 5 has not started. No live hand-control collection is part of the remaining plan. Level 5 will train and qualify five core skills—reach, pick, place-held-object, push, and button press—through a supervised runtime. It will validate a coherent tabletop workcell assistant on workspace clearing, inspection-station operation, and workspace setup; dial rotation is optional. Learned drop recovery, general arbitrary-object grasping, tools, kitchen tasks, cutting, pouring, deformables, and open-world scenes are deferred.

Clean setup

Install Git and a Conda distribution such as Miniconda or Miniforge, clone this repository, and run the following commands from the repository root:

conda env create -f environment.yml
conda activate dexvision
python -m dexvision.apps.health_check

The environment specification installs Python 3.11, the runtime dependencies, pytest, and Ruff. To update an existing environment after the specification changes:

conda env update --name dexvision --file environment.yml --prune
conda activate dexvision

PyTorch is used for learning work and is an optional dependency:

python -m pip install -e ".[learning]"

The live demo uses these repository assets:

assets/models/hand_landmarker.task
assets/mujoco/hand_scene.xml
configs/level1_teleop.yaml

Level 1 Demo (Legacy)

This historical hand-tracking demonstration remains reproducible, but it is not used to generate Level 4 data, corrections, or final skill behavior.

macOS

Run viewer-based applications from Terminal or iTerm with mjpython, which is installed with MuJoCo:

mjpython -m dexvision.apps.run_level1_teleop --camera-id 0 --show-camera-window --print-interval 10

macOS may ask for camera permission the first time the application runs. The automated checks do not open a camera or GUI.

Windows

Run the same module from Anaconda Prompt or PowerShell after activating the environment:

python -m dexvision.apps.run_level1_teleop --camera-id 0 --show-camera-window --print-interval 10

Allow camera access in Windows privacy settings if the live application cannot open the selected camera.

The demo opens the MuJoCo hand viewer and a camera overlay with landmarks, finger-control bars, tracking confidence, FPS, and tracking-loss status. Press q in the camera overlay, close the viewer, or press Ctrl-C to stop.

For a short demo video or GIF, record the camera overlay and MuJoCo viewer with the operating system screen recorder while the demo is running.

Development checks

Always run checks in the dexvision environment:

conda activate dexvision
ruff check dexvision tests
pytest
python -m dexvision.apps.health_check

Automated tests use synthetic camera data and do not require a webcam, GPU, or visible MuJoCo GUI.

Demonstration data

Working demonstrations under data/demos/ are local data and are intentionally ignored by Git. Creating or updating the environment does not remove them. An immutable Level 2 snapshot is published separately in this repository through Git LFS; see Versioned Dataset Releases for download, verification, and extraction commands. See the Level 2 dataset runbook for the editable collection layout and commands.

The first reach-policy offline split, rollout scenarios, and acceptance gates are frozen before training in the Level 3 evaluation protocol. Reserved held-out target poses are closed-loop MuJoCo rollout conditions, not required demonstration samples.

Level 3.1 through Level 3.8 are complete: the repository now has a goal-conditioned dataset loader, a schema-bound MLP, reproducible CPU behavior-cloning, validation-only best-checkpoint selection, and frozen headless evaluation for reach, button press, and cube push. The final Level 3 feasibility report records the no-go decision for Level 2-trained policies and the evidence-backed Level 4 requirements. Train the three preserved baselines from the repository root with:

python -m dexvision.apps.train_policy --config configs/level3_reach_bc_v2.yaml
python -m dexvision.apps.train_policy --config configs/level3_button_bc.yaml
python -m dexvision.apps.train_policy --config configs/level3_push_bc.yaml

Each command saves distinct best-validation and last checkpoints with SHA-256 sidecars. Checkpoint metadata includes the exact model and schema contract, dataset/split/normalization digests, training-only normalization statistics, loss history, selected epoch, seed, and environment information. Evaluate the selected checkpoints against their unchanged frozen protocols with:

python -m dexvision.apps.evaluate_policy --training-config configs/level3_reach_bc_v2.yaml
python -m dexvision.apps.evaluate_policy --training-config configs/level3_button_bc.yaml
python -m dexvision.apps.evaluate_policy --training-config configs/level3_push_bc.yaml

All three Level 3.5B closed-loop baselines failed their frozen gates. Level 3.6 then measured action-space and safety effects, Level 3.7 correctly skipped an untriggered temporal model, and Level 3.8 closed the feasibility study. The negative results and every rollout failure remain preserved under outputs/level3/ rather than being tuned away.

Project staging and checkpoint status are documented in CURRENT_STATUS and the active Level 4 progress file. Completed Level 3 is documented in the Level 3 progress file. The later roadmap is defined in the Level 5 full-scale learning plan, Level 6 polish plan, and future Level 7 orchestration boundary.

Completed Level 2 dataset status

Level 2.10B is complete. The three manipulation datasets currently marked Level 3-ready are:

• reach_touch_target: 55 clean successes with balanced target coverage (18/18/19)
• button_press: 55 clean successes across nine configured button/depth goals
• push_cube_to_target: 101 clean successes across three lane-aligned goals (33/34/34)

Every retained push-cube episode validates, completes semantic headless replay, recomputes as successful, and passes quality filtering. Its versioned split also reserves three interpolated cube start/target-pose states for Level 3 evaluation.

Policy-free Level 2 skill metadata can be exported from each implemented task spec and the matching dataset-summary group:

python -m dexvision.apps.export_skill_metadata --task reach_touch_target
python -m dexvision.apps.export_skill_metadata --task button_press
python -m dexvision.apps.export_skill_metadata --task push_cube_to_target

These Level 2 release stubs intentionally remain policy-free; Level 3 training checkpoints are saved separately under outputs/level3/.

Level 2.8 adds a second, approximate retargeting baseline alongside the existing curl retargeter. It converts MediaPipe-compatible hand landmarks into five palm-local fingertip targets normalized by palm width, maps fingertip extension to bounded Shadow Hand controls, and safely falls back when target extraction or solving fails. Run its synthetic checks with:

pytest tests/test_fingertip_retargeter.py

Level 2.9 adds an optimization retargeter that minimizes palm-local fingertip error, configured actuator-limit violations, and temporal changes in finger controls. It uses bounded SciPy optimization when available, retains a deterministic projected-gradient path when SciPy is absent, clips outputs, and falls back to last-valid or safe-open targets. Run its synthetic checks with:

pytest tests/test_optimization_retargeter.py

Level 2.10B validates all three retargeters on all 101 saved push_cube_to_target episodes (7,176 frames). The shared human feature and landmark streams feed each retargeter, and counterfactual headless MuJoCo replay combines each method's finger targets with the recorded base actions. Values in brackets are deterministic 95% episode-bootstrap intervals from 2,000 resamples with seed 0:

Retargeter	Latency ms [95% CI]	Jerk [95% CI]	Surrogate tip error [95% CI]	Success [95% CI]	Tip-cube distance m [95% CI]	Tip-contact rate [95% CI]
Curl	0.0766 [0.0763, 0.0768]	0.01547 [0.01465, 0.01634]	0.4854 [0.4767, 0.4938]	0.871 [0.802, 0.931]	0.10573 [0.10541, 0.10604]	0.00867 [0.00592, 0.01168]
Fingertip	0.1014 [0.0998, 0.1041]	0.01872 [0.01736, 0.02022]	0.4546 [0.4454, 0.4637]	0.832 [0.762, 0.901]	0.10632 [0.10604, 0.10658]	0.00574 [0.00368, 0.00815]
Optimization	1.3040 [1.2661, 1.3696]	0.01798 [0.01668, 0.01941]	0.4546 [0.4454, 0.4637]	0.871 [0.802, 0.931]	0.10585 [0.10552, 0.10616]	0.00988 [0.00715, 0.01286]

All three methods had zero joint-limit violations. Latency varies by machine; action jerk is normalized actuator units per frame cubed, surrogate fingertip error is normalized by palm width, and tip-cube measurements use the five distal collision geoms in MuJoCo. Generate the versioned JSON, flattened CSV, and six-panel SVG artifacts with:

python -m dexvision.apps.benchmark_retargeters --task push_cube_to_target --episodes 101 --bootstrap-samples 2000

The historical demonstrations and base trajectories were collected with curl retargeting. Reusing those base trajectories makes this a controlled counterfactual finger-retargeting comparison, but it does not remove that curl bias. No independent live fingertip/optimization trajectories were collected or claimed, and no follow-up human-control collection is planned.

Level 2.11 also publishes the immutable dataset snapshot through Git LFS and freezes the first reach-policy evaluation protocol. Level 3 completed offline training, frozen closed-loop rollout, diagnostics, and its feasibility report for reach, button press, and cube push. Level 7 orchestration has not been implemented.

Known Limitations

This is a simulated manipulation-learning and dataset pipeline, not a real- robot controller. The legacy Level 1 hand-control demonstration remains limited by monocular tracking, camera placement, and approximate finger retargeting; those limitations are why it is excluded from the active skill-data path. Use --assume-mirrored-input only for selfie-mirrored camera feeds. The Level 2.8 fingertip baseline is a geometric approximation rather than a numerical robot-model IK solve. The Level 2.10 fingertip-error metric uses the same palm-local surrogate for a consistent comparison and is not a measured physical robot fingertip distance.