Follow-up Architecture (DTRA v3.0.0): Our standalone successor scaling to 100 sequential 7-DoF LIBERO-100 tasks (
87.00 ± 0.86%final SR across 3 seeds,99.90%routing,27frozen shared motor bases) and 8 continual 3D Meta-World tasks with an$O(1)$ active decoding core and Two-Timescale Elastic Snap-Back on$\mathbb{S}^{63}$ is published at https://github.com/5Color/dtra (Zenodo v3.0.0 DOI:10.5281/zenodo.24297711).
Can a lightweight visuomotor policy learn sequential multi-stroke manipulation tasks without catastrophic forgetting, without an experience replay buffer, and without ground-truth
Task-IDlabels at inference time?
ABK-VLA (Autonomous B-spline Knot-based Visuomotor Policy) addresses sequential continual learning in compact closed-loop trajectory control. While massive foundation models dilute forgetting through over-parameterization and large-scale replay, lightweight visuomotor policies suffer severe trajectory collapse (46.44%–89.33% forgetting) under sequential fine-tuning.
By compressing 4.59M shared parameters) with lightweight residual Winner-Take-All (WTA) slot heads (0.33M parameters per residual slot, 7.16% of the backbone, 94.44% total model frozen at Select-then-Blend, 99.98% average diagonal routing confidence (4/4 slot match on canonical targets), 1.66% forgetting rate, and 85.59% average final apple → sprout → star → cat)—recovering 92.1% of the external-Task-ID Oracle baseline (92.96%) without any replay data or test-time task labels.
(For the earlier Phase 1 closed-loop ACT baseline and crowdsourced web benchmark, see paper/main.tex, Drawing_VLA.pdf, and 5Color/drawing-vla.)
After sequentially learning all four tasks, naive fine-tuning baselines (Model 1, Model 2) collapse on earlier tasks, and experience replay (Model 3) degrades on complex multi-part geometry (Cat Model 5) preserves structural fidelity across all four tasks without external test-time task labels.
| Sequential |
Task-ID-Free Softmax Routing ( |
|---|---|
![]() |
![]() |
All five models were evaluated on Intel(R) Arc Graphics (xpu, FP32, PyTorch 2.15.0+xpu, seed=42) using identical dataset splits (85% train / 15% val by episode), identical curriculum and augmentation schedules (T0: apple 10 epochs × 1× aug, T1: sprout 12 epochs × 3× aug, T2: star 12 epochs × 5× aug, T3: cat 16 epochs × 5× aug, batch_size=64, WeightedRandomSampler for multi-stroke tasks), identical spatial pointers (compute_res_cue + compute_spatial_pointer), post-hoc chunk-boundary exec_steps=16, max_iters=100, base patience=8 dynamically extended to effective_patience=16 when
Protocol Note on Optimizer & Head Differences: In
experiments/benchmark_autonomous_cl.py,T0useslr=3e-4for all models; forT1–T3, single-modeModels 1–3uselr=3e-4, whereas slot-isolated 6-mode WTAModels 4–5uselr=1e-3(Model 5prior_weightuses5e-3).Model 4usesWTASlotHead(0.30M) with cross-entropy gating loss, whereasModel 5usesResidualWTASlotHead(0.33M) with pointer-conditioned mode selection, focal gating loss, and pen-consistent supervision.
| Evaluation Axis | Metric | Model 1GRU + Naive FT
|
Model 2ABK + Naive FT
|
Model 3ABK + Replay
|
Model 4 (Oracle)ABK Slots (C)
|
Model 5 (Ours)Autonomous ABK (D)
|
|---|---|---|---|---|---|---|
| Conditions | Inference Task-ID |
None | None | None | Required (Oracle) | None (Autonomous) |
| Replay Buffer | None | None |
256 samples/task |
None | None (Zero Replay) | |
| Slot Head / Modes | GRU (H=16) |
Single-mode MLP | Single-mode MLP | 6-mode WTA (0.30M) |
6-mode Residual WTA (0.33M) |
|
| Task Precision |
Apple (T0) |
6.5% | 4.7% | 54.4% | 99.7% | 88.5% |
Sprout (T1) |
11.7% | 5.4% | 88.8% | 92.2% | 81.1% | |
Star (T2) |
28.7% | 8.3% | 82.4% | 87.1% | 96.4% | |
Cat (T3) |
1.6% | 68.0% | 25.9% | 92.9% | 76.5% | |
| Avg Final |
12.12% | 21.61% | 62.87% | 92.96% | 85.59% (92.1% of Oracle) | |
| Continual Learning | Forgetting Rate |
46.44% | 89.33% | 14.13% | 0.00% | 1.66% (54.0× vs Naive FT) |
| Backward Transfer (BWT) | -46.44% | -89.33% | +1.95% | 0.00% | -1.66% | |
| Geometry & Control | Chamfer Distance |
0.1043 | 0.0984 | 0.0327 | 0.0098 | 0.0192 (5.4× vs GRU) |
| Normalized Mean Jerk |
5.05 | 5.91 | 5.92 | 3.09 | 3.33 | |
| Component Recall |
0.0% | 6.3% | 50.0% | 87.5% | 62.5% (+12.5%p vs Replay) | |
| Efficiency | Episode Latency |
601.6 ms | 108.6 ms | 564.9 ms | 346.6 ms | 464.3 ms (1.30× vs GRU) |
| T3 Trainable Params | 5.08M (Full) | 4.72M (Full) | 4.72M (Full) | 0.30M (Slot) | 0.33M (94.44% Frozen) |
-
Cubic B-Spline Trajectory Compression (
BSplineDecoder): Instead of predicting$H = 16$ raw Cartesian waypoints, each slot predicts$N_k = 6$ sigmoid-bounded control knots$\mathbf{P} \in (0, 1)^{6 \times 2}$ and step-wise pen logits$\mathbf{u} \in \mathbb{R}^{16}$ , mapped analytically via a frozen cubic B-spline design matrix$\hat{\mathbf{Y}} = \mathbf{B}\mathbf{P}$ . During closed-loop rollout (rollout_bspline.py), continuous pen-down transitions apply post-hoc chunk-boundary smoothing ($\hat{\mathbf{y}}0 \leftarrow \mathbf{p}{\text{prev}}$ for$C^0$ , and $\hat{\mathbf{y}}1 \leftarrow 0.7,\hat{\mathbf{y}}1 + 0.3,(\mathbf{p}{\text{prev}} + \mathbf{v}{\text{prev}})$ when$\cos\alpha > 0$ for$C^1$ tangent continuity), yielding smooth control (NMJ = 3.33). -
Thin-Stroke Spatial Pointer & Residual Cue (
compute_spatial_pointer,compute_res_cue): From the uncovered target residual $\mathbf{M}{\text{uncov}} = \max(0, \mathbf{I}{\text{target}} - \mathbf{I}_{\text{canvas}})$, the policy extracts:- A 64-d
$8 \times 8$ adaptive average-pooled residual cue (compute_res_cue, scaled by4.0). - A 7-d geometric spatial pointer (
[cx, cy, rem_ratio, dx_center, dy_center, dx_near, dy_near]), where nearest uncovered pixels are located via a blended salience map0.7 * uncov + 0.3 * AvgPool3x3(uncov)(thresholded at> 0.15, fallback> 0.08) to preserve thin 1 px stems, leaves, and whiskers.
- A 64-d
-
5,376-d Hybrid Prototype Key & Select-then-Blend Residual Routing (
AutonomousABKPolicy):-
Prototype Key (
_compute_target_key): Concatenates L2-normalized Conv44×4pooled features (4,096-d, weight0.5) with L2-normalized multi-scale target silhouette average pooling (32×32smoothed +16×16=1,280-d, weight1.0), followed by final L2 normalization (5,376-d). -
Select-then-Blend Inference (
predict): Rather than hardargmaxrouting over a single slot,AutonomousABKPolicycaches the static target key and softmax routing weights$\mathbf{w}$ ($T = 0.05$ ) per episode (reset_cache()), evaluates all$K$ ResidualWTASlotHeadmodules sequentially with logit-space residual composition (sigmoid(prior_weight * logit(base_cp) + delta_cp)), selects the highest-scoring WTA mode within each slot (argmaxoversel_head), and computes a$\mathbf{w}$ -weighted average across all slots. -
Routing & Forgetting Trade-off: On the 4 canonical target templates, diagonal softmax routing weights are
99.99%(Apple),100.00%(Sprout),99.97%(Star), and99.96%(Cat) (99.98%mean diagonal weight,4/4slot match). Because soft Select-then-Blend includes tiny off-diagonal tail weights ($<0.04%$ ) from subsequently added residual slots, closed-loop rollout on frozen earlier tasks (Apple:92.48% → 88.46%,Star:97.30% → 96.36%) exhibits a slight1.66%forgetting rate. -
Episode Latency: Evaluating all 4 slots with
reset_cache()yields464.3 msper episode (7.02 msper chunk) inside the 5-model sequential training-and-evaluation benchmark (benchmark_autonomous_cl.py, Table 1)—1.30×faster thanGRU(601.6 ms) and1.22×faster thanABK + Replay(564.9 ms), while remaining slower than single-slotModel 4(346.6 ms) and early-stallingModel 2(108.6 ms). (Note: When re-evaluating the saved checkpointoutputs/autonomous_model_d_final.pthstandalone inExp 3,Blend T=0.05runs at387.8 msvs357.5 msforHard Argmaxunder identical standalone timing conditions.)
-
Prototype Key (
-
Full-Chunk Horizon Alignment (
exec_steps=16): Executing the complete$H = 16$ horizon (exec_steps=16) prevents premature truncation of pen-up jump trajectories before touchdown on disconnected components.
To stress-test every theoretical and empirical claim under strict continual learning conditions, we added four follow-up benchmarks:
| Experiment | Script | Key Verified Result | Figure |
|---|---|---|---|
| Exp 1: OOD Routing & Key Ablation | experiments/eval_routing_perturbation.py |
Hybrid key (0.5·Conv4 + 1.0·Spatial) keeps 100.0% routing up to ±8 px shift, ±15° rotation, ±15% scale, sigma=0.40 noise, and 20% occlusion (92.5% at 40%); balanced 50/50 reaches 90.0% at ±12 px. (Closed-loop OOD F1: 65.04% Blend / 56.97% Argmax at 4 px shift, 72.78% / 72.59% at 8° rot, 29.55% / 22.49% at sigma=0.15 noise.) |
Perturbation Curve |
| Exp 2: 3-Seed & Task-Order Robustness | experiments/benchmark_multiseed_order.py |
Across seeds {42, 43, 44}, Forward achieves 85.15 ± 1.06% F1 (FR = 0.61 ± 0.74%) under Blend and 84.58 ± 1.14% (FR = 0.00 ± 0.00%) under Argmax; Reverse achieves 79.50 ± 4.97% (FR = 0.06 ± 0.06%) vs 78.16 ± 5.53% (FR = 0.00 ± 0.00%). |
Multi-Seed Order |
| Exp 3: Select-then-Blend vs Hard Argmax | experiments/eval_argmax_vs_blend.py |
On saved checkpoint outputs/autonomous_model_d_final.pth, switching to hard_argmax eliminates residual tail-weight forgetting completely (FR: 1.66% → 0.00%, standalone Latency 387.8 ms → 357.5 ms) while Default Blend (T=0.05) preserves slightly higher transfer (85.59% vs 84.39% F1). |
Argmax vs Blend |
| Exp 4: 3D Meta-World Manipulation | experiments/benchmark_metaworld_cl.py |
Ported directly to 4-DoF 3D MuJoCo manipulation (button-press-v3 -> door-open-v3 -> window-open-v3 -> faucet-open-v3) with 128×128 RGB + 4-d EEF & DoG+Conv4 key: 91.7% Avg SR / 0.0% FR / +11.1% BWT (100.0% autonomous routing accuracy without test-time Task-ID, 99.71% conf) vs Naive FT 43.3% / 75.6% FR, Replay 88.3% / 13.3% FR, Oracle Slots 90.0%. |
Meta-World 3D CL |
| Method | Test-Time Task-ID? |
Replay? | Final Avg SR |
FR |
BWT |
Autonomous Routing (No Task-ID) |
Total Params |
|---|---|---|---|---|---|---|---|
| Naive FT (1-Head B-Spline) | Not Used | No | 43.3% | 75.6% | -75.6% | — | 1.60M |
| Experience Replay (256/task) | Not Used | Yes | 88.3% | 13.3% | -13.3% | — | 1.60M |
| Oracle Slots (External ID) | Required | No | 90.0% | 0.0% | +0.0% | Oracle-Selected | 2.19M |
| Autonomous 3D ABK (Hard Argmax) | None | No | 86.7% | 0.0% | +0.0% |
100.0% (No Task-ID) |
2.19M |
| Autonomous 3D ABK (Select-then-Blend) | None | No | 91.7% | 0.0% | +11.1% |
100.0% (No Task-ID) |
2.19M |
To ensure transparent scientific interpretation, we explicitly define the operational boundaries of this study:
-
Goal-Image / Visual-Only Conditioning & Simulation Testbeds: In this repository, "ABK-VLA" denotes a compact vision-and-proprioception visuomotor action-chunking policy validated on a 2D multi-stroke drawing testbed and a 3D Meta-World (
MuJoCo) continual manipulation benchmark (Experiments 1–4). The current implementation does not include a language encoder/VLM backbone or physical hardware validation. -
Training Task Boundaries, OOD Sensitivity & Narrow 3D Margin: Inference is
Task-ID-free, though sequential training invokesregister_or_expand_slotat discrete task boundaries. Experiment 1 confirms100.0%routing accuracy up to ±8 px shift, ±15° rotation, ±15% scale,sigma=0.40noise, and 20% occlusion (92.5%at 40%), while closed-loop OOD rollout F1 drops under stronger target corruptions (65.04%at 4 px shift,29.55%atsigma=0.15noise). In 3D Meta-World, subtracting a 15×15 local average (DoG residual) and using 8 initial-frame exemplars (T=0.003,threshold=0.996) yields a raw$T_3$ cosine margin of0.0168(0.9792vs.0.996) and assumes distinct initial ($t=0$ ) scene layouts. -
Linear
$O(K)$ Slot Chaining & Blend-vs-Argmax Trade-Off: Both slot memory (0.33M/slot in 2D,5.90Mtotal at$T_3$ ;0.19M/slot in 3D,2.19Mtotal at$T_3$ ) and sequential residual chaining scale linearly as$O(K)$ , even underhard_argmax. Experiment 3 confirms that switching fromSelect-then-Blend(T=0.05) tohard_argmaxeliminates the residual tail-weight forgetting (1.66%→0.00%FR; standalone latency387.8 ms→357.5 ms) at the cost of a minor-1.20 pptransfer reduction (84.39%vs.85.59%F1).
abk-vla/
├── README.md # Project overview, benchmark tables, and visual results
├── requirements.txt # Core Python dependencies (+ optional 3D Meta-World packages)
├── .zenodo.json # Zenodo DOI archival metadata (Son, Seungjoon / MIT)
├── RELEASE_NOTES_v1.1.2.md # Release notes for GitHub Release v1.1.2
├── RELEASE_NOTES_v1.1.3.md # Release notes for GitHub Release v1.1.3
├── paper/
│ ├── ABK_VLA_Paper.tex # Full two-column LaTeX preprint manuscript
│ ├── ABK_VLA_Preprint.pdf # Compiled 9-page OpenType preprint PDF
│ └── main.tex # Phase 1 Drawing-VLA ACT manuscript
├── data/
│ └── bspline/ # Preprocessed 4-task B-spline datasets (H=16, Nk=6, stride=8)
│ ├── bspline_vla_dataset_apple_m6_h16_s8.npz
│ ├── bspline_vla_dataset_sprout_m6_h16_s8.npz
│ ├── bspline_vla_dataset_star_m6_h16_s8.npz
│ └── bspline_vla_dataset_cat_m6_h16_s8.npz
├── train/
│ ├── bspline_policy.py # AutonomousABKPolicy (blend & argmax), BSplineModelCPolicy, GRU, MLP
│ ├── rollout_bspline.py # Closed-loop rollout engine, post-hoc C0/C1 & stall patience
│ └── dataset.py # Observation rendering and trajectory utilities
├── tools/
│ ├── bspline_core.py # Clamped cubic B-spline basis, BVLS fitting, BSplineDecoder
│ └── bspline_augment.py # Equivariant affine/elastic/covariate augmentation & pen loss
├── experiments/
│ ├── benchmark_autonomous_cl.py # 5-model continual learning benchmark runner
│ ├── benchmark_three_models.py # Shared data loader, device utilities, and 3-model runner
│ ├── eval_routing_perturbation.py # [Exp 1] OOD perturbation routing curves, key ablation & rollout F1
│ ├── benchmark_multiseed_order.py # [Exp 2] 3-seed ({42,43,44}) Forward & Reverse task-order benchmark
│ ├── eval_argmax_vs_blend.py # [Exp 3] Select-then-Blend (T sweep) vs Hard Argmax comparison
│ └── benchmark_metaworld_cl.py # [Exp 4] 3D Meta-World MuJoCo 4-task continual manipulation benchmark
├── tests/
│ ├── test_bspline_core.py # Unit tests for B-spline basis and BVLS fitting
│ └── test_bspline_policy.py # Unit tests for policy architectures, blend/argmax, and loss interfaces
└── outputs/
├── autonomous_cl_benchmark.json # Raw 5-model benchmark metrics & routing matrix
├── autonomous_cl_rollout_grid.png # 4x5 qualitative rollout comparison grid
├── autonomous_cl_heatmap.png # 5-model F1 retention heatmaps
├── autonomous_routing_matrix.png # 4x4 Task-ID-free softmax routing matrix
├── autonomous_model_d_rollout.png # Model D standalone rollout grid
├── autonomous_model_d_final.pth # Trained 4-task AutonomousABKPolicy checkpoint
├── exp1_routing_perturbation.{json,png}
├── exp2_multiseed_order.{json,png}
├── exp3_argmax_vs_blend.{json,png}
└── exp4_metaworld_cl.{json,png}
pip install -r requirements.txtpython experiments/benchmark_autonomous_cl.py \
--tasks apple sprout star cat \
--epochs-t0 10 \
--epochs-slot 12 \
--batch-size 64 \
--max-iters 100 \
--exec-steps 16 \
--seed 42python experiments/benchmark_autonomous_cl.py \
--only-model-d \
--tasks apple sprout star cat \
--exec-steps 16 \
--num-modes 6 \
--temperature 0.05 \
--novelty-threshold 0.85# Core B-spline & policy unit tests (9 tests)
python -m unittest tests/test_bspline_core.py tests/test_bspline_policy.py -v
# Exp 1: OOD perturbation routing curves, key weight ablation & closed-loop OOD F1
# (reads outputs/autonomous_model_d_final.pth & outputs/autonomous_cl_benchmark.json)
python experiments/eval_routing_perturbation.py
# Exp 2: 3-seed ({42,43,44}) Forward & Reverse task-order benchmark
python experiments/benchmark_multiseed_order.py --seeds 42 43 44 --orders forward reverse --epochs-t0 10 --epochs-slot 12
# Exp 3: Select-then-Blend (T sweep) vs Hard Argmax slot routing
# (reads outputs/autonomous_model_d_final.pth & outputs/autonomous_cl_benchmark.json)
python experiments/eval_argmax_vs_blend.py
# Exp 4: 3D Meta-World 4-task continual manipulation (requires gymnasium, mujoco, metaworld)
python experiments/benchmark_metaworld_cl.py --epochs 55 --n-demos 30 --n-eval 15 --seed 42





