Skip to content

Add PEAR embodied experiment qualification and tracking - #783

Draft
rpuns wants to merge 87 commits into
codex/leact-efe-20261115/09-adapter-checkpoint-gatherfrom
codex/leact-efe-20261115/10-pearplusplus-preparation
Draft

rpuns wants to merge 87 commits into
codex/leact-efe-20261115/09-adapter-checkpoint-gatherfrom
codex/leact-efe-20261115/10-pearplusplus-preparation

Conversation

@rpuns

@rpuns rpuns commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

This change adds the PEAR preparation, launch and tracking pipeline for the embodied multitask study. Primary training requires a qualified full curriculum, frozen-router numerical checks and native observation/action/reward validation. Diagnostic updates remain separate from primary optimizer history.

  • Qualify full-model frozen-router updates and restarts, on-policy score centering, distributed queue recovery and private checkpoint upload/readback/restore.
  • Add eight procedural reasoning families, native reward controls, protected-scene checks and recorded RGB audits.
  • Launch separate trainer, sampler and native simulator allocations with explicit Ray CPU budgets. Retry infrastructure failures, honor persistent user pauses, cancel only owned workers and wait for the current model owner's startup rather than accepting a previous attempt's completion marker.
  • Honor per-pool HIGH/LOW placement, compatible shared-volume retries and persistent user pause controls. Add independent graphics capacity through a validated external file relay while retaining the existing model and working simulator allocations. Support differing filesystems through private object transport and pinned asset placement; that path remains gated on a real native cross-pool reset/action/verdict check and renderer migration smokes.
  • Keep native float32 action bounds and observations consistent through short, bit-preserving decimal serialization. Gate added workers on a real reset, RGB frames, executed boundary action and finalized native verdict. Validate transport-probe commands through the policy parser and expose reduced native-container concurrency for recovery.
  • Add an opt-in task curriculum with SGS over tasks and reset variants. Fill 128-by-eight batches from available mixed-outcome groups and balance family targets over eight updates; preserve the full admitted mixture, exploration, training-only histories and restart state. Gate the approved transition on exact driver composition and a native-success known-case replay; preserve any completed optimizer progress. Matched replays share the production final JSON decoder so reasoning wrappers are handled consistently. Bound replay completions to the serving context and retain HTTP error bodies.
  • Retire canceled or expired unclaimed simulator requests with retained receipts. Retry recognized native infrastructure failures with bounded backoff while continuing to fail on persistent source errors.
  • Keep model and simulator allocations alive across child-job failures and recipe changes. Add versioned job submission inside retained nodes, primary/standby selection with a shared writer guard, and explicit-user-only allocation release. Retry expired LOW standby requests without changing live clusters; scope telemetry caches to the admitted run directory.
  • Finalize native terminal rewards before environment cleanup on token/parse endings, without adding sampled actions or tokens. Keep missing verdicts as infrastructure failures and tag diagnostic pre-repair runs. Add an early sampling preference for confirmed native training tasks plus explicit LIBERO/RoboDojo training probes; decay it over eight updates while retaining broad coverage. Apply validated replacements inside retained allocations and stage fresh sources before standby takeover.
  • Scope hosted telemetry, log folding and snapshot overlays to the admitted curriculum identity so older recipes cannot supply the current arm's rewards, metadata or optimizer history.
  • Reap owned Ray/vLLM descendants across retained-node retries using allocation, workflow and attempt identities, with PID-reuse checks. Verify detached-process cleanup on Linux and release old GPU allocations inside the container before restarting the same recipe.
  • Isolate retained native-worker tool/source/socket paths to avoid overwriting a running conmon binary while reusing cached image layers. Add requested L40-03 rollout capacity and deferred in-allocation retargeting. Stage a separately identified 32-group startup variant with eight rollouts per group and proportionate rolling family targets; gate handoff on exact CPU composition and preservation of any completed optimizer work.
  • Reset keeper ownership metadata on fresh scheduler retries, preventing an inherited old run path from blocking the rest of simulator setup. Add the user-requested hourly local chat heartbeat with duplicate-send protection and supported Codex queue/resume delivery to the existing thread.
  • Track validation, current scheduler state, optimizer metrics and task rewards on the independent dashboard. Read telemetry through separate workers so exec probes cannot block the primary OSMO startup barrier. Completed scientific results require observed updates and evaluations.

Validation: 715 research tests passed, 4 skipped, 8 subtests passed; JavaScript syntax and diff checks passed. The required repository-wide unit command was attempted, but local collection is blocked by the missing lightning package. Private sources, task catalogs, credentials and experiment artifacts are excluded.

Primary RL progress is tracked separately from diagnostic updates; no held-out improvement is claimed. Static asset integrity checks do not establish native rollout readiness.

  • Fix the shared simulator retirement race that crashed YAM and RoboDojo workers after concurrent abandoned-request cleanup. Preserve inconsistent-duplicate errors, and retry OSMO controller-ended idle standbys without stopping live allocations.

  • Suppress identical replays of retired simulator requests while rejecting changed contents. Automatically stage this transport-only repair for the exact failed simulator child inside its retained allocation, with current-owner and all-task startup guards; preserve healthy peers and simulator specs.

  • Reconcile scheduler-created simulator children only when their frozen-spec initial ID matches the current desired request. Include native EFE lane logs in recognized transport-failure recovery; keep healthy jobs running.

  • Serve live simulator requests oldest-first instead of sorting random UUIDs, while preserving request deadlines, retirement grace, atomic ownership, replay safety and grading. Close the owned native trial on worker termination. Verify source checksums and queue order on the allocated Linux node before replacing only the affected child inside its retained allocation.

  • Permit an explicit extension of sequential LOW standby retries after the default software limit is exhausted. Preserve terminal-failure checks, pinned sources, live allocations and user cancellations; never queue a second standby while one is pending or running.

@rpuns rpuns changed the title Prepare expanded PEAR++ arms and rp_jhh checkpoint names Prepare PEAR++ launch and integrate the multitask curriculum Nov 17, 2026
@rpuns rpuns changed the title Prepare PEAR++ launch and integrate the multitask curriculum Prepare PEAR++ and prioritize EFE LeAct R1 evaluations Nov 17, 2026
@rpuns rpuns changed the title Prepare PEAR++ and prioritize EFE LeAct R1 evaluations Recover EFE LeAct R1 scoring and prepare PEAR++ Nov 17, 2026
@rpuns rpuns changed the title Recover EFE LeAct R1 scoring and prepare PEAR++ Recover EFE LeAct R1 and add PEAR planning workspace Nov 17, 2026
rpunamiya added 7 commits November 17, 2026 15:26
Persist per-task RL telemetry, hosted OSMO collection and interactive metric charts. Keep native split assignments in private artifacts and freeze expert-selection routers for future MoE training behind qualification gates.

Add pinned base/warmstart RealWorldQA evaluation with resume checks, paired statistics and a separate batched-serving qualification. Primary training remains paused.

Validation: 464 research tests passed, 3 skipped, 8 subtests passed; desktop/mobile and synthetic interactive curves checked. Root unit collection requires the unavailable lightning package. GPU router and full RL instrumentation qualification remain pending.
Evaluate the same seeded random subset for base and warmstart with the unchanged inference budget. Run warmstart first, preserve the subset on infrastructure retries, and label preliminary results and progress counts throughout the hosted dashboard.

Validation: 466 research tests passed, 3 skipped, 8 subtests passed. Live H100 execution confirmed the 40-question panel and warmstart-first order. Hosted desktop/mobile checks passed. Root unit collection is blocked by the missing lightning dependency.
Qualify frozen-router updates and restarts, audit source reset identities and recorded RGB, and add eight independently generated reasoning families with native PEAR transport checks. Keep discarded numerical and on-policy checks outside primary run history, and require the full curriculum before primary training.

Track environment admission and renderer hardware requirements on the independent dashboard. Preserve native control/verifier semantics and cache exact simulator bases; private simulator source and catalogs remain excluded.

Validation: 500 research tests passed, 3 skipped, 8 subtests passed; shell/JavaScript syntax and staged diff checks passed. Root tests/unit collection remains unavailable because the local venv lacks lightning. Main RL is not yet launched; simulator admission and on-policy qualification are still pending.
@rpuns rpuns changed the title Recover EFE LeAct R1 and add PEAR planning workspace Add PEAR embodied experiment qualification and tracking Nov 18, 2026
rpunamiya added 30 commits November 19, 2026 04:52
Keep the eight-GPU trainer and fixed training recipe while allowing a six-node backup allocation. Preserve its five-node inference layout on first-ready promotion.

Validation: 47 focused primary-ownership and standby tests passed; the staged workflow passed OSMO validation and was accepted. The repository unit suite remains blocked by the existing missing lightning dependency.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant