4.0 - #402
Open
jsuarez5341 wants to merge 856 commits into
Open
Conversation
Tb bristol 4.0 boxoban
remove outdated installation tests
Merging this here, will pull it on to 5c and test there
Nethack hypers
Nethack head gating, 5-6k score
The star history chart in the README is currently broken due to GitHub stargazer API restrictions. Update the chart links and images to point to our working star history service so the chart displays correctly again.
Fix broken star history chart in README
Physics: replace the single semi-implicit Euler step with RK4 plus configurable substeps. Acceleration is factored out of integrate_physics into dp_accel so it can be evaluated at RK4 trial states; the dynamics themselves are unchanged. Measured energy drift over a 10s episode with zero force falls from 123% to 0.6355% at substeps=1 and 0.0006% at substeps=4. That matters here because a swing-up is entirely energy bookkeeping -- an integrator that leaks energy makes the catch easy for the wrong reason. Reward: replace 0.5*height + hold-streak bonus with a DeepMind Control Suite-style dense blend. The streak bookkeeping is kept because hold_time is derived from it, which keeps that metric reward-independent. On accurate physics the reward choice is decisive. Building on the terminal-signal fix in this branch's parent, at 500M steps from a dead hang, hold_time out of 600: Euler + streak 358 (previous behaviour) RK4 + streak 0 -- never catches RK4 + DeepMind 407 The streak reward pays 0.5100 for the first held step against 0.5000 for flying through the top, a 2% signal that Euler's energy drift was masking by acting as unintended dissipation. The DeepMind blend pays 4.2x at the same decision point. Note this removes the old reward rather than keeping it behind a flag, so existing double_pendulum results are not reproducible from here on. The standalone demo sets substeps and balance_bonus_weight explicitly. Its designated initializer would otherwise zero them, leaving the interactive binary on RK4x1 with w=0 while training runs RK4x4 with w=0.5. Also fix min_lr_ratio = 0 -> 0.0. The CLI infers each flag's type from its config default, so the int made --train.min-lr-ratio reject float values. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fix double pendulum env
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR will not be merged. We are targeting EoY and 4.0 will just become master. Key goals:
TBD: cpp/barracuda, final constellation features, xlstm, advantage calc tweaks