Skill Reuse as Compression in Agentic RL

Zhikun Xu¹, Yu Feng², Jacob Dineen¹, Taiwei Shi³, Jieyu Zhao³, Ben Zhou¹

¹Arizona State University · ²University of Pennsylvania · ³University of Southern California

---

Overview

ReuseRL grounds agentic RL in the Minimum Description Length principle. It extracts a shared skill dictionary from successful trajectories online with a greedy BPE-style merge, then augments the GRPO objective with a segmentation cost that penalizes idiosyncratic, one-off behaviors. Because the penalty is gated on success, it never discourages task completion.

This README covers how to use the codebase. For the method, the theory, and the results, see the paper.

Citation

@article{xu2026reuserl,
  title={Skill Reuse as Compression in Agentic RL},
  author={Xu, Zhikun and Feng, Yu and Dineen, Jacob and Shi, Taiwei and Zhao, Jieyu and Zhou, Ben},
  journal={arXiv preprint arXiv:2605.31509},
  year={2026}
}

What is here

This repository builds on SkillRL and its vendored verl-agent trainer. ReuseRL is implemented as a reward-shaping module inside the GRPO loop, not as a separate training stage. Core additions:

• agent_system/reward_manager/ — the atomic skill alphabet, the BPE dictionary extraction, and the SegCost penalty, one module per environment: efficiency.py (ALFWorld), efficiency_tw_cooking.py, efficiency_countdown_stepwise.py. The alphabet is defined at the top of each file (e.g. _TW_ATOMIC_SKILLS).
• agent_system/environments/env_package/ — the environments and their rule-based skill projections: alfworld/, tw_cooking/, countdown_stepwise/.
• verl/trainer/ppo/ray_trainer.py — routes (env_name, efficiency_cls) to the right reward manager.
• examples/grpo_trainer/ — one run script per (environment × method × model).
• examples/data_preprocess/generate_tw_cooking_games.py — offline generator for the TextWorld-Cooking game pool.
• sanity_check_bpe_dictionary_vs_bruteforce.py — validates the greedy BPE merge schedule against the brute-force MDL optimum.
• vllm_eval_alfworld.py, vllm_eval_tw_cooking.py, vllm_eval_countdown_stepwise.py — batched vLLM evaluation harnesses.

The three environments above are the ones reported in the paper, and the only ones supported here. The repository still carries the environments inherited from SkillRL and the upstream trainer (sokoban/, webshop/, search/, gym_cards/, appworld/) together with their original run scripts; they are untouched by ReuseRL and unsupported.

Setup

Python 3.8+ with CUDA is required (vllm and flash-attn are pinned to CUDA builds).

git clone https://github.com/ARC-ASU/ReuseRL.git
cd ReuseRL

pip install -r requirements.txt
pip install vllm==0.11.0
pip install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir
pip install -e .

# Required by the Countdown-Stepwise environment; not in requirements.txt.
pip install reasoning-gym==0.1.25 gym==0.26.2

Every run script resolves the repository root from its own location and cds there, so it can be launched from any directory. Two settings are still hard-coded per script and must match your machine:

export CUDA_VISIBLE_DEVICES=2,3   # your GPUs
trainer.n_gpus_per_node=2         # must match CUDA_VISIBLE_DEVICES

TMPDIR defaults to /tmp; export it before launching if /tmp is too small for the Ray and vLLM scratch space.

The parquet files under examples/data/ are placeholders that only carry the modality and data size — the actual task instances come from the environment, not from disk. Each run script regenerates them by calling examples.data_preprocess.prepare, so no manual download is needed.

Environment assets

ALFWorld

pip install alfworld gymnasium==0.29.1 stable-baselines3==2.6.0
alfworld-download -f

TextWorld-Cooking — requires TextWorld with Inform7 available for game compilation. Generate the game pool once before the first run (~3 GB, 6000 compiled games):

pip install textworld

python -m examples.data_preprocess.generate_tw_cooking_games \
    --out agent_system/environments/env_package/tw_cooking/games \
    --num_train 5000 --num_test 1000

The generator takes no random seed: game seeds are 0 … N/5 - 1 within each of five fixed difficulty levels, so this command reproduces the exact pool byte-for-byte. It writes 5000 games to games/train/ (challenge split=train) and 1000 to games/val/ (challenge split=test), as .z8 + .json + .ni triples. The pool is gitignored and must be regenerated per machine.

Countdown-Stepwise — no external assets. Problems are generated in-process by reasoning_gym from the Hydra config, using env.seed=0 for train and env.seed + 100000000 for val to keep the splits disjoint. Reproducibility depends on the reasoning_gym version — the paper's runs used 0.1.25.

Running

Each environment has the same four method variants, one run script each. Using TextWorld-Cooking as the example:

cd examples/grpo_trainer   # or launch from anywhere; the scripts self-locate

# (1) Vanilla GRPO — no compression signal
bash run_tw_cooking.sh

# (2) Pure round-length penalty — the singleton-dictionary degenerate baseline
bash run_tw_cooking_pure_round_length_penalty.sh

# (3) ReuseRL-SegCost — dictionary from batch + global success buffer
bash run_tw_cooking_compression_BPE_idio_value.sh

# (4) ReuseRL-SegCost, no buffer — dictionary fit on the current batch only
bash run_tw_cooking_compression_BPE_idio_value_noGlobalBuffer.sh

The same four variants exist as run_countdown_stepwise_* and as run_alfworld_* (where the baseline is named run_alfworld_pure_length_penalty.sh). Per-model variants are suffixed _qwen2.5_3b, _qwen2.5_7b, _qwen3_4b; buffer-size sweeps are suffixed _buf1024, _buf4096.

Configuration

Behaviour is controlled by these Hydra overrides inside the run scripts:

Flag	Values	Description
actor_rollout_ref.actor.use_efficiency_reward	bool	Master switch for the compression penalty. Absent in the vanilla scripts.
actor_rollout_ref.actor.efficiency_cls	see below	Which penalty to apply.
actor_rollout_ref.actor.efficiency_reward_coef	float (default 10.0)	The efficiency coefficient λ, overridable via EFFICIENCY_REWARD_COEF.
env.env_name	alfworld/AlfredTWEnv, tw_cooking/Cooking, countdown_stepwise	Also selects the reward manager.
env.max_steps	int	Step budget T, the SegCost denominator.

Values for efficiency_cls:

• pure_round_length_penalty — the singleton-only fixed-code baseline
• skill_compression_BPE_idio_value — ReuseRL-SegCost (batch + global buffer)
• skill_compression_BPE_idio_value_noGlobalBuffer — dictionary fit per-batch only

ALFWorld uses different spellings for the last two. Its reward manager (efficiency.py) predates the others and names them skill_compression_BPE_noParser_idio_value and skill_compression_BPE_noParser_idio_value_noGlobalBuffer; passing the un-prefixed names raises ValueError. The ALFWorld scripts already set the right value, and take it as their second positional argument (bash run_alfworld_compression_BPE_idio_value.sh vllm <efficiency_cls>).

The global success buffer size is set per environment via environment variable (default 256):

export SKILL_COMPRESSION_BPE_GLOBAL_BUFFER_SIZE=1024 # For AlfWorld
export TW_COOKING_SKILL_COMPRESSION_BPE_BUFFER_SIZE=1024
export COUNTDOWN_STEPWISE_SKILL_COMPRESSION_BPE_BUFFER_SIZE=1024

Evaluation

Evaluation is two steps: merge the sharded FSDP checkpoint into a HuggingFace model directory, then run the environment's harness on it.

python3 scripts/model_merger.py merge --backend fsdp \
    --local_dir  checkpoints/<project>/<experiment_name>/global_step_300/actor \
    --target_dir checkpoints/<project>/<experiment_name>/hf_model_300

The commands below are the ones used for the paper's runs.

TextWorld-Cooking — Pass@1 over 1000 held-out val games:

CUDA_VISIBLE_DEVICES=0 python3 vllm_eval_tw_cooking.py \
    --policy-model checkpoints/verl_agent_tw_cooking/<experiment_name>/hf_model_300 \
    --disable-atomic-skill-judge --env-num 256 \
    --output results/tw_cooking_step300.json --seed 42

Countdown-Stepwise — Pass@1 over 1024 held-out val problems:

CUDA_VISIBLE_DEVICES=0 python3 vllm_eval_countdown_stepwise.py \
    --policy-model checkpoints/verl_agent_countdown_stepwise/<experiment_name>/hf_model_300 \
    --num-eval-games 1024 --env-num 128 \
    --output results/countdown_stepwise_step300.json

--num-eval-games 1024 is required to reproduce the reported numbers — the default is only 128.

ALFWorld — self-consistency SC@7, in-distribution and out-of-distribution:

# In-distribution
CUDA_VISIBLE_DEVICES=0 python3 vllm_eval_alfworld.py \
    --policy-model checkpoints/verl_agent_alfworld/<experiment_name>/hf_model_300 \
    --env-num 140 --history-length 10 \
    --use-self-consistency-eval --self-consistency-num 7 \
    --disable-atomic-skill-judge \
    --output results/alfworld_iid_step300.json

# Out-of-distribution: same command plus
    --eval-dataset eval_out_of_distribution

To validate the greedy BPE merge schedule against the brute-force MDL optimum:

python sanity_check_bpe_dictionary_vs_bruteforce.py

License

Apache License 2.0 (see LICENSE). Files authored for ReuseRL carry a Copyright 2026 The ReuseRL Authors notice; everything inherited from SkillRL, verl-agent and verl (Bytedance, Apache 2.0) retains its original license header. See Notice.txt for the full attribution chain.

Acknowledgement

Built on SkillRL, verl-agent and verl. We also thank ALFWorld, TextWorld, reasoning-gym, and Qwen. Please cite those works as well when using this codebase.