Build your own robot-use agents. Give an LLM a goal, camera views and robot tools, and let it observe, act and adjust. We're open sourcing the parts of our stack that make that loop easy to build.
- Rust server
- Python SDK and agent examples
- TypeScript SDK and agent CLI example
- Rust SDK and command-line station client
- LiveKit camera preview
For a physical SO-101, follow the source quickstart
to calibrate the arm with so101 setup, start RUA with --profile auto and
run your first Python agent.
For learned pose control, configure a compatible ONNX policy.
Start an agent project with uv:
uv init my-robot-agent
cd my-robot-agent
uv add "hebbian-rua[openai,anthropic,gemini]"Already have a uv project? Start with uv add. Choose just the extras you need.
With the published RUA 0.1.2 wheel, use an
explicit profile and LeRobot calibration to connect a physical arm.
Prebuilt wheels support Linux x86-64 (glibc 2.39+) and macOS 14+ on Apple Silicon. On Windows, use WSL2.
See getting started and the Python examples to try it, and the example applications for example toy projects.
Try the SO-101 without hardware using the MuJoCo quickstart, or connect your own simulator. To watch your cameras, follow the LiveKit web preview setup.
RUA supplies robot tools, camera observations and execution. You bring the model and the loop.
Here's an example agent loop. The complete example includes the connection, lease and arm setup, plus loops for Claude and Gemini:
from hebbian_rua.client.llm.openai import OpenAIRobotTools
robot_tools = OpenAIRobotTools(robot)
messages = [{"role": "user", "content": "Wave hello"}]
for _ in range(20):
observation = await robot_tools.observe(wait_for_recovery=True)
messages.append(observation.message)
response = await client.responses.create(
model=model,
instructions=instructions,
tools=robot_tools.tools,
input=messages,
)
turn = await robot_tools.execute(response, expected_revisions=observation.revisions)
messages.extend(turn.messages)
if turn.should_stop or not turn.results:
breakThe model chooses tools, RUA executes them, and fresh camera frames and robot
state come back for the next turn. You can change the prompt, model and loop.
Use AnthropicRobotTools from hebbian_rua.client.llm.anthropic or GeminiRobotTools from
hebbian_rua.client.llm.gemini with the corresponding SDK. See the
Python examples.
Choose what reaches the model with output formatters.
RUA compiles action sequences automatically. The Robot Use Compiler checks and translates the requested actions. The Robot Use Runtime prepares each step against live robot state and admits it for execution. Advanced users can compile and inspect before running. See how execution works.
Hardware setup, camera tools and recording utilities have their own guides: SO-101, LiveKit Cameras and MCAP Tools. RUA handles their integration with your robot and agent.
Some reading and projects around robot-use agents:
- Robot-Use Agents — Phillip Isola's overview of the idea.
- Anthropic's robotics evaluations and Fable 5.2 robot-use examples.
- Google's Gemini 4 Argon for Robotics — robot control with a general-purpose model.
- NVIDIA's Astranomy — studying Astra as a robot policy.
- PEAR — training reasoning models for robot use with reinforcement learning in simulation.
- ASPIRE — agentic robot skill discovery.
- ENPIRE — robot policy improvement through physical experiments.
- RAX — agents learning robot programs through online exploration.
- RoboColosseum — real-world robot evaluation.
See Documentation and the development reference for the station, integration tests, and release verification.
See contributing. Code is MIT; models and dependencies keep their own licenses. See third-party notices.