Starter code for the Aaltonomous Humanoid challenge: make Väiski, a Unitree G1 with two Dex3 hands, sort coloured objects into bins fully autonomously. Develop in sim, hand in a Docker policy server, the best teams run on the real robot in the finals.
55 s of simulation footage from unedited runs of our policy: grasps, places, and an unseen frying pan lifted by its
handle. Download the video.
The submitted code is tagged v1.0-submission.
Scoring drives the design. A wrong drop costs -1 and no drop costs 0, so every stage would rather hold or abstain than
guess. On the known tasks no learned model is on the control path. A local vision-language model (VLM) only steps in
for what the classical stack cannot name: an unseen object or an open instruction. Details and file references are in
docs/ARCHITECTURE.md. The finals overlay is
aaltonomous/baselines/classical/overlays/finals_r25_gen.yaml.
flowchart TD
IN["head RGB-D + joint state + instruction"] --> AD{"sim or real?<br/>profile_detect.py<br/>depth holes + frame-to-frame depth flicker"}
AD --> P["Perception<br/>HSV + depth, top-face shape, bin vote<br/>novel_objects.py: geometry of unknown objects"]
IN --> R["Instruction resolver<br/>normalize.py -> frozen regex resolver<br/>strict: unparsed or ambiguous = abstain"]
P --> D["Decision<br/>known bin, or no pick"]
R --> D
D -- "known task" --> PL
D -- "unseen noun or open instruction" --> BR["Brain: local Qwen3-VL-8B on llama.cpp<br/>closed set of tools (find, pick_place, skip, done ...)<br/>emits data, never joints"]
BR -- "every step dry-run by the classical planner" --> PL["Planner + IK<br/>Pinocchio on the G1 URDF, time-scaled trajectories"]
PL --> SH["Safety shell<br/>NaN guard, output limiter, hold last target on a deadline miss"]
SH --> SF["Organisers' SafetyFilter<br/>per-tick velocity bound, workspace box, hold / stop"]
SF --> ROBOT["Unitree G1 + Dex3, or the simulator"]
- The brain is gated. With
brain.known_task_hard_off, a known task (the resolver covers the sentence and no noun is unknown) makes no brain call at all. The action stream is then identical to running with the brain disabled (TestKnownTaskHardOffintest_universal.py). The brain may never change a bin the classical stack already knows. - It never blocks the 30 Hz loop. Brain requests run on a worker thread, and the server's health is re-probed at
every reset. If the server is down on an open instruction, the arms go home and the policy re-probes every 3 s for
up to 60 s, then carries on or stops (
test_brain_ready.py;act()p95 stays under 30 ms during the wait). - Sim or real is detected from the first frames, not from a flag. A simulator render has no invalid depth pixels
and is bit-identical between frames of a still scene. The D435i always has a shadow band of holes and re-measures
every pixel. The policy calls a run real if either cue fires, and sim only if both are quiet. Real-only fixes are
then switched on, and
profile_auto.sim_overridesrestores the sim values.
-
Unseen objects are understood from their shape, not their name.
novel_objects.pytakes an above-table depth blob that matches no known colour and looks for an elongated part sticking out of the body: a handle. The grasp goes across the handle's width. A part narrower than 15 mm is rejected, so a screwdriver is held by its handle, not its shaft. For a frying pan (a 20 cm disc with a 12 x 3 cm handle),test_handle_grasp.pychecks 9 cases:- the handle is grasped across its width, and the planner closes on it;
- a plain disc is not taken for a handle;
- a pan cut off by the image edge keeps its visible handle;
- a wok is grounded by the VLM and grasped by its handle.
No word in the instruction picks the grasp point.
-
Unknown nouns. "Put the pan in the left bin" names nothing the colour resolver knows. The VLM grounds the noun in the head image (
brain/find.py). The result becomes an ordinary detection, and the classical planner plans the grasp. -
Unseen wordings. We fed the resolver 60 phrasings it was not written for: polite, sloppy, negated, contradictory and nonsense. It resolved 37 as intended and abstained on 19. Three of the other four map orange and purple to red and blue, which is by design: orange and purple are the real objects that stand in for the sim colours. The fourth drops the unknown noun in "the stapler and the red ones" and keeps "red goes left". Of 10 negations, 8 abstain ("don't put red on the left") and 2 "except" forms resolve: "everything except purple goes right" leaves the purple objects where they are and moves the rest. With the brain on, an abstained open sentence goes to the brain planner, whose plan is checked step by step.
-
Abstain rather than guess. Strict mapping, the red/yellow hue guard, the lid guard and the novel-object checks all end in "no pick" when unsure.
All three fixes are real-only. The sim/real detector switches them on, and the sim keeps its tested values.
- Table height. In the organisers' debug log of our real run, the robot saw no objects at all. The committed table
height was 3.2 cm off the plane the head camera measured, so the height gate rejected every object. The fix:
- The policy now measures the table on every run: a dominant-plane fit on head depth within ±0.30 m, gated by tilt,
inlier count and RMS (
calib.plane.robust_height). - The older fit is kept only when it agrees with that measurement within 1 cm (
calib.plane.robust_agree_m). - The nominal height is used only when nothing could be measured.
- Detections handed to the planner are mapped into the calibration camera's frame (
perception.plan_frame).
- The policy now measures the table on every run: a dominant-plane fit on head depth within ±0.30 m, gated by tilt,
inlier count and RMS (
- Arm sag. The runner's arm PD has no gravity term (kp 80 at the shoulders and elbow, 40 at the wrists), so the
arm sags by roughly gravity torque / kp.
- We send the URDF gravity torque in the organisers'
arm_tau_fffield. - On top of that,
sag_comp.pyadds a bounded position offset, q_cmd = q_plan + kappa * tau_g / kp. - kappa is estimated online from still ticks. It comes out as 0 when the runner already compensates, so the output is then bit-identical to running without the fix.
- The offset stays below the runner's 0.12 rad clip, its rate is capped at 0.5 rad/s, and it is applied before the output limiter.
- We send the URDF gravity torque in the organisers'
- Reach clamp. The IK reaches a drop point at x = 0.52 m only from a release height of 0.25 m or more. Drop points are clamped to x ≤ 0.52 m with release z ≥ 0.25 m, for voted bins and for fallback bins. An unvoted bin is centred so that its first drop slot sits just inside the near rim.
- Drills are not graspable. The handle finder takes the chuck for a handle, because it is about as long as a pan
handle. The real grip sticks out at right angles to the body. Telling the two apart needs a cue we do not have yet. Hammers, pliers,
wrenches, mug handles, knives and spatulas are recorded as known failures in
test_tool_grasp.py. Of its 23 cases, 8 pass: the screwdriver at both angles, fruit, and hammer and pliers detection. - Lids abstain. For
insert_shapes, the lid holes are 3 to 10 cm beyond the arm's IK reach, and a release over the lid bin scores -1.policy.insert_guardtherefore picks nothing: 0 points by design. - The sim/real thresholds were set without real depth data. They come from sim renders and the sensor's known behaviour, because we had no recorded real depth to tune them on.
- The brain has not been tested on the robot. End-to-end runs with the real llama.cpp server take place on a GPU simulator box. The unit tests cover only a server that is down or late.
In the finals the policy scored 0 points in both real runs (sort_fixed, seed 0, 15 Nov 2026). In simulation the same
code scored 478 of 527 objects over 128 episodes (64 seeds x 2 tasks), with 2 objects placed in the wrong bin. This
section records what happened on the robot, the cause, and what we changed afterwards. The brain never ran on the
real robot.
The bins at the venue were wrapped in red-orange paper. In the simulator they are plain orange-yellow cardboard. In both runs the colour gate kept 8 of 9 candidates, the policy picked "red", and the right arm went to a grasp point on the right bin. The hand closed on the lower edge of the bin's front wall and dragged the bin out of position. The held check passed, the release check failed twice, and the policy stopped at about 39 s of a 120 s run. No object was moved.
- Gate row cutoff. The image gate (
gate_below_bins) only counted bin runs starting above row 216 (first_run_top_frac0.45 of 480). The right bin's front face starts at rows 215 to 222, so the gate put its bottom edge near row 89 instead of about 277 and let the wall through. - Defences skipped while the gate is on. With the gate active, the 3D footprint cut, the
in_binreject and the colour gate's workspace and bin rules were bypassed. The gate was meant as an extra layer and acted as a switch for the others. - Blob splitting.
split_blobscut the wall strip into cube-sized pieces before thetoo_bigcheck ran. Colour could not separate them: robust red covers hue 0 to 16, the paper sits at 4 to 8 and the orange objects at 11 to 12. - Hard-coded mapping.
policy_server.pyalways injected the venue colour-to-bin mapping forsort_fixed, so the instruction-first path was never reached. The instruction said red goes left, the mapping said right. bin.votedisagreed in both runs (confidence 0.3), so live bin positions were not trusted. The real bins were 8 to 15 cm from the nominal footprint. The logs do not show why the vote disagreed.- Held detector and early stop. Finger sum was 0.266 at close and 0.046 at lift against a 0.045 threshold, so a bin wall counted as held. After two failed release checks the policy stopped picking with 80 s left.
The hackathon hotfix (bin_size_guard) did not help: the surviving strip was below its size thresholds.
tools/venue_replay.py runs perception on 11 labelled frames from the two real runs (RGB only, no depth, no GPU) and
counts accepted colour candidates that lie on a bin.
| Config | Candidates on a bin (11 frames) | Objects found |
|---|---|---|
| As run on the robot | 20 | 55/55 |
As run + hotfix bin_size_guard |
20 | 55/55 |
| After the fix | 2 | 55/55 |
The two remaining candidates are the robot's own fingers, which the arm mask removes on the robot. On the frame the robot used to choose its target, the old code ranks the right bin's wall first as "red"; the fix ranks the orange cube first.
We built a venue-twin scene (aaltonomous/sim/VENUE_TWIN.md: paper-coloured bin walls, brown bin interiors, venue
table and object colours sampled from the run video) and ran an A/B in Isaac Sim on sort_fixed, seeds 1000 to 1003:
the submitted code and the fix, each on the venue twin and on the nominal scene.
| Arm | Scene | Objects correct / total | Clips | Rate |
|---|---|---|---|---|
| submitted | nominal | 0 / 15 | 0 | ~30 Hz |
| submitted | venue twin | 0 / 15 | 0 | ~30 Hz |
| fix | nominal | 0 / 15 | 0 | ~30 Hz |
| fix | venue twin | 0 / 15 | 0 | ~30 Hz |
All four arms scored zero. The images were built with the real profile forced (PROFILE_MODE=real), and that profile
does not sort in simulation at all, so this run neither reproduces the failure nor validates the fix. No
[grasp_diag] lift values were captured. The only validation of the fix is the offline replay above.
The fix turns on bin_image_guard (rejects dark-warm components of bin size in the image), raises
first_run_top_frac to 0.55, keeps the in_bin defence active while the gate is on, and makes the sort_fixed
mapping instruction-first. bin.vote now logs its voters, and bin.vote.tape_body_reject ignores paper-wrapped bin
bodies (7.9k to 19k px in the replay frames) when looking for tape. Unit tests cover each piece.
Not covered:
- Not validated on real depth or on the robot.
grasp.wrong_object_recovery(a stricter post-lift finger check and a bounded re-look after a failed release) is implemented but off: it needs a lift finger sum of at least 0.09, and only the venue bin wall (0.046 and 0.088) has ever been measured, never a genuine grasp.- The reason
bin.votedisagreed is unknown, and the organisers' colour-to-bin mapping is unconfirmed.
- Colour thresholds tuned on sim materials did not transfer: bin paper, orange objects and the red range overlapped.
- A guard that disables the footprint,
in_binandtoo_bigchecks turns three independent layers into one. - Eleven frames and an RGB-only replay reproduced the failure offline. We had no venue-coloured data in the loop before the finals.
- The abstain rules in the resolver never fired because perception handed over a confident wrong candidate, and the held detector then confirmed it.
- Two failed release checks ended a 120 s run at 39 s. A bounded retry would have given a second attempt.
To run it without building, pull the submitted image. It is a byte-identical, digest-pinned copy published by
.github/workflows/publish-image.yml:
docker pull ghcr.io/cufelix/hackjunction:finalThe final image (ghcr.io/dymyt-ry/pp9gi2f:final) is built from this repo in two steps on an x86_64 host with
Docker and BuildKit. The brain stage compiles llama.cpp with CUDA, so it needs a few minutes and about 30 GB of disk.
# 1. Stage what is not in git: the G1 URDF, llama.cpp b12110 source, Qwen3-VL-8B-Instruct Q8_0 + mmproj
# (about 9.5 GB, all sha256-checked). This is the only step that needs internet.
bash tools/brain/stage_brain.sh
# 2. Policy base, then the brain layer on top (llama-server and the model inside the image, no downloads at start)
docker build -f tools/submission.Dockerfile --build-arg OVERLAY=finals_r25_gen --build-arg PROFILE_MODE=auto \
-t hackjunction/policy:final-base .
docker build -f tools/brain/Dockerfile.policy_brain --build-arg POLICY_IMAGE=hackjunction/policy:final-base \
--build-arg BRAIN_ENABLED=1 --build-arg BRAIN_OVERLAY=finals_r25_gen --build-arg PROFILE_MODE=auto \
-t hackjunction/policy:final .
# 3. Check it: server up within 120 s, brain answers, replay above 27 Hz
bash tools/brain/test_brain_image.sh hackjunction/policy:final 8371
python -m aaltonomous.deploy.replay --synthetic 100 --container hackjunction/policy:finalPROFILE_MODE=autolets the policy tell sim from the real robot at start-up.simorrealforces one profile.BRAIN_ENABLED=0builds the same image with the brain off. Step 2's first command alone gives the CPU-only image.- Both Dockerfiles run the release checks (
tools/flags.py,tools/check_release_flags.py) during the build, so a build with a wrong overlay or flag set fails.
| Start here | |
|---|---|
| Challenge brief | Task, rungs, timeline, what to read next |
| Quickstart | Fresh cloud GPU instance → baseline running in sim (~15 min) |
| Rules and scoring | Success rule, points, finalist selection |
| Interface | Observations, actions, joint order, limits, cameras |
| Submission | Docker policy-server contract (websocket + msgpack, port 8000) |
| Starter paths | Classical, VLM, IL/VLA, RL, real2sim |
| FAQ |
| Path | What |
|---|---|
aaltonomous/interface/ |
AaltoEnv, joint spec, the safety filter every action goes through |
aaltonomous/deploy/, aaltonomous/docker/ |
Policy-server template and submission Dockerfiles |
aaltonomous/sim/ |
Isaac Lab task Aalto-Sort-G129-Dex3-Joint, evaluator, oracle, training DR config |
aaltonomous/scene/ |
Scene geometry (scene.yaml), success checker, printable assets |
aaltonomous/baselines/classical/ |
Classical perception + IK baseline |
aaltonomous/data/ |
Real-demo recorder and LeRobot / GR00T converters |
aaltonomous/cloud/ |
Instance setup, aalto CLI, sim image Dockerfile |
Unit tests (no GPU needed): python -m unittest aaltonomous/interface/test_safety.py aaltonomous/interface/test_env.py aaltonomous/scene/test_scene.py.
Upstream dependencies (unitree_sim_isaaclab, xr_teleoperate for the G1 URDF, Isaac Lab) are cloned into
third_party/ by aaltonomous/cloud/setup_instance.sh; the sim Docker image already contains them.
Organisers: DigiTwin, Aalto University. This repo is synced from our internal repo; please open issues or ask in the challenge channel rather than sending PRs during the event.


