Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 93 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,99 @@ All three fixes are real-only. The sim/real detector switches them on, and the s
- **The brain has not been tested on the robot.** End-to-end runs with the real llama.cpp server take place on a GPU
simulator box. The unit tests cover only a server that is down or late.

## Real-robot postmortem

In the finals the policy scored 0 points in both real runs (`sort_fixed`, seed 0, 15 Nov 2026). In simulation the same
code scored 478 of 527 objects over 128 episodes (64 seeds x 2 tasks), with 2 objects placed in the wrong bin. This
section records what happened on the robot, the cause, and what we changed afterwards. The brain never ran on the
real robot.

### What happened

The bins at the venue were wrapped in red-orange paper. In the simulator they are plain orange-yellow cardboard. In both
runs the colour gate kept 8 of 9 candidates, the policy picked "red", and the right arm went to a grasp point on the
right bin. The hand closed on the lower edge of the bin's front wall and dragged the bin out of position. The held
check passed, the release check failed twice, and the policy stopped at about 39 s of a 120 s run. No object was moved.

![Head camera at t = 11 s. Green marks the pixels the colour gate still accepted as red: three of the five objects, and a block on the right bin's front wall.](docs/postmortem/gate_overlay.jpg)

![Run 1, t = 11 to 35 s. The right hand reaches the right bin's front wall and the bin slides out of position.](docs/postmortem/bin_dragged_sequence.jpg)

### Cause

1. **Gate row cutoff.** The image gate (`gate_below_bins`) only counted bin runs starting above row 216
(`first_run_top_frac` 0.45 of 480). The right bin's front face starts at rows 215 to 222, so the gate put its
bottom edge near row 89 instead of about 277 and let the wall through.
2. **Defences skipped while the gate is on.** With the gate active, the 3D footprint cut, the `in_bin` reject and the
colour gate's workspace and bin rules were bypassed. The gate was meant as an extra layer and acted as a switch
for the others.
3. **Blob splitting.** `split_blobs` cut the wall strip into cube-sized pieces before the `too_big` check ran. Colour
could not separate them: robust red covers hue 0 to 16, the paper sits at 4 to 8 and the orange objects at 11 to 12.
4. **Hard-coded mapping.** `policy_server.py` always injected the venue colour-to-bin mapping for `sort_fixed`, so the
instruction-first path was never reached. The instruction said red goes left, the mapping said right.
5. **`bin.vote` disagreed** in both runs (confidence 0.3), so live bin positions were not trusted. The real bins were
8 to 15 cm from the nominal footprint. The logs do not show why the vote disagreed.
6. **Held detector and early stop.** Finger sum was 0.266 at close and 0.046 at lift against a 0.045 threshold, so a
bin wall counted as held. After two failed release checks the policy stopped picking with 80 s left.

The hackathon hotfix (`bin_size_guard`) did not help: the surviving strip was below its size thresholds.

### Offline replay

`tools/venue_replay.py` runs perception on 11 labelled frames from the two real runs (RGB only, no depth, no GPU) and
counts accepted colour candidates that lie on a bin.

| Config | Candidates on a bin (11 frames) | Objects found |
|---|---|---|
| As run on the robot | 20 | 55/55 |
| As run + hotfix `bin_size_guard` | 20 | 55/55 |
| After the fix | 2 | 55/55 |

The two remaining candidates are the robot's own fingers, which the arm mask removes on the robot. On the frame the
robot used to choose its target, the old code ranks the right bin's wall first as "red"; the fix ranks the orange
cube first.

### Simulation check: inconclusive

We built a venue-twin scene (`aaltonomous/sim/VENUE_TWIN.md`: paper-coloured bin walls, brown bin interiors, venue
table and object colours sampled from the run video) and ran an A/B in Isaac Sim on `sort_fixed`, seeds 1000 to 1003:
the submitted code and the fix, each on the venue twin and on the nominal scene.

| Arm | Scene | Objects correct / total | Clips | Rate |
|---|---|---|---|---|
| submitted | nominal | 0 / 15 | 0 | ~30 Hz |
| submitted | venue twin | 0 / 15 | 0 | ~30 Hz |
| fix | nominal | 0 / 15 | 0 | ~30 Hz |
| fix | venue twin | 0 / 15 | 0 | ~30 Hz |

All four arms scored zero. The images were built with the real profile forced (`PROFILE_MODE=real`), and that profile
does not sort in simulation at all, so this run neither reproduces the failure nor validates the fix. No
`[grasp_diag]` lift values were captured. The only validation of the fix is the offline replay above.

### The fix and what it does not cover

The fix turns on `bin_image_guard` (rejects dark-warm components of bin size in the image), raises
`first_run_top_frac` to 0.55, keeps the `in_bin` defence active while the gate is on, and makes the `sort_fixed`
mapping instruction-first. `bin.vote` now logs its voters, and `bin.vote.tape_body_reject` ignores paper-wrapped bin
bodies (7.9k to 19k px in the replay frames) when looking for tape. Unit tests cover each piece.

Not covered:
- Not validated on real depth or on the robot.
- `grasp.wrong_object_recovery` (a stricter post-lift finger check and a bounded re-look after a failed release) is
implemented but **off**: it needs a lift finger sum of at least 0.09, and only the venue bin wall (0.046 and 0.088)
has ever been measured, never a genuine grasp.
- The reason `bin.vote` disagreed is unknown, and the organisers' colour-to-bin mapping is unconfirmed.

### Lessons

- Colour thresholds tuned on sim materials did not transfer: bin paper, orange objects and the red range overlapped.
- A guard that disables the footprint, `in_bin` and `too_big` checks turns three independent layers into one.
- Eleven frames and an RGB-only replay reproduced the failure offline. We had no venue-coloured data in the loop
before the finals.
- The abstain rules in the resolver never fired because perception handed over a confident wrong candidate, and the
held detector then confirmed it.
- Two failed release checks ended a 120 s run at 39 s. A bounded retry would have given a second attempt.

## Build the submitted image

To run it without building, pull the submitted image. It is a byte-identical, digest-pinned copy published by
Expand Down
60 changes: 52 additions & 8 deletions aaltonomous/baselines/classical/bin_vote.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,20 +98,23 @@ def _box_edge_dist(Q, ctr, half):
return np.abs(sd)


def rim_votes(perception, depth, centers, v, transform_snapshot=None):
def rim_votes(perception, depth, centers, v, transform_snapshot=None, diag=None):
"""{bin: xy|None}: per-bin fit of the outer+inner rim rectangles to elevated depth points, +-search_m window."""
sc, cfg = perception.scene, perception.cfg
c = cfg.get("self_calibration") or {}
b = sc.cfg["bins"]
out = {n: None for n in centers}
diag = {} if diag is None else diag
if depth is None or np.ndim(depth) != 2:
diag["why"] = "no depth"
return out
clean_depth = np.nan_to_num(np.asarray(depth, float), nan=0.0)
if transform_snapshot is None:
P, coef, why = perception._counter_plane(clean_depth, c)
else:
P, coef, why = perception._counter_plane(clean_depth, c, transform_snapshot=transform_snapshot)
if why:
diag["why"] = f"plane: {why}"
return out
height = P[:, 2] - (np.c_[P[:, :2], np.ones(len(P))] @ coef)
keep = (height >= float(c.get("min_wall_height_m", 0.008))) & (height <= float(b["wall_height"]) + 0.015)
Expand All @@ -122,6 +125,7 @@ def rim_votes(perception, depth, centers, v, transform_snapshot=None):
trunc = 0.015
for n, Q in _nearest_bin_points(P, centers, outer, search).items():
if len(Q) < int(v["min_rim_points"]):
diag[n] = f"{len(Q)} pts < min {int(v['min_rim_points'])}"
continue
Q = Q[np.linspace(0, len(Q) - 1, min(len(Q), 1500)).astype(int)]
signed = bool(v.get("signed_rim_fit", False))
Expand All @@ -142,7 +146,10 @@ def search_grid(ctr, span, step):
off = search_grid(off, 0.002, 0.0005)
fit = Q - off if signed else Q + off
d = np.minimum(_box_edge_dist(fit, centers[n], outer), _box_edge_dist(fit, centers[n], inner))
if float(np.mean(d <= 0.006)) < 0.35:
inl = float(np.mean(d <= 0.006))
diag[n] = f"{len(Q)} pts, inlier={inl:.2f}, off=({1000 * off[0]:+.0f},{1000 * off[1]:+.0f})mm"
if inl < 0.35:
diag[n] += " REJECTED (inlier < 0.35)"
continue
out[n] = np.asarray(centers[n], float) + off
return out
Expand Down Expand Up @@ -178,25 +185,37 @@ def _bin_blob_vote(blobs, elong):
return _blob_vote(blobs[0], elong)


def tape_votes(perception, rgb, depth, centers, v, transform_snapshot=None):
def tape_votes(perception, rgb, depth, centers, v, transform_snapshot=None, diag=None):
"""{bin: xy|None}: per orange-tape blob near the bin's rim (8-connected in the image), lifted via depth or via the
rim-height plane where depth is missing; the blob votes its centroid, or its inner end when elongated (see
_blob_vote). Ambiguous or missing -> None. # ponytail: no per-edge tape layout model; upgrade once the real
tape layout is known."""
import cv2
sc = perception.scene
out = {n: None for n in centers}
diag = {} if diag is None else diag
if rgb is None or np.ndim(rgb) != 3:
diag["why"] = "no rgb"
return out
hsv = cv2.cvtColor(np.ascontiguousarray(rgb), cv2.COLOR_RGB2HSV)
lo, hi = v["tape_hsv"]
mask = cv2.inRange(hsv, np.array(lo, np.uint8), np.array(hi, np.uint8))
mask[int(mask.shape[0] * TAPE_V_MAX_FRAC):] = 0
vv, uu = np.nonzero(mask)
if len(vv) < int(v["min_tape_px"]):
diag["why"] = f"{len(vv)} tape-coloured px < min"
return out
labels = cv2.connectedComponents(mask, connectivity=8)[1]
lab = labels[vv, uu]
sizes = np.bincount(lab)
diag["blob_px"] = sorted((int(x) for x in sizes[1:] if x >= int(v["min_tape_px"])), reverse=True)[:8]
max_px = int(v.get("tape_max_px", 0) or 0) if v.get("tape_body_reject", False) else 0
if max_px > 0: # venue: the red-orange paper-wrapped bin body (8-19k px) matches tape_hsv; real tape is a thin strip
big = sizes[lab] > max_px
diag["rejected_body_px"] = sorted((int(x) for x in np.unique(sizes[lab[big]])), reverse=True)
vv, uu, lab = vv[~big], uu[~big], lab[~big]
if len(vv) < int(v["min_tape_px"]):
return out
rim_z = sc.counter_z + float(sc.cfg["bins"]["wall_height"])
d = np.zeros(len(vv))
if depth is not None and np.ndim(depth) == 2:
Expand Down Expand Up @@ -241,6 +260,7 @@ def tape_votes(perception, rgb, depth, centers, v, transform_snapshot=None):
per_bin[names[k]].append(B)
for n in names:
out[n] = _bin_blob_vote(per_bin[n], float(v["tape_elong_aspect"]))
diag["blobs_per_bin"] = {n: [len(b) for b in per_bin[n]] for n in names}
return out


Expand All @@ -249,26 +269,50 @@ def _med(frames, n):
return np.median(xs, axis=0) if xs else None


def _fmt(x):
return "none" if x is None else f"({x[0]:.3f},{x[1]:.3f})"


def _log_voters(log, names, priors, rim, tape, diags):
"""[bin.vote.voters]: every voter's estimate (pelvis xy, m) + distance to the prior + why a voter was missing, so a
disagreement (conf 0.3) is explained by the next real run's log. Never raises."""
try:
for n in names:
parts = []
for k, est in (("rim", rim[n]), ("tape", tape[n])):
dist = "" if est is None else f" d_prior={1000 * float(np.linalg.norm(np.asarray(est) - priors[n])):.0f}mm"
parts.append(f"{k}={_fmt(est)}{dist}")
log(f"[bin.vote.voters] {n}: prior={_fmt(priors[n])} " + " ".join(parts))
rd, td = diags[0]
log(f"[bin.vote.voters] frame0 rim: {dict(rd)}")
log(f"[bin.vote.voters] frame0 tape: {dict(td)}")
except Exception as exc:
log(f"[bin.vote.voters] log failed ({type(exc).__name__})")


def vote_bins(perception, planner, frames, cfg, log=print):
"""frames: [{"head_rgb", "head_depth"}]. -> (shift, drop_extra, conf). Never raises (prior on any error)."""
v = cfg["bin"]["vote"]
names = list(planner.scene.bin_names)
priors = {n: np.asarray(planner.bin_xy(n), float) for n in names}
try:
rims, tapes = [], []
rims, tapes, diags = [], [], []
for f in frames:
rd, td = {}, {}
diags.append((rd, td))
if v.get("frame_transform_snapshot", False):
T = f["transform_snapshot"]
rims.append(rim_votes(perception, f.get("head_depth"), priors, v, T))
tapes.append(tape_votes(perception, f.get("head_rgb"), f.get("head_depth"), priors, v, T))
rims.append(rim_votes(perception, f.get("head_depth"), priors, v, T, diag=rd))
tapes.append(tape_votes(perception, f.get("head_rgb"), f.get("head_depth"), priors, v, T, diag=td))
else:
frame = f.get("transform") or perception.frame_transform()
with perception.use_frame_transform(frame):
rims.append(rim_votes(perception, f.get("head_depth"), priors, v))
tapes.append(tape_votes(perception, f.get("head_rgb"), f.get("head_depth"), priors, v))
rims.append(rim_votes(perception, f.get("head_depth"), priors, v, diag=rd))
tapes.append(tape_votes(perception, f.get("head_rgb"), f.get("head_depth"), priors, v, diag=td))
rim = {n: _med(rims, n) for n in names}
tape = {n: _med(tapes, n) for n in names}
shift, extra, info = resolve(priors, rim, tape, v)
_log_voters(log, names, priors, rim, tape, diags)
except Exception as exc: # optional feature: never terminate the control loop
log(f"[bin.vote] failed ({type(exc).__name__}: {exc}); prior positions")
return {}, {n: float(v["low_conf_extra_m"]) for n in names}, {n: CONF_PRIOR for n in names}
Expand Down
9 changes: 7 additions & 2 deletions aaltonomous/baselines/classical/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,9 @@
"perception.prism_speckle_floor": ("prism floor", False, "color_gate speckle floor: orange/red blobs with a triangular outline (minEnclosingTriangle fill) use prism_speckle_floor_params.area_frac instead of color_gate.area_frac"),
"perception.real_border_partial": ("real perc", False, "real only: color_gate speckle floor uses real_border_partial_params.area_frac for blobs cut by the left/right/bottom image border"),
"perception.bin_bottom_first_run": ("real perc", False, "real only: gate_below_bins observed bin-bottom row = first vertical warm run per column (a shadowed orange object on the mat is not bin mass)"),
"perception.bin_size_guard": ("real perc", False, "real only (hotfix): a colour blob that is bin-sized (area / 3D footprint) or bin-tall is rejected before split_blobs (bin_size_guard_params)"),
"perception.bin_image_guard": ("real perc", False, "real only (venue replay): a colour blob that lies on a bin-sized dark-warm image component (bin wall in shade, V < 150) is rejected before split_blobs; needs no depth (bin_image_guard_params). Also keeps the nominal in_bin reject active while the image gate_below_bins is on"),
"perception.gate_below_bins_params.first_run_top_frac": ("real perc", 0.45, "gate_below_bins + bin_bottom_first_run: a warm run counts as bin wall only if it starts in the top this-fraction of the frame (venue: 0.45 cut the shaded bin wall; finals_r25_gen 0.55)"),
"perception.fk_colour_rescue": ("real perc", False, "real only: colour pixels inside the FK arm mask stay candidates; detections overlapping the mask must sit at object height (fk_colour_rescue_params.z_above_counter_m)"),
"perception.real_merge_split": ("real perc", False, "real only: split_blob tries merged same-colour pairs from real_merge_split_params.area_factor x expected area (default 0.9 instead of 1.5)"),
"perception.gate_below_bins": ("gate fix", False, "when live bins are untrusted, gate candidates below observed bin mass in image space"),
Expand Down Expand Up @@ -266,6 +269,8 @@
"bin.vote.max_shift_m": ("bin vote", 0.10, "a bin vote result further than this from the prior is discarded (prior kept); real venue bins measure +8..+15 cm vs nominal"),
"bin.vote.enabled": ("bin vote", False, "per-bin position from 3 votes (depth rim, tape, prior) over the first frames"),
"bin.vote.signed_rim_fit": ("C3", False, "fit physical rim displacement with its signed +x/+y direction"),
"bin.vote.tape_body_reject": ("venue", False, "bin.vote: a tape-coloured blob larger than bin.vote.tape_max_px is a paper-wrapped bin body (venue: 8-19k px), not tape: it casts no tape vote (else the body, lifted to rim height, disagrees with prior and rim -> conf 0.3)"),
"grasp.wrong_object_recovery": ("venue", False, "real: (a) the post-lift finger read must clear grasp.wrong_object_recovery_params.lift_margin_mult x threshold (and optionally lift_min_ratio x the close read) else open + retreat + blacklist the spot (bin wall mistaken for a cube); (b) a failed release_confirm opens fully, goes home and keeps picking (max_recoveries per episode) instead of stopping"),
"bin.vote.async_extract": ("H5", False, "extract bin votes in the low-priority look worker; act only polls/holds"),
"bin.vote.frame_transform_snapshot": ("H4", False, "one immutable camera transform per frame shared by rim and tape"),
"hand.closed_tight": ("pinch fix", STOCK, "the closed_tight posture (LEFT DDS order, right mirrored)"),
Expand Down Expand Up @@ -461,7 +466,7 @@ def check_flag_values(cfg):
if not isinstance(mc, int) or isinstance(mc, bool) or mc < 0:
raise ValueError(f"instruction.max_chars = {mc!r}: want an int >= 0 (0 = off)")
for key in ("safety.nan_guard", "policy.generation_workers", "grasp.held_calibration.enabled",
"grasp.drop_slot_grid"):
"grasp.drop_slot_grid", "bin.vote.tape_body_reject", "grasp.wrong_object_recovery"):
if not isinstance(_get(cfg, key), bool):
raise ValueError(f"{key}: want true or false")
cb = _get(cfg, "planner.candidate_budget")
Expand Down Expand Up @@ -489,7 +494,7 @@ def check_flag_values(cfg):
p = cfg.get("perception") or {}
for key in ("sim_center_principal_point", "split_by_solidity", "tape_value_cut", "gate_below_bins",
"bins_from_image", "reject_log", "still_vote", "prism_speckle_floor", "real_border_partial", "real_merge_split",
"bin_bottom_first_run", "fk_colour_rescue"):
"bin_bottom_first_run", "fk_colour_rescue", "bin_size_guard", "bin_image_guard"):
if not isinstance(p.get(key), bool):
raise ValueError(f"perception.{key}: want true or false")
aim = p.get("aim_z_frac", 0.25)
Expand Down
Loading
Loading