Skip to content

FundingCAPTCHA: self-learning depth background, no boot-time standback - #170

Merged
charliehuge merged 2 commits into
mainfrom
claude/adaptive-background-166
Nov 10, 2026
Merged

charliehuge merged 2 commits into
mainfrom
claude/adaptive-background-166

Conversation

@ii-impy

@ii-impy ii-impy commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Closes #166.

What

None of the FundingCAPTCHA entry points (app.py, shader_app.py/playlist_app.py, body_grid_tester.py) has a standback or calibration screen any more. The depth background is learned continuously from the first frame, with people in view: the farthest steady depth wins (ADR-0023).

IIVision

  • New AdaptiveBackground / AdaptiveBackgroundConfig (IIVision/adaptive_background.py). All of it is vectorised numpy, with an injectable clock, push(frame) and reseed(). It implements the issue's five rules: seed, EMA at background, reveal after reveal_frames, absorb after absorb_s, and the bump guard.
  • BlobTracker now reads a small BackgroundModel protocol: background, valid_mask, threshold(depth_delta_mm). Calibration gains threshold(), which uses the same 3×IQR formula as before, so FlowerBeds and Calibrator behave exactly as they did.
  • Per-frame update time is logged every background.stats_log_s (30 s). It measures about 5 ms at 640×480 on this machine, against a 66 ms budget at 15 fps.

FundingCAPTCHA

  • app.py: the BG_CAL state, the countdown screen, the collect gate and the cal_progress/cal_done handshake are gone. It boots straight into SCREENSAVER. R reseeds the background from the next frame. /api/restart still restarts the camera thread, which now starts a fresh model and returns to SCREENSAVER instead of BG_CAL.
  • shader_app.py: the standback timer and the collect event are gone. playlist_app.py shares this code, so it gets the change too.
  • body_grid_tester.py: the same change. Its C views (background, valid mask, noise σ, threshold, delta) now show the live model, refreshed every 0.5 s. R reseeds.
  • captcha-settings.json: the calibration_* keys are replaced by a background block with the issue's defaults.

Docs

  • New ADR-0023. ADR-0013 (BG_CAL section) and ADR-0021 (startup consequence) now point to it. docs/FundingCAPTCHA.md, docs/agents/architecture.md and ADR-0007's /captcha/restart line are updated.

One deviation from the issue

The issue says σ starts at depth_delta_mm "so thr starts at depth_delta_mm". With thr = max(depth_delta_mm, noise_k × σ), it actually starts at noise_k × depth_delta_mm, which is 120 mm with our settings. I kept σ seeded at depth_delta_mm as the issue's table says. Starting high means early frames read nearly everything as background and the σ estimate settles down from above (time constant about 20 frames). Starting low would bias the estimate low, because it only learns from samples inside thr, and could flash false foreground just after boot. The cost is that for the first few seconds, something within about 120 mm of its background (feet on the floor) isn't foreground. ADR-0023 records the rule as implemented.

Tests

  • IIVision/test_adaptive_background.py covers every case the issue lists: a person present from frame 0 who then walks off, a person standing still for 120 s, a static object absorbed at absorb_s, a single-frame far spike, Gaussian noise (σ 20 mm at 3 m, < 0.1 % raw foreground, none after despeckle), a bump nearer that reseeds and a bump farther that heals without one, and the update-time log. Fidget, crowd and reseed-request cases are covered too.

  • ShowControl/FundingCAPTCHA/test_live_background.py: shader_app.start_camera with MockCamera (bodies present from frame 0) shows a body in the Body texture within 1.5 s. In practice it's frame 10, about 0.63 s. The other tests check that the BG_CAL state and calibration settings are gone.

  • app.py --mock-camera and body_grid_tester.py --mock-camera both smoke-ran headless without errors. The app reached the Arc 5.6 s after launch, 5 s of which is the attract dwell.

  • Python suite: 498 passed. Two things fail or can't run, and both are the same on main:

    • test_hunter_bird.py::test_person_has_white_fill_and_ink_outline fails.
    • FlowerBeds/test_layout_tool.py can't be collected because fastapi isn't installed on this machine.

    Node isn't installed here, so the node --test half of make test didn't run. Nothing JS changed.

Needs hardware after merge

  • Orbbec: with a person in view at launch, they show as foreground after their first step, and the empty floor never does.
  • Timing: the logged Background update … mean/max ms line stays within budget at 640×480 @ 15 fps on the show machine.
  • Tuning: absorb_s against real dwell times, and whether a dense crowd sets off the bump guard.
  • The model now learns while the projector is showing content. ADR-0004's black-frame rule came from the touch-screen era, before ADR-0013. It's worth a glance that projector IR doesn't disturb the outward-facing camera.

🤖 Generated with Claude Code

ii-impy and others added 2 commits November 8, 2026 14:23
#166)

IIVision gains AdaptiveBackground: "the farthest steady depth wins". A
person can only make a depth pixel nearer, so each pixel's background is
the farthest depth it has held steadily, learned from the first frame
with people in view. Rules: seed, slow EMA at background, reveal after
reveal_frames farther, absorb after absorb_s steady nearer, and a bump
guard that reseeds when most of the scene stays foreground.

BlobTracker now reads a small BackgroundModel protocol (background,
valid_mask, threshold()), implemented by both Calibration and
AdaptiveBackground. Calibrator stays for FlowerBeds.

app.py, shader_app.py (and so playlist_app.py) and body_grid_tester.py
drop BG_CAL, the standback timer and the cal_progress/cal_done
handshake; R now reseeds the background. captcha-settings.json swaps the
calibration_* keys for a background block. Per-frame update time is
logged every stats_log_s. ADR-0023 supersedes ADR-0013's BG_CAL section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On real hardware, launch left small blobs that lasted until absorb_s:

- The Gemini 336's first frames read up to ~400 mm too far (1 frame
  warm, ~5 cold). Seeding bg from them made the true surface "nearer".
  New background.warmup_s (1.0) ignores the start of a stream.
- Seeding bg from one sample is fragile: on a noisy far pixel a single
  far outlier left every later reading nearer, never at bg, too jumpy
  to absorb. bg and sigma now come from the mean and spread of the first
  min_valid_frames samples.

Checked against four recorded starts and a live run: no persistent
foreground. Live update time is 12-14 ms mean at 640x480.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ii-impy

ii-impy commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Hardware testing found small blobs left on the screen after launch. They lasted until absorb_s. Fixed in 59444d5:

  • The camera's first frames read too far. On the Gemini 336, the first frames of a stream read up to ~400 mm too far: 1 frame when the camera is warm, about 5 from cold. Seeding the background from them made the true floor look nearer, so it showed as foreground. The new background.warmup_s setting (default 1.0) ignores the start of each stream.
  • Seeding from a single sample was fragile. On a noisy far pixel (σ ≈ 80 mm at 3.7 m), one far outlier left every later reading "nearer", and the pixel was too jumpy to ever be absorbed. Each pixel's background and noise level now come from the mean and spread of its first min_valid_frames samples.

Checked against four recorded camera starts and a live 30 s run: no persistent foreground. Two numbers in the description above change:

  • Boot-to-live is now about 2 s, not under 1 s.
  • The live per-frame update is 12–14 ms mean, not the ~5 ms measured on synthetic frames. That's still well inside the 66 ms budget.

🤖 Generated with Claude Code

@charliehuge
charliehuge merged commit 77a65db into main Nov 10, 2026
1 check passed
@charliehuge
charliehuge deleted the claude/adaptive-background-166 branch November 10, 2026 11:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FundingCAPTCHA: self-learning depth background, no boot-time standback

2 participants