Skip to content

[Fix] Read the browser screenshot from the capture's result object, not by pattern search - #60

Open
chaimaerachdi wants to merge 2 commits into
ThinkFlowLab:mainfrom
chaimaerachdi:screenshot-structural-parse
Open

chaimaerachdi wants to merge 2 commits into
ThinkFlowLab:mainfrom
chaimaerachdi:screenshot-structural-parse

Conversation

@chaimaerachdi

@chaimaerachdi chaimaerachdi commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Why

Follow-up to the review of #33 (P3). The screenshot an image-reading decision model (OmniJev) decides over was found with a regex over json.dumps of the whole run-code output. That output also carries the page state playwright-mcp appends after the result, so PNG-like text there could be taken for the picture, and a change in the wrapping could make it miss the real one.

How

_png_from_run_code (s1a/browser/decision_model.py) opens the known layers in turn: payload (compact RPC envelope), MCP content text, result/text/data, and JSON text after ### Result, possibly encoded twice (the same shapes the runtime's own _unwrap_mcp_text_result and extract_json_object handle). It reads the png field of the object SCREENSHOT_JS returns and accepts it only when it is valid base64 that decodes to a PNG signature. Anything else is "no screenshot", which takes the existing path: one retry, then the step goes on without a picture.

What

  • s1a/browser/decision_model.py: _SCREENSHOT_RE is replaced by _png_from_run_code; no flag or caller change.
  • tests/test_browser_screenshot.py: 4 tests. Every wrapping is opened; PNG-like page text outside the result object is not the picture (the old regex takes it); a png field that is not base64 of a PNG, or not a string, is no image; unknown shapes are no image.
  • CHANGELOG.md: entry under Fixed.

Verification

  • uv run ruff format --check . && uv run ruff check . && uv run ty check: all clean
  • uv run pytest -q: 784 passed, 47 skipped; scripts/smoke.sh: ok
  • CHANGELOG.md and the docs say what the code does now

The page-text case was checked against the regex from #33: it finds a picture there, and the new code returns none.

Demo / evidence

  • Video link, or exemption/blocker with follow-up: N/A, minor-change exemption (parsing of one result field, no change in agent behaviour on a well-formed capture).
  • Application/task → agent actions → independently checked result: N/A
  • System1-Agents / System1-Omni commits, checkpoint and worker/frontend provenance: base main 320bb08, head 4236efb.

Note

Low Risk
Localized parsing change in browser screenshot extraction with broad unit tests; agent behavior is unchanged when captures return a well-formed png field.

Overview
Browser screenshot parsing for image-reading decision models (e.g. OmniJev) no longer uses a regex over the entire run-code JSON. It now unwraps the known result envelopes (compact RPC payload, MCP text blocks, result/text/data, and JSON after ### Result, including double-encoded cases) and reads only the png field from the object SCREENSHOT_JS returns.

Decoded bytes must be valid base64 and start with a PNG signature; otherwise the step treats the capture as missing (same retry-then-text-only behavior as before). That blocks false positives when playwright-mcp appends page state containing PNG-like text after a failed or empty capture.

Tests cover each wrapping shape, rejection of page-state decoys, invalid png values, and unknown shapes; CHANGELOG documents the fix under Fixed.

Reviewed by Cursor Bugbot for commit 4236efb. Bugbot is set up for automated code reviews on this repo. Configure here.

chaimaerachdi and others added 2 commits October 9, 2026 16:38
…search

The PNG was found by a regex over json.dumps of the whole run-code output, so
PNG-like text elsewhere in that output (the page state playwright-mcp appends)
could be taken for the picture. _png_from_run_code opens the known layers in
turn (payload, MCP content text, result/text/data, JSON text after
"### Result", possibly encoded twice), reads the png field, and accepts it only
when it is valid base64 of a PNG. Follow-up to the review of ThinkFlowLab#33 (P3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant