Repository navigation
[Fix] Read the browser screenshot from the capture's result object, not by pattern search - #60
Open
chaimaerachdi wants to merge 2 commits into
Open
chaimaerachdi wants to merge 2 commits into
chaimaerachdi wants to merge 2 commits into
Conversation
…search The PNG was found by a regex over json.dumps of the whole run-code output, so PNG-like text elsewhere in that output (the page state playwright-mcp appends) could be taken for the picture. _png_from_run_code opens the known layers in turn (payload, MCP content text, result/text/data, JSON text after "### Result", possibly encoded twice), reads the png field, and accepts it only when it is valid base64 of a PNG. Follow-up to the review of ThinkFlowLab#33 (P3). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…al-parse # Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Follow-up to the review of #33 (P3). The screenshot an image-reading decision model (OmniJev) decides over was found with a regex over
json.dumpsof the whole run-code output. That output also carries the page state playwright-mcp appends after the result, so PNG-like text there could be taken for the picture, and a change in the wrapping could make it miss the real one.How
_png_from_run_code(s1a/browser/decision_model.py) opens the known layers in turn:payload(compact RPC envelope), MCP content text,result/text/data, and JSON text after### Result, possibly encoded twice (the same shapes the runtime's own_unwrap_mcp_text_resultandextract_json_objecthandle). It reads thepngfield of the objectSCREENSHOT_JSreturns and accepts it only when it is valid base64 that decodes to a PNG signature. Anything else is "no screenshot", which takes the existing path: one retry, then the step goes on without a picture.What
s1a/browser/decision_model.py:_SCREENSHOT_REis replaced by_png_from_run_code; no flag or caller change.tests/test_browser_screenshot.py: 4 tests. Every wrapping is opened; PNG-like page text outside the result object is not the picture (the old regex takes it); apngfield that is not base64 of a PNG, or not a string, is no image; unknown shapes are no image.CHANGELOG.md: entry under Fixed.Verification
uv run ruff format --check . && uv run ruff check . && uv run ty check: all cleanuv run pytest -q: 784 passed, 47 skipped;scripts/smoke.sh: okCHANGELOG.mdand the docs say what the code does nowThe page-text case was checked against the regex from #33: it finds a picture there, and the new code returns none.
Demo / evidence
Note
Low Risk
Localized parsing change in browser screenshot extraction with broad unit tests; agent behavior is unchanged when captures return a well-formed
pngfield.Overview
Browser screenshot parsing for image-reading decision models (e.g. OmniJev) no longer uses a regex over the entire run-code JSON. It now unwraps the known result envelopes (compact RPC
payload, MCP text blocks,result/text/data, and JSON after### Result, including double-encoded cases) and reads only thepngfield from the objectSCREENSHOT_JSreturns.Decoded bytes must be valid base64 and start with a PNG signature; otherwise the step treats the capture as missing (same retry-then-text-only behavior as before). That blocks false positives when playwright-mcp appends page state containing PNG-like text after a failed or empty capture.
Tests cover each wrapping shape, rejection of page-state decoys, invalid
pngvalues, and unknown shapes; CHANGELOG documents the fix under Fixed.Reviewed by Cursor Bugbot for commit 4236efb. Bugbot is set up for automated code reviews on this repo. Configure here.