Skip to content

feat(provider): forward image attachments to Cursor SDK - #133

Merged
Nomadcxx merged 1 commit into
Nomadcxx:mainfrom
breath57:feat/image-attachments
Sep 29, 2026
Merged

Nomadcxx merged 1 commit into
Nomadcxx:mainfrom
breath57:feat/image-attachments

Conversation

@breath57

Copy link
Copy Markdown
Contributor

What

Forwards image attachments from the incoming OpenAI-style request to Cursor's SDK, so images attached in opencode reach the model instead of being silently dropped.

Why

Today the plugin collapses every request into a text-only prompt string. Image content parts ({ type: "image_url", image_url: { url: "data:..." } }) are discarded — even though the Cursor SDK natively supports them:

const run = await agent.send({
  text: "...",
  images: [{ data: base64Png, mimeType: "image/png" }],
});

(SDKUserMessage.images?: SDKImage[], see @cursor/sdk options.d.ts)

How

  • New pure extractor extractImagesFromMessages(messages) (src/proxy/image-extract.ts):
    • scans from the end, taking the last user message that carries image parts (the current turn)
    • data: URLs → { data, mimeType }; other URLs → { url }; malformed data URLs are skipped (never forwarded as bare URLs); duplicates removed
  • Threaded through the existing transport: proxy handler → sdk-child.sendRequest(..., images) → NDJSON (field written only when non-empty) → runner
  • scripts/sdk-runner.mjs: new pure buildUserMessage(prompt, images) returns { text, images } when images exist, the plain string otherwise (byte-identical behavior without images)
  • cursor-agent backend ignores the field (verified: its runner destructures only known keys)

Requirement for end-to-end use (not changed in this patch)

For opencode to actually send attachments, the model entry must declare support, e.g.:

"claude-opus-5": {
  "attachment": true,
  "modalities": { "input": ["text", "image"], "output": ["text"] }
}

The plugin's model sync does not add these flags today — documented as a follow-up (kept out of this diff on purpose).

Tests

  • Unit: extraction (data URL, multiple parts, mixed text+image, no images, malformed, last-message-only, dedupe), NDJSON payload presence/absence, buildUserMessage string-vs-object
  • E2E (real API): a programmatically generated 128×128 four-colour PNG sent through the patched runner → the model describes the image (shape/colours); the same request without images behaves exactly as before (no regression)

Notes

  • No changes to model discovery/pricing/sync.
  • Follow-up (separate PR): auto-declaring attachment/modalities on synced model entries + docs note.


Re-submission note

This is a re-submission: the previous PR (#131) was closed pending local verification on real hardware. GitHub refuses to reopen it (state cannot be changed — the branch was force-pushed or recreated), so a new PR is opened from the same branch. Branch content is unchanged (1 commit ahead of main, 0 behind). Now verified locally before submitting.

Comment thread src/proxy/image-extract.ts Outdated
const image = extractFromPart(part);
if (image) found.push(image);
}
if (found.length > 0) {

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks mate, one small gap here: when the latest message has no image, this keeps looking back and picks up an image from an earlier message. That means a text-only turn can send an old image again. It should stop at the latest message and send no image when that message has none. The test for a text-only message should check this too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review — good catch, and you were right.

The old loop scanned backwards and skipped user messages without images, so a text-only turn could resurrect an image from an earlier turn. Fixed as you described:

  • extractImagesFromMessages now resolves only the last user message and stops there. If that message has no usable images (or its content is not an array) it returns undefined, so no image is sent — it never looks past the latest message.
  • One implementation note: instead of Array.prototype.findLast I used a reverse loop that breaks at the last user message, because the project targets lib: ES2022 (findLast is ES2023 → TS2550).
  • Tests: the previous case does not pick images from a user message that has none if an earlier one does asserted exactly the old behaviour, so I inverted it to returns undefined when the last user message has no images even if an earlier one does. The "last user message carries an image" case is still covered and still passes.
  • Also rebased onto current main (the branch had conflicts with the OpenCode 2.0 work — resolved by keeping both the params and images fields) and added the two new test files to test:ci:unit so they actually run in CI.

CI: unit + integration both green. Thanks again!

@breath57
breath57 force-pushed the feat/image-attachments branch from 7ef5ee0 to f2925ab Compare September 28, 2026 07:07
@Nomadcxx
Nomadcxx merged commit e305e22 into Nomadcxx:main Sep 29, 2026
2 checks passed
@Nomadcxx

Copy link
Copy Markdown
Owner

Thanks mate for the resubmission. I verified the wiring against the installed SDK, agent.send() takes a string or { text, images }, and both image forms are valid. Requests without images stay byte-identical, and the cursor-agent backend ignores the field. Merged.

I'll land one small fix on main: the dedupe key compares data.length instead of the payload, so two images of equal encoded length collapse into one. You don't need to touch anything here.

Cheers RAMA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants