识图模型工具已实现,测试全部通过 - #14
Open
opencode-agent[bot] wants to merge 1 commit into
Open
Conversation
Co-authored-by: PANDAJSR <PANDAJSR@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation complete. Here's a summary of what was done for this issue.
问题与方案
痛点:当主模型没有图片输入能力时(如 deepseek 等纯文本模型),现有唯一的识图路径
look_at会把图片块回塞给主模型,完全不可用。核心思路:把识图能力抽象成一个独立、可配置的后端模型,并以工具形式暴露给主 Agent——主 Agent 调用
secagent__look_at_image(path, prompt),由独立的识图子模型返回文字,图片永远不会回传给无视觉的主模型。改动内容
配置层
src/types.ts:SecAgentConfig.defaults与SettingsPayload新增visionModelId。src/config.ts:OFFICIAL_VISION_MODEL = "virtual-vision"(官方识图虚拟模型)。readSettings/saveSettings读写visionModelId。resolveModelConfig(拷贝 config 后useConfiguredModel,不修改入参)和resolveVisionAgentConfig(显式配置优先;官方模式下兜底virtual-vision;模型 id 失效时安全返回undefined)。运行时
src/pi-tools.ts:抽取共享的readImageFile(扩展名/文件/12MB 校验),look_at与识图工具共用,行为完全一致。src/system-prompt.ts:新增VISION_SYSTEM_PROMPT(单轮、只输出答案的子模型提示词)。src/runtime.ts:SecAgentRuntime新增可选visionConfig,内部构造无工具、单轮的识图子 Agent;仅在配置了识图模型时向主 Agent 暴露secagent__look_at_image工具;callVision读取图片→构造ChatAttachment→调用子 Agent→返回{ path, name, text },emit + 审计照常且脱敏;abort signal 透传给识图子请求,停止生成可中断识图。接线
src/electron/main.ts:models:list在官方模式放行virtual-vision并打vision标记;sessions:send解析并传入识图配置。src/index.ts:CLIopenRuntime同样传入识图配置。渲染层
SettingsApp.tsx:「默认模型」旁新增「识图模型」下拉(官方模式含virtual-vision,自定义模式可选任意已配置模型,可留空禁用)。App.tsx/utils.ts/vite-env.d.ts:主模型下拉/默认选择过滤掉virtual-vision(避免误当主 Agent),新增isOfficialVisionModel与vision/visionModelId类型。验证
npm run build(tsc + electron-vite)通过。npm test:137 通过 / 1 跳过 / 0 失败。新增 8 个测试覆盖:visionModelId持久化往返、resolveVisionAgentConfig三种情况(显式/官方兜底/失效)、识图工具成功路径(子请求含图片 dataUrl 与 prompt)、未配置时报错、图片校验失败不调用识图模型、单轮子 Agent 的 attachments 编码。剩余事项(仓库外)
官方中继服务需要:
/models返回virtual-vision(virtual: true);model: "virtual-vision"时路由到有视觉能力的上游模型——这正是 issue 里"其它虚拟模型可能有时有识图能力有时没有"的解法:客户端始终指向virtual-vision,上游可随时调整而不影响客户端。客户端在官方模式下即使中继尚未上线该模型,也会暴露识图工具,失败时安全降级为给主 Agent 返回错误信息,不影响整轮对话。
Closes #6
opencode session | github run