gui-agents

Agent S: an open agentic framework that uses computers like a human

STEP-GUI: The top GUI agent solution in the galaxy. Developed by the StepFun-GELab team and powered by StepFun’s cutting-edge research capabilities.

💻 A curated list of papers and resources for multi-modal Graphical User Interface (GUI) agents.

[ICLR 2026 Oral] ScaleCUA is the open-sourced computer use agents that can operate on cross-platform environments (Windows, macOS, Ubuntu, Android).

An understudy watches. Then performs.

The AI agent that physically operates a phone — the way you do.

[ICLR'25 Oral] UGround: Universal GUI Visual Grounding for GUI Agents

[ECCV 2026] Official Repository for "DualTAP: A Dual-Task Adversarial Protector for Mobile MLLM Agents"

Official implementation of UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning

[AAAI 2026] Test-Time Reinforcement Learning for GUI Grounding via Region Consistency https://arxiv.org/abs/2508.05615

DataClaw: Agentic Tailoring Multimodal Data from Raw Streams — coming soon (code, weights, dataset & DataClaw-val upon acceptance).

[CVPR 2025] Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents

Code repo for "Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding"

[ACL'25 (Findings)] Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents

Curated resources about automated GUI computer-use via LLMs. Highly opinionated, focus is on quality vs quantity.

PreAct: Computer-Using Agents that Get Faster on Repeated Tasks (arXiv:2606.17929). Compile successful runs into verified state-machine programs and replay them 8.5–13× faster with no per-step LLM calls.

Official code for "Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents".

Papers, benchmarks, demos, and tools for multimodal coding agents: visual feedback for UI, CAD, 3D, games, charts, and robotics.

A Gradio-based demonstration for the Microsoft Fara-7B model, designed as a computer use agent. Users upload UI screenshots (e.g., desktop or app interfaces), provide task instructions (e.g., "Click on the search bar"), and receive parsed actions with visualized indicators overlaid on the image.