AgentCPM-GUI: An on-device GUI agent for operating Android apps, enhancing reasoning ability with reinforcement fine-tuning for efficient task execution.
AgentCPM-GUI: An on-device GUI agent for operating Android apps, enhancing reasoning ability with reinforcement fine-tuning for efficient task execution.
【English | 中文】
Overview • Quick Start • Model • Evaluation Data • Technical Report
AgentCPM-GUI is an open-source on-device LLM agent model jointly developed by THUNLP, Renmin University of China and ModelBest. Built on MiniCPM-V with 8 billion parameters, it accepts smartphone screenshots as input and autonomously executes user-specified tasks.
Key features include:
Demo Case (1x speed):
https://github.com/user-attachments/assets/694d3c2c-12ce-4084-8feb-4937ca9ad247
git clone https://github.com/OpenBMB/AgentCPM-GUI
cd AgentCPM-GUI
conda create -n gui_agent python=3.11
conda activate gui_agent
pip install -r requirements.txtDownload AgentCPM-GUI from Hugging Face and place it in model/AgentCPM-GUI.
…Expected output:
{"thought":"任务目标是点击屏幕上的‘会员’按钮。当前界面显示了应用的推荐页面，顶部有一个导航栏。点击‘会员’按钮可以访问应用的会员相关内容。","POINT":[729,69]}Note: AgentCPM-GUI outputs relative coordinates ranging from 0-1000. The conversions are as follows:
rel_x, rel_y = [int(abs_x / width * 1000), int(abs_y / height * 1000)]
abs_x, abs_y = [int(rel_x / 1000 * width), int(rel_y / 1000 * height)]where width and height refer to the original width and height of the image, respectively.
# Launch the vLLM server
# If run out of VRAM, try add --max_model_len 2048
vllm serve model/AgentCPM-GUI --served-model-name AgentCPM-GUI --tensor_parallel_size 1 --trust-remote-code --limit-mm-per-prompt image=10…At each step, the agent outputs is a single JSON object that contains:
duration, thought) and/or a task-level flag (STATUS).Note that all keywords are case-sensitive, and we use compact JSON (i.e., no extra whitespace), which affects the tokenizer’s behavior.
| Action | Required field(s) | Optional field(s) | Purpose | Example |
|---|---|---|---|---|
| Click | POINT:[x,y] |
duration,thought,STATUS |
Single tap at the normalized screen coordinate (0–1000, origin = top-left). | {"POINT":[480,320]} |
| Long Press | POINT:[x,y]duration:1000 |
duration,thought,STATUS |
Touch-and-hold at coordinate (set a longer duration, e.g. >200 ms). | {"POINT":[480,320],"duration":1000} |
| Swipe | POINT:[x,y]to:"up" | "down" | "left" | "right" or to:[x,y] |
duration,thought,STATUS |
Swipe from the start point toward a direction or another coordinate. | {"POINT":[500,200],"to":"down"} |
| Press key | PRESS:"HOME" | "BACK" | "ENTER" |
duration,thought,STATUS |
Trigger a hardware / navigation button. | {"PRESS":"HOME"} |
| Type text | TYPE:"<text>" |
duration,thought,STATUS |
Insert the given text at the current input focus. | {"TYPE":"Hello, world!"} |
| Wait | duration |
thought,STATUS |
Idle for the specified time without any other action. | {"duration":500} |
| Task-level status | STATUS:"start" | "continue" | "finish" | "satisfied" | "impossible" | "interrupt" | "need_feedback" |
duration,thought |
Report task progress; may appear alone or with a primitive action. | {"STATUS":"finish"} |
Source code for SFT and RFT training is provided — see SFT and RFT.
| Model | Fun2Point | Text2Point | Bbox2text | Average |
|---|---|---|---|---|
| AgentCPM-GUI-8B | 79.1 | 76.5 | 58.2 | 71.3 |
| Qwen2.5-VL-7B | 59.8 | 59.3 | 50.0 | 56.4 |
| Intern2.5-VL-8B | 17.2 | 24.2 | 45.9 | 29.1 |
| Intern2.5-VL-26B | 14.8 | 16.6 | 36.3 | 22.6 |
| OS-Genesis-7B | 8.3 | 5.8 | 4.0 | 6.0 |
| UI-TARS-7B | 56.8 | 66.7 | 1.4 | 41.6 |
| OS-Atlas-7B | 53.6 | 60.7 | 0.4 | 38.2 |
| Aguvis-7B | 60.8 | 76.5 | 0.2 | 45.8 |
| GPT-4o | 22.1 | 19.9 | 14.3 | 18.8 |
| GPT-4o with Grounding | 44.3 | 44.0 | 14.3 | 44.2 |
| Dataset | Android Control-Low TM | Android Control-Low EM | Android Control-High TM | Android Control-High EM | GUI-Odyssey TM | GUI-Odyssey EM | AITZ TM | AITZ EM | Chinese APP (CAGUI) TM | Chinese APP (CAGUI) EM |
|---|---|---|---|---|---|---|---|---|---|---|
| AgentCPM-GUI-8B | 94.39 | 90.20 | 77.70 | 69.17 | 90.85 | 74.96 | 85.71 | 76.38 | 96.86 | 91.28 |
| Qwen2.5-VL-7B | 94.14 | 84.96 | 75.10 | 62.90 | 59.54 | 46.28 | 78.41 | 54.61 | 74.18 | 55.16 |
| UI-TARS-7B | 95.24 | 91.79 | 81.63 | 74.43 | 86.06 | 67.90 | 80.42 | 65.77 | 88.62 | 70.26 |
| OS-Genesis-7B | 90.74 | 74.22 | 65.92 | 44.43 | 11.67 | 3.63 | 19.98 | 8.45 | 38.10 | 14.50 |
| OS-Atlas-7B | 73.03 | 67.25 | 70.36 | 56.53 | 91.83* | 76.76* | 74.13 | 58.45 | 81.53 | 55.89 |
| Aguvis-7B | 93.85 | 89.40 | 65.56 | 54.18 | 26.71 | 13.54 | 35.71 | 18.99 | 67.43 | 38.20 |
| OdysseyAgent-7B | 65.10 | 39.16 | 58.80 | 32.74 | 90.83 | 73.67 | 59.17 | 31.60 | 67.56 | 25.44 |
| GPT-4o | - | 19.49 | - | 20.80 | - | 20.39 | 70.00 | 35.30 | 3.67 | 3.67 |
| Gemini 2.0 | - | 28.50 | - | 60.20 | - | 3.27 | - | - | - | - |
| Claude | - | 19.40 | - | 12.50 | 60.90 | - | - | - | - | - |
*Different train/test splits
TM and EM stand for the Type Match and Exact Match, respectively. All evaluation data and code are open-sourced — see here for details.
We provide CAGUI, an evaluation benchmark for Chinese apps covering grounding and agent tasks. See the dataset on Hugging Face.
Click here to view the FAQs.
If AgentCPM-GUI is useful for your research, please cite:
…No open issues yet, or sync has not completed.