docs: add Antigravity x ARTEMIS autonomous testing workflow showcase and update banner

This commit is contained in:
somew1nd committed 2026-08-18 15:13:39 -07:00
1 parent 9a5aaf99b7
commit a306258d26
9 files changed
+229 -78

No files matched your search

+64 -27
View File
@@ -3,16 +3,18 @@
</p> </p>
<p align="center"> <p align="center">
<strong>ARTEMIS: Autonomous AI Assistant for Mobile Automation</strong> <strong>ARTEMIS: Next-Gen AI Mobile Test Automation & Autonomous Assistant Platform</strong><br>
<sub><b>A</b>utonomous <b>R</b>eal-time <b>T</b>esting, <b>E</b>xploration &amp; <b>M</b>obile <b>I</b>nteraction <b>S</b>ystem</sub>
</p> </p>
<p align="center"> <p align="center">
<em>⚡ Drive real phones from Cursor & Claude Code • Cross-App Automation • Zero-Maintenance UI Testing • Flash Mode as Fast as 3-5s</em> <em>⚡ Drive Real Devices from Antigravity & Claude Code • Cross-App Automation • Zero-Maintenance UI Testing • Bug Repro & Logcat Diagnostics</em>
</p> </p>
<p align="center"> <p align="center">
<a href="./README.md"><b>English</b></a> • <a href="./README.md"><b>English</b></a> •
<a href="./README_CN.md">中文文档</a> • <a href="./README_CN.md">中文文档</a> •
<a href="#workflow-showcase">Workflow Showcase</a> •
<a href="#quick-start">Quick Start</a> • <a href="#quick-start">Quick Start</a> •
<a href="#mcp-setup">MCP for IDEs</a> • <a href="#mcp-setup">MCP for IDEs</a> •
<a href="#benchmarks">Benchmarks</a> • <a href="#benchmarks">Benchmarks</a> •
@@ -36,12 +38,45 @@
## ✨ Key Highlights ## ✨ Key Highlights
* ⚡ **Lightning Fast**: First-of-its-kind **Optimistic Asynchronous Pipeline** decouples the foreground execution stream from heavy LLM reasoning; * 🤖 **Cross-App Automation & Autonomous AI Assistant**: Operates not just as a robust testing framework, but as an autonomous agent capable of handling complex cross-app workflows and daily tasks via natural language;
* 🔋 **Ultra-Long Autonomous Execution**: **Pro mode** runs stably for over 10 hours, while monitoring tasks support around-the-clock 24/7 execution; * 🧪 **Zero-Maintenance UI Test Automation**: Built upon a "Dynamic-First, Coordinate-Fallback" multimodal locating engine, eliminating fragile XPath/ID selector maintenance and remaining resilient to UI redesigns, system updates, and resolution drift;
* 🔌 **Native Model Context Protocol (MCP)**: Empowers **Cursor, Claude Code, OpenClaw, Windsurf** with direct mobile device manipulation from within your chat prompts; * 🐞 **One-Click Bug Repro & Logcat Diagnostics in IDE**: Native **Model Context Protocol (MCP)** integration allows **Antigravity, Claude Code, and Windsurf** to drive physical test devices via natural language, automatically capturing crash stacks from **Logcat** and keyframe screenshots;
* 💾 **Visual Console & Persistent Debugging**: Real-time Web UI screen projection, execution replay, and task management with natural language test generation and high-fidelity historical review; * ⚡ **Ultra-Fast Execution (3–5s per Step)**: Pioneered an **Optimistic Asynchronous Pipeline** that completely decouples UI interaction from heavy LLM reasoning, achieving rapid regression throughput in Flash mode;
* 🛡️ **Popup Self-Healing & 10+ Hour Exploration**: Proprietary **Safety Net** double-checks targets before action execution to intercept and clear interfering system popups; Pro mode supports **10+ hours** of continuous exploratory & monkey-plus stability testing;
* 🏆 **Industry-Leading SOTA**: Achieved **99%+ task completion** on Google Research's **AndroidWorld** benchmark (100+ complex multi-step tasks). * 🏆 **Industry-Leading SOTA**: Achieved **99%+ task completion** on Google Research's **AndroidWorld** benchmark (100+ complex multi-step tasks).
<a id="workflow-showcase"></a>
## 🤝 Antigravity × ARTEMIS: Autonomous Testing Workflow
Experience seamless collaboration between **Antigravity** and **ARTEMIS** via native MCP integration — taking you from a natural language requirement to a production-grade diagnostic report in four automated steps:
<table width="100%">
<tr>
<td width="50%" align="center">
<b>1️⃣ Prompt Input (Task Dispatch)</b><br>
<sub>Describe your test scenario and target metrics in Antigravity</sub><br><br>
<img src="./docs/assets/workflow-1-prompt.png" width="100%" alt="Step 1: Prompt Input in Antigravity" />
</td>
<td width="50%" align="center">
<b>2️⃣ Test Plan Generation</b><br>
<sub>Antigravity formulates a step-by-step test plan & architecture for review</sub><br><br>
<img src="./docs/assets/workflow-2-plan.png" width="100%" alt="Step 2: Test Plan Generation" />
</td>
</tr>
<tr>
<td width="50%" align="center">
<b>3️⃣ Autonomous Test Execution</b><br>
<sub>ARTEMIS drives the real device, navigates UI, and profiles performance</sub><br><br>
<img src="./docs/assets/workflow-3-exec.png" width="100%" alt="Step 3: Autonomous Test Execution" />
</td>
<td width="50%" align="center">
<b>4️⃣ Comprehensive Final Report</b><br>
<sub>Delivers structured audit findings, metric tables, and raw datasets</sub><br><br>
<img src="./docs/assets/workflow-4-report.png" width="100%" alt="Step 4: Final Report" />
</td>
</tr>
</table>
<a id="quick-start"></a> <a id="quick-start"></a>
## ⚡ Quick Start ## ⚡ Quick Start
@@ -64,7 +99,7 @@ start.bat
<a id="mcp-setup"></a> <a id="mcp-setup"></a>
<a id="mcp"></a> <a id="mcp"></a>
<details> <details>
<summary><b>🔌 MCP Setup for Cursor / Claude Code / Windsurf (Click to expand)</b></summary> <summary><b>🔌 MCP Setup for Antigravity / Claude Code / Windsurf (Click to expand)</b></summary>
<br> <br>
@@ -75,14 +110,14 @@ ARTEMIS includes a native **Model Context Protocol (MCP)** server. Connect your
Run the built-in generator to produce ready-to-use JSON configuration: Run the built-in generator to produce ready-to-use JSON configuration:
```bash ```bash
artemis mcp --generate-config cursor artemis mcp --generate-config antigravity
# Or generate configs for all IDEs: # Or generate configs for all IDEs:
artemis mcp --generate-config all artemis mcp --generate-config all
``` ```
### 2. Copy Config to Your IDE ### 2. Copy Config to Your IDE
* **Cursor** (`.cursor/mcp.json` or Settings ➔ MCP Servers): * **Antigravity** (MCP config file or Settings ➔ MCP Servers):
```json ```json
{ {
"mcpServers": { "mcpServers": {
@@ -112,7 +147,7 @@ artemis mcp --generate-config all
``` ```
### 3. Prompt Your Phone in the IDE Chat ### 3. Prompt Your Phone in the IDE Chat
In Cursor or Claude Code, simply prompt: In Antigravity or Claude Code, simply prompt:
> 💬 *"Build the latest changes into an APK, install it on the connected device, open the login screen with a test account, verify if there are any unexpected popups after login, and return screenshots of the final page."* > 💬 *"Build the latest changes into an APK, install it on the connected device, open the login screen with a test account, verify if there are any unexpected popups after login, and return screenshots of the final page."*
</details> </details>
@@ -131,15 +166,17 @@ from artemis.interfaces.sdk import ArtemisClient
async def main(): async def main():
# Initialize client (choose "flash" for speed or "pro" for deep reasoning) # Initialize client (choose "flash" for fast UI checks or "pro" for deep reasoning & self-healing)
client = ArtemisClient(default_profile="flash") client = ArtemisClient(default_profile="flash")
# Run autonomous task # Execute natural language end-to-end test case
result = await client.run("Open calculator and compute (128 * 45) + 330") result = await client.run(
"Open System Settings, go to 'Battery', verify battery percentage is displayed, and check for any crash dialogs."
)
print(f"Status: {result.status}") # Structured assertions & execution tracing
print(f"Turns taken: {result.turns}") assert result.status == "SUCCESS", f"Test failed: {result.failure_reason}"
print(f"Trace ID: {result.trace_id}") print(f"✅ Test Passed! Turns: {result.turns} | Trace ID: {result.trace_id}")
if __name__ == "__main__": if __name__ == "__main__":
@@ -156,20 +193,20 @@ if __name__ == "__main__":
<sub>💡 <b>Console Overview</b>: <b>① View Switcher</b> (Home / Workspace) · <b>② Model & Replay</b> (Flash/Pro status & video replay) · <b>③ Live Agent Stream</b> (Action perception, target coordinates & structured results) · <b>④ Prompt Dock</b> (Natural language dispatch) · <b>⑤ Task Queue & Dashboard</b> (Lifecycle & history)</sub> <sub>💡 <b>Console Overview</b>: <b>① View Switcher</b> (Home / Workspace) · <b>② Model & Replay</b> (Flash/Pro status & video replay) · <b>③ Live Agent Stream</b> (Action perception, target coordinates & structured results) · <b>④ Prompt Dock</b> (Natural language dispatch) · <b>⑤ Task Queue & Dashboard</b> (Lifecycle & history)</sub>
</p> </p>
* 🖥️ **Web Visual Console (`artemis ui`)**: Real-time screen projection and interactive panel, supporting natural language task dispatch, live reasoning telemetry, action trajectories, and execution replay; * 🖥️ **Web Visual Test Console (`artemis ui`)**: Real-time screen projection and interactive panel, supporting natural language test dispatch, live reasoning telemetry, action trajectories, and execution replay;
* 🔌 **Native MCP Protocol (IDE Integration)**: Operates as a standard MCP server seamlessly integrating with **Cursor, Claude Code, OpenClaw**, etc., directly invoking mobile device manipulation inside the IDE; * 🔌 **Native MCP Protocol (IDE Collaboration)**: Operates as a standard MCP server seamlessly integrating with **Antigravity, Claude Code, Windsurf**, etc., directly driving real devices inside the IDE to verify bugs and run test cases;
* 💻 **Developer CLI (`artemis run`)**: Direct terminal execution for automated tasks, test cases, or AndroidWorld benchmarks with high-fidelity structured terminal output; * 💻 **Developer CLI (`artemis run`)**: Direct terminal execution for automated test cases, exploratory stability inspection, or AndroidWorld benchmarks with high-fidelity structured terminal output;
* 🐍 **Python SDK**: Integrates as a standard Python library into existing automated testing frameworks or pipelines with strongly typed Pydantic structured outputs. * 🐍 **Python SDK**: Integrates as a standard Python library into existing automated testing frameworks (e.g., pytest) or CI/CD pipelines with strongly typed Pydantic structured outputs and assertion support.
## 📊 Head-to-Head Comparison ## 📊 Head-to-Head Comparison
| Scenario / Dimension | Traditional Automation | Generic Screen AI Agents | **ARTEMIS ☕** | | Evaluation Dimension | Traditional Test Automation (Appium / Maestro) | Generic Mobile VLM Agents | **ARTEMIS ☕ (Next-Gen AI Testing)** |
| :--- | :--- | :--- | :--- | | :--- | :--- | :--- | :--- |
| **Natural Language Execution** | ❌ Rigid hard-coded scripts only | ⚠️ Sluggish (waits 20-30s per tap) | ⚡ **High Throughput**: As fast as 3–5s per step in Flash mode with fluid responses | | **Test Case Maintenance** | ❌ Fragile XPath/ID dependencies; UI changes cause test failures | ⚠️ Unreliable execution; cannot be reused as regression tests | 🧪 **Zero Maintenance**: Natural language test cases resilient to UI drift & redesigns |
| **UI Redesign Resilience** | ❌ Layout changes break XPath selectors | ⚠️ Coordinate guessing leads to drift | 🎯 **Semantic Grounding**: Blends text, icons & UI tree structure to adapt to UI changes | | **Execution Latency & Throughput** | ⚡ Fast script execution, but extreme setup and locator debugging costs | ❌ Sluggish 20–30s per step; too slow for regression testing | ⚡ **High Throughput**: Optimistic Async Pipeline runs at 3–5s per step |
| **Popup Interception** | ❌ Blocked and crashed immediately | ❌ Loops endlessly on unfamiliar screens | 🩹 **Pre-Execution Self-Healing**: Verifies target state before clicking; dismisses popups | | **Popup Resilience & Self-Healing** | ❌ System popups or permissions immediately crash the script | ❌ Easily gets stuck or loops endlessly on unexpected dialogs | 🛡️ **Pre-Execution Safety Net**: Automatically intercepts and clears interfering popups |
| **Dynamic & Video Tasks** | ❌ Blind static sleep only | ❌ Blind to dynamic video playback | ⏱️ **Real-Time Multimedia**: Live video stream analysis, countdown timers & speculative chaining | | **Diagnostics & Multimedia** | ❌ Blind static waits (sleep); cannot assert dynamic video/animations | ❌ Static screenshots only; no system logs or underlying state | 🐞 **Deep Diagnostics**: Live video stream analysis & **Logcat crash stack capture** |
| **IDE & Tooling Integration** | ❌ Standalone test runner | ❌ Web-only with high integration friction | 🔌 **Native MCP Server**: Directly callable inside Cursor, Claude Desktop, etc. | | **Dev Environment Integration** | ❌ Standalone runner; requires manual log collection upon failure | ❌ Isolated web demos; difficult to embed into dev pipelines | 🔌 **Native MCP & SDK**: Drive physical test devices and debug directly inside Antigravity / Claude Code |
<a id="benchmarks"></a> <a id="benchmarks"></a>
## 🏆 Benchmarks: AndroidWorld (SOTA 99%+) ## 🏆 Benchmarks: AndroidWorld (SOTA 99%+)
@@ -230,7 +267,7 @@ Evaluated on [AndroidWorld](https://github.com/google-research/android_world)
- [x] **Optimistic Asynchronous Pipeline**: Ultra-lean main loop + background context compression & milestone checks. - [x] **Optimistic Asynchronous Pipeline**: Ultra-lean main loop + background context compression & milestone checks.
- [x] **Pre-Execution Safety Net**: Millisecond pre-check gate & speculative chained actions. - [x] **Pre-Execution Safety Net**: Millisecond pre-check gate & speculative chained actions.
- [x] **Time-Sensitive Media Tasks**: Fullscreen video and audio stream analysis with `scrcpy` & `ffmpeg`. - [x] **Time-Sensitive Media Tasks**: Fullscreen video and audio stream analysis with `scrcpy` & `ffmpeg`.
- [x] **Native MCP Server**: Seamless integration with tools like Cursor and Claude Desktop. - [x] **Native MCP Server**: Seamless integration with tools like Antigravity and Claude Desktop.
- [x] **Web Visual Console**: Live screen projection, interactive playground, and trajectory review. - [x] **Web Visual Console**: Live screen projection, interactive playground, and trajectory review.
- [x] **AndroidWorld SOTA**: Achieved 99%+ task completion rate. - [x] **AndroidWorld SOTA**: Achieved 99%+ task completion rate.
- [ ] **Cross-Platform Extensions**: Exploring iOS and desktop Web perception and execution. - [ ] **Cross-Platform Extensions**: Exploring iOS and desktop Web perception and execution.
+66 -28
View File
@@ -3,16 +3,18 @@
</p> </p>
<p align="center"> <p align="center">
<strong>ARTEMIS: 全自动操控手机的 AI 个人助理</strong> <strong>ARTEMIS: 下一代 AI 驱动的移动端自动化测试与个人助理平台</strong><br>
<sub><b>A</b>utonomous <b>R</b>eal-time <b>T</b>esting, <b>E</b>xploration &amp; <b>M</b>obile <b>I</b>nteraction <b>S</b>ystem</sub>
</p> </p>
<p align="center"> <p align="center">
<em>⚡ 让 Cursor / Claude Code 接管真机 • 跨 App 复杂自动化 • 零脚本维护 UI 测试 • Flash 模式单步最快 3-5 秒</em> <em>⚡ 让 Antigravity / Claude Code 接管真机 • 跨 App 复杂自动化 • 零脚本维护 UI 测试 • 智能 Bug 复现与 Logcat 诊断</em>
</p> </p>
<p align="center"> <p align="center">
<a href="./README.md">English</a> • <a href="./README.md">English</a> •
<a href="./README_CN.md"><b>中文文档</b></a> • <a href="./README_CN.md"><b>中文文档</b></a> •
<a href="#workflow-showcase">全流程演示</a> •
<a href="#quick-start">快速上手</a> • <a href="#quick-start">快速上手</a> •
<a href="#mcp-setup">MCP 接入 IDE</a> • <a href="#mcp-setup">MCP 接入 IDE</a> •
<a href="#benchmarks">基准评测</a> • <a href="#benchmarks">基准评测</a> •
@@ -36,11 +38,45 @@
## ✨ 核心亮点 ## ✨ 核心亮点
* ⚡ **极致响应速度**:首创**全链路乐观异步流水线 (Optimistic Async Pipeline)**,主干执行流与后台重度推理完全解耦; * 🤖 **跨 App 复杂自动化与 AI 个人助理**:不仅能执行严格的自动化测试用例,更具备接管真机的自主决策与长流程操作能力,一句话处理跨应用复杂业务与任务;
* 🔋 **超长程稳定运行**:**Pro 模式**可运行超 10 小时,监控任务甚至可以全天候运行; * 🧪 **零脚本维护 UI 自动化测试**:基于“动态优先、坐标兜底”的多模态语义定位,彻底告别传统 XPath / 控件 ID 频繁失效的痛点,无惧 App 改版、分辨率差异与系统升级;
* 🔌 **原生 MCP (Model Context Protocol) 赋能**:让 **Cursor、Claude Code、OpenClaw、Windsurf** 直接长出“操作真机的双手”,在聊天窗口一句话完成真机测试与 Bug 复现; * 🐞 **IDE 内一键 Bug 复现与 Logcat 诊断**:原生支持 **MCP 协议**,可在 **Antigravity、Claude Code、Windsurf** 中直接用自然语言驱动真机复现缺陷,自动抓取关键帧截图与 **Logcat 崩溃堆栈**,完成研发测试闭环;
* 💾 **可视化控制台与持久化调试**:提供 Web UI 实时投屏、操作回放与任务持久化管理,支持通过自然语言生成测试用例,并对历史执行进行高保真可视化审查; * ⚡ **极速执行吞吐(单步 3–5 秒)**:首创**全链路乐观异步流水线 (Optimistic Async Pipeline)**,解耦重度推理与页面操作,在 Flash 模式下实现丝滑的高频交互与快速回归验证;
* 🏆 **业界顶尖 SOTA**:在 Google Research **AndroidWorld** 基准评测(100+ 复杂长程任务)中取得 **99%+ 任务完成率**。 * 🛡️ **干扰弹窗自愈与超长程巡检**:独创 **Safety Net** 执行前校验机制,自动识别并清除系统权限弹窗、通知遮挡等异常;Pro 模式支持连续 **10+ 小时** 无人值守稳定性巡检与探索性测试 (Exploratory / Monkey-plus Testing);
* 🏆 **业界顶尖 SOTA**:在 Google Research **AndroidWorld** 基准评测(100+ 复杂长程系统与应用交互任务)中取得 **99%+ 任务完成率**。
<a id="workflow-showcase"></a>
<a id="全流程演示"></a>
## 🤝 Antigravity × ARTEMIS:全流程自主测试演示
通过原生 MCP 协议,**Antigravity** 与 **ARTEMIS** 深度协同——只需一句自然语言指令,即可自动完成从需求理解、测试规划、真机执行到深度报告输出的完整闭环:
<table width="100%">
<tr>
<td width="50%" align="center">
<b>1️⃣ 输入测试提示词 (Task Dispatch)</b><br>
<sub>在 Antigravity 中用自然语言描述测试需求与目标指标</sub><br><br>
<img src="./docs/assets/workflow-1-prompt.png" width="100%" alt="步骤一:输入测试提示词" />
</td>
<td width="50%" align="center">
<b>2️⃣ 生成测试方案 (Test Plan Generation)</b><br>
<sub>Antigravity 自动拆解任务,生成详细测试步骤与架构图供确认</sub><br><br>
<img src="./docs/assets/workflow-2-plan.png" width="100%" alt="步骤二:生成测试方案" />
</td>
</tr>
<tr>
<td width="50%" align="center">
<b>3️⃣ 自主执行测试 (Autonomous Test Execution)</b><br>
<sub>ARTEMIS 接管真机自动化操作,跳过广告并实时采集性能指标</sub><br><br>
<img src="./docs/assets/workflow-3-exec.png" width="100%" alt="步骤三:自主执行测试" />
</td>
<td width="50%" align="center">
<b>4️⃣ 交付最终报告 (Comprehensive Final Report)</b><br>
<sub>生成结构化测试分析报告,包含性能数据表、结论与原始数据集</sub><br><br>
<img src="./docs/assets/workflow-4-report.png" width="100%" alt="步骤四:交付最终报告" />
</td>
</tr>
</table>
<a id="quick-start"></a> <a id="quick-start"></a>
<a id="快速上手"></a> <a id="快速上手"></a>
@@ -65,7 +101,7 @@ start.bat
<a id="mcp-setup"></a> <a id="mcp-setup"></a>
<a id="mcp"></a> <a id="mcp"></a>
<details> <details>
<summary><b>🔌 接入 Cursor / Claude Code / Windsurf (MCP)(点击展开)</b></summary> <summary><b>🔌 接入 Antigravity / Claude Code / Windsurf (MCP)(点击展开)</b></summary>
<br> <br>
@@ -76,14 +112,14 @@ ARTEMIS 内置原生 **Model Context Protocol (MCP)** 服务。只需将以下
运行内置命令一键获取当前环境的完整配置 JSON: 运行内置命令一键获取当前环境的完整配置 JSON:
```bash ```bash
artemis mcp --generate-config cursor artemis mcp --generate-config antigravity
# 或生成所有 IDE 配置: # 或生成所有 IDE 配置:
artemis mcp --generate-config all artemis mcp --generate-config all
``` ```
### 2. 复制配置到 IDE ### 2. 复制配置到 IDE
* **Cursor** (`.cursor/mcp.json` 或 设置 ➔ MCP Servers): * **Antigravity** (MCP 配置文件或 设置 ➔ MCP Servers):
```json ```json
{ {
"mcpServers": { "mcpServers": {
@@ -113,7 +149,7 @@ artemis mcp --generate-config all
``` ```
### 3. 在 IDE 中体验真机协同 ### 3. 在 IDE 中体验真机协同
在 Cursor / Claude Code 对话框中直接输入: 在 Antigravity / Claude Code 对话框中直接输入:
> 💬 *"请帮我把刚刚修改的代码编译成 APK 并安装到手机上,打开登录页面输入测试账号,验证登录后是否有异常弹窗,并把最终页面截图回传。"* > 💬 *"请帮我把刚刚修改的代码编译成 APK 并安装到手机上,打开登录页面输入测试账号,验证登录后是否有异常弹窗,并把最终页面截图回传。"*
</details> </details>
@@ -132,15 +168,17 @@ from artemis.interfaces.sdk import ArtemisClient
async def main(): async def main():
# 初始化客户端(支持 "flash" 极速模式 或 "pro" 深度推理模式) # 初始化测试客户端(支持 "flash" 极速校验模式 或 "pro" 深度推理与自愈模式)
client = ArtemisClient(default_profile="flash") client = ArtemisClient(default_profile="flash")
# 执行自然语言任务 # 执行自然语言端到端测试用例
result = await client.run("打开计算器,计算 (128 * 45) + 330 的结果") result = await client.run(
"打开系统设置,进入『电池』页面,验证是否正常显示电量百分比,确认页面无异常报错弹窗。"
)
print(f"执行状态: {result.status}") # 结构化断言与执行追溯
print(f"消耗步数: {result.turns}") assert result.status == "SUCCESS", f"测试执行失败: {result.failure_reason}"
print(f"Trace ID: {result.trace_id}") print(f"✅ 测试通过!耗时步数: {result.turns} | Trace ID: {result.trace_id}")
if __name__ == "__main__": if __name__ == "__main__":
@@ -157,20 +195,20 @@ if __name__ == "__main__":
<sub>💡 <b>控制台功能布局</b>:<b>① 顶栏视图切换</b>(主页与工作区) · <b>② 运行模式与录屏回放</b>(Flash/Pro 状态与视频回放) · <b>③ 实时感知推理流</b>(动作分解、点击坐标与结构化总结) · <b>④ 自然语言下发胶囊</b>(自然语言驱动真机) · <b>⑤ 任务队列看板</b>(状态流转与历史回溯)</sub> <sub>💡 <b>控制台功能布局</b>:<b>① 顶栏视图切换</b>(主页与工作区) · <b>② 运行模式与录屏回放</b>(Flash/Pro 状态与视频回放) · <b>③ 实时感知推理流</b>(动作分解、点击坐标与结构化总结) · <b>④ 自然语言下发胶囊</b>(自然语言驱动真机) · <b>⑤ 任务队列看板</b>(状态流转与历史回溯)</sub>
</p> </p>
* 🖥️ **Web 可视化控制台 (`artemis ui`)**:集成设备实时投屏与交互面板,支持通过自然语言下发任务,实时观测推理过程、操作轨迹与状态回放; * 🖥️ **Web 可视化测试控制台 (`artemis ui`)**:集成设备实时投屏与交互面板,支持通过自然语言下发测试用例,实时观测推理步骤、操作轨迹、截图留存与异常状态回放;
* 🔌 **MCP 协议集成 (IDE 联动)**:作为标准 MCP 服务器无缝接入 **Cursor、Claude Code、OpenClaw** 等环境,直接在 IDE 中调用移动端真机操作能力; * 🔌 **MCP 协议集成 (AI IDE 协同)**:作为标准 MCP 服务器无缝接入 **Antigravity、Claude Code、Windsurf** 等开发环境,在 IDE 中直接驱动真机完成自动化测试与 Bug 复现验证;
* 💻 **命令行工具 (`artemis run`)**:支持通过 CLI 直接执行自动化任务、测试用例或 AndroidWorld 基准评测,提供高保真结构化终端输出; * 💻 **命令行工具 (`artemis run`)**:支持通过 CLI 直接执行自动化用例、稳定性巡检或 AndroidWorld 基准评测,提供高保真结构化终端输出;
* 🐍 **Python SDK**:作为标准 Python 库集成至现有自动化测试框架或流水线,提供基于 Pydantic 的强类型结构化输出。 * 🐍 **Python SDK**:作为标准 Python 库无缝集成至现有自动化测试框架(如 pytest)或 CI/CD 流水线,提供基于 Pydantic 的强类型结构化结果与断言支持。
## 📊 方案横向对比 ## 📊 方案横向对比
| 场景 / 维度 | 传统自动化框架 | 常见屏幕 Agent | **ARTEMIS ☕** | | 评估维度 | 传统自动化测试框架 (Appium / Maestro) | 常见移动端 VLM Agent | **ARTEMIS ☕ (下一代 AI 测试平台)** |
| :--- | :--- | :--- | :--- | | :--- | :--- | :--- | :--- |
| **自然语言指令支持** | ❌ 仅支持硬编码脚本,无语义理解 | ⚠️ 响应较慢,单步交互通常需 20-30 秒 | ⚡ **高效执行**:Flash 模式下单步最快约 3-5 秒,交互流畅 | | **用例编写与维护** | ❌ 强依赖 XPath/ID,UI 微调即导致大面积报错 | ⚠️ 缺乏工程化封装,执行不可靠,无法作为用例复用 | 🧪 **零脚本维护**:自然语言直接定义用例,无惧 UI 漂移与改版 |
| **UI 改版适应能力** | ❌ 元素变更易导致脚本执行中断 | ⚠️ 依赖绝对坐标,存在位置偏移风险 | 🎯 **多模态语义定位**:结合文本、图标与层次结构,自适应 UI 变化 | | **执行延迟与吞吐** | ⚡ 脚本执行快,但编写与定位调试耗时极长 | ❌ 单步推理动辄 20-30 秒,无法满足回归测试要求 | ⚡ **高吞吐低延迟**:首创乐观异步流水线,单步仅需 3-5 秒 |
| **异常弹窗处理** | ❌ 弹窗遮挡易导致定位失败 | ❌ 面对未知弹窗易陷入无效循环 | 🩹 **执行前校验与自愈**:操作前确认目标状态,自动识别并处理干扰弹窗 | | **异常遮挡与自愈** | ❌ 遇到系统权限弹窗或意外通知时直接中断报错 | ❌ 遇到非预期弹窗极易卡死或陷入无意义循环 | 🛡️ **执行前校验与自愈**:Safety Net 自动拦截并处理干扰弹窗 |
| **动态与视频场景** | ❌ 仅支持静态等待 (sleep),无动态感知 | ❌ 无法识别视频与动态流媒体内容 | ⏱️ **流媒体分析**:支持视频流分析、倒计时识别与预测性连点 | | **缺陷诊断与多媒体** | ❌ 仅支持静态等待,难以对视频流/动效做自动化验证 | ❌ 仅看静态截图,无法获取底层日志及系统状态 | 🐞 **深层诊断**:支持流媒体分析与 **Logcat 崩溃堆栈抓取** |
| **开发环境集成** | ❌ 独立运行,难以直接与 AI IDE 联动 | ❌ 通常仅提供独立网页,集成成本高 | 🔌 **原生 MCP 支持**:可在 Cursor、Claude Desktop 等环境中直接调用 | | **开发环境集成** | ❌ 独立运行,发现问题后需人工抓日志提单 | ❌ 多为独立网页 Demo,难以融入研发工具链 | 🔌 **原生 MCP & SDK**:在 Antigravity/Claude Code 中直接驱动真机测 Bug |
<a id="benchmarks"></a> <a id="benchmarks"></a>
<a id="基准评测"></a> <a id="基准评测"></a>
@@ -232,7 +270,7 @@ if __name__ == "__main__":
- [x] **全链路乐观异步流水线**:主流程轻量化,后台并发执行记忆压缩与断言校验。 - [x] **全链路乐观异步流水线**:主流程轻量化,后台并发执行记忆压缩与断言校验。
- [x] **Safety Net 执行前校验**:动作下发前双层核验与预测性链式执行。 - [x] **Safety Net 执行前校验**:动作下发前双层核验与预测性链式执行。
- [x] **时间敏感多媒体处理**:集成 `scrcpy` 与 `ffmpeg` 分析全屏视频与音频流。 - [x] **时间敏感多媒体处理**:集成 `scrcpy` 与 `ffmpeg` 分析全屏视频与音频流。
- [x] **原生 MCP 服务支持**:支持与 Cursor、Claude Desktop 等工具无缝集成。 - [x] **原生 MCP 服务支持**:支持与 Antigravity、Claude Desktop 等工具无缝集成。
- [x] **Web 可视化控制台**:提供设备投屏、交互演练与轨迹回溯。 - [x] **Web 可视化控制台**:提供设备投屏、交互演练与轨迹回溯。
- [x] **AndroidWorld SOTA**:达到 99%+ 的任务完成率。 - [x] **AndroidWorld SOTA**:达到 99%+ 的任务完成率。
- [ ] **跨平台扩展**:探索 iOS 与 Web 端的定位与操作能力。 - [ ] **跨平台扩展**:探索 iOS 与 Web 端的定位与操作能力。
Binary file not shown.

Before

Width:  |  Height:  |  Size: 189 KiB

After

Width:  |  Height:  |  Size: 205 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 189 KiB

After

Width:  |  Height:  |  Size: 205 KiB

+99 -23
View File
@@ -154,9 +154,23 @@ def create_banner_html():
filter: drop-shadow(0 0 35px rgba(66, 133, 244, 0.4)); filter: drop-shadow(0 0 35px rgba(66, 133, 244, 0.4));
} }
.brand-acronym {
font-size: 13px;
font-weight: 600;
color: #94A3B8;
letter-spacing: 0.015em;
font-family: 'Google Sans Mono', 'Roboto Mono', monospace;
margin-top: -3px;
}
.brand-acronym span {
color: #60A5FA;
font-weight: 800;
}
/* Clean Single Subtitle (Preserving Android) */ /* Clean Single Subtitle (Preserving Android) */
.brand-tagline { .brand-tagline {
font-size: 24px; font-size: 23px;
font-weight: 700; font-weight: 700;
line-height: 1.35; line-height: 1.35;
color: #F8FAFC; color: #F8FAFC;
@@ -216,7 +230,8 @@ def create_banner_html():
} }
.prompt-text { .prompt-text {
font-size: 13.5px; font-size: 12px;
line-height: 1.35;
font-family: 'Google Sans Mono', 'Roboto Mono', monospace; font-family: 'Google Sans Mono', 'Roboto Mono', monospace;
color: #E2E8F0; color: #E2E8F0;
font-weight: 500; font-weight: 500;
@@ -379,6 +394,46 @@ def create_banner_html():
font-family: 'Google Sans Mono', 'Roboto Mono', monospace; font-family: 'Google Sans Mono', 'Roboto Mono', monospace;
} }
/* Host Telemetry Metrics Panel in Console */
.metrics-panel {
background: rgba(15, 23, 42, 0.85);
border: 1px solid rgba(255, 255, 255, 0.08);
border-radius: 8px;
padding: 7px 10px;
display: flex;
flex-direction: column;
gap: 4px;
margin-top: 1px;
}
.metric-row {
display: flex;
align-items: center;
justify-content: space-between;
font-family: 'Google Sans Mono', 'Roboto Mono', monospace;
font-size: 9.5px;
}
.metric-label {
color: #94A3B8;
}
.metric-value {
color: #F1F5F9;
font-weight: 700;
}
.metric-value.cpu-highlight {
color: #34D399;
background: rgba(52, 211, 153, 0.15);
padding: 1px 5px;
border-radius: 3px;
}
.metric-value.status-ok {
color: #60A5FA;
}
/* Connected Mobile Phone Mockup */ /* Connected Mobile Phone Mockup */
.phone-frame { .phone-frame {
position: relative; position: relative;
@@ -657,9 +712,14 @@ def create_banner_html():
<h1 class="brand-title">ARTEMIS</h1> <h1 class="brand-title">ARTEMIS</h1>
</div> </div>
<!-- Write out what ARTEMIS stands for -->
<div class="brand-acronym">
<span>A</span>utonomous <span>R</span>eal-time <span>T</span>esting, <span>E</span>xploration &amp; <span>M</span>obile <span>I</span>nteraction <span>S</span>ystem
</div>
<!-- Clear, Crisp Subtitle --> <!-- Clear, Crisp Subtitle -->
<div class="brand-tagline"> <div class="brand-tagline">
Control Any Android App with <span>Autonomous AI</span> Test Any Pixel / Android App with <span>Autonomous AI</span>
</div> </div>
<!-- User Prompt Input Example --> <!-- User Prompt Input Example -->
@@ -672,7 +732,7 @@ def create_banner_html():
</div> </div>
</div> </div>
<div class="prompt-text"> <div class="prompt-text">
<span>&gt;</span> Order an iced caramel latte with oat milk <span>&gt;</span> Connect to Bluetooth earbuds, play a song, and capture CPU usage metrics
</div> </div>
</div> </div>
</div> </div>
@@ -685,70 +745,86 @@ def create_banner_html():
<div class="console-header"> <div class="console-header">
<div class="console-title-group"> <div class="console-title-group">
<div class="console-dot"></div> <div class="console-dot"></div>
<span class="console-title">Agent Execution</span> <span class="console-title">Host Agent Console</span>
</div> </div>
<span class="console-badge">STEP 3 / 4</span> <span class="console-badge">MONITORING</span>
</div> </div>
<div class="step-list"> <div class="step-list">
<div class="step-item"> <div class="step-item">
<span class="step-icon-done">✓</span> <span class="step-icon-done">✓</span>
<span class="step-text-done">1. Open app & search catalog</span> <span class="step-text-done">1. Connect to Bluetooth earbuds</span>
</div> </div>
<div class="step-item"> <div class="step-item">
<span class="step-icon-done">✓</span> <span class="step-icon-done">✓</span>
<span class="step-text-done">2. Select item & customize</span> <span class="step-text-done">2. Launch music player & play song</span>
</div> </div>
<!-- Active Step --> <!-- Active Step -->
<div class="step-active-card"> <div class="step-active-card">
<div class="step-active-header"> <div class="step-active-header">
<span class="step-active-title">▶ 3. Tap "Place Order"</span> <span class="step-active-title">▶ 3. Capture CPU usage metrics</span>
<span class="step-active-time">140ms</span> <span class="step-active-time">Active</span>
</div> </div>
<div class="step-active-desc"> <div class="step-active-desc">
Vision target grounded at <span>(540, 1680)</span> ADB polling top -n 1 <span>(pid: 14208)</span>
</div>
</div>
<!-- Host Telemetry Metrics Box -->
<div class="metrics-panel">
<div class="metric-row">
<span class="metric-label">Target App CPU</span>
<span class="metric-value cpu-highlight">4.2%</span>
</div>
<div class="metric-row">
<span class="metric-label">Bluetooth Audio</span>
<span class="metric-value status-ok">● Connected</span>
</div>
<div class="metric-row">
<span class="metric-label">Sample Rate</span>
<span class="metric-value">48kHz / A2DP</span>
</div> </div>
</div> </div>
<div class="step-pending"> <div class="step-pending">
<span>○</span> <span>○</span>
<span>4. Verify receipt & return</span> <span>4. Assert CPU &lt; 5.0% threshold</span>
</div> </div>
</div> </div>
</div> </div>
<!-- Real Connected Phone Executing the Tap --> <!-- Real Connected Phone Displaying Actual App UI -->
<div class="phone-frame"> <div class="phone-frame">
<div class="phone-screen"> <div class="phone-screen">
<div class="phone-island"></div> <div class="phone-island"></div>
<div class="screen-header"> <div class="screen-header">
<span class="screen-title">Food & Drink</span> <span class="screen-title">Now Playing</span>
<span class="device-tag">● ANDROID</span> <span class="device-tag">● PIXEL LAB</span>
</div> </div>
<div class="mock-item-card"> <div class="mock-item-card">
<div class="item-thumb">☕</div> <div class="item-thumb">🎵</div>
<div class="item-info"> <div class="item-info">
<span class="item-title">Iced Caramel Latte</span> <span class="item-title">Coldplay - Yellow</span>
<span class="item-sub">Double Shot · Oat Milk</span> <span class="item-sub">Output: Pixel Buds Pro</span>
</div> </div>
<span class="item-price">$5.40</span> <span class="item-price">Playing ▶</span>
</div> </div>
<div class="mock-secondary-card"> <div class="mock-secondary-card">
<span class="secondary-text">Delivery Address</span> <span class="secondary-text">Active Audio Output Device</span>
<span class="secondary-val">Desk #42</span> <span class="secondary-val">🎧 Pixel Buds Pro (A2DP)</span>
</div> </div>
<div class="action-target-wrapper"> <div class="action-target-wrapper">
<!-- AI Grounding Box on Target Button --> <!-- AI Grounding Box on Real Mobile Target -->
<div class="grounding-box"> <div class="grounding-box">
<div class="grounding-tag">🎯 GROUNDED (0.99)</div> <div class="grounding-tag">🎯 GROUNDED (0.99)</div>
</div> </div>
<div class="mock-btn"> <div class="mock-btn">
Place Order ($5.40) Media Output Controls
</div> </div>
<!-- Tap Ripple --> <!-- Tap Ripple -->
<div class="tap-point"> <div class="tap-point">
Binary file not shown.

After

Width:  |  Height:  |  Size: 260 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 674 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 732 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 728 KiB