docs: streamline execution profiles and refocus roadmap

- Remove redundant head-to-head comparison section
- Simplify Flash vs. Pro profiles with precise latencies, benefits, and limitations
- Refocus roadmap on future milestones prioritizing Android Studio integration, followed by iOS
This commit is contained in:
somew1nd committed 2026-08-25 21:21:42 -07:00
1 parent bb16991f2c
commit 06643e955b
2 files changed
+17 -55

No files matched your search

+8 -27
View File
@@ -244,16 +244,6 @@ if __name__ == "__main__":
* **Developer CLI (`uv run artemis run`)**: Direct terminal execution for automated test cases, exploratory stability inspection, or AndroidWorld benchmarks with high-fidelity structured terminal output; * **Developer CLI (`uv run artemis run`)**: Direct terminal execution for automated test cases, exploratory stability inspection, or AndroidWorld benchmarks with high-fidelity structured terminal output;
* **Python SDK**: Integrates as a standard Python library into existing automated testing frameworks (e.g., pytest) or CI/CD pipelines with strongly typed Pydantic structured outputs and assertion support. * **Python SDK**: Integrates as a standard Python library into existing automated testing frameworks (e.g., pytest) or CI/CD pipelines with strongly typed Pydantic structured outputs and assertion support.
## Head-to-Head Comparison
| Evaluation Dimension | Traditional Test Automation (Appium / Maestro) | Generic Mobile VLM Agents | **ARTEMIS (Next-Gen AI Testing)** |
| :--- | :--- | :--- | :--- |
| **Test Case Maintenance** | ❌ Fragile XPath/ID dependencies; UI changes cause test failures | ⚠️ Unreliable execution; cannot be reused as regression tests | **Zero Maintenance**: Natural language test cases resilient to UI drift & redesigns |
| **Execution Latency & Throughput** | ⚡ Fast script execution, but extreme setup and locator debugging costs | ❌ Sluggish 20–30s per step; too slow for regression testing | **High Throughput**: Optimistic Async Pipeline runs at 3–5s per step |
| **Popup Resilience & Self-Healing** | ❌ System popups or permissions immediately crash the script | ❌ Easily gets stuck or loops endlessly on unexpected dialogs | **Pre-Execution Safety Net**: Automatically intercepts and clears interfering popups |
| **Diagnostics & Multimedia** | ❌ Blind static waits (sleep); cannot assert dynamic video/animations | ❌ Static screenshots only; no system logs or underlying state | **Deep Diagnostics**: Live video stream analysis & **Logcat crash stack capture** |
| **Dev Environment Integration** | ❌ Standalone runner; requires manual log collection upon failure | ❌ Isolated web demos; difficult to embed into dev pipelines | **Native MCP & SDK**: Drive physical test devices and debug directly inside Antigravity / Claude Code |
<a id="benchmarks"></a> <a id="benchmarks"></a>
## Benchmarks: AndroidWorld (SOTA 99%+) ## Benchmarks: AndroidWorld (SOTA 99%+)
@@ -275,26 +265,17 @@ Evaluated on [AndroidWorld](https://github.com/google-research/android_world)
## Execution Profiles: Flash vs. Pro ## Execution Profiles: Flash vs. Pro
| Feature / Dimension | **ARTEMIS Flash** (`--profile flash`) | **ARTEMIS Pro** (`--profile pro`) | ARTEMIS supports two execution profiles tailored for different automation requirements:
| :--- | :--- | :--- |
| **Design Purpose** | **Lightweight & Fast**: Direct deterministic UI actions | **Deep Reasoning**: Multi-step planning & complex self-healing | * **Flash Profile (`--profile flash`)**: Fast and token-efficient reactive loop (~3–5s per step). Ideal for routine, deterministic UI tasks within 25–30 steps. *Limitations*: Does not support long-term state monitoring, video stream analysis, or multi-step failure self-healing.
| **Step Latency** | **3–5 seconds** / step | **15–30 seconds** / turn (includes planning & verification) | * **Pro Profile (`--profile pro`)**: Our most capable model architecture (~15–40s per step). Powered by a multi-agent graph with planning, visual verification, and automated recovery. Capable of handling 100+ step long-horizon workflows, continuous device state monitoring, and multimedia video analysis.
| **Task Duration** | Minute-level short tasks (typically ≤35 steps) | Runs stably for **10+ hours**; monitoring tasks support **24/7 execution** |
| **Best Suited For** | Well-defined standard UI tasks | Complex cross-app workflows, failure self-healing, continuous monitoring |
| **Self-Healing** | Local step retries | **Safety Net Gate** + dialog suppression + crash recovery + snapshot rollback |
| **Media Analysis** | **Basic visual perception** + High-Speed OCR | Full `scrcpy`/`ffmpeg` video stream analysis + Logcat logs |
## Roadmap ## Roadmap
- [x] **Optimistic Asynchronous Pipeline**: Ultra-lean main loop + background context compression & milestone checks. - [ ] **Android Studio Integration**: Native IDE plugin and workflow integration to enable in-editor debugging, test recording, and automated device control directly within Android Studio.
- [x] **Pre-Execution Safety Net**: Millisecond pre-check gate & speculative chained actions. - [ ] **iOS Platform Expansion**: Extending multimodal perception and mobile automation to iOS devices and simulators.
- [x] **Time-Sensitive Media Tasks**: Fullscreen video and audio stream analysis with `scrcpy` & `ffmpeg`. - [ ] **On-Device Lightweight VLMs**: Local execution with lightweight edge vision models for low-latency, privacy-first automation.
- [x] **Native MCP Server**: Seamless integration with tools like Antigravity and Claude Desktop. - [ ] **Real-time Duplex Voice Interaction**: Voice-driven task dispatch with real-time conversational control and interruption handling.
- [x] **Web Visual Console**: Live screen projection, interactive playground, and trajectory review.
- [x] **AndroidWorld SOTA**: Achieved 99%+ task completion rate.
- [ ] **Cross-Platform Extensions**: Exploring iOS and desktop Web perception and execution.
- [ ] **On-Device Lightweight VLMs**: Zero-cloud local execution with lightweight edge vision models.
- [ ] **Real-time Duplex Voice Mode**: Natural voice input with real-time interruption (barge-in) control.
## Community & Contributing ## Community & Contributing
+9 -28
View File
@@ -246,16 +246,6 @@ if __name__ == "__main__":
* **命令行工具 (`uv run artemis run`)**:支持通过 CLI 直接执行自动化用例、稳定性巡检或 AndroidWorld 基准评测,提供高保真结构化终端输出; * **命令行工具 (`uv run artemis run`)**:支持通过 CLI 直接执行自动化用例、稳定性巡检或 AndroidWorld 基准评测,提供高保真结构化终端输出;
* **Python SDK**:作为标准 Python 库无缝集成至现有自动化测试框架(如 pytest)或 CI/CD 流水线,提供基于 Pydantic 的强类型结构化结果与断言支持。 * **Python SDK**:作为标准 Python 库无缝集成至现有自动化测试框架(如 pytest)或 CI/CD 流水线,提供基于 Pydantic 的强类型结构化结果与断言支持。
## 方案横向对比
| 评估维度 | 传统自动化测试框架 (Appium / Maestro) | 常见移动端 VLM Agent | **ARTEMIS (下一代 AI 测试平台)** |
| :--- | :--- | :--- | :--- |
| **用例编写与维护** | ❌ 强依赖 XPath/ID,UI 微调即导致大面积报错 | ⚠️ 缺乏工程化封装,执行不可靠,无法作为用例复用 | 🧪 **零脚本维护**:自然语言直接定义用例,无惧 UI 漂移与改版 |
| **执行延迟与吞吐** | ⚡ 脚本执行快,但编写与定位调试耗时极长 | ❌ 单步推理动辄 20-30 秒,无法满足回归测试要求 | ⚡ **高吞吐低延迟**:首创乐观异步流水线,单步仅需 3-5 秒 |
| **异常遮挡与自愈** | ❌ 遇到系统权限弹窗或意外通知时直接中断报错 | ❌ 遇到非预期弹窗极易卡死或陷入无意义循环 | 🛡️ **执行前校验与自愈**:Safety Net 自动拦截并处理干扰弹窗 |
| **缺陷诊断与多媒体** | ❌ 仅支持静态等待,难以对视频流/动效做自动化验证 | ❌ 仅看静态截图,无法获取底层日志及系统状态 | 🐞 **深层诊断**:支持流媒体分析与 **Logcat 崩溃堆栈抓取** |
| **开发环境集成** | ❌ 独立运行,发现问题后需人工抓日志提单 | ❌ 多为独立网页 Demo,难以融入研发工具链 | 🔌 **原生 MCP & SDK**:在 Antigravity/Claude Code 中直接驱动真机测 Bug |
<a id="benchmarks"></a> <a id="benchmarks"></a>
<a id="基准评测"></a> <a id="基准评测"></a>
## 基准评测:AndroidWorld (SOTA 99%+) ## 基准评测:AndroidWorld (SOTA 99%+)
@@ -276,28 +266,19 @@ if __name__ == "__main__":
<img src="./docs/assets/artemis_architecture_diagram.png" alt="ARTEMIS 架构系统示意图" width="100%" /> <img src="./docs/assets/artemis_architecture_diagram.png" alt="ARTEMIS 架构系统示意图" width="100%" />
</p> </p>
## 运行模式对比:Flash vs. Pro ## 运行模式:Flash vs. Pro
| 特性对比 | **ARTEMIS Flash** (`--profile flash`) | **ARTEMIS Pro** (`--profile pro`) | ARTEMIS 提供两种运行模式以适应不同的自动化需求:
| :--- | :--- | :--- |
| **设计定位** | **轻量极速**:面向常规确定性操作 | **深度规划**:面向长流程与复杂自愈 | * **Flash 模式 (`--profile flash`)**:轻量极速且省 Token 的响应式循环(单步约 3–5 秒),适合 25–30 步以内的常规确定性 UI 操作。*局限性*:不支持长时间状态监控、视频流分析以及复杂的多步自愈恢复。
| **单步延迟** | **3–5 秒** / 步 | **15–30 秒** / 轮 (包含深度推理与全局校验) | * **Pro 模式 (`--profile pro`)**:最强大的模型架构(单步约 15–40 秒),基于多智能体图编排,具备自主规划、视觉校验与闭环自愈能力,支持 100+ 步的长程复杂业务流、设备持续状态监控与多媒体视频分析。
| **任务时长** | 分钟级短程任务(通常 ≤35 步) | 可稳定运行**超 10 小时**,监控任务可**全天候运行** |
| **适用场景** | 目标明确的常规 UI 操作 | 复杂的跨 App 长流程、故障自愈、持续状态观察 |
| **自愈机制** | 基础局部重试 | **Safety Net 校验** + 弹窗处理 + 崩溃恢复 + 快照回滚 |
| **多媒体分析** | 基础视觉感知 + 高速 OCR | 完整 `scrcpy`/`ffmpeg` 视频流分析 + Logcat 日志采集 |
## 路线图 ## 路线图
- [x] **全链路乐观异步流水线**:主流程轻量化,后台并发执行记忆压缩与断言校验。 - [ ] **Android Studio 深度集成**:推出官方 IDE 插件与协同工作流,支持在 Android Studio 内直接进行自动化测试、设备交互与断点调试。
- [x] **Safety Net 执行前校验**:动作下发前双层核验与预测性链式执行。 - [ ] **iOS 跨平台支持**:将视觉感知与自动化执行引擎拓展至 iOS 真机与模拟器。
- [x] **时间敏感多媒体处理**:集成 `scrcpy` 与 `ffmpeg` 分析全屏视频与音频流。 - [ ] **端侧轻量化模型**:支持离线运行的轻量级 Edge VLM,实现低延迟与隐私安全的本地自动化。
- [x] **原生 MCP 服务支持**:支持与 Antigravity、Claude Desktop 等工具无缝集成。 - [ ] **实时语音双工交互**:支持自然语音下发任务与实时打断(Barge-in)控制。
- [x] **Web 可视化控制台**:提供设备投屏、交互演练与轨迹回溯。
- [x] **AndroidWorld SOTA**:达到 99%+ 的任务完成率。
- [ ] **跨平台扩展**:探索 iOS 与 Web 端的定位与操作能力。
- [ ] **端侧轻量化模型**:支持在设备本地离线运行的轻量级 Edge VLM 模型。
- [ ] **实时语音双工交互**:支持自然语音输入与实时打断控制。
## 社区与贡献 ## 社区与贡献