diff --git a/README.md b/README.md index 7fbca12..042c704 100644 --- a/README.md +++ b/README.md @@ -265,38 +265,14 @@ Evaluated on [AndroidWorld](https://github.com/google-research/android_world) ## 🚀 How ARTEMIS is Architected -* ⚡ **Optimistic Asynchronous Pipeline**: The front-facing loop responds in milliseconds, while memory pruning and assertion verification run concurrently in the background without blocking execution; -* 🛡️ **Safety Net Pre-Execution Gate**: Dual-layer pre-check validates target availability milliseconds before action dispatch, instantly intercepting unexpected popups to eliminate blind clicks; -* ⏱️ **Time-Sensitive Speculative Chaining**: Overcomes LLM inference latency for transient UI elements (e.g. video fullscreen) by predicting target coordinates and executing rapid chained taps. - -
-🔍 Click to expand: Architecture Deep Dive & Pipeline Diagram - -
- -### 1. ⚡ Optimistic Asynchronous Pipeline -* **Status Quo & Pain Points**: Conventional mobile agents rely on a **fully synchronous blocking model** — every single action must wait sequentially for the LLM to prune historical context, check milestone assertions, and audit long-term plans. This inflates per-step latency to 20–40 seconds, creating a sluggish user experience. -* **Artemis's Architectural Solution**: Inspired by **Optimistic Concurrency Control (OCC) and Snapshot Isolation** in database systems, Artemis completely decouples the main execution loop from heavy auxiliary computation: - * **High-Throughput Main Loop**: The front-facing execution path is strictly narrowed to a high-speed "Perception → Decision → Safety Gate → Execution" pipeline; - * **Background Concurrent Tasks**: Context token pruning, milestone checkers (`Checker`), and planner validations (`Planner`) run dynamically in parallel without halting device interaction; - * **Snapshot Isolation & Rollback**: The agent optimistically charges forward. If background verification detects a deviation, Artemis instantly **rolls back via state snapshots (Rollback)** and injects self-healing feedback. +* 🛡️ **Pre-Touch Pixel Gate & Speculative Chaining**: Eliminates "silent misclicks" from inference latency race conditions. Milliseconds before dispatch, a local UI guard intercepts unexpected dialogs (0 tokens, 0 cloud wait), a Micro-ROI gate verifies target stability, and speculative chained taps hit transient UI (e.g. auto-fading video controls) before they expire; +* 🎯 **Three-Layer Progressive Grounding Engine**: Fuses local OCR with accessibility hierarchies (~150ms, 0 tokens) to drive 85%+ of standard actions via drift-free numeric indices, gracefully falling back to spatial vision models for custom Canvas/Compose/Flutter UI and sandboxed CV probing for subtle pixel states; +* 🧠 **Elastic Dual Engine with In-Flight Context Compactor**: Seamlessly toggles between high-throughput reactive CI loops (Flash Mode, 3–5s/step) and multi-step cognitive state graphs (Pro Mode), using background visual deltas and DOM pruning to slash token consumption by >70% for 10+ hours of continuous, unattended soak testing.

- Artemis Optimistic Async Pipeline Architecture + ARTEMIS System Architecture Diagram

-### 2. 🛡️ Pre-Execution Safety Net & Time-Sensitive Speculative Chaining -* **Status Quo & The Transient UI Dilemma**: - * Mobile applications feature numerous **time-sensitive transient UI controls** (e.g. video fullscreen: tapping the screen wakes up the floating overlay, followed immediately by tapping the fullscreen icon). - * Traditional agents tap the screen to reveal controls, take a new screenshot, and wait 3–15 seconds for LLM reasoning. By the time the click is dispatched, **the player controls have already auto-faded away** — causing the click to strike the underlying video, triggering an endless loop of accidental pausing and waking. -* **Artemis's Architectural Solution**: - * **Speculative Chained Actions**: Upon recognizing time-sensitive dependencies, the agent dispatches compound chained actions (Wakeup → Millisecond Chained Tap) to hit the target within its transient visibility window; - * **Two Pillars Ensuring Reliable Chaining**: - 1. **Historical UI Prior Prediction**: Predicts the target control's wake-up coordinates based on prior interaction history and app layout heuristics; - 2. **Safety Net Pre-Execution Gate**: Milliseconds before the chained action lands, the Safety Net instantly verifies that the target control was successfully revealed at the expected coordinates. If the wakeup failed or an unexpected popup intercepted it, execution is **immediately blocked to prevent blind clicks**. - -
- ## ⚡ Execution Profiles: Flash vs. Pro | Feature / Dimension | ⚡ **ARTEMIS Flash** (`--profile flash`) | 🧠 **ARTEMIS Pro** (`--profile pro`) | diff --git a/README_CN.md b/README_CN.md index 9297857..abc5ebf 100644 --- a/README_CN.md +++ b/README_CN.md @@ -268,38 +268,14 @@ if __name__ == "__main__": ## 🚀 ARTEMIS 是如何建构的 -* ⚡ **全链路乐观异步 (Optimistic Async)**：主交互循环毫秒级响应下发，记忆修剪与阶段断言全部在后台并发静默运行，告别死等； -* 🛡️ **Safety Net 动作守门**：动作下发前毫秒级双重校验目标，突发弹窗即刻拦截自愈，绝不盲点空气； -* ⏱️ **时效敏感预测性连点 (Speculative Chaining)**：针对视频全屏等瞬态交互，结合历史先验预测坐标并极速连点，彻底解决大模型延迟导致控件消失的问题。 - -
-🔍 点击展开：核心架构设计细节与流水线流程图 - -
- -### 1. ⚡ 全链路乐观异步流水线 (Optimistic Asynchronous Pipeline) -* **现有方案与问题**：传统移动端智能体普遍采用**全同步阻塞模型** —— 每一步交互都必须同步串行等待大模型完成历史上下文修剪、阶段性断言检查与长程规划审计。这导致主交互循环极度臃肿，单步延迟动辄 20–40 秒，交互体验迟缓； -* **Artemis 的架构解法**：借鉴数据库的**乐观并发控制 (OCC) 与快照隔离**思想，将主干执行流与重度计算彻底解耦： - * **高吞吐前台主循环**：前台严格收敛为「感知 → 决策 → 守门 → 执行」极简流水线，单步响应压缩至秒级； - * **后台并发异步处理**：上下文修剪、里程碑断言校验（Checker）、全局规划审计（Planner）全部在后台并发静默运行，零阻塞前台； - * **快照隔离与无损回滚**：前台以前瞻视角乐观执行；一旦后台异步断言检测到状态偏离，系统基于快照机制**一键无损回滚 (Rollback)** 并注入纠偏反馈，兼得极低延迟与可靠自愈。 +* 🛡️ **Pre-Touch 触控前像素守门与预测性连点**：彻底杜绝大模型推理延迟造成的「静默误触」。动作下发前毫秒级拦截意外弹窗（0 Token、0 云端等待），Micro-ROI 局部校验目标稳定性；针对视频全屏等瞬态控件，结合历史先验毫秒级连击唤醒，在淡出窗口期内瞬时闭环； +* 🎯 **三层递进式目标定位引擎 (Progressive Grounding)**：首层通过本地 OCR 与无障碍树几何融合（~150ms、0 Token），以无偏移数字索引承载 85%+ 常见交互；二层对 Flutter/Compose/Canvas 等自绘组件自动回退至空间视觉模型，三层辅以沙箱 CV 探针核验细微像素状态； +* 🧠 **弹性双执行引擎与运行时上下文动态压缩**：兼顾 3–5 秒秒级响应的 CI 高吞吐回归（Flash 模式）与深度长程认知状态图探索（Pro 模式），后台异步生成视觉增量并剪除冗余 DOM，长程任务 Token 消耗锐降 70%+，支持 10+ 小时无人值守稳定性压测。

- Artemis 乐观异步流水线架构 + ARTEMIS 架构系统示意图

-### 2. 🛡️ 动作守门机制 (Safety Net) 与时效敏感预测性连点 (Speculative Chaining) -* **现有方案与时效困境**： - * 移动端存在大量**时效敏感型交互**（如视频全屏播放：必须先轻点屏幕唤醒浮层控件，再点击全屏按钮）。 - * 传统智能体在点击屏幕唤醒控件后，重新截屏并等待大模型推理决策耗时数秒；而此时**播放器控件早已自动淡出隐藏**，导致后续点击直接打在视频画面上，陷入“唤醒 → 控件消失 → 误触暂停 → 再次唤醒”的死循环。 -* **Artemis 的架构解法**： - * **预测性链式连点 (Speculative Chaining)**：智能体在识别此类时序依赖后，直接下发复合连击指令（唤醒 → 毫秒级连击全屏），在控件有效窗口期内瞬时完成交互； - * **保障连点绝对可靠的两大支柱**： - 1. **历史先验位置预测**：基于历史交互轨迹与应用 UI 先验知识，精准预判控件唤醒后的目标坐标； - 2. **Safety Net 毫秒级拦截守门**：在连击动作落下的前一刻，瞬间核验目标控件是否已被成功唤醒并位于预判位置。若唤醒失败或遭遇意外弹窗遮挡，**立刻前置拦截熔断，杜绝盲点误触**。 - -
- ## ⚡ 运行模式对比：Flash vs. Pro | 特性对比 | ⚡ **ARTEMIS Flash** (`--profile flash`) | 🧠 **ARTEMIS Pro** (`--profile pro`) | diff --git a/docs/assets/androidworld_benchmark_comparison.png b/docs/assets/androidworld_benchmark_comparison.png index 7794f2e..8d12381 100644 Binary files a/docs/assets/androidworld_benchmark_comparison.png and b/docs/assets/androidworld_benchmark_comparison.png differ diff --git a/docs/assets/artemis_architecture_diagram.png b/docs/assets/artemis_architecture_diagram.png new file mode 100644 index 0000000..6847561 Binary files /dev/null and b/docs/assets/artemis_architecture_diagram.png differ