+ 
+
+ ARTEMIS: Autonomous AI Assistant for Mobile Automation +
-[](https://www.python.org/downloads/) -[](LICENSE) -[](https://github.com/langchain-ai/langgraph) -[](https://ai.google.dev/) ++ ⚡ Drive real phones from Cursor & Claude Code • Cross-App Automation • Zero-Maintenance UI Testing • Flash Mode as Fast as 3-5s +
-ARTEMIS is a next-generation, production-grade autonomous agent and testing framework engineered for Android devices and emulators. Combining state-of-the-art multimodal Large Vision-Language Models (VLMs) with hierarchical closed-loop planning, resilient self-healing, and deep Android subsystem integration (ADB, UIAutomator2, accessibility trees), ARTEMIS enables autonomous UI navigation, robust automated testing, and intelligent device diagnostics. ++ English • + 中文文档 • + Quick Start • + MCP for IDEs • + Benchmarks • + Discord Community +
---- +
+ 
+ 
+ 
+ 
+ 
+
+ 
+
+ Live Demo: Setup driving routes and calculate total durations in Google Maps, then open YouTube to play a Coldplay song.
+
+
+ +ARTEMIS includes a native **Model Context Protocol (MCP)** server. Connect your real phone directly into AI IDEs: + +### 1. Generate MCP Config + +Run the built-in generator to produce ready-to-use JSON configuration: ```bash -# Install dependencies into a virtual environment -uv sync --dev +artemis mcp --generate-config cursor +# Or generate configs for all IDEs: +artemis mcp --generate-config all ``` -Or using standard `pip`: +### 2. Copy Config to Your IDE -```bash -python3 -m venv .venv -source .venv/bin/activate -pip install -e ".[dev]" +* **Cursor** (`.cursor/mcp.json` or Settings ➔ MCP Servers): +```json +{ + "mcpServers": { + "artemis": { + "command": "python", + "args": ["-m", "mcp_server"], + "cwd": "/path/to/artemis", + "env": { + "PYTHONUNBUFFERED": "1" + } + } + } +} ``` -### 3. Quick Setup (10 Seconds) - -Run the interactive setup wizard to configure your API key and connect your device: - -```bash -artemis init +* **Claude Desktop** (`claude_desktop_config.json`): +```json +{ + "mcpServers": { + "artemis": { + "command": "python", + "args": ["-m", "mcp_server"], + "cwd": "/path/to/artemis" + } + } +} ``` -Verify your environment and connected devices anytime with the doctor diagnostic tool: +### 3. Prompt Your Phone in the IDE Chat +In Cursor or Claude Code, simply prompt: +> 💬 *"Build the latest changes into an APK, install it on the connected device, open the login screen with a test account, verify if there are any unexpected popups after login, and return screenshots of the final page."* -```bash -artemis doctor -``` +
-*(Alternatively, copy `.env.example` to `.env` and set `GEMINI_API_KEY` for LLM reasoning and `OCR_API_KEY` for Vision perception.)*
+
+🔌 MCP Setup for Cursor / Claude Code / Windsurf (Click to expand)
+ ++ +ARTEMIS includes a native **Model Context Protocol (MCP)** server. Connect your real phone directly into AI IDEs: + +### 1. Generate MCP Config + +Run the built-in generator to produce ready-to-use JSON configuration: ```bash -# Install dependencies into a virtual environment -uv sync --dev +artemis mcp --generate-config cursor +# Or generate configs for all IDEs: +artemis mcp --generate-config all ``` -Or using standard `pip`: +### 2. Copy Config to Your IDE -```bash -python3 -m venv .venv -source .venv/bin/activate -pip install -e ".[dev]" +* **Cursor** (`.cursor/mcp.json` or Settings ➔ MCP Servers): +```json +{ + "mcpServers": { + "artemis": { + "command": "python", + "args": ["-m", "mcp_server"], + "cwd": "/path/to/artemis", + "env": { + "PYTHONUNBUFFERED": "1" + } + } + } +} ``` -### 3. Quick Setup (10 Seconds) - -Run the interactive setup wizard to configure your API key and connect your device: - -```bash -artemis init +* **Claude Desktop** (`claude_desktop_config.json`): +```json +{ + "mcpServers": { + "artemis": { + "command": "python", + "args": ["-m", "mcp_server"], + "cwd": "/path/to/artemis" + } + } +} ``` -Verify your environment and connected devices anytime with the doctor diagnostic tool: +### 3. Prompt Your Phone in the IDE Chat +In Cursor or Claude Code, simply prompt: +> 💬 *"Build the latest changes into an APK, install it on the connected device, open the login screen with a test account, verify if there are any unexpected popups after login, and return screenshots of the final page."* -```bash -artemis doctor -``` +
+
-## 💡 Quick Start - -### Command Line Interface (CLI) - -ARTEMIS installs a unified command-line tool `artemis`: - -```bash -# 1. Environment & Device Diagnostics -artemis doctor - -# 2. Interactive Setup Wizard -artemis init - -# 3. Basic task execution (ARTEMIS Pro by default) -artemis run "Open YouTube and search for Lo-Fi Hip Hop" - -# 4. Fast reactive profile (ARTEMIS Flash) -artemis run "Open Settings and verify Airplane Mode is turned off" --profile flash - -# 5. Execute with trace recording and structured output -artemis run "Open Maps, search for coffee shops, and extract the top 3 results" \ - --test-name "coffee_search_test" \ - --traces-path "./traces" \ - --output-description '{"places": "list of shop names", "ratings": "list of rating scores"}' - -# 6. Enable dynamic video recording analysis -artemis run "Play the latest video in the feed and verify playback" --with-video-recording-tools -``` - -### Python SDK - -Use the fluent SDK to integrate ARTEMIS into your test suites or automation workflows: +Embed the mobile automation engine into your Python workflows in just a few lines: ```python import asyncio -from pydantic import BaseModel, Field -from artemis.sdk import Agent - - -class VideoSearchResult(BaseModel): - query: str = Field(..., description="The search term used") - top_video_title: str = Field(..., description="Title of the first video result") - channel_name: str = Field(..., description="Channel name of the first video") +from artemis.interfaces.sdk import ArtemisClient async def main(): - # 1. Initialize the agent - agent = Agent() - await agent.init() + # Initialize client (choose "flash" for speed or "pro" for deep reasoning) + client = ArtemisClient(default_profile="flash") - try: - # 2. Build task specification - task = ( - agent.new_task("Search for 'Pixel 9 Pro' and get the first video details") - .with_name("youtube_search_verification") - .with_locked_app_package("com.google.android.youtube") - .with_output_format(VideoSearchResult) - .with_max_steps(50) - .build() - ) + # Run autonomous task + result = await client.run("Open calculator and compute (128 * 45) + 330") - # 3. Execute task - result: VideoSearchResult = await agent.run_task(request=task) - - if result: - print("Search Results:") - print(f"- Query: {result.query}") - print(f"- First Video: {result.top_video_title}") - print(f"- Channel: {result.channel_name}") - else: - print("Task did not complete successfully.") - - finally: - # 4. Clean up agent session & device connections - await agent.clean() + print(f"Status: {result.status}") + print(f"Turns taken: {result.turns}") + print(f"Trace ID: {result.trace_id}") if __name__ == "__main__": asyncio.run(main()) ``` ---- +
+
+## 🕹️ Usage Modes
+
+🐍 Python SDK Integration (Click to expand)
---- +-## 💡 Quick Start - -### Command Line Interface (CLI) - -ARTEMIS installs a unified command-line tool `artemis`: - -```bash -# 1. Environment & Device Diagnostics -artemis doctor - -# 2. Interactive Setup Wizard -artemis init - -# 3. Basic task execution (ARTEMIS Pro by default) -artemis run "Open YouTube and search for Lo-Fi Hip Hop" - -# 4. Fast reactive profile (ARTEMIS Flash) -artemis run "Open Settings and verify Airplane Mode is turned off" --profile flash - -# 5. Execute with trace recording and structured output -artemis run "Open Maps, search for coffee shops, and extract the top 3 results" \ - --test-name "coffee_search_test" \ - --traces-path "./traces" \ - --output-description '{"places": "list of shop names", "ratings": "list of rating scores"}' - -# 6. Enable dynamic video recording analysis -artemis run "Play the latest video in the feed and verify playback" --with-video-recording-tools -``` - -### Python SDK - -Use the fluent SDK to integrate ARTEMIS into your test suites or automation workflows: +Embed the mobile automation engine into your Python workflows in just a few lines: ```python import asyncio -from pydantic import BaseModel, Field -from artemis.sdk import Agent - - -class VideoSearchResult(BaseModel): - query: str = Field(..., description="The search term used") - top_video_title: str = Field(..., description="Title of the first video result") - channel_name: str = Field(..., description="Channel name of the first video") +from artemis.interfaces.sdk import ArtemisClient async def main(): - # 1. Initialize the agent - agent = Agent() - await agent.init() + # Initialize client (choose "flash" for speed or "pro" for deep reasoning) + client = ArtemisClient(default_profile="flash") - try: - # 2. Build task specification - task = ( - agent.new_task("Search for 'Pixel 9 Pro' and get the first video details") - .with_name("youtube_search_verification") - .with_locked_app_package("com.google.android.youtube") - .with_output_format(VideoSearchResult) - .with_max_steps(50) - .build() - ) + # Run autonomous task + result = await client.run("Open calculator and compute (128 * 45) + 330") - # 3. Execute task - result: VideoSearchResult = await agent.run_task(request=task) - - if result: - print("Search Results:") - print(f"- Query: {result.query}") - print(f"- First Video: {result.top_video_title}") - print(f"- Channel: {result.channel_name}") - else: - print("Task did not complete successfully.") - - finally: - # 4. Clean up agent session & device connections - await agent.clean() + print(f"Status: {result.status}") + print(f"Turns taken: {result.turns}") + print(f"Trace ID: {result.trace_id}") if __name__ == "__main__": asyncio.run(main()) ``` ---- +
+ 
+
+ 💡 Console Overview: ① View Switcher (Home / Workspace) · ② Model & Replay (Flash/Pro status & video replay) · ③ Live Agent Stream (Action perception, target coordinates & structured results) · ④ Prompt Dock (Natural language dispatch) · ⑤ Task Queue & Dashboard (Lifecycle & history)
+
+ 
+
+
+ +### 1. ⚡ Optimistic Asynchronous Pipeline +* **Status Quo & Pain Points**: Conventional mobile agents rely on a **fully synchronous blocking model** — every single action must wait sequentially for the LLM to prune historical context, check milestone assertions, and audit long-term plans. This inflates per-step latency to 20–40 seconds, creating a sluggish user experience. +* **Artemis's Architectural Solution**: Inspired by **Optimistic Concurrency Control (OCC) and Snapshot Isolation** in database systems, Artemis completely decouples the main execution loop from heavy auxiliary computation: + * **High-Throughput Main Loop**: The front-facing execution path is strictly narrowed to a high-speed "Perception → Decision → Safety Gate → Execution" pipeline; + * **Background Concurrent Tasks**: Context token pruning, milestone checkers (`Checker`), and planner validations (`Planner`) run dynamically in parallel without halting device interaction; + * **Snapshot Isolation & Rollback**: The agent optimistically charges forward. If background verification detects a deviation, Artemis instantly **rolls back via state snapshots (Rollback)** and injects self-healing feedback. + +
## ⚡ Execution Profiles: Flash vs. Pro
-| Feature | ARTEMIS Flash (`--profile flash`) | ARTEMIS Pro (`--profile default`) |
+| Feature / Dimension | ⚡ **ARTEMIS Flash** (`--profile flash`) | 🧠 **ARTEMIS Pro** (`--profile pro`) |
| :--- | :--- | :--- |
-| **Model Loop** | Single reactive agent (Observe-Think-Act) | Multi-node LangGraph orchestration |
-| **Best For** | Short, deterministic tasks (< 35 steps) | Deep, complex, multi-stage workflows |
-| **Step Latency** | ~3–5 seconds / step | ~15–30 seconds / turn (with planning & validation) |
-| **Planning** | Direct goal execution | Hierarchical multi-step plan decomposition |
-| **Diagnostics** | Basic error reporting | Deep diagnostics, log analysis & self-healing |
+| **Design Purpose** | **Lightweight & Fast**: Direct deterministic UI actions | **Deep Reasoning**: Multi-step planning & complex self-healing |
+| **Step Latency** | **3–5 seconds** / step | **15–30 seconds** / turn (includes planning & verification) |
+| **Task Duration** | Minute-level short tasks (typically ≤35 steps) | Runs stably for **10+ hours**; monitoring tasks support **24/7 execution** |
+| **Best Suited For** | Well-defined standard UI tasks | Complex cross-app workflows, failure self-healing, continuous monitoring |
+| **Self-Healing** | Local step retries | **Safety Net Gate** + dialog suppression + crash recovery + snapshot rollback |
+| **Media Analysis** | **Basic visual perception** + High-Speed OCR | Full `scrcpy`/`ffmpeg` video stream analysis + Logcat logs |
----
+## 🗺️ Roadmap
-## 🔌 Model Context Protocol (MCP) Integration
+- [x] **Optimistic Asynchronous Pipeline**: Ultra-lean main loop + background context compression & milestone checks.
+- [x] **Pre-Execution Safety Net**: Millisecond pre-check gate & speculative chained actions.
+- [x] **Time-Sensitive Media Tasks**: Fullscreen video and audio stream analysis with `scrcpy` & `ffmpeg`.
+- [x] **Native MCP Server**: Seamless integration with tools like Cursor and Claude Desktop.
+- [x] **Web Visual Console**: Live screen projection, interactive playground, and trajectory review.
+- [x] **AndroidWorld SOTA**: Achieved 99%+ task completion rate.
+- [ ] **Cross-Platform Extensions**: Exploring iOS and desktop Web perception and execution.
+- [ ] **On-Device Lightweight VLMs**: Zero-cloud local execution with lightweight edge vision models.
+- [ ] **Real-time Duplex Voice Mode**: Natural voice input with real-time interruption (barge-in) control.
-ARTEMIS includes built-in MCP servers and tools to allow AI IDEs (such as Claude Desktop, Cursor, and Gemini Jetski) to control Android devices directly:
+## 🤝 Community & Contributing
-| Tool Name | Description |
-| :--- | :--- |
-| `mobile_run_task` | Launches an autonomous background automation task on the connected device. |
-| `mobile_manage_task` | Inspects status, terminates tasks, or injects real-time steering instructions. |
-| `mobile_get_device_state` | Retrieves real-time screenshots or simplified accessibility UI trees. |
-| `mobile_inspect_trace` | Inspects step-by-step screenshots, visual action overlays, and execution logs. |
-
----
-
-## ⚙️ Configuration Guide
-
-All model configurations, thinking budgets, agent capabilities, and tool filtering are consolidated into a single self-explanatory configuration file: [`config/artemis.jsonc`](config/artemis.jsonc).
-
-```jsonc
-{
- // 1. Global Default Model (Applied across all sub-agents unless overridden)
- "default": {
- "provider": "google",
- "model": "gemini-3.6-flash",
- "thinking_level": "medium",
- "fallback": {
- "provider": "google",
- "model": "gemini-3.5-flash"
- }
- },
-
- // 2. Built-in Presets
- "presets": {
- "gemini-flagship": { "provider": "google", "model": "gemini-3.6-flash" },
- "openai-gpt4o": { "provider": "openai", "model": "gpt-4o" },
- "cost-saving": { "provider": "google", "model": "gemini-3.5-flash-lite" },
- "local-ollama": { "provider": "openai", "model": "llama3.2-vision" }
- },
-
- // 3. Agent & Sub-Agent Node Customizations (Optional overrides)
- "nodes": {
- // 🧠 Planner: High-level goal decomposition and strategic step planning
- "planner": { "thinking_level": "medium" },
-
- // 🎯 Operator: Screen perception, element interaction, and direct execution
- "operator": { "thinking_level": "medium", "include_thoughts": true },
-
- // 🛡️ Validator & Failure Analyzer: Self-healing and auto-recovery
- "validator_failure_analyzer": { "thinking_level": "medium" }
- },
-
- // 4. Agent Capabilities & Tool Filtering
- "agent": {
- "explorer_versions": { "operator": "flash", "validator": "flash" },
- "denylisted_tools": {
- "explorer": ["ask_image_processor", "get_ocr_list", "inspect_region"]
- }
- }
-}
-```
-
-> [!TIP]
-> **Zero-Friction Global Change**: To switch the entire platform from Gemini to GPT-4o or Claude, simply edit the `"default"` block once in `config/artemis.jsonc`. All 15 sub-agents automatically inherit the new model!
-
-### Using Non-Google LLMs
-
-ARTEMIS supports multi-provider LLM backends via standard LangChain integrations (Google Gemini, OpenAI, OpenRouter, xAI / Grok, or Google Vertex AI).
-
-#### Supported Providers & API Key Setup
-
-| Provider | Provider Key | Environment Variable | Base URL Override | Example Models |
-| :--- | :--- | :--- | :--- | :--- |
-| **Google Gemini** | `google` *(default)* | `GEMINI_API_KEY` | N/A | `gemini-3.6-flash`, `gemini-3.5-flash` |
-| **Google Vision OCR** | `ocr` | `OCR_API_KEY` | N/A | Google Cloud Vision Text Detection |
-| **Google Vertex AI** | `vertexai` | Google ADC (`gcloud auth application-default login`) | N/A | `gemini-3.6-flash` |
-| **OpenAI** | `openai` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | `gpt-4o`, `gpt-4o-mini`, `o3` |
-| **OpenRouter** | `openrouter` | `OPEN_ROUTER_API_KEY` | N/A | `anthropic/claude-3.5-sonnet`, `meta-llama/llama-3.3-70b-instruct` |
-| **xAI (Grok)** | `xai` | `XAI_API_KEY` | N/A | `grok-2-vision-1212`, `grok-beta` |
-
-> [!NOTE]
-> ARTEMIS uses multimodal visual grounding (screenshots) for UI navigation. When assigning non-Google models to vision-dependent agents (such as `operator`), ensure the selected model supports image/vision inputs.
-
-#### Switching Provider in `config/artemis.jsonc`
-
-To use OpenAI or Claude globally, simply change the `default` section in [`config/artemis.jsonc`](config/artemis.jsonc):
-
-```jsonc
-{
- "default": {
- "provider": "openai",
- "model": "gpt-4o",
- "fallback": {
- "provider": "openai",
- "model": "gpt-4o-mini"
- }
- }
-}
-```
-
-#### Custom / Local OpenAI-Compatible Endpoints (Ollama, vLLM, LocalAI)
-
-To connect ARTEMIS to local or self-hosted LLM endpoints (e.g., Ollama or vLLM), set `OPENAI_BASE_URL` in `.env` and specify `openai` as the provider in your configuration:
-
-```bash
-# Route OpenAI provider calls to local Ollama / vLLM endpoint
-OPENAI_API_KEY=ollama # dummy key if required by client
-OPENAI_BASE_URL=http://localhost:11434/v1
-```
-
----
-
-## 📊 Trace Inspection & Observability
-
-Every task execution can record rich execution artifacts stored in `./traces/🔍 Click to expand: Architecture Deep Dive & Pipeline Diagram
+ ++ +### 1. ⚡ Optimistic Asynchronous Pipeline +* **Status Quo & Pain Points**: Conventional mobile agents rely on a **fully synchronous blocking model** — every single action must wait sequentially for the LLM to prune historical context, check milestone assertions, and audit long-term plans. This inflates per-step latency to 20–40 seconds, creating a sluggish user experience. +* **Artemis's Architectural Solution**: Inspired by **Optimistic Concurrency Control (OCC) and Snapshot Isolation** in database systems, Artemis completely decouples the main execution loop from heavy auxiliary computation: + * **High-Throughput Main Loop**: The front-facing execution path is strictly narrowed to a high-speed "Perception → Decision → Safety Gate → Execution" pipeline; + * **Background Concurrent Tasks**: Context token pruning, milestone checkers (`Checker`), and planner validations (`Planner`) run dynamically in parallel without halting device interaction; + * **Snapshot Isolation & Rollback**: The agent optimistically charges forward. If background verification detects a deviation, Artemis instantly **rolls back via state snapshots (Rollback)** and injects self-healing feedback. + +
+ 
+
+
+
+ ARTEMIS: 全自动操控手机的 AI 个人助理 +
+ ++ ⚡ 让 Cursor / Claude Code 接管真机 • 跨 App 复杂自动化 • 零脚本维护 UI 测试 • Flash 模式单步最快 3-5 秒 +
+ ++ English • + 中文文档 • + 快速上手 • + MCP 接入 IDE • + 基准评测 • + Discord 社区 +
+ +
+
+
+
+
+
+
+
+
+ 实机演示：在 Google Maps 中设置驾车路线并计算总耗时，随后打开 YouTube 播放 Coldplay 的歌曲。
+
+
+ +ARTEMIS 内置原生 **Model Context Protocol (MCP)** 服务。只需将以下配置加入你的 IDE 配置文件中，即可在编写代码时直接驱动真机： + +### 1. 生成或查看配置 + +运行内置命令一键获取当前环境的完整配置 JSON： + +```bash +artemis mcp --generate-config cursor +# 或生成所有 IDE 配置： +artemis mcp --generate-config all +``` + +### 2. 复制配置到 IDE + +* **Cursor** (`.cursor/mcp.json` 或 设置 ➔ MCP Servers)： +```json +{ + "mcpServers": { + "artemis": { + "command": "python", + "args": ["-m", "mcp_server"], + "cwd": "/path/to/artemis", + "env": { + "PYTHONUNBUFFERED": "1" + } + } + } +} +``` + +* **Claude Desktop** (`claude_desktop_config.json`)： +```json +{ + "mcpServers": { + "artemis": { + "command": "python", + "args": ["-m", "mcp_server"], + "cwd": "/path/to/artemis" + } + } +} +``` + +### 3. 在 IDE 中体验真机协同 +在 Cursor / Claude Code 对话框中直接输入： +> 💬 *"请帮我把刚刚修改的代码编译成 APK 并安装到手机上，打开登录页面输入测试账号，验证登录后是否有异常弹窗，并把最终页面截图回传。"* + +
+
+
+🔌 接入 Cursor / Claude Code / Windsurf (MCP)（点击展开）
+ ++ +ARTEMIS 内置原生 **Model Context Protocol (MCP)** 服务。只需将以下配置加入你的 IDE 配置文件中，即可在编写代码时直接驱动真机： + +### 1. 生成或查看配置 + +运行内置命令一键获取当前环境的完整配置 JSON： + +```bash +artemis mcp --generate-config cursor +# 或生成所有 IDE 配置： +artemis mcp --generate-config all +``` + +### 2. 复制配置到 IDE + +* **Cursor** (`.cursor/mcp.json` 或 设置 ➔ MCP Servers)： +```json +{ + "mcpServers": { + "artemis": { + "command": "python", + "args": ["-m", "mcp_server"], + "cwd": "/path/to/artemis", + "env": { + "PYTHONUNBUFFERED": "1" + } + } + } +} +``` + +* **Claude Desktop** (`claude_desktop_config.json`)： +```json +{ + "mcpServers": { + "artemis": { + "command": "python", + "args": ["-m", "mcp_server"], + "cwd": "/path/to/artemis" + } + } +} +``` + +### 3. 在 IDE 中体验真机协同 +在 Cursor / Claude Code 对话框中直接输入： +> 💬 *"请帮我把刚刚修改的代码编译成 APK 并安装到手机上，打开登录页面输入测试账号，验证登录后是否有异常弹窗，并把最终页面截图回传。"* + +
+
+ +只需几行代码，将移动端自动化智能体无缝嵌入你的 Python 脚本与 CI/CD 流水线中： + +```python +import asyncio +from artemis.interfaces.sdk import ArtemisClient + + +async def main(): + # 初始化客户端（支持 "flash" 极速模式 或 "pro" 深度推理模式） + client = ArtemisClient(default_profile="flash") + + # 执行自然语言任务 + result = await client.run("打开计算器，计算 (128 * 45) + 330 的结果") + + print(f"执行状态: {result.status}") + print(f"消耗步数: {result.turns}") + print(f"Trace ID: {result.trace_id}") + + +if __name__ == "__main__": + asyncio.run(main()) +``` + +
+
+## 🕹️ 使用方式
+
+🐍 Python SDK 极简集成（点击展开）
+ ++ +只需几行代码，将移动端自动化智能体无缝嵌入你的 Python 脚本与 CI/CD 流水线中： + +```python +import asyncio +from artemis.interfaces.sdk import ArtemisClient + + +async def main(): + # 初始化客户端（支持 "flash" 极速模式 或 "pro" 深度推理模式） + client = ArtemisClient(default_profile="flash") + + # 执行自然语言任务 + result = await client.run("打开计算器，计算 (128 * 45) + 330 的结果") + + print(f"执行状态: {result.status}") + print(f"消耗步数: {result.turns}") + print(f"Trace ID: {result.trace_id}") + + +if __name__ == "__main__": + asyncio.run(main()) +``` + +
+ 
+
+ 💡 控制台功能布局：① 顶栏视图切换（主页与工作区） · ② 运行模式与录屏回放（Flash/Pro 状态与视频回放） · ③ 实时感知推理流（动作分解、点击坐标与结构化总结） · ④ 自然语言下发胶囊（自然语言驱动真机） · ⑤ 任务队列看板（状态流转与历史回溯）
+
+
+
+
+ +### 1. ⚡ 全链路乐观异步流水线 (Optimistic Asynchronous Pipeline) +* **现有方案与问题**：传统移动端智能体普遍采用**全同步阻塞模型** —— 每一步交互都必须同步串行等待大模型完成历史上下文修剪、阶段性断言检查与长程规划审计。这导致主交互循环极度臃肿，单步延迟动辄 20–40 秒，交互体验迟缓； +* **Artemis 的架构解法**：借鉴数据库的**乐观并发控制 (OCC) 与快照隔离**思想，将主干执行流与重度计算彻底解耦： + * **高吞吐前台主循环**：前台严格收敛为「感知 → 决策 → 守门 → 执行」极简流水线，单步响应压缩至秒级； + * **后台并发异步处理**：上下文修剪、里程碑断言校验（Checker）、全局规划审计（Planner）全部在后台并发静默运行，零阻塞前台； + * **快照隔离与无损回滚**：前台以前瞻视角乐观执行；一旦后台异步断言检测到状态偏离，系统基于快照机制**一键无损回滚 (Rollback)** 并注入纠偏反馈，兼得极低延迟与可靠自愈。 + +
+
+## ⚡ 运行模式对比：Flash vs. Pro
+
+| 特性对比 | ⚡ **ARTEMIS Flash** (`--profile flash`) | 🧠 **ARTEMIS Pro** (`--profile pro`) |
+| :--- | :--- | :--- |
+| **设计定位** | **轻量极速**：面向常规确定性操作 | **深度规划**：面向长流程与复杂自愈 |
+| **单步延迟** | **3–5 秒** / 步 | **15–30 秒** / 轮 (包含深度推理与全局校验) |
+| **任务时长** | 分钟级短程任务（通常 ≤35 步） | 可稳定运行**超 10 小时**，监控任务可**全天候运行** |
+| **适用场景** | 目标明确的常规 UI 操作 | 复杂的跨 App 长流程、故障自愈、持续状态观察 |
+| **自愈机制** | 基础局部重试 | **Safety Net 校验** + 弹窗处理 + 崩溃恢复 + 快照回滚 |
+| **多媒体分析** | 基础视觉感知 + 高速 OCR | 完整 `scrcpy`/`ffmpeg` 视频流分析 + Logcat 日志采集 |
+
+## 🗺️ 路线图
+
+- [x] **全链路乐观异步流水线**：主流程轻量化，后台并发执行记忆压缩与断言校验。
+- [x] **Safety Net 执行前校验**：动作下发前双层核验与预测性链式执行。
+- [x] **时间敏感多媒体处理**：集成 `scrcpy` 与 `ffmpeg` 分析全屏视频与音频流。
+- [x] **原生 MCP 服务支持**：支持与 Cursor、Claude Desktop 等工具无缝集成。
+- [x] **Web 可视化控制台**：提供设备投屏、交互演练与轨迹回溯。
+- [x] **AndroidWorld SOTA**：达到 99%+ 的任务完成率。
+- [ ] **跨平台扩展**：探索 iOS 与 Web 端的定位与操作能力。
+- [ ] **端侧轻量化模型**：支持在设备本地离线运行的轻量级 Edge VLM 模型。
+- [ ] **实时语音双工交互**：支持自然语音输入与实时打断控制。
+
+## 🤝 社区与贡献
+
+欢迎通过以下方式参与项目建设：
+* ⭐ **Star 本项目**以关注最新进展与更新
+* 💬 加入 [Discord 社区](https://discord.gg/wF2FN4WHGY) 参与技术探讨与功能建议
+* 🐛 提交 [Issue](https://github.com/google/artemis/issues) 反馈 Bug，欢迎发起 [Pull Request](https://github.com/google/artemis/pulls) 贡献代码
+
+## 📄 开源许可证
+
+本项目基于 [Apache License 2.0](LICENSE) 协议开源。
diff --git a/apps/README.md b/apps/README.md
deleted file mode 100644
index 486c62f..0000000
--- a/apps/README.md
+++ /dev/null
@@ -1,26 +0,0 @@
-# 📱 Artemis Applications Ecosystem (`apps/`)
-
-This directory houses the two primary applications and user interfaces of the Artemis ecosystem.
-
----
-
-## 📂 Subsystems Overview
-
-```
-apps/
-├── 🛠️ admin_console/ # [Interface 1] Full-fidelity Trace Inspector & Task Control Console (FastAPI + Web)
-└── ✨ showcase_ui/ # [Interface 2] Aesthetic User Presentation & Interaction Workspace (Angular 19)
-```
-
----
-
-## 🔍 Detailed Comparison of the Two UIs
-
-| Dimension | 🛠️ Admin & Trace Console (`admin_console`) | ✨ Showcase & Workspace UI (`showcase_ui`) |
-| :--- | :--- | :--- |
-| **Primary Audience** | Developers, Test Engineers, System Admins | End Users, Customers, Product Demo Audiences |
-| **UI Paradigm** | High-density, full-fidelity tree logs, step-by-step inspector | Dynamic Aurora Glow, Glassmorphism, dual-pane stream |
-| **Core Capabilities** | • Full hierarchical Call Tree & LLM thinking inspection🔍 点击展开：核心架构设计细节与流水线流程图
+ ++ +### 1. ⚡ 全链路乐观异步流水线 (Optimistic Asynchronous Pipeline) +* **现有方案与问题**：传统移动端智能体普遍采用**全同步阻塞模型** —— 每一步交互都必须同步串行等待大模型完成历史上下文修剪、阶段性断言检查与长程规划审计。这导致主交互循环极度臃肿，单步延迟动辄 20–40 秒，交互体验迟缓； +* **Artemis 的架构解法**：借鉴数据库的**乐观并发控制 (OCC) 与快照隔离**思想，将主干执行流与重度计算彻底解耦： + * **高吞吐前台主循环**：前台严格收敛为「感知 → 决策 → 守门 → 执行」极简流水线，单步响应压缩至秒级； + * **后台并发异步处理**：上下文修剪、里程碑断言校验（Checker）、全局规划审计（Planner）全部在后台并发静默运行，零阻塞前台； + * **快照隔离与无损回滚**：前台以前瞻视角乐观执行；一旦后台异步断言检测到状态偏离，系统基于快照机制**一键无损回滚 (Rollback)** 并注入纠偏反馈，兼得极低延迟与可靠自愈。 + +
+
+
• Pre/post action screenshots & visual overlay verification
• Video playback & step replay engine (`/api/replay`)
• Queue tasks & live state toggle | • Real-time Agent thought & action progress streaming
• Interactive natural language chat sidebar
• Aesthetic visual representation of mobile autonomy | -| **Tech Stack** | Python (FastAPI) + Vanilla Responsive Web | TypeScript, Angular 19, SCSS | -| **Start Command** | `uv run uvicorn apps.admin_console.server:app --port 8000 --reload` | `cd apps/showcase_ui && npm start` (Port 4200) | - diff --git a/apps/__init__.py b/apps/__init__.py index 2a5c098..89afab8 100644 --- a/apps/__init__.py +++ b/apps/__init__.py @@ -12,5 +12,4 @@ # See the License for the specific language governing permissions and # limitations under the License. - -"""Artemis / Latte Applications Package.""" +"""Artemis Applications Package.""" diff --git a/apps/admin_console/README.md b/apps/admin_console/README.md deleted file mode 100644 index fd63072..0000000 --- a/apps/admin_console/README.md +++ /dev/null @@ -1,51 +0,0 @@ -# 🛠️ Artemis Admin & Trace Console - -A comprehensive developer/administrator control plane and trace inspection console for the Latte/Artemis mobile automation system. - ---- - -## 🎯 Key Features - -1. **Full Trace Inspection (`/api/sessions`, `/api/steps`)**: - - Hierarchical Call Tree exploration (LLM thoughts, tool calls, agent motivations). - - Step-by-step pre/post screenshots and action overlays (tap circles, swipe paths). - - Streaming logs and video recording playback. -2. **Task Control Center (`/api/run`)**: - - Directly launch autonomous tasks with custom goals and model profiles (Flash / Pro). - - Pause, resume, and manage FIFO task execution queues. -3. **Step Replay & Debugging Engine (`/api/replay`)**: - - Replay individual steps with isolated inputs, state inspection, and live device execution. - ---- - -## 🚀 How to Start - -### Option 1: Using `uv` (Recommended) -From the repository root: -```bash -uv run uvicorn apps.admin_console.server:app --host 0.0.0.0 --port 8000 --reload -``` -Or directly using Python module: -```bash -uv run python -m apps.admin_console.server -``` - -### Option 2: From the `apps/admin_console` directory -```bash -cd apps/admin_console -uv run uvicorn server:app --host 0.0.0.0 --port 8000 --reload -``` - -Once started, open your browser and navigate to: -👉 **`http://localhost:8000`** - ---- - -## 🎥 Video Debugging Mode - -By default, `VideoAnalyzer` cleans up local temporary video files after execution. To retain video recordings for full playback in the console: - -```bash -KEEP_VIDEOS=1 artemis run "your task goal" -``` -Or set `ARTEMIS_DEBUG=1` in your environment. diff --git a/apps/admin_console/core/config.py b/apps/admin_console/core/config.py index 7f80a96..95385ba 100644 --- a/apps/admin_console/core/config.py +++ b/apps/admin_console/core/config.py @@ -63,6 +63,12 @@ except ImportError: REPLAY_BASE_DIR = TRACES_PATH / "replay" TEST_DATA_DIR = REPLAY_BASE_DIR / "data" TEST_OUTPUTS_DIR = REPLAY_BASE_DIR / "outputs" - IMAGES_DIR = TRACES_PATH / "images" PAUSE_FILE = WORKSPACE_ROOT / ".artemis_paused" + +# Ensure all essential runtime directories exist automatically +for _d in (TRACES_PATH, REPLAY_BASE_DIR, TEST_DATA_DIR, TEST_OUTPUTS_DIR, IMAGES_DIR): + try: + _d.mkdir(parents=True, exist_ok=True) + except Exception: + pass diff --git a/apps/admin_console/core/state.py b/apps/admin_console/core/state.py index d37dd67..e4e73e5 100644 --- a/apps/admin_console/core/state.py +++ b/apps/admin_console/core/state.py @@ -36,25 +36,96 @@ class ServerState: self.active_connections: dict[str, dict[str, Any]] = {} self.active_session_id: str | None = None self.was_stopped_manually: bool = False + self.cancelled_session_ids: set[str] = set() - # FIFO Task queue - self.task_queue: asyncio.Queue = asyncio.Queue() - self.queue_tasks: list[dict[str, Any]] = [] + # Unified single source of truth for task queue + self.queue_items: list[dict[str, Any]] = [] + self._wake_event: asyncio.Event | None = None self.worker_task: asyncio.Task | None = None + @property + def wake_event(self) -> asyncio.Event: + try: + loop = asyncio.get_running_loop() + except RuntimeError: + loop = None + + if self._wake_event is None or ( + loop + and getattr(self._wake_event, "_loop", None) is not None + and self._wake_event._loop != loop + ): + self._wake_event = asyncio.Event() + return self._wake_event + + @property + def task_queue(self) -> list[dict[str, Any]]: + """Backward compatibility alias for queue_items.""" + return self.queue_items + + @property + def queue_tasks(self) -> list[dict[str, Any]]: + """Returns all currently pending tasks in the queue.""" + return [t for t in self.queue_items if isinstance(t, dict) and t.get("status") == "pending"] + + @queue_tasks.setter + def queue_tasks(self, val: list[dict[str, Any]]): + # Compatibility setter + self.queue_items = list(val) + @property def queue_goals(self) -> list[str]: """Backward compatibility helper for queue goals.""" - return [t.get("goal", "") for t in self.queue_tasks if isinstance(t, dict)] + return [t.get("goal", "") for t in self.queue_tasks] @queue_goals.setter def queue_goals(self, val: list[Any]): - # Allow setting if legacy code updates it pass @property def is_running(self) -> bool: - return self.current_process is not None and self.current_process.returncode is None + has_proc = False + if self.current_process is not None: + if self.current_process.returncode is not None: + has_proc = False + self.current_process = None + else: + pid = getattr(self.current_process, "pid", None) + if pid: + try: + import os + + os.kill(pid, 0) + has_proc = True + except (ProcessLookupError, PermissionError, OSError): + has_proc = False + self.current_process = None + else: + has_proc = False + self.current_process = None + + has_running_item = any( + isinstance(t, dict) and t.get("status") == "running" for t in self.queue_items + ) + + has_live_connection = False + for sid, conn in list(self.active_connections.items()): + c_pid = conn.get("pid") + if c_pid: + try: + import os + + os.kill(c_pid, 0) + has_live_connection = True + except (ProcessLookupError, PermissionError, OSError): + self.active_connections.pop(sid, None) + + if not has_proc and not has_running_item and not has_live_connection: + self.active_session_id = None + self.current_process = None + return False + + return True @property def is_paused(self) -> bool: @@ -69,13 +140,10 @@ class ServerState: self.ipc_subscribers.remove(callback) def clear_queue(self): - while not self.task_queue.empty(): - try: - self.task_queue.get_nowait() - self.task_queue.task_done() - except (asyncio.QueueEmpty, ValueError): - break - self.queue_tasks.clear() + """Clears all pending items from the task queue.""" + self.queue_items = [t for t in self.queue_items if t.get("status") == "running"] + if self._wake_event: + self._wake_event.set() # Global shared instance diff --git a/apps/admin_console/database/connection.py b/apps/admin_console/database/connection.py index 5de31d3..a7f2530 100644 --- a/apps/admin_console/database/connection.py +++ b/apps/admin_console/database/connection.py @@ -15,7 +15,6 @@ from contextlib import contextmanager from pathlib import Path import sqlite3 -from fastapi import HTTPException try: from admin_console.core.config import DB_PATH @@ -23,13 +22,29 @@ except ImportError: from apps.admin_console.core.config import DB_PATH +_initialized_dbs = set() + + def get_db(db_path: Path | None = None) -> sqlite3.Connection: """Returns a SQLite connection with timeout and Row row_factory.""" path = db_path or DB_PATH - if not path.exists(): - raise HTTPException(status_code=500, detail=f"Database not found at {path}") + path_key = str(path) + if path_key not in _initialized_dbs: + path.parent.mkdir(parents=True, exist_ok=True) + try: + from artemis.data_engine.storage import StorageManager + + StorageManager(db_path=path, base_trace_dir=path.parent) + _initialized_dbs.add(path_key) + except Exception: + pass conn = sqlite3.connect(path, timeout=10.0) conn.row_factory = sqlite3.Row + try: + conn.execute("PRAGMA journal_mode=WAL") + conn.execute("PRAGMA busy_timeout=10000") + except Exception: + pass return conn diff --git a/apps/admin_console/database/repositories/__init__.py b/apps/admin_console/database/repositories/__init__.py index 30cb974..58d482e 100644 --- a/apps/admin_console/database/repositories/__init__.py +++ b/apps/admin_console/database/repositories/__init__.py @@ -10,4 +10,4 @@ # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and -# limitations under the License. \ No newline at end of file +# limitations under the License. diff --git a/apps/admin_console/database/repositories/session_repository.py b/apps/admin_console/database/repositories/session_repository.py index 4b0b3cc..9821ea8 100644 --- a/apps/admin_console/database/repositories/session_repository.py +++ b/apps/admin_console/database/repositories/session_repository.py @@ -33,13 +33,6 @@ class SessionRepository: cursor.execute("SELECT * FROM sessions ORDER BY start_time DESC") return [dict(r) for r in cursor.fetchall()] - def get_session_by_id(self, session_id: str) -> dict[str, Any] | None: - with db_session(self.db_path) as conn: - cursor = conn.cursor() - cursor.execute("SELECT * FROM sessions WHERE session_id = ?", (str(session_id),)) - row = cursor.fetchone() - return dict(row) if row else None - def get_video_recordings_map(self) -> dict[str, str]: with db_session(self.db_path) as conn: cursor = conn.cursor() diff --git a/apps/admin_console/database/repositories/step_repository.py b/apps/admin_console/database/repositories/step_repository.py index 7070999..71d0d7b 100644 --- a/apps/admin_console/database/repositories/step_repository.py +++ b/apps/admin_console/database/repositories/step_repository.py @@ -13,6 +13,7 @@ # limitations under the License. import json +import re from typing import Any try: @@ -183,9 +184,27 @@ class StepRepository: ) args = payload_obj.get("args", payload_obj) - if isinstance(args, dict): - return {"args": self._clean_value(args)} - return {"args": self._clean_value(args)} + cleaned_args = ( + self._clean_value(args) if isinstance(args, dict) else self._clean_value(args) + ) + result_payload: dict[str, Any] = {"args": cleaned_args} + + # Extract structured action execution results + res = payload_obj.get("result") + if isinstance(res, dict): + if res.get("post_image_name"): + result_payload["post_image_name"] = res["post_image_name"] + if res.get("pre_image_name"): + result_payload["pre_image_name"] = res["pre_image_name"] + if res.get("outcome"): + result_payload["outcome"] = res["outcome"] + elif res: + res_str = str(res) + match = re.search(r"([a-f0-9]{64})", res_str) + if match: + result_payload["post_image_name"] = match.group(1) + + return result_payload def get_session_steps(self, session_id: str, client: str | None = None) -> list[dict[str, Any]]: with db_session(self.db_path) as conn: @@ -221,7 +240,7 @@ class StepRepository: if pre_img and step_dict.get("post_image_name") == pre_img: step_dict["post_image_name"] = None - # Fetch generic tool traces for this step + # Fetch action and generic tool traces for this step step_id = step_dict.get("step_id") if step_id: cursor.execute( @@ -229,7 +248,7 @@ class StepRepository: " t1.type, t1.name, t1.timestamp, t1.duration, t1.status," " t1.payload, t2.name as agent_name FROM traces t1 LEFT" " JOIN traces t2 ON t1.parent_trace_id = t2.trace_id WHERE" - " t1.step_id = ? AND (t1.type = 'tool' OR (t1.type =" + " t1.step_id = ? AND (t1.type = 'tool' OR t1.type = 'action' OR (t1.type =" " 'llm_call' AND t1.status = 'failed')) ORDER BY" " t1.timestamp ASC", (step_id,), diff --git a/apps/admin_console/index.html b/apps/admin_console/index.html index 69aad2d..aaac56e 100644 --- a/apps/admin_console/index.html +++ b/apps/admin_console/index.html @@ -20,6 +20,11 @@
+
+
+
+
+
+
+
+
+
+
+
+
+ ARTEMIS
+
+ Control Any Android App with Autonomous AI
+
+
+
+
+
+
+ Goal / Prompt
+
+
+
+ RUNNING ON DEVICE
+
+
+ > Order an iced caramel latte with oat milk
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Agent Execution
+
+ STEP 3 / 4
+
+
+
+
+ 1. Open app & search catalog
+
+
+
+ 2. Select item & customize
+
+
+
+
+
+
+
+ ▶ 3. Tap "Place Order"
+ 140ms
+
+
+ Vision target grounded at (540, 1680)
+
+
+ ○
+ 4. Verify receipt & return
+
+
+
+
+
+
+
+
+
+ Food & Drink
+ ● ANDROID
+
+
+
+
+
+ ☕
+
+ Iced Caramel Latte
+ Double Shot · Oat Milk
+
+ $5.40
+
+ Delivery Address
+ Desk #42
+
+
+
+
+
+
+
+ 🎯 GROUNDED (0.99)
+
+ Place Order ($5.40)
+
+
+
+
+
+
+