Artemis-Eapil/README.md

ARTEMIS: Autonomous AI Assistant for Mobile Automation

⚡ Drive real phones from Cursor & Claude Code • Cross-App Automation • Zero-Maintenance UI Testing • Flash Mode as Fast as 3-5s

English • 中文文档 • Quick Start • MCP for IDEs • Benchmarks • Discord Community

Live Demo: Setup driving routes and calculate total durations in Google Maps, then open YouTube to play a Coldplay song.

✨ Key Highlights

• ⚡ Lightning Fast: First-of-its-kind Optimistic Asynchronous Pipeline decouples the foreground execution stream from heavy LLM reasoning;
• 🔋 Ultra-Long Autonomous Execution: Pro mode runs stably for over 10 hours, while monitoring tasks support around-the-clock 24/7 execution;
• 🔌 Native Model Context Protocol (MCP): Empowers Cursor, Claude Code, OpenClaw, Windsurf with direct mobile device manipulation from within your chat prompts;
• 💾 Visual Console & Persistent Debugging: Real-time Web UI screen projection, execution replay, and task management with natural language test generation and high-fidelity historical review;
• 🏆 Industry-Leading SOTA: Achieved 99%+ task completion on Google Research's AndroidWorld benchmark (100+ complex multi-step tasks).

⚡ Quick Start

Ensure an Android device (with USB Debugging enabled) or emulator is connected.

# 1. Clone repo & navigate to directory
git clone https://github.com/google/artemis.git && cd artemis

# 2. One-click launch (automatically installs ADB, scrcpy, FFmpeg, uv runtime, and opens console)
# 🍎 macOS & 🐧 Linux
./start.sh

# 🪟 Windows (CMD / PowerShell)
start.bat

💡 Tip: Opens http://localhost:8000 in your default browser with a device connection wizard, live screen mirroring, prompt sandbox, and execution replays. You can also run directly from CLI: artemis run "Open Settings, find Battery and tell me current level" --profile flash.

🔌 MCP Setup for Cursor / Claude Code / Windsurf (Click to expand)

ARTEMIS includes a native Model Context Protocol (MCP) server. Connect your real phone directly into AI IDEs:

1. Generate MCP Config

Run the built-in generator to produce ready-to-use JSON configuration:

artemis mcp --generate-config cursor
# Or generate configs for all IDEs:
artemis mcp --generate-config all

2. Copy Config to Your IDE

• Cursor (.cursor/mcp.json or Settings ➔ MCP Servers):

{
  "mcpServers": {
    "artemis": {
      "command": "python",
      "args": ["-m", "mcp_server"],
      "cwd": "/path/to/artemis",
      "env": {
        "PYTHONUNBUFFERED": "1"
      }
    }
  }
}

• Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "artemis": {
      "command": "python",
      "args": ["-m", "mcp_server"],
      "cwd": "/path/to/artemis"
    }
  }
}

3. Prompt Your Phone in the IDE Chat

In Cursor or Claude Code, simply prompt:

💬 "Build the latest changes into an APK, install it on the connected device, open the login screen with a test account, verify if there are any unexpected popups after login, and return screenshots of the final page."

🐍 Python SDK Integration (Click to expand)

Embed the mobile automation engine into your Python workflows in just a few lines:

import asyncio
from artemis.interfaces.sdk import ArtemisClient


async def main():
    # Initialize client (choose "flash" for speed or "pro" for deep reasoning)
    client = ArtemisClient(default_profile="flash")

    # Run autonomous task
    result = await client.run("Open calculator and compute (128 * 45) + 330")

    print(f"Status: {result.status}")
    print(f"Turns taken: {result.turns}")
    print(f"Trace ID: {result.trace_id}")


if __name__ == "__main__":
    asyncio.run(main())

🕹️ Usage Modes

_{💡 Console Overview: ① View Switcher (Home / Workspace) · ② Model & Replay (Flash/Pro status & video replay) · ③ Live Agent Stream (Action perception, target coordinates & structured results) · ④ Prompt Dock (Natural language dispatch) · ⑤ Task Queue & Dashboard (Lifecycle & history)}

• 🖥️ Web Visual Console (artemis ui): Real-time screen projection and interactive panel, supporting natural language task dispatch, live reasoning telemetry, action trajectories, and execution replay;
• 🔌 Native MCP Protocol (IDE Integration): Operates as a standard MCP server seamlessly integrating with Cursor, Claude Code, OpenClaw, etc., directly invoking mobile device manipulation inside the IDE;
• 💻 Developer CLI (artemis run): Direct terminal execution for automated tasks, test cases, or AndroidWorld benchmarks with high-fidelity structured terminal output;
• 🐍 Python SDK: Integrates as a standard Python library into existing automated testing frameworks or pipelines with strongly typed Pydantic structured outputs.

📊 Head-to-Head Comparison

Scenario / Dimension	Traditional Automation	Generic Screen AI Agents	ARTEMIS ☕
Natural Language Execution	❌ Rigid hard-coded scripts only	⚠️ Sluggish (waits 20-30s per tap)	⚡ High Throughput: As fast as 3–5s per step in Flash mode with fluid responses
UI Redesign Resilience	❌ Layout changes break XPath selectors	⚠️ Coordinate guessing leads to drift	🎯 Semantic Grounding: Blends text, icons & UI tree structure to adapt to UI changes
Popup Interception	❌ Blocked and crashed immediately	❌ Loops endlessly on unfamiliar screens	🩹 Pre-Execution Self-Healing: Verifies target state before clicking; dismisses popups
Dynamic & Video Tasks	❌ Blind static sleep only	❌ Blind to dynamic video playback	⏱️ Real-Time Multimedia: Live video stream analysis, countdown timers & speculative chaining
IDE & Tooling Integration	❌ Standalone test runner	❌ Web-only with high integration friction	🔌 Native MCP Server: Directly callable inside Cursor, Claude Desktop, etc.

🏆 Benchmarks: AndroidWorld (SOTA 99%+)

Evaluated on AndroidWorld — Google Research's gold-standard benchmark spanning 20+ real apps and 100+ complex multi-step tasks: Artemis demonstrated exceptional robustness across the entire benchmark suite, achieving a 99%+ completion rate.

🚀 How ARTEMIS is Architected

• ⚡ Optimistic Asynchronous Pipeline: The front-facing loop responds in milliseconds, while memory pruning and assertion verification run concurrently in the background without blocking execution;
• 🛡️ Safety Net Pre-Execution Gate: Dual-layer pre-check validates target availability milliseconds before action dispatch, instantly intercepting unexpected popups to eliminate blind clicks;
• ⏱️ Time-Sensitive Speculative Chaining: Overcomes LLM inference latency for transient UI elements (e.g. video fullscreen) by predicting target coordinates and executing rapid chained taps.

🔍 Click to expand: Architecture Deep Dive & Pipeline Diagram

1. ⚡ Optimistic Asynchronous Pipeline

• Status Quo & Pain Points: Conventional mobile agents rely on a fully synchronous blocking model — every single action must wait sequentially for the LLM to prune historical context, check milestone assertions, and audit long-term plans. This inflates per-step latency to 20–40 seconds, creating a sluggish user experience.
• Artemis's Architectural Solution: Inspired by Optimistic Concurrency Control (OCC) and Snapshot Isolation in database systems, Artemis completely decouples the main execution loop from heavy auxiliary computation:
  • High-Throughput Main Loop: The front-facing execution path is strictly narrowed to a high-speed "Perception → Decision → Safety Gate → Execution" pipeline;
  • Background Concurrent Tasks: Context token pruning, milestone checkers (Checker), and planner validations (Planner) run dynamically in parallel without halting device interaction;
  • Snapshot Isolation & Rollback: The agent optimistically charges forward. If background verification detects a deviation, Artemis instantly rolls back via state snapshots (Rollback) and injects self-healing feedback.

2. 🛡️ Pre-Execution Safety Net & Time-Sensitive Speculative Chaining

• Status Quo & The Transient UI Dilemma:
  • Mobile applications feature numerous time-sensitive transient UI controls (e.g. video fullscreen: tapping the screen wakes up the floating overlay, followed immediately by tapping the fullscreen icon).
  • Traditional agents tap the screen to reveal controls, take a new screenshot, and wait 3–15 seconds for LLM reasoning. By the time the click is dispatched, the player controls have already auto-faded away — causing the click to strike the underlying video, triggering an endless loop of accidental pausing and waking.
• Artemis's Architectural Solution:
  • Speculative Chained Actions: Upon recognizing time-sensitive dependencies, the agent dispatches compound chained actions (Wakeup → Millisecond Chained Tap) to hit the target within its transient visibility window;
  • Two Pillars Ensuring Reliable Chaining:
    1. Historical UI Prior Prediction: Predicts the target control's wake-up coordinates based on prior interaction history and app layout heuristics;
    2. Safety Net Pre-Execution Gate: Milliseconds before the chained action lands, the Safety Net instantly verifies that the target control was successfully revealed at the expected coordinates. If the wakeup failed or an unexpected popup intercepted it, execution is immediately blocked to prevent blind clicks.

⚡ Execution Profiles: Flash vs. Pro

Feature / Dimension	⚡ ARTEMIS Flash (--profile flash)	🧠 ARTEMIS Pro (--profile pro)
Design Purpose	Lightweight & Fast: Direct deterministic UI actions	Deep Reasoning: Multi-step planning & complex self-healing
Step Latency	3–5 seconds / step	15–30 seconds / turn (includes planning & verification)
Task Duration	Minute-level short tasks (typically ≤35 steps)	Runs stably for 10+ hours; monitoring tasks support 24/7 execution
Best Suited For	Well-defined standard UI tasks	Complex cross-app workflows, failure self-healing, continuous monitoring
Self-Healing	Local step retries	Safety Net Gate + dialog suppression + crash recovery + snapshot rollback
Media Analysis	Basic visual perception + High-Speed OCR	Full scrcpy/ffmpeg video stream analysis + Logcat logs

🗺️ Roadmap

• Optimistic Asynchronous Pipeline: Ultra-lean main loop + background context compression & milestone checks.
• Pre-Execution Safety Net: Millisecond pre-check gate & speculative chained actions.
• Time-Sensitive Media Tasks: Fullscreen video and audio stream analysis with scrcpy & ffmpeg.
• Native MCP Server: Seamless integration with tools like Cursor and Claude Desktop.
• Web Visual Console: Live screen projection, interactive playground, and trajectory review.
• AndroidWorld SOTA: Achieved 99%+ task completion rate.
• Cross-Platform Extensions: Exploring iOS and desktop Web perception and execution.
• On-Device Lightweight VLMs: Zero-cloud local execution with lightweight edge vision models.
• Real-time Duplex Voice Mode: Natural voice input with real-time interruption (barge-in) control.

🤝 Community & Contributing

Contributions are warmly welcomed!

• ⭐ Star the repo to follow updates and releases
• 💬 Join the Discord Community for technical discussions
• 🐛 Open an Issue or submit a Pull Request

📄 License

This project is licensed under the Apache License 2.0.