feat: introduce MCP server, cross-platform runtime, and Showcase UI reorganization

- Showcase UI Reorganization:
  * Move web UI dashboard from root directory into apps/showcase_ui/ for cleaner project structure
  * Add 'artemis ui' CLI command (artemis/interfaces/cli/commands/ui.py) and launcher scripts for the interactive showcase console
- Standalone MCP Server & Notifiers (mcp_server/):
  * Implement dedicated MCP server package with mobile automation tools (device_state, inspect_trace, task_manager, task_runner)
  * Add multi-channel notifiers (AgentAPI, Desktop notifications, Webhooks, File output, and Composite routing)
  * Introduce background task runner and trace store for autonomous mobile execution monitoring
- Cross-Platform Runtime & Toolchain Support:
  * Add artemis/platform/ with OS-specific implementations for Linux, macOS (Darwin), and Windows
  * Add artemis/runtime/supervisor.py for resilient process and agent runtime supervision
  * Add artemis/toolchain/ with dependency resolution and SDK descriptors
  * Add cross-platform launcher scripts (start.sh, start.bat) and installer scripts (scripts/install_deps.*)
- Core Enhancements & Diagnostics:
  * Enhance coordinate normalization, OCR API fallback, and perception pipeline
  * Add credential validation (credentials_validator.py) and system diagnostics
  * Update bilingual documentation (README.md, README_CN.md) and architecture diagrams
- Test Suite Expansion:
  * Add integration tests for cloud gateway, remote ADB, and Artemis login
  * Add unit tests covering core diagnostics, platform detection, toolchain, credentials validator, optional OCR, and MCP tools
This commit is contained in:
somew1nd committed 2026-08-18 13:59:27 -07:00
1 parent 78e478f502
commit 9a5aaf99b7
249 files changed
+37226 -9339

No files matched your search

+68
View File
@@ -0,0 +1,68 @@
# 🔌 Universal MCP Server for ARTEMIS
This directory contains the universal **Model Context Protocol (MCP)** server for **ARTEMIS**, enabling seamless mobile device automation inside AI IDEs and agents like **Cursor**, **Claude Code**, **OpenClaw**, **Windsurf**, and **Jetski**.
## 🏗️ Architecture
```
mcp_server/
├── __init__.py # Package exports (mcp, main)
├── base.py # Shared FastMCP instance
├── server.py # Server entrypoint (stdio / sse)
├── rules.md # AI Agent Testing Mindset & system prompt rules for IDEs
├── background/ # Background process execution
│ └── task_runner.py # Subprocess runner for Artemis Agent
├── notifiers/ # Multi-environment notification adapters
│ ├── base.py # BaseNotifier abstract class
│ ├── agentapi.py # Jetski / Antigravity AgentAPI notifier
│ ├── webhook.py # OpenClaw / CI / Discord / Slack webhook notifier
│ ├── desktop.py # Native OS toast notifier
│ ├── file.py # File audit logger
│ └── composite.py # Multi-channel notification dispatcher
├── tools/ # Modular standard MCP tools
│ ├── task_runner.py # mobile_run_task
│ ├── task_manager.py # mobile_manage_task
│ ├── device_state.py # mobile_get_device_state
│ └── inspect_trace.py # mobile_inspect_trace
└── utils/ # Environment, device, and trace utilities
├── device_utils.py # Cross-platform ADB and emulator resolver
├── env_utils.py # Python interpreter and process manager
└── trace_store.py # Traces directory and status manager
```
## 🧠 AI Agent Behavioral Rules (`rules.md`)
When connecting ARTEMIS to AI coding assistants like **Claude Code**, **Cursor**, or **Windsurf**, providing the assistant with domain-specific testing discipline is critical for generating reliable test code.
The included [`rules.md`](./rules.md) file contains the **Mobile Testing Mindset (ARTEMIS Integration)** guideline. It teaches the AI agent how to properly collaborate with ARTEMIS:
1. **The Runnable Code Principle & Active Exploration**: Instructs the AI to never guess or hallucinate UI transitions. Instead, it must first explore the live app via ARTEMIS MCP tools (`mobile_run_task`, etc.) to discover and verify real-world interaction paths before writing test scripts.
2. **Flash vs. Pro Routing Strategy**: Guides the AI to choose **Flash** for rapid, straightforward UI actions (< 30 steps) and **Pro** for complex multi-agent planning, polling loops, or video/log diagnosis.
3. **Latency & Timing Compensation**: Clarifies the difference between AI exploratory latency (e.g., model decision intervals) and the deterministic timing requirements of final test code.
4. **"Dynamic-First, Coordinate-Fallback" Locator Pattern**: Teaches the AI to prioritize dynamic UI locators (Resource IDs, OCR text, semantics) for layout resilience, while implementing absolute coordinate fallbacks for maximum execution reliability.
### How to Apply `rules.md` in Your IDE
* **Claude Code**: Copy or include the contents of `rules.md` in your project's `CLAUDE.md` file (or reference it directly in your instructions).
* **Cursor**: Copy the contents of `rules.md` into your `.cursorrules` or create a new rule file in `.cursor/rules/artemis.mdc`.
* **Windsurf / OpenClaw**: Add the rules to your global/workspace rules or agent system prompts so the assistant always follows verified mobile testing principles.
## 🛠️ MCP Tools Overview
* **`mobile_run_task`**: Asynchronously launches an autonomous mobile automation task (`Flash` or `Pro` model).
* **`mobile_manage_task`**: Manages task lifecycle (`status`, `stop`, `inject_instruction`).
* **`mobile_get_device_state`**: Real-time observer (`screenshot` or OCR+XML `hierarchy`).
* **`mobile_inspect_trace`**: Granular trace inspection, visual action overlays, and agent reasoning.
## 🚀 How to Run & Configure
### CLI Execution
```bash
# Start server over stdio
python -m mcp_server
# Or via Artemis CLI
artemis mcp
```
### IDE Configuration
Run `artemis mcp --generate-config all` to generate ready-to-use configuration JSON for Cursor, Claude Code, OpenClaw, and Windsurf.
+20
View File
@@ -0,0 +1,20 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Universal Model Context Protocol (MCP) Server for ARTEMIS."""
from mcp_server.base import mcp
from mcp_server.server import main
__all__ = ["mcp", "main"]
+20
View File
@@ -0,0 +1,20 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Direct execution entrypoint for mcp_server."""
from mcp_server.server import main
if __name__ == "__main__":
main()
+15
View File
@@ -0,0 +1,15 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""MCP Background execution package."""
+321
View File
@@ -0,0 +1,321 @@
#!/usr/bin/env python3
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Background execution runner for MCP autonomous mobile automation tasks."""
import argparse
import asyncio
import json
import logging
import os
import sys
import traceback
from dotenv import load_dotenv
PROJECT_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
if PROJECT_ROOT not in sys.path:
sys.path.insert(0, PROJECT_ROOT)
try:
from artemis.config.paths import get_app_dir
app_dir = str(get_app_dir())
global_env = os.path.join(app_dir, ".env")
if os.path.exists(global_env):
load_dotenv(global_env)
else:
load_dotenv(os.path.join(PROJECT_ROOT, ".env"))
except Exception:
load_dotenv(os.path.join(PROJECT_ROOT, ".env"))
from mcp_server.notifiers import notify
from mcp_server.utils import device_utils, trace_store
def resolve_profile_file() -> str | None:
"""Resolves the LLM configuration profile across multiple locations."""
candidates = []
artemis_config_dir = os.getenv("ARTEMIS_CONFIG_DIR")
if artemis_config_dir:
candidates.extend(
[
os.path.join(artemis_config_dir, "llm-config.override.jsonc"),
os.path.join(artemis_config_dir, "llm-config.json"),
]
)
try:
from artemis.config.paths import get_app_dir
app_dir = str(get_app_dir())
candidates.extend(
[
os.path.join(app_dir, "llm-config.override.jsonc"),
os.path.join(app_dir, "llm-config.json"),
]
)
except Exception:
pass
candidates.extend(
[
os.path.join(PROJECT_ROOT, "llm-config.override.jsonc"),
os.path.join(PROJECT_ROOT, "llm-config.json"),
os.path.join(PROJECT_ROOT, "config", "llm-config.json"),
]
)
for cand in candidates:
if os.path.exists(cand):
return cand
return None
async def run_task(
trace_id: str,
task_desc: str,
model: str,
conversation_id: str,
locked_app_package: str | None = None,
app_path: str | None = None,
expected_output_desc: str | None = None,
):
"""Executes the mobile automation agent task and logs all actions/results."""
trace_dir = trace_store.get_trace_dir(trace_id)
os.makedirs(trace_dir, exist_ok=True)
stdout_log_path = os.path.join(trace_dir, "stdout.log")
stderr_log_path = os.path.join(trace_dir, "stderr.log")
original_stdout = sys.stdout
original_stderr = sys.stderr
log_file_out = open(stdout_log_path, "w", buffering=1, encoding="utf-8")
log_file_err = open(stderr_log_path, "w", buffering=1, encoding="utf-8")
sys.stdout = log_file_out
sys.stderr = log_file_err
logging.basicConfig(level=logging.INFO, force=True)
logging.getLogger("absl").setLevel(logging.WARNING)
logging.getLogger("artemis").setLevel(logging.INFO)
print(f"Starting task execution for Trace ID: {trace_id}")
print(f"Goal: {task_desc}")
print(f"Model: {model}")
print(f"Conversation ID: {conversation_id}")
print("--------------------------------------------------")
agent = None
adb_path = device_utils.resolve_adb_path()
try:
from artemis.sdk import Agent
from artemis.sdk.builders import Builders
from artemis.sdk.types import AgentProfile
connected_devices = device_utils.get_connected_devices(adb_path)
if connected_devices:
print(
f"✅ Detected active connected device(s): {connected_devices}. "
f"Using the first available device: '{connected_devices[0]}'."
)
else:
print("❌ No active connected devices detected. Booting emulator...")
if not device_utils.ensure_emulator(adb_path=adb_path):
raise RuntimeError("Failed to start or connect to the Android emulator.")
print("Initializing Artemis Agent...")
from artemis.config import initialize_llm_config, settings
profile_file = resolve_profile_file()
if profile_file:
profile = AgentProfile(name="default", from_file=profile_file)
else:
profile = AgentProfile(name="default", llm_config=initialize_llm_config())
config_builder = Builders.AgentConfig.with_default_profile(profile)
if settings.ADB_HOST:
config_builder.with_adb_server(host=settings.ADB_HOST, port=settings.ADB_PORT)
config = config_builder.build()
agent = Agent(config=config)
await agent.init(
retry_count=int(os.getenv("ARTEMIS_HEALTH_RETRIES", 5)),
retry_wait_seconds=int(os.getenv("ARTEMIS_HEALTH_DELAY", 2)),
)
print("Running task on agent...")
task_builder = (
agent.new_task(goal=task_desc)
.with_name(f"task_{trace_id}")
.with_trace_recording(enabled=True, path=trace_store.TRACES_DIR)
)
if locked_app_package:
task_builder.with_locked_app_package(package_name=locked_app_package)
if app_path:
task_builder.with_app_path(app_path=app_path)
if expected_output_desc:
task_builder.with_output_description(description=expected_output_desc)
if model.lower() == "flash":
task_builder.using_profile("flash")
result = await agent.run_task(request=task_builder.build())
print(f"Task completed. Result: {result}")
if isinstance(result, dict) and "status" in result and result.get("status") != "completed":
error_explanation = result.get("explanation", "Task execution returned failed status.")
print(f"Task finished with non-completed status: {error_explanation}", file=sys.stderr)
trace_store.update_trace_status(
trace_id, "failed", error=error_explanation, result=result
)
formatted_result = json.dumps(result, indent=2, ensure_ascii=False)
failure_msg = (
f"Artemis background task finished with non-completed status.\n\n"
f"Trace ID: {trace_id}\n"
f"Goal: {task_desc}\n"
f"Explanation: {error_explanation}\n\n"
f"Result:\n```json\n{formatted_result}\n```\n\n"
f'Check the logs using mobile_inspect_trace(action="view_summary", trace_id="{trace_id}") '
f"or inspect stderr log: [stderr.log](file://{stderr_log_path})."
)
notify(
conversation_id=conversation_id,
message=failure_msg,
title="Artemis Task Incomplete",
event_type="failed",
payload={"trace_id": trace_id, "goal": task_desc, "result": result},
)
return
if not result:
if model.lower() == "flash":
result = "Task executed successfully."
else:
result = (
"Task executed successfully. You can check the notes under "
"notes_dir to view more details."
)
trace_store.update_trace_status(trace_id, "completed", result=result)
if isinstance(result, dict):
formatted_result = json.dumps(result, indent=2, ensure_ascii=False)
result_str = f"Result:\n```json\n{formatted_result}\n```\n"
else:
formatted_result = str(result)
result_str = f"Result: {formatted_result}\n"
success_msg = (
"Artemis background task completed successfully.\n\n"
f"Trace ID: {trace_id}\n"
f"Goal: {task_desc}\n"
f"{result_str}"
)
notify(
conversation_id=conversation_id,
message=success_msg,
title="Artemis Task Completed",
event_type="completed",
payload={"trace_id": trace_id, "goal": task_desc, "result": result},
)
except asyncio.CancelledError:
print("Task was cancelled (asyncio.CancelledError)", file=sys.stderr)
trace_store.update_trace_status(trace_id, "cancelled", error="Task was cancelled")
cancel_msg = (
f"Artemis background task was cancelled.\n\nTrace ID: {trace_id}\nGoal: {task_desc}\n"
)
notify(
conversation_id=conversation_id,
message=cancel_msg,
title="Artemis Task Cancelled",
event_type="cancelled",
payload={"trace_id": trace_id, "goal": task_desc},
)
raise
except Exception as e:
error_type = type(e).__name__
raw_error_msg = str(e)
tb_str = traceback.format_exc()
print(
f"CRITICAL ERROR during task execution: {error_type}: {raw_error_msg}\n{tb_str}",
file=sys.stderr,
)
exc_type, exc_obj, tb = sys.exc_info()
while tb and tb.tb_next:
tb = tb.tb_next
frame_summary = f" (at {tb.tb_frame.f_code.co_name}:{tb.tb_lineno})" if tb else ""
full_error_desc = f"{error_type}: {raw_error_msg}{frame_summary}"
trace_store.update_trace_status(trace_id, "failed", error=full_error_desc)
failure_msg = (
f"Artemis background task failed.\n\n"
f"Trace ID: {trace_id}\n"
f"Goal: {task_desc}\n"
f"Error: {full_error_desc}\n\n"
f'Check the logs using mobile_inspect_trace(action="view_summary", trace_id="{trace_id}") '
f"or inspect stderr log: [stderr.log](file://{stderr_log_path})."
)
notify(
conversation_id=conversation_id,
message=failure_msg,
title="Artemis Task Failed",
event_type="failed",
payload={"trace_id": trace_id, "goal": task_desc, "error": full_error_desc},
)
finally:
print("Cleaning up resources...")
if agent:
try:
await agent.clean()
except Exception as e:
print(f"Error cleaning agent: {e}", file=sys.stderr)
sys.stdout = original_stdout
sys.stderr = original_stderr
log_file_out.close()
log_file_err.close()
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Artemis Background Task Runner")
parser.add_argument("--trace-id", required=True, help="Unique trace identifier")
parser.add_argument("--task-desc", required=True, help="Description of the task to run")
parser.add_argument("--model", required=True, help="Model to use ('Flash' or 'Pro')")
parser.add_argument("--conversation-id", required=True, help="Active conversation ID")
parser.add_argument("--locked-app-package", help="Package name of app to lock execution to")
parser.add_argument("--app-path", help="Path to local APK to install before running task")
parser.add_argument("--expected-output-desc", help="Expected output description")
args = parser.parse_args()
asyncio.run(
run_task(
trace_id=args.trace_id,
task_desc=args.task_desc,
model=args.model,
conversation_id=args.conversation_id,
locked_app_package=args.locked_app_package,
app_path=args.app_path,
expected_output_desc=args.expected_output_desc,
)
)
+29
View File
@@ -0,0 +1,29 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Shared FastMCP instance definition for the mobile automation MCP server."""
from mcp.server.fastmcp import FastMCP
# Define the shared FastMCP instance for external IDE and agent clients.
mcp = FastMCP(
"artemis",
instructions=(
"ARTEMIS is an autonomous mobile AI agent and Android UI automation engine. "
"Use mobile_run_task to launch autonomous UI workflows on connected Android devices or emulators, "
"mobile_manage_task to check status or steer execution, "
"mobile_get_device_state to inspect real-time device screen/hierarchy, "
"and mobile_inspect_trace to inspect detailed execution steps and visual action overlays."
),
)
+66
View File
@@ -0,0 +1,66 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Universal MCP Multi-Environment Notification System."""
from typing import Any, Dict, Optional
from mcp_server.notifiers.agentapi import AgentApiNotifier
from mcp_server.notifiers.base import BaseNotifier
from mcp_server.notifiers.composite import CompositeNotifier
from mcp_server.notifiers.desktop import DesktopNotifier
from mcp_server.notifiers.file import FileNotifier
from mcp_server.notifiers.webhook import WebhookNotifier
_default_notifier = CompositeNotifier()
def get_default_notifier() -> CompositeNotifier:
"""Returns the global default composite notifier instance."""
return _default_notifier
def notify(
conversation_id: str,
message: str,
title: str | None = None,
event_type: str = "completed",
payload: dict[str, Any] | None = None,
) -> bool:
"""Dispatches a notification across all available channels (AgentAPI, Webhook, Desktop, File)."""
return _default_notifier.notify(
conversation_id=conversation_id,
message=message,
title=title,
event_type=event_type,
payload=payload,
)
def notify_jetski(conversation_id: str, message: str) -> bool:
"""Backward-compatible alias for notify()."""
return notify(conversation_id=conversation_id, message=message, event_type="completed")
__all__ = [
"BaseNotifier",
"AgentApiNotifier",
"WebhookNotifier",
"DesktopNotifier",
"FileNotifier",
"CompositeNotifier",
"get_default_notifier",
"notify",
"notify_jetski",
]
+170
View File
@@ -0,0 +1,170 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""AgentAPI Notifier for Jetski / Antigravity environments."""
import json
import logging
import os
import shutil
import subprocess
from typing import Any
from mcp_server.notifiers.base import BaseNotifier
logger = logging.getLogger("mcp_server.notifiers.agentapi")
class AgentApiNotifier(BaseNotifier):
"""Notifier implementation for environments supporting the agentapi CLI (e.g. Jetski / Antigravity)."""
@property
def name(self) -> str:
return "agentapi"
def _find_agentapi_path(self) -> str | None:
"""Locates the agentapi executable."""
which_path = shutil.which("agentapi")
if which_path:
return which_path
candidates = [
os.path.expanduser("~/.gemini/jetski/bin/agentapi"),
os.path.expanduser("~/.artemis/bin/agentapi"),
os.path.expanduser("~/bin/agentapi"),
"/usr/local/bin/agentapi",
]
for cand in candidates:
if os.path.exists(cand) and os.access(cand, os.X_OK):
return cand
return None
def is_available(self) -> bool:
"""Returns True if agentapi binary exists or LS address is detected."""
if self._find_agentapi_path():
return True
if "ANTIGRAVITY_LS_ADDRESS" in os.environ:
return True
return False
def _load_shared_env(self, force_proc_scan: bool = False) -> None:
"""Recovers and loads ANTIGRAVITY_LS_ADDRESS and ANTIGRAVITY_CSRF_TOKEN."""
if not force_proc_scan:
try:
shared_candidates = [
os.path.expanduser("~/.gemini/jetski/.jetski_env"),
os.path.expanduser("~/.artemis/.artemis_env"),
]
# Project root fallback
parent_dir = os.path.dirname(
os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
)
shared_candidates.append(os.path.join(parent_dir, ".jetski_env"))
for shared_file in shared_candidates:
if os.path.exists(shared_file):
with open(shared_file, encoding="utf-8") as f:
env_data = json.load(f)
for key, val in env_data.items():
if val:
os.environ[key] = val
logger.debug(f"[EnvRecovery] Loaded {key} from {shared_file}.")
return
except Exception as e:
logger.debug(f"[EnvRecovery] Error reading shared env file: {e}")
# Fallback to psutil process scan
logger.debug("[EnvRecovery] Scanning process table for active agent session...")
found_addr = None
found_token = None
try:
import psutil
for proc in psutil.process_iter(["environ"]):
try:
env = proc.info.get("environ")
if env and "ANTIGRAVITY_LS_ADDRESS" in env and "ANTIGRAVITY_CSRF_TOKEN" in env:
found_addr = env["ANTIGRAVITY_LS_ADDRESS"]
found_token = env["ANTIGRAVITY_CSRF_TOKEN"]
break
except (psutil.AccessDenied, psutil.NoSuchProcess):
continue
except ImportError:
# Fallback on Linux /proc
import glob
for env_path in glob.glob("/proc/[0-9]*/environ"):
try:
with open(env_path, "rb") as f:
content = f.read()
env_vars = content.split(b"\x00")
local_addr = None
local_token = None
for var in env_vars:
if var.startswith(b"ANTIGRAVITY_LS_ADDRESS="):
local_addr = var.split(b"=", 1)[1].decode("utf-8", errors="ignore")
elif var.startswith(b"ANTIGRAVITY_CSRF_TOKEN="):
local_token = var.split(b"=", 1)[1].decode("utf-8", errors="ignore")
if local_addr and local_token:
found_addr = local_addr
found_token = local_token
break
except Exception:
pass
if found_addr and found_token:
os.environ["ANTIGRAVITY_LS_ADDRESS"] = found_addr
os.environ["ANTIGRAVITY_CSRF_TOKEN"] = found_token
logger.debug(f"[EnvRecovery] Recovered session from process. Address: {found_addr}")
def notify(
self,
conversation_id: str,
message: str,
title: str | None = None,
event_type: str = "completed",
payload: dict[str, Any] | None = None,
) -> bool:
"""Dispatches a notification via the agentapi send-message command."""
if not conversation_id or conversation_id.lower() in ("default", "none", ""):
return False
agentapi_path = self._find_agentapi_path()
if not agentapi_path:
return False
if "ANTIGRAVITY_LS_ADDRESS" not in os.environ or "ANTIGRAVITY_CSRF_TOKEN" not in os.environ:
self._load_shared_env()
cmd = [agentapi_path, "send-message"]
if title:
cmd.extend(["--title", title])
cmd.extend([conversation_id, message])
try:
subprocess.run(cmd, capture_output=True, text=True, check=True, timeout=10)
logger.info("AgentAPI notification sent successfully.")
return True
except (subprocess.CalledProcessError, FileNotFoundError, Exception):
try:
self._load_shared_env(force_proc_scan=True)
if "ANTIGRAVITY_LS_ADDRESS" in os.environ:
subprocess.run(cmd, capture_output=True, text=True, check=True, timeout=10)
logger.info("AgentAPI notification sent successfully on retry.")
return True
except Exception as retry_err:
logger.debug(f"AgentAPI notification retry failed: {retry_err}")
return False
return False
+45
View File
@@ -0,0 +1,45 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Base interface for MCP notification adapters."""
from abc import ABC, abstractmethod
from typing import Any
class BaseNotifier(ABC):
"""Abstract base class for environment notification providers."""
@property
@abstractmethod
def name(self) -> str:
"""Name of the notifier adapter."""
pass
@abstractmethod
def is_available(self) -> bool:
"""Checks if the notifier is configured and available in the current runtime environment."""
pass
@abstractmethod
def notify(
self,
conversation_id: str,
message: str,
title: str | None = None,
event_type: str = "completed",
payload: dict[str, Any] | None = None,
) -> bool:
"""Sends a notification to the target conversation / client."""
pass
+78
View File
@@ -0,0 +1,78 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Composite Notifier coordinating multi-channel notifications."""
import logging
from typing import Any
from mcp_server.notifiers.agentapi import AgentApiNotifier
from mcp_server.notifiers.base import BaseNotifier
from mcp_server.notifiers.desktop import DesktopNotifier
from mcp_server.notifiers.file import FileNotifier
from mcp_server.notifiers.webhook import WebhookNotifier
logger = logging.getLogger("mcp_server.notifiers.composite")
class CompositeNotifier(BaseNotifier):
"""Aggregates multiple notifiers and broadcasts events across all available channels."""
def __init__(self, notifiers: list[BaseNotifier] | None = None):
if notifiers is not None:
self._notifiers = notifiers
else:
self._notifiers = [
FileNotifier(),
AgentApiNotifier(),
WebhookNotifier(),
DesktopNotifier(),
]
@property
def name(self) -> str:
return "composite"
def is_available(self) -> bool:
return any(n.is_available() for n in self._notifiers)
def register_notifier(self, notifier: BaseNotifier) -> None:
"""Registers an additional custom notifier."""
self._notifiers.append(notifier)
def notify(
self,
conversation_id: str,
message: str,
title: str | None = None,
event_type: str = "completed",
payload: dict[str, Any] | None = None,
) -> bool:
any_success = False
for notifier in self._notifiers:
if notifier.is_available():
try:
success = notifier.notify(
conversation_id=conversation_id,
message=message,
title=title,
event_type=event_type,
payload=payload,
)
if success:
any_success = True
except Exception as e:
logger.debug(f"Notifier '{notifier.name}' encountered error: {e}")
return any_success
+92
View File
@@ -0,0 +1,92 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Desktop Toast / OS notification adapter."""
import logging
import os
import shutil
import subprocess
import sys
from typing import Any
from mcp_server.notifiers.base import BaseNotifier
logger = logging.getLogger("mcp_server.notifiers.desktop")
class DesktopNotifier(BaseNotifier):
"""Notifier that displays native desktop notifications when explicitly enabled."""
@property
def name(self) -> str:
return "desktop"
def is_available(self) -> bool:
return os.getenv("ARTEMIS_DESKTOP_NOTIFY", "").lower() in ("1", "true", "yes")
def notify(
self,
conversation_id: str,
message: str,
title: str | None = None,
event_type: str = "completed",
payload: dict[str, Any] | None = None,
) -> bool:
if not self.is_available():
return False
header = title or f"☕ Artemis Task {event_type.capitalize()}"
clean_body = message.split("\n\n")[0][:120]
try:
if sys.platform == "linux":
if shutil.which("notify-send"):
subprocess.run(
["notify-send", header, clean_body],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
timeout=3,
)
return True
elif sys.platform == "darwin":
if shutil.which("osascript"):
script = f'display notification "{clean_body}" with title "{header}"'
subprocess.run(
["osascript", "-e", script],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
timeout=3,
)
return True
elif sys.platform == "win32":
ps_cmd = (
f"[Windows.UI.Notifications.ToastNotificationManager, Windows.UI.Notifications, ContentType = WindowsRuntime] > $null; "
f"$template = [Windows.UI.Notifications.ToastNotificationManager]::GetTemplateContent([Windows.UI.Notifications.ToastTemplateType]::ToastText02); "
f'$textNodes = $template.GetElementsByTagName("text"); '
f'$textNodes.Item(0).AppendChild($template.CreateTextNode("{header}")) > $null; '
f'$textNodes.Item(1).AppendChild($template.CreateTextNode("{clean_body}")) > $null; '
f"$toast = [Windows.UI.Notifications.ToastNotification]::new($template); "
f'[Windows.UI.Notifications.ToastNotificationManager]::CreateToastNotifier("Artemis").Show($toast);'
)
subprocess.run(
["powershell", "-NoProfile", "-Command", ps_cmd],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
timeout=5,
)
return True
except Exception as e:
logger.debug(f"Desktop notification failed: {e}")
return False
+69
View File
@@ -0,0 +1,69 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""File-based notification recorder."""
import json
import logging
import os
import time
from typing import Any
from mcp_server.notifiers.base import BaseNotifier
from mcp_server.utils import trace_store
logger = logging.getLogger("mcp_server.notifiers.file")
class FileNotifier(BaseNotifier):
"""Notifier that records all notification events directly to the trace directory."""
@property
def name(self) -> str:
return "file"
def is_available(self) -> bool:
return True
def notify(
self,
conversation_id: str,
message: str,
title: str | None = None,
event_type: str = "completed",
payload: dict[str, Any] | None = None,
) -> bool:
trace_id = (payload or {}).get("trace_id")
if not trace_id:
return False
try:
trace_dir = trace_store.get_trace_dir(trace_id)
os.makedirs(trace_dir, exist_ok=True)
log_file = os.path.join(trace_dir, "notifications.jsonl")
entry = {
"timestamp": time.time(),
"event_type": event_type,
"title": title,
"conversation_id": conversation_id,
"message": message,
"payload": payload or {},
}
with open(log_file, "a", encoding="utf-8") as f:
f.write(json.dumps(entry, ensure_ascii=False) + "\n")
return True
except Exception as e:
logger.debug(f"File notification record failed: {e}")
return False
+87
View File
@@ -0,0 +1,87 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Webhook Notifier for OpenClaw, CI pipelines, and external gateways."""
import json
import logging
import os
import time
from typing import Any
import urllib.request
from mcp_server.notifiers.base import BaseNotifier
logger = logging.getLogger("mcp_server.notifiers.webhook")
class WebhookNotifier(BaseNotifier):
"""Notifier that posts execution events to HTTP/HTTPS webhooks (e.g. OpenClaw / CI / Slack)."""
ENV_VARS = [
"OPENCLAW_WEBHOOK_URL",
"MCP_NOTIFICATION_WEBHOOK",
"ARTEMIS_WEBHOOK_URL",
]
@property
def name(self) -> str:
return "webhook"
def _get_webhook_url(self) -> str | None:
for var in self.ENV_VARS:
val = os.getenv(var)
if val and val.startswith(("http://", "https://")):
return val
return None
def is_available(self) -> bool:
return self._get_webhook_url() is not None
def notify(
self,
conversation_id: str,
message: str,
title: str | None = None,
event_type: str = "completed",
payload: dict[str, Any] | None = None,
) -> bool:
url = self._get_webhook_url()
if not url:
return False
data = {
"event": event_type,
"title": title or f"Artemis Task {event_type.capitalize()}",
"conversation_id": conversation_id,
"message": message,
"timestamp": time.time(),
"payload": payload or {},
}
try:
req_data = json.dumps(data).encode("utf-8")
req = urllib.request.Request(
url,
data=req_data,
headers={"Content-Type": "application/json", "User-Agent": "Artemis-MCP/3.0"},
method="POST",
)
with urllib.request.urlopen(req, timeout=5) as response:
if response.status in (200, 201, 202, 204):
logger.info(f"Webhook notification delivered to {url}.")
return True
except Exception as e:
logger.warning(f"Failed to deliver webhook notification to {url}: {e}")
return False
+48
View File
@@ -0,0 +1,48 @@
# Mobile Testing Mindset (ARTEMIS Integration)
As a senior test engineer, you must ensure that all test code you write is **fully executable, stable, and exceptionally robust**. You must never guess or assume how an interaction behaves on a mobile device, no matter how simple the task may seem. Instead, you must leverage **ARTEMIS**, a powerful mobile UI automation framework, to discover and verify the exact execution flow, and author your test scripts based strictly on these concrete, verified steps. Furthermore, if you need to investigate or research specific software or hardware behaviors, you should utilize ARTEMIS to perform active explorations on the device, whether by mimicking human interactions or executing precise ADB commands.
### 1. The Runnable Code Principle & ARTEMIS Exploration
- **When tasked with authoring tests**, your primary goal is to deliver **runnable, production-grade test code**.
- Before writing any test code, you must use the ARTEMIS MCP tools to interactively run and explore the target application. This allows you to discover the exact sequence of UI states, transitions, and required interactions.
- Analyze the user's target testing framework to exploit its native capabilities and maximize test stability.
- **Timing & Latency Management (Exploration vs. Execution)**:
- **Precise Timing in Final Code**: While ARTEMIS's AI exploration inherently involves model latency and is not strictly time-precise, you must bridge this gap in your final deliverables. Use ARTEMIS to discover and verify the interaction path, then implement exact, deterministic timing and wait conditions (`sleep`, explicit/implicit waits) in your authored test scripts, as local test execution runs without LLM overhead.
- **Compensating for Model Latency During Exploration**: When delegating exploratory tasks to ARTEMIS that involve waiting periods (e.g., waiting 30 seconds for a page load or timer), adjust the requested wait duration in your task description based on the selected model's natural step interval:
- **Flash Model**: The average processing interval between steps is roughly **5 seconds**. For a required 30-second delay, instruct the agent to wait for approximately **25 seconds**.
- **Pro Model**: The processing interval between turns is typically **~30 seconds** (due to multi-agent planning and verification). The natural pipeline delay often covers the required waiting time without adding long explicit delay commands.
- **Pragmatic Timing Judgment**: When a user request mentions performing an action for a specific duration (e.g., "stay on this screen for 2 minutes"), evaluate whether exact timing is functionally critical. Often, these durations are rough guidelines rather than strict test constraints—exercise flexibility and pragmatic engineering judgment to achieve the verification goal efficiently.
### 2. ARTEMIS Closed-Loop Architecture Mastery
Deeply understand and select between ARTEMIS's dual execution models (**ARTEMIS Flash** and **ARTEMIS Pro**) based on the task scenario, and master their corresponding workflows:
- **ARTEMIS Flash (Fast / Reactive Model)**:
- **Applicable Scenarios**: Designed for simple, highly deterministic, short-to-medium range (typically `< 30-35` steps) lightweight UI operations or direct automation workflows.
- **Execution Mechanism (Reactive Loop)**: Does not enter the complex LangGraph multi-node graph orchestration, and has no independent Planner, Validator, or Summarizer nodes. Instead, `FlashRunner` directly receives the user's goal, the latest structural layout (XML), and the real-time screenshot, rapidly performing "Observe-Think-Act" via a single LLM, calling action tools until finally reporting the task status.
- **ARTEMIS Pro (Deep / Multi-Agent Graph-Driven Closed-Loop Model)**:
- **Applicable Scenarios**: Designed for long-range, highly complex, dynamic multi-branch tasks, or tasks requiring deep system diagnostics, asynchronous validation, and self-healing exploratory retries.
- **Execution Mechanism (Plan-Execute-Analyze-Summarize Closed-Loop)**:
- **Planner**: Deconstructs complex, high-level testing goals into a structured, step-by-step execution plan.
- **Operator**: Consumes the plan, analyzes the current screen state (via screenshots and XML layout trees), and executes precise device interactions.
- **Safety Net & Self-Healing (Failure Analyzer)**: Every action is vetted by an execution safety net. If an action fails or is blocked, the **Failure Analyzer** is triggered to run local step-by-step recovery sequences, achieving autonomous self-healing. This loop continues iteratively until the task terminates.
- **Outputter (Optional)**: Synthesizes the entire execution trace into a human-readable report detailing every action step and visual result.
### 3. Device & Environment Constraints
- **ADB & File Transfer**: While ARTEMIS excels at device automation, it operates within the device boundary. To extract diagnostic files, logs, or test artifacts from the mobile device to the host PC for further analysis, you must manually write and execute appropriate `adb` commands (e.g., `adb pull`).
- **Hardware Prerequisites**: Running ARTEMIS requires a physically connected, fully authorized Android device (e.g., a Pixel phone) or an active emulator.
- **Single-Device Policy**: ARTEMIS can only operate a single device at any given time. Ensure your workflows and code respect this concurrency limit.
### 4. Robust Test Code Design (The "Dynamic-First, Coordinate-Fallback" Philosophy)
*If your task involves authoring test code, you must adhere to the following design principles for maximum reliability:*
- **Adapt to Framework Capabilities**: You must first assess what locating mechanisms the user's test framework supports (e.g., resource IDs, XPath, text matching, OCR, image template matching, or absolute/relative coordinates).
- **The Core Principle: Dynamic-First, Coordinate-Fallback**:
- Wherever supported by the framework, **always prioritize dynamic element locating** (using IDs, text, OCR, etc.) to ensure the test code can withstand UI layout drifts and resolution changes.
- Use **absolute coordinates as a reliable fallback** to guarantee execution success when dynamic locators fail or are unavailable.
- **Implementing Resilient Locating Patterns**:
- **Dual-Capable Frameworks**: If the framework supports both dynamic and coordinate-based locating, implement the **Try-Catch Fallback Pattern**:
1. **Try**: Attempt to interact with the element using dynamic locators (IDs, OCR, text) for maximum resilience against UI changes.
2. **Catch**: If the dynamic attempt fails, fall back to the precise absolute coordinates verified during your ARTEMIS exploration.
- **Coordinate-Only Frameworks**: If the framework only supports coordinates, ensure the coordinates are well-documented, and where possible, parameterized or made relative to screen boundaries to mitigate resolution differences.
- **Dynamic-Only Frameworks**: If the framework does not support coordinate-based clicks, focus entirely on generating highly robust dynamic locators, leveraging ARTEMIS's element descriptions and XML tree analysis.
+69
View File
@@ -0,0 +1,69 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Universal Model Context Protocol (MCP) Server for ARTEMIS."""
import datetime
import os
import sys
# Ensure repository root is in sys.path
PROJECT_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
if PROJECT_ROOT not in sys.path:
sys.path.insert(0, PROJECT_ROOT)
# Debug logging and stderr tee to capture crashes without breaking MCP stdio JSON-RPC
try:
_mcp_log_dir = os.path.join(PROJECT_ROOT, "scratch")
os.makedirs(_mcp_log_dir, exist_ok=True)
with open(os.path.join(_mcp_log_dir, "mcp_launch_debug.log"), "a", encoding="utf-8") as _f:
_f.write(
f"[{datetime.datetime.now()}] Artemis MCP Server launched! sys.argv: {sys.argv}, CWD: {os.getcwd()}\n"
)
class StderrTee:
def __init__(self, original, log_path):
self.original = original
self.log_file = open(log_path, "a", encoding="utf-8")
def write(self, data):
self.original.write(data)
self.log_file.write(data)
self.log_file.flush()
def flush(self):
self.original.flush()
self.log_file.flush()
sys.stderr = StderrTee(sys.stderr, os.path.join(_mcp_log_dir, "mcp_stderr.log"))
except Exception:
pass
# 1. Import shared FastMCP instance
from mcp_server.base import mcp
# 2. Import tools to register them
def main(transport: str = "stdio", host: str = "127.0.0.1", port: int = 8001):
"""Main entrypoint to run the Artemis Mobile Agent MCP server."""
if transport.lower() == "sse":
mcp.run(transport="sse", host=host, port=port)
else:
mcp.run(transport="stdio")
if __name__ == "__main__":
main()
+27
View File
@@ -0,0 +1,27 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""MCP Tools package for ARTEMIS."""
from mcp_server.tools.device_state import mobile_get_device_state
from mcp_server.tools.inspect_trace import mobile_inspect_trace
from mcp_server.tools.task_manager import mobile_manage_task
from mcp_server.tools.task_runner import mobile_run_task
__all__ = [
"mobile_run_task",
"mobile_manage_task",
"mobile_get_device_state",
"mobile_inspect_trace",
]
+104
View File
@@ -0,0 +1,104 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""MCP Tool: mobile_get_device_state."""
import base64
import os
from mcp_server.base import mcp
from artemis.mcp.adb_server import _get_controller
from mcp_server.utils import env_utils
from artemis.utils.ocr_xml_fusion import (
fuse_ocr_with_xml,
_detect_status_bar_height,
_crop_image_remove_status_bar,
_map_coordinates_back,
)
from artemis.utils.ocr_api import perform_ocr
from artemis.utils.visualization import format_minimal_list_with_elements
@mcp.tool()
async def mobile_get_device_state(view_type: str) -> str:
"""Real-time mobile device state observer (for debugging and validation).
Use this tool to inspect the latest mobile status or to assist in tracking
the subagent's task progress.
Retrieves a real-time screenshot (returns local file URI) or a simplified
element tree text list of the currently connected mobile device.
Args:
view_type: The observation type. Supported values: - "screenshot":
Captures the real-time mobile screen and saves it to the workspace,
returning the local URI path of the image file (e.g.,
"file:///path/to/screenshot.jpg"). - "hierarchy": Directly returns the
simplified text-labeled element list, which is identical to what the
mobile UI automation subagent sees when making decisions.
"""
try:
controller = _get_controller()
device_width = controller.ctx.device.device_width
device_height = controller.ctx.device.device_height
except Exception as e:
return f"Error: Failed to initialize/lock Android device controller: {e}"
try:
device_data = await controller.get_screen_data()
latest_screenshot_b64 = device_data.base64
xml_hierarchy = device_data.elements
project_root = env_utils.get_project_root()
if view_type == "screenshot":
screenshot_bytes = base64.b64decode(latest_screenshot_b64)
device_id = controller.ctx.device.device_id
safe_device_id = "".join(
[c if c.isalnum() or c in ("-", "_") else "_" for c in device_id]
)
screenshot_filename = f"live_screenshot_{safe_device_id}.jpg"
screenshot_path = os.path.join(project_root, screenshot_filename)
with open(screenshot_path, "wb") as f:
f.write(screenshot_bytes)
return f"file://{screenshot_path}"
elif view_type == "hierarchy":
try:
screen_height = device_data.height
status_bar_height = _detect_status_bar_height(xml_hierarchy, screen_height)
if status_bar_height > 0:
cropped_b64, _, _ = _crop_image_remove_status_bar(
latest_screenshot_b64, status_bar_height
)
raw_ocr_results = await perform_ocr(cropped_b64)
ocr_results = _map_coordinates_back(raw_ocr_results, status_bar_height)
else:
ocr_results = await perform_ocr(latest_screenshot_b64)
except Exception:
ocr_results = []
fused_xml = fuse_ocr_with_xml(xml_hierarchy, ocr_results)
hierarchy_text, elements, labels = format_minimal_list_with_elements(
fused_xml, width=device_width, height=device_height
)
return hierarchy_text
else:
return f"Error: Invalid view_type '{view_type}'. Supported types are 'screenshot' and 'hierarchy'."
except Exception as e:
return f"Error: An unexpected error occurred while communicating with the device: {e}"
+465
View File
@@ -0,0 +1,465 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""MCP Tool: mobile_inspect_trace."""
import json
import math
import os
import sqlite3
import sys
from typing import Any
from PIL import Image, ImageDraw
from mcp_server.base import mcp
from mcp_server.utils import env_utils, trace_store
from artemis.utils.task_tree import (
_render_step_detailed,
build_plan_and_history,
)
def _draw_action_overlay(bg_image_path: str, action_obj: dict, output_path: str) -> bool:
"""Draws the action (tap, swipe, etc.) on the image and saves it."""
try:
with Image.open(bg_image_path) as img:
draw_img = img.convert("RGBA")
draw = ImageDraw.Draw(draw_img)
act_type = action_obj.get("action") or action_obj.get("name")
coords = action_obj.get("coordinates") or action_obj.get("target")
if not act_type or not coords:
return False
fill_color = (255, 0, 0, 80)
outline_color = (255, 0, 0, 255)
outline_width = 4
if act_type in (
"tap",
"click",
"long_press",
"focus_and_input_text",
"focus_and_clear_text",
):
if isinstance(coords, list) and len(coords) == 2:
x, y = coords[0], coords[1]
r = 25
draw.ellipse(
[x - r, y - r, x + r, y + r],
fill=fill_color,
outline=outline_color,
width=outline_width,
)
r_center = 4
draw.ellipse(
[
x - r_center,
y - r_center,
x + r_center,
y + r_center,
],
fill=outline_color,
outline=outline_color,
)
elif act_type == "swipe":
if isinstance(coords, list) and len(coords) == 4:
x1, y1, x2, y2 = coords[0], coords[1], coords[2], coords[3]
draw.line([x1, y1, x2, y2], fill=outline_color, width=6)
r_start = 8
draw.ellipse(
[
x1 - r_start,
y1 - r_start,
x1 + r_start,
y1 + r_start,
],
fill=outline_color,
)
dx = x2 - x1
dy = y2 - y1
angle = math.atan2(dy, dx)
arrow_length = 30
arrow_angle = math.pi / 6
p1_x = x2 - arrow_length * math.cos(angle - arrow_angle)
p1_y = y2 - arrow_length * math.sin(angle - arrow_angle)
p2_x = x2 - arrow_length * math.cos(angle + arrow_angle)
p2_y = y2 - arrow_length * math.sin(angle + arrow_angle)
draw.polygon([x2, y2, p1_x, p1_y, p2_x, p2_y], fill=outline_color)
final_img = draw_img.convert("RGB")
final_img.save(output_path, "JPEG", quality=85)
return True
except Exception as e:
print(f"Error drawing action overlay: {e}", file=sys.stderr)
return False
@mcp.tool()
async def mobile_inspect_trace(action: str, trace_id: str, step_number: int | None = None) -> Any:
"""A specialized trace data retrieval tool designed to extract execution details of mobile automation tasks.
Use this tool DURING a running task or AFTER it has finished (or failed) to
monitor progress, verify answers, diagnose agent errors, and debug.
It provides comprehensive runtime information, raw agent outputs, and visual
screenshots for both **Flash** and **Pro** tasks.
### Available Actions:
- **'view_summary'**: Retrieves a high-level execution summary across all
steps.
* **Pro Model**: Displays a hierarchical task plan summary, showing the
status and descriptions of all completed, active, and pending steps.
* **Flash Model**: Displays an all-in-one execution chain containing the
complete reasoning (thoughts/motivation) and action taken (`click`,
`input_text`, etc.) for every single step.
- **'view_step_screenshots'**: Retrieves local file paths of screenshots for
a specific step (supports both Flash and Pro). Returns:
1. `before_screenshot`: The original device screenshot observed by the
agent right before taking the action.
2. `after_screenshot`: A post-action screenshot capturing the failure or
mismatch state (if the action failed or was intercepted). Note: This is
conditional/optional and will be `None` (`null`) for normal/successful
steps where no extra post-action screenshot was captured.
3. `action_overlay_screenshot`: The `before_screenshot` with the agent's
action visually marked (e.g., red circle for taps, red arrow for
swipes). CRITICAL for verifying if the agent tapped the correct UI
element.
- **'view_step_details'**: Retrieves deep runtime logs, raw VLM thoughts,
and exact actions for a specific step.
* **Pro Model**: Fully supported. Returns detailed per-step diagnostic
data.
* **Flash Model**: DO NOT USE. (Flash reasoning and actions are already
fully included in 'view_summary', making this action redundant).
Args:
action: STR. **REQUIRED**. The retrieval action to perform
(`"view_summary"`, `"view_step_screenshots"`, or
`"view_step_details"`).
trace_id: STR. **REQUIRED**. The unique session identifier of the task
(returned by `mobile_run_task`).
step_number: INT. **CONDITIONAL**. The 1-indexed step number to query. -
You MUST provide this if `action="view_step_screenshots"` or
`action="view_step_details"`. - You MUST omit or leave this empty if
`action="view_summary"`.
"""
project_root = env_utils.get_project_root()
db_path = os.path.join(trace_store.TRACES_DIR, "data_engine.db")
if not os.path.exists(db_path):
db_path = os.path.join(project_root, "traces", "data_engine.db")
if not os.path.exists(db_path):
return {
"error": "Database not found",
"message": f"Data engine database does not exist at {db_path} yet.",
}
# 1. Handle Action: "view_summary"
if action == "view_summary":
plan_content = None
plan_path = os.path.join(trace_store.get_trace_dir(trace_id), "notes", "task_plan.md")
if not os.path.exists(plan_path):
plan_path = os.path.join(trace_store.TRACES_DIR, "notes", "task_plan.md")
if os.path.exists(plan_path):
try:
with open(plan_path, encoding="utf-8") as f:
plan_content = f.read()
except Exception:
pass
try:
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
cursor = conn.cursor()
cursor.execute(
"SELECT start_time FROM sessions WHERE session_id = ?",
(trace_id,),
)
session_row = cursor.fetchone()
session_start_time = session_row["start_time"] if session_row else None
cursor.execute(
"SELECT * FROM steps WHERE session_id = ? ORDER BY step_number ASC",
(trace_id,),
)
rows = cursor.fetchall()
steps = []
for r in rows:
step_dict = dict(r)
for col in ("action_taken", "last_execution_result", "extra_metadata"):
if step_dict.get(col):
try:
step_dict[col] = json.loads(step_dict[col])
except Exception:
pass
cursor.execute(
"SELECT type, name, status, timestamp, duration, payload FROM traces WHERE step_id = ? ORDER BY timestamp ASC",
(step_dict["step_id"],),
)
trace_rows = cursor.fetchall()
events = []
for tr in trace_rows:
t_dict = dict(tr)
if t_dict.get("payload"):
try:
t_dict["payload"] = json.loads(t_dict["payload"])
except Exception:
pass
events.append(t_dict)
step_dict["interleaved_events"] = events
if session_start_time and step_dict.get("timestamp"):
relative_time_val = step_dict["timestamp"] - session_start_time
step_dict["relative_time"] = f"{relative_time_val:.1f}s"
else:
step_dict["relative_time"] = "N/A"
steps.append(step_dict)
conn.close()
except Exception as e:
return {
"error": "Database query error",
"message": f"Failed to retrieve steps from database: {e}",
}
status_data = trace_store.read_status(trace_id)
is_flash = status_data and status_data.get("model", "").lower() == "flash"
if is_flash:
summary_lines = [
"# ARTEMIS Flash Execution Summary",
f"**Session ID:** `{trace_id}` | **Model:** Flash\n",
"---",
"## Step-by-Step Execution Chain\n",
]
if not steps:
summary_lines.append("_No action steps have been recorded yet._")
else:
for s in steps:
s_num = s.get("step_number", "?")
rel_t = s.get("relative_time", "N/A")
dur_val = s.get("duration")
if dur_val is None and s.get("interleaved_events"):
events = s["interleaved_events"]
timestamps = [e.get("timestamp", 0) for e in events if e.get("timestamp")]
if timestamps:
dur_val = max(timestamps) - min(timestamps)
dur_str = f"{dur_val:.1f}s" if dur_val is not None and dur_val > 0 else "< 0.5s"
thought = s.get("operator_raw_thinking")
if not thought or not thought.strip():
for ev in s.get("interleaved_events", []):
if ev.get("type") in ("raw_thinking", "llm_call") and ev.get("payload"):
p = ev["payload"]
if isinstance(p, dict) and p.get("response_text"):
thought = p["response_text"]
break
if not thought or not thought.strip():
thought = "(Direct action without preliminary text reasoning)"
act = s.get("action_taken") or "N/A"
if isinstance(act, dict) and "action" in act:
act_str = f"{act['action']}({act.get('args', act.get('coordinates', ''))})"
else:
act_str = str(act)
res = s.get("last_execution_result") or "N/A"
summary_lines.append(f"### Step {s_num} (at {rel_t}, duration {dur_str})")
summary_lines.append(f"- **Thinking / Motivation:** {thought}")
summary_lines.append(f"- **Action Taken:** `{act_str}` -> `{res}`\n")
return "\n".join(summary_lines)
try:
summary_markdown = build_plan_and_history(
task_plan=plan_content or "No task plan created yet.",
steps=steps,
current_subgoal_hash="default",
min_summaries=len(steps),
last_n_detailed=0,
all_detailed=False,
)
return summary_markdown
except Exception as e:
return {
"error": "Formatting error",
"message": f"Failed to format plan and history using Artemis task tree: {e}",
}
# 2. Handle Actions requiring step_number
elif action in ("view_step_screenshots", "view_step_details"):
status_data = trace_store.read_status(trace_id)
is_flash = status_data and status_data.get("model", "").lower() == "flash"
if action == "view_step_details" and is_flash:
return {
"error": "Not applicable for Flash model",
"message": (
"Action 'view_step_details' is not needed for Flash model runs. "
"For Flash tasks, 'view_summary' already outputs the complete "
"step-by-step reasoning (thoughts) and executed actions for the entire run."
),
}
if step_number is None:
return {
"error": "Missing parameter",
"message": f"Parameter 'step_number' is required for action '{action}'.",
}
try:
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
cursor = conn.cursor()
cursor.execute(
"SELECT * FROM steps WHERE session_id = ? AND step_number = ?",
(trace_id, step_number),
)
row = cursor.fetchone()
if not row:
conn.close()
return {
"error": "Step not found",
"message": f"Step number {step_number} not found for trace '{trace_id}'.",
}
step_dict = dict(row)
for col in ("action_taken", "last_execution_result", "extra_metadata"):
if step_dict.get(col):
try:
step_dict[col] = json.loads(step_dict[col])
except Exception:
pass
cursor.execute(
"SELECT start_time FROM sessions WHERE session_id = ?",
(trace_id,),
)
session_row = cursor.fetchone()
session_start_time = session_row["start_time"] if session_row else None
cursor.execute(
"SELECT type, name, status, timestamp, duration, payload FROM traces WHERE step_id = ? ORDER BY timestamp ASC",
(step_dict["step_id"],),
)
trace_rows = cursor.fetchall()
events = []
for tr in trace_rows:
t_dict = dict(tr)
if t_dict.get("payload"):
try:
t_dict["payload"] = json.loads(t_dict["payload"])
except Exception:
pass
events.append(t_dict)
step_dict["interleaved_events"] = events
conn.close()
except Exception as e:
return {
"error": "Database error",
"message": f"Database query failed: {e}",
}
# 3. Handle Action: "view_step_screenshots"
if action == "view_step_screenshots":
images_dir = os.path.join(trace_store.TRACES_DIR, "images")
if not os.path.exists(images_dir):
images_dir = os.path.join(project_root, "traces", "images")
pre_image = None
post_image = None
overlay_image = None
if step_dict.get("pre_image_name"):
name = step_dict["pre_image_name"]
if not name.endswith(".jpg"):
name += ".jpg"
path = os.path.join(images_dir, name)
if os.path.exists(path):
pre_image = f"file://{path}"
overlay_dir = trace_store.get_trace_dir(trace_id)
os.makedirs(overlay_dir, exist_ok=True)
overlay_filename = f"step_{step_dict['step_number']}_overlay.jpg"
overlay_abs_path = os.path.join(overlay_dir, overlay_filename)
action_obj = step_dict.get("action_taken")
action_parsed = (
action_obj[0] if isinstance(action_obj, list) and action_obj else action_obj
)
if isinstance(action_parsed, dict) and action_parsed.get("action"):
if not os.path.exists(overlay_abs_path):
success_draw = _draw_action_overlay(
path, action_parsed, overlay_abs_path
)
if success_draw:
overlay_image = f"file://{overlay_abs_path}"
else:
overlay_image = f"file://{overlay_abs_path}"
if step_dict.get("post_image_name"):
name = step_dict["post_image_name"]
if not name.endswith(".jpg"):
name += ".jpg"
path = os.path.join(images_dir, name)
if os.path.exists(path):
post_image = f"file://{path}"
return {
"trace_id": trace_id,
"step_number": step_dict["step_number"],
"before_screenshot": pre_image,
"after_screenshot": post_image,
"action_overlay_screenshot": overlay_image,
}
# 4. Handle Action: "view_step_details"
elif action == "view_step_details":
relative_time_str = "N/A"
if session_start_time and step_dict.get("timestamp"):
relative_time_val = step_dict["timestamp"] - session_start_time
relative_time_str = f"{relative_time_val:.1f}s"
rendered_text = _render_step_detailed(
step=step_dict,
relative_time=relative_time_str,
summary=step_dict.get("summary"),
action=step_dict.get("action_taken"),
result=step_dict.get("last_execution_result"),
is_most_recent=False,
for_failure_analyzer=False,
)
return {
"trace_id": trace_id,
"step_number": step_dict["step_number"],
"details": rendered_text,
}
else:
return {
"error": "Invalid action",
"message": (
f"Action '{action}' is not supported. Supported actions: "
"'view_summary', 'view_step_screenshots', 'view_step_details'."
),
}
+335
View File
@@ -0,0 +1,335 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""MCP Tool: mobile_manage_task."""
import json
import os
import signal
import sqlite3
import sys
import time
from typing import Any
from mcp_server.base import mcp
from mcp_server.notifiers import notify
from mcp_server.utils import env_utils, trace_store
@mcp.tool()
def mobile_manage_task(
action: str, trace_id: str, instruction: str | None = None
) -> dict[str, Any]:
"""Manages the lifecycle and retrieves the status of a background mobile automation task.
This is your primary diagnostic and control tool. You MUST use this tool to
poll the task status
when your 1-minute fallback timer triggers (as requested by
`mobile_run_task`). You can also use
it to gracefully abort a task or forcefully correct the subagent's behavior
mid-flight.
### Available Actions:
- **'status'**: Retrieves execution progress, elapsed time, and metadata.
Returns a JSON object.
* **How to use the output**:
- For **Pro Model**, check `progress.task_plan` to see if the
agent's long-term plan aligns with your goal.
- For **Flash Model**, monitor `current_turn`, `latest_action`, and
`latest_thought`. If the turn count increases rapidly without
progressing, or the thought indicates it is stuck, you should
intervene.
- **'inject_instruction'**: Injects real-time guidance when the subagent
makes a mistake, gets stuck, or loops.
* **Pro Model**: Applied at the start of the next turn's planning phase.
* **Flash Model**: Applied at the start of the next reactive execution
loop.
* *Note*: You MUST provide the `instruction` argument for this action.
- **'stop'**: Forcefully terminates the subagent (Pro or Flash), immediately
halting device interactions and releasing the device. Use this if the task
is complete, irreparably broken, or running out of control.
Args:
action: STR. **REQUIRED**. The management action to perform. Must be
exactly `"status"`, `"stop"`, or `"inject_instruction"`.
trace_id: STR. **REQUIRED**. The unique session identifier of the task
(returned by `mobile_run_task`).
instruction: STR. **CONDITIONAL**. The real-time correction or guidance
string. - You MUST provide this if `action="inject_instruction"`. -
You MUST leave this empty or omit it if `action="status"` or
`action="stop"`.
"""
status_data = trace_store.read_status(trace_id)
if not status_data:
return {
"trace_id": trace_id,
"status": "unknown",
"message": f"Trace ID '{trace_id}' not found.",
}
current_status = status_data.get("status", "unknown")
pid = status_data.get("pid")
is_alive = False
if pid and current_status in ("running", "pending"):
try:
os.kill(pid, 0)
is_alive = True
except OSError:
current_status = "failed"
error_text = "Task runner process terminated unexpectedly."
trace_store.update_trace_status(trace_id, "failed", error=error_text)
conv_id = status_data.get("conversation_id")
if conv_id:
try:
notify(
conversation_id=conv_id,
message=f"Artemis background task died unexpectedly for trace '{trace_id}'. Error: {error_text}",
event_type="failed",
payload={"trace_id": trace_id, "error": error_text},
)
except Exception:
pass
status_data = trace_store.read_status(trace_id) or status_data
trace_dir = trace_store.get_trace_dir(trace_id)
if action == "status":
start_time = status_data.get("start_time")
end_time = status_data.get("end_time") or time.time()
elapsed = round(end_time - start_time, 1) if start_time else 0
response: dict[str, Any] = {
"trace_id": trace_id,
"status": current_status,
"elapsed_seconds": elapsed,
"task_desc": status_data.get("task_desc"),
"model": status_data.get("model"),
"stdout_log": os.path.join(trace_dir, "stdout.log"),
"stderr_log": os.path.join(trace_dir, "stderr.log"),
}
if status_data.get("model", "").lower() != "flash":
response["notes_dir"] = os.path.join(trace_dir, "notes")
if current_status == "failed":
response["error"] = status_data.get("error")
elif current_status == "completed":
response["result"] = status_data.get("result")
if current_status in ("running", "pending") and is_alive:
progress: dict[str, Any] = {}
project_root = env_utils.get_project_root()
db_path = os.path.join(trace_store.TRACES_DIR, "data_engine.db")
if not os.path.exists(db_path):
db_path = os.path.join(project_root, "traces", "data_engine.db")
session_id = None
if os.path.exists(db_path):
try:
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
cursor = conn.cursor()
cursor.execute(
"SELECT session_id FROM sessions WHERE pid = ? ORDER BY start_time DESC LIMIT 1",
(pid,),
)
session_row = cursor.fetchone()
if session_row:
session_id = session_row["session_id"]
conn.close()
except Exception as db_err:
print(f"Error querying data_engine.db for session: {db_err}", file=sys.stderr)
if session_id:
if status_data.get("model", "").lower() == "flash":
progress["phase"] = "running_flash_loop"
if os.path.exists(db_path):
try:
conn2 = sqlite3.connect(db_path)
conn2.row_factory = sqlite3.Row
cur2 = conn2.cursor()
cur2.execute(
"SELECT count(*) as cnt FROM traces WHERE session_id = ? AND type = 'llm_call'",
(session_id,),
)
row_cnt = cur2.fetchone()
if row_cnt:
progress["current_turn"] = row_cnt["cnt"]
cur2.execute(
"SELECT payload FROM traces WHERE session_id = ? AND type in ('raw_thinking', 'llm_call') "
"AND status = 'success' ORDER BY timestamp DESC LIMIT 1",
(session_id,),
)
row_thought = cur2.fetchone()
if row_thought and row_thought["payload"]:
try:
p_obj = json.loads(row_thought["payload"])
if isinstance(p_obj, dict):
if "thought" in p_obj:
progress["latest_thought"] = p_obj["thought"]
elif (
"response" in p_obj
and isinstance(p_obj["response"], list)
and p_obj["response"]
):
progress["latest_thought"] = p_obj["response"][0].get(
"text", ""
)
except Exception:
pass
cur2.execute(
"SELECT name, payload FROM traces WHERE session_id = ? AND type = 'tool' "
"AND status = 'success' ORDER BY timestamp DESC LIMIT 1",
(session_id,),
)
row_action = cur2.fetchone()
if row_action:
act_name = row_action["name"]
act_payload = {}
if row_action["payload"]:
try:
act_payload = json.loads(row_action["payload"]).get(
"args", {}
)
except Exception:
pass
progress["latest_action"] = (
f"{act_name}({act_payload})" if act_payload else act_name
)
conn2.close()
except Exception as q_err:
print(
f"Error querying flash progress from data_engine.db: {q_err}",
file=sys.stderr,
)
else:
plan_content = None
plan_path = os.path.join(
trace_store.TRACES_DIR, session_id, "notes", "task_plan.md"
)
if not os.path.exists(plan_path):
plan_path = os.path.join(trace_dir, "notes", "task_plan.md")
if os.path.exists(plan_path):
try:
with open(plan_path, encoding="utf-8") as f:
plan_content = f.read()
except Exception:
pass
progress["task_plan"] = plan_content
else:
progress["phase"] = "initializing"
progress["status_message"] = "Spawning background runner..."
response["progress"] = progress
return response
elif action == "stop":
if current_status not in ("running", "pending"):
return {
"trace_id": trace_id,
"status": current_status,
"message": f"Cannot stop task because it is already in a terminal state: {current_status}",
}
if not pid:
return {
"trace_id": trace_id,
"status": current_status,
"message": "Process ID (PID) is missing from the task status. Cannot stop task.",
}
try:
os.kill(pid, signal.SIGTERM)
trace_store.update_trace_status(
trace_id, "cancelled", error="Task aborted by user request."
)
return {
"trace_id": trace_id,
"status": "cancelled",
"message": f"Successfully sent termination signal to background process {pid}.",
}
except ProcessLookupError:
trace_store.update_trace_status(
trace_id, "cancelled", error="Task aborted (process was already stopped)."
)
return {
"trace_id": trace_id,
"status": "cancelled",
"message": f"Process {pid} was not found; it may have already exited. Status marked as cancelled.",
}
except Exception as e:
return {
"trace_id": trace_id,
"status": current_status,
"message": f"Failed to terminate process {pid}: {e}",
}
elif action == "inject_instruction":
if not instruction:
return {
"trace_id": trace_id,
"status": current_status,
"message": "Missing required argument 'instruction' for action 'inject_instruction'.",
}
if current_status not in ("running", "pending"):
return {
"trace_id": trace_id,
"status": current_status,
"message": f"Cannot inject instruction because task is in a terminal state: {current_status}",
}
if not os.path.exists(trace_dir):
return {
"trace_id": trace_id,
"status": current_status,
"message": f"Trace directory not found: {trace_dir}",
}
instruction_path = os.path.join(trace_dir, "injected_instruction.json")
try:
with open(instruction_path, "w", encoding="utf-8") as f:
json.dump(
{
"instruction": instruction,
"timestamp": time.time(),
"status": "pending",
},
f,
indent=2,
)
return {
"trace_id": trace_id,
"status": current_status,
"message": f"Successfully injected instruction: '{instruction}'",
}
except Exception as e:
return {
"trace_id": trace_id,
"status": current_status,
"message": f"Failed to inject instruction: {e}",
}
else:
return {
"trace_id": trace_id,
"status": current_status,
"message": f"Invalid action '{action}'. Supported actions are 'status', 'stop', and 'inject_instruction'.",
}
+205
View File
@@ -0,0 +1,205 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""MCP Tool: mobile_run_task."""
import os
import subprocess
from typing import Any
import uuid
from mcp_server.base import mcp
from mcp_server.utils import env_utils, trace_store
@mcp.tool()
def mobile_run_task(
task_desc: str,
conversation_id: str,
model: str = "Flash",
locked_app_package: str | None = None,
app_path: str | None = None,
expected_output_desc: str | None = None,
) -> dict[str, Any]:
"""Starts a specialized mobile UI automation subagent to autonomously plan and execute tasks on a connected Android device.
With this tool, you can delegate complex mobile workflows and UI
interactions to a background agent.
This tool supports two execution models: **Flash** (for rapid,
straightforward UI scripts) and **Pro** (for complex, exploratory tasks
requiring deep reasoning).
### Model Selection Guide (CRITICAL FOR ROUTING)
You MUST choose the appropriate `model` based on the task complexity:
- **Flash**: Best for simple, deterministic tasks that can be completed
within **30 UI steps**. You should prioritize using this model to complete
the task.
*Limitations*: Flash CANNOT analyze video recordings, query system logs,
execute arbitrary ADB shell commands, maintain a structured task plan, or
write custom execution notes/reports. Do not use Flash if the task
requires exploration, complex error recovery, continuous monitoring/polling
(due to step count limits and lack of state persistence), or outputting
large amounts of detailed information (as Flash cannot access the
notes/scratchpad system to retain extensive details across steps; use
**Pro** instead).
- **Pro**: Required for complex, non-linear, or exploratory tasks (e.g.,
browsing deep lists, troubleshooting app crashes, tasks requiring multi-step
planning), continuous monitoring or polling tasks (e.g., periodically checking
UI state, polling until specific conditions or events occur, watching progress,
or waiting for background changes), OR when you need the subagent to extract,
remember, and output large amounts of detailed information. Pro is slower but
fully featured.
*Capabilities*: Supports full video analysis, system log querying, ADB
shell access, robust monitoring/polling execution loops with multi-agent
planning, and generating custom text outputs/reports via the expected
output agent and structured notes (`notes_dir`).
Regardless of the model, precise timing for waiting actions cannot be
guaranteed due to latency. If you need to implement a wait, please note that
the total waiting time comprises both the time required for the Artemis large
model to generate output and the actual waiting duration. (For Flash, the
interval between output steps is typically 3-7 seconds, whereas Pro requires
25-35 seconds.)
For long-term repetitive tasks, run the workflow once with this tool to
discover the path, then author a dedicated automation script instead of
repeatedly delegating.
This tool is non-blocking and will immediately return a dictionary
containing:
- `trace_id`: The unique session identifier of the task for status tracking
and inspection.
- `notes_dir`: (Pro Model Only) The directory where the subagent's execution
files/notes are located.
- `stdout_log`: The path to the execution framework log.
- `stderr_log`: The path to the error log containing critical failures and
Python tracebacks (essential for debugging).
Upon task completion (success or failure), the system will trigger a
Reactive Wakeup to notify you.
CRITICAL BEHAVIORAL RULES:
1. **Fallback Timer Constraint**: You **MUST** set a background timer to
check the task status at least once every 1 minute as a safety fallback.
2. **Post-Task Inspection**: After completion, retrieve and inspect the
execution logs (`stderr_log`/`stdout_log`) to verify success.
3. **Inspecting Notes (Pro Model Only)**: If running with Pro, you can
inspect the notes/plans recorded in `notes_dir` to extract critical
execution results passed back by the subagent.
Args:
task_desc: A clear, highly detailed, actionable task description for the
mobile subagent. Since execution is autonomous and one-shot, provide
all necessary context, exact UI actions to take, and clear termination
conditions. If `model="Pro"`, you can explicitly instruct the subagent
on what specific information it must record in the notes.
conversation_id: Your current active Conversation ID. Critical for
routing the wakeup notification.
model: Must be either `"Flash"` or `"Pro"`. - Choose `"Flash"` for fast,
simple UI tasks under 30 steps without log/video analysis or
monitoring/polling needs. - Choose `"Pro"` for complex tasks requiring
planning, ADB shell commands, logs, video exploration, or continuous
monitoring and polling tasks.
locked_app_package: Optional package name of the app to lock execution
to. The agent will auto-launch and restrict actions to this app.
app_path: Optional path to a local APK to install on the device before
running the task.
expected_output_desc: Optional (Pro Model Only). If provided, a
summarization agent will aggregate the interaction history
(motivation, actions, videos) into a summary report saved as
"output.md" in `notes_dir`. Ignored if `model="Flash"`.
"""
# 0. Validate and normalize model
if model.lower() not in ("flash", "pro"):
raise ValueError(f"Invalid model '{model}'. Must be either 'Flash' or 'Pro'.")
canonical_model = "Flash" if model.lower() == "flash" else "Pro"
# 1. Generate a unique trace_id
trace_id = str(uuid.uuid4())
# 2. Initialize the trace store directory and status.json
trace_store.init_trace(
trace_id=trace_id,
task_desc=task_desc,
model=canonical_model,
conversation_id=conversation_id,
)
# 3. Resolve project root, python executable, and background runner module
project_root = env_utils.get_project_root()
python_exe = env_utils.resolve_python_executable(project_root)
# 4. Spawn the background task runner as an independent subprocess
try:
cmd = [
python_exe,
"-m",
"mcp_server.background.task_runner",
"--trace-id",
trace_id,
"--task-desc",
task_desc,
"--model",
canonical_model,
"--conversation-id",
conversation_id,
]
if locked_app_package:
cmd.extend(["--locked-app-package", locked_app_package])
if app_path:
cmd.extend(["--app-path", app_path])
if expected_output_desc and canonical_model != "Flash":
cmd.extend(["--expected-output-desc", expected_output_desc])
env = os.environ.copy()
env["ARTEMIS_SESSION_ID"] = trace_id
proc_kwargs = env_utils.get_detached_process_kwargs()
proc = subprocess.Popen(
cmd,
env=env,
cwd=project_root,
**proc_kwargs,
)
# 5. Record the PID of the spawned subprocess
status_data = trace_store.read_status(trace_id)
if status_data:
status_data["pid"] = proc.pid
trace_store.write_status(trace_id, status_data)
trace_dir = trace_store.get_trace_dir(trace_id)
stdout_log_path = os.path.join(trace_dir, "stdout.log")
stderr_log_path = os.path.join(trace_dir, "stderr.log")
response_dict: dict[str, Any] = {
"trace_id": trace_id,
"message": "Successfully started background task.",
"stdout_log": stdout_log_path,
"stderr_log": stderr_log_path,
}
if canonical_model != "Flash":
response_dict["notes_dir"] = os.path.join(trace_dir, "notes")
return response_dict
except Exception as e:
trace_store.update_trace_status(
trace_id, "failed", error=f"Failed to spawn background runner: {e}"
)
return {
"trace_id": trace_id,
"status": "failed",
"message": f"Error starting background task: {e}",
}
+15
View File
@@ -0,0 +1,15 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""MCP Server Utilities package."""
+172
View File
@@ -0,0 +1,172 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Device discovery, ADB resolution, and emulator management for MCP workflows."""
import os
import shutil
import subprocess
import sys
import time
def resolve_adb_path() -> str:
"""Resolves the absolute path to the adb binary across platforms."""
# 1. Check if adb is in PATH
which_adb = shutil.which("adb")
if which_adb:
return which_adb
# 2. Check ANDROID_HOME / ANDROID_SDK_ROOT
for env_var in ("ANDROID_HOME", "ANDROID_SDK_ROOT"):
sdk_root = os.getenv(env_var)
if sdk_root:
candidate = os.path.join(
sdk_root, "platform-tools", "adb.exe" if sys.platform == "win32" else "adb"
)
if os.path.exists(candidate):
return candidate
# 3. Check standard OS-specific SDK locations
candidates = []
if sys.platform == "win32":
local_appdata = os.getenv("LOCALAPPDATA", "")
if local_appdata:
candidates.append(
os.path.join(local_appdata, "Android", "Sdk", "platform-tools", "adb.exe")
)
elif sys.platform == "darwin":
candidates.extend(
[
os.path.expanduser("~/Library/Android/sdk/platform-tools/adb"),
"/opt/homebrew/bin/adb",
"/usr/local/bin/adb",
]
)
else: # Linux
candidates.extend(
[
"/usr/bin/adb",
os.path.expanduser("~/Android/Sdk/platform-tools/adb"),
"/usr/local/bin/adb",
]
)
for cand in candidates:
if os.path.exists(cand):
return cand
return "adb"
def resolve_emulator_path() -> str | None:
"""Resolves the path to the Android emulator executable."""
which_emulator = shutil.which("emulator")
if which_emulator:
return which_emulator
candidates = [
os.path.expanduser("~/Android/Sdk/emulator/emulator"),
os.path.expanduser("~/Library/Android/sdk/emulator/emulator"),
]
if sys.platform == "win32":
local_appdata = os.getenv("LOCALAPPDATA", "")
if local_appdata:
candidates.append(
os.path.join(local_appdata, "Android", "Sdk", "emulator", "emulator.exe")
)
for cand in candidates:
if os.path.exists(cand):
return cand
return None
def get_connected_devices(adb_path: str | None = None) -> list[str]:
"""Returns a list of connected and authorized Android device serials."""
adb = adb_path or resolve_adb_path()
try:
res = subprocess.run(
[adb, "devices"],
capture_output=True,
text=True,
timeout=5,
)
devices: list[str] = []
for line in res.stdout.splitlines():
line_str = line.strip()
if line_str and not line_str.startswith("List of devices"):
parts = line_str.split()
if len(parts) >= 2 and parts[1] == "device":
devices.append(parts[0])
return devices
except Exception:
return []
def is_emulator_running(adb_path: str | None = None) -> bool:
"""Checks if an Android emulator (e.g. emulator-5554) is running and accessible."""
adb = adb_path or resolve_adb_path()
devices = get_connected_devices(adb)
return any(d.startswith("emulator-") for d in devices)
def ensure_emulator(
avd_name: str = "AndroidWorldAvd",
adb_path: str | None = None,
emulator_path: str | None = None,
timeout_seconds: int = 90,
) -> bool:
"""Ensures an emulator is running. If not, boots the specified AVD in the background."""
adb = adb_path or resolve_adb_path()
if is_emulator_running(adb):
return True
emu_exe = emulator_path or resolve_emulator_path()
if not emu_exe or not os.path.exists(emu_exe):
return False
popen_kwargs = {"stdout": subprocess.DEVNULL, "stderr": subprocess.DEVNULL}
if sys.platform == "win32":
popen_kwargs["creationflags"] = (
subprocess.CREATE_NEW_PROCESS_GROUP | subprocess.DETACHED_PROCESS # type: ignore[attr-defined]
)
else:
popen_kwargs["start_new_session"] = True
try:
subprocess.Popen(
[emu_exe, "-avd", avd_name, "-no-snapshot", "-grpc", "8554"],
**popen_kwargs,
)
except Exception:
return False
start_time = time.time()
while time.time() - start_time < timeout_seconds:
try:
res = subprocess.run(
[adb, "-s", "emulator-5554", "shell", "getprop", "sys.boot_completed"],
capture_output=True,
text=True,
timeout=3,
)
if "1" in res.stdout:
time.sleep(2) # Stabilization
return True
except Exception:
pass
time.sleep(2)
return False
+61
View File
@@ -0,0 +1,61 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Environment and runtime discovery utilities for MCP."""
import os
import subprocess
import sys
from typing import Any
def get_project_root() -> str:
"""Returns the absolute path to the project root repository."""
try:
from artemis.config.paths import ROOT_DIR
return str(ROOT_DIR)
except Exception:
# mcp_server/utils/env_utils.py -> root is 3 levels up
return os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
def resolve_python_executable(project_root: str | None = None) -> str:
"""Resolves the preferred Python executable (.venv or current sys.executable)."""
root = project_root or get_project_root()
if sys.platform == "win32":
venv_python = os.path.join(root, ".venv", "Scripts", "python.exe")
else:
venv_python = os.path.join(root, ".venv", "bin", "python")
if os.path.exists(venv_python):
return venv_python
return sys.executable
def get_detached_process_kwargs() -> dict[str, Any]:
"""Returns kwargs for subprocess.Popen to run a fully detached background process cross-platform."""
kwargs: dict[str, Any] = {
"stdout": subprocess.DEVNULL,
"stderr": subprocess.DEVNULL,
}
if sys.platform == "win32":
kwargs["creationflags"] = (
subprocess.CREATE_NEW_PROCESS_GROUP | subprocess.DETACHED_PROCESS # type: ignore[attr-defined]
)
else:
kwargs["start_new_session"] = True
return kwargs
+108
View File
@@ -0,0 +1,108 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Trace storage and lifecycle management for MCP background automation tasks."""
import json
import os
import time
from typing import Any
try:
from artemis.config.paths import get_traces_dir
TRACES_DIR = str(get_traces_dir())
except Exception:
PROJECT_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
TRACES_DIR = os.path.join(PROJECT_ROOT, "traces")
def get_trace_dir(trace_id: str) -> str:
"""Returns the absolute path to the directory for a given trace_id."""
return os.path.join(TRACES_DIR, trace_id)
def get_status_path(trace_id: str) -> str:
"""Returns the absolute path to the status.json file for a given trace_id."""
return os.path.join(get_trace_dir(trace_id), "status.json")
def init_trace(
trace_id: str,
task_desc: str,
model: str,
conversation_id: str,
) -> dict[str, Any]:
"""Initializes the trace directory and creates the initial status.json file."""
trace_dir = get_trace_dir(trace_id)
os.makedirs(trace_dir, exist_ok=True)
status_data: dict[str, Any] = {
"trace_id": trace_id,
"task_desc": task_desc,
"model": model,
"conversation_id": conversation_id,
"status": "running",
"start_time": time.time(),
"end_time": None,
"error": None,
"result": None,
}
write_status(trace_id, status_data)
return status_data
def read_status(trace_id: str) -> dict[str, Any] | None:
"""Reads and returns the status.json for a given trace_id, or None if it doesn't exist."""
path = get_status_path(trace_id)
if not os.path.exists(path):
return None
try:
with open(path, encoding="utf-8") as f:
return json.load(f)
except Exception:
return None
def write_status(trace_id: str, data: dict[str, Any]) -> None:
"""Writes the given status dictionary to the trace's status.json."""
path = get_status_path(trace_id)
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, indent=2, ensure_ascii=False)
def update_trace_status(
trace_id: str,
status: str,
error: str | None = None,
result: Any | None = None,
) -> dict[str, Any] | None:
"""Updates specific fields of the status.json for a given trace_id."""
data = read_status(trace_id)
if not data:
return None
data["status"] = status
if status in ("completed", "failed", "cancelled"):
data["end_time"] = time.time()
if error is not None:
data["error"] = error
if result is not None:
data["result"] = result
write_status(trace_id, data)
return data