feat: extract in-process MCP action layer, concurrent task dispatch, and spawn/reconcile hardening

- Extract device action execution into artemis/mcp action layer:
  typed ActionResult, capability manifest with startup validation,
  AdbActuator as the single physical implementation, in-memory
  ActionSession (replaces stdio subprocess MCP in validator), and
  McpActionExecutor preserving the legacy executor contract.
  adb_server keeps its external stdio/SSE surface unchanged and now
  reuses the actuator implementation; import-time side effects moved
  into configure_stdio_mode().
- Capability-gated prompt assembly (prompt_assembly.py): tool enums and
  teaching segments render per available backend tools; full-set output
  is byte-identical to the previous static text (locked by tests).
- Admin console task queue: serial worker replaced by a per-device
  concurrent dispatcher (ARTEMIS_CONCURRENCY_MODE=global restores
  serial semantics), per-session cancellation, strict device binding on
  enqueue, unified stdio piping with tee to trace stdout.log, and
  shutdown cleanup of all active runs. Fix _pid_exists exception order
  so PermissionError is treated as alive.
- mcp_server reliability: spawn watchdog kills runners stuck before
  interpreter startup and releases device reservations; task state
  reconcile uses psutil liveness, prefers runner-reported pids, and
  adds a 45s startup grace to stop false failure reports on Windows;
  conversation_id is now optional end-to-end; explicit device_serial is
  validated at submission.
- CLI install now writes ~/.claude/rules/artemis.md for Claude Code and
  migrates/removes the legacy CLAUDE.md injection block; ~/.claude.json
  is never rewritten when it fails to parse.
- Tests: Windows-portable stdio lifecycle and path assertions; new
  suites for action layer, prompt assembly, spawn watchdog, and task
  reconcile (895 passed, 3 skipped).
This commit is contained in:
weilinliu55 committed 2026-08-27 18:25:27 -07:00
1 parent 60312abbb0
commit cbdca1497c
52 files changed
+5735 -1649

No files matched your search

+1 -1
View File
@@ -185,7 +185,7 @@ To ensure your AI coding assistant acts with the rigor of a senior mobile test e
You can mount or copy [`mcp_server/rules.md`](./mcp_server/rules.md) into your AI IDE's rule configuration: You can mount or copy [`mcp_server/rules.md`](./mcp_server/rules.md) into your AI IDE's rule configuration:
* **Antigravity**: Add the contents of `rules.md` to your Workspace Rules, Global Rules settings, or agent instructions. * **Antigravity**: Add the contents of `rules.md` to your Workspace Rules, Global Rules settings, or agent instructions.
* **Claude Code**: Copy or include the contents of `rules.md` in your project's `CLAUDE.md` file. * **Claude Code**: Run `artemis mcp --install claude` to install the rules to `~/.claude/rules/artemis.md` (install to exactly one location — Claude Code loads both `~/.claude/CLAUDE.md` and `~/.claude/rules/*.md`, so duplicating the rules wastes context).
* **Cursor**: Copy the contents into `.cursorrules` or create a rule file at `.cursor/rules/artemis.mdc`. * **Cursor**: Copy the contents into `.cursorrules` or create a rule file at `.cursor/rules/artemis.mdc`.
* **Codex**: Add the contents to `~/.codex/AGENTS.md` (or the active `AGENTS.override.md`). * **Codex**: Add the contents to `~/.codex/AGENTS.md` (or the active `AGENTS.override.md`).
* **Windsurf / OpenClaw**: Add the rules to your workspace rules or global system prompts. * **Windsurf / OpenClaw**: Add the rules to your workspace rules or global system prompts.
+1 -1
View File
@@ -187,7 +187,7 @@ PYTHONPATH = "/path/to/artemis"
你可以将 [`mcp_server/rules.md`](./mcp_server/rules.md) 挂载或复制到你的 AI IDE 规则配置中: 你可以将 [`mcp_server/rules.md`](./mcp_server/rules.md) 挂载或复制到你的 AI IDE 规则配置中:
* **Antigravity**:将 `rules.md` 内容添加至工作区规则(Workspace Rules)或全局规则设置中。 * **Antigravity**:将 `rules.md` 内容添加至工作区规则(Workspace Rules)或全局规则设置中。
* **Claude Code**:将 `rules.md` 内容复制或引入至项目根目录的 `CLAUDE.md` 文件中。 * **Claude Code**:运行 `artemis mcp --install claude` 自动安装规则至 `~/.claude/rules/artemis.md`(只安装到单一位置——Claude Code 会同时加载 `~/.claude/CLAUDE.md` 与 `~/.claude/rules/*.md`,重复安装会浪费上下文)。
* **Cursor**:将内容复制到 `.cursorrules` 文件或在 `.cursor/rules/artemis.mdc` 中创建新规则。 * **Cursor**:将内容复制到 `.cursorrules` 文件或在 `.cursor/rules/artemis.mdc` 中创建新规则。
* **Codex**:将内容添加至 `~/.codex/AGENTS.md`(或当前生效的 `AGENTS.override.md`)。 * **Codex**:将内容添加至 `~/.codex/AGENTS.md`(或当前生效的 `AGENTS.override.md`)。
* **Windsurf / OpenClaw**:将内容添加到工作区规则或全局 System Prompt 中。 * **Windsurf / OpenClaw**:将内容添加到工作区规则或全局 System Prompt 中。
+58
View File
@@ -19,6 +19,29 @@ from typing import Any
from artemis.config import PAUSE_FILE from artemis.config import PAUSE_FILE
def _pid_exists(pid: int) -> bool:
"""Best-effort liveness probe; returns True when liveness cannot be determined.
psutil is preferred because on Windows ``os.kill(pid, 0)`` does not report
dead pids reliably.
"""
try:
import psutil
return psutil.pid_exists(pid)
except Exception:
pass
try:
import os
os.kill(pid, 0)
return True
except PermissionError:
return True
except (ProcessLookupError, OSError):
return False
class ServerState: class ServerState:
"""Encapsulates all runtime states of the debug server.""" """Encapsulates all runtime states of the debug server."""
@@ -40,6 +63,12 @@ class ServerState:
self.cancelled_session_ids: set[str] = set() self.cancelled_session_ids: set[str] = set()
self.startup_progress: dict[str, list[dict[str, Any]]] = {} self.startup_progress: dict[str, list[dict[str, Any]]] = {}
# Concurrent task executions keyed by session_id. Each value holds
# {"process", "device_id", "goal", "profile"}. `current_process` /
# `active_session_id` mirror the most recently launched run for
# backward compatibility with single-task consumers.
self.active_runs: dict[str, dict[str, Any]] = {}
# Unified single source of truth for task queue # Unified single source of truth for task queue
self.queue_items: list[dict[str, Any]] = [] self.queue_items: list[dict[str, Any]] = []
self._wake_event: asyncio.Event | None = None self._wake_event: asyncio.Event | None = None
@@ -101,8 +130,37 @@ class ServerState:
def queue_goals(self, val: list[Any]): def queue_goals(self, val: list[Any]):
pass pass
def prune_finished_runs(self) -> None:
"""Drop finished entries from active_runs.
Besides reaped processes (returncode set), also drops entries whose pid no
longer exists: if a run coroutine dies without reaping its child, the entry
must not permanently block the scheduler.
"""
for sid, run in list(self.active_runs.items()):
proc = run.get("process")
if proc is None or proc.returncode is not None:
self.active_runs.pop(sid, None)
continue
pid = getattr(proc, "pid", None)
if pid and not _pid_exists(pid):
self.active_runs.pop(sid, None)
@property
def busy_device_ids(self) -> set[str]:
"""Device serials currently owned by an in-flight run."""
self.prune_finished_runs()
return {
str(run["device_id"])
for run in self.active_runs.values()
if run.get("device_id")
}
@property @property
def is_running(self) -> bool: def is_running(self) -> bool:
self.prune_finished_runs()
if self.active_runs:
return True
has_proc = False has_proc = False
if self.current_process is not None: if self.current_process is not None:
if self.current_process.returncode is not None: if self.current_process.returncode is not None:
+8 -3
View File
@@ -103,10 +103,15 @@ async def run_task(request: RunRequest):
if device_probe and device_probe.metadata.get("active_device"): if device_probe and device_probe.metadata.get("active_device"):
verified_serial = device_probe.metadata["active_device"].get("serial") verified_serial = device_probe.metadata["active_device"].get("serial")
if verified_serial: # Only auto-selected targets may be re-bound to the probed device. An
# explicitly requested serial is never silently replaced -- if it is
# invalid, enqueue_tasks rejects the submission with a clear error.
if verified_serial and not request.device_serial:
target_serial = verified_serial target_serial = verified_serial
if readiness_engine.get_active_device_serial() != verified_serial: # The probe verified a live device either way: keep the global active
readiness_engine.set_active_device_serial(verified_serial) # serial in sync even when the caller pinned the target explicitly.
if verified_serial and readiness_engine.get_active_device_serial() != verified_serial:
readiness_engine.set_active_device_serial(verified_serial)
return await task_queue_service.enqueue_tasks( return await task_queue_service.enqueue_tasks(
incoming_goals, incoming_goals,
+17
View File
@@ -147,6 +147,23 @@ async def on_shutdown():
pass pass
state.worker_task = None state.worker_task = None
# Cancel in-flight run coroutines and wait for their finalizers (DB status,
# session_ended broadcast, trace sync, recording recovery) to run.
run_tasks = [t for t in list(task_queue_service._run_tasks) if not t.done()]
for run_task in run_tasks:
run_task.cancel()
if run_tasks:
try:
await asyncio.wait_for(
asyncio.gather(*run_tasks, return_exceptions=True), timeout=5.0
)
except (TimeoutError, asyncio.CancelledError):
pass
for run in list(state.active_runs.values()):
run_proc = run.get("process")
if run_proc is not None and run_proc.returncode is None:
await task_queue_service._terminate_worker_process(run_proc)
state.active_runs.clear()
if state.current_process is not None and state.current_process.returncode is None: if state.current_process is not None and state.current_process.returncode is None:
await task_queue_service._terminate_worker_process(state.current_process) await task_queue_service._terminate_worker_process(state.current_process)
for item in state.queue_items: for item in state.queue_items:
+493 -267
View File
@@ -46,6 +46,10 @@ class TaskQueueService:
and startup tasks. and startup tasks.
""" """
# Strong references to in-flight _execute_task_item tasks (asyncio itself only
# keeps weak references to running tasks).
_run_tasks: set[asyncio.Task] = set()
@classmethod @classmethod
def _broadcast_event(cls, event_type: str, data: Any): def _broadcast_event(cls, event_type: str, data: Any):
"""Broadcasts an event safely to all registered subscribers.""" """Broadcasts an event safely to all registered subscribers."""
@@ -128,39 +132,71 @@ class TaskQueueService:
A new process group alone is insufficient on Windows: the worker still A new process group alone is insufficient on Windows: the worker still
shares the parent's console, so a CTRL_C_EVENT generated anywhere in shares the parent's console, so a CTRL_C_EVENT generated anywhere in
that console can reach the UI server. CREATE_NO_WINDOW removes that that console can reach the UI server. CREATE_NO_WINDOW removes that
shared console boundary. Output is captured and forwarded explicitly shared console boundary.
so detached workers remain visible in the server terminal.
Output is captured on every platform so it can be forwarded to the
server terminal and teed into the trace's stdout.log (the daemon itself
is often spawned with its stdio discarded, so inheriting would lose the
worker's logs entirely).
""" """
kwargs: dict[str, Any] = {
"stdout": asyncio.subprocess.PIPE,
"stderr": asyncio.subprocess.STDOUT,
}
if sys.platform == "win32": if sys.platform == "win32":
return { kwargs["creationflags"] = (
"creationflags": ( subprocess.CREATE_NEW_PROCESS_GROUP | subprocess.CREATE_NO_WINDOW
subprocess.CREATE_NEW_PROCESS_GROUP | subprocess.CREATE_NO_WINDOW )
), return kwargs
"stdout": asyncio.subprocess.PIPE,
"stderr": asyncio.subprocess.STDOUT,
}
return {}
@staticmethod @staticmethod
async def _forward_worker_output(stream: asyncio.StreamReader | None) -> None: async def _forward_worker_output(
"""Forward a detached Windows worker's combined output without corrupting UTF-8.""" stream: asyncio.StreamReader | None, log_path: str | None = None
) -> None:
"""Forward a worker's combined output without corrupting UTF-8.
When ``log_path`` is given, the output is also teed into that file so
the trace's advertised stdout.log actually exists for diagnostics.
"""
if stream is None: if stream is None:
return return
decoder = codecs.getincrementaldecoder("utf-8")(errors="replace") log_file = None
while True: if log_path:
chunk = await stream.read(4096) try:
if not chunk: os.makedirs(os.path.dirname(log_path), exist_ok=True)
break log_file = open(log_path, "w", buffering=1, encoding="utf-8", errors="replace")
text = decoder.decode(chunk) except Exception as exc:
if text: print(f"[QueueWorker] Could not open worker log file '{log_path}': {exc}")
sys.stdout.write(text)
sys.stdout.flush()
tail = decoder.decode(b"", final=True) def _emit(text: str) -> None:
if tail: sys.stdout.write(text)
sys.stdout.write(tail)
sys.stdout.flush() sys.stdout.flush()
if log_file is not None:
try:
log_file.write(text)
except Exception:
pass
try:
decoder = codecs.getincrementaldecoder("utf-8")(errors="replace")
while True:
chunk = await stream.read(4096)
if not chunk:
break
text = decoder.decode(chunk)
if text:
_emit(text)
tail = decoder.decode(b"", final=True)
if tail:
_emit(tail)
finally:
if log_file is not None:
try:
log_file.close()
except Exception:
pass
@staticmethod @staticmethod
async def _finish_output_forwarder(output_task: asyncio.Task[None] | None) -> None: async def _finish_output_forwarder(output_task: asyncio.Task[None] | None) -> None:
@@ -231,269 +267,386 @@ class TaskQueueService:
except RuntimeError: except RuntimeError:
pass pass
@classmethod
def _concurrency_mode(cls) -> str:
"""Resolve the scheduler concurrency mode.
Mirrors DeviceExecutionLock's contract (ARTEMIS_CONCURRENCY_MODE /
ARTEMIS_MAX_CONCURRENT_TASKS):
- "global": strict serial -- one task at a time across all devices.
- "per_device": one task per device -- distinct devices run concurrently.
Defaults to per-device concurrency, matching the lock layer's default.
"""
mode = os.environ.get("ARTEMIS_CONCURRENCY_MODE", "").strip().lower()
if mode in ("global", "serial", "1"):
return "global"
if mode in ("per_device", "device", "parallel", "0"):
return "per_device"
try:
if int(os.environ.get("ARTEMIS_MAX_CONCURRENT_TASKS", 0) or 0) == 1:
return "global"
except (TypeError, ValueError):
pass
return "per_device"
@classmethod @classmethod
async def queue_worker(cls): async def queue_worker(cls):
"""Persistent, self-healing background worker loop processing tasks in FIFO order.""" """Persistent dispatcher scheduling pending tasks onto devices.
print("[QueueWorker] Background worker loop initialized and running.")
Scans the queue and launches each eligible task as an independent
coroutine. Admission is governed by _concurrency_mode(): "global" admits
one task at a time overall, "per_device" admits one task per device so
distinct devices execute concurrently. Per-device FIFO ordering and
cross-process mutual exclusion remain enforced by DeviceExecutionLock
inside each worker process.
"""
print(
"[QueueWorker] Dispatcher initialized "
f"(concurrency mode: {cls._concurrency_mode()})."
)
try: try:
DeviceExecutionLock.cleanup_stale_locks() DeviceExecutionLock.cleanup_stale_locks()
except Exception as exc: except Exception as exc:
print(f"[QueueWorker] Initial stale lock cleanup notice: {exc}") print(f"[QueueWorker] Initial stale lock cleanup notice: {exc}")
while True: try:
task_item = None while True:
sess_id = None try:
goal = None cls._dispatch_pending_tasks()
profile = "flash" except Exception as exc:
output_task: asyncio.Task[None] | None = None print(f"[QueueWorker] Dispatch error: {exc}")
try: try:
if state.is_running: await asyncio.wait_for(state.wake_event.wait(), timeout=0.3)
await asyncio.sleep(0.2) state.wake_event.clear()
except TimeoutError:
pass
except asyncio.CancelledError:
print("[QueueWorker] Dispatcher received cancellation signal.")
for run in list(state.active_runs.values()):
run_proc = run.get("process")
if run_proc is not None and run_proc.returncode is None:
await cls._terminate_worker_process(run_proc)
raise
@classmethod
def _dispatch_pending_tasks(cls) -> None:
"""Launch every pending task admissible under the current concurrency mode."""
mode = cls._concurrency_mode()
state.prune_finished_runs()
if mode == "global" and state.is_running:
return
# Dispatched-but-not-yet-spawned runs are only visible as queue items in
# "running" state, so admission must count those too -- active_runs alone
# lags behind by the subprocess startup latency.
in_flight = [
i
for i in state.queue_items
if isinstance(i, dict) and i.get("status") == "running"
]
busy_devices = state.busy_device_ids | {
str(i.get("device_serial")) for i in in_flight if i.get("device_serial")
}
dispatched_any = False
loop = asyncio.get_running_loop()
for item in list(state.queue_items):
if not (isinstance(item, dict) and item.get("status") == "pending"):
continue
sess_id = item.get("session_id")
if sess_id and str(sess_id) in state.cancelled_session_ids:
cls._remove_task(sess_id)
continue
device = item.get("device_serial")
if mode == "per_device":
# A task without a resolved device may bind to any serial, so it
# only launches on an otherwise idle scheduler; the device lock
# then allocates freely without contending against active runs.
if device is None and (state.active_runs or in_flight or dispatched_any):
continue
if device is not None and str(device) in busy_devices:
continue continue
# 1. Fetch the next pending task item["status"] = "running"
task_item = cls._get_next_pending_task() dispatched_any = True
if task_item is None: if device is not None:
await asyncio.sleep(0.3) busy_devices.add(str(device))
continue run_task = loop.create_task(cls._execute_task_item(item))
# Hold a strong reference: asyncio keeps only weak refs to running
# tasks, and a collected run would strand its queue item forever.
cls._run_tasks.add(run_task)
run_task.add_done_callback(cls._run_tasks.discard)
if mode == "global":
break
# 2. Extract and initialize task execution metadata @classmethod
sess_id = task_item.get("session_id") async def _execute_task_item(cls, task_item: dict[str, Any]) -> None:
if sess_id and str(sess_id) in getattr(state, "cancelled_session_ids", set()): """Run one queued task to completion in its own worker subprocess."""
cls._remove_task(sess_id) sess_id = task_item.get("session_id")
continue run_key = str(sess_id) if sess_id else uuid.uuid4().hex
goal = task_item.get("goal")
profile = task_item.get("profile", "flash")
proc: asyncio.subprocess.Process | None = None
output_task: asyncio.Task[None] | None = None
try:
expected_output = task_item.get("expected_output")
enable_outputter = task_item.get("enable_outputter")
locked_app = task_item.get("locked_app_package") or task_item.get("locked_app")
app_path = task_item.get("app_path")
task_item["status"] = "running"
task_item["start_time"] = time.time()
goal = task_item.get("goal") # A fresh launch clears a stale stop request from a previous task,
profile = task_item.get("profile", "flash") # matching the historical serial worker's per-task reset. Per-session
expected_output = task_item.get("expected_output") # stops are tracked in cancelled_session_ids and are unaffected.
enable_outputter = task_item.get("enable_outputter") state.was_stopped_manually = False
locked_app = task_item.get("locked_app_package") or task_item.get("locked_app") state.current_goal = goal
app_path = task_item.get("app_path") state.current_profile = profile
task_item["status"] = "running" state.active_session_id = sess_id
task_item["start_time"] = time.time()
state.current_goal = goal cls._broadcast_startup_progress(
state.current_profile = profile sess_id, "launching", "Starting the execution process"
state.active_session_id = sess_id )
state.was_stopped_manually = False
cls._broadcast_startup_progress( # Broadcast session_started so all connected clients know the task has started
sess_id, "launching", "Starting the execution process" cls._broadcast_event(
"session_started",
{
"session_id": sess_id,
"initial_goal": goal,
"profile": profile,
"device_serial": task_item.get("device_serial"),
},
)
test_name = f"web_{int(time.time())}_{run_key[:8]}"
env = os.environ.copy()
pythonpath_parts = [
str(WORKSPACE_ROOT),
str(WORKSPACE_ROOT / "apps" / "admin_console"),
str(WORKSPACE_ROOT / "apps" / "cloud_service"),
env.get("PYTHONPATH", ""),
]
env["PYTHONPATH"] = os.pathsep.join([p for p in pythonpath_parts if p])
env["PYTHONUTF8"] = "1"
env["PYTHONUNBUFFERED"] = "1"
if state.ipc_port is not None:
env["ARTEMIS_IPC_PORT"] = str(state.ipc_port)
if sess_id:
env["ARTEMIS_SESSION_ID"] = str(sess_id)
env["ARTEMIS_TASK_INGRESS"] = str(task_item.get("ingress", "frontend"))
env["ARTEMIS_TASK_WORKER"] = "1"
queue_ticket = task_item.get("queue_ticket")
if queue_ticket:
env[DeviceExecutionLock.QUEUE_TICKET_ENV] = str(queue_ticket)
cmd = [
sys.executable,
"-m",
"artemis.main",
goal,
"--profile",
profile,
"--test-name",
test_name,
]
if sess_id:
cmd.extend(["--session-id", str(sess_id)])
if expected_output:
cmd.extend(["--output-description", str(expected_output)])
if enable_outputter is not None:
cmd.append("--enable-outputter" if enable_outputter else "--disable-outputter")
if locked_app:
cmd.extend(["--locked-app", str(locked_app)])
if app_path:
cmd.extend(["--app-path", str(app_path)])
device_serial = task_item.get("device_serial")
if device_serial:
cmd.extend(["--device-serial", str(device_serial)])
env["ADB_DEVICE_SERIAL"] = str(device_serial)
print(
f"[QueueWorker] Starting task [{sess_id}]: '{goal}' (profile: {profile}, device: {device_serial or 'auto'}, outputter: {bool(expected_output or enable_outputter)})"
)
proc = await asyncio.create_subprocess_exec(
*cmd,
cwd=str(WORKSPACE_ROOT),
env=env,
**cls._subprocess_creation_kwargs(),
)
state.current_process = proc
task_item["pid"] = proc.pid
state.active_runs[run_key] = {
"process": proc,
"device_id": str(device_serial) if device_serial else None,
"goal": goal,
"profile": profile,
}
if sess_id:
try:
from mcp_server.utils import trace_store
st = trace_store.read_status(str(sess_id))
if st:
st["pid"] = proc.pid
trace_store.write_status(str(sess_id), st)
except Exception:
pass
cls._broadcast_startup_progress(
sess_id, "process_ready", "Execution process started"
)
ingress_type = str(task_item.get("ingress", "frontend"))
DeviceExecutionLock.transfer_reservation(
str(queue_ticket),
proc.pid,
description=f"{ingress_type} task: {goal[:120]}",
device_id=device_serial or "pending",
session_id=str(sess_id) if sess_id else None,
ingress=ingress_type,
)
# Forward the worker's combined output and tee it into the trace's
# stdout.log so the log paths advertised by the MCP API exist.
log_path = None
if sess_id:
try:
from mcp_server.utils import trace_store
log_path = trace_store.get_trace_stdout_log_path(str(sess_id))
except Exception:
log_path = None
if isinstance(proc.stdout, asyncio.StreamReader):
output_task = asyncio.create_task(
cls._forward_worker_output(proc.stdout, log_path)
) )
# Broadcast session_started so all connected clients know the task has started if sess_id and (
str(sess_id) in getattr(state, "cancelled_session_ids", set())
or state.was_stopped_manually
):
print(
f"[QueueWorker] Task [{sess_id}] was cancelled during launch. Terminating."
)
await cls._terminate_worker_process(proc)
# 3. Await subprocess completion
returncode = await cls._wait_for_worker_process(proc)
print(f"[QueueWorker] Task [{sess_id}] exited with returncode {returncode}")
manual_stop = state.was_stopped_manually or bool(
sess_id and str(sess_id) in state.cancelled_session_ids
)
# 4. Perform fallback database status update and notification
if sess_id:
current_status = session_repo.get_session_status(sess_id)
new_status, should_persist = cls._resolve_terminal_status(
current_status,
returncode,
manual_stop,
)
if should_persist:
session_repo.update_session_status(sess_id, new_status, time.time())
print(f"[QueueWorker] Updated session {sess_id} status to '{new_status}'")
else:
print(
f"[QueueWorker] Preserved authoritative session {sess_id} "
f"status '{new_status}'"
)
try:
from mcp_server.utils import trace_store
if trace_store.read_status(str(sess_id)):
canonical_mcp_status = "completed" if new_status in ("completed", "success") else new_status
trace_store.update_trace_status(
str(sess_id),
canonical_mcp_status,
)
except Exception:
pass
rec_info = session_repo.get_video_recording_for_session(sess_id)
rec_status = (rec_info or {}).get("status")
recovered_video_path = None
if rec_status != "ready":
try:
video_rec_map = session_repo.get_video_recordings_map()
video_idx = await asyncio.to_thread(media_service.build_video_index)
recovered_url = await asyncio.to_thread(
media_service.resolve_video_url,
{"session_id": sess_id},
video_rec_map,
video_idx,
)
if recovered_url:
session_repo.mark_recording_ready(sess_id, recovered_url)
cls._broadcast_event(
"recording_ready",
{"session_id": sess_id, "video_url": recovered_url},
)
recovered_video_path = recovered_url
except Exception as rec_err:
print(f"[QueueWorker] Error attempting recording recovery: {rec_err}")
if not recovered_video_path and rec_status != "ready":
recording_error = "Task worker exited before recording finalization completed"
if session_repo.mark_recording_failed_if_pending(sess_id, recording_error):
cls._broadcast_event(
"recording_failed",
{"session_id": sess_id, "error": recording_error},
)
state.active_connections.pop(sess_id, None)
cls._broadcast_event( cls._broadcast_event(
"session_started", "session_ended",
{ {
"session_id": sess_id, "session_id": sess_id,
"initial_goal": goal, "status": new_status,
"profile": profile, "was_stopped_manually": manual_stop,
"device_serial": task_item.get("device_serial"),
}, },
) )
test_name = f"web_{int(time.time())}" conversation_id = task_item.get("conversation_id") if task_item else None
env = os.environ.copy() if conversation_id or task_item.get("ingress") == "mcp":
pythonpath_parts = [
str(WORKSPACE_ROOT),
str(WORKSPACE_ROOT / "apps" / "admin_console"),
str(WORKSPACE_ROOT / "apps" / "cloud_service"),
env.get("PYTHONPATH", ""),
]
env["PYTHONPATH"] = os.pathsep.join([p for p in pythonpath_parts if p])
env["PYTHONUTF8"] = "1"
env["PYTHONUNBUFFERED"] = "1"
if state.ipc_port is not None:
env["ARTEMIS_IPC_PORT"] = str(state.ipc_port)
if sess_id:
env["ARTEMIS_SESSION_ID"] = str(sess_id)
env["ARTEMIS_TASK_INGRESS"] = str(task_item.get("ingress", "frontend"))
env["ARTEMIS_TASK_WORKER"] = "1"
queue_ticket = task_item.get("queue_ticket")
if queue_ticket:
env[DeviceExecutionLock.QUEUE_TICKET_ENV] = str(queue_ticket)
cmd = [
sys.executable,
"-m",
"artemis.main",
goal,
"--profile",
profile,
"--test-name",
test_name,
]
if sess_id:
cmd.extend(["--session-id", str(sess_id)])
if expected_output:
cmd.extend(["--output-description", str(expected_output)])
if enable_outputter is not None:
cmd.append("--enable-outputter" if enable_outputter else "--disable-outputter")
if locked_app:
cmd.extend(["--locked-app", str(locked_app)])
if app_path:
cmd.extend(["--app-path", str(app_path)])
device_serial = task_item.get("device_serial")
if device_serial:
cmd.extend(["--device-serial", str(device_serial)])
env["ADB_DEVICE_SERIAL"] = str(device_serial)
print(
f"[QueueWorker] Starting task [{sess_id}]: '{goal}' (profile: {profile}, device: {device_serial or 'auto'}, outputter: {bool(expected_output or enable_outputter)})"
)
proc = await asyncio.create_subprocess_exec(
*cmd,
cwd=str(WORKSPACE_ROOT),
env=env,
**cls._subprocess_creation_kwargs(),
)
state.current_process = proc
task_item["pid"] = proc.pid
if sess_id:
try: try:
from mcp_server.utils import trace_store from mcp_server.notifiers import notify
st = trace_store.read_status(str(sess_id)) notify(
if st: conversation_id=conversation_id or "",
st["pid"] = proc.pid message=f"Artemis autonomous task '{goal}' finished with status '{new_status}'.\nTrace ID: {sess_id}",
trace_store.write_status(str(sess_id), st) title=f"Task {new_status.capitalize()}: {goal[:40]}",
except Exception: event_type=new_status,
pass payload={
cls._broadcast_startup_progress( "trace_id": sess_id,
sess_id, "process_ready", "Execution process started" "session_id": sess_id,
) "status": new_status,
ingress_type = str(task_item.get("ingress", "frontend")) "goal": goal,
DeviceExecutionLock.transfer_reservation( },
str(queue_ticket),
proc.pid,
description=f"{ingress_type} task: {goal[:120]}",
device_id=device_serial or "pending",
session_id=str(sess_id) if sess_id else None,
ingress=ingress_type,
)
if sys.platform == "win32" and isinstance(proc.stdout, asyncio.StreamReader):
output_task = asyncio.create_task(cls._forward_worker_output(proc.stdout))
if sess_id and (
str(sess_id) in getattr(state, "cancelled_session_ids", set())
or state.was_stopped_manually
):
print(
f"[QueueWorker] Task [{sess_id}] was cancelled during launch. Terminating."
)
await cls._terminate_worker_process(proc)
# 3. Await subprocess completion
returncode = await cls._wait_for_worker_process(proc)
print(f"[QueueWorker] Task [{sess_id}] exited with returncode {returncode}")
# 4. Perform fallback database status update and notification
if sess_id:
current_status = session_repo.get_session_status(sess_id)
new_status, should_persist = cls._resolve_terminal_status(
current_status,
returncode,
state.was_stopped_manually,
)
if should_persist:
session_repo.update_session_status(sess_id, new_status, time.time())
print(f"[QueueWorker] Updated session {sess_id} status to '{new_status}'")
else:
print(
f"[QueueWorker] Preserved authoritative session {sess_id} "
f"status '{new_status}'"
) )
try: except Exception as notif_err:
from mcp_server.utils import trace_store print(f"[QueueWorker] Notification dispatch notice: {notif_err}")
if trace_store.read_status(str(sess_id)): except asyncio.CancelledError:
canonical_mcp_status = "completed" if new_status in ("completed", "success") else new_status print(f"[QueueWorker] Task [{sess_id}] received cancellation signal.")
trace_store.update_trace_status( if proc is not None and proc.returncode is None:
str(sess_id), await cls._terminate_worker_process(proc)
canonical_mcp_status, raise
) except Exception as e:
except Exception: print(f"[QueueWorker] Unexpected error executing task [{sess_id}]: {e}")
pass finally:
rec_info = session_repo.get_video_recording_for_session(sess_id) await cls._finish_output_forwarder(output_task)
rec_status = (rec_info or {}).get("status") # 5. Clean up the finished task and release this run's scheduling slot
recovered_video_path = None if sess_id:
if rec_status != "ready": cls._remove_task(sess_id)
try: state.cancelled_session_ids.discard(str(sess_id))
video_rec_map = session_repo.get_video_recordings_map() state.cancelled_session_ids.discard(run_key)
video_idx = await asyncio.to_thread(media_service.build_video_index) state.active_runs.pop(run_key, None)
recovered_url = await asyncio.to_thread( if proc is not None and state.current_process is proc:
media_service.resolve_video_url,
{"session_id": sess_id},
video_rec_map,
video_idx,
)
if recovered_url:
session_repo.mark_recording_ready(sess_id, recovered_url)
cls._broadcast_event(
"recording_ready",
{"session_id": sess_id, "video_url": recovered_url},
)
recovered_video_path = recovered_url
except Exception as rec_err:
print(f"[QueueWorker] Error attempting recording recovery: {rec_err}")
if not recovered_video_path and rec_status != "ready":
recording_error = "Task worker exited before recording finalization completed"
if session_repo.mark_recording_failed_if_pending(sess_id, recording_error):
cls._broadcast_event(
"recording_failed",
{"session_id": sess_id, "error": recording_error},
)
state.active_connections.pop(sess_id, None)
cls._broadcast_event(
"session_ended",
{
"session_id": sess_id,
"status": new_status,
"was_stopped_manually": state.was_stopped_manually,
},
)
conversation_id = task_item.get("conversation_id") if task_item else None
if conversation_id or task_item.get("ingress") == "mcp":
try:
from mcp_server.notifiers import notify
notify(
conversation_id=conversation_id or "",
message=f"Artemis autonomous task '{goal}' finished with status '{new_status}'.\nTrace ID: {sess_id}",
title=f"Task {new_status.capitalize()}: {goal[:40]}",
event_type=new_status,
payload={
"trace_id": sess_id,
"session_id": sess_id,
"status": new_status,
"goal": goal,
},
)
except Exception as notif_err:
print(f"[QueueWorker] Notification dispatch notice: {notif_err}")
except asyncio.CancelledError:
print("[QueueWorker] Task received cancellation signal.")
if state.current_process and state.current_process.returncode is None:
await cls._terminate_worker_process(state.current_process)
break
except Exception as e:
print(f"[QueueWorker] Unexpected error executing task: {e}")
await asyncio.sleep(0.5)
finally:
await cls._finish_output_forwarder(output_task)
# 5. Clean up completed task from queue and reset execution state
if sess_id:
cls._remove_task(sess_id)
state.current_process = None state.current_process = None
if not state.active_runs:
state.active_session_id = None state.active_session_id = None
state.current_goal = None state.current_goal = None
state.current_profile = None state.current_profile = None
state.was_stopped_manually = False state.was_stopped_manually = False
state.wake_event.set() state.wake_event.set()
@classmethod @classmethod
async def enqueue_tasks( async def enqueue_tasks(
@@ -564,6 +717,38 @@ class TaskQueueService:
"total_queued": len(state.queue_tasks), "total_queued": len(state.queue_tasks),
} }
# Strict device binding: reject an explicitly requested serial that is not
# attached and authorized, instead of silently running on another device.
if device_serial:
try:
from artemis.runtime import device_pool
norm = DeviceExecutionLock._normalize_device_id
attached = {
norm(d.serial): d.state
for d in await device_pool.list_devices_async()
}
except Exception:
attached = None
# Empty enumeration (no adb / transient failure) skips validation so the
# task can proceed and fail downstream with a clear no-device error,
# mirroring mcp_server.tools.task_runner._validate_device_serial.
if attached and attached.get(norm(device_serial)) != "device":
dev_state = attached.get(norm(device_serial))
error_detail = (
f"Device '{device_serial}' is not connected. "
f"Attached devices: {sorted(attached) if attached else 'none'}."
if dev_state is None
else f"Device '{device_serial}' is attached but not ready (state: '{dev_state}')."
)
return {
"status": "rejected",
"error": error_detail,
"tasks": [],
"enqueued_count": 0,
"total_queued": len(state.queue_tasks),
}
for i, goal in enumerate(goals): for i, goal in enumerate(goals):
sess_id = (session_id if (session_id and len(goals) == 1) else str(uuid.uuid4())) sess_id = (session_id if (session_id and len(goals) == 1) else str(uuid.uuid4()))
assigned_serial = device_serial assigned_serial = device_serial
@@ -690,8 +875,22 @@ class TaskQueueService:
}, },
) )
if state.current_process: if state.active_runs or state.current_process:
state.was_stopped_manually = True state.was_stopped_manually = True
for run_key, run in list(state.active_runs.items()):
# Track the stop per run so each finalizer resolves its terminal
# status as cancelled even if the global flag is reset meanwhile.
state.cancelled_session_ids.add(str(run_key))
run_proc = run.get("process")
if run_proc is not None and run_proc.returncode is None:
try:
run_proc.kill()
except Exception:
pass
# active_runs entries are popped by each run's finalizer once the
# process exit is observed; clearing them here would free the device
# slots before the processes are actually gone.
if state.current_process:
try: try:
state.current_process.kill() state.current_process.kill()
except Exception: except Exception:
@@ -749,7 +948,25 @@ class TaskQueueService:
else False else False
) )
owner_pid = owner.pid if owner else None owner_pid = owner.pid if owner else None
local_pid = getattr(state.current_process, "pid", None)
# Resolve the locally-managed run for this target (concurrent scheduling
# keeps one subprocess per run in state.active_runs).
local_run = None
if target_sid:
local_run = state.active_runs.get(target_sid)
elif target_device:
local_run = next(
(
run
for run in state.active_runs.values()
if run.get("device_id") and str(run["device_id"]) == target_device
),
None,
)
elif len(state.active_runs) == 1:
local_run = next(iter(state.active_runs.values()))
local_proc = (local_run or {}).get("process") or state.current_process
local_pid = getattr(local_proc, "pid", None)
is_local_owner = bool(owner_pid and local_pid and owner_pid == local_pid) is_local_owner = bool(owner_pid and local_pid and owner_pid == local_pid)
running_item = next( running_item = next(
@@ -795,19 +1012,24 @@ class TaskQueueService:
# Never fall back to a frontend PID while another process has an # Never fall back to a frontend PID while another process has an
# owner record that is still being published or cannot be parsed. # owner record that is still being published or cannot be parsed.
return False return False
elif owner is None and state.current_process and ( elif owner is None and local_proc is not None and (
not target_sid or str(state.active_session_id) == target_sid local_run is not None
or not target_sid
or str(state.active_session_id) == target_sid
): ):
# The frontend worker can be stopped during its short initialization # The locally-managed worker can be stopped during its short
# window before Agent acquires the global device lease. # initialization window before Agent acquires the device lease.
state.was_stopped_manually = True if target_sid:
state.cancelled_session_ids.add(target_sid)
else:
state.was_stopped_manually = True
if local_pid: if local_pid:
try: try:
stopped = process_supervisor.terminate_tree(local_pid) stopped = process_supervisor.terminate_tree(local_pid)
except Exception: except Exception:
stopped = False stopped = False
try: try:
state.current_process.kill() local_proc.kill()
stopped = True stopped = True
except Exception: except Exception:
pass pass
@@ -849,10 +1071,14 @@ class TaskQueueService:
return False return False
if is_local_owner: if is_local_owner:
state.was_stopped_manually = True
state.current_process = None
if stopped_session_id: if stopped_session_id:
# Per-session cancellation: the run's own finalizer resolves the
# terminal status without affecting other concurrent runs.
state.cancelled_session_ids.add(str(stopped_session_id)) state.cancelled_session_ids.add(str(stopped_session_id))
else:
state.was_stopped_manually = True
if local_proc is not None and state.current_process is local_proc:
state.current_process = None
for sid, conn_info in list(state.active_connections.items()): for sid, conn_info in list(state.active_connections.items()):
if (stopped_session_id and str(sid) == str(stopped_session_id)) or ( if (stopped_session_id and str(sid) == str(stopped_session_id)) or (
+2 -2
View File
@@ -15,7 +15,7 @@ You operate in a dynamic observation-reasoning-execution loop. **CRITICAL RULE:
# 2. DYNAMIC ENVIRONMENT & TIME AWARENESS # 2. DYNAMIC ENVIRONMENT & TIME AWARENESS
Keep in mind that the mobile device is an asynchronous, constantly evolving environment. However, your actions entail an inherent "turn latency" (a 3–7 second delay caused by model inference and network transmission between normal consecutive actions). Keep in mind that the mobile device is an asynchronous, constantly evolving environment. However, your actions entail an inherent "turn latency" (a 3–7 second delay caused by model inference and network transmission between normal consecutive actions).
- **State Drift & Progression Tolerance**: While you think, video playback advances, banners scroll, and web pages finish loading. If the latest observation has progressed further than expected (e.g., an ad automatically skipped or a redirect landed), **do not treat it as an error**. Adapt smoothly to the live screen in front of you. - **State Drift & Progression Tolerance**: While you think, video playback advances, banners scroll, and web pages finish loading. If the latest observation has progressed further than expected (e.g., an ad automatically skipped or a redirect landed), **do not treat it as an error**. Adapt smoothly to the live screen in front of you.
- **Timed Waiting & Transitions**: When facing mid-transition animations, loading spinners, required watch durations, or ad countdown timers, use the precise `wait_for_delay` command to allow time to pass or the UI state to stabilize. Some transition states are subtle and may even resemble a phone bug or system freeze; it is always advisable to wait first, and only consider retrying or choosing an alternative path if the state remains broken. {% if "wait_for_delay" in available_tools %}- **Timed Waiting & Transitions**: When facing mid-transition animations, loading spinners, required watch durations, or ad countdown timers, use the precise `wait_for_delay` command to allow time to pass or the UI state to stabilize. Some transition states are subtle and may even resemble a phone bug or system freeze; it is always advisable to wait first, and only consider retrying or choosing an alternative path if the state remains broken.{% endif %}
- **Time-Sensitive Tasks & Transient UI**: First, you must understand that each of your decision turns takes several seconds of model inference and network delay, while the mobile device state is continuously changing. A target element that you see on the current screenshot (e.g., auto-dismissing overlays, temporary message bars, menus or control bars with short timeouts) may have already vanished by the time your physical click command reaches the device! This is one of the most common reasons why the UI does not change as expected after a single step. We call these **Time-Sensitive Tasks**. - **Time-Sensitive Tasks & Transient UI**: First, you must understand that each of your decision turns takes several seconds of model inference and network delay, while the mobile device state is continuously changing. A target element that you see on the current screenshot (e.g., auto-dismissing overlays, temporary message bars, menus or control bars with short timeouts) may have already vanished by the time your physical click command reaches the device! This is one of the most common reasons why the UI does not change as expected after a single step. We call these **Time-Sensitive Tasks**.
- *Mental Model*: When facing these tasks, any attempt to interact step-by-step (such as tapping once to wake up a menu/bar and waiting for the next turn to tap the target button) WILL fail because the element will auto-dismiss during your decision delay. You MUST combine the summon action on the trigger element (`Trigger`) and the target action (`Target`) into a single atomic sequence. - *Mental Model*: When facing these tasks, any attempt to interact step-by-step (such as tapping once to wake up a menu/bar and waiting for the next turn to tap the target button) WILL fail because the element will auto-dismiss during your decision delay. You MUST combine the summon action on the trigger element (`Trigger`) and the target action (`Target`) into a single atomic sequence.
- *Chained Execution*: Use `click_sequence(sequence=[[trigger_x, trigger_y], [target_x, target_y]])` in `0-1000` normalized scale to execute both taps consecutively with a 50ms delay, catching the transient element before the timeout closes it. - *Chained Execution*: Use `click_sequence(sequence=[[trigger_x, trigger_y], [target_x, target_y]])` in `0-1000` normalized scale to execute both taps consecutively with a 50ms delay, catching the transient element before the timeout closes it.
@@ -32,7 +32,7 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
1. **Normalized Coordinates (`0-1000` Scale)**: Always use normalized `[x, y]` coordinates in `0-1000` scale for any location-based target (e.g., `target=[500, 600]` or `click_sequence=[[300, 400], [500, 600]]`). 1. **Normalized Coordinates (`0-1000` Scale)**: Always use normalized `[x, y]` coordinates in `0-1000` scale for any location-based target (e.g., `target=[500, 600]` or `click_sequence=[[300, 400], [500, 600]]`).
2. **Do not determine element coordinates based on guesswork.** If the element is visible in the screenshot but its coordinates are missing from the provided XML tree, please use the `ask_explorer` tool to locate the element first. 2. **Do not determine element coordinates based on guesswork.** If the element is visible in the screenshot but its coordinates are missing from the provided XML tree, please use the `ask_explorer` tool to locate the element first.
3. **Screenshot Pruning Awareness**: To optimize speed, the system **prunes old intermediate screenshots from your history**—you only ever see the *latest* live screenshot and UI tree. Therefore, if you spot critical text in an earlier step that you will need later (e.g., verification codes, specific titles, search lists), **you must explicitly verbalize and write those details into your text thinking/notes** so they persist across turns. 3. **Screenshot Pruning Awareness**: To optimize speed, the system **prunes old intermediate screenshots from your history**—you only ever see the *latest* live screenshot and UI tree. Therefore, if you spot critical text in an earlier step that you will need later (e.g., verification codes, specific titles, search lists), **you must explicitly verbalize and write those details into your text thinking/notes** so they persist across turns.
4. **App Launching**: To start or reset an application, always use `manage_app(action="launch", app_name="...")` or `action="stop"`. Do not manually swipe across home screens. {% if "manage_app" in available_tools %}4. **App Launching**: To start or reset an application, always use `manage_app(action="launch", app_name="...")` or `action="stop"`. Do not manually swipe across home screens.{% endif %}
--- ---
+24 -8
View File
@@ -36,10 +36,8 @@ from langchain_core.messages import (
from artemis.agents.flash.context_compressor import compress_flash_messages from artemis.agents.flash.context_compressor import compress_flash_messages
from artemis.agents.flash.summarizer import VisualStepSummarizer from artemis.agents.flash.summarizer import VisualStepSummarizer
from artemis.agents.validator.tool_declarations import ( from artemis.agents.validator.tool_declarations import (
ACTION_TOOL_NAMES,
ASK_EXPLORER_TOOL, ASK_EXPLORER_TOOL,
CLICK_SEQUENCE_TOOL, CLICK_SEQUENCE_TOOL,
MobileActionExecutor,
REPORT_TASK_STATUS_TOOL, REPORT_TASK_STATUS_TOOL,
VALIDATOR_TOOLS_DECLARATION, VALIDATOR_TOOLS_DECLARATION,
capture_screenshot_and_parse_ui, capture_screenshot_and_parse_ui,
@@ -51,6 +49,7 @@ from artemis.controllers.unified_controller import UnifiedMobileController
from artemis.data_engine.trace import trace from artemis.data_engine.trace import trace
from artemis.graph.perception import _check_injected_instruction_file from artemis.graph.perception import _check_injected_instruction_file
from artemis.graph.state import State from artemis.graph.state import State
from artemis.mcp.action_executor import McpActionExecutor
from artemis.services.llm import ( from artemis.services.llm import (
RobustChatModelWrapper, RobustChatModelWrapper,
get_google_llm, get_google_llm,
@@ -78,7 +77,7 @@ class FlashRunner:
self.step_summarizer_cfg = StepSummarizerConfig() self.step_summarizer_cfg = StepSummarizerConfig()
self.controller = UnifiedMobileController(ctx) self.controller = UnifiedMobileController(ctx)
self.executor = MobileActionExecutor(ctx, self.controller) self.executor = McpActionExecutor(ctx, self.controller)
self.summarizer = ( self.summarizer = (
VisualStepSummarizer(ctx, model_name=self.step_summarizer_cfg.model) VisualStepSummarizer(ctx, model_name=self.step_summarizer_cfg.model)
if self.step_summarizer_cfg.enabled if self.step_summarizer_cfg.enabled
@@ -90,6 +89,14 @@ class FlashRunner:
tools.insert(1, CLICK_SEQUENCE_TOOL) tools.insert(1, CLICK_SEQUENCE_TOOL)
tools.append(ASK_EXPLORER_TOOL) tools.append(ASK_EXPLORER_TOOL)
tools.append(REPORT_TASK_STATUS_TOOL) tools.append(REPORT_TASK_STATUS_TOOL)
# With an actuator installed, drop declarations for device actions the
# backend does not implement (and append its extension tools); without
# one, the full historical declaration set is kept.
actuator = getattr(self.ctx, "actuator", None)
if actuator is not None:
from artemis.mcp.action_manifest import filter_declarations
tools = filter_declarations(tools, actuator, "flash")
return tools return tools
def _prune_intermediate_screenshots(self, messages: list[BaseMessage]) -> None: def _prune_intermediate_screenshots(self, messages: list[BaseMessage]) -> None:
@@ -144,10 +151,15 @@ class FlashRunner:
} }
) )
# Render System Prompt # Render System Prompt. Tool-teaching segments are gated on the available
# tool set so an absent tool leaves no trace in the prompt; until an actuator
# is wired in, the full manifest set reproduces the historical prompt.
prompt_path = Path(__file__).parent / "flash_runner.md" prompt_path = Path(__file__).parent / "flash_runner.md"
prompt_template = prompt_path.read_text(encoding="utf-8") prompt_template = prompt_path.read_text(encoding="utf-8")
system_prompt = Template(prompt_template).render(goal=self.goal) available_tools = frozenset(t.name for t in tools_declaration)
system_prompt = Template(prompt_template).render(
goal=self.goal, available_tools=available_tools
)
messages: list[BaseMessage] = [ messages: list[BaseMessage] = [
SystemMessage(content=system_prompt), SystemMessage(content=system_prompt),
@@ -432,8 +444,12 @@ class FlashRunner:
try: try:
exec_result = await self.executor.execute(name, args, tc_id, state) exec_result = await self.executor.execute(name, args, tc_id, state)
# Dynamic dispatch set: manifest device actions plus any backend
# extension tools, so extension steps are recorded like actions.
action_names = self.executor.action_tool_names
post_img_bytes = exec_result.screenshot_bytes post_img_bytes = exec_result.screenshot_bytes
if not post_img_bytes and name in ACTION_TOOL_NAMES: if not post_img_bytes and name in action_names:
try: try:
controller = UnifiedMobileController(self.ctx) controller = UnifiedMobileController(self.ctx)
screen_data = await controller.get_screen_data() screen_data = await controller.get_screen_data()
@@ -447,7 +463,7 @@ class FlashRunner:
# Record telemetry / step in DataEngine # Record telemetry / step in DataEngine
recorded_step_id = None recorded_step_id = None
if self.ctx.data_engine and name in ACTION_TOOL_NAMES: if self.ctx.data_engine and name in action_names:
try: try:
if self.ctx.data_engine.current_step_id is None: if self.ctx.data_engine.current_step_id is None:
self.ctx.data_engine.allocate_step_id() self.ctx.data_engine.allocate_step_id()
@@ -517,7 +533,7 @@ class FlashRunner:
logger.warning(f"Error recording step in FlashRunner: {step_err}") logger.warning(f"Error recording step in FlashRunner: {step_err}")
# ⚡ Non-blocking dispatch of objective visual transition summarizer # ⚡ Non-blocking dispatch of objective visual transition summarizer
if self.summarizer and name in ACTION_TOOL_NAMES: if self.summarizer and name in action_names:
action_sequence += 1 action_sequence += 1
self.summarizer.dispatch( self.summarizer.dispatch(
step_number=action_sequence, step_number=action_sequence,
+44 -25
View File
@@ -114,6 +114,30 @@ class OperatorNode:
logger.error(f"Failed to load operator prompts: {e}") logger.error(f"Failed to load operator prompts: {e}")
self.prompts = {} self.prompts = {}
def _available_device_actions(self) -> frozenset[str]:
"""Device actions the installed actuator backend provides.
Without an explicit ``ctx.actuator`` the full manifest set applies (the
default AdbActuator implements everything), so behavior is unchanged until a
partial backend is actually installed.
"""
from artemis.mcp.action_manifest import (
OPTIONAL_ACTIONS,
REQUIRED_ACTIONS,
available_device_actions,
)
actuator = getattr(self.ctx, "actuator", None)
if actuator is not None and callable(getattr(actuator, "capabilities", None)):
try:
extension_names = frozenset(
e.name for e in actuator.extensions() if "operator" in e.targets
)
return available_device_actions(actuator) | extension_names
except Exception as e:
logger.warning(f"Failed to read actuator capabilities: {e}")
return REQUIRED_ACTIONS | OPTIONAL_ACTIONS
def _check_infinite_loop(self, state: State): def _check_infinite_loop(self, state: State):
subagent_calls = state.subagent_calls or [] subagent_calls = state.subagent_calls or []
if len(subagent_calls) >= 3: if len(subagent_calls) >= 3:
@@ -386,15 +410,9 @@ class OperatorNode:
current_messages.append(response) current_messages.append(response)
action_tool_names = [ # Manifest-driven, so a backend's extension actions are classified as
"click", # turn-ending device actions without touching this file.
"input_text", action_tool_names = sorted(self._available_device_actions())
"swipe",
"press_key",
"manage_app",
"wait_for_delay",
"long_press",
]
def normalize_name(name: str) -> str: def normalize_name(name: str) -> str:
return name.split(":")[-1] if ":" in name else name return name.split(":")[-1] if ":" in name else name
@@ -804,23 +822,24 @@ class OperatorNode:
current_step_num = len(steps) + 1 current_step_num = len(steps) + 1
# 5. Prepare Tools # 5. Prepare Tools. Device-action declarations are assembled against the
click_tool = self._get_click_tool() # installed actuator backend's capabilities: an action the backend does not
input_text_tool = self._get_input_text_tool() # implement is simply never declared (and the prompt assembly drops its
swipe_tool = self._get_swipe_tool() # teaching segments in lockstep).
press_key_tool = self._get_press_key_tool() available_actions = self._available_device_actions()
manage_app_tool = self._get_manage_app_tool() action_tool_factories = {
wait_for_delay_tool = self._get_wait_for_delay_tool() "click": self._get_click_tool,
long_press_tool = self._get_long_press_tool() "input_text": self._get_input_text_tool,
"swipe": self._get_swipe_tool,
"press_key": self._get_press_key_tool,
"manage_app": self._get_manage_app_tool,
"wait_for_delay": self._get_wait_for_delay_tool,
"long_press": self._get_long_press_tool,
}
all_tools = [ all_tools = [
click_tool, factory()
input_text_tool, for name, factory in action_tool_factories.items()
swipe_tool, if name in available_actions
press_key_tool,
manage_app_tool,
wait_for_delay_tool,
long_press_tool,
] + self.tools ] + self.tools
if self._has_checker_feedback(): if self._has_checker_feedback():
+8 -2
View File
@@ -39,12 +39,18 @@ class OperatorPromptBuilder:
"""Builds prompt messages and multimodal contents for the Operator Agent.""" """Builds prompt messages and multimodal contents for the Operator Agent."""
@classmethod @classmethod
def build_system_message(cls, template_name: str = "main_template") -> str: def build_system_message(
cls,
template_name: str = "main_template",
available_tools: frozenset[str] | None = None,
) -> str:
from artemis.agents.operator.prompts import apply_operator_prompt_contract from artemis.agents.operator.prompts import apply_operator_prompt_contract
prompts = load_operator_prompts() prompts = load_operator_prompts()
try: try:
return apply_operator_prompt_contract(prompts[template_name]) return apply_operator_prompt_contract(
prompts[template_name], available_tools=available_tools
)
except KeyError as exc: except KeyError as exc:
raise ValueError(f"Unknown Operator template: {template_name}") from exc raise ValueError(f"Unknown Operator template: {template_name}") from exc
+206 -13
View File
@@ -33,21 +33,139 @@ from artemis.utils.task_tree import get_active_subgoal_hashes
logger = get_logger(__name__) logger = get_logger(__name__)
_LEGACY_TOOL_CALLING_RULE = """- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.""" from artemis.agents.prompt_assembly import (
gate_segment,
render_tool_enum,
resolve_available,
)
_TOOL_CALLING_RULE = """- **Tool Calling Contract**: Tools that return information needed for the decision (`ask_explorer`, `ask_diagnoser`, `video_analyzer`, `run_adb_command`, `manage_task`, `analyze_task_output`, `read_note`, `list_notes`, and `save_note`) are result-dependent Pre-Decision Tools. Do not submit them in the same tool-call list as a Turn-Ending Action; inspect their result first. The write-through tools `update_note` and `append_note` may accompany at most one Turn-Ending Action when their content is based entirely on evidence already available before that action. Never write an action's expected outcome as if it had already been observed.""" _LEGACY_TOOL_CALLING_RULE = """- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action."""
_LEGACY_TOOL_PROTOCOL = """1. **Pre-Decision Exploratory Tools**: _LEGACY_TOOL_PROTOCOL = """1. **Pre-Decision Exploratory Tools**:
- *What they are*: Helper/Subagent tools (such as `ask_explorer`, `ask_diagnoser`, `video_analyzer`), ADB command tools (`run_adb_command`, `manage_task`), and memory note tools (`read_note`, `list_notes`, `save_note`, `update_note`, `append_note`). - *What they are*: Helper/Subagent tools (such as `ask_explorer`, `ask_diagnoser`, `video_analyzer`), ADB command tools (`run_adb_command`, `manage_task`), and memory note tools (`read_note`, `list_notes`, `save_note`, `update_note`, `append_note`).
- *How they work*: These tools gather details or update/read memory. When you invoke these tools, the framework will immediately execute them and return the results, allowing you to continue thinking and make your final decision. - *How they work*: These tools gather details or update/read memory. When you invoke these tools, the framework will immediately execute them and return the results, allowing you to continue thinking and make your final decision.
2. **Turn-Ending Actions**:""" 2. **Turn-Ending Actions**:"""
_TOOL_PROTOCOL = """1. **Result-Dependent Pre-Decision Tools**: # --- Assembled tool references -------------------------------------------------------
- *What they are*: Helper/Subagent tools (`ask_explorer`, `ask_diagnoser`, `video_analyzer`), ADB/task tools (`run_adb_command`, `manage_task`, `analyze_task_output`), and memory tools whose result must be inspected (`read_note`, `list_notes`, `save_note`). # The orderings below reproduce the historical prompt wording exactly; with the full
- *How they work*: Invoke these without a Turn-Ending Action, inspect the returned result, and then decide. # tool set the rendered output is byte-identical to the previous literals. See
2. **Write-Through Memory Tools**: # artemis/agents/prompt_assembly.py for the assembly rationale.
- `update_note` and `append_note` may run alongside at most one Turn-Ending Action only when recording facts already observed in the current context. If the note content depends on the action's result, wait for the next observation before writing it.
3. **Turn-Ending Actions**:""" _PRE_DECISION_HELPER_TOOLS = ("ask_explorer", "ask_diagnoser", "video_analyzer")
_PRE_DECISION_ADB_TOOLS = ("run_adb_command", "manage_task", "analyze_task_output")
_PRE_DECISION_MEMORY_TOOLS = ("read_note", "list_notes", "save_note")
_PRE_DECISION_ALL_TOOLS = (
_PRE_DECISION_HELPER_TOOLS + _PRE_DECISION_ADB_TOOLS + _PRE_DECISION_MEMORY_TOOLS
)
# The two device-action enumeration slots in operator.json (identical in both
# templates), in their historical orders.
_PHYSICAL_ACTIONS_ORDER = (
"click",
"input_text",
"swipe",
"press_key",
"manage_app",
"wait_for_delay",
"long_press",
)
_TURN_ENDING_ORDER = (
"click",
"swipe",
"input_text",
"long_press",
"press_key",
"manage_app",
"wait_for_delay",
)
_PHYSICAL_ACTIONS_SLOT = (
"Physical device actions (`click`, `input_text`, `swipe`, `press_key`,"
" `manage_app`, `wait_for_delay`, `long_press`)"
)
_TURN_ENDING_SLOT = (
"Turn-Ending Action (`click`, `swipe`, `input_text`, `long_press`,"
" `press_key`, `manage_app`, or `wait_for_delay`)"
)
# Self-contained instruction segments that teach a single tool. Removed wholesale when
# the tool is unavailable, so nothing in the prompt dangles.
_WAIT_FOR_DELAY_TRANSITIONS_SEGMENT = (
"- **Loading & Transitions**: When entering page transitions (e.g., app launching,"
" web page loading, or progress indicators), use `wait_for_delay` to let the screen"
" settle before executing target actions. Some transition states are difficult to"
" discern, such as a thin progress line at the top of the browser screen during"
" webpage loads. Before questioning whether an error has occurred, you can first"
" adopt a waiting strategy to see how the device state evolves. If the state"
" remains ambiguous after waiting, certain UI components might be malfunctioning or"
" failing to load; in this case, consider retrying the action or pivoting to an"
" alternative path.\n"
)
_WAIT_FOR_DELAY_INTERVAL_PARENTHETICAL = (
" (convert minutes to milliseconds accurately for `wait_for_delay`)"
)
_VIDEO_ANALYZER_TRANSIENT_SEGMENT = (
" - *Missed Transient States*: If the current state jumped ahead and you suspect"
" you missed an important initial or intermediate screen (e.g., a number or flash"
" message that was overwritten by your last action), consider using the"
" `video_analyzer` tool to look back at the recent history.\n"
)
_VIDEO_ANALYZER_SNAPSHOT_SENTENCE = (
" If you have already missed certain critical information and cannot re-view it by"
" going back or other operations, you can use the `video_analyzer` to look back at"
" historical information."
)
# Tail of the "Action Scope & Interaction Precision" bullet: everything past the two
# generic gesture-scope sentences is swipe/drag teaching.
_SWIPE_PRECISION_SEGMENT = (
" For any task requiring continuous range interactions, sliding, or boundary"
" selection, prefer using targeted swipe/drag gestures spanning from the element's"
" start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is"
" strictly confined to the target element. For drag-and-drop, list reordering, or"
" sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag"
" slightly PAST the target position to overcome touch slop and reliably trigger the"
" drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%),"
" always swipe fully to the extreme boundary to guarantee reaching the absolute"
" limits."
)
#: Every tool name the operator prompt slots can reference. ``available_tools=None``
#: resolves to this set, preserving the historical output.
OPERATOR_PROMPT_TOOLSET: frozenset[str] = frozenset(
_PRE_DECISION_ALL_TOOLS + _PHYSICAL_ACTIONS_ORDER
)
def _tool_calling_rule(available: frozenset[str]) -> str:
tools = render_tool_enum(_PRE_DECISION_ALL_TOOLS, available, final_sep="and")
return (
"- **Tool Calling Contract**: Tools that return information needed for the"
f" decision ({tools}) are result-dependent Pre-Decision Tools. Do not submit"
" them in the same tool-call list as a Turn-Ending Action; inspect their result"
" first. The write-through tools `update_note` and `append_note` may accompany"
" at most one Turn-Ending Action when their content is based entirely on"
" evidence already available before that action. Never write an action's"
" expected outcome as if it had already been observed."
)
def _tool_protocol(available: frozenset[str]) -> str:
helper = render_tool_enum(_PRE_DECISION_HELPER_TOOLS, available)
adb = render_tool_enum(_PRE_DECISION_ADB_TOOLS, available)
memory = render_tool_enum(_PRE_DECISION_MEMORY_TOOLS, available)
return (
"1. **Result-Dependent Pre-Decision Tools**:\n"
f" - *What they are*: Helper/Subagent tools ({helper}), ADB/task tools"
f" ({adb}), and memory tools whose result must be inspected ({memory}).\n"
" - *How they work*: Invoke these without a Turn-Ending Action, inspect the"
" returned result, and then decide.\n"
"2. **Write-Through Memory Tools**:\n"
" - `update_note` and `append_note` may run alongside at most one Turn-Ending"
" Action only when recording facts already observed in the current context. If"
" the note content depends on the action's result, wait for the next"
" observation before writing it.\n"
"3. **Turn-Ending Actions**:"
)
_LEGACY_CHECKER_REJECTION_TRIGGER = """ - **Validation/Checker Rejection**: The verification agent (Checker) rejected your subgoal completion (i.e., you are in troubleshooter mode and this is a retry).""" _LEGACY_CHECKER_REJECTION_TRIGGER = """ - **Validation/Checker Rejection**: The verification agent (Checker) rejected your subgoal completion (i.e., you are in troubleshooter mode and this is a retry)."""
@@ -70,15 +188,28 @@ _LARGE_LIST_STRATEGY = """- **Large List & Long Text Exploration Strategy (Zero-
4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for "double-checking", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content.""" 4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for "double-checking", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content."""
def apply_operator_prompt_contract(prompt_template: str) -> str: def apply_operator_prompt_contract(
"""Align prompt-level tool and recovery rules with Operator runtime semantics.""" prompt_template: str,
available_tools: frozenset[str] | None = None,
) -> str:
"""Align prompt-level tool and recovery rules with Operator runtime semantics.
Args:
prompt_template: One of the raw operator.json templates.
available_tools: Tools that actually exist this run. ``None`` renders with the
full historical tool set, producing byte-identical output to the
pre-assembly contract. Unavailable tools disappear from every enumeration
slot, and instruction segments teaching them are removed wholesale.
"""
available = frozenset(resolve_available(available_tools, OPERATOR_PROMPT_TOOLSET))
prompt_template = prompt_template.replace( prompt_template = prompt_template.replace(
_LEGACY_TOOL_CALLING_RULE, _LEGACY_TOOL_CALLING_RULE,
_TOOL_CALLING_RULE, _tool_calling_rule(available),
) )
prompt_template = prompt_template.replace( prompt_template = prompt_template.replace(
_LEGACY_TOOL_PROTOCOL, _LEGACY_TOOL_PROTOCOL,
_TOOL_PROTOCOL, _tool_protocol(available),
) )
prompt_template = prompt_template.replace( prompt_template = prompt_template.replace(
_LEGACY_CHECKER_REJECTION_TRIGGER, _LEGACY_CHECKER_REJECTION_TRIGGER,
@@ -92,6 +223,44 @@ def apply_operator_prompt_contract(prompt_template: str) -> str:
_LEGACY_LARGE_LIST_STRATEGY, _LEGACY_LARGE_LIST_STRATEGY,
_LARGE_LIST_STRATEGY, _LARGE_LIST_STRATEGY,
) )
# Device-action enumeration slots (identical in both templates).
prompt_template = prompt_template.replace(
_PHYSICAL_ACTIONS_SLOT,
"Physical device actions ("
+ render_tool_enum(_PHYSICAL_ACTIONS_ORDER, available)
+ ")",
)
prompt_template = prompt_template.replace(
_TURN_ENDING_SLOT,
"Turn-Ending Action ("
+ render_tool_enum(_TURN_ENDING_ORDER, available, final_sep="or")
+ ")",
)
# Instruction segments that teach a single tool.
prompt_template = prompt_template.replace(
_WAIT_FOR_DELAY_TRANSITIONS_SEGMENT,
gate_segment(_WAIT_FOR_DELAY_TRANSITIONS_SEGMENT, available, "wait_for_delay"),
)
prompt_template = prompt_template.replace(
_WAIT_FOR_DELAY_INTERVAL_PARENTHETICAL,
gate_segment(
_WAIT_FOR_DELAY_INTERVAL_PARENTHETICAL, available, "wait_for_delay"
),
)
prompt_template = prompt_template.replace(
_VIDEO_ANALYZER_TRANSIENT_SEGMENT,
gate_segment(_VIDEO_ANALYZER_TRANSIENT_SEGMENT, available, "video_analyzer"),
)
prompt_template = prompt_template.replace(
_VIDEO_ANALYZER_SNAPSHOT_SENTENCE,
gate_segment(_VIDEO_ANALYZER_SNAPSHOT_SENTENCE, available, "video_analyzer"),
)
prompt_template = prompt_template.replace(
_SWIPE_PRECISION_SEGMENT,
gate_segment(_SWIPE_PRECISION_SEGMENT, available, "swipe"),
)
return prompt_template return prompt_template
@@ -143,7 +312,31 @@ class TemplatePromptComponent(PromptComponent):
"Failed to format prompt, template not found in operator prompts config." "Failed to format prompt, template not found in operator prompts config."
) )
prompt_template = apply_operator_prompt_contract(prompt_template) # video_analyzer is bound conditionally (graph.py gates it on
# video_recording_tools_enabled); the prompt must not advertise it when the
# tool is not actually available this run.
available = set(OPERATOR_PROMPT_TOOLSET)
setup = getattr(ctx, "execution_setup", None)
if not (setup and getattr(setup, "video_recording_tools_enabled", False)):
available.discard("video_analyzer")
# Device actions the installed actuator backend does not implement disappear
# from the prompt in lockstep with their tool declarations.
actuator = getattr(ctx, "actuator", None)
if actuator is not None and callable(getattr(actuator, "capabilities", None)):
try:
from artemis.mcp.action_manifest import (
DEVICE_ACTIONS,
available_device_actions,
)
available -= DEVICE_ACTIONS - available_device_actions(actuator)
except Exception as e:
logger.warning(f"Failed to assemble prompt against actuator: {e}")
prompt_template = apply_operator_prompt_contract(
prompt_template, available_tools=frozenset(available)
)
plan_and_history = kwargs.get("plan_and_history", "No plan or history yet.") plan_and_history = kwargs.get("plan_and_history", "No plan or history yet.")
+79
View File
@@ -0,0 +1,79 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Helpers for assembling tool references into otherwise-static prompts.
Prompts stay hand-written text; only the spots that mention tools are assembled:
* **Enumeration slots** -- lists like ``(`click`, `swipe`, ... , or `wait_for_delay`)``
are re-rendered from the actually-available tool set, so an absent tool never
appears. With the full tool set the rendering reproduces the original wording
byte-for-byte (ordering, backticks, Oxford ``or``/``and``), which is what the
regression tests pin.
* **Instruction segments** -- self-contained blocks that teach one tool (e.g. the
``manage_app`` app-launching rule) are included only while that tool exists.
An unavailable tool therefore leaves no trace in the prompt at all -- the executable
definition of "an unimplemented tool must cost the model nothing".
"""
from collections.abc import Iterable, Sequence, Set
__all__ = ["gate_segment", "render_tool_enum", "resolve_available"]
def render_tool_enum(
tools: Sequence[str],
available: Set[str],
final_sep: str | None = None,
) -> str:
"""Renders an ordered, backticked tool enumeration limited to available tools.
Args:
tools: Tool names in the exact order the prompt historically used.
available: Tools that currently exist.
final_sep: ``"or"``/``"and"`` to join the last item Oxford-style
(``"`a`, `b`, or `c`"``); ``None`` for a plain comma list.
Returns:
The rendered enumeration, or ``""`` when none of the tools are available.
"""
ticked = [f"`{name}`" for name in tools if name in available]
if not ticked:
return ""
if len(ticked) == 1:
return ticked[0]
if final_sep is None:
return ", ".join(ticked)
if len(ticked) == 2:
return f"{ticked[0]} {final_sep} {ticked[1]}"
return ", ".join(ticked[:-1]) + f", {final_sep} " + ticked[-1]
def gate_segment(segment: str, available: Set[str], *tools: str) -> str:
"""Returns ``segment`` when every tool it teaches is available, else ``""``."""
if all(name in available for name in tools):
return segment
return ""
def resolve_available(available: Iterable[str] | None, full_set: Set[str]) -> Set[str]:
"""Normalizes an optional availability argument.
``None`` means "everything" -- callers that have no actuator wired yet keep
today's output unchanged. Passing an explicit set opts into assembly.
"""
if available is None:
return full_set
return frozenset(available)
+50 -11
View File
@@ -36,7 +36,7 @@ from langchain_core.messages import (
from artemis.agents.validator.tool_declarations import ( from artemis.agents.validator.tool_declarations import (
ASK_EXPLORER_TOOL, ASK_EXPLORER_TOOL,
CLICK_SEQUENCE_TOOL, CLICK_SEQUENCE_TOOL,
MobileActionExecutor, MobileActionExecutor, # Legacy shims below; the loop itself uses McpActionExecutor.
ToolDeclaration, ToolDeclaration,
VALIDATOR_TOOLS_DECLARATION, VALIDATOR_TOOLS_DECLARATION,
capture_screenshot_and_parse_ui, capture_screenshot_and_parse_ui,
@@ -46,6 +46,7 @@ from artemis.context import ArtemisContext
from artemis.controllers.unified_controller import UnifiedMobileController from artemis.controllers.unified_controller import UnifiedMobileController
from artemis.data_engine.trace import trace from artemis.data_engine.trace import trace
from artemis.graph.state import State from artemis.graph.state import State
from artemis.mcp.action_executor import McpActionExecutor
from artemis.services.llm import get_llm from artemis.services.llm import get_llm
from artemis.utils.logger import get_logger from artemis.utils.logger import get_logger
from artemis.utils.notes import get_note_file_path from artemis.utils.notes import get_note_file_path
@@ -230,6 +231,16 @@ class FailureAnalysisStrategy:
def get_tools(self) -> list: def get_tools(self) -> list:
return VALIDATOR_TOOLS_DECLARATION return VALIDATOR_TOOLS_DECLARATION
def required_tools(self) -> frozenset[str]:
"""Tools this strategy's prompt depends on structurally.
A strategy whose prompt is built around a tool (rather than merely listing it)
must declare it here; strategy selection falls back to the base strategy when
the current backend does not provide every required tool, so the dependent
prompt is never rendered against a backend that cannot satisfy it.
"""
return frozenset()
def get_user_message_suffix(self, pre_screenshot: bool, post_screenshot: bool) -> str: def get_user_message_suffix(self, pre_screenshot: bool, post_screenshot: bool) -> str:
suffix = ( suffix = (
"Your objective is:\n" "Your objective is:\n"
@@ -268,6 +279,11 @@ class TargetDisappearedStrategy(FailureAnalysisStrategy):
tools.append(ASK_EXPLORER_TOOL) tools.append(ASK_EXPLORER_TOOL)
return tools return tools
def required_tools(self) -> frozenset[str]:
# The whole prompt is built around atomic chained execution to defeat turn
# latency, which *is* click_sequence.
return frozenset({"click_sequence"})
def get_user_message_suffix(self, pre_screenshot: bool, post_screenshot: bool) -> str: def get_user_message_suffix(self, pre_screenshot: bool, post_screenshot: bool) -> str:
return "" return ""
@@ -284,6 +300,9 @@ class PixelTargetDisappearedStrategy(FailureAnalysisStrategy):
tools.append(ASK_EXPLORER_TOOL) tools.append(ASK_EXPLORER_TOOL)
return tools return tools
def required_tools(self) -> frozenset[str]:
return frozenset({"click_sequence"})
def get_user_message_suffix(self, pre_screenshot: bool, post_screenshot: bool) -> str: def get_user_message_suffix(self, pre_screenshot: bool, post_screenshot: bool) -> str:
return "" return ""
@@ -295,15 +314,33 @@ class FailureAnalyzer:
self.ctx = ctx self.ctx = ctx
self.max_iterations = 15 self.max_iterations = 15
def _select_strategy(self, error_category: ValidationErrorCategory) -> FailureAnalysisStrategy: def _select_strategy(
self,
error_category: ValidationErrorCategory,
available_tools: frozenset[str] | None = None,
) -> FailureAnalysisStrategy:
if error_category == ValidationErrorCategory.PIXEL_TARGET_DISAPPEARED: if error_category == ValidationErrorCategory.PIXEL_TARGET_DISAPPEARED:
logger.info("Routing failure analysis to specialized PixelTargetDisappearedStrategy.") logger.info("Routing failure analysis to specialized PixelTargetDisappearedStrategy.")
return PixelTargetDisappearedStrategy() strategy: FailureAnalysisStrategy = PixelTargetDisappearedStrategy()
if error_category == ValidationErrorCategory.TARGET_DISAPPEARED: elif error_category == ValidationErrorCategory.TARGET_DISAPPEARED:
logger.info("Routing failure analysis to specialized TargetDisappearedStrategy.") logger.info("Routing failure analysis to specialized TargetDisappearedStrategy.")
return TargetDisappearedStrategy() strategy = TargetDisappearedStrategy()
logger.info("Routing failure analysis to DefaultAnalysisStrategy.") else:
return FailureAnalysisStrategy() logger.info("Routing failure analysis to DefaultAnalysisStrategy.")
strategy = FailureAnalysisStrategy()
# Capability gating: a strategy and its prompt are one unit. If the backend
# cannot satisfy the tools the prompt is built around, fall back to the base
# strategy so the dependent prompt is never rendered. `None` = no gating.
if available_tools is not None:
missing = strategy.required_tools() - available_tools
if missing:
logger.warning(
f"Strategy {type(strategy).__name__} requires unavailable tool(s)"
f" {sorted(missing)}; falling back to base FailureAnalysisStrategy."
)
return FailureAnalysisStrategy()
return strategy
def _get_active_subgoal_hash(self) -> str: def _get_active_subgoal_hash(self) -> str:
if not self.ctx.data_engine: if not self.ctx.data_engine:
@@ -363,7 +400,12 @@ class FailureAnalyzer:
if not pre_screenshot: if not pre_screenshot:
raise ValueError("Failure analysis requires valid screenshots.") raise ValueError("Failure analysis requires valid screenshots.")
strategy = self._select_strategy(error_category) controller = UnifiedMobileController(self.ctx)
executor = McpActionExecutor(self.ctx, controller)
strategy = self._select_strategy(
error_category, available_tools=frozenset(executor.action_tool_names)
)
prompt_template_name = strategy.get_prompt_template_name() prompt_template_name = strategy.get_prompt_template_name()
tools_declaration = strategy.get_tools() tools_declaration = strategy.get_tools()
@@ -372,9 +414,6 @@ class FailureAnalyzer:
except Exception: except Exception:
llm = get_llm(self.ctx, name="validator") llm = get_llm(self.ctx, name="validator")
controller = UnifiedMobileController(self.ctx)
executor = MobileActionExecutor(self.ctx, controller)
# Pre-Action and Post-Action XML formatting # Pre-Action and Post-Action XML formatting
pre_xml_list_str = "Not available." pre_xml_list_str = "Not available."
if hasattr(state, "latest_ui_hierarchy") and state.latest_ui_hierarchy: if hasattr(state, "latest_ui_hierarchy") and state.latest_ui_hierarchy:
+84 -207
View File
@@ -15,35 +15,31 @@
"""Universal Tool Declarations, Execution Results, and Dispatcher for Artemis.""" """Universal Tool Declarations, Execution Results, and Dispatcher for Artemis."""
import ast import ast
import asyncio
import base64 import base64
import hashlib import hashlib
import inspect
import json import json
from pathlib import Path from pathlib import Path
import time
from typing import Any, Literal from typing import Any, Literal
from google.genai import types as genai_types from google.genai import types as genai_types
from langchain_core.messages import ToolMessage from langchain_core.messages import ToolMessage
from pydantic import BaseModel, Field from pydantic import BaseModel, Field
from artemis.config import get_temp_dir, resolve_explorer_version from artemis.config import resolve_explorer_version
from artemis.context import ArtemisContext from artemis.context import ArtemisContext
from artemis.controllers.unified_controller import UnifiedMobileController from artemis.controllers.unified_controller import UnifiedMobileController
from artemis.core.tool_declaration import ToolDeclaration from artemis.core.tool_declaration import ToolDeclaration
from artemis.data_engine.trace import trace from artemis.data_engine.trace import trace
from artemis.mcp.action_types import ActionCode
from artemis.mcp.actuators.adb import AdbActuator
from artemis.mcp.observation import observe
from artemis.tools.explorer_tool import _run_explorer_logic from artemis.tools.explorer_tool import _run_explorer_logic
from artemis.graph.state import State from artemis.graph.state import State
from artemis.tools.mobile.launch_app import find_package
from artemis.utils.app_launch_utils import launch_app_with_retries
from artemis.utils.coordinates import ( from artemis.utils.coordinates import (
compute_smart_swipe_coordinates, compute_smart_swipe_coordinates,
parse_swipe_parameters, parse_swipe_parameters,
) )
from artemis.utils.logger import get_logger from artemis.utils.logger import get_logger
from artemis.utils.ocr_xml_fusion import fuse_ocr_with_xml
from artemis.utils.visualization import format_minimal_list_with_elements
from artemis.utils.notes import ( from artemis.utils.notes import (
LIST_NOTES_DOCSTRING, LIST_NOTES_DOCSTRING,
READ_NOTE_ARG_KEY_DESC, READ_NOTE_ARG_KEY_DESC,
@@ -186,65 +182,22 @@ async def capture_screenshot_and_parse_ui(
controller: UnifiedMobileController, controller: UnifiedMobileController,
skip_settling: bool = False, skip_settling: bool = False,
) -> tuple[str | None, bytes | None, str | None]: ) -> tuple[str | None, bytes | None, str | None]:
"""Captures screenshot and parses fused XML tree after optional screen settling delay.""" """Captures screenshot and parses fused XML tree after optional screen settling delay.
try:
if skip_settling:
logger.info("Retrieving screen data directly (skip settling)...")
device_data = await controller.get_screen_data()
latest_screenshot_b64 = device_data.base64
else:
logger.info("Delaying 0.4s before capturing screen state...")
await asyncio.sleep(0.4)
device_data = await controller.get_screen_data()
latest_screenshot_b64 = device_data.base64
latest_screenshot_bytes = base64.b64decode(latest_screenshot_b64) Thin adapter over :func:`artemis.mcp.observation.observe` that adds the LangGraph
except Exception as e: ``State`` write-back (indexed points/elements) the agents rely on.
logger.error(f"Failed to capture screen state: {e}") """
obs, screenshot_bytes = await observe(
ctx, controller, settle_ms=0 if skip_settling else 400
)
if not obs.ok:
return None, None, None return None, None, None
try: if obs.hierarchy_ok:
xml_hierarchy = device_data.elements state.indexed_points = [el["center"] for el in obs.elements]
width = device_data.width or 1080 state.indexed_elements = obs.elements
height = device_data.height or 2400
if ctx.device:
ctx.device.device_width = width
ctx.device.device_height = height
ocr_results = [] return obs.screenshot_path, screenshot_bytes, obs.elements_text
fused_xml = fuse_ocr_with_xml(xml_hierarchy, ocr_results)
minimal_list_str, elements, _ = format_minimal_list_with_elements(fused_xml, width, height)
state.indexed_points = [el["center"] for el in elements]
state.indexed_elements = elements
except Exception as e:
logger.error(f"Failed to fetch or fuse UI hierarchy: {e}")
minimal_list_str = "Not available due to hierarchy error."
screenshot_path = None
if ctx.data_engine:
try:
image_name = ctx.data_engine.get_or_create_image(
latest_screenshot_bytes,
ui_tree=xml_hierarchy,
ocr_result=ocr_results,
)
screenshot_path = str(ctx.data_engine.get_image_path(image_name))
except Exception as e:
logger.error(f"Failed to save image in DataEngine: {e}")
if not screenshot_path:
try:
temp_dir = get_temp_dir("screenshots")
temp_file = temp_dir / f"screenshot_{int(time.time())}.jpg"
temp_file.write_bytes(latest_screenshot_bytes)
screenshot_path = str(temp_file)
except Exception as e:
logger.error(f"Failed to save fallback screenshot: {e}")
return screenshot_path, latest_screenshot_bytes, minimal_list_str
def normalize_coordinate_target(target: Any) -> int | list[int] | Any: def normalize_coordinate_target(target: Any) -> int | list[int] | Any:
@@ -313,11 +266,23 @@ def serialize_mobile_action_result(result: Any) -> dict[str, Any]:
class MobileActionExecutor: class MobileActionExecutor:
"""Encapsulates device action execution, observation capture, and result reporting.""" """Adapts LLM tool calls onto an actuator backend and captures observations.
def __init__(self, ctx: ArtemisContext, controller: UnifiedMobileController | None = None): Device-call bodies live in the actuator (``artemis/mcp/actuators/``); this class
keeps the agent-side responsibilities: argument normalization, index resolution
against LangGraph ``State``, act-then-observe screenshot capture, tracing, and the
legacy 4-tuple result contract.
"""
def __init__(
self,
ctx: ArtemisContext,
controller: UnifiedMobileController | None = None,
actuator: "AdbActuator | None" = None,
):
self.ctx = ctx self.ctx = ctx
self.controller = controller or UnifiedMobileController(ctx) self.actuator = actuator or AdbActuator(ctx, controller)
self.controller = self.actuator.controller
@trace(type="action", name="click", serializer=serialize_mobile_action_result) @trace(type="action", name="click", serializer=serialize_mobile_action_result)
async def exec_click( async def exec_click(
@@ -327,27 +292,16 @@ class MobileActionExecutor:
times: int = 1, times: int = 1,
delay_ms: int = 100, delay_ms: int = 100,
) -> tuple[str, bytes | None, str | None, str | None]: ) -> tuple[str, bytes | None, str | None, str | None]:
width = getattr(self.ctx.device, "device_width", 1080)
height = getattr(self.ctx.device, "device_height", 2400)
try: try:
target = normalize_coordinate_target(target) target = normalize_coordinate_target(target)
if not isinstance(target, (list, tuple)) or len(target) != 2: if not isinstance(target, (list, tuple)) or len(target) != 2:
raise ValueError(f"Invalid target format: {target}") raise ValueError(f"Invalid target format: {target}")
nx, ny = target nx, ny = target
x = int(max(0, min(width - 1, int(nx) * width / 1000))) res = await self.actuator.click(int(nx), int(ny), times=times, delay_ms=delay_ms)
y = int(max(0, min(height - 1, int(ny) * height / 1000))) outcome = res.message
result = await self.controller.tap_at(x, y, times=times, delay_ms=delay_ms)
err = getattr(result, "error", None) if result else None
success = err is None
outcome = (
f"Clicked at [{nx}, {ny}] (normalized) successfully."
if success
else f"Error executing click: {err}"
)
img_bytes, shot_path, xml_list = None, None, None img_bytes, shot_path, xml_list = None, None, None
if success: if res.ok:
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui( shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller self.ctx, state, self.controller
) )
@@ -385,15 +339,20 @@ class MobileActionExecutor:
None, None,
) )
outcomes = [] # Resolve each entry to normalized coordinates: integer element indices go
for i, raw_target in enumerate(sequence): # through state.indexed_points (pixel centers -> normalized), coordinate
# pairs are already normalized.
norm_points: list[tuple[int, int]] = []
for raw_target in sequence:
target = normalize_coordinate_target(raw_target) target = normalize_coordinate_target(raw_target)
if isinstance(target, int): if isinstance(target, int):
idx = target idx = target
points = getattr(state, "indexed_points", []) or [] points = getattr(state, "indexed_points", []) or []
if 1 <= idx <= len(points): if 1 <= idx <= len(points):
pt = points[idx - 1] pt = points[idx - 1]
x, y = int(pt[0]), int(pt[1]) px, py = int(pt[0]), int(pt[1])
nx = int(max(0, min(1000, round(px * 1000 / max(1, width)))))
ny = int(max(0, min(1000, round(py * 1000 / max(1, height)))))
else: else:
return ( return (
( (
@@ -405,9 +364,7 @@ class MobileActionExecutor:
None, None,
) )
elif isinstance(target, (list, tuple)) and len(target) == 2: elif isinstance(target, (list, tuple)) and len(target) == 2:
nx, ny = target nx, ny = int(target[0]), int(target[1])
x = int(max(0, min(width - 1, int(nx) * width / 1000)))
y = int(max(0, min(height - 1, int(ny) * height / 1000)))
else: else:
return ( return (
f"Error during click sequence: Invalid target format: {raw_target}", f"Error during click sequence: Invalid target format: {raw_target}",
@@ -415,25 +372,16 @@ class MobileActionExecutor:
None, None,
None, None,
) )
norm_points.append((nx, ny))
result = await self.controller.tap_at(x, y) res = await self.actuator.click_sequence(norm_points, delay_ms=delay_ms)
err = getattr(result, "error", None) if result else None if not res.ok:
if err: return res.message, None, None, None
return (
f"Error executing click at step {i + 1}: {err}",
None,
None,
None,
)
outcomes.append(f"Tapped at [{x}, {y}]")
if i < len(sequence) - 1:
await asyncio.sleep(max(0, delay_ms) / 1000.0)
outcome_str = f"Sequence clicked successfully: {'; '.join(outcomes)}"
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui( shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller self.ctx, state, self.controller
) )
return outcome_str, img_bytes, shot_path, xml_list return res.message, img_bytes, shot_path, xml_list
except Exception as e: except Exception as e:
return f"Error executing click sequence: {e}", None, None, None return f"Error executing click sequence: {e}", None, None, None
@@ -445,8 +393,6 @@ class MobileActionExecutor:
duration_ms: int = 1000, duration_ms: int = 1000,
duration: int | None = None, duration: int | None = None,
) -> tuple[str, bytes | None, str | None, str | None]: ) -> tuple[str, bytes | None, str | None, str | None]:
width = getattr(self.ctx.device, "device_width", 1080)
height = getattr(self.ctx.device, "device_height", 2400)
if duration is not None: if duration is not None:
duration_ms = duration duration_ms = duration
try: try:
@@ -454,22 +400,11 @@ class MobileActionExecutor:
if not isinstance(target, (list, tuple)) or len(target) != 2: if not isinstance(target, (list, tuple)) or len(target) != 2:
raise ValueError(f"Invalid target format: {target}") raise ValueError(f"Invalid target format: {target}")
nx, ny = target nx, ny = target
x = int(max(0, min(width - 1, int(nx) * width / 1000))) res = await self.actuator.long_press(int(nx), int(ny), duration_ms=duration_ms)
y = int(max(0, min(height - 1, int(ny) * height / 1000))) outcome = res.message
result = await self.controller.tap_at(
x, y, long_press=True, long_press_duration=duration_ms
)
err = getattr(result, "error", None) if result else None
success = err is None
outcome = (
f"Long pressed at [{nx}, {ny}] (normalized) for {duration_ms}ms successfully."
if success
else f"Error executing long press: {err}"
)
img_bytes, shot_path, xml_list = None, None, None img_bytes, shot_path, xml_list = None, None, None
if success: if res.ok:
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui( shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller self.ctx, state, self.controller
) )
@@ -487,24 +422,16 @@ class MobileActionExecutor:
state: State, state: State,
clear_exist: bool = True, clear_exist: bool = True,
) -> tuple[str, bytes | None, str | None, str | None]: ) -> tuple[str, bytes | None, str | None, str | None]:
width = getattr(self.ctx.device, "device_width", 1080)
height = getattr(self.ctx.device, "device_height", 2400)
try: try:
norm_target: tuple[int, int] | None = None
if target: if target:
target = normalize_coordinate_target(target) target = normalize_coordinate_target(target)
if not isinstance(target, (list, tuple)) or len(target) != 2: if not isinstance(target, (list, tuple)) or len(target) != 2:
raise ValueError(f"Invalid target format: {target}") raise ValueError(f"Invalid target format: {target}")
nx, ny = target norm_target = (int(target[0]), int(target[1]))
x = int(max(0, min(width - 1, int(nx) * width / 1000)))
y = int(max(0, min(height - 1, int(ny) * height / 1000)))
await self.controller.tap_at(x, y) res = await self.actuator.input_text(text, norm_target, clear_exist=clear_exist)
if clear_exist and hasattr(self.controller, "erase_text"): outcome = res.message
await self.controller.erase_text()
if hasattr(self.controller, "type_text"):
await self.controller.type_text(text)
outcome = f"Executed typing '{text}'."
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui( shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller self.ctx, state, self.controller
) )
@@ -540,6 +467,8 @@ class MobileActionExecutor:
try: try:
if kind == "direction": if kind == "direction":
dir_name = target dir_name = target
# Smart-swipe resolution needs State (indexed elements + ui tree), so
# it stays client-side; the actuator only receives the two points.
x1, y1, x2, y2, smart_dur = compute_smart_swipe_coordinates( x1, y1, x2, y2, smart_dur = compute_smart_swipe_coordinates(
direction=target, direction=target,
target=swipe_input.get("target"), target=swipe_input.get("target"),
@@ -550,19 +479,21 @@ class MobileActionExecutor:
duration=final_duration, duration=final_duration,
) )
final_duration = smart_dur final_duration = smart_dur
err = await self.controller.swipe_coords(x1, y1, x2, y2, final_duration) nx1 = int(max(0, min(1000, round(x1 * 1000 / max(1, width)))))
if err: ny1 = int(max(0, min(1000, round(y1 * 1000 / max(1, height)))))
return f"Error swiping {dir_name}: {err}", None, None, None nx2 = int(max(0, min(1000, round(x2 * 1000 / max(1, width)))))
ny2 = int(max(0, min(1000, round(y2 * 1000 / max(1, height)))))
res = await self.actuator.swipe((nx1, ny1), (nx2, ny2), final_duration)
if not res.ok:
return f"Error swiping {dir_name}: {res.detail}", None, None, None
outcome = f"Swipe completed successfully. Swiped {dir_name}." outcome = f"Swipe completed successfully. Swiped {dir_name}."
elif kind == "coords": elif kind == "coords":
x1, y1, x2, y2 = target x1, y1, x2, y2 = target
px1 = int(max(0, min(width - 1, int(x1) * width / 1000))) res = await self.actuator.swipe(
py1 = int(max(0, min(height - 1, int(y1) * height / 1000))) (int(x1), int(y1)), (int(x2), int(y2)), final_duration
px2 = int(max(0, min(width - 1, int(x2) * width / 1000))) )
py2 = int(max(0, min(height - 1, int(y2) * height / 1000))) if not res.ok:
err = await self.controller.swipe_coords(px1, py1, px2, py2, final_duration) return f"Error dragging: {res.detail}", None, None, None
if err:
return f"Error dragging: {err}", None, None, None
outcome = f"Swipe completed successfully. Swiped from [{x1}, {y1}] to [{x2}, {y2}]." outcome = f"Swipe completed successfully. Swiped from [{x1}, {y1}] to [{x2}, {y2}]."
else: else:
return ( return (
@@ -589,24 +520,11 @@ class MobileActionExecutor:
state: State, state: State,
) -> tuple[str, bytes | None, str | None, str | None]: ) -> tuple[str, bytes | None, str | None, str | None]:
try: try:
valid_keys = {"home", "back", "enter", "delete", "tab", "search", "menu", "app_switch"} res = await self.actuator.press_key(key)
key_str = str(key).lower() if not res.ok:
if key_str not in valid_keys: return res.message, None, None, None
return f"Error executing key press '{key}'.", None, None, None
if key_str == "back" and hasattr(self.controller, "go_back"): outcome = res.message
await self.controller.go_back()
elif key_str == "home" and hasattr(self.controller, "go_home"):
await self.controller.go_home()
elif key_str == "enter" and hasattr(self.controller, "press_enter"):
await self.controller.press_enter()
elif hasattr(self.controller, "press_key"):
res = await self.controller.press_key(key)
err = getattr(res, "error", None) if res else None
if err:
return f"Error executing key press '{key}': {err}", None, None, None
outcome = f"Executed key press '{key}'."
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui( shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller self.ctx, state, self.controller
) )
@@ -624,35 +542,13 @@ class MobileActionExecutor:
state: State, state: State,
) -> tuple[str, bytes | None, str | None, str | None]: ) -> tuple[str, bytes | None, str | None, str | None]:
try: try:
res = find_package(self.ctx, app_name, use_fallback=False) res = await self.actuator.manage_app(action, app_name)
pkg = await res if inspect.iscoroutine(res) else res # Argument-level failures return without an observation, exactly as the
if not pkg: # historical early returns did; a failed launch still captures the screen.
return f"Error finding package for app: {app_name}", None, None, None if res.code in (ActionCode.PACKAGE_NOT_FOUND, ActionCode.INVALID_ARGS):
target_pkg = pkg return res.message, None, None, None
if action.lower() == "launch":
res_launch = launch_app_with_retries(self.ctx, target_pkg)
if inspect.iscoroutine(res_launch):
res_launch = await res_launch
if isinstance(res_launch, tuple):
success, error_msg = res_launch
else:
success, error_msg = bool(res_launch), ""
outcome = (
f"Launched app '{app_name}' ({target_pkg}) successfully."
if success
else f"Failed to launch app '{app_name}': {error_msg}"
)
elif action.lower() == "stop":
if hasattr(self.controller, "terminate_app"):
res_term = self.controller.terminate_app(target_pkg)
if inspect.iscoroutine(res_term):
await res_term
outcome = f"Terminated app '{app_name}' successfully."
else:
return f"Invalid manage_app action: {action}", None, None, None
outcome = res.message
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui( shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller self.ctx, state, self.controller
) )
@@ -669,9 +565,8 @@ class MobileActionExecutor:
state: State, state: State,
) -> tuple[str, bytes | None, str | None, str | None]: ) -> tuple[str, bytes | None, str | None, str | None]:
try: try:
delay_s = max(0.0, float(time_in_ms) / 1000.0) res = await self.actuator.wait_for_delay(time_in_ms)
await asyncio.sleep(delay_s) outcome = res.message
outcome = f"Waited for {time_in_ms}ms successfully."
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui( shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller self.ctx, state, self.controller
) )
@@ -690,27 +585,9 @@ class MobileActionExecutor:
state: State, state: State,
) -> tuple[str, bytes | None, str | None, str | None]: ) -> tuple[str, bytes | None, str | None, str | None]:
try: try:
timeout_s = (timeout_ms or 5000) / 1000.0 res = await self.actuator.wait_for_text(text, wait_state, timeout_ms)
start = time.time() outcome = res.message
found = False # A timeout still captures the screen, as the original loop did.
target_state = (wait_state or "appear").lower()
while time.time() - start < timeout_s:
tree = await self.controller.get_ui_elements()
text_present = text.lower() in (tree or "").lower()
if target_state == "appear" and text_present:
found = True
break
elif target_state == "disappear" and not text_present:
found = True
break
await asyncio.sleep(0.5)
outcome = (
f"Successfully waited for text '{text}' to {target_state}."
if found
else f"Timed out waiting for text '{text}' to {target_state}."
)
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui( shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller self.ctx, state, self.controller
) )
+64 -460
View File
@@ -17,39 +17,31 @@ import base64
import difflib import difflib
import json import json
import math import math
import os
from pathlib import Path from pathlib import Path
import psutil
import re import re
import sys
import time import time
import traceback import traceback
import uuid import uuid
from uuid import UUID from uuid import UUID
from adbutils import AdbClient
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from langchain_core.messages import HumanMessage, SystemMessage from langchain_core.messages import HumanMessage, SystemMessage
from artemis.agents.validator.failure_analyzer import ( from artemis.agents.validator.failure_analyzer import (
FailureAnalyzer, FailureAnalyzer,
ValidationErrorCategory, ValidationErrorCategory,
) )
from artemis.clients.ui_automator_client import UIAutomatorClient
from artemis.constants import ( from artemis.constants import (
VALIDATOR_POLL_INTERVAL, VALIDATOR_POLL_INTERVAL,
VALIDATOR_POLL_TIMEOUT, VALIDATOR_POLL_TIMEOUT,
VALIDATOR_UI_HIERARCHY_TIMEOUT, VALIDATOR_UI_HIERARCHY_TIMEOUT,
) )
from artemis.context import ArtemisContext, DeviceContext, DevicePlatform from artemis.context import ArtemisContext
from artemis.controllers.unified_controller import UnifiedMobileController from artemis.controllers.unified_controller import UnifiedMobileController
from artemis.data_engine.trace import CURRENT_TRACE_ID, trace from artemis.data_engine.trace import CURRENT_TRACE_ID, trace
from artemis.graph.state import State from artemis.graph.state import State
from artemis.platform import platform from artemis.mcp.action_names import to_canonical_call
from artemis.mcp.action_session import ActionSession, get_action_session
from artemis.services.llm import get_llm from artemis.services.llm import get_llm
from artemis.tools.mobile.launch_app import find_package from artemis.utils import image_diff, visualization
from artemis.utils import app_launch_utils, image_diff, visualization
from artemis.utils.decorators import wrap_with_callbacks from artemis.utils.decorators import wrap_with_callbacks
from artemis.utils.logger import get_logger from artemis.utils.logger import get_logger
@@ -85,151 +77,14 @@ class ValidatorNode:
def __init__(self, ctx: ArtemisContext): def __init__(self, ctx: ArtemisContext):
self.ctx = ctx self.ctx = ctx
def _kill_mcp_server_instantly(self): async def _get_mcp_session(self) -> ActionSession:
"""Instantly terminate MCP ADB server child processes.""" """Returns the in-process unified action session (created lazily on ctx).
try: The previous stdio-subprocess spawn -- and its local-controller fallback,
current_process = psutil.Process() which on Windows was the *only* path that ever ran -- is gone: the server now
children = current_process.children(recursive=True) lives in-process, so there is no spawn to fail and exactly one execution path.
for child in children:
try:
cmdline = child.cmdline()
if any("adb_server.py" in part for part in cmdline):
logger.warning(f"Instantly killing MCP Server child process: {child.pid}")
child.kill()
except psutil.NoSuchProcess:
pass
except Exception as e:
logger.error(f"Failed to instantly kill MCP server child processes: {e}")
async def _close_failed_mcp_session(
self,
*,
session,
session_entered: bool,
client_ctx,
client_entered: bool,
) -> bool:
"""Exit partially-entered MCP contexts in reverse order.
The MCP stdio transport owns an AnyIO task group and cancellation
scope. Dropping an entered transport without calling ``__aexit__``
leaves that scope attached to the validator task and can cancel an
unrelated action later in the same node.
Returns ``True`` when every entered context closed cleanly.
""" """
self.ctx.mcp_session = None return await get_action_session(self.ctx)
self.ctx.mcp_client_ctx = None
cleanup_succeeded = True
for manager, entered, label in (
(session, session_entered, "MCP session"),
(client_ctx, client_entered, "MCP stdio transport"),
):
if not entered or manager is None:
continue
try:
await manager.__aexit__(None, None, None)
except asyncio.CancelledError:
cleanup_succeeded = False
logger.warning(f"{label} cleanup was cancelled.")
except Exception as cleanup_error:
cleanup_succeeded = False
logger.warning(f"Failed to close {label}: {cleanup_error}")
return cleanup_succeeded
async def _get_mcp_session(self):
if getattr(self.ctx, "mcp_session", None) is not None:
return self.ctx.mcp_session
root_dir = Path(__file__).parent.parent.parent
venv_python = root_dir / ".venv" / "bin" / "python3"
python_exe = str(venv_python) if venv_python.exists() else sys.executable
server_env = os.environ.copy()
# The parent Artemis process already owns the device-awake policy.
# Starting it again in this short-lived child can block the MCP
# handshake on cross-process ADB/lease cleanup.
server_env["ARTEMIS_KEEP_DEVICE_AWAKE"] = "false"
server_params = StdioServerParameters(
command=python_exe,
args=[str(root_dir / "mcp" / "adb_server.py")],
env=server_env,
)
logger.info("Starting persistent MCP server session...")
client_ctx = stdio_client(server_params)
session = None
client_entered = False
session_entered = False
self.ctx.mcp_client_ctx = client_ctx
try:
read, write = await client_ctx.__aenter__()
client_entered = True
session = ClientSession(read, write)
self.ctx.mcp_session = session
await session.__aenter__()
session_entered = True
# Handshake timeout protection
await asyncio.wait_for(session.initialize(), timeout=15.0)
return session
except asyncio.CancelledError:
await self._close_failed_mcp_session(
session=session,
session_entered=session_entered,
client_ctx=client_ctx,
client_entered=client_entered,
)
raise
except Exception as e:
err_stack = traceback.format_exc()
logger.critical(
"CRITICAL: Failed to initialize MCP server session:"
f" {e}\n{err_stack}. Cleaning up child processes."
)
cleanup_succeeded = await self._close_failed_mcp_session(
session=session,
session_entered=session_entered,
client_ctx=client_ctx,
client_entered=client_entered,
)
if not cleanup_succeeded:
self._kill_mcp_server_instantly()
logger.warning(
"Falling back to robust local in-process execution bypassing MCP protocol."
)
try:
adb = AdbClient(host="localhost", port=5037)
devices = adb.device_list()
if devices:
device_id = devices[0].serial
ui_client = UIAutomatorClient(device_id=device_id)
try:
ui_data = ui_client.get_screen_data()
w, h = ui_data.width, ui_data.height
except Exception:
w, h = 1080, 2400
local_ctx = ArtemisContext(
trace_id="local-fallback",
device=DeviceContext(
host_platform=platform.os_type.name,
mobile_platform=DevicePlatform.ANDROID,
device_id=device_id,
device_width=w,
device_height=h,
),
adb_client=adb,
ui_adb_client=ui_client,
)
self._local_controller = UnifiedMobileController(local_ctx)
except Exception as le:
logger.error(f"Failed to setup local fallback controller: {le}")
return None
def _parse_decisions(self, structured_decisions: str) -> tuple[list[dict] | None, str | None]: def _parse_decisions(self, structured_decisions: str) -> tuple[list[dict] | None, str | None]:
if not structured_decisions: if not structured_decisions:
@@ -270,20 +125,11 @@ class ValidatorNode:
error_msg = "" error_msg = ""
while time.time() - start_time < timeout: while time.time() - start_time < timeout:
if session is not None: try:
result = await session.call_tool("take_screenshot", {}) content_img = await session.screenshot_b64()
success_img, content_img = self._parse_mcp_result(result) success_img = True
elif getattr(self, "_local_controller", None): except Exception as e:
try: success_img, content_img = False, str(e)
content_img = await self._local_controller.take_screenshot()
success_img = True
except Exception as e:
success_img, content_img = False, str(e)
else:
success_img, content_img = (
False,
"No session or local controller",
)
if success_img: if success_img:
post_screenshot_b64 = content_img post_screenshot_b64 = content_img
img_before_bytes = base64.b64decode(pre_screenshot_b64) img_before_bytes = base64.b64decode(pre_screenshot_b64)
@@ -465,15 +311,7 @@ class ValidatorNode:
# Capture the live mismatch screenshot to provide context for FailureAnalyzer # Capture the live mismatch screenshot to provide context for FailureAnalyzer
try: try:
if session is not None: post_screenshot_b64 = await session.screenshot_b64()
result = await session.call_tool("take_screenshot", {})
success_img, content = self._parse_mcp_result(result)
if success_img and content:
post_screenshot_b64 = content
else:
logger.warning(f"Take screenshot via MCP returned non-success: {content}")
elif getattr(self, "_local_controller", None):
post_screenshot_b64 = await self._local_controller.take_screenshot()
except Exception as e: except Exception as e:
logger.error(f"Failed to capture live screenshot for failure analysis: {e}") logger.error(f"Failed to capture live screenshot for failure analysis: {e}")
post_screenshot_b64 = None post_screenshot_b64 = None
@@ -504,17 +342,9 @@ class ValidatorNode:
if not success and not post_screenshot_b64: if not success and not post_screenshot_b64:
logger.info("Action failed, capturing failure screenshot...") logger.info("Action failed, capturing failure screenshot...")
try: try:
if session is not None: post_screenshot_b64 = await session.screenshot_b64()
result = await session.call_tool("take_screenshot", {})
success_img, content = self._parse_mcp_result(result)
if success_img and content:
post_screenshot_b64 = content
else:
logger.warning(f"Take failure screenshot via MCP returned non-success: {content}")
elif getattr(self, "_local_controller", None):
post_screenshot_b64 = await self._local_controller.take_screenshot()
except Exception as e: except Exception as e:
logger.error(f"Failed to capture failure screenshot: {e}") logger.error(f"Failed to capture failure screenshot: {e!r}")
post_screenshot_b64 = None post_screenshot_b64 = None
# Enrich and record execution # Enrich and record execution
@@ -661,10 +491,9 @@ class ValidatorNode:
try: try:
return await self._execute_validation_loop(state) return await self._execute_validation_loop(state)
except asyncio.CancelledError: except asyncio.CancelledError:
logger.warning( # In-process session teardown happens in ArtemisContext.__aexit__; there
"Validator task cancelled. Instantly killing MCP server processes." # is no subprocess left to kill.
) logger.warning("Validator task cancelled (Stop requested).")
self._kill_mcp_server_instantly()
raise raise
except Exception as e: except Exception as e:
err_stack = traceback.format_exc() err_stack = traceback.format_exc()
@@ -681,226 +510,42 @@ class ValidatorNode:
) )
except Exception: except Exception:
pass pass
self._kill_mcp_server_instantly()
raise e raise e
finally: finally:
if self.ctx.data_engine: if self.ctx.data_engine:
self.ctx.data_engine.current_step_id = original_step_id self.ctx.data_engine.current_step_id = original_step_id
async def _exec_action( async def _exec_action(self, session: ActionSession, action_item: dict) -> tuple[bool, str]:
self, session: ClientSession | None, action_item: dict """Executes one Operator action item through the unified action session.
) -> tuple[bool, str]:
The Operator's internal verbs (tap, long_press_on, focus_and_input_text, ...)
are translated to canonical tool calls in ``to_canonical_call`` using the
``normalized_coordinates`` the Operator already computed. Success comes from
the structured ``ActionResult.ok`` -- no string parsing.
"""
action_name = action_item.get("action") action_name = action_item.get("action")
coordinates = action_item.get("coordinates")
# Intercept non-wait_for_text actions for local fallback if action_name == "wait_for_delay":
if session is None and getattr(self, "_local_controller", None) is not None: time_in_ms = action_item.get("time_in_ms", 0)
ctrl = self._local_controller logger.info(f"Validator performing wait_for_delay for {time_in_ms}ms...")
logger.info(f"Executing in-process action fallback: {action_name}") await asyncio.sleep(time_in_ms / 1000.0)
try: return True, ""
if action_name == "tap":
if coordinates and len(coordinates) == 2:
times = action_item.get("times", 1)
delay_ms = action_item.get("delay_ms", 100)
res = await ctrl.tap_at(
coordinates[0],
coordinates[1],
times=times,
delay_ms=delay_ms,
)
return (True, "Success") if not res.error else (False, res.error)
return False, "Invalid coords"
elif action_name == "long_press_on":
d = action_item.get("duration", 1000)
if coordinates and len(coordinates) == 2:
res = await ctrl.tap_at(
coordinates[0],
coordinates[1],
long_press=True,
long_press_duration=d,
)
return (True, "Success") if not res.error else (False, res.error)
return False, "Invalid coords"
elif action_name == "swipe":
d = action_item.get("duration", 400)
if coordinates and len(coordinates) == 4:
err = await ctrl.swipe_coords(
coordinates[0],
coordinates[1],
coordinates[2],
coordinates[3],
duration=d,
)
return (True, "Success") if not err else (False, str(err))
return False, "Invalid coords"
elif action_name == "focus_and_input_text":
t = action_item.get("text")
clear_before = action_item.get("clear_before_input", False)
if coordinates and len(coordinates) == 2:
await ctrl.tap_at(coordinates[0], coordinates[1])
await asyncio.sleep(0.5) width = getattr(self.ctx.device, "device_width", 1080) if self.ctx.device else 1080
if clear_before: height = getattr(self.ctx.device, "device_height", 2400) if self.ctx.device else 2400
await ctrl.erase_text()
else:
await ctrl.press_key("123") # Move cursor to end for clean append
success = await ctrl.type_text(t, clear_existing=False)
return (True, "Success") if success else (False, "Failed typing")
return False, "Invalid coords"
elif action_name == "focus_and_clear_text":
if coordinates and len(coordinates) == 2:
await ctrl.tap_at(coordinates[0], coordinates[1])
await asyncio.sleep(0.5)
success = await ctrl.erase_text()
return (True, "Success") if success else (False, "Failed erase")
return False, "Invalid coords"
elif action_name == "erase_one_char":
success = await ctrl.erase_text(nb_chars=1)
return (True, "Success") if success else (False, "Failed erase")
elif action_name == "press_key":
keycode = action_item.get("keycode")
success = await ctrl.press_key(keycode)
return (True, "Success") if success else (False, f"Failed press {keycode}")
elif action_name == "back":
success = await ctrl.go_back()
return (True, "Success") if success else (False, "Failed back")
elif action_name == "launch_app":
app_name = action_item.get("app_name")
package_name = await find_package(self.ctx, app_name, use_fallback=False)
if package_name:
success, err_msg = await app_launch_utils.launch_app_with_retries(
ctrl.ctx, package_name
)
return (True, "Success") if success else (False, err_msg)
return False, "Package not found"
elif action_name == "stop_app":
app_name = action_item.get("app_name")
package_name = await find_package(self.ctx, app_name, use_fallback=False)
if package_name:
success = await ctrl.terminate_app(package_name)
return (True, "Success") if success else (False, "Failed terminate")
return False, "Package not found"
elif action_name == "open_link":
url = action_item.get("url")
success = await ctrl.open_url(url)
return (True, "Success") if success else (False, "Failed open url")
elif action_name == "wait_for_delay":
time_in_ms = action_item.get("time_in_ms", 0)
await asyncio.sleep(time_in_ms / 1000.0)
return True, ""
else:
return (
False,
f"Unsupported action for local fallback: {action_name}",
)
except Exception as e:
return False, f"In-process fallback failed: {e}"
# Normal MCP Execution Path
try: try:
if action_name == "tap": tool_name, wire_args = to_canonical_call(action_item, dims=(width, height))
coordinates = action_item.get("coordinates") except ValueError as e:
times = action_item.get("times") or action_item.get("click_times") or 1 return False, str(e)
delay_ms = action_item.get("delay_ms") or action_item.get("delay") or 100
result = await session.call_tool(
"tap",
{
"coordinates": coordinates,
"times": times,
"delay_ms": delay_ms,
},
)
return self._parse_mcp_result(result)
elif action_name == "long_press_on": try:
coordinates = action_item.get("coordinates") res = await session.call(tool_name, wire_args)
duration = action_item.get("duration", 1000)
result = await session.call_tool(
"long_press_on",
{"coordinates": coordinates, "duration": duration},
)
return self._parse_mcp_result(result)
elif action_name == "swipe":
coordinates = action_item.get("coordinates")
duration = action_item.get("duration", 400)
result = await session.call_tool(
"swipe", {"coordinates": coordinates, "duration": duration}
)
return self._parse_mcp_result(result)
elif action_name == "focus_and_input_text":
coordinates = action_item.get("coordinates")
text = action_item.get("text", "")
clear_before_input = action_item.get("clear_before_input", False)
result = await session.call_tool(
"focus_and_input_text",
{
"coordinates": coordinates,
"text": text,
"clear_before_input": clear_before_input,
},
)
return self._parse_mcp_result(result)
elif action_name == "focus_and_clear_text":
coordinates = action_item.get("coordinates")
result = await session.call_tool(
"focus_and_clear_text", {"coordinates": coordinates}
)
return self._parse_mcp_result(result)
elif action_name == "erase_one_char":
result = await session.call_tool("erase_one_char", {})
return self._parse_mcp_result(result)
elif action_name == "press_key":
keycode = action_item.get("keycode", "")
result = await session.call_tool("press_key", {"keycode": keycode})
return self._parse_mcp_result(result)
elif action_name == "back":
result = await session.call_tool("back", {})
return self._parse_mcp_result(result)
elif action_name == "launch_app":
app_name = action_item.get("app_name")
package_name = await find_package(self.ctx, app_name, use_fallback=False)
if not package_name:
return False, f"Failed to find package for app: {app_name}"
success, err_msg = await app_launch_utils.launch_app_with_retries(
self.ctx, package_name
)
return success, "Success" if success else err_msg
elif action_name == "stop_app":
app_name = action_item.get("app_name")
package_name = await find_package(self.ctx, app_name, use_fallback=False)
if not package_name:
return False, f"Failed to find package for app: {app_name}"
result = await session.call_tool("stop_app", {"package_name": package_name})
return self._parse_mcp_result(result)
elif action_name == "open_link":
url = action_item.get("url")
result = await session.call_tool("open_link", {"url": url})
return self._parse_mcp_result(result)
elif action_name == "wait_for_delay":
time_in_ms = action_item.get("time_in_ms", 0)
logger.info(f"Validator performing wait_for_delay for {time_in_ms}ms...")
await asyncio.sleep(time_in_ms / 1000.0)
return True, ""
else:
return False, f"Unsupported action: {action_name}"
except Exception as e: except Exception as e:
return False, f"MCP call failed: {e}" return False, f"MCP call failed: {e}"
return (True, "") if res.ok else (False, res.message)
@trace(type="tool", name="safety_net_validation") @trace(type="tool", name="safety_net_validation")
async def _validate_action_precondition( async def _validate_action_precondition(
self, session, action_item: dict, state: State | None = None self, session, action_item: dict, state: State | None = None
@@ -1001,45 +646,21 @@ class ValidatorNode:
try: try:
elements = None elements = None
if session is not None: try:
try: # The timeout must ride inside the session (MCP read timeout), never an
result = await asyncio.wait_for( # outer asyncio.wait_for: cancelling call_tool mid-flight corrupts the
session.call_tool("get_ui_hierarchy", {}), # in-memory transport and bricks the session for all later callers.
timeout=VALIDATOR_UI_HIERARCHY_TIMEOUT, elements = await session.ui_hierarchy(timeout=VALIDATOR_UI_HIERARCHY_TIMEOUT)
) except Exception as e:
if hasattr(result, "content") and result.content: logger.warning(
text_content = ( f"Failed to get live XML via MCP: {e!r}. Falling back to"
result.content[0].text " Pixel-based validation."
if hasattr(result.content[0], "text") )
else str(result.content[0]) return (
) False,
elements = json.loads(text_content) ValidationErrorCategory.XML_BYPASSED,
except Exception as e: f"XML hierarchy fetch timed out or errored: {e!r}",
logger.warning( )
f"Failed to get live XML via MCP: {e}. Falling back to"
" Pixel-based validation."
)
return (
False,
ValidationErrorCategory.XML_BYPASSED,
f"XML hierarchy fetch timed out or errored: {e}",
)
elif getattr(self, "_local_controller", None):
try:
elements = await asyncio.wait_for(
self._local_controller.get_ui_elements(),
timeout=VALIDATOR_UI_HIERARCHY_TIMEOUT,
)
except Exception as e:
logger.warning(
f"Failed to get live XML via local controller: {e}."
" Falling back to Pixel-based validation."
)
return (
False,
ValidationErrorCategory.XML_BYPASSED,
f"XML hierarchy fetch timed out or errored: {e}",
)
if not elements or not isinstance(elements, list): if not elements or not isinstance(elements, list):
logger.warning( logger.warning(
@@ -1275,6 +896,9 @@ class ValidatorNode:
f" {new_coords}." f" {new_coords}."
) )
action_item["coordinates"] = new_coords action_item["coordinates"] = new_coords
# Drop the now-stale normalized coordinates so the canonical
# translation re-derives them from the healed pixel center.
action_item.pop("normalized_coordinates", None)
else: else:
logger.info( logger.info(
"Pre-execution validation SUCCESS" "Pre-execution validation SUCCESS"
@@ -1435,23 +1059,6 @@ class ValidatorNode:
return passed, category, reason return passed, category, reason
def _parse_mcp_result(self, result) -> tuple[bool, str]:
"""Helper to parse MCP tool result."""
if hasattr(result, "content") and result.content:
text_content = (
result.content[0].text
if hasattr(result.content[0], "text")
else str(result.content[0])
)
if text_content.startswith("Error:") or text_content.startswith("Failed:"):
return False, text_content
elif text_content == "Success":
return True, ""
elif text_content == "Failed":
return False, "Action failed on server"
return True, text_content
return True, ""
@trace(type="tool", name="safety_net_pixel_validation") @trace(type="tool", name="safety_net_pixel_validation")
async def _validate_action_precondition_pixel( async def _validate_action_precondition_pixel(
self, self,
@@ -1535,13 +1142,10 @@ class ValidatorNode:
try: try:
# A. Take fresh live screenshot (captures state changes / dynamic loading settling) # A. Take fresh live screenshot (captures state changes / dynamic loading settling)
live_screenshot_b64 = None live_screenshot_b64 = None
if session is not None: try:
result = await session.call_tool("take_screenshot", {}) live_screenshot_b64 = await session.screenshot_b64()
success_img, live_screenshot_b64 = self._parse_mcp_result(result)
elif getattr(self, "_local_controller", None):
live_screenshot_b64 = await self._local_controller.take_screenshot()
success_img = True success_img = True
else: except Exception:
success_img, live_screenshot_b64 = False, None success_img, live_screenshot_b64 = False, None
if not success_img or not live_screenshot_b64: if not success_img or not live_screenshot_b64:
+15
View File
@@ -176,11 +176,26 @@ class ArtemisContext(BaseModel):
_mobile_controller: Any | None = PrivateAttr(default=None) _mobile_controller: Any | None = PrivateAttr(default=None)
mcp_client_ctx: Any | None = None mcp_client_ctx: Any | None = None
mcp_session: Any | None = None mcp_session: Any | None = None
action_session: Any | None = None
"""In-process unified action MCP session (artemis.mcp.action_session)."""
actuator: Any | None = None
"""Optional actuator backend override (artemis.mcp.actuators). ``None`` selects
the default AdbActuator; installing e.g. a robot-arm backend happens here."""
async def __aenter__(self) -> ArtemisContext: async def __aenter__(self) -> ArtemisContext:
return self return self
async def __aexit__(self, exc_type, exc_val, exc_tb): async def __aexit__(self, exc_type, exc_val, exc_tb):
# Close the in-process action session before draining background tasks so its
# owner task exits cleanly rather than being cancelled below.
if self.action_session is not None:
try:
await self.action_session.aclose()
except Exception:
pass
finally:
self.action_session = None
if self.background_tasks: if self.background_tasks:
tasks = list(self.background_tasks) tasks = list(self.background_tasks)
pending_tasks = [task for task in tasks if not task.done()] pending_tasks = [task for task in tasks if not task.done()]
+79 -14
View File
@@ -157,11 +157,15 @@ def _get_config_snippet(client: str, python_exe: str, project_root: str) -> dict
return {"mcpServers": {"artemis": config_body}} return {"mcpServers": {"artemis": config_body}}
def _parse_json_lenient(text: str) -> dict: def _parse_json_lenient(text: str) -> dict | None:
"""Parses JSON or JSONC (JSON with comments / trailing commas) safely.""" """Parses JSON or JSONC (JSON with comments / trailing commas) safely.
Returns ``None`` when the text cannot be parsed as a JSON object, so callers
can tell a genuinely empty ``{}`` file apart from a corrupt one.
"""
try: try:
data = json.loads(text) data = json.loads(text)
return data if isinstance(data, dict) else {} return data if isinstance(data, dict) else None
except Exception: except Exception:
pass pass
# Remove // single-line comments # Remove // single-line comments
@@ -172,9 +176,9 @@ def _parse_json_lenient(text: str) -> dict:
cleaned = re.sub(r",\s*([}\]])", r"\1", cleaned) cleaned = re.sub(r",\s*([}\]])", r"\1", cleaned)
try: try:
data = json.loads(cleaned) data = json.loads(cleaned)
return data if isinstance(data, dict) else {} return data if isinstance(data, dict) else None
except Exception: except Exception:
return {} return None
def _merge_json_file( def _merge_json_file(
@@ -183,15 +187,29 @@ def _merge_json_file(
server_config: dict, server_config: dict,
key_name: str | tuple[str, ...] = "mcpServers", key_name: str | tuple[str, ...] = "mcpServers",
remove_paths: tuple[tuple[str, ...], ...] = (), remove_paths: tuple[tuple[str, ...], ...] = (),
strict_parse: bool = False,
) -> bool: ) -> bool:
"""Merges server configuration into a target JSON file without overwriting existing servers.""" """Merges server configuration into a target JSON file without overwriting existing servers.
With ``strict_parse=True``, an existing non-empty file that cannot be parsed
aborts the merge instead of being replaced. Use this for files that hold
state beyond MCP config (e.g. ~/.claude.json), where a degraded rewrite
would destroy unrelated data.
"""
try: try:
file_path.parent.mkdir(parents=True, exist_ok=True) file_path.parent.mkdir(parents=True, exist_ok=True)
data: dict = {} data: dict = {}
if file_path.exists() and file_path.stat().st_size > 0: if file_path.exists() and file_path.stat().st_size > 0:
raw_text = file_path.read_text(encoding="utf-8") raw_text = file_path.read_text(encoding="utf-8")
data = _parse_json_lenient(raw_text) parsed = _parse_json_lenient(raw_text)
if not data and raw_text.strip(): data = parsed or {}
if parsed is None and raw_text.strip():
if strict_parse:
logger.warning(
f"Existing JSON in {file_path} could not be parsed; refusing to rewrite it. "
"Fix or remove the file, then re-run the install."
)
return False
# Backup unparseable existing file to prevent accidental loss # Backup unparseable existing file to prevent accidental loss
backup_path = file_path.with_suffix(file_path.suffix + ".bak") backup_path = file_path.with_suffix(file_path.suffix + ".bak")
try: try:
@@ -305,6 +323,36 @@ def _merge_codex_toml(file_path: Path, server_config: dict) -> bool:
return False return False
def _remove_rules_block(file_path: Path) -> bool:
"""Removes a previously injected managed ARTEMIS block from a rules file.
Used to migrate clients that load both their main rules file and a
standalone rule file (e.g. Claude Code loads ~/.claude/CLAUDE.md AND
~/.claude/rules/*.md), where keeping the injected block would duplicate
the rules in the model's context. If the file contains nothing but the
managed block, the file is deleted entirely.
"""
try:
if not file_path.exists():
return True
begin_marker = "<!-- BEGIN ARTEMIS MOBILE TESTING RULES -->"
end_marker = "<!-- END ARTEMIS MOBILE TESTING RULES -->"
existing = file_path.read_text(encoding="utf-8")
if begin_marker not in existing or end_marker not in existing:
return True
prefix = existing.split(begin_marker)[0]
suffix = existing.split(end_marker)[1]
remainder = f"{prefix.rstrip()}\n\n{suffix.lstrip()}".strip()
if remainder:
file_path.write_text(remainder + "\n", encoding="utf-8")
else:
file_path.unlink()
return True
except Exception as e:
logger.warning(f"Could not remove rules block from {file_path}: {e}")
return False
def _inject_rules_block(file_path: Path, content: str) -> bool: def _inject_rules_block(file_path: Path, content: str) -> bool:
"""Injects or updates a managed ARTEMIS block in a global rules file without overwriting user content.""" """Injects or updates a managed ARTEMIS block in a global rules file without overwriting user content."""
try: try:
@@ -419,13 +467,22 @@ def install_rules(client: str, project_root: str) -> list[str]:
installed_paths.append(str(cursor_rules_file)) installed_paths.append(str(cursor_rules_file))
if _write_cursor_mdc(global_mdc, raw_rules): if _write_cursor_mdc(global_mdc, raw_rules):
installed_paths.append(str(global_mdc)) installed_paths.append(str(global_mdc))
elif target in ("claude", "claude_code", "claude_desktop"): elif target in ("claude", "claude_code"):
# Claude Code loads BOTH ~/.claude/CLAUDE.md and ~/.claude/rules/*.md
# into context every session, so install the rules in exactly one
# place (the standalone rule file) and migrate away any block a
# previous version injected into CLAUDE.md.
claude_md = Path.home() / ".claude" / "CLAUDE.md" claude_md = Path.home() / ".claude" / "CLAUDE.md"
global_rule = Path.home() / ".claude" / "rules" / "artemis.md" global_rule = Path.home() / ".claude" / "rules" / "artemis.md"
if _inject_rules_block(claude_md, raw_rules): _remove_rules_block(claude_md)
installed_paths.append(str(claude_md))
if _write_rule_file(global_rule, raw_rules): if _write_rule_file(global_rule, raw_rules):
installed_paths.append(str(global_rule)) installed_paths.append(str(global_rule))
elif target == "claude_desktop":
# Claude Desktop only reads ~/.claude/CLAUDE.md (not rules/*.md), so
# the rules block stays injected there for this target.
claude_md = Path.home() / ".claude" / "CLAUDE.md"
if _inject_rules_block(claude_md, raw_rules):
installed_paths.append(str(claude_md))
elif target == "windsurf": elif target == "windsurf":
global_rule = Path.home() / ".codeium" / "windsurf" / "rules" / "artemis.md" global_rule = Path.home() / ".codeium" / "windsurf" / "rules" / "artemis.md"
global_mem = Path.home() / ".codeium" / "windsurf" / "memories" / "global_rules.md" global_mem = Path.home() / ".codeium" / "windsurf" / "memories" / "global_rules.md"
@@ -516,9 +573,11 @@ def install_mcp_config(client: str, python_exe: str, project_root: str) -> list[
if _merge_json_file(claude_path, "artemis", server_cfg): if _merge_json_file(claude_path, "artemis", server_cfg):
installed_paths.append(str(claude_path)) installed_paths.append(str(claude_path))
# Also install to Claude Code CLI global config (~/.claude.json) # Also install to Claude Code CLI global config (~/.claude.json).
# This file holds Claude Code state well beyond MCP config, so a
# parse failure must abort the merge rather than rewrite the file.
claude_code_path = Path.home() / ".claude.json" claude_code_path = Path.home() / ".claude.json"
if _merge_json_file(claude_code_path, "artemis", server_cfg): if _merge_json_file(claude_code_path, "artemis", server_cfg, strict_parse=True):
installed_paths.append(str(claude_code_path)) installed_paths.append(str(claude_code_path))
elif target == "cursor": elif target == "cursor":
server_cfg = snippet["mcpServers"]["artemis"] server_cfg = snippet["mcpServers"]["artemis"]
@@ -737,10 +796,16 @@ def mcp_command(
else: else:
agent_mcp.run(transport="stdio") agent_mcp.run(transport="stdio")
elif st == "adb": elif st == "adb":
logger.info(f"Starting Artemis ADB MCP Server over {transport}...")
if transport.lower() == "sse": if transport.lower() == "sse":
logger.info(f"Starting Artemis ADB MCP Server over {transport}...")
adb_mcp.run(transport="sse", host=host, port=port) adb_mcp.run(transport="sse", host=host, port=port)
else: else:
# stdio carries the MCP JSON-RPC stream: redirect logging and detach
# from the parent DataEngine before any output can corrupt it.
from artemis.mcp.adb_server import configure_stdio_mode
configure_stdio_mode()
logger.info("Starting Artemis ADB MCP Server over stdio...")
adb_mcp.run(transport="stdio") adb_mcp.run(transport="stdio")
elif st == "xml": elif st == "xml":
from artemis.mcp.xml_search_server import mcp as xml_mcp from artemis.mcp.xml_search_server import mcp as xml_mcp
+546
View File
@@ -0,0 +1,546 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""MCP-backed action executor for FlashRunner and FailureAnalyzer.
Same public contract as the legacy ``MobileActionExecutor`` --
``execute(name, args, tool_call_id, state) -> ToolExecutionResult`` -- but execution
routes through the in-process action MCP session, and dispatch is table-driven so a
backend's extension tools are callable without touching this file.
Kept agent-side (by design, not omission): argument normalization, element-index
resolution against LangGraph ``State``, smart-swipe resolution, act-then-observe
capture and ``State`` write-back, tracing (a ``ContextVar`` cannot cross the server's
task boundary -- see the plan's R2), and the non-device tools (``read_note``,
``list_notes``, ``ask_explorer``).
Status is carried by ``ActionResult.ok`` -- the historical substring sniffing on
``"Error"``/``"Failed"`` is gone, so typing the literal text "Failed" into a field no
longer marks the step as failed, and a successful tap with a failed screenshot is no
longer misreported as an error.
"""
import ast
import hashlib
import json
from pathlib import Path
from typing import Any
from artemis.config import resolve_explorer_version
from artemis.context import ArtemisContext
from artemis.controllers.unified_controller import UnifiedMobileController
from artemis.data_engine.trace import TraceSpan
from artemis.mcp.action_manifest import OPTIONAL_ACTIONS, REQUIRED_ACTIONS
from artemis.mcp.action_session import ActionSession, get_action_session
from artemis.mcp.action_types import ActionCode, ActionResult
from artemis.mcp.actuators.adb import AdbActuator
from artemis.agents.validator.tool_declarations import (
ToolExecutionResult,
normalize_coordinate_target,
)
from artemis.utils.coordinates import (
compute_smart_swipe_coordinates,
parse_swipe_parameters,
)
from artemis.utils.logger import get_logger
from artemis.utils.notes import (
format_list_notes_failure,
format_list_notes_success,
format_read_note_failure,
list_notes_info,
read_note_content,
)
logger = get_logger(__name__)
__all__ = ["McpActionExecutor"]
# Wording for wrapped exceptions; matches the historical executor `except` arms.
_EXCEPTION_PREFIX = {
"click": "Error during click",
"click_sequence": "Error executing click sequence",
"long_press": "Error during long press",
"input_text": "Error during input text",
"swipe": "Error during swipe",
"press_key": "Error during press_key",
"manage_app": "Error during manage_app",
"wait_for_delay": "Error during wait_for_delay",
"wait_for_text": "Error during wait_for_text",
}
class _ArgError(ValueError):
"""Argument-translation failure whose message is already fully formatted."""
class McpActionExecutor:
"""Routes agent tool calls through the unified action MCP session."""
def __init__(
self,
ctx: ArtemisContext,
controller: UnifiedMobileController | None = None,
actuator: AdbActuator | None = None,
):
self.ctx = ctx
self.actuator = (
actuator or getattr(ctx, "actuator", None) or AdbActuator(ctx, controller)
)
self.controller = self.actuator.controller
self._session: ActionSession | None = None
@property
def action_tool_names(self) -> frozenset[str]:
"""Device actions plus backend extension names -- the dynamic dispatch set."""
return (
(REQUIRED_ACTIONS | OPTIONAL_ACTIONS)
| {e.name for e in self.actuator.extensions()}
)
async def _session_or_start(self) -> ActionSession:
if self._session is None or not self._session.started:
self._session = await get_action_session(self.ctx, actuator=self.actuator)
return self._session
# --- Public entry ----------------------------------------------------------------
async def execute(
self,
name: str,
args: dict[str, Any],
tool_call_id: str,
state: Any,
) -> ToolExecutionResult:
"""Executes the named tool and wraps the outcome for the calling agent."""
raw_name = name.split(":")[-1] if ":" in name else name
if raw_name in ("read_note", "list_notes", "ask_explorer"):
return await self._execute_agent_tool(raw_name, name, args, tool_call_id, state)
if raw_name in self.action_tool_names:
return await self._execute_device_action(
raw_name, name, args, tool_call_id, state
)
return ToolExecutionResult(
tool_call_id=tool_call_id,
tool_name=name,
status="error",
text_summary=f"Error: Tool '{name}' not supported.",
)
# --- Device actions --------------------------------------------------------------
async def _execute_device_action(
self,
raw_name: str,
name: str,
args: dict[str, Any],
tool_call_id: str,
state: Any,
) -> ToolExecutionResult:
span = TraceSpan(name=raw_name, trace_type="action", ctx=self.ctx)
span.payload = {"args": args}
message = ""
res: ActionResult | None = None
img_bytes: bytes | None = None
shot_path: str | None = None
xml_list: str | None = None
with span:
try:
session = await self._session_or_start()
wire_name, wire_args, finalize = self._translate(raw_name, args, state)
extension = raw_name not in (REQUIRED_ACTIONS | OPTIONAL_ACTIONS)
if extension:
raw_result = await session.call_raw(wire_name, wire_args)
message = next(
(b.text for b in raw_result.content if getattr(b, "type", "") == "text"),
f"Extension tool '{raw_name}' completed.",
)
res = ActionResult.success(raw_name, message)
else:
res = await session.call(wire_name, wire_args)
message = finalize(res) if finalize else res.message
if res.ok or self._observe_despite_failure(raw_name, res):
obs = await session.observe(settle_ms=400)
if obs.ok:
shot_path = obs.screenshot_path
xml_list = obs.elements_text
if obs.hierarchy_ok:
state.indexed_points = [el["center"] for el in obs.elements]
state.indexed_elements = obs.elements
if shot_path:
try:
img_bytes = Path(shot_path).read_bytes()
except Exception as read_err:
logger.warning(
f"Failed to read observed screenshot: {read_err}"
)
except _ArgError as e:
res = ActionResult.failure(raw_name, str(e), code=ActionCode.INVALID_ARGS)
message = str(e)
except Exception as e:
prefix = _EXCEPTION_PREFIX.get(raw_name, f"Error during {raw_name}")
message = f"{prefix}: {e}"
res = ActionResult.failure(raw_name, message, detail=repr(e))
if shot_path:
state.latest_screenshot = shot_path
status = "success" if res.ok else "error"
post_image_name = None
if shot_path:
post_image_name = Path(shot_path).stem
elif img_bytes:
post_image_name = hashlib.sha256(img_bytes).hexdigest()
span.result = {
"outcome": message,
"post_image_name": post_image_name,
"has_xml": bool(xml_list),
"status": status,
}
if status == "error":
span.status = "failed"
span.error = message
return ToolExecutionResult(
tool_call_id=tool_call_id,
tool_name=name,
status=status,
text_summary=message,
screenshot_bytes=img_bytes,
screenshot_path=shot_path,
ui_elements_text=xml_list,
metadata={
"code": res.code.value,
"normalized_coordinates": res.normalized_coordinates,
},
)
@staticmethod
def _observe_despite_failure(raw_name: str, res: ActionResult) -> bool:
"""Failures that still observe the screen, mirroring the legacy executor.
A failed app launch and a wait_for_text timeout captured the screen; argument
and package errors returned without one.
"""
if raw_name == "manage_app":
return res.code not in (ActionCode.PACKAGE_NOT_FOUND, ActionCode.INVALID_ARGS)
if raw_name == "wait_for_text":
return res.code is ActionCode.TIMEOUT
return False
# --- Argument translation --------------------------------------------------------
def _translate(
self, raw_name: str, args: dict[str, Any], state: Any
) -> tuple[str, dict[str, Any], Any]:
"""Maps agent-facing args to wire args; returns (name, args, finalize).
``finalize(res)`` post-processes the ActionResult message where the historical
wording depended on client-side context (direction swipes).
"""
if raw_name == "click":
target = self._require_pair(
args.get("target") or args.get("coordinates") or [], "click"
)
return (
"click",
{
"target": list(target),
"times": args.get("times", 1),
"delay_ms": args.get("delay_ms", 100),
},
None,
)
if raw_name == "long_press":
target = self._require_pair(
args.get("target") or args.get("coordinates") or [], "long press"
)
return (
"long_press",
{
"target": list(target),
"duration_ms": args.get("duration_ms", args.get("duration", 1000)),
},
None,
)
if raw_name == "input_text":
raw_target = args.get("target") or args.get("coordinates")
target = None
if raw_target:
target = list(self._require_pair(raw_target, "input text"))
return (
"input_text",
{
"text": args.get("text", ""),
"target": target,
"clear_exist": args.get("clear_exist", True),
},
None,
)
if raw_name == "click_sequence":
return (
"click_sequence",
{
"sequence": self._resolve_sequence(args.get("sequence") or [], state),
"delay_ms": args.get("delay_ms", 50),
},
None,
)
if raw_name == "swipe":
return self._translate_swipe(args, state)
if raw_name == "press_key":
return "press_key", {"key": args.get("key", "BACK")}, None
if raw_name == "manage_app":
return (
"manage_app",
{
"action": args.get("action", "launch"),
"app_name": args.get("app_name", ""),
},
None,
)
if raw_name == "wait_for_delay":
return "wait_for_delay", {"time_in_ms": args.get("time_in_ms", 1000)}, None
if raw_name == "wait_for_text":
return (
"wait_for_text",
{
"text": args.get("text", ""),
"wait_state": args.get("wait_state") or "appear",
"timeout_ms": args.get("timeout_ms", 5000),
},
None,
)
# Backend extension: pass the arguments straight through.
return raw_name, dict(args), None
@staticmethod
def _require_pair(raw: Any, label: str) -> tuple[int, int]:
target = normalize_coordinate_target(raw)
if not isinstance(target, (list, tuple)) or len(target) != 2:
prefix = {
"click": "Error during click",
"long press": "Error during long press",
"input text": "Error during input text",
}[label]
raise _ArgError(f"{prefix}: Invalid target format: {raw}")
return int(target[0]), int(target[1])
def _resolve_sequence(self, sequence: Any, state: Any) -> list[list[int]]:
"""Resolves click_sequence entries (indices or pairs) to normalized pairs."""
if isinstance(sequence, str):
sequence_str = sequence.strip()
try:
sequence = json.loads(sequence_str)
except Exception:
try:
sequence = ast.literal_eval(sequence_str)
except Exception:
pass
if not isinstance(sequence, (list, tuple)):
raise _ArgError(
f"Error during click sequence: Invalid sequence format: {sequence}"
)
width = getattr(self.ctx.device, "device_width", 1080) if self.ctx.device else 1080
height = getattr(self.ctx.device, "device_height", 2400) if self.ctx.device else 2400
resolved: list[list[int]] = []
for raw_target in sequence:
target = normalize_coordinate_target(raw_target)
if isinstance(target, int):
idx = target
points = getattr(state, "indexed_points", []) or []
if not 1 <= idx <= len(points):
raise _ArgError(
"Error during click sequence: Index"
f" {idx} is out of range (available: {len(points)})"
)
px, py = int(points[idx - 1][0]), int(points[idx - 1][1])
nx = int(max(0, min(1000, round(px * 1000 / max(1, width)))))
ny = int(max(0, min(1000, round(py * 1000 / max(1, height)))))
elif isinstance(target, (list, tuple)) and len(target) == 2:
nx, ny = int(target[0]), int(target[1])
else:
raise _ArgError(
f"Error during click sequence: Invalid target format: {raw_target}"
)
resolved.append([nx, ny])
return resolved
def _translate_swipe(
self, args: dict[str, Any], state: Any
) -> tuple[str, dict[str, Any], Any]:
width = getattr(self.ctx.device, "device_width", 1080) if self.ctx.device else 1080
height = getattr(self.ctx.device, "device_height", 2400) if self.ctx.device else 2400
default_duration = args.get("duration", 400)
kind, target, final_duration = parse_swipe_parameters(
dict(args), default_duration=default_duration
)
if kind == "direction":
dir_name = target
x1, y1, x2, y2, smart_dur = compute_smart_swipe_coordinates(
direction=target,
target=args.get("target"),
indexed_elements=getattr(state, "indexed_elements", None) if state else None,
ui_hierarchy=getattr(state, "ui_tree", None) if state else None,
width=width,
height=height,
duration=final_duration,
)
def _norm(v: float, size: int) -> int:
return int(max(0, min(1000, round(v * 1000 / max(1, size)))))
def finalize(res: ActionResult) -> str:
if not res.ok:
return f"Error swiping {dir_name}: {res.detail}"
return f"Swipe completed successfully. Swiped {dir_name}."
return (
"swipe",
{
"start": [_norm(x1, width), _norm(y1, height)],
"end": [_norm(x2, width), _norm(y2, height)],
"duration_ms": smart_dur,
},
finalize,
)
if kind == "coords":
x1, y1, x2, y2 = target
def finalize(res: ActionResult) -> str:
if not res.ok:
return f"Error dragging: {res.detail}"
return (
"Swipe completed successfully. Swiped from"
f" [{x1}, {y1}] to [{x2}, {y2}]."
)
return (
"swipe",
{
"start": [int(x1), int(y1)],
"end": [int(x2), int(y2)],
"duration_ms": final_duration,
},
finalize,
)
raise _ArgError(f"Error during swipe: Invalid direction: {args}")
# --- Non-device tools (stay agent-side) ------------------------------------------
async def _execute_agent_tool(
self,
raw_name: str,
name: str,
args: dict[str, Any],
tool_call_id: str,
state: Any,
) -> ToolExecutionResult:
span = TraceSpan(name=raw_name, trace_type="tool", ctx=self.ctx)
span.payload = {"args": args}
with span:
if raw_name == "read_note":
text = await self._read_note(
args.get("key", ""), args.get("start_line"), args.get("end_line")
)
elif raw_name == "list_notes":
text = await self._list_notes()
else:
text = await self._ask_explorer(
args.get("query") or args.get("task_description") or "",
args.get("context_feedback"),
state,
)
span.result = {"outcome": text}
# These texts come from our own failure formatters, so the historical
# "Error" marker check is reliable here (unlike free-form device output).
return ToolExecutionResult(
tool_call_id=tool_call_id,
tool_name=name,
status="error" if "Error" in (text or "") else "success",
text_summary=text,
)
async def _read_note(
self, key: str, start_line: int | None, end_line: int | None
) -> str:
try:
base_dir = self.ctx.data_engine.base_dir if self.ctx.data_engine else None
if not base_dir:
return format_read_note_failure(key, "DataEngine not initialized.")
return read_note_content(base_dir, key, start_line, end_line)
except Exception as e:
return format_read_note_failure(key, str(e))
async def _list_notes(self) -> str:
try:
base_dir = self.ctx.data_engine.base_dir if self.ctx.data_engine else None
if not base_dir:
return format_list_notes_failure("DataEngine not initialized.")
return format_list_notes_success(list_notes_info(base_dir))
except Exception as e:
return format_list_notes_failure(str(e))
async def _ask_explorer(
self, query: str, context_feedback: str | None, state: Any
) -> str:
# Imported lazily: the explorer stack is heavy and recursive (it spawns an LLM
# sub-agent), which is also why it must never live behind the action server.
from artemis.tools.explorer_tool import _run_explorer_logic
try:
active_version = resolve_explorer_version(
self.ctx,
explicit_version=None,
agent_or_profile_name="validator",
)
result = await _run_explorer_logic(
self.ctx,
state,
query,
context_feedback or "",
version=active_version,
)
if isinstance(result, list):
texts = [
item["text"]
for item in result
if isinstance(item, dict) and item.get("type") == "text"
]
return "\n".join(texts) if texts else str(result)
return str(result)
except Exception as e:
return f"Error executing ask_explorer: {e}"
+319
View File
@@ -0,0 +1,319 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Contract layer between actuator backends and the agent-facing tool surface.
This module is the single place that declares *what a backend must provide*. It sits
between the actuator implementations (``artemis/mcp/actuators/``) and the agents that
consume tools, so that a backend -- ADB today, a robot arm tomorrow -- can omit tools
it cannot perform, and add tools nobody anticipated, without any agent knowing.
Two rules govern the split between required and optional:
1. A tool is REQUIRED only when a prompt depends on it *structurally*, i.e. the prompt
cannot be assembled around its absence. Today that is ``click_sequence`` alone: the
two ``*_disappeared_analyzer.md`` prompts are built around "atomic chained execution
to defeat turn latency", which *is* ``click_sequence``.
2. Everything else is OPTIONAL, because every other prompt reference lives in a tool
enumeration slot or a self-contained instruction block that the assembly layer
(``artemis/agents/prompt_assembly.py``) renders conditionally. An optional tool that
a backend does not implement disappears from the declarations *and* from the prompt,
so it costs the model nothing.
``tests/unit/mcp/test_action_manifest.py`` re-derives rule 2 by scanning the prompt
sources, so adding a tool name to a prompt without classifying it here fails CI.
"""
from collections.abc import Awaitable, Callable
from dataclasses import dataclass, field
from typing import Any
__all__ = [
"ActuatorContractError",
"AGENT_ROLES",
"ALL_KNOWN_TOOLS",
"BACKEND_INDEPENDENT_TOOLS",
"DEVICE_ACTIONS",
"ExtensionTool",
"INTERNAL_ACTIONS",
"MIN_EXTENSION_DESCRIPTION_LENGTH",
"OPTIONAL_ACTIONS",
"REQUIRED_ACTIONS",
"available_device_actions",
"filter_declarations",
"validate_actuator",
]
# --- Tool classification -------------------------------------------------------------
#: Device actions a backend MUST implement. See rule 1 in the module docstring.
#:
#: ``click_sequence`` implies the ability to tap, so a backend providing it can
#: trivially provide ``click`` as well; ``click`` is nonetheless optional because no
#: prompt depends on it structurally once the enumeration slots are assembled.
REQUIRED_ACTIONS: frozenset[str] = frozenset(
{
"click_sequence",
}
)
#: Device actions a backend MAY implement. Absent ones vanish from both the tool
#: declarations and the rendered prompts.
OPTIONAL_ACTIONS: frozenset[str] = frozenset(
{
"click",
"long_press",
"input_text",
"swipe",
"press_key",
"manage_app",
"wait_for_delay",
# The four below appear in no prompt at all. `wait_for_text` corroborates the
# classification: it is listed in the legacy ACTION_TOOL_NAMES but has never had
# a ToolDeclaration, i.e. it has always been unreachable by any LLM.
"wait_for_text",
"open_link",
"erase_one_char",
"focus_and_clear_text",
}
)
#: Used by the client-side adapter only; never declared to any LLM, so the
#: "no cognitive load" rule does not apply. The adapter cannot function without them,
#: so they are validated as strictly as REQUIRED_ACTIONS.
INTERNAL_ACTIONS: frozenset[str] = frozenset(
{
"observe_screen",
"take_screenshot",
"get_ui_hierarchy",
}
)
#: Tools that are not device control at all -- notes, sub-agents, diagnostics, and the
#: loop-termination sentinels. They are unaffected by which actuator is installed.
#:
#: ``run_adb_command`` and ``manage_app`` ride the *system channel* rather than physical
#: actuation: a robot arm driving a handset still has ADB attached, so ``am start``
#: suffices and no icon-hunting is required. (``manage_app`` is nonetheless classified
#: as an optional device action above, since it is declared through the same path.)
BACKEND_INDEPENDENT_TOOLS: frozenset[str] = frozenset(
{
"read_note",
"list_notes",
"save_note",
"update_note",
"append_note",
"ask_explorer",
"ask_diagnoser",
"ask_committee",
"video_analyzer",
"run_adb_command",
"manage_task",
"analyze_task_output",
"reply_to_checker",
"report_task_status",
"report_failure_analysis",
}
)
#: Everything an actuator may legitimately claim in ``capabilities()``.
DEVICE_ACTIONS: frozenset[str] = REQUIRED_ACTIONS | OPTIONAL_ACTIONS | INTERNAL_ACTIONS
#: Every name the manifest knows about. Extension tools must not collide with these.
ALL_KNOWN_TOOLS: frozenset[str] = DEVICE_ACTIONS | BACKEND_INDEPENDENT_TOOLS
#: Agent roles an extension tool may target.
AGENT_ROLES: frozenset[str] = frozenset(
{
"operator",
"flash",
"failure_analyzer",
"validator",
}
)
#: Extension descriptions carry the entire usage contract (see ``ExtensionTool``), so a
#: one-word description is a bug rather than a terse style choice.
MIN_EXTENSION_DESCRIPTION_LENGTH = 40
class ActuatorContractError(RuntimeError):
"""Raised when an actuator does not satisfy the manifest contract.
Always raised at session start rather than mid-run, so a misconfigured backend
fails before the agent has taken any action on the device.
"""
@dataclass(frozen=True)
class ExtensionTool:
"""A tool contributed by a backend that the manifest does not know about.
Extension tools are invisible to the prompt layer by construction: prompts are
static text that cannot anticipate future backends, so an extension gets **no**
prompt-level teaching. Its ``description`` must therefore be self-contained --
stating when to reach for it and what it does -- or the model will never use it
correctly. :func:`validate_actuator` enforces a length floor as a crude guard.
"""
name: str
description: str
input_schema: dict[str, Any]
handler: Callable[..., Awaitable[Any]]
targets: frozenset[str] = field(default=AGENT_ROLES)
# --- Contract validation -------------------------------------------------------------
def validate_actuator(actuator: Any) -> None:
"""Checks an actuator against the manifest, raising on any violation.
Called once from ``get_action_session`` before the first device interaction.
"""
caps = frozenset(actuator.capabilities())
missing_required = REQUIRED_ACTIONS - caps
if missing_required:
raise ActuatorContractError(
f"{type(actuator).__name__} does not implement required action(s): "
f"{sorted(missing_required)}. These are structurally depended on by prompts "
f"and cannot be assembled away."
)
missing_internal = INTERNAL_ACTIONS - caps
if missing_internal:
raise ActuatorContractError(
f"{type(actuator).__name__} does not implement internal action(s): "
f"{sorted(missing_internal)}. The client-side adapter cannot observe the "
f"device without them."
)
unknown = caps - DEVICE_ACTIONS
if unknown:
raise ActuatorContractError(
f"{type(actuator).__name__} claims capabilities unknown to the manifest: "
f"{sorted(unknown)}. Backend-specific tools belong in extensions(), not "
f"capabilities()."
)
_validate_extensions(actuator)
def _validate_extensions(actuator: Any) -> None:
"""Validates extension tools for name collisions and self-sufficient descriptions."""
extensions = list(actuator.extensions())
seen: set[str] = set()
for ext in extensions:
if ext.name in ALL_KNOWN_TOOLS:
raise ActuatorContractError(
f"Extension tool '{ext.name}' collides with a manifest tool of the same "
f"name. Extension names must be unique across the whole tool surface."
)
if ext.name in seen:
raise ActuatorContractError(
f"{type(actuator).__name__} registers extension '{ext.name}' more than once."
)
seen.add(ext.name)
unknown_targets = frozenset(ext.targets) - AGENT_ROLES
if unknown_targets:
raise ActuatorContractError(
f"Extension tool '{ext.name}' targets unknown agent role(s): "
f"{sorted(unknown_targets)}. Valid roles: {sorted(AGENT_ROLES)}."
)
if len(ext.description.strip()) < MIN_EXTENSION_DESCRIPTION_LENGTH:
raise ActuatorContractError(
f"Extension tool '{ext.name}' has a description of "
f"{len(ext.description.strip())} characters. Extensions receive no "
f"prompt-level guidance, so the description must state when to use the "
f"tool and what it does (minimum "
f"{MIN_EXTENSION_DESCRIPTION_LENGTH} characters)."
)
# --- Surface derivation --------------------------------------------------------------
def available_device_actions(actuator: Any) -> frozenset[str]:
"""Returns the LLM-facing device actions this actuator supports.
Internal actions are excluded: they are adapter plumbing and never reach a model.
"""
caps = frozenset(actuator.capabilities())
return (REQUIRED_ACTIONS | OPTIONAL_ACTIONS) & caps
def filter_declarations(
declarations: list[Any],
actuator: Any,
agent: str,
) -> list[Any]:
"""Filters a declaration list down to what ``actuator`` actually provides.
Declarations for known tools keep coming from the hand-tuned ``ToolDeclaration``
constants -- this layer only removes entries, it never rewrites descriptions.
Extension declarations are appended from the actuator itself, since the manifest
cannot know them in advance.
Args:
declarations: Declarations the agent would use if every tool existed.
actuator: The installed backend.
agent: One of :data:`AGENT_ROLES`; selects which extensions apply.
Returns:
Declarations in their original order, with unavailable device actions dropped
and this agent's extension tools appended.
"""
if agent not in AGENT_ROLES:
raise ValueError(f"Unknown agent role '{agent}'. Valid roles: {sorted(AGENT_ROLES)}.")
available = available_device_actions(actuator)
kept = [
decl
for decl in declarations
# Only device actions are gated. Backend-independent tools and anything the
# manifest does not classify (e.g. an agent's own bespoke tool) pass through.
if _declaration_name(decl) not in DEVICE_ACTIONS
or _declaration_name(decl) in available
]
for ext in actuator.extensions():
if agent in ext.targets:
kept.append(_extension_declaration(ext))
return kept
def _declaration_name(declaration: Any) -> str:
"""Reads a tool name off either a ToolDeclaration mapping or a LangChain tool."""
name = getattr(declaration, "name", None)
if name is None and isinstance(declaration, dict):
name = declaration.get("name")
return name or ""
def _extension_declaration(ext: ExtensionTool) -> Any:
"""Renders an ExtensionTool into the same ToolDeclaration shape as known tools."""
from artemis.core.tool_declaration import ToolDeclaration
return ToolDeclaration(
name=ext.name,
description=ext.description,
parameters=ext.input_schema,
)
+172
View File
@@ -0,0 +1,172 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""The one place where the historical verb split is resolved.
Canonical tool names are the agent-facing ones (``click``, ``long_press``,
``input_text``, ``manage_app``, ...) because those are what the prompts teach and the
prompts are the contract. The Operator/Validator-internal vocabulary
(``tap``, ``long_press_on``, ``focus_and_input_text``, ``launch_app``/``stop_app``,
``back``) is translated here and nowhere else.
``to_canonical_call`` converts a Validator action item (the dicts Operator writes into
``state.structured_decisions``) into a canonical ``(name, kwargs)`` pair for the
actuator/session layer, using the ``normalized_coordinates`` the Operator already
computed so the canonical surface never needs a pixel escape hatch.
"""
from typing import Any
__all__ = ["OPERATOR_ACTION_TO_CANONICAL", "to_canonical_call"]
#: Operator-internal action verb -> canonical tool name.
OPERATOR_ACTION_TO_CANONICAL: dict[str, str] = {
"tap": "click",
"long_press_on": "long_press",
"swipe": "swipe",
"focus_and_input_text": "input_text",
"focus_and_clear_text": "focus_and_clear_text",
"erase_one_char": "erase_one_char",
"press_key": "press_key",
"back": "press_key",
"launch_app": "manage_app",
"stop_app": "manage_app",
"open_link": "open_link",
"wait_for_delay": "wait_for_delay",
}
def _norm_pair(
action_item: dict[str, Any], dims: tuple[int, int] | None = None
) -> tuple[int, int] | None:
"""Reads the Operator-computed normalized [nx, ny] off an action item.
Falls back to converting the pixel ``coordinates`` when the item predates the
``normalized_coordinates`` field and screen ``dims`` are provided.
"""
coords = action_item.get("normalized_coordinates")
if isinstance(coords, (list, tuple)) and len(coords) >= 2:
return int(coords[0]), int(coords[1])
pixel = action_item.get("coordinates")
if dims and isinstance(pixel, (list, tuple)) and len(pixel) >= 2:
width, height = dims
nx = int(max(0, min(1000, round(int(pixel[0]) * 1000 / max(1, width)))))
ny = int(max(0, min(1000, round(int(pixel[1]) * 1000 / max(1, height)))))
return nx, ny
return None
def to_canonical_call(
action_item: dict[str, Any], dims: tuple[int, int] | None = None
) -> tuple[str, dict[str, Any]]:
"""Translates a Validator action item into a canonical (tool, kwargs) call.
Args:
action_item: One entry of the Operator's structured decisions.
dims: Optional (width, height) for pixel->normalized fallback on legacy items
lacking ``normalized_coordinates``.
Raises:
ValueError: for an unknown action verb or an item missing the coordinates the
canonical call requires.
"""
verb = str(action_item.get("action", ""))
canonical = OPERATOR_ACTION_TO_CANONICAL.get(verb)
if canonical is None:
raise ValueError(f"Unsupported action: {verb}")
if verb == "tap":
pair = _norm_pair(action_item, dims)
if pair is None:
raise ValueError("Invalid coords")
return "click", {
"target": [pair[0], pair[1]],
"times": action_item.get("times") or action_item.get("click_times") or 1,
"delay_ms": action_item.get("delay_ms") or action_item.get("delay") or 100,
}
if verb == "long_press_on":
pair = _norm_pair(action_item, dims)
if pair is None:
raise ValueError("Invalid coords")
return "long_press", {
"target": [pair[0], pair[1]],
"duration_ms": action_item.get("duration", 1000),
}
if verb == "swipe":
coords = action_item.get("normalized_coordinates")
if not (isinstance(coords, (list, tuple)) and len(coords) == 4):
pixel = action_item.get("coordinates")
if dims and isinstance(pixel, (list, tuple)) and len(pixel) == 4:
width, height = dims
coords = [
int(max(0, min(1000, round(int(pixel[0]) * 1000 / max(1, width))))),
int(max(0, min(1000, round(int(pixel[1]) * 1000 / max(1, height))))),
int(max(0, min(1000, round(int(pixel[2]) * 1000 / max(1, width))))),
int(max(0, min(1000, round(int(pixel[3]) * 1000 / max(1, height))))),
]
else:
raise ValueError("Invalid coords")
return "swipe", {
"start": [int(coords[0]), int(coords[1])],
"end": [int(coords[2]), int(coords[3])],
"duration_ms": action_item.get("duration", 400),
}
if verb == "focus_and_input_text":
pair = _norm_pair(action_item, dims)
if pair is None:
raise ValueError("Invalid coords")
return "input_text", {
"text": action_item.get("text", ""),
"target": [pair[0], pair[1]],
"clear_exist": bool(action_item.get("clear_before_input", False)),
}
if verb == "focus_and_clear_text":
pair = _norm_pair(action_item, dims)
if pair is None:
raise ValueError("Invalid coords")
return "focus_and_clear_text", {"target": [pair[0], pair[1]]}
if verb == "erase_one_char":
return "erase_one_char", {}
if verb == "press_key":
keycode = str(action_item.get("keycode", ""))
# Operator emits Android-style KEYCODE_* names; the canonical tool takes the
# bare key word for the common keys. Anything else keeps its original spelling
# so the driver can forward it verbatim (arbitrary KEYCODE_* / numeric codes).
bare = keycode.removeprefix("KEYCODE_").lower() if keycode else ""
known = {"home", "back", "enter", "delete", "tab", "search", "menu", "app_switch"}
return "press_key", {"key": bare if bare in known else keycode}
if verb == "back":
return "press_key", {"key": "back"}
if verb in ("launch_app", "stop_app"):
return "manage_app", {
"action": "launch" if verb == "launch_app" else "stop",
"app_name": action_item.get("app_name", ""),
}
if verb == "open_link":
return "open_link", {"url": action_item.get("url", "")}
if verb == "wait_for_delay":
return "wait_for_delay", {"time_in_ms": action_item.get("time_in_ms", 0)}
raise ValueError(f"Unsupported action: {verb}")
+353
View File
@@ -0,0 +1,353 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""The unified device action MCP server.
``build_action_server`` turns an actuator into a ``FastMCP`` server that registers
**only** the tools the actuator implements, plus its extension tools -- so
``list_tools()`` is clean by construction and an absent optional tool is invisible to
every consumer.
Return protocol: every device action returns a ``CallToolResult`` whose
``structuredContent`` is an ``ActionResult`` -- ``ok``/``code`` carry the real status
(killing the historical ``"Error" in text`` sniffing) while ``content`` keeps the
historical human-readable message. ``isError`` is reserved for protocol-level
failures; a device refusing an action is ``isError=False, ok=False`` so it reaches the
model as an observation, not an exception.
Exceptions raised by the actuator are wrapped here with the historical
``"Error during X: {e}"`` wording the agents' transcripts already contain.
"""
import asyncio
from typing import Annotated, Any
from mcp.server.fastmcp import FastMCP
from mcp.types import CallToolResult, ImageContent, TextContent
from artemis.mcp.action_manifest import validate_actuator
from artemis.mcp.action_types import ActionCode, ActionResult, ObserveResult
from artemis.mcp.actuators.base import Actuator
from artemis.mcp.observation import observe as observe_impl
from artemis.utils.logger import get_logger
logger = get_logger(__name__)
__all__ = ["build_action_server"]
# Exception wrapping per tool; wording matches the historical executor `except` arms.
_EXCEPTION_PREFIX = {
"click": "Error during click",
"click_sequence": "Error executing click sequence",
"long_press": "Error during long press",
"input_text": "Error during input text",
"swipe": "Error during swipe",
"press_key": "Error during press_key",
"manage_app": "Error during manage_app",
"wait_for_delay": "Error during wait_for_delay",
"wait_for_text": "Error during wait_for_text",
}
def _wrap(res: ActionResult) -> CallToolResult:
return CallToolResult(
content=[TextContent(type="text", text=res.message)],
structuredContent=res.model_dump(mode="json"),
isError=False,
)
def _wrap_exception(action: str, exc: Exception) -> CallToolResult:
prefix = _EXCEPTION_PREFIX.get(action, f"Error during {action}")
return _wrap(
ActionResult.failure(action, f"{prefix}: {exc}", detail=repr(exc))
)
def build_action_server(actuator: Actuator, name: str = "artemis_actions") -> FastMCP:
"""Builds a FastMCP server exposing exactly what ``actuator`` implements.
Raises ``ActuatorContractError`` when the actuator violates the manifest, so a
misconfigured backend fails before serving anything.
"""
validate_actuator(actuator)
caps = actuator.capabilities()
mcp = FastMCP(name)
# --- Device actions (registered only when implemented) ---------------------------
if "click" in caps:
@mcp.tool()
async def click(
target: list[int], times: int = 1, delay_ms: int = 100
) -> Annotated[CallToolResult, ActionResult]:
"""Tap at a 0-1000 normalized [x, y] coordinate."""
try:
return _wrap(
await actuator.click(
int(target[0]), int(target[1]), times=times, delay_ms=delay_ms
)
)
except Exception as e:
return _wrap_exception("click", e)
if "click_sequence" in caps:
@mcp.tool()
async def click_sequence(
sequence: list[list[int]], delay_ms: int = 50
) -> Annotated[CallToolResult, ActionResult]:
"""Tap a series of 0-1000 normalized [x, y] points in one atomic burst."""
try:
points = [(int(p[0]), int(p[1])) for p in sequence]
return _wrap(await actuator.click_sequence(points, delay_ms=delay_ms))
except Exception as e:
return _wrap_exception("click_sequence", e)
if "long_press" in caps:
@mcp.tool()
async def long_press(
target: list[int], duration_ms: int = 1000
) -> Annotated[CallToolResult, ActionResult]:
"""Long-press at a 0-1000 normalized [x, y] coordinate."""
try:
return _wrap(
await actuator.long_press(
int(target[0]), int(target[1]), duration_ms=duration_ms
)
)
except Exception as e:
return _wrap_exception("long_press", e)
if "input_text" in caps:
@mcp.tool()
async def input_text(
text: str,
target: list[int] | None = None,
clear_exist: bool = True,
) -> Annotated[CallToolResult, ActionResult]:
"""Type text, optionally focusing a 0-1000 normalized [x, y] target first."""
try:
norm = (int(target[0]), int(target[1])) if target else None
return _wrap(
await actuator.input_text(text, norm, clear_exist=clear_exist)
)
except Exception as e:
return _wrap_exception("input_text", e)
if "swipe" in caps:
@mcp.tool()
async def swipe(
start: list[int], end: list[int], duration_ms: int = 800
) -> Annotated[CallToolResult, ActionResult]:
"""Swipe between two 0-1000 normalized [x, y] points."""
try:
return _wrap(
await actuator.swipe(
(int(start[0]), int(start[1])),
(int(end[0]), int(end[1])),
duration_ms,
)
)
except Exception as e:
return _wrap_exception("swipe", e)
if "press_key" in caps:
@mcp.tool()
async def press_key(key: str) -> Annotated[CallToolResult, ActionResult]:
"""Press a device key (home, back, enter, delete, tab, search, menu, app_switch)."""
try:
return _wrap(await actuator.press_key(key))
except Exception as e:
return _wrap_exception("press_key", e)
if "manage_app" in caps:
@mcp.tool()
async def manage_app(
action: str, app_name: str
) -> Annotated[CallToolResult, ActionResult]:
"""Launch or stop an app by human-readable name or package."""
try:
return _wrap(await actuator.manage_app(action, app_name))
except Exception as e:
return _wrap_exception("manage_app", e)
if "wait_for_delay" in caps:
@mcp.tool()
async def wait_for_delay(
time_in_ms: int,
) -> Annotated[CallToolResult, ActionResult]:
"""Wait for a fixed number of milliseconds."""
try:
return _wrap(await actuator.wait_for_delay(time_in_ms))
except Exception as e:
return _wrap_exception("wait_for_delay", e)
if "wait_for_text" in caps:
@mcp.tool()
async def wait_for_text(
text: str, wait_state: str = "appear", timeout_ms: int = 5000
) -> Annotated[CallToolResult, ActionResult]:
"""Wait for text to appear on or disappear from the screen."""
try:
return _wrap(await actuator.wait_for_text(text, wait_state, timeout_ms))
except Exception as e:
return _wrap_exception("wait_for_text", e)
if "open_link" in caps:
@mcp.tool()
async def open_link(url: str) -> Annotated[CallToolResult, ActionResult]:
"""Open a URL on the device."""
try:
return _wrap(await actuator.open_link(url))
except Exception as e:
return _wrap_exception("open_link", e)
if "erase_one_char" in caps:
@mcp.tool()
async def erase_one_char() -> Annotated[CallToolResult, ActionResult]:
"""Erase a single character in the focused field."""
try:
return _wrap(await actuator.erase_one_char())
except Exception as e:
return _wrap_exception("erase_one_char", e)
if "focus_and_clear_text" in caps:
@mcp.tool()
async def focus_and_clear_text(
target: list[int],
) -> Annotated[CallToolResult, ActionResult]:
"""Focus the field at a 0-1000 normalized [x, y] and clear its text."""
try:
return _wrap(
await actuator.focus_and_clear_text(int(target[0]), int(target[1]))
)
except Exception as e:
return _wrap_exception("focus_and_clear_text", e)
# --- Internal observation tools (never declared to an LLM) -----------------------
if "observe_screen" in caps:
@mcp.tool()
async def observe_screen(
settle_ms: int = 400, include_image: bool = False
) -> Annotated[CallToolResult, ObserveResult]:
"""Capture and index the current screen (adapter-internal)."""
try:
obs, img_bytes = await observe_impl(
getattr(actuator, "ctx", None),
actuator.controller,
settle_ms=settle_ms,
)
content: list[Any] = [TextContent(type="text", text=obs.message)]
if include_image and img_bytes:
import base64 as _b64
content.append(
ImageContent(
type="image",
data=_b64.b64encode(img_bytes).decode("utf-8"),
mimeType="image/jpeg",
)
)
return CallToolResult(
content=content,
structuredContent=obs.model_dump(mode="json"),
isError=False,
)
except Exception as e:
obs = ObserveResult(
ok=False,
code=ActionCode.DEVICE_ERROR,
message=f"Error during observe_screen: {e}",
hierarchy_ok=False,
)
return CallToolResult(
content=[TextContent(type="text", text=obs.message)],
structuredContent=obs.model_dump(mode="json"),
isError=False,
)
# Timeouts on the two observation tools run INSIDE the tool, around the actuator
# call only. They must never be expressed as a transport-level read timeout
# (``read_timeout_seconds``) or a caller-side ``asyncio.wait_for``: both produce a
# late response to an abandoned request, which crashes the mcp-1.29 client
# receive loop (``_handle_response`` sends into the request's already-closed
# response stream) and bricks the in-memory session for every later caller.
# A server-side timeout yields a normal error payload and no late response.
if "take_screenshot" in caps:
@mcp.tool()
async def take_screenshot(timeout_ms: int | None = None) -> CallToolResult:
"""Return the current screen as base64 (adapter-internal, no persistence)."""
try:
coro = actuator.take_screenshot()
if timeout_ms:
coro = asyncio.wait_for(coro, timeout=timeout_ms / 1000.0)
b64 = await coro
payload: dict[str, Any] = {"ok": True, "image_b64": b64}
except (TimeoutError, asyncio.TimeoutError):
payload = {"ok": False, "error": f"screenshot timed out after {timeout_ms}ms"}
except Exception as e:
payload = {"ok": False, "error": str(e)}
return CallToolResult(
content=[TextContent(type="text", text="screenshot" if payload["ok"] else str(payload))],
structuredContent=payload,
isError=False,
)
if "get_ui_hierarchy" in caps:
@mcp.tool()
async def get_ui_hierarchy(timeout_ms: int | None = None) -> CallToolResult:
"""Return the raw UI element hierarchy (adapter-internal)."""
try:
coro = actuator.get_ui_elements()
if timeout_ms:
coro = asyncio.wait_for(coro, timeout=timeout_ms / 1000.0)
elements = await coro
payload: dict[str, Any] = {"ok": True, "elements": elements}
except (TimeoutError, asyncio.TimeoutError):
payload = {"ok": False, "error": f"ui hierarchy fetch timed out after {timeout_ms}ms"}
except Exception as e:
payload = {"ok": False, "error": str(e)}
return CallToolResult(
content=[TextContent(type="text", text="ui_hierarchy" if payload["ok"] else str(payload))],
structuredContent=payload,
isError=False,
)
# --- Backend extension tools -----------------------------------------------------
for ext in actuator.extensions():
mcp.add_tool(ext.handler, name=ext.name, description=ext.description)
logger.info(
f"Action server '{name}' built with {len(caps)} device actions and"
f" {len(actuator.extensions())} extension tools."
)
return mcp
+249
View File
@@ -0,0 +1,249 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""In-process MCP session over the unified action server.
The in-memory transport (``create_connected_server_and_client_session``) wraps the
server in an ``anyio`` task group whose cancel scope must be entered and exited **in
the same task**. ``ActionSession`` therefore runs the whole context manager inside one
owner task and communicates through events: ``call()`` and ``aclose()`` are safe from
any task, which is what lets the session be created lazily inside an agent node and
torn down from ``ArtemisContext.__aexit__``.
Transport operations additionally run **only in the owner task**, via a request
queue: callers enqueue ``(future, name, args, timeout)`` and await the future. A
caller that times out or is cancelled only abandons its future -- the underlying
``call_tool`` is never cancelled mid-flight. This matters because cancelling
``call_tool`` on the in-memory transport corrupts the shared memory streams and
bricks the session for every subsequent caller (observed live: one 1-second
``asyncio.wait_for`` around a slow hierarchy fetch killed a 24-minute run, with
every later call failing on an empty-message ``BrokenResourceError``).
MCP's native ``read_timeout_seconds`` is just as fatal here (also observed live):
mcp 1.29's ``_handle_response`` delivers a LATE response into the timed-out
request's already-closed response stream, which crashes the client receive loop.
Per-call timeouts therefore travel as a ``timeout_ms`` tool argument and fire
inside the server tool, around the actuator call only -- a timed-out operation
produces a normal error payload and never a late response.
If the transport does die despite all this, the owner loop exits, which flips
``started`` to False so the next ``get_action_session`` builds a fresh session
instead of leaving the rest of the run to fail on a corpse.
The single owner loop also serializes calls, which enforces ARTEMIS's
single-device policy at the one choke point all agents share -- in particular it
stops a nested Validator -> FailureAnalyzer flow from interleaving two device
actions.
"""
import asyncio
from typing import Any
import anyio
from mcp.shared.memory import create_connected_server_and_client_session
from artemis.mcp.action_types import ActionResult, ObserveResult
from artemis.utils.logger import get_logger
logger = get_logger(__name__)
__all__ = ["ActionSession", "get_action_session"]
_SHUTDOWN = object() # queue sentinel
class ActionSession:
"""Owns the in-memory MCP client/server pair for one device run."""
def __init__(self, server: Any):
self._server = server
self._session: Any = None
self._ready = asyncio.Event()
self._owner: asyncio.Task | None = None
self._error: BaseException | None = None
self._requests: asyncio.Queue = asyncio.Queue()
async def _run(self) -> None:
try:
# Enter and exit strictly within this task (anyio cancel-scope rule).
async with create_connected_server_and_client_session(self._server) as session:
self._session = session
self._ready.set()
await self._serve(session)
except BaseException as e:
self._error = e
self._ready.set()
raise
finally:
self._session = None
self._fail_pending("ActionSession owner task has exited.")
async def _serve(self, session: Any) -> None:
"""Executes queued transport calls until the shutdown sentinel arrives."""
while True:
item = await self._requests.get()
if item is _SHUTDOWN:
return
fut, name, args = item
if fut.cancelled():
continue # caller gave up before we started; nothing was sent
try:
result = await session.call_tool(name, args)
except BaseException as e:
if not fut.cancelled():
fut.set_exception(e)
if isinstance(e, asyncio.CancelledError):
raise
if isinstance(e, (anyio.ClosedResourceError, anyio.BrokenResourceError)):
# The transport is dead; no future call on this session can
# succeed. Exit so `started` flips False and the next
# get_action_session() builds a fresh session.
logger.error(
"Action session transport failed (%r); retiring this session.", e
)
return
else:
if not fut.cancelled():
fut.set_result(result)
def _fail_pending(self, reason: str) -> None:
while not self._requests.empty():
item = self._requests.get_nowait()
if item is _SHUTDOWN:
continue
fut = item[0]
if not fut.done():
fut.set_exception(RuntimeError(reason))
async def start(self) -> "ActionSession":
if self._owner is not None:
return self
self._owner = asyncio.create_task(self._run(), name="artemis-action-session")
await self._ready.wait()
if self._error is not None:
error = self._error
self._owner = None
self._error = None
raise RuntimeError(f"Failed to start action session: {error}") from error
return self
@property
def started(self) -> bool:
return self._session is not None
async def call_raw(self, name: str, args: dict[str, Any]) -> Any:
"""Calls a tool and returns the raw ``CallToolResult``.
There is deliberately no transport-level timeout here (see the module
docstring). Tools that support one take a ``timeout_ms`` argument and time
out server-side. Wrapping this coroutine in ``asyncio.wait_for`` merely
abandons the caller's future -- harmless, because the transport call itself
runs in the owner task and cannot be cancelled from here.
"""
if self._session is None:
raise RuntimeError("ActionSession is not started.")
fut: asyncio.Future = asyncio.get_running_loop().create_future()
self._requests.put_nowait((fut, name, args))
result = await fut
if getattr(result, "isError", False):
# Protocol-level failure (unknown tool, schema violation) -- a programming
# error, not a device observation.
text = ""
for block in getattr(result, "content", []) or []:
if getattr(block, "type", None) == "text":
text = block.text
break
raise RuntimeError(f"MCP tool '{name}' failed: {text}")
return result
async def call(self, name: str, args: dict[str, Any]) -> ActionResult:
"""Calls a device action tool and returns its typed result."""
result = await self.call_raw(name, args)
return ActionResult.model_validate(result.structuredContent)
async def observe(
self, settle_ms: int = 400, include_image: bool = False
) -> ObserveResult:
"""Captures and indexes the current screen via the internal observe tool."""
result = await self.call_raw(
"observe_screen", {"settle_ms": settle_ms, "include_image": include_image}
)
return ObserveResult.model_validate(result.structuredContent)
async def screenshot_b64(self, timeout: float | None = None) -> str:
"""Returns the current screen as a base64 string (no persistence)."""
args = {"timeout_ms": int(timeout * 1000)} if timeout else {}
result = await self.call_raw("take_screenshot", args)
payload = result.structuredContent or {}
# FastMCP wraps plain-dict returns as {"result": ...} only for non-dict types;
# dicts pass through.
data = payload.get("result", payload)
if not data.get("ok"):
raise RuntimeError(f"take_screenshot failed: {data.get('error')}")
return data["image_b64"]
async def ui_hierarchy(self, timeout: float | None = None) -> Any:
"""Returns the raw UI element hierarchy."""
args = {"timeout_ms": int(timeout * 1000)} if timeout else {}
result = await self.call_raw("get_ui_hierarchy", args)
payload = result.structuredContent or {}
data = payload.get("result", payload)
if not data.get("ok"):
raise RuntimeError(f"get_ui_hierarchy failed: {data.get('error')}")
return data["elements"]
async def aclose(self) -> None:
"""Shuts the session down; safe to call from any task."""
if self._owner is None:
return
self._requests.put_nowait(_SHUTDOWN)
try:
await asyncio.wait_for(asyncio.shield(self._owner), timeout=5.0)
except (TimeoutError, asyncio.CancelledError, Exception) as e:
logger.warning(f"Action session owner task did not exit cleanly: {e!r}")
finally:
self._owner = None
self._ready = asyncio.Event()
async def get_action_session(ctx: Any, actuator: Any = None) -> ActionSession:
"""Returns the context's action session, creating and starting it on first use.
Contract validation happens here (inside ``build_action_server``), so a backend
that cannot satisfy the manifest fails before the first device interaction.
"""
existing = getattr(ctx, "action_session", None)
if existing is not None and existing.started:
return existing
if existing is not None:
# A dead session still owns a task and memory streams: release them
# before the replacement is built.
try:
await existing.aclose()
except Exception:
logger.debug("Closing a dead action session failed; replacing it anyway.")
from artemis.mcp.action_server import build_action_server
from artemis.mcp.actuators.adb import AdbActuator
actuator = actuator or getattr(ctx, "actuator", None) or AdbActuator(ctx)
server = build_action_server(actuator)
session = ActionSession(server)
await session.start()
try:
ctx.action_session = session
except Exception:
# Contexts that forbid new attributes (strict mocks) can still use the session.
logger.debug("Could not store action session on ctx; proceeding unattached.")
return session
+86
View File
@@ -0,0 +1,86 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Typed results for the device action layer.
These models replace the ``"Error" in text`` substring sniffing: success is carried by
``ok``/``code``, while ``message`` remains the human/LLM-facing summary and keeps its
historical wording. Kept dependency-free (pydantic only) so servers, adapters, and
tests can all import it cheaply.
"""
from enum import Enum
from typing import Any
from pydantic import BaseModel, Field
__all__ = ["ActionCode", "ActionResult", "ObserveResult"]
class ActionCode(str, Enum):
"""Machine-readable outcome classification for a device action."""
OK = "OK"
INVALID_ARGS = "INVALID_ARGS"
TARGET_NOT_FOUND = "TARGET_NOT_FOUND"
DEVICE_ERROR = "DEVICE_ERROR"
PACKAGE_NOT_FOUND = "PACKAGE_NOT_FOUND"
TIMEOUT = "TIMEOUT"
UNSUPPORTED = "UNSUPPORTED"
class ActionResult(BaseModel):
"""Outcome of one device action, independent of any transport or agent."""
ok: bool
code: ActionCode
action: str
message: str
"""Agent-facing summary; wording matches the historical executor outcomes."""
detail: str | None = None
"""Raw driver/controller error for diagnostics; never shown to the LLM."""
normalized_coordinates: list[int] | None = None
duration_ms: int | None = None
@classmethod
def success(cls, action: str, message: str, **kwargs: Any) -> "ActionResult":
return cls(ok=True, code=ActionCode.OK, action=action, message=message, **kwargs)
@classmethod
def failure(
cls,
action: str,
message: str,
code: ActionCode = ActionCode.DEVICE_ERROR,
**kwargs: Any,
) -> "ActionResult":
return cls(ok=False, code=code, action=action, message=message, **kwargs)
class ObserveResult(BaseModel):
"""Post-action observation: persisted screenshot plus the parsed element list."""
ok: bool
code: ActionCode = ActionCode.OK
message: str = ""
screenshot_path: str | None = None
image_name: str | None = None
"""Stem of ``screenshot_path``; stable key used by trace/replay consumers."""
width: int | None = None
height: int | None = None
elements_text: str | None = None
elements: list[dict[str, Any]] = Field(default_factory=list)
hierarchy_ok: bool = True
"""False when the screenshot succeeded but the UI hierarchy parse failed; callers
must then keep their previous element index rather than clobbering it."""
+21
View File
@@ -0,0 +1,21 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Actuator backends: pluggable implementations of the device action contract."""
from artemis.mcp.actuators.base import Actuator
from artemis.mcp.actuators.adb import AdbActuator
from artemis.mcp.actuators.mock import MockActuator
__all__ = ["Actuator", "AdbActuator", "MockActuator"]
+419
View File
@@ -0,0 +1,419 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""ADB-backed actuator: the reference implementation of the actuator contract.
The device-call bodies here were extracted from
``artemis.agents.validator.tool_declarations.MobileActionExecutor``; the outcome
message wording is preserved byte-for-byte because agent transcripts, trace payloads,
and existing tests assert on it.
Error convention: device-level refusals the controller *reports* (an ``error`` field,
a falsy success flag) come back as ``ActionResult(ok=False)`` with the historical
message. Unexpected exceptions are allowed to propagate -- the caller (executor or MCP
server layer) owns the ``"Error during X: {e}"`` wrapping, exactly as the original
executor's ``try/except`` did.
"""
import asyncio
import inspect
import re
import time
from typing import Any
from artemis.context import ArtemisContext
from artemis.controllers.unified_controller import UnifiedMobileController
from artemis.mcp.action_manifest import DEVICE_ACTIONS, ExtensionTool
from artemis.mcp.action_types import ActionCode, ActionResult
from artemis.utils.logger import get_logger
logger = get_logger(__name__)
__all__ = ["AdbActuator", "find_element_at_coords", "ensure_focus_at_coords"]
def find_element_at_coords(elements: list[dict], x: int, y: int) -> dict | None:
"""Finds the smallest (leaf-most) focusable element containing pixel [x, y].
Moved from ``artemis.mcp.adb_server`` so both the stdio server and in-process
actuator share one implementation.
"""
matching_element = None
min_area = float("inf")
for elem in elements:
is_focusable = (
elem.get("focusable") == "true"
or elem.get("clickable") == "true"
or "EditText" in str(elem.get("class", ""))
)
if not is_focusable:
continue
bounds_str = elem.get("bounds")
if bounds_str and isinstance(bounds_str, str):
match = re.match(r"\[(\-?\d+),(\-?\d+)\]\[(\-?\d+),(\-?\d+)\]", bounds_str)
if match:
x1, y1, x2, y2 = map(int, match.groups())
if x1 <= x <= x2 and y1 <= y <= y2:
area = (x2 - x1) * (y2 - y1)
if area < min_area:
min_area = area
matching_element = elem
return matching_element
async def ensure_focus_at_coords(controller, x: int, y: int) -> str | None:
"""Ensures the element at pixel [x, y] is focused, tapping only if needed.
Returns an error string on failure, ``None`` on success. Moved from
``artemis.mcp.adb_server``.
"""
try:
elements = await controller.get_ui_elements()
elem = find_element_at_coords(elements, x, y)
if elem and elem.get("focused") == "true":
logger.info(f"Element under [{x}, {y}] is already focused. Skipping tap.")
return None
except Exception as e:
logger.warning(f"Failed to check focus status: {e}. Falling back to unconditional tap.")
result = await controller.tap_at(x=x, y=y)
if hasattr(result, "error") and result.error:
return result.error
# Wait for the UI to settle and keyboard to pop up after tapping
await asyncio.sleep(1.0)
return None
class AdbActuator:
"""Drives an Android device through ``UnifiedMobileController`` over ADB."""
def __init__(
self,
ctx: ArtemisContext,
controller: UnifiedMobileController | None = None,
):
self.ctx = ctx
self.controller = controller or UnifiedMobileController(ctx)
# --- Contract --------------------------------------------------------------------
def capabilities(self) -> frozenset[str]:
return DEVICE_ACTIONS
def extensions(self) -> list[ExtensionTool]:
return []
# --- Coordinate helpers ----------------------------------------------------------
def _dims(self) -> tuple[int, int]:
width = getattr(self.ctx.device, "device_width", 1080) if self.ctx.device else 1080
height = getattr(self.ctx.device, "device_height", 2400) if self.ctx.device else 2400
return width or 1080, height or 2400
def _to_px(self, nx: int, ny: int) -> tuple[int, int]:
width, height = self._dims()
x = int(max(0, min(width - 1, int(nx) * width / 1000)))
y = int(max(0, min(height - 1, int(ny) * height / 1000)))
return x, y
# --- Required action -------------------------------------------------------------
async def click_sequence(
self, points: list[tuple[int, int]], delay_ms: int = 50
) -> ActionResult:
outcomes = []
for i, (nx, ny) in enumerate(points):
x, y = self._to_px(nx, ny)
result = await self.controller.tap_at(x, y)
err = getattr(result, "error", None) if result else None
if err:
return ActionResult.failure(
"click_sequence",
f"Error executing click at step {i + 1}: {err}",
detail=str(err),
)
outcomes.append(f"Tapped at [{x}, {y}]")
if i < len(points) - 1:
await asyncio.sleep(max(0, delay_ms) / 1000.0)
return ActionResult.success(
"click_sequence",
f"Sequence clicked successfully: {'; '.join(outcomes)}",
)
# --- Optional actions ------------------------------------------------------------
async def click(
self, nx: int, ny: int, times: int = 1, delay_ms: int = 100
) -> ActionResult:
x, y = self._to_px(nx, ny)
result = await self.controller.tap_at(x, y, times=times, delay_ms=delay_ms)
err = getattr(result, "error", None) if result else None
if err is not None:
return ActionResult.failure(
"click", f"Error executing click: {err}", detail=str(err)
)
return ActionResult.success(
"click",
f"Clicked at [{nx}, {ny}] (normalized) successfully.",
normalized_coordinates=[int(nx), int(ny)],
)
async def long_press(self, nx: int, ny: int, duration_ms: int = 1000) -> ActionResult:
x, y = self._to_px(nx, ny)
result = await self.controller.tap_at(
x, y, long_press=True, long_press_duration=duration_ms
)
err = getattr(result, "error", None) if result else None
if err is not None:
return ActionResult.failure(
"long_press", f"Error executing long press: {err}", detail=str(err)
)
return ActionResult.success(
"long_press",
f"Long pressed at [{nx}, {ny}] (normalized) for {duration_ms}ms successfully.",
normalized_coordinates=[int(nx), int(ny)],
duration_ms=duration_ms,
)
async def input_text(
self,
text: str,
target: tuple[int, int] | None = None,
clear_exist: bool = True,
) -> ActionResult:
coords = None
if target:
nx, ny = target
x, y = self._to_px(nx, ny)
coords = [int(nx), int(ny)]
err = await ensure_focus_at_coords(self.controller, x, y)
if err:
return ActionResult.failure(
"input_text",
f"Error focusing element: {err}",
detail=str(err),
normalized_coordinates=coords,
)
if clear_exist:
if hasattr(self.controller, "erase_text") and not await self.controller.erase_text():
return ActionResult.failure(
"input_text",
"Failed to clear existing text",
normalized_coordinates=coords,
)
elif hasattr(self.controller, "press_key"):
# KEYCODE_MOVE_END: move cursor to the end for reliable append.
await self.controller.press_key("123")
if hasattr(self.controller, "type_text") and not await self.controller.type_text(
text, clear_existing=False
):
return ActionResult.failure(
"input_text",
f"Failed typing '{text}'.",
normalized_coordinates=coords,
)
return ActionResult.success(
"input_text",
f"Executed typing '{text}'.",
normalized_coordinates=coords,
)
async def swipe(
self,
start: tuple[int, int],
end: tuple[int, int],
duration_ms: int = 800,
) -> ActionResult:
nx1, ny1 = start
nx2, ny2 = end
px1, py1 = self._to_px(nx1, ny1)
px2, py2 = self._to_px(nx2, ny2)
err = await self.controller.swipe_coords(px1, py1, px2, py2, duration_ms)
if err:
return ActionResult.failure("swipe", f"Error dragging: {err}", detail=str(err))
return ActionResult.success(
"swipe",
f"Swipe completed successfully. Swiped from [{nx1}, {ny1}] to [{nx2}, {ny2}].",
normalized_coordinates=[int(nx1), int(ny1), int(nx2), int(ny2)],
duration_ms=duration_ms,
)
async def press_key(self, key: str) -> ActionResult:
key_str = str(key).lower()
if not key_str:
return ActionResult.failure(
"press_key",
f"Error executing key press '{key}'.",
code=ActionCode.INVALID_ARGS,
)
if key_str == "back" and hasattr(self.controller, "go_back"):
await self.controller.go_back()
elif key_str == "home" and hasattr(self.controller, "go_home"):
await self.controller.go_home()
elif key_str == "enter" and hasattr(self.controller, "press_enter"):
await self.controller.press_enter()
elif hasattr(self.controller, "press_key"):
# Pass unrecognized keycodes through verbatim (KEYCODE_* names, numeric
# codes): the driver resolves or forwards them, matching the historical
# adb_server behavior of accepting any Android key event.
res = await self.controller.press_key(key)
err = getattr(res, "error", None) if res else None
if err:
return ActionResult.failure(
"press_key",
f"Error executing key press '{key}': {err}",
detail=str(err),
)
if not res:
return ActionResult.failure(
"press_key", f"Error executing key press '{key}'."
)
return ActionResult.success("press_key", f"Executed key press '{key}'.")
async def manage_app(self, action: str, app_name: str) -> ActionResult:
# Imported lazily: launch_app pulls in tool wrappers that are costly at import.
from artemis.tools.mobile.launch_app import find_package
from artemis.utils.app_launch_utils import launch_app_with_retries
res = find_package(self.ctx, app_name, use_fallback=False)
pkg = await res if inspect.iscoroutine(res) else res
if not pkg:
return ActionResult.failure(
"manage_app",
f"Error finding package for app: {app_name}",
code=ActionCode.PACKAGE_NOT_FOUND,
)
target_pkg = pkg
if action.lower() == "launch":
res_launch = launch_app_with_retries(self.ctx, target_pkg)
if inspect.iscoroutine(res_launch):
res_launch = await res_launch
if isinstance(res_launch, tuple):
success, error_msg = res_launch
else:
success, error_msg = bool(res_launch), ""
if success:
return ActionResult.success(
"manage_app",
f"Launched app '{app_name}' ({target_pkg}) successfully.",
)
return ActionResult.failure(
"manage_app",
f"Failed to launch app '{app_name}': {error_msg}",
detail=error_msg or None,
)
elif action.lower() == "stop":
if hasattr(self.controller, "terminate_app"):
res_term = self.controller.terminate_app(target_pkg)
if inspect.iscoroutine(res_term):
await res_term
return ActionResult.success(
"manage_app", f"Terminated app '{app_name}' successfully."
)
return ActionResult.failure(
"manage_app",
f"Invalid manage_app action: {action}",
code=ActionCode.INVALID_ARGS,
)
async def wait_for_delay(self, time_in_ms: int) -> ActionResult:
delay_s = max(0.0, float(time_in_ms) / 1000.0)
await asyncio.sleep(delay_s)
return ActionResult.success(
"wait_for_delay",
f"Waited for {time_in_ms}ms successfully.",
duration_ms=int(time_in_ms),
)
async def wait_for_text(
self, text: str, wait_state: str | None = None, timeout_ms: int | None = None
) -> ActionResult:
timeout_s = (timeout_ms or 5000) / 1000.0
start = time.time()
found = False
target_state = (wait_state or "appear").lower()
while time.time() - start < timeout_s:
tree = await self.controller.get_ui_elements()
text_present = text.lower() in str(tree or "").lower()
if target_state == "appear" and text_present:
found = True
break
elif target_state == "disappear" and not text_present:
found = True
break
await asyncio.sleep(0.5)
if found:
return ActionResult.success(
"wait_for_text",
f"Successfully waited for text '{text}' to {target_state}.",
)
return ActionResult.failure(
"wait_for_text",
f"Timed out waiting for text '{text}' to {target_state}.",
code=ActionCode.TIMEOUT,
)
async def open_link(self, url: str) -> ActionResult:
success = await self.controller.open_url(url)
if success:
return ActionResult.success("open_link", f"Opened link '{url}' successfully.")
return ActionResult.failure("open_link", f"Failed to open link '{url}'.")
async def erase_one_char(self) -> ActionResult:
success = await self.controller.erase_text(nb_chars=1)
if success:
return ActionResult.success("erase_one_char", "Erased one character.")
return ActionResult.failure("erase_one_char", "Failed to erase one character.")
async def focus_and_clear_text(self, nx: int, ny: int) -> ActionResult:
x, y = self._to_px(nx, ny)
err = await ensure_focus_at_coords(self.controller, x, y)
if err:
return ActionResult.failure(
"focus_and_clear_text",
f"Error focusing element: {err}",
detail=str(err),
normalized_coordinates=[int(nx), int(ny)],
)
success = await self.controller.erase_text()
if success:
return ActionResult.success(
"focus_and_clear_text",
f"Cleared text at [{nx}, {ny}] (normalized) successfully.",
normalized_coordinates=[int(nx), int(ny)],
)
return ActionResult.failure("focus_and_clear_text", "Failed to erase text.")
# --- Internal observation primitives ---------------------------------------------
async def take_screenshot(self) -> str:
return await self.controller.take_screenshot()
async def get_ui_elements(self) -> Any:
return await self.controller.get_ui_elements()
async def get_screen_data(self) -> Any:
return await self.controller.get_screen_data()
+113
View File
@@ -0,0 +1,113 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""The actuator protocol: what any backend must look like to drive a device.
An actuator answers exactly one question -- *how to perform a physical action* -- in
the canonical 0-1000 normalized coordinate space. It knows nothing about LangGraph
``State``, prompts, agents, screenshots-as-observations, or MCP. Pixel geometry (or a
robot arm's motion planning) is a private detail behind this boundary.
A backend advertises what it can do through :meth:`Actuator.capabilities` (names from
``artemis.mcp.action_manifest.DEVICE_ACTIONS``) and may contribute tools nobody
anticipated through :meth:`Actuator.extensions`. ``validate_actuator`` in the manifest
enforces the contract at session start.
Methods for actions a backend does not implement should simply not exist (or raise
``NotImplementedError``); callers consult ``capabilities()`` first.
"""
from typing import Any, Protocol, runtime_checkable
from artemis.mcp.action_manifest import ExtensionTool
from artemis.mcp.action_types import ActionResult
__all__ = ["Actuator"]
@runtime_checkable
class Actuator(Protocol):
"""Protocol every actuator backend implements.
All coordinates are 0-1000 normalized. Every action coroutine returns an
:class:`ActionResult` and must not raise for device-level failures -- those are
reported through ``ok=False`` so a failed tap reaches the agent as an observation
rather than an exception.
"""
def capabilities(self) -> frozenset[str]:
"""Names from ``DEVICE_ACTIONS`` this backend implements."""
...
def extensions(self) -> list[ExtensionTool]:
"""Backend-specific tools unknown to the manifest."""
...
# --- Required action -------------------------------------------------------------
async def click_sequence(
self, points: list[tuple[int, int]], delay_ms: int = 50
) -> ActionResult: ...
# --- Optional actions ------------------------------------------------------------
async def click(
self, nx: int, ny: int, times: int = 1, delay_ms: int = 100
) -> ActionResult: ...
async def long_press(self, nx: int, ny: int, duration_ms: int = 1000) -> ActionResult: ...
async def input_text(
self,
text: str,
target: tuple[int, int] | None = None,
clear_exist: bool = True,
) -> ActionResult: ...
async def swipe(
self,
start: tuple[int, int],
end: tuple[int, int],
duration_ms: int = 800,
) -> ActionResult: ...
async def press_key(self, key: str) -> ActionResult: ...
async def manage_app(self, action: str, app_name: str) -> ActionResult: ...
async def wait_for_delay(self, time_in_ms: int) -> ActionResult: ...
async def wait_for_text(
self, text: str, wait_state: str | None = None, timeout_ms: int | None = None
) -> ActionResult: ...
async def open_link(self, url: str) -> ActionResult: ...
async def erase_one_char(self) -> ActionResult: ...
async def focus_and_clear_text(self, nx: int, ny: int) -> ActionResult: ...
# --- Internal observation primitives ---------------------------------------------
async def take_screenshot(self) -> str:
"""Returns the current screen as a base64-encoded JPEG/PNG string."""
...
async def get_ui_elements(self) -> Any:
"""Returns the raw UI hierarchy elements as provided by the device."""
...
async def get_screen_data(self) -> Any:
"""Returns combined screen data (screenshot + elements + dimensions)."""
...
+70
View File
@@ -0,0 +1,70 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Mock actuator for tests: real actuator code paths over an in-memory device.
Subclasses :class:`AdbActuator` deliberately, so tests exercise the same coordinate
conversion and controller dispatch as production -- only the driver is in-memory.
``capabilities``/``extensions`` are overridable to model partial and extended
backends.
"""
from artemis.context import ArtemisContext, DeviceContext, DevicePlatform
from artemis.drivers.mock.mock_driver import MockDeviceDriver
from artemis.mcp.action_manifest import DEVICE_ACTIONS, ExtensionTool
from artemis.mcp.actuators.adb import AdbActuator
from artemis.platform import platform
__all__ = ["MockActuator"]
class MockActuator(AdbActuator):
"""An :class:`AdbActuator` bound to a :class:`MockDeviceDriver`."""
def __init__(
self,
width: int = 1080,
height: int = 2400,
capabilities: frozenset[str] | None = None,
extensions: list[ExtensionTool] | None = None,
):
driver = MockDeviceDriver(width=width, height=height)
ctx = ArtemisContext(
trace_id="mock-actuator",
device=DeviceContext(
host_platform=platform.os_type.name,
mobile_platform=DevicePlatform.ANDROID,
device_id=driver.device_id,
device_width=width,
device_height=height,
),
)
# get_driver(ctx) returns a cached `_active_driver` when present
# (artemis/drivers/factory.py), the established test seam.
ctx._active_driver = driver
super().__init__(ctx)
self.driver = driver
self._capabilities = capabilities if capabilities is not None else DEVICE_ACTIONS
self._extensions = list(extensions or [])
def capabilities(self) -> frozenset[str]:
return self._capabilities
def extensions(self) -> list[ExtensionTool]:
return self._extensions
@property
def action_history(self) -> list[dict]:
"""The mock driver's recorded actions, for assertions."""
return self.driver.action_history
+46 -72
View File
@@ -17,7 +17,6 @@ import json
import logging import logging
import os import os
from pathlib import Path from pathlib import Path
import re
import sys import sys
from typing import Any from typing import Any
@@ -43,33 +42,46 @@ from artemis.controllers.unified_controller import UnifiedMobileController
from artemis.platform import platform from artemis.platform import platform
from artemis.utils.app_launch_utils import launch_app_with_retries from artemis.utils.app_launch_utils import launch_app_with_retries
# Redirect stdio to prevent logs from breaking MCP JSON-RPC protocol and causing deadlocks.
try:
from artemis.config import settings
log_path = Path(settings.TRACES_PATH) / "mcp_server.log"
except Exception:
log_path = Path("traces") / "mcp_server.log"
log_path.parent.mkdir(parents=True, exist_ok=True)
for name in ["artemis", __name__]:
log_instance = logging.getLogger(name)
log_instance.setLevel(logging.DEBUG)
log_instance.propagate = False
# Remove standard streams handlers
log_instance.handlers = [
h for h in log_instance.handlers if not isinstance(h, logging.StreamHandler)
]
fh = logging.FileHandler(log_path, encoding="utf-8")
fh.setFormatter(logging.Formatter("%(asctime)s | %(name)s | %(levelname)s | %(message)s"))
log_instance.addHandler(fh)
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
logger.info("MCP Server Python process fully started. Configuring FastMCP service...")
# Avoid concurrent SQLite WAL locks by blocking child processes from connecting to primary DataEngine.
os.environ["ARTEMIS_IPC_PORT"] = "" def configure_stdio_mode() -> None:
"""Applies process-wide settings required when serving MCP over stdio.
These are destructive to the host process (console logging is removed and the
DataEngine IPC port is cleared), so they must only run when this module owns the
process -- i.e. from the ``__main__`` block. Importing this module for its tools
or for :func:`_get_controller` must leave the caller's logging and environment
untouched.
"""
# Redirect stdio to prevent logs from breaking MCP JSON-RPC protocol and causing deadlocks.
try:
from artemis.config import settings
log_path = Path(settings.TRACES_PATH) / "mcp_server.log"
except Exception:
log_path = Path("traces") / "mcp_server.log"
log_path.parent.mkdir(parents=True, exist_ok=True)
for name in ["artemis", __name__]:
log_instance = logging.getLogger(name)
log_instance.setLevel(logging.DEBUG)
log_instance.propagate = False
# Remove standard streams handlers
log_instance.handlers = [
h for h in log_instance.handlers if not isinstance(h, logging.StreamHandler)
]
fh = logging.FileHandler(log_path, encoding="utf-8")
fh.setFormatter(logging.Formatter("%(asctime)s | %(name)s | %(levelname)s | %(message)s"))
log_instance.addHandler(fh)
logger.info("MCP Server Python process fully started. Configuring FastMCP service...")
# Avoid concurrent SQLite WAL locks by blocking child processes from connecting
# to the primary DataEngine.
os.environ["ARTEMIS_IPC_PORT"] = ""
# Create minimal MCP server # Create minimal MCP server
mcp = FastMCP("Android_ADB_Controller") mcp = FastMCP("Android_ADB_Controller")
@@ -161,52 +173,13 @@ def _get_controller(device_serial: str | None = None):
return controller return controller
def _find_element_at_coords(elements: list[dict], x: int, y: int) -> dict | None: # Shared with the in-process actuator layer; this stdio server keeps its legacy
"""Finds the smallest (leaf-most) focusable element containing [x, y].""" # pixel-space contract for external MCP clients, but the element/focus logic has a
matching_element = None # single implementation in artemis.mcp.actuators.adb.
min_area = float("inf") from artemis.mcp.actuators.adb import ( # noqa: E402 pylint: disable=wrong-import-position
ensure_focus_at_coords as _ensure_focus_at_coords,
for elem in elements: find_element_at_coords as _find_element_at_coords,
is_focusable = ( )
elem.get("focusable") == "true"
or elem.get("clickable") == "true"
or "EditText" in str(elem.get("class", ""))
)
if not is_focusable:
continue
bounds_str = elem.get("bounds")
if bounds_str and isinstance(bounds_str, str):
match = re.match(r"\[(\-?\d+),(\-?\d+)\]\[(\-?\d+),(\-?\d+)\]", bounds_str)
if match:
x1, y1, x2, y2 = map(int, match.groups())
if x1 <= x <= x2 and y1 <= y <= y2:
area = (x2 - x1) * (y2 - y1)
if area < min_area:
min_area = area
matching_element = elem
return matching_element
async def _ensure_focus_at_coords(controller, x: int, y: int) -> str | None:
"""Ensures the element at [x, y] is focused, tapping only if it is not already focused."""
try:
elements = await controller.get_ui_elements()
elem = _find_element_at_coords(elements, x, y)
if elem and elem.get("focused") == "true":
logger.info(f"Element under [{x}, {y}] is already focused. Skipping tap.")
return None
except Exception as e:
logger.warning(f"Failed to check focus status: {e}. Falling back to unconditional tap.")
result = await controller.tap_at(x=x, y=y)
if hasattr(result, "error") and result.error:
return result.error
# Wait for the UI to settle and keyboard to pop up after tapping
await asyncio.sleep(1.0)
return None
@mcp.tool() @mcp.tool()
@@ -480,6 +453,7 @@ async def get_ui_hierarchy(ctx: Context) -> str:
if __name__ == "__main__": if __name__ == "__main__":
from artemis.runtime import shutdown_awake_service, start_awake_service from artemis.runtime import shutdown_awake_service, start_awake_service
configure_stdio_mode()
start_awake_service() start_awake_service()
try: try:
mcp.run(transport="stdio") mcp.run(transport="stdio")
+133
View File
@@ -0,0 +1,133 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Screen observation: capture, fuse, index, and persist -- without LangGraph State.
This is the act-then-observe half that used to be welded into every executor action
(``capture_screenshot_and_parse_ui``). Split out so that actuation and observation are
separate calls: the Flash/FailureAnalyzer adapter observes after each action, while the
Validator -- which runs its own UI-change polling -- simply never calls it.
"""
import asyncio
import base64
import time
from artemis.config import get_temp_dir
from artemis.context import ArtemisContext
from artemis.controllers.unified_controller import UnifiedMobileController
from artemis.mcp.action_types import ActionCode, ObserveResult
from artemis.utils.logger import get_logger
from artemis.utils.ocr_xml_fusion import fuse_ocr_with_xml
from artemis.utils.visualization import format_minimal_list_with_elements
logger = get_logger(__name__)
__all__ = ["observe"]
async def observe(
ctx: ArtemisContext,
controller: UnifiedMobileController,
settle_ms: int = 400,
ocr_results: list | None = None,
) -> tuple[ObserveResult, bytes | None]:
"""Captures and indexes the current screen.
Returns the structured observation plus the raw screenshot bytes (an in-process
convenience; the serialized ``ObserveResult`` carries only the persisted path).
Mirrors the historical ``capture_screenshot_and_parse_ui`` behavior exactly:
a settling delay, screen capture, XML fusion and element indexing, persistence via
the data engine with a temp-file fallback -- and the same failure semantics: a
failed capture yields ``ok=False``; a failed hierarchy parse still returns the
screenshot with ``hierarchy_ok=False``.
"""
try:
if settle_ms > 0:
logger.info(f"Delaying {settle_ms / 1000:.1f}s before capturing screen state...")
await asyncio.sleep(settle_ms / 1000.0)
else:
logger.info("Retrieving screen data directly (skip settling)...")
device_data = await controller.get_screen_data()
screenshot_bytes = base64.b64decode(device_data.base64)
except Exception as e:
logger.error(f"Failed to capture screen state: {e}")
return (
ObserveResult(
ok=False,
code=ActionCode.DEVICE_ERROR,
message=f"Failed to capture screen state: {e}",
hierarchy_ok=False,
),
None,
)
xml_hierarchy = None
ocr_results = ocr_results or []
elements: list[dict] = []
hierarchy_ok = True
width = height = None
try:
xml_hierarchy = device_data.elements
width = device_data.width or 1080
height = device_data.height or 2400
if ctx.device:
ctx.device.device_width = width
ctx.device.device_height = height
fused_xml = fuse_ocr_with_xml(xml_hierarchy, ocr_results)
minimal_list_str, elements, _ = format_minimal_list_with_elements(fused_xml, width, height)
except Exception as e:
logger.error(f"Failed to fetch or fuse UI hierarchy: {e}")
minimal_list_str = "Not available due to hierarchy error."
hierarchy_ok = False
screenshot_path = None
if ctx.data_engine:
try:
image_name = ctx.data_engine.get_or_create_image(
screenshot_bytes,
ui_tree=xml_hierarchy,
ocr_result=ocr_results,
)
screenshot_path = str(ctx.data_engine.get_image_path(image_name))
except Exception as e:
logger.error(f"Failed to save image in DataEngine: {e}")
if not screenshot_path:
try:
temp_dir = get_temp_dir("screenshots")
temp_file = temp_dir / f"screenshot_{int(time.time())}.jpg"
temp_file.write_bytes(screenshot_bytes)
screenshot_path = str(temp_file)
except Exception as e:
logger.error(f"Failed to save fallback screenshot: {e}")
from pathlib import Path
return (
ObserveResult(
ok=True,
message="Screen observed.",
screenshot_path=screenshot_path,
image_name=Path(screenshot_path).stem if screenshot_path else None,
width=width,
height=height,
elements_text=minimal_list_str,
elements=elements,
hierarchy_ok=hierarchy_ok,
),
screenshot_bytes,
)
+5 -1
View File
@@ -366,7 +366,11 @@ if __name__ == "__main__":
parser.add_argument("--trace-id", required=True, help="Unique trace identifier") parser.add_argument("--trace-id", required=True, help="Unique trace identifier")
parser.add_argument("--task-desc", required=True, help="Description of the task to run") parser.add_argument("--task-desc", required=True, help="Description of the task to run")
parser.add_argument("--model", required=True, help="Model to use ('Flash' or 'Pro')") parser.add_argument("--model", required=True, help="Model to use ('Flash' or 'Pro')")
parser.add_argument("--conversation-id", required=True, help="Active conversation ID") parser.add_argument(
"--conversation-id",
default="",
help="Conversation ID for wakeup notification routing (optional; empty disables routing)",
)
parser.add_argument("--locked-app-package", help="Package name of app to lock execution to") parser.add_argument("--locked-app-package", help="Package name of app to lock execution to")
parser.add_argument("--app-path", help="Path to local APK to install before running task") parser.add_argument("--app-path", help="Path to local APK to install before running task")
parser.add_argument("--expected-output-desc", help="Expected output description") parser.add_argument("--expected-output-desc", help="Expected output description")
+4 -2
View File
@@ -35,5 +35,7 @@ mcp = FastMCP(
), ),
) )
# Import tools so they are registered with the mcp instance upon module load # Tool registration is intentionally NOT done here: mcp_server/__init__.py
import mcp_server.tools # noqa: E402, F401 # imports mcp_server.tools (which imports this module), so importing this
# module from anywhere already registers every tool. Importing tools here as
# well would create a circular import (base -> tools -> base).
+27 -2
View File
@@ -24,24 +24,49 @@ if PROJECT_ROOT not in sys.path:
sys.path.insert(0, PROJECT_ROOT) sys.path.insert(0, PROJECT_ROOT)
# Debug logging and stderr tee to capture crashes without breaking MCP stdio JSON-RPC # Debug logging and stderr tee to capture crashes without breaking MCP stdio JSON-RPC
_LOG_MAX_BYTES = 5 * 1024 * 1024
def _rotate_log(path: str, max_bytes: int = _LOG_MAX_BYTES) -> None:
"""Rotates the log to <path>.old once it exceeds max_bytes, so long-lived
server processes cannot grow the debug logs without bound."""
try:
if os.path.exists(path) and os.path.getsize(path) > max_bytes:
old_path = path + ".old"
if os.path.exists(old_path):
os.remove(old_path)
os.replace(path, old_path)
except Exception:
pass
try: try:
_mcp_log_dir = os.path.join(PROJECT_ROOT, "scratch") _mcp_log_dir = os.path.join(PROJECT_ROOT, "scratch")
os.makedirs(_mcp_log_dir, exist_ok=True) os.makedirs(_mcp_log_dir, exist_ok=True)
with open(os.path.join(_mcp_log_dir, "mcp_launch_debug.log"), "a", encoding="utf-8") as _f: _launch_log = os.path.join(_mcp_log_dir, "mcp_launch_debug.log")
_rotate_log(_launch_log)
with open(_launch_log, "a", encoding="utf-8") as _f:
_f.write( _f.write(
f"[{datetime.datetime.now()}] Artemis MCP Server launched! sys.argv: {sys.argv}, CWD: {os.getcwd()}\n" f"[{datetime.datetime.now()}] Artemis MCP Server launched! sys.argv: {sys.argv}, CWD: {os.getcwd()}\n"
) )
class StderrTee: class StderrTee:
def __init__(self, original, log_path): def __init__(self, original, log_path, max_bytes=_LOG_MAX_BYTES):
self.original = original self.original = original
self.log_path = log_path
self.max_bytes = max_bytes
_rotate_log(log_path, max_bytes)
self.log_file = open(log_path, "a", encoding="utf-8") self.log_file = open(log_path, "a", encoding="utf-8")
def write(self, data): def write(self, data):
self.original.write(data) self.original.write(data)
self.log_file.write(data) self.log_file.write(data)
self.log_file.flush() self.log_file.flush()
if self.log_file.tell() > self.max_bytes:
self.log_file.close()
_rotate_log(self.log_path, self.max_bytes)
self.log_file = open(self.log_path, "a", encoding="utf-8")
def flush(self): def flush(self):
self.original.flush() self.original.flush()
+12 -23
View File
@@ -34,31 +34,20 @@ from artemis.utils.visualization import format_minimal_list_with_elements
async def mobile_get_device_state(view_type: str, device_serial: str | None = None) -> str: async def mobile_get_device_state(view_type: str, device_serial: str | None = None) -> str:
"""Real-time mobile device state observer (for debugging and validation). """Real-time mobile device state observer (for debugging and validation).
Use this tool to inspect the latest mobile status or to assist in tracking Retrieves a real-time screenshot or a simplified UI element tree from the
the subagent's task progress. target device — useful for inspecting device status or tracking a
subagent's progress.
Retrieves a real-time screenshot (returns local file URI) or a simplified
element tree text list of the target mobile device.
### Device Selection Modes:
1. **Direct Device Specification**: Provide `device_serial` to observe a specific
phone or emulator (e.g. `device_serial="63191FDKX00062"`).
2. **Automatic Device Selection**: Omit `device_serial` or leave it `None` to
automatically inspect the default/connected device.
**Diagnostics & User Preference**:
- When multiple devices are attached, prioritize letting the user specify or confirm the device.
- Run `adb devices` to check connected devices and identify their serial numbers.
Args: Args:
view_type: The observation type. Supported values: view_type: Observation type:
- "screenshot": Captures the real-time mobile screen and saves it to the workspace, - "screenshot": captures the screen, saves it to the workspace, and
returning the local URI path of the image file (e.g., "file:///path/to/screenshot.jpg"). returns the image's local file URI.
- "hierarchy": Directly returns the simplified text-labeled element list, which is - "hierarchy": returns the simplified text-labeled element list —
identical to what the mobile UI automation subagent sees when making decisions. exactly what the automation subagent sees when making decisions.
device_serial: Optional Android device serial number (e.g., "63191FDKX00062" or device_serial: Optional device serial (e.g. "emulator-5554") to inspect
"emulator-5554"). If specified, inspects state from that specific device. If omitted, a specific device; omitted → the default connected device. With
inspects the default connected device. several devices attached, confirm the target with the user
(`adb devices -l` lists serials).
""" """
try: try:
controller = _get_controller(device_serial=device_serial) controller = _get_controller(device_serial=device_serial)
+20 -44
View File
@@ -74,54 +74,30 @@ def _draw_action_overlay(bg_image_path: str, action_obj: dict, output_path: str)
@mcp.tool() @mcp.tool()
async def mobile_inspect_trace(action: str, trace_id: str, step_number: int | None = None) -> Any: async def mobile_inspect_trace(action: str, trace_id: str, step_number: int | None = None) -> Any:
"""A specialized trace data retrieval tool designed to extract execution details of mobile automation tasks. """Retrieves execution details of a mobile automation task (running or finished).
Use this tool DURING a running task or AFTER it has finished (or failed) to Use it to monitor progress, verify answers, and diagnose agent errors for
monitor progress, verify answers, diagnose agent errors, and debug. both Flash and Pro tasks. Every result includes the assigned `device_serial`.
It provides comprehensive runtime information (including the assigned
`device_serial`), raw agent outputs, and visual screenshots for both
**Flash** and **Pro** tasks.
### Available Actions: ### Actions
- **'view_summary'**: Retrieves a high-level execution summary across all - **'view_summary'**: High-level execution summary across all steps.
steps, including the target `device_serial`. Pro: hierarchical task plan with per-step status; Flash: full execution
* **Pro Model**: Displays a hierarchical task plan summary, showing the chain with each step's reasoning and action.
status, device serial, and descriptions of all completed, active, and pending steps. - **'view_step_screenshots'**: Local file paths of one step's screenshots:
* **Flash Model**: Displays an all-in-one execution chain containing the `before_screenshot` (what the agent saw), `after_screenshot` (only set
complete reasoning (thoughts/motivation), device serial, and action taken (`click`, when the action failed/was intercepted, else null), and
`input_text`, etc.) for every single step. `action_overlay_screenshot` (the action visually marked — e.g. red circle
for taps — key for verifying the agent tapped the right element).
- **'view_step_screenshots'**: Retrieves local file paths of screenshots for - **'view_step_details'**: Deep per-step diagnostics (raw VLM thoughts,
a specific step (supports both Flash and Pro), along with `device_serial`. Returns: exact actions). Pro only — DO NOT use for Flash tasks ('view_summary'
1. `before_screenshot`: The original device screenshot observed by the already carries their full execution chain).
agent right before taking the action.
2. `after_screenshot`: A post-action screenshot capturing the failure or
mismatch state (if the action failed or was intercepted). Note: This is
conditional/optional and will be `None` (`null`) for normal/successful
steps where no extra post-action screenshot was captured.
3. `action_overlay_screenshot`: The `before_screenshot` with the agent's
action visually marked (e.g., red circle for taps, red arrow for
swipes). CRITICAL for verifying if the agent tapped the correct UI
element.
4. `device_serial`: The serial of the device on which this step was executed.
- **'view_step_details'**: Retrieves deep runtime logs, raw VLM thoughts,
and exact actions for a specific step, along with `device_serial`.
* **Pro Model**: Fully supported. Returns detailed per-step diagnostic
data.
* **Flash Model**: DO NOT USE. (Flash reasoning and actions are already
fully included in 'view_summary', making this action redundant).
Args: Args:
action: STR. **REQUIRED**. The retrieval action to perform action: `"view_summary"`, `"view_step_screenshots"`, or
(`"view_summary"`, `"view_step_screenshots"`, or `"view_step_details"`.
`"view_step_details"`). trace_id: The task's session identifier from `mobile_run_task`.
trace_id: STR. **REQUIRED**. The unique session identifier of the task step_number: 1-indexed step to query; required for the two per-step
(returned by `mobile_run_task`). actions, omit for `view_summary`.
step_number: INT. **CONDITIONAL**. The 1-indexed step number to query. -
You MUST provide this if `action="view_step_screenshots"` or
`action="view_step_details"`. - You MUST omit or leave this empty if
`action="view_summary"`.
""" """
project_root = env_utils.get_project_root() project_root = env_utils.get_project_root()
db_path = os.path.join(trace_store.TRACES_DIR, "data_engine.db") db_path = os.path.join(trace_store.TRACES_DIR, "data_engine.db")
+178 -100
View File
@@ -27,6 +27,161 @@ from mcp_server.notifiers import notify
from mcp_server.utils import env_utils, trace_store from mcp_server.utils import env_utils, trace_store
from artemis.runtime import DeviceExecutionLock, process_supervisor from artemis.runtime import DeviceExecutionLock, process_supervisor
_LIVENESS_FAILURE_ERROR = "Task runner process terminated unexpectedly."
# Grace window covering the spawn race: the launcher pid may already have exited
# while the real runner has not yet registered its own pid in the DataEngine DB.
_STARTUP_GRACE_SECONDS = 45.0
def _find_data_engine_db() -> str | None:
db_path = os.path.join(trace_store.TRACES_DIR, "data_engine.db")
if os.path.exists(db_path):
return db_path
db_path = os.path.join(env_utils.get_project_root(), "traces", "data_engine.db")
return db_path if os.path.exists(db_path) else None
def _pid_alive(pid: int | None) -> bool:
if not pid:
return False
try:
import psutil
proc = psutil.Process(int(pid))
return proc.is_running() and proc.status() != psutil.STATUS_ZOMBIE
except Exception:
try:
os.kill(int(pid), 0)
return True
except OSError:
return False
def _session_tracked_by_lock(trace_id: str) -> bool:
"""True when the device-lock layer still tracks this session (queued or active owner)."""
try:
for q_item in DeviceExecutionLock.get_queued_tasks():
if str(q_item.get("session_id")) == trace_id:
return True
except Exception:
pass
try:
for owner in DeviceExecutionLock.get_active_owners().values():
if owner.session_id and str(owner.session_id) == trace_id:
return True
except Exception:
pass
return False
def _reconcile_task_state(
trace_id: str, status_data: dict[str, Any]
) -> tuple[str, int | None, bool]:
"""Reconcile status.json with the DataEngine DB and real process liveness.
The DB session row is written by the runner process itself (``os.getpid()``),
so its pid and terminal status outrank the pid recorded at spawn time -- on
Windows that spawn pid is a short-lived venv launcher, not the runner. A dead
or missing pid alone never fails a task that the DB, the device-lock queue,
or an active lock owner still reports as alive.
Returns ``(current_status, pid, is_alive)`` and persists any correction back
to status.json (including recovering from a previously misreported failure).
"""
current_status = status_data.get("status", "unknown")
pid = status_data.get("pid")
dirty = False
db_status: str | None = None
db_pid: int | None = None
db_path = _find_data_engine_db()
if db_path:
try:
conn = sqlite3.connect(db_path)
conn.row_factory = sqlite3.Row
row = conn.execute(
"SELECT status, pid FROM sessions WHERE session_id = ? ORDER BY start_time DESC LIMIT 1",
(trace_id,),
).fetchone()
conn.close()
if row:
db_status = row["status"]
db_pid = row["pid"]
except Exception:
pass
if db_pid and db_pid != pid:
pid = db_pid
status_data["pid"] = pid
dirty = True
# The DB terminal verdict only fills in a status.json that has not reached a
# terminal state itself, or corrects a liveness-inferred failure. It never
# overrides a completed result or an explicit user cancellation.
liveness_failed = (
current_status == "failed" and status_data.get("error") == _LIVENESS_FAILURE_ERROR
)
if db_status in ("completed", "success", "failed", "cancelled") and (
current_status in ("running", "pending") or liveness_failed
):
canonical = "completed" if db_status in ("completed", "success") else db_status
if current_status != canonical:
current_status = canonical
status_data["status"] = canonical
if canonical != "failed" and status_data.get("error") == _LIVENESS_FAILURE_ERROR:
status_data["error"] = None
if not status_data.get("end_time"):
status_data["end_time"] = time.time()
dirty = True
is_alive = False
elif current_status in ("running", "pending"):
if not pid:
# No pid recorded (daemon dispatch never writes one): liveness cannot be
# inferred, so the task is assumed alive -- matching the historical
# behavior of only checking liveness when a pid exists.
is_alive = True
else:
is_alive = _pid_alive(pid)
if not is_alive:
if _session_tracked_by_lock(trace_id):
# Still queued for a device or owned by a live lock holder.
is_alive = True
elif db_status in ("running", "pending") and db_pid:
# The runner registered itself and its process is gone: truly dead.
current_status = "failed"
_mark_liveness_failure(trace_id, status_data)
elif (time.time() - (status_data.get("start_time") or 0)) < _STARTUP_GRACE_SECONDS:
is_alive = True
else:
current_status = "failed"
_mark_liveness_failure(trace_id, status_data)
else:
is_alive = False
if dirty:
trace_store.write_status(trace_id, status_data)
return current_status, pid, is_alive
def _mark_liveness_failure(trace_id: str, status_data: dict[str, Any]) -> None:
trace_store.update_trace_status(trace_id, "failed", error=_LIVENESS_FAILURE_ERROR)
status_data["status"] = "failed"
status_data["error"] = _LIVENESS_FAILURE_ERROR
conv_id = status_data.get("conversation_id")
if conv_id:
try:
notify(
conversation_id=conv_id,
message=(
f"Artemis background task died unexpectedly for trace '{trace_id}'. "
f"Error: {_LIVENESS_FAILURE_ERROR}"
),
event_type="failed",
payload={"trace_id": trace_id, "error": _LIVENESS_FAILURE_ERROR},
)
except Exception:
pass
@mcp.tool() @mcp.tool()
def mobile_manage_task( def mobile_manage_task(
@@ -34,53 +189,31 @@ def mobile_manage_task(
) -> dict[str, Any]: ) -> dict[str, Any]:
"""Manages the lifecycle and retrieves the status of a background mobile automation task. """Manages the lifecycle and retrieves the status of a background mobile automation task.
This is your primary diagnostic and control tool. You MUST use this tool to This is your primary diagnostic and control tool for tasks started by
poll the task status `mobile_run_task`. You MUST use it to poll the task status when your
when your 1-minute fallback timer triggers (as requested by 1-minute fallback timer triggers (as required by `mobile_run_task`). You
`mobile_run_task`). You can also use can also steer a subagent that is stuck or off-track, or abort a task.
it to gracefully abort a task or forcefully correct the subagent's behavior
mid-flight.
### Available Actions: ### Actions
- **'status'**: Retrieves execution progress, elapsed time, and metadata. - **'status'**: Returns `trace_id`, `status` ('running'/'completed'/'failed'/
Returns a JSON object containing: 'cancelled'), `device_serial` (which phone owns this task in multi-device
* `trace_id`: Unique identifier for the task. setups), `task_desc`, `model`, `elapsed_seconds`, and `progress` (Flash:
* `status`: Task lifecycle state ('running', 'completed', 'failed', 'cancelled'). current turn, latest thought/action — intervene if turns climb without
* `device_serial`: The Android device serial number assigned to/running this task progress or the thought indicates it is stuck; Pro: the active task plan
(essential in multi-device environments to know which phone this task belongs to). — check it still aligns with your goal).
* `task_desc`: The goal/description of the task. - **'inject_instruction'**: Injects real-time guidance mid-flight when the
* `model`: 'Flash' or 'Pro'. subagent errs, stalls, or loops. Applied at the start of the next
* `elapsed_seconds`: Execution duration in seconds. planning turn (Pro) or the next reactive loop (Flash). Requires
* `progress`: Live progress data (current turn, latest thoughts and actions for Flash; `instruction`.
active task plan for Pro). - **'stop'**: Forcefully terminates the subagent, immediately halting
* **How to use the output**: device interactions and releasing the device. Use when the task is done,
- For **Pro Model**, check `progress.task_plan` to see if the irreparably broken, or running out of control.
agent's long-term plan aligns with your goal.
- For **Flash Model**, monitor `current_turn`, `latest_action`, and
`latest_thought`. If the turn count increases rapidly without
progressing, or the thought indicates it is stuck, you should
intervene.
- **'inject_instruction'**: Injects real-time guidance when the subagent
makes a mistake, gets stuck, or loops.
* **Pro Model**: Applied at the start of the next turn's planning phase.
* **Flash Model**: Applied at the start of the next reactive execution
loop.
* *Note*: You MUST provide the `instruction` argument for this action.
- **'stop'**: Forcefully terminates the subagent (Pro or Flash), immediately
halting device interactions and releasing the device. Use this if the task
is complete, irreparably broken, or running out of control.
Args: Args:
action: STR. **REQUIRED**. The management action to perform. Must be action: `"status"`, `"inject_instruction"`, or `"stop"`.
exactly `"status"`, `"stop"`, or `"inject_instruction"`. trace_id: The task's session identifier from `mobile_run_task`.
trace_id: STR. **REQUIRED**. The unique session identifier of the task instruction: Guidance string; required for `inject_instruction`, omit
(returned by `mobile_run_task`). otherwise.
instruction: STR. **CONDITIONAL**. The real-time correction or guidance
string. - You MUST provide this if `action="inject_instruction"`. -
You MUST leave this empty or omit it if `action="status"` or
`action="stop"`.
""" """
status_data = trace_store.read_status(trace_id) status_data = trace_store.read_status(trace_id)
if not status_data: if not status_data:
@@ -90,62 +223,7 @@ def mobile_manage_task(
"message": f"Trace ID '{trace_id}' not found.", "message": f"Trace ID '{trace_id}' not found.",
} }
current_status = status_data.get("status", "unknown") current_status, pid, is_alive = _reconcile_task_state(trace_id, status_data)
pid = status_data.get("pid")
is_alive = False
project_root = env_utils.get_project_root()
db_path = os.path.join(trace_store.TRACES_DIR, "data_engine.db")
if not os.path.exists(db_path):
db_path = os.path.join(project_root, "traces", "data_engine.db")
# If status is still running or pending, sync from database if available
if current_status in ("running", "pending") and os.path.exists(db_path):
try:
conn_st = sqlite3.connect(db_path)
conn_st.row_factory = sqlite3.Row
cur_st = conn_st.cursor()
cur_st.execute(
"SELECT status, pid FROM sessions WHERE session_id = ? ORDER BY start_time DESC LIMIT 1",
(trace_id,),
)
row_st = cur_st.fetchone()
if row_st:
db_status = row_st["status"]
db_pid = row_st["pid"]
if db_pid and not pid:
pid = db_pid
status_data["pid"] = pid
trace_store.write_status(trace_id, status_data)
if db_status in ("completed", "success", "failed", "cancelled"):
canonical_status = "completed" if db_status in ("completed", "success") else db_status
current_status = canonical_status
status_data["status"] = canonical_status
trace_store.write_status(trace_id, status_data)
conn_st.close()
except Exception:
pass
if pid and current_status in ("running", "pending"):
try:
os.kill(pid, 0)
is_alive = True
except OSError:
current_status = "failed"
error_text = "Task runner process terminated unexpectedly."
trace_store.update_trace_status(trace_id, "failed", error=error_text)
conv_id = status_data.get("conversation_id")
if conv_id:
try:
notify(
conversation_id=conv_id,
message=f"Artemis background task died unexpectedly for trace '{trace_id}'. Error: {error_text}",
event_type="failed",
payload={"trace_id": trace_id, "error": error_text},
)
except Exception:
pass
status_data = trace_store.read_status(trace_id) or status_data
trace_dir = trace_store.get_trace_dir(trace_id) trace_dir = trace_store.get_trace_dir(trace_id)
+218 -106
View File
@@ -16,134 +16,222 @@
import os import os
import subprocess import subprocess
import sys
import threading
import time
from typing import Any from typing import Any
import uuid import uuid
from mcp_server.base import mcp from mcp_server.base import mcp
from mcp_server.notifiers import notify
from mcp_server.utils import env_utils, trace_store from mcp_server.utils import env_utils, trace_store
from artemis.runtime import DeviceExecutionLock from artemis.runtime import DeviceExecutionLock
# Seconds the spawned runner gets to finish its imports and open its log files
# before the spawn is declared dead. Normal startup creates stdout.log within a
# few seconds; a runner that produced no logs by this deadline is hung in
# interpreter/DLL startup and will never recover (observed on Windows when the
# MCP server process context degrades).
SPAWN_WATCHDOG_SECONDS = 60
def _kill_process_tree(pid: int) -> None:
"""Terminate a spawned runner and its children (the venv shim re-execs python)."""
if sys.platform == "win32":
subprocess.run(
["taskkill", "/PID", str(pid), "/T", "/F"],
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
check=False,
)
else:
try:
os.kill(pid, 9)
except OSError:
pass
def _watch_spawn(
trace_id: str,
pid: int,
queue_ticket: str,
conversation_id: str | None,
deadline_seconds: float = SPAWN_WATCHDOG_SECONDS,
poll_interval: float = 2.0,
) -> bool:
"""Watch a freshly spawned runner; kill and fail the task if it never boots.
The runner opens stdout.log/stderr.log immediately after module imports, so
their absence after the deadline means the process is wedged before running
any task code. Without this, a wedged spawn leaves the task in 'running'
forever and holds its device-queue ticket.
Returns True when the runner booted (or was already terminal), False when it
was declared hung and killed.
"""
trace_dir = trace_store.get_trace_dir(trace_id)
log_paths = (
os.path.join(trace_dir, "stdout.log"),
os.path.join(trace_dir, "stderr.log"),
)
deadline = time.monotonic() + deadline_seconds
while time.monotonic() < deadline:
if any(os.path.exists(p) for p in log_paths):
return True # runner booted normally
status_data = trace_store.read_status(trace_id)
if status_data and status_data.get("status") not in ("running", "pending"):
return True # already terminal (e.g. stopped by the user)
time.sleep(poll_interval)
if any(os.path.exists(p) for p in log_paths):
return True
status_data = trace_store.read_status(trace_id)
if status_data and status_data.get("status") not in ("running", "pending"):
return True
error_text = (
f"Background runner (pid {pid}) produced no logs within "
f"{deadline_seconds:.0f}s of spawn and was killed: the process hung "
"during interpreter startup. Restarting the MCP server usually clears "
"this; the task can be resubmitted afterwards."
)
_kill_process_tree(pid)
DeviceExecutionLock.cancel_reservation(queue_ticket)
trace_store.update_trace_status(trace_id, "failed", error=error_text)
if conversation_id:
try:
notify(
conversation_id=conversation_id,
message=f"Artemis task '{trace_id}' failed to start: {error_text}",
event_type="failed",
payload={"trace_id": trace_id, "error": error_text},
)
except Exception:
pass
return False
def _start_spawn_watchdog(
trace_id: str,
pid: int,
queue_ticket: str,
conversation_id: str | None,
) -> None:
"""Run _watch_spawn on a daemon thread so the tool call returns immediately."""
threading.Thread(
target=_watch_spawn,
args=(trace_id, pid, queue_ticket, conversation_id),
name=f"spawn-watchdog-{trace_id[:8]}",
daemon=True,
).start()
def _validate_device_serial(device_serial: str) -> dict[str, Any] | None:
"""Reject a task whose explicitly requested device is not attached and authorized.
Returns a failure response dict when the serial must be rejected, or None when
the device is usable. If device enumeration itself fails (no adb available),
validation is skipped rather than blocking task submission.
"""
try:
from artemis.runtime import device_pool
from artemis.runtime.device_lock import DeviceExecutionLock
norm = DeviceExecutionLock._normalize_device_id
attached = {norm(d.serial): d.state for d in device_pool.list_devices()}
except Exception:
return None
if not attached:
# Enumeration unavailable (no adb / transient failure): let the task
# proceed and fail downstream with a clear no-device error instead.
return None
dev_state = attached.get(norm(device_serial))
if dev_state == "device":
return None
if dev_state is None:
detail = (
f"Device '{device_serial}' is not connected. "
f"Attached devices: {sorted(attached) if attached else 'none'}."
)
else:
detail = f"Device '{device_serial}' is attached but not ready (state: '{dev_state}')."
return {
"status": "failed",
"error": detail,
"message": (
f"{detail} Task rejected to prevent execution on an unintended device. "
"Run `adb devices -l` to inspect attached hardware, then resubmit with a "
"valid serial (or omit device_serial for automatic selection)."
),
}
@mcp.tool() @mcp.tool()
def mobile_run_task( def mobile_run_task(
task_desc: str, task_desc: str,
conversation_id: str, conversation_id: str | None = None,
model: str = "Flash", model: str = "Flash",
locked_app_package: str | None = None, locked_app_package: str | None = None,
app_path: str | None = None, app_path: str | None = None,
expected_output_desc: str | None = None, expected_output_desc: str | None = None,
device_serial: str | None = None, device_serial: str | None = None,
) -> dict[str, Any]: ) -> dict[str, Any]:
"""Starts a specialized mobile UI automation subagent to autonomously plan and execute tasks on a connected Android device. """Starts an autonomous mobile UI automation subagent on a connected Android device.
With this tool, you can delegate complex mobile workflows and UI Delegates a mobile workflow to a background agent. Non-blocking: returns
interactions to a background agent. immediately with `trace_id` (for `mobile_manage_task` / `mobile_inspect_trace`),
This tool supports two execution models: **Flash** (for rapid, `device_serial`, `stdout_log`/`stderr_log` paths, and `notes_dir` (Pro only).
straightforward UI scripts) and **Pro** (for complex, exploratory tasks On completion (success or failure) a Reactive Wakeup notifies you — but only
requiring deep reasoning). when `conversation_id` was provided; without it, rely on your fallback timer.
### Device Selection & Multi-Device Support (CRITICAL) ### Model selection (routing)
ARTEMIS supports multi-device execution and provides two device selection modes: - **Flash** (default, preferred): simple, deterministic tasks completable
1. **Direct Device Specification**: Specify `device_serial` explicitly within ~30 UI steps. No log/video analysis, no ADB shell, no persistent
(e.g., `device_serial="63191FDKX00062"` or `device_serial="emulator-5554"`). plan or notes — unsuitable for exploration, complex error recovery,
Tasks targeting different devices run concurrently via per-device locking. monitoring/polling, or tasks that must report back large amounts of
2. **Automatic Device Selection**: If `device_serial` is omitted or `None`, detail.
ARTEMIS will automatically select the connected device or allocate an idle device - **Pro**: complex, exploratory, or multi-branch tasks; continuous
from the device pool. monitoring/polling; tasks needing ADB shell, system logs, video analysis,
multi-step planning, or detailed written output (via notes and
`expected_output_desc`).
**Device Selection Policy & Diagnostics**: Timing is not precise: the agent's own inference adds ~5 s per step (Flash)
- **Prioritize User Choice**: When multiple devices or emulators are connected (or when or ~30 s per turn (Pro) on top of any requested waits — account for this
user intent is not explicitly specified), **ALWAYS PRIORITIZE ASKING THE USER** to when the task involves waiting. For recurring workflows, run once to
choose or confirm the target device before launching tasks. discover the path, then author a deterministic script instead of
- **Device Diagnosis with `adb devices`**: Always use `adb devices` (or `adb devices -l`) re-delegating.
to inspect connected hardware, verify authorization states (`device` vs `unauthorized`),
and retrieve active device serials when diagnosing or before delegating tasks.
### Model Selection Guide (CRITICAL FOR ROUTING) ### Follow-up protocol (CRITICAL)
You MUST choose the appropriate `model` based on the task complexity: 1. **Fallback timer**: you MUST set a background timer and check the task
- **Flash**: Best for simple, deterministic tasks that can be completed status via `mobile_manage_task` at least once every 1 minute — mobile
within **30 UI steps**. You should prioritize using this model to complete tasks can stall silently, so never rely on the completion wakeup alone.
the task. 2. **Post-task inspection**: after completion, inspect the execution logs
*Limitations*: Flash CANNOT analyze video recordings, query system logs, to verify success (`stderr_log` carries critical failures and Python
execute arbitrary ADB shell commands, maintain a structured task plan, or tracebacks — essential for debugging).
write custom execution notes/reports. Do not use Flash if the task 3. **Pro notes**: for Pro runs, read `notes_dir` to extract the results and
requires exploration, complex error recovery, continuous monitoring/polling plans the subagent recorded.
(due to step count limits and lack of state persistence), or outputting
large amounts of detailed information (as Flash cannot access the
notes/scratchpad system to retain extensive details across steps; use
**Pro** instead).
- **Pro**: Required for complex, non-linear, or exploratory tasks (e.g.,
browsing deep lists, troubleshooting app crashes, tasks requiring multi-step
planning), continuous monitoring or polling tasks (e.g., periodically checking
UI state, polling until specific conditions or events occur, watching progress,
or waiting for background changes), OR when you need the subagent to extract,
remember, and output large amounts of detailed information. Pro is slower but
fully featured.
*Capabilities*: Supports full video analysis, system log querying, ADB
shell access, robust monitoring/polling execution loops with multi-agent
planning, and generating custom text outputs/reports via the expected
output agent and structured notes (`notes_dir`).
Regardless of the model, precise timing for waiting actions cannot be
guaranteed due to latency. If you need to implement a wait, please note that
the total waiting time comprises both the time required for the Artemis large
model to generate output and the actual waiting duration. (For Flash, the
interval between output steps is typically 3-7 seconds, whereas Pro requires
25-35 seconds.)
For long-term repetitive tasks, run the workflow once with this tool to
discover the path, then author a dedicated automation script instead of
repeatedly delegating.
This tool is non-blocking and will immediately return a dictionary
containing:
- `trace_id`: The unique session identifier of the task for status tracking
and inspection.
- `device_serial`: The assigned target device serial (or "auto-select").
- `notes_dir`: (Pro Model Only) The directory where the subagent's execution
files/notes are located.
- `stdout_log`: The path to the execution framework log.
- `stderr_log`: The path to the error log containing critical failures and
Python tracebacks (essential for debugging).
Upon task completion (success or failure), the system will trigger a
Reactive Wakeup to notify you.
CRITICAL BEHAVIORAL RULES:
1. **Fallback Timer Constraint**: You **MUST** set a background timer to
check the task status at least once every 1 minute as a safety fallback.
2. **Post-Task Inspection**: After completion, retrieve and inspect the
execution logs (`stderr_log`/`stdout_log`) to verify success.
3. **Inspecting Notes (Pro Model Only)**: If running with Pro, you can
inspect the notes/plans recorded in `notes_dir` to extract critical
execution results passed back by the subagent.
Args: Args:
task_desc: A clear, highly detailed, actionable task description for the task_desc: Clear, self-contained task description with all context,
mobile subagent. Since execution is autonomous and one-shot, provide exact UI actions, and termination conditions (execution is one-shot).
all necessary context, exact UI actions to take, and clear termination For Pro, may also specify what the subagent must record in its notes.
conditions. If `model="Pro"`, you can explicitly instruct the subagent conversation_id: Optional conversation ID used to route the completion
on what specific information it must record in the notes. wakeup notification back to the caller. Omit if unknown.
conversation_id: Your current active Conversation ID. Critical for model: `"Flash"` or `"Pro"` — see model selection above.
routing the wakeup notification. locked_app_package: Optional package name to lock execution to; the
model: Must be either `"Flash"` or `"Pro"`. - Choose `"Flash"` for fast, agent auto-launches it and restricts actions to that app.
simple UI tasks under 30 steps without log/video analysis or app_path: Optional local APK path to install before running.
monitoring/polling needs. - Choose `"Pro"` for complex tasks requiring expected_output_desc: Optional, Pro only. If set, a summarization agent
planning, ADB shell commands, logs, video exploration, or continuous writes a report to `output.md` in `notes_dir`. Ignored for Flash.
monitoring and polling tasks. device_serial: Optional device serial (e.g. "emulator-5554") to bind
locked_app_package: Optional package name of the app to lock execution execution to a specific device; distinct devices run concurrently.
to. The agent will auto-launch and restrict actions to this app. If omitted, an available device is selected automatically. When
app_path: Optional path to a local APK to install on the device before several devices are attached, confirm the target with the user first
running the task. (`adb devices -l` lists serials and authorization states).
expected_output_desc: Optional (Pro Model Only). If provided, a
summarization agent will aggregate the interaction history
(motivation, actions, videos) into a summary report saved as
"output.md" in `notes_dir`. Ignored if `model="Flash"`.
device_serial: Optional Android device serial number (e.g. "63191FDKX00062"
or "emulator-5554"). If specified, binds execution strictly to that device,
allowing multi-device concurrent automation. If omitted, ARTEMIS will
automatically select an available device. Prioritize letting the user select
the target device if multiple devices are attached.
""" """
# 0. Validate and normalize model # 0. Validate and normalize model
if model.lower() not in ("flash", "pro"): if model.lower() not in ("flash", "pro"):
@@ -162,6 +250,16 @@ def mobile_run_task(
device_serial=device_serial, device_serial=device_serial,
) )
# 2b. Strict device binding: an explicitly requested serial must be attached and
# authorized. Rejecting here prevents the task from silently running on a
# different device than the caller asked for. Runs after init_trace so the
# rejection carries a trace_id like every other response of this tool.
if device_serial:
rejection = _validate_device_serial(device_serial)
if rejection:
trace_store.update_trace_status(trace_id, "failed", error=rejection["error"])
return {"trace_id": trace_id, **rejection}
# 3. Dispatch via unified Artemis Daemon scheduler if available (unless standalone forced) # 3. Dispatch via unified Artemis Daemon scheduler if available (unless standalone forced)
if os.environ.get("ARTEMIS_STANDALONE") != "1": if os.environ.get("ARTEMIS_STANDALONE") != "1":
try: try:
@@ -181,6 +279,15 @@ def mobile_run_task(
conversation_id=conversation_id, conversation_id=conversation_id,
base_url=base_url, base_url=base_url,
) )
if resp and resp.get("status") == "rejected":
rejection_error = resp.get("error") or "Task rejected by Artemis Daemon."
trace_store.update_trace_status(trace_id, "failed", error=rejection_error)
return {
"trace_id": trace_id,
"status": "failed",
"error": rejection_error,
"message": f"Task rejected by Artemis Daemon: {rejection_error}",
}
if resp and resp.get("tasks"): if resp and resp.get("tasks"):
assigned_sid = resp["tasks"][0].get("session_id", trace_id) assigned_sid = resp["tasks"][0].get("session_id", trace_id)
trace_store.update_trace_status( trace_store.update_trace_status(
@@ -275,7 +382,7 @@ def mobile_run_task(
"--model", "--model",
canonical_model, canonical_model,
"--conversation-id", "--conversation-id",
conversation_id, conversation_id or "",
] ]
if locked_app_package: if locked_app_package:
cmd.extend(["--locked-app-package", locked_app_package]) cmd.extend(["--locked-app-package", locked_app_package])
@@ -330,6 +437,11 @@ def mobile_run_task(
status_data["queue_ticket"] = queue_ticket status_data["queue_ticket"] = queue_ticket
trace_store.write_status(trace_id, status_data) trace_store.write_status(trace_id, status_data)
# 6. Arm a watchdog: a runner that never opens its log files is hung in
# interpreter startup and must be killed instead of squatting on the
# queue as 'running' forever.
_start_spawn_watchdog(trace_id, proc.pid, queue_ticket, conversation_id)
trace_dir = trace_store.get_trace_dir(trace_id) trace_dir = trace_store.get_trace_dir(trace_id)
stdout_log_path = os.path.join(trace_dir, "stdout.log") stdout_log_path = os.path.join(trace_dir, "stdout.log")
stderr_log_path = os.path.join(trace_dir, "stderr.log") stderr_log_path = os.path.join(trace_dir, "stderr.log")
+1 -1
View File
@@ -57,7 +57,7 @@ def init_trace(
trace_id: str, trace_id: str,
task_desc: str, task_desc: str,
model: str, model: str,
conversation_id: str, conversation_id: str | None = None,
device_serial: str | None = None, device_serial: str | None = None,
) -> dict[str, Any]: ) -> dict[str, Any]:
"""Initializes the trace directory and creates the initial status.json file.""" """Initializes the trace directory and creates the initial status.json file."""
@@ -667,6 +667,10 @@ async def test_queue_worker_notifies_conversation():
patch("apps.admin_console.services.task_queue_service.media_service"), patch("apps.admin_console.services.task_queue_service.media_service"),
patch("mcp_server.notifiers.notify") as mock_notify, patch("mcp_server.notifiers.notify") as mock_notify,
patch("apps.admin_console.services.task_queue_service.session_repo") as mock_repo, patch("apps.admin_console.services.task_queue_service.session_repo") as mock_repo,
patch(
"artemis.runtime.device_pool.device_pool.list_devices_async",
new=AsyncMock(return_value=[]),
),
): ):
mock_repo.get_running_session_id.return_value = None mock_repo.get_running_session_id.return_value = None
@@ -849,6 +853,11 @@ async def test_enqueue_tasks_debounces_rapid_identical_submissions():
with ( with (
patch("apps.admin_console.services.task_queue_service.session_repo"), patch("apps.admin_console.services.task_queue_service.session_repo"),
patch("apps.admin_console.services.task_queue_service.DeviceExecutionLock.reserve", return_value="ticket-456"), patch("apps.admin_console.services.task_queue_service.DeviceExecutionLock.reserve", return_value="ticket-456"),
patch(
"artemis.runtime.device_pool.device_pool.list_devices_async",
new=AsyncMock(return_value=[]),
),
patch.object(TaskQueueService, "ensure_worker_running"),
): ):
res1 = await task_queue_service.enqueue_tasks(["Rapid duplicate goal"], device_serial="dev-1") res1 = await task_queue_service.enqueue_tasks(["Rapid duplicate goal"], device_serial="dev-1")
assert len(state.queue_items) == 1 assert len(state.queue_items) == 1
@@ -304,9 +304,11 @@ async def test_input_text_happy_path(mock_context, mock_controller):
ctx=mock_context, ctx=mock_context,
) )
assert "Executed typing 'hello'." in outcome assert "Executed typing 'hello'." in outcome
mock_controller.tap_at.assert_called_once_with(540, 1200) # Focus goes through ensure_focus_at_coords (keyword-style tap), and typing
# never re-clears: the clear already happened via erase_text.
mock_controller.tap_at.assert_called_once_with(x=540, y=1200)
mock_controller.erase_text.assert_called_once() mock_controller.erase_text.assert_called_once()
mock_controller.type_text.assert_called_once_with("hello") mock_controller.type_text.assert_called_once_with("hello", clear_existing=False)
@pytest.mark.asyncio @pytest.mark.asyncio
@@ -396,8 +398,10 @@ async def test_press_key_input_validation(mock_context, mock_controller):
controller=mock_controller, controller=mock_controller,
ctx=mock_context, ctx=mock_context,
) )
# The current execution doesn't raise error on unknown keys, just logs error # Unknown keycodes are forwarded verbatim to the driver (historical adb_server
assert "Error executing key press 'INVALID_KEY'." in outcome # behavior); the mock driver accepts them, so the press reports success.
assert "Executed key press 'INVALID_KEY'." in outcome
mock_controller.press_key.assert_called_once_with("INVALID_KEY")
# ========================================== # ==========================================
@@ -409,8 +413,10 @@ async def test_press_key_input_validation(mock_context, mock_controller):
async def test_manage_app_happy_path(mock_context, mock_controller): async def test_manage_app_happy_path(mock_context, mock_controller):
state = DummyState() state = DummyState()
with ( with (
patch("artemis.agents.validator.tool_declarations.find_package") as mock_find_package, # The device-call bodies moved into the actuator layer, which imports these
patch("artemis.agents.validator.tool_declarations.launch_app_with_retries") as mock_launch, # lazily from their source modules -- patch them at the source.
patch("artemis.tools.mobile.launch_app.find_package") as mock_find_package,
patch("artemis.utils.app_launch_utils.launch_app_with_retries") as mock_launch,
): ):
mock_find_package.return_value = "com.google.android.youtube" mock_find_package.return_value = "com.google.android.youtube"
mock_launch.return_value = (True, "") mock_launch.return_value = (True, "")
@@ -430,7 +436,7 @@ async def test_manage_app_happy_path(mock_context, mock_controller):
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_manage_app_input_validation(mock_context, mock_controller): async def test_manage_app_input_validation(mock_context, mock_controller):
state = DummyState() state = DummyState()
with patch("artemis.agents.validator.tool_declarations.find_package") as mock_find_package: with patch("artemis.tools.mobile.launch_app.find_package") as mock_find_package:
mock_find_package.return_value = None mock_find_package.return_value = None
outcome, _, _, _ = await _exec_manage_app( outcome, _, _, _ = await _exec_manage_app(
+233
View File
@@ -0,0 +1,233 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Prompt assembly tests: full-set output is byte-stable, absent tools leave no trace."""
from pathlib import Path
import re
from jinja2 import Template
import pytest
from artemis.agents.operator.prompt_builder import load_operator_prompts
from artemis.agents.operator.prompts import (
OPERATOR_PROMPT_TOOLSET,
apply_operator_prompt_contract,
)
from artemis.agents.prompt_assembly import gate_segment, render_tool_enum
from artemis.agents.validator.failure_analyzer import (
FailureAnalysisStrategy,
FailureAnalyzer,
PixelTargetDisappearedStrategy,
TargetDisappearedStrategy,
)
from artemis.agents.validator.validator import ValidationErrorCategory
REPO_ROOT = Path(__file__).resolve().parents[3]
OPERATOR_DEVICE_TOOLS = (
"click",
"input_text",
"swipe",
"press_key",
"manage_app",
"wait_for_delay",
"long_press",
)
def _tool_refs(text: str, tool: str) -> list[str]:
"""Finds real tool references (backticked or call form), ignoring prose."""
return re.findall(rf"(`{tool}`|\b{tool}\()", text)
# --- render_tool_enum ----------------------------------------------------------------
def test_enum_plain_list():
out = render_tool_enum(("a", "b", "c"), {"a", "b", "c"})
assert out == "`a`, `b`, `c`"
def test_enum_oxford_or_and():
tools = ("a", "b", "c")
assert render_tool_enum(tools, set(tools), final_sep="or") == "`a`, `b`, or `c`"
assert render_tool_enum(tools, set(tools), final_sep="and") == "`a`, `b`, and `c`"
def test_enum_two_items_no_oxford_comma():
assert render_tool_enum(("a", "b"), {"a", "b"}, final_sep="or") == "`a` or `b`"
def test_enum_single_and_empty():
assert render_tool_enum(("a", "b"), {"a"}) == "`a`"
assert render_tool_enum(("a", "b"), set()) == ""
def test_gate_segment():
assert gate_segment("teach", {"swipe"}, "swipe") == "teach"
assert gate_segment("teach", set(), "swipe") == ""
assert gate_segment("teach", {"a"}, "a", "b") == ""
# --- Operator prompt assembly --------------------------------------------------------
def test_full_toolset_is_the_default():
"""``available_tools=None`` must equal the explicit full set, byte for byte."""
for template in load_operator_prompts().values():
assert apply_operator_prompt_contract(template) == apply_operator_prompt_contract(
template, available_tools=OPERATOR_PROMPT_TOOLSET
)
@pytest.mark.parametrize("tool", OPERATOR_DEVICE_TOOLS + ("video_analyzer",))
def test_removed_tool_leaves_no_reference(tool):
"""The executable definition of 'an absent tool costs the model nothing'."""
for name, template in load_operator_prompts().items():
out = apply_operator_prompt_contract(
template, available_tools=OPERATOR_PROMPT_TOOLSET - {tool}
)
assert not _tool_refs(out, tool), f"'{tool}' still referenced in {name}"
def test_removed_swipe_takes_its_teaching_along():
"""Gating removes the drag/slider teaching, not just the backticked name."""
template = load_operator_prompts()["main_template"]
out = apply_operator_prompt_contract(
template, available_tools=OPERATOR_PROMPT_TOOLSET - {"swipe"}
)
assert "swipe/drag" not in out
assert "sliders/SeekBars" not in out
# The generic gesture-scope sentences survive.
assert "Action Scope & Interaction Precision" in out
def test_removed_wait_for_delay_keeps_interval_line_coherent():
"""Only the wait_for_delay parenthetical goes; the SOP grammar line stays."""
template = load_operator_prompts()["main_template"]
out = apply_operator_prompt_contract(
template, available_tools=OPERATOR_PROMPT_TOOLSET - {"wait_for_delay"}
)
assert "`Interval: <Interval>`: Declared polling cadence." in out
assert "Loading & Transitions" not in out
def test_reduced_enums_stay_well_formed():
template = load_operator_prompts()["main_template"]
out = apply_operator_prompt_contract(
template,
available_tools=OPERATOR_PROMPT_TOOLSET - {"manage_app", "wait_for_delay"},
)
assert (
"Physical device actions (`click`, `input_text`, `swipe`, `press_key`,"
" `long_press`)" in out
)
assert (
"Turn-Ending Action (`click`, `swipe`, `input_text`, `long_press`, or"
" `press_key`)" in out
)
def test_full_set_enum_slots_render_verbatim():
"""With every tool present the historical enumeration wording is reproduced."""
template = load_operator_prompts()["main_template"]
out = apply_operator_prompt_contract(template)
assert (
"Physical device actions (`click`, `input_text`, `swipe`, `press_key`,"
" `manage_app`, `wait_for_delay`, `long_press`)" in out
)
assert (
"Turn-Ending Action (`click`, `swipe`, `input_text`, `long_press`,"
" `press_key`, `manage_app`, or `wait_for_delay`)" in out
)
assert (
"(`ask_explorer`, `ask_diagnoser`, `video_analyzer`, `run_adb_command`,"
" `manage_task`, `analyze_task_output`, `read_note`, `list_notes`, and"
" `save_note`)" in out
)
# --- Flash prompt assembly -----------------------------------------------------------
def _render_flash(available_tools: frozenset[str]) -> str:
template_text = (REPO_ROOT / "artemis/agents/flash/flash_runner.md").read_text(
encoding="utf-8"
)
return Template(template_text).render(goal="test goal", available_tools=available_tools)
FLASH_FULL = frozenset({"manage_app", "wait_for_delay", "click_sequence", "click", "swipe"})
def test_flash_full_set_keeps_both_segments():
out = _render_flash(FLASH_FULL)
assert "**App Launching**" in out
assert "**Timed Waiting & Transitions**" in out
assert "{%" not in out
def test_flash_without_manage_app_drops_app_launching():
out = _render_flash(FLASH_FULL - {"manage_app"})
assert "**App Launching**" not in out
assert not _tool_refs(out, "manage_app")
# The neighbouring numbered rule survives.
assert "**Screenshot Pruning Awareness**" in out
def test_flash_without_wait_for_delay_drops_timed_waiting():
out = _render_flash(FLASH_FULL - {"wait_for_delay"})
assert "**Timed Waiting & Transitions**" not in out
assert not _tool_refs(out, "wait_for_delay")
assert "**State Drift & Progression Tolerance**" in out
# --- FailureAnalyzer strategy gating -------------------------------------------------
def _analyzer() -> FailureAnalyzer:
return FailureAnalyzer.__new__(FailureAnalyzer) # _select_strategy needs no ctx
def test_strategy_selection_unchanged_without_gating():
analyzer = _analyzer()
assert isinstance(
analyzer._select_strategy(ValidationErrorCategory.TARGET_DISAPPEARED),
TargetDisappearedStrategy,
)
assert isinstance(
analyzer._select_strategy(ValidationErrorCategory.PIXEL_TARGET_DISAPPEARED),
PixelTargetDisappearedStrategy,
)
def test_strategy_falls_back_when_required_tool_missing():
"""Strategy + prompt are one unit: no click_sequence, no disappeared-prompts."""
analyzer = _analyzer()
strategy = analyzer._select_strategy(
ValidationErrorCategory.TARGET_DISAPPEARED,
available_tools=frozenset({"click", "swipe"}),
)
assert type(strategy) is FailureAnalysisStrategy
assert strategy.get_prompt_template_name() == "failure_analyzer.md"
def test_strategy_kept_when_required_tool_present():
analyzer = _analyzer()
strategy = analyzer._select_strategy(
ValidationErrorCategory.PIXEL_TARGET_DISAPPEARED,
available_tools=frozenset({"click_sequence"}),
)
assert isinstance(strategy, PixelTargetDisappearedStrategy)
+112 -251
View File
@@ -12,13 +12,25 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
"""ValidatorNode tests over the in-process unified action session.
The Validator no longer spawns a stdio MCP subprocess; it calls a shared
``ActionSession`` whose device actions return structured ``ActionResult`` objects.
``FakeActionSession`` below stands in for it: ``action_handler`` decides each
action's ``(ok, message)``, ``screenshot`` feeds the polling loop, and ``hierarchy``
feeds the safety-net XML validation.
"""
import json import json
from unittest.mock import AsyncMock, MagicMock, Mock, patch from unittest.mock import AsyncMock, MagicMock, Mock, patch
from langchain_core.messages import AIMessage from langchain_core.messages import AIMessage
import pytest
from artemis.agents.validator.failure_analyzer import ValidationErrorCategory from artemis.agents.validator.failure_analyzer import ValidationErrorCategory
from artemis.agents.validator.validator import ValidatorNode from artemis.agents.validator.validator import ValidatorNode
from artemis.context import ArtemisContext from artemis.context import ArtemisContext
import pytest from artemis.mcp.action_types import ActionCode, ActionResult
class DummyState: class DummyState:
@@ -38,6 +50,46 @@ class DummyState:
return update return update
class FakeActionSession:
"""Test double for artemis.mcp.action_session.ActionSession."""
def __init__(self):
self.started = True
self.calls: list[tuple[str, dict]] = []
# (name, args) -> (ok, message)
self.action_handler = lambda name, args: (True, "")
# str | callable | Exception
self.screenshot = "ZHVtbXlfZGF0YQ==" # b64("dummy_data")
self.hierarchy: list | Exception = []
async def call(self, name, args):
self.calls.append((name, args))
ok, msg = self.action_handler(name, args)
return ActionResult(
ok=ok,
code=ActionCode.OK if ok else ActionCode.DEVICE_ERROR,
action=name,
message=msg,
)
async def screenshot_b64(self, timeout=None):
shot = self.screenshot
if isinstance(shot, Exception):
raise shot
return shot() if callable(shot) else shot
async def ui_hierarchy(self, timeout=None):
if isinstance(self.hierarchy, Exception):
raise self.hierarchy
return self.hierarchy
async def aclose(self):
pass
def calls_for(self, name: str) -> list[dict]:
return [args for n, args in self.calls if n == name]
@pytest.fixture @pytest.fixture
def mock_context(tmp_path): def mock_context(tmp_path):
ctx = Mock(spec=ArtemisContext) ctx = Mock(spec=ArtemisContext)
@@ -45,6 +97,9 @@ def mock_context(tmp_path):
ctx.data_engine = Mock() ctx.data_engine = Mock()
ctx.data_engine.base_dir = tmp_path ctx.data_engine.base_dir = tmp_path
ctx.data_engine.get_relative_time.return_value = 1.0 ctx.data_engine.get_relative_time.return_value = 1.0
ctx.device = Mock()
ctx.device.device_width = 1080
ctx.device.device_height = 2400
return ctx return ctx
@@ -57,31 +112,18 @@ def temp_screenshot(tmp_path):
@pytest.fixture @pytest.fixture
def mock_mcp(): def mock_mcp():
with ( fake = FakeActionSession()
patch("mcp.client.stdio.stdio_client") as mock_stdio,
patch("artemis.agents.validator.validator.ClientSession") as mock_session_cls,
):
mock_ctx = AsyncMock()
mock_ctx.__aenter__.return_value = (Mock(), Mock())
mock_stdio.return_value = mock_ctx
mock_session = AsyncMock() async def _get_session(ctx, actuator=None):
mock_session.initialize = AsyncMock() return fake
mock_session_cls.return_value = mock_session
yield mock_session with patch("artemis.agents.validator.validator.get_action_session", _get_session):
yield fake
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_validator_success(mock_mcp, mock_context, temp_screenshot): async def test_validator_success(mock_mcp, mock_context, temp_screenshot):
"""Test that ValidatorNode executes actions successfully.""" """Test that ValidatorNode executes actions successfully."""
# Mock MCP calls: screenshot, tap, screenshot
mock_mcp.call_tool.side_effect = [
Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")]), # pre-screenshot
Mock(content=[Mock(text="Success")]), # tap action
Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")]), # post-screenshot
]
decisions = json.dumps([{"action": "tap", "coordinates": [105, 205]}]) decisions = json.dumps([{"action": "tap", "coordinates": [105, 205]}])
state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot) state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot)
@@ -95,78 +137,23 @@ async def test_validator_success(mock_mcp, mock_context, temp_screenshot):
assert len(report["execution"]) == 1 assert len(report["execution"]) == 1
assert report["execution"][0]["action"] == "tap" assert report["execution"][0]["action"] == "tap"
assert "attempts" not in report["execution"][0] assert "attempts" not in report["execution"][0]
# The Operator's tap verb was translated to the canonical click tool.
assert len(mock_mcp.calls_for("click")) == 1
@pytest.mark.asyncio # NOTE: upstream's test_failed_mcp_handshake_exits_contexts_and_disables_child_awake_policy
async def test_failed_mcp_handshake_exits_contexts_and_disables_child_awake_policy(): # guarded the stdio-subprocess handshake teardown, machinery this migration removed
"""A handshake failure must not leave AnyIO cancellation scopes behind.""" # entirely. Its hazard class (leaked AnyIO cancel scopes) is now pinned by
events = [] # tests/unit/mcp/test_action_session.py against the in-process ActionSession.
class FakeClientContext:
async def __aenter__(self):
events.append("client_enter")
return Mock(), Mock()
async def __aexit__(self, exc_type, exc, traceback):
events.append("client_exit")
class FakeSession:
async def __aenter__(self):
events.append("session_enter")
return self
async def __aexit__(self, exc_type, exc, traceback):
events.append("session_exit")
async def initialize(self):
raise TimeoutError("handshake timed out")
ctx = Mock(spec=ArtemisContext)
ctx.mcp_client_ctx = None
ctx.mcp_session = None
fake_client_ctx = FakeClientContext()
fake_session = FakeSession()
captured_parameters = []
def fake_stdio_client(parameters):
captured_parameters.append(parameters)
return fake_client_ctx
node = ValidatorNode(ctx)
with (
patch("artemis.agents.validator.validator.stdio_client", fake_stdio_client),
patch("artemis.agents.validator.validator.ClientSession", return_value=fake_session),
patch("artemis.agents.validator.validator.AdbClient") as adb_client,
patch.object(node, "_kill_mcp_server_instantly") as emergency_kill,
):
adb_client.return_value.device_list.return_value = []
result = await node._get_mcp_session()
assert result is None
assert events == ["client_enter", "session_enter", "session_exit", "client_exit"]
assert ctx.mcp_session is None
assert ctx.mcp_client_ctx is None
assert captured_parameters[0].env["ARTEMIS_KEEP_DEVICE_AWAKE"] == "false"
emergency_kill.assert_not_called()
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_validator_failure_analysis(mock_mcp, mock_context, temp_screenshot): async def test_validator_failure_analysis(mock_mcp, mock_context, temp_screenshot):
"""Test that ValidatorNode triggers failure analysis on error and handles """Test that ValidatorNode triggers failure analysis on error and handles
cannot_fix via tool. cannot_fix via tool.
""" """
tap_count = 0 mock_mcp.action_handler = lambda name, args: (
(False, "Error: Element not found") if name == "click" else (True, "")
def mock_call_tool(name, args): )
nonlocal tap_count
if name == "take_screenshot":
return Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")])
if name == "tap":
tap_count += 1
return Mock(content=[Mock(text="Error: Element not found")])
return Mock(content=[Mock(text="Success")])
mock_mcp.call_tool.side_effect = mock_call_tool
decisions = json.dumps([{"action": "tap", "coordinates": [105, 205]}]) decisions = json.dumps([{"action": "tap", "coordinates": [105, 205]}])
state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot) state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot)
@@ -203,18 +190,15 @@ async def test_validator_repair_success(mock_mcp, mock_context, temp_screenshot)
""" """
tap_count = 0 tap_count = 0
def mock_call_tool(name, args): def handler(name, args):
nonlocal tap_count nonlocal tap_count
if name == "take_screenshot": if name == "click":
return Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")])
if name == "tap":
tap_count += 1 tap_count += 1
if tap_count <= 2: # First tap fails if tap_count <= 2: # First action fails on both validator attempts
return Mock(content=[Mock(text="Error: Element not found")]) return False, "Error: Element not found"
return Mock(content=[Mock(text="Success")]) return True, ""
return Mock(content=[Mock(text="Success")])
mock_mcp.call_tool.side_effect = mock_call_tool mock_mcp.action_handler = handler
# Two actions planned initially # Two actions planned initially
decisions = json.dumps( decisions = json.dumps(
@@ -254,12 +238,6 @@ async def test_validator_repair_success(mock_mcp, mock_context, temp_screenshot)
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_validator_wait_for_delay(mock_mcp, mock_context, temp_screenshot): async def test_validator_wait_for_delay(mock_mcp, mock_context, temp_screenshot):
"""Test that wait_for_delay executes successfully.""" """Test that wait_for_delay executes successfully."""
def mock_call(name, args):
return Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")])
mock_mcp.call_tool.side_effect = mock_call
decisions = json.dumps([{"action": "wait_for_delay", "time_in_ms": 10}]) decisions = json.dumps([{"action": "wait_for_delay", "time_in_ms": 10}])
state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot) state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot)
@@ -272,19 +250,13 @@ async def test_validator_wait_for_delay(mock_mcp, mock_context, temp_screenshot)
assert "execution" in report assert "execution" in report
assert len(report["execution"]) == 1 assert len(report["execution"]) == 1
assert report["execution"][0]["action"] == "wait_for_delay" assert report["execution"][0]["action"] == "wait_for_delay"
# wait_for_delay is a pure client-side sleep, never a session call.
assert mock_mcp.calls_for("wait_for_delay") == []
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_validator_focus_and_clear_text_no_ui_change(mock_mcp, mock_context, temp_screenshot): async def test_validator_focus_and_clear_text_no_ui_change(mock_mcp, mock_context, temp_screenshot):
"""Test that focus_and_clear_text executes successfully even if no UI change is detected.""" """Test that focus_and_clear_text executes successfully even if no UI change is detected."""
def mock_call(name, args):
if name == "focus_and_clear_text":
return Mock(content=[Mock(text="Success")])
return Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")])
mock_mcp.call_tool.side_effect = mock_call
decisions = json.dumps([{"action": "focus_and_clear_text", "coordinates": [400, 500]}]) decisions = json.dumps([{"action": "focus_and_clear_text", "coordinates": [400, 500]}])
state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot) state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot)
@@ -297,6 +269,7 @@ async def test_validator_focus_and_clear_text_no_ui_change(mock_mcp, mock_contex
assert "execution" in report assert "execution" in report
assert len(report["execution"]) == 1 assert len(report["execution"]) == 1
assert report["execution"][0]["action"] == "focus_and_clear_text" assert report["execution"][0]["action"] == "focus_and_clear_text"
assert len(mock_mcp.calls_for("focus_and_clear_text")) == 1
@pytest.mark.asyncio @pytest.mark.asyncio
@@ -304,16 +277,6 @@ async def test_validator_silent_failure_treated_as_success(mock_mcp, mock_contex
"""Test that silent failure (exec succeeds but no UI change) is treated as """Test that silent failure (exec succeeds but no UI change) is treated as
success and does not trigger FailureAnalyzer. success and does not trigger FailureAnalyzer.
""" """
def mock_call_tool(name, args):
if name == "take_screenshot":
return Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")])
if name == "tap":
return Mock(content=[Mock(text="Success")])
return Mock(content=[Mock(text="Success")])
mock_mcp.call_tool.side_effect = mock_call_tool
decisions = json.dumps([{"action": "tap", "coordinates": [105, 205]}]) decisions = json.dumps([{"action": "tap", "coordinates": [105, 205]}])
state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot) state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot)
@@ -344,13 +307,7 @@ async def test_validator_records_skipped_actions_on_cannot_fix(
mock_mcp, mock_context, temp_screenshot mock_mcp, mock_context, temp_screenshot
): ):
"""Test that ValidatorNode marks unexecuted actions as Skipped when repair fails.""" """Test that ValidatorNode marks unexecuted actions as Skipped when repair fails."""
mock_mcp.action_handler = lambda name, args: (False, "Error: Connection lost")
def mock_call_tool(name, args):
if name == "take_screenshot":
return Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")])
return Mock(content=[Mock(text="Error: Connection lost")])
mock_mcp.call_tool.side_effect = mock_call_tool
# Two actions planned # Two actions planned
decisions = json.dumps( decisions = json.dumps(
@@ -399,7 +356,7 @@ async def test_validator_pre_execution_validation_and_repair(
"""Test that pre-execution validation fails when element changes, and is """Test that pre-execution validation fails when element changes, and is
repaired by FailureAnalyzer. repaired by FailureAnalyzer.
""" """
mock_live_elements = [ mock_mcp.hierarchy = [
{ {
"text": "Sign Up", "text": "Sign Up",
"bounds": "[100,200][300,300]", "bounds": "[100,200][300,300]",
@@ -407,17 +364,6 @@ async def test_validator_pre_execution_validation_and_repair(
} }
] ]
def mock_call_tool(name, args):
if name == "take_screenshot":
return Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")])
if name == "get_ui_hierarchy":
return Mock(content=[Mock(text=json.dumps(mock_live_elements))])
if name == "tap":
return Mock(content=[Mock(text="Success")])
return Mock(content=[Mock(text="Success")])
mock_mcp.call_tool.side_effect = mock_call_tool
decisions = json.dumps( decisions = json.dumps(
[ [
{ {
@@ -477,7 +423,7 @@ async def test_validator_pre_execution_validation_self_healing(
mock_mcp, mock_context, temp_screenshot mock_mcp, mock_context, temp_screenshot
): ):
"""Test that pre-execution validation succeeds and self-heals when element shifts slightly.""" """Test that pre-execution validation succeeds and self-heals when element shifts slightly."""
mock_live_elements = [ mock_mcp.hierarchy = [
{ {
"text": "Login", "text": "Login",
"bounds": "[100,220][300,320]", "bounds": "[100,220][300,320]",
@@ -485,21 +431,6 @@ async def test_validator_pre_execution_validation_self_healing(
} }
] ]
tapped_coordinates = None
def mock_call_tool(name, args):
nonlocal tapped_coordinates
if name == "take_screenshot":
return Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")])
if name == "get_ui_hierarchy":
return Mock(content=[Mock(text=json.dumps(mock_live_elements))])
if name == "tap":
tapped_coordinates = args.get("coordinates")
return Mock(content=[Mock(text="Success")])
return Mock(content=[Mock(text="Success")])
mock_mcp.call_tool.side_effect = mock_call_tool
decisions = json.dumps( decisions = json.dumps(
[ [
{ {
@@ -518,8 +449,12 @@ async def test_validator_pre_execution_validation_self_healing(
with patch("artemis.utils.image_diff.check_ui_change", return_value=True): with patch("artemis.utils.image_diff.check_ui_change", return_value=True):
await node(state) await node(state)
# Center of [100, 220][300, 320] is [200, 270] # Center of [100, 220][300, 320] is pixel [200, 270]; the healed coordinates
assert tapped_coordinates == [200, 270] # travel to the canonical click tool normalized to 0-1000 on a 1080x2400 screen:
# round(200 * 1000 / 1080) = 185, round(270 * 1000 / 2400) = 112.
clicks = mock_mcp.calls_for("click")
assert len(clicks) == 1
assert clicks[0]["target"] == [185, 112]
@pytest.mark.asyncio @pytest.mark.asyncio
@@ -531,7 +466,7 @@ async def test_validator_pre_execution_validation_anonymous_occupant(
""" """
# The live screen contains an anonymous clickable view covering # The live screen contains an anonymous clickable view covering
# target coordinates [200, 250] # target coordinates [200, 250]
mock_live_elements = [ mock_mcp.hierarchy = [
{ {
"text": "", "text": "",
"bounds": "[150,200][250,300]", "bounds": "[150,200][250,300]",
@@ -540,15 +475,6 @@ async def test_validator_pre_execution_validation_anonymous_occupant(
} }
] ]
def mock_call_tool(name, args):
if name == "take_screenshot":
return Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")])
if name == "get_ui_hierarchy":
return Mock(content=[Mock(text=json.dumps(mock_live_elements))])
return Mock(content=[Mock(text="Success")])
mock_mcp.call_tool.side_effect = mock_call_tool
decisions = json.dumps( decisions = json.dumps(
[ [
{ {
@@ -596,12 +522,6 @@ async def test_validator_pre_execution_validation_anonymous_occupant(
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_validator_pixel_validation_success(mock_mcp, mock_context, temp_screenshot): async def test_validator_pixel_validation_success(mock_mcp, mock_context, temp_screenshot):
"""Test that the pixel safety net succeeds when Gemini reports target is present.""" """Test that the pixel safety net succeeds when Gemini reports target is present."""
mock_mcp.call_tool.side_effect = [
Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")]), # live screenshot for pixel check
Mock(content=[Mock(text="Success")]), # tap action
Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")]), # post-screenshot
]
decisions = json.dumps([{"action": "tap", "coordinates": [100, 200]}]) decisions = json.dumps([{"action": "tap", "coordinates": [100, 200]}])
state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot) state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot)
@@ -645,14 +565,6 @@ async def test_validator_pixel_validation_failure(mock_mcp, mock_context, temp_s
"""Test that the pixel safety net fails and triggers FailureAnalyzer """Test that the pixel safety net fails and triggers FailureAnalyzer
when Gemini reports target is missing. when Gemini reports target is missing.
""" """
def mock_call_tool(name, args):
if name == "take_screenshot":
return Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")])
return Mock(content=[Mock(text="Success")])
mock_mcp.call_tool.side_effect = mock_call_tool
decisions = json.dumps([{"action": "tap", "coordinates": [100, 200]}]) decisions = json.dumps([{"action": "tap", "coordinates": [100, 200]}])
state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot) state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot)
@@ -697,23 +609,15 @@ async def test_validator_pixel_validation_failure(mock_mcp, mock_context, temp_s
@pytest.mark.asyncio @pytest.mark.asyncio
@patch("artemis.agents.validator.validator.find_package") async def test_validator_launch_app_routes_through_manage_app(
@patch("artemis.utils.app_launch_utils.launch_app_with_retries") mock_mcp, mock_context, temp_screenshot
async def test_validator_launch_app_passes_use_fallback_mcp(
mock_launch_app, mock_find_package, mock_mcp, mock_context, temp_screenshot
): ):
"""Test that ValidatorNode calls find_package with use_fallback=False """Test that a launch_app action item routes to the canonical manage_app tool.
for launch_app action in MCP mode.
Package resolution (find_package with use_fallback=False) now happens inside the
actuator's manage_app implementation and is covered by the failure-analyzer tool
tests; the Validator's contract is the translation and single dispatch.
""" """
mock_find_package.return_value = "com.example.app"
mock_launch_app.return_value = (True, None)
# Mock MCP calls: screenshot (pre), screenshot (post)
mock_mcp.call_tool.side_effect = [
Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")]), # pre-screenshot
Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")]), # post-screenshot
]
decisions = json.dumps([{"action": "launch_app", "app_name": "My App"}]) decisions = json.dumps([{"action": "launch_app", "app_name": "My App"}])
state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot) state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot)
@@ -721,27 +625,17 @@ async def test_validator_launch_app_passes_use_fallback_mcp(
node = ValidatorNode(mock_context) node = ValidatorNode(mock_context)
result = await node(state) result = await node(state)
mock_find_package.assert_called_once_with(mock_context, "My App", use_fallback=False) manage_calls = mock_mcp.calls_for("manage_app")
mock_launch_app.assert_called_once_with(mock_context, "com.example.app") assert manage_calls == [{"action": "launch", "app_name": "My App"}]
assert "last_execution_result" in result assert "last_execution_result" in result
@pytest.mark.asyncio @pytest.mark.asyncio
@patch("artemis.agents.validator.validator.find_package") async def test_validator_launch_app_failure_no_retry(mock_mcp, mock_context, temp_screenshot):
@patch("artemis.utils.app_launch_utils.launch_app_with_retries")
async def test_validator_launch_app_failure_no_retry(
mock_launch_app, mock_find_package, mock_mcp, mock_context, temp_screenshot
):
"""Test that ValidatorNode does NOT retry launch_app action on failure.""" """Test that ValidatorNode does NOT retry launch_app action on failure."""
mock_find_package.return_value = "com.example.app" mock_mcp.action_handler = lambda name, args: (
mock_launch_app.return_value = (False, "Error: Force close") (False, "Error: Force close") if name == "manage_app" else (True, "")
)
def mock_call_tool(name, args):
if name == "take_screenshot":
return Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")])
return Mock(content=[Mock(text="Success")])
mock_mcp.call_tool.side_effect = mock_call_tool
decisions = json.dumps([{"action": "launch_app", "app_name": "My App"}]) decisions = json.dumps([{"action": "launch_app", "app_name": "My App"}])
state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot) state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot)
@@ -760,13 +654,9 @@ async def test_validator_launch_app_failure_no_retry(
node = ValidatorNode(mock_context) node = ValidatorNode(mock_context)
result = await node(state) result = await node(state)
# Verify find_package was called only once # launch_app allows a single attempt -- no validator-level retry.
mock_find_package.assert_called_once_with(mock_context, "My App", use_fallback=False) assert len(mock_mcp.calls_for("manage_app")) == 1
# Verify launch_app_with_retries was called only once
# (no retry by ValidatorNode itself since it relies on launch_app_with_retries)
mock_launch_app.assert_called_once_with(mock_context, "com.example.app")
# Verify attempts list has only 1 element
assert "last_execution_result" in result assert "last_execution_result" in result
report = result["last_execution_result"] report = result["last_execution_result"]
assert len(report["execution"]) == 1 assert len(report["execution"]) == 1
@@ -783,7 +673,7 @@ async def test_validator_pre_execution_validation_disappeared_not_shifted(
""" """
# Live hierarchy only contains a giant background/container # Live hierarchy only contains a giant background/container
# containing the text "Sign Up" # containing the text "Sign Up"
mock_live_elements = [ mock_mcp.hierarchy = [
{ {
"text": "Sign Up", "text": "Sign Up",
"bounds": "[0,0][1080,2400]", "bounds": "[0,0][1080,2400]",
@@ -791,15 +681,6 @@ async def test_validator_pre_execution_validation_disappeared_not_shifted(
} }
] ]
def mock_call_tool(name, args):
if name == "take_screenshot":
return Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")])
if name == "get_ui_hierarchy":
return Mock(content=[Mock(text=json.dumps(mock_live_elements))])
return Mock(content=[Mock(text="Success")])
mock_mcp.call_tool.side_effect = mock_call_tool
decisions = json.dumps( decisions = json.dumps(
[ [
{ {
@@ -850,12 +731,6 @@ async def test_validator_pre_execution_validation_ocr_direct_to_pixel(
"""Test that OCR-derived elements bypass XML validation and """Test that OCR-derived elements bypass XML validation and
directly use pixel-based validation. directly use pixel-based validation.
""" """
mock_mcp.call_tool.side_effect = [
Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")]), # live screenshot for pixel check
Mock(content=[Mock(text="Success")]), # tap action
Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")]), # post-screenshot
]
decisions = json.dumps( decisions = json.dumps(
[ [
{ {
@@ -905,7 +780,7 @@ async def test_validator_pre_execution_xml_failure_not_overridden_when_pixel_byp
"""Test that when XML validation fails AND Pixel validation is bypassed """Test that when XML validation fails AND Pixel validation is bypassed
(PIXEL_BYPASSED), the XML failure is NOT overridden. (PIXEL_BYPASSED), the XML failure is NOT overridden.
""" """
mock_live_elements = [ mock_mcp.hierarchy = [
{ {
"text": "Sign Up", "text": "Sign Up",
"bounds": "[100,200][300,300]", "bounds": "[100,200][300,300]",
@@ -913,15 +788,6 @@ async def test_validator_pre_execution_xml_failure_not_overridden_when_pixel_byp
} }
] ]
def mock_call_tool(name, args):
if name == "take_screenshot":
return Mock(content=[Mock(text="ZHVtbXlfZGF0YQ==")])
if name == "get_ui_hierarchy":
return Mock(content=[Mock(text=json.dumps(mock_live_elements))])
return Mock(content=[Mock(text="Success")])
mock_mcp.call_tool.side_effect = mock_call_tool
decisions = json.dumps( decisions = json.dumps(
[ [
{ {
@@ -970,17 +836,13 @@ async def test_validator_pre_execution_xml_failure_not_overridden_when_pixel_byp
async def test_validator_failure_screenshot_mcp_error_handled_safely( async def test_validator_failure_screenshot_mcp_error_handled_safely(
mock_mcp, mock_context, temp_screenshot mock_mcp, mock_context, temp_screenshot
): ):
"""Test that when take_screenshot fails on MCP server and returns an error text, """Test that when take_screenshot fails on the session, the ValidatorNode does
the ValidatorNode does not crash with base64 decoding error. not crash -- the failure screenshot is simply absent.
""" """
def mock_call_tool(name, args): mock_mcp.screenshot = RuntimeError("Error: ADB connection lost")
if name == "take_screenshot": mock_mcp.action_handler = lambda name, args: (
return Mock(content=[Mock(text="Error: ADB connection lost")]) (False, "Error: Target not found") if name == "click" else (True, "")
if name == "tap": )
return Mock(content=[Mock(text="Error: Target not found")])
return Mock(content=[Mock(text="Success")])
mock_mcp.call_tool.side_effect = mock_call_tool
decisions = json.dumps([{"action": "tap", "coordinates": [105, 205]}]) decisions = json.dumps([{"action": "tap", "coordinates": [105, 205]}])
state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot) state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot)
@@ -1002,4 +864,3 @@ async def test_validator_failure_screenshot_mcp_error_handled_safely(
assert report["status"] == "failed" assert report["status"] == "failed"
assert len(report["execution"]) == 1 assert len(report["execution"]) == 1
assert report["execution"][0]["action"] == "tap" assert report["execution"][0]["action"] == "tap"
+3 -1
View File
@@ -55,7 +55,9 @@ def test_darwin_platform_paths():
"""Verify macOS platform paths adhere to Apple Library standards.""" """Verify macOS platform paths adhere to Apple Library standards."""
darwin_p = DarwinPlatform() darwin_p = DarwinPlatform()
assert darwin_p.os_type == OSType.MACOS assert darwin_p.os_type == OSType.MACOS
assert "Library/Application Support" in str(darwin_p.paths.config_dir) # Compare path components so the assertion holds regardless of the host
# OS path separator used by str().
assert darwin_p.paths.config_dir.parts[-3:] == ("Library", "Application Support", "artemis")
assert darwin_p.process.path_separator == ":" assert darwin_p.process.path_separator == ":"
@@ -392,10 +392,12 @@ async def test_validator_pre_execution_loop_reverted_to_exact_safety_contract(
mock_artemis_ctx.data_engine = MagicMock() mock_artemis_ctx.data_engine = MagicMock()
node = ValidatorNode(mock_artemis_ctx) node = ValidatorNode(mock_artemis_ctx)
# The Validator now talks to the in-process unified action session.
session = MagicMock() session = MagicMock()
session.call_tool = AsyncMock( session.started = True
return_value=MagicMock(content=[MagicMock(text="ZHVtbXlfZGF0YQ==")]) session.screenshot_b64 = AsyncMock(return_value="ZHVtbXlfZGF0YQ==")
) session.ui_hierarchy = AsyncMock(return_value=[])
session.call = AsyncMock()
decisions = json.dumps( decisions = json.dumps(
[ [
@@ -421,6 +423,10 @@ async def test_validator_pre_execution_loop_reverted_to_exact_safety_contract(
) )
with ( with (
patch(
"artemis.agents.validator.validator.get_action_session",
AsyncMock(return_value=session),
),
patch.object( patch.object(
ValidatorNode, "_get_initial_screenshot", new_callable=AsyncMock ValidatorNode, "_get_initial_screenshot", new_callable=AsyncMock
) as mock_get_screen, ) as mock_get_screen,
+294
View File
@@ -0,0 +1,294 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Prompt <-> manifest consistency and actuator contract tests.
The scan below re-derives the manifest classification from the prompt sources, so
adding a tool name to a prompt without classifying it in
``artemis/mcp/action_manifest.py`` fails here rather than at runtime.
"""
import json
from pathlib import Path
import re
import pytest
from artemis.core.tool_declaration import ToolDeclaration
from artemis.mcp.action_manifest import (
ActuatorContractError,
AGENT_ROLES,
ALL_KNOWN_TOOLS,
BACKEND_INDEPENDENT_TOOLS,
DEVICE_ACTIONS,
ExtensionTool,
INTERNAL_ACTIONS,
OPTIONAL_ACTIONS,
REQUIRED_ACTIONS,
available_device_actions,
filter_declarations,
validate_actuator,
)
REPO_ROOT = Path(__file__).resolve().parents[3]
# Prompt sources belonging to the three agents this manifest governs (plus the
# Validator's safety net). Other agents (log_analyzer, image_processor, ...) have their
# own tool universes and are deliberately out of scope.
SCOPED_PROMPT_FILES = [
"artemis/agents/flash/flash_runner.md",
"artemis/agents/validator/failure_analyzer.md",
"artemis/agents/validator/target_disappeared_analyzer.md",
"artemis/agents/validator/pixel_target_disappeared_analyzer.md",
"artemis/agents/validator/pixel_safety_net.md",
]
# Backtick-quoted identifiers in scoped prompts that are not tools: parameter names and
# note keys. Anything new landing here should be a conscious decision.
NON_TOOL_IDENTIFIERS = frozenset({"analysis", "task_plan"})
_TOOL_REF = re.compile(r"(?:`([a-z][a-z0-9_]{2,})`|\b([a-z][a-z0-9_]{2,})\()")
def _scoped_prompt_sources() -> dict[str, str]:
"""Collects all in-scope prompt texts, keyed by a short source label."""
sources: dict[str, str] = {}
for rel in SCOPED_PROMPT_FILES:
path = REPO_ROOT / rel
sources[path.name] = path.read_text(encoding="utf-8")
operator_json = REPO_ROOT / "artemis/agents/operator/operator.json"
for key, text in json.loads(operator_json.read_text(encoding="utf-8")).items():
sources[f"operator.json:{key}"] = text
# Only the prompt string constants from prompts.py -- scanning the module as text
# would pick up Python code (`len(`, `getattr(` ...).
from artemis.agents.operator import prompts as operator_prompts
for name in dir(operator_prompts):
value = getattr(operator_prompts, name)
if name.startswith("_") and isinstance(value, str) and len(value) > 80:
sources[f"prompts.py:{name}"] = value
return sources
def _referenced_tools() -> dict[str, set[str]]:
"""Returns {identifier: source labels} for every tool-shaped reference in scope."""
found: dict[str, set[str]] = {}
for label, text in _scoped_prompt_sources().items():
for match in _TOOL_REF.finditer(text):
name = match.group(1) or match.group(2)
found.setdefault(name, set()).add(label)
return found
# --- Prompt <-> manifest invariants --------------------------------------------------
def test_scan_finds_prompt_sources():
"""The scan must cover all expected sources; silent scope loss would void the suite."""
sources = _scoped_prompt_sources()
assert "flash_runner.md" in sources
assert "operator.json:main_template" in sources
assert "operator.json:troubleshooter_template" in sources
assert any(label.startswith("prompts.py:") for label in sources)
def test_every_prompt_tool_reference_is_classified():
"""Every tool referenced by an in-scope prompt must be known to the manifest."""
unclassified = {
name: sorted(labels)
for name, labels in _referenced_tools().items()
if name not in ALL_KNOWN_TOOLS and name not in NON_TOOL_IDENTIFIERS
}
assert not unclassified, (
"Prompts reference tools the manifest does not classify. Add each to the "
f"appropriate set in artemis/mcp/action_manifest.py: {unclassified}"
)
def test_required_actions_are_structurally_referenced():
"""A REQUIRED action must actually appear in a prompt; otherwise demote it."""
referenced = set(_referenced_tools())
unreferenced = REQUIRED_ACTIONS - referenced
assert not unreferenced, (
f"REQUIRED_ACTIONS not referenced by any prompt: {sorted(unreferenced)}. "
"Required status exists only to protect structural prompt dependencies."
)
def test_internal_actions_never_appear_in_prompts():
"""INTERNAL actions are adapter plumbing and must stay invisible to every model."""
leaked = INTERNAL_ACTIONS & set(_referenced_tools())
assert not leaked, f"Internal actions leaked into prompts: {sorted(leaked)}"
def test_manifest_sets_are_disjoint():
sets = {
"REQUIRED_ACTIONS": REQUIRED_ACTIONS,
"OPTIONAL_ACTIONS": OPTIONAL_ACTIONS,
"INTERNAL_ACTIONS": INTERNAL_ACTIONS,
"BACKEND_INDEPENDENT_TOOLS": BACKEND_INDEPENDENT_TOOLS,
}
names = list(sets)
for i, a in enumerate(names):
for b in names[i + 1 :]:
overlap = sets[a] & sets[b]
assert not overlap, f"{a} and {b} overlap: {sorted(overlap)}"
# --- Actuator contract ---------------------------------------------------------------
class _StubActuator:
"""Minimal actuator test double with configurable capabilities and extensions."""
def __init__(self, caps=None, exts=None):
self._caps = frozenset(caps if caps is not None else DEVICE_ACTIONS)
self._exts = list(exts or [])
def capabilities(self):
return self._caps
def extensions(self):
return self._exts
def _extension(**overrides) -> ExtensionTool:
async def _noop(**_kwargs):
return "ok"
defaults = dict(
name="calibrate_arm",
description=(
"Runs the robot arm's touch calibration sweep. Use before the first "
"physical action of a session, or after any missed tap."
),
input_schema={"type": "object", "properties": {}},
handler=_noop,
targets=frozenset({"operator", "flash"}),
)
defaults.update(overrides)
return ExtensionTool(**defaults)
def test_full_actuator_passes_validation():
validate_actuator(_StubActuator())
def test_minimal_actuator_passes_validation():
"""REQUIRED + INTERNAL alone is a legal backend; every optional action may be absent."""
validate_actuator(_StubActuator(caps=REQUIRED_ACTIONS | INTERNAL_ACTIONS))
def test_missing_required_action_fails_fast():
caps = (DEVICE_ACTIONS - {"click_sequence"})
with pytest.raises(ActuatorContractError, match="click_sequence"):
validate_actuator(_StubActuator(caps=caps))
def test_missing_internal_action_fails_fast():
caps = DEVICE_ACTIONS - {"observe_screen"}
with pytest.raises(ActuatorContractError, match="observe_screen"):
validate_actuator(_StubActuator(caps=caps))
def test_unknown_capability_is_rejected():
"""Backend-specific tools must go through extensions(), not capabilities()."""
with pytest.raises(ActuatorContractError, match="calibrate_arm"):
validate_actuator(_StubActuator(caps=DEVICE_ACTIONS | {"calibrate_arm"}))
def test_extension_name_collision_fails_fast():
with pytest.raises(ActuatorContractError, match="collides"):
validate_actuator(_StubActuator(exts=[_extension(name="click")]))
def test_duplicate_extension_names_fail_fast():
with pytest.raises(ActuatorContractError, match="more than once"):
validate_actuator(_StubActuator(exts=[_extension(), _extension()]))
def test_extension_with_thin_description_is_rejected():
"""Extensions get no prompt-level teaching; the description carries the contract."""
with pytest.raises(ActuatorContractError, match="description"):
validate_actuator(_StubActuator(exts=[_extension(description="Calibrates.")]))
def test_extension_with_unknown_target_is_rejected():
with pytest.raises(ActuatorContractError, match="unknown agent role"):
validate_actuator(_StubActuator(exts=[_extension(targets=frozenset({"planner"}))]))
# --- Surface derivation --------------------------------------------------------------
def _declarations(*names: str) -> list[ToolDeclaration]:
return [
ToolDeclaration(name=n, description=f"{n} tool", parameters={"type": "object"})
for n in names
]
def test_available_device_actions_excludes_internal():
actions = available_device_actions(_StubActuator())
assert actions == REQUIRED_ACTIONS | OPTIONAL_ACTIONS
assert not actions & INTERNAL_ACTIONS
def test_filter_drops_unimplemented_optional_actions():
actuator = _StubActuator(caps=DEVICE_ACTIONS - {"swipe", "manage_app"})
decls = _declarations("click", "swipe", "manage_app", "read_note")
kept = [d.name for d in filter_declarations(decls, actuator, agent="flash")]
assert kept == ["click", "read_note"]
def test_filter_passes_through_unclassified_agent_tools():
"""Agent-specific tools the manifest does not know must never be gated."""
actuator = _StubActuator(caps=REQUIRED_ACTIONS | INTERNAL_ACTIONS)
decls = _declarations("click_sequence", "reply_to_checker", "some_bespoke_tool")
kept = [d.name for d in filter_declarations(decls, actuator, agent="operator")]
assert kept == ["click_sequence", "reply_to_checker", "some_bespoke_tool"]
def test_filter_appends_extensions_for_targeted_agents_only():
actuator = _StubActuator(exts=[_extension(targets=frozenset({"flash"}))])
decls = _declarations("click")
flash = [d.name for d in filter_declarations(decls, actuator, agent="flash")]
operator = [d.name for d in filter_declarations(decls, actuator, agent="operator")]
assert flash == ["click", "calibrate_arm"]
assert operator == ["click"]
def test_filter_extension_declaration_shape():
"""Extension declarations must be indistinguishable in shape from known ones."""
actuator = _StubActuator(exts=[_extension()])
decls = filter_declarations([], actuator, agent="operator")
assert len(decls) == 1
ext_decl = decls[0]
assert isinstance(ext_decl, ToolDeclaration)
assert ext_decl.name == "calibrate_arm"
assert ext_decl.parameters == {"type": "object", "properties": {}}
def test_filter_rejects_unknown_agent_role():
with pytest.raises(ValueError, match="Unknown agent role"):
filter_declarations([], _StubActuator(), agent="planner")
def test_all_agent_roles_are_covered():
assert AGENT_ROLES == {"operator", "flash", "failure_analyzer", "validator"}
+156
View File
@@ -0,0 +1,156 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""End-to-end tests of the unified action server over the in-memory transport."""
import pytest
from artemis.mcp.action_manifest import (
ActuatorContractError,
DEVICE_ACTIONS,
INTERNAL_ACTIONS,
REQUIRED_ACTIONS,
ExtensionTool,
)
from artemis.mcp.action_server import build_action_server
from artemis.mcp.action_session import ActionSession
from artemis.mcp.actuators import MockActuator
pytestmark = pytest.mark.asyncio
async def _session_for(actuator) -> ActionSession:
return await ActionSession(build_action_server(actuator)).start()
async def test_click_end_to_end():
actuator = MockActuator(width=1000, height=2000)
session = await _session_for(actuator)
try:
res = await session.call("click", {"target": [500, 500]})
assert res.ok
assert res.message == "Clicked at [500, 500] (normalized) successfully."
assert actuator.action_history[-1]["action"] == "tap"
assert actuator.action_history[-1]["x"] == 500
assert actuator.action_history[-1]["y"] == 1000
finally:
await session.aclose()
async def test_device_refusal_is_not_a_protocol_error():
"""A failed action reaches the caller as ok=False, never as an exception."""
actuator = MockActuator()
session = await _session_for(actuator)
try:
res = await session.call("press_key", {"key": ""})
assert not res.ok
assert res.code.value == "INVALID_ARGS"
assert res.message == "Error executing key press ''."
finally:
await session.aclose()
async def test_partial_actuator_hides_optional_tools():
"""An unimplemented optional tool does not exist anywhere on the surface."""
minimal = MockActuator(capabilities=REQUIRED_ACTIONS | INTERNAL_ACTIONS)
session = await _session_for(minimal)
try:
listed = {t.name for t in (await session._session.list_tools()).tools}
assert "click_sequence" in listed
assert "observe_screen" in listed
for absent in ("click", "swipe", "manage_app", "wait_for_text", "open_link"):
assert absent not in listed
finally:
await session.aclose()
async def test_missing_required_action_fails_at_build_time():
broken = MockActuator(capabilities=DEVICE_ACTIONS - {"click_sequence"})
with pytest.raises(ActuatorContractError, match="click_sequence"):
build_action_server(broken)
async def test_extension_tool_is_served_and_callable():
async def calibrate(sweep_speed: int = 1) -> str:
return f"calibrated at speed {sweep_speed}"
ext = ExtensionTool(
name="calibrate_arm",
description=(
"Runs the robot arm's touch calibration sweep. Use before the first "
"physical action of a session, or after any missed tap."
),
input_schema={"type": "object", "properties": {}},
handler=calibrate,
targets=frozenset({"flash"}),
)
actuator = MockActuator(extensions=[ext])
session = await _session_for(actuator)
try:
listed = {t.name for t in (await session._session.list_tools()).tools}
assert "calibrate_arm" in listed
result = await session.call_raw("calibrate_arm", {"sweep_speed": 3})
text = next(b.text for b in result.content if b.type == "text")
assert text == "calibrated at speed 3"
finally:
await session.aclose()
async def test_extension_name_collision_fails_at_build_time():
async def bogus() -> str:
return "x"
ext = ExtensionTool(
name="click",
description="An extension improperly reusing a canonical tool name, which "
"must be rejected at build time.",
input_schema={"type": "object"},
handler=bogus,
)
with pytest.raises(ActuatorContractError, match="collides"):
build_action_server(MockActuator(extensions=[ext]))
async def test_unknown_tool_raises_protocol_error():
session = await _session_for(MockActuator())
try:
with pytest.raises(RuntimeError, match="not_a_tool"):
await session.call_raw("not_a_tool", {})
finally:
await session.aclose()
async def test_screenshot_and_hierarchy_helpers():
session = await _session_for(MockActuator(width=1000, height=2000))
try:
b64 = await session.screenshot_b64()
assert isinstance(b64, str) and len(b64) > 0
elements = await session.ui_hierarchy()
assert elements is not None
finally:
await session.aclose()
async def test_structured_content_carries_real_status():
"""The wire result exposes ok/code so no consumer ever sniffs substrings again."""
session = await _session_for(MockActuator())
try:
raw = await session.call_raw("click", {"target": [10, 10]})
sc = raw.structuredContent
assert sc["ok"] is True
assert sc["code"] == "OK"
assert sc["action"] == "click"
assert raw.isError is False
finally:
await session.aclose()
+188
View File
@@ -0,0 +1,188 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""ActionSession lifecycle tests, including the anyio same-task exit hazard."""
import asyncio
import pytest
from artemis.mcp.action_server import build_action_server
from artemis.mcp.action_session import ActionSession, get_action_session
from artemis.mcp.actuators import MockActuator
pytestmark = pytest.mark.asyncio
async def test_start_and_close():
session = ActionSession(build_action_server(MockActuator()))
await session.start()
assert session.started
await session.aclose()
assert not session.started
async def test_close_from_different_task():
"""The regression guard for the anyio cancel-scope same-task rule.
The in-memory transport's task group is entered inside the owner task; closing
from any *other* task must not raise ``RuntimeError: Attempted to exit cancel
scope in a different task``.
"""
session = ActionSession(build_action_server(MockActuator()))
await session.start()
async def _closer():
await session.aclose()
await asyncio.create_task(_closer())
assert not session.started
async def test_caller_timeout_does_not_brick_the_session():
"""Regression: cancelling a caller must never corrupt the shared transport.
Observed live on-device: ``asyncio.wait_for(session.ui_hierarchy(), 1.0)`` around
a slow hierarchy fetch cancelled ``call_tool`` mid-flight, corrupting the
in-memory streams -- every later call failed with an empty-message
``BrokenResourceError`` and the server never received another request. The owner
request queue makes an abandoned caller harmless; this test pins that down.
"""
actuator = MockActuator(width=1000, height=2000)
async def _slow_get_ui_elements():
await asyncio.sleep(2.0)
return []
actuator.get_ui_elements = _slow_get_ui_elements
session = await ActionSession(build_action_server(actuator)).start()
try:
with pytest.raises(asyncio.TimeoutError):
await asyncio.wait_for(session.ui_hierarchy(), timeout=0.1)
# The session must still be fully functional afterwards.
result = await session.call("click", {"target": [500, 500]})
assert result.ok
taps = [h for h in actuator.action_history if h["action"] == "tap"]
assert len(taps) == 1
finally:
await session.aclose()
async def test_per_call_timeout_fires_server_side_and_session_survives():
"""A per-call ``timeout`` runs inside the server tool, never on the transport.
mcp 1.29's ``read_timeout_seconds`` is NOT safe on the in-memory transport: a
late response to a timed-out request crashes the client receive loop
(``_handle_response`` sends into the request's already-closed response stream) --
observed live on-device as a session bricked after a single clean
``McpError`` timeout. Server-side timeouts produce an ordinary error payload
and no late response, so the session stays healthy by construction.
"""
actuator = MockActuator(width=1000, height=2000)
async def _slow_get_ui_elements():
await asyncio.sleep(2.0)
return []
actuator.get_ui_elements = _slow_get_ui_elements
session = await ActionSession(build_action_server(actuator)).start()
try:
with pytest.raises(RuntimeError, match="timed out"):
await session.ui_hierarchy(timeout=0.1)
result = await session.call("click", {"target": [500, 500]})
assert result.ok
finally:
await session.aclose()
async def test_transport_death_retires_session_for_rebuild():
"""A dead transport must flip ``started`` so get_action_session rebuilds.
Without this, one transport failure leaves the rest of a run failing on a
corpse (observed live: 24 minutes of empty-message ``ClosedResourceError``).
"""
import anyio
session = await ActionSession(build_action_server(MockActuator())).start()
real_call_tool = session._session.call_tool
async def _dying_call_tool(name, args, **kw):
raise anyio.ClosedResourceError()
session._session.call_tool = _dying_call_tool
with pytest.raises(anyio.ClosedResourceError):
await session.call("click", {"target": [1, 1]})
# Owner loop retires; the session reports itself unusable. The owner task may
# finish with an exception group from the transport teardown -- aclose() is the
# API that swallows that, so here we just wait for `started` to flip.
for _ in range(50):
if not session.started:
break
await asyncio.sleep(0.1)
assert not session.started
class Ctx:
action_session = session
fresh = await get_action_session(Ctx(), actuator=MockActuator())
assert fresh is not session
assert fresh.started
await fresh.aclose()
del real_call_tool
async def test_calls_are_serialized_by_the_lock():
"""Two concurrent calls must not interleave on the single device."""
actuator = MockActuator(width=1000, height=2000)
session = await ActionSession(build_action_server(actuator)).start()
try:
await asyncio.gather(
session.call("click", {"target": [100, 100]}),
session.call("click", {"target": [900, 900]}),
)
taps = [h for h in actuator.action_history if h["action"] == "tap"]
assert len(taps) == 2
finally:
await session.aclose()
async def test_call_before_start_raises():
session = ActionSession(build_action_server(MockActuator()))
with pytest.raises(RuntimeError, match="not started"):
await session.call("click", {"target": [1, 1]})
async def test_double_start_is_idempotent():
session = ActionSession(build_action_server(MockActuator()))
await session.start()
owner = session._owner
await session.start()
assert session._owner is owner
await session.aclose()
async def test_get_action_session_caches_on_ctx():
class Ctx:
action_session = None
ctx = Ctx()
actuator = MockActuator()
session = await get_action_session(ctx, actuator=actuator)
assert ctx.action_session is session
again = await get_action_session(ctx)
assert again is session
await session.aclose()
+146
View File
@@ -0,0 +1,146 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Actuator layer tests: coordinate conversion, message parity, driver dispatch."""
import pytest
from artemis.mcp.action_manifest import DEVICE_ACTIONS
from artemis.mcp.action_types import ActionCode
from artemis.mcp.actuators import Actuator, AdbActuator, MockActuator
@pytest.fixture
def actuator() -> MockActuator:
return MockActuator(width=1000, height=2000)
def test_mock_actuator_satisfies_protocol(actuator):
assert isinstance(actuator, Actuator)
assert isinstance(actuator, AdbActuator)
assert actuator.capabilities() == DEVICE_ACTIONS
assert actuator.extensions() == []
@pytest.mark.asyncio
async def test_click_converts_normalized_to_pixels(actuator):
res = await actuator.click(500, 500)
assert res.ok and res.code is ActionCode.OK
# Historical wording is part of the contract: transcripts and traces assert on it.
assert res.message == "Clicked at [500, 500] (normalized) successfully."
assert res.normalized_coordinates == [500, 500]
assert actuator.action_history[-1] == {
"action": "tap",
"x": 500,
"y": 1000,
"times": 1,
"duration_ms": 100,
}
@pytest.mark.asyncio
async def test_click_sequence_taps_each_point(actuator):
res = await actuator.click_sequence([(100, 100), (900, 900)], delay_ms=1)
assert res.ok
assert res.message == (
"Sequence clicked successfully: Tapped at [100, 200]; Tapped at [900, 1800]"
)
taps = [h for h in actuator.action_history if h["action"] == "tap"]
assert [(t["x"], t["y"]) for t in taps] == [(100, 200), (900, 1800)]
@pytest.mark.asyncio
async def test_long_press_message_and_duration(actuator):
res = await actuator.long_press(250, 750, duration_ms=1500)
assert res.ok
assert res.message == (
"Long pressed at [250, 750] (normalized) for 1500ms successfully."
)
assert res.duration_ms == 1500
@pytest.mark.asyncio
async def test_swipe_reports_normalized_endpoints(actuator):
res = await actuator.swipe((500, 800), (500, 200), 400)
assert res.ok
assert res.message == "Swipe completed successfully. Swiped from [500, 800] to [500, 200]."
assert res.normalized_coordinates == [500, 800, 500, 200]
@pytest.mark.asyncio
async def test_press_key_forwards_unknown_keycode_to_driver(actuator):
# Arbitrary keycodes (KEYCODE_* names, numeric codes) pass through to the driver,
# matching the historical adb_server behavior of accepting any Android key event.
res = await actuator.press_key("KEYCODE_DPAD_DOWN")
assert res.ok
assert res.message == "Executed key press 'KEYCODE_DPAD_DOWN'."
assert actuator.action_history[-1]["action"] == "press_key"
@pytest.mark.asyncio
async def test_press_key_rejects_empty_key(actuator):
res = await actuator.press_key("")
assert not res.ok
assert res.code is ActionCode.INVALID_ARGS
@pytest.mark.asyncio
async def test_input_text_append_moves_cursor_and_preserves_text(actuator, monkeypatch):
monkeypatch.setattr("asyncio.sleep", _instant_sleep)
res = await actuator.input_text("hello", (500, 300), clear_exist=False)
assert res.ok
assert res.message == "Executed typing 'hello'."
actions = [h["action"] for h in actuator.action_history]
# Append semantics: focus tap, cursor-to-end (KEYCODE_MOVE_END), then type without
# clearing — matching the historical adb_server focus_and_input_text contract.
assert actions == ["tap", "press_key", "input_text"]
typed = actuator.action_history[-1]
assert typed["clear_existing"] is False
cursor = actuator.action_history[-2]
assert cursor["key"] == "123"
@pytest.mark.asyncio
async def test_input_text_clear_erases_then_types_without_reclearing(actuator, monkeypatch):
monkeypatch.setattr("asyncio.sleep", _instant_sleep)
res = await actuator.input_text("hello", (500, 300), clear_exist=True)
assert res.ok
actions = [h["action"] for h in actuator.action_history]
# The controller's full clear runs as a shell keyevent batch on this driver.
assert actions == ["tap", "execute_shell", "input_text"]
assert actuator.action_history[-1]["clear_existing"] is False
async def _instant_sleep(_delay, *args, **kwargs):
return None
@pytest.mark.asyncio
async def test_wait_for_delay(actuator):
res = await actuator.wait_for_delay(1)
assert res.ok
assert res.message == "Waited for 1ms successfully."
def test_partial_capabilities_are_reported():
partial = MockActuator(capabilities=frozenset({"click_sequence"}))
assert partial.capabilities() == frozenset({"click_sequence"})
@pytest.mark.asyncio
async def test_observation_primitives(actuator):
shot = await actuator.take_screenshot()
assert isinstance(shot, str) and len(shot) > 0
data = await actuator.get_screen_data()
assert data.width == 1000 and data.height == 2000
+47 -18
View File
@@ -20,8 +20,10 @@ subprocess stdin isolation against pipe hijacking, and stdout purity.
import io import io
import json import json
import os import os
import queue
import subprocess import subprocess
import sys import sys
import threading
import time import time
from unittest.mock import MagicMock, patch from unittest.mock import MagicMock, patch
@@ -34,6 +36,31 @@ from artemis.utils.logger import get_logger
from mcp_server.utils import device_utils, env_utils from mcp_server.utils import device_utils, env_utils
def _readline_with_timeout(pipe, timeout: float) -> str | None:
"""Read one line from a subprocess pipe with a timeout, portably.
select.select only supports sockets on Windows, so poll via a reader thread
instead of selecting on the pipe handle.
"""
result: queue.Queue = queue.Queue()
def _reader():
try:
result.put(pipe.readline())
except Exception:
result.put(b"")
reader = threading.Thread(target=_reader, daemon=True)
reader.start()
try:
line = result.get(timeout=timeout)
except queue.Empty:
return None
if not line:
return None
return line.decode("utf-8").strip()
def test_mcp_stdio_handshake_immediate_input(): def test_mcp_stdio_handshake_immediate_input():
"""Verify that MCP server over stdio responds to an immediate initialize request """Verify that MCP server over stdio responds to an immediate initialize request
@@ -51,6 +78,24 @@ def test_mcp_stdio_handshake_immediate_input():
"PYTHONUNBUFFERED": "1", "PYTHONUNBUFFERED": "1",
"PATH": os.environ.get("PATH", ""), "PATH": os.environ.get("PATH", ""),
"PYTHONPATH": project_root, "PYTHONPATH": project_root,
# Windows: the interpreter's socket stack (Winsock/_overlapped),
# Path.home(), and tempfile handling need these system variables.
**{
key: os.environ[key]
for key in (
"SYSTEMROOT",
"SYSTEMDRIVE",
"WINDIR",
"TEMP",
"TMP",
"USERPROFILE",
"HOMEDRIVE",
"HOMEPATH",
"APPDATA",
"LOCALAPPDATA",
)
if key in os.environ
},
}, },
cwd=project_root, cwd=project_root,
) )
@@ -71,15 +116,7 @@ def test_mcp_stdio_handshake_immediate_input():
p.stdin.flush() p.stdin.flush()
# Read initialize response with a strict timeout # Read initialize response with a strict timeout
start = time.time() init_resp_line = _readline_with_timeout(p.stdout, 6.0)
init_resp_line = None
while time.time() - start < 6.0:
import select
r, _, _ = select.select([p.stdout], [], [], 0.2)
if r:
init_resp_line = p.stdout.readline().decode("utf-8").strip()
break
assert init_resp_line is not None, "MCP server failed to respond to initialize request within 6 seconds (deadlock detected)!" assert init_resp_line is not None, "MCP server failed to respond to initialize request within 6 seconds (deadlock detected)!"
init_data = json.loads(init_resp_line) init_data = json.loads(init_resp_line)
@@ -93,15 +130,7 @@ def test_mcp_stdio_handshake_immediate_input():
p.stdin.write(json.dumps(tools_req).encode("utf-8") + b"\n") p.stdin.write(json.dumps(tools_req).encode("utf-8") + b"\n")
p.stdin.flush() p.stdin.flush()
tools_resp_line = None tools_resp_line = _readline_with_timeout(p.stdout, 4.0)
start = time.time()
while time.time() - start < 4.0:
import select
r, _, _ = select.select([p.stdout], [], [], 0.2)
if r:
tools_resp_line = p.stdout.readline().decode("utf-8").strip()
break
assert tools_resp_line is not None, "MCP server failed to respond to tools/list request!" assert tools_resp_line is not None, "MCP server failed to respond to tools/list request!"
tools_data = json.loads(tools_resp_line) tools_data = json.loads(tools_resp_line)
+97
View File
@@ -0,0 +1,97 @@
"""Tests for the spawn watchdog guarding mobile_run_task's standalone fallback."""
import json
import os
from unittest.mock import patch
from mcp_server.tools import task_runner as spawn_tool
def _write_status(trace_dir: str, trace_id: str, status: str) -> None:
os.makedirs(trace_dir, exist_ok=True)
with open(os.path.join(trace_dir, "status.json"), "w", encoding="utf-8") as f:
json.dump({"trace_id": trace_id, "status": status}, f)
def _patch_trace_dir(tmp_path):
return patch.object(spawn_tool.trace_store, "TRACES_DIR", str(tmp_path))
def test_watchdog_passes_when_runner_creates_logs(tmp_path):
trace_id = "wd-boot-ok"
trace_dir = tmp_path / trace_id
_write_status(str(trace_dir), trace_id, "running")
(trace_dir / "stdout.log").write_text("booted", encoding="utf-8")
with _patch_trace_dir(tmp_path), patch.object(spawn_tool, "_kill_process_tree") as kill:
assert spawn_tool._watch_spawn(
trace_id, pid=999999, queue_ticket="t", conversation_id=None,
deadline_seconds=0.2, poll_interval=0.05,
)
kill.assert_not_called()
def test_watchdog_skips_already_terminal_task(tmp_path):
trace_id = "wd-cancelled"
trace_dir = tmp_path / trace_id
_write_status(str(trace_dir), trace_id, "cancelled")
with _patch_trace_dir(tmp_path), patch.object(spawn_tool, "_kill_process_tree") as kill:
assert spawn_tool._watch_spawn(
trace_id, pid=999999, queue_ticket="t", conversation_id=None,
deadline_seconds=0.2, poll_interval=0.05,
)
kill.assert_not_called()
def test_watchdog_kills_and_fails_hung_runner(tmp_path):
trace_id = "wd-hung"
trace_dir = tmp_path / trace_id
_write_status(str(trace_dir), trace_id, "running")
with (
_patch_trace_dir(tmp_path),
patch.object(spawn_tool, "_kill_process_tree") as kill,
patch.object(spawn_tool.DeviceExecutionLock, "cancel_reservation") as cancel,
patch.object(spawn_tool, "notify") as notifier,
):
assert not spawn_tool._watch_spawn(
trace_id, pid=424242, queue_ticket="ticket-1", conversation_id="conv-1",
deadline_seconds=0.2, poll_interval=0.05,
)
kill.assert_called_once_with(424242)
cancel.assert_called_once_with("ticket-1")
notifier.assert_called_once()
with open(trace_dir / "status.json", encoding="utf-8") as f:
status = json.load(f)
assert status["status"] == "failed"
assert "no logs" in status.get("error", "")
def test_watchdog_tolerates_logs_appearing_at_deadline(tmp_path):
"""Logs that appear between the last poll and the deadline must not cause a kill."""
trace_id = "wd-late-boot"
trace_dir = tmp_path / trace_id
_write_status(str(trace_dir), trace_id, "running")
real_read_status = spawn_tool.trace_store.read_status
calls = {"n": 0}
def read_status_and_create_log(tid):
calls["n"] += 1
if calls["n"] == 1:
(trace_dir / "stderr.log").write_text("late boot", encoding="utf-8")
return real_read_status(tid)
with (
_patch_trace_dir(tmp_path),
patch.object(spawn_tool.trace_store, "read_status", side_effect=read_status_and_create_log),
patch.object(spawn_tool, "_kill_process_tree") as kill,
):
assert spawn_tool._watch_spawn(
trace_id, pid=999999, queue_ticket="t", conversation_id=None,
deadline_seconds=0.15, poll_interval=0.2,
)
kill.assert_not_called()
+133
View File
@@ -0,0 +1,133 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Regression tests for mobile_manage_task's state reconciliation.
Guards the invariants restored after the reconcile rewrite:
- a DB terminal verdict never overrides a status.json that already reached its
own terminal state (a completed task must keep its result),
- except to correct a previously liveness-inferred failure,
- and a task without a recorded pid (daemon dispatch) is never inferred dead.
"""
import sqlite3
import time
import pytest
import mcp_server.tools.task_manager as task_manager
from mcp_server.tools.task_manager import _LIVENESS_FAILURE_ERROR, _reconcile_task_state
@pytest.fixture
def no_persistence(monkeypatch):
"""Keep reconciliation in-memory: no DB, no status writes, no lock queries."""
monkeypatch.setattr(task_manager, "_find_data_engine_db", lambda: None)
monkeypatch.setattr(task_manager, "_session_tracked_by_lock", lambda _tid: False)
monkeypatch.setattr(task_manager.trace_store, "write_status", lambda *_a, **_k: None)
monkeypatch.setattr(
task_manager.trace_store, "update_trace_status", lambda *_a, **_k: None
)
monkeypatch.setattr(task_manager, "notify", lambda *_a, **_k: None)
def _db_with_session(tmp_path, trace_id: str, status: str, pid: int | None = 4242):
db_path = tmp_path / "artemis.db"
conn = sqlite3.connect(db_path)
conn.execute(
"CREATE TABLE sessions (session_id TEXT, status TEXT, pid INTEGER, start_time REAL)"
)
conn.execute(
"INSERT INTO sessions VALUES (?, ?, ?, ?)", (trace_id, status, pid, time.time())
)
conn.commit()
conn.close()
return str(db_path)
@pytest.fixture
def use_db(tmp_path, monkeypatch, no_persistence):
def _install(trace_id: str, status: str, pid: int | None = 4242):
db_path = _db_with_session(tmp_path, trace_id, status, pid)
monkeypatch.setattr(task_manager, "_find_data_engine_db", lambda: db_path)
return _install
def test_db_failed_never_overrides_completed_status(use_db):
use_db("t1", "failed")
status_data = {"status": "completed", "result": "the answer", "pid": 111}
current_status, _pid, is_alive = _reconcile_task_state("t1", status_data)
assert current_status == "completed"
assert status_data["status"] == "completed"
assert status_data["result"] == "the answer"
assert is_alive is False
def test_db_verdict_corrects_liveness_inferred_failure(use_db):
use_db("t2", "completed")
status_data = {"status": "failed", "error": _LIVENESS_FAILURE_ERROR, "pid": 111}
current_status, _pid, _is_alive = _reconcile_task_state("t2", status_data)
assert current_status == "completed"
assert status_data["error"] is None
def test_db_never_overrides_user_cancellation(use_db):
use_db("t3", "completed")
status_data = {"status": "cancelled", "pid": 111}
current_status, _pid, _is_alive = _reconcile_task_state("t3", status_data)
assert current_status == "cancelled"
def test_pidless_running_task_is_assumed_alive(no_persistence):
# Daemon dispatch writes no pid into status.json; liveness must not be
# inferred even long past the startup grace window.
status_data = {"status": "running", "start_time": time.time() - 3600}
current_status, _pid, is_alive = _reconcile_task_state("t4", status_data)
assert current_status == "running"
assert is_alive is True
assert status_data.get("error") is None
def test_dead_pid_past_grace_is_marked_failed(no_persistence, monkeypatch):
monkeypatch.setattr(task_manager, "_pid_alive", lambda _pid: False)
status_data = {
"status": "running",
"pid": 99999,
"start_time": time.time() - 3600,
}
current_status, _pid, is_alive = _reconcile_task_state("t5", status_data)
assert current_status == "failed"
assert status_data["error"] == _LIVENESS_FAILURE_ERROR
assert is_alive is False
def test_dead_pid_within_grace_is_assumed_alive(no_persistence, monkeypatch):
monkeypatch.setattr(task_manager, "_pid_alive", lambda _pid: False)
status_data = {"status": "running", "pid": 99999, "start_time": time.time()}
current_status, _pid, is_alive = _reconcile_task_state("t6", status_data)
assert current_status == "running"
assert is_alive is True
+4 -3
View File
@@ -14,6 +14,7 @@
"""Unit tests for MCP trace store.""" """Unit tests for MCP trace store."""
import os
import shutil import shutil
import tempfile import tempfile
import uuid import uuid
@@ -80,6 +81,6 @@ def test_read_nonexistent_status(temp_trace_env):
def test_trace_paths(temp_trace_env): def test_trace_paths(temp_trace_env):
trace_id = "test-paths-id" trace_id = "test-paths-id"
trace_dir = trace_store.get_trace_dir(trace_id) trace_dir = trace_store.get_trace_dir(trace_id)
assert trace_store.get_trace_notes_dir(trace_id) == f"{trace_dir}/notes" assert trace_store.get_trace_notes_dir(trace_id) == os.path.join(trace_dir, "notes")
assert trace_store.get_trace_stdout_log_path(trace_id) == f"{trace_dir}/stdout.log" assert trace_store.get_trace_stdout_log_path(trace_id) == os.path.join(trace_dir, "stdout.log")
assert trace_store.get_trace_stderr_log_path(trace_id) == f"{trace_dir}/stderr.log" assert trace_store.get_trace_stderr_log_path(trace_id) == os.path.join(trace_dir, "stderr.log")
+66 -1
View File
@@ -222,7 +222,9 @@ def test_cli_mcp_install_all(tmp_path, monkeypatch):
assert (tmp_path / ".gemini" / "rules" / "artemis.md").exists() assert (tmp_path / ".gemini" / "rules" / "artemis.md").exists()
assert (tmp_path / ".cursorrules").exists() assert (tmp_path / ".cursorrules").exists()
assert (tmp_path / ".cursor" / "rules" / "artemis.mdc").exists() assert (tmp_path / ".cursor" / "rules" / "artemis.mdc").exists()
assert (tmp_path / ".claude" / "CLAUDE.md").exists() # Claude Code loads both CLAUDE.md and rules/*.md, so rules are installed
# only as the standalone rule file to avoid duplicated context.
assert not (tmp_path / ".claude" / "CLAUDE.md").exists()
assert (tmp_path / ".claude" / "rules" / "artemis.md").exists() assert (tmp_path / ".claude" / "rules" / "artemis.md").exists()
assert (tmp_path / ".codeium" / "windsurf" / "memories" / "global_rules.md").exists() assert (tmp_path / ".codeium" / "windsurf" / "memories" / "global_rules.md").exists()
assert (tmp_path / ".codeium" / "windsurf" / "rules" / "artemis.md").exists() assert (tmp_path / ".codeium" / "windsurf" / "rules" / "artemis.md").exists()
@@ -236,6 +238,69 @@ def test_cli_mcp_install_all(tmp_path, monkeypatch):
assert (tmp_path / ".codex" / "AGENTS.md").exists() assert (tmp_path / ".codex" / "AGENTS.md").exists()
def test_cli_mcp_install_claude_migrates_legacy_claude_md(tmp_path, monkeypatch):
"""Verify the claude target removes a previously injected CLAUDE.md block while preserving user content."""
monkeypatch.setattr("pathlib.Path.home", lambda: tmp_path)
monkeypatch.setenv("APPDATA", str(tmp_path / "AppData" / "Roaming"))
monkeypatch.setattr("mcp_server.utils.env_utils.get_project_root", lambda: str(tmp_path))
claude_dir = tmp_path / ".claude"
claude_dir.mkdir(parents=True)
claude_md = claude_dir / "CLAUDE.md"
claude_md.write_text(
"# My own instructions\n\n"
"<!-- BEGIN ARTEMIS MOBILE TESTING RULES -->\nold injected rules\n"
"<!-- END ARTEMIS MOBILE TESTING RULES -->\n",
encoding="utf-8",
)
result = runner.invoke(app, ["mcp", "--install", "claude"])
assert result.exit_code == 0
remaining = claude_md.read_text(encoding="utf-8")
assert "My own instructions" in remaining
assert "ARTEMIS MOBILE TESTING RULES" not in remaining
rule_file = claude_dir / "rules" / "artemis.md"
assert rule_file.exists()
assert "Mobile Testing Mindset" in rule_file.read_text(encoding="utf-8")
def test_cli_mcp_install_claude_deletes_block_only_claude_md(tmp_path, monkeypatch):
"""Verify a CLAUDE.md consisting solely of the injected block is deleted outright."""
monkeypatch.setattr("pathlib.Path.home", lambda: tmp_path)
monkeypatch.setenv("APPDATA", str(tmp_path / "AppData" / "Roaming"))
monkeypatch.setattr("mcp_server.utils.env_utils.get_project_root", lambda: str(tmp_path))
claude_dir = tmp_path / ".claude"
claude_dir.mkdir(parents=True)
claude_md = claude_dir / "CLAUDE.md"
claude_md.write_text(
"<!-- BEGIN ARTEMIS MOBILE TESTING RULES -->\nold injected rules\n"
"<!-- END ARTEMIS MOBILE TESTING RULES -->\n",
encoding="utf-8",
)
result = runner.invoke(app, ["mcp", "--install", "claude"])
assert result.exit_code == 0
assert not claude_md.exists()
assert (claude_dir / "rules" / "artemis.md").exists()
def test_cli_mcp_install_claude_refuses_unparseable_claude_json(tmp_path, monkeypatch):
"""Verify an unparseable ~/.claude.json is left untouched instead of being rewritten."""
monkeypatch.setattr("pathlib.Path.home", lambda: tmp_path)
monkeypatch.setenv("APPDATA", str(tmp_path / "AppData" / "Roaming"))
monkeypatch.setattr("mcp_server.utils.env_utils.get_project_root", lambda: str(tmp_path))
corrupt = "{ this is not json at all"
claude_json = tmp_path / ".claude.json"
claude_json.write_text(corrupt, encoding="utf-8")
result = runner.invoke(app, ["mcp", "--install", "claude"])
assert result.exit_code == 0
assert claude_json.read_text(encoding="utf-8") == corrupt
def test_cli_mcp_install_codex_preserves_config_and_is_idempotent(tmp_path, monkeypatch): def test_cli_mcp_install_codex_preserves_config_and_is_idempotent(tmp_path, monkeypatch):
"""Verify Codex TOML merging preserves other servers and updates only Artemis.""" """Verify Codex TOML merging preserves other servers and updates only Artemis."""
monkeypatch.setattr("pathlib.Path.home", lambda: tmp_path) monkeypatch.setattr("pathlib.Path.home", lambda: tmp_path)