feat: extract in-process MCP action layer, concurrent task dispatch, and spawn/reconcile hardening

- Extract device action execution into artemis/mcp action layer:
  typed ActionResult, capability manifest with startup validation,
  AdbActuator as the single physical implementation, in-memory
  ActionSession (replaces stdio subprocess MCP in validator), and
  McpActionExecutor preserving the legacy executor contract.
  adb_server keeps its external stdio/SSE surface unchanged and now
  reuses the actuator implementation; import-time side effects moved
  into configure_stdio_mode().
- Capability-gated prompt assembly (prompt_assembly.py): tool enums and
  teaching segments render per available backend tools; full-set output
  is byte-identical to the previous static text (locked by tests).
- Admin console task queue: serial worker replaced by a per-device
  concurrent dispatcher (ARTEMIS_CONCURRENCY_MODE=global restores
  serial semantics), per-session cancellation, strict device binding on
  enqueue, unified stdio piping with tee to trace stdout.log, and
  shutdown cleanup of all active runs. Fix _pid_exists exception order
  so PermissionError is treated as alive.
- mcp_server reliability: spawn watchdog kills runners stuck before
  interpreter startup and releases device reservations; task state
  reconcile uses psutil liveness, prefers runner-reported pids, and
  adds a 45s startup grace to stop false failure reports on Windows;
  conversation_id is now optional end-to-end; explicit device_serial is
  validated at submission.
- CLI install now writes ~/.claude/rules/artemis.md for Claude Code and
  migrates/removes the legacy CLAUDE.md injection block; ~/.claude.json
  is never rewritten when it fails to parse.
- Tests: Windows-portable stdio lifecycle and path assertions; new
  suites for action layer, prompt assembly, spawn watchdog, and task
  reconcile (895 passed, 3 skipped).
This commit is contained in:
weilinliu55 committed 2026-08-27 18:25:27 -07:00
1 parent 60312abbb0
commit cbdca1497c
52 files changed
+5735 -1649

No files matched your search

+2 -2
View File
@@ -15,7 +15,7 @@ You operate in a dynamic observation-reasoning-execution loop. **CRITICAL RULE:
# 2. DYNAMIC ENVIRONMENT & TIME AWARENESS
Keep in mind that the mobile device is an asynchronous, constantly evolving environment. However, your actions entail an inherent "turn latency" (a 3–7 second delay caused by model inference and network transmission between normal consecutive actions).
- **State Drift & Progression Tolerance**: While you think, video playback advances, banners scroll, and web pages finish loading. If the latest observation has progressed further than expected (e.g., an ad automatically skipped or a redirect landed), **do not treat it as an error**. Adapt smoothly to the live screen in front of you.
- **Timed Waiting & Transitions**: When facing mid-transition animations, loading spinners, required watch durations, or ad countdown timers, use the precise `wait_for_delay` command to allow time to pass or the UI state to stabilize. Some transition states are subtle and may even resemble a phone bug or system freeze; it is always advisable to wait first, and only consider retrying or choosing an alternative path if the state remains broken.
{% if "wait_for_delay" in available_tools %}- **Timed Waiting & Transitions**: When facing mid-transition animations, loading spinners, required watch durations, or ad countdown timers, use the precise `wait_for_delay` command to allow time to pass or the UI state to stabilize. Some transition states are subtle and may even resemble a phone bug or system freeze; it is always advisable to wait first, and only consider retrying or choosing an alternative path if the state remains broken.{% endif %}
- **Time-Sensitive Tasks & Transient UI**: First, you must understand that each of your decision turns takes several seconds of model inference and network delay, while the mobile device state is continuously changing. A target element that you see on the current screenshot (e.g., auto-dismissing overlays, temporary message bars, menus or control bars with short timeouts) may have already vanished by the time your physical click command reaches the device! This is one of the most common reasons why the UI does not change as expected after a single step. We call these **Time-Sensitive Tasks**.
- *Mental Model*: When facing these tasks, any attempt to interact step-by-step (such as tapping once to wake up a menu/bar and waiting for the next turn to tap the target button) WILL fail because the element will auto-dismiss during your decision delay. You MUST combine the summon action on the trigger element (`Trigger`) and the target action (`Target`) into a single atomic sequence.
- *Chained Execution*: Use `click_sequence(sequence=[[trigger_x, trigger_y], [target_x, target_y]])` in `0-1000` normalized scale to execute both taps consecutively with a 50ms delay, catching the transient element before the timeout closes it.
@@ -32,7 +32,7 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
1. **Normalized Coordinates (`0-1000` Scale)**: Always use normalized `[x, y]` coordinates in `0-1000` scale for any location-based target (e.g., `target=[500, 600]` or `click_sequence=[[300, 400], [500, 600]]`).
2. **Do not determine element coordinates based on guesswork.** If the element is visible in the screenshot but its coordinates are missing from the provided XML tree, please use the `ask_explorer` tool to locate the element first.
3. **Screenshot Pruning Awareness**: To optimize speed, the system **prunes old intermediate screenshots from your history**—you only ever see the *latest* live screenshot and UI tree. Therefore, if you spot critical text in an earlier step that you will need later (e.g., verification codes, specific titles, search lists), **you must explicitly verbalize and write those details into your text thinking/notes** so they persist across turns.
4. **App Launching**: To start or reset an application, always use `manage_app(action="launch", app_name="...")` or `action="stop"`. Do not manually swipe across home screens.
{% if "manage_app" in available_tools %}4. **App Launching**: To start or reset an application, always use `manage_app(action="launch", app_name="...")` or `action="stop"`. Do not manually swipe across home screens.{% endif %}
---
+24 -8
View File
@@ -36,10 +36,8 @@ from langchain_core.messages import (
from artemis.agents.flash.context_compressor import compress_flash_messages
from artemis.agents.flash.summarizer import VisualStepSummarizer
from artemis.agents.validator.tool_declarations import (
ACTION_TOOL_NAMES,
ASK_EXPLORER_TOOL,
CLICK_SEQUENCE_TOOL,
MobileActionExecutor,
REPORT_TASK_STATUS_TOOL,
VALIDATOR_TOOLS_DECLARATION,
capture_screenshot_and_parse_ui,
@@ -51,6 +49,7 @@ from artemis.controllers.unified_controller import UnifiedMobileController
from artemis.data_engine.trace import trace
from artemis.graph.perception import _check_injected_instruction_file
from artemis.graph.state import State
from artemis.mcp.action_executor import McpActionExecutor
from artemis.services.llm import (
RobustChatModelWrapper,
get_google_llm,
@@ -78,7 +77,7 @@ class FlashRunner:
self.step_summarizer_cfg = StepSummarizerConfig()
self.controller = UnifiedMobileController(ctx)
self.executor = MobileActionExecutor(ctx, self.controller)
self.executor = McpActionExecutor(ctx, self.controller)
self.summarizer = (
VisualStepSummarizer(ctx, model_name=self.step_summarizer_cfg.model)
if self.step_summarizer_cfg.enabled
@@ -90,6 +89,14 @@ class FlashRunner:
tools.insert(1, CLICK_SEQUENCE_TOOL)
tools.append(ASK_EXPLORER_TOOL)
tools.append(REPORT_TASK_STATUS_TOOL)
# With an actuator installed, drop declarations for device actions the
# backend does not implement (and append its extension tools); without
# one, the full historical declaration set is kept.
actuator = getattr(self.ctx, "actuator", None)
if actuator is not None:
from artemis.mcp.action_manifest import filter_declarations
tools = filter_declarations(tools, actuator, "flash")
return tools
def _prune_intermediate_screenshots(self, messages: list[BaseMessage]) -> None:
@@ -144,10 +151,15 @@ class FlashRunner:
}
)
# Render System Prompt
# Render System Prompt. Tool-teaching segments are gated on the available
# tool set so an absent tool leaves no trace in the prompt; until an actuator
# is wired in, the full manifest set reproduces the historical prompt.
prompt_path = Path(__file__).parent / "flash_runner.md"
prompt_template = prompt_path.read_text(encoding="utf-8")
system_prompt = Template(prompt_template).render(goal=self.goal)
available_tools = frozenset(t.name for t in tools_declaration)
system_prompt = Template(prompt_template).render(
goal=self.goal, available_tools=available_tools
)
messages: list[BaseMessage] = [
SystemMessage(content=system_prompt),
@@ -432,8 +444,12 @@ class FlashRunner:
try:
exec_result = await self.executor.execute(name, args, tc_id, state)
# Dynamic dispatch set: manifest device actions plus any backend
# extension tools, so extension steps are recorded like actions.
action_names = self.executor.action_tool_names
post_img_bytes = exec_result.screenshot_bytes
if not post_img_bytes and name in ACTION_TOOL_NAMES:
if not post_img_bytes and name in action_names:
try:
controller = UnifiedMobileController(self.ctx)
screen_data = await controller.get_screen_data()
@@ -447,7 +463,7 @@ class FlashRunner:
# Record telemetry / step in DataEngine
recorded_step_id = None
if self.ctx.data_engine and name in ACTION_TOOL_NAMES:
if self.ctx.data_engine and name in action_names:
try:
if self.ctx.data_engine.current_step_id is None:
self.ctx.data_engine.allocate_step_id()
@@ -517,7 +533,7 @@ class FlashRunner:
logger.warning(f"Error recording step in FlashRunner: {step_err}")
# ⚡ Non-blocking dispatch of objective visual transition summarizer
if self.summarizer and name in ACTION_TOOL_NAMES:
if self.summarizer and name in action_names:
action_sequence += 1
self.summarizer.dispatch(
step_number=action_sequence,
+44 -25
View File
@@ -114,6 +114,30 @@ class OperatorNode:
logger.error(f"Failed to load operator prompts: {e}")
self.prompts = {}
def _available_device_actions(self) -> frozenset[str]:
"""Device actions the installed actuator backend provides.
Without an explicit ``ctx.actuator`` the full manifest set applies (the
default AdbActuator implements everything), so behavior is unchanged until a
partial backend is actually installed.
"""
from artemis.mcp.action_manifest import (
OPTIONAL_ACTIONS,
REQUIRED_ACTIONS,
available_device_actions,
)
actuator = getattr(self.ctx, "actuator", None)
if actuator is not None and callable(getattr(actuator, "capabilities", None)):
try:
extension_names = frozenset(
e.name for e in actuator.extensions() if "operator" in e.targets
)
return available_device_actions(actuator) | extension_names
except Exception as e:
logger.warning(f"Failed to read actuator capabilities: {e}")
return REQUIRED_ACTIONS | OPTIONAL_ACTIONS
def _check_infinite_loop(self, state: State):
subagent_calls = state.subagent_calls or []
if len(subagent_calls) >= 3:
@@ -386,15 +410,9 @@ class OperatorNode:
current_messages.append(response)
action_tool_names = [
"click",
"input_text",
"swipe",
"press_key",
"manage_app",
"wait_for_delay",
"long_press",
]
# Manifest-driven, so a backend's extension actions are classified as
# turn-ending device actions without touching this file.
action_tool_names = sorted(self._available_device_actions())
def normalize_name(name: str) -> str:
return name.split(":")[-1] if ":" in name else name
@@ -804,23 +822,24 @@ class OperatorNode:
current_step_num = len(steps) + 1
# 5. Prepare Tools
click_tool = self._get_click_tool()
input_text_tool = self._get_input_text_tool()
swipe_tool = self._get_swipe_tool()
press_key_tool = self._get_press_key_tool()
manage_app_tool = self._get_manage_app_tool()
wait_for_delay_tool = self._get_wait_for_delay_tool()
long_press_tool = self._get_long_press_tool()
# 5. Prepare Tools. Device-action declarations are assembled against the
# installed actuator backend's capabilities: an action the backend does not
# implement is simply never declared (and the prompt assembly drops its
# teaching segments in lockstep).
available_actions = self._available_device_actions()
action_tool_factories = {
"click": self._get_click_tool,
"input_text": self._get_input_text_tool,
"swipe": self._get_swipe_tool,
"press_key": self._get_press_key_tool,
"manage_app": self._get_manage_app_tool,
"wait_for_delay": self._get_wait_for_delay_tool,
"long_press": self._get_long_press_tool,
}
all_tools = [
click_tool,
input_text_tool,
swipe_tool,
press_key_tool,
manage_app_tool,
wait_for_delay_tool,
long_press_tool,
factory()
for name, factory in action_tool_factories.items()
if name in available_actions
] + self.tools
if self._has_checker_feedback():
+8 -2
View File
@@ -39,12 +39,18 @@ class OperatorPromptBuilder:
"""Builds prompt messages and multimodal contents for the Operator Agent."""
@classmethod
def build_system_message(cls, template_name: str = "main_template") -> str:
def build_system_message(
cls,
template_name: str = "main_template",
available_tools: frozenset[str] | None = None,
) -> str:
from artemis.agents.operator.prompts import apply_operator_prompt_contract
prompts = load_operator_prompts()
try:
return apply_operator_prompt_contract(prompts[template_name])
return apply_operator_prompt_contract(
prompts[template_name], available_tools=available_tools
)
except KeyError as exc:
raise ValueError(f"Unknown Operator template: {template_name}") from exc
+206 -13
View File
@@ -33,21 +33,139 @@ from artemis.utils.task_tree import get_active_subgoal_hashes
logger = get_logger(__name__)
_LEGACY_TOOL_CALLING_RULE = """- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action."""
from artemis.agents.prompt_assembly import (
gate_segment,
render_tool_enum,
resolve_available,
)
_TOOL_CALLING_RULE = """- **Tool Calling Contract**: Tools that return information needed for the decision (`ask_explorer`, `ask_diagnoser`, `video_analyzer`, `run_adb_command`, `manage_task`, `analyze_task_output`, `read_note`, `list_notes`, and `save_note`) are result-dependent Pre-Decision Tools. Do not submit them in the same tool-call list as a Turn-Ending Action; inspect their result first. The write-through tools `update_note` and `append_note` may accompany at most one Turn-Ending Action when their content is based entirely on evidence already available before that action. Never write an action's expected outcome as if it had already been observed."""
_LEGACY_TOOL_CALLING_RULE = """- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action."""
_LEGACY_TOOL_PROTOCOL = """1. **Pre-Decision Exploratory Tools**:
- *What they are*: Helper/Subagent tools (such as `ask_explorer`, `ask_diagnoser`, `video_analyzer`), ADB command tools (`run_adb_command`, `manage_task`), and memory note tools (`read_note`, `list_notes`, `save_note`, `update_note`, `append_note`).
- *How they work*: These tools gather details or update/read memory. When you invoke these tools, the framework will immediately execute them and return the results, allowing you to continue thinking and make your final decision.
2. **Turn-Ending Actions**:"""
_TOOL_PROTOCOL = """1. **Result-Dependent Pre-Decision Tools**:
- *What they are*: Helper/Subagent tools (`ask_explorer`, `ask_diagnoser`, `video_analyzer`), ADB/task tools (`run_adb_command`, `manage_task`, `analyze_task_output`), and memory tools whose result must be inspected (`read_note`, `list_notes`, `save_note`).
- *How they work*: Invoke these without a Turn-Ending Action, inspect the returned result, and then decide.
2. **Write-Through Memory Tools**:
- `update_note` and `append_note` may run alongside at most one Turn-Ending Action only when recording facts already observed in the current context. If the note content depends on the action's result, wait for the next observation before writing it.
3. **Turn-Ending Actions**:"""
# --- Assembled tool references -------------------------------------------------------
# The orderings below reproduce the historical prompt wording exactly; with the full
# tool set the rendered output is byte-identical to the previous literals. See
# artemis/agents/prompt_assembly.py for the assembly rationale.
_PRE_DECISION_HELPER_TOOLS = ("ask_explorer", "ask_diagnoser", "video_analyzer")
_PRE_DECISION_ADB_TOOLS = ("run_adb_command", "manage_task", "analyze_task_output")
_PRE_DECISION_MEMORY_TOOLS = ("read_note", "list_notes", "save_note")
_PRE_DECISION_ALL_TOOLS = (
_PRE_DECISION_HELPER_TOOLS + _PRE_DECISION_ADB_TOOLS + _PRE_DECISION_MEMORY_TOOLS
)
# The two device-action enumeration slots in operator.json (identical in both
# templates), in their historical orders.
_PHYSICAL_ACTIONS_ORDER = (
"click",
"input_text",
"swipe",
"press_key",
"manage_app",
"wait_for_delay",
"long_press",
)
_TURN_ENDING_ORDER = (
"click",
"swipe",
"input_text",
"long_press",
"press_key",
"manage_app",
"wait_for_delay",
)
_PHYSICAL_ACTIONS_SLOT = (
"Physical device actions (`click`, `input_text`, `swipe`, `press_key`,"
" `manage_app`, `wait_for_delay`, `long_press`)"
)
_TURN_ENDING_SLOT = (
"Turn-Ending Action (`click`, `swipe`, `input_text`, `long_press`,"
" `press_key`, `manage_app`, or `wait_for_delay`)"
)
# Self-contained instruction segments that teach a single tool. Removed wholesale when
# the tool is unavailable, so nothing in the prompt dangles.
_WAIT_FOR_DELAY_TRANSITIONS_SEGMENT = (
"- **Loading & Transitions**: When entering page transitions (e.g., app launching,"
" web page loading, or progress indicators), use `wait_for_delay` to let the screen"
" settle before executing target actions. Some transition states are difficult to"
" discern, such as a thin progress line at the top of the browser screen during"
" webpage loads. Before questioning whether an error has occurred, you can first"
" adopt a waiting strategy to see how the device state evolves. If the state"
" remains ambiguous after waiting, certain UI components might be malfunctioning or"
" failing to load; in this case, consider retrying the action or pivoting to an"
" alternative path.\n"
)
_WAIT_FOR_DELAY_INTERVAL_PARENTHETICAL = (
" (convert minutes to milliseconds accurately for `wait_for_delay`)"
)
_VIDEO_ANALYZER_TRANSIENT_SEGMENT = (
" - *Missed Transient States*: If the current state jumped ahead and you suspect"
" you missed an important initial or intermediate screen (e.g., a number or flash"
" message that was overwritten by your last action), consider using the"
" `video_analyzer` tool to look back at the recent history.\n"
)
_VIDEO_ANALYZER_SNAPSHOT_SENTENCE = (
" If you have already missed certain critical information and cannot re-view it by"
" going back or other operations, you can use the `video_analyzer` to look back at"
" historical information."
)
# Tail of the "Action Scope & Interaction Precision" bullet: everything past the two
# generic gesture-scope sentences is swipe/drag teaching.
_SWIPE_PRECISION_SEGMENT = (
" For any task requiring continuous range interactions, sliding, or boundary"
" selection, prefer using targeted swipe/drag gestures spanning from the element's"
" start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is"
" strictly confined to the target element. For drag-and-drop, list reordering, or"
" sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag"
" slightly PAST the target position to overcome touch slop and reliably trigger the"
" drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%),"
" always swipe fully to the extreme boundary to guarantee reaching the absolute"
" limits."
)
#: Every tool name the operator prompt slots can reference. ``available_tools=None``
#: resolves to this set, preserving the historical output.
OPERATOR_PROMPT_TOOLSET: frozenset[str] = frozenset(
_PRE_DECISION_ALL_TOOLS + _PHYSICAL_ACTIONS_ORDER
)
def _tool_calling_rule(available: frozenset[str]) -> str:
tools = render_tool_enum(_PRE_DECISION_ALL_TOOLS, available, final_sep="and")
return (
"- **Tool Calling Contract**: Tools that return information needed for the"
f" decision ({tools}) are result-dependent Pre-Decision Tools. Do not submit"
" them in the same tool-call list as a Turn-Ending Action; inspect their result"
" first. The write-through tools `update_note` and `append_note` may accompany"
" at most one Turn-Ending Action when their content is based entirely on"
" evidence already available before that action. Never write an action's"
" expected outcome as if it had already been observed."
)
def _tool_protocol(available: frozenset[str]) -> str:
helper = render_tool_enum(_PRE_DECISION_HELPER_TOOLS, available)
adb = render_tool_enum(_PRE_DECISION_ADB_TOOLS, available)
memory = render_tool_enum(_PRE_DECISION_MEMORY_TOOLS, available)
return (
"1. **Result-Dependent Pre-Decision Tools**:\n"
f" - *What they are*: Helper/Subagent tools ({helper}), ADB/task tools"
f" ({adb}), and memory tools whose result must be inspected ({memory}).\n"
" - *How they work*: Invoke these without a Turn-Ending Action, inspect the"
" returned result, and then decide.\n"
"2. **Write-Through Memory Tools**:\n"
" - `update_note` and `append_note` may run alongside at most one Turn-Ending"
" Action only when recording facts already observed in the current context. If"
" the note content depends on the action's result, wait for the next"
" observation before writing it.\n"
"3. **Turn-Ending Actions**:"
)
_LEGACY_CHECKER_REJECTION_TRIGGER = """ - **Validation/Checker Rejection**: The verification agent (Checker) rejected your subgoal completion (i.e., you are in troubleshooter mode and this is a retry)."""
@@ -70,15 +188,28 @@ _LARGE_LIST_STRATEGY = """- **Large List & Long Text Exploration Strategy (Zero-
4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for "double-checking", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content."""
def apply_operator_prompt_contract(prompt_template: str) -> str:
"""Align prompt-level tool and recovery rules with Operator runtime semantics."""
def apply_operator_prompt_contract(
prompt_template: str,
available_tools: frozenset[str] | None = None,
) -> str:
"""Align prompt-level tool and recovery rules with Operator runtime semantics.
Args:
prompt_template: One of the raw operator.json templates.
available_tools: Tools that actually exist this run. ``None`` renders with the
full historical tool set, producing byte-identical output to the
pre-assembly contract. Unavailable tools disappear from every enumeration
slot, and instruction segments teaching them are removed wholesale.
"""
available = frozenset(resolve_available(available_tools, OPERATOR_PROMPT_TOOLSET))
prompt_template = prompt_template.replace(
_LEGACY_TOOL_CALLING_RULE,
_TOOL_CALLING_RULE,
_tool_calling_rule(available),
)
prompt_template = prompt_template.replace(
_LEGACY_TOOL_PROTOCOL,
_TOOL_PROTOCOL,
_tool_protocol(available),
)
prompt_template = prompt_template.replace(
_LEGACY_CHECKER_REJECTION_TRIGGER,
@@ -92,6 +223,44 @@ def apply_operator_prompt_contract(prompt_template: str) -> str:
_LEGACY_LARGE_LIST_STRATEGY,
_LARGE_LIST_STRATEGY,
)
# Device-action enumeration slots (identical in both templates).
prompt_template = prompt_template.replace(
_PHYSICAL_ACTIONS_SLOT,
"Physical device actions ("
+ render_tool_enum(_PHYSICAL_ACTIONS_ORDER, available)
+ ")",
)
prompt_template = prompt_template.replace(
_TURN_ENDING_SLOT,
"Turn-Ending Action ("
+ render_tool_enum(_TURN_ENDING_ORDER, available, final_sep="or")
+ ")",
)
# Instruction segments that teach a single tool.
prompt_template = prompt_template.replace(
_WAIT_FOR_DELAY_TRANSITIONS_SEGMENT,
gate_segment(_WAIT_FOR_DELAY_TRANSITIONS_SEGMENT, available, "wait_for_delay"),
)
prompt_template = prompt_template.replace(
_WAIT_FOR_DELAY_INTERVAL_PARENTHETICAL,
gate_segment(
_WAIT_FOR_DELAY_INTERVAL_PARENTHETICAL, available, "wait_for_delay"
),
)
prompt_template = prompt_template.replace(
_VIDEO_ANALYZER_TRANSIENT_SEGMENT,
gate_segment(_VIDEO_ANALYZER_TRANSIENT_SEGMENT, available, "video_analyzer"),
)
prompt_template = prompt_template.replace(
_VIDEO_ANALYZER_SNAPSHOT_SENTENCE,
gate_segment(_VIDEO_ANALYZER_SNAPSHOT_SENTENCE, available, "video_analyzer"),
)
prompt_template = prompt_template.replace(
_SWIPE_PRECISION_SEGMENT,
gate_segment(_SWIPE_PRECISION_SEGMENT, available, "swipe"),
)
return prompt_template
@@ -143,7 +312,31 @@ class TemplatePromptComponent(PromptComponent):
"Failed to format prompt, template not found in operator prompts config."
)
prompt_template = apply_operator_prompt_contract(prompt_template)
# video_analyzer is bound conditionally (graph.py gates it on
# video_recording_tools_enabled); the prompt must not advertise it when the
# tool is not actually available this run.
available = set(OPERATOR_PROMPT_TOOLSET)
setup = getattr(ctx, "execution_setup", None)
if not (setup and getattr(setup, "video_recording_tools_enabled", False)):
available.discard("video_analyzer")
# Device actions the installed actuator backend does not implement disappear
# from the prompt in lockstep with their tool declarations.
actuator = getattr(ctx, "actuator", None)
if actuator is not None and callable(getattr(actuator, "capabilities", None)):
try:
from artemis.mcp.action_manifest import (
DEVICE_ACTIONS,
available_device_actions,
)
available -= DEVICE_ACTIONS - available_device_actions(actuator)
except Exception as e:
logger.warning(f"Failed to assemble prompt against actuator: {e}")
prompt_template = apply_operator_prompt_contract(
prompt_template, available_tools=frozenset(available)
)
plan_and_history = kwargs.get("plan_and_history", "No plan or history yet.")
+79
View File
@@ -0,0 +1,79 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Helpers for assembling tool references into otherwise-static prompts.
Prompts stay hand-written text; only the spots that mention tools are assembled:
* **Enumeration slots** -- lists like ``(`click`, `swipe`, ... , or `wait_for_delay`)``
are re-rendered from the actually-available tool set, so an absent tool never
appears. With the full tool set the rendering reproduces the original wording
byte-for-byte (ordering, backticks, Oxford ``or``/``and``), which is what the
regression tests pin.
* **Instruction segments** -- self-contained blocks that teach one tool (e.g. the
``manage_app`` app-launching rule) are included only while that tool exists.
An unavailable tool therefore leaves no trace in the prompt at all -- the executable
definition of "an unimplemented tool must cost the model nothing".
"""
from collections.abc import Iterable, Sequence, Set
__all__ = ["gate_segment", "render_tool_enum", "resolve_available"]
def render_tool_enum(
tools: Sequence[str],
available: Set[str],
final_sep: str | None = None,
) -> str:
"""Renders an ordered, backticked tool enumeration limited to available tools.
Args:
tools: Tool names in the exact order the prompt historically used.
available: Tools that currently exist.
final_sep: ``"or"``/``"and"`` to join the last item Oxford-style
(``"`a`, `b`, or `c`"``); ``None`` for a plain comma list.
Returns:
The rendered enumeration, or ``""`` when none of the tools are available.
"""
ticked = [f"`{name}`" for name in tools if name in available]
if not ticked:
return ""
if len(ticked) == 1:
return ticked[0]
if final_sep is None:
return ", ".join(ticked)
if len(ticked) == 2:
return f"{ticked[0]} {final_sep} {ticked[1]}"
return ", ".join(ticked[:-1]) + f", {final_sep} " + ticked[-1]
def gate_segment(segment: str, available: Set[str], *tools: str) -> str:
"""Returns ``segment`` when every tool it teaches is available, else ``""``."""
if all(name in available for name in tools):
return segment
return ""
def resolve_available(available: Iterable[str] | None, full_set: Set[str]) -> Set[str]:
"""Normalizes an optional availability argument.
``None`` means "everything" -- callers that have no actuator wired yet keep
today's output unchanged. Passing an explicit set opts into assembly.
"""
if available is None:
return full_set
return frozenset(available)
+50 -11
View File
@@ -36,7 +36,7 @@ from langchain_core.messages import (
from artemis.agents.validator.tool_declarations import (
ASK_EXPLORER_TOOL,
CLICK_SEQUENCE_TOOL,
MobileActionExecutor,
MobileActionExecutor, # Legacy shims below; the loop itself uses McpActionExecutor.
ToolDeclaration,
VALIDATOR_TOOLS_DECLARATION,
capture_screenshot_and_parse_ui,
@@ -46,6 +46,7 @@ from artemis.context import ArtemisContext
from artemis.controllers.unified_controller import UnifiedMobileController
from artemis.data_engine.trace import trace
from artemis.graph.state import State
from artemis.mcp.action_executor import McpActionExecutor
from artemis.services.llm import get_llm
from artemis.utils.logger import get_logger
from artemis.utils.notes import get_note_file_path
@@ -230,6 +231,16 @@ class FailureAnalysisStrategy:
def get_tools(self) -> list:
return VALIDATOR_TOOLS_DECLARATION
def required_tools(self) -> frozenset[str]:
"""Tools this strategy's prompt depends on structurally.
A strategy whose prompt is built around a tool (rather than merely listing it)
must declare it here; strategy selection falls back to the base strategy when
the current backend does not provide every required tool, so the dependent
prompt is never rendered against a backend that cannot satisfy it.
"""
return frozenset()
def get_user_message_suffix(self, pre_screenshot: bool, post_screenshot: bool) -> str:
suffix = (
"Your objective is:\n"
@@ -268,6 +279,11 @@ class TargetDisappearedStrategy(FailureAnalysisStrategy):
tools.append(ASK_EXPLORER_TOOL)
return tools
def required_tools(self) -> frozenset[str]:
# The whole prompt is built around atomic chained execution to defeat turn
# latency, which *is* click_sequence.
return frozenset({"click_sequence"})
def get_user_message_suffix(self, pre_screenshot: bool, post_screenshot: bool) -> str:
return ""
@@ -284,6 +300,9 @@ class PixelTargetDisappearedStrategy(FailureAnalysisStrategy):
tools.append(ASK_EXPLORER_TOOL)
return tools
def required_tools(self) -> frozenset[str]:
return frozenset({"click_sequence"})
def get_user_message_suffix(self, pre_screenshot: bool, post_screenshot: bool) -> str:
return ""
@@ -295,15 +314,33 @@ class FailureAnalyzer:
self.ctx = ctx
self.max_iterations = 15
def _select_strategy(self, error_category: ValidationErrorCategory) -> FailureAnalysisStrategy:
def _select_strategy(
self,
error_category: ValidationErrorCategory,
available_tools: frozenset[str] | None = None,
) -> FailureAnalysisStrategy:
if error_category == ValidationErrorCategory.PIXEL_TARGET_DISAPPEARED:
logger.info("Routing failure analysis to specialized PixelTargetDisappearedStrategy.")
return PixelTargetDisappearedStrategy()
if error_category == ValidationErrorCategory.TARGET_DISAPPEARED:
strategy: FailureAnalysisStrategy = PixelTargetDisappearedStrategy()
elif error_category == ValidationErrorCategory.TARGET_DISAPPEARED:
logger.info("Routing failure analysis to specialized TargetDisappearedStrategy.")
return TargetDisappearedStrategy()
logger.info("Routing failure analysis to DefaultAnalysisStrategy.")
return FailureAnalysisStrategy()
strategy = TargetDisappearedStrategy()
else:
logger.info("Routing failure analysis to DefaultAnalysisStrategy.")
strategy = FailureAnalysisStrategy()
# Capability gating: a strategy and its prompt are one unit. If the backend
# cannot satisfy the tools the prompt is built around, fall back to the base
# strategy so the dependent prompt is never rendered. `None` = no gating.
if available_tools is not None:
missing = strategy.required_tools() - available_tools
if missing:
logger.warning(
f"Strategy {type(strategy).__name__} requires unavailable tool(s)"
f" {sorted(missing)}; falling back to base FailureAnalysisStrategy."
)
return FailureAnalysisStrategy()
return strategy
def _get_active_subgoal_hash(self) -> str:
if not self.ctx.data_engine:
@@ -363,7 +400,12 @@ class FailureAnalyzer:
if not pre_screenshot:
raise ValueError("Failure analysis requires valid screenshots.")
strategy = self._select_strategy(error_category)
controller = UnifiedMobileController(self.ctx)
executor = McpActionExecutor(self.ctx, controller)
strategy = self._select_strategy(
error_category, available_tools=frozenset(executor.action_tool_names)
)
prompt_template_name = strategy.get_prompt_template_name()
tools_declaration = strategy.get_tools()
@@ -372,9 +414,6 @@ class FailureAnalyzer:
except Exception:
llm = get_llm(self.ctx, name="validator")
controller = UnifiedMobileController(self.ctx)
executor = MobileActionExecutor(self.ctx, controller)
# Pre-Action and Post-Action XML formatting
pre_xml_list_str = "Not available."
if hasattr(state, "latest_ui_hierarchy") and state.latest_ui_hierarchy:
+84 -207
View File
@@ -15,35 +15,31 @@
"""Universal Tool Declarations, Execution Results, and Dispatcher for Artemis."""
import ast
import asyncio
import base64
import hashlib
import inspect
import json
from pathlib import Path
import time
from typing import Any, Literal
from google.genai import types as genai_types
from langchain_core.messages import ToolMessage
from pydantic import BaseModel, Field
from artemis.config import get_temp_dir, resolve_explorer_version
from artemis.config import resolve_explorer_version
from artemis.context import ArtemisContext
from artemis.controllers.unified_controller import UnifiedMobileController
from artemis.core.tool_declaration import ToolDeclaration
from artemis.data_engine.trace import trace
from artemis.mcp.action_types import ActionCode
from artemis.mcp.actuators.adb import AdbActuator
from artemis.mcp.observation import observe
from artemis.tools.explorer_tool import _run_explorer_logic
from artemis.graph.state import State
from artemis.tools.mobile.launch_app import find_package
from artemis.utils.app_launch_utils import launch_app_with_retries
from artemis.utils.coordinates import (
compute_smart_swipe_coordinates,
parse_swipe_parameters,
)
from artemis.utils.logger import get_logger
from artemis.utils.ocr_xml_fusion import fuse_ocr_with_xml
from artemis.utils.visualization import format_minimal_list_with_elements
from artemis.utils.notes import (
LIST_NOTES_DOCSTRING,
READ_NOTE_ARG_KEY_DESC,
@@ -186,65 +182,22 @@ async def capture_screenshot_and_parse_ui(
controller: UnifiedMobileController,
skip_settling: bool = False,
) -> tuple[str | None, bytes | None, str | None]:
"""Captures screenshot and parses fused XML tree after optional screen settling delay."""
try:
if skip_settling:
logger.info("Retrieving screen data directly (skip settling)...")
device_data = await controller.get_screen_data()
latest_screenshot_b64 = device_data.base64
else:
logger.info("Delaying 0.4s before capturing screen state...")
await asyncio.sleep(0.4)
device_data = await controller.get_screen_data()
latest_screenshot_b64 = device_data.base64
"""Captures screenshot and parses fused XML tree after optional screen settling delay.
latest_screenshot_bytes = base64.b64decode(latest_screenshot_b64)
except Exception as e:
logger.error(f"Failed to capture screen state: {e}")
Thin adapter over :func:`artemis.mcp.observation.observe` that adds the LangGraph
``State`` write-back (indexed points/elements) the agents rely on.
"""
obs, screenshot_bytes = await observe(
ctx, controller, settle_ms=0 if skip_settling else 400
)
if not obs.ok:
return None, None, None
try:
xml_hierarchy = device_data.elements
width = device_data.width or 1080
height = device_data.height or 2400
if ctx.device:
ctx.device.device_width = width
ctx.device.device_height = height
if obs.hierarchy_ok:
state.indexed_points = [el["center"] for el in obs.elements]
state.indexed_elements = obs.elements
ocr_results = []
fused_xml = fuse_ocr_with_xml(xml_hierarchy, ocr_results)
minimal_list_str, elements, _ = format_minimal_list_with_elements(fused_xml, width, height)
state.indexed_points = [el["center"] for el in elements]
state.indexed_elements = elements
except Exception as e:
logger.error(f"Failed to fetch or fuse UI hierarchy: {e}")
minimal_list_str = "Not available due to hierarchy error."
screenshot_path = None
if ctx.data_engine:
try:
image_name = ctx.data_engine.get_or_create_image(
latest_screenshot_bytes,
ui_tree=xml_hierarchy,
ocr_result=ocr_results,
)
screenshot_path = str(ctx.data_engine.get_image_path(image_name))
except Exception as e:
logger.error(f"Failed to save image in DataEngine: {e}")
if not screenshot_path:
try:
temp_dir = get_temp_dir("screenshots")
temp_file = temp_dir / f"screenshot_{int(time.time())}.jpg"
temp_file.write_bytes(latest_screenshot_bytes)
screenshot_path = str(temp_file)
except Exception as e:
logger.error(f"Failed to save fallback screenshot: {e}")
return screenshot_path, latest_screenshot_bytes, minimal_list_str
return obs.screenshot_path, screenshot_bytes, obs.elements_text
def normalize_coordinate_target(target: Any) -> int | list[int] | Any:
@@ -313,11 +266,23 @@ def serialize_mobile_action_result(result: Any) -> dict[str, Any]:
class MobileActionExecutor:
"""Encapsulates device action execution, observation capture, and result reporting."""
"""Adapts LLM tool calls onto an actuator backend and captures observations.
def __init__(self, ctx: ArtemisContext, controller: UnifiedMobileController | None = None):
Device-call bodies live in the actuator (``artemis/mcp/actuators/``); this class
keeps the agent-side responsibilities: argument normalization, index resolution
against LangGraph ``State``, act-then-observe screenshot capture, tracing, and the
legacy 4-tuple result contract.
"""
def __init__(
self,
ctx: ArtemisContext,
controller: UnifiedMobileController | None = None,
actuator: "AdbActuator | None" = None,
):
self.ctx = ctx
self.controller = controller or UnifiedMobileController(ctx)
self.actuator = actuator or AdbActuator(ctx, controller)
self.controller = self.actuator.controller
@trace(type="action", name="click", serializer=serialize_mobile_action_result)
async def exec_click(
@@ -327,27 +292,16 @@ class MobileActionExecutor:
times: int = 1,
delay_ms: int = 100,
) -> tuple[str, bytes | None, str | None, str | None]:
width = getattr(self.ctx.device, "device_width", 1080)
height = getattr(self.ctx.device, "device_height", 2400)
try:
target = normalize_coordinate_target(target)
if not isinstance(target, (list, tuple)) or len(target) != 2:
raise ValueError(f"Invalid target format: {target}")
nx, ny = target
x = int(max(0, min(width - 1, int(nx) * width / 1000)))
y = int(max(0, min(height - 1, int(ny) * height / 1000)))
result = await self.controller.tap_at(x, y, times=times, delay_ms=delay_ms)
err = getattr(result, "error", None) if result else None
success = err is None
outcome = (
f"Clicked at [{nx}, {ny}] (normalized) successfully."
if success
else f"Error executing click: {err}"
)
res = await self.actuator.click(int(nx), int(ny), times=times, delay_ms=delay_ms)
outcome = res.message
img_bytes, shot_path, xml_list = None, None, None
if success:
if res.ok:
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller
)
@@ -385,15 +339,20 @@ class MobileActionExecutor:
None,
)
outcomes = []
for i, raw_target in enumerate(sequence):
# Resolve each entry to normalized coordinates: integer element indices go
# through state.indexed_points (pixel centers -> normalized), coordinate
# pairs are already normalized.
norm_points: list[tuple[int, int]] = []
for raw_target in sequence:
target = normalize_coordinate_target(raw_target)
if isinstance(target, int):
idx = target
points = getattr(state, "indexed_points", []) or []
if 1 <= idx <= len(points):
pt = points[idx - 1]
x, y = int(pt[0]), int(pt[1])
px, py = int(pt[0]), int(pt[1])
nx = int(max(0, min(1000, round(px * 1000 / max(1, width)))))
ny = int(max(0, min(1000, round(py * 1000 / max(1, height)))))
else:
return (
(
@@ -405,9 +364,7 @@ class MobileActionExecutor:
None,
)
elif isinstance(target, (list, tuple)) and len(target) == 2:
nx, ny = target
x = int(max(0, min(width - 1, int(nx) * width / 1000)))
y = int(max(0, min(height - 1, int(ny) * height / 1000)))
nx, ny = int(target[0]), int(target[1])
else:
return (
f"Error during click sequence: Invalid target format: {raw_target}",
@@ -415,25 +372,16 @@ class MobileActionExecutor:
None,
None,
)
norm_points.append((nx, ny))
result = await self.controller.tap_at(x, y)
err = getattr(result, "error", None) if result else None
if err:
return (
f"Error executing click at step {i + 1}: {err}",
None,
None,
None,
)
outcomes.append(f"Tapped at [{x}, {y}]")
if i < len(sequence) - 1:
await asyncio.sleep(max(0, delay_ms) / 1000.0)
res = await self.actuator.click_sequence(norm_points, delay_ms=delay_ms)
if not res.ok:
return res.message, None, None, None
outcome_str = f"Sequence clicked successfully: {'; '.join(outcomes)}"
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller
)
return outcome_str, img_bytes, shot_path, xml_list
return res.message, img_bytes, shot_path, xml_list
except Exception as e:
return f"Error executing click sequence: {e}", None, None, None
@@ -445,8 +393,6 @@ class MobileActionExecutor:
duration_ms: int = 1000,
duration: int | None = None,
) -> tuple[str, bytes | None, str | None, str | None]:
width = getattr(self.ctx.device, "device_width", 1080)
height = getattr(self.ctx.device, "device_height", 2400)
if duration is not None:
duration_ms = duration
try:
@@ -454,22 +400,11 @@ class MobileActionExecutor:
if not isinstance(target, (list, tuple)) or len(target) != 2:
raise ValueError(f"Invalid target format: {target}")
nx, ny = target
x = int(max(0, min(width - 1, int(nx) * width / 1000)))
y = int(max(0, min(height - 1, int(ny) * height / 1000)))
result = await self.controller.tap_at(
x, y, long_press=True, long_press_duration=duration_ms
)
err = getattr(result, "error", None) if result else None
success = err is None
outcome = (
f"Long pressed at [{nx}, {ny}] (normalized) for {duration_ms}ms successfully."
if success
else f"Error executing long press: {err}"
)
res = await self.actuator.long_press(int(nx), int(ny), duration_ms=duration_ms)
outcome = res.message
img_bytes, shot_path, xml_list = None, None, None
if success:
if res.ok:
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller
)
@@ -487,24 +422,16 @@ class MobileActionExecutor:
state: State,
clear_exist: bool = True,
) -> tuple[str, bytes | None, str | None, str | None]:
width = getattr(self.ctx.device, "device_width", 1080)
height = getattr(self.ctx.device, "device_height", 2400)
try:
norm_target: tuple[int, int] | None = None
if target:
target = normalize_coordinate_target(target)
if not isinstance(target, (list, tuple)) or len(target) != 2:
raise ValueError(f"Invalid target format: {target}")
nx, ny = target
x = int(max(0, min(width - 1, int(nx) * width / 1000)))
y = int(max(0, min(height - 1, int(ny) * height / 1000)))
norm_target = (int(target[0]), int(target[1]))
await self.controller.tap_at(x, y)
if clear_exist and hasattr(self.controller, "erase_text"):
await self.controller.erase_text()
if hasattr(self.controller, "type_text"):
await self.controller.type_text(text)
outcome = f"Executed typing '{text}'."
res = await self.actuator.input_text(text, norm_target, clear_exist=clear_exist)
outcome = res.message
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller
)
@@ -540,6 +467,8 @@ class MobileActionExecutor:
try:
if kind == "direction":
dir_name = target
# Smart-swipe resolution needs State (indexed elements + ui tree), so
# it stays client-side; the actuator only receives the two points.
x1, y1, x2, y2, smart_dur = compute_smart_swipe_coordinates(
direction=target,
target=swipe_input.get("target"),
@@ -550,19 +479,21 @@ class MobileActionExecutor:
duration=final_duration,
)
final_duration = smart_dur
err = await self.controller.swipe_coords(x1, y1, x2, y2, final_duration)
if err:
return f"Error swiping {dir_name}: {err}", None, None, None
nx1 = int(max(0, min(1000, round(x1 * 1000 / max(1, width)))))
ny1 = int(max(0, min(1000, round(y1 * 1000 / max(1, height)))))
nx2 = int(max(0, min(1000, round(x2 * 1000 / max(1, width)))))
ny2 = int(max(0, min(1000, round(y2 * 1000 / max(1, height)))))
res = await self.actuator.swipe((nx1, ny1), (nx2, ny2), final_duration)
if not res.ok:
return f"Error swiping {dir_name}: {res.detail}", None, None, None
outcome = f"Swipe completed successfully. Swiped {dir_name}."
elif kind == "coords":
x1, y1, x2, y2 = target
px1 = int(max(0, min(width - 1, int(x1) * width / 1000)))
py1 = int(max(0, min(height - 1, int(y1) * height / 1000)))
px2 = int(max(0, min(width - 1, int(x2) * width / 1000)))
py2 = int(max(0, min(height - 1, int(y2) * height / 1000)))
err = await self.controller.swipe_coords(px1, py1, px2, py2, final_duration)
if err:
return f"Error dragging: {err}", None, None, None
res = await self.actuator.swipe(
(int(x1), int(y1)), (int(x2), int(y2)), final_duration
)
if not res.ok:
return f"Error dragging: {res.detail}", None, None, None
outcome = f"Swipe completed successfully. Swiped from [{x1}, {y1}] to [{x2}, {y2}]."
else:
return (
@@ -589,24 +520,11 @@ class MobileActionExecutor:
state: State,
) -> tuple[str, bytes | None, str | None, str | None]:
try:
valid_keys = {"home", "back", "enter", "delete", "tab", "search", "menu", "app_switch"}
key_str = str(key).lower()
if key_str not in valid_keys:
return f"Error executing key press '{key}'.", None, None, None
res = await self.actuator.press_key(key)
if not res.ok:
return res.message, None, None, None
if key_str == "back" and hasattr(self.controller, "go_back"):
await self.controller.go_back()
elif key_str == "home" and hasattr(self.controller, "go_home"):
await self.controller.go_home()
elif key_str == "enter" and hasattr(self.controller, "press_enter"):
await self.controller.press_enter()
elif hasattr(self.controller, "press_key"):
res = await self.controller.press_key(key)
err = getattr(res, "error", None) if res else None
if err:
return f"Error executing key press '{key}': {err}", None, None, None
outcome = f"Executed key press '{key}'."
outcome = res.message
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller
)
@@ -624,35 +542,13 @@ class MobileActionExecutor:
state: State,
) -> tuple[str, bytes | None, str | None, str | None]:
try:
res = find_package(self.ctx, app_name, use_fallback=False)
pkg = await res if inspect.iscoroutine(res) else res
if not pkg:
return f"Error finding package for app: {app_name}", None, None, None
target_pkg = pkg
if action.lower() == "launch":
res_launch = launch_app_with_retries(self.ctx, target_pkg)
if inspect.iscoroutine(res_launch):
res_launch = await res_launch
if isinstance(res_launch, tuple):
success, error_msg = res_launch
else:
success, error_msg = bool(res_launch), ""
outcome = (
f"Launched app '{app_name}' ({target_pkg}) successfully."
if success
else f"Failed to launch app '{app_name}': {error_msg}"
)
elif action.lower() == "stop":
if hasattr(self.controller, "terminate_app"):
res_term = self.controller.terminate_app(target_pkg)
if inspect.iscoroutine(res_term):
await res_term
outcome = f"Terminated app '{app_name}' successfully."
else:
return f"Invalid manage_app action: {action}", None, None, None
res = await self.actuator.manage_app(action, app_name)
# Argument-level failures return without an observation, exactly as the
# historical early returns did; a failed launch still captures the screen.
if res.code in (ActionCode.PACKAGE_NOT_FOUND, ActionCode.INVALID_ARGS):
return res.message, None, None, None
outcome = res.message
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller
)
@@ -669,9 +565,8 @@ class MobileActionExecutor:
state: State,
) -> tuple[str, bytes | None, str | None, str | None]:
try:
delay_s = max(0.0, float(time_in_ms) / 1000.0)
await asyncio.sleep(delay_s)
outcome = f"Waited for {time_in_ms}ms successfully."
res = await self.actuator.wait_for_delay(time_in_ms)
outcome = res.message
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller
)
@@ -690,27 +585,9 @@ class MobileActionExecutor:
state: State,
) -> tuple[str, bytes | None, str | None, str | None]:
try:
timeout_s = (timeout_ms or 5000) / 1000.0
start = time.time()
found = False
target_state = (wait_state or "appear").lower()
while time.time() - start < timeout_s:
tree = await self.controller.get_ui_elements()
text_present = text.lower() in (tree or "").lower()
if target_state == "appear" and text_present:
found = True
break
elif target_state == "disappear" and not text_present:
found = True
break
await asyncio.sleep(0.5)
outcome = (
f"Successfully waited for text '{text}' to {target_state}."
if found
else f"Timed out waiting for text '{text}' to {target_state}."
)
res = await self.actuator.wait_for_text(text, wait_state, timeout_ms)
outcome = res.message
# A timeout still captures the screen, as the original loop did.
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
self.ctx, state, self.controller
)
+64 -460
View File
@@ -17,39 +17,31 @@ import base64
import difflib
import json
import math
import os
from pathlib import Path
import psutil
import re
import sys
import time
import traceback
import uuid
from uuid import UUID
from adbutils import AdbClient
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from langchain_core.messages import HumanMessage, SystemMessage
from artemis.agents.validator.failure_analyzer import (
FailureAnalyzer,
ValidationErrorCategory,
)
from artemis.clients.ui_automator_client import UIAutomatorClient
from artemis.constants import (
VALIDATOR_POLL_INTERVAL,
VALIDATOR_POLL_TIMEOUT,
VALIDATOR_UI_HIERARCHY_TIMEOUT,
)
from artemis.context import ArtemisContext, DeviceContext, DevicePlatform
from artemis.context import ArtemisContext
from artemis.controllers.unified_controller import UnifiedMobileController
from artemis.data_engine.trace import CURRENT_TRACE_ID, trace
from artemis.graph.state import State
from artemis.platform import platform
from artemis.mcp.action_names import to_canonical_call
from artemis.mcp.action_session import ActionSession, get_action_session
from artemis.services.llm import get_llm
from artemis.tools.mobile.launch_app import find_package
from artemis.utils import app_launch_utils, image_diff, visualization
from artemis.utils import image_diff, visualization
from artemis.utils.decorators import wrap_with_callbacks
from artemis.utils.logger import get_logger
@@ -85,151 +77,14 @@ class ValidatorNode:
def __init__(self, ctx: ArtemisContext):
self.ctx = ctx
def _kill_mcp_server_instantly(self):
"""Instantly terminate MCP ADB server child processes."""
async def _get_mcp_session(self) -> ActionSession:
"""Returns the in-process unified action session (created lazily on ctx).
try:
current_process = psutil.Process()
children = current_process.children(recursive=True)
for child in children:
try:
cmdline = child.cmdline()
if any("adb_server.py" in part for part in cmdline):
logger.warning(f"Instantly killing MCP Server child process: {child.pid}")
child.kill()
except psutil.NoSuchProcess:
pass
except Exception as e:
logger.error(f"Failed to instantly kill MCP server child processes: {e}")
async def _close_failed_mcp_session(
self,
*,
session,
session_entered: bool,
client_ctx,
client_entered: bool,
) -> bool:
"""Exit partially-entered MCP contexts in reverse order.
The MCP stdio transport owns an AnyIO task group and cancellation
scope. Dropping an entered transport without calling ``__aexit__``
leaves that scope attached to the validator task and can cancel an
unrelated action later in the same node.
Returns ``True`` when every entered context closed cleanly.
The previous stdio-subprocess spawn -- and its local-controller fallback,
which on Windows was the *only* path that ever ran -- is gone: the server now
lives in-process, so there is no spawn to fail and exactly one execution path.
"""
self.ctx.mcp_session = None
self.ctx.mcp_client_ctx = None
cleanup_succeeded = True
for manager, entered, label in (
(session, session_entered, "MCP session"),
(client_ctx, client_entered, "MCP stdio transport"),
):
if not entered or manager is None:
continue
try:
await manager.__aexit__(None, None, None)
except asyncio.CancelledError:
cleanup_succeeded = False
logger.warning(f"{label} cleanup was cancelled.")
except Exception as cleanup_error:
cleanup_succeeded = False
logger.warning(f"Failed to close {label}: {cleanup_error}")
return cleanup_succeeded
async def _get_mcp_session(self):
if getattr(self.ctx, "mcp_session", None) is not None:
return self.ctx.mcp_session
root_dir = Path(__file__).parent.parent.parent
venv_python = root_dir / ".venv" / "bin" / "python3"
python_exe = str(venv_python) if venv_python.exists() else sys.executable
server_env = os.environ.copy()
# The parent Artemis process already owns the device-awake policy.
# Starting it again in this short-lived child can block the MCP
# handshake on cross-process ADB/lease cleanup.
server_env["ARTEMIS_KEEP_DEVICE_AWAKE"] = "false"
server_params = StdioServerParameters(
command=python_exe,
args=[str(root_dir / "mcp" / "adb_server.py")],
env=server_env,
)
logger.info("Starting persistent MCP server session...")
client_ctx = stdio_client(server_params)
session = None
client_entered = False
session_entered = False
self.ctx.mcp_client_ctx = client_ctx
try:
read, write = await client_ctx.__aenter__()
client_entered = True
session = ClientSession(read, write)
self.ctx.mcp_session = session
await session.__aenter__()
session_entered = True
# Handshake timeout protection
await asyncio.wait_for(session.initialize(), timeout=15.0)
return session
except asyncio.CancelledError:
await self._close_failed_mcp_session(
session=session,
session_entered=session_entered,
client_ctx=client_ctx,
client_entered=client_entered,
)
raise
except Exception as e:
err_stack = traceback.format_exc()
logger.critical(
"CRITICAL: Failed to initialize MCP server session:"
f" {e}\n{err_stack}. Cleaning up child processes."
)
cleanup_succeeded = await self._close_failed_mcp_session(
session=session,
session_entered=session_entered,
client_ctx=client_ctx,
client_entered=client_entered,
)
if not cleanup_succeeded:
self._kill_mcp_server_instantly()
logger.warning(
"Falling back to robust local in-process execution bypassing MCP protocol."
)
try:
adb = AdbClient(host="localhost", port=5037)
devices = adb.device_list()
if devices:
device_id = devices[0].serial
ui_client = UIAutomatorClient(device_id=device_id)
try:
ui_data = ui_client.get_screen_data()
w, h = ui_data.width, ui_data.height
except Exception:
w, h = 1080, 2400
local_ctx = ArtemisContext(
trace_id="local-fallback",
device=DeviceContext(
host_platform=platform.os_type.name,
mobile_platform=DevicePlatform.ANDROID,
device_id=device_id,
device_width=w,
device_height=h,
),
adb_client=adb,
ui_adb_client=ui_client,
)
self._local_controller = UnifiedMobileController(local_ctx)
except Exception as le:
logger.error(f"Failed to setup local fallback controller: {le}")
return None
return await get_action_session(self.ctx)
def _parse_decisions(self, structured_decisions: str) -> tuple[list[dict] | None, str | None]:
if not structured_decisions:
@@ -270,20 +125,11 @@ class ValidatorNode:
error_msg = ""
while time.time() - start_time < timeout:
if session is not None:
result = await session.call_tool("take_screenshot", {})
success_img, content_img = self._parse_mcp_result(result)
elif getattr(self, "_local_controller", None):
try:
content_img = await self._local_controller.take_screenshot()
success_img = True
except Exception as e:
success_img, content_img = False, str(e)
else:
success_img, content_img = (
False,
"No session or local controller",
)
try:
content_img = await session.screenshot_b64()
success_img = True
except Exception as e:
success_img, content_img = False, str(e)
if success_img:
post_screenshot_b64 = content_img
img_before_bytes = base64.b64decode(pre_screenshot_b64)
@@ -465,15 +311,7 @@ class ValidatorNode:
# Capture the live mismatch screenshot to provide context for FailureAnalyzer
try:
if session is not None:
result = await session.call_tool("take_screenshot", {})
success_img, content = self._parse_mcp_result(result)
if success_img and content:
post_screenshot_b64 = content
else:
logger.warning(f"Take screenshot via MCP returned non-success: {content}")
elif getattr(self, "_local_controller", None):
post_screenshot_b64 = await self._local_controller.take_screenshot()
post_screenshot_b64 = await session.screenshot_b64()
except Exception as e:
logger.error(f"Failed to capture live screenshot for failure analysis: {e}")
post_screenshot_b64 = None
@@ -504,17 +342,9 @@ class ValidatorNode:
if not success and not post_screenshot_b64:
logger.info("Action failed, capturing failure screenshot...")
try:
if session is not None:
result = await session.call_tool("take_screenshot", {})
success_img, content = self._parse_mcp_result(result)
if success_img and content:
post_screenshot_b64 = content
else:
logger.warning(f"Take failure screenshot via MCP returned non-success: {content}")
elif getattr(self, "_local_controller", None):
post_screenshot_b64 = await self._local_controller.take_screenshot()
post_screenshot_b64 = await session.screenshot_b64()
except Exception as e:
logger.error(f"Failed to capture failure screenshot: {e}")
logger.error(f"Failed to capture failure screenshot: {e!r}")
post_screenshot_b64 = None
# Enrich and record execution
@@ -661,10 +491,9 @@ class ValidatorNode:
try:
return await self._execute_validation_loop(state)
except asyncio.CancelledError:
logger.warning(
"Validator task cancelled. Instantly killing MCP server processes."
)
self._kill_mcp_server_instantly()
# In-process session teardown happens in ArtemisContext.__aexit__; there
# is no subprocess left to kill.
logger.warning("Validator task cancelled (Stop requested).")
raise
except Exception as e:
err_stack = traceback.format_exc()
@@ -681,226 +510,42 @@ class ValidatorNode:
)
except Exception:
pass
self._kill_mcp_server_instantly()
raise e
finally:
if self.ctx.data_engine:
self.ctx.data_engine.current_step_id = original_step_id
async def _exec_action(
self, session: ClientSession | None, action_item: dict
) -> tuple[bool, str]:
async def _exec_action(self, session: ActionSession, action_item: dict) -> tuple[bool, str]:
"""Executes one Operator action item through the unified action session.
The Operator's internal verbs (tap, long_press_on, focus_and_input_text, ...)
are translated to canonical tool calls in ``to_canonical_call`` using the
``normalized_coordinates`` the Operator already computed. Success comes from
the structured ``ActionResult.ok`` -- no string parsing.
"""
action_name = action_item.get("action")
coordinates = action_item.get("coordinates")
# Intercept non-wait_for_text actions for local fallback
if session is None and getattr(self, "_local_controller", None) is not None:
ctrl = self._local_controller
logger.info(f"Executing in-process action fallback: {action_name}")
try:
if action_name == "tap":
if coordinates and len(coordinates) == 2:
times = action_item.get("times", 1)
delay_ms = action_item.get("delay_ms", 100)
res = await ctrl.tap_at(
coordinates[0],
coordinates[1],
times=times,
delay_ms=delay_ms,
)
return (True, "Success") if not res.error else (False, res.error)
return False, "Invalid coords"
elif action_name == "long_press_on":
d = action_item.get("duration", 1000)
if coordinates and len(coordinates) == 2:
res = await ctrl.tap_at(
coordinates[0],
coordinates[1],
long_press=True,
long_press_duration=d,
)
return (True, "Success") if not res.error else (False, res.error)
return False, "Invalid coords"
elif action_name == "swipe":
d = action_item.get("duration", 400)
if coordinates and len(coordinates) == 4:
err = await ctrl.swipe_coords(
coordinates[0],
coordinates[1],
coordinates[2],
coordinates[3],
duration=d,
)
return (True, "Success") if not err else (False, str(err))
return False, "Invalid coords"
elif action_name == "focus_and_input_text":
t = action_item.get("text")
clear_before = action_item.get("clear_before_input", False)
if coordinates and len(coordinates) == 2:
await ctrl.tap_at(coordinates[0], coordinates[1])
if action_name == "wait_for_delay":
time_in_ms = action_item.get("time_in_ms", 0)
logger.info(f"Validator performing wait_for_delay for {time_in_ms}ms...")
await asyncio.sleep(time_in_ms / 1000.0)
return True, ""
await asyncio.sleep(0.5)
if clear_before:
await ctrl.erase_text()
else:
await ctrl.press_key("123") # Move cursor to end for clean append
success = await ctrl.type_text(t, clear_existing=False)
return (True, "Success") if success else (False, "Failed typing")
return False, "Invalid coords"
elif action_name == "focus_and_clear_text":
if coordinates and len(coordinates) == 2:
await ctrl.tap_at(coordinates[0], coordinates[1])
width = getattr(self.ctx.device, "device_width", 1080) if self.ctx.device else 1080
height = getattr(self.ctx.device, "device_height", 2400) if self.ctx.device else 2400
await asyncio.sleep(0.5)
success = await ctrl.erase_text()
return (True, "Success") if success else (False, "Failed erase")
return False, "Invalid coords"
elif action_name == "erase_one_char":
success = await ctrl.erase_text(nb_chars=1)
return (True, "Success") if success else (False, "Failed erase")
elif action_name == "press_key":
keycode = action_item.get("keycode")
success = await ctrl.press_key(keycode)
return (True, "Success") if success else (False, f"Failed press {keycode}")
elif action_name == "back":
success = await ctrl.go_back()
return (True, "Success") if success else (False, "Failed back")
elif action_name == "launch_app":
app_name = action_item.get("app_name")
package_name = await find_package(self.ctx, app_name, use_fallback=False)
if package_name:
success, err_msg = await app_launch_utils.launch_app_with_retries(
ctrl.ctx, package_name
)
return (True, "Success") if success else (False, err_msg)
return False, "Package not found"
elif action_name == "stop_app":
app_name = action_item.get("app_name")
package_name = await find_package(self.ctx, app_name, use_fallback=False)
if package_name:
success = await ctrl.terminate_app(package_name)
return (True, "Success") if success else (False, "Failed terminate")
return False, "Package not found"
elif action_name == "open_link":
url = action_item.get("url")
success = await ctrl.open_url(url)
return (True, "Success") if success else (False, "Failed open url")
elif action_name == "wait_for_delay":
time_in_ms = action_item.get("time_in_ms", 0)
await asyncio.sleep(time_in_ms / 1000.0)
return True, ""
else:
return (
False,
f"Unsupported action for local fallback: {action_name}",
)
except Exception as e:
return False, f"In-process fallback failed: {e}"
# Normal MCP Execution Path
try:
if action_name == "tap":
coordinates = action_item.get("coordinates")
times = action_item.get("times") or action_item.get("click_times") or 1
delay_ms = action_item.get("delay_ms") or action_item.get("delay") or 100
result = await session.call_tool(
"tap",
{
"coordinates": coordinates,
"times": times,
"delay_ms": delay_ms,
},
)
return self._parse_mcp_result(result)
tool_name, wire_args = to_canonical_call(action_item, dims=(width, height))
except ValueError as e:
return False, str(e)
elif action_name == "long_press_on":
coordinates = action_item.get("coordinates")
duration = action_item.get("duration", 1000)
result = await session.call_tool(
"long_press_on",
{"coordinates": coordinates, "duration": duration},
)
return self._parse_mcp_result(result)
elif action_name == "swipe":
coordinates = action_item.get("coordinates")
duration = action_item.get("duration", 400)
result = await session.call_tool(
"swipe", {"coordinates": coordinates, "duration": duration}
)
return self._parse_mcp_result(result)
elif action_name == "focus_and_input_text":
coordinates = action_item.get("coordinates")
text = action_item.get("text", "")
clear_before_input = action_item.get("clear_before_input", False)
result = await session.call_tool(
"focus_and_input_text",
{
"coordinates": coordinates,
"text": text,
"clear_before_input": clear_before_input,
},
)
return self._parse_mcp_result(result)
elif action_name == "focus_and_clear_text":
coordinates = action_item.get("coordinates")
result = await session.call_tool(
"focus_and_clear_text", {"coordinates": coordinates}
)
return self._parse_mcp_result(result)
elif action_name == "erase_one_char":
result = await session.call_tool("erase_one_char", {})
return self._parse_mcp_result(result)
elif action_name == "press_key":
keycode = action_item.get("keycode", "")
result = await session.call_tool("press_key", {"keycode": keycode})
return self._parse_mcp_result(result)
elif action_name == "back":
result = await session.call_tool("back", {})
return self._parse_mcp_result(result)
elif action_name == "launch_app":
app_name = action_item.get("app_name")
package_name = await find_package(self.ctx, app_name, use_fallback=False)
if not package_name:
return False, f"Failed to find package for app: {app_name}"
success, err_msg = await app_launch_utils.launch_app_with_retries(
self.ctx, package_name
)
return success, "Success" if success else err_msg
elif action_name == "stop_app":
app_name = action_item.get("app_name")
package_name = await find_package(self.ctx, app_name, use_fallback=False)
if not package_name:
return False, f"Failed to find package for app: {app_name}"
result = await session.call_tool("stop_app", {"package_name": package_name})
return self._parse_mcp_result(result)
elif action_name == "open_link":
url = action_item.get("url")
result = await session.call_tool("open_link", {"url": url})
return self._parse_mcp_result(result)
elif action_name == "wait_for_delay":
time_in_ms = action_item.get("time_in_ms", 0)
logger.info(f"Validator performing wait_for_delay for {time_in_ms}ms...")
await asyncio.sleep(time_in_ms / 1000.0)
return True, ""
else:
return False, f"Unsupported action: {action_name}"
try:
res = await session.call(tool_name, wire_args)
except Exception as e:
return False, f"MCP call failed: {e}"
return (True, "") if res.ok else (False, res.message)
@trace(type="tool", name="safety_net_validation")
async def _validate_action_precondition(
self, session, action_item: dict, state: State | None = None
@@ -1001,45 +646,21 @@ class ValidatorNode:
try:
elements = None
if session is not None:
try:
result = await asyncio.wait_for(
session.call_tool("get_ui_hierarchy", {}),
timeout=VALIDATOR_UI_HIERARCHY_TIMEOUT,
)
if hasattr(result, "content") and result.content:
text_content = (
result.content[0].text
if hasattr(result.content[0], "text")
else str(result.content[0])
)
elements = json.loads(text_content)
except Exception as e:
logger.warning(
f"Failed to get live XML via MCP: {e}. Falling back to"
" Pixel-based validation."
)
return (
False,
ValidationErrorCategory.XML_BYPASSED,
f"XML hierarchy fetch timed out or errored: {e}",
)
elif getattr(self, "_local_controller", None):
try:
elements = await asyncio.wait_for(
self._local_controller.get_ui_elements(),
timeout=VALIDATOR_UI_HIERARCHY_TIMEOUT,
)
except Exception as e:
logger.warning(
f"Failed to get live XML via local controller: {e}."
" Falling back to Pixel-based validation."
)
return (
False,
ValidationErrorCategory.XML_BYPASSED,
f"XML hierarchy fetch timed out or errored: {e}",
)
try:
# The timeout must ride inside the session (MCP read timeout), never an
# outer asyncio.wait_for: cancelling call_tool mid-flight corrupts the
# in-memory transport and bricks the session for all later callers.
elements = await session.ui_hierarchy(timeout=VALIDATOR_UI_HIERARCHY_TIMEOUT)
except Exception as e:
logger.warning(
f"Failed to get live XML via MCP: {e!r}. Falling back to"
" Pixel-based validation."
)
return (
False,
ValidationErrorCategory.XML_BYPASSED,
f"XML hierarchy fetch timed out or errored: {e!r}",
)
if not elements or not isinstance(elements, list):
logger.warning(
@@ -1275,6 +896,9 @@ class ValidatorNode:
f" {new_coords}."
)
action_item["coordinates"] = new_coords
# Drop the now-stale normalized coordinates so the canonical
# translation re-derives them from the healed pixel center.
action_item.pop("normalized_coordinates", None)
else:
logger.info(
"Pre-execution validation SUCCESS"
@@ -1435,23 +1059,6 @@ class ValidatorNode:
return passed, category, reason
def _parse_mcp_result(self, result) -> tuple[bool, str]:
"""Helper to parse MCP tool result."""
if hasattr(result, "content") and result.content:
text_content = (
result.content[0].text
if hasattr(result.content[0], "text")
else str(result.content[0])
)
if text_content.startswith("Error:") or text_content.startswith("Failed:"):
return False, text_content
elif text_content == "Success":
return True, ""
elif text_content == "Failed":
return False, "Action failed on server"
return True, text_content
return True, ""
@trace(type="tool", name="safety_net_pixel_validation")
async def _validate_action_precondition_pixel(
self,
@@ -1535,13 +1142,10 @@ class ValidatorNode:
try:
# A. Take fresh live screenshot (captures state changes / dynamic loading settling)
live_screenshot_b64 = None
if session is not None:
result = await session.call_tool("take_screenshot", {})
success_img, live_screenshot_b64 = self._parse_mcp_result(result)
elif getattr(self, "_local_controller", None):
live_screenshot_b64 = await self._local_controller.take_screenshot()
try:
live_screenshot_b64 = await session.screenshot_b64()
success_img = True
else:
except Exception:
success_img, live_screenshot_b64 = False, None
if not success_img or not live_screenshot_b64: