feat: extract in-process MCP action layer, concurrent task dispatch, and spawn/reconcile hardening
- Extract device action execution into artemis/mcp action layer: typed ActionResult, capability manifest with startup validation, AdbActuator as the single physical implementation, in-memory ActionSession (replaces stdio subprocess MCP in validator), and McpActionExecutor preserving the legacy executor contract. adb_server keeps its external stdio/SSE surface unchanged and now reuses the actuator implementation; import-time side effects moved into configure_stdio_mode(). - Capability-gated prompt assembly (prompt_assembly.py): tool enums and teaching segments render per available backend tools; full-set output is byte-identical to the previous static text (locked by tests). - Admin console task queue: serial worker replaced by a per-device concurrent dispatcher (ARTEMIS_CONCURRENCY_MODE=global restores serial semantics), per-session cancellation, strict device binding on enqueue, unified stdio piping with tee to trace stdout.log, and shutdown cleanup of all active runs. Fix _pid_exists exception order so PermissionError is treated as alive. - mcp_server reliability: spawn watchdog kills runners stuck before interpreter startup and releases device reservations; task state reconcile uses psutil liveness, prefers runner-reported pids, and adds a 45s startup grace to stop false failure reports on Windows; conversation_id is now optional end-to-end; explicit device_serial is validated at submission. - CLI install now writes ~/.claude/rules/artemis.md for Claude Code and migrates/removes the legacy CLAUDE.md injection block; ~/.claude.json is never rewritten when it fails to parse. - Tests: Windows-portable stdio lifecycle and path assertions; new suites for action layer, prompt assembly, spawn watchdog, and task reconcile (895 passed, 3 skipped).
This commit is contained in:
1 parent
60312abbb0
commit
cbdca1497c
52 files changed
+5735
-1649
No files matched your search
@@ -15,7 +15,7 @@ You operate in a dynamic observation-reasoning-execution loop. **CRITICAL RULE:
|
||||
# 2. DYNAMIC ENVIRONMENT & TIME AWARENESS
|
||||
Keep in mind that the mobile device is an asynchronous, constantly evolving environment. However, your actions entail an inherent "turn latency" (a 3–7 second delay caused by model inference and network transmission between normal consecutive actions).
|
||||
- **State Drift & Progression Tolerance**: While you think, video playback advances, banners scroll, and web pages finish loading. If the latest observation has progressed further than expected (e.g., an ad automatically skipped or a redirect landed), **do not treat it as an error**. Adapt smoothly to the live screen in front of you.
|
||||
- **Timed Waiting & Transitions**: When facing mid-transition animations, loading spinners, required watch durations, or ad countdown timers, use the precise `wait_for_delay` command to allow time to pass or the UI state to stabilize. Some transition states are subtle and may even resemble a phone bug or system freeze; it is always advisable to wait first, and only consider retrying or choosing an alternative path if the state remains broken.
|
||||
{% if "wait_for_delay" in available_tools %}- **Timed Waiting & Transitions**: When facing mid-transition animations, loading spinners, required watch durations, or ad countdown timers, use the precise `wait_for_delay` command to allow time to pass or the UI state to stabilize. Some transition states are subtle and may even resemble a phone bug or system freeze; it is always advisable to wait first, and only consider retrying or choosing an alternative path if the state remains broken.{% endif %}
|
||||
- **Time-Sensitive Tasks & Transient UI**: First, you must understand that each of your decision turns takes several seconds of model inference and network delay, while the mobile device state is continuously changing. A target element that you see on the current screenshot (e.g., auto-dismissing overlays, temporary message bars, menus or control bars with short timeouts) may have already vanished by the time your physical click command reaches the device! This is one of the most common reasons why the UI does not change as expected after a single step. We call these **Time-Sensitive Tasks**.
|
||||
- *Mental Model*: When facing these tasks, any attempt to interact step-by-step (such as tapping once to wake up a menu/bar and waiting for the next turn to tap the target button) WILL fail because the element will auto-dismiss during your decision delay. You MUST combine the summon action on the trigger element (`Trigger`) and the target action (`Target`) into a single atomic sequence.
|
||||
- *Chained Execution*: Use `click_sequence(sequence=[[trigger_x, trigger_y], [target_x, target_y]])` in `0-1000` normalized scale to execute both taps consecutively with a 50ms delay, catching the transient element before the timeout closes it.
|
||||
@@ -32,7 +32,7 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
|
||||
1. **Normalized Coordinates (`0-1000` Scale)**: Always use normalized `[x, y]` coordinates in `0-1000` scale for any location-based target (e.g., `target=[500, 600]` or `click_sequence=[[300, 400], [500, 600]]`).
|
||||
2. **Do not determine element coordinates based on guesswork.** If the element is visible in the screenshot but its coordinates are missing from the provided XML tree, please use the `ask_explorer` tool to locate the element first.
|
||||
3. **Screenshot Pruning Awareness**: To optimize speed, the system **prunes old intermediate screenshots from your history**—you only ever see the *latest* live screenshot and UI tree. Therefore, if you spot critical text in an earlier step that you will need later (e.g., verification codes, specific titles, search lists), **you must explicitly verbalize and write those details into your text thinking/notes** so they persist across turns.
|
||||
4. **App Launching**: To start or reset an application, always use `manage_app(action="launch", app_name="...")` or `action="stop"`. Do not manually swipe across home screens.
|
||||
{% if "manage_app" in available_tools %}4. **App Launching**: To start or reset an application, always use `manage_app(action="launch", app_name="...")` or `action="stop"`. Do not manually swipe across home screens.{% endif %}
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -36,10 +36,8 @@ from langchain_core.messages import (
|
||||
from artemis.agents.flash.context_compressor import compress_flash_messages
|
||||
from artemis.agents.flash.summarizer import VisualStepSummarizer
|
||||
from artemis.agents.validator.tool_declarations import (
|
||||
ACTION_TOOL_NAMES,
|
||||
ASK_EXPLORER_TOOL,
|
||||
CLICK_SEQUENCE_TOOL,
|
||||
MobileActionExecutor,
|
||||
REPORT_TASK_STATUS_TOOL,
|
||||
VALIDATOR_TOOLS_DECLARATION,
|
||||
capture_screenshot_and_parse_ui,
|
||||
@@ -51,6 +49,7 @@ from artemis.controllers.unified_controller import UnifiedMobileController
|
||||
from artemis.data_engine.trace import trace
|
||||
from artemis.graph.perception import _check_injected_instruction_file
|
||||
from artemis.graph.state import State
|
||||
from artemis.mcp.action_executor import McpActionExecutor
|
||||
from artemis.services.llm import (
|
||||
RobustChatModelWrapper,
|
||||
get_google_llm,
|
||||
@@ -78,7 +77,7 @@ class FlashRunner:
|
||||
self.step_summarizer_cfg = StepSummarizerConfig()
|
||||
|
||||
self.controller = UnifiedMobileController(ctx)
|
||||
self.executor = MobileActionExecutor(ctx, self.controller)
|
||||
self.executor = McpActionExecutor(ctx, self.controller)
|
||||
self.summarizer = (
|
||||
VisualStepSummarizer(ctx, model_name=self.step_summarizer_cfg.model)
|
||||
if self.step_summarizer_cfg.enabled
|
||||
@@ -90,6 +89,14 @@ class FlashRunner:
|
||||
tools.insert(1, CLICK_SEQUENCE_TOOL)
|
||||
tools.append(ASK_EXPLORER_TOOL)
|
||||
tools.append(REPORT_TASK_STATUS_TOOL)
|
||||
# With an actuator installed, drop declarations for device actions the
|
||||
# backend does not implement (and append its extension tools); without
|
||||
# one, the full historical declaration set is kept.
|
||||
actuator = getattr(self.ctx, "actuator", None)
|
||||
if actuator is not None:
|
||||
from artemis.mcp.action_manifest import filter_declarations
|
||||
|
||||
tools = filter_declarations(tools, actuator, "flash")
|
||||
return tools
|
||||
|
||||
def _prune_intermediate_screenshots(self, messages: list[BaseMessage]) -> None:
|
||||
@@ -144,10 +151,15 @@ class FlashRunner:
|
||||
}
|
||||
)
|
||||
|
||||
# Render System Prompt
|
||||
# Render System Prompt. Tool-teaching segments are gated on the available
|
||||
# tool set so an absent tool leaves no trace in the prompt; until an actuator
|
||||
# is wired in, the full manifest set reproduces the historical prompt.
|
||||
prompt_path = Path(__file__).parent / "flash_runner.md"
|
||||
prompt_template = prompt_path.read_text(encoding="utf-8")
|
||||
system_prompt = Template(prompt_template).render(goal=self.goal)
|
||||
available_tools = frozenset(t.name for t in tools_declaration)
|
||||
system_prompt = Template(prompt_template).render(
|
||||
goal=self.goal, available_tools=available_tools
|
||||
)
|
||||
|
||||
messages: list[BaseMessage] = [
|
||||
SystemMessage(content=system_prompt),
|
||||
@@ -432,8 +444,12 @@ class FlashRunner:
|
||||
try:
|
||||
exec_result = await self.executor.execute(name, args, tc_id, state)
|
||||
|
||||
# Dynamic dispatch set: manifest device actions plus any backend
|
||||
# extension tools, so extension steps are recorded like actions.
|
||||
action_names = self.executor.action_tool_names
|
||||
|
||||
post_img_bytes = exec_result.screenshot_bytes
|
||||
if not post_img_bytes and name in ACTION_TOOL_NAMES:
|
||||
if not post_img_bytes and name in action_names:
|
||||
try:
|
||||
controller = UnifiedMobileController(self.ctx)
|
||||
screen_data = await controller.get_screen_data()
|
||||
@@ -447,7 +463,7 @@ class FlashRunner:
|
||||
|
||||
# Record telemetry / step in DataEngine
|
||||
recorded_step_id = None
|
||||
if self.ctx.data_engine and name in ACTION_TOOL_NAMES:
|
||||
if self.ctx.data_engine and name in action_names:
|
||||
try:
|
||||
if self.ctx.data_engine.current_step_id is None:
|
||||
self.ctx.data_engine.allocate_step_id()
|
||||
@@ -517,7 +533,7 @@ class FlashRunner:
|
||||
logger.warning(f"Error recording step in FlashRunner: {step_err}")
|
||||
|
||||
# ⚡ Non-blocking dispatch of objective visual transition summarizer
|
||||
if self.summarizer and name in ACTION_TOOL_NAMES:
|
||||
if self.summarizer and name in action_names:
|
||||
action_sequence += 1
|
||||
self.summarizer.dispatch(
|
||||
step_number=action_sequence,
|
||||
|
||||
Reference in new issue
Block a user