feat: introduce MCP server, cross-platform runtime, and Showcase UI reorganization

- Showcase UI Reorganization:
  * Move web UI dashboard from root directory into apps/showcase_ui/ for cleaner project structure
  * Add 'artemis ui' CLI command (artemis/interfaces/cli/commands/ui.py) and launcher scripts for the interactive showcase console
- Standalone MCP Server & Notifiers (mcp_server/):
  * Implement dedicated MCP server package with mobile automation tools (device_state, inspect_trace, task_manager, task_runner)
  * Add multi-channel notifiers (AgentAPI, Desktop notifications, Webhooks, File output, and Composite routing)
  * Introduce background task runner and trace store for autonomous mobile execution monitoring
- Cross-Platform Runtime & Toolchain Support:
  * Add artemis/platform/ with OS-specific implementations for Linux, macOS (Darwin), and Windows
  * Add artemis/runtime/supervisor.py for resilient process and agent runtime supervision
  * Add artemis/toolchain/ with dependency resolution and SDK descriptors
  * Add cross-platform launcher scripts (start.sh, start.bat) and installer scripts (scripts/install_deps.*)
- Core Enhancements & Diagnostics:
  * Enhance coordinate normalization, OCR API fallback, and perception pipeline
  * Add credential validation (credentials_validator.py) and system diagnostics
  * Update bilingual documentation (README.md, README_CN.md) and architecture diagrams
- Test Suite Expansion:
  * Add integration tests for cloud gateway, remote ADB, and Artemis login
  * Add unit tests covering core diagnostics, platform detection, toolchain, credentials validator, optional OCR, and MCP tools

artemis/agents/checker/checker.json

@@ -1,4 +1,4 @@
 {
   "checker_system": "You are a precise and reliable verification expert (Checker). Your task is to determine whether the Operator has successfully completed the active subgoal by combining visual evidence from the current screenshot with data stored in persistent notes.",
-  "checker_prompt": "# Inputs\n- Active Subgoal: {{ subgoal }}\n- Execution History:\n{{ plan_and_history }}\n- Dialogue History: {{ dialogue_history }}\n- Visual State: The provided screenshot.\n\n# Guidelines\n- **Visual + Data Verification**: You must combine the screenshot with note contents. If the subgoal involves extracting, saving, or recording information (e.g., \"save contacts\", \"record address\"), you **MUST** verify that the content actually exists in the note. Do not assume success just because the information is visible on the screen. If the signs of success are not obvious, you can check the history, especially the most recent steps, to see if the operator performed their own check and whether that check was reasonable, to determine if the subgoal was truly successful.\n- **Visual Priority (Anti-History-Bias)**: Your primary ground truth is the visual evidence on the **Current Screenshot**. The `Execution History` lists previous successes or failures, which may be outdated due to dynamic background recovery or self-healing. If the screenshot demonstrably shows the active subgoal's target state is achieved, you **MUST** mark `success: true`, regardless of previous failed steps. Trust what you see, not past FAILED texts.\n- **Constraint Verification**: For subgoals containing specific parameters (such as names, brands, topics, events, numbers, or dates), confirm that the visible text precisely reflects these specified details. A successful state is defined by a clear, direct match with the subgoal's constraints.\n- **Realistic Expectations**: You cannot scroll the screen. If a subgoal requires saving all items but only some are visible on the screen, verify that at least the **visible items** are recorded in the note. As long as the visible part is recorded and there are no obvious errors, consider it a success. Do not fail the task simply because off-screen content cannot be verified.\n- **Success Criteria**: Set `success` to `true` if the core objective of the subgoal is met and it does not block subsequent steps. Set `success` to `false` if the state is incorrect, required data was not saved, or the operator is stuck/looping.\n\n# Output Protocol\nInvoke the `submit_verification` tool with `success` (boolean) and `reason` (string) to submit your final decision. If you need to inspect note data first, use `read_note`."
+    "checker_prompt": "# Inputs\n- Active Subgoal: {{ subgoal }}\n- Execution History:\n{{ plan_and_history }}\n- Dialogue History: {{ dialogue_history }}\n- Visual State: The provided screenshot.\n\n# Guidelines\n- **Visual + Data Verification**: You must combine the screenshot with note contents. If the subgoal involves extracting, saving, or recording information (e.g., \"save contacts\", \"record address\"), you **MUST** verify that the content actually exists in the note. Do not assume success just because the information is visible on the screen. If the signs of success are not obvious, you can check the history, especially the most recent steps, to see if the operator performed their own check and whether that check was reasonable, to determine if the subgoal was truly successful.\n- **Visual Priority (Anti-History-Bias)**: Your primary ground truth is the visual evidence on the **Current Screenshot**. The `Execution History` lists previous successes or failures, which may be outdated due to dynamic background recovery or self-healing. If the screenshot demonstrably shows the active subgoal's target state is achieved, you **MUST** mark `success: true`, regardless of previous failed steps. Trust what you see, not past FAILED texts.\n- **Constraint Verification**: For subgoals containing specific parameters (such as names, brands, topics, events, numbers, or dates), confirm that the visible text precisely reflects these specified details. A successful state is defined by a clear, direct match with the subgoal's constraints.\n- **Iteration & Monitoring Subgoals Validation**: For concrete iteration sub-tasks (e.g., inspecting Candidate #1) or periodic polling subgoals (e.g., Polling Round #1 check), verify whether the Operator correctly executed the step and logged the progress/findings into notes or the task plan. Do NOT mark a monitoring or iteration step as failed simply because the overall monitoring session is ongoing or no new incoming events were detected in that specific polling cycle. As long as the discrete check was performed and recorded, mark `success: true`.\n- **Realistic Expectations**: You cannot scroll the screen. If a subgoal requires saving all items but only some are visible on the screen, verify that at least the **visible items** are recorded in the note. As long as the visible part is recorded and there are no obvious errors, consider it a success. Do not fail the task simply because off-screen content cannot be verified.\n- **Success Criteria**: Set `success` to `true` if the core objective of the subgoal is met and it does not block subsequent steps. Set `success` to `false` if the state is incorrect, required data was not saved, or the operator is stuck/looping.\n\n# Output Protocol\nInvoke the `submit_verification` tool with `success` (boolean) and `reason` (string) to submit your final decision. If you need to inspect note data first, use `read_note`."
 }

artemis/agents/checker/checker.py

@@ -42,7 +42,7 @@ from artemis.tools.tool_wrapper import (
 )
 from artemis.utils.logger import get_logger
 from artemis.utils.notes import get_note_file_path, get_notes_dir
-from artemis.utils.ocr_api import perform_ocr
+from artemis.utils.ocr_api import is_ocr_configured, perform_ocr
 from artemis.utils.ocr_xml_fusion import (
     _crop_image_remove_status_bar,
     _detect_status_bar_height,
@@ -143,18 +143,19 @@ async def run_async_check(
             height = device_data.height
 
             ocr_results = []
-            try:
-                status_bar_height = _detect_status_bar_height(xml_hierarchy, height)
-                if status_bar_height > 0:
-                    cropped_b64, _, _ = _crop_image_remove_status_bar(
-                        screenshot_b64, status_bar_height
-                    )
-                    raw_ocr_results = await perform_ocr(cropped_b64)
-                    ocr_results = _map_coordinates_back(raw_ocr_results, status_bar_height)
-                else:
-                    ocr_results = await perform_ocr(screenshot_b64)
-            except Exception as e:
-                logger.error(f"Checker OCR failed, proceeding without OCR: {e}")
+            if is_ocr_configured():
+                try:
+                    status_bar_height = _detect_status_bar_height(xml_hierarchy, height)
+                    if status_bar_height > 0:
+                        cropped_b64, _, _ = _crop_image_remove_status_bar(
+                            screenshot_b64, status_bar_height
+                        )
+                        raw_ocr_results = await perform_ocr(cropped_b64)
+                        ocr_results = _map_coordinates_back(raw_ocr_results, status_bar_height)
+                    else:
+                        ocr_results = await perform_ocr(screenshot_b64)
+                except Exception as e:
+                    logger.warning(f"Checker OCR failed, proceeding without OCR: {e}")
 
             fused_xml = fuse_ocr_with_xml(xml_hierarchy, ocr_results)
 
@@ -356,10 +357,10 @@ async def run_async_check(
                     tool_status = "success"
                     try:
                         if ":" in tool_name:
-                            tool_to_run = get_tool_by_name(tool_name, traced_tools)
+                            tool_to_run = get_tool_by_name(tool_name, tools_to_bind)
                         else:
                             tool_to_run = next(
-                                (t for t in traced_tools if t.name == tool_name),
+                                (t for t in tools_to_bind if t.name == tool_name),
                                 None,
                             )
                         if tool_to_run:

artemis/agents/diagnoser/diagnoser.json

@@ -11,6 +11,7 @@
     "- **No Direct Operations**: Do not perform user actions or modify device state (e.g., tap, swipe, text input, BACK/HOME keys, or force-starting activities). Focus solely on analysis.",
     "- **Tool Constraints**: Video analysis is expensive. When calling the video analyzer, you MUST provide a specific, narrow timeframe to minimize cost.",
     "- **Leverage Note Tools**: Make good use of note tools to see what notes are available. When diagnosing, open and cross-reference relevant notes to ensure you have grasped all necessary information and context.",
+    "- **Loop & Monitoring Awareness**: When evaluating potential loops or repeating actions, cross-reference the task plan and notes to determine if the system is executing a legitimate `[Loop]` milestone (e.g., candidate iteration, periodic list polling with wait intervals, or continuous swipes for battery discharge). Do NOT misdiagnose planned iterations or paced polling checks as bugs or screen freezes.",
     "- **Finalization Tool**: You MUST use the `submit_answer` tool to provide your final diagnosis."
   ],
   "checker_feedback_template": [

artemis/agents/explorer/constants.py

@@ -32,14 +32,14 @@ EXPLORE_DESCRIPTIONS = {
     "pro": {
         "description": (
             "[EXPLORER] Call this to ask the UI Explorer agent to locate"
-            " coordinates on the screen layout by executing XML, OCR,"
+            " coordinates on the screen layout by executing UI search,"
             " coordinate search, and object detection."
         ),
         "query_description": (
             "The target element or information to search for, including descriptions."
         ),
         "rule_prompt": (
-            "Call the explorer tool to search, OCR, or inspect it (maximum {max_iterations} tries)."
+            "Call the explorer tool to search or inspect it (maximum {max_iterations} tries)."
         ),
         "version_prompt": (
             "You are running with a maximum of {max_iterations} turns: on your"

artemis/agents/explorer/explorer.py

@@ -53,7 +53,7 @@ from artemis.mcp.xml_search_server import (
     search_ui as search_ui_func,
 )
 from artemis.utils.logger import get_logger
-from artemis.utils.ocr_api import perform_ocr
+from artemis.utils.ocr_api import is_ocr_configured, perform_ocr
 from artemis.utils.ocr_xml_fusion import fuse_ocr_with_xml
 from artemis.utils.visualization import draw_dots, format_minimal_list_with_points
 
@@ -73,7 +73,7 @@ UNIVERSAL_EXPLORER_TOOLS = [
                 "properties": {
                     "search_query": {
                         "type": "string",
-                        "description": "Required. Text query to search for in XML/OCR elements.",
+                        "description": "Required. Text query to search for in UI elements.",
                     },
                     "nx": {
                         "type": "integer",
@@ -234,8 +234,8 @@ class Explorer:
                 " in parallel, including database search, coordinate search,"
                 " and visual object detection on the screen.\n\nOutput"
                 " Format:\nReturns:'text' (str): A single string consolidating"
-                " results with 0-1000 normalized coordinates, including xml"
-                " search matches labeled [X], ocr search matches labeled [O],"
+                " results with 0-1000 normalized coordinates, including UI"
+                " search matches labeled [X],"
                 " coordinate search matches labeled [X], and visually detected"
                 " object targets labeled [D].\n(Annotated screenshots are also"
                 " added to the context containing corresponding colored label"
@@ -246,7 +246,7 @@ class Explorer:
                 properties={
                     "search_query": types.Schema(
                         type=types.Type.STRING,
-                        description=("Required. Text query to search for in XML/OCR elements."),
+                        description=("Required. Text query to search for in UI elements."),
                     ),
                     "nx": types.Schema(
                         type=types.Type.INTEGER,
@@ -463,10 +463,10 @@ class Explorer:
         llm_config = getattr(ctx, "llm_config", None)
         llm_cfg = getattr(llm_config, "explorer", None) if llm_config else None
         model_str = (
-            llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.6-flash"
+            llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.7-flash"
         ).lower()
         self.model_name = (
-            llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.6-flash"
+            llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.7-flash"
         )
         if "/" in self.model_name:
             self.model_name = self.model_name.split("/")[-1]
@@ -514,8 +514,11 @@ class Explorer:
             ]
 
     def get_exposed_tools(self, only_submit: bool = False) -> list[types.FunctionDeclaration]:
-        """Returns the list of tools, filtering out denylisted ones."""
-        tools = [tool for tool in self.TOOLS if tool.name not in self.denylisted_tools]
+        """Returns the list of tools, filtering out denylisted and unconfigured ones."""
+        denylisted = set(self.denylisted_tools)
+        if not is_ocr_configured():
+            denylisted.add("get_ocr_list")
+        tools = [tool for tool in self.TOOLS if tool.name not in denylisted]
         if only_submit:
             tools = [tool for tool in tools if tool.name == "submit_answer"]
         return tools
@@ -855,7 +858,7 @@ class Explorer:
 
         xml_text = results[0].get("text", "No matches found.")
         if xml_text and xml_text != "No matches found.":
-            text_parts.append(f"XML/OCR Text Search Results are: {xml_text}")
+            text_parts.append(f"Text Search Results are: {xml_text}")
 
         coord_text = results[1].get("text", "No elements found.")
         if coord_text and coord_text != "No elements found.":
@@ -1012,7 +1015,7 @@ class Explorer:
             record = storage.get_ui_record(self.image_name)
             if not record or not record.ocr_result:
                 return {
-                    "text": ("No OCR results found in Data Engine for the current screen."),
+                    "text": "No text elements detected on the screen.",
                     "image_path": None,
                 }
 
@@ -1143,11 +1146,12 @@ class Explorer:
         """Executes Explorer reasoning loop via Universal LangChain ChatModel."""
         llm = get_llm(self.ctx, name="explorer")
 
-        # Filter universal tools based on denylist
+        # Filter universal tools based on denylist and OCR configuration
+        denylisted = set(self.denylisted_tools)
+        if not is_ocr_configured():
+            denylisted.add("get_ocr_list")
         exposed_tools = [
-            t
-            for t in UNIVERSAL_EXPLORER_TOOLS
-            if t["function"]["name"] not in self.denylisted_tools
+            t for t in UNIVERSAL_EXPLORER_TOOLS if t["function"]["name"] not in denylisted
         ]
         bound_llm = llm.bind_tools(exposed_tools)
 
@@ -1447,6 +1451,9 @@ class Explorer:
                 "inspect_region",
             }
 
+        if not is_ocr_configured():
+            version_denylist.add("get_ocr_list")
+
         try:
             denylisted_config = (
                 self.ctx.agent_config.denylisted_tools.get("explorer", [])
@@ -1517,7 +1524,7 @@ class Explorer:
                 if "/" in fallback_model:
                     fallback_model = fallback_model.split("/")[-1]
         else:
-            model_name = "gemini-3.6-flash"
+            model_name = "gemini-3.7-flash"
 
         # 5. Construct Prompt & Initial Message List
         prompt_path = Path(__file__).parent / "explorer.json"
@@ -1567,13 +1574,20 @@ class Explorer:
                 ui_tree = record.ui_tree
                 ocr_results = getattr(record, "ocr_result", None)
                 if ocr_results is None:
-                    try:
-                        logger.info("Previous screenshot OCR is missing. Running OCR on-the-fly...")
-                        with open(screenshot_path, "rb") as img_file:
-                            img_b64 = base64.b64encode(img_file.read()).decode("utf-8")
-                        ocr_results = await perform_ocr(img_b64, client=self.http_client)
-                    except Exception as ocr_err:
-                        logger.error(f"On-the-fly OCR failed for previous screenshot: {ocr_err}")
+                    if is_ocr_configured():
+                        try:
+                            logger.info(
+                                "Previous screenshot OCR is missing. Running OCR on-the-fly..."
+                            )
+                            with open(screenshot_path, "rb") as img_file:
+                                img_b64 = base64.b64encode(img_file.read()).decode("utf-8")
+                            ocr_results = await perform_ocr(img_b64, client=self.http_client)
+                        except Exception as ocr_err:
+                            logger.error(
+                                f"On-the-fly OCR failed for previous screenshot: {ocr_err}"
+                            )
+                            ocr_results = []
+                    else:
                         ocr_results = []
 
                 fused_xml = fuse_ocr_with_xml(ui_tree, ocr_results or [])

artemis/agents/flash/runner.py

@@ -22,6 +22,7 @@ import asyncio
 import base64
 import json
 from pathlib import Path
+import re
 import uuid
 
 from jinja2 import Template
@@ -439,13 +440,13 @@ class FlashRunner:
                                 elif kind == "direction" and isinstance(target_val, str):
                                     g_lower = target_val.lower()
                                     if "up" in g_lower:
-                                        norm_coords = [600, 800, 600, 200]
+                                        norm_coords = [600, 700, 600, 300]
                                     elif "down" in g_lower:
-                                        norm_coords = [600, 200, 600, 800]
+                                        norm_coords = [600, 300, 600, 700]
                                     elif "left" in g_lower:
-                                        norm_coords = [800, 500, 200, 500]
+                                        norm_coords = [750, 500, 250, 500]
                                     elif "right" in g_lower:
-                                        norm_coords = [200, 500, 800, 500]
+                                        norm_coords = [250, 500, 750, 500]
                                     if norm_coords:
                                         norm_start = norm_coords[:2]
                                         norm_end = norm_coords[2:]

artemis/agents/image_processor/image_processor.md

@@ -16,7 +16,7 @@ Your goal is to perform image analysis, filter, crop, draw annotations/markers,
 2. **DO NOT** use `cv2.resize()` to scale the image.
 Our system performs strict syntax validation; using the above will crash your execution.
 
-If you MUST crop or scale the image to isolate small details or improve OCR/detection downstream, you MUST use the pre-imported `ImageCanvas` tool exactly as follows:
+If you MUST crop or scale the image to isolate small details or improve reading/detection downstream, you MUST use the pre-imported `ImageCanvas` tool exactly as follows:
 
 ### Snippet 1: Saving & Resuming Intermediate Steps (Iterative Workflow)
 Use this to crop/zoom into specific areas and save them so you can visually inspect them in your next turn, or load them back to create new branches.

artemis/agents/operator/operator.json

@@ -1,4 +1,4 @@
 {
-  "main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List/Text/Complex UI Exploration Strategy (Zero Omission, Zero Duplication, Single-Pass Rule)**:\n  When traversing large lists, long text bodies, or continuous content streams, adhere strictly to the **Single-Pass Exploration Standard**:\n  1. **Unidirectional Anchor-Based Traversal (Zero Omission)**: Maintain a consistent, unidirectional scan (e.g., scrolling steadily from top to bottom). Before each scroll, use the bottom-most visible item as a visual anchor. Ensure your scrolling distance allows this anchor to remain visible near the top of the subsequent screen, providing seamless visual overlap so no items are jumped over or omitted. Conclude exploration when reaching the definite boundary (e.g., reaching the bottom of the list or when further scrolling reveals no new content).\n  2. **Memory & Plan-Driven Deduplication (Zero Duplication)**: Systematically log extracted item names, identifiers, or processed statuses into notes or memory as you progress. Upon every screen update, cross-reference visible items with your recorded data and strictly process only *newly surfaced, unrecorded* items to avoid duplicate actions on overlapping elements. Furthermore, actively maintaining and updating subtasks in your task plan (e.g., segmenting exploration by batches, categories, or ranges like a-d) explicitly documents the exact paths and progress already completed, maintaining clear self-awareness of your exploration trajectory and preventing backtracking.\n  3. **Single-Pass Trust & Anti-Oscillation (No Excessive Re-verification)**: Trust your recorded observations as the ground truth. Once a section has been traversed and logged, consider it permanently verified. Never scroll back anLine truncated
-  "troubleshooter_template": "# ROLE\nYou are the Expert Android Device Troubleshooter. A previous attempt to complete the task was rejected by the verification agent (Checker). Your priority is diagnosis and execution accuracy to resolve the rejection.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List/Text/Complex UI Exploration Strategy (Zero Omission, Zero Duplication, Single-Pass Rule)**:\n  When traversing large lists, long text bodies, or continuous content streams, adhere strictly to the **Single-Pass Exploration Standard**:\n  1. **Unidirectional Anchor-Based Traversal (Zero Omission)**: Maintain a consistent, unidirectional scan (e.g., scrolling steadily from top to bottom). Before each scroll, use the bottom-most visible item as a visual anchor. Ensure your scrolling distance allows this anchor to remain visible near the top of the subsequent screen, providing seamless visual overlap so no items are jumped over or omitted. Conclude exploration when reaching the definite boundary (e.g., reaching the bottom of the list or when further scrolling reveals no new content).\n  2. **Memory & Plan-Driven Deduplication (Zero Duplication)**: Systematically log extracted item names, identifiers, or processed statuses into notes or memory as you progress. Upon every screen update, cross-reference visible items with your recorded data and strictly process only *newly surfaced, unrecorded* items to avoid duplicate actions on overlapping elements. Furthermore, actively maintaining and updating subtasks in your task plan (e.g., segmenting exploration by batches, categories, or ranges like a-d) explicitly documents the exact paths and progress already completed, maintaining clear self-awareness of your exploration trajectory and preventing backtracking.\n  3. **Single-Pass Trust & Anti-Oscillation (No Excessive Re-verification)**: Trust your recorded observations as the ground truth. Once a section has been traversed and loggedLine truncated
+  "main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n  1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor. Calibrate your scroll distance so that this anchor remains visible near the top of the subsequent screen, creating a seamless visual overlap that guarantees zero missed items. Terminate exploration when a definitive boundary is reached (e.g., list bottom reached or no new content appears after scrolling).\n  2. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n  3. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\". Only perform incremental re-verification if the list has been refreshed or contentLine truncated
+  "troubleshooter_template": "# ROLE\nYou are the Expert Android Device Troubleshooter. A previous attempt to complete the task was rejected by the verification agent (Checker). Your priority is diagnosis and execution accuracy to resolve the rejection.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n  1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor. Calibrate your scroll distance so that this anchor remains visible near the top of the subsequent screen, creating a seamless visual overlap that guarantees zero missed items. Terminate exploration when a definitive boundary is reached (e.g., list bottom reached or no new content appears after scrolling).\n  2. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n  3. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\". Only perform incremental rLine truncated
 }

artemis/agents/operator/operator.py

@@ -46,7 +46,10 @@ from artemis.tools.tool_wrapper import (
     get_tool_result_content,
     invoke_tool_with_injection,
 )
-from artemis.utils.coordinates import parse_swipe_parameters
+from artemis.utils.coordinates import (
+    compute_smart_swipe_coordinates,
+    parse_swipe_parameters,
+)
 from artemis.utils.decorators import wrap_with_callbacks
 from artemis.utils.file import create_snapshot
 from artemis.utils.logger import get_logger
@@ -1056,23 +1059,25 @@ class OperatorNode:
         def swipe(
             direction: Annotated[
                 Literal["up", "down", "left", "right"] | None,
-                "Smart directional swipe ('up', 'down', 'left', 'right') for general page"
-                " scrolling/browsing. Automatically computes safe swipe vectors (locking axes"
-                " to prevent sidebar/edge mis-clicks), retains a ~40% visual overlap anchor,"
-                " and prevents inertial flings: 'up' (drags bottom-to-top, scrolls down to reveal content below),"
-                " 'down' (drags top-to-bottom, scrolls up to reveal content above),"
-                " 'left' (drags right-to-left, scrolls right), 'right' (drags left-to-right, scrolls left).",
+                "Direction for scrolling and swiping: 'up' (drags bottom-to-top, scrolling down to reveal content below),"
+                " 'down' (drags top-to-bottom, scrolling up to reveal content above),"
+                " 'left' (drags right-to-left, scrolling right),"
+                " 'right' (drags left-to-right, scrolling left).",
             ] = None,
             start: Annotated[
                 list[int] | None,
-                "Start point normalized coordinates [start_x, start_y] in 0-1000 scale for precise,"
+                "Start normalized coordinates [start_x, start_y] in 0-1000 scale for precise,"
                 " local interactions (e.g. adjusting sliders, SeekBars, fine range selection, or drag-and-drop).",
             ] = None,
             end: Annotated[
                 list[int] | None,
-                "End point normalized coordinates [end_x, end_y] in 0-1000 scale for precise,"
+                "End normalized coordinates [end_x, end_y] in 0-1000 scale for precise,"
                 " local interactions (e.g. adjusting sliders, SeekBars, fine range selection, or drag-and-drop).",
             ] = None,
+            target: Annotated[
+                int | list[int] | str | None,
+                "Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within.",
+            ] = None,
             gesture: Annotated[
                 Literal["up", "down", "left", "right"] | list[int] | None,
                 "Backward-compatible swipe gesture: smart direction string ('up', 'down', 'left', 'right')"
@@ -1080,23 +1085,23 @@ class OperatorNode:
             ] = None,
             duration: Annotated[
                 int | None,
-                "Optional swipe/drag duration in milliseconds. If specified, the gesture will run at this exact speed."
-                " For drag-and-drop, list reordering, or sliding/adjusting sliders (e.g., volume, brightness, SeekBars),"
-                " set duration >= 1000 (e.g. 1500). If omitted for directional swipe, the system automatically"
-                " calculates the best duration based on the swipe distance to prevent unintended inertial flings.",
+                "Optional swipe/drag duration in milliseconds (default 800). For drag-and-drop,"
+                " list reordering, or sliding/adjusting sliders (e.g., volume, brightness, SeekBars),"
+                " set duration >= 1000 (e.g. 1500). If omitted for directional swipe, duration is computed automatically.",
             ] = None,
         ):
             """[ACTION] Perform a swipe, drag, or slider-adjustment gesture on the screen.
 
-            • Smart Directional Scrolling ('direction'): Best for general browsing and page scrolling. Automatically computes safe swipe vectors (locking axes to prevent edge/sidebar mis-clicks), retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings.
-            • Precise Coordinate Gestures ('start', 'end'): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars) or drag-and-drop / list reordering. Drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.
+            • Directional Scrolling ('direction'): Recommended for general browsing and standard page scrolling in most scenarios. Automatically computes safe swipe vectors and adaptive duration, retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings. Supports scoping to a sub-container via 'target'. If it fails on certain custom layouts, fall back to specifying exact coordinates ('start' and 'end') directly.
+            • Precise Coordinate Gestures ('start', 'end'): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars), drag-and-drop / list reordering, or as a reliable fallback when directional scrolling fails on specific containers. Always drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), swipe fully to the extreme boundary.
 
             Args:
-                direction: Smart directional scrolling ('up', 'down', 'left', 'right'). Automatically computes safe swipe vectors, retaining 40% of the previous screen's visible content as an overlapping anchor: 'up' (drags bottom-to-top, scrolling down to reveal content below), 'down' (drags top-to-bottom, scrolling up to reveal content above), 'left' (drags right-to-left), 'right' (drags left-to-right).
-                start: Start normalized coordinates [start_x, start_y] in 0-1000 scale for precise/local interactions.
-                end: End normalized coordinates [end_x, end_y] in 0-1000 scale for precise/local interactions.
+                direction: Smart directional scrolling ('up', 'down', 'left', 'right'). Automatically computes safe swipe vectors, retaining 40% visual overlap: 'up' (reveal content below), 'down' (reveal content above), 'left', 'right'.
+                start: Start normalized coordinates [start_x, start_y] in 0-1000 scale.
+                end: End normalized coordinates [end_x, end_y] in 0-1000 scale.
+                target: Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within.
                 gesture: Backward-compatible parameter: direction string OR custom coordinates list [start_x, start_y, end_x, end_y] in 0-1000 scale.
-                duration: Optional gesture duration in milliseconds. If omitted for directional swipe, the system automatically computes a reliable duration based on the swipe distance to prevent unintended inertial flings.
+                duration: Optional gesture duration in milliseconds (default 800).
             """
             return "Action Recorded"
 
@@ -1134,16 +1139,16 @@ class OperatorNode:
         def wait_for_delay(
             time_in_ms: Annotated[
                 int,
-                "The duration to wait in milliseconds (e.g., 2000 for 2"
-                " seconds, 5000 for 5 seconds). Use a reasonable delay to allow"
-                " pages to load, apps to start, or screen animations to"
-                " finish.",
+                "The exact duration to wait in milliseconds. Accurately convert the"
+                " required time duration into milliseconds based on your objective"
+                " or plan (e.g., 2000 for 2s, 5000 for 5s, 60000 for 1 minute,"
+                " 180000 for 3 minutes, 300000 for 5 minutes).",
             ],
         ):
-            """[ACTION] Pause execution and wait for a fixed duration.
+            """[ACTION] Pause execution and wait for a specified duration in milliseconds.
 
-            Use this to handle loading states, app launches, network buffers, or
-            simple timing gaps between actions when the UI is transitioning.
+            Use this whenever you need time to elapse—whether for UI loading, animations,
+            screen transitions, or longer scheduled delays and intervals specified in the task.
             """
             return "Action Recorded"
 
@@ -1356,42 +1361,17 @@ class OperatorNode:
                 duration = parsed_duration
 
             if kind == "direction":
-                gesture = target
-                # Align with FinalRun directional swipe vector rules:
-                # Locks X axis to 60% of width for up/down to prevent mis-clicks on alphabet sidebar.
-                # Horizontal Y axis is locked to 60% of height.
-                if gesture == "up":
-                    coords = [
-                        int(width * 0.6),
-                        int(height * 0.8),
-                        int(width * 0.6),
-                        int(height * 0.2),
-                    ]
-                elif gesture == "down":
-                    coords = [
-                        int(width * 0.6),
-                        int(height * 0.2),
-                        int(width * 0.6),
-                        int(height * 0.8),
-                    ]
-                elif gesture == "left":
-                    coords = [
-                        int(width * 0.8),
-                        int(height * 0.6),
-                        int(width * 0.2),
-                        int(height * 0.6),
-                    ]
-                elif gesture == "right":
-                    coords = [
-                        int(width * 0.2),
-                        int(height * 0.6),
-                        int(width * 0.8),
-                        int(height * 0.6),
-                    ]
-                else:
-                    return [], f"Error: Invalid swipe direction '{gesture}'."
-
-                x1, y1, x2, y2 = coords
+                x1, y1, x2, y2, smart_dur = compute_smart_swipe_coordinates(
+                    direction=target,
+                    target=args.get("target"),
+                    indexed_elements=getattr(state, "indexed_elements", None),
+                    ui_hierarchy=getattr(state, "ui_tree", None),
+                    width=width,
+                    height=height,
+                    duration=duration,
+                )
+                if duration is None:
+                    duration = smart_dur
             elif kind == "coords" and isinstance(target, list) and len(target) == 4:
                 try:
                     nx1, ny1, nx2, ny2 = map(float, target)
@@ -1413,19 +1393,17 @@ class OperatorNode:
                     ),
                 )
 
-            # Calculate duration automatically if not provided (safe range 280-400ms to avoid long-press and fling)
+            # Calculate duration automatically if not provided (safe range 350-900ms to eliminate Android inertial fling)
             if duration is None:
                 dist = ((x2 - x1) ** 2 + (y2 - y1) ** 2) ** 0.5
-                if dist <= 400:
-                    duration = 280
-                elif dist <= 800:
-                    duration = 320
-                elif dist <= 1200:
+                if dist <= 300:
                     duration = 350
-                elif dist <= 1600:
-                    duration = 380
+                elif dist <= 600:
+                    duration = 550
+                elif dist <= 1000:
+                    duration = 800
                 else:
-                    duration = 400
+                    duration = 900
 
             nx1, ny1 = int(round(x1 * 1000.0 / width)), int(round(y1 * 1000.0 / height))
             nx2, ny2 = int(round(x2 * 1000.0 / width)), int(round(y2 * 1000.0 / height))

artemis/agents/outputter/outputter.py

@@ -35,6 +35,7 @@ from artemis.tools.scratchpad import (
     get_update_note_tool_pure,
 )
 from artemis.tools.video_tool import get_video_analyzer_tool_pure
+from artemis.tools.tool_wrapper import invoke_tool_with_injection
 from artemis.utils.logger import get_logger
 from artemis.utils.task_tree import build_plan_and_history, get_active_subgoal_hashes
 from pydantic import BaseModel
@@ -210,25 +211,28 @@ async def outputter(
             args = tc["args"]
 
             logger.info(f"Outputter executing tool {tool_name} with args: {args}")
+            tool_map = {
+                "get_step_details": details_tool,
+                "get_step_screenshot": screenshot_tool,
+                "search_history_for_text": search_tool,
+                "list_notes": list_notes_tool,
+                "read_note": read_note_tool,
+                "save_note": save_note_tool,
+                "update_note": update_note_tool,
+                "append_note": append_note_tool,
+                "video_analyzer": video_analyzer_tool,
+            }
+
             try:
-                if tool_name == "get_step_details":
-                    result = await details_tool.ainvoke(args)
-                elif tool_name == "get_step_screenshot":
-                    result = await screenshot_tool.ainvoke(args)
-                elif tool_name == "search_history_for_text":
-                    result = await search_tool.ainvoke(args)
-                elif tool_name == "list_notes":
-                    result = await list_notes_tool.ainvoke(args)
-                elif tool_name == "read_note":
-                    result = await read_note_tool.ainvoke(args)
-                elif tool_name == "save_note":
-                    result = await save_note_tool.ainvoke(args)
-                elif tool_name == "update_note":
-                    result = await update_note_tool.ainvoke(args)
-                elif tool_name == "append_note":
-                    result = await append_note_tool.ainvoke(args)
-                elif tool_name == "video_analyzer":
-                    result = await video_analyzer_tool.ainvoke(args)
+                selected_tool = tool_map.get(tool_name)
+                if selected_tool:
+                    result = await invoke_tool_with_injection(
+                        tool=selected_tool,
+                        args=args,
+                        tool_call_id=tc["id"],
+                        state=graph_output,
+                        record_trace=True,
+                    )
                 else:
                     result = f"Error: Tool {tool_name} is not supported."
                 status = (

artemis/agents/planner/planner.json

@@ -1,18 +1,18 @@
 {
   "blocks": {
-    "role": "## Role\nYou are the Planner agent. Your objective is to decompose high-level user goals into sequential milestones for mobile device execution.",
+    "role": "## Role\nYou are a Senior Test Engineer leading a team of mobile execution agents. Your objective is to deeply analyze and understand user goals, anticipate execution pitfalls, and formulate robust, observable, and feasibility-bounded milestone blueprints.",
+    "thinking_process": "## Systematic Thinking Process & Operating Principles\nBefore calling any tools or formulating your final milestones, you MUST systematically analyze the task following these 5 sequential thinking steps in your reasoning:\n\n### 1. Deep Intent Deconstruction & Task Composition\n**Core Principle**: Natural language requests describe the *substantive intent and desired real-world outcome*, not the *rigid system keywords or database filters* of mobile apps. Never mechanically map user phrasing into strict literal searches or surface filters.\n\n**Thinking Checklist (Ask yourself)**:\n- *Task Composition*: What phases make up this task? Is it a sequence of one-off linear steps (e.g., open app, configure settings), an iterative loop over multiple items (e.g., inspecting a list of search results), or a periodic polling loop (e.g., refreshing to monitor new messages, waiting for battery level)?\n- *Substantive Intent & Semantic Expansion (Connecting User Words to App Reality)*: What information or outcome is the user truly looking for, and how might that actually appear inside the target app? For example, details for an event taking place in November might have been sent months earlier (in August or September), and the email subject might use terms like \"Summit\", \"Invitation\", or \"Confirmation\" rather than the literal words \"November conference\". Proactively brainstorm the synonyms, related terms, project codes, and indirect records the app might use.\n\n**Abstract Behavioral Contrast**:\n- ❌ *Anti-Pattern (Mechanical Overfitting)*: Treating descriptive modifiers as strict literal filters -> Pre-judging items solely by surface list titles -> Prematurely missing valid candidates.\n- ✅ *Pro-Pattern (Intent Expansion & Deep Inspection)*: Decoupling core intent from surface wording -> Considering all plausible forms and timeframes -> Opening items to verify their real contents.\n\n### 2. Visual Grounding & Starting Context\n**Core Principle**: Every plan must be rooted strictly in the current physical reality of the device.\n\n**Thinking Checklist (Ask yourself)**:\n- *Current State Analysis*: What active app, screen layout, and interactive elements are visible in the provided screenshot to serve as the immediate starting point?\n\n### 3. Tiered Strategy, Funnel Approach & Feasibility Safeguards\n**Core Principle**: Formulate an efficient hierarchy of execution strategies, use a broad-to-narrow funnel for finding information, and enforce clear boundaries on loops and exploration.\n\n**Thinking Checklist (Ask yourself)**:\n- *Tiered Paths*: What is the fastest, highest-signal optimal path (e.g., targeted in-app search, direct shortcuts)? If that fails or yields no results, what is the pragmatic fallback path (e.g., browsing categories, chronological scanning)?\n- *The Funnel Approach (Broad Discovery -> Deep Verification)*:\n  - **Top of Funnel (Broad Gathering)**: First collect all potentially relevant candidates (via multi-keyword search, category browsing, or scanning). Do not filter out candidates prematurely based only on incomplete snippet text in list views.\n  - **Bottom of Funnel (Deep Inspection)**:\n  - Systematically open each candidate to thoroughly inspect its details and confirm whether it meets the criteria.\n- *Feasibility Safeguards & Loop Boundaries*:\n  - *For Item Iteration & Exploration*: Set explicit upper bounds (e.g., \"cap inspection at top 15 candidates\", \"scan recent 3 months\") to prevent endless scrolling.\n  - *For Periodic Monitoring / Polling*: Specify the check action (e.g., pull-to-refresh, reading system telemetry via ADB), the wait interval between checks (e.g., check every 5 minutes to conserve resources), and the exit criteria (e.g., stop when condition is met, or continue until manually stopped).\n\n### 4. Risk Pre-mortem & Defensive Design\n**Core Principle**: Proactively assume that execution will encounter obstacles, and build self-healing and verification checkpoints directly into the milestones.\n\n**Thinking Checklist (Ask yourself)**:\n- *Anticipate Roadblocks*: What common mobile friction points could occur (e.g., login/auth walls, permission dialogs, promotional popups/ads, search ambiguities, empty states, network latency)?\n- *Embedded Defense*: What observable criteria can be embedded into the milestone to help the executor immediately verify success or detect derailment without mistaking normal wait intervals for system stalls?\n\n### 5. Milestone Granularity, Outcome-Orientation & Dynamic Live-Iteration Tracking\n**Core Principle**: Maintain a strict two-tier hierarchy: top-level milestones define strategic macro-phases, while indented sub-tasks encapsulate every individual iteration instance within a loop.\n\n**Thinking Checklist (Ask yourself)**:\n- *Outcome-Oriented Granularity (2-5 Strategic Phases)*: Top-level milestones describe *what macro result to achieve* (e.g., \"Search for candidate emails and extract confirmed itiLine truncated
+    "capabilities_operator": "## Execution Capabilities\n\nThe execution agent primarily interacts with the mobile device and analyzes screen recordings and logs. It can perform the following types of actions:\n- **UI Interaction**: Tapping, swiping, and entering text on the screen.\n- **App Management**: Launching and stopping specific applications.\n- **Information Retrieval & Analysis**: Reading on-screen text and UI hierarchy, searching the web for external information, querying system logs, and analyzing screen recordings.\n- **Terminal & Background Operations**: Executing arbitrary ADB shell commands on the device for exploration. It is capable of running long-running ADB commands asynchronously in the background as tasks.\n\n### Capability Boundaries & Planning Rules\n- **Cross-App Operations**: When a task requires interacting with multiple applications, split the operations into distinct milestones for each app. Avoid mixing actions for different apps in a single milestone.",
+    "output_protocol_initial": "## Task Plan & Scratchpad Protocol\n\n1. **Articulate Thinking**: Begin by reasoning through the 5-step **Systematic Thinking Process** to establish your strategy, funnel approach, bounds, and pre-mortem checks.\n2. **Designate Notes in Milestones & Pre-generate Answer Cards**: If the task involves collecting findings, logging periodic monitoring events, recording candidates, or analytical summaries, designate clear, descriptive note names directly in the milestone descriptions (e.g., `candidates_list`, `monitoring_log`, `schedule_summary`). You can and should invoke the `save_note` tool to pre-generate these structured template files (answer card / scratchpad skeletons with Markdown headers, sections, or tables) before submitting `task_plan`. This provides downstream executors with a clear, ready-to-fill schema.\n3. **Submit Plan**: Submit your initial plan by invoking the `save_note` tool with `key='task_plan'`. Avoid writing the plan only in your text response, as it will only be recognized by the system if submitted via the tool.\n\n### Plan Formatting Rules\n- Format the plan strictly as a Markdown task list starting with `- [ ]` for uncompleted items.\n- For linear macro-milestones: `- [ ] <Phase Description>`\n- For repetitive, iterative, or periodic polling milestones, tag the item with `[Loop]` and declare its Exit Boundary and Interval (e.g., `- [ ] [Loop] <Overall Loop / Monitoring Goal> (Exit: <Condition or Continuous until manually stopped>; Interval: <e.g., every 5 minutes>)`).\n- **Strict Two-Tier Hierarchy & Loop Subtask Contract**:\n  1. **Subtask Nesting Rule**: ALL individual iteration instances, polling checks, candidate inspections, or baseline setups MUST be formatted as 2-space indented sub-tasks (`  - [ ]`) directly nested under their parent `[Loop]` milestone.\n  2. **Mandatory First-Instance Initialization**: Every `[Loop]` milestone MUST be initialized with its first concrete child sub-task (e.g., `  - [ ] Inspect Candidate #1: ...` or `  - [ ] Polling Check #1 (Baseline & Initial Sync): ...`).\n  3. **No Flattened Cycles**: NEVER elevate individual checks, recurring polling cycles, or candidate inspections to top-level milestones outside the `[Loop]`. Top-level items are reserved exclusively for strategic macro-phases.\n- Ensure the content of the `task_plan` note contains only the task list, excluding introductions, explanations, conversational text, or headers.",
+    "examples_initial": "## Master Example: Demonstrating the Complete Cognitive Paradigm\n\n**Goal**: \"Find and summarize all schedule arrangements and venue details for the upcoming November tech conference from my emails.\"\n\n### Naive / Incorrect Approaches:\n1. *Flawed Semantics & Too Granular*:\n- [ ] Tap search icon in Gmail\n- [ ] Type \"November\"\n- [ ] Open first email and tap back\n*(Fails: Micromanages atomic UI taps; assumes emails for November arrangements must have \"November\" in the title or be received in November)*\n2. *Vague & Unbounded Trap*:\n- [ ] Open email and check all emails for conference details\n*(Fails: Unbounded scan trap; causes infinite scrolling and step budget exhaustion on mobile)*\n\n---\n\n### Correct Approach (Full Cognitive Thinking & Robust Plan):\n\n#### Step-by-Step Thinking:\n1. **Intent & Semantic Expansion**:\n   - *Core Objective*: Extract confirmed schedule dates, sessions, and venue info for the November tech conference.\n   - *Semantic Insight*: Conference communications are often sent months in advance (e.g., ticket confirmations or registration emails in August/September). The subject line may say \"Global Developer Summit\", \"Tech Expo Agenda\", or \"Registration Confirmed\" rather than literally \"November conference\".\n   - *Task Composition*: Linear setup (open email app and search) -> Iterative candidate inspection loop (examine each promising email) -> Final consolidation.\n2. **Visual Grounding**:\n   - The device screenshot shows the Home screen with the Gmail app icon visible. The plan must start by opening the email client.\n3. **Tiered Strategy, Funnel Approach & Safeguards**:\n   - *Tier 1 (Optimal)*: Use in-app search with rich semantic keywords (\"conference\", \"summit\", \"schedule\", \"agenda\", \"venue\", \"November\").\n   - *Tier 2 (Fallback)*: If search is unavailable or yields zero results, fallback to chronological scanning of recent 3 months.\n   - *Funnel (Broad Gathering -> Deep Inspection)*: Gather candidate emails from search results; do not discard candidates based solely on list previews. Open each candidate thread to inspect full body details.\n   - *Feasibility Boundary*: Cap the candidate email inspection pool at top 15 emails to prevent infinite scrolling.\n4. **Risk Pre-mortem**:\n   - *Risks*: Promotional marketing spam; conflicting drafts with preliminary dates; cancelled sessions.\n   - *Defensive Check*: Instruct downstream executor to filter out promotional newsletters, verify session timestamps, and confirm final venue address.\n5. **Milestone Granularity & Scratchpad Blueprint**:\n   - Focus milestones on business outcomes (search, inspect candidates, consolidate), avoiding micromanaged click-by-click steps.\n   - Use indented sub-tasks to guide the loop of inspecting each candidate email, initializing with Item #1 and adding subsequent items as they are identified. This prevents the agent from losing state or re-inspecting the same items.\n   - Designate note `november_conference_summary` as the destination for recording findings and pre-generate the structured template note.\n\n#### Tool Invocation:\n1. Pre-generate designated answer card note template:\n`save_note(key='november_conference_summary', content='# November Tech Conference Summary\\n\\n## Key Details\\n- Conference Name: \\n- Dates: \\n- Venue / Address: \\n\\n## Daily Schedule & Sessions\\n(To be populated during email inspection)\\n')`\n\n2. Submit milestone plan:\n`save_note(key='task_plan', content='- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\\n  - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\\n- [ ] Review and consolidate all extracted information in `november_conference_summary` into a clear, finalized conference itinerary')`\n\n#### Content of `task_plan` Note:\n- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\n  - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\n- [ ] ReviLine truncated
     "role_validator": "## Role\nYou are the Task Plan Validator. The Operator has attempted to modify the top-level task plan. Your job is to ensure the proposed new plan still aligns with the user's original intent and is viable.",
-    "guidelines": "## Operating Principles\n\n### 1. Visual Grounding (Start from Current State)\nBase all planning decisions strictly on the provided screenshot of the current device state. Your plan must begin from the current context of the screen.\n\n### 2. Milestone Granularity\nDefine high-level milestones rather than granular UI operations. The milestones you establish are essential checkpoints that the task must pass through; your goal is to keep the task on the right track, not to think about how the task should be completed step-by-step.\n\n### 3. Flexibility and Possibility Retention (Avoid Over-Specification)\nAvoid unnecessarily restricting the execution path or specific tools/applications unless strictly mandated by the user. For example, if the user asks to perform a task that can be accomplished via multiple interfaces (such as a native app or a web browser), describe the milestone in terms of the desired outcome rather than forcing a specific application or interface (e.g. forcing the use of a specific native app when a web browser could also serve). Allow the downstream execution agent the flexibility to choose the most viable path based on the dynamic state of the device.\n\n### 4. Heuristics for Complex & Iterative Operations\nWhen a milestone requires the downstream agent to perform complex iterations, maintain state, or map computed data back to the UI (e.g., adding a pre-selected list of items to a system), you MUST provide heuristics via indented sub-tasks. These sub-tasks should instruct the agent on the exact iterative process (e.g., searching for each item one by one from a note, applying it, and verifying the final state).",
-    "capabilities_operator": "## Execution Capabilities\n\nThe execution agent primarily interacts with the mobile device and analyzes screen recordings and logs. It can perform the following types of actions:\n- **UI Interaction**: Tapping, swiping, and entering text on the screen.\n- **App Management**: Launching and stopping specific applications.\n- **Information Retrieval & Analysis**: Using OCR to read text on screen, searching the web for external information, querying system logs, and analyzing screen recordings.\n- **Terminal & Background Operations**: Executing arbitrary ADB shell commands on the device for exploration. It is capable of running long-running ADB commands asynchronously in the background as tasks.\n\n### Capability Boundaries & Planning Rules\n- **Cross-App Operations**: When a task requires interacting with multiple applications, split the operations into distinct milestones for each app. Avoid mixing actions for different apps in a single milestone.",
-    "output_protocol_initial": "## Task Plan & Scratchpad\nSubmit your initial plan by invoking the `save_note` tool with `key='task_plan'`. Avoid writing the plan in your text response, as it will only be recognized if submitted via the tool.\n\nFormat the plan as a Markdown task list starting with `- [ ]` for uncompleted items. You can use indented sub-tasks (e.g., `  - [ ]`) to break down complex exploratory steps and provide heuristics to guide the downstream agent. Ensure the content of the `task_plan` note contains only the task list, excluding any introductions, explanations, or headers.\n\n### Creating Additional Notes\nYou are highly recommended to create additional empty notes (like answer sheets) for the downstream executor to fill out with findings, lists, or analytical information relevant to the task. For example, if the user asks you to analyze specific YouTube videos, you should pre-create an empty note with a descriptive name (e.g., `analysis_of_xxx_videos`), and make sure to put the note name directly inside your milestones to guide the downstream executor in filling out this answer sheet.",
-    "examples_initial": "## Examples\n\n### Example 1: Basic Task\n**Goal**: \"Send 'I will be late' to Alice on WhatsApp\"\n\n**Incorrect (Too granular)**:\n- [ ] Tap search icon\n- [ ] Type Alice\n- [ ] Tap Alice's avatar\n\n**Correct (Milestones)**:\n- [ ] Open the chat window with Alice in WhatsApp\n- [ ] Successfully send the message \"I will be late\" in Alice's chat window\n\n### Example 2: Information Retrieval & Scratchpad\n**Goal**: \"Find 10 nature videos on YouTube\"\n\n**Incorrect (Vague or directly jumping to end)**:\n- [ ] Open YouTube\n- [ ] Search nature videos and pick 10\n\n**Correct (Leveraging Scratchpad & Sub-tasks Heuristics)**:\n*(Note: As the Planner, pre-create the required empty notes `nature_videos_candidates` and `nature_videos` using the note-taking tools before completing the plan)*\n- [ ] Search for \"nature videos\" on YouTube\n- [ ] Save a list of 15 candidate videos into a temporary note called `nature_videos_candidates`\n  - [ ] Add candidate 1\n  - [ ] Add candidate 2\n  - [ ] (Continue until 15 candidates are found)\n- [ ] Evaluate candidate videos from `nature_videos_candidates` one by one, select exactly 10 correct ones, and save them into `nature_videos`\n  - [ ] Evaluate candidate 1: watch, filter out if irrelevant, save if correct\n  - [ ] Evaluate candidate 2\n  - [ ] (Continue evaluating one by one until 10 are selected)",
-    "validator_guidelines": "## Validation Principles\n- **Goal Alignment**: Ensure the new plan ultimately accomplishes the `Initial Goal` requested by the user. If the Operator has abandoned the core objective or fundamentally altered it, you must reject the plan.\n- **Logical Coherence**: Verify that the proposed milestones form a logical, achievable sequence to reach the goal. Reject the plan if it introduces contradictory steps, infinite loops, or obvious logical fallacies.\n- **Adaptive Acceptance**: The Operator operates in a dynamic environment and may encounter unexpected UI states. You should approve alternative strategies, extra intermediate steps, or necessary workarounds as long as they serve the primary goal effectively.\n- **Constructive Feedback**: If you must reject the plan, provide clear, objective, and actionable feedback explaining the logical flaw or goal deviation, and suggest how the Operator should correct the plan."
+    "validator_guidelines": "## Validation Principles\n- **Goal Alignment**: Ensure the new plan ultimately accomplishes the `Initial Goal` requested by the user. If the Operator has abandoned the core objective or fundamentally altered it, you must reject the plan.\n- **Logical Coherence**: Verify that the proposed milestones form a logical, achievable sequence to reach the goal. Support valid ongoing iteration/polling subgoals and trigger branches without misconstruing periodic checks as infinite loops. Reject the plan if it introduces contradictory steps, unhandled deadlocks, or obvious logical fallacies.\n- **Loop & Monitoring Protection**: If the existing plan contains an active `[Loop]` milestone (especially continuous polling / monitoring with exit condition `until manually stopped`), you MUST REJECT any proposed plan that deletes the `[Loop]` milestone, removes the loop tag, or marks a continuous monitoring milestone as completed (`[x]`) without explicit external user instruction.\n- **Adaptive Acceptance**: The Operator operates in a dynamic environment and may encounter unexpected UI states or dynamic event triggers. You should approve alternative strategies, extra intermediate steps, loop status tracking, or necessary workarounds as long as they serve the primary goal effectively.\n- **Constructive Feedback**: If you must reject the plan, provide clear, objective, and actionable feedback explaining the logical flaw or goal deviation, and suggest how the Operator should correct the plan."
   },
   "modes": {
     "initial_plan": {
       "system": [
         "role",
-        "guidelines",
+        "thinking_process",
         "capabilities_operator",
         "output_protocol_initial",
         "examples_initial"
@@ -27,4 +27,4 @@
       "human": "Initial Goal: {{ initial_goal }}\n\n### Execution History\n{{ history_str }}\n\n### Operator's Thoughts Before Modifying\n{% if operator_native_thinking %}Native Thinking:\n{{ operator_native_thinking }}\n\n{% endif %}{% if operator_raw_thinking %}Raw Thinking:\n{{ operator_raw_thinking }}\n\n{% endif %}### Old Plan\n{{ content_before }}\n\n### Proposed New Plan\n{{ content_after }}\n\nPlease evaluate if the proposed new plan should be approved."
     }
   }
-}
+}

artemis/agents/summarizer/summarizer.md

@@ -13,16 +13,19 @@ You are the Summarizer Agent for a mobile automation system. Your task is to sum
    - **BANNED WORDS**: "successfully", "completed", "entered", "navigated to", "failed", "unsuccessful", "could not", "achieved".
 3. **Focus on Intent & Strategy**: The specific physical action (exact coordinates, swipes) is now recorded automatically. Focus primarily on the high-level Intent and reasoning. If you realize the Operator is on the wrong path and needs to backtrack or pivot, describe this as a shift in strategy.
 4. **Intended Context Transitions**: Describe the intended navigation path ("From Where -> Intended Where"). Do not declare that the transition actually happened. (e.g., "From the Home screen, I intended to open the Network Settings view").
-5. **Preserve Verifications & Missing Items**: If the Operator verified any state in this step, or explicitly mentioned what prerequisites or items are still missing to accomplish the goal, you MUST state these contents in detail in the summary.
+5. **Preserve Verifications & Iteration Progress**: If the Operator verified any state in this step, processed a specific candidate item/polling round, or explicitly mentioned what prerequisites or items are still missing, you MUST succinctly preserve these concrete progress details and verified items in the summary.
 6. **Objective Intervention Logs**: If the Failure Analyzer intervened, faithfully summarize what triggered the intervention and what the analyzer executed, without judging the outcome.
 
 # LOOP & STAGNATION DETECTION
 Use the `RECENT 10 STEPS HISTORY` to intelligently detect loops or stagnation:
-1. **Physical Action Loops**: Be vigilant for loops spanning multiple steps based on physical interactions. If you detect a valid action loop, explicitly mention the pattern and the current iteration count (e.g., "This is the 3rd cycle of attempting to access the video group folder").
-2. **Cognitive Stagnation**: While you generally ignore internal background tool calls (e.g., reading logs, checking states) when counting physical action loops, there is an exception: if you detect continuous multiple steps of purely internal tool calls without any physical actions, you MUST objectively record this strategic "stagnation" or "analysis loop".
+1. **Physical Action Loops vs. Planned Iterations**:
+   - *Planned Iterations / Monitoring (Valid Progress)*: If the Operator is systematically processing different candidate items in a loop, or executing planned periodic polling cycles with wait intervals under a `[Loop]` milestone, record this objectively as planned progress (e.g., "Inspected Candidate #2", "Completed Polling Round #2 for new messages"). Do NOT flag planned iterations across different items or spaced polling as abnormal loops.
+   - *Unintended Action Loops (Anomalies)*: If the Operator is repeatedly clicking the exact same target, failing to progress on the same item, or oscillating blindly between identical states without updating progress notes, explicitly record the anomalous loop pattern and iteration count (e.g., "This is the 3rd cycle of failing to open the same folder").
+2. **Cognitive Stagnation**: While you generally ignore internal background tool calls (e.g., reading logs, checking states) when counting physical action loops, if you detect continuous multiple steps of purely internal tool calls without any physical actions or note updates, objectively record this "stagnation" or "analysis loop".
 
 # INPUT STRUCTURE
 You will receive:
+- **CURRENT PLAN & TASK PLAN**: The current task plan including milestones, active subgoals, and live iteration checklists to help you understand the strategic context and loop phase.
 - **RECENT 10 STEPS HISTORY**: A brief history of the last 10 steps to help you detect loops and stagnation patterns.
 - **CHRONOLOGICAL STEP TRACE (CURRENT STEP)**: The step-by-step chronological history of the current execution turn, including:
   - `[Operator Monologue] / [Operator Native Thought]`: The operator's planning, intentions, progress counting, reasoning, verified states, and explicit mentions of missing prerequisites.

artemis/agents/validator/tool_declarations.py

@@ -17,8 +17,10 @@
 import ast
 import asyncio
 import base64
+import hashlib
 import inspect
 import json
+from pathlib import Path
 import time
 from typing import Any, Literal
 
@@ -35,7 +37,10 @@ from artemis.tools.explorer_tool import _run_explorer_logic
 from artemis.graph.state import State
 from artemis.tools.mobile.launch_app import find_package
 from artemis.utils.app_launch_utils import launch_app_with_retries
-from artemis.utils.coordinates import parse_swipe_parameters
+from artemis.utils.coordinates import (
+    compute_smart_swipe_coordinates,
+    parse_swipe_parameters,
+)
 from artemis.utils.logger import get_logger
 from artemis.utils.ocr_xml_fusion import fuse_ocr_with_xml
 from artemis.utils.visualization import format_minimal_list_with_elements
@@ -282,6 +287,31 @@ def normalize_coordinate_target(target: Any) -> int | list[int] | Any:
     return target
 
 
+def serialize_mobile_action_result(result: Any) -> dict[str, Any]:
+    """Serializes MobileActionExecutor execution results into a clean structured dict.
+
+    Extracts the post-action screenshot hash from the captured image to eliminate guesswork.
+    """
+    if not isinstance(result, tuple) or len(result) < 3:
+        return {"outcome": str(result) if result is not None else ""}
+    outcome, img_bytes, shot_path, *rest = result
+    xml_list = rest[0] if rest else None
+
+    post_image_name = None
+    if shot_path:
+        post_image_name = Path(shot_path).stem
+    elif img_bytes:
+        post_image_name = hashlib.sha256(img_bytes).hexdigest()
+
+    is_err = bool(outcome and ("Error" in outcome or "Failed" in outcome))
+    return {
+        "outcome": outcome,
+        "post_image_name": post_image_name,
+        "has_xml": bool(xml_list),
+        "status": "error" if is_err else "success",
+    }
+
+
 class MobileActionExecutor:
     """Encapsulates device action execution, observation capture, and result reporting."""
 
@@ -289,7 +319,7 @@ class MobileActionExecutor:
         self.ctx = ctx
         self.controller = controller or UnifiedMobileController(ctx)
 
-    @trace(type="tool", name="click")
+    @trace(type="action", name="click", serializer=serialize_mobile_action_result)
     async def exec_click(
         self,
         target: list[int] | Any,
@@ -327,7 +357,7 @@ class MobileActionExecutor:
 
         return outcome, img_bytes, shot_path, xml_list
 
-    @trace(type="tool", name="click_sequence")
+    @trace(type="action", name="click_sequence", serializer=serialize_mobile_action_result)
     async def exec_click_sequence(
         self,
         sequence: list[int | list[int] | str] | str,
@@ -407,7 +437,7 @@ class MobileActionExecutor:
         except Exception as e:
             return f"Error executing click sequence: {e}", None, None, None
 
-    @trace(type="tool", name="long_press")
+    @trace(type="action", name="long_press", serializer=serialize_mobile_action_result)
     async def exec_long_press(
         self,
         target: list[int] | Any,
@@ -449,7 +479,7 @@ class MobileActionExecutor:
 
         return outcome, img_bytes, shot_path, xml_list
 
-    @trace(type="tool", name="input_text")
+    @trace(type="action", name="input_text", serializer=serialize_mobile_action_result)
     async def exec_input_text(
         self,
         text: str,
@@ -484,11 +514,11 @@ class MobileActionExecutor:
 
         return outcome, img_bytes, shot_path, xml_list
 
-    @trace(type="tool", name="swipe")
+    @trace(type="action", name="swipe", serializer=serialize_mobile_action_result)
     async def exec_swipe(
         self,
         action: str | list[int] | Any = None,
-        duration: int = 400,
+        duration: int = 800,
         state: State | None = None,
         duration_ms: int | None = None,
         **kwargs: Any,
@@ -510,33 +540,16 @@ class MobileActionExecutor:
         try:
             if kind == "direction":
                 dir_name = target
-                dir_map = {
-                    "up": (
-                        int(width * 0.6),
-                        int(height * 0.8),
-                        int(width * 0.6),
-                        int(height * 0.2),
-                    ),
-                    "down": (
-                        int(width * 0.6),
-                        int(height * 0.2),
-                        int(width * 0.6),
-                        int(height * 0.8),
-                    ),
-                    "left": (
-                        int(width * 0.8),
-                        int(height * 0.5),
-                        int(width * 0.2),
-                        int(height * 0.5),
-                    ),
-                    "right": (
-                        int(width * 0.2),
-                        int(height * 0.5),
-                        int(width * 0.8),
-                        int(height * 0.5),
-                    ),
-                }
-                x1, y1, x2, y2 = dir_map[dir_name]
+                x1, y1, x2, y2, smart_dur = compute_smart_swipe_coordinates(
+                    direction=target,
+                    target=swipe_input.get("target"),
+                    indexed_elements=getattr(state, "indexed_elements", None) if state else None,
+                    ui_hierarchy=getattr(state, "ui_tree", None) if state else None,
+                    width=width,
+                    height=height,
+                    duration=final_duration,
+                )
+                final_duration = smart_dur
                 err = await self.controller.swipe_coords(x1, y1, x2, y2, final_duration)
                 if err:
                     return f"Error swiping {dir_name}: {err}", None, None, None
@@ -569,7 +582,7 @@ class MobileActionExecutor:
 
         return outcome, img_bytes, shot_path, xml_list
 
-    @trace(type="tool", name="press_key")
+    @trace(type="action", name="press_key", serializer=serialize_mobile_action_result)
     async def exec_press_key(
         self,
         key: str,
@@ -603,7 +616,7 @@ class MobileActionExecutor:
 
         return outcome, img_bytes, shot_path, xml_list
 
-    @trace(type="tool", name="manage_app")
+    @trace(type="action", name="manage_app", serializer=serialize_mobile_action_result)
     async def exec_manage_app(
         self,
         action: str,
@@ -649,7 +662,7 @@ class MobileActionExecutor:
 
         return outcome, img_bytes, shot_path, xml_list
 
-    @trace(type="tool", name="wait_for_delay")
+    @trace(type="action", name="wait_for_delay", serializer=serialize_mobile_action_result)
     async def exec_wait_for_delay(
         self,
         time_in_ms: int,
@@ -668,7 +681,7 @@ class MobileActionExecutor:
 
         return outcome, img_bytes, shot_path, xml_list
 
-    @trace(type="tool", name="wait_for_text")
+    @trace(type="action", name="wait_for_text", serializer=serialize_mobile_action_result)
     async def exec_wait_for_text(
         self,
         text: str,
@@ -979,24 +992,51 @@ INPUT_TEXT_TOOL = ToolDeclaration(
 SWIPE_TOOL = ToolDeclaration(
     name="swipe",
     description=(
-        "[ACTION] Scroll or swipe on the screen. The screen after swipe will be"
-        " returned automatically. Set duration >= 1000 to drag-and-drop."
+        "[ACTION] Perform a swipe, drag, or slider-adjustment gesture on the screen. The screen and UI hierarchy after swipe will be returned automatically.\n\n"
+        "• Directional Scrolling ('direction' or 'action'): Recommended for general browsing and standard page scrolling in most scenarios. Automatically computes safe swipe vectors and adaptive duration, retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings. Supports scoping to a sub-container via 'target'. If it fails on certain custom layouts, fall back to specifying exact coordinates ('start' and 'end') directly.\n"
+        "• Precise Coordinate Gestures ('start', 'end' or coordinates list): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars), drag-and-drop / list reordering, or as a reliable fallback when directional scrolling fails on specific containers. Always drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), swipe fully to the extreme boundary."
     ),
     parameters={
         "type": "object",
         "properties": {
+            "direction": {
+                "type": "string",
+                "enum": ["up", "down", "left", "right"],
+                "description": (
+                    "Direction for scrolling and swiping: 'up' (drags bottom-to-top, scrolling down to reveal content below),"
+                    " 'down' (drags top-to-bottom, scrolling up to reveal content above),"
+                    " 'left' (drags right-to-left, scrolling right),"
+                    " 'right' (drags left-to-right, scrolling left)."
+                ),
+            },
+            "start": {
+                "type": "array",
+                "items": {"type": "integer"},
+                "description": "Start normalized coordinates [start_x, start_y] in 0-1000 scale.",
+            },
+            "end": {
+                "type": "array",
+                "items": {"type": "integer"},
+                "description": "End normalized coordinates [end_x, end_y] in 0-1000 scale.",
+            },
+            "target": {
+                "description": "Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within.",
+            },
             "action": {
                 "description": (
-                    "Swipe action direction string ('up', 'down', 'left', 'right') OR custom"
-                    " coordinates [start_x, start_y, end_x, end_y] in 0-1000 scale."
-                )
+                    "Backward-compatible swipe gesture: smart direction string ('up', 'down', 'left', 'right')"
+                    " OR precise custom coordinates [start_x, start_y, end_x, end_y] in 0-1000 scale."
+                ),
             },
             "duration": {
                 "type": "integer",
-                "description": "Swipe duration in milliseconds (default 400).",
+                "description": (
+                    "Optional swipe/drag duration in milliseconds (default 800). For drag-and-drop,"
+                    " list reordering, or sliding/adjusting sliders (e.g., volume, brightness, SeekBars),"
+                    " set duration >= 1000 (e.g. 1500). If omitted for directional swipe, duration is computed automatically."
+                ),
             },
         },
-        "required": ["action"],
     },
 )
 
@@ -1074,7 +1114,7 @@ MANAGE_APP_TOOL = ToolDeclaration(
 WAIT_FOR_DELAY_TOOL = ToolDeclaration(
     name="wait_for_delay",
     description=(
-        "[ACTION] Pause execution and wait for a fixed duration in milliseconds."
+        "[ACTION] Pause execution and wait for a specified duration in milliseconds."
         " The screen after pause will be returned automatically."
     ),
     parameters={
@@ -1082,7 +1122,11 @@ WAIT_FOR_DELAY_TOOL = ToolDeclaration(
         "properties": {
             "time_in_ms": {
                 "type": "integer",
-                "description": "The exact duration to pause in milliseconds (e.g., 2000).",
+                "description": (
+                    "The exact duration to pause in milliseconds (e.g., 2000 for 2s,"
+                    " 60000 for 1 min, 180000 for 3 mins, 300000 for 5 mins)."
+                    " Convert any required waiting duration into milliseconds."
+                ),
             }
         },
         "required": ["time_in_ms"],

artemis/agents/validator/validator.py

@@ -47,6 +47,7 @@ from artemis.context import ArtemisContext, DeviceContext, DevicePlatform
 from artemis.controllers.unified_controller import UnifiedMobileController
 from artemis.data_engine.trace import CURRENT_TRACE_ID, trace
 from artemis.graph.state import State
+from artemis.platform import platform
 from artemis.services.llm import get_llm
 from artemis.tools.mobile.launch_app import find_package
 from artemis.utils import app_launch_utils, image_diff, visualization
@@ -153,7 +154,7 @@ class ValidatorNode:
                     local_ctx = ArtemisContext(
                         trace_id="local-fallback",
                         device=DeviceContext(
-                            host_platform="LINUX",
+                            host_platform=platform.os_type.name,
                             mobile_platform=DevicePlatform.ANDROID,
                             device_id=device_id,
                             device_width=w,

artemis/agents/video_analyzer/video_analyzer.md

@@ -24,7 +24,7 @@ You must carefully choose the appropriate tool based on the nature of the task:
 
 - **Must use Visual Analysis (`spawn_sub_agent`)**:
   - **UI Interactions**: Identifying clicks, gestures, typing, navigation.
-  - **Visual State**: Reading on-screen text (OCR), checking UI status (toggles, dialogs, loading states).
+  - **Visual State**: Reading on-screen text, checking UI status (toggles, dialogs, loading states).
   - **Visual Content**: Identifying images, layout issues, animations.
 - **Can use Audio-Only Analysis (`analyze_audio_only`)**:
   - **Speech-centric**: Summarizing spoken content, identifying speakers, transcription.

artemis/agents/video_analyzer/video_analyzer.py

@@ -40,7 +40,7 @@ from langchain_core.messages import (
 from artemis.agents.video_analyzer.conflict_resolution import (
     ConflictResolutionService,
 )
-from artemis.config import settings
+from artemis.config import get_temp_dir, settings
 from artemis.constants import SAFETY_SETTINGS_BLOCK_NONE
 from artemis.context import ArtemisContext
 from artemis.controllers.controller_factory import get_controller
@@ -461,10 +461,10 @@ class VideoAnalyzer:
         utils_cfg = getattr(llm_config, "utils", None) if llm_config else None
         llm_cfg = getattr(utils_cfg, "video_analyzer", None) if utils_cfg else None
         model_str = (
-            llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.6-flash"
+            llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.7-flash"
         ).lower()
         self.model_name = (
-            llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.6-flash"
+            llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.7-flash"
         )
         if "/" in self.model_name:
             self.model_name = self.model_name.split("/")[-1]
@@ -1142,7 +1142,7 @@ class VideoAnalyzer:
                                         confidence = float(event.get("confidence_score", 0.0))
                                         relative_offset = max(0.0, verification_ts - actual_start)
 
-                                        out_dir = Path("/tmp/video_analyzer")
+                                        out_dir = get_temp_dir("video_analyzer")
                                         out_dir.mkdir(parents=True, exist_ok=True)
                                         existing_imgs = glob.glob(f"{out_dir}/img_*.jpg")
                                         max_idx = -1
@@ -1692,7 +1692,7 @@ class VideoAnalyzer:
             screenshot_path_str = None
             try:
                 rel_offset = max(0.0, v_ts - actual_start)
-                out_dir = Path("/tmp/video_analyzer")
+                out_dir = get_temp_dir("video_analyzer")
                 out_dir.mkdir(parents=True, exist_ok=True)
 
                 existing_imgs = glob.glob(f"{out_dir}/img_*.jpg")
@@ -1899,7 +1899,7 @@ class VideoAnalyzer:
                 if getattr(llm_cfg, "thinking_level", None) is not None:
                     thinking_level = llm_cfg.thinking_level
             else:
-                self.model_name = "gemini-3.6-flash"
+                self.model_name = "gemini-3.7-flash"
 
             # Track files for cleanup
             self.local_files_to_cleanup = set()