feat: unify history tools and improve execution reliability

artemis/agents/operator/operator.json

@@ -1,3 +1,3 @@
 {
-  "main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n[% if \"wait_for_delay\" in available_tools %]- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n[% endif %]- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n  1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.\n  2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading state. This boundary probe must remain unidirectional; never reverse direction merely to prove completion. If the task declares an earlier exit condition (for example, find one verified match or process N items), that explicit condition may end the traversal without reaching the list boundary.\n  3. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n  4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content.\n- **Dynamic Monitoring & Reactive Loop Protocol (Monitor-Trigger-Action)**: When executing an active milestone matching `[Loop] <Goal> into note <TargetNote> (Exit: <ExitCondition>; Interval: <Interval>)` or `[Loop:continuous] <Goal> into note <TargetNote> (Interval: <Interval>)` (the milestone grammar itself is defined under Task Plan Grammar below), run this 5-step cyclic state machine:\n  - **Step 1 (Baseline Sync)**: On the initial cycle, observe all visible entries/metrics in the primary monitoring view and save the full snaLine truncated
+  "main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n[% if \"wait_for_delay\" in available_tools %]- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n[% endif %]- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n  1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.\n  2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading state. This boundary probe must remain unidirectional; never reverse direction merely to prove completion. If the task declares an earlier exit condition (for example, find one verified match or process N items), that explicit condition may end the traversal without reaching the list boundary.\n  3. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n  4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content.\n- **Dynamic Monitoring & Reactive Loop Protocol (Monitor-Trigger-Action)**: When executing an active milestone matching `[Loop] <Goal> into note <TargetNote> (Exit: <ExitCondition>; Interval: <Interval>)` or `[Loop:continuous] <Goal> into note <TargetNote> (Interval: <Interval>)` (the milestone grammar itself is defined under Task Plan Grammar below), run this 5-step cyclic state machine:\n  - **Step 1 (Baseline Sync)**: On the initial cycle, observe all visible entries/metrics in the primary monitoring view and save the full snaLine truncated
 }

artemis/agents/operator/operator.py

@@ -37,8 +37,11 @@ from artemis.tools.index import get_tool_by_name
 from artemis.tools.tool_wrapper import (
     get_tool_result_content,
     invoke_tool_with_injection,
+    tool_result_messages,
 )
 from artemis.utils.coordinates import (
+    COORDINATE_SPACE_KEY,
+    COORDINATE_SPACE_PIXEL,
     compute_smart_swipe_coordinates,
     parse_swipe_parameters,
 )
@@ -58,7 +61,9 @@ DEFERRING_TOOLS = {
     "video_analyzer",
     "read_note",
     "list_notes",
-    "recall_history",
+    "search_history",
+    "replay_steps",
+    "get_step_screenshot",
     "run_adb_command",
     "manage_task",
     "analyze_task_output",
@@ -639,11 +644,15 @@ class OperatorNode:
                         results.append((f"Error: Tool {tool_name} not supported", "error"))
 
                 for tc, (content, status) in zip(other_calls, results):
-                    tool_outputs.append(
-                        ToolMessage(
-                            tool_call_id=tc["id"],
-                            content=content,
+                    # A step screenshot (content blocks) travels in the carrier
+                    # this model's provider accepts; text stays a ToolMessage.
+                    tool_outputs.extend(
+                        tool_result_messages(
+                            tc["id"],
+                            content,
+                            name=normalize_name(tc["name"]),
                             status=status,
+                            llm=base_llm,
                         )
                     )
 
@@ -732,8 +741,8 @@ class OperatorNode:
                         )
                     action_calls_to_translate = []
                 elif len(action_calls) > burst_cap:
-                    # A multi-action turn is a fast-action burst that the Validator
-                    # fires without the safety net; cap its length before it runs.
+                    # A multi-action turn is a fast-action burst: the Validator vets its first
+                    # member, fires the rest unvetted; cap its length before it runs.
                     validation_errors = True
                     for tc in action_calls:
                         tool_outputs.append(
@@ -1168,6 +1177,7 @@ class OperatorNode:
                 {
                     "action": "tap",
                     "coordinates": el["center"],
+                    COORDINATE_SPACE_KEY: COORDINATE_SPACE_PIXEL,
                     "normalized_coordinates": norm_c,
                     "times": times,
                     "delay_ms": delay_ms,
@@ -1193,6 +1203,7 @@ class OperatorNode:
                 {
                     "action": "long_press_on",
                     "coordinates": el["center"],
+                    COORDINATE_SPACE_KEY: COORDINATE_SPACE_PIXEL,
                     "normalized_coordinates": norm_c,
                     "duration": duration,
                     "target_text": el.get("text"),
@@ -1229,6 +1240,7 @@ class OperatorNode:
                 {
                     "action": "focus_and_input_text",
                     "coordinates": el["center"],
+                    COORDINATE_SPACE_KEY: COORDINATE_SPACE_PIXEL,
                     "normalized_coordinates": norm_c,
                     "text": text,
                     "clear_before_input": clear_exist,
@@ -1310,6 +1322,7 @@ class OperatorNode:
                 {
                     "action": "swipe",
                     "coordinates": [x1, y1, x2, y2],
+                    COORDINATE_SPACE_KEY: COORDINATE_SPACE_PIXEL,
                     "normalized_coordinates": [nx1, ny1, nx2, ny2],
                     "normalized_start_coordinates": [nx1, ny1],
                     "normalized_end_coordinates": [nx2, ny2],

artemis/agents/operator/prompts.py

@@ -20,6 +20,8 @@ from pathlib import Path
 
 from jinja2 import Environment, StrictUndefined, Template
 from langchain_core.messages import HumanMessage, SystemMessage
+
+from artemis.tools.history import HISTORY_TOOL_NAMES
 from PIL import Image
 
 from artemis.context import ArtemisContext
@@ -62,7 +64,13 @@ def load_operator_prompts() -> dict[str, str]:
 
 _PRE_DECISION_HELPER_TOOLS = ("ask_explorer", "ask_diagnoser", "video_analyzer")
 _PRE_DECISION_ADB_TOOLS = ("run_adb_command", "manage_task")
-_PRE_DECISION_MEMORY_TOOLS = ("read_note", "list_notes", "recall_history")
+_PRE_DECISION_MEMORY_TOOLS = (
+    "read_note",
+    "list_notes",
+    "search_history",
+    "replay_steps",
+    "get_step_screenshot",
+)
 _PRE_DECISION_ALL_TOOLS = (
     _PRE_DECISION_HELPER_TOOLS + _PRE_DECISION_ADB_TOOLS + _PRE_DECISION_MEMORY_TOOLS
 )
@@ -188,10 +196,12 @@ def resolve_operator_prompt_tools(ctx: ArtemisContext) -> frozenset[str]:
     if not (setup and getattr(setup, "video_recording_tools_enabled", False)):
         available.discard("video_analyzer")
 
-    # recall_history (M4) needs a DataEngine to search and is config-gated;
-    # the prompt must not advertise it when the tool is not bound this run.
-    if getattr(ctx, "data_engine", None) is None or not _recall_enabled():
-        available.discard("recall_history")
+    # The history tools need a DataEngine to read, and search_history is also
+    # config-gated; the prompt must not advertise a tool that is not bound.
+    if getattr(ctx, "data_engine", None) is None:
+        available.difference_update(HISTORY_TOOL_NAMES)
+    elif not _recall_enabled():
+        available.discard("search_history")
 
     # Device actions the installed actuator backend does not implement disappear
     # from the prompt in lockstep with their tool declarations.
@@ -438,14 +448,13 @@ def _last_successful_action_before(steps: list, step_number: int | None) -> tupl
 
 def _incident_target_label(incident: dict) -> str:
     action = incident.get("action") or {}
+    # The Operator only ever sees 0–1000 coordinates: the recorded pixel
+    # target is the controller's business and is never rendered here.
     normalized = action.get("normalized_coordinates")
-    pixels = action.get("coordinates")
     text = action.get("target_text")
     parts = []
     if normalized:
         parts.append(f"normalized {list(normalized)}")
-    elif pixels:
-        parts.append(f"pixel {list(pixels)}")
     if text:
         parts.append(f'"{text}"')
     return " ".join(parts) if parts else "the recorded target"
@@ -844,7 +853,7 @@ class ScreenshotSimilarityPromptComponent(PromptComponent):
 
 
 def _recall_enabled() -> bool:
-    """Whether the recall_history tool is enabled by configuration (M4)."""
+    """Whether the search_history tool is enabled by configuration."""
     try:
         from artemis.config import load_agent_config
 
@@ -861,7 +870,7 @@ class HistoricalStateHintPromptComponent(PromptComponent):
     including steps already chunk-compressed out of the visible transcript) —
     an O(n) integer scan, no model call, no historical image bytes. On a
     close match to a step *older* than the recent window it injects a one-line
-    hint pointing at ``recall_history``.
+    hint pointing at ``search_history``.
 
     Division of labor with :class:`ScreenshotSimilarityPromptComponent`: that
     component's pixel-exact 3-step look-back covers "stuck on the same
@@ -936,5 +945,5 @@ class HistoricalStateHintPromptComponent(PromptComponent):
             builder.add_human_content(
                 f"Historical state hint: current screen closely resembles the"
                 f" post-action screen from Step {best_step_number}. Use"
-                " recall_history only if its details are needed."
+                " search_history / replay_steps only if its details are needed."
             )