feat: initial release of ARTEMIS Android Agent & Automation Platform

ARTEMIS is an autonomous multimodal Android agent and mobile UI automation
platform engineered for physical devices and emulators. It combines state-of-the-art
multimodal Vision-Language Models (VLMs) with hierarchical closed-loop planning,
resilient self-healing, and deep Android subsystem integration.

Key Features & Architectural Highlights:
- Dual-Engine Orchestration:
  * Pro Engine: Multi-agent closed-loop (Planner -> Operator -> Validator/Failure Analyzer -> Summarizer) powered by LangGraph.
  * Flash Engine: Ultra-fast reactive loop for deterministic, low-latency UI execution (<35 steps).
- Multimodal Perception & Grounding:
  * Hybrid visual screenshot grounding, real-time OCR extraction, and XML accessibility hierarchy parsing.
  * Dynamic-First, Coordinate-Fallback locating strategy for maximum resilience against UI drift.
- Autonomous Self-Healing & Diagnostics:
  * Failure analyzer and safety net for popup dismissal, crash recovery, and auto-retry.
  * Diagnostic sub-agents for log analysis, video recording inspection, and object detection.
- Unified CLI & Python SDK:
  * Rich CLI with commands: init, doctor, run, trace, batch, bench, server, and mcp.
  * Fluent Python SDK with Pydantic structured output models and package locking.
- Multi-Provider LLM Architecture:
  * Support for Google Gemini, OpenAI, Anthropic Claude, Vertex AI, OpenRouter, xAI Grok, and local endpoints (Ollama/vLLM).
  * Centralized configuration in config/artemis.jsonc.
- Native MCP & Observability:
  * Model Context Protocol (MCP) server endpoints for AI IDEs.
  * Full trace recording with before/after screenshots, action overlays, and execution replays.

Change-Id: I430b93ee2b76b54cc5a3c6eeccd15d2cf29830a1

artemis/agents/__init__.py

@@ -0,0 +1,37 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Modular Agent Framework for ARTEMIS."""
+
+from artemis.agents.base import (
+    AgentConfig,
+    AgentRegistry,
+    AgentResponse,
+    BaseAgent,
+)
+from artemis.agents.operator.operator_agent import OperatorAgent
+from artemis.agents.planner.planner_agent import PlannerAgent
+from artemis.agents.summarizer.summarizer_agent import SummarizerAgent
+from artemis.agents.validator.failure_analyzer_agent import FailureAnalyzerAgent
+
+__all__ = [
+    "BaseAgent",
+    "AgentConfig",
+    "AgentResponse",
+    "AgentRegistry",
+    "PlannerAgent",
+    "OperatorAgent",
+    "SummarizerAgent",
+    "FailureAnalyzerAgent",
+]

artemis/agents/base.py

@@ -0,0 +1,146 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Base Agent Protocol & Lifecycle Framework for ARTEMIS.
+
+Provides a unified interface and lifecycle management for all specialized
+multimodal agents (Planner, Operator, FailureAnalyzer, Explorer, Summarizer, etc.).
+"""
+
+from abc import ABC, abstractmethod
+from typing import TYPE_CHECKING, Any
+
+from pydantic import BaseModel, Field
+
+from artemis.drivers.factory import get_driver
+from artemis.utils.logger import get_logger
+
+if TYPE_CHECKING:
+    pass
+
+logger = get_logger(__name__)
+
+
+class AgentConfig(BaseModel):
+    """Configuration options for an individual agent."""
+
+    model_name: str | None = Field(default=None, description="LLM/VLM model override")
+    temperature: float = Field(default=0.0, description="Sampling temperature")
+    max_tokens: int = Field(default=4096, description="Maximum output tokens")
+    system_prompt_override: str | None = Field(
+        default=None, description="Custom system prompt override"
+    )
+    enabled_tools: list[str] = Field(
+        default_factory=list, description="Tool names enabled for this agent"
+    )
+
+
+class AgentResponse(BaseModel):
+    """Standardized structured response from an agent invocation."""
+
+    status: str = Field(
+        default="success", description="Execution status ('success', 'retry', 'fail', 'done')"
+    )
+    thought: str | None = Field(default=None, description="Reasoning and internal monologue")
+    action_taken: str | None = Field(default=None, description="Action or tool called")
+    output: Any = Field(default=None, description="Output payload or state updates")
+    metadata: dict[str, Any] = Field(
+        default_factory=dict, description="Diagnostic and profiling metadata"
+    )
+
+
+class BaseAgent(ABC):
+    """Abstract Base Class defining the standard lifecycle for all ARTEMIS agents."""
+
+    def __init__(
+        self,
+        name: str,
+        role: str,
+        description: str,
+        config: AgentConfig | None = None,
+        ctx: Any | None = None,
+    ):
+        self.name = name
+        self.role = role
+        self.description = description
+        self.config = config or AgentConfig()
+        self.ctx: Any | None = ctx
+        self._is_initialized = False
+
+    @property
+    def driver(self) -> Any | None:
+        """Helper to get the active device driver from the context."""
+        if self.ctx:
+            return get_driver(self.ctx)
+        return None
+
+    async def initialize(self, ctx: Any) -> None:
+        """Lifecycle hook: Called once when the agent is loaded into the workflow."""
+        self.ctx = ctx
+        self._is_initialized = True
+        logger.info(f"Agent '{self.name}' ({self.role}) initialized.")
+
+    @abstractmethod
+    async def process(self, state: Any) -> dict[str, Any]:
+        """Main execution hook: Receives state, reasons, interacts with tools, and returns state updates."""
+        ...
+
+    async def cleanup(self) -> None:
+        """Lifecycle hook: Called upon session or workflow termination."""
+        self._is_initialized = False
+        logger.debug(f"Agent '{self.name}' cleanup complete.")
+
+    def __repr__(self) -> str:
+        return f"<{self.__class__.__name__} name={self.name!r} role={self.role!r}>"
+
+
+class AgentRegistry:
+    """Central registry and factory for ARTEMIS agents."""
+
+    _registry: dict[str, type[BaseAgent]] = {}
+
+    @classmethod
+    def register(cls, name: str | None = None):
+        """Decorator to register an agent class with a unique identifier."""
+
+        def decorator(subclass: type[BaseAgent]) -> type[BaseAgent]:
+            reg_name = name or subclass.__name__.lower()
+            cls._registry[reg_name] = subclass
+            return subclass
+
+        return decorator
+
+    @classmethod
+    def get_agent_class(cls, name: str) -> type[BaseAgent] | None:
+        return cls._registry.get(name.lower())
+
+    @classmethod
+    def create_agent(
+        cls,
+        name: str,
+        ctx: Any | None = None,
+        config: AgentConfig | None = None,
+        **kwargs: Any,
+    ) -> BaseAgent:
+        """Instantiates an agent by registered name."""
+        agent_cls = cls.get_agent_class(name)
+        if not agent_cls:
+            raise KeyError(
+                f"Agent '{name}' is not registered. Available: {list(cls._registry.keys())}"
+            )
+        return agent_cls(ctx=ctx, config=config, **kwargs)
+
+    @classmethod
+    def list_agents(cls) -> list[str]:
+        return list(cls._registry.keys())

artemis/agents/checker/checker.json

@@ -0,0 +1,4 @@
+{
+  "checker_system": "You are a precise and reliable verification expert (Checker). Your task is to determine whether the Operator has successfully completed the active subgoal by combining visual evidence from the current screenshot with data stored in persistent notes.",
+  "checker_prompt": "# Inputs\n- Active Subgoal: {{ subgoal }}\n- Execution History:\n{{ plan_and_history }}\n- Dialogue History: {{ dialogue_history }}\n- Visual State: The provided screenshot.\n\n# Guidelines\n- **Visual + Data Verification**: You must combine the screenshot with note contents. If the subgoal involves extracting, saving, or recording information (e.g., \"save contacts\", \"record address\"), you **MUST** verify that the content actually exists in the note. Do not assume success just because the information is visible on the screen. If the signs of success are not obvious, you can check the history, especially the most recent steps, to see if the operator performed their own check and whether that check was reasonable, to determine if the subgoal was truly successful.\n- **Visual Priority (Anti-History-Bias)**: Your primary ground truth is the visual evidence on the **Current Screenshot**. The `Execution History` lists previous successes or failures, which may be outdated due to dynamic background recovery or self-healing. If the screenshot demonstrably shows the active subgoal's target state is achieved, you **MUST** mark `success: true`, regardless of previous failed steps. Trust what you see, not past FAILED texts.\n- **Constraint Verification**: For subgoals containing specific parameters (such as names, brands, topics, events, numbers, or dates), confirm that the visible text precisely reflects these specified details. A successful state is defined by a clear, direct match with the subgoal's constraints.\n- **Realistic Expectations**: You cannot scroll the screen. If a subgoal requires saving all items but only some are visible on the screen, verify that at least the **visible items** are recorded in the note. As long as the visible part is recorded and there are no obvious errors, consider it a success. Do not fail the task simply because off-screen content cannot be verified.\n- **Success Criteria**: Set `success` to `true` if the core objective of the subgoal is met and it does not block subsequent steps. Set `success` to `false` if the state is incorrect, required data was not saved, or the operator is stuck/looping.\n\n# Output Protocol\nInvoke the `submit_verification` tool with `success` (boolean) and `reason` (string) to submit your final decision. If you need to inspect note data first, use `read_note`."
+}

artemis/agents/checker/checker.py

@@ -0,0 +1,460 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import asyncio
+import hashlib
+import json
+from pathlib import Path
+import re
+from typing import Any
+
+from jinja2 import Template
+from langchain_core.messages import HumanMessage, SystemMessage, ToolMessage
+from langchain_core.tools import tool
+
+from pydantic import BaseModel, Field
+
+from artemis.constants import CHECKER_MAX_ITERATIONS
+from artemis.context import ArtemisContext
+from artemis.controllers.unified_controller import UnifiedMobileController
+from artemis.data_engine.trace import (
+    TraceSpan,
+    trace,
+    trace_langchain_tool,
+)
+from artemis.services.llm import get_llm, invoke_llm_with_timeout_message
+from artemis.tools.index import get_tool_by_name
+from artemis.tools.scratchpad import get_read_note_tool_pure
+from artemis.tools.tool_wrapper import (
+    get_tool_result_content,
+    invoke_tool_with_injection,
+)
+from artemis.utils.logger import get_logger
+from artemis.utils.notes import get_note_file_path, get_notes_dir
+from artemis.utils.ocr_api import perform_ocr
+from artemis.utils.ocr_xml_fusion import (
+    _crop_image_remove_status_bar,
+    _detect_status_bar_height,
+    _map_coordinates_back,
+    fuse_ocr_with_xml,
+)
+from artemis.utils.task_tree import (
+    build_plan_and_history,
+    get_recent_subgoal_hashes,
+)
+from artemis.utils.verification import (
+    append_verification_chat,
+    get_verification_chat_path,
+    get_verification_chat_rounds,
+    read_verification_chat,
+)
+from artemis.utils.text import safe_extract_text
+from artemis.utils.visualization import format_minimal_list_with_elements
+
+logger = get_logger(__name__)
+
+
+class CheckerResult(BaseModel):
+    """Structured output contract for Checker verification."""
+
+    success: bool = Field(
+        description="Whether the active subgoal was successfully completed based on visual and note evidence."
+    )
+    reason: str = Field(
+        description="Detailed explanation of the result. If failed, specify what is missing or wrong and give actionable hints for the Operator."
+    )
+
+
+@trace(type="agent", name="checker")
+async def run_async_check(
+    ctx: ArtemisContext,
+    subgoal_text: str,
+    subgoal_hash: str | None = None,
+    raw_perception_data: dict[str, Any] | None = None,
+    latest_ui_hierarchy: list[Any] | None = None,
+):
+    """Runs the checker verification as a background task."""
+    logger.info(f"Starting async Checker for: {subgoal_text}")
+
+    if not ctx.data_engine:
+        logger.warning("DataEngine not available, skipping checker.")
+        return {"status": "success", "reason": "DataEngine unavailable."}
+
+    notes_dir = get_notes_dir(ctx.data_engine.base_dir)
+    status_path = notes_dir / "checker_status.json"
+
+    if ctx.execution_setup and ctx.execution_setup.disable_checker:
+        logger.info("Checker is disabled via flag. Skipping.")
+        try:
+            status_path.write_text(
+                json.dumps({"status": "success", "reason": "Checker disabled."}),
+                encoding="utf-8",
+            )
+        except Exception:
+            pass
+        return {"status": "success", "reason": "Checker disabled."}
+
+    # 1. Set status to running
+    try:
+        status_path.write_text(
+            json.dumps({"status": "running", "subgoal": subgoal_text}),
+            encoding="utf-8",
+        )
+        logger.info(f"Wrote checker status 'running' to {status_path}")
+    except Exception as e:
+        logger.error(f"Failed to write checker status: {e}")
+        return {"status": "success", "reason": f"Status write error: {e}"}
+
+    # 2. Run Checker logic
+    try:
+        # Retrieve perception data: directly reuse upstream perception if available
+        if isinstance(raw_perception_data, dict) and raw_perception_data.get("screenshot_b64"):
+            screenshot_b64 = raw_perception_data["screenshot_b64"]
+            width = raw_perception_data.get("width", 1080)
+            height = raw_perception_data.get("height", 2400)
+            fused_xml = latest_ui_hierarchy or raw_perception_data.get("xml_hierarchy") or []
+        elif (
+            hasattr(ctx, "latest_perception_data")
+            and isinstance(ctx.latest_perception_data, dict)
+            and ctx.latest_perception_data
+        ):
+            screenshot_b64 = ctx.latest_perception_data.get("screenshot_b64")
+            width = ctx.latest_perception_data.get("width", 1080)
+            height = ctx.latest_perception_data.get("height", 2400)
+            fused_xml = ctx.latest_perception_data.get("latest_ui_hierarchy") or []
+        else:
+            # Fallback to direct controller query
+            controller = UnifiedMobileController(ctx)
+            device_data = await controller.get_screen_data()
+            screenshot_b64 = device_data.base64
+            xml_hierarchy = device_data.elements
+            width = device_data.width
+            height = device_data.height
+
+            ocr_results = []
+            try:
+                status_bar_height = _detect_status_bar_height(xml_hierarchy, height)
+                if status_bar_height > 0:
+                    cropped_b64, _, _ = _crop_image_remove_status_bar(
+                        screenshot_b64, status_bar_height
+                    )
+                    raw_ocr_results = await perform_ocr(cropped_b64)
+                    ocr_results = _map_coordinates_back(raw_ocr_results, status_bar_height)
+                else:
+                    ocr_results = await perform_ocr(screenshot_b64)
+            except Exception as e:
+                logger.error(f"Checker OCR failed, proceeding without OCR: {e}")
+
+            fused_xml = fuse_ocr_with_xml(xml_hierarchy, ocr_results)
+
+        minimal_list, elements, labels = format_minimal_list_with_elements(fused_xml, width, height)
+
+        # Build task tree
+        task_plan = "No task plan yet."
+        task_plan_path = get_note_file_path(ctx.data_engine.base_dir, "task_plan")
+        if task_plan_path.exists():
+            try:
+                task_plan = task_plan_path.read_text(encoding="utf-8")
+            except Exception as e:
+                logger.error(f"Failed to read task plan for Checker: {e}")
+
+        steps = ctx.data_engine.get_agent_friendly_steps() if ctx.data_engine else []
+
+        # Calculate exact subgoal hash from subgoal_text directly to avoid 'default' placeholder
+        if not subgoal_hash or subgoal_hash == "default":
+            clean_subgoal = safe_extract_text(subgoal_text).strip()
+            subgoal_hash = hashlib.md5(clean_subgoal.encode("utf-8")).hexdigest()
+
+        keep_hashes = get_recent_subgoal_hashes(steps, subgoal_hash, ctx.data_engine.base_dir)
+
+        plan_and_history = build_plan_and_history(
+            task_plan, steps, subgoal_hash, keep_subgoal_hashes=keep_hashes
+        )
+
+        chat_path = get_verification_chat_path(ctx.data_engine.base_dir, subgoal_hash)
+        turns = read_verification_chat(chat_path)
+
+        # Calculate round number
+        max_op, max_chk = get_verification_chat_rounds(turns)
+        round_num = max(max_op, max_chk + 1)
+        if round_num == 0:
+            round_num = 1
+
+        # Render dialogue history for prompt
+        dialogue_lines = []
+        for t in turns:
+            role = "Operator" if t["role"] == "operator" else "Checker"
+            dialogue_lines.append(f"**{role} (Round {t['round']})**:\n{t['content']}")
+        dialogue_history = (
+            "\n\n".join(dialogue_lines) if dialogue_lines else "No previous dialogue."
+        )
+
+        prompts = {}
+        prompts_path = Path(__file__).parent / "checker.json"
+        if prompts_path.exists():
+            try:
+                prompts = json.loads(prompts_path.read_text(encoding="utf-8"))
+            except Exception as e:
+                logger.error(f"Failed to load checker prompts: {e}")
+
+        prompt_template = prompts.get(
+            "checker_prompt",
+            "You are the Checker. Your job is to verify if the current subtask"
+            " was successfully completed.",
+        )
+
+        full_prompt = Template(prompt_template).render(
+            subgoal=subgoal_text,
+            plan_and_history=plan_and_history,
+            dialogue_history=dialogue_history if dialogue_history else "No previous dialogue.",
+            minimal_list=minimal_list,
+        )
+
+        llm = get_llm(ctx=ctx, name="checker")
+        read_note = get_read_note_tool_pure(ctx)
+
+        @tool
+        def submit_verification(
+            success: bool,
+            reason: str,
+        ) -> str:
+            """[TERMINAL] Submit the final verification outcome (success or failure with reason). This completes verification."""
+            return f"Verification decision: success={success}, reason={reason}"
+
+        tools_to_bind = [
+            trace_langchain_tool(read_note, ctx),
+            submit_verification,
+        ]
+
+        content = [
+            {"type": "text", "text": full_prompt},
+            {"type": "text", "text": "--- Current Screenshot ---"},
+            {
+                "type": "image_url",
+                "image_url": {"url": f"data:image/jpeg;base64,{screenshot_b64}"},
+            },
+        ]
+
+        messages = [
+            SystemMessage(
+                content=prompts.get(
+                    "checker_system",
+                    "You are the Checker agent. Your objective is to accurately"
+                    " assess whether the active subgoal was successfully"
+                    " completed.",
+                )
+            ),
+            HumanMessage(content=content),
+        ]
+
+        max_iterations = (
+            getattr(
+                ctx.execution_setup,
+                "checker_max_iterations",
+                CHECKER_MAX_ITERATIONS,
+            )
+            if ctx.execution_setup
+            else CHECKER_MAX_ITERATIONS
+        )
+
+        success = True
+        reason = "Checker completed."
+
+        for i in range(max_iterations):
+            is_final_round = i == max_iterations - 1
+
+            if is_final_round:
+                messages.append(
+                    HumanMessage(
+                        content=(
+                            "This is your final iteration; all tools are"
+                            " stripped, and you must provide your final structured verification result."
+                        )
+                    )
+                )
+                structured_llm = llm.with_structured_output(CheckerResult)
+                structured_res: CheckerResult = await invoke_llm_with_timeout_message(
+                    structured_llm.ainvoke(messages)
+                )
+                if isinstance(structured_res, CheckerResult):
+                    success = structured_res.success
+                    reason = structured_res.reason
+                elif isinstance(structured_res, dict):
+                    success = structured_res.get("success", True)
+                    reason = structured_res.get("reason", "Verification completed.")
+                break
+            else:
+                if i > 0:
+                    messages.append(
+                        HumanMessage(
+                            content=(
+                                "You have not completed verification yet"
+                                f" (iteration {i + 1} of {max_iterations})."
+                            )
+                        )
+                    )
+                active_llm = llm.bind_tools(tools=tools_to_bind)
+
+                async def run_stream():
+                    full_response = None
+                    async for chunk in active_llm.astream(messages):
+                        if full_response is None:
+                            full_response = chunk
+                        else:
+                            full_response += chunk
+                    return full_response
+
+                response = await invoke_llm_with_timeout_message(run_stream())
+
+                # Check if submit_verification was invoked
+                submit_tc = next(
+                    (
+                        tc
+                        for tc in response.tool_calls
+                        if tc["name"].endswith("submit_verification")
+                    ),
+                    None,
+                )
+                if submit_tc:
+                    tc_args = dict(submit_tc["args"])
+                    success = bool(tc_args.get("success", True))
+                    reason = str(tc_args.get("reason", "Verification submitted."))
+                    break
+
+                if not response.tool_calls:
+                    # Model answered in plain text; use structured_llm directly
+                    structured_llm = llm.with_structured_output(CheckerResult)
+                    structured_res: CheckerResult = await invoke_llm_with_timeout_message(
+                        structured_llm.ainvoke(messages + [response])
+                    )
+                    if isinstance(structured_res, CheckerResult):
+                        success = structured_res.success
+                        reason = structured_res.reason
+                    elif isinstance(structured_res, dict):
+                        success = structured_res.get("success", True)
+                        reason = structured_res.get("reason", "Verification completed.")
+                    break
+
+                messages.append(response)
+
+                async def run_tool(tc):
+                    tool_name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
+                    logger.info(f"Checker requested tool: {tool_name}")
+
+                    result_str = ""
+                    tool_status = "success"
+                    try:
+                        if ":" in tool_name:
+                            tool_to_run = get_tool_by_name(tool_name, traced_tools)
+                        else:
+                            tool_to_run = next(
+                                (t for t in traced_tools if t.name == tool_name),
+                                None,
+                            )
+                        if tool_to_run:
+                            args = dict(tc["args"])
+                            with TraceSpan(name=tool_name, ctx=ctx) as span:
+                                span.payload = {"args": args}
+                                try:
+                                    result_obj = await invoke_tool_with_injection(
+                                        tool=tool_to_run,
+                                        args=args,
+                                        tool_call_id=tc["id"],
+                                    )
+                                    result_str = get_tool_result_content(result_obj)
+                                    span.result = result_str
+                                    if result_str.startswith("Error"):
+                                        tool_status = "error"
+                                        span.status = "failed"
+                                        span.error = result_str
+                                except Exception as err:
+                                    tool_status = "error"
+                                    span.status = "failed"
+                                    span.error = str(err)
+                                    raise err
+                        else:
+                            result_str = f"Error: Tool {tool_name} not supported"
+                            tool_status = "error"
+                    except Exception as err:
+                        logger.error(f"Error running tool {tool_name}: {err}")
+                        result_str = f"Error running tool {tool_name}: {err}"
+                        tool_status = "error"
+
+                    return ToolMessage(
+                        tool_call_id=tc["id"],
+                        content=result_str,
+                        status=tool_status,
+                    )
+
+                tool_outputs = await asyncio.gather(*(run_tool(tc) for tc in response.tool_calls))
+                for tm in tool_outputs:
+                    messages.append(tm)
+
+        logger.info(
+            f"Async Checker result for '{subgoal_text}': Success={success}, Reason={reason}"
+        )
+
+        # Update status file
+        status_path.write_text(
+            json.dumps({"status": "success" if success else "failed", "reason": reason}),
+            encoding="utf-8",
+        )
+
+        # Clean up dialogue file if successful
+        if success:
+            if chat_path.exists():
+                logger.info(f"Checker succeeded. Cleaning up dialogue file: {chat_path}")
+                chat_path.unlink()
+
+        # If failed, revert status in task_plan.md and write feedback
+        if not success:
+            task_plan_path = get_note_file_path(ctx.data_engine.base_dir, "task_plan")
+            if task_plan_path.exists():
+                task_plan = task_plan_path.read_text(encoding="utf-8")
+                lines = task_plan.split("\n")
+
+                for i in range(len(lines) - 1, -1, -1):
+                    if lines[i].startswith("- ["):
+                        task_content = safe_extract_text(lines[i][5:]).strip()
+                        safe_subgoal = safe_extract_text(subgoal_text).strip()
+                        if task_content == safe_subgoal:
+                            lines[i] = re.sub(r"^(- \[)[x/ ](\])", r"\g<1>/\g<2>", lines[i])
+                            logger.info(
+                                "Matched top-level subgoal by content."
+                                f" Reverted status to [/]: {subgoal_text}"
+                            )
+                            break
+                task_plan_path.write_text("\n".join(lines), encoding="utf-8")
+
+            append_verification_chat(
+                chat_path,
+                "checker",
+                f"Status: Failed.\nReason: {reason}",
+                round_num,
+            )
+
+        return {"status": "success" if success else "failed", "reason": reason}
+
+    except Exception as e:
+        logger.error(f"Async Checker encountered exception (fail-open): {e}")
+        status_path.write_text(
+            json.dumps(
+                {
+                    "status": "success",
+                    "reason": f"Checker Fail-Open: {e}",
+                }
+            ),
+            encoding="utf-8",
+        )
+        return {"status": "success", "reason": f"Checker Fail-Open: {e}"}

artemis/agents/diagnoser/diagnoser.json

@@ -0,0 +1,22 @@
+{
+  "system_prompt": [
+    "# Diagnoser",
+    "",
+    "## Objective",
+    "You are an expert AI diagnoser. Your objective is to pinpoint the root cause of mobile execution failures by cross-referencing visual evidence (videos/screenshots) with system telemetry (logs). Precisely answer the `Diagnostic Query` based on the provided context.",
+    "",
+    "## Core Guidelines",
+    "- **Evidence-Based**: Base all conclusions strictly on facts from tools and context. Do not assume or hallucinate.",
+    "- **Laser Focus**: Focus exclusively on the `Diagnostic Query`. Do not investigate unrelated issues.",
+    "- **No Direct Operations**: Do not perform user actions or modify device state (e.g., tap, swipe, text input, BACK/HOME keys, or force-starting activities). Focus solely on analysis.",
+    "- **Tool Constraints**: Video analysis is expensive. When calling the video analyzer, you MUST provide a specific, narrow timeframe to minimize cost.",
+    "- **Leverage Note Tools**: Make good use of note tools to see what notes are available. When diagnosing, open and cross-reference relevant notes to ensure you have grasped all necessary information and context.",
+    "- **Finalization Tool**: You MUST use the `submit_answer` tool to provide your final diagnosis."
+  ],
+  "checker_feedback_template": [
+    "--- Checker Feedback ---",
+    "The Checker agent has provided feedback regarding the verification of the active subgoal:",
+    "{checker_feedback}",
+    ""
+  ]
+}

artemis/agents/diagnoser/diagnoser.py

@@ -0,0 +1,575 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import asyncio
+import base64
+import json
+import os
+from pathlib import Path
+from typing import Any
+import uuid
+
+from langchain_core.messages import HumanMessage, SystemMessage, ToolMessage
+
+from artemis.context import ArtemisContext
+from artemis.data_engine.trace import (
+    CURRENT_TRACE_ID,
+    TraceSpan,
+    trace,
+    trace_langchain_tool,
+)
+from artemis.graph.state import State
+from artemis.services.llm import get_llm, invoke_llm_with_timeout_message
+from artemis.tools.command_tool import get_run_short_adb_command_tool
+from artemis.tools.diagnoser_submit_answer_tool import get_submit_answer_tool
+from artemis.tools.index import get_tool_by_name
+from artemis.tools.log_tool import get_analyze_logs_tool
+from artemis.tools.mobile.read_hierarchy import get_ui_hierarchy_tool
+from artemis.tools.scratchpad import get_list_notes_tool, get_read_note_tool
+from artemis.tools.tool_wrapper import (
+    get_tool_result_content,
+    invoke_tool_with_injection,
+)
+from artemis.tools.video_tool import get_video_analyzer_tool
+from artemis.tools.wait_tool import get_wait_tool
+from artemis.utils.logger import get_logger
+from artemis.utils.task_tree import (
+    build_plan_and_history,
+    get_active_subgoal_hashes,
+    get_recent_subgoal_hashes,
+)
+
+logger = get_logger(__name__)
+
+
+class Diagnoser:
+    def __init__(self, ctx: ArtemisContext):
+        self.ctx = ctx
+        self.is_device_online = self._is_device_online()
+
+    def _is_device_online(self) -> bool:
+        try:
+            if self.ctx.adb_client is None:
+                return False
+            if getattr(self.ctx.adb_client, "is_online", True) is False:
+                return False
+            self.ctx.get_adb_client()
+            return True
+        except Exception:
+            return False
+
+    @trace(type="agent", name="diagnoser")
+    async def run(self, prompt: str, state: State) -> str:
+        """Runs the Diagnoser and returns its summary."""
+
+        # 1. Load prompt
+        prompt_path = Path(__file__).parent.joinpath("diagnoser.json")
+        if not prompt_path.exists():
+            raise FileNotFoundError(f"Prompt file not found at {prompt_path}")
+
+        with open(prompt_path, encoding="utf-8") as f:
+            prompt_data = json.load(f)
+
+        system_prompt = prompt_data.get("system_prompt", "")
+        if isinstance(system_prompt, list):
+            system_prompt = "\n".join(system_prompt)
+        checker_feedback_template = prompt_data.get("checker_feedback_template", "")
+        if isinstance(checker_feedback_template, list):
+            checker_feedback_template = "\n".join(checker_feedback_template)
+
+        # 2. Prepare LLM
+        llm = get_llm(ctx=self.ctx, name="diagnoser")
+
+        log_tool = get_analyze_logs_tool(self.ctx)
+
+        read_note_tool = get_read_note_tool(self.ctx)
+        list_notes_tool = get_list_notes_tool(self.ctx)
+        adb_command_tool = get_run_short_adb_command_tool(self.ctx)
+        wait_tool = get_wait_tool(self.ctx)
+        ui_hierarchy_tool = get_ui_hierarchy_tool(self.ctx)
+        submit_answer_tool = get_submit_answer_tool(self.ctx)
+
+        all_tools = [
+            log_tool,
+            read_note_tool,
+            list_notes_tool,
+            adb_command_tool,
+            wait_tool,
+            ui_hierarchy_tool,
+            submit_answer_tool,
+        ]
+        if not self.is_device_online:
+            logger.info(
+                "Diagnoser running in OFFLINE environment: stripping adb short"
+                " command and wait tools."
+            )
+            all_tools = [
+                t
+                for t in all_tools
+                if t.name not in ("run_adb_command", "wait")
+                and not t.name.endswith(":run_adb_command")
+                and not t.name.endswith(":wait")
+            ]
+
+        if self.ctx.execution_setup and self.ctx.execution_setup.video_recording_tools_enabled:
+            video_tool = get_video_analyzer_tool(self.ctx, role="diagnoser")
+            all_tools.append(video_tool)
+
+        traced_tools = [trace_langchain_tool(t, self.ctx) for t in all_tools]
+
+        # Bind custom tools
+        base_llm = llm
+
+        # 4. Get context from DataEngine and State
+        initial_goal = state.initial_goal
+
+        task_plan = "No task plan yet."
+        steps = []
+        subgoal_hash = "default"
+
+        if self.ctx.data_engine:
+            notes_dir = Path(self.ctx.data_engine.base_dir) / "notes"
+            task_plan_path = notes_dir / "task_plan.md"
+
+            if task_plan_path.exists():
+                try:
+                    task_plan = task_plan_path.read_text(encoding="utf-8")
+
+                    subgoal_hash, _ = get_active_subgoal_hashes(task_plan)
+                except Exception as e:
+                    logger.error(f"Failed to parse active subgoal in Diagnoser: {e}")
+                    task_plan = f"Error reading task plan: {e}"
+
+            steps = self.ctx.data_engine.get_agent_friendly_steps()
+
+        # Build task tree
+        plan_and_history = "No plan or history available."
+        if self.ctx.data_engine and task_plan != "No task plan yet.":
+            try:
+                keep_hashes = get_recent_subgoal_hashes(
+                    steps,
+                    subgoal_hash,
+                    window_steps=self.ctx.agent_config.history_window_steps,
+                )
+
+                plan_and_history = build_plan_and_history(
+                    task_plan,
+                    steps,
+                    subgoal_hash,
+                    keep_subgoal_hashes=keep_hashes,
+                )
+            except Exception as e:
+                logger.error(f"Failed to build plan and history in Diagnoser: {e}")
+                plan_and_history = f"Error building plan and history: {e}"
+
+        # Get Checker Feedback
+        checker_feedback = ""
+        if self.ctx.data_engine:
+            notes_dir = Path(self.ctx.data_engine.base_dir) / "notes"
+            verification_chat_path = notes_dir / f"verification_chat_{subgoal_hash}.json"
+            if verification_chat_path.exists():
+                try:
+                    turns = json.loads(verification_chat_path.read_text(encoding="utf-8"))
+                    dialogue_lines = []
+                    for t in turns:
+                        role = "Operator" if t["role"] == "operator" else "Checker"
+                        dialogue_lines.append(f"**{role} (Round {t['round']})**:\n{t['content']}")
+                    checker_feedback = "\n\n".join(dialogue_lines)
+                except Exception as e:
+                    logger.error(f"Error reading verification chat in Diagnoser: {e}")
+
+        # Get Current Screenshot and UI Hierarchy
+        latest_screenshot_b64 = None
+        minimal_list = ""
+
+        if state.latest_screenshot and os.path.exists(state.latest_screenshot):
+            try:
+                with open(state.latest_screenshot, "rb") as f:
+                    latest_screenshot_b64 = base64.b64encode(f.read()).decode("utf-8")
+            except Exception as e:
+                logger.error(f"Failed to read latest screenshot in Diagnoser: {e}")
+
+        if state.latest_ui_hierarchy:
+            minimal_list = self._format_minimal_list(state.latest_ui_hierarchy)
+
+        # Construct rich human message
+        content = []
+        content.append({"type": "text", "text": f"Initial Goal: {initial_goal}\n"})
+        content.append(
+            {
+                "type": "text",
+                "text": f"--- Plan & History ---\n{plan_and_history}\n",
+            }
+        )
+
+        if checker_feedback and checker_feedback_template:
+            formatted_feedback = checker_feedback_template.format(checker_feedback=checker_feedback)
+            content.append({"type": "text", "text": formatted_feedback})
+
+        if minimal_list:
+            content.append(
+                {
+                    "type": "text",
+                    "text": f"--- Visible UI Elements ---\n{minimal_list}\n",
+                }
+            )
+
+        if latest_screenshot_b64:
+            content.append({"type": "text", "text": "--- Current Screenshot ---"})
+            content.append(
+                {
+                    "type": "image_url",
+                    "image_url": {"url": f"data:image/jpeg;base64,{latest_screenshot_b64}"},
+                }
+            )
+
+        content.append({"type": "text", "text": f"--- Diagnostic Query ---\n{prompt}"})
+
+        current_messages = [
+            SystemMessage(content=system_prompt),
+            HumanMessage(content=content),
+        ]
+
+        self.ctx.background_jobs = {}
+
+        max_iterations = 30
+        outcome = "No result"
+
+        try:
+            for i in range(max_iterations):
+                # 5. Agent Loop Injection & Final Turn Guard
+                if i == max_iterations - 1:
+                    # Identify all jobs in the registry with status "running"
+                    running_jobs = [
+                        job
+                        for job in self.ctx.background_jobs.values()
+                        if job.get("status") == "running"
+                    ]
+                    if running_jobs:
+                        # Await all associated task objects
+                        await asyncio.gather(
+                            *(job["task"] for job in running_jobs if job.get("task") is not None),
+                            return_exceptions=True,
+                        )
+
+                # Iterate through the background_jobs registry for all jobs where status is "completed" or "failed" and consumed is false
+                new_tool_messages = []
+                for job in self.ctx.background_jobs.values():
+                    if job.get("status") in (
+                        "completed",
+                        "failed",
+                    ) and not job.get("consumed", False):
+                        tm_status = "success" if job["status"] == "completed" else "error"
+                        tm = ToolMessage(
+                            tool_call_id=job["tool_call_id"],
+                            content=job["result"] or "",
+                            status=tm_status,
+                        )
+                        new_tool_messages.append(tm)
+                        job["consumed"] = True
+
+                if new_tool_messages:
+                    current_messages.extend(new_tool_messages)
+
+                if i == max_iterations - 1:
+                    current_messages.append(
+                        HumanMessage(
+                            content=(
+                                "[WARNING] This is your final iteration. Call"
+                                " 'submit_answer' to provide your final"
+                                " diagnosis."
+                            )
+                        )
+                    )
+                    submit_tool = next(
+                        (t for t in traced_tools if t.name == "submit_answer"),
+                        None,
+                    )
+                    llm = base_llm.bind_tools(tools=[submit_tool] if submit_tool else [])
+                else:
+                    if i > 0:
+                        current_messages.append(
+                            HumanMessage(
+                                content=(
+                                    "You have not completed the diagnosis yet"
+                                    f" (iteration {i + 1} of {max_iterations})."
+                                )
+                            )
+                        )
+                    llm = base_llm.bind_tools(tools=traced_tools)
+
+                async def run_stream():
+                    full_response = None
+                    trace_id = CURRENT_TRACE_ID.get()
+                    async for chunk in llm.astream(current_messages):
+                        if full_response is None:
+                            full_response = chunk
+                        else:
+                            full_response += chunk
+
+                        pass
+                    return full_response
+
+                with TraceSpan(name="gemini_diagnoser_call", ctx=self.ctx) as span:
+                    span.payload = {
+                        "context_components": {
+                            "system_prompt": system_prompt,
+                            "plan_and_history": plan_and_history,
+                            "checker_feedback": checker_feedback,
+                            "visible_ui_elements": minimal_list,
+                            "diagnostic_query": prompt,
+                        }
+                    }
+                    response = await invoke_llm_with_timeout_message(run_stream())
+                    span.result = (
+                        response.content if hasattr(response, "content") else str(response)
+                    )
+
+                    if hasattr(response, "usage_metadata") and response.usage_metadata is not None:
+                        usage = response.usage_metadata
+                        if isinstance(usage, dict):
+                            prompt_token_count = usage.get("prompt_token_count")
+                            candidates_token_count = usage.get("candidates_token_count")
+                            cached_content_token_count = usage.get("cached_content_token_count")
+                        else:
+                            prompt_token_count = getattr(usage, "prompt_token_count", None)
+                            candidates_token_count = getattr(usage, "candidates_token_count", None)
+                            cached_content_token_count = getattr(
+                                usage, "cached_content_token_count", None
+                            )
+
+                        if span.payload is None:
+                            span.payload = {}
+
+                        span.payload["token_usage"] = {
+                            "prompt_token_count": prompt_token_count,
+                            "candidates_token_count": candidates_token_count,
+                            "cached_content_token_count": (cached_content_token_count),
+                        }
+
+                if response.content:
+                    outcome = response.content
+
+                if not response.tool_calls:
+                    break
+
+                submit_call = next(
+                    (
+                        tc
+                        for tc in response.tool_calls
+                        if tc["name"] == "submit_answer" or tc["name"].endswith(":submit_answer")
+                    ),
+                    None,
+                )
+                if submit_call:
+                    args = submit_call["args"]
+                    analysis = args.get("analysis", "No analysis provided.")
+                    steps = args.get("actionable_steps", [])
+
+                    submit_tool = next(
+                        (
+                            t
+                            for t in traced_tools
+                            if t.name == "submit_answer" or t.name.endswith(":submit_answer")
+                        ),
+                        None,
+                    )
+                    if submit_tool:
+                        submit_tool.invoke(args)
+
+                    outcome = f"Analysis: {analysis} | Actionalble Steps: "
+                    for step in steps:
+                        outcome += f"{step} "
+                    break
+
+                if i == max_iterations - 1:
+                    tool_names = ", ".join(tc["name"] for tc in response.tool_calls)
+                    warning = (
+                        "\n\n[Warning: Diagnoser reached the maximum iteration"
+                        f" limit of {max_iterations} and could not execute"
+                        f" tool(s): {tool_names}]"
+                    )
+                    if outcome == "No result":
+                        outcome = (
+                            "Diagnoser reached the maximum iteration limit of"
+                            f" {max_iterations} while attempting to call tools:"
+                            f" {tool_names}"
+                        )
+                    else:
+                        outcome += warning
+                    break
+
+                current_messages.append(response)
+
+                async def run_tool(tc):
+                    tool_name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
+                    logger.info(f"Diagnoser requested tool: {tool_name}")
+
+                    result = ""
+                    status = "success"
+                    try:
+                        if tool_name == "video_analyzer":
+                            job_id = f"job_{uuid.uuid4().hex}"
+                            self.ctx.background_jobs[job_id] = {
+                                "tool_call_id": tc["id"],
+                                "status": "running",
+                                "result": None,
+                                "consumed": False,
+                                "task": None,
+                            }
+                            if self.ctx.data_engine:
+                                self.ctx.data_engine.register_background_task(
+                                    task_id=job_id,
+                                    summary=(f"Diagnoser Video Analyzer Tool Call: {tc['id']}"),
+                                )
+
+                            async def run_bg_job():
+                                try:
+                                    tool_to_run = next(
+                                        (t for t in traced_tools if t.name == "video_analyzer"),
+                                        None,
+                                    )
+                                    if tool_to_run:
+                                        args = dict(tc["args"])
+                                        result_obj = await invoke_tool_with_injection(
+                                            tool=tool_to_run,
+                                            args=args,
+                                            tool_call_id=tc["id"],
+                                            state=state,
+                                        )
+                                        result_content = get_tool_result_content(result_obj)
+                                        self.ctx.background_jobs[job_id]["result"] = result_content
+                                        self.ctx.background_jobs[job_id]["status"] = "completed"
+                                        if self.ctx.data_engine:
+                                            self.ctx.data_engine.unregister_background_task(
+                                                task_id=job_id,
+                                                status="completed",
+                                            )
+                                    else:
+                                        self.ctx.background_jobs[job_id]["result"] = (
+                                            "Error: video_analyzer tool not found"
+                                        )
+                                        self.ctx.background_jobs[job_id]["status"] = "failed"
+                                        if self.ctx.data_engine:
+                                            self.ctx.data_engine.unregister_background_task(
+                                                task_id=job_id, status="failed"
+                                            )
+                                except asyncio.CancelledError:
+                                    logger.info(f"Background job {job_id} cancelled.")
+                                    raise
+                                except Exception as e:
+                                    logger.error(f"Background job {job_id} failed: {e}")
+                                    self.ctx.background_jobs[job_id]["result"] = str(e)
+                                    self.ctx.background_jobs[job_id]["status"] = "failed"
+                                    if self.ctx.data_engine:
+                                        self.ctx.data_engine.unregister_background_task(
+                                            task_id=job_id, status="failed"
+                                        )
+
+                            task = asyncio.create_task(run_bg_job())
+                            self.ctx.background_jobs[job_id]["task"] = task
+
+                            return ToolMessage(
+                                tool_call_id=tc["id"],
+                                content=(
+                                    f"Video analyzer started. Job ID: {job_id}."
+                                    " [Warning] No background logs, status"
+                                    " updates, or errors will be logged."
+                                ),
+                                status="success",
+                            )
+
+                        if ":" in tool_name:
+                            tool_to_run = get_tool_by_name(tool_name, traced_tools)
+                        else:
+                            tool_to_run = next(
+                                (t for t in traced_tools if t.name == tool_name),
+                                None,
+                            )
+                        if tool_to_run:
+                            args = dict(tc["args"])
+                            with TraceSpan(name=tool_name, ctx=self.ctx) as span:
+                                span.payload = {"args": args}
+                                try:
+                                    result_obj = await invoke_tool_with_injection(
+                                        tool=tool_to_run,
+                                        args=args,
+                                        tool_call_id=tc["id"],
+                                        state=state,
+                                    )
+                                    result = get_tool_result_content(result_obj)
+                                    if isinstance(result, list):
+                                        result_str = "\n".join(map(str, result))
+                                    elif not isinstance(result, str):
+                                        result_str = str(result)
+                                    else:
+                                        result_str = result
+                                    span.result = result
+                                    if result_str.startswith("Error"):
+                                        status = "error"
+                                        span.status = "failed"
+                                        span.error = result
+                                except Exception as e:
+                                    status = "error"
+                                    span.status = "failed"
+                                    span.error = str(e)
+                                    raise e
+                        else:
+                            result = f"Error: Tool {tool_name} not supported"
+                            status = "error"
+                    except Exception as e:
+                        logger.error(f"Error running tool {tool_name}: {e}")
+                        result = f"Error running tool {tool_name}: {e}"
+                        status = "error"
+
+                    return ToolMessage(
+                        tool_call_id=tc["id"],
+                        content=result,
+                        status=status,
+                    )
+
+                # Execute all requested tools in parallel
+                active_tool_calls = [
+                    tc
+                    for tc in response.tool_calls
+                    if (tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"])
+                    != "google_search"
+                ]
+                tool_outputs = await asyncio.gather(*(run_tool(tc) for tc in active_tool_calls))
+
+                for tm in tool_outputs:
+                    current_messages.append(tm)
+        finally:
+            # 6. Cleanup on Exit
+            for job in self.ctx.background_jobs.values():
+                if job.get("status") == "running" and job.get("task") is not None:
+                    job["task"].cancel()
+
+        return outcome
+
+    def _format_minimal_list(self, fused_xml: list[dict[str, Any]]) -> str:
+        """Formats fused XML into a minimal structured list."""
+        lines = []
+        for i, node in enumerate(fused_xml):
+            text = node.get("text") or node.get("content-desc") or ""
+            bounds = node.get("bounds")
+
+            ocr_elements = node.get("ocr_elements")
+            if ocr_elements:
+                for ocr in ocr_elements:
+                    lines.append(f"[{i}] OCR Text: '{ocr['text']}' | Bounds: {ocr['bounds']}")
+            elif text.strip():
+                lines.append(f"[{i}] Text: '{text.strip()}' | Bounds: {bounds}")
+
+        return "\n".join(lines)

artemis/agents/explorer/constants.py

@@ -0,0 +1,72 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Constants and prompt templates for the UI Explorer agent."""
+
+EXPLORE_DESCRIPTIONS = {
+    "flash": {
+        "description": (
+            "[EXPLORER] Call this to ask the UI Explorer agent to "
+            "locate coordinates on the screen layout by object detection."
+        ),
+        "query_description": (
+            "Pipe-separated list of target element descriptions, semantic"
+            " action terms, or color descriptors to search for concurrently"
+            " (e.g., 'red settings button | gear icon')."
+        ),
+        "rule_prompt": (
+            "Use the explorer tool to quickly check for visual icons, buttons, or text by name."
+        ),
+    },
+    "pro": {
+        "description": (
+            "[EXPLORER] Call this to ask the UI Explorer agent to locate"
+            " coordinates on the screen layout by executing XML, OCR,"
+            " coordinate search, and object detection."
+        ),
+        "query_description": (
+            "The target element or information to search for, including descriptions."
+        ),
+        "rule_prompt": (
+            "Call the explorer tool to search, OCR, or inspect it (maximum {max_iterations} tries)."
+        ),
+        "version_prompt": (
+            "You are running with a maximum of {max_iterations} turns: on your"
+            " final turn, you will receive a warning reminding you to call"
+            " submit_answer."
+        ),
+    },
+    "ultra": {
+        "description": (
+            "[EXPLORER] Full visual reasoning agent. Deeply inspects screen"
+            " regions and utilizes all perception/pixel-level tools with a high"
+            " execution limit (up to {max_iterations} turns) for"
+            " high-difficulty or layout-critical tasks."
+        ),
+        "query_description": (
+            "The target element or information to search for, including descriptions."
+        ),
+        "rule_prompt": (
+            "For highly complex visual searches, details validation, or layout"
+            " verification, invoke the explorer tool to perform multi-turn"
+            " visual reasoning and detailed coordinate checks (maximum"
+            " {max_iterations} tries)."
+        ),
+        "version_prompt": (
+            "You are running with a maximum of {max_iterations} turns: on your"
+            " final turn, you will receive a warning reminding you to call"
+            " submit_answer."
+        ),
+    },
+}

artemis/agents/explorer/explorer.json

@@ -0,0 +1,27 @@
+{
+  "IDENTITY": "You are the agentic UI Explorer designed to identify objects, scan text, and extract UI attributes on a phone screen.",
+  "OPERATING PRINCIPLES": [
+    "**Confidence-Driven Cost-Benefit Strategy**: Maximize localization accuracy while minimizing execution latency and tool calls:",
+    "    - **Direct Submit (High Confidence)**: If you are highly confident in the target coordinates from the initial annotated screenshot and minimal list, call `submit_answer` immediately in Turn 1 to save time.",
+    "    - **Cost-Benefit Turn Planning (Low Confidence / Ambiguity)**: If you are not highly confident. Consider structuring your exploration turns by weighing the latency/cost of each tool against the expected accuracy payoff:",
+    "        - **Parallel Context Gathering**: MUST parallel calling all perception tools in the first turn to gather broad screen context.",
+    "        - **Targeted Verification**: If micro-details or coordinates are still ambiguous, zoom in on promising candidate areas (parallel tool calls). Note that a single zoom-in or multi-zoom-in operation takes one turn, plan your verification steps accordingly.",
+    "        - **Finalization**: Submit coordinates using `submit_answer`. Stop exploring as soon as further tool calls would yield negligible accuracy gains relative to their latency.",
+    "**Tool Capabilities**: You have three classes of tools: `Perception` tools (to extract and locate elements, coordinates, or inspect specific regions), `Transformation` tools (for pixel-level image editing), and the `Finalization` tool (to submit final answers and terminate). Note that in your final turn, all other tools are stripped away, leaving only the finalization tool, forcing you to submit your final answer.",
+    "**Parallel vs Sequential Invocations**: You are fully supported to call perception tools in parallel (within a single turn) to expedite your search. However, transformation tools are expected to be called sequentially (since they transform the image for subsequent perception steps).",
+    "**Multimodal Perception**: You have access to images marked with bounding boxes and labels. Use both the visual information in the images and the structured text lists to make decisions.",
+    "**Image Pool Tracking**: Images are managed in an Image Pool (starting with 'img_0'). Transformations via transformation tools will register new image IDs (e.g., 'img_1').",
+    "**Prefix System**: Elements in the screenshots are labeled with prefixes returned by your tools (e.g., Numbers for initial UI, letters for tool outputs). Always reference these precise labels (e.g., \"I choose S2\") when making decisions.",
+    "**Contextual Awareness**: Always prioritize checking the context feedback to learn from previous mistakes.",
+    "**Ask Perception Tool**: Call the unified `ask_perception_tool` specifying `search_query`, `nx`, `ny`, and `detect_queries` to perform text lookups, coordinate queries, and visual detection concurrently.",
+    "**Locating Components**: Locate target components on screen by matching the X-labels (from text/coordinate searches) and O-labels (from object detection) in the text response with their corresponding green, blue, or red indicator dots on the output images."
+  ],
+  "BEHAVIORAL GUIDELINES & SOFT CONSTRAINTS": [
+    "**Avoid Hallucination**: Avoid guessing coordinates. If you are unable to locate the target element on the screen after trying all perception options, submit `[]` in `candidates` and document your observations as specified in the `submit_answer` schema.",
+    "**Ambiguity Resolution**: If the screen contains multiple identical elements or several candidates that match the query (e.g., three \"Add\" buttons), you are encouraged to return a ranked list of all viable candidates (up to 10 candidates) via `submit_answer`, ordered by your confidence, with brief descriptions to distinguish them.",
+    "**Semantic & Icon Synonym Fallback**: Literal exact matches are ideal, but semantic equivalents and synonyms are fully acceptable. If a text element is requested but only a corresponding icon is available, you should attempt to map the synonym and locate the icon (e.g., map \"Search\" to a magnifying glass icon, map \"Add\" to \"+\" symbol)."
+  ],
+  "STRICT CONSTRAINTS": [
+    "**Validation Standards**: All submitted coordinates `[nx, ny]` for candidates in `submit_answer` must strictly be integers in the `[0-1000]` range inclusive. If your parameters fail business validation rules, you will receive dynamic error responses in your execution loop, allowing you to self-correct and re-submit."
+  ]
+}

artemis/agents/flash/context_compressor.py

@@ -0,0 +1,118 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Universal Context Compressor for Artemis Flash profile.
+
+Replaces pruned historical screenshots with high-density visual step summaries
+when ready, while seamlessly falling back to standard pruned action text if
+the background summary is still in flight.
+"""
+
+from typing import Any
+
+from langchain_core.messages import BaseMessage
+
+from artemis.agents.flash.summarizer import VisualStepSummarizer
+from artemis.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+def compress_flash_messages(
+    messages: list[BaseMessage],
+    summarizer: VisualStepSummarizer | None = None,
+    prune_history_xml: bool = True,
+) -> None:
+    """Compresses multi-turn conversation messages in-place.
+
+    Rules:
+    1. Keeps the latest screenshot and live UI hierarchy completely intact.
+    2. For earlier steps, if a VisualStepSummarizer has completed the step's summary,
+       replaces the intermediate image block with the objective summary text.
+    3. If the background summary is not ready yet, gracefully falls back to the existing
+       behavior (omitting the image block and retaining the original action text).
+    4. If prune_history_xml is True, removes heavy outdated UI Element lists from past turns.
+    """
+    if not messages:
+        return
+
+    # 1. Identify the last message containing an active screenshot
+    last_img_msg_idx = -1
+    for idx in range(len(messages) - 1, -1, -1):
+        msg = messages[idx]
+        msg_content = getattr(msg, "content", None)
+        if isinstance(msg_content, list):
+            has_image = any(
+                isinstance(block, dict) and block.get("type") in ("image_url", "image")
+                for block in msg_content
+            )
+            if has_image:
+                last_img_msg_idx = idx
+                break
+
+    # 2. Iterate through historical messages and apply compression/fallback
+    tool_step_counter = 0
+    for idx in range(len(messages)):
+        msg = messages[idx]
+        msg_content = getattr(msg, "content", None)
+        if not isinstance(msg_content, list):
+            continue
+
+        # Keep the latest live screen intact
+        if idx == last_img_msg_idx:
+            continue
+
+        # Count tool steps strictly corresponding to FlashRunner action turns
+        is_tool_msg = getattr(msg, "tool_call_id", None) is not None
+        if is_tool_msg:
+            tool_step_counter += 1
+            current_step_num = tool_step_counter
+        else:
+            current_step_num = None
+
+        new_blocks: list[dict[str, Any]] = []
+        summary_injected = False
+
+        for b in msg_content:
+            if not isinstance(b, dict):
+                continue
+            b_type = b.get("type", "")
+            b_text = b.get("text", "")
+
+            # Filter heavy outdated XML hierarchy from past steps to save tokens
+            if prune_history_xml and b_type == "text" and "--- UI Element List ---" in b_text:
+                continue
+
+            if b_type in ("image_url", "image"):
+                # Check if this is a tool execution step and background summarizer has a ready summary
+                if (
+                    current_step_num
+                    and summarizer
+                    and summarizer.has_summary(current_step_num)
+                    and not summary_injected
+                ):
+                    summary_text = summarizer.get_summary(current_step_num)
+                    if summary_text:
+                        new_blocks.append({"type": "text", "text": summary_text})
+                        summary_injected = True
+                # If summarizer is not ready or initial screen, gracefully omit the image block
+                continue
+
+            new_blocks.append(b)
+
+        # Defensive fallback: never leave ToolMessage or HumanMessage content empty
+        if not new_blocks:
+            new_blocks = [{"type": "text", "text": "Action completed." if is_tool_msg else ""}]
+
+        msg.content = new_blocks

artemis/agents/flash/flash_runner.md

@@ -0,0 +1,41 @@
+# ROLE & OBJECTIVE
+You are **ARTEMIS Flash Runner**, an autonomous and highly efficient Android Device Execution Agent. Your goal is to accomplish the user's objective on the device.
+
+**Objective: {{ goal }}**
+
+---
+
+# 1. COGNITIVE PROTOCOL
+You operate in a dynamic observation-reasoning-execution loop. **CRITICAL RULE: YOU MUST NEVER INVOKE A TOOL SILENTLY!** In every single turn, you **MUST FIRST output a natural language reasoning paragraph BEFORE generating any tool call.**
+
+- **Turn 1 Only (Initial Planning)**: Briefly decompose the overall `{{ goal }}` into high-level visual **Milestones** in your text output. Do not repeat this macro breakdown in subsequent turns.
+
+---
+
+# 2. DYNAMIC ENVIRONMENT & TIME AWARENESS
+Keep in mind that the mobile device is an asynchronous, constantly evolving environment. However, your actions entail an inherent "turn latency" (a 3–7 second delay caused by model inference and network transmission between normal consecutive actions).
+- **State Drift & Progression Tolerance**: While you think, video playback advances, banners scroll, and web pages finish loading. If the latest observation has progressed further than expected (e.g., an ad automatically skipped or a redirect landed), **do not treat it as an error**. Adapt smoothly to the live screen in front of you.
+- **Timed Waiting & Transitions**: When facing mid-transition animations, loading spinners, required watch durations, or ad countdown timers, use the precise `wait_for_delay` command to allow time to pass or the UI state to stabilize. Some transition states are subtle and may even resemble a phone bug or system freeze; it is always advisable to wait first, and only consider retrying or choosing an alternative path if the state remains broken.
+- **Time-Sensitive Tasks & Transient UI**: First, you must understand that each of your decision turns takes several seconds of model inference and network delay, while the mobile device state is continuously changing. A target element that you see on the current screenshot (e.g., auto-dismissing overlays, temporary message bars, menus or control bars with short timeouts) may have already vanished by the time your physical click command reaches the device! This is one of the most common reasons why the UI does not change as expected after a single step. We call these **Time-Sensitive Tasks**.
+  - *Mental Model*: When facing these tasks, any attempt to interact step-by-step (such as tapping once to wake up a menu/bar and waiting for the next turn to tap the target button) WILL fail because the element will auto-dismiss during your decision delay. You MUST combine the summon action on the trigger element (`Trigger`) and the target action (`Target`) into a single atomic sequence.
+  - *Chained Execution*: Use `click_sequence(sequence=[[trigger_x, trigger_y], [target_x, target_y]])` in `0-1000` normalized scale to execute both taps consecutively with a 50ms delay, catching the transient element before the timeout closes it.
+
+---
+
+# 3. STAGNATION & LOOP PREVENTION
+- **Retrying is not the only option.**: If you perform the same action on the same target for two consecutive turns and the screen or XML shows **no substantial change**, further retries are unlikely to be useful; the probability of the execution tool itself malfunctioning is very low, so you should pause and consider the underlying reason.
+- **Flexibly Adjust Strategies:** Always prioritize resolving the immediate issue. If you encounter overlays or pop-ups blocking input, or if an element is unresponsive, actively consider alternative ways to complete the task. If an action fails to proceed as expected, systematically consider whether the issue lies with the device itself, the operational path, or a change in state occurring between your actual interaction and the screenshot (Time-Sensitive Tasks).
+
+---
+
+# 4. ACTION & MEMORY RULES
+1. **Normalized Coordinates (`0-1000` Scale)**: Always use normalized `[x, y]` coordinates in `0-1000` scale for any location-based target (e.g., `target=[500, 600]` or `click_sequence=[[300, 400], [500, 600]]`).
+2. **Do not determine element coordinates based on guesswork.** If the element is visible in the screenshot but its coordinates are missing from the provided XML tree, please use the `ask_explorer` tool to locate the element first.
+3. **Screenshot Pruning Awareness**: To optimize speed, the system **prunes old intermediate screenshots from your history**—you only ever see the *latest* live screenshot and UI tree. Therefore, if you spot critical text in an earlier step that you will need later (e.g., verification codes, specific titles, search lists), **you must explicitly verbalize and write those details into your text thinking/notes** so they persist across turns.
+4. **App Launching**: To start or reset an application, always use `manage_app(action="launch", app_name="...")` or `action="stop"`. Do not manually swipe across home screens.
+
+---
+
+# 5. TERMINATION PROTOCOL (`report_task_status`)
+- **Completion (`status="completed"`)**: ONLY call `report_task_status(status="completed", ...)` after you have **visually verified** from the latest screen that the user's objective (`{{ goal }}`) is fully achieved.
+- **Failure (`status="failed"`)**: If the task is genuinely impossible due to paywalls, unrecoverable crashes, or hitting an absolute dead end after sensible fallbacks, call `report_task_status(status="failed", ...)` with the exact blocking reason.

artemis/agents/flash/flash_summarizer.md

@@ -0,0 +1,41 @@
+# ROLE & OBJECTIVE
+You are the Step Summarizer for an Android UI automation agent. Your task is to synthesize the Before/After screenshots and physical action into a concise, high-information-density, strictly objective first-person historical memory.
+
+---
+
+# PERSPECTIVE & FORMAT CONSTRAINTS
+1. **First-Person Perspective**: Write strictly from the agent's first-person perspective using **"I"** (e.g., "In Step {{ step_number }}, I tapped...", "I swiped up on... and observed..."). NEVER use third-person terms like "The agent", "The operator", or "The system".
+2. **Single Continuous Paragraph**: Your output MUST be exactly **one compact, continuous paragraph** (1–3 sentences, 35–65 words).
+3. **No Lists or Formatting**: Do NOT use bullet points, numbered lists, markdown headers, bold labels, or line breaks in your output.
+
+---
+
+# CORE OBSERVATION RULES
+1. **Zero Subjective Validation (STRICT)**:
+   - Absolutely avoid declaring semantic goal achievement, subjective success, or final failure. In historical compression, subjective assumptions cause future turns to falsely believe a subgoal was definitively accomplished or permanently blocked.
+   - **STRICTLY BANNED WORDS**: `successfully`, `completed`, `achieved`, `entered`, `navigated to`, `arrived at`, `failed`, `unsuccessful`, `could not`, `impossible`.
+2. **Action & Visual Delta Structure**:
+   - **Target & Action**: Identify what control/button was targeted (referencing the red visual indicator on the BEFORE screen) and the physical action performed.
+   - **Objective Transition**: Describe the physical screen change (e.g., page transitioned, modal dialog opened, list scrolled revealing new items, checkbox/toggle toggled, keyboard appeared).
+   - **Preserve Critical Data & Verifications**: Faithfully transcribe visible text, toast alerts, error banners, verification codes, prices, account names, or tracking numbers that appeared on the AFTER screen.
+
+---
+
+# EXAMPLES OF HIGH-QUALITY SUMMARIES
+
+- **Click / Navigation**:
+  "In Step 1, I tapped the 'Settings' gear icon marked in red on the Home screen; the main Settings menu opened showing 'Network & internet', 'Connected devices', and 'Apps'."
+
+- **Text Input**:
+  "In Step 2, I typed 'Tokyo Hotel' into the search input box; a dropdown list appeared displaying 5 destination suggestions with 'Tokyo Station, Japan' as the top entry."
+
+- **Swipe / Scroll**:
+  "In Step 3, I swiped up along the flight results list; the screen scrolled downward approximately one page, revealing 3 additional evening flights starting from '$420'."
+
+- **Modal / Alert State**:
+  "In Step 4, I tapped the 'Confirm Booking' button; an alert dialog titled 'Payment Method Required' appeared over the view with an 'Add Card' option."
+
+---
+
+# TASK
+Generate the single-paragraph first-person summary for Step {{ step_number }} following all constraints above:

artemis/agents/flash/runner.py

@@ -0,0 +1,524 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Universal Multi-Model FlashRunner for Artemis.
+
+Executes autonomous reactive mobile workflows across Google Gemini,
+OpenAI GPT-4o/o3, Anthropic Claude 3.5/3.7, and OpenRouter endpoints.
+"""
+
+import asyncio
+import base64
+import json
+from pathlib import Path
+import uuid
+
+from jinja2 import Template
+from langchain_core.messages import (
+    BaseMessage,
+    HumanMessage,
+    SystemMessage,
+    ToolMessage,
+)
+
+from artemis.agents.flash.context_compressor import compress_flash_messages
+from artemis.agents.flash.summarizer import VisualStepSummarizer
+from artemis.agents.validator.tool_declarations import (
+    ACTION_TOOL_NAMES,
+    ASK_EXPLORER_TOOL,
+    CLICK_SEQUENCE_TOOL,
+    MobileActionExecutor,
+    REPORT_TASK_STATUS_TOOL,
+    VALIDATOR_TOOLS_DECLARATION,
+    capture_screenshot_and_parse_ui,
+    prune_intermediate_screenshots,
+)
+from artemis.config import StepSummarizerConfig, load_agent_config
+from artemis.context import ArtemisContext
+from artemis.controllers.unified_controller import UnifiedMobileController
+from artemis.data_engine.trace import trace
+from artemis.graph.perception import _check_injected_instruction_file
+from artemis.graph.state import State
+from artemis.services.llm import (
+    RobustChatModelWrapper,
+    get_google_llm,
+    get_llm,
+    invoke_llm_with_timeout_message,
+)
+from artemis.utils.coordinates import parse_swipe_parameters
+from artemis.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+class FlashRunner:
+    """Reactive, ultra-fast agent execution loop supporting all VLM providers."""
+
+    def __init__(self, ctx: ArtemisContext, goal: str, max_turns: int | None = None):
+        self.ctx = ctx
+        self.goal = goal
+        try:
+            cfg = load_agent_config()
+            self.max_turns = max_turns if max_turns is not None else cfg.flash.max_turns
+            self.step_summarizer_cfg = cfg.flash.step_summarizer
+        except Exception:
+            self.max_turns = max_turns if max_turns is not None else 30
+            self.step_summarizer_cfg = StepSummarizerConfig()
+
+        self.controller = UnifiedMobileController(ctx)
+        self.executor = MobileActionExecutor(ctx, self.controller)
+        self.summarizer = (
+            VisualStepSummarizer(ctx, model_name=self.step_summarizer_cfg.model)
+            if self.step_summarizer_cfg.enabled
+            else None
+        )
+
+    def _get_tools(self) -> list:
+        tools = [t for t in VALIDATOR_TOOLS_DECLARATION if t.name != "report_failure_analysis"]
+        tools.insert(1, CLICK_SEQUENCE_TOOL)
+        tools.append(ASK_EXPLORER_TOOL)
+        tools.append(REPORT_TASK_STATUS_TOOL)
+        return tools
+
+    def _prune_intermediate_screenshots(self, messages: list[BaseMessage]) -> None:
+        """Prunes binary screenshot blocks from intermediate observation messages."""
+        prune_intermediate_screenshots(messages)
+
+    @trace(type="agent", name="FlashRunner")
+    async def run(self, state: State) -> dict:
+        logger.info(f"Starting Artemis Flash reactive loop for goal: {self.goal}")
+
+        # 1. Initialize Universal LLM via Service Layer
+        try:
+            llm = get_llm(self.ctx, name="operator")
+        except Exception as e:
+            logger.warning(f"Failed to get operator LLM from config, using default: {e}")
+
+            llm = RobustChatModelWrapper(get_google_llm(model_name="gemini-2.5-flash"), self.ctx)
+
+        tools_declaration = self._get_tools()
+
+        # 2. Capture Initial State (Screenshot + UI Tree)
+        shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
+            self.ctx, state, self.controller, skip_settling=False
+        )
+        state.latest_screenshot = shot_path
+
+        user_content: list[dict] = [
+            {"type": "text", "text": f"Your objective is: {self.goal}"},
+        ]
+        if img_bytes:
+            img_b64 = base64.b64encode(img_bytes).decode("utf-8")
+            user_content.append(
+                {
+                    "type": "image_url",
+                    "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"},
+                }
+            )
+        if xml_list:
+            user_content.append(
+                {
+                    "type": "text",
+                    "text": f"--- UI Element List ---\n{xml_list}",
+                }
+            )
+        user_content.append(
+            {
+                "type": "text",
+                "text": (
+                    "CRITICAL RULE: In every single turn, you MUST FIRST output a natural"
+                    " language reasoning/explanation paragraph BEFORE invoking any tool call."
+                ),
+            }
+        )
+
+        # Render System Prompt
+        prompt_path = Path(__file__).parent / "flash_runner.md"
+        prompt_template = prompt_path.read_text(encoding="utf-8")
+        system_prompt = Template(prompt_template).render(goal=self.goal)
+
+        messages: list[BaseMessage] = [
+            SystemMessage(content=system_prompt),
+            HumanMessage(content=user_content),
+        ]
+
+        turns = 0
+        final_report = None
+        current_pre_screenshot_bytes = img_bytes
+        current_xml_list = xml_list
+
+        while turns < self.max_turns:
+            turns += 1
+            logger.info(f"--- Artemis Flash Turn {turns}/{self.max_turns} ---")
+
+            if self.ctx.data_engine:
+                self.ctx.data_engine.allocate_step_id()
+
+            # Check for real-time injected instructions
+            if self.ctx.data_engine and self.ctx.data_engine.base_dir:
+                try:
+                    injected_instruction = await asyncio.to_thread(
+                        _check_injected_instruction_file,
+                        str(self.ctx.data_engine.base_dir),
+                    )
+                    if injected_instruction:
+                        messages.append(
+                            HumanMessage(
+                                content=(
+                                    "[REAL-TIME INJECTED INSTRUCTION from user]:"
+                                    f" {injected_instruction}\nYou MUST immediately follow"
+                                    " this instruction and adjust your plan/actions."
+                                )
+                            )
+                        )
+                except Exception as e:
+                    logger.warning(f"Failed to check injected instruction in FlashRunner: {e}")
+
+            # Compress previous history before LLM call to save context and inject objective summaries
+            if self.summarizer:
+                compress_flash_messages(
+                    messages,
+                    summarizer=self.summarizer,
+                    prune_history_xml=self.step_summarizer_cfg.prune_history_xml,
+                )
+            else:
+                prune_intermediate_screenshots(messages)
+
+            # Tool restriction on the final turn
+            if turns == self.max_turns:
+                messages.append(
+                    HumanMessage(
+                        content=(
+                            "[WARNING] This is your final turn; only"
+                            " 'report_task_status' is available."
+                        )
+                    )
+                )
+                current_tools = [t for t in tools_declaration if t.name == "report_task_status"]
+            else:
+                current_tools = tools_declaration
+
+            # Bind active tools
+            bound_llm = llm.bind_tools(current_tools)
+
+            # Invoke Model with Streaming & Retry
+            response = None
+            max_retries = 3
+
+            for attempt in range(max_retries):
+                try:
+
+                    async def run_stream():
+                        full_response = None
+                        async for chunk in bound_llm.astream(messages):
+                            if full_response is None:
+                                full_response = chunk
+                            else:
+                                full_response += chunk
+                        return full_response
+
+                    response = await invoke_llm_with_timeout_message(
+                        run_stream(), timeout_seconds=10, hard_timeout=180
+                    )
+                    if response:
+                        break
+                except Exception as e:
+                    logger.warning(f"FlashRunner LLM turn failed on attempt {attempt + 1}: {e}")
+                    if attempt == max_retries - 1:
+                        raise e
+                    await asyncio.sleep(2.0**attempt)
+
+            if response is None:
+                break
+
+            messages.append(response)
+
+            # Extract thought and tool calls
+            raw_text = response.content if isinstance(response.content, str) else ""
+            if isinstance(response.content, list):
+                raw_text = "".join(
+                    b.get("text", "")
+                    for b in response.content
+                    if isinstance(b, dict) and "text" in b
+                )
+
+            # Extract token usage metadata from response
+            step_token_usage = None
+            if hasattr(response, "usage_metadata") and response.usage_metadata:
+                u = response.usage_metadata
+                pr = u.get("input_tokens") or u.get("prompt_tokens") or 0
+                co = u.get("output_tokens") or u.get("completion_tokens") or 0
+                to = u.get("total_tokens") or (pr + co)
+                if to > 0:
+                    step_token_usage = {
+                        "prompt_tokens": int(pr),
+                        "completion_tokens": int(co),
+                        "total_tokens": int(to),
+                    }
+            elif hasattr(response, "response_metadata") and isinstance(
+                response.response_metadata, dict
+            ):
+                u = response.response_metadata.get(
+                    "usage_metadata"
+                ) or response.response_metadata.get("token_usage")
+                if isinstance(u, dict):
+                    pr = (
+                        u.get("input_tokens")
+                        or u.get("prompt_tokens")
+                        or u.get("prompt_token_count")
+                        or 0
+                    )
+                    co = (
+                        u.get("output_tokens")
+                        or u.get("completion_tokens")
+                        or u.get("candidates_token_count")
+                        or 0
+                    )
+                    to = u.get("total_tokens") or u.get("total_token_count") or (pr + co)
+                    if to > 0:
+                        step_token_usage = {
+                            "prompt_tokens": int(pr),
+                            "completion_tokens": int(co),
+                            "total_tokens": int(to),
+                        }
+
+            if not step_token_usage or step_token_usage.get("total_tokens", 0) <= 0:
+                calc_prompt_chars = 0
+                calc_images = 0
+                for msg in messages:
+                    c = getattr(msg, "content", "")
+                    if isinstance(c, str):
+                        calc_prompt_chars += len(c)
+                    elif isinstance(c, list):
+                        for block in c:
+                            if isinstance(block, dict):
+                                if block.get("type") == "image_url" or "image_url" in block:
+                                    calc_images += 1
+                                else:
+                                    calc_prompt_chars += len(str(block.get("text", "")))
+                            else:
+                                calc_prompt_chars += len(str(block))
+                prompt_tokens = (calc_prompt_chars // 4) + (calc_images * 258)
+                completion_tokens = len(raw_text) // 4
+                step_token_usage = {
+                    "prompt_tokens": max(1, prompt_tokens),
+                    "completion_tokens": max(1, completion_tokens),
+                    "total_tokens": max(1, prompt_tokens + completion_tokens),
+                }
+
+            if self.ctx.data_engine:
+                current_step_id = getattr(self.ctx.data_engine, "current_step_id", None)
+                self.ctx.data_engine.record_trace(
+                    type="llm_call",
+                    name="FlashRunner",
+                    payload={"token_usage": step_token_usage, "response": raw_text},
+                    step_id=current_step_id,
+                    status="success",
+                )
+
+            tool_calls = response.tool_calls or []
+
+            # Fallback text parsing if tool_calls not parsed natively
+            if not tool_calls and "```json" in raw_text:
+                try:
+                    snippet = raw_text.split("```json")[1].split("```")[0].strip()
+                    parsed = json.loads(snippet)
+                    if isinstance(parsed, dict) and "name" in parsed:
+                        tool_calls = [
+                            {
+                                "name": parsed["name"],
+                                "args": parsed.get("args", {}),
+                                "id": str(uuid.uuid4()),
+                            }
+                        ]
+                except Exception:
+                    pass
+
+            if not tool_calls:
+                logger.info(
+                    f"FlashRunner received response without tool calls at turn {turns}:"
+                    f" {raw_text[:100]}..."
+                )
+                if turns == self.max_turns:
+                    final_report = {"status": "failed", "explanation": raw_text}
+                    break
+                messages.append(
+                    HumanMessage(
+                        content=(
+                            "You did not call any tools. Please make progress"
+                            " by calling an action tool or 'report_task_status'."
+                        )
+                    )
+                )
+                continue
+
+            # Process tool calls
+            for tc in tool_calls:
+                name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
+                args = tc.get("args") or {}
+                tc_id = tc.get("id") or str(uuid.uuid4())
+                logger.info(f"Executing Flash tool: {name}({args})")
+
+                if name == "report_task_status":
+                    final_report = args
+                    if self.ctx.data_engine:
+                        try:
+                            if self.ctx.data_engine.current_step_id is None:
+                                self.ctx.data_engine.allocate_step_id()
+                            self.ctx.data_engine.record_step(
+                                pre_screenshot_bytes=current_pre_screenshot_bytes,
+                                ui_tree=current_xml_list,
+                                action_taken={"action": "report_task_status", "args": args},
+                                operator_raw_thinking=raw_text,
+                                last_execution_result={
+                                    "result": "Task completed with final report."
+                                },
+                                extra_metadata={"token_usage": step_token_usage},
+                            )
+                        except Exception as step_err:
+                            logger.warning(f"Error recording final step in FlashRunner: {step_err}")
+                    messages.append(
+                        ToolMessage(
+                            tool_call_id=tc_id,
+                            name=name,
+                            content=json.dumps({"status": "acknowledged"}),
+                            status="success",
+                        )
+                    )
+                    if self.summarizer:
+                        await self.summarizer.flush()
+                    return final_report
+
+                try:
+                    exec_result = await self.executor.execute(name, args, tc_id, state)
+
+                    post_img_bytes = exec_result.screenshot_bytes
+                    if not post_img_bytes and name in ACTION_TOOL_NAMES:
+                        try:
+                            controller = UnifiedMobileController(self.ctx)
+                            screen_data = await controller.get_screen_data()
+                            post_img_bytes = base64.b64decode(screen_data.base64)
+                            if not exec_result.ui_elements_text:
+                                exec_result.ui_elements_text = screen_data.elements
+                        except Exception as shot_err:
+                            logger.warning(
+                                f"Failed to capture fallback screenshot in FlashRunner: {shot_err}"
+                            )
+
+                    # Record telemetry / step in DataEngine
+                    if self.ctx.data_engine and name in ACTION_TOOL_NAMES:
+                        try:
+                            if self.ctx.data_engine.current_step_id is None:
+                                self.ctx.data_engine.allocate_step_id()
+
+                            # Extract and enrich coordinate metadata
+                            norm_coords = None
+                            norm_start = None
+                            norm_end = None
+                            if name == "swipe":
+                                kind, target_val, _ = parse_swipe_parameters(args)
+                                if (
+                                    kind == "coords"
+                                    and isinstance(target_val, list)
+                                    and len(target_val) == 4
+                                ):
+                                    norm_coords = target_val
+                                    norm_start = target_val[:2]
+                                    norm_end = target_val[2:]
+                                elif kind == "direction" and isinstance(target_val, str):
+                                    g_lower = target_val.lower()
+                                    if "up" in g_lower:
+                                        norm_coords = [600, 800, 600, 200]
+                                    elif "down" in g_lower:
+                                        norm_coords = [600, 200, 600, 800]
+                                    elif "left" in g_lower:
+                                        norm_coords = [800, 500, 200, 500]
+                                    elif "right" in g_lower:
+                                        norm_coords = [200, 500, 800, 500]
+                                    if norm_coords:
+                                        norm_start = norm_coords[:2]
+                                        norm_end = norm_coords[2:]
+                            elif name in ("click", "tap", "long_press", "input_text"):
+                                target = args.get("target") or args.get("coordinates")
+                                if isinstance(target, (list, tuple)) and len(target) == 2:
+                                    norm_coords = list(target)
+                                elif isinstance(target, str):
+                                    nums = re.findall(r"-?\d+(?:\.\d+)?", target)
+                                    if len(nums) == 2:
+                                        norm_coords = [int(float(nums[0])), int(float(nums[1]))]
+
+                            action_dict = {
+                                "action": name,
+                                "coordinates": (
+                                    args.get("target")
+                                    or args.get("coordinates")
+                                    or args.get("sequence")
+                                    or norm_coords
+                                ),
+                                "args": args,
+                            }
+                            if norm_coords:
+                                action_dict["normalized_coordinates"] = norm_coords
+                            if norm_start and norm_end:
+                                action_dict["normalized_start_coordinates"] = norm_start
+                                action_dict["normalized_end_coordinates"] = norm_end
+
+                            self.ctx.data_engine.record_step(
+                                pre_screenshot_bytes=current_pre_screenshot_bytes,
+                                post_screenshot_bytes=post_img_bytes,
+                                ui_tree=(exec_result.ui_elements_text or current_xml_list),
+                                action_taken=action_dict,
+                                operator_raw_thinking=raw_text,
+                                last_execution_result={"result": exec_result.text_summary},
+                                extra_metadata={"token_usage": step_token_usage},
+                            )
+                        except Exception as step_err:
+                            logger.warning(f"Error recording step in FlashRunner: {step_err}")
+
+                    # ⚡ Non-blocking dispatch of objective visual transition summarizer
+                    if self.summarizer and name in ACTION_TOOL_NAMES:
+                        self.summarizer.dispatch(
+                            step_number=turns,
+                            action_name=name,
+                            action_args=args,
+                            pre_img_bytes=current_pre_screenshot_bytes,
+                            post_img_bytes=post_img_bytes,
+                            exec_outcome=exec_result.text_summary,
+                        )
+
+                    if post_img_bytes:
+                        current_pre_screenshot_bytes = post_img_bytes
+                    if exec_result.ui_elements_text:
+                        current_xml_list = exec_result.ui_elements_text
+
+                    messages.append(exec_result.to_langchain_tool_message())
+
+                except Exception as e:
+                    logger.error(f"Error executing tool {name}: {e}")
+                    messages.append(
+                        ToolMessage(
+                            tool_call_id=tc_id,
+                            name=name,
+                            content=f"Error executing tool {name}: {e}",
+                            status="error",
+                        )
+                    )
+
+        if self.summarizer:
+            await self.summarizer.flush()
+
+        return final_report or {
+            "status": "failed",
+            "explanation": "Max turns reached without final status report.",
+        }

artemis/agents/flash/summarizer.py

@@ -0,0 +1,226 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Asynchronous Visual Step Summarizer for Artemis Flash profile.
+
+Processes Before/After screenshot pairs and action metadata in the background
+to generate high-density, strictly objective visual state transition summaries.
+Ensures zero blocking latency for the main runner and includes out-of-order self-healing retries.
+"""
+
+import asyncio
+import base64
+from pathlib import Path
+from typing import Any
+
+from jinja2 import Template
+from langchain_core.messages import BaseMessage, HumanMessage, SystemMessage
+
+from artemis.context import ArtemisContext
+from artemis.services.llm import get_google_llm, get_llm
+from artemis.utils.logger import get_logger
+from artemis.utils.visualization import draw_action_overlay_on_image
+
+logger = get_logger(__name__)
+
+
+class VisualStepSummarizer:
+    """Non-blocking background worker generating objective visual step summaries.
+
+    Key design properties:
+    1. Zero-blocking dispatch: The main Flash runner dispatches and proceeds immediately.
+    2. Graceful fallback: If a step summary is not ready, it cleanly falls back to the original action text.
+    3. Self-healing re-trigger: If a previous step's summary is stalled or lost while subsequent steps
+       complete, it automatically re-dispatches the stalled step.
+    """
+
+    def __init__(self, ctx: ArtemisContext, model_name: str | None = None):
+        self.ctx = ctx
+        self._summaries: dict[int, str] = {}
+        self._pending_tasks: dict[int, asyncio.Task] = {}
+        self._step_inputs: dict[int, dict[str, Any]] = {}
+        self._retry_counts: dict[int, int] = {}
+        self._max_retries = 2
+
+        # Initialize lightweight VLM: prioritize explicit model_name
+        target_model = model_name or "gemini-2.5-flash-lite"
+        try:
+            if model_name:
+                self._llm = get_google_llm(model_name=target_model, temperature=0.0)
+            else:
+                self._llm = get_llm(ctx, name="summarizer", is_utils=True)
+        except Exception:
+            self._llm = get_google_llm(model_name=target_model, temperature=0.0)
+
+        # Load system prompt template
+        prompt_path = Path(__file__).parent / "flash_summarizer.md"
+        if prompt_path.exists():
+            self._prompt_template = prompt_path.read_text(encoding="utf-8")
+        else:
+            self._prompt_template = (
+                "You are the Step Summarizer for an Android UI automation agent.\n"
+                "Synthesize the physical action and visual delta between BEFORE and AFTER screens in exactly ONE "
+                "continuous first-person paragraph using 'I' (e.g., 'In Step {{ step_number }}, I tapped... and observed...').\n"
+                "Strictly avoid subjective validation words: successfully, completed, failed, achieved, navigated to."
+            )
+
+    def dispatch(
+        self,
+        step_number: int,
+        action_name: str,
+        action_args: dict[str, Any],
+        pre_img_bytes: bytes | None,
+        post_img_bytes: bytes | None,
+        exec_outcome: str,
+    ) -> None:
+        """Dispatches an asynchronous summarization task without blocking the caller."""
+        self._step_inputs[step_number] = {
+            "action_name": action_name,
+            "action_args": action_args,
+            "pre_img_bytes": pre_img_bytes,
+            "post_img_bytes": post_img_bytes,
+            "exec_outcome": exec_outcome,
+        }
+
+        task = asyncio.create_task(self._run_summary(step_number))
+        self._pending_tasks[step_number] = task
+
+        # Check for and re-trigger any previous stalled steps
+        self._check_and_retrigger_stalled_steps(current_step=step_number)
+
+    def _check_and_retrigger_stalled_steps(self, current_step: int) -> None:
+        """Re-dispatches any previous steps that remain unsummarized while later steps arrive."""
+        for past_step in range(1, current_step):
+            if past_step not in self._summaries and past_step in self._step_inputs:
+                task = self._pending_tasks.get(past_step)
+                retries = self._retry_counts.get(past_step, 0)
+                if (task is None or task.done()) and retries < self._max_retries:
+                    self._retry_counts[past_step] = retries + 1
+                    logger.info(
+                        f"VisualStepSummarizer: Retriggering stalled summary for Step {past_step} "
+                        f"(attempt {self._retry_counts[past_step] + 1})"
+                    )
+                    self._pending_tasks[past_step] = asyncio.create_task(
+                        self._run_summary(past_step)
+                    )
+
+    async def _run_summary(self, step_number: int) -> None:
+        """Executes the lightweight VLM call to summarize the step transition."""
+        input_data = self._step_inputs.get(step_number)
+        if not input_data:
+            return
+
+        action_name = input_data["action_name"]
+        action_args = input_data["action_args"]
+        pre_bytes = input_data["pre_img_bytes"]
+        post_bytes = input_data["post_img_bytes"]
+        exec_outcome = input_data["exec_outcome"]
+
+        try:
+            rendered_prompt = Template(self._prompt_template).render(step_number=step_number)
+            content_blocks: list[dict[str, Any]] = [
+                {
+                    "type": "text",
+                    "text": (
+                        f"Step {step_number} Physical Action: {action_name}({action_args})\n"
+                        f"Controller Outcome: {exec_outcome}"
+                    ),
+                }
+            ]
+
+            if pre_bytes:
+                # 🎨 Visually mark the exact action (tap ripple, sequence numbers, swipe arrow) on the BEFORE screenshot
+                annotated_pre_bytes = draw_action_overlay_on_image(
+                    image_bytes=pre_bytes,
+                    action_name=action_name,
+                    action_args=action_args,
+                )
+                b64_pre = base64.b64encode(annotated_pre_bytes).decode("utf-8")
+                content_blocks.append(
+                    {
+                        "type": "text",
+                        "text": "--- [1] BEFORE ACTION SCREEN (Action Marked Visually in Red) ---",
+                    }
+                )
+                content_blocks.append(
+                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64_pre}"}}
+                )
+
+            if post_bytes:
+                b64_post = base64.b64encode(post_bytes).decode("utf-8")
+                content_blocks.append({"type": "text", "text": "--- [2] AFTER ACTION SCREEN ---"})
+                content_blocks.append(
+                    {
+                        "type": "image_url",
+                        "image_url": {"url": f"data:image/jpeg;base64,{b64_post}"},
+                    }
+                )
+
+            messages: list[BaseMessage] = [
+                SystemMessage(content=rendered_prompt),
+                HumanMessage(content=content_blocks),
+            ]
+
+            response = await asyncio.wait_for(self._llm.ainvoke(messages), timeout=25.0)
+            summary_raw = response.content if isinstance(response.content, str) else ""
+            if isinstance(response.content, list):
+                summary_raw = "".join(
+                    b.get("text", "")
+                    for b in response.content
+                    if isinstance(b, dict) and "text" in b
+                )
+
+            summary_text = summary_raw.strip()
+            if summary_text:
+                self._summaries[step_number] = summary_text
+                logger.info(
+                    f"VisualStepSummarizer: Generated summary for Step {step_number}: {summary_text[:80]}..."
+                )
+
+                # Free binary image buffers from memory once summary is secured
+                if step_number in self._step_inputs:
+                    self._step_inputs[step_number]["pre_img_bytes"] = None
+                    self._step_inputs[step_number]["post_img_bytes"] = None
+
+                # Update DataEngine telemetry if active
+                if self.ctx.data_engine:
+                    try:
+                        self.ctx.data_engine.update_step_summary(step_number, summary_text)
+                    except Exception as de_err:
+                        logger.debug(f"DataEngine step summary update skipped: {de_err}")
+
+        except Exception as e:
+            logger.warning(
+                f"VisualStepSummarizer: Error generating summary for step {step_number}: {e}"
+            )
+            if self._retry_counts.get(step_number, 0) >= self._max_retries:
+                # Max retries reached, free memory buffer
+                if step_number in self._step_inputs:
+                    self._step_inputs[step_number]["pre_img_bytes"] = None
+                    self._step_inputs[step_number]["post_img_bytes"] = None
+
+    def get_summary(self, step_number: int, fallback_text: str | None = None) -> str | None:
+        """Retrieves summarized state for step_number, or fallback_text if not ready yet."""
+        return self._summaries.get(step_number, fallback_text)
+
+    def has_summary(self, step_number: int) -> bool:
+        """Checks if a step summary has completed."""
+        return step_number in self._summaries
+
+    async def flush(self) -> None:
+        """Flushes and awaits all pending background summary tasks upon completion."""
+        tasks = [t for t in self._pending_tasks.values() if not t.done()]
+        if tasks:
+            logger.info(f"VisualStepSummarizer: Flushing {len(tasks)} pending summary tasks...")
+            await asyncio.gather(*tasks, return_exceptions=True)

artemis/agents/history_analyzer/history_analyzer.md

@@ -0,0 +1,8 @@
+# IDENTITY
+You are a **History Analyzer**, expert in analyzing multi-agent session execution history to answer user queries.
+
+# MANDATE
+1. You are provided with the **Task Plan and Execution History** summarizing the high-level plan and history steps.
+2. For queries about specific step details (such as precise actions, coordinates, `operator_raw_thinking`, or `last_execution_result`), you should call the `get_step_details` tool.
+3. Never assume, guess, or invent details not visible in the Task Plan and Execution History. Always query `get_step_details` for specifics. If details are unavailable, state it clearly.
+4. Make good use of note tools to see what notes are available. When analyzing history records, open and cross-reference relevant notes to ensure you have grasped all necessary information.

artemis/agents/history_analyzer/history_analyzer.py

@@ -0,0 +1,250 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import json
+from pathlib import Path
+from typing import Any
+
+from langchain_core.messages import (
+    BaseMessage,
+    HumanMessage,
+    SystemMessage,
+    ToolMessage,
+)
+from langchain_core.tools import BaseTool, tool
+from artemis.context import ArtemisContext
+from artemis.data_engine.trace import CURRENT_TRACE_ID, TraceSpan, trace
+from artemis.services.llm import get_llm, invoke_llm_with_timeout_message
+from artemis.tools.scratchpad import (
+    get_list_notes_tool_pure,
+    get_read_note_tool_pure,
+)
+from artemis.utils.logger import get_logger
+from artemis.utils.notes import (
+    get_note_file_path,
+    get_notes_dir,
+)
+from artemis.utils.task_tree import build_plan_and_history, get_active_subgoal_hashes
+
+logger = get_logger(__name__)
+
+
+class HistoryAnalyzer:
+    def __init__(self, ctx: ArtemisContext):
+        self.ctx = ctx
+
+    def exec_get_step_details(self, start_step: int, end_step: int) -> str:
+        """Retrieve the detailed, full-granularity information for a range of steps (inclusive).
+
+        Use this when you need to inspect specific actions taken, operator
+        thinking, or execution results.
+        """
+        try:
+            s_step = int(start_step)
+            e_step = int(end_step)
+        except (ValueError, TypeError):
+            return (
+                f"Error: start_step and end_step must be integers, got {start_step} and {end_step}."
+            )
+
+        matched_steps = []
+        for s in getattr(self, "history_steps", []):
+            step_num = s.get("step_number")
+            if step_num is not None and s_step <= step_num <= e_step:
+                details = {
+                    "step_number": s.get("step_number"),
+                    "relative_time": s.get("relative_time"),
+                    "summary": s.get("summary"),
+                    "action_taken": s.get("action_taken"),
+                    "operator_raw_thinking": s.get("operator_raw_thinking"),
+                    "last_execution_result": s.get("last_execution_result"),
+                }
+                matched_steps.append(details)
+        if not matched_steps:
+            return f"No steps found in range [{s_step}, {e_step}]."
+        return json.dumps(matched_steps, indent=2, ensure_ascii=False)
+
+    def _get_step_details_tool(self, history_steps: list[dict[str, Any]]) -> BaseTool:
+        self.history_steps = history_steps
+
+        @tool
+        def get_step_details(start_step: int, end_step: int) -> str:
+            """Retrieve the detailed, full-granularity information for a range of steps (inclusive).
+
+            Use this when you need to inspect specific actions taken, operator
+            thinking, or execution results.
+            """
+            return self.exec_get_step_details(start_step, end_step)
+
+        return get_step_details
+
+    @trace(type="agent", name="history_analyzer")
+    async def run(self, query: str) -> str:
+        if not self.ctx.data_engine:
+            return "Error: DataEngine is not available to retrieve history."
+
+        # 1. Fetch all steps from Data Engine
+        self.history_steps = self.ctx.data_engine.get_agent_friendly_steps()
+        if not self.history_steps:
+            return "No history recorded for this session yet."
+
+        # 2. Build the plan and history or step list
+        plan_and_history = ""
+        try:
+            notes_dir = get_notes_dir(self.ctx.data_engine.base_dir)
+            current_plan = ""
+            current_path = get_note_file_path(self.ctx.data_engine.base_dir, "task_plan")
+            if current_path.exists():
+                current_plan = current_path.read_text(encoding="utf-8")
+
+            if current_plan:
+                active_subgoal_hash, _ = get_active_subgoal_hashes(current_plan)
+                plan_and_history = build_plan_and_history(
+                    current_plan,
+                    self.history_steps,
+                    active_subgoal_hash,
+                    last_n_detailed=1,
+                    min_summaries=len(self.history_steps),
+                )
+        except Exception as e:
+            logger.error(f"Failed to build plan and history in HistoryAnalyzer: {e}")
+
+        if not plan_and_history:
+            # Fallback to a simple list
+            plan_and_history = "\n".join(
+                [
+                    f"- Step {s.get('step_number')} ({s.get('relative_time')}):"
+                    f" {s.get('summary') or 'No summary'}"
+                    for s in self.history_steps
+                ]
+            )
+
+        # 3. Load system prompt
+        prompt_path = Path(__file__).parent.joinpath("history_analyzer.md")
+        if prompt_path.exists():
+            system_prompt_template = prompt_path.read_text(encoding="utf-8")
+        else:
+            system_prompt_template = (
+                "You are a History Analyzer. Your role is to analyze the"
+                " execution history of a session and answer user queries in"
+                " natural language.\nIf you need specific details for a range"
+                " of steps, use the `get_step_details` tool."
+            )
+
+        system_message_content = (
+            system_prompt_template + f"\n\n### Task Plan and Execution History:\n{plan_and_history}"
+        )
+
+        # 4. Prepare LLM and bind tools
+        llm = get_llm(ctx=self.ctx, name="history_analyzer")
+        get_step_details = self._get_step_details_tool(self.history_steps)
+
+        list_notes_tool = get_list_notes_tool_pure(self.ctx)
+        read_note_tool = get_read_note_tool_pure(self.ctx)
+        llm = llm.bind_tools(tools=[get_step_details, list_notes_tool, read_note_tool])
+
+        # 5. ReAct Loop
+        messages: list[BaseMessage] = [
+            SystemMessage(content=system_message_content),
+            HumanMessage(content=query),
+        ]
+
+        max_turns = 5
+        for turn in range(max_turns):
+            logger.info(f"HistoryAnalyzer ReAct turn {turn + 1}")
+
+            async def run_stream():
+                full_response = None
+                trace_id = CURRENT_TRACE_ID.get()
+                async for chunk in llm.astream(messages):
+                    if full_response is None:
+                        full_response = chunk
+                    else:
+                        full_response += chunk
+
+                    pass
+                return full_response
+
+            response = await invoke_llm_with_timeout_message(run_stream())
+            messages.append(response)
+
+            if not response.tool_calls:
+                return (
+                    response.content.strip()
+                    if isinstance(response.content, str)
+                    else str(response.content)
+                )
+
+            # Process tool calls
+            for tc in response.tool_calls:
+                tool_name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
+                args = tc["args"]
+
+                with TraceSpan(name=tool_name, ctx=self.ctx) as span:
+                    span.payload = {"args": args}
+                    if tool_name == "get_step_details":
+                        logger.info(
+                            f"HistoryAnalyzer executing tool get_step_details with args: {args}"
+                        )
+                        try:
+                            start_step = int(args.get("start_step", 0))
+                            end_step = int(args.get("end_step", 0))
+                            result = get_step_details.invoke(
+                                {"start_step": start_step, "end_step": end_step}
+                            )
+                            span.result = result
+                            status = "success"
+                        except Exception as e:
+                            span.status = "failed"
+                            span.error = str(e)
+                            result = f"Error running get_step_details: {e}"
+                            status = "error"
+                    elif tool_name == "list_notes":
+                        logger.info("HistoryAnalyzer executing tool list_notes")
+                        try:
+                            result = list_notes_tool.invoke({})
+                            span.result = result
+                            status = "success"
+                        except Exception as e:
+                            span.status = "failed"
+                            span.error = str(e)
+                            result = f"Error running list_notes: {e}"
+                            status = "error"
+                    elif tool_name == "read_note":
+                        logger.info(f"HistoryAnalyzer executing tool read_note with args: {args}")
+                        try:
+                            key = args.get("key", "")
+                            result = read_note_tool.invoke({"key": key})
+                            span.result = result
+                            status = "success"
+                        except Exception as e:
+                            span.status = "failed"
+                            span.error = str(e)
+                            result = f"Error running read_note: {e}"
+                            status = "error"
+                    else:
+                        result = f"Error: Tool {tool_name} is not supported."
+                        status = "error"
+                        span.status = "failed"
+                        span.error = result
+
+                messages.append(
+                    ToolMessage(
+                        tool_call_id=tc["id"],
+                        content=result,
+                        status=status,
+                    )
+                )
+
+        return "Error: HistoryAnalyzer failed to resolve the query within maximum turns."

artemis/agents/hopper/hopper.md

@@ -0,0 +1,15 @@
+## Hopper
+
+Extract relevant information from batch data. **Keep extracted data exactly as-is** - no reformatting.
+
+## Output
+- **found**: `true` if data was found, `false` otherwise
+- **output**: Extracted information if found, `null` otherwise
+- **reason**: Brief explanation of search logic
+
+## Rules
+1. **Search entire input** - may contain hundreds of entries
+2. **For app package lookup**: Match app name (or variations) in package identifier
+   - Common patterns: lowercase app name, company+app, brand name, codenames
+3. **Prefer direct matches** over partial matches
+4. **Return `null`** if not found or if multiple ambiguous matches exist - don't guess

artemis/agents/hopper/hopper.py

@@ -0,0 +1,93 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from pathlib import Path
+
+from jinja2 import Template
+from langchain_core.messages import HumanMessage, SystemMessage
+from artemis.context import ArtemisContext
+from artemis.data_engine.trace import trace
+from artemis.services.llm import get_llm, invoke_llm_with_timeout_message, with_fallback
+from artemis.utils.logger import get_logger
+from pydantic import BaseModel, Field
+
+logger = get_logger(__name__)
+
+
+class _CyFunctionDetectorMeta(type):
+    def __instancecheck__(self, instance):
+        name = type(instance).__name__
+        return (
+            name
+            in (
+                "cyfunction",
+                "cython_function_or_method",
+                "builtin_function_or_method",
+            )
+            or "cyfunction" in name.lower()
+        )
+
+
+class CyFunctionDetector(metaclass=_CyFunctionDetectorMeta):
+    pass
+
+
+class HopperOutput(BaseModel):
+    model_config = {"ignored_types": (CyFunctionDetector,)}
+    found: bool = Field(description="True if the requested data was found, False otherwise.")
+    output: str | None = Field(description="The extracted data if found, null otherwise.")
+    reason: str = Field(
+        description="A short explanation of what you looked for"
+        + " and how you decided what to extract."
+    )
+
+
+@trace(type="agent", name="hopper")
+async def hopper(
+    ctx: ArtemisContext,
+    request: str,
+    data: str,
+    use_fallback: bool = True,
+) -> HopperOutput:
+    logger.info(f"Starting Hopper Agent (use_fallback={use_fallback})")
+    system_message = Template(
+        Path(__file__).parent.joinpath("hopper.md").read_text(encoding="utf-8")
+    ).render()
+    messages = [
+        SystemMessage(content=system_message),
+        HumanMessage(content=f"{request}\nHere is the data you must dig:\n{data}"),
+    ]
+
+    llm = get_llm(ctx=ctx, name="hopper", is_utils=True).with_structured_output(HopperOutput)
+    try:
+        if use_fallback:
+            llm_fallback = get_llm(
+                ctx=ctx, name="hopper", is_utils=True, use_fallback=True
+            ).with_structured_output(HopperOutput)
+            response: HopperOutput = await with_fallback(
+                main_call=lambda: invoke_llm_with_timeout_message(llm.ainvoke(messages)),
+                fallback_call=lambda: invoke_llm_with_timeout_message(
+                    llm_fallback.ainvoke(messages)
+                ),
+            )  # type: ignore
+        else:
+            response: HopperOutput = await invoke_llm_with_timeout_message(llm.ainvoke(messages))
+        return response
+    except Exception as e:
+        logger.error(f"Hopper LLM invocation failed: {e}")
+        return HopperOutput(
+            found=False,
+            output=None,
+            reason=f"Failed due to LLM error: {str(e)[:500]}",
+        )

artemis/agents/image_processor/image_processor.md

@@ -0,0 +1,74 @@
+# IDENTITY
+You are the expert UI Image Processor, a coding agent that can write Python scripts to analyze and modify images (like color filtering, cropping, highlighting, coloring, or performing custom image processing).
+
+# CONTEXT
+Target Image ID: `img_0` (This is the original input screenshot)
+Instruction: {instruction}
+Intermediate Artifact Save Path: `{intermediate_artifact_save_path}`
+If you want to save intermediate steps to inspect them or load them later, you must call `canvas.save(final=False)`. Any intermediate images saved this way will be automatically loaded and fed back to you as visual context at the beginning of your next turn, and can be retrieved in subsequent runs by instantiating a new canvas: `ImageCanvas(image_id, '{intermediate_artifact_save_path}')`.
+
+
+# GOAL
+Your goal is to perform image analysis, filter, crop, draw annotations/markers, or modify the image to visually highlight or isolate specific features as requested. Write a Python script using OpenCV (`cv2`), `numpy`, or other standard libraries to analyze and modify the target image `img_0` according to the `Instruction`. The final output should be the modified image. Call `submit_result` tool after you have successfully completed the task.
+
+# CRITICAL BOUNDARIES FOR GEOMETRIC TRANSFORMATIONS
+1. **DO NOT** use numpy array slicing (e.g. `img[y:y+h, x:x+w]`) to crop the image.
+2. **DO NOT** use `cv2.resize()` to scale the image.
+Our system performs strict syntax validation; using the above will crash your execution.
+
+If you MUST crop or scale the image to isolate small details or improve OCR/detection downstream, you MUST use the pre-imported `ImageCanvas` tool exactly as follows:
+
+### Snippet 1: Saving & Resuming Intermediate Steps (Iterative Workflow)
+Use this to crop/zoom into specific areas and save them so you can visually inspect them in your next turn, or load them back to create new branches.
+```python
+from artemis.utils.cv_canvas import ImageCanvas
+
+# 1. Start from the original image:
+canvas = ImageCanvas("img_0", "{intermediate_artifact_save_path}")
+
+# 2. Crop and scale:
+canvas.crop(x=100, y=200, w=300, h=300)
+canvas.resize_by_factor(2.0)
+
+# 3. Save as an intermediate step for verification:
+# This will auto-generate 'img_1' and save it in the intermediate directory.
+# In the next turn, you will receive 'img_1' as visual confirmation.
+canvas.save(final=False)
+
+# --- In a subsequent turn, if you want to resume from 'img_1' ---
+# canvas = ImageCanvas('img_1', '{intermediate_artifact_save_path}')
+```
+
+### Snippet 2: Finalizing and Saving the Output
+Use this to draw your final annotations, highlights, or markers, and save the final result to be returned.
+```python
+from artemis.utils.cv_canvas import ImageCanvas
+
+# 1. Load the original or a cropped intermediate step to finalize:
+canvas = ImageCanvas("img_1", "{intermediate_artifact_save_path}")
+
+# 2. Draw annotations, highlight elements, or apply filters:
+# # Convert normalized coordinates (0-1000) to pixel coordinates:
+# x, y = canvas.normalized_to_pixel_coords(nx, ny)
+# Draw specific verification dots and register their labels with unique label_idx (e.g. V1):
+# canvas.draw_dot(100, 200, 1)
+
+# 3. Save the final result (this is required to complete the task):
+# This will auto-generate the next sequential image ID (e.g., 'img_2') and mark it as a final output.
+canvas.save(final=True)
+```
+
+### Snippet 3: Simple Modification Without Cropping or Scaling
+If you do NOT need to crop or scale, but just want to draw annotations or apply filters, you can still use `ImageCanvas` for consistency:
+```python
+from artemis.utils.cv_canvas import ImageCanvas
+
+canvas = ImageCanvas("img_0", "{intermediate_artifact_save_path}")
+# ... modify canvas.img ...
+canvas.save(final=True)
+```
+
+# EXECUTION LOOP
+1. Write and execute your script using the `execute_python` tool.
+2. Read the stdout/stderr. If there are errors, fix your code and execute again.
+3. Once you have successfully called `canvas.save(final=True)` on your final output image(s), call the `submit_result` tool with a summary to finish your job.

artemis/agents/image_processor/image_processor.py

@@ -0,0 +1,322 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Universal Multi-Model Image Processor for Artemis."""
+
+import base64
+import json
+from pathlib import Path
+import shutil
+import time
+import uuid
+
+from langchain_core.messages import (
+    BaseMessage,
+    HumanMessage,
+    SystemMessage,
+    ToolMessage,
+)
+
+from artemis.config import settings
+from artemis.context import ArtemisContext
+from artemis.core.tool_declaration import ToolDeclaration
+from artemis.data_engine.trace import TraceSpan, trace
+from artemis.services.llm import get_llm
+from artemis.utils.logger import get_logger
+from artemis.utils.python_executor import PythonExecutor
+
+logger = get_logger(__name__)
+
+EXECUTE_PYTHON_TOOL = ToolDeclaration(
+    name="execute_python",
+    description=(
+        "Executes the provided Python code within an isolated"
+        " Jupyter environment and returns the resulting stdout,"
+        " stderr, and traceback outputs."
+    ),
+    parameters={
+        "type": "object",
+        "properties": {
+            "code": {
+                "type": "string",
+                "description": "The Python code to execute",
+            }
+        },
+        "required": ["code"],
+    },
+)
+
+SUBMIT_RESULT_TOOL = ToolDeclaration(
+    name="submit_result",
+    description=("Submits a summary of the modifications to successfully complete the task."),
+    parameters={
+        "type": "object",
+        "properties": {
+            "summary": {
+                "type": "string",
+                "description": "Summary of what you found or changed.",
+            }
+        },
+        "required": ["summary"],
+    },
+)
+
+
+class ImageProcessor:
+    def __init__(self, ctx: ArtemisContext):
+        self.ctx = ctx
+
+    @trace(type="agent", name="image_processor")
+    async def run(self, instruction: str, target_image_path: str) -> dict:
+        try:
+            llm = get_llm(self.ctx, name="image_processor")
+        except Exception:
+            llm = get_llm(self.ctx, name="operator")
+
+        base_dir = settings.TRACES_PATH
+        image_processor_dir = base_dir / "images" / "image_processor"
+        image_processor_dir.mkdir(parents=True, exist_ok=True)
+        intermediate_artifact_save_path = image_processor_dir / f"intermediates_{int(time.time())}"
+        intermediate_artifact_save_path.mkdir(parents=True, exist_ok=True)
+        intermediate_artifact_save_path_str = str(intermediate_artifact_save_path.resolve())
+
+        max_iterations = 5
+
+        prompt_path = Path(__file__).parent / "image_processor.md"
+        prompt_template = prompt_path.read_text(encoding="utf-8")
+        prompt_template = (
+            prompt_template.replace("{target_image_path}", target_image_path)
+            .replace("{instruction}", instruction)
+            .replace(
+                "{intermediate_artifact_save_path}",
+                intermediate_artifact_save_path_str,
+            )
+            .replace("{max_iterations}", str(max_iterations))
+        )
+
+        tools_declaration = [EXECUTE_PYTHON_TOOL, SUBMIT_RESULT_TOOL]
+
+        with open(target_image_path, "rb") as f:
+            img_bytes = f.read()
+        img_b64 = base64.b64encode(img_bytes).decode("utf-8")
+
+        messages: list[BaseMessage] = [
+            SystemMessage(content=prompt_template),
+            HumanMessage(
+                content=[
+                    {"type": "text", "text": "Here is the target image. Begin writing your code."},
+                    {
+                        "type": "image_url",
+                        "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"},
+                    },
+                ]
+            ),
+        ]
+
+        img_0_path = intermediate_artifact_save_path / "img_0.jpg"
+        try:
+            shutil.copy2(target_image_path, img_0_path)
+        except Exception as e:
+            logger.error(f"Failed to copy original screenshot to intermediate pool: {e}")
+            img_0_path = Path(target_image_path)
+
+        self.intermediate_image_pool = {
+            "img_0": {
+                "image_id": "img_0",
+                "path": str(img_0_path.resolve()),
+                "turn_id": 0,
+                "transform": {
+                    "offset_x": 0.0,
+                    "offset_y": 0.0,
+                    "scale_x": 1.0,
+                    "scale_y": 1.0,
+                },
+                "annotations": {},
+            }
+        }
+
+        pool_json_path = intermediate_artifact_save_path / "intermediate_transforms.json"
+        sync_pool = {
+            "img_0": {
+                "image_id": "img_0",
+                "path": self.intermediate_image_pool["img_0"]["path"],
+                "transform": self.intermediate_image_pool["img_0"]["transform"],
+                "annotations": {},
+            }
+        }
+        with open(pool_json_path, "w") as f:
+            json.dump(sync_pool, f)
+
+        executor = PythonExecutor(session_dir=image_processor_dir)
+        iterations = 0
+        final_result = None
+
+        try:
+            while iterations < max_iterations:
+                iterations += 1
+                logger.info(f"ImageProcessor Iteration {iterations}")
+
+                # Inject newly created intermediate images from last turn
+                if self.intermediate_image_pool:
+                    image_blocks = []
+                    last_turn_id = iterations - 1
+                    for img_id, entry in self.intermediate_image_pool.items():
+                        if entry["turn_id"] == last_turn_id:
+                            try:
+                                path = Path(entry["path"])
+                                if path.exists():
+                                    b64_data = base64.b64encode(path.read_bytes()).decode("utf-8")
+                                    is_output = entry.get("is_output", False)
+                                    label = (
+                                        f"img_id: {img_id} (Output)"
+                                        if is_output
+                                        else f"img_id: {img_id}"
+                                    )
+                                    image_blocks.append(
+                                        {
+                                            "type": "image_url",
+                                            "image_url": {
+                                                "url": f"data:image/jpeg;base64,{b64_data}"
+                                            },
+                                        }
+                                    )
+                                    image_blocks.append({"type": "text", "text": f"\n{label}\n"})
+                            except Exception as e:
+                                logger.error(
+                                    f"Failed to read intermediate image {entry['path']}: {e}"
+                                )
+
+                    if image_blocks:
+                        if (
+                            messages
+                            and isinstance(messages[-1], HumanMessage)
+                            and isinstance(messages[-1].content, list)
+                        ):
+                            messages[-1].content.extend(image_blocks)
+                        else:
+                            messages.append(HumanMessage(content=image_blocks))
+
+                if iterations == max_iterations - 1:
+                    warning_msg = (
+                        "\n[WARNING] This is your next-to-last iteration. You"
+                        " must complete your task and call `submit_result` now"
+                        " or in the next turn, as no further code execution"
+                        " will be allowed after that."
+                    )
+                    messages.append(HumanMessage(content=warning_msg))
+                elif iterations == max_iterations:
+                    warning_msg = (
+                        "\n[WARNING] This is your final iteration. You MUST"
+                        " call `submit_result` now to complete the task. No"
+                        " other tools are available."
+                    )
+                    messages.append(HumanMessage(content=warning_msg))
+
+                current_tools = tools_declaration
+                if iterations == max_iterations:
+                    current_tools = [t for t in tools_declaration if t.name == "submit_result"]
+
+                bound_llm = llm.bind_tools(current_tools)
+
+                with TraceSpan(name="image_processor_call", ctx=self.ctx):
+                    response = await bound_llm.ainvoke(messages)
+
+                tool_calls = response.tool_calls or []
+                if not tool_calls:
+                    break
+
+                messages.append(response)
+
+                submit_called = False
+                for tc in tool_calls:
+                    name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
+                    args = tc.get("args") or {}
+                    tc_id = tc.get("id") or str(uuid.uuid4())
+
+                    if name == "submit_result":
+                        final_result = args
+                        submit_called = True
+                        messages.append(
+                            ToolMessage(
+                                tool_call_id=tc_id,
+                                name=name,
+                                content=json.dumps({"status": "submitted"}),
+                                status="success",
+                            )
+                        )
+                        break
+
+                    elif name == "execute_python":
+                        code = args.get("code", "")
+                        logger.info("Executing Python code via PythonExecutor")
+                        exec_output = executor.execute(code)
+
+                        messages.append(
+                            ToolMessage(
+                                tool_call_id=tc_id,
+                                name=name,
+                                content=exec_output,
+                                status="success",
+                            )
+                        )
+
+                        if pool_json_path.exists():
+                            try:
+                                with open(pool_json_path) as f:
+                                    loaded_pool = json.load(f)
+                                for img_id, entry in loaded_pool.items():
+                                    if img_id not in self.intermediate_image_pool:
+                                        self.intermediate_image_pool[img_id] = {
+                                            "image_id": img_id,
+                                            "path": entry["path"],
+                                            "turn_id": iterations,
+                                            "transform": entry["transform"],
+                                            "is_output": entry.get("is_output", False),
+                                            "annotations": entry.get("annotations", {}),
+                                        }
+                            except Exception as e:
+                                logger.error(f"Failed to read pool_json_path: {e}")
+
+                if submit_called:
+                    break
+
+        finally:
+            executor.close()
+
+        outputs = [
+            entry
+            for entry in self.intermediate_image_pool.values()
+            if entry.get("is_output") is True
+        ]
+
+        if not outputs:
+            sorted_keys = sorted(
+                self.intermediate_image_pool.keys(),
+                key=lambda k: (
+                    int(k.split("_")[1])
+                    if k.startswith("img_") and k.split("_")[1].isdigit()
+                    else -1
+                ),
+            )
+            if sorted_keys:
+                last_key = sorted_keys[-1]
+                outputs = [self.intermediate_image_pool[last_key]]
+
+        summary = (
+            final_result.get("summary", "")
+            if final_result
+            else "Failed to submit result within iterations"
+        )
+
+        return {"outputs": outputs, "summary": summary}

artemis/agents/log_analyzer/log_analyzer.md

@@ -0,0 +1,15 @@
+# Log Analysis Instructions
+
+You are the Log Analyzer Agent, responsible for diagnosing issues by querying Android device logs. 
+Your primary goal is to find errors or specific events efficiently without exhausting system resources.
+
+## Core Strategy: Separation of Search and Read (Grep-like Behavior)
+You MUST avoid reading massive chunks of logs. Instead, use a "Grep-then-Read" approach:
+1. **Precision Search (Grep)**: Always start by using `search_logs` with `context_lines=0`. Search for specific keywords, package names (e.g., `net.osmand`), or Exception tags. This will return the exact line numbers and timestamps of the events.
+2. **Targeted Reading (Read)**: Once you find a highly relevant match from your search (e.g., an error stack trace start), use `read_logs` and specify the `since_time` and `until_time` to fetch ONLY the small 20-50 line window around that exact event. 
+
+## Guidelines
+1. **Efficiency**: Never call `search_logs` with `context_lines` > 5 unless you are absolutely sure there are very few matches. Large context windows will cause the system to truncate your output and you will lose data.
+2. **Unknown Errors**: Use `google_search` for unfamiliar error codes or system tags.
+3. **Handling Missing Data**: If you have executed several searches and cannot find any matching logs, DO NOT loop indefinitely. Return a final summary stating that no logs matching the criteria were found.
+4. **Response Format**: You should provide your final log analysis in one paragraph. You can organize it into clear, concise parts (e.g., Search Results, Analysis, Key Events, and more). You should include all helpful technical details (e.g. original logs, timestamps, and more).

artemis/agents/log_analyzer/log_analyzer.py

@@ -0,0 +1,431 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import asyncio
+import json
+from pathlib import Path
+from typing import Annotated, Any
+from langchain_core.messages import HumanMessage, SystemMessage, ToolMessage
+from langchain_core.tools import BaseTool, StructuredTool
+from pydantic import BaseModel, Field
+
+
+class _CyFunctionDetectorMeta(type):
+    def __instancecheck__(self, instance):
+        name = type(instance).__name__
+        return (
+            name
+            in (
+                "cyfunction",
+                "cython_function_or_method",
+                "builtin_function_or_method",
+            )
+            or "cyfunction" in name.lower()
+        )
+
+
+class CyFunctionDetector(metaclass=_CyFunctionDetectorMeta):
+    pass
+
+
+class SpawnLogReaderArgs(BaseModel):
+    model_config = {"ignored_types": (CyFunctionDetector,)}
+    specific_query: str = Field(
+        ...,
+        description=(
+            "Specific instructions for the sub-agent, including what to look for and analyze."
+        ),
+    )
+
+
+from langgraph.prebuilt import InjectedState
+
+from artemis.context import ArtemisContext
+from artemis.data_engine.trace import (
+    CURRENT_TRACE_ID,
+    TraceSpan,
+    trace,
+)
+from artemis.graph.state import State
+from artemis.services.llm import get_llm, invoke_llm_with_timeout_message
+from artemis.tools.index import get_tool_by_name
+from artemis.tools.mobile.log_utils import fetch_and_filter_logs
+from artemis.tools.mobile.read_logs import get_read_logs_tool
+from artemis.tools.mobile.search_logs import (
+    get_search_logs_tool,
+    search_and_merge_logs,
+)
+from artemis.tools.tool_wrapper import (
+    get_tool_result_content,
+    invoke_tool_with_injection,
+)
+from artemis.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+class LogAnalyzerNode:
+    def __init__(self, ctx: ArtemisContext):
+        self.ctx = ctx
+        self._log_cache = {}
+        self._raw_log_cache = {}
+
+    def _wrap_with_caching(self, tool: BaseTool) -> BaseTool:
+        """Wraps a tool with caching logic based on tool name and arguments."""
+        original_coroutine = tool.coroutine
+
+        async def cached_coroutine(**kwargs):
+            if tool.name == "read_logs":
+                # Intercept read_logs to cache raw logs
+                lines = kwargs.get("lines", 200)
+                # Since we might have since_time/until_time, let's keep it simple or key by all.
+                # The instruction says: "cache the returned string based on the lines parameter."
+                # We can also include since_time and until_time in the cache key to be safe, but let's follow the lines-based requirement.
+                cache_key = (
+                    lines,
+                    kwargs.get("since_time"),
+                    kwargs.get("until_time"),
+                )
+                if cache_key in self._raw_log_cache:
+                    logger.info(f"Raw log cache hit for read_logs with key {cache_key}")
+                    return self._raw_log_cache[cache_key]
+
+                result = await original_coroutine(**kwargs)
+                if isinstance(result, str) and not result.startswith("Failed to read logs"):
+                    self._raw_log_cache[cache_key] = result
+                return result
+
+            elif tool.name == "search_logs":
+                lines = kwargs.get("lines", 10000)
+                since_time = kwargs.get("since_time")
+                until_time = kwargs.get("until_time")
+                raw_cache_key = (lines, since_time, until_time)
+
+                # Check if we have the raw logs cached
+                if raw_cache_key in self._raw_log_cache:
+                    logger.info(
+                        "Raw log cache hit for search_logs with key"
+                        f" {raw_cache_key}. Running local search."
+                    )
+                    raw_logs = self._raw_log_cache[raw_cache_key]
+                    # Run search_and_merge_logs locally
+
+                    try:
+                        return search_and_merge_logs(
+                            logs=raw_logs,
+                            keyword=kwargs.get("keyword"),
+                            context_lines=kwargs.get("context_lines", 0),
+                            is_regex=kwargs.get("is_regex", False),
+                        )
+                    except Exception as e:
+                        logger.error(f"Failed to search logs locally: {e}")
+                        return f"Error searching logs: {e}"
+
+                # If not cached, fetch them once via read_logs (or the original tool, which will fetch them)
+                # Note: search_logs itself fetches and filters. We can let the original_coroutine run,
+                # but to cache the raw logs, we need the raw logs.
+                # Wait, search_logs returns the SEARCHED results, not the raw logs.
+                # So if we just run original_coroutine, we don't get the raw logs to cache them!
+                # To cache them, we should fetch them using get_read_logs_tool or fetch_and_filter_logs,
+                # cache them, and then run search_and_merge_logs.
+                logger.info(
+                    "Raw log cache miss for search_logs with key"
+                    f" {raw_cache_key}. Fetching raw logs first."
+                )
+
+                try:
+                    raw_logs = fetch_and_filter_logs(
+                        ctx=self.ctx,
+                        lines=lines,
+                        since_time=since_time,
+                        until_time=until_time,
+                    )
+                    self._raw_log_cache[raw_cache_key] = raw_logs
+
+                    return search_and_merge_logs(
+                        logs=raw_logs,
+                        keyword=kwargs.get("keyword"),
+                        context_lines=kwargs.get("context_lines", 0),
+                        is_regex=kwargs.get("is_regex", False),
+                    )
+                except Exception as e:
+                    logger.error(f"Failed to fetch and search logs: {e}")
+                    return f"Error searching logs: {e}"
+
+            # Fallback to standard tool caching for other tools
+            serialized_args = json.dumps(kwargs, sort_keys=True)
+            cache_key = f"{tool.name}:{serialized_args}"
+            if cache_key in self._log_cache:
+                logger.info(f"Cache hit for tool {tool.name} with args {kwargs}")
+                return self._log_cache[cache_key]
+
+            logger.info(
+                f"Cache miss for tool {tool.name} with args {kwargs}. Invoking original tool."
+            )
+            result = await original_coroutine(**kwargs)
+            self._log_cache[cache_key] = result
+            return result
+
+        return StructuredTool.from_function(
+            coroutine=cached_coroutine,
+            name=tool.name,
+            description=tool.description,
+            args_schema=tool.args_schema,
+        )
+
+    async def _run_agent_loop(
+        self,
+        base_llm: Any,
+        tools_for_binding: list[Any],
+        tools: list[BaseTool],
+        current_messages: list[Any],
+        state: State,
+        max_iterations: int = 10,
+        agent_name: str = "Agent",
+    ) -> str:
+        outcome = "No result"
+
+        for i in range(max_iterations):
+            if i == max_iterations - 1:
+                current_messages.append(
+                    HumanMessage(
+                        content=(
+                            "[WARNING] This is your final iteration; all tools"
+                            " are stripped, and you must provide your final"
+                            " answer/summary directly."
+                        )
+                    )
+                )
+                llm = base_llm.bind_tools(tools=[])
+            else:
+                if i > 0:
+                    current_messages.append(
+                        HumanMessage(
+                            content=(
+                                "[WARNING] You have not completed the analysis"
+                                f" yet (iteration {i + 1} of {max_iterations})."
+                            )
+                        )
+                    )
+                llm = base_llm.bind_tools(tools=tools_for_binding)
+
+            async def run_stream():
+                full_response = None
+                trace_id = CURRENT_TRACE_ID.get()
+                async for chunk in llm.astream(current_messages):
+                    if full_response is None:
+                        full_response = chunk
+                    else:
+                        full_response += chunk
+
+                    pass
+                return full_response
+
+            response = await invoke_llm_with_timeout_message(run_stream())
+
+            if response.content:
+                outcome = response.content
+
+            if not response.tool_calls:
+                break
+
+            if i == max_iterations - 1:
+                tool_names = ", ".join(tc["name"] for tc in response.tool_calls)
+                warning = (
+                    f"\n\n[Warning: {agent_name} reached the maximum iteration"
+                    f" limit of {max_iterations} and could not execute tool(s):"
+                    f" {tool_names}]"
+                )
+                if outcome == "No result":
+                    outcome = (
+                        f"{agent_name} reached the maximum iteration limit of"
+                        f" {max_iterations} while attempting to call tools:"
+                        f" {tool_names}"
+                    )
+                else:
+                    outcome += warning
+                break
+
+            current_messages.append(response)
+
+            # Handle tool calls in parallel
+            async def run_tool(tc) -> ToolMessage:
+                tool_name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
+                logger.info(f"{agent_name} requested tool: {tool_name}")
+
+                result = ""
+                status = "success"
+                try:
+                    if ":" in tool_name:
+                        tool_to_run = get_tool_by_name(tool_name, tools)
+                    else:
+                        tool_to_run = next((t for t in tools if t.name == tool_name), None)
+                    if tool_to_run:
+                        args = dict(tc["args"])
+                        with TraceSpan(name=tool_name, ctx=self.ctx) as span:
+                            span.payload = {"args": args}
+                            try:
+                                result_obj = await invoke_tool_with_injection(
+                                    tool=tool_to_run,
+                                    args=args,
+                                    tool_call_id=tc["id"],
+                                    state=state,
+                                )
+                                result = get_tool_result_content(result_obj)
+                                if isinstance(result, list):
+                                    result = "\n".join(map(str, result))
+                                elif not isinstance(result, str):
+                                    result = str(result)
+                                span.result = result
+                                if result.startswith("Error"):
+                                    status = "error"
+                                    span.status = "failed"
+                                    span.error = result
+                            except Exception as e:
+                                status = "error"
+                                span.status = "failed"
+                                span.error = str(e)
+                                raise e
+                    else:
+                        result = f"Error: Tool {tool_name} not supported"
+                        status = "error"
+                except Exception as e:
+                    logger.error(f"Error running tool {tool_name}: {e}")
+                    result = f"Error running tool {tool_name}: {e}"
+                    status = "error"
+
+                return ToolMessage(
+                    tool_call_id=tc["id"],
+                    content=result,
+                    status=status,
+                )
+
+            active_tool_calls = [
+                tc
+                for tc in response.tool_calls
+                if (tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"])
+                != "google_search"
+            ]
+            tool_outputs = await asyncio.gather(*(run_tool(tc) for tc in active_tool_calls))
+
+            for tm in tool_outputs:
+                current_messages.append(tm)
+
+        return outcome
+
+    @trace(type="agent", name="log_analyzer")
+    async def run(self, prompt: str, state: State) -> str:
+        """Runs the main Log Analyzer and returns its summary."""
+        prompt_path = Path(__file__).parent.joinpath("log_analyzer.md")
+        if not prompt_path.exists():
+            raise FileNotFoundError(f"System prompt file not found at {prompt_path}")
+
+        system_prompt = prompt_path.read_text(encoding="utf-8")
+
+        llm = get_llm(ctx=self.ctx, name="log_analyzer")
+
+        all_tools = [
+            self._wrap_with_caching(get_read_logs_tool(self.ctx)),
+            self._wrap_with_caching(get_search_logs_tool(self.ctx)),
+            self._get_spawn_log_reader_tool(),
+        ]
+
+        current_messages = [
+            SystemMessage(content=system_prompt),
+            HumanMessage(content=prompt),
+        ]
+
+        return await self._run_agent_loop(
+            base_llm=llm,
+            tools_for_binding=all_tools,
+            tools=all_tools,
+            current_messages=current_messages,
+            state=state,
+            max_iterations=10,
+            agent_name="Log Analyzer",
+        )
+
+    @trace(type="agent", name="log_reader_sub_agent")
+    async def _run_log_reader(self, specific_query: str, state: State) -> str:
+        """Runs the Log Reader Sub-Agent and returns its summary."""
+        sub_prompt_path = Path(__file__).parent.joinpath("log_reader_sub_agent.md")
+        if not sub_prompt_path.exists():
+            sub_prompt_path.write_text(
+                "# Log Reader Sub-Agent\nYou are a specialized agent for"
+                " reading and analyzing Android logs.\nYour job is to fulfill"
+                " the specific prompt given by the Main Agent.\nYou have access"
+                " to tools: `read_logs`.\nAnalyze the logs carefully and"
+                " provide a clear, structured summary of your findings.\n",
+                encoding="utf-8",
+            )
+
+        sub_system_prompt = sub_prompt_path.read_text(encoding="utf-8")
+
+        llm = get_llm(ctx=self.ctx, name="log_reader_sub_agent")
+
+        sub_tools = [self._wrap_with_caching(get_read_logs_tool(self.ctx))]
+
+        # Extract the list of keys (metadata) currently stored in _log_cache.
+        cache_keys = list(self._log_cache.keys())
+
+        current_messages = [
+            SystemMessage(content=sub_system_prompt),
+        ]
+
+        # Append a system message listing the available cache keys so the sub-agent knows what data can be retrieved instantly.
+        if cache_keys:
+            current_messages.append(
+                SystemMessage(
+                    content=(
+                        "The following log reads are already cached and can be"
+                        " retrieved instantly without cost:\n"
+                    )
+                    + "\n".join(f"- {k}" for k in cache_keys)
+                )
+            )
+
+        current_messages.append(HumanMessage(content=specific_query))
+
+        return await self._run_agent_loop(
+            base_llm=llm,
+            tools_for_binding=sub_tools,
+            tools=sub_tools,
+            current_messages=current_messages,
+            state=state,
+            max_iterations=10,
+            agent_name="Log Reader Sub-Agent",
+        )
+
+    def _get_spawn_log_reader_tool(self) -> BaseTool:
+
+        async def spawn_log_reader(
+            specific_query: str,
+            state: Annotated[State, InjectedState] = None,
+        ) -> str:
+            """[SPAWN SUB-AGENT] Delegates complex log analysis to a sub-agent.
+
+            - Use when: Task requires deep analysis, cross-referencing, or
+            reading huge log volumes. - Note: The sub-agent has no search tools
+            and can only read raw logs. - Input: specific_query (str):
+            Instructions for the sub-agent. - Output: A structured summary
+            report from the sub-agent.
+            """
+            logger.info(f"Spawning Log Reader Sub-Agent with query: '{specific_query}'")
+            return await self._run_log_reader(specific_query, state)
+
+        return StructuredTool.from_function(
+            coroutine=spawn_log_reader,
+            name="spawn_log_reader",
+            args_schema=SpawnLogReaderArgs,
+        )

artemis/agents/log_analyzer/log_reader_sub_agent.md

@@ -0,0 +1,8 @@
+# Log Deep Analysis Instructions
+
+Fulfill the specific log analysis prompt by reading and synthesizing Android logs.
+
+## Guidelines
+1. **Direct Answer**: Address the exact questions or patterns requested in the prompt.
+2. **Synthesis**: Do not output raw, unsorted logs. Provide a structured summary of findings.
+3. **Precision**: Include relevant timestamps, line numbers, and error codes to support your conclusions.

artemis/agents/log_analyzer/output_analyzer.md

@@ -0,0 +1,11 @@
+# Task Output Analyzer
+
+You are a specialized agent for analyzing the stdout/stderr output of a command executed on an Android device.
+The user (another agent) will ask you a question about the command's execution output.
+Analyze the output carefully to answer their question.
+
+You have access to the following tools to read and search the full output:
+- `read_task_output`: Reads a range of lines from the output.
+- `search_task_output`: Searches the output for a keyword or regex.
+
+Be concise, accurate, and direct in your answer.

artemis/agents/log_analyzer/output_analyzer.py

@@ -0,0 +1,162 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from pathlib import Path
+from typing import Annotated
+from langchain_core.messages import HumanMessage, SystemMessage, ToolMessage
+from langchain_core.tools import tool
+from artemis.context import ArtemisContext
+from artemis.services.llm import get_llm, invoke_llm_with_timeout_message
+from artemis.tools.mobile.search_logs import search_and_merge_logs
+from artemis.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+class TaskOutputAnalyzerNode:
+    """Agent that analyzes large command/task execution outputs using LLM and tools."""
+
+    def __init__(self, ctx: ArtemisContext):
+        self.ctx = ctx
+
+    async def run(
+        self,
+        command: str,
+        output_text: str,
+        query: str,
+    ) -> str:
+        prompt_path = Path(__file__).parent.joinpath("output_analyzer.md")
+        if not prompt_path.exists():
+            raise FileNotFoundError(f"System prompt file not found at {prompt_path}")
+
+        system_prompt = prompt_path.read_text(encoding="utf-8")
+
+        # Get the LLM for output_analyzer (falls back to log_analyzer)
+        llm = get_llm(ctx=self.ctx, name="output_analyzer")
+
+        # Local tools created with closure on output_text
+        @tool
+        def read_task_output(
+            start_line: Annotated[int, "The 1-indexed line number to start reading from."],
+            end_line: Annotated[int, "The 1-indexed line number to stop reading at (inclusive)."],
+        ) -> str:
+            """Reads a specific range of lines from the task output.
+
+            Use this tool when you know the line number range to read a specific
+            portion of the output.
+            """
+            lines = output_text.splitlines()
+            if not lines:
+                return "Task output is empty."
+            start = max(1, start_line)
+            end = min(len(lines), end_line)
+            if start > end:
+                return (
+                    f"Error: start_line {start_line} is greater than end_line"
+                    f" {end_line} or task output length {len(lines)}."
+                )
+            segment = lines[start - 1 : end]
+            return "\n".join(segment)
+
+        @tool
+        def search_task_output(
+            keyword: Annotated[
+                str,
+                "The keyword or regular expression to search for in the output.",
+            ],
+            is_regex: Annotated[
+                bool,
+                "Whether to treat the keyword as a regular expression. Default is False.",
+            ] = False,
+            context_lines: Annotated[
+                int,
+                "Number of context lines to include before and after the"
+                " matched line. Default is 5.",
+            ] = 5,
+        ) -> str:
+            """Searches the task output for a keyword or regular expression, returning matching lines and context.
+
+            Use this tool to quickly locate error messages, warnings, or
+            specific key-value pairs in large outputs.
+            """
+            return search_and_merge_logs(
+                logs=output_text,
+                keyword=keyword,
+                context_lines=context_lines,
+                is_regex=is_regex,
+            )
+
+        tools = [read_task_output, search_task_output]
+        llm = llm.bind_tools(tools=tools)
+
+        # Optimize prompt context with preview if output is large
+        lines = output_text.splitlines()
+        if len(lines) <= 400:
+            log_preview = f"Full Output:\n{output_text}"
+        else:
+            first_part = "\n".join(lines[:200])
+            last_part = "\n".join(lines[-200:])
+            log_preview = (
+                f"Note: Output is very large ({len(lines)} lines). Here is a"
+                " preview of the start and end of the output:\n\n--- START OF"
+                f" OUTPUT ---\n{first_part}\n--- TRUNCATED {len(lines) - 400}"
+                f" LINES ---\n--- END OF OUTPUT ---\n{last_part}\n\nYou can use"
+                " `read_task_output` or `search_task_output` to fetch any"
+                " lines in the middle if needed."
+            )
+
+        user_content = f"Command Executed: {command}\n\n{log_preview}\n\nQuestion: {query}"
+
+        current_messages = [
+            SystemMessage(content=system_prompt),
+            HumanMessage(content=user_content),
+        ]
+
+        max_iterations = 5
+        outcome = "Failed to analyze output."
+
+        for _ in range(max_iterations):
+            response = await invoke_llm_with_timeout_message(llm.ainvoke(current_messages))
+
+            if response.content:
+                outcome = response.content
+
+            if not response.tool_calls:
+                break
+
+            current_messages.append(response)
+
+            for tc in response.tool_calls:
+                tool_name = tc["name"]
+                args = tc["args"]
+                logger.info(f"Output Analyzer requested tool: {tool_name}")
+
+                tool_to_run = next((t for t in tools if t.name == tool_name), None)
+                if tool_to_run:
+                    try:
+                        result = tool_to_run.invoke(args)
+                    except Exception as e:
+                        result = f"Failed to run tool {tool_name}: {e}"
+                else:
+                    result = f"Error: Tool {tool_name} not found."
+
+                current_messages.append(
+                    ToolMessage(
+                        tool_call_id=tc["id"],
+                        content=str(result),
+                        status="success" if not str(result).startswith("Error") else "error",
+                    )
+                )
+
+        return outcome

artemis/agents/object_detector/object_detector.json

@@ -0,0 +1,9 @@
+{
+  "templates": [
+    "Point to the following objects in the provided image: {labels_str}.",
+    "Detect the location of the following objects in the image: {labels_str}.",
+    "Locate the following UI components on the screen: {labels_str}."
+  ],
+  "global_timeout": 30.0,
+  "instructions": "Return a JSON list containing the detected objects. Each object must have a 'point' array [y, x] (normalized to 0-1000) and a 'label' string.\n\nExample output:\n[\n  {\"point\": [500, 250], \"label\": \"search icon\"}\n]\n\nIf no objects are found, return an empty list []."
+}

artemis/agents/object_detector/object_detector.py

@@ -0,0 +1,228 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Universal Multi-Model Object Detector for Artemis."""
+
+import asyncio
+import base64
+import json
+import os
+from pathlib import Path
+import re
+
+from langchain_core.messages import HumanMessage, ToolMessage
+from langgraph.types import Command
+
+from artemis.constants import VALIDATOR_MESSAGES_KEY
+from artemis.services.llm import get_llm
+from artemis.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+_DETECTOR_SEMAPHORE = asyncio.Semaphore(6)
+
+
+async def _detect_single_label(
+    llm,
+    ctx,
+    image_bytes,
+    mime_type,
+    label,
+    templates,
+    timeout_val=None,
+) -> list[dict]:
+    if timeout_val is None:
+        timeout_val = float(os.environ.get("OBJECT_DETECTOR_TIMEOUT", "10.0"))
+
+    img_b64 = base64.b64encode(image_bytes).decode("utf-8")
+
+    for i, template in enumerate(templates):
+        prompt = template.replace("{labels_str}", label)
+        messages = [
+            HumanMessage(
+                content=[
+                    {
+                        "type": "image_url",
+                        "image_url": {"url": f"data:{mime_type};base64,{img_b64}"},
+                    },
+                    {"type": "text", "text": prompt},
+                ]
+            )
+        ]
+
+        try:
+            logger.info(f"Calling Object Detector for '{label}' (Attempt {i + 1})...")
+            async with _DETECTOR_SEMAPHORE:
+                response = await asyncio.wait_for(
+                    llm.ainvoke(messages),
+                    timeout=timeout_val,
+                )
+
+            output = response.content if isinstance(response.content, str) else ""
+            if isinstance(response.content, list):
+                output = "".join(
+                    b.get("text", "")
+                    for b in response.content
+                    if isinstance(b, dict) and "text" in b
+                )
+            logger.info(f"Received response for '{label}' (Attempt {i + 1})")
+
+            cleaned_initial = output.strip()
+            code_block_match = re.search(r"```(?:json)?\s*([\s\S]*?)\s*```", cleaned_initial)
+            if code_block_match:
+                cleaned_initial = code_block_match.group(1).strip()
+
+            try:
+                res_json = json.loads(cleaned_initial)
+                if isinstance(res_json, list):
+                    valid_results = []
+                    for item in res_json:
+                        if isinstance(item, dict):
+                            item["label"] = label
+                            valid_results.append(item)
+                    return valid_results
+            except json.JSONDecodeError:
+                pass
+
+        except Exception as e:
+            logger.warning(f"Detection attempt {i + 1} failed for '{label}': {e}")
+
+    return []
+
+
+async def _run_object_detection(
+    ctx,
+    image_bytes: bytes | str | Path | None = None,
+    queries: list[str] | None = None,
+    templates: list[str] | None = None,
+    mime_type: str = "image/jpeg",
+    global_timeout: float = 30.0,
+    image_path: str | Path | None = None,
+) -> dict:
+    target_img = image_bytes if image_bytes is not None else image_path
+    if target_img is None:
+        raise ValueError(
+            "Either image_bytes or image_path must be provided to _run_object_detection"
+        )
+    if isinstance(target_img, (str, Path)):
+        image_data = Path(target_img).read_bytes()
+    else:
+        image_data = target_img
+
+    queries = queries or []
+    templates = templates or ["Point to the following objects: {labels_str}"]
+    try:
+        llm = get_llm(ctx, name="object_detector")
+    except Exception:
+        llm = get_llm(ctx, name="operator")
+
+    raw_timeout = getattr(getattr(ctx, "llm_config", None), "timeout", None)
+    if isinstance(raw_timeout, (int, float)):
+        timeout_val = float(raw_timeout)
+    else:
+        timeout_val = float(os.environ.get("OBJECT_DETECTOR_TIMEOUT", "10.0"))
+
+    tasks = [
+        asyncio.create_task(
+            _detect_single_label(
+                llm,
+                ctx,
+                image_data,
+                mime_type,
+                label=query,
+                templates=templates,
+                timeout_val=timeout_val,
+            )
+        )
+        for query in queries
+    ]
+
+    done, pending = await asyncio.wait(tasks, timeout=global_timeout)
+
+    for task in pending:
+        task.cancel()
+    if pending:
+        await asyncio.gather(*pending, return_exceptions=True)
+
+    fused_results = []
+    for task in done:
+        try:
+            res = task.result()
+            if res:
+                fused_results.extend(res)
+        except Exception as e:
+            logger.error(f"Task raised exception: {e}")
+
+    # Swap coordinates from [y, x] normalized to [x, y] normalized if needed
+    for item in fused_results:
+        if (
+            isinstance(item, dict)
+            and "point" in item
+            and isinstance(item["point"], list)
+            and len(item["point"]) == 2
+        ):
+            y_norm, x_norm = item["point"]
+            item["point"] = [x_norm, y_norm]
+
+    detected_labels = set(
+        item["label"] for item in fused_results if isinstance(item, dict) and "label" in item
+    )
+    failed_queries = [query for query in queries if query not in detected_labels]
+
+    result_dict = {"detected": fused_results, "failed": failed_queries}
+
+    if failed_queries:
+        result_dict["message"] = (
+            "Detection failed for some targets. Please try changing to a more"
+            " detailed description (e.g., describe its shape, color, or nearby"
+            " text)."
+        )
+
+    logger.info("Fused results from parallel inference.")
+    return result_dict
+
+
+async def _create_error_command(ctx, state, tool_call_id, error_message, wrapper):
+    tool_message = ToolMessage(
+        tool_call_id=tool_call_id or "default_tool_call",
+        content=wrapper.on_failure_fn(error_message),
+        additional_kwargs={"error": error_message},
+        status="error",
+    )
+    return Command(
+        update=await state.asanitize_update(
+            ctx=ctx,
+            update={
+                VALIDATOR_MESSAGES_KEY: [tool_message],
+            },
+            agent="validator",
+        ),
+    )
+
+
+async def _create_success_command(ctx, state, tool_call_id, output, wrapper):
+    tool_message = ToolMessage(
+        tool_call_id=tool_call_id or "default_tool_call",
+        content=wrapper.on_success_fn(output),
+        status="success",
+    )
+    return Command(
+        update=await state.asanitize_update(
+            ctx=ctx,
+            update={
+                VALIDATOR_MESSAGES_KEY: [tool_message],
+            },
+            agent="validator",
+        ),
+    )

artemis/agents/operator/loop_detector.py

@@ -0,0 +1,42 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Loop, deadlock, and action repetition detector for Operator Agent."""
+
+from typing import Any
+from artemis.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+class LoopDetector:
+    """Monitors recent actions to detect and break infinite interaction loops."""
+
+    def __init__(self, repetition_threshold: int = 3):
+        self.repetition_threshold = repetition_threshold
+        self._action_signatures: list[str] = []
+
+    def record_action(self, action_name: str, params: dict[str, Any]) -> None:
+        sig = f"{action_name}:{sorted(params.items())}"
+        self._action_signatures.append(sig)
+
+    def is_loop_detected(self) -> bool:
+        """Returns True if the identical action has been repeated >= threshold times."""
+        if len(self._action_signatures) < self.repetition_threshold:
+            return False
+        last_sig = self._action_signatures[-1]
+        return all(s == last_sig for s in self._action_signatures[-self.repetition_threshold :])
+
+    def reset(self) -> None:
+        self._action_signatures.clear()

artemis/agents/operator/operator.json

@@ -0,0 +1,4 @@
+{
+  "main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List/Text/Complex UI Exploration Strategy (Zero Omission, Zero Duplication, Single-Pass Rule)**:\n  When traversing large lists, long text bodies, or continuous content streams, adhere strictly to the **Single-Pass Exploration Standard**:\n  1. **Unidirectional Anchor-Based Traversal (Zero Omission)**: Maintain a consistent, unidirectional scan (e.g., scrolling steadily from top to bottom). Before each scroll, use the bottom-most visible item as a visual anchor. Ensure your scrolling distance allows this anchor to remain visible near the top of the subsequent screen, providing seamless visual overlap so no items are jumped over or omitted. Conclude exploration when reaching the definite boundary (e.g., reaching the bottom of the list or when further scrolling reveals no new content).\n  2. **Memory & Plan-Driven Deduplication (Zero Duplication)**: Systematically log extracted item names, identifiers, or processed statuses into notes or memory as you progress. Upon every screen update, cross-reference visible items with your recorded data and strictly process only *newly surfaced, unrecorded* items to avoid duplicate actions on overlapping elements. Furthermore, actively maintaining and updating subtasks in your task plan (e.g., segmenting exploration by batches, categories, or ranges like a-d) explicitly documents the exact paths and progress already completed, maintaining clear self-awareness of your exploration trajectory and preventing backtracking.\n  3. **Single-Pass Trust & Anti-Oscillation (No Excessive Re-verification)**: Trust your recorded observations as the ground truth. Once a section has been traversed and logged, consider it permanently verified. Never scroll back anLine truncated
+  "troubleshooter_template": "# ROLE\nYou are the Expert Android Device Troubleshooter. A previous attempt to complete the task was rejected by the verification agent (Checker). Your priority is diagnosis and execution accuracy to resolve the rejection.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List/Text/Complex UI Exploration Strategy (Zero Omission, Zero Duplication, Single-Pass Rule)**:\n  When traversing large lists, long text bodies, or continuous content streams, adhere strictly to the **Single-Pass Exploration Standard**:\n  1. **Unidirectional Anchor-Based Traversal (Zero Omission)**: Maintain a consistent, unidirectional scan (e.g., scrolling steadily from top to bottom). Before each scroll, use the bottom-most visible item as a visual anchor. Ensure your scrolling distance allows this anchor to remain visible near the top of the subsequent screen, providing seamless visual overlap so no items are jumped over or omitted. Conclude exploration when reaching the definite boundary (e.g., reaching the bottom of the list or when further scrolling reveals no new content).\n  2. **Memory & Plan-Driven Deduplication (Zero Duplication)**: Systematically log extracted item names, identifiers, or processed statuses into notes or memory as you progress. Upon every screen update, cross-reference visible items with your recorded data and strictly process only *newly surfaced, unrecorded* items to avoid duplicate actions on overlapping elements. Furthermore, actively maintaining and updating subtasks in your task plan (e.g., segmenting exploration by batches, categories, or ranges like a-d) explicitly documents the exact paths and progress already completed, maintaining clear self-awareness of your exploration trajectory and preventing backtracking.\n  3. **Single-Pass Trust & Anti-Oscillation (No Excessive Re-verification)**: Trust your recorded observations as the ground truth. Once a section has been traversed and loggedLine truncated
+}

artemis/agents/operator/operator_agent.py

@@ -0,0 +1,72 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Modular Operator Agent implementation adhering to BaseAgent lifecycle."""
+
+from typing import Any
+from artemis.agents.base import AgentConfig, AgentRegistry, BaseAgent
+from artemis.agents.operator.loop_detector import LoopDetector
+from artemis.agents.operator.prompt_builder import OperatorPromptBuilder
+from artemis.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+@AgentRegistry.register("operator")
+class OperatorAgent(BaseAgent):
+    """Specialized agent responsible for real-time screen analysis and UI interaction decisions."""
+
+    def __init__(
+        self,
+        config: AgentConfig | None = None,
+        ctx: Any | None = None,
+        **kwargs: Any,
+    ):
+        super().__init__(
+            name="operator",
+            role="Mobile UI Interaction Operator",
+            description="Perceives mobile UI and executes touch/key gestures to progress tasks.",
+            config=config,
+            ctx=ctx,
+        )
+        self.loop_detector = LoopDetector(repetition_threshold=3)
+
+    async def process(self, state: Any) -> dict[str, Any]:
+        """Observes screen state, reasons next action, and dispatches tool call."""
+        if not self.driver:
+            raise RuntimeError("OperatorAgent requires an active device driver in context.")
+
+        # 1. Capture screen data
+        screen_data = await self.driver.get_screen_data()
+
+        # 2. Check for loop deadlock
+        if self.loop_detector.is_loop_detected():
+            logger.warning(
+                "Operator detected repetitive interaction loop! Injecting recovery back-press."
+            )
+            await self.driver.press_key("back")
+            self.loop_detector.reset()
+            return {"action_taken": "press_key:back", "status": "recovering"}
+
+        # 3. Formulate prompt & messages
+        goal = getattr(state, "task_goal", "")
+        human_text = OperatorPromptBuilder.build_human_message(goal=goal)
+
+        return {
+            "thought": "Observed screen and deciding next interaction.",
+            "action_taken": "click",
+            "screen_width": screen_data.width,
+            "screen_height": screen_data.height,
+            "status": "success",
+        }

artemis/agents/operator/prompt_builder.py

@@ -0,0 +1,59 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Dynamic prompt template builder for Operator Agent."""
+
+from jinja2 import Template
+
+OPERATOR_SYSTEM_PROMPT = """You are ARTEMIS Operator, an expert autonomous mobile device interaction agent.
+Your objective is to observe the current Android mobile screen and take precise actions to achieve the user's goal.
+
+Available action commands:
+- click(target=[x, y]): Normalized coordinates 0-1000 scale.
+- swipe(action="up"|"down"|"left"|"right"|[x1, y1, x2, y2]): Drag across screen.
+- input_text(text="...", target=[x, y]): Click field and type.
+- press_key(key="home"|"back"|"enter"|"delete"): Press hardware key.
+- wait_for_delay(seconds=...): Pause execution.
+"""
+
+OPERATOR_HUMAN_TEMPLATE = """Goal: {{ goal }}
+Active Sub-Goal: {{ sub_goal }}
+Current Turn: {{ current_turn }}
+Execution History:
+{{ history }}
+"""
+
+
+class OperatorPromptBuilder:
+    """Builds prompt messages and multimodal contents for the Operator Agent."""
+
+    @classmethod
+    def build_system_message(cls) -> str:
+        return OPERATOR_SYSTEM_PROMPT
+
+    @classmethod
+    def build_human_message(
+        cls,
+        goal: str,
+        sub_goal: str = "",
+        current_turn: int = 1,
+        history: str = "",
+    ) -> str:
+        template = Template(OPERATOR_HUMAN_TEMPLATE)
+        return template.render(
+            goal=goal,
+            sub_goal=sub_goal,
+            current_turn=current_turn,
+            history=history,
+        )

artemis/agents/operator/prompts.py

@@ -0,0 +1,376 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import base64
+import io
+import json
+from pathlib import Path
+
+from jinja2 import Template
+from langchain_core.messages import HumanMessage, SystemMessage
+from PIL import Image
+
+from artemis.context import ArtemisContext
+from artemis.graph.state import State
+from artemis.tools.command_tool import (
+    _format_long_output_response,
+    _is_output_long,
+)
+from artemis.agents.explorer.constants import EXPLORE_DESCRIPTIONS
+from artemis.config import resolve_explorer_version
+from artemis.utils.logger import get_logger
+from artemis.utils.task_tree import get_active_subgoal_hashes
+
+logger = get_logger(__name__)
+
+
+class PromptBuilder:
+    def __init__(self):
+        self.system_parts = []
+        self.human_parts = []
+        self.human_footer = None
+
+    def add_system_text(self, text: str):
+        self.system_parts.append(text)
+
+    def add_human_content(self, content: str | dict):
+        self.human_parts.append(content)
+
+    def set_human_footer(self, content: str):
+        self.human_footer = content
+
+    def build(self) -> list[SystemMessage | HumanMessage]:
+        system_content = "".join(self.system_parts)
+        human_content = []
+        for p in self.human_parts:
+            if isinstance(p, str):
+                human_content.append({"type": "text", "text": p})
+            else:
+                human_content.append(p)
+
+        if self.human_footer:
+            human_content.append({"type": "text", "text": self.human_footer})
+
+        return [
+            SystemMessage(content=system_content),
+            HumanMessage(content=human_content),
+        ]
+
+
+class PromptComponent:
+    async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
+        raise NotImplementedError
+
+
+class TemplatePromptComponent(PromptComponent):
+    async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
+        prompts = kwargs.get("prompts", {})
+        template_name = kwargs.get("template_name", "main_template")
+        prompt_template = prompts.get(template_name)
+        if not prompt_template:
+            raise KeyError(
+                "Failed to format prompt, template not found in operator prompts config."
+            )
+
+        op_version = resolve_explorer_version(ctx, agent_or_profile_name="operator")
+        rule_prompt_info = EXPLORE_DESCRIPTIONS.get(op_version, EXPLORE_DESCRIPTIONS["pro"])
+        rule_prompt = rule_prompt_info["rule_prompt"]
+        if "{max_iterations}" in rule_prompt:
+            max_iterations = 8 if op_version == "ultra" else 3
+            rule_prompt = rule_prompt.format(max_iterations=max_iterations)
+
+        prompt_template = prompt_template.replace(
+            "# OPERATIONAL PRINCIPLES\n",
+            f"# OPERATIONAL PRINCIPLES\n- **Visual Explorer Rule**: {rule_prompt}\n",
+        )
+
+        plan_and_history = kwargs.get("plan_and_history", "No plan or history yet.")
+
+        full_prompt = Template(prompt_template).render(
+            initial_goal=state.initial_goal,
+            subgoals_status="",
+            plan_and_history=plan_and_history,
+            unified_history="",
+        )
+
+        parts = full_prompt.split("# CURRENT OBSERVATION")
+        builder.add_system_text(parts[0] + "# CURRENT OBSERVATION\n")
+
+        if len(parts) > 1:
+            builder.set_human_footer(parts[1])
+
+
+class ObservationPromptComponent(PromptComponent):
+    async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
+        latest_screenshot_b64 = kwargs.get("latest_screenshot_b64")
+        minimal_list = kwargs.get("minimal_list")
+
+        builder.add_human_content("--- Current Screenshot ---")
+        builder.add_human_content(
+            {
+                "type": "image_url",
+                "image_url": {"url": f"data:image/jpeg;base64,{latest_screenshot_b64}"},
+            }
+        )
+        builder.add_human_content(f"--- Visible UI Elements ---\n{minimal_list}")
+
+
+class CheckerFeedbackPromptComponent(PromptComponent):
+    async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
+        if not ctx.data_engine:
+            return
+
+        notes_dir = Path(ctx.data_engine.base_dir) / "notes"
+
+        subgoal_hash = "default"
+        task_plan_path = notes_dir / "task_plan.md"
+        if task_plan_path.exists():
+            try:
+                content = task_plan_path.read_text(encoding="utf-8")
+
+                parent_hash, _ = get_active_subgoal_hashes(content)
+                subgoal_hash = parent_hash
+            except Exception as e:
+                logger.error(f"Failed to parse active subgoal in component: {e}")
+
+        verification_chat_path = notes_dir / f"verification_chat_{subgoal_hash}.json"
+        turns = []
+        if verification_chat_path.exists():
+            try:
+                turns = json.loads(verification_chat_path.read_text(encoding="utf-8"))
+                logger.info(f"Read verification chat for {subgoal_hash}: {len(turns)} turns")
+            except Exception as e:
+                logger.error(f"Error reading verification chat: {e}")
+
+        if turns:
+            dialogue_lines = []
+            for t in turns:
+                role = "Operator" if t["role"] == "operator" else "Checker"
+                dialogue_lines.append(f"**{role} (Round {t['round']})**:\n{t['content']}")
+            checker_feedback = "\n\n".join(dialogue_lines)
+
+            feedback_prompt = f"--- Checker Feedback ---\n{checker_feedback}"
+
+            builder.add_human_content(feedback_prompt)
+
+
+class BackgroundTasksPromptComponent(PromptComponent):
+    async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
+        active_tasks = kwargs.get("active_background_tasks", [])
+        if active_tasks:
+            lines = [
+                "--- Active Background ADB Tasks ---",
+            ]
+            for task in active_tasks:
+                lines.append(
+                    f"- TaskId: {task['task_id']}\n  Command:"
+                    f" `{task['command']}`\n  Cwd: `{task['cwd']}`\n "
+                    f" TerminalID: `{task['terminal_id']}`\n  Accumulated"
+                    f" Output: {task['output_line_count']} lines of logs"
+                )
+            builder.add_human_content("\n".join(lines) + "\n")
+
+        newly_finished_tasks = kwargs.get("newly_finished_tasks", [])
+        if newly_finished_tasks:
+            lines = [
+                "--- NEWLY FINISHED ADB TASKS (Since last step) ---",
+            ]
+            for task in newly_finished_tasks:
+                task_id = task["task_id"]
+                command = task["command"]
+                status = task["status"]
+                output_text = task.get("output_text", "")
+
+                intro = f"- TaskId: {task_id}\n  Command: `{command[:60]}...`\n  Status: {status}"
+
+                if _is_output_long(output_text):
+                    formatted = _format_long_output_response(task_id, output_text, intro)
+                    # format_long_output_response is multi-line, let's indent it nicely
+                    indented = "\n".join(f"  {line}" for line in formatted.splitlines())
+                    lines.append(indented)
+                else:
+                    lines.append(f"{intro}\n  Final Output:\n  {output_text.strip()}")
+            builder.add_human_content("\n".join(lines) + "\n")
+
+
+class ShortTermMemoryPromptComponent(PromptComponent):
+    async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
+        if state.short_term_memory:
+            builder.add_human_content(
+                f"--- Short-Term Memory (Scratchpad) ---\n{state.short_term_memory}\n"
+            )
+
+
+class TaskPlanWarningPromptComponent(PromptComponent):
+    async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
+        steps = kwargs.get("steps", [])
+        if len(steps) >= 2:
+            last_two_steps = steps[-2:]
+            modified_in_last_two = False
+            for step in last_two_steps:
+                tool_calls = step.get("tool_calls", [])
+                for tc in tool_calls:
+                    if tc.get("name") in [
+                        "update_note",
+                        "save_note",
+                        "append_note",
+                    ]:
+                        args = tc.get("args", {})
+                        note_key = args.get("key") or args.get("name")
+                        if note_key == "task_plan":
+                            modified_in_last_two = True
+                            break
+                if modified_in_last_two:
+                    break
+
+            if not modified_in_last_two:
+                builder.add_human_content(
+                    "\nReminder: You have not updated the task_plan for two"
+                    " consecutive turns. Please check the latest progress,"
+                    " reflect on whether the task planning is detailed enough,"
+                    " and whether every pending item is listed as a subtask."
+                    " Please update the completed tasks and pending tasks in"
+                    " detail."
+                )
+
+
+class ToolLimitWarningPromptComponent(PromptComponent):
+    async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
+        if getattr(state, "operator_tool_limit_exceeded", False):
+            builder.add_human_content(
+                "\n Warning: You did not execute any screen interaction actions"
+                " in your last turn. Please re-examine the task goal and revise"
+                " your plan; your current approach may not be the correct path."
+                " Actively calling ask_diagnoser can help you diagnose the"
+                " issue."
+            )
+
+
+class InjectedInstructionPromptComponent(PromptComponent):
+    async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
+        injected = getattr(state, "injected_instruction", None)
+        if injected:
+            builder.add_human_content(
+                f"\n--- User Guidance ---\n"
+                f"The user observing your progress has provided the following"
+                f" feedback or correction:\n"
+                f'"{injected}"\n\n'
+                f"Please review this guidance, evaluate it against your current"
+                f" screen state and recent history, "
+                f"and integrate it into your reasoning. Use it to refine your"
+                f" task plan and determine the "
+                f"most appropriate next action."
+            )
+
+
+class ScreenshotSimilarityPromptComponent(PromptComponent):
+    """Background check: compare current screen against post-action screenshots of the last few steps.
+
+    Inject a note if any are identical. This helps detect if the same screen
+    keeps reappearing unexpectedly.
+    """
+
+    NUM_STEPS_BACK: int = 3
+    MAX_ALLOWED_DIFF_PIXELS: int = 3
+    COLOR_TOLERANCE: int = 8
+
+    async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
+        if not ctx or not ctx.data_engine:
+            return
+        latest_screenshot_b64 = kwargs.get("latest_screenshot_b64")
+        steps = kwargs.get("steps") or []
+        if not latest_screenshot_b64 or not steps:
+            return
+        # 1. Load current live image and re-encode to JPEG for symmetric compression matching
+        try:
+            # Strip data URI header if present
+            if "," in latest_screenshot_b64:
+                latest_screenshot_b64 = latest_screenshot_b64.split(",", 1)[1]
+            curr_img_bytes = base64.b64decode(latest_screenshot_b64)
+            raw_img = Image.open(io.BytesIO(curr_img_bytes)).convert("RGB")
+            # Symmetric in-memory re-encoding to match JPEG compression artifacts
+            jpeg_buf = io.BytesIO()
+            raw_img.save(jpeg_buf, format="JPEG", quality=75)
+            jpeg_buf.seek(0)
+            curr_img = Image.open(jpeg_buf).convert("RGB")
+        except Exception:
+            return
+        # 2. Get last steps with a recorded post_image_name or pre_image_name
+        history_steps = [s for s in steps if s.get("post_image_name") or s.get("pre_image_name")]
+        recent_steps = history_steps[-self.NUM_STEPS_BACK :]
+        matched_step_nums = []
+        images_dir = Path(ctx.data_engine.global_base_dir) / "images"
+        # 3. Compare current image with each past step's post-action screenshot
+        for step_rec in recent_steps:
+            step_num = step_rec.get("step_number")
+            if step_num is None:
+                continue
+            image_name = step_rec.get("post_image_name") or step_rec.get("pre_image_name")
+            if not image_name:
+                image_name = step_rec.get("post_image_name")
+            if not image_name:
+                continue
+            image_path = images_dir / f"{image_name}.jpg"
+            if not image_path.exists():
+                continue
+            try:
+                past_img = Image.open(image_path).convert("RGB")
+
+                # Must be same dimensions to compare pixel-by-pixel
+                if curr_img.size != past_img.size:
+                    continue
+                # Count differing pixels
+                diff_count = self._count_differing_pixels(
+                    curr_img, past_img, max_allowed=self.MAX_ALLOWED_DIFF_PIXELS
+                )
+                if diff_count <= self.MAX_ALLOWED_DIFF_PIXELS:
+                    matched_step_nums.append(str(step_num))
+            except Exception:
+                continue
+        # 4. Inject note if any identical screenshots are found
+        if matched_step_nums:
+            steps_str = ", ".join(matched_step_nums)
+            note_text = (
+                f"Note: Screenshot in step {steps_str} seem to be identical to"
+                " current screen. This could be intended since not all actions"
+                " alter screens."
+            )
+            builder.add_human_content(note_text)
+
+    def _count_differing_pixels(
+        self,
+        img1: Image.Image,
+        img2: Image.Image,
+        max_allowed: int | None = None,
+    ) -> int:
+        """Count pixels that differ between two RGB images.
+
+        Early-exits if diff_count exceeds max_allowed for performance.
+        """
+        if max_allowed is None:
+            max_allowed = self.MAX_ALLOWED_DIFF_PIXELS
+        data1 = img1.getdata()
+        data2 = img2.getdata()
+        diff_count = 0
+        tolerance = self.COLOR_TOLERANCE
+        for p1, p2 in zip(data1, data2):
+            if (
+                abs(p1[0] - p2[0]) > tolerance
+                or abs(p1[1] - p2[1]) > tolerance
+                or abs(p1[2] - p2[2]) > tolerance
+            ):
+                diff_count += 1
+                if diff_count > max_allowed:
+                    break
+        return diff_count

artemis/agents/outputter/outputter.md

@@ -0,0 +1,28 @@
+# Context & Inputs
+
+## User's Initial Goal
+> {{ initial_goal }}
+
+{% if plan_and_history %}
+## Execution History & Plan
+```markdown
+{{ plan_and_history }}
+```
+{% endif %}
+
+{% if output_description and not structured_output %}
+## Target Information to Extract
+The user explicitly requested to extract the following information:
+> "{{ output_description }}"
+{% endif %}
+
+---
+
+# Instruction
+
+1. **Assess Confidence & Verify**: Based on the execution history, final state, and the target information to extract, determine if you can confidently formulate the final answer. If the current evidence is insufficient or you require more granular proof, you **MUST** actively gather information and details using your tools first. If your tools fail (e.g., return errors or time out), or if the steps were skipped/not observed, you **MUST** report this lack of evidence honestly. Do not assume, guess, or fabricate any unobserved steps.
+2. **Write Persistent Report**: Once your investigation is complete, you **MUST** synthesize your findings into a beautifully formatted Markdown summary report (detailing Status [SUCCESS/FAILED/PARTIAL], Evidence, and a clear explanation). You must write this report to persistent memory under the key "output" by calling:
+   `save_note(key="output", content="...your markdown report...")`
+3. **Final Response**: Only after saving the report, write a direct, helpful, and jargon-free final summary response to the user. Do not use any XML tags, code block wrappers, or special delimiters—just state the conclusion clearly and naturally, ensuring any extracted target information is clearly presented.
+
+

artemis/agents/outputter/outputter.py

@@ -0,0 +1,317 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import json
+from pathlib import Path
+
+from jinja2 import Template
+from langchain_core.messages import BaseMessage, HumanMessage, SystemMessage, ToolMessage
+from artemis.agents.outputter.tools import (
+    get_search_history_tool,
+    get_step_details_tool,
+    get_step_screenshot_tool,
+)
+from artemis.config import OutputConfig
+from artemis.context import ArtemisContext
+from artemis.data_engine.trace import trace
+from artemis.graph.state import State
+from artemis.services.llm import get_llm, invoke_llm_with_timeout_message, with_fallback
+from artemis.tools.scratchpad import (
+    get_append_note_tool_pure,
+    get_list_notes_tool_pure,
+    get_read_note_tool_pure,
+    get_save_note_tool_pure,
+    get_update_note_tool_pure,
+)
+from artemis.tools.video_tool import get_video_analyzer_tool_pure
+from artemis.utils.logger import get_logger
+from artemis.utils.task_tree import build_plan_and_history, get_active_subgoal_hashes
+from pydantic import BaseModel
+
+logger = get_logger(__name__)
+
+
+@trace(type="agent", name="outputter")
+async def outputter(
+    ctx: ArtemisContext,
+    output_config: OutputConfig,
+    graph_output: State,
+    plan_and_history: str | None = None,
+) -> dict:
+    logger.info("Starting Outputter Agent")
+
+    # Fetch plan and history if not provided
+    if plan_and_history is None and ctx.data_engine:
+        try:
+            history = ctx.data_engine.get_agent_friendly_steps()
+
+            # Read current plan if it exists and base_dir is valid
+            current_plan = ""
+            base_dir = getattr(ctx.data_engine, "base_dir", None)
+            if base_dir and isinstance(base_dir, (str, Path)):
+                notes_dir = Path(base_dir) / "notes"
+                current_path = notes_dir / "task_plan.md"
+                if current_path.exists():
+                    try:
+                        current_plan = current_path.read_text(encoding="utf-8")
+                    except Exception as e:
+                        logger.error(f"Failed to read current plan for outputter: {e}")
+
+            if history:
+                active_subgoal_hash = "default"
+                if current_plan:
+                    try:
+                        active_subgoal_hash, _ = get_active_subgoal_hashes(current_plan)
+                    except Exception as e:
+                        logger.error(f"Failed to parse active subgoal in outputter: {e}")
+
+                plan_and_history = build_plan_and_history(
+                    current_plan,
+                    history,
+                    active_subgoal_hash,
+                    last_n_detailed=0,
+                    min_summaries=len(history),
+                    strict_milestone_pruning=False,
+                )
+        except Exception as e:
+            logger.error(f"Failed to resolve plan and history in outputter: {e}")
+
+    system_message = (
+        "You are the Output Synthesis Agent for an Android UI automation"
+        " system. Your sole objective is to verify whether the user's initial"
+        " goal was achieved and synthesize the final report.\n\n## Core"
+        " Principles\n1. STRICTLY EVIDENCE-BASED & NO HALLUCINATION: Rely ONLY"
+        " on the actual observed execution history and successful tool outputs."
+        " Do not assume, reconstruct, or hallucinate any actions, elements, or"
+        " steps that did not actually occur during this specific run. If a step"
+        " was skipped or not observed (e.g., because the device was already in"
+        " the target state), you must explicitly state that it was 'Not"
+        " Observed / Already in State' rather than fabricating the interaction."
+        " If a tool call (like `video_analyzer`) fails or returns an error, you"
+        " must report the failure and the resulting lack of evidence, rather"
+        " than assuming success or guessing the details. If the user requests a"
+        " step-by-step guide, you must only include steps that were actually"
+        " executed and observed; do not invent 'standard' steps to fill in"
+        " gaps.\n2. NO JARGON IN FINAL ANSWER: The 'Final Answer' must be"
+        " written in clear, user-friendly language, completely free of"
+        " agent-specific jargon (e.g., do not mention 'nodes', 'XPath',"
+        " 'selectors', 'ReAct', or 'tools').\n\n## Tool Usage Guidelines\n-"
+        " LAZY VERIFICATION: If the execution history and the final screenshot"
+        " already in your context provide undeniable proof of the outcome, do"
+        " not call tools. Directly output your conclusion.\n- ACTIVE RETRIEVAL:"
+        " You should only call tools (e.g., `get_step_details`,"
+        " `get_step_screenshot`, `search_history_for_text`) if you need to"
+        " extract specific hidden data (like verification codes, tracking"
+        " numbers) or if the final state is ambiguous.\n- VIDEO ANALYSIS COST:"
+        " The `video_analyzer` tool is expensive. Use it only when necessary"
+        " (e.g., to verify video playback or motion).\n- PERSISTENT WRITE: If"
+        " you extract important information requested by the user (such as a"
+        " verification code, tracking number, or a text summary), you should"
+        " use `save_note` or `update_note` to write it directly into persistent"
+        " notes so it can be retrieved by other agents or the system."
+    )
+
+    render_kwargs = {
+        "initial_goal": graph_output.initial_goal,
+        "structured_output": output_config.structured_output,
+        "output_description": output_config.output_description,
+        "plan_and_history": plan_and_history,
+    }
+
+    human_message = Template(
+        Path(__file__).parent.joinpath("outputter.md").read_text(encoding="utf-8")
+    ).render(**render_kwargs)
+
+    screenshot_b64 = (
+        graph_output.operator_raw_data.get("screenshot_b64")
+        if graph_output.operator_raw_data
+        else None
+    )
+
+    human_message_content = [{"type": "text", "text": human_message}]
+    if screenshot_b64:
+        human_message_content.append(
+            {
+                "type": "image_url",
+                "image_url": {"url": f"data:image/jpeg;base64,{screenshot_b64}"},
+            }
+        )
+
+    messages: list[BaseMessage] = [
+        SystemMessage(content=system_message),
+        HumanMessage(content=human_message_content),
+    ]
+
+    # Setup tools
+    history_steps = []
+    if ctx.data_engine:
+        history_steps = ctx.data_engine.get_agent_friendly_steps()
+
+    details_tool = get_step_details_tool(history_steps)
+    screenshot_tool = get_step_screenshot_tool(ctx, history_steps)
+    search_tool = get_search_history_tool(ctx)
+    list_notes_tool = get_list_notes_tool_pure(ctx)
+    read_note_tool = get_read_note_tool_pure(ctx)
+    save_note_tool = get_save_note_tool_pure(ctx)
+    update_note_tool = get_update_note_tool_pure(ctx)
+    append_note_tool = get_append_note_tool_pure(ctx)
+    video_analyzer_tool = get_video_analyzer_tool_pure(ctx)
+
+    tools = [
+        details_tool,
+        screenshot_tool,
+        search_tool,
+        list_notes_tool,
+        read_note_tool,
+        save_note_tool,
+        update_note_tool,
+        append_note_tool,
+        video_analyzer_tool,
+    ]
+
+    llm = get_llm(ctx=ctx, name="outputter", is_utils=True)
+    llm_fallback = get_llm(ctx=ctx, name="outputter", is_utils=True, use_fallback=True)
+
+    llm_with_tools = llm.bind_tools(tools=tools)
+    llm_fallback_with_tools = llm_fallback.bind_tools(tools=tools)
+
+    # ReAct Loop
+    max_turns = 20
+    raw_answer = None
+    for turn in range(max_turns):
+        logger.info(f"Outputter ReAct turn {turn + 1}")
+
+        response = await with_fallback(
+            main_call=lambda: invoke_llm_with_timeout_message(llm_with_tools.ainvoke(messages)),
+            fallback_call=lambda: invoke_llm_with_timeout_message(
+                llm_fallback_with_tools.ainvoke(messages)
+            ),
+        )
+
+        messages.append(response)
+
+        if not response.tool_calls:
+            raw_answer = response.content
+            break
+
+        for tc in response.tool_calls:
+            tool_name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
+            args = tc["args"]
+
+            logger.info(f"Outputter executing tool {tool_name} with args: {args}")
+            try:
+                if tool_name == "get_step_details":
+                    result = await details_tool.ainvoke(args)
+                elif tool_name == "get_step_screenshot":
+                    result = await screenshot_tool.ainvoke(args)
+                elif tool_name == "search_history_for_text":
+                    result = await search_tool.ainvoke(args)
+                elif tool_name == "list_notes":
+                    result = await list_notes_tool.ainvoke(args)
+                elif tool_name == "read_note":
+                    result = await read_note_tool.ainvoke(args)
+                elif tool_name == "save_note":
+                    result = await save_note_tool.ainvoke(args)
+                elif tool_name == "update_note":
+                    result = await update_note_tool.ainvoke(args)
+                elif tool_name == "append_note":
+                    result = await append_note_tool.ainvoke(args)
+                elif tool_name == "video_analyzer":
+                    result = await video_analyzer_tool.ainvoke(args)
+                else:
+                    result = f"Error: Tool {tool_name} is not supported."
+                status = (
+                    "success"
+                    if not (isinstance(result, str) and result.startswith("Error"))
+                    else "error"
+                )
+            except Exception as e:
+                logger.error(f"Error running tool {tool_name}: {e}")
+                result = f"Error running tool {tool_name}: {e}"
+                status = "error"
+
+            messages.append(
+                ToolMessage(
+                    tool_call_id=tc["id"],
+                    content=result if isinstance(result, (str, list)) else str(result),
+                    status=status,
+                )
+            )
+
+    if raw_answer is None:
+        raw_answer = "Error: Outputter failed to resolve the query within maximum turns."
+        logger.error(raw_answer)
+
+    # Final Formatting if structured output is requested
+    if output_config.structured_output:
+        logger.info("Formatting raw output to structured format...")
+        schema = None
+        so = output_config.structured_output
+        if isinstance(so, dict):
+            schema = so
+        elif isinstance(so, BaseModel):
+            schema = type(so)
+        elif isinstance(so, type) and issubclass(so, BaseModel):
+            schema = so
+
+        if schema is not None:
+            structured_llm = llm.with_structured_output(schema)
+            structured_llm_fallback = llm_fallback.with_structured_output(schema)
+
+            format_messages = [
+                SystemMessage(
+                    content=(
+                        "You are a helpful assistant. Your task is to take the"
+                        " raw execution summary and format it into the"
+                        " requested structured output schema. Do not invent any"
+                        " information. If the raw summary does not contain the"
+                        " required info, leave those fields empty or null."
+                    )
+                ),
+                HumanMessage(
+                    content=(
+                        f"Initial Goal: {graph_output.initial_goal}\nRaw Summary:\n{raw_answer}"
+                    )
+                ),
+            ]
+
+            response = await with_fallback(
+                main_call=lambda: invoke_llm_with_timeout_message(
+                    structured_llm.ainvoke(format_messages)
+                ),
+                fallback_call=lambda: invoke_llm_with_timeout_message(
+                    structured_llm_fallback.ainvoke(format_messages)
+                ),
+            )
+
+            if isinstance(response, BaseModel):
+                return response.model_dump()
+            return response
+
+    # Fallback to old behavior if it didn't match the template
+    if isinstance(raw_answer, str):
+        raw_answer_stripped = raw_answer.strip()
+        if raw_answer_stripped.startswith("{") and raw_answer_stripped.endswith("}"):
+            try:
+                return json.loads(raw_answer_stripped)
+            except Exception:
+                pass
+        elif raw_answer_stripped.startswith("```json") and raw_answer_stripped.endswith("```"):
+            try:
+                content = raw_answer_stripped[7:-3].strip()
+                return json.loads(content)
+            except Exception:
+                pass
+
+    return raw_answer

artemis/agents/outputter/tools.py

@@ -0,0 +1,170 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import base64
+import json
+from pathlib import Path
+from typing import Any
+
+from langchain_core.tools import BaseTool, tool
+from artemis.context import ArtemisContext
+from artemis.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+def get_step_details_tool(history_steps: list[dict[str, Any]]) -> BaseTool:
+    @tool
+    def get_step_details(
+        start_step: int,
+        end_step: int,
+    ) -> str:
+        """Retrieve the detailed, full-granularity information for a range of steps (inclusive).
+
+        Use this when you need to inspect specific actions taken, operator
+        thinking, or execution results.
+        """
+        try:
+            s_step = int(start_step)
+            e_step = int(end_step)
+        except (ValueError, TypeError):
+            return (
+                f"Error: start_step and end_step must be integers, got {start_step} and {end_step}."
+            )
+
+        matched_steps = []
+        for s in history_steps:
+            step_num = s.get("step_number")
+            if step_num is not None and s_step <= step_num <= e_step:
+                details = {
+                    "step_number": s.get("step_number"),
+                    "relative_time": s.get("relative_time"),
+                    "summary": s.get("summary"),
+                    "action_taken": s.get("action_taken"),
+                    "operator_raw_thinking": s.get("operator_raw_thinking"),
+                    "operator_native_thinking": s.get("operator_native_thinking"),
+                    "last_execution_result": s.get("last_execution_result"),
+                    "interleaved_events": s.get("interleaved_events"),
+                }
+                matched_steps.append(details)
+        if not matched_steps:
+            return f"No steps found in range [{s_step}, {e_step}]."
+        return json.dumps(matched_steps, indent=2, ensure_ascii=False)
+
+    return get_step_details
+
+
+def get_step_screenshot_tool(ctx: ArtemisContext, history_steps: list[dict[str, Any]]) -> BaseTool:
+    @tool
+    def get_step_screenshot(step_number: int) -> list[dict[str, Any]] | str:
+        """Retrieve the screenshot for a specific step.
+
+        Returns a multimodal content block containing the image.
+        """
+        if not ctx.data_engine:
+            return "Error: DataEngine not available."
+
+        try:
+            step_num = int(step_number)
+        except (ValueError, TypeError):
+            return f"Error: step_number must be an integer, got {step_number}."
+
+        # Find the step to get the image name
+        step_record = next((s for s in history_steps if s.get("step_number") == step_num), None)
+        if not step_record:
+            return f"Error: Step {step_num} not found in history."
+
+        image_name = step_record.get("pre_image_name") or step_record.get("post_image_name")
+        if not image_name:
+            return f"Error: No screenshot recorded for step {step_num}."
+
+        images_dir = Path(ctx.data_engine.global_base_dir) / "images"
+        image_path = images_dir / f"{image_name}.jpg"
+
+        if not image_path.exists():
+            return f"Error: Screenshot file for step {step_num} does not exist on disk."
+
+        try:
+            with open(image_path, "rb") as f:
+                img_base64 = base64.b64encode(f.read()).decode("utf-8")
+        except Exception as e:
+            return f"Error reading screenshot file: {e}"
+
+        # Return a LangChain multimodal content structure
+        return [
+            {"type": "text", "text": f"Screenshot for step {step_num}:"},
+            {
+                "type": "image_url",
+                "image_url": {"url": f"data:image/jpeg;base64,{img_base64}"},
+            },
+        ]
+
+    return get_step_screenshot
+
+
+def get_search_history_tool(ctx: ArtemisContext) -> BaseTool:
+    @tool
+    def search_history_for_text(query: str) -> str:
+        """Search for a specific text query within the OCR and UI hierarchy of all historical steps."""
+        if not ctx.data_engine:
+            return "Error: DataEngine not available."
+
+        session_id = ctx.data_engine.current_session_id
+        if not session_id:
+            return "Error: No active session."
+
+        try:
+            steps = ctx.data_engine.storage.get_steps(session_id)
+        except Exception as e:
+            return f"Error retrieving steps from storage: {e}"
+
+        results = []
+        for step in steps:
+            step_matches = []
+            # Search pre-action image
+            if step.pre_image_name:
+                try:
+                    matches = ctx.data_engine.storage.search_ui_by_hash(step.pre_image_name, query)
+                    if matches:
+                        step_matches.extend(matches)
+                except Exception as e:
+                    logger.error(f"Failed to search UI by hash {step.pre_image_name}: {e}")
+
+            # Search post-action image
+            if step.post_image_name:
+                try:
+                    matches = ctx.data_engine.storage.search_ui_by_hash(step.post_image_name, query)
+                    if matches:
+                        step_matches.extend(matches)
+                except Exception as e:
+                    logger.error(f"Failed to search UI by hash {step.post_image_name}: {e}")
+
+            if step_matches:
+                # Clean up match details to not bloat context, keep top 3
+                clean_matches = []
+                for m in step_matches[:3]:
+                    clean_matches.append(
+                        {
+                            "type": m.get("type"),
+                            "matched_text": m.get("matched_text"),
+                            "score": m.get("score"),
+                        }
+                    )
+                results.append({"step_number": step.step_number, "matches": clean_matches})
+
+        if not results:
+            return f"No matches found for '{query}' in history."
+        return json.dumps(results, indent=2, ensure_ascii=False)
+
+    return search_history_for_text

artemis/agents/planner/planner.json

@@ -0,0 +1,30 @@
+{
+  "blocks": {
+    "role": "## Role\nYou are the Planner agent. Your objective is to decompose high-level user goals into sequential milestones for mobile device execution.",
+    "role_validator": "## Role\nYou are the Task Plan Validator. The Operator has attempted to modify the top-level task plan. Your job is to ensure the proposed new plan still aligns with the user's original intent and is viable.",
+    "guidelines": "## Operating Principles\n\n### 1. Visual Grounding (Start from Current State)\nBase all planning decisions strictly on the provided screenshot of the current device state. Your plan must begin from the current context of the screen.\n\n### 2. Milestone Granularity\nDefine high-level milestones rather than granular UI operations. The milestones you establish are essential checkpoints that the task must pass through; your goal is to keep the task on the right track, not to think about how the task should be completed step-by-step.\n\n### 3. Flexibility and Possibility Retention (Avoid Over-Specification)\nAvoid unnecessarily restricting the execution path or specific tools/applications unless strictly mandated by the user. For example, if the user asks to perform a task that can be accomplished via multiple interfaces (such as a native app or a web browser), describe the milestone in terms of the desired outcome rather than forcing a specific application or interface (e.g. forcing the use of a specific native app when a web browser could also serve). Allow the downstream execution agent the flexibility to choose the most viable path based on the dynamic state of the device.\n\n### 4. Heuristics for Complex & Iterative Operations\nWhen a milestone requires the downstream agent to perform complex iterations, maintain state, or map computed data back to the UI (e.g., adding a pre-selected list of items to a system), you MUST provide heuristics via indented sub-tasks. These sub-tasks should instruct the agent on the exact iterative process (e.g., searching for each item one by one from a note, applying it, and verifying the final state).",
+    "capabilities_operator": "## Execution Capabilities\n\nThe execution agent primarily interacts with the mobile device and analyzes screen recordings and logs. It can perform the following types of actions:\n- **UI Interaction**: Tapping, swiping, and entering text on the screen.\n- **App Management**: Launching and stopping specific applications.\n- **Information Retrieval & Analysis**: Using OCR to read text on screen, searching the web for external information, querying system logs, and analyzing screen recordings.\n- **Terminal & Background Operations**: Executing arbitrary ADB shell commands on the device for exploration. It is capable of running long-running ADB commands asynchronously in the background as tasks.\n\n### Capability Boundaries & Planning Rules\n- **Cross-App Operations**: When a task requires interacting with multiple applications, split the operations into distinct milestones for each app. Avoid mixing actions for different apps in a single milestone.",
+    "output_protocol_initial": "## Task Plan & Scratchpad\nSubmit your initial plan by invoking the `save_note` tool with `key='task_plan'`. Avoid writing the plan in your text response, as it will only be recognized if submitted via the tool.\n\nFormat the plan as a Markdown task list starting with `- [ ]` for uncompleted items. You can use indented sub-tasks (e.g., `  - [ ]`) to break down complex exploratory steps and provide heuristics to guide the downstream agent. Ensure the content of the `task_plan` note contains only the task list, excluding any introductions, explanations, or headers.\n\n### Creating Additional Notes\nYou are highly recommended to create additional empty notes (like answer sheets) for the downstream executor to fill out with findings, lists, or analytical information relevant to the task. For example, if the user asks you to analyze specific YouTube videos, you should pre-create an empty note with a descriptive name (e.g., `analysis_of_xxx_videos`), and make sure to put the note name directly inside your milestones to guide the downstream executor in filling out this answer sheet.",
+    "examples_initial": "## Examples\n\n### Example 1: Basic Task\n**Goal**: \"Send 'I will be late' to Alice on WhatsApp\"\n\n**Incorrect (Too granular)**:\n- [ ] Tap search icon\n- [ ] Type Alice\n- [ ] Tap Alice's avatar\n\n**Correct (Milestones)**:\n- [ ] Open the chat window with Alice in WhatsApp\n- [ ] Successfully send the message \"I will be late\" in Alice's chat window\n\n### Example 2: Information Retrieval & Scratchpad\n**Goal**: \"Find 10 nature videos on YouTube\"\n\n**Incorrect (Vague or directly jumping to end)**:\n- [ ] Open YouTube\n- [ ] Search nature videos and pick 10\n\n**Correct (Leveraging Scratchpad & Sub-tasks Heuristics)**:\n*(Note: As the Planner, pre-create the required empty notes `nature_videos_candidates` and `nature_videos` using the note-taking tools before completing the plan)*\n- [ ] Search for \"nature videos\" on YouTube\n- [ ] Save a list of 15 candidate videos into a temporary note called `nature_videos_candidates`\n  - [ ] Add candidate 1\n  - [ ] Add candidate 2\n  - [ ] (Continue until 15 candidates are found)\n- [ ] Evaluate candidate videos from `nature_videos_candidates` one by one, select exactly 10 correct ones, and save them into `nature_videos`\n  - [ ] Evaluate candidate 1: watch, filter out if irrelevant, save if correct\n  - [ ] Evaluate candidate 2\n  - [ ] (Continue evaluating one by one until 10 are selected)",
+    "validator_guidelines": "## Validation Principles\n- **Goal Alignment**: Ensure the new plan ultimately accomplishes the `Initial Goal` requested by the user. If the Operator has abandoned the core objective or fundamentally altered it, you must reject the plan.\n- **Logical Coherence**: Verify that the proposed milestones form a logical, achievable sequence to reach the goal. Reject the plan if it introduces contradictory steps, infinite loops, or obvious logical fallacies.\n- **Adaptive Acceptance**: The Operator operates in a dynamic environment and may encounter unexpected UI states. You should approve alternative strategies, extra intermediate steps, or necessary workarounds as long as they serve the primary goal effectively.\n- **Constructive Feedback**: If you must reject the plan, provide clear, objective, and actionable feedback explaining the logical flaw or goal deviation, and suggest how the Operator should correct the plan."
+  },
+  "modes": {
+    "initial_plan": {
+      "system": [
+        "role",
+        "guidelines",
+        "capabilities_operator",
+        "output_protocol_initial",
+        "examples_initial"
+      ],
+      "human": "Goal: {{ initial_goal }}"
+    },
+    "validator": {
+      "system": [
+        "role_validator",
+        "validator_guidelines"
+      ],
+      "human": "Initial Goal: {{ initial_goal }}\n\n### Execution History\n{{ history_str }}\n\n### Operator's Thoughts Before Modifying\n{% if operator_native_thinking %}Native Thinking:\n{{ operator_native_thinking }}\n\n{% endif %}{% if operator_raw_thinking %}Raw Thinking:\n{{ operator_raw_thinking }}\n\n{% endif %}### Old Plan\n{{ content_before }}\n\n### Proposed New Plan\n{{ content_after }}\n\nPlease evaluate if the proposed new plan should be approved."
+    }
+  }
+}

artemis/agents/planner/planner.py

@@ -0,0 +1,422 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import base64
+import json
+from pathlib import Path
+
+from jinja2 import Template
+from langchain_core.messages import HumanMessage, SystemMessage, ToolMessage
+from langchain_core.tools import BaseTool
+from pydantic import BaseModel, Field
+
+from artemis.context import ArtemisContext
+from artemis.controllers.unified_controller import UnifiedMobileController
+from artemis.data_engine.trace import CURRENT_TRACE_ID, TraceSpan, trace
+from artemis.graph.state import State
+from artemis.services.llm import (
+    get_llm,
+    invoke_llm_with_timeout_message,
+    with_fallback,
+)
+from artemis.tools.index import get_tool_by_name, get_tools_from_wrappers
+from artemis.tools.scratchpad import (
+    list_notes_wrapper,
+    read_note_wrapper,
+    save_note_wrapper,
+    update_note_wrapper,
+)
+from artemis.tools.tool_wrapper import (
+    get_tool_result_content,
+    invoke_tool_with_injection,
+)
+from artemis.utils.decorators import wrap_with_callbacks
+from artemis.utils.logger import get_logger
+from artemis.utils.notes import get_note_file_path
+from artemis.utils.task_tree import build_plan_and_history
+
+logger = get_logger(__name__)
+
+
+class _CyFunctionDetectorMeta(type):
+    def __instancecheck__(self, instance):
+        name = type(instance).__name__
+        return (
+            name
+            in (
+                "cyfunction",
+                "cython_function_or_method",
+                "builtin_function_or_method",
+            )
+            or "cyfunction" in name.lower()
+        )
+
+
+class CyFunctionDetector(metaclass=_CyFunctionDetectorMeta):
+    pass
+
+
+class ValidationResult(BaseModel):
+    model_config = {"ignored_types": (CyFunctionDetector,)}
+    is_approved: bool = Field(
+        description=(
+            "True if the new plan is acceptable and aligns with the initial goal, False otherwise."
+        )
+    )
+    feedback: str = Field(
+        description=(
+            "If rejected, provide constructive feedback explaining why and what"
+            " to do instead. If approved, can be empty."
+        )
+    )
+
+
+async def run_async_planner_validation(
+    ctx: ArtemisContext,
+    initial_goal: str,
+    content_before: str,
+    content_after: str,
+    operator_raw_thinking: str | None = None,
+    operator_native_thinking: str | None = None,
+) -> dict:
+    """Asynchronously validates a top-level plan change proposed by the Operator."""
+    logger.info("Starting Async Planner Validation...")
+    try:
+        prompts_path = Path(__file__).parent / "planner.json"
+        with open(prompts_path, encoding="utf-8") as f:
+            prompts_data = json.load(f)
+
+        mode_config = prompts_data["modes"]["validator"]
+        system_blocks = mode_config["system"]
+        system_content = "\n\n".join(prompts_data["blocks"][block] for block in system_blocks)
+        system_message = Template(system_content).render()
+
+        history_str = "No history available."
+        if ctx.data_engine:
+            try:
+                steps = ctx.data_engine.get_agent_friendly_steps()
+                history_str = build_plan_and_history(
+                    "",  # Pass empty string to avoid rendering the task plan in the history
+                    steps,
+                    "default",
+                    last_n_detailed=2,
+                    strict_milestone_pruning=True,
+                    recent_window_size=5,
+                    chronological_last_step=True,
+                )
+
+            except Exception as e:
+                logger.error(f"Failed to fetch execution history for validator: {e}")
+
+        human_content = mode_config["human"]
+        human_message = Template(human_content).render(
+            initial_goal=initial_goal,
+            content_before=content_before,
+            content_after=content_after,
+            operator_raw_thinking=operator_raw_thinking,
+            operator_native_thinking=operator_native_thinking,
+            history_str=history_str,
+        )
+
+        messages = [
+            SystemMessage(content=system_message),
+            HumanMessage(content=human_message),
+        ]
+
+        llm = get_llm(ctx=ctx, name="planner").with_structured_output(ValidationResult)
+
+        result: ValidationResult = await invoke_llm_with_timeout_message(llm.ainvoke(messages))
+
+        return {
+            "status": "success" if result.is_approved else "failed",
+            "feedback": result.feedback,
+        }
+    except Exception as e:
+        logger.error(f"Planner async validation failed: {e}")
+        # In case of error, we fallback to approving so we don't block the agent unnecessarily
+        return {
+            "status": "success",
+            "feedback": f"Validation failed with error: {e}",
+        }
+
+
+class PlannerNode:
+    def __init__(self, ctx: ArtemisContext, tools: list[BaseTool] | None = None):
+        self.ctx = ctx
+        self.tools = tools or []
+
+    @wrap_with_callbacks(
+        before=lambda: logger.info("Starting Planner Agent..."),
+        on_success=lambda _: logger.success("Planner Agent"),
+        on_failure=lambda _: logger.error("Planner Agent"),
+    )
+    @trace(type="agent", name="planner")
+    async def __call__(self, state: State):
+        prompts_path = Path(__file__).parent / "planner.json"
+        with open(prompts_path, encoding="utf-8") as f:
+            prompts_data = json.load(f)
+
+        mode = "initial_plan"
+        mode_config = prompts_data["modes"][mode]
+
+        system_blocks = mode_config["system"]
+        system_content = "\n\n".join(prompts_data["blocks"][block] for block in system_blocks)
+
+        system_message = Template(system_content).render()
+
+        # Get screenshot
+        screenshot_b64 = None
+        screenshot_path = state.latest_screenshot if hasattr(state, "latest_screenshot") else None
+
+        if screenshot_path and Path(screenshot_path).exists():
+            try:
+                with open(screenshot_path, "rb") as f:
+                    screenshot_b64 = base64.b64encode(f.read()).decode("utf-8")
+                logger.info(f"Planner loaded screenshot from file: {screenshot_path}")
+            except Exception as e:
+                logger.error(f"Failed to read screenshot from {screenshot_path}: {e}")
+
+        # Fallback for Initial Plan: if no screenshot in state, take one on the spot
+        if not screenshot_b64:
+            logger.info("No screenshot found in state. Taking screenshot for Initial Plan...")
+            try:
+                controller = UnifiedMobileController(self.ctx)
+                screenshot_b64 = await controller.take_screenshot()
+
+                # Save to Data Engine if available to keep trace
+                if self.ctx.data_engine:
+                    img_bytes = base64.b64decode(screenshot_b64)
+                    image_name = self.ctx.data_engine.get_or_create_image(img_bytes)
+                    saved_path = str(self.ctx.data_engine.get_image_path(image_name))
+                    # Update local state object so it can be used later if needed
+                    state.latest_screenshot = saved_path
+                    logger.info(f"Saved initial plan screenshot to {saved_path}")
+            except Exception as e:
+                logger.error(f"Failed to take screenshot for Initial Plan: {e}")
+
+        if not screenshot_b64:
+            logger.error("Planner failed to obtain a screenshot.")
+            raise ValueError("Planner failed to obtain a screenshot.")
+
+        human_content = mode_config["human"]
+        render_kwargs = {
+            "initial_goal": state.initial_goal,
+        }
+
+        human_message = Template(human_content).render(**render_kwargs)
+
+        human_message_content = [{"type": "text", "text": human_message}]
+        if screenshot_b64:
+            human_message_content.append(
+                {
+                    "type": "image_url",
+                    "image_url": {"url": f"data:image/jpeg;base64,{screenshot_b64}"},
+                }
+            )
+        messages = [
+            SystemMessage(content=system_message),
+            HumanMessage(content=human_message_content),
+        ]
+
+        llm = get_llm(ctx=self.ctx, name="planner")
+        llm_fallback = get_llm(ctx=self.ctx, name="planner", use_fallback=True)
+
+        # Ensure tools are populated (fallback if not passed in __init__, e.g. in tests)
+        if not self.tools:
+            note_wrappers = [
+                save_note_wrapper,
+                read_note_wrapper,
+                update_note_wrapper,
+                list_notes_wrapper,
+            ]
+            self.tools = get_tools_from_wrappers(self.ctx, note_wrappers)
+
+        all_tools = list(self.tools)
+
+        if all_tools:
+            llm = llm.bind_tools(all_tools)
+            llm_fallback = llm_fallback.bind_tools(all_tools)
+
+        def validate_plan_format(content: str) -> tuple[bool, str]:
+            lines = content.splitlines()
+            has_subgoal = False
+            for line in lines:
+                stripped = line.strip()
+                if stripped.startswith("- ["):
+                    has_subgoal = True
+                    if not (
+                        stripped.startswith("- [ ]")
+                        or stripped.startswith("- [x]")
+                        or stripped.startswith("- [!]")
+                    ):
+                        return (
+                            False,
+                            (
+                                f"Invalid task status in line: {line}. Must be"
+                                " '[ ]', '[x]', or '[!]'."
+                            ),
+                        )
+                elif stripped.startswith("#") or not stripped:
+                    continue
+                else:
+                    continue
+
+            if not has_subgoal:
+                return (
+                    False,
+                    ("Plan must contain at least one subgoal starting with '- [ ]'."),
+                )
+
+            return True, ""
+
+        current_messages = messages
+        max_validation_attempts = 3
+        max_iterations = 10
+
+        for validation_attempt in range(max_validation_attempts):
+            tool_called = False
+
+            async def run_stream(llm_model):
+                full_response = None
+                trace_id = CURRENT_TRACE_ID.get()
+                async for chunk in llm_model.astream(current_messages):
+                    if full_response is None:
+                        full_response = chunk
+                    else:
+                        full_response += chunk
+
+                    pass
+                return full_response
+
+            for _ in range(max_iterations):
+                response = await with_fallback(
+                    main_call=lambda: invoke_llm_with_timeout_message(run_stream(llm)),
+                    fallback_call=lambda: invoke_llm_with_timeout_message(run_stream(llm_fallback)),
+                )
+
+                if response is None or not response.tool_calls:
+                    break
+
+                current_messages.append(response)
+
+                for tc in response.tool_calls:
+                    tool_name = tc["name"]
+                    logger.info(f"Planner called tool: {tool_name} with args: {tc['args']}")
+                    if ":" in tool_name:
+                        tool_to_run = get_tool_by_name(tool_name, all_tools)
+                    else:
+                        tool_to_run = next((t for t in all_tools if t.name == tool_name), None)
+
+                    if tool_to_run and tool_to_run.name in [
+                        "save_note",
+                        "update_note",
+                    ]:
+                        tool_called = True
+                    if tool_to_run:
+                        args = dict(tc["args"])
+                        with TraceSpan(name=tool_name, ctx=self.ctx) as span:
+                            span.payload = {"args": args}
+                            try:
+                                logger.info(
+                                    f"Tool {tool_name} expected args:"
+                                    f" {list(tool_to_run.args.keys())}"
+                                )
+
+                                logger.info(f"Invoking {tool_name} with args: {list(args.keys())}")
+                                result_obj = await invoke_tool_with_injection(
+                                    tool=tool_to_run,
+                                    args=args,
+                                    tool_call_id=tc["id"],
+                                    state=state,
+                                )
+                                result_content = get_tool_result_content(result_obj)
+                                span.result = result_content
+
+                                current_messages.append(
+                                    ToolMessage(
+                                        tool_call_id=tc["id"],
+                                        content=result_content,
+                                    )
+                                )
+                            except Exception as e:
+                                span.status = "failed"
+                                span.error = str(e)
+                                current_messages.append(
+                                    ToolMessage(
+                                        tool_call_id=tc["id"],
+                                        content=f"Error: {e}",
+                                        status="error",
+                                    )
+                                )
+                    else:
+                        current_messages.append(
+                            ToolMessage(
+                                tool_call_id=tc["id"],
+                                content=f"Error: Tool {tool_name} not found",
+                                status="error",
+                            )
+                        )
+
+            # Post-execution validation
+            plan_exists = False
+            plan_valid = False
+            error_message = ""
+
+            if not tool_called:
+                error_message = (
+                    "You did not call `save_note` or `update_note` to submit/update the plan."
+                )
+            elif self.ctx.data_engine:
+                task_plan_path = get_note_file_path(self.ctx.data_engine.base_dir, "task_plan")
+                if task_plan_path.exists():
+                    plan_exists = True
+                    try:
+                        content = task_plan_path.read_text(encoding="utf-8")
+                        plan_valid, error_message = validate_plan_format(content)
+                    except Exception as e:
+                        error_message = f"Failed to read task plan for validation: {e}"
+                else:
+                    error_message = (
+                        "Task plan under key 'task_plan' does not exist despite tool call."
+                    )
+
+            if tool_called and plan_exists and plan_valid:
+                logger.info("Plan submitted and validated successfully.")
+                break
+
+            logger.warning(
+                f"Plan validation failed (attempt {validation_attempt + 1}): {error_message}"
+            )
+
+            # Feedback to model
+            current_messages.append(
+                HumanMessage(
+                    content=(
+                        f"Plan validation failed: {error_message}\nPlease"
+                        " correct the plan and try again by calling the"
+                        " appropriate tool."
+                    )
+                )
+            )
+
+        else:
+            logger.error(
+                f"Failed to generate a valid plan after {max_validation_attempts} attempts."
+            )
+
+        return await state.asanitize_update(
+            ctx=self.ctx,
+            update={},
+            agent="planner",
+        )

artemis/agents/planner/planner_agent.py

@@ -0,0 +1,53 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Planner Agent implementation adhering to BaseAgent lifecycle."""
+
+from typing import Any
+from artemis.agents.base import AgentConfig, AgentRegistry, BaseAgent
+from artemis.agents.planner.planner import PlannerNode
+from artemis.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+@AgentRegistry.register("planner")
+class PlannerAgent(BaseAgent):
+    """Specialized agent responsible for high-level goal decomposition and strategic task planning."""
+
+    def __init__(
+        self,
+        config: AgentConfig | None = None,
+        ctx: Any | None = None,
+        **kwargs: Any,
+    ):
+        super().__init__(
+            name="planner",
+            role="Strategic Task Planner",
+            description="Decomposes user objectives into structured, observable milestone steps.",
+            config=config,
+            ctx=ctx,
+        )
+        self._node = PlannerNode(ctx=ctx, **kwargs) if ctx else None
+
+    async def initialize(self, ctx: Any) -> None:
+        await super().initialize(ctx)
+        self._node = PlannerNode(ctx=ctx)
+
+    async def process(self, state: Any) -> dict[str, Any]:
+        if not self._node and self.ctx:
+            self._node = PlannerNode(ctx=self.ctx)
+        if not self._node:
+            raise RuntimeError("PlannerAgent has not been initialized with a ArtemisContext.")
+        return await self._node(state)

artemis/agents/planner/types.py

@@ -0,0 +1,85 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from datetime import datetime
+from enum import Enum
+from typing import Annotated
+
+from pydantic import BaseModel
+
+
+class _CyFunctionDetectorMeta(type):
+    def __instancecheck__(self, instance):
+        name = type(instance).__name__
+        return (
+            name
+            in (
+                "cyfunction",
+                "cython_function_or_method",
+                "builtin_function_or_method",
+            )
+            or "cyfunction" in name.lower()
+        )
+
+
+class CyFunctionDetector(metaclass=_CyFunctionDetectorMeta):
+    pass
+
+
+class PlannerSubgoalOutput(BaseModel):
+    model_config = {"ignored_types": (CyFunctionDetector,)}
+    description: str
+
+
+class PlannerOutput(BaseModel):
+    model_config = {"ignored_types": (CyFunctionDetector,)}
+    subgoals: list[PlannerSubgoalOutput]
+
+
+class SubgoalStatus(Enum):
+    NOT_STARTED = "NOT_STARTED"
+    PENDING = "PENDING"
+    SUCCESS = "SUCCESS"
+    FAILURE = "FAILURE"
+
+
+class Subgoal(BaseModel):
+    model_config = {"ignored_types": (CyFunctionDetector,)}
+    id: Annotated[str, "Unique identifier of the subgoal"]
+    description: Annotated[str, "Description of the subgoal"]
+    completion_reason: Annotated[
+        str | None, "Reason why the subgoal was completed (failure or success)"
+    ] = None
+    status: SubgoalStatus
+    started_at: Annotated[datetime | None, "When the subgoal started"] = None
+    ended_at: Annotated[datetime | None, "When the subgoal ended"] = None
+
+    def __str__(self):
+        status_emoji = "❓"
+        if self.status == SubgoalStatus.SUCCESS:
+            status_emoji = "✅"
+        elif self.status == SubgoalStatus.FAILURE:
+            status_emoji = "❌"
+        elif self.status == SubgoalStatus.PENDING:
+            status_emoji = "⏳"
+        elif self.status == SubgoalStatus.NOT_STARTED:
+            status_emoji = "(not started yet)"
+
+        output = f"- [ID:{self.id}]: {self.description} : {status_emoji}."
+        if self.completion_reason:
+            output += f" Completion reason: {self.completion_reason}"
+        return output
+
+    def __repr__(self):
+        return str(self)

artemis/agents/planner/utils.py

@@ -0,0 +1,89 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+try:
+    from datetime import UTC, datetime
+except ImportError:
+    from datetime import datetime
+
+    UTC = UTC
+import random
+import string
+
+from artemis.agents.planner.types import Subgoal, SubgoalStatus
+
+
+def get_current_subgoal(subgoals: list[Subgoal]) -> Subgoal | None:
+    return next((s for s in subgoals if s.status == SubgoalStatus.PENDING), None)
+
+
+def get_subgoals_by_ids(subgoals: list[Subgoal], ids: list[str]) -> list[Subgoal]:
+    return [s for s in subgoals if s.id in ids]
+
+
+def get_next_subgoal(subgoals: list[Subgoal]) -> Subgoal | None:
+    return next((s for s in subgoals if s.status == SubgoalStatus.NOT_STARTED), None)
+
+
+def nothing_started(subgoals: list[Subgoal]) -> bool:
+    return all(s.status == SubgoalStatus.NOT_STARTED for s in subgoals)
+
+
+def complete_current_subgoal(subgoals: list[Subgoal]) -> list[Subgoal]:
+    current_subgoal = get_current_subgoal(subgoals)
+    if not current_subgoal:
+        return subgoals
+    current_subgoal.status = SubgoalStatus.SUCCESS
+    current_subgoal.ended_at = datetime.now(UTC)
+    return subgoals
+
+
+def complete_subgoals_by_ids(subgoals: list[Subgoal], ids: list[str]) -> list[Subgoal]:
+    for subgoal in subgoals:
+        if subgoal.id in ids:
+            subgoal.status = SubgoalStatus.SUCCESS
+            subgoal.ended_at = datetime.now(UTC)
+    return subgoals
+
+
+def fail_current_subgoal(subgoals: list[Subgoal]) -> list[Subgoal]:
+    current_subgoal = get_current_subgoal(subgoals)
+    if not current_subgoal:
+        return subgoals
+    current_subgoal.status = SubgoalStatus.FAILURE
+    current_subgoal.ended_at = datetime.now(UTC)
+    return subgoals
+
+
+def all_completed(subgoals: list[Subgoal]) -> bool:
+    return all(s.status == SubgoalStatus.SUCCESS for s in subgoals)
+
+
+def one_of_them_is_failure(subgoals: list[Subgoal]) -> bool:
+    return any(s.status == SubgoalStatus.FAILURE for s in subgoals)
+
+
+def start_next_subgoal(subgoals: list[Subgoal]) -> list[Subgoal]:
+    next_subgoal = get_next_subgoal(subgoals)
+    if not next_subgoal:
+        return subgoals
+    next_subgoal.status = SubgoalStatus.PENDING
+    next_subgoal.started_at = datetime.now(UTC)
+    return subgoals
+
+
+def generate_id(length: int = 6) -> str:
+    """Generates a small and distinct random string ID."""
+    chars = string.ascii_lowercase + string.digits
+    return "".join(random.choice(chars) for _ in range(length))

artemis/agents/summarizer/summarizer.md

@@ -0,0 +1,31 @@
+# ROLE
+You are the Summarizer Agent for a mobile automation system. Your task is to summarize the latest execution step, providing a high-information-density and succinct history summary of the current step without losing any information.
+
+# OUTPUT FORMAT
+- **Format Constraint**: You MUST output the summary as a **single, continuous paragraph**.
+- **Prohibited Formats**: Do NOT use bullet points, numbered lists, markdown formatting, or multiple paragraphs.
+- **Goal**: Maintain a consistent, easily parsable narrative for the Operator's historical record.
+
+# CORE PHILOSOPHY & RULES
+
+1. **Perspective**: You MUST write the summary from the Operator's first-person perspective using "I" (e.g., "I intended to...", "I clicked..."). NEVER use third-person terms like "The operator" or "The agent", nor second-person terms like "You" when referring to the Operator. For other agents (e.g., Failure Analyzer, Explorer), you can use their names.
+2. **Zero Subjective Validation**: Absolutely avoid declaring semantic success, completion, or failure. In historical compression, any subjective validation will cause future planners to falsely believe a goal was definitively met or permanently broken.
+   - **BANNED WORDS**: "successfully", "completed", "entered", "navigated to", "failed", "unsuccessful", "could not", "achieved".
+3. **Focus on Intent & Strategy**: The specific physical action (exact coordinates, swipes) is now recorded automatically. Focus primarily on the high-level Intent and reasoning. If you realize the Operator is on the wrong path and needs to backtrack or pivot, describe this as a shift in strategy.
+4. **Intended Context Transitions**: Describe the intended navigation path ("From Where -> Intended Where"). Do not declare that the transition actually happened. (e.g., "From the Home screen, I intended to open the Network Settings view").
+5. **Preserve Verifications & Missing Items**: If the Operator verified any state in this step, or explicitly mentioned what prerequisites or items are still missing to accomplish the goal, you MUST state these contents in detail in the summary.
+6. **Objective Intervention Logs**: If the Failure Analyzer intervened, faithfully summarize what triggered the intervention and what the analyzer executed, without judging the outcome.
+
+# LOOP & STAGNATION DETECTION
+Use the `RECENT 10 STEPS HISTORY` to intelligently detect loops or stagnation:
+1. **Physical Action Loops**: Be vigilant for loops spanning multiple steps based on physical interactions. If you detect a valid action loop, explicitly mention the pattern and the current iteration count (e.g., "This is the 3rd cycle of attempting to access the video group folder").
+2. **Cognitive Stagnation**: While you generally ignore internal background tool calls (e.g., reading logs, checking states) when counting physical action loops, there is an exception: if you detect continuous multiple steps of purely internal tool calls without any physical actions, you MUST objectively record this strategic "stagnation" or "analysis loop".
+
+# INPUT STRUCTURE
+You will receive:
+- **RECENT 10 STEPS HISTORY**: A brief history of the last 10 steps to help you detect loops and stagnation patterns.
+- **CHRONOLOGICAL STEP TRACE (CURRENT STEP)**: The step-by-step chronological history of the current execution turn, including:
+  - `[Operator Monologue] / [Operator Native Thought]`: The operator's planning, intentions, progress counting, reasoning, verified states, and explicit mentions of missing prerequisites.
+  - `[Operator Tool Call] / [Operator Final Action]`: The actual action(s) planned and executed.
+  - `[Pre-Execution Safety Net]`: Validation checks prior to action execution.
+  - `[Failure Analyzer Recovery Loop]`: Interventions, thoughts, auxiliary tool calls, and outcomes of the Failure Analyzer.

artemis/agents/summarizer/summarizer.py

@@ -0,0 +1,231 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import asyncio
+import json
+from pathlib import Path
+from uuid import UUID
+
+from langchain_core.messages import (
+    HumanMessage,
+    SystemMessage,
+)
+
+from artemis.context import ArtemisContext
+from artemis.data_engine.trace import CURRENT_TRACE_ID, trace
+from artemis.graph.state import State
+from artemis.services.llm import get_llm
+from artemis.utils.coordinates import normalize_any_structure
+from artemis.utils.logger import get_logger
+from artemis.utils.task_tree import (
+    build_plan_and_history,
+    get_active_subgoal_hashes,
+)
+
+logger = get_logger(__name__)
+
+
+class SummarizerNode:
+    def __init__(self, ctx: ArtemisContext):
+        self.ctx = ctx
+
+    @trace(type="agent", name="summarizer")
+    async def __call__(self, state: State):
+        # 1. Trigger background task for summary and data engine update
+        if self.ctx.data_engine and state.current_step_id:
+            step_id = UUID(state.current_step_id)
+            decisions = state.structured_decisions
+            operator_raw_thinking = getattr(state, "operator_raw_thinking", None)
+            operator_native_thinking = getattr(state, "operator_native_thinking", None)
+            last_execution_result = getattr(state, "last_execution_result", None)
+
+            # Spawn background task
+            trace_id = CURRENT_TRACE_ID.get()
+            task = asyncio.create_task(
+                self._generate_summary_and_update_engine(
+                    step_id,
+                    decisions,
+                    operator_raw_thinking,
+                    operator_native_thinking,
+                    last_execution_result,
+                    trace_id,
+                ),
+                name=f"summarizer_step_{step_id}",
+            )
+            if hasattr(self.ctx, "background_tasks") and isinstance(
+                self.ctx.background_tasks, list
+            ):
+                self.ctx.background_tasks.append(task)
+
+        return {}
+
+    async def _generate_summary_and_update_engine(
+        self,
+        step_id,
+        decisions,
+        operator_raw_thinking,
+        operator_native_thinking,
+        last_execution_result,
+        trace_id,
+    ):
+        """Background task to generate summary and update Data Engine with retries."""
+
+        max_retries = 3
+        retry_delay = 2
+
+        for attempt in range(max_retries):
+            try:
+                logger.info(f"Generating summary for step {step_id} (attempt {attempt + 1})")
+
+                # Prepare prompt for summary
+                llm = get_llm(ctx=self.ctx, name="summarizer")
+
+                prompt_path = Path(__file__).parent.joinpath("summarizer.md")
+                if not prompt_path.exists():
+                    prompt = (
+                        "You are a summarizer. Summarize the latest action and"
+                        " its result in 1-2 concise sentences. Focus on: Action"
+                        " taken -> Result -> Observation."
+                    )
+                else:
+                    prompt = prompt_path.read_text(encoding="utf-8")
+
+                # Normalize coordinates before presenting to the summarizer LLM
+                width = (
+                    getattr(self.ctx.device, "device_width", 1080)
+                    if self.ctx and self.ctx.device
+                    else 1080
+                )
+                height = (
+                    getattr(self.ctx.device, "device_height", 2400)
+                    if self.ctx and self.ctx.device
+                    else 2400
+                )
+
+                content = []
+
+                # Consistently use optimal build_plan_and_history to construct accurate context
+                plan_and_history_str = ""
+                if self.ctx.data_engine:
+                    try:
+                        friendly_steps = self.ctx.data_engine.get_agent_friendly_steps()
+                        if isinstance(friendly_steps, list):
+                            # 1. Trim history to only include steps up to the current step being summarized (inclusive)
+                            current_steps = []
+                            for s in friendly_steps:
+                                current_steps.append(s)
+                                if str(s.get("step_id")) == str(step_id):
+                                    break
+
+                            # 2. Read task plan to provide goal orientation to the summarizer
+                            task_plan = ""
+                            base_dir = getattr(self.ctx.data_engine, "base_dir", None)
+                            if base_dir and isinstance(base_dir, (str, Path)):
+                                notes_dir = Path(base_dir) / "notes"
+                                task_plan_path = notes_dir / "task_plan.md"
+                                if task_plan_path.exists():
+                                    task_plan = task_plan_path.read_text(encoding="utf-8")
+
+                            # 3. Unified rendering: concise for first N steps (with self-healing facts), detailed for current step (without redundant summary header)
+
+                            subgoal_hash = "default"
+                            if task_plan:
+                                subgoal_hash, _ = get_active_subgoal_hashes(task_plan)
+
+                            plan_and_history_str = build_plan_and_history(
+                                task_plan=task_plan,
+                                steps=current_steps,
+                                current_subgoal_hash=subgoal_hash,
+                                last_n_detailed=1,
+                                min_summaries=10,
+                                strict_milestone_pruning=False,
+                            )
+                    except Exception as e:
+                        logger.error(f"Failed to build plan and history for Summarizer: {e}")
+
+                if plan_and_history_str:
+                    content.append(plan_and_history_str)
+                else:
+                    # Fallback to standard raw fields if chronological trace couldn't be generated
+                    if decisions:
+                        try:
+                            dec_obj = json.loads(decisions)
+                            dec_obj = normalize_any_structure(dec_obj, width, height)
+                            content.append(
+                                f"Decisions made: {json.dumps(dec_obj, ensure_ascii=False)}"
+                            )
+                        except Exception:
+                            content.append(f"Decisions made: {decisions}")
+
+                    if operator_raw_thinking:
+                        content.append(f"Operator explicit thoughts:\n{operator_raw_thinking}")
+
+                    if operator_native_thinking:
+                        content.append(f"Operator native thoughts:\n{operator_native_thinking}")
+
+                messages = [
+                    SystemMessage(content=prompt),
+                    HumanMessage(content="\n\n".join(content)),
+                ]
+
+                generated_text = ""
+                aggregated_chunk = None
+                async for chunk in llm.astream(messages):
+                    if aggregated_chunk is None:
+                        aggregated_chunk = chunk
+                    else:
+                        aggregated_chunk += chunk
+
+                    if chunk.content:
+                        text_to_stream = ""
+                        if isinstance(chunk.content, str):
+                            text_to_stream = chunk.content
+                        elif isinstance(chunk.content, list):
+                            for item in chunk.content:
+                                if isinstance(item, dict) and item.get("type") == "text":
+                                    text_to_stream += item.get("text", "")
+
+                        if text_to_stream:
+                            generated_text += text_to_stream
+                    pass
+
+                summary = generated_text.strip()
+                logger.info(f"Summary generated: {summary}")
+
+                if aggregated_chunk and getattr(aggregated_chunk, "usage_metadata", None):
+                    usage = aggregated_chunk.usage_metadata
+                    cached_tokens = usage.get("input_token_details", {}).get("cache_read", 0)
+                    logger.info(
+                        "Summarizer LLM usage:"
+                        f" prompt_tokens={usage.get('input_tokens')}"
+                        f" (cached={cached_tokens}),"
+                        f" completion_tokens={usage.get('output_tokens')}"
+                    )
+
+                # Update Data Engine
+                self.ctx.data_engine.update_step_summary(step_id, summary)
+                logger.info(f"Data Engine updated with summary for step {step_id}")
+                return  # Success
+
+            except Exception as e:
+                logger.error(
+                    f"Failed to generate summary or update engine (attempt {attempt + 1}): {e}"
+                )
+                if attempt < max_retries - 1:
+                    await asyncio.sleep(retry_delay)
+                    retry_delay *= 2  # Exponential backoff
+                else:
+                    logger.error(
+                        f"Max retries reached for step {step_id}. Summary generation failed."
+                    )

artemis/agents/summarizer/summarizer_agent.py

@@ -0,0 +1,53 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Summarizer Agent implementation adhering to BaseAgent lifecycle."""
+
+from typing import Any
+from artemis.agents.base import AgentConfig, AgentRegistry, BaseAgent
+from artemis.agents.summarizer.summarizer import SummarizerNode
+from artemis.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+@AgentRegistry.register("summarizer")
+class SummarizerAgent(BaseAgent):
+    """Specialized agent responsible for step execution summarization and trajectory logging."""
+
+    def __init__(
+        self,
+        config: AgentConfig | None = None,
+        ctx: Any | None = None,
+        **kwargs: Any,
+    ):
+        super().__init__(
+            name="summarizer",
+            role="Execution History Summarizer",
+            description="Synthesizes actions and observations into concise milestone summaries.",
+            config=config,
+            ctx=ctx,
+        )
+        self._node = SummarizerNode(ctx=ctx) if ctx else None
+
+    async def initialize(self, ctx: Any) -> None:
+        await super().initialize(ctx)
+        self._node = SummarizerNode(ctx=ctx)
+
+    async def process(self, state: Any) -> dict[str, Any]:
+        if not self._node and self.ctx:
+            self._node = SummarizerNode(ctx=self.ctx)
+        if not self._node:
+            raise RuntimeError("SummarizerAgent has not been initialized with a ArtemisContext.")
+        return await self._node(state)

artemis/agents/validator/failure_analyzer.md

@@ -0,0 +1,25 @@
+# Identity & Goal
+You are a specialized error recovery agent. You will be provided with a failed action. Your ONLY task is to repair this specific failure and restore the system state so the main system can resume.
+
+# Context
+- **Initial Goal**: {{ initial_goal }}
+- **History & Plan**:
+{{ plan_and_history }}
+
+# Core Workflow
+1. **Analyze**: Compare the two states provided to you:
+   - **Decision State (`Screenshot Seen During System Decision`)**: This is the screen the system saw when it originally planned the action. It shows the target element successfully located.
+   - **Failed State (`Latest Screenshot (Failed State)`)**: This is the live screen captured at the moment you were summoned. It shows the current state where the target is missing or unresponsive.
+   Your goal in this comparison is to:
+   - Read the Operator's thought process in the decision loop of the failed step to understand what they were originally trying to accomplish with this failed action.
+   - Understand the expected target layout.
+   - Identify the cause of the failure based on the error message and the visual mismatch.
+2. **Recover & Execute**: Restore the target state and execute the failed action.
+   - **Flexibility**: Focus on outcome over form. Take alternative/adaptive steps (e.g., scroll, navigate, different click paths) if original coordinates are invalid.
+3. **Report**: Call `report_failure_analysis` to conclude:
+   - `status="fixed"`: Successfully executed the failed action, reached the destination state, or false alarm.
+   - `status="cannot_fix"`: Target is unrecoverable, the app is broken, or the 15-step budget is exceeded.
+   - `analysis`: Summarize the root cause, the fix applied, and explicitly confirming that control is handed back to the main system.
+
+# Constraints
+The most recent step (Step {{ failed_step_number }}: {{ failed_action_description }}) failed to execute. You must resolve the issue and report the results in 15 interactive steps or fewer. Now please begin the repair analysis. Please note that during the **Recover & Execute** phase, you only need to resolve the currently failed step; do not attempt to execute any subsequent steps or complete the overall goal yourself. Once the failure is resolved, please immediately enter the **Report** phase and return control to the system.

artemis/agents/validator/failure_analyzer.py

@@ -0,0 +1,607 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Universal Multi-Model Failure Analyzer for Artemis.
+
+Analyzes execution failures and performs autonomous self-healing across
+Google Gemini, OpenAI GPT-4o/o3, Anthropic Claude 3.5/3.7, and OpenRouter endpoints.
+"""
+
+import asyncio
+from enum import StrEnum
+import json
+from pathlib import Path
+from typing import Any
+import uuid
+
+from jinja2 import Template
+from langchain_core.messages import (
+    BaseMessage,
+    HumanMessage,
+    SystemMessage,
+    ToolMessage,
+)
+
+from artemis.agents.validator.tool_declarations import (
+    ASK_EXPLORER_TOOL,
+    CLICK_SEQUENCE_TOOL,
+    MobileActionExecutor,
+    ToolDeclaration,
+    VALIDATOR_TOOLS_DECLARATION,
+    capture_screenshot_and_parse_ui,
+    prune_intermediate_screenshots,
+)
+from artemis.context import ArtemisContext
+from artemis.controllers.unified_controller import UnifiedMobileController
+from artemis.data_engine.trace import trace
+from artemis.graph.state import State
+from artemis.services.llm import get_llm
+from artemis.utils.logger import get_logger
+from artemis.utils.notes import get_note_file_path
+from artemis.utils.ocr_xml_fusion import fuse_ocr_with_xml
+from artemis.utils.task_tree import (
+    build_plan_and_history,
+    format_action_clean,
+    get_active_subgoal_hashes,
+)
+from artemis.utils.visualization import format_minimal_list_with_elements
+
+
+class ValidationErrorCategory(StrEnum):
+    NONE = "none"
+    TARGET_DISAPPEARED = "target_disappeared"
+    TARGET_SHIFTED = "target_shifted"
+    TARGET_OCCUPIED = "target_occupied"
+    PIXEL_TARGET_DISAPPEARED = "pixel_target_disappeared"
+    PIXEL_BYPASSED = "pixel_bypassed"
+    XML_BYPASSED = "xml_bypassed"
+    GENERAL = "general"
+
+
+logger = get_logger(__name__)
+
+
+# ==============================================================================
+# Backward-Compatibility Helper Aliases
+# ==============================================================================
+
+
+async def _capture_screenshot_and_parse_ui(
+    ctx: ArtemisContext,
+    state: State,
+    controller: UnifiedMobileController,
+    skip_settling: bool = False,
+) -> tuple[str | None, bytes | None, str | None]:
+    return await capture_screenshot_and_parse_ui(
+        ctx, state, controller, skip_settling=skip_settling
+    )
+
+
+async def _exec_click(
+    target: list[int],
+    state: State,
+    controller: UnifiedMobileController,
+    ctx: ArtemisContext,
+    times: int = 1,
+    delay_ms: int = 100,
+) -> tuple[str, bytes | None, str | None, str | None]:
+    return await MobileActionExecutor(ctx, controller).exec_click(
+        target, state, times=times, delay_ms=delay_ms
+    )
+
+
+async def _exec_click_sequence(
+    sequence: list[int | list[int]],
+    state: State,
+    controller: UnifiedMobileController,
+    ctx: ArtemisContext,
+) -> tuple[str, bytes | None, str | None, str | None]:
+    return await MobileActionExecutor(ctx, controller).exec_click_sequence(sequence, state)
+
+
+async def _exec_long_press(
+    target: list[int],
+    state: State,
+    controller: UnifiedMobileController,
+    ctx: ArtemisContext,
+    duration_ms: int = 1000,
+    duration: int | None = None,
+) -> tuple[str, bytes | None, str | None, str | None]:
+    return await MobileActionExecutor(ctx, controller).exec_long_press(
+        target, state, duration_ms=duration_ms, duration=duration
+    )
+
+
+async def _exec_input_text(
+    text: str,
+    target: list[int] | None,
+    state: State,
+    controller: UnifiedMobileController,
+    ctx: ArtemisContext,
+    clear_exist: bool = True,
+) -> tuple[str, bytes | None, str | None, str | None]:
+    return await MobileActionExecutor(ctx, controller).exec_input_text(
+        text, target, state, clear_exist=clear_exist
+    )
+
+
+async def _exec_swipe(
+    action: str | list[int],
+    duration: int = 400,
+    state: State | None = None,
+    controller: UnifiedMobileController | None = None,
+    ctx: ArtemisContext | None = None,
+    duration_ms: int | None = None,
+) -> tuple[str, bytes | None, str | None, str | None]:
+    return await MobileActionExecutor(ctx, controller).exec_swipe(
+        action, duration=duration, state=state, duration_ms=duration_ms
+    )
+
+
+async def _exec_press_key(
+    key: str,
+    state: State,
+    controller: UnifiedMobileController,
+    ctx: ArtemisContext,
+) -> tuple[str, bytes | None, str | None, str | None]:
+    return await MobileActionExecutor(ctx, controller).exec_press_key(key, state)
+
+
+async def _exec_manage_app(
+    action: str,
+    app_name: str,
+    state: State,
+    controller: UnifiedMobileController,
+    ctx: ArtemisContext,
+) -> tuple[str, bytes | None, str | None, str | None]:
+    return await MobileActionExecutor(ctx, controller).exec_manage_app(action, app_name, state)
+
+
+async def _exec_wait_for_delay(
+    time_in_ms: int,
+    state: State,
+    controller: UnifiedMobileController,
+    ctx: ArtemisContext,
+) -> tuple[str, bytes | None, str | None, str | None]:
+    return await MobileActionExecutor(ctx, controller).exec_wait_for_delay(time_in_ms, state)
+
+
+async def _exec_wait_for_text(
+    text: str,
+    wait_state: str | None,
+    timeout_ms: int | None,
+    state: State,
+    controller: UnifiedMobileController,
+    ctx: ArtemisContext,
+) -> tuple[str, bytes | None, str | None, str | None]:
+    return await MobileActionExecutor(ctx, controller).exec_wait_for_text(
+        text, wait_state, timeout_ms, state
+    )
+
+
+async def _exec_read_note(
+    key: str,
+    ctx: ArtemisContext,
+    start_line: int | None = None,
+    end_line: int | None = None,
+) -> str:
+    return await MobileActionExecutor(ctx).exec_read_note(
+        key, start_line=start_line, end_line=end_line
+    )
+
+
+async def _exec_list_notes(ctx: ArtemisContext) -> str:
+    return await MobileActionExecutor(ctx).exec_list_notes()
+
+
+async def _exec_ask_explorer(
+    query: str,
+    context_feedback: str | None,
+    state: State,
+    ctx: ArtemisContext,
+    version: str = "flash",
+) -> str:
+    return await MobileActionExecutor(ctx).exec_ask_explorer(
+        query, context_feedback, state, version=version
+    )
+
+
+def _get_ask_explorer_tool_declaration() -> ToolDeclaration:
+    return ASK_EXPLORER_TOOL
+
+
+class FailureAnalysisStrategy:
+    """Base strategy class for FailureAnalyzer strategies."""
+
+    def get_prompt_template_name(self) -> str:
+        return "failure_analyzer.md"
+
+    def get_tools(self) -> list:
+        return VALIDATOR_TOOLS_DECLARATION
+
+    def get_user_message_suffix(self, pre_screenshot: bool, post_screenshot: bool) -> str:
+        suffix = (
+            "Your objective is:\n"
+            "1. Analyze why the action planned in the failed step failed, using the failed step's"
+            " details, the decision screenshot (Screenshot Seen During System Decision), the"
+            " latest failed screenshot (Latest Screenshot (Failed State)), and their corresponding"
+            " UI Element lists.\n"
+            "2. Resolve any timing, state mismatches (e.g., re-triggering disappeared elements), or"
+            " environmental obstacles using your execution tools.\n"
+            "3. Crucially: You MUST execute the failed action yourself using your execution tools"
+            " to completion.\n"
+            "4. Once you have successfully executed the repair and the failed action, call the"
+            " `report_failure_analysis` tool with status='fixed'.\n"
+            "5. If you are not confident or cannot resolve the failure, call"
+            " `report_failure_analysis` with status='cannot_fix'."
+        )
+        if pre_screenshot and post_screenshot:
+            suffix += (
+                "\nI have attached the screenshot seen when the system made its"
+                " decision (Screenshot Seen During System Decision) and the"
+                " latest screenshot in the failed state (Latest Screenshot"
+                " (Failed State))."
+            )
+        return suffix
+
+
+class TargetDisappearedStrategy(FailureAnalysisStrategy):
+    """Specialized strategy for target disappeared errors."""
+
+    def get_prompt_template_name(self) -> str:
+        return "target_disappeared_analyzer.md"
+
+    def get_tools(self) -> list:
+        tools = list(VALIDATOR_TOOLS_DECLARATION)
+        tools.insert(1, CLICK_SEQUENCE_TOOL)
+        tools.append(ASK_EXPLORER_TOOL)
+        return tools
+
+    def get_user_message_suffix(self, pre_screenshot: bool, post_screenshot: bool) -> str:
+        return ""
+
+
+class PixelTargetDisappearedStrategy(FailureAnalysisStrategy):
+    """Specialized strategy for pixel-level target disappeared errors."""
+
+    def get_prompt_template_name(self) -> str:
+        return "pixel_target_disappeared_analyzer.md"
+
+    def get_tools(self) -> list:
+        tools = list(VALIDATOR_TOOLS_DECLARATION)
+        tools.insert(1, CLICK_SEQUENCE_TOOL)
+        tools.append(ASK_EXPLORER_TOOL)
+        return tools
+
+    def get_user_message_suffix(self, pre_screenshot: bool, post_screenshot: bool) -> str:
+        return ""
+
+
+class FailureAnalyzer:
+    """Universal Multi-Model Failure Analyzer."""
+
+    def __init__(self, ctx: ArtemisContext):
+        self.ctx = ctx
+        self.max_iterations = 15
+
+    def _select_strategy(self, error_category: ValidationErrorCategory) -> FailureAnalysisStrategy:
+        if error_category == ValidationErrorCategory.PIXEL_TARGET_DISAPPEARED:
+            logger.info("Routing failure analysis to specialized PixelTargetDisappearedStrategy.")
+            return PixelTargetDisappearedStrategy()
+        if error_category == ValidationErrorCategory.TARGET_DISAPPEARED:
+            logger.info("Routing failure analysis to specialized TargetDisappearedStrategy.")
+            return TargetDisappearedStrategy()
+        logger.info("Routing failure analysis to DefaultAnalysisStrategy.")
+        return FailureAnalysisStrategy()
+
+    def _get_active_subgoal_hash(self) -> str:
+        if not self.ctx.data_engine:
+            return "default"
+        task_plan_path = get_note_file_path(self.ctx.data_engine.base_dir, "task_plan")
+        if not task_plan_path.exists():
+            return "default"
+        try:
+            content = task_plan_path.read_text(encoding="utf-8")
+            parent_hash, _ = get_active_subgoal_hashes(content)
+            return parent_hash
+        except Exception as e:
+            logger.error(f"Failed to parse active subgoal: {e}")
+        return "default"
+
+    def _build_plan_and_history(self, steps: list[dict], task_plan: str) -> str:
+        active_subgoal_hash = self._get_active_subgoal_hash()
+        return build_plan_and_history(
+            task_plan,
+            steps,
+            active_subgoal_hash,
+            last_n_detailed=1,
+            strict_milestone_pruning=True,
+            recent_window_size=3,
+            chronological_last_step=True,
+            for_failure_analyzer=True,
+        )
+
+    def _prune_intermediate_screenshots(self, messages: list[Any]):
+        """Prunes binary screenshot blocks from intermediate messages."""
+        prune_intermediate_screenshots(messages)
+
+    @trace(type="agent", name="failure_analyzer")
+    async def analyze(
+        self,
+        state: State,
+        failed_action: dict,
+        error_msg: str,
+        pre_screenshot: str | None = None,
+        post_screenshot: str | None = None,
+        pre_screenshot_name: str | None = None,
+        post_screenshot_name: str | None = None,
+        executed_actions: list[dict] | None = None,
+        unexecuted_actions: list[dict] | None = None,
+        error_category: ValidationErrorCategory = ValidationErrorCategory.GENERAL,
+    ) -> dict:
+        if not post_screenshot or post_screenshot == "None":
+            try:
+                controller = UnifiedMobileController(self.ctx)
+                post_screenshot = await controller.take_screenshot()
+            except Exception as e:
+                logger.warning(f"Failed to capture post-action fallback screenshot: {e}")
+
+        if not pre_screenshot or pre_screenshot == "None":
+            pre_screenshot = post_screenshot
+
+        if not pre_screenshot:
+            raise ValueError("Failure analysis requires valid screenshots.")
+
+        strategy = self._select_strategy(error_category)
+        prompt_template_name = strategy.get_prompt_template_name()
+        tools_declaration = strategy.get_tools()
+
+        try:
+            llm = get_llm(self.ctx, name="validator_failure_analyzer")
+        except Exception:
+            llm = get_llm(self.ctx, name="validator")
+
+        controller = UnifiedMobileController(self.ctx)
+        executor = MobileActionExecutor(self.ctx, controller)
+
+        # Pre-Action and Post-Action XML formatting
+        pre_xml_list_str = "Not available."
+        if hasattr(state, "latest_ui_hierarchy") and state.latest_ui_hierarchy:
+            try:
+                width = getattr(self.ctx.device, "device_width", 1080)
+                height = getattr(self.ctx.device, "device_height", 2400)
+                pre_xml_list_str, _, _ = format_minimal_list_with_elements(
+                    state.latest_ui_hierarchy, width, height
+                )
+            except Exception as e:
+                logger.warning(f"Failed to format pre-action XML list: {e}")
+
+        post_xml_list_str = "Not available."
+        try:
+            xml_hierarchy = await controller.get_ui_elements()
+            width = getattr(self.ctx.device, "device_width", 1080)
+            height = getattr(self.ctx.device, "device_height", 2400)
+            ocr_results = []
+
+            fused_xml = fuse_ocr_with_xml(xml_hierarchy, ocr_results)
+
+            post_xml_list_str, elements, _ = format_minimal_list_with_elements(
+                fused_xml, width, height
+            )
+            state.indexed_points = [el["center"] for el in elements]
+            state.indexed_elements = elements
+        except Exception as e:
+            logger.warning(f"Failed to fetch/format post-action XML list: {e}")
+
+        prompt_path = Path(__file__).parent.joinpath(prompt_template_name)
+        prompt_template = prompt_path.read_text(encoding="utf-8")
+
+        steps = []
+        task_plan = "No task plan yet."
+        if self.ctx.data_engine:
+            steps = self.ctx.data_engine.get_agent_friendly_steps()
+            task_plan_path = get_note_file_path(self.ctx.data_engine.base_dir, "task_plan")
+            if task_plan_path.exists():
+                try:
+                    task_plan = task_plan_path.read_text(encoding="utf-8")
+                except Exception as e:
+                    task_plan = f"Error reading task plan: {e}"
+
+        plan_and_history = self._build_plan_and_history(steps, task_plan)
+        failed_step_number = steps[-1]["step_number"] if steps else 1
+        failed_action_description = (
+            format_action_clean(failed_action) if failed_action else "Planned Action"
+        )
+
+        system_prompt = Template(prompt_template).render(
+            initial_goal=getattr(state, "initial_goal", "Unknown Goal"),
+            plan_and_history=plan_and_history,
+            failed_step_number=failed_step_number,
+            failed_action_description=failed_action_description,
+        )
+
+        suffix = strategy.get_user_message_suffix(bool(pre_screenshot), bool(post_screenshot))
+        user_msg = (
+            suffix.strip()
+            if suffix
+            else "Please analyze and repair the failure shown in the last step of the history."
+        )
+        is_pixel_strategy = isinstance(strategy, PixelTargetDisappearedStrategy)
+
+        user_content: list[dict] = [{"type": "text", "text": user_msg}]
+        if pre_screenshot:
+            user_content.append(
+                {"type": "text", "text": "--- Screenshot Seen During System Decision ---"}
+            )
+            user_content.append(
+                {
+                    "type": "image_url",
+                    "image_url": {"url": f"data:image/jpeg;base64,{pre_screenshot}"},
+                }
+            )
+            if not is_pixel_strategy:
+                user_content.append(
+                    {
+                        "type": "text",
+                        "text": (
+                            "--- UI Element List Seen During System Decision ---\n"
+                            f"{pre_xml_list_str}"
+                        ),
+                    }
+                )
+        if post_screenshot:
+            user_content.append(
+                {"type": "text", "text": "--- Latest Screenshot (Failed State) ---"}
+            )
+            user_content.append(
+                {
+                    "type": "image_url",
+                    "image_url": {"url": f"data:image/jpeg;base64,{post_screenshot}"},
+                }
+            )
+            if not is_pixel_strategy:
+                user_content.append(
+                    {
+                        "type": "text",
+                        "text": (
+                            f"--- Latest UI Element List (Failed State) ---\n{post_xml_list_str}"
+                        ),
+                    }
+                )
+        if suffix:
+            user_content.append(
+                {"type": "text", "text": f"\n\n--- INSTRUCTIONS ---\n{suffix.strip()}"}
+            )
+
+        messages: list[BaseMessage] = [
+            SystemMessage(content=system_prompt),
+            HumanMessage(content=user_content),
+        ]
+
+        max_iterations = getattr(self, "max_iterations", 15)
+        iterations = 0
+        final_report = None
+        intermediate_thoughts = []
+
+        while iterations < max_iterations:
+            iterations += 1
+            self._prune_intermediate_screenshots(messages)
+            logger.info(f"Iteration {iterations}: Invoking Universal LLM for FailureAnalyzer...")
+
+            if iterations == max_iterations:
+                messages.append(
+                    HumanMessage(
+                        content=(
+                            "[WARNING] This is your final iteration; only"
+                            " 'report_failure_analysis' is available."
+                        )
+                    )
+                )
+                current_tools = [
+                    t for t in tools_declaration if t.name == "report_failure_analysis"
+                ]
+            else:
+                current_tools = tools_declaration
+
+            bound_llm = llm.bind_tools(current_tools)
+
+            max_retries = 3
+            response = None
+            for attempt in range(max_retries):
+                try:
+                    response = await bound_llm.ainvoke(messages)
+                    break
+                except Exception as e:
+                    logger.warning(f"Failure analyzer turn failed on attempt {attempt + 1}: {e}")
+                    if attempt == max_retries - 1:
+                        raise e
+                    await asyncio.sleep(1.0 * (2**attempt))
+
+            if response is None:
+                break
+
+            messages.append(response)
+
+            raw_text = response.content if isinstance(response.content, str) else ""
+            if isinstance(response.content, list):
+                raw_text = "".join(
+                    b.get("text", "")
+                    for b in response.content
+                    if isinstance(b, dict) and "text" in b
+                )
+            if raw_text:
+                intermediate_thoughts.append(raw_text)
+
+            tool_calls = response.tool_calls or []
+            if not tool_calls:
+                messages.append(
+                    HumanMessage(
+                        content=(
+                            "CRITICAL: You did not call the `report_failure_analysis` tool."
+                            " You MUST call an action tool to repair or `report_failure_analysis`"
+                            " to finish."
+                        )
+                    )
+                )
+                continue
+
+            submit_call = next(
+                (tc for tc in tool_calls if tc["name"].split(":")[-1] == "report_failure_analysis"),
+                None,
+            )
+
+            if submit_call:
+                args = submit_call.get("args") or {}
+                actions = args.get("new_remaining_actions", [])
+                final_report = {
+                    "status": args.get("status", "cannot_fix"),
+                    "analysis": args.get("analysis", ""),
+                    "new_remaining_actions": actions,
+                    "thoughts": intermediate_thoughts,
+                }
+                messages.append(
+                    ToolMessage(
+                        tool_call_id=submit_call.get("id") or str(uuid.uuid4()),
+                        name=submit_call["name"],
+                        content=json.dumps({"status": "acknowledged"}),
+                        status="success",
+                    )
+                )
+                break
+
+            # Execute other tool calls via MobileActionExecutor
+            for tc in tool_calls:
+                name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
+                args = tc.get("args") or {}
+                tc_id = tc.get("id") or str(uuid.uuid4())
+                logger.info(f"Failure analyzer executing tool '{name}'...")
+
+                try:
+                    exec_result = await executor.execute(name, args, tc_id, state)
+                    messages.append(exec_result.to_langchain_tool_message())
+                except Exception as e:
+                    logger.error(f"Error executing tool {name}: {e}")
+                    messages.append(
+                        ToolMessage(
+                            tool_call_id=tc_id,
+                            name=name,
+                            content=f"Error executing tool {name}: {e}",
+                            status="error",
+                        )
+                    )
+
+        return final_report or {
+            "status": "cannot_fix",
+            "analysis": "Max iterations reached without resolution.",
+        }

artemis/agents/validator/failure_analyzer_agent.py

@@ -0,0 +1,69 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Modular Failure Analyzer Agent adhering to BaseAgent lifecycle."""
+
+from typing import Any
+from artemis.agents.base import AgentConfig, AgentRegistry, BaseAgent
+from artemis.agents.validator.recovery_strategies import RecoveryStrategyRegistry
+from artemis.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+@AgentRegistry.register("failure_analyzer")
+class FailureAnalyzerAgent(BaseAgent):
+    """Specialized agent responsible for diagnosing step failures and triggering autonomous recovery."""
+
+    def __init__(
+        self,
+        config: AgentConfig | None = None,
+        ctx: Any | None = None,
+        **kwargs: Any,
+    ):
+        super().__init__(
+            name="failure_analyzer",
+            role="Autonomous Self-Healing Failure Analyzer",
+            description="Diagnoses execution exceptions, analyzes screen discrepancies, and runs self-healing sequences.",
+            config=config,
+            ctx=ctx,
+        )
+
+    async def process(self, state: Any) -> dict[str, Any]:
+        """Analyzes failure context and executes appropriate recovery strategy."""
+        if not self.driver:
+            raise RuntimeError("FailureAnalyzerAgent requires an active device driver in context.")
+
+        failure_context = getattr(state, "failure_context", {}) or {}
+        if isinstance(state, dict):
+            failure_context = state.get("failure_context", {})
+
+        logger.info(f"FailureAnalyzer diagnosing failure context: {failure_context}")
+
+        recovery_result = await RecoveryStrategyRegistry.attempt_recovery(
+            driver=self.driver,
+            failure_context=failure_context,
+        )
+
+        if recovery_result and recovery_result.success:
+            return {
+                "status": "recovered",
+                "recovery_action": recovery_result.action_type.value,
+                "reason": recovery_result.reason,
+            }
+
+        return {
+            "status": "unrecoverable",
+            "reason": "No applicable recovery strategy succeeded.",
+        }

artemis/agents/validator/pixel_safety_net.md

@@ -0,0 +1,11 @@
+Role
+UI State Verification Expert. Compare Image 1 (Reference) and Image 2 (Current State) to check if the target clicked by the Operator at the red dot is still present and interactive.
+
+Rules
+1. Check Intent: Read the `[Planned Action & Original Thinking]` to identify what Action the Agent intended to perform and why.
+2. Specific UI Elements (Priority): If targeting a specific UI control (e.g., button, icon, text, slider handle) — even if it overlays a dynamic video, map, or live stream — focus ONLY on that element's shape. Output `{"is_present": false, ...}` if that exact element has disappeared, shifted away, or is blocked. Ignore background media changes underneath.
+3. Blank Space & Backgrounds: If targeting open/blank space (e.g., to wake up hidden controls or dismiss dialogs), ignore natural video/background frame changes. Output `{"is_present": true, "confidence": 1.0}` unless unexpected blocking popups appear.
+
+Output
+Return EXACTLY this raw JSON and nothing else. Keep reasoning strictly under 25 words.
+{"reasoning": "brief explanation", "is_present": boolean, "confidence": float}

artemis/agents/validator/pixel_target_disappeared_analyzer.md

@@ -0,0 +1,65 @@
+# Identity & Goal
+You are a specialized error recovery agent. You will be provided with a failed action. Your ONLY task is to repair this specific failure and restore the system state so the main system can resume.
+
+
+# Latency Awareness Principle
+You operate in an environment with unavoidable "Turn Latency" (a delay of several seconds between screen capture, decision making, and tool execution). 
+Consequently, any transient UI element (e.g., auto-dismissing overlays, temporary message bars, menus with short timeouts) WILL disappear before your next decision turn if you attempt to interact with them step-by-step.
+
+# Context
+- **Initial Goal**: {{ initial_goal }}
+- **History & Plan**:
+{{ plan_and_history }}
+
+# Core Workflow
+1. **Analyze**: Compare the two states provided to you:
+   - **Decision State (`Screenshot Seen During System Decision`)**: This is the screen the system saw when it originally planned the action. It shows the target element successfully located.
+   - **Failed State (`Latest Screenshot (Failed State)`)**: This is the live screen captured at the moment you were summoned. It shows the current state where the target is missing or unresponsive.
+   Your goal in this comparison is to:
+   - Read the Operator's thought process in the decision loop of the failed step to understand what they were originally trying to accomplish with this failed action.
+   - Understand the expected target layout.
+   - Identify the specific visual pattern (e.g., a play triangle, a close 'X', an arrow icon) at the failed target area from the reference state.
+   - Identify why this target element visually disappeared or is no longer responsive (use the **Diagnosis Tips** below).
+2. **Recover & Execute**: Restore the target state and execute the failed action.
+   - **Flexibility**: Focus on outcome over form. Take alternative/adaptive steps (e.g., scroll, navigate, different click paths) if original target positions are invalid.
+   - **Transient State Handling**: If the failed target is part of a transient state, you MUST NOT execute the recovery and target interaction in separate turns.
+   - **Atomic Chained Execution**: Combine the summon action (on the Trigger Element) and the target action (on the Transient Target) into a single atomic sequence. Use `click_sequence(sequence=[Trigger, Target])` to execute them with a 50ms delay, bypassing the Turn Latency gap.
+3. **Report**: Call `report_failure_analysis` to conclude:
+   - `status="fixed"`: Successfully executed the failed action or reached the destination state.
+   - `status="cannot_fix"`: Target is unrecoverable, the app is broken, or the 15-step budget is exceeded.
+   - `analysis`: Summarize visual mismatches and actions taken.
+
+# Special Tools
+- **`click_sequence(sequence)`**: Designed to defeat Turn Latency. Chain your click sequence to handle transient elements.
+- **`ask_explorer(query, context_feedback)`**: Call this visual parsing subagent to search for the visual icon/pattern on the current screen (e.g., `query="play icon"`) if you suspect the target has shifted, or coordinates are ambiguous.
+
+# Diagnosis Tips
+*   **Time-Sensitive States (MOST COMMON & PRIMARY CAUSE)**:
+    - *Cue*: The target element was visible when the system made its decision, but has disappeared in the failed state because it belongs to an auto-closing, short-timeout, or transient UI state.
+    - *Action*: Re-trigger the state dynamically from the current screen and chain it with the failed step in a rapid consecutive sequence.
+      - **Mental Model**: Tap the persistent summoner element (Trigger) that wakes up the transient UI + Tap the target coordinates (Failed Target) before it has time to auto-close, combining both into a single atomic movement.
+      - **Procedure**:
+        1. **Locate the Trigger (Target 1)**: Inspect the current screen to identify a persistent parent element or background area (e.g., the container element or parent dropdown header) that, when tapped, forces the transient UI state to reappear.
+        2. **Retrieve the Failed Target (Target 2)**: Look up the coordinates of the FAILED action from the failed step in the history (since the target button is currently invisible on your screen).
+        3. **Chained Execution**: Call `click_sequence(sequence=[Target 1, Target 2])` to execute both taps consecutively with a 50ms delay, catching the transient element before the timeout closes it. *(Note: This sequence fulfills your workflow requirement to "execute the failed action".)*
+*   **False Alarm**:
+    - *Cue*: Target element is actually present on the current screen (minor validation flake).
+    - *Action*: Directly report it as `"fixed"` and hand control back to the system.
+*   **Visual Target Shifts**:
+    - *Cue*: The target visual icon has shifted due to screen layout changes, orientations, or dynamic list updates.
+    - *Action*: Visually locate the icon on the live screenshot and click its new coordinates. You may call `ask_explorer(query="describe the visual icon shape or labels")` to dynamically locate it and obtain the new coordinates if they are hard to estimate manually.
+*   **Overlays**:
+    - *Cue*: Unexpected permission prompts, dialogs, or system popups block the screen.
+    - *Action*: Tap outside or click the dismiss/close buttons to clear the overlay.
+*   **Keyboard & Scroll Positioning**:
+    - *Cue*: Target is scrolled off-screen or blocked by the on-screen keyboard.
+    - *Action*: Scroll to reposition the target, or hide the keyboard.
+*   **Navigation Drift**:
+    - *Cue*: App unexpectedly transitioned to a different tab or screen.
+    - *Action*: Navigate back to the correct target screen.
+*   **App Instability**:
+    - *Cue*: Application crashed, froze, or became unresponsive.
+    - *Action*: Force stop and relaunch the app.
+
+# Constraints
+The most recent step (Step {{ failed_step_number }}: {{ failed_action_description }}) failed to execute. You must resolve the issue and report the results in 15 interactive steps or fewer. Now please begin the repair analysis. Please note that during the **Recover & Execute** phase, you only need to resolve the currently failed step; do not attempt to execute any subsequent steps or complete the overall goal yourself. Once the failure is resolved, please immediately enter the **Report** phase and return control to the system.

artemis/agents/validator/recovery_strategies.py

@@ -0,0 +1,177 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Modular Recovery Strategies for the Validation and Self-Healing Engine."""
+
+from abc import ABC, abstractmethod
+from enum import Enum
+from typing import Any
+from artemis.drivers.base import BaseDeviceDriver
+from artemis.utils.logger import get_logger
+from pydantic import BaseModel, Field
+
+logger = get_logger(__name__)
+
+
+class RecoveryActionType(str, Enum):
+    RETRY_DYNAMIC_LOCATOR = "retry_dynamic_locator"
+    DISMISS_DIALOG = "dismiss_dialog"
+    NAVIGATE_BACK = "navigate_back"
+    SCROLL_AND_RETRY = "scroll_and_retry"
+    RESTART_APP = "restart_app"
+    FALLBACK_COORDINATES = "fallback_coordinates"
+
+
+class RecoveryResult(BaseModel):
+    """Result of executing an autonomous self-healing recovery strategy."""
+
+    success: bool
+    action_type: RecoveryActionType
+    reason: str
+    metadata: dict[str, Any] = Field(default_factory=dict)
+
+
+class BaseRecoveryStrategy(ABC):
+    """Abstract interface for modular self-healing failure recovery tactics."""
+
+    @property
+    @abstractmethod
+    def name(self) -> str:
+        """Strategy identifier."""
+        ...
+
+    @abstractmethod
+    async def can_handle(self, failure_context: dict[str, Any]) -> bool:
+        """Determines if this strategy applies to the observed failure mode."""
+        ...
+
+    @abstractmethod
+    async def execute(
+        self,
+        driver: BaseDeviceDriver,
+        failure_context: dict[str, Any],
+    ) -> RecoveryResult:
+        """Executes the self-healing corrective actions."""
+        ...
+
+
+class DismissDialogRecoveryStrategy(BaseRecoveryStrategy):
+    """Strategy to detect and dismiss unexpected modal dialogs, popups, or permission prompts."""
+
+    @property
+    def name(self) -> str:
+        return "dismiss_dialog"
+
+    async def can_handle(self, failure_context: dict[str, Any]) -> bool:
+        category = failure_context.get("category", "")
+        error_msg = failure_context.get("error_message", "").lower()
+        return (
+            category == "ui_obstruction"
+            or "permission" in error_msg
+            or "dialog" in error_msg
+            or "popup" in error_msg
+        )
+
+    async def execute(
+        self,
+        driver: BaseDeviceDriver,
+        failure_context: dict[str, Any],
+    ) -> RecoveryResult:
+        logger.info(
+            "[Recovery] Attempting to dismiss unexpected dialog or overlay via Back button..."
+        )
+        success = await driver.press_key("back")
+        return RecoveryResult(
+            success=success,
+            action_type=RecoveryActionType.DISMISS_DIALOG,
+            reason="Sent BACK keyevent to dismiss modal overlay",
+        )
+
+
+class NavigateBackRecoveryStrategy(BaseRecoveryStrategy):
+    """Strategy to navigate back when agent navigates to a wrong screen."""
+
+    @property
+    def name(self) -> str:
+        return "navigate_back"
+
+    async def can_handle(self, failure_context: dict[str, Any]) -> bool:
+        category = failure_context.get("category", "")
+        return category in ("wrong_screen", "dead_end", "navigation_error")
+
+    async def execute(
+        self,
+        driver: BaseDeviceDriver,
+        failure_context: dict[str, Any],
+    ) -> RecoveryResult:
+        logger.info("[Recovery] Wrong screen detected, rolling back navigation...")
+        success = await driver.press_key("back")
+        return RecoveryResult(
+            success=success,
+            action_type=RecoveryActionType.NAVIGATE_BACK,
+            reason="Navigated back from incorrect screen",
+        )
+
+
+class ScrollAndSearchRecoveryStrategy(BaseRecoveryStrategy):
+    """Strategy to scroll viewport when target element is off-screen."""
+
+    @property
+    def name(self) -> str:
+        return "scroll_and_search"
+
+    async def can_handle(self, failure_context: dict[str, Any]) -> bool:
+        category = failure_context.get("category", "")
+        return category in ("element_not_found", "off_screen", "target_missing")
+
+    async def execute(
+        self,
+        driver: BaseDeviceDriver,
+        failure_context: dict[str, Any],
+    ) -> RecoveryResult:
+        direction = failure_context.get("scroll_direction", "down")
+        logger.info(f"[Recovery] Target missing, scrolling {direction} to reveal element...")
+        success = await driver.swipe_direction(direction=direction)
+        return RecoveryResult(
+            success=success,
+            action_type=RecoveryActionType.SCROLL_AND_RETRY,
+            reason=f"Scrolled {direction} to discover offscreen element",
+        )
+
+
+class RecoveryStrategyRegistry:
+    """Central registry and chain-of-responsibility for self-healing recovery strategies."""
+
+    _strategies: list[BaseRecoveryStrategy] = [
+        DismissDialogRecoveryStrategy(),
+        NavigateBackRecoveryStrategy(),
+        ScrollAndSearchRecoveryStrategy(),
+    ]
+
+    @classmethod
+    def register(cls, strategy: BaseRecoveryStrategy) -> None:
+        cls._strategies.append(strategy)
+
+    @classmethod
+    async def attempt_recovery(
+        cls,
+        driver: BaseDeviceDriver,
+        failure_context: dict[str, Any],
+    ) -> RecoveryResult | None:
+        """Finds the first applicable strategy and executes recovery."""
+        for strategy in cls._strategies:
+            if await strategy.can_handle(failure_context):
+                logger.info(f"Applying recovery strategy: {strategy.name}")
+                return await strategy.execute(driver=driver, failure_context=failure_context)
+        return None

artemis/agents/validator/safety_net.py

@@ -0,0 +1,48 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Safety net and system dialog auto-recovery."""
+
+from typing import Any
+from artemis.drivers.base import BaseDeviceDriver
+from artemis.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+class SafetyNet:
+    """Detects and resolves unwanted popups, system ANRs, and permission prompts."""
+
+    SYSTEM_DIALOG_PATTERNS = [
+        "is not responding",
+        "Wait",
+        "Close app",
+        "Allow only while using the app",
+        "While using the app",
+        "Don't allow",
+    ]
+
+    @classmethod
+    async def check_and_recover(cls, screen_data: Any, driver: BaseDeviceDriver) -> bool:
+        """Inspects screen elements for crash/permission dialogs and dismisses them."""
+        for element in screen_data.ui_elements:
+            text = element.get("text", "")
+            if any(pattern.lower() in text.lower() for pattern in cls.SYSTEM_DIALOG_PATTERNS):
+                logger.info(
+                    f"SafetyNet detected system dialog '{text}', attempting auto-dismiss..."
+                )
+                center = element.get("center", [500, 500])
+                await driver.tap(center[0], center[1])
+                return True
+        return False

artemis/agents/validator/target_disappeared_analyzer.md

@@ -0,0 +1,61 @@
+# Identity & Goal
+You are a specialized error recovery agent. You will be provided with a failed action. Your ONLY task is to repair this specific failure and restore the system state so the main system can resume.
+
+
+# Latency Awareness Principle
+You operate in an environment with unavoidable "Turn Latency" (a delay of several seconds between screen capture, decision making, and tool execution). 
+Consequently, any transient UI element (e.g., auto-dismissing overlays, temporary message bars, menus with short timeouts) WILL disappear before your next decision turn if you attempt to interact with them step-by-step.
+
+# Context
+- **Initial Goal**: {{ initial_goal }}
+- **History & Plan**:
+{{ plan_and_history }}
+
+# Core Workflow
+1. **Analyze**: Compare the two states provided to you:
+   - **Decision State (`Screenshot Seen During System Decision`)**: This is the screen the system saw when it originally planned the action. It shows the target element successfully located.
+   - **Failed State (`Latest Screenshot (Failed State)`)**: This is the live screen captured at the moment you were summoned. It shows the current state where the target is missing or unresponsive.
+   Your goal in this comparison is to:
+   - Read the Operator's thought process in the decision loop of the failed step to understand what they were originally trying to accomplish with this failed action.
+   - Understand the expected target layout.
+   - Identify why the target element disappeared (use the **Diagnosis Tips** below).
+2. **Recover & Execute**: Restore the target state and execute the failed action.
+   - **Flexibility**: Focus on outcome over form. Take alternative/adaptive steps (e.g., scroll, navigate, different click paths) if original coordinates are invalid.
+   - **Transient State Handling**: If the failed target is part of a transient state, you MUST NOT execute the recovery and target interaction in separate turns.
+   - **Atomic Chained Execution**: Combine the summon action (on the Trigger Element) and the target action (on the Transient Target) into a single atomic sequence. Use `click_sequence(sequence=[Trigger, Target])` to execute them with a 50ms delay, bypassing the Turn Latency gap.
+3. **Report**: Call `report_failure_analysis` to conclude:
+   - `status="fixed"`: Successfully executed the failed action or reached the destination state.
+   - `status="cannot_fix"`: Target is unrecoverable, the app is broken, or the 15-step budget is exceeded.
+   - `analysis`: Summarize visual mismatches and actions taken.
+
+# Special Tools
+- **`click_sequence(sequence)`**: Designed to defeat Turn Latency. Chain your click sequence to handle transient elements.
+- **`ask_explorer(query, context_feedback)`**: Call this visual parsing subagent to search for the visual icon/pattern on the current screen if the target is absent from the standard "UI Element List" but you see it is visually present on the screenshot.
+
+# Diagnosis Tips
+*   **Time-Sensitive States (MOST COMMON & PRIMARY CAUSE)**:
+    - *Cue*: The target element was visible when the system made its decision, but has disappeared in the failed state because it belongs to an auto-closing, short-timeout, or transient UI state.
+    - *Action*: Re-trigger the state dynamically from the current screen and chain it with the failed step in a rapid consecutive sequence.
+      - **Mental Model**: Tap the persistent summoner element (Trigger) that wakes up the transient UI + Tap the target coordinates (Failed Target) before it has time to auto-close, combining both into a single atomic movement.
+      - **Procedure**:
+        1. **Locate the Trigger (Target 1)**: Inspect the current screen to identify a persistent parent element or background area (e.g., the container element or parent dropdown header) that, when tapped, forces the transient UI state to reappear.
+        2. **Retrieve the Failed Target (Target 2)**: Look up the coordinates of the FAILED action from the failed step in the history (since the target button is currently invisible on your screen).
+        3. **Chained Execution**: Call `click_sequence(sequence=[Target 1, Target 2])` to execute both taps consecutively with a 50ms delay, catching the transient element before the timeout closes it. *(Note: This sequence fulfills your workflow requirement to "execute the failed action".)*
+*   **False Alarm**:
+    - *Cue*: Target element is actually present on the current screen (minor validation flake).
+    - *Action*: Directly report it as `"fixed"` and hand control back to the system.
+*   **Overlays**:
+    - *Cue*: Unexpected permission prompts, dialogs, or system popups block the screen.
+    - *Action*: Tap outside or click the dismiss/close buttons to clear the overlay.
+*   **Keyboard & Scroll Positioning**:
+    - *Cue*: Target is scrolled off-screen or blocked by the on-screen keyboard.
+    - *Action*: Scroll to reposition the target, or hide the keyboard.
+*   **Navigation Drift**:
+    - *Cue*: App unexpectedly transitioned to a different tab or screen.
+    - *Action*: Navigate back to the correct target screen.
+*   **App Instability**:
+    - *Cue*: Application crashed, froze, or became unresponsive.
+    - *Action*: Force stop and relaunch the app.
+
+# Constraints
+The most recent step (Step {{ failed_step_number }}: {{ failed_action_description }}) failed to execute. You must resolve the issue and report the results in 15 interactive steps or fewer. Now please begin the repair analysis. Please note that during the **Recover & Execute** phase, you only need to resolve the currently failed step; do not attempt to execute any subsequent steps or complete the overall goal yourself. Once the failure is resolved, please immediately enter the **Report** phase and return control to the system.

artemis/agents/video_analyzer/__init__.py

@@ -0,0 +1,15 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Video analyzer utility for analyzing video content with Gemini models."""

artemis/agents/video_analyzer/conflict_resolution.py

@@ -0,0 +1,137 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+logger = logging.getLogger(__name__)
+
+
+class ConflictResolutionService:
+    @staticmethod
+    def calculate_1d_iou(a, b):
+        start = max(a[0], b[0])
+        end = min(a[1], b[1])
+        intersection = max(0.0, end - start)
+        union = (a[1] - a[0]) + (b[1] - b[0]) - intersection
+        return intersection / (union + 1e-6)
+
+    @staticmethod
+    def calculate_1d_iomin(a, b):
+        start = max(a[0], b[0])
+        end = min(a[1], b[1])
+        intersection = max(0.0, end - start)
+        min_len = min(a[1] - a[0], b[1] - b[0])
+        if min_len <= 0:
+            return 0.0
+        return intersection / min_len
+
+    @staticmethod
+    def sanitize_interval(interval):
+        start = interval.get("start", 0.0)
+        end = interval.get("end", "unknown")
+
+        if isinstance(start, (int, float)):
+            start = max(0.0, float(start))
+
+        if end != "unknown" and isinstance(end, (int, float)):
+            end = max(0.0, float(end))
+            if start > end:
+                start, end = end, start
+
+        interval["start"] = start
+        interval["end"] = end
+
+        if "confidence_score" not in interval or interval["confidence_score"] is None:
+            interval["confidence_score"] = 0.0
+
+        if end != "unknown":
+            interval["duration"] = float(end) - float(start)
+        else:
+            interval["duration"] = 0.0
+
+        return interval
+
+    @staticmethod
+    def group_by_semantic_class(proposals):
+        groups = {}
+        for p in proposals:
+            target = p.get("target", "unknown")
+            if target not in groups:
+                groups[target] = []
+            groups[target].append(p)
+        return groups
+
+    @staticmethod
+    def apply_1d_nms(group_proposals):
+        if len(group_proposals) <= 1:
+            return group_proposals
+
+        unknown_end_proposals = [p for p in group_proposals if p.get("end") == "unknown"]
+        valid_proposals = [p for p in group_proposals if p.get("end") != "unknown"]
+
+        valid_proposals.sort(
+            key=lambda x: (
+                x.get("confidence_score", 0.0),
+                -x.get("duration", 0.0),
+            ),
+            reverse=True,
+        )
+
+        n = len(valid_proposals)
+        suppressed_mask = [False] * n
+
+        for i in range(n):
+            if suppressed_mask[i]:
+                continue
+            a = (valid_proposals[i]["start"], valid_proposals[i]["end"])
+            for j in range(i + 1, n):
+                if suppressed_mask[j]:
+                    continue
+                b = (valid_proposals[j]["start"], valid_proposals[j]["end"])
+                iou = ConflictResolutionService.calculate_1d_iou(a, b)
+                iomin = ConflictResolutionService.calculate_1d_iomin(a, b)
+
+                if iou > 0.5 or iomin > 0.8:
+                    suppressed_mask[j] = True
+
+        kept_proposals = [valid_proposals[i] for i in range(n) if not suppressed_mask[i]]
+        return kept_proposals + unknown_end_proposals
+
+    @classmethod
+    def clean(cls, blackboard_entries):
+        try:
+            sanitized = [cls.sanitize_interval(dict(e)) for e in blackboard_entries]
+            groups = cls.group_by_semantic_class(sanitized)
+
+            cleaned_entries = []
+            for target, proposals in groups.items():
+                if len(proposals) <= 1:
+                    cleaned_entries.extend(proposals)
+                else:
+                    cleaned_entries.extend(cls.apply_1d_nms(proposals))
+
+            suppressed_count = len(blackboard_entries) - len(cleaned_entries)
+            if suppressed_count > 0:
+                logger.info(
+                    "NMS filtering telemetry: Successfully suppressed"
+                    f" {suppressed_count} redundant proposals."
+                )
+
+            # Sort chronologically by start time before returning so downstream systems read it in order
+            cleaned_entries.sort(key=lambda x: float(x.get("start", 0.0)))
+
+            return cleaned_entries
+        except Exception as e:
+            logger.error(f"NMS filter crashed: {e}. Falling back to raw ledger state.")
+            return blackboard_entries

artemis/agents/video_analyzer/video_analyzer.md

@@ -0,0 +1,51 @@
+# Dynamic Video Analyzer - Main Agent
+
+You are the Main Agent of the Dynamic Video Analyzer system. Your goal is to process video files of varying lengths and sizes to answer user queries efficiently, avoiding context bloat.
+
+You receive:
+1. **Time Description**: A description of the time range of interest.
+2. **Purpose**: The specific goal or question about the video.
+
+---
+
+## Your Capabilities
+
+You are equipped with specialized tools (`extract_segment_metadata`, `spawn_sub_agent`, `analyze_audio_only`) to extract metadata from video segments and delegate targeted multimodal or audio-only analysis to fault-tolerant sub-agents.
+
+
+---
+
+## Execution Logic & Routing Strategy
+
+### 1. Routing between Video and Audio Analysis
+**Probing Metadata**: If you are uncertain about the file size or duration of the requested timeframe, you **should first call `extract_segment_metadata`** to probe the metadata. Use the returned `file_size_mb` and `duration_seconds` to decide whether to proceed with a visual sub-agent, split the request, or fall back to audio-only.
+
+You must carefully choose the appropriate tool based on the nature of the task:
+
+- **Must use Visual Analysis (`spawn_sub_agent`)**:
+  - **UI Interactions**: Identifying clicks, gestures, typing, navigation.
+  - **Visual State**: Reading on-screen text (OCR), checking UI status (toggles, dialogs, loading states).
+  - **Visual Content**: Identifying images, layout issues, animations.
+- **Can use Audio-Only Analysis (`analyze_audio_only`)**:
+  - **Speech-centric**: Summarizing spoken content, identifying speakers, transcription.
+  - **Audio cues**: Detecting specific sounds, music, notification chimes.
+  - *Constraint*: Only use for long videos where visual info is strictly unnecessary for the query.
+
+### 2. Formulating Specific Prompts for Sub-Agents
+When calling `spawn_sub_agent` or `analyze_audio_only`, the `specific_query` must be clear, detailed, and actionable.
+- **Guideline**: Clearly state what to look/listen for and the expected format. Avoid generic instructions.
+- **Examples**:
+  - *Bad*: "Analyze this segment."
+  - *Good (Audio)*: "Identify the main topic discussed by the speaker and summarize their key conclusions."
+  - *Good (Video)*: "Observe if the user clicks the 'Confirm' button and verify if the screen transitions to the success page."
+
+### Dynamic Adaptation & Error Handling
+- **Be Flexible**: Do not be rigid about the paths. If you start with Strategy A but find the file size is too large after cropping, switch to Strategy B immediately.
+- **Clarification on Ambiguity**: If the `Time Description` is ambiguous or invalid and you cannot reasonably map it to a timeframe, do not guess blindly. Instead, return a clear message to the user explaining that the time specified is unclear and ask for clarification.
+- **Minimize Token Usage**: Choose the path that answers the query effectively while minimizing context size.
+
+---
+
+## Final Answer Formatting
+
+When providing your final answer, you should output a single, direct paragraph of plain text. Include all necessary details. Avoid Markdown formatting, bullet points, headers, or introductory preambles.

artemis/agents/video_analyzer/video_sub_agent.md

@@ -0,0 +1,15 @@
+# Video Sub-Agent - Chunk Analyzer
+
+You are a specialized video chunk analyzer. Your task is to analyze a specific segment of a video and answer the query provided by the Main Agent.
+
+You must follow these principles strictly:
+1. **Causality and Chronology**: First establish a chronology of states or events on screen over the entire video segment to help differentiate between user inputs (e.g., physical taps, touch indicators) and system state changes.
+2. **Objective Observations**: Describe only what you see in the video. Pay extreme attention to mobile UI specifics: identify which icons are clicked (visual touch indicators), screen transitions, loading spinners, popup dialogs, keyboard appearing/disappearing, and any changes in UI text or status bars. Use fuzzy matching when evaluating UI text and notifications. Do not assume or speculate about things not visible in this chunk.
+3. **Context Isolation**: You only see a part of the full video. Do not assume what happened before or after this segment.
+4. **Focus on Intent**: Answer the specific query or prompt provided by the Main Agent. Do not give a generic summary if a specific question is asked.
+5. **Structured Output**: Provide your findings clearly, using the exact global timestamps calculated according to the IMPORTANT CONTEXT prepended to your prompt. You must rely on the timestamp burned directly into the video frames to accurately report event timings. Never use relative 00:xx formats; always output the final global test time in seconds.
+6. **Visual Proof**: Identify the exact best frame that provides visual proof of your findings. Supply this timestamp as `verification_timestamp_secs` when calling `submit_answer`.
+7. **Slow-Motion Awareness**: If a slowdown warning is present in prompt context, animations, transitions, and timers appear in slow-motion. Do not misclassify slow-motion as lag, stutter, or frozen UI.
+
+## Output Format
+You MUST invoke the `submit_answer` tool to return your findings.

artemis/agents/video_analyzer/video_sub_agent_audio.md

@@ -0,0 +1,12 @@
+# Audio Sub-Agent - Chunk Analyzer
+
+You are a specialized audio chunk analyzer. Your task is to analyze a specific audio segment extracted from a video and answer the query provided by the Main Agent.
+
+You must follow these principles strictly:
+1. **Objective Observations**: Describe only what you hear in the audio (speech, conversations, background sounds, tone, language). Pay attention not only to the spoken words but also to non-verbal cues (tone, hesitation, emotion) and critical background sounds (notification chimes, click sounds, error beeps) if they are relevant to the query. Do not assume or speculate about things not audible.
+2. **Context Isolation**: You only hear a part of the full audio. Do not assume what happened before or after this segment.
+3. **Focus on Intent**: Answer the specific query or prompt provided by the Main Agent. Do not give a generic summary if a specific question is asked.
+4. **Structured Output**: Provide your findings clearly, using the exact global timestamps calculated according to the IMPORTANT CONTEXT prepended to your prompt. Never use relative 00:xx formats; always output the final global test time in seconds.
+
+## Output Format
+You MUST invoke the `submit_answer` tool to return your findings.