chenmingxuan/Artemis-Eapil
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 32 Wiki Activity

feat: harden agent verification and runtime contracts

Browse Source 
Introduce two-gate checker settlement, explicit graph visibility and checkpoints, structured LLM reliability, centralized action specifications, and corresponding tests and design documentation.
This commit is contained in:
somew1nd committed 2026-08-31 20:42:24 -07:00
1 parent ddfa420945
commit cbe8f4f0c6
106 files changed
+11542 -4445
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

artemis/agents/operator/operator.json
+1 -2
View File
@@ -1,4 +1,3 @@
{
"main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor. Calibrate your scroll distance so that this anchor remains visible near the top of the subsequent screen, creating a seamless visual overlap that guarantees zero missed items. Terminate exploration when a definitive boundary is reached (e.g., list bottom reached or no new content appears after scrolling).\n 2. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n 3. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\". Only perform incremental re-verification if the list has been refreshed or contentLine truncated
"troubleshooter_template": "# ROLE\nYou are the Expert Android Device Troubleshooter. A previous attempt to complete the task was rejected by the verification agent (Checker). Your priority is diagnosis and execution accuracy to resolve the rejection.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor. Calibrate your scroll distance so that this anchor remains visible near the top of the subsequent screen, creating a seamless visual overlap that guarantees zero missed items. Terminate exploration when a definitive boundary is reached (e.g., list bottom reached or no new content appears after scrolling).\n 2. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n 3. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\". Only perform incremental rLine truncated
"main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen.[% if \"swipe\" in available_tools %] For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.[% endif %]\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Contract**: Tools that return information needed for the decision ([[ tool_enum(pre_decision_tools, \"and\") ]]) are result-dependent Pre-Decision Tools. Do not submit them in the same tool-call list as a Turn-Ending Action; inspect their result first. The write-through tools `update_note` and `append_note` may accompany at most one Turn-Ending Action when their content is based entirely on evidence already available before that action. Never write an action's expected outcome as if it had already been observed.\n[% if \"wait_for_delay\" in available_tools %]- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n[% endif %]- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.\n 2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading state. This boundary probe must remain unidLine truncated
}
artemis/agents/operator/operator.py
+52 -456
View File
@@ -12,21 +12,14 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import asyncio
import json
from pathlib import Path
import re
from typing import Annotated, Any, Literal
from typing import Any
from uuid import uuid4
from langchain_core.messages import SystemMessage, ToolMessage
from langchain_core.tools import BaseTool, tool
from langgraph.types import Command
from langchain_core.tools import BaseTool
from artemis.constants import (
OPERATOR_MAX_CHAT_ROUNDS,
VALIDATOR_MESSAGES_KEY,
)
from artemis.context import ArtemisContext
from artemis.data_engine.trace import (
TraceSpan,
@@ -34,7 +27,9 @@ from artemis.data_engine.trace import (
trace_langchain_tool,
)
from artemis.graph.state import State
from artemis.services.llm import get_llm, invoke_llm_with_timeout_message
from artemis.graph.visibility import strict_state
from artemis.mcp.action_specs import OPERATOR_SHELL_ORDER, operator_shell_tool
from artemis.services.llm import acomplete, get_llm, invoke_llm_with_timeout_message
from artemis.tools.command_tool import (
_BACKGROUND_TASKS,
_FINISHED_TASKS_LOGS,
@@ -51,19 +46,12 @@ from artemis.utils.coordinates import (
parse_swipe_parameters,
)
from artemis.utils.decorators import wrap_with_callbacks
from artemis.utils.file import create_snapshot
from artemis.utils.logger import get_logger
from artemis.utils.notes import get_note_file_path, get_notes_dir
from artemis.utils.notes import get_note_file_path
from artemis.utils.task_tree import (
build_plan_and_history,
get_active_subgoal_hashes,
)
from artemis.utils.verification import (
append_verification_chat,
get_verification_chat_path,
get_verification_chat_rounds,
read_verification_chat,
)
from artemis.utils.visualization import format_minimal_list_with_elements
logger = get_logger(__name__)
@@ -86,7 +74,8 @@ from artemis.agents.operator.prompts import (
TemplatePromptComponent,
ObservationPromptComponent,
ScreenshotSimilarityPromptComponent,
CheckerFeedbackPromptComponent,
FeedbackPromptComponent,
CheckItemsExplainerPromptComponent,
BackgroundTasksPromptComponent,
ShortTermMemoryPromptComponent,
TaskPlanWarningPromptComponent,
@@ -169,56 +158,6 @@ class OperatorNode:
return steps, task_plan
def _get_verification_chat_rounds(self) -> tuple[int, int]:
if not self.ctx.data_engine:
return 0, 0
subgoal_hash = self._get_active_subgoal_hash()
chat_path = get_verification_chat_path(self.ctx.data_engine.base_dir, subgoal_hash)
turns = read_verification_chat(chat_path)
return get_verification_chat_rounds(turns)
def _has_checker_feedback(self) -> bool:
if not self.ctx.data_engine:
return False
subgoal_hash = self._get_active_subgoal_hash()
chat_path = get_verification_chat_path(self.ctx.data_engine.base_dir, subgoal_hash)
return chat_path.exists()
def _get_reply_to_checker_tool(self) -> BaseTool:
@tool
def reply_to_checker(
reasoning: Annotated[
str,
"Your reasoning or observations regarding the verification feedback.",
],
):
"""[TERMINAL] Use this tool to reply to the verification feedback with your reasoning or observations.
This ends your turn.
"""
if self.ctx.data_engine:
subgoal_hash = self._get_active_subgoal_hash()
chat_path = get_verification_chat_path(self.ctx.data_engine.base_dir, subgoal_hash)
turns = read_verification_chat(chat_path)
max_op, max_chk = get_verification_chat_rounds(turns)
round_num = max(max_op + 1, max_chk)
if round_num == 0:
round_num = 1
if append_verification_chat(chat_path, "operator", reasoning, round_num):
return "Successfully recorded your reply to the Checker."
else:
return "Failed to save reply."
else:
return "Error: DataEngine not available."
return reply_to_checker
async def _build_prompt(
self,
state: State,
@@ -235,39 +174,26 @@ class OperatorNode:
builder = PromptBuilder()
has_feedback = self._has_checker_feedback()
# One constant component list: the prompt template is never switched by
# verification results. Check-related components are append-only and
# render nothing when there is nothing to say.
components = self.prompt_components
if not components:
if has_feedback:
components = [
(
TemplatePromptComponent(),
{"template_name": "troubleshooter_template"},
),
(ObservationPromptComponent(), {}),
(ScreenshotSimilarityPromptComponent(), {}),
(InjectedInstructionPromptComponent(), {}),
(ShortTermMemoryPromptComponent(), {}),
(BackgroundTasksPromptComponent(), {}),
(CheckerFeedbackPromptComponent(), {}),
(TaskPlanWarningPromptComponent(), {}),
(ToolLimitWarningPromptComponent(), {}),
]
else:
components = [
(
TemplatePromptComponent(),
{"template_name": "main_template"},
),
(ObservationPromptComponent(), {}),
(ScreenshotSimilarityPromptComponent(), {}),
(InjectedInstructionPromptComponent(), {}),
(ShortTermMemoryPromptComponent(), {}),
(BackgroundTasksPromptComponent(), {}),
(TaskPlanWarningPromptComponent(), {}),
(ToolLimitWarningPromptComponent(), {}),
]
components = [
(
TemplatePromptComponent(),
{"template_name": "main_template"},
),
(CheckItemsExplainerPromptComponent(), {}),
(ObservationPromptComponent(), {}),
(ScreenshotSimilarityPromptComponent(), {}),
(InjectedInstructionPromptComponent(), {}),
(ShortTermMemoryPromptComponent(), {}),
(BackgroundTasksPromptComponent(), {}),
(FeedbackPromptComponent(), {}),
(TaskPlanWarningPromptComponent(), {}),
(ToolLimitWarningPromptComponent(), {}),
]
else:
components = [(c, {}) for c in components] + [(ToolLimitWarningPromptComponent(), {})]
@@ -298,31 +224,13 @@ class OperatorNode:
traced_tools: list,
new_subagent_calls: list,
state: State,
) -> tuple[list[dict] | None, bool, str | None, str | None, bool]:
) -> tuple[list[dict] | None, str | None, str | None, bool]:
max_iterations = 20
action_result = None
requested_argue = False
raw_thoughts = []
native_thoughts = []
tool_limit_exceeded = False
async def read_stream_and_accumulate(target_llm, msgs):
try:
full_response = None
has_chunks = False
async for chunk in target_llm.astream(msgs):
has_chunks = True
if full_response is None:
full_response = chunk
else:
full_response += chunk
if has_chunks and full_response is not None:
return full_response
except Exception as e:
logger.debug(f"astream failed or not supported: {e}. Falling back to ainvoke.")
return await target_llm.ainvoke(msgs)
for iteration in range(max_iterations):
if iteration == max_iterations - 1:
logger.warning(
@@ -340,9 +248,7 @@ class OperatorNode:
)
bound_llm = base_llm.bind_tools(tools=traced_tools)
response = await invoke_llm_with_timeout_message(
read_stream_and_accumulate(bound_llm, current_messages)
)
response = await invoke_llm_with_timeout_message(acomplete(bound_llm, current_messages))
if hasattr(response, "response_metadata") and response.response_metadata:
usage = (
@@ -420,23 +326,19 @@ class OperatorNode:
action_calls = [
tc for tc in response.tool_calls if normalize_name(tc["name"]) in action_tool_names
]
reply_to_checker_calls = [
tc for tc in response.tool_calls if normalize_name(tc["name"]) == "reply_to_checker"
]
other_calls = [
tc
for tc in response.tool_calls
if normalize_name(tc["name"]) not in action_tool_names + ["reply_to_checker"]
if normalize_name(tc["name"]) not in action_tool_names
]
tool_outputs = []
validation_errors = False
requested_argue = False
other_tool_failed = False
other_tool_failure_msg = ""
has_terminal_calls = bool(action_calls or reply_to_checker_calls)
has_terminal_calls = bool(action_calls)
if other_calls:
results = []
@@ -478,12 +380,8 @@ class OperatorNode:
content = get_tool_result_content(result_obj)
status = "success"
if isinstance(result_obj, Command):
updates = result_obj.update
if VALIDATOR_MESSAGES_KEY in updates:
msgs = updates[VALIDATOR_MESSAGES_KEY]
if msgs and hasattr(msgs[0], "status"):
status = msgs[0].status
if isinstance(result_obj, ToolMessage) and result_obj.status:
status = result_obj.status
is_err = status == "error"
if not is_err:
@@ -567,29 +465,6 @@ class OperatorNode:
)
action_calls = []
if reply_to_checker_calls:
logger.warning(
"Scratchpad/helper tool call failed. Deferring"
f" reply to checker: {other_tool_failure_msg}"
)
for tc in reply_to_checker_calls:
tool_outputs.append(
ToolMessage(
tool_call_id=tc["id"],
content=(
"Your reply to the checker was"
" rejected because an accompanying"
" tool call failed. Error:"
f" {other_tool_failure_msg}. Currently,"
" no response has been sent to the"
" checker. Please correct the error"
" and try again."
),
status="error",
)
)
reply_to_checker_calls = []
elif has_terminal_calls:
has_deferring_calls = any(
normalize_name(tc["name"]) in DEFERRING_TOOLS for tc in other_calls
@@ -623,28 +498,7 @@ class OperatorNode:
)
)
for tc in reply_to_checker_calls:
tool_outputs.append(
ToolMessage(
tool_call_id=tc["id"],
content=(
"Your pre-decision tools have been"
" successfully processed. However, your"
" response to the checker was rejected"
" because you are not allowed to"
" simultaneously use result-dependent"
" pre-decision tools and reply to the"
" checker. Currently, no response has been"
" sent to the checker. Please review"
" your updated context and re-output"
" your reply."
),
status="success",
)
)
action_calls = []
reply_to_checker_calls = []
if action_calls:
logger.info(
@@ -676,32 +530,10 @@ class OperatorNode:
if not validation_errors:
action_result = actions_list
if reply_to_checker_calls:
logger.info("Operator replied to checker.")
tc = reply_to_checker_calls[0]
requested_argue = True
# Execute it immediately
result = await self._run_other_tool(
tc,
traced_tools,
new_subagent_calls,
state,
raw_thoughts,
native_thoughts,
)
tool_outputs.append(
ToolMessage(
tool_call_id=tc["id"],
content=result,
status="success" if not result.startswith("Error") else "error",
)
)
for tm in tool_outputs:
current_messages.append(tm)
if (action_calls and not validation_errors) or requested_argue:
if action_calls and not validation_errors:
break
if validation_errors:
@@ -714,7 +546,6 @@ class OperatorNode:
native_thinking = "\n".join(native_thoughts) if native_thoughts else None
return (
action_result,
requested_argue,
raw_thinking,
native_thinking,
tool_limit_exceeded,
@@ -765,18 +596,10 @@ class OperatorNode:
)
@trace(type="agent", name="operator")
async def __call__(self, state: State):
state = strict_state(state, "operator")
# 1. Check for infinite loops
self._check_infinite_loop(state)
# Create snapshot for optimistic execution
if self.ctx.data_engine:
notes_dir = get_notes_dir(self.ctx.data_engine.base_dir)
if notes_dir.exists():
snapshot_dir = notes_dir.with_name("notes_snapshot")
await asyncio.to_thread(create_snapshot, notes_dir, snapshot_dir)
self.ctx.task_plan_snapshot = snapshot_dir
logger.info(f"Created snapshot of notes at {snapshot_dir}")
# 2. Get perception data from state (populated by Perception node)
operator_raw_data = getattr(state, "operator_raw_data", {}) or {}
latest_screenshot_b64 = operator_raw_data.get("screenshot_b64")
@@ -822,45 +645,16 @@ class OperatorNode:
current_step_num = len(steps) + 1
# 5. Prepare Tools. Device-action declarations are assembled against the
# installed actuator backend's capabilities: an action the backend does not
# implement is simply never declared (and the prompt assembly drops its
# teaching segments in lockstep).
# 5. Prepare Tools. Device-action shells come from the canonical manifest
# (artemis/mcp/action_specs.py) and are assembled against the installed
# actuator backend's capabilities: an action the backend does not implement
# is simply never declared (and the prompt assembly drops its teaching
# segments in lockstep).
available_actions = self._available_device_actions()
action_tool_factories = {
"click": self._get_click_tool,
"input_text": self._get_input_text_tool,
"swipe": self._get_swipe_tool,
"press_key": self._get_press_key_tool,
"manage_app": self._get_manage_app_tool,
"wait_for_delay": self._get_wait_for_delay_tool,
"long_press": self._get_long_press_tool,
}
all_tools = [
factory()
for name, factory in action_tool_factories.items()
if name in available_actions
operator_shell_tool(name) for name in OPERATOR_SHELL_ORDER if name in available_actions
] + self.tools
if self._has_checker_feedback():
max_op, max_chk = self._get_verification_chat_rounds()
max_rounds = (
getattr(
getattr(self.ctx, "execution_setup", None),
"checker_max_chat_rounds",
OPERATOR_MAX_CHAT_ROUNDS,
)
if self.ctx and getattr(self.ctx, "execution_setup", None)
else OPERATOR_MAX_CHAT_ROUNDS
)
if max_op < max_rounds:
reply_to_checker_tool = self._get_reply_to_checker_tool()
all_tools.append(reply_to_checker_tool)
else:
logger.info(
f"Maximum chat rounds ({max_rounds}) reached. Removing reply_to_checker tool."
)
# 5. Evaluate dynamic tools and prepare background tasks
has_long_output = False
for tinfo in _FINISHED_TASKS_LOGS.values():
@@ -924,7 +718,6 @@ class OperatorNode:
new_subagent_calls = []
(
action_result,
requested_argue,
raw_thinking,
native_thinking,
tool_limit_exceeded,
@@ -955,22 +748,17 @@ class OperatorNode:
if stm_match:
short_term_memory = stm_match.group(1).strip()
return await state.asanitize_update(
ctx=self.ctx,
update={
"structured_decisions": structured_decisions,
"operator_raw_thinking": raw_thinking,
"operator_native_thinking": native_thinking,
"short_term_memory": short_term_memory,
"indexed_points": state.indexed_points,
"complete_subgoals_by_ids": [],
"current_step_id": state.current_step_id,
"subagent_calls": ((state.subagent_calls or []) + new_subagent_calls),
"operator_replied": requested_argue,
"operator_tool_limit_exceeded": tool_limit_exceeded,
},
agent="operator",
)
return {
"structured_decisions": structured_decisions,
"operator_raw_thinking": raw_thinking,
"operator_native_thinking": native_thinking,
"short_term_memory": short_term_memory,
"indexed_points": state.indexed_points,
"indexed_elements": state.indexed_elements,
"current_step_id": state.current_step_id,
"subagent_calls": ((state.subagent_calls or []) + new_subagent_calls),
"operator_tool_limit_exceeded": tool_limit_exceeded,
}
def _get_active_subgoal_hash(self) -> str:
"""Parses task_plan.md to find the active top-level subgoal hash."""
@@ -1007,198 +795,6 @@ class OperatorNode:
chronological_last_step=True,
)
def _get_long_press_tool(self) -> BaseTool:
@tool
def long_press(
target: Annotated[
int | list[int],
"Long press target. Can be an element index number (int, e.g."
" 3) OR normalized coordinates (list of 2 integers, e.g. [500,"
" 600]).",
],
duration: Annotated[int, "Long press duration in milliseconds (default 1000)."] = 1000,
):
"""[ACTION] Long press on the target location on the screen (supports element index or absolute normalized coordinates)."""
return "Action Recorded"
return long_press
def _get_click_tool(self) -> BaseTool:
@tool
def click(
target: Annotated[
int | list[int],
"Click target. Can be an element index number (int, e.g. 3) OR"
" normalized coordinates (list of 2 integers, e.g. [500,"
" 600]).",
],
times: Annotated[
int,
"Number of consecutive clicks on this target. Use this for"
" double-clicks or multi-clicks (e.g. 7 to enter developer"
" mode). Default is 1.",
] = 1,
delay_ms: Annotated[
int,
"Delay in milliseconds between consecutive clicks. Default is 100.",
] = 100,
):
"""[ACTION] Click on the target location on the screen (supports element index or absolute normalized coordinates)."""
return "Action Recorded"
return click
def _get_input_text_tool(self) -> BaseTool:
@tool
def input_text(
text: Annotated[
str,
"The text content to input. Supports multi-line content with '\\n'.",
],
target: Annotated[
int | list[int],
"Input target field. Can be an input box element index number"
" (int, e.g. 3) OR normalized coordinates (list of 2 integers,"
" e.g. [500, 600]).",
],
clear_exist: Annotated[
bool,
"Whether to clear existing text before typing. True (default):"
" clear/replace entire text. False: append at the end of"
" existing content.",
] = True,
):
"""[ACTION] Type text into the target input field (supports replacing whole text or appending to the end, and multi-line strings with '\\n')."""
return "Action Recorded"
return input_text
def _get_swipe_tool(self) -> BaseTool:
@tool
def swipe(
direction: Annotated[
Literal["up", "down", "left", "right"] | None,
"Direction for scrolling and swiping: 'up' (drags bottom-to-top, scrolling down to reveal content below),"
" 'down' (drags top-to-bottom, scrolling up to reveal content above),"
" 'left' (drags right-to-left, scrolling right),"
" 'right' (drags left-to-right, scrolling left).",
] = None,
start: Annotated[
list[int] | None,
"Start normalized coordinates [start_x, start_y] in 0-1000 scale for precise,"
" local interactions (e.g. adjusting sliders, SeekBars, fine range selection, or drag-and-drop).",
] = None,
end: Annotated[
list[int] | None,
"End normalized coordinates [end_x, end_y] in 0-1000 scale for precise,"
" local interactions (e.g. adjusting sliders, SeekBars, fine range selection, or drag-and-drop).",
] = None,
target: Annotated[
int | list[int] | str | None,
"Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within.",
] = None,
gesture: Annotated[
Literal["up", "down", "left", "right"] | list[int] | None,
"Backward-compatible swipe gesture: smart direction string ('up', 'down', 'left', 'right')"
" OR precise custom coordinates [start_x, start_y, end_x, end_y] in 0-1000 scale.",
] = None,
duration: Annotated[
int | None,
"Optional swipe/drag duration in milliseconds (default 800). For drag-and-drop,"
" list reordering, or sliding/adjusting sliders (e.g., volume, brightness, SeekBars),"
" set duration >= 1000 (e.g. 1500). If omitted for directional swipe, duration is computed automatically.",
] = None,
):
"""[ACTION] Perform a swipe, drag, or slider-adjustment gesture on the screen.
• Directional Scrolling ('direction'): Recommended for general browsing and standard page scrolling in most scenarios. Automatically computes safe swipe vectors and adaptive duration, retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings. Supports scoping to a sub-container via 'target'. If it fails on certain custom layouts, fall back to specifying exact coordinates ('start' and 'end') directly.
• Precise Coordinate Gestures ('start', 'end'): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars), drag-and-drop / list reordering, or as a reliable fallback when directional scrolling fails on specific containers. Always drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), swipe fully to the extreme boundary.
Args:
direction: Smart directional scrolling ('up', 'down', 'left', 'right'). Automatically computes safe swipe vectors, retaining 40% visual overlap: 'up' (reveal content below), 'down' (reveal content above), 'left', 'right'.
start: Start normalized coordinates [start_x, start_y] in 0-1000 scale.
end: End normalized coordinates [end_x, end_y] in 0-1000 scale.
target: Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within.
gesture: Backward-compatible parameter: direction string OR custom coordinates list [start_x, start_y, end_x, end_y] in 0-1000 scale.
duration: Optional gesture duration in milliseconds (default 800).
"""
return "Action Recorded"
return swipe
def _get_press_key_tool(self) -> BaseTool:
@tool
def press_key(
key: Annotated[
Literal["ENTER", "BACK", "HOME", "APP_SWITCH"],
"Standard Android system button name (ENTER, BACK, HOME, APP_SWITCH).",
],
):
"""[ACTION] Press a physical or virtual system button (e.g.
ENTER, BACK, HOME, APP_SWITCH).
"""
return "Action Recorded"
return press_key
def _get_manage_app_tool(self) -> BaseTool:
@tool
def manage_app(
action: Annotated[Literal["launch", "stop"], "The action type."],
app_name: Annotated[str, "Display name or package name of the application."],
):
"""[ACTION] Launch or force stop a specified application."""
return "Action Recorded"
return manage_app
def _get_wait_for_delay_tool(self) -> BaseTool:
@tool
def wait_for_delay(
time_in_ms: Annotated[
int,
"The exact duration to wait in milliseconds. Accurately convert the"
" required time duration into milliseconds based on your objective"
" or plan (e.g., 2000 for 2s, 5000 for 5s, 60000 for 1 minute,"
" 180000 for 3 minutes, 300000 for 5 minutes).",
],
):
"""[ACTION] Pause execution and wait for a specified duration in milliseconds.
Use this whenever you need time to elapse—whether for UI loading, animations,
screen transitions, or longer scheduled delays and intervals specified in the task.
"""
return "Action Recorded"
return wait_for_delay
def _get_wait_for_text_tool(self) -> BaseTool:
@tool
def wait_for_text(
text: Annotated[
str,
"A simple, distinct keyword (e.g., 'Success', 'Done',"
" 'Loading') to watch on the screen. Try to pass a clean word"
" without trailing punctuation (e.g. 'Loading' instead of"
" 'Loading...') for optimal matching reliability.",
],
state: Annotated[
Literal["visible", "hidden"],
"Wait for the keyword to appear ('visible') or disappear"
" ('hidden') from the screen.",
] = "visible",
timeout_ms: Annotated[int, "Maximum timeout in milliseconds (default 5000)."] = 5000,
):
"""[ACTION] Pause execution and wait intelligently for a specific, distinct text or keyword to appear or disappear on the screen.
Use this when waiting for a specific loading screen to finish or a
specific success/confirmation state to appear.
"""
return "Action Recorded"
return wait_for_text
def _translate_and_validate_tool(self, tc: dict, state: State) -> tuple[list[dict], str | None]:
actions, err = self._translate_and_validate_tool_inner(tc, state)
if err:
@@ -1384,7 +980,7 @@ class OperatorNode:
direction=target,
target=args.get("target"),
indexed_elements=getattr(state, "indexed_elements", None),
ui_hierarchy=getattr(state, "ui_tree", None),
ui_hierarchy=getattr(state, "latest_ui_hierarchy", None),
width=width,
height=height,
duration=duration,
artemis/agents/operator/prompts.py
+106 -217
View File
@@ -14,10 +14,9 @@
import base64
import io
import json
from pathlib import Path
from jinja2 import Template
from jinja2 import Environment, StrictUndefined, Template
from langchain_core.messages import HumanMessage, SystemMessage
from PIL import Image
@@ -28,27 +27,27 @@ from artemis.tools.command_tool import (
_is_output_long,
)
from artemis.utils.logger import get_logger
from artemis.utils.task_tree import get_active_subgoal_hashes
from artemis.utils.notes import get_note_file_path
from artemis.utils.plan_grammar import parse_plan, render_plan_grammar_spec
logger = get_logger(__name__)
from artemis.agents.prompt_assembly import (
gate_segment,
render_tool_enum,
resolve_available,
)
_LEGACY_TOOL_CALLING_RULE = """- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action."""
_LEGACY_TOOL_PROTOCOL = """1. **Pre-Decision Exploratory Tools**:
- *What they are*: Helper/Subagent tools (such as `ask_explorer`, `ask_diagnoser`, `video_analyzer`), ADB command tools (`run_adb_command`, `manage_task`), and memory note tools (`read_note`, `list_notes`, `save_note`, `update_note`, `append_note`).
- *How they work*: These tools gather details or update/read memory. When you invoke these tools, the framework will immediately execute them and return the results, allowing you to continue thinking and make your final decision.
2. **Turn-Ending Actions**:"""
from artemis.agents.prompt_assembly import render_tool_enum, resolve_available
from artemis.mcp.action_specs import OPERATOR_SHELL_ORDER
# --- Assembled tool references -------------------------------------------------------
# The operator.json templates carry availability slots rendered by
# ``apply_operator_prompt_contract``: ``[[ tool_enum(...) ]]`` expands an ordered,
# backticked enumeration limited to the available tool set, and
# ``[% if "x" in available_tools %]...[% endif %]`` gates a self-contained teaching
# segment. An unavailable tool therefore leaves no trace in the prompt at all. The
# slot delimiters are square-bracketed so the standard ``{{ ... }}`` context
# variables (initial_goal, plan_and_history, plan_grammar, ...) pass through
# untouched for the later context render.
#
# The orderings below reproduce the historical prompt wording exactly; with the full
# tool set the rendered output is byte-identical to the previous literals. See
# tool set the rendered output is byte-identical to the pre-assembly literals. See
# artemis/agents/prompt_assembly.py for the assembly rationale.
_PRE_DECISION_HELPER_TOOLS = ("ask_explorer", "ask_diagnoser", "video_analyzer")
@@ -59,16 +58,9 @@ _PRE_DECISION_ALL_TOOLS = (
)
# The two device-action enumeration slots in operator.json (identical in both
# templates), in their historical orders.
_PHYSICAL_ACTIONS_ORDER = (
"click",
"input_text",
"swipe",
"press_key",
"manage_app",
"wait_for_delay",
"long_press",
)
# templates), in their historical orders. The physical order is the canonical
# manifest's shell-binding order; the turn-ending order is a prompt property.
_PHYSICAL_ACTIONS_ORDER = OPERATOR_SHELL_ORDER
_TURN_ENDING_ORDER = (
"click",
"swipe",
@@ -79,120 +71,31 @@ _TURN_ENDING_ORDER = (
"wait_for_delay",
)
_PHYSICAL_ACTIONS_SLOT = (
"Physical device actions (`click`, `input_text`, `swipe`, `press_key`,"
" `manage_app`, `wait_for_delay`, `long_press`)"
)
_TURN_ENDING_SLOT = (
"Turn-Ending Action (`click`, `swipe`, `input_text`, `long_press`,"
" `press_key`, `manage_app`, or `wait_for_delay`)"
)
# Self-contained instruction segments that teach a single tool. Removed wholesale when
# the tool is unavailable, so nothing in the prompt dangles.
_WAIT_FOR_DELAY_TRANSITIONS_SEGMENT = (
"- **Loading & Transitions**: When entering page transitions (e.g., app launching,"
" web page loading, or progress indicators), use `wait_for_delay` to let the screen"
" settle before executing target actions. Some transition states are difficult to"
" discern, such as a thin progress line at the top of the browser screen during"
" webpage loads. Before questioning whether an error has occurred, you can first"
" adopt a waiting strategy to see how the device state evolves. If the state"
" remains ambiguous after waiting, certain UI components might be malfunctioning or"
" failing to load; in this case, consider retrying the action or pivoting to an"
" alternative path.\n"
)
_WAIT_FOR_DELAY_INTERVAL_PARENTHETICAL = (
" (convert minutes to milliseconds accurately for `wait_for_delay`)"
)
_VIDEO_ANALYZER_TRANSIENT_SEGMENT = (
" - *Missed Transient States*: If the current state jumped ahead and you suspect"
" you missed an important initial or intermediate screen (e.g., a number or flash"
" message that was overwritten by your last action), consider using the"
" `video_analyzer` tool to look back at the recent history.\n"
)
_VIDEO_ANALYZER_SNAPSHOT_SENTENCE = (
" If you have already missed certain critical information and cannot re-view it by"
" going back or other operations, you can use the `video_analyzer` to look back at"
" historical information."
)
# Tail of the "Action Scope & Interaction Precision" bullet: everything past the two
# generic gesture-scope sentences is swipe/drag teaching.
_SWIPE_PRECISION_SEGMENT = (
" For any task requiring continuous range interactions, sliding, or boundary"
" selection, prefer using targeted swipe/drag gestures spanning from the element's"
" start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is"
" strictly confined to the target element. For drag-and-drop, list reordering, or"
" sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag"
" slightly PAST the target position to overcome touch slop and reliably trigger the"
" drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%),"
" always swipe fully to the extreme boundary to guarantee reaching the absolute"
" limits."
)
#: Every tool name the operator prompt slots can reference. ``available_tools=None``
#: resolves to this set, preserving the historical output.
OPERATOR_PROMPT_TOOLSET: frozenset[str] = frozenset(
_PRE_DECISION_ALL_TOOLS + _PHYSICAL_ACTIONS_ORDER
)
def _tool_calling_rule(available: frozenset[str]) -> str:
tools = render_tool_enum(_PRE_DECISION_ALL_TOOLS, available, final_sep="and")
return (
"- **Tool Calling Contract**: Tools that return information needed for the"
f" decision ({tools}) are result-dependent Pre-Decision Tools. Do not submit"
" them in the same tool-call list as a Turn-Ending Action; inspect their result"
" first. The write-through tools `update_note` and `append_note` may accompany"
" at most one Turn-Ending Action when their content is based entirely on"
" evidence already available before that action. Never write an action's"
" expected outcome as if it had already been observed."
)
def _tool_protocol(available: frozenset[str]) -> str:
helper = render_tool_enum(_PRE_DECISION_HELPER_TOOLS, available)
adb = render_tool_enum(_PRE_DECISION_ADB_TOOLS, available)
memory = render_tool_enum(_PRE_DECISION_MEMORY_TOOLS, available)
return (
"1. **Result-Dependent Pre-Decision Tools**:\n"
f" - *What they are*: Helper/Subagent tools ({helper}), ADB/task tools"
f" ({adb}), and memory tools whose result must be inspected ({memory}).\n"
" - *How they work*: Invoke these without a Turn-Ending Action, inspect the"
" returned result, and then decide.\n"
"2. **Write-Through Memory Tools**:\n"
" - `update_note` and `append_note` may run alongside at most one Turn-Ending"
" Action only when recording facts already observed in the current context. If"
" the note content depends on the action's result, wait for the next"
" observation before writing it.\n"
"3. **Turn-Ending Actions**:"
)
_LEGACY_CHECKER_REJECTION_TRIGGER = """ - **Validation/Checker Rejection**: The verification agent (Checker) rejected your subgoal completion (i.e., you are in troubleshooter mode and this is a retry)."""
_AMBIGUOUS_CHECKER_REJECTION_TRIGGER = """ - **Ambiguous Validation/Checker Rejection**: Use `ask_diagnoser` only when the rejection cause is unclear, the evidence conflicts, or no safe local correction is evident. A Checker rejection alone is not a mandatory diagnosis trigger."""
_LEGACY_TROUBLESHOOTER_CHALLENGE = """ - *Challenge Checker*: If you visually confirm the subgoal's target state is *already achieved*, you **MUST** invoke the `reply_to_checker` tool to state your observation."""
_TROUBLESHOOTER_CHALLENGE = """ - *Challenge Checker*: If current evidence clearly proves the subgoal is already achieved, invoke `reply_to_checker` directly. This branch takes precedence over diagnosis; do not call `ask_diagnoser` first.
- *Clear Local Correction*: If the rejection cause is explicit and a safe correction is evident from the current screen, perform that correction directly. This branch also takes precedence over generic diagnosis triggers. Use `ask_diagnoser` only for ambiguity, conflicting evidence, or repeated failure."""
_LEGACY_LARGE_LIST_STRATEGY = """- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:
1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor. Calibrate your scroll distance so that this anchor remains visible near the top of the subsequent screen, creating a seamless visual overlap that guarantees zero missed items. Terminate exploration when a definitive boundary is reached (e.g., list bottom reached or no new content appears after scrolling).
2. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.
3. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for "double-checking". Only perform incremental re-verification if the list has been refreshed or content has mutated, while maintaining strict single-pass handling of new events."""
_LARGE_LIST_STRATEGY = """- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:
1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.
2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading state. This boundary probe must remain unidirectional; never reverse direction merely to prove completion. If the task declares an earlier exit condition (for example, find one verified match or process N items), that explicit condition may end the traversal without reaching the list boundary.
3. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.
4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for "double-checking", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content."""
#: Renders the availability slots in operator.json. Square-bracket delimiters keep
#: the standard ``{{ ... }}`` context placeholders inert during this phase.
_CONTRACT_ENV = Environment(
block_start_string="[%",
block_end_string="%]",
variable_start_string="[[",
variable_end_string="]]",
comment_start_string="[#",
comment_end_string="#]",
keep_trailing_newline=True,
undefined=StrictUndefined,
)
def apply_operator_prompt_contract(
prompt_template: str,
available_tools: frozenset[str] | None = None,
) -> str:
"""Align prompt-level tool and recovery rules with Operator runtime semantics.
"""Renders a template's availability slots against the actually-available tools.
Args:
prompt_template: One of the raw operator.json templates.
@@ -203,65 +106,19 @@ def apply_operator_prompt_contract(
"""
available = frozenset(resolve_available(available_tools, OPERATOR_PROMPT_TOOLSET))
prompt_template = prompt_template.replace(
_LEGACY_TOOL_CALLING_RULE,
_tool_calling_rule(available),
)
prompt_template = prompt_template.replace(
_LEGACY_TOOL_PROTOCOL,
_tool_protocol(available),
)
prompt_template = prompt_template.replace(
_LEGACY_CHECKER_REJECTION_TRIGGER,
_AMBIGUOUS_CHECKER_REJECTION_TRIGGER,
)
prompt_template = prompt_template.replace(
_LEGACY_TROUBLESHOOTER_CHALLENGE,
_TROUBLESHOOTER_CHALLENGE,
)
prompt_template = prompt_template.replace(
_LEGACY_LARGE_LIST_STRATEGY,
_LARGE_LIST_STRATEGY,
)
def tool_enum(names, final_sep: str | None = None) -> str:
return render_tool_enum(tuple(names), available, final_sep=final_sep)
# Device-action enumeration slots (identical in both templates).
prompt_template = prompt_template.replace(
_PHYSICAL_ACTIONS_SLOT,
"Physical device actions ("
+ render_tool_enum(_PHYSICAL_ACTIONS_ORDER, available)
+ ")",
return _CONTRACT_ENV.from_string(prompt_template).render(
available_tools=available,
tool_enum=tool_enum,
pre_decision_tools=_PRE_DECISION_ALL_TOOLS,
helper_tools=_PRE_DECISION_HELPER_TOOLS,
adb_tools=_PRE_DECISION_ADB_TOOLS,
memory_tools=_PRE_DECISION_MEMORY_TOOLS,
physical_actions=_PHYSICAL_ACTIONS_ORDER,
turn_ending_actions=_TURN_ENDING_ORDER,
)
prompt_template = prompt_template.replace(
_TURN_ENDING_SLOT,
"Turn-Ending Action ("
+ render_tool_enum(_TURN_ENDING_ORDER, available, final_sep="or")
+ ")",
)
# Instruction segments that teach a single tool.
prompt_template = prompt_template.replace(
_WAIT_FOR_DELAY_TRANSITIONS_SEGMENT,
gate_segment(_WAIT_FOR_DELAY_TRANSITIONS_SEGMENT, available, "wait_for_delay"),
)
prompt_template = prompt_template.replace(
_WAIT_FOR_DELAY_INTERVAL_PARENTHETICAL,
gate_segment(
_WAIT_FOR_DELAY_INTERVAL_PARENTHETICAL, available, "wait_for_delay"
),
)
prompt_template = prompt_template.replace(
_VIDEO_ANALYZER_TRANSIENT_SEGMENT,
gate_segment(_VIDEO_ANALYZER_TRANSIENT_SEGMENT, available, "video_analyzer"),
)
prompt_template = prompt_template.replace(
_VIDEO_ANALYZER_SNAPSHOT_SENTENCE,
gate_segment(_VIDEO_ANALYZER_SNAPSHOT_SENTENCE, available, "video_analyzer"),
)
prompt_template = prompt_template.replace(
_SWIPE_PRECISION_SEGMENT,
gate_segment(_SWIPE_PRECISION_SEGMENT, available, "swipe"),
)
return prompt_template
class PromptBuilder:
@@ -340,11 +197,22 @@ class TemplatePromptComponent(PromptComponent):
plan_and_history = kwargs.get("plan_and_history", "No plan or history yet.")
# Grammar spec assembly is a function of configuration: with both check
# gates disabled, the check-line grammar never enters any prompt.
include_checks = bool(setup and getattr(setup, "checks_enabled", False))
# The rejection/finding diagnosis trigger only exists while a mechanism
# that can produce rejections or findings is active.
verification_active = include_checks or bool(
setup and not getattr(setup, "disable_planner_validation", True)
)
full_prompt = Template(prompt_template).render(
initial_goal=state.initial_goal,
subgoals_status="",
plan_and_history=plan_and_history,
unified_history="",
plan_grammar=render_plan_grammar_spec(include_checks),
verification_active=verification_active,
)
parts = full_prompt.split("# CURRENT OBSERVATION")
@@ -369,43 +237,64 @@ class ObservationPromptComponent(PromptComponent):
builder.add_human_content(f"--- Visible UI Elements ---\n{minimal_list}")
class CheckerFeedbackPromptComponent(PromptComponent):
class FeedbackPromptComponent(PromptComponent):
"""Append-only injection of source-tagged findings.
Reads ``state.operator_feedback`` (written by ``execution_check_node`` at
harvest / planner rejection and by exit settlement on a bounce-back).
Renders nothing when there are no findings — the prompt template itself
is never switched.
"""
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
if not ctx.data_engine:
findings = getattr(state, "operator_feedback", None)
if not findings:
return
lines = "\n".join(f"- {f}" for f in findings)
builder.add_human_content(
"--- Verification Findings ---\n"
f"{lines}\n"
"Each finding is tagged with its source. Checker verdicts"
" ([verify failed], [final check]) are independent judgments —"
" address any reverted subgoal accordingly; do not re-litigate"
" them. [planner] findings explain why a plan change was rejected"
" and rolled back — factor the reason into your next strategy."
)
notes_dir = Path(ctx.data_engine.base_dir) / "notes"
subgoal_hash = "default"
task_plan_path = notes_dir / "task_plan.md"
if task_plan_path.exists():
try:
content = task_plan_path.read_text(encoding="utf-8")
class CheckItemsExplainerPromptComponent(PromptComponent):
"""Behavioral guidance for check lines, rendered iff the CURRENT plan
actually contains check lines (content-driven, not switch-driven: a resumed
plan carrying check lines still gets the explanation)."""
parent_hash, _ = get_active_subgoal_hashes(content)
subgoal_hash = parent_hash
except Exception as e:
logger.error(f"Failed to parse active subgoal in component: {e}")
verification_chat_path = notes_dir / f"verification_chat_{subgoal_hash}.json"
turns = []
if verification_chat_path.exists():
try:
turns = json.loads(verification_chat_path.read_text(encoding="utf-8"))
logger.info(f"Read verification chat for {subgoal_hash}: {len(turns)} turns")
except Exception as e:
logger.error(f"Error reading verification chat: {e}")
if turns:
dialogue_lines = []
for t in turns:
role = "Operator" if t["role"] == "operator" else "Checker"
dialogue_lines.append(f"**{role} (Round {t['round']})**:\n{t['content']}")
checker_feedback = "\n\n".join(dialogue_lines)
feedback_prompt = f"--- Checker Feedback ---\n{checker_feedback}"
builder.add_human_content(feedback_prompt)
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
if not ctx or not getattr(ctx, "data_engine", None):
return
try:
task_plan_path = get_note_file_path(ctx.data_engine.base_dir, "task_plan")
if not task_plan_path.exists():
return
snapshot = parse_plan(task_plan_path.read_text(encoding="utf-8"))
except Exception as e:
logger.error(f"Failed to parse plan for check-items explainer: {e}")
return
if not snapshot.all_check_items:
return
builder.add_human_content(
"--- About the plan's check lines ---\n"
"The task plan declares `- verify:` / `- assert:` check lines."
" `verify:` lines are the acceptance criteria for their subgoal — use"
" them to confirm your work is complete, but the judgment is made by"
" an independent Checker: never declare a check passed yourself or"
" record conclusions on its behalf. `assert:` lines are test"
" assertions — take NO extra actions for them and never construct or"
" fake state to satisfy one; a failing assertion is a legitimate test"
" result. Check lines must not be deleted or reworded (deletions are"
" automatically restored by the system); keep them verbatim when"
" rewriting the plan — adding new ones is allowed. Simply mark"
" completions per the plan grammar as usual; checking runs"
" asynchronously in the background and does not block you."
)
class BackgroundTasksPromptComponent(PromptComponent):