feat: improve operator progress and recording accuracy

artemis/agents/operator/operator.json

@@ -1,3 +1,3 @@
 {
-  "main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n[% if \"wait_for_delay\" in available_tools %]- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n[% endif %]- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n  1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.\n  2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading state. This boundary probe must remain unidirectional; never reverse direction merely to prove completion. If the task declares an earlier exit condition (for example, find one verified match or process N items), that explicit condition may end the traversal without reaching the list boundary.\n  3. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n  4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content.\n- **Dynamic Monitoring & Reactive Loop Protocol (Monitor-Trigger-Action)**: When executing an active milestone matching `[Loop] <Goal> into note <TargetNote> (Exit: <ExitCondition>; Interval: <Interval>)` or `[Loop:continuous] <Goal> into note <TargetNote> (Interval: <Interval>)` (the milestone grammar itself is defined under Task Plan Grammar below), run this 5-step cyclic state machine:\n  - **Step 1 (Baseline Sync)**: On the initial cycle, observe all visible entries/metrics in the primary monitoring view and save the full snaLine truncated
+  "main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n[% if \"wait_for_delay\" in available_tools %]- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n[% endif %]- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n  1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.\n  2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading state. This boundary probe must remain unidirectional; never reverse direction merely to prove completion. If the task declares an earlier exit condition (for example, find one verified match or process N items), that explicit condition may end the traversal without reaching the list boundary.\n  3. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n  4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content.\n- **Dynamic Monitoring & Reactive Loop Protocol (Monitor-Trigger-Action)**: When executing an active milestone matching `[Loop] <Goal> into note <TargetNote> (Exit: <ExitCondition>; Interval: <Interval>)` or `[Loop:continuous] <Goal> into note <TargetNote> (Interval: <Interval>)` (the milestone grammar itself is defined under Task Plan Grammar below), run this 5-step cyclic state machine:\n  - **Step 1 (Baseline Sync)**: On the initial cycle, observe all visible entries/metrics in the primary monitoring view and save the full snaLine truncated
 }

artemis/agents/operator/operator.py

@@ -46,6 +46,7 @@ from artemis.utils.decorators import wrap_with_callbacks
 from artemis.utils.element_hit_test import find_element_at_point
 from artemis.utils.logger import get_logger
 from artemis.utils.notes import get_note_file_path
+from artemis.utils.plan_grammar import parse_plan
 from artemis.memory.context_policy import build_history_for
 from artemis.utils.task_tree import get_active_subgoal_hashes
 from artemis.utils.visualization import format_minimal_list_with_elements
@@ -77,7 +78,8 @@ from artemis.agents.operator.prompts import (
     ExecutionIncidentPromptComponent,
     CheckItemsExplainerPromptComponent,
     BackgroundTasksPromptComponent,
-    TaskPlanWarningPromptComponent,
+    render_plan_ledger_bounce,
+    unwritten_action_streak,
     ToolLimitWarningPromptComponent,
     InjectedInstructionPromptComponent,
     render_transcript_static_system,
@@ -120,6 +122,8 @@ class OperatorNode:
         # Index into the rendered message list where this turn's tail begins;
         # set by the transcript build and consumed after the tool loop.
         self._transcript_turn_base: int | None = None
+        self._turn_plan_baseline: str | None = None
+        self._turn_ledger_streak: int = 0
 
     def _available_device_actions(self) -> frozenset[str]:
         """Device actions the installed actuator backend provides.
@@ -216,7 +220,6 @@ class OperatorNode:
                 (InjectedInstructionPromptComponent(), {}),
                 (BackgroundTasksPromptComponent(), {}),
                 (FeedbackPromptComponent(), {}),
-                (TaskPlanWarningPromptComponent(), {}),
                 (ToolLimitWarningPromptComponent(), {}),
             ]
         else:
@@ -230,6 +233,7 @@ class OperatorNode:
             kwargs_to_pass = {
                 "prompts": self.prompts,
                 "plan_and_history": plan_and_history,
+                "task_plan": task_plan,
                 "latest_screenshot_b64": latest_screenshot_b64,
                 "minimal_list": minimal_list,
                 "current_step_num": current_step_num,
@@ -378,7 +382,6 @@ class OperatorNode:
             (InjectedInstructionPromptComponent(), {}),
             (BackgroundTasksPromptComponent(), {}),
             (FeedbackPromptComponent(), {}),
-            (TaskPlanWarningPromptComponent(), {}),
             (ToolLimitWarningPromptComponent(), {}),
         ]
         for component, extra_kwargs in components:
@@ -433,6 +436,7 @@ class OperatorNode:
         raw_thoughts = []
         native_thoughts = []
         tool_limit_exceeded = False
+        plan_gate_bounced = False
 
         for iteration in range(max_iterations):
             if iteration == max_iterations - 1:
@@ -709,7 +713,25 @@ class OperatorNode:
                 )
                 actions_list = []
                 burst_cap = self._max_burst_actions()
-                if len(action_calls) > burst_cap:
+                bounce = None
+                if (
+                    not plan_gate_bounced
+                    and iteration < max_iterations - 1
+                    and len(action_calls) == 1
+                    and not getattr(state, "open_incident", None)
+                    and self._plan_changed_this_turn() is False
+                ):
+                    bounce = self._plan_ledger_bounce_reason()
+                if bounce:
+                    plan_gate_bounced = True
+                    validation_errors = True
+                    logger.info(f"Plan ledger gate: bouncing action. {bounce}")
+                    for tc in action_calls:
+                        tool_outputs.append(
+                            ToolMessage(tool_call_id=tc["id"], content=bounce, status="error")
+                        )
+                    action_calls_to_translate = []
+                elif len(action_calls) > burst_cap:
                     # A multi-action turn is a fast-action burst that the Validator
                     # fires without the safety net; cap its length before it runs.
                     validation_errors = True
@@ -730,6 +752,11 @@ class OperatorNode:
                         )
                     action_calls_to_translate = []
                 else:
+                    if plan_gate_bounced and self._plan_changed_this_turn() is False:
+                        logger.warning(
+                            "Plan ledger gate: executing actions after one bounce without"
+                            " a task_plan change (ledger gap this turn)."
+                        )
                     action_calls_to_translate = action_calls
                 for tc in action_calls_to_translate:
                     translated_actions, error = self._translate_and_validate_tool(tc, state)
@@ -838,6 +865,8 @@ class OperatorNode:
 
         # 3. Get history and plan
         steps, task_plan = self._get_history_and_plan()
+        self._turn_plan_baseline = self._plan_ledger_baseline(task_plan)
+        self._turn_ledger_streak = unwritten_action_streak(steps)
 
         # 4. Format minimal list
 
@@ -940,6 +969,63 @@ class OperatorNode:
             "operator_tool_limit_exceeded": tool_limit_exceeded,
         }
 
+    def _plan_ledger_gate_enabled(self) -> bool:
+        try:
+            from artemis.config import load_agent_config
+
+            return bool(load_agent_config().pro.execution.plan_ledger_gate)
+        except (AttributeError, OSError, TypeError, ValueError):
+            return True
+
+    def _read_task_plan_text(self) -> str | None:
+        """Current ``task_plan`` note text, or ``None`` when it cannot be read."""
+        engine = getattr(self.ctx, "data_engine", None)
+        if not engine:
+            return None
+        try:
+            path = get_note_file_path(engine.base_dir, "task_plan")
+            if not path.exists():
+                return None
+            return path.read_text(encoding="utf-8")
+        except (AttributeError, OSError, TypeError) as e:
+            logger.warning(f"Plan ledger gate: failed to read task_plan: {e}")
+            return None
+
+    def _plan_ledger_baseline(self, task_plan: str) -> str | None:
+        """Return the plan text used as the current turn's baseline."""
+        if not self._plan_ledger_gate_enabled() or not getattr(self.ctx, "data_engine", None):
+            return None
+        if not parse_plan(task_plan).has_top_level:
+            return None
+        return task_plan
+
+    def _plan_ledger_stale_turns(self) -> int:
+        try:
+            from artemis.config import load_agent_config
+
+            return int(load_agent_config().pro.execution.plan_ledger_stale_turns)
+        except (AttributeError, OSError, TypeError, ValueError):
+            return 4
+
+    def _plan_ledger_bounce_reason(self) -> str | None:
+        """Bounce message when the (unchanged) ledger is broken or stale, else ``None``."""
+        current = self._read_task_plan_text()
+        if current is None:
+            return None
+        return render_plan_ledger_bounce(
+            current, self._turn_ledger_streak, self._plan_ledger_stale_turns()
+        )
+
+    def _plan_changed_this_turn(self) -> bool | None:
+        """Whether ``task_plan`` differs from the turn's baseline (``None``: gate inactive)."""
+        baseline = self._turn_plan_baseline
+        if baseline is None:
+            return None
+        current = self._read_task_plan_text()
+        if current is None:
+            return None
+        return current != baseline
+
     def _get_active_subgoal_hash(self) -> str:
         """Parses task_plan.md to find the active top-level subgoal hash."""
 

artemis/agents/operator/prompts.py

@@ -652,41 +652,69 @@ class BackgroundTasksPromptComponent(PromptComponent):
             builder.add_human_content("\n".join(lines) + "\n")
 
 
-class TaskPlanWarningPromptComponent(PromptComponent):
-    """Suggests a task_plan update after an action turn that did not touch it.
+_PLAN_NOTE_TOOLS = ("update_note", "save_note", "append_note")
 
-    Only the most recent step is inspected: if it executed a Turn-Ending
-    Action and no note tool wrote to ``task_plan``, a soft suggestion is
-    injected into the next turn. Pure diagnosis / waiting / observation turns
-    legitimately leave the plan alone and get no reminder.
-    """
 
-    NOTE_TOOLS = ("update_note", "save_note", "append_note")
+def step_updated_task_plan(step: dict) -> bool:
+    """Whether a recorded step's tool calls wrote the ``task_plan`` note."""
+    for tc in step.get("tool_calls", []):
+        if tc.get("name") not in _PLAN_NOTE_TOOLS:
+            continue
+        args = tc.get("args", {})
+        note_key = args.get("key") or args.get("name")
+        if note_key == "task_plan":
+            return True
+    return False
 
-    @classmethod
-    def _step_updated_task_plan(cls, step: dict) -> bool:
-        for tc in step.get("tool_calls", []):
-            if tc.get("name") not in cls.NOTE_TOOLS:
-                continue
-            args = tc.get("args", {})
-            note_key = args.get("key") or args.get("name")
-            if note_key == "task_plan":
-                return True
-        return False
 
-    async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
-        steps = kwargs.get("steps", [])
-        if not steps:
-            return
-        last = steps[-1]
-        if not last.get("action_taken") or self._step_updated_task_plan(last):
-            return
-        builder.add_human_content(
-            "\nReminder: your last turn executed a Turn-Ending Action without"
-            " updating task_plan. If progress was made, record it now with"
-            " surgical `update_note` edits: mark what completed and add the"
-            " subtasks you are now pursuing."
+def unwritten_action_streak(steps: list[dict]) -> int:
+    """Count trailing action steps since the last ``task_plan`` write."""
+    streak = 0
+    for step in reversed(steps or []):
+        if step_updated_task_plan(step):
+            break
+        if step.get("action_taken"):
+            streak += 1
+    return streak
+
+
+def _clip(text: str, limit: int = 80) -> str:
+    text = " ".join(text.split())
+    return text if len(text) <= limit else text[: limit - 1] + "\u2026"
+
+
+_LEDGER_BOUNCE_TAIL = (
+    " with `update_note` (key `task_plan`), then re-issue the action in the same"
+    " tool-call list (Live Sub-goal Ledger)."
+)
+
+
+def render_plan_ledger_bounce(task_plan: str, streak: int, stale_turns: int) -> str | None:
+    """Return a plan-ledger validation error, if any."""
+    snapshot = parse_plan(task_plan or "")
+    if not snapshot.has_top_level or snapshot.all_top_level_done:
+        return None
+    milestone = snapshot.active_milestone()
+    if milestone is None:
+        return (
+            "Not executed: no milestone is marked in progress. Mark the one you are"
+            " executing `[/]` and open its `[/]` sub-goal" + _LEDGER_BOUNCE_TAIL
         )
+    leaf = snapshot.active_leaf(milestone)
+    if leaf is None:
+        return (
+            f'Not executed: the active milestone "{_clip(milestone.text)}" has no'
+            " in-progress `[/]` sub-goal beneath it. Open one for what you are doing now"
+            + _LEDGER_BOUNCE_TAIL
+        )
+    turns = streak + 1
+    if stale_turns > 0 and turns >= stale_turns:
+        return (
+            f'Not executed: the sub-goal "{_clip(leaf.text)}" has stayed in progress for'
+            f" {turns} action turns without any plan change. Settle it (`[x]` / `[!]`) or"
+            " record its progress and open the next leaf" + _LEDGER_BOUNCE_TAIL
+        )
+    return None
 
 
 class ToolLimitWarningPromptComponent(PromptComponent):