fix: preserve cross-device evidence and render recorded reports

artemis/agents/checker/checker.json

@@ -4,7 +4,7 @@
   "verify_semantics": "# `verify` items (completion acceptance criteria)\n- A `verify` item asks: was the anchored subgoal actually accomplished to this standard?\n- A `failed` verify verdict can send its milestone back for repair, so it must include a `suggestion`: a concrete, actionable hint telling the executor what is missing or wrong.\n- The release decision (whether execution proceeds) is computed outside of you and is fail-open on errors; your job is only the truthful verdict value.",
   "assert_semantics": "# `assert` items (test assertions)\n- An `assert` item is a test expectation, not an instruction. A failed assertion is a legitimate, valuable test result: record it truthfully with evidence. Nothing will (or should) try to repair it.\n- Never mark an assert `failed` merely because evidence is absent — absence of captured evidence is `inconclusive` with an explicit note that the evidence is missing.\n- Precondition first: before judging an assert, confirm from the execution history (step replay, notes, anchor evidence) that the action(s) meant to trigger the asserted behavior were actually and correctly performed by the executor (the toggle was really tapped, the intended screen was really reached).\n- If that precondition is not established — the executor never performed the trigger, performed the wrong action, or the evidence cannot show it — output `inconclusive` and state explicitly that the precondition failed / the triggering action is not confirmed as executed. Never output `failed` here: a failed assert is attributed to the device under test, and an execution problem must not be reported as a device defect.\n- Output `failed` only when the precondition is established AND the device state, probe output, or recorded evidence still contradicts the expectation. The evidence must cite the concrete step(s) that performed the trigger plus the probe output / observation that contradicts the expectation, so the verdict reads as a device-defect diagnosis.\n- `suggestion` is not needed for assert verdicts.",
   "anchor_guide": "# Evidence anchoring (checkpoint audit)\n- You are auditing a subgoal at its completion moment. The authoritative evidence is the execution history around the anchor step (the anchor step's pre-action screenshot and the preceding step's post-action state) plus persistently probeable device state.\n- For transient expectations (toasts, dialogs, momentary states) and ordering expectations, you MUST inspect the recorded history near the anchor step — the live device has already moved on and is NOT valid evidence for them.\n- Prefer persistent state probes and recorded notes over inference from summaries.",
-  "final_guide": "# Final review (task exit audit)\n- Audit against the user's original goal as amended by the user guidance received during the run (listed in the prompt), not merely the plan's checkboxes. Guidance outranks the plan and its check lines: judge a check line the guidance retired or changed by the guidance, not by its original wording.\n- `@end` items: judge them using the FINAL device state provided to you.\n- `on_complete` items: cite the verdict ledger — an already-recorded PASS/FAIL from the item's checkpoint attempt is authoritative (an assert that ever failed stays failed). For items never checked, judge from the historical evidence near their subgoal's completion; if that evidence was never captured, output `inconclusive` stating the evidence is missing. NEVER re-judge an `on_complete` item from the current screen: the current screen only proves the present, not the past moment the item was declared for.\n- `unmet_subgoals`: list, verbatim, any plan subgoal marked complete whose verify criteria are demonstrably not met. Leave it empty otherwise; failed asserts alone do NOT make a subgoal unmet.",
+  "final_guide": "# Final review (task exit audit)\n- Audit against the user's original goal as amended by the user guidance received during the run (listed in the prompt), not merely the plan's checkboxes. Guidance outranks the plan and its check lines: judge a check line the guidance retired or changed by the guidance, not by its original wording.\n- `@end` items: judge them using the FINAL device state provided to you.\n- `on_complete` items: cite the verdict ledger — an already-recorded PASS/FAIL from the item's checkpoint attempt is authoritative (an assert that ever failed stays failed). For items never checked, judge from the historical evidence near their subgoal's completion; if that evidence was never captured, output `inconclusive` stating the evidence is missing. NEVER re-judge an `on_complete` item from the current screen: the current screen only proves the present, not the past moment the item was declared for.\n- `unmet_subgoals`: list, verbatim, any plan subgoal marked complete whose verify criteria are demonstrably not met. Leave it empty otherwise; failed asserts alone do NOT make a subgoal unmet.\n- Recorded evidence is labelled with its resource, step and capture time. A live primary-device image cannot represent other resources. Repeated captures with the same hash establish an unchanged image, not that input was delivered or that a page reached its bottom. Combine them with visible end markers. If a resource is absent from the preview or its evidence is ambiguous, inspect get_step_screenshot before declaring missing evidence.",
   "probe_guide": "# Device probes (read-only)\n- `probe_device` runs a fixed, enumerated set of read-only state probes. Prefer probing persistent state over inferring it: alarms (`alarms`), battery status (`battery`), foreground activities (`foreground`), notifications (`notifications`), a settings value (`setting` with `namespace` + `key`), a content provider query (`content` with `uri`), installed packages (`packages`), a system property (`prop` with `key`).\n- Probes outside this table do not exist; you cannot run arbitrary commands.",
-  "case_result_guide": "# Overall test result (final review, for Flash and Pro)\nAlways produce `test_result`, even when no plan/check items exist. Infer acceptance criteria directly from the user's original task prompt and subsequent guidance; do not ask for an extra expected-results form. Evaluate explicit expectations and the observable outcome of requested actions. Do not invent requirements, turn examples into expectations, or relax an expectation because the executor says it completed the task.\nCompare the executor's final explanation with actual screenshots, UI elements, step history, notes, probes and the verdict ledger. The explanation is a claim to verify, never proof by itself. A final screen only proves the final state: use recorded history for intermediate/transient results and required sequences. Use read-only tools only. Do not repair a failed test or repeat device actions.\n- passed: all requirements in the prompt/guidance are supported by concrete evidence, and no active assertion failed. Cite the specific step/screenshot/probe observations.\n- failed: the triggering actions and prerequisites are confirmed as correctly executed, but concrete evidence contradicts at least one stated expectation. Cite both the trigger and the contradictory result. A confirmed failure stays failed even when other requirements are unverified.\n- inconclusive: evidence is missing/ambiguous, the task has no discernible acceptance criterion, or the conclusion cannot be checked. Explain which requirement/evidence is missing; never claim passed or failed merely from the report or an absent exception.\n- blocked: an execution or environment obstacle prevented the required actions/validation. This is not evidence of a product defect.\nWrite `reason` in concise Simplified Chinese, explaining the expected and actual result. `evidence` contains concrete observations with step numbers when available. Preserve user-supplied values, UI text and logs exactly.\nThe test result evaluates the system under test, NOT whether the testing agent obeyed the instruction to inspect/report. Explicit expectations (预期, 应当, 必须, should, expected) are test assertions even without plan checkboxes. Reporting a mismatch correctly does not satisfy that expectation. Example: 'Check the search box; expected Search settings; if mismatched report the actual text and finish.' If the box visibly says 搜索设置, output test_result.status=failed, not passed: the inspection completed but the expected product result was not met. 'Report any failures and stop' never converts a detected failure into a pass. Likewise, confirming a missing required entry is a failed test when the correct page was reached. First list the expected product behavior in reason, then compare the actual evidence; judge execution/reporting separately. If no product expectation is given, judge the observable result of the requested actions."
+  "case_result_guide": "# Overall test result (final review, for Flash and Pro)\nAlways produce `test_result`, even when no plan/check items exist. Infer acceptance criteria directly from the user's original task prompt and subsequent guidance; do not ask for an extra expected-results form. Evaluate explicit expectations and the observable outcome of requested actions. Do not invent requirements, turn examples into expectations, or relax an expectation because the executor says it completed the task.\nCompare the executor's final explanation with actual screenshots, UI elements, step history, notes, probes and the verdict ledger. The explanation is a claim to verify, never proof by itself. A final screen only proves the final state: use recorded history for intermediate/transient results and required sequences. Use read-only tools only. Do not repair a failed test or repeat device actions.\n- passed: all requirements in the prompt/guidance are supported by concrete evidence, and no active assertion failed. Cite the specific step/screenshot/probe observations.\n- failed: the triggering actions and prerequisites are confirmed as correctly executed, but concrete evidence contradicts at least one stated expectation. Cite both the trigger and the contradictory result. A confirmed failure stays failed even when other requirements are unverified.\n- inconclusive: evidence is missing/ambiguous, the task has no discernible acceptance criterion, or the conclusion cannot be checked. Explain which requirement/evidence is missing; never claim passed or failed merely from the report or an absent exception.\n- blocked: an execution or environment obstacle prevented the required actions/validation. This is not evidence of a product defect.\nWrite `reason` in concise Simplified Chinese, explaining the expected and actual result. `evidence` contains concrete observations with step numbers when available. Preserve user-supplied values, UI text and logs exactly.\nThe test result evaluates the system under test, NOT whether the testing agent obeyed the instruction to inspect/report. Explicit expectations (预期, 应当, 必须, should, expected) are test assertions even without plan checkboxes. Reporting a mismatch correctly does not satisfy that expectation. Example: 'Check the search box; expected Search settings; if mismatched report the actual text and finish.' If the box visibly says 搜索设置, output test_result.status=failed, not passed: the inspection completed but the expected product result was not met. 'Report any failures and stop' never converts a detected failure into a pass. Likewise, confirming a missing required entry is a failed test when the correct page was reached. First list the expected product behavior in reason, then compare the actual evidence; judge execution/reporting separately. If no product expectation is given, judge the observable result of the requested actions.\nFor each important acceptance criterion, make evidence a concise Chinese line starting 已确认 / 发现问题 / 待确认 / 受阻, followed by the criterion, observed facts and step references. For missing evidence, name the smallest missing observation; do not collapse confirmed and unverified resources into an undifferentiated conclusion."
 }

artemis/agents/checker/checker.py

@@ -902,8 +902,20 @@ async def run_final_check(
     )
 
     content: list = [{"type": "text", "text": human_text}]
+    from artemis.agents.checker.evidence import final_evidence
+
+    try:
+        content.extend(final_evidence(ctx.data_engine))
+    except (OSError, ValueError, AttributeError) as exc:
+        logger.warning("Final recorded evidence unavailable: %s", exc)
     if screenshot_b64:
-        content.append({"type": "text", "text": "--- Final Screenshot ---"})
+        target = getattr(ctx, "execution_target", "android")
+        content.append(
+            {
+                "type": "text",
+                "text": f"--- Live primary-device screenshot ({target}); does not represent other resources ---",
+            }
+        )
         content.append(
             {
                 "type": "image_url",

artemis/agents/checker/evidence.py

@@ -0,0 +1,102 @@
+"""Bounded, source-labelled end-of-run evidence from the existing step ledger."""
+
+import base64
+from pathlib import Path
+import re
+
+from artemis.data_engine.step_evidence import step_image_name
+
+
+def final_evidence(engine, *, max_sources: int = 8) -> list[dict]:
+    """Attach the last two independently captured frames per resource.
+
+    These are historical captures, never advertised as live device state.
+    Equal hashes retain both capture labels but share one image block. No
+    screenshots from another resource are substituted for missing evidence.
+    """
+    if engine is None:
+        return []
+    steps = engine.storage.get_steps(engine.current_session_id) or []
+    sources = {}
+    for step in steps:
+        result = step.last_execution_result
+        execution = result.get("execution") if isinstance(result, dict) else None
+        actions = execution if isinstance(execution, list) else step.action_taken
+        actions = actions if isinstance(actions, list) else [actions]
+        for action in actions:
+            if not isinstance(action, dict):
+                continue
+            plugin = action.get("plugin_id")
+            source = action.get("source") or {}
+            if plugin:
+                identity = (plugin, source.get("kind", "other"), source.get("target_id", ""))
+                name = action.get("post_image_name")
+                if len(actions) == 1:
+                    name = name or step_image_name(step, "post")
+                pre = action.get("pre_image_name") if action.get("pre_image_valid") else None
+            else:
+                # A pure helper/report's decision frame is not a fresh post-action observation.
+                if str(action.get("action", "")).startswith("plug_"):
+                    continue
+                identity = ("device", "android", "primary")
+                name, pre = step.post_image_name, step.pre_image_name
+            if not name and not plugin:
+                continue
+            label = f"Step {step.step_number} post-action; {identity}; {source.get('label', '')}"
+            label += f"; captured_at={source.get('captured_at') or step.timestamp}"
+            if pre and pre == name:
+                label += (
+                    "; fresh post-capture identical to pre-capture (not proof of success by itself)"
+                )
+            frames = sources.pop(identity, [])
+            sources[identity] = [*frames[-1:], (name, label)]
+    selected = list(sources.values())[-max_sources:]
+    blocks = []
+    attached = set()
+    for frames in selected:
+        for name, label in frames[-2:]:
+            if not name:
+                blocks.append(
+                    {
+                        "type": "text",
+                        "text": f"Recorded evidence: {label}; no post-action image. Earlier captures do not prove the state after this call.",
+                    }
+                )
+                continue
+            if not isinstance(name, str) or not re.fullmatch(r"[0-9a-f]{64}", name):
+                continue
+            path = Path(engine.get_image_path(name))
+            blocks.append({"type": "text", "text": f"Recorded evidence: {label}; image={name}"})
+            if name in attached:
+                blocks.append(
+                    {"type": "text", "text": "Same image as the preceding capture with this hash."}
+                )
+                continue
+            try:
+                image = path.read_bytes()
+            except OSError:
+                blocks.append(
+                    {
+                        "type": "text",
+                        "text": "Recorded image unavailable; do not infer its contents.",
+                    }
+                )
+                continue
+            mime = "image/png" if image.startswith(b"\x89PNG") else "image/jpeg"
+            if image.startswith(b"RIFF") and image[8:12] == b"WEBP":
+                mime = "image/webp"
+            blocks.append(
+                {
+                    "type": "image_url",
+                    "image_url": {"url": f"data:{mime};base64,{base64.b64encode(image).decode()}"},
+                }
+            )
+            attached.add(name)
+    if len(sources) > max_sources:
+        blocks.append(
+            {
+                "type": "text",
+                "text": "Additional resources omitted from this bounded preview; inspect them with get_step_screenshot before judging.",
+            }
+        )
+    return blocks

artemis/agents/flash/flash_runner.md

@@ -59,3 +59,7 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
 - **Completion (`status="completed"`)**: ONLY call `report_task_status(status="completed", ...)` after the latest screenshot and UI elements support the claimed result. If those sources disagree during a transition, wait and observe again before concluding. Tie each claimed page title or value to an observed screenshot or UI element; mark any missing evidence as unverified.
 - **Deliver the requested output**: Completing device actions does not complete a requested report. Before finishing, check the original goal and user guidance for required deliverables. Put the actual report/answer in the explanation field, covering the requested sections with observed findings, source-specific evidence and untested or blocked scope. Do not replace it with "completed" or a promise to provide it later. The short-paragraph rule above applies to turn commentary, not to this final deliverable. Use the existing record; retrieve only missing evidence instead of repeating the whole test. Do not invent findings or treat finding a product defect as execution failure when the goal is to investigate and report it.
 - **Failure (`status="failed"`)**: If the task is genuinely impossible due to paywalls, unrecoverable crashes, or hitting an absolute dead end after sensible fallbacks, call `report_task_status(status="failed", ...)` with the exact blocking reason.
+
+
+## 收尾与跨端验收
+切换到另一端之前，核对当前端的必需验收动作是否已完成并留下证据。提交 report_task_status 前逐项对照用户要求，缺少必要且仍可安全取得的观察时只补该步骤，不重跑已完成部分；无法补齐时明确列出缺项。看到页脚不自动等于滚动已到底，若用户要求验证位置不变，需追加滚动并比较本端前后画面。截图来源、步骤和实际内容共同支持结论，执行成功回执本身不证明验收通过。

artemis/agents/flash/runner.py

@@ -63,6 +63,7 @@ from artemis.agents.validator.tool_declarations import (
     LIST_NOTES_TOOL,
     READ_NOTE_TOOL,
     REPORT_TASK_STATUS_TOOL,
+    ToolExecutionResult,
     VALIDATOR_TOOLS_DECLARATION,
     capture_screenshot_and_parse_ui,
     prune_intermediate_screenshots,
@@ -1014,7 +1015,8 @@ class FlashRunner:
         return (
             "当前使用插件观察；实际画面和控件见对应工具返回，不是安卓画面。"
             "安卓旧截图、元素编号和坐标已失效。连续操作电脑无需重复观察手机；"
-            "切回安卓时先单独调用 wait_for_delay(time_in_ms=0) 读取最新手机画面，"
+            "等待电脑加载可用正时长 wait_for_delay，等待本身不会刷新画面，之后调用插件观察工具；"
+            "只有切回安卓时才单独调用 wait_for_delay(time_in_ms=0) 读取最新手机画面，"
             "收到后再决定手机动作。不要将电脑控件编号用于手机。"
         )
 
@@ -1027,14 +1029,17 @@ class FlashRunner:
         return image, elements
 
     @staticmethod
-    def _defer_tool_calls(messages: list[BaseMessage], tool_calls: list) -> None:
+    def _defer_tool_calls(
+        messages: list[BaseMessage], tool_calls: list, *, reason: str | None = None
+    ) -> None:
         """Close every pending call without executing decisions made on stale evidence."""
         for tc in tool_calls:
             messages.append(
                 ToolMessage(
                     tool_call_id=tc.get("id") or str(uuid.uuid4()),
                     name=tc["name"],
-                    content="未执行此调用：已切回安卓并重新观察。请先读取最新手机画面和控件，再重新决定动作或结论。",
+                    content=reason
+                    or "未执行此调用：已切回安卓并重新观察。请先读取最新手机画面和控件，再重新决定动作或结论。",
                     status="error",
                 )
             )
@@ -1104,6 +1109,52 @@ class FlashRunner:
             args = tc.get("args") or {}
             tc_id = tc.get("id") or str(uuid.uuid4())
 
+            # A positive wait on the PC means elapsed time, not a phone refresh.
+            if (
+                getattr(self, "_android_observation_stale", False)
+                and name == "wait_for_delay"
+                and isinstance(args.get("time_in_ms"), (int, float))
+                and 0 < args["time_in_ms"] < float("inf")
+            ):
+                operation = asyncio.sleep(args["time_in_ms"] / 1000)
+                if isinstance(control, TaskControl):
+                    interrupted, _ = await control.run_interruptibly(
+                        operation,
+                        on_pause=getattr(self.ctx, "on_task_pause", None),
+                        on_resume=getattr(self.ctx, "on_task_resume", None),
+                    )
+                    if interrupted:
+                        self._defer_tool_calls(
+                            messages,
+                            tool_calls[call_index:],
+                            reason="等待被用户打断，请重新观察当前目标再继续。",
+                        )
+                        return None, pre_screenshot_bytes, xml_list, action_sequence
+                else:
+                    await operation
+                message = f"已等待 {args['time_in_ms']} 毫秒，保持当前插件操作端；未采集新画面，请用插件观察工具核对加载结果。"
+                result = ToolExecutionResult(
+                    tool_name=name, tool_call_id=tc_id, status="success", text_summary=message
+                )
+                step_id = None
+                if self.ctx.data_engine:
+                    step_id = self._record_action_step(
+                        name,
+                        args,
+                        result,
+                        raw_text,
+                        step_token_usage,
+                        None,
+                        "",
+                        None,
+                        injected=injected,
+                        native_text=native_text,
+                    )
+                turn.step_keys.append(str(step_id or tc_id))
+                turn.actions.append((name, "success", message))
+                messages.append(ToolMessage(tool_call_id=tc_id, name=name, content=message))
+                continue
+
             returning_to_android = (
                 getattr(self, "_android_observation_stale", False) and name in android_tools
             )

artemis/agents/flash/summarizer.py

@@ -276,15 +276,18 @@ class VisualStepSummarizer(StepMemoryService):
             "post_ui_elements_text": post_ui_elements_text,
             "observation_settled": observation_settled,
         }
-        if receipt_only:
+        unchanged = bool(pre_img_bytes and post_img_bytes and pre_img_bytes == post_img_bytes)
+        if receipt_only or unchanged:
             from artemis.config.constants import PLUGIN_SUMMARY_TEXT_CHARS
             from artemis.plugins.prompt_budget import bounded_result_text
 
             summary = (
-                f"步骤 {step_number}：只读观察 {action_name}；未执行写操作。"
+                f"步骤 {step_number}：{action_name}；重新采集的动作后截图与动作前截图完全相同。"
+                "只能确认本次画面未变化，不能单凭此判定动作生效、已到页底或测试通过。\n"
+                if unchanged
+                else f"步骤 {step_number}：只读观察 {action_name}；未执行写操作。"
                 "以下为工具返回摘录，截图仍在本步骤证据中；不代表视觉验收通过。\n"
-                + bounded_result_text(exec_outcome, limit=PLUGIN_SUMMARY_TEXT_CHARS)
-            )
+            ) + bounded_result_text(exec_outcome, limit=PLUGIN_SUMMARY_TEXT_CHARS)
             # Evidence has already been persisted by the caller. As with a
             # completed visual summary, do not retain every frame in memory.
             payload.update(pre_img_bytes=None, post_img_bytes=None, focus=None)
@@ -295,7 +298,10 @@ class VisualStepSummarizer(StepMemoryService):
             if data_engine_step_id is not None and self.ctx.data_engine:
                 try:
                     self.ctx.data_engine.update_step_summary(
-                        data_engine_step_id, summary, status="ready", source="tool_receipt"
+                        data_engine_step_id,
+                        summary,
+                        status="ready",
+                        source="unchanged_capture" if unchanged else "tool_receipt",
                     )
                 except (OSError, ValueError, sqlite3.Error) as de_err:
                     logger.debug(f"DataEngine receipt summary update skipped: {de_err}")
@@ -442,7 +448,8 @@ class VisualStepSummarizer(StepMemoryService):
             # ACTION section and forbids synthesizing a transition.
             dual = bool(pre_bytes) and bool(post_bytes)
             template = self._prompt_template if dual else self._single_prompt_template
-            rendered_prompt = Template(template).render(step_number=step_number)
+            # The ordinal belongs to the dynamic input, not the cacheable rules.
+            rendered_prompt = Template(template).render(step_number="N")
             phrase = self._action_phrase(action_name, action_args)
             lead_lines = [f"Step {step_number} Physical Action: {phrase}"]
             # The verbatim arguments are worth a line only when the phrase
@@ -455,7 +462,8 @@ class VisualStepSummarizer(StepMemoryService):
                 lead_lines.append(
                     "Observation status: screen still changing; page state is not confirmed."
                 )
-            focus = input_data.get("focus")
+            focus = dict(input_data.get("focus") or {})
+            goal = focus.pop("goal", "")
             if isinstance(focus, dict) and focus:
                 lead_lines.extend(["", render_focus_block(focus, dual=dual)])
             content_blocks: list[dict[str, Any]] = [{"type": "text", "text": "\n".join(lead_lines)}]
@@ -513,7 +521,10 @@ class VisualStepSummarizer(StepMemoryService):
                 )
 
             messages: list[BaseMessage] = [
-                SystemMessage(content=rendered_prompt),
+                SystemMessage(
+                    content=rendered_prompt
+                    + f"\n\nTask goal (attention context, not observed evidence):\n{goal}"
+                ),
                 HumanMessage(content=content_blocks),
             ]