feat: add shared test acceptance and release 1.5.0

artemis/agents/checker/checker.json

@@ -5,5 +5,6 @@
   "assert_semantics": "# `assert` items (test assertions)\n- An `assert` item is a test expectation, not an instruction. A failed assertion is a legitimate, valuable test result: record it truthfully with evidence. Nothing will (or should) try to repair it.\n- Never mark an assert `failed` merely because evidence is absent — absence of captured evidence is `inconclusive` with an explicit note that the evidence is missing.\n- Precondition first: before judging an assert, confirm from the execution history (step replay, notes, anchor evidence) that the action(s) meant to trigger the asserted behavior were actually and correctly performed by the executor (the toggle was really tapped, the intended screen was really reached).\n- If that precondition is not established — the executor never performed the trigger, performed the wrong action, or the evidence cannot show it — output `inconclusive` and state explicitly that the precondition failed / the triggering action is not confirmed as executed. Never output `failed` here: a failed assert is attributed to the device under test, and an execution problem must not be reported as a device defect.\n- Output `failed` only when the precondition is established AND the device state, probe output, or recorded evidence still contradicts the expectation. The evidence must cite the concrete step(s) that performed the trigger plus the probe output / observation that contradicts the expectation, so the verdict reads as a device-defect diagnosis.\n- `suggestion` is not needed for assert verdicts.",
   "anchor_guide": "# Evidence anchoring (checkpoint audit)\n- You are auditing a subgoal at its completion moment. The authoritative evidence is the execution history around the anchor step (the anchor step's pre-action screenshot and the preceding step's post-action state) plus persistently probeable device state.\n- For transient expectations (toasts, dialogs, momentary states) and ordering expectations, you MUST inspect the recorded history near the anchor step — the live device has already moved on and is NOT valid evidence for them.\n- Prefer persistent state probes and recorded notes over inference from summaries.",
   "final_guide": "# Final review (task exit audit)\n- Audit against the user's original goal as amended by the user guidance received during the run (listed in the prompt), not merely the plan's checkboxes. Guidance outranks the plan and its check lines: judge a check line the guidance retired or changed by the guidance, not by its original wording.\n- `@end` items: judge them using the FINAL device state provided to you.\n- `on_complete` items: cite the verdict ledger — an already-recorded PASS/FAIL from the item's checkpoint attempt is authoritative (an assert that ever failed stays failed). For items never checked, judge from the historical evidence near their subgoal's completion; if that evidence was never captured, output `inconclusive` stating the evidence is missing. NEVER re-judge an `on_complete` item from the current screen: the current screen only proves the present, not the past moment the item was declared for.\n- `unmet_subgoals`: list, verbatim, any plan subgoal marked complete whose verify criteria are demonstrably not met. Leave it empty otherwise; failed asserts alone do NOT make a subgoal unmet.",
-  "probe_guide": "# Device probes (read-only)\n- `probe_device` runs a fixed, enumerated set of read-only state probes. Prefer probing persistent state over inferring it: alarms (`alarms`), battery status (`battery`), foreground activities (`foreground`), notifications (`notifications`), a settings value (`setting` with `namespace` + `key`), a content provider query (`content` with `uri`), installed packages (`packages`), a system property (`prop` with `key`).\n- Probes outside this table do not exist; you cannot run arbitrary commands."
+  "probe_guide": "# Device probes (read-only)\n- `probe_device` runs a fixed, enumerated set of read-only state probes. Prefer probing persistent state over inferring it: alarms (`alarms`), battery status (`battery`), foreground activities (`foreground`), notifications (`notifications`), a settings value (`setting` with `namespace` + `key`), a content provider query (`content` with `uri`), installed packages (`packages`), a system property (`prop` with `key`).\n- Probes outside this table do not exist; you cannot run arbitrary commands.",
+  "case_result_guide": "# Overall test result (final review, for Flash and Pro)\nAlways produce `test_result`, even when no plan/check items exist. Infer acceptance criteria directly from the user's original task prompt and subsequent guidance; do not ask for an extra expected-results form. Evaluate explicit expectations and the observable outcome of requested actions. Do not invent requirements, turn examples into expectations, or relax an expectation because the executor says it completed the task.\nCompare the executor's final explanation with actual screenshots, UI elements, step history, notes, probes and the verdict ledger. The explanation is a claim to verify, never proof by itself. A final screen only proves the final state: use recorded history for intermediate/transient results and required sequences. Use read-only tools only. Do not repair a failed test or repeat device actions.\n- passed: all requirements in the prompt/guidance are supported by concrete evidence, and no active assertion failed. Cite the specific step/screenshot/probe observations.\n- failed: the triggering actions and prerequisites are confirmed as correctly executed, but concrete evidence contradicts at least one stated expectation. Cite both the trigger and the contradictory result. A confirmed failure stays failed even when other requirements are unverified.\n- inconclusive: evidence is missing/ambiguous, the task has no discernible acceptance criterion, or the conclusion cannot be checked. Explain which requirement/evidence is missing; never claim passed or failed merely from the report or an absent exception.\n- blocked: an execution or environment obstacle prevented the required actions/validation. This is not evidence of a product defect.\nWrite `reason` in concise Simplified Chinese, explaining the expected and actual result. `evidence` contains concrete observations with step numbers when available. Preserve user-supplied values, UI text and logs exactly.\nThe test result evaluates the system under test, NOT whether the testing agent obeyed the instruction to inspect/report. Explicit expectations (预期, 应当, 必须, should, expected) are test assertions even without plan checkboxes. Reporting a mismatch correctly does not satisfy that expectation. Example: 'Check the search box; expected Search settings; if mismatched report the actual text and finish.' If the box visibly says 搜索设置, output test_result.status=failed, not passed: the inspection completed but the expected product result was not met. 'Report any failures and stop' never converts a detected failure into a pass. Likewise, confirming a missing required entry is a failed test when the correct page was reached. First list the expected product behavior in reason, then compare the actual evidence; judge execution/reporting separately. If no product expectation is given, judge the observable result of the requested actions."
 }

artemis/agents/checker/checker.py

@@ -46,6 +46,7 @@ from pydantic import BaseModel, Field, ValidationError
 
 from artemis.core.tool_failure import ToolFailure, is_tool_failure
 from artemis.constants import CHECKER_MAX_ITERATIONS
+from artemis.runtime.case_result import CaseTestResult, normalize_case_result
 from artemis.context import ArtemisContext
 from artemis.data_engine.context_vars import CURRENT_TRACE_ID
 from artemis.data_engine.trace import trace, trace_langchain_tool
@@ -98,6 +99,10 @@ class CheckVerdict(BaseModel):
 
 class CheckReport(BaseModel):
     verdicts: list[CheckVerdict] = Field(default_factory=list)
+    test_result: CaseTestResult | None = Field(
+        default=None,
+        description="Final review only: overall test verdict against the user's prompt and guidance, independent of execution completion.",
+    )
     unmet_subgoals: list[str] = Field(
         default_factory=list,
         description=(
@@ -287,6 +292,7 @@ def assemble_checker_prompt_segments(
         segments.append(prompts.get("anchor_guide", ""))
     else:
         segments.append(prompts.get("final_guide", ""))
+        segments.append(prompts.get("case_result_guide", ""))
     if probe_tool_registered:
         segments.append(prompts.get("probe_guide", ""))
     return "\n\n".join(s for s in segments if s)
@@ -458,7 +464,11 @@ def _normalize_report(report: CheckReport, check_items: list) -> CheckReport:
                     evidence="no verdict produced for this item",
                 )
             )
-    return CheckReport(verdicts=verdicts, unmet_subgoals=list(report.unmet_subgoals))
+    return CheckReport(
+        verdicts=verdicts,
+        unmet_subgoals=list(report.unmet_subgoals),
+        test_result=normalize_case_result(report.test_result) if report.test_result else None,
+    )
 
 
 async def _run_check_loop(
@@ -796,6 +806,7 @@ async def run_final_check(
     ledger: list[dict],
     check_items,
     attempt_id: str | None = None,
+    execution_report: dict | str | None = None,
 ) -> CheckReport:
     """Final review at task exit: audits the user's original goal and all
     declared check items, with the final screen state available."""
@@ -828,6 +839,8 @@ async def run_final_check(
     human_text = (
         f"# User's Original Goal\n{goal}\n\n"
         f"# User guidance received during the run\n{_format_user_guidance(ctx, steps)}\n\n"
+        f"# Executor's Final Report (claims to verify, not independent evidence)\n"
+        f"{json.dumps(execution_report, ensure_ascii=False, default=str) if execution_report is not None else '(not provided)'}\n\n"
         f"# Final Task Plan\n{plan_text or '(no plan)'}\n\n"
         f"# Declared Check Items\n{_format_check_items(items)}\n\n"
         f"# Verdict Ledger (checkpoint results, authoritative for on_complete"
@@ -852,7 +865,15 @@ async def run_final_check(
     ]
     with CheckerStreamCapture(ctx) as capture:
         try:
-            return await _run_check_loop(ctx, messages, tools, items)
+            report = await _run_check_loop(ctx, messages, tools, items)
+            report.test_result = (
+                normalize_case_result(report.test_result)
+                if report.test_result
+                else CaseTestResult(
+                    status="inconclusive", reason="验收未生成测试结论，不能确认用例是否通过。"
+                )
+            )
+            return report
         finally:
             _persist_stream(
                 ctx, capture, attempt_id=attempt_id, phase="final", checkpoint_id="final"

artemis/agents/checker/flash.py

@@ -0,0 +1,113 @@
+"""Flash's exit-only acceptance check uses the same read-only Checker as Pro."""
+
+import asyncio
+
+from artemis.agents.checker.checker import run_final_check
+from artemis.context import ArtemisContext
+from artemis.graph.checkpoints import (
+    RunOutcome,
+    TestSummary,
+    final_check_enabled,
+    publish_checker_event,
+    verdict_payloads,
+    write_run_outcome,
+)
+from artemis.runtime.case_result import CaseTestResult, resolve_case_result
+from artemis.utils.logger import get_logger
+
+logger = get_logger(__name__)
+
+
+async def settle_flash_outcome(
+    ctx: ArtemisContext,
+    goal: str,
+    execution_report: dict,
+    *,
+    final_report_received: bool = False,
+) -> dict:
+    """Check once at exit, without operating the device or rerunning the task."""
+    status = str(execution_report.get("status") or "failed")
+    result = None
+    if status == "completed" or final_report_received:
+        if final_check_enabled(ctx):
+            timeout = float(getattr(ctx.execution_setup, "checkpoint_timeout", 180.0))
+            publish_checker_event(
+                ctx,
+                {
+                    "event": "attempt_started",
+                    "phase": "final",
+                    "attempt_id": "final#1",
+                    "checkpoint_id": "final",
+                    "check_items": [],
+                    "status": "running",
+                },
+            )
+            try:
+                report = await asyncio.wait_for(
+                    run_final_check(
+                        ctx,
+                        goal=goal,
+                        plan_text="",
+                        ledger=[],
+                        check_items=[],
+                        attempt_id="final#1",
+                        execution_report=execution_report,
+                    ),
+                    timeout=timeout,
+                )
+                result = report.test_result
+                publish_checker_event(
+                    ctx,
+                    {
+                        "event": "attempt_finished",
+                        "phase": "final",
+                        "attempt_id": "final#1",
+                        "checkpoint_id": "final",
+                        "status": "done",
+                        "verdicts": verdict_payloads(report.verdicts),
+                        "test_result": result.model_dump() if result else None,
+                        "route": "end",
+                    },
+                )
+            except Exception as exc:
+                detail = (
+                    f"超过 {timeout:g} 秒"
+                    if isinstance(exc, TimeoutError)
+                    else f"{type(exc).__name__}: {exc}"
+                )
+                logger.warning(f"Flash 测试验收未完成：{detail}")
+                result = CaseTestResult(
+                    status="inconclusive", reason=f"操作已结束，但测试验收未完成：{detail}。"
+                )
+                publish_checker_event(
+                    ctx,
+                    {
+                        "event": "attempt_finished",
+                        "phase": "final",
+                        "attempt_id": "final#1",
+                        "checkpoint_id": "final",
+                        "status": "error",
+                        "error": result.reason,
+                        "route": "end",
+                    },
+                )
+        else:
+            result = CaseTestResult(
+                status="inconclusive", reason="已关闭最终检查，未进行测试结果验收。"
+            )
+
+    outcome = RunOutcome(
+        task_status="completed" if status == "completed" else "blocked", tests=TestSummary()
+    )
+    payload = outcome.model_dump()
+    if result:
+        payload["test_result"] = result.model_dump()
+    resolved = resolve_case_result(payload, status, str(execution_report.get("explanation") or ""))
+    payload["test_result"] = resolved.model_dump() if resolved else None
+    if ctx.data_engine:
+        write_run_outcome(
+            ctx.data_engine.base_dir, outcome, {"test_result": payload["test_result"]}
+        )
+    publish_checker_event(ctx, {"event": "run_outcome", "phase": "outcome", **payload})
+    ctx.run_outcome = payload
+    return payload