refactor: unify agent history and harden runtime diagnostics

Preserve user guidance through history compression, stabilize Flash index targets, and retire withdrawn checks without applying stale verdicts.

Improve runtime diagnostics, tool responses, and history rendering. Simplify documentation and comments.

Validation: 2076 tests passed in the full suite; 381 affected tests and 26 frontend tests passed after cleanup. Ruff and the quality ratchet passed.
This commit is contained in:
somew1nd committed 2026-09-09 21:35:40 -07:00
1 parent e983180b5b
commit 5d92939d91
127 files changed
+13738 -1988

No files matched your search

+3 -3
View File
@@ -1,9 +1,9 @@
{
"system": "You are a precise, evidence-driven verification expert (Checker). You audit declared check items against recorded execution evidence and issue verdicts. You never operate the device, never modify any state, and never negotiate with the executing agent — you only observe, probe read-only state where allowed, and judge.",
"base_rules": "# Verdict Rules (always apply)\n- Quote every check item verbatim in `item_text` — never paraphrase.\n- Every verdict MUST carry concrete evidence: a specific probe output, a specific note excerpt, or a specific on-screen/screenshot observation. A `failed` verdict without concrete evidence is invalid and will be downgraded.\n- `inconclusive` means the evidence needed for a judgment was never captured or is genuinely ambiguous. When evidence was never collected, output `inconclusive` and state explicitly that the evidence is missing — do NOT output `failed` for lack of evidence.\n- Judge only what the check item declares. Do not invent additional criteria and do not relax the declared ones.",
"verify_semantics": "# `verify` items (completion acceptance criteria)\n- A `verify` item asks: was the anchored subgoal actually accomplished to this standard?\n- `failed` verify verdicts trigger a repair loop, so a failed verdict MUST include a `suggestion`: a concrete, actionable hint telling the executor what is missing or wrong.\n- The release decision (whether execution proceeds) is computed outside of you and is fail-open on errors; your job is only the truthful verdict value.",
"assert_semantics": "# `assert` items (test assertions)\n- An `assert` item is a test expectation, not an instruction. A failed assertion is a legitimate, valuable test result: record it truthfully with evidence. Nothing will (or should) try to repair it.\n- Never mark an assert `failed` merely because evidence is absent — absence of captured evidence is `inconclusive` with an explicit note that the evidence is missing.\n- `suggestion` is not needed for assert verdicts.",
"verify_semantics": "# `verify` items (completion acceptance criteria)\n- A `verify` item asks: was the anchored subgoal actually accomplished to this standard?\n- A `failed` verify verdict can send its milestone back for repair, so it must include a `suggestion`: a concrete, actionable hint telling the executor what is missing or wrong.\n- The release decision (whether execution proceeds) is computed outside of you and is fail-open on errors; your job is only the truthful verdict value.",
"assert_semantics": "# `assert` items (test assertions)\n- An `assert` item is a test expectation, not an instruction. A failed assertion is a legitimate, valuable test result: record it truthfully with evidence. Nothing will (or should) try to repair it.\n- Never mark an assert `failed` merely because evidence is absent — absence of captured evidence is `inconclusive` with an explicit note that the evidence is missing.\n- Precondition first: before judging an assert, confirm from the execution history (step replay, notes, anchor evidence) that the action(s) meant to trigger the asserted behavior were actually and correctly performed by the executor (the toggle was really tapped, the intended screen was really reached).\n- If that precondition is not established — the executor never performed the trigger, performed the wrong action, or the evidence cannot show it — output `inconclusive` and state explicitly that the precondition failed / the triggering action is not confirmed as executed. Never output `failed` here: a failed assert is attributed to the device under test, and an execution problem must not be reported as a device defect.\n- Output `failed` only when the precondition is established AND the device state, probe output, or recorded evidence still contradicts the expectation. The evidence must cite the concrete step(s) that performed the trigger plus the probe output / observation that contradicts the expectation, so the verdict reads as a device-defect diagnosis.\n- `suggestion` is not needed for assert verdicts.",
"anchor_guide": "# Evidence anchoring (checkpoint audit)\n- You are auditing a subgoal at its completion moment. The authoritative evidence is the execution history around the anchor step (the anchor step's pre-action screenshot and the preceding step's post-action state) plus persistently probeable device state.\n- For transient expectations (toasts, dialogs, momentary states) and ordering expectations, you MUST inspect the recorded history near the anchor step — the live device has already moved on and is NOT valid evidence for them.\n- Prefer persistent state probes and recorded notes over inference from summaries.",
"final_guide": "# Final review (task exit audit)\n- Audit against the USER'S ORIGINAL GOAL, not merely the plan's checkboxes.\n- `@end` items: judge them using the FINAL device state provided to you.\n- `on_complete` items: cite the verdict ledger — an already-recorded PASS/FAIL from the item's checkpoint attempt is authoritative (an assert that ever failed stays failed). For items never checked, judge from the historical evidence near their subgoal's completion; if that evidence was never captured, output `inconclusive` stating the evidence is missing. NEVER re-judge an `on_complete` item from the current screen: the current screen only proves the present, not the past moment the item was declared for.\n- `unmet_subgoals`: list, verbatim, any plan subgoal marked complete whose verify criteria are demonstrably not met. Leave it empty otherwise; failed asserts alone do NOT make a subgoal unmet.",
"final_guide": "# Final review (task exit audit)\n- Audit against the user's original goal as amended by the user guidance received during the run (listed in the prompt), not merely the plan's checkboxes. Guidance outranks the plan and its check lines: judge a check line the guidance retired or changed by the guidance, not by its original wording.\n- `@end` items: judge them using the FINAL device state provided to you.\n- `on_complete` items: cite the verdict ledger — an already-recorded PASS/FAIL from the item's checkpoint attempt is authoritative (an assert that ever failed stays failed). For items never checked, judge from the historical evidence near their subgoal's completion; if that evidence was never captured, output `inconclusive` stating the evidence is missing. NEVER re-judge an `on_complete` item from the current screen: the current screen only proves the present, not the past moment the item was declared for.\n- `unmet_subgoals`: list, verbatim, any plan subgoal marked complete whose verify criteria are demonstrably not met. Leave it empty otherwise; failed asserts alone do NOT make a subgoal unmet.",
"probe_guide": "# Device probes (read-only)\n- `probe_device` runs a fixed, enumerated set of read-only state probes. Prefer probing persistent state over inferring it: alarms (`alarms`), battery status (`battery`), foreground activities (`foreground`), notifications (`notifications`), a settings value (`setting` with `namespace` + `key`), a content provider query (`content` with `uri`), installed packages (`packages`), a system property (`prop` with `key`).\n- Probes outside this table do not exist; you cannot run arbitrary commands."
}
+59 -7
View File
@@ -319,21 +319,54 @@ def build_checker_tools(
# --- History formatting --------------------------------------------------------------
def _format_history(ctx: ArtemisContext, limit: int = 60) -> str:
def _load_steps(ctx: ArtemisContext) -> list[dict] | str:
"""The agent-friendly step records, or the reason they are unavailable."""
if not ctx.data_engine:
return "No execution history available."
try:
steps = ctx.data_engine.get_agent_friendly_steps() or []
return list(ctx.data_engine.get_agent_friendly_steps() or [])
except Exception as e:
return f"Failed to load execution history: {e}"
def _step_offset_label(step: dict, session_start: float | None) -> str:
"""Session-relative ``T+mm:ss`` label of a step: the clock every agent shares.
The recorded timestamp against the session start is authoritative; a
record that only carries the engine's ``"97.7s"`` label is converted.
"""
from artemis.memory.transcript import format_session_offset
ts = step.get("timestamp")
if isinstance(session_start, (int, float)) and isinstance(ts, (int, float)):
return format_session_offset(float(ts) - float(session_start))
rel = str(step.get("relative_time") or "").strip()
if rel.startswith("T+"):
return rel
if rel.endswith("s"):
try:
return format_session_offset(float(rel[:-1]))
except ValueError:
pass
return "T+??:??"
def _format_history(ctx: ArtemisContext, limit: int = 60, steps: list[dict] | None = None) -> str:
if steps is None:
steps = _load_steps(ctx)
if isinstance(steps, str):
return steps
if not steps:
return "No execution history available."
session_start = getattr(ctx.data_engine, "session_start_time", None)
lines = []
for s in steps[-limit:]:
num = s.get("step_number")
rel = s.get("relative_time", "")
summary = s.get("summary") or ""
summary = (s.get("summary") or "").strip()
action = s.get("action_taken")
if not summary and not action:
# Nothing to audit on this step: an empty row is only noise.
continue
action_str = ""
if action and not summary:
# The shared renderer keeps the target's provenance visible: a
@@ -343,8 +376,25 @@ def _format_history(ctx: ArtemisContext, limit: int = 60) -> str:
action_str = format_actions_clean(action)[:160]
except Exception:
action_str = str(action)[:160]
lines.append(f"- Step {num} ({rel}): {summary or action_str}".rstrip())
return "\n".join(lines)
label = _step_offset_label(s, session_start)
lines.append(f"- Step {num} ({label}): {summary or action_str}".rstrip())
return "\n".join(lines) or "No execution history available."
def _format_user_guidance(ctx: ArtemisContext, steps: list[dict] | None = None) -> str:
"""Every mid-run user instruction, read from the step it was stamped on."""
if steps is None:
steps = _load_steps(ctx)
if isinstance(steps, str) or not steps:
return "(none)"
session_start = getattr(ctx.data_engine, "session_start_time", None)
lines = []
for s in steps:
instruction = (s.get("extra_metadata") or {}).get("injected_instruction")
if isinstance(instruction, str) and instruction.strip():
label = _step_offset_label(s, session_start)
lines.append(f'- Step {s.get("step_number")} ({label}): "{instruction.strip()}"')
return "\n".join(lines) or "(none)"
def _format_check_items(check_items: list) -> str:
@@ -773,14 +823,16 @@ async def run_final_check(
)
screenshot_b64, minimal_list = await _capture_final_screen(ctx)
steps = _load_steps(ctx)
human_text = (
f"# User's Original Goal\n{goal}\n\n"
f"# User guidance received during the run\n{_format_user_guidance(ctx, steps)}\n\n"
f"# Final Task Plan\n{plan_text or '(no plan)'}\n\n"
f"# Declared Check Items\n{_format_check_items(items)}\n\n"
f"# Verdict Ledger (checkpoint results, authoritative for on_complete"
f" items)\n{_format_ledger(ledger)}\n\n"
f"# Execution History (summaries)\n{_format_history(ctx)}\n\n"
f"# Execution History (summaries)\n{_format_history(ctx, steps=steps)}\n\n"
f"# Final Screen Elements\n{minimal_list}"
)
+4 -3
View File
@@ -356,9 +356,10 @@ class Diagnoser:
if submit_tool:
submit_tool.invoke(args)
outcome = f"Analysis: {analysis} | Actionable Steps: "
for step in steps:
outcome += f"{step} "
outcome = f"Analysis: {analysis}"
if steps:
numbered = "\n".join(f"{i}. {step}" for i, step in enumerate(steps, start=1))
outcome += f"\nActionable steps:\n{numbered}"
return outcome
def _max_iteration_outcome(self, response, outcome: str) -> str:
+2 -2
View File
@@ -61,7 +61,7 @@ UNIVERSAL_EXPLORER_TOOLS = [
"function": {
"name": "detect_objects",
"description": (
"[Perception] Locates elements, shapes and more using powerful VLM."
"[Perception] Locates elements, shapes and more using a vision-language model."
" Returns their normalized coordinates (0-1000 scale) with label dots."
),
"parameters": {
@@ -235,7 +235,7 @@ NATIVE_EXPLORER_TOOL_DECLARATIONS = [
types.FunctionDeclaration(
name="detect_objects",
description=(
"[Perception] Locates elements, shapes and more using powerful"
"[Perception] Locates elements, shapes and more using a"
" VLM. Returns their normalized coordinates (in a [0, 1000]"
" scale) labeled with [D1], [D2] etc. in the text output, along"
" with an annotated image containing corresponding visual label"
+249 -118
View File
@@ -14,10 +14,14 @@
"""Context compression shared by Flash and Pro.
:class:`ScrubEdgeCompressor` strips XML at depth 1 and resolves screenshots
at depth K, then freezes the message. A late summary never backfills a frozen
message. FlashRunner uses this implementation; the Pro transcript ledger
reuses it over its active region (see the class docstring).
:class:`ScrubEdgeCompressor` maintains two edges over the observation history:
a shallow text edge (``xml_scrub_depth``, default 1: as soon as a newer
observation exists) where the heavy text blocks and the per-turn ephemeral
blocks leave, and the screenshot edge at depth K where the image is resolved
in place into its visual summary. A message past both edges is frozen; a late
summary never backfills a frozen message. FlashRunner uses this implementation;
the Pro transcript ledger reuses it over its active region (see the class
docstring).
"""
from typing import Any, Callable
@@ -25,6 +29,7 @@ from typing import Any, Callable
from langchain_core.messages import BaseMessage
from artemis.memory.step_memory import StepMemoryService
from artemis.memory.transcript import EPHEMERAL_BLOCKS_KEY
from artemis.utils.logger import get_logger
logger = get_logger(__name__)
@@ -35,6 +40,15 @@ HISTORY_SUMMARY_PREFIX = "--- Historical Visual Transition ---\n"
_HISTORY_SUMMARY_PREFIX = HISTORY_SUMMARY_PREFIX
_UI_LIST_MARKER = "--- UI Element List ---"
#: Text of the label block both observation shapes place directly above the
#: screenshot. It only makes sense next to an image, so it leaves together
#: with the image it labels (the visual summary carries its own header).
SCREENSHOT_LABEL = "--- Current Screenshot ---"
#: Sentinel: the screenshot stays for now (summary pending inside the grace
#: window); the message is revisited for the image swap only.
_KEEP_IMAGE = object()
def _without_marked_suffix(text: str, markers: tuple[str, ...] = (_UI_LIST_MARKER,)) -> str:
"""Remove a marked heavy block without discarding text before its marker."""
@@ -49,22 +63,32 @@ def _without_marked_suffix(text: str, markers: tuple[str, ...] = (_UI_LIST_MARKE
class ScrubEdgeCompressor:
"""Replace older observation details with summaries near the message tail.
"""Replace older observation details with summaries at two scrub edges.
Mutation discipline — every message is touched at most twice, always near
the tail, and is then frozen forever:
Depth 1 == the most recent image-bearing message. Messages newer than an
edge are byte-identical to what the model was first shown. Each message
is edited at most twice and is then frozen:
1. Depth 1 (XML): once a newer observation exists, historical UI Element
lists are stripped (same semantics as the legacy ``prune_history_xml``).
2. Depth K (image, ``image_scrub_depth``): the K-th most recent
image-bearing message has its screenshot resolved — replaced by the
ready visual summary; if the summary is still pending the image is
retained for up to ``pending_grace_steps`` further image-depths, after
which it is replaced by a pending placeholder. A failed summary is
replaced by an unavailable placeholder immediately. Both placeholders
carry the DataEngine step reference.
3. Frozen: after resolution the message is never read or written again —
a late summary never backfills a frozen message.
1. Text edge (depth > ``xml_scrub_depth``, default 1, i.e. as soon as a
newer observation exists), one pass, exactly once per message: the
heavy blocks behind ``strip_markers`` (the UI element list; for Pro
also the plan recitation) are cut at their marker with the text before
the marker kept, and every content block whose index is listed under
:data:`~artemis.memory.transcript.EPHEMERAL_BLOCKS_KEY` in the
message's ``additional_kwargs`` (reminders, hints, per-turn notices)
is removed. Only the live observation therefore ever carries an
indexed element list: a stale ``[n]`` index from an older screen can
never be picked up as a target.
2. Screenshot edge (depth >= ``image_scrub_depth``), IN PLACE: the ready
visual summary block takes the image block's position and the
:data:`SCREENSHOT_LABEL` block above it is removed. A failed summary
becomes an unavailable placeholder immediately; a summary still
pending keeps the image (and its label) for up to
``pending_grace_steps`` further depths, after which a pending
placeholder replaces it. Both placeholders carry the DataEngine step
reference. An image with no summary job at all is dropped silently.
3. Frozen: after the image is resolved the message is never read or
written again; a late summary never backfills a frozen message.
Contract: the message list is append-only (FlashRunner never removes or
reorders entries), so bookkeeping is index-based and each pass only
@@ -72,15 +96,14 @@ class ScrubEdgeCompressor:
whole history.
The Pro transcript ledger uses this discipline over its active region.
Its observation messages carry no
``tool_call_id``, so ``summary_key_getter`` supplies the summary-job key
(the DataEngine step id) per message; ``strip_markers`` extends the
depth-1 strip to the Pro tail blocks (UI element list + plan recitation);
and ``tail_offset=1`` accounts for the live observation living outside
the tracked list (the ledger renders it as a separate tail), so depth
arithmetic and the keep-window stay aligned with the Flash semantics
where the live observation is *inside* the list. Defaults leave the
Flash behavior byte-identical.
Its observation messages carry no ``tool_call_id``, so
``summary_key_getter`` supplies the summary-job key (the DataEngine step
id) per message; ``strip_markers`` extends the strip to the Pro tail
blocks (UI element list + plan recitation); and ``tail_offset=1``
accounts for the live observation living outside the tracked list (the
ledger renders it as a separate tail), so depth arithmetic stays aligned
with the Flash semantics where the live observation is *inside* the
list.
"""
def __init__(
@@ -90,7 +113,7 @@ class ScrubEdgeCompressor:
prune_history_xml: bool = True,
image_scrub_depth: int = 3,
pending_grace_steps: int = 3,
xml_scrub_depth: int = 1,
xml_scrub_depth: int | None = 1,
summary_key_getter: Callable[[BaseMessage], str | None] | None = None,
strip_markers: tuple[str, ...] = (_UI_LIST_MARKER,),
tail_offset: int = 0,
@@ -99,7 +122,9 @@ class ScrubEdgeCompressor:
self._prune_history_xml = prune_history_xml
self._image_scrub_depth = max(1, image_scrub_depth)
self._pending_grace_steps = max(0, pending_grace_steps)
self._xml_scrub_depth = max(1, xml_scrub_depth)
# Text edge: the heavy text blocks and ephemeral blocks leave once a
# message is deeper than this (None keeps the default of 1).
self._xml_scrub_depth = max(1, int(xml_scrub_depth)) if xml_scrub_depth is not None else 1
self._summary_key_getter = summary_key_getter
self._strip_markers = tuple(strip_markers) or (_UI_LIST_MARKER,)
self._tail_offset = max(0, tail_offset)
@@ -110,15 +135,28 @@ class ScrubEdgeCompressor:
# {"idx", "key" (str tool_call_id | None), "legacy" (ordinal | None), "is_tool"}
self._tracked: list[dict[str, Any]] = []
self._frozen: set[int] = set()
# Messages whose one-time text edits (strip + ephemeral deletion) are
# done; a message inside the pending-grace window sits here unfrozen.
self._text_scrubbed: set[int] = set()
# Strip-marker carriers without an image (e.g. a step whose screenshot
# failed); they are stripped when the edge passes their position.
self._xml_candidates: list[int] = []
def compress(self, messages: list[BaseMessage]) -> None:
"""Advance the scrub edge over newly appended messages in-place."""
def compress(
self, messages: list[BaseMessage], *, image_scrub_depth: int | None = None
) -> None:
"""Advance the scrub edge over newly appended messages in-place.
``image_scrub_depth`` overrides the constructor depth for this pass
(the transcript ledger passes its occupancy-driven depth). Because a
scrubbed message is frozen, a deeper pass after a shallower one never
backfills anything: relaxing the depth only lets *new* images live
longer, tightening it scrubs the extra depths once.
"""
self._discover(messages)
if not self._tracked:
if not self._tracked and not self._xml_candidates:
return
self._scrub_xml(messages)
self._scrub_images(messages)
self._scrub_edge(messages, image_scrub_depth)
# ------------------------------------------------------------------
# Discovery (incremental; frozen prefix is never rescanned)
@@ -144,8 +182,6 @@ class ScrubEdgeCompressor:
elif block.get("type") == "text" and self._has_strip_marker(block.get("text", "")):
has_xml = True
if has_xml:
self._xml_candidates.append(idx)
if has_image:
key = str(tool_call_id) if tool_call_id is not None else None
if key is None and self._summary_key_getter is not None:
@@ -163,62 +199,24 @@ class ScrubEdgeCompressor:
"is_tool": tool_call_id is not None,
}
)
elif has_xml:
self._xml_candidates.append(idx)
self._scanned_until = len(messages)
def _has_strip_marker(self, text: str) -> bool:
return any(marker in text for marker in self._strip_markers)
# ------------------------------------------------------------------
# Depth-1 XML scrub
# The edges: text strip + ephemeral deletion (shallow), image swap (K)
# ------------------------------------------------------------------
def _scrub_xml(self, messages: list[BaseMessage]) -> None:
if not self._prune_history_xml:
return
# Keep the UI list only in the newest xml_scrub_depth observation
# messages; everything older (or non-image XML carriers) is stripped.
# With a live tail outside the tracked list (tail_offset > 0) the tail
# itself occupies the newest slots of the keep window.
keep_count = max(0, self._xml_scrub_depth - self._tail_offset)
keep_indices = {rec["idx"] for rec in self._tracked[-keep_count:]} if keep_count else set()
still_pending: list[int] = []
for idx in self._xml_candidates:
if idx in keep_indices:
still_pending.append(idx)
continue
if idx in self._frozen:
continue
self._strip_xml_blocks(messages[idx])
self._xml_candidates = still_pending
def _strip_xml_blocks(self, msg: BaseMessage) -> None:
content = getattr(msg, "content", None)
if not isinstance(content, list):
return
new_blocks: list[Any] = []
for block in content:
if (
isinstance(block, dict)
and block.get("type") == "text"
and self._has_strip_marker(block.get("text", ""))
):
retained_text = _without_marked_suffix(block.get("text", ""), self._strip_markers)
if retained_text:
retained_block = dict(block)
retained_block["text"] = retained_text
new_blocks.append(retained_block)
continue
new_blocks.append(block)
msg.content = new_blocks
# ------------------------------------------------------------------
# Depth-K image scrub
# ------------------------------------------------------------------
def _scrub_images(self, messages: list[BaseMessage]) -> None:
def _scrub_edge(
self, messages: list[BaseMessage], image_scrub_depth: int | None = None
) -> None:
total = len(self._tracked)
scrub_depth = self._image_scrub_depth
scrub_depth = max(1, int(image_scrub_depth or self._image_scrub_depth))
grace_limit = scrub_depth + self._pending_grace_steps
text_depth = self._xml_scrub_depth
for pos, rec in enumerate(self._tracked):
if rec["idx"] in self._frozen:
@@ -226,12 +224,25 @@ class ScrubEdgeCompressor:
# 1 == most recent image-bearing message; a live tail outside the
# tracked list (tail_offset > 0) counts as the newest depths.
depth = total - pos + self._tail_offset
if depth < scrub_depth:
continue # active window
if self._tail_offset == 0 and pos == total - 1:
continue # the live observation is never scrubbed
past_text_edge = depth > text_depth
past_image_edge = depth >= scrub_depth
if not past_text_edge and not past_image_edge:
continue # newer than both edges: byte-identical
if not past_image_edge:
# Past the text edge only: strip once, keep the screenshot.
if rec["idx"] not in self._text_scrubbed:
self._rewrite(
messages[rec["idx"]],
rec["idx"],
replacement=_KEEP_IMAGE,
is_tool=rec["is_tool"],
)
continue
summary, pending, failed, step_no = self._lookup(rec)
replacement: Any
if summary is not None:
replacement = {
"type": "text",
@@ -243,7 +254,7 @@ class ScrubEdgeCompressor:
"text": f"[visual summary unavailable; evidence at DataEngine step {step_no}]",
}
elif pending and depth <= grace_limit:
continue # grace period: retain the image, revisit next pass
replacement = _KEEP_IMAGE # grace: text edits now, image swap later
elif pending:
replacement = {
"type": "text",
@@ -252,7 +263,26 @@ class ScrubEdgeCompressor:
else:
replacement = None # no summary job exists; drop the image silently
self._resolve(messages, rec, replacement)
self._resolve(messages, rec, replacement, text_edits=past_text_edge)
if self._xml_candidates:
# Imageless strip-marker carriers (a step whose screenshot failed)
# follow the same text edge, counted over every observation —
# image-bearing or not — plus the live tail: with the default
# depth 1 the list leaves as soon as any newer observation exists,
# so only the live observation ever carries an indexed element
# list. A carrier that is itself the newest observation stays.
observations = sorted({rec["idx"] for rec in self._tracked} | set(self._xml_candidates))
remaining: list[int] = []
for idx in self._xml_candidates:
newer = sum(1 for other in observations if other > idx) + self._tail_offset
if newer < text_depth:
remaining.append(idx)
continue
if idx not in self._frozen:
self._rewrite(messages[idx], idx, replacement=None, is_tool=False)
self._frozen.add(idx)
self._xml_candidates = remaining
def _lookup(self, rec: dict[str, Any]) -> tuple[str | None, bool, bool, int | None]:
"""Mirror the legacy keying: tool_call_id first, ordinal fallback."""
@@ -280,42 +310,143 @@ class ScrubEdgeCompressor:
self,
messages: list[BaseMessage],
rec: dict[str, Any],
replacement: dict[str, Any] | None,
replacement: Any,
*,
text_edits: bool = True,
) -> None:
"""Replace the message's image blocks and freeze it permanently."""
msg = messages[rec["idx"]]
content = getattr(msg, "content", None)
new_blocks: list[Any] = []
if isinstance(content, list):
for block in content:
if isinstance(block, dict) and block.get("type") in ("image_url", "image"):
continue
if (
self._prune_history_xml
and isinstance(block, dict)
and block.get("type") == "text"
and self._has_strip_marker(block.get("text", ""))
):
# Freezing implies fully scrubbed: no stale UI list survives
# even when xml_scrub_depth exceeds the image depth.
retained_text = _without_marked_suffix(
block.get("text", ""), self._strip_markers
)
if retained_text:
retained_block = dict(block)
retained_block["text"] = retained_text
new_blocks.append(retained_block)
continue
new_blocks.append(block)
"""Rewrite one image-bearing message at the screenshot edge; freeze it
unless the screenshot is kept for the pending-grace window. With a text
edge deeper than the screenshot edge the text edits wait for their own
edge (``text_edits=False``); the message is then frozen only once both
have happened."""
idx = rec["idx"]
self._rewrite(
messages[idx],
idx,
replacement=replacement,
is_tool=rec["is_tool"],
text_edits=text_edits,
)
if not text_edits:
return # text edge still ahead: revisit for the strip
if replacement is not _KEEP_IMAGE:
self._frozen.add(idx)
if replacement is not None:
new_blocks.append(replacement)
def _rewrite(
self,
msg: BaseMessage,
idx: int,
*,
replacement: Any,
is_tool: bool,
text_edits: bool = True,
) -> None:
"""The single-pass edit of one message (see the class docstring).
``replacement`` is the block that takes the first image's place (a
summary or placeholder), ``None`` to drop the image, or
:data:`_KEEP_IMAGE` to leave the image and its label in place. The
text edits (strip + ephemeral deletion) run only the first time the
message is rewritten and consume the ephemeral indices exactly once.
Any rewrite that happens *before* that first edit (the image swap when
the screenshot edge is shallower than the text edge, or a summary
arriving inside the grace window) can remove the label and the image
and so shift every later block; the surviving ephemeral indices are
then remapped to the new layout in ``additional_kwargs`` so the later
text edit still deletes the blocks that were marked, never a
neighbour that slid into their old position.
"""
content = getattr(msg, "content", None)
if not isinstance(content, list):
self._text_scrubbed.add(idx)
return
first_edit = idx not in self._text_scrubbed and text_edits
ephemeral: set[int] = set()
if first_edit:
ephemeral = self._ephemeral_indices(msg)
keep_image = replacement is _KEEP_IMAGE
new_blocks: list[Any] = []
# old position -> new position of every block that survives as itself
# (a replacement block takes the image's slot but is not the image).
position_map: dict[int, int] = {}
image_placed = False
for position, block in enumerate(content):
if position in ephemeral:
continue
if isinstance(block, dict):
kind = block.get("type")
if kind in ("image_url", "image"):
if keep_image:
position_map[position] = len(new_blocks)
new_blocks.append(block)
elif replacement is not None and not image_placed:
new_blocks.append(replacement) # in place of the image
image_placed = True
continue
if kind == "text":
text = block.get("text", "")
if not keep_image and text.strip() == SCREENSHOT_LABEL:
continue # the label leaves with the image it labelled
if first_edit and self._prune_history_xml and self._has_strip_marker(text):
retained_text = _without_marked_suffix(text, self._strip_markers)
if retained_text:
retained_block = dict(block)
retained_block["text"] = retained_text
position_map[position] = len(new_blocks)
new_blocks.append(retained_block)
continue
position_map[position] = len(new_blocks)
new_blocks.append(block)
# Defensive fallback: never leave ToolMessage or HumanMessage content empty
if not new_blocks:
new_blocks = [{"type": "text", "text": "Action dispatched." if rec["is_tool"] else ""}]
new_blocks = [{"type": "text", "text": "Action dispatched." if is_tool else ""}]
position_map = {}
msg.content = new_blocks
self._frozen.add(rec["idx"])
if rec["idx"] in self._xml_candidates:
self._xml_candidates.remove(rec["idx"])
if first_edit:
self._text_scrubbed.add(idx)
if ephemeral:
# Consumed: the indices no longer describe the block layout.
msg.additional_kwargs.pop(EPHEMERAL_BLOCKS_KEY, None)
else:
self._remap_ephemeral(msg, position_map)
@staticmethod
def _remap_ephemeral(msg: BaseMessage, position_map: dict[int, int]) -> None:
"""Rewrite the not-yet-consumed ephemeral indices of ``msg`` through
``position_map`` (old block position -> new block position); indices of
blocks that did not survive are dropped and the key is removed when
nothing survives. Only ``additional_kwargs`` is touched, the content
blocks sent to the provider stay exactly as assembled."""
kwargs = getattr(msg, "additional_kwargs", None)
if not isinstance(kwargs, dict) or EPHEMERAL_BLOCKS_KEY not in kwargs:
return
remapped: list[int] = []
for value in kwargs.get(EPHEMERAL_BLOCKS_KEY) or []:
try:
old_position = int(value)
except (TypeError, ValueError):
continue
new_position = position_map.get(old_position)
if new_position is not None and new_position not in remapped:
remapped.append(new_position)
if remapped:
kwargs[EPHEMERAL_BLOCKS_KEY] = remapped
else:
kwargs.pop(EPHEMERAL_BLOCKS_KEY, None)
@staticmethod
def _ephemeral_indices(msg: BaseMessage) -> set[int]:
kwargs = getattr(msg, "additional_kwargs", None)
if not isinstance(kwargs, dict):
return set()
indices: set[int] = set()
for value in kwargs.get(EPHEMERAL_BLOCKS_KEY) or []:
try:
indices.add(int(value))
except (TypeError, ValueError):
continue
return indices
+8 -7
View File
@@ -6,9 +6,9 @@ You are an autonomous and highly efficient Android Device Execution Agent. Your
---
# 1. COGNITIVE PROTOCOL
You operate in a dynamic observation-reasoning-execution loop. **CRITICAL RULE: YOU MUST NEVER INVOKE A TOOL SILENTLY!** In every single turn, you **MUST FIRST output a natural language reasoning paragraph BEFORE generating any tool call.**
In every turn, think through observation, planning and action in that order. Before calling an action tool or `report_task_status` (the Turn-Ending Action), your reply carries one short paragraph of connected prose reflecting those three phases; that written text is kept in the execution history and shown to the user.
- **Turn 1 Only (Initial Planning)**: Briefly decompose the overall `{{ goal }}` into high-level visual **Milestones** in your text output. Do not repeat this macro breakdown in subsequent turns.
- **Turn 1 Only (Initial Planning)**: Briefly decompose the objective into high-level visual **Milestones** in your text output. Do not repeat this macro breakdown in subsequent turns.
---
@@ -23,7 +23,7 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
---
# 3. READING YOUR HISTORY & THE SESSION CLOCK
- **Conversation = execution history**: The earlier turns of this conversation are your raw execution history. Each observation opens with `# CURRENT OBSERVATION [T+mm:ss]`, and each turn that executed a device action ends with an `--- Action Execution Result (T+mm:ss) ---` message (`Status: dispatched` means the device accepted the action; whether it had the intended effect is yours to judge from the observation that follows. `Status: failed` carries the `Error:` line). Every timestamp is a session-relative `T+mm:ss` offset from the start of this session; it is the only clock you have, for pacing waits as well as for reasoning about elapsed time.
- **Conversation = execution history**: The earlier turns of this conversation are your raw execution history. Each observation opens with `# CURRENT OBSERVATION [T+mm:ss]`. Every device action runs immediately and its tool result states what the device did (e.g. `Swiped up.`); whether it had the intended effect is yours to judge from the observation that follows. Only a turn whose action failed ends with an `--- Action Execution Result (T+mm:ss) ---` message (`Status: failed` with the `Error:` line). Every timestamp is a session-relative `T+mm:ss` offset from the start of this session; it is the only clock you have, for pacing waits as well as for reasoning about elapsed time.
- **Compression**: Screenshots older than a few turns are replaced by short `--- Historical Visual Transition ---` summaries and old `--- Visible UI Elements ---` lists are dropped. Long spans render as `[Era N | Steps a–b | T+.. → T+..]` blocks (a merged synopsis, segment titles, and a per-step action ledger). Because of this, if you spot critical text that you will need later (e.g., verification codes, specific titles, search lists), **you must explicitly verbalize and write those details into your text reasoning** so they persist across turns.
{% if "search_history" in available_tools %}- **History Lookup (`search_history`)**: Compressed history keeps the step numbers and `T+` offsets but not every detail. When you need exactly what you did or saw at an earlier step (a value you read, the result of an action, which screen you were on), call `search_history` with keywords and/or the `[start, end]` step range shown in a chunk block; it searches the full stored history (screen descriptions, actions, results, reasoning, tool calls, notes, on-screen text) and every hit carries its step number. It is a pre-decision helper: read its answer, then act.{% endif %}
{% if "replay_steps" in available_tools %}- **Step Replay (`replay_steps`)**: Replays one earlier step (or a short range) exactly as your context showed it — screen description, reasoning, tool calls with their results, the action and its execution result. Use the step numbers shown in the compressed history{% if "search_history" in available_tools %} or returned by `search_history`{% endif %}.{% endif %}
@@ -39,12 +39,13 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
---
# 5. ACTION RULES
1. **Normalized Coordinates (`0-1000` Scale)**: Always use normalized `[x, y]` coordinates in `0-1000` scale for any location-based target (e.g., `target=[500, 600]` or `click_sequence=[[300, 400], [500, 600]]`). Every coordinate target must carry `target_description` (one entry per point for `click_sequence`): a short, meaningful name for what you are aiming at, as a user would call it (e.g. 'play button', 'video body'), never a generic 'element' or 'button'. Nothing is inferred on your behalf: it is recorded as your own statement, and your history shows it marked as self-described.
2. **Do not determine element coordinates based on guesswork.** If the element is visible in the screenshot but its coordinates are missing from the provided `--- Visible UI Elements ---` list, please use the `ask_explorer` tool to locate the element first.
{% if "manage_app" in available_tools %}3. **App Launching**: To start or reset an application, always use `manage_app(action="launch", app_name="...")` or `action="stop"`. Do not manually swipe across home screens.{% endif %}
1. **Prefer the element index.** The `--- Visible UI Elements ---` list numbers every element it found (`[3] Text: 'Wi-Fi' | Bounds: ...`). When your target is listed, address it by that index (`target=3`, a bare integer) with `click`, `long_press` or `input_text`: the tap lands on the element's center and the element's own text and bounds are recorded with the action. Heed the overlap warnings in the list; an overlapped element may not receive the tap. Indices belong to the screen they were listed on: swiping, scrolling or a page update renumbers them, so only ever use an index from the list in the current observation, never one remembered from an earlier turn. Within one turn, every index you call (even after an earlier action in the same turn) refers to the list in the current observation, not to the screen your earlier action produced.
2. **Normalized Coordinates (`0-1000` Scale) otherwise.** When the target is visible but not listed, use normalized `[x, y]` coordinates in `0-1000` scale (e.g., `target=[500, 600]`; `click_sequence` always takes coordinate pairs, e.g. `sequence=[[300, 400], [500, 600]]`). Every coordinate target must carry `target_description` (one entry per point for `click_sequence`): a short, meaningful name for what you are aiming at, as a user would call it (e.g. 'play button', 'video body'), never a generic 'element' or 'button'. Nothing is inferred on your behalf: it is recorded as your own statement, and your history shows it marked as self-described.
3. **Do not determine element coordinates based on guesswork.** If the element is visible in the screenshot but neither listed nor safely locatable from the screenshot, use the `ask_explorer` tool to locate it first; its candidates join the element list and can be addressed by index or by coordinate.
{% if "manage_app" in available_tools %}4. **App Launching**: To start or reset an application, always use `manage_app(action="launch", app_name="...")` or `action="stop"`. Do not manually swipe across home screens.{% endif %}
---
# 6. TERMINATION PROTOCOL (`report_task_status`)
- **Completion (`status="completed"`)**: ONLY call `report_task_status(status="completed", ...)` after you have **visually verified** from the latest screen that the user's objective (`{{ goal }}`) is fully achieved.
- **Completion (`status="completed"`)**: ONLY call `report_task_status(status="completed", ...)` after you have **visually verified** from the latest screen that the objective is fully achieved.
- **Failure (`status="failed"`)**: If the task is genuinely impossible due to paywalls, unrecoverable crashes, or hitting an absolute dead end after sensible fallbacks, call `report_task_status(status="failed", ...)` with the exact blocking reason.
+302 -99
View File
@@ -22,10 +22,19 @@ the prompt is built every turn from a session :class:`TranscriptLedger` —
a byte-stable system prefix, the committed earlier turns (append-only; old UI
lists stripped at depth 1, screenshots resolved to visual summaries at depth
K, long spans chunk-compressed), and a fresh tail carrying the current
observation under a ``# CURRENT OBSERVATION [T+mm:ss]`` header. Each
committed turn ends with an ``--- Action Execution Result (T+mm:ss) ---``
message, so every timestamp the model sees is a session-relative offset — the
same clock the video analyzer uses for the session recording.
observation under a ``# CURRENT OBSERVATION [T+mm:ss]`` header. Actions run
synchronously and their tool messages already carry the outcome; only a turn
whose action failed ends with an ``--- Action Execution Result (T+mm:ss) ---``
message (it carries the error). Every timestamp the model sees is a
session-relative offset — the same clock the video analyzer uses for the
session recording.
Per-turn notices on the tail (the reasoning reminder after a silent turn, the
no-tool-call nudge, the final-turn warning, the user-guidance wrapper) are
marked ephemeral: the ledger's scrub edge deletes them once the message leaves
the recent window, so they never reach the frozen history or a chunk capsule.
The verbatim body of a user instruction is the exception: it stays in the
active window until its turn is chunked, whose user lines then carry it on.
"""
import asyncio
@@ -45,9 +54,12 @@ from langchain_core.messages import (
)
from artemis.agents.flash.summarizer import VisualStepSummarizer, build_focus_context
from artemis.agents.operator.prompts import REASONING_REMINDER, UserGuidance, render_user_guidance
from artemis.agents.validator.tool_declarations import (
ASK_EXPLORER_TOOL,
CLICK_SEQUENCE_TOOL,
LIST_NOTES_TOOL,
READ_NOTE_TOOL,
REPORT_TASK_STATUS_TOOL,
VALIDATOR_TOOLS_DECLARATION,
capture_screenshot_and_parse_ui,
@@ -66,7 +78,8 @@ from artemis.graph.perception import _check_injected_instruction_file
from artemis.graph.state import State
from artemis.llm.structured import ParseFailure, parse_structured
from artemis.mcp.action_executor import McpActionExecutor
from artemis.memory.transcript import PRO_UI_LIST_MARKER, TranscriptLedger
from artemis.mcp.observation import observe
from artemis.memory.transcript import PRO_UI_LIST_MARKER, TranscriptLedger, mark_ephemeral
from artemis.services.llm import (
RobustChatModelWrapper,
acomplete,
@@ -85,12 +98,6 @@ from artemis.utils.logger import get_logger
logger = get_logger(__name__)
#: Reminder carried by the first observation only (the system prompt states the rule).
_REASONING_FIRST_REMINDER = (
"CRITICAL RULE: In every single turn, you MUST FIRST output a natural"
" language reasoning/explanation paragraph BEFORE invoking any tool call."
)
_NO_TOOL_CALL_NOTICE = (
"You did not call any tools last turn. Please make progress by calling an"
" action tool or 'report_task_status'."
@@ -116,17 +123,17 @@ class _TurnRecord:
def result(self) -> dict | None:
"""The turn's execution result in the validator-report shape.
``None`` when no device action ran (helper-only turns have no result
message, as in Pro).
``None`` unless an action failed: Flash executes synchronously and each
action's tool message already carries its outcome, so a result message
would only repeat it. A failure is worth its own message because it
carries the error the turn ended on.
"""
if not self.actions:
return None
for name, status, text in self.actions:
if status != "success":
# Executor messages name the action themselves ("Error
# executing click: ..."); prefix only the ones that don't.
return {"status": "failed", "error": text if name in text else f"{name}: {text}"}
return {"status": "dispatched"}
return None
class FlashRunner:
@@ -164,7 +171,7 @@ class FlashRunner:
if self.step_summarizer_cfg.enabled
else None
)
# Publish the service on the composition root (ctx.step_memory, M2) so
# Publish the service on the composition root (ctx.step_memory) so
# any co-resident consumer shares this run's summary runtime.
if self.summarizer is not None and getattr(ctx, "step_memory", None) is None:
try:
@@ -187,7 +194,13 @@ class FlashRunner:
return getattr(setup, "video_recording_tools_enabled", False) is True
def _get_tools(self) -> list:
tools = list(VALIDATOR_TOOLS_DECLARATION)
# Flash has no tool that writes notes and its prompt never teaches
# them, so the Validator set's note readers are left out here.
tools = [
t
for t in VALIDATOR_TOOLS_DECLARATION
if t is not READ_NOTE_TOOL and t is not LIST_NOTES_TOOL
]
tools.insert(1, CLICK_SEQUENCE_TOOL)
tools.append(ASK_EXPLORER_TOOL)
# Helper tools shared with the Pro operator: the history tools are
@@ -235,6 +248,9 @@ class FlashRunner:
image_scrub_depth=getattr(cfg, "image_scrub_depth", 3),
pending_grace_steps=getattr(cfg, "pending_grace_steps", 3),
xml_scrub_depth=getattr(cfg, "xml_scrub_depth", 1),
image_scrub_depth_relaxed=getattr(cfg, "image_scrub_depth_relaxed", None),
context_budget_tokens=getattr(cfg, "context_budget_tokens", None),
start_ratio=getattr(cfg, "start_ratio", None),
session_start=session_start if isinstance(session_start, (int, float)) else None,
)
if engine is not None:
@@ -286,14 +302,19 @@ class FlashRunner:
# Per-turn helpers (observe / think)
# ------------------------------------------------------------------
async def _read_injected_instruction(self) -> tuple[str | None, str | None]:
"""Consumes this turn's injected instruction: ``(instruction, notice)``.
async def _read_injected_instruction(
self, offset_label: str | None = None
) -> tuple[str | None, UserGuidance | None]:
"""Consumes this turn's injected instruction: ``(instruction, guidance)``.
``instruction`` is the user's text verbatim — what the step record,
the chunk ledger and the visual-transition lens receive (the same
shape Pro stamps). ``notice`` wraps it with the operator-facing
directive and rides only on the observation tail. Both are None
when nothing was injected.
shape Pro stamps). ``guidance`` is the shared user-guidance pair
(:func:`render_user_guidance`, the Pro rendering without a plan to
edit): the ephemeral ``--- User Guidance ---`` wrapper and the
persistent verbatim body, both riding on the observation tail.
``offset_label`` stamps the body with the session offset. Both are
None when nothing was injected.
"""
if not (self.ctx.data_engine and self.ctx.data_engine.base_dir):
return None, None
@@ -308,18 +329,13 @@ class FlashRunner:
if not (injected_payload and injected_payload.get("instruction")):
return None, None
instruction = str(injected_payload["instruction"])
notice = (
"[REAL-TIME INJECTED INSTRUCTION from user]:"
f" {instruction}\nYou MUST immediately"
" follow this instruction and adjust your plan/actions."
guidance = render_user_guidance(
instruction,
release_loop=bool(injected_payload.get("release_loop")),
has_plan=False,
offset_label=offset_label,
)
if injected_payload.get("release_loop"):
notice += (
"\nThe user has explicitly authorized stopping any"
" ongoing monitoring loop; you may now wrap up and"
" complete the task."
)
return instruction, notice
return instruction, guidance
def _build_tail(
self,
@@ -328,19 +344,28 @@ class FlashRunner:
img_bytes,
xml_list,
*,
injected: str | None = None,
injected: UserGuidance | None = None,
notices: list[str] | None = None,
is_final: bool = False,
previous_turn_silent: bool = False,
) -> HumanMessage:
"""Builds this turn's observation tail (Pro observation shape).
Header, screenshot and UI list carry the same markers as the Pro
operator's tail so the ledger's scrub edge treats both alike.
operator's tail so the ledger's scrub edge treats both alike. The
objective is stated once, in the system prompt; the tail never repeats
it. Every per-turn notice (nudges, the user-guidance wrapper, the
final-turn warning, the reasoning reminder after a silent turn) is
marked ephemeral so it leaves the transcript with the recent window.
The verbatim body of an injected instruction is the one exception: it
is a regular block, so a standing instruction stays in the active
window until the turn is chunked (the chunk's user lines carry it on).
"""
blocks: list[dict] = []
if turns == 1:
blocks.append({"type": "text", "text": f"Your objective is: {self.goal}"})
blocks.append({"type": "text", "text": f"# CURRENT OBSERVATION [{ledger.elapsed_label()}]"})
# ``turns`` is kept for call-site symmetry: the tail has the same shape
# on every turn.
blocks: list[dict] = [
{"type": "text", "text": f"# CURRENT OBSERVATION [{ledger.elapsed_label()}]"}
]
if img_bytes:
img_b64 = base64.b64encode(img_bytes).decode("utf-8")
blocks.append({"type": "text", "text": "--- Current Screenshot ---"})
@@ -349,15 +374,26 @@ class FlashRunner:
)
if xml_list:
blocks.append({"type": "text", "text": f"{PRO_UI_LIST_MARKER}\n{xml_list}"})
for notice in notices or []:
blocks.append({"type": "text", "text": notice})
ephemeral: list[int] = []
# (text, ephemeral) in tail order.
per_turn: list[tuple[str, bool]] = [(text, True) for text in (notices or [])]
if injected:
blocks.append({"type": "text", "text": injected})
# The wrapper is per-turn; the verbatim body right after it is a
# regular block that stays until the turn is chunked.
per_turn.append((injected.wrapper, True))
per_turn.append((injected.body, False))
if is_final:
blocks.append({"type": "text", "text": _FINAL_TURN_WARNING})
if turns == 1:
blocks.append({"type": "text", "text": _REASONING_FIRST_REMINDER})
return HumanMessage(content=blocks)
per_turn.append((_FINAL_TURN_WARNING, True))
if previous_turn_silent:
per_turn.append((REASONING_REMINDER, True))
for text, is_ephemeral in per_turn:
if is_ephemeral:
ephemeral.append(len(blocks))
blocks.append({"type": "text", "text": text})
tail = HumanMessage(content=blocks)
if ephemeral:
mark_ephemeral(tail, ephemeral)
return tail
def _extract_response_text(self, response) -> str:
"""Extracts the natural-language thought text from the model response."""
@@ -368,6 +404,22 @@ class FlashRunner:
)
return raw_text
def _extract_response_thinking(self, response) -> str:
"""Extracts the model's native thinking blocks (thought summaries), if any.
Providers running with ``include_thoughts`` return them as ``thinking``
content blocks next to the ``text`` blocks; they are recorded on the step
as ``operator_native_thinking`` and never fed back as reply text.
"""
if not isinstance(response.content, list):
return ""
parts = [
str(b.get("thinking") or "")
for b in response.content
if isinstance(b, dict) and b.get("type") == "thinking"
]
return "\n".join(p for p in parts if p.strip())
def _token_usage_from_response(self, response) -> dict | None:
"""Extracts token usage metadata from the model response, if present."""
step_token_usage = None
@@ -435,12 +487,20 @@ class FlashRunner:
"total_tokens": max(1, prompt_tokens + completion_tokens),
}
def _resolve_token_usage(self, response, messages: list[BaseMessage], raw_text: str) -> dict:
"""Extracts token usage from response metadata, estimating as fallback."""
def _resolve_token_usage(
self, response, messages: list[BaseMessage], raw_text: str
) -> tuple[dict, bool]:
"""Token usage for the call and whether the provider measured it.
Falls back to an estimate when the response carries no usage
metadata; the flag lets the caller keep an estimate out of the
ledger's chars-per-token calibration (an estimate is itself derived
from character counts, so calibrating on it would be circular).
"""
step_token_usage = self._token_usage_from_response(response)
if not step_token_usage or step_token_usage.get("total_tokens", 0) <= 0:
step_token_usage = self._estimate_token_usage(messages, raw_text)
return step_token_usage
if step_token_usage and step_token_usage.get("total_tokens", 0) > 0:
return step_token_usage, True
return self._estimate_token_usage(messages, raw_text), False
def _record_llm_trace(self, step_token_usage: dict, raw_text: str) -> None:
"""Records the llm_call trace for this turn in the DataEngine."""
@@ -490,6 +550,7 @@ class FlashRunner:
pre_screenshot_bytes,
xml_list,
messages: list[BaseMessage],
native_text: str | None = None,
) -> dict:
"""Records the final step, acknowledges the tool call, and flushes."""
final_report = args
@@ -502,6 +563,7 @@ class FlashRunner:
ui_tree=xml_list,
action_taken={"action": "report_task_status", "args": args},
operator_raw_thinking=raw_text,
operator_native_thinking=native_text or None,
last_execution_result={"result": "Task completed with final report."},
extra_metadata={"token_usage": step_token_usage},
)
@@ -519,25 +581,56 @@ class FlashRunner:
await self.summarizer.flush()
return final_report
async def _capture_post_screenshot(self, exec_result, name: str, action_names):
"""Returns the post-action screenshot, capturing a fallback if needed."""
async def _capture_post_screenshot(
self, exec_result, name: str, action_names, state: State | None = None
):
"""Returns the post-action screenshot, capturing a fallback if needed.
When a device action produced no observation (a failed action, an
unknown tool, ...) the screen is observed through the very same
pipeline the executor's success path uses (:func:`observe`), so the
next observation tail carries the indexed minimal UI list and
``state`` gets the same refreshed screenshot path and indexed
elements. Nothing is captured for non-action tools.
"""
post_img_bytes = exec_result.screenshot_bytes
if not post_img_bytes and name in action_names:
try:
controller = UnifiedMobileController(self.ctx)
screen_data = await controller.get_screen_data()
post_img_bytes = base64.b64decode(screen_data.base64)
if not exec_result.ui_elements_text:
exec_result.ui_elements_text = screen_data.elements
except Exception as shot_err:
logger.warning(f"Failed to capture fallback screenshot in FlashRunner: {shot_err}")
if post_img_bytes or name not in action_names:
return post_img_bytes
try:
obs, screenshot_bytes = await observe(self.ctx, self.controller, settle_ms=0)
except Exception as shot_err:
logger.warning(f"Failed to capture fallback screenshot in FlashRunner: {shot_err}")
return post_img_bytes
if not obs.ok:
logger.warning(f"Failed to capture fallback screenshot in FlashRunner: {obs.message}")
return post_img_bytes
post_img_bytes = screenshot_bytes
if not exec_result.ui_elements_text:
exec_result.ui_elements_text = obs.elements_text
if not exec_result.screenshot_path and obs.screenshot_path:
exec_result.screenshot_path = obs.screenshot_path
if state is not None:
if obs.hierarchy_ok:
state.indexed_points = [el["center"] for el in obs.elements]
state.indexed_elements = obs.elements
if obs.screenshot_path:
state.latest_screenshot = obs.screenshot_path
return post_img_bytes
def _extract_normalized_coordinates(self, name: str, args: dict):
"""Extracts and enriches coordinate metadata for the recorded action."""
def _extract_normalized_coordinates(self, name: str, args: dict, resolved=None):
"""Extracts and enriches coordinate metadata for the recorded action.
``resolved`` is the normalized point the executor sent a point action
to; it is the coordinate of record when the model addressed an element
by index (the arguments then carry no point at all).
"""
norm_coords = None
norm_start = None
norm_end = None
if name in ("click", "tap", "long_press", "input_text") and (
isinstance(resolved, (list, tuple)) and len(resolved) == 2
):
return [int(resolved[0]), int(resolved[1])], None, None
if name == "swipe":
kind, target_val, _ = parse_swipe_parameters(args)
if kind == "coords" and isinstance(target_val, list) and len(target_val) == 4:
@@ -567,6 +660,40 @@ class FlashRunner:
norm_coords = [int(float(nums[0])), int(float(nums[1]))]
return norm_coords, norm_start, norm_end
def _build_action_record(self, name: str, args: dict, exec_result) -> dict:
"""The action as the step record and the visual lens see it.
Flash records the model's own 0–1000 coordinates verbatim (an index
target records the point the executor resolved it to); the explicit
space marker keeps every later normalization pass (agent-friendly
steps, MCP inspector) a no-op on this record. The tool arguments ride
along under ``args``, and the target semantics the executor validated
(observed element fields for an index, the model's own
``target_description`` for a point) land top level in the shape Pro
records.
"""
metadata = getattr(exec_result, "metadata", None) or {}
resolved = metadata.get("target_coordinates")
norm_coords, norm_start, norm_end = self._extract_normalized_coordinates(
name, args, resolved=resolved
)
given = args.get("target") or args.get("coordinates") or args.get("sequence")
if isinstance(given, (int, float)) and not isinstance(given, bool):
given = None # An element index is not a coordinate.
action_dict = {
"action": name,
"coordinates": given or norm_coords,
COORDINATE_SPACE_KEY: COORDINATE_SPACE_NORMALIZED,
"args": args,
}
if norm_coords:
action_dict["normalized_coordinates"] = norm_coords
if norm_start and norm_end:
action_dict["normalized_start_coordinates"] = norm_start
action_dict["normalized_end_coordinates"] = norm_end
action_dict.update(metadata.get("target_semantics") or {})
return action_dict
def _record_action_step(
self,
name: str,
@@ -578,6 +705,8 @@ class FlashRunner:
xml_list,
post_img_bytes,
injected: str | None = None,
native_text: str | None = None,
action_dict: dict | None = None,
):
"""Records telemetry / step in DataEngine; returns the step id or None."""
recorded_step_id = None
@@ -585,31 +714,8 @@ class FlashRunner:
if self.ctx.data_engine.current_step_id is None:
self.ctx.data_engine.allocate_step_id()
norm_coords, norm_start, norm_end = self._extract_normalized_coordinates(name, args)
# Flash records the model's own 0–1000 coordinates verbatim; the
# explicit space marker keeps every later normalization pass
# (agent-friendly steps, MCP inspector) a no-op on this record.
action_dict = {
"action": name,
"coordinates": (
args.get("target")
or args.get("coordinates")
or args.get("sequence")
or norm_coords
),
COORDINATE_SPACE_KEY: COORDINATE_SPACE_NORMALIZED,
"args": args,
}
if norm_coords:
action_dict["normalized_coordinates"] = norm_coords
if norm_start and norm_end:
action_dict["normalized_start_coordinates"] = norm_start
action_dict["normalized_end_coordinates"] = norm_end
# The model's own statement of what it aimed at (target_description),
# validated and shaped by the executor into the fields Pro records.
action_dict.update((exec_result.metadata or {}).get("target_semantics") or {})
if action_dict is None:
action_dict = self._build_action_record(name, args, exec_result)
succeeded = exec_result.status == "success"
last_execution_result = {
@@ -631,6 +737,7 @@ class FlashRunner:
ui_tree=(exec_result.ui_elements_text or xml_list),
action_taken=action_dict,
operator_raw_thinking=raw_text,
operator_native_thinking=native_text or None,
last_execution_result=last_execution_result,
extra_metadata=extra_metadata,
)
@@ -664,21 +771,35 @@ class FlashRunner:
action_sequence: int,
turn: _TurnRecord,
injected: str | None = None,
native_text: str | None = None,
index_elements: list[dict] | None = None,
):
"""Executes one tool call, records it, and updates loop state.
The tool message carries the outcome text only; the post-action
screenshot and UI list become the next turn's observation tail.
Returns the (possibly updated) pre_screenshot_bytes, xml_list, and
action_sequence for the next iteration.
``index_elements`` is the turn's element snapshot index targets
resolve against (see :meth:`_process_tool_calls`); ``None`` resolves
against the state's current list. Returns the (possibly updated)
pre_screenshot_bytes, xml_list, and action_sequence for the next
iteration.
"""
# Dynamic dispatch set: manifest device actions plus any backend
# extension tools, so extension steps are recorded like actions.
action_names = self.executor.action_tool_names
try:
exec_result = await self.executor.execute(name, args, tc_id, state)
exec_result = await self.executor.execute(
name, args, tc_id, state, index_elements=index_elements
)
post_img_bytes = await self._capture_post_screenshot(exec_result, name, action_names)
post_img_bytes = await self._capture_post_screenshot(
exec_result, name, action_names, state
)
# One record shape for the DataEngine step and the visual lens.
action_dict = (
self._build_action_record(name, args, exec_result) if name in action_names else None
)
# Record telemetry / step in DataEngine
recorded_step_id = None
@@ -693,6 +814,8 @@ class FlashRunner:
xml_list,
post_img_bytes,
injected=injected,
native_text=native_text,
action_dict=action_dict,
)
if name in action_names:
@@ -705,7 +828,9 @@ class FlashRunner:
self.summarizer.dispatch(
step_number=action_sequence,
action_name=name,
action_args=args,
# The recorded action minus its verb: the same shape the Pro
# summarizer hands the lens, so the phrase renders alike.
action_args={k: v for k, v in action_dict.items() if k != "action"},
pre_img_bytes=pre_screenshot_bytes,
post_img_bytes=post_img_bytes,
exec_outcome=exec_result.text_summary,
@@ -780,12 +905,33 @@ class FlashRunner:
action_sequence: int,
turn: _TurnRecord,
injected: str | None = None,
native_text: str | None = None,
):
"""Dispatches the turn's tool calls.
Every index target of the turn resolves against the element list the
model decided on: the list of the observation this turn was built
from, snapshotted here before the first action. Each action still
refreshes ``state.indexed_elements`` (that refreshed list is what the
next observation shows), so without the snapshot ``click(5)`` after
``click(3)`` would be resolved against the screen after the first
click.
The snapshot is the very list object the observation left on the
state (not a copy), and observations replace the state's list rather
than mutating it, so ``ask_explorer`` candidates registered before
any action land in the snapshot by construction. When a helper tool
appends candidates after an action has already replaced the state's
list, the explorer numbered them against that replaced list, so the
snapshot is re-taken from it: indices the explorer just handed out
resolve correctly, at the cost of the pre-action numbering (mixing
the two in one turn is the model's error, not a supported case).
Returns (final_report, pre_screenshot_bytes, xml_list, action_sequence);
final_report is non-None only when 'report_task_status' was called.
"""
action_names = self.executor.action_tool_names
index_elements = self._turn_index_snapshot(state)
for tc in tool_calls:
name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
args = tc.get("args") or {}
@@ -802,9 +948,12 @@ class FlashRunner:
pre_screenshot_bytes,
xml_list,
messages,
native_text=native_text,
)
return final_report, pre_screenshot_bytes, xml_list, action_sequence
elements_before = getattr(state, "indexed_elements", None)
before_len = len(elements_before) if isinstance(elements_before, list) else 0
pre_screenshot_bytes, xml_list, action_sequence = await self._execute_and_record_action(
name,
args,
@@ -818,9 +967,46 @@ class FlashRunner:
action_sequence,
turn,
injected=injected,
native_text=native_text,
index_elements=index_elements,
)
if name not in action_names:
index_elements = self._extend_index_snapshot(state, index_elements, before_len)
return None, pre_screenshot_bytes, xml_list, action_sequence
@staticmethod
def _turn_index_snapshot(state: State) -> list[dict]:
"""The element list this turn's index targets resolve against.
The state's list object itself, so in-place appends (explorer
candidates) before any action join the snapshot; an empty list is
installed on the state when there is none, for the same reason.
"""
elements = getattr(state, "indexed_elements", None)
if not isinstance(elements, list):
elements = []
try:
state.indexed_elements = elements
except (AttributeError, TypeError, ValueError):
pass
return elements
@staticmethod
def _extend_index_snapshot(state: State, snapshot: list[dict], before_len: int) -> list[dict]:
"""Re-takes the snapshot after a helper tool appended candidates to a
list that is no longer the snapshot (an action replaced it first).
Appends into the snapshot itself need nothing: the explorer numbered
them against the snapshot. Only growth of a different list object
moves the snapshot, since that is the list the new numbers refer to.
"""
current = getattr(state, "indexed_elements", None)
if not isinstance(current, list) or current is snapshot:
return snapshot
if len(current) > before_len:
return current
return snapshot
async def _prepare_conversation(self, state: State, tools_declaration: list):
"""Installs the static prefix and captures the initial device state."""
ledger = self._build_ledger()
@@ -883,8 +1069,10 @@ class FlashRunner:
previous_turn = None
# Check for real-time injected instructions: the verbatim text
# is stamped on the step, the wrapped notice goes to the tail.
injected, injected_notice = await self._read_injected_instruction()
# is stamped on the step, the guidance pair goes to the tail.
injected, injected_guidance = await self._read_injected_instruction(
ledger.elapsed_label()
)
# Tool restriction on the final turn (bounded loops only)
is_final = limit is not None and turns == limit
@@ -893,9 +1081,12 @@ class FlashRunner:
turns,
current_pre_screenshot_bytes,
current_xml_list,
injected=injected_notice,
injected=injected_guidance,
notices=pending_notices,
is_final=is_final,
# The ledger judged the turn it just committed: a bare tool
# call (or thinking only) earns a one-line reminder, no bounce.
previous_turn_silent=bool(getattr(ledger, "last_turn_silent", False)),
)
pending_notices = []
@@ -913,9 +1104,20 @@ class FlashRunner:
# Extract thought and tool calls
raw_text = self._extract_response_text(response)
native_text = self._extract_response_thinking(response)
# Extract token usage metadata from response
step_token_usage = self._resolve_token_usage(response, messages, raw_text)
step_token_usage, measured = self._resolve_token_usage(response, messages, raw_text)
# The ledger's compaction thresholds read the runner's own prompt
# size, never the session-wide meter (which any agent overwrites).
# The prompt as sent (``messages`` minus the just-appended reply)
# rides along so the ledger can calibrate its chars-per-token
# ratio — only from a provider measurement, never from the
# estimate (same as the Pro operator).
ledger.record_prompt_tokens(
step_token_usage.get("prompt_tokens"),
messages=messages[:-1] if measured else None,
)
self._record_llm_trace(step_token_usage, raw_text)
@@ -952,6 +1154,7 @@ class FlashRunner:
action_sequence,
turn,
injected=injected,
native_text=native_text,
)
ledger.stage_turn(messages[turn_base:])
previous_turn = turn
+73 -11
View File
@@ -260,24 +260,83 @@ class VisualStepSummarizer(StepMemoryService):
def _action_phrase(action_name: str, action_args: dict[str, Any] | None) -> str:
"""The action as every other history reader sees it (``format_actions_clean``).
A described coordinate target renders as ``'play button' (self-described)``,
an observed element as ``'Play'``; a Pro burst lists every member.
``action_args`` is the recorded action minus its verb (what the Pro
summarizer extracts from ``action_taken`` and what the Flash runner
builds from the same record shape). The action name always wins over
an argument of the same key: ``manage_app``'s own ``action="launch"``
argument becomes the ``intent`` the renderer reads, never the verb. A
described coordinate target renders as ``'play button'
(self-described)``, an observed element as ``'Play'``; a Pro burst
lists every member.
"""
args = dict(action_args or {})
extra = args.pop("additional_actions", None)
own_verb = args.pop("action", None)
if own_verb is not None and own_verb != action_name:
args.setdefault("intent", own_verb)
# Flash's coordinate swipe names its points ``start``/``end``; the
# history renderer reads ``start_coordinates``/``end_coordinates``.
if action_name == "swipe" and "start" in args and "end" in args:
args.setdefault("start_coordinates", args["start"])
args.setdefault("end_coordinates", args["end"])
if action_name == "swipe":
raw = args.get("args") if isinstance(args.get("args"), dict) else {}
start = args.get("start", raw.get("start"))
end = args.get("end", raw.get("end"))
if start is not None and end is not None:
args.setdefault("start_coordinates", start)
args.setdefault("end_coordinates", end)
try:
first = {"action": action_name, **args}
first = {**args, "action": action_name}
actions = [first, *extra] if isinstance(extra, list) and extra else first
return format_actions_clean(actions)
except Exception as exc:
logger.debug(f"Action phrase rendering fell back to raw args: {exc}", exc_info=True)
return f"{action_name}({action_args})"
@staticmethod
def _verbatim_args(action_args: dict[str, Any] | None) -> dict[str, Any]:
"""The agent-facing arguments of the action.
A Flash record keeps the tool call's arguments under ``args``; a Pro
record is the arguments themselves. Bookkeeping keys never count.
"""
args = action_args or {}
raw = args.get("args")
if isinstance(raw, dict):
return dict(raw)
internal = {
"additional_actions",
"coordinate_space",
"normalized_coordinates",
"normalized_start_coordinates",
"normalized_end_coordinates",
}
return {k: v for k, v in args.items() if k not in internal}
@staticmethod
def _phrase_carries(phrase: str, args: dict[str, Any]) -> bool:
"""Whether every argument value is visible in the rendered phrase.
Booleans and ``None`` never count (the phrase spells their effect, not
their value); everything else must appear verbatim, so an index
target, a repeat count or a duration the phrase folded away keeps the
verbatim argument line.
"""
haystack = phrase.lower()
def _shown(value: Any) -> bool:
if isinstance(value, (list, tuple)):
# A point renders whole (``at [880, 410]``); a sequence or a
# description list renders member by member.
return str(list(value)).lower() in haystack or (
bool(value) and all(_shown(v) for v in value)
)
return str(value).lower() in haystack
return all(
_shown(value)
for value in args.values()
if value is not None and not isinstance(value, bool)
)
def _meter_lens_call(self, response: Any) -> None:
"""Meter one raw-model lens call as an ``llm_usage`` trace, best-effort.
@@ -338,11 +397,14 @@ class VisualStepSummarizer(StepMemoryService):
dual = bool(pre_bytes) and bool(post_bytes)
template = self._prompt_template if dual else self._single_prompt_template
rendered_prompt = Template(template).render(step_number=step_number)
lead_lines = [
f"Step {step_number} Physical Action: {self._action_phrase(action_name, action_args)}",
f"Action arguments (verbatim): {action_name}({action_args})",
f"Controller Outcome: {exec_outcome}",
]
phrase = self._action_phrase(action_name, action_args)
lead_lines = [f"Step {step_number} Physical Action: {phrase}"]
# The verbatim arguments are worth a line only when the phrase
# folded some of them away (an index target, a repeat count, ...).
verbatim = self._verbatim_args(action_args)
if verbatim and not self._phrase_carries(phrase, verbatim):
lead_lines.append(f"Action arguments (verbatim): {action_name}({verbatim})")
lead_lines.append(f"Controller Outcome: {exec_outcome}")
focus = input_data.get("focus")
if isinstance(focus, dict) and focus:
lead_lines.extend(["", render_focus_block(focus, dual=dual)])
+1 -1
View File
@@ -1,3 +1,3 @@
{
"main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n[% if \"wait_for_delay\" in available_tools %]- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n[% endif %]- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.\n 2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading state. This boundary probe must remain unidirectional; never reverse direction merely to prove completion. If the task declares an earlier exit condition (for example, find one verified match or process N items), that explicit condition may end the traversal without reaching the list boundary.\n 3. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n 4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content.\n- **Dynamic Monitoring & Reactive Loop Protocol (Monitor-Trigger-Action)**: When executing an active milestone matching `[Loop] <Goal> into note <TargetNote> (Exit: <ExitCondition>; Interval: <Interval>)` or `[Loop:continuous] <Goal> into note <TargetNote> (Interval: <Interval>)` (the milestone grammar itself is defined under Task Plan Grammar below), run this 5-step cyclic state machine:\n - **Step 1 (Baseline Sync)**: On the initial cycle, observe all visible entries/metrics in the primary monitoring view and save the full snaLine truncated
"main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n[% if \"wait_for_delay\" in available_tools %]- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n[% endif %]- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.\n 2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading state. This boundary probe must remain unidirectional; never reverse direction merely to prove completion. If the task declares an earlier exit condition (for example, find one verified match or process N items), that explicit condition may end the traversal without reaching the list boundary.\n 3. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n 4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content.\n- **Dynamic Monitoring & Reactive Loop Protocol (Monitor-Trigger-Action)**: When executing an active milestone matching `[Loop] <Goal> into note <TargetNote> (Exit: <ExitCondition>; Interval: <Interval>)` or `[Loop:continuous] <Goal> into note <TargetNote> (Interval: <Interval>)` (the milestone grammar itself is defined under Task Plan Grammar below), run this 5-step cyclic state machine:\n - **Step 1 (Baseline Sync)**: On the initial cycle, observe all visible entries/metrics in the primary monitoring view and save the full snaLine truncated
}
+114 -37
View File
@@ -71,6 +71,9 @@ DEFERRING_TOOLS = {
"ask_explorer",
}
# Bare key names press_key accepts (case-insensitive, optional KEYCODE_ prefix).
SUPPORTED_PRESS_KEYS = ("ENTER", "BACK", "HOME", "APP_SWITCH")
from artemis.agents.operator.prompts import (
OPERATOR_MAX_TOOL_ITERATIONS,
PromptBuilder,
@@ -82,9 +85,9 @@ from artemis.agents.operator.prompts import (
HistoricalStateHintPromptComponent,
FeedbackPromptComponent,
ExecutionIncidentPromptComponent,
CheckItemsExplainerPromptComponent,
BackgroundTasksPromptComponent,
render_plan_ledger_bounce,
ReasoningReminderPromptComponent,
unwritten_action_streak,
ToolLimitWarningPromptComponent,
InjectedInstructionPromptComponent,
@@ -137,6 +140,24 @@ class OperatorNode:
self._transcript_turn_base: int | None = None
self._turn_plan_baseline: str | None = None
self._turn_ledger_streak: int = 0
# Whether the previous turn's replies carried no visible reasoning text;
# the next observation tail then carries a one-line reminder (no extra
# model call). Legacy-path fallback only: with a transcript ledger the
# ledger's own ``last_turn_silent`` is authoritative.
self._previous_turn_silent: bool = False
def _previous_turn_was_silent(self) -> bool:
"""Whether the previous turn carried no visible reasoning text.
The transcript ledger judges its committed turns itself; the local flag
only serves the legacy path, which has no ledger.
"""
from artemis.memory import TranscriptLedger
ledger = getattr(self.ctx, "transcript_ledger", None)
if isinstance(ledger, TranscriptLedger):
return bool(ledger.last_turn_silent)
return self._previous_turn_silent
def _available_device_actions(self) -> frozenset[str]:
"""Device actions the installed actuator backend provides.
@@ -225,7 +246,6 @@ class OperatorNode:
TemplatePromptComponent(),
{"template_name": "main_template"},
),
(CheckItemsExplainerPromptComponent(), {}),
(ExecutionIncidentPromptComponent(), {}),
(ObservationPromptComponent(), {}),
(ScreenshotSimilarityPromptComponent(), {}),
@@ -233,10 +253,14 @@ class OperatorNode:
(InjectedInstructionPromptComponent(), {}),
(BackgroundTasksPromptComponent(), {}),
(FeedbackPromptComponent(), {}),
(ReasoningReminderPromptComponent(), {}),
(ToolLimitWarningPromptComponent(), {}),
]
else:
components = [(c, {}) for c in components] + [(ToolLimitWarningPromptComponent(), {})]
components = [(c, {}) for c in components] + [
(ReasoningReminderPromptComponent(), {}),
(ToolLimitWarningPromptComponent(), {}),
]
# Active and finished tasks are now passed as arguments
active_tasks = active_background_tasks or []
@@ -252,6 +276,7 @@ class OperatorNode:
"current_step_num": current_step_num,
"active_background_tasks": active_tasks,
"newly_finished_tasks": newly_finished,
"previous_turn_silent": self._previous_turn_was_silent(),
"steps": steps,
}
kwargs_to_pass.update(extra_kwargs)
@@ -280,6 +305,9 @@ class OperatorNode:
image_scrub_depth=getattr(cfg, "image_scrub_depth", 3),
pending_grace_steps=getattr(cfg, "pending_grace_steps", 3),
xml_scrub_depth=getattr(cfg, "xml_scrub_depth", 1),
image_scrub_depth_relaxed=getattr(cfg, "image_scrub_depth_relaxed", None),
context_budget_tokens=getattr(cfg, "context_budget_tokens", None),
start_ratio=getattr(cfg, "start_ratio", None),
session_start=session_start if isinstance(session_start, (int, float)) else None,
)
# L2/L3 chunk compression (M3) rides the same flag; without a
@@ -337,6 +365,8 @@ class OperatorNode:
"""
from langchain_core.messages import HumanMessage
from artemis.memory.transcript import mark_ephemeral
ledger = self._ensure_transcript_ledger(state)
# 1. S region: rendered exactly once per session.
@@ -387,7 +417,6 @@ class OperatorNode:
builder.add_human_content(f"# CURRENT OBSERVATION [{ledger.elapsed_label()}]")
components = [
(PlanRecitationPromptComponent(), {}),
(CheckItemsExplainerPromptComponent(), {}),
(ExecutionIncidentPromptComponent(), {}),
(ObservationPromptComponent(), {}),
(ScreenshotSimilarityPromptComponent(), {}),
@@ -395,6 +424,7 @@ class OperatorNode:
(InjectedInstructionPromptComponent(), {}),
(BackgroundTasksPromptComponent(), {}),
(FeedbackPromptComponent(), {}),
(ReasoningReminderPromptComponent(), {}),
(ToolLimitWarningPromptComponent(), {}),
]
for component, extra_kwargs in components:
@@ -407,6 +437,7 @@ class OperatorNode:
"active_background_tasks": active_background_tasks or [],
"newly_finished_tasks": newly_finished_tasks or [],
"steps": steps,
"previous_turn_silent": self._previous_turn_was_silent(),
}
kwargs_to_pass.update(extra_kwargs)
await component(builder, state, self.ctx, **kwargs_to_pass)
@@ -418,6 +449,10 @@ class OperatorNode:
else:
tail_content.append(part)
tail = HumanMessage(content=tail_content)
# Per-turn notices (reminders, hints, guidance, incident blocks) are
# deleted by the scrub edge once the turn leaves the recent window.
if builder.ephemeral_indices:
mark_ephemeral(tail, builder.ephemeral_indices)
messages = ledger.render([tail])
self._transcript_turn_base = len(messages) - 1
@@ -471,12 +506,27 @@ class OperatorNode:
response = await invoke_llm_with_timeout_message(acomplete(bound_llm, current_messages))
usage = usage_from_message(response)
# The ledger's compaction thresholds read the operator's own
# prompt size, never the session-wide meter (which the
# Planner/Checker/sub-agent calls overwrite). The messages ride
# along so the ledger can calibrate its chars-per-token ratio.
transcript_ledger = getattr(self.ctx, "transcript_ledger", None)
if usage is not None:
logger.info(
f"LLM usage: prompt_tokens={usage['prompt_tokens']}"
f" (cached={usage['cached_tokens']}),"
f" completion_tokens={usage['completion_tokens']}"
)
if hasattr(transcript_ledger, "record_prompt_tokens"):
transcript_ledger.record_prompt_tokens(
usage.get("prompt_tokens"), messages=current_messages
)
elif hasattr(transcript_ledger, "record_prompt_tokens"):
# No usage metadata from the provider: record an estimate so
# the start gate and thresholds still see a context base.
from artemis.memory.transcript import estimate_prompt_tokens
transcript_ledger.record_prompt_tokens(estimate_prompt_tokens(current_messages))
if response.content:
if isinstance(response.content, str):
@@ -638,18 +688,22 @@ class OperatorNode:
"Scratchpad/helper tool call failed. Deferring"
f" screen actions: {other_tool_failure_msg}"
)
for tc in action_calls:
# Every rejected call needs its own ToolMessage, but the
# full explanation travels once; the rest point at it.
for idx, tc in enumerate(action_calls):
tool_outputs.append(
ToolMessage(
tool_call_id=tc["id"],
name=normalize_name(tc["name"]),
content=(
"This screen action was rejected"
" because an accompanying tool call"
" failed. Error:"
f" {other_tool_failure_msg}. Currently,"
" no screen actions have been"
" executed. Please correct the error"
" and try again."
"Not executed: this screen action was"
" rejected because an accompanying tool"
f" call failed. Error: {other_tool_failure_msg}."
" No screen action ran this turn. Correct"
" the error, then re-issue the action."
if idx == 0
else "Not executed: see the accompanying"
" tool-call error above."
),
status="error",
)
@@ -667,25 +721,23 @@ class OperatorNode:
" Deferring terminal actions."
)
for tc in action_calls:
for idx, tc in enumerate(action_calls):
tool_outputs.append(
ToolMessage(
tool_call_id=tc["id"],
name=normalize_name(tc["name"]),
content=(
"Your pre-decision tools have been"
" successfully processed. However, the"
" screen action you attempted to"
" execute was rejected because you are"
" not allowed to simultaneously"
" use result-dependent pre-decision"
" tools and execute physical actions in"
" the same turn."
" Currently, no screen actions have"
" been executed. Please review your"
" updated context and re-output your"
" intended screen action."
"Not executed: this screen action was"
" deferred because the same turn also"
" called a result-dependent pre-decision"
" tool. No screen action ran this turn."
" Read the tool results above, then"
" re-issue the action."
if idx == 0
else "Not executed: deferred for the same"
" reason as the screen action above."
),
status="success",
status="error",
)
)
@@ -713,24 +765,36 @@ class OperatorNode:
logger.info(f"Plan ledger gate: bouncing action. {bounce}")
for tc in action_calls:
tool_outputs.append(
ToolMessage(tool_call_id=tc["id"], content=bounce, status="error")
ToolMessage(
tool_call_id=tc["id"],
name=normalize_name(tc["name"]),
content=bounce,
status="error",
)
)
action_calls_to_translate = []
elif len(action_calls) > burst_cap:
# A multi-action turn is a fast-action burst that the Validator
# fires without the safety net; cap its length before it runs.
validation_errors = True
for tc in action_calls:
burst_error = (
f"Error: {len(action_calls)} Turn-Ending Actions in one"
" turn exceed the fast-action burst limit of"
f" {burst_cap}. Nothing was executed. Re-issue at most"
f" {burst_cap} actions (a burst is for sub-second"
" sequences on transient UI, not for batching normal"
" steps), or a single vetted action."
)
# One ToolMessage per rejected call, full text only on the first.
for idx, tc in enumerate(action_calls):
tool_outputs.append(
ToolMessage(
tool_call_id=tc["id"],
name=normalize_name(tc["name"]),
content=(
f"Error: {len(action_calls)} Turn-Ending Actions in one"
" turn exceed the fast-action burst limit of"
f" {burst_cap}. Nothing was executed. Re-issue at most"
f" {burst_cap} actions (a burst is for sub-second"
" sequences on transient UI, not for batching normal"
" steps), or a single vetted action."
burst_error
if idx == 0
else "Not executed: see the burst-limit error above."
),
status="error",
)
@@ -750,6 +814,7 @@ class OperatorNode:
tool_outputs.append(
ToolMessage(
tool_call_id=tc["id"],
name=normalize_name(tc["name"]),
content=error,
status="error",
)
@@ -759,6 +824,7 @@ class OperatorNode:
tool_outputs.append(
ToolMessage(
tool_call_id=tc["id"],
name=normalize_name(tc["name"]),
content="Action Recorded",
status="success",
)
@@ -896,6 +962,7 @@ class OperatorNode:
new_subagent_calls=new_subagent_calls,
state=state,
)
self._previous_turn_silent = not (raw_thinking and raw_thinking.strip())
# 8b. Transcript path: hold this turn's messages (observation tail +
# tool-loop products) for commit at the next build, when the turn's
@@ -1276,6 +1343,10 @@ class OperatorNode:
"normalized_start_coordinates": [nx1, ny1],
"normalized_end_coordinates": [nx2, ny2],
"duration": duration,
# A directional scroll keeps the direction the agent issued so
# history and replays render "Swiped up (...)" rather than a
# bare coordinate path.
**({"direction": str(target)} if kind == "direction" else {}),
**({"target_description": described} if described else {}),
}
], None
@@ -1290,9 +1361,15 @@ class OperatorNode:
f"Error: 'key' must be a string, got {type(key).__name__}.",
)
normalized_key = key[8:] if key.startswith("KEYCODE_") else key
if normalized_key not in ["ENTER", "BACK", "HOME", "APP_SWITCH"]:
return [], f"Error: Unsupported key '{key}'."
normalized_key = key.strip().upper()
if normalized_key.startswith("KEYCODE_"):
normalized_key = normalized_key[len("KEYCODE_") :]
if normalized_key not in SUPPORTED_PRESS_KEYS:
return (
[],
f"Error: Unsupported key '{key}'. Supported keys:"
f" {', '.join(SUPPORTED_PRESS_KEYS)}.",
)
return [{"action": "press_key", "keycode": f"KEYCODE_{normalized_key}"}], None
elif tool_name == "manage_app":
+200 -87
View File
@@ -17,6 +17,7 @@ from functools import lru_cache
import io
import json
from pathlib import Path
from typing import NamedTuple
from jinja2 import Environment, StrictUndefined, Template
from langchain_core.messages import HumanMessage, SystemMessage
@@ -31,9 +32,8 @@ from artemis.tools.command_tool import (
_is_output_long,
)
from artemis.utils.logger import get_logger
from artemis.utils.notes import get_note_file_path
from artemis.utils.plan_grammar import parse_plan, render_plan_grammar_spec
from artemis.utils.task_tree import SELF_DESCRIBED_MARKER
from artemis.utils.task_tree import SELF_DESCRIBED_MARKER, action_intent_phrase
logger = get_logger(__name__)
@@ -149,11 +149,19 @@ class PromptBuilder:
self.system_parts = []
self.human_parts = []
self.human_footer = None
#: Indices into ``human_parts`` of blocks that only matter this turn (see
#: ``artemis.memory.transcript.EPHEMERAL_BLOCKS_KEY``).
self.ephemeral_indices: list[int] = []
def add_system_text(self, text: str):
self.system_parts.append(text)
def add_human_content(self, content: str | dict):
def add_human_content(self, content: str | dict, *, ephemeral: bool = False):
"""Append a block to the observation. ``ephemeral=True`` marks it as valid
for this turn only: the transcript scrub edge deletes it at depth K, so it
never reaches the frozen history or a chunk capsule."""
if ephemeral:
self.ephemeral_indices.append(len(self.human_parts))
self.human_parts.append(content)
def set_human_footer(self, content: str):
@@ -171,10 +179,12 @@ class PromptBuilder:
if self.human_footer:
human_content.append({"type": "text", "text": self.human_footer})
return [
SystemMessage(content=system_content),
HumanMessage(content=human_content),
]
human = HumanMessage(content=human_content)
if self.ephemeral_indices:
from artemis.memory.transcript import mark_ephemeral
mark_ephemeral(human, self.ephemeral_indices)
return [SystemMessage(content=system_content), human]
class PromptComponent:
@@ -221,18 +231,42 @@ def resolve_operator_prompt_tools(ctx: ArtemisContext) -> frozenset[str]:
return frozenset(available)
def _operator_grammar_flags(ctx: ArtemisContext) -> tuple[bool, bool]:
"""(include_checks, verification_active) for the template render."""
def _operator_grammar_flags(ctx: ArtemisContext) -> tuple[bool, bool, bool]:
"""(midway_checks, final_check, verification_active) for the template render.
The two check gates are read separately so the grammar and the check-line
guidance are worded for what actually runs: with midway checks off there is
no repair loop to teach, and with both off the check-line grammar never
enters any prompt.
"""
setup = getattr(ctx, "execution_setup", None)
# Grammar spec assembly is a function of configuration: with both check
# gates disabled, the check-line grammar never enters any prompt.
include_checks = bool(setup and getattr(setup, "checks_enabled", False))
midway = bool(setup and getattr(setup, "midway_checks_enabled", False))
final = bool(setup and getattr(setup, "final_check_enabled", False))
# The rejection/finding diagnosis trigger only exists while a mechanism
# that can produce rejections or findings is active.
verification_active = include_checks or bool(
setup and not getattr(setup, "disable_planner_validation", True)
verification_active = (
midway or final or bool(setup and not getattr(setup, "disable_planner_validation", True))
)
return include_checks, verification_active
return midway, final, verification_active
def _checkpoint_max_repairs(ctx: ArtemisContext) -> int:
"""The midway repair budget recited in the static prompt."""
setup = getattr(ctx, "execution_setup", None)
max_repairs = getattr(setup, "checkpoint_max_repairs", None)
return max_repairs if isinstance(max_repairs, int) else 2
def _grammar_render_context(ctx: ArtemisContext) -> dict:
"""Template variables shared by the static and legacy system renders."""
midway, final, verification_active = _operator_grammar_flags(ctx)
return {
"plan_grammar": render_plan_grammar_spec(midway=midway, final=final),
"verification_active": verification_active,
"checks_active": midway or final,
"midway_checks_active": midway,
"checkpoint_max_repairs": _checkpoint_max_repairs(ctx),
}
# --- M2 template split -----------------------------------------------------------
@@ -282,18 +316,15 @@ def render_transcript_static_system(
available = resolve_operator_prompt_tools(ctx)
static_template = apply_operator_prompt_contract(static_template, available_tools=available)
include_checks, verification_active = _operator_grammar_flags(ctx)
return Template(static_template).render(
initial_goal=state.initial_goal,
subgoals_status="",
plan_and_history="",
unified_history="",
plan_grammar=render_plan_grammar_spec(include_checks),
verification_active=verification_active,
checks_active=include_checks,
transcript_history=True,
max_burst_actions=_max_burst_actions_for_prompt(),
max_tool_calls=OPERATOR_MAX_TOOL_ITERATIONS,
**_grammar_render_context(ctx),
)
@@ -326,19 +357,15 @@ class TemplatePromptComponent(PromptComponent):
plan_and_history = kwargs.get("plan_and_history", "No plan or history yet.")
include_checks, verification_active = _operator_grammar_flags(ctx)
full_prompt = Template(prompt_template).render(
initial_goal=state.initial_goal,
subgoals_status="",
plan_and_history=plan_and_history,
unified_history="",
plan_grammar=render_plan_grammar_spec(include_checks),
verification_active=verification_active,
checks_active=include_checks,
transcript_history=False,
max_burst_actions=_max_burst_actions_for_prompt(),
max_tool_calls=OPERATOR_MAX_TOOL_ITERATIONS,
**_grammar_render_context(ctx),
)
parts = full_prompt.split("# CURRENT OBSERVATION")
@@ -400,7 +427,8 @@ class FeedbackPromptComponent(PromptComponent):
" address any reverted subgoal accordingly; do not re-litigate"
" them. [planner] findings are advisory: a lightweight reviewer"
" had a concern about a plan change that stayed applied — weigh"
" the reason against your own observations."
" the reason against your own observations.",
ephemeral=True,
)
@@ -472,7 +500,9 @@ def render_execution_incident(incident: dict, steps: list) -> str:
kind = incident.get("kind") or "exec_error"
category = str(incident.get("category") or "general")
consecutive = int(incident.get("consecutive_failures") or 1)
description = incident.get("action_description") or "the planned action"
# Intent form ("launch app 'X'"): the action did not happen, so the
# past-tense outcome phrase would misreport it as done.
description = action_intent_phrase(incident.get("action_description") or "the planned action")
reason = str(incident.get("reason") or "").strip()
burst_size = int(incident.get("burst_size") or 1)
index = int(incident.get("action_index") or 0)
@@ -505,8 +535,8 @@ def render_execution_incident(incident: dict, steps: list) -> str:
)
else:
lines.append(
f"What happened: your planned action `{description}` was dispatched, but the"
f" device/executor reported: {reason}"
f"What happened: your planned action `{description}` could"
f" not be executed; the device/executor reported: {reason}"
)
# --- Category-specific evidence -------------------------------------------------
@@ -547,7 +577,7 @@ def render_closed_incident(closed: dict) -> str:
"""One-turn notice after an incident closes: settle the original intent."""
opened = closed.get("step_number")
closed_at = closed.get("closed_at_step")
description = closed.get("action_description") or "the blocked action"
description = action_intent_phrase(closed.get("action_description") or "the blocked action")
header = (
f"--- Execution Incident (CLOSED at Step {closed_at}) ---"
if closed_at
@@ -567,61 +597,22 @@ class ExecutionIncidentPromptComponent(PromptComponent):
then a one-turn CLOSED notice asking the Operator to settle the intent.
Reads ``state.open_incident`` / ``state.last_closed_incident`` (both written
by the Validator). The block is its own text part of the observation tail,
so the transcript ledger keeps it verbatim: the whole resolution effort
stays legible across turns.
by the Validator). Both blocks are ephemeral: the open block is re-rendered
every turn the incident stays open, and the step record already carries the
failed execution result, so older copies add nothing to the history.
"""
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
incident = getattr(state, "open_incident", None)
if isinstance(incident, dict) and incident.get("reason"):
builder.add_human_content(
render_execution_incident(incident, kwargs.get("steps") or [])
render_execution_incident(incident, kwargs.get("steps") or []),
ephemeral=True,
)
return
closed = getattr(state, "last_closed_incident", None)
if isinstance(closed, dict) and closed.get("kind"):
builder.add_human_content(render_closed_incident(closed))
class CheckItemsExplainerPromptComponent(PromptComponent):
"""Behavioral guidance for check lines, rendered iff the CURRENT plan
actually contains check lines (content-driven, not switch-driven: a resumed
plan carrying check lines still gets the explanation)."""
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
if not ctx or not getattr(ctx, "data_engine", None):
return
try:
task_plan_path = get_note_file_path(ctx.data_engine.base_dir, "task_plan")
if not task_plan_path.exists():
return
snapshot = parse_plan(task_plan_path.read_text(encoding="utf-8"))
except Exception as e:
logger.error(f"Failed to parse plan for check-items explainer: {e}")
return
if not snapshot.all_check_items:
return
setup = getattr(ctx, "execution_setup", None)
max_repairs = getattr(setup, "checkpoint_max_repairs", None)
if not isinstance(max_repairs, int):
max_repairs = 2
builder.add_human_content(
"--- About the plan's check lines ---\n"
"The task plan declares `- verify:` / `- assert:` check lines (see Task"
" Plan Grammar). Use `verify:` lines to confirm your work is complete,"
" but never declare a check passed yourself or record conclusions on"
" its behalf. Take NO extra actions for `assert:` lines and never"
" construct or fake state to satisfy one. Check lines must not be"
" deleted or reworded (deletions are automatically restored by the"
" system); keep them verbatim when rewriting the plan — adding new ones"
" is allowed. Simply mark completions per the plan grammar as usual;"
" checking runs asynchronously in the background and does not block you."
f" A failed `verify:` reopens its milestone at most {max_repairs} time(s);"
" once that repair budget is exhausted, its standing `finding:` line"
" disappears and the failure stands as the recorded result — do not"
" keep repairing it."
)
builder.add_human_content(render_closed_incident(closed), ephemeral=True)
class BackgroundTasksPromptComponent(PromptComponent):
@@ -700,6 +691,26 @@ _LEDGER_BOUNCE_TAIL = (
)
#: One-line tail notice for the turn after a turn whose replies carried no visible
#: reasoning text. Thinking-mode models satisfy "reason first" inside their thought
#: channel and emit bare tool calls; the reminder costs no extra model call.
REASONING_REMINDER = (
"--- Reminder ---\n"
"Your previous turn carried no visible reasoning text (internal thinking is not"
" shown). This turn, write one short paragraph of plain prose before calling a"
" Turn-Ending Action: what the screen shows, whether the last action worked, and"
" what you do next and why."
)
class ReasoningReminderPromptComponent(PromptComponent):
"""Renders :data:`REASONING_REMINDER` when the previous turn was silent."""
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
if kwargs.get("previous_turn_silent"):
builder.add_human_content(REASONING_REMINDER, ephemeral=True)
def render_plan_ledger_bounce(task_plan: str, streak: int, stale_turns: int) -> str | None:
"""Return a plan-ledger validation error, if any."""
snapshot = parse_plan(task_plan or "")
@@ -735,27 +746,128 @@ class ToolLimitWarningPromptComponent(PromptComponent):
"\nWarning: your last turn used up its tool-call budget without a"
" Turn-Ending Action. Re-examine the task goal and your plan;"
" the current approach may not be the right path. If the cause"
" is unclear, ask_diagnoser can help."
" is unclear, ask_diagnoser can help.",
ephemeral=True,
)
#: Header of the user-guidance block (shared by the Flash runner and the Pro
#: operator; the operator's static prompt explains the channel once).
USER_GUIDANCE_MARKER = "--- User Guidance ---"
#: Prefix of the persistent user-instruction line (see :func:`render_user_guidance`).
USER_INSTRUCTION_PREFIX = "User instruction"
class UserGuidance(NamedTuple):
"""The two observation blocks delivering a mid-run user instruction.
``wrapper`` is the per-turn ``--- User Guidance ---`` framing (relay
notice, rank, release-loop wording): it is marked ephemeral and leaves
with the recent window. ``body`` is the verbatim instruction as one
self-contained line; it is added as a regular block so it stays in the
active window until the turn is chunk-compressed, at which point the
chunk's ``user_lines`` carry it verbatim. Standing instructions remain
visible after the turn in which they arrived.
"""
body: str
wrapper: str
def render_user_instruction(instruction: str, *, offset_label: str | None = None) -> str:
"""The persistent one-line body: ``User instruction (T+mm:ss): "..."``.
``offset_label`` is the session-relative time the instruction was
delivered when the call site has it; omitted otherwise.
"""
when = f" ({offset_label})" if offset_label else ""
return f'{USER_INSTRUCTION_PREFIX}{when}: "{instruction}"'
def render_user_guidance(
instruction: str,
*,
release_loop: bool = False,
has_plan: bool = True,
offset_label: str | None = None,
) -> UserGuidance:
"""The observation blocks delivering a mid-run user instruction.
One rendering for both profiles: the instruction is stated as what it is (an
external interruption from the user, relayed by the system) rather than as
advice to be weighed, and it explicitly outranks the plan and its check lines.
Returns ``(body, wrapper)``; callers add ``wrapper`` with ``ephemeral=True``
and ``body`` without (see :class:`UserGuidance`).
"""
rank = (
" It outranks the task plan and its check lines: edit the plan to match,"
" check lines included."
if has_plan
else " It outranks your current milestones: adjust them to match."
)
lines = [
USER_GUIDANCE_MARKER,
"The user watching this run sent the instruction quoted below; the system"
" relays it as an external interruption, like a stop signal." + rank,
"Act on it from this turn: reconcile it with the current screen and recent"
" history, then choose the next action. The quoted line stays in your"
" history as a standing instruction until the user sends another.",
]
if release_loop:
lines.append(
"The user has explicitly authorized stopping any ongoing monitoring loop;"
" you may now wrap up and complete the task."
)
return UserGuidance(
body=render_user_instruction(instruction, offset_label=offset_label),
wrapper=chr(10).join(lines),
)
def add_user_guidance(builder: PromptBuilder, guidance: UserGuidance) -> None:
"""Adds the two guidance blocks: the wrapper ephemeral, the body persistent."""
builder.add_human_content(guidance.wrapper, ephemeral=True)
builder.add_human_content(guidance.body)
class InjectedInstructionPromptComponent(PromptComponent):
"""The mid-run user instruction: an ephemeral wrapper plus the persistent
verbatim body (:class:`UserGuidance`).
The step record also keeps the instruction verbatim (``extra_metadata``),
which is what the chunk ledger renders once the turn is compressed.
"""
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
injected = getattr(state, "injected_instruction", None)
if injected:
builder.add_human_content(
f"\n--- User Guidance ---\n"
f"The user observing your progress has provided the following"
f" feedback or correction:\n"
f'"{injected}"\n\n'
f"Please review this guidance, evaluate it against your current"
f" screen state and recent history, "
f"and integrate it into your reasoning. Use it to refine your"
f" task plan and determine the "
f"most appropriate next action."
release_loop = bool(getattr(state, "user_stop_requested", False))
add_user_guidance(
builder,
render_user_guidance(
injected,
release_loop=release_loop,
has_plan=True,
offset_label=_session_offset_label(ctx),
),
)
def _session_offset_label(ctx: ArtemisContext | None) -> str | None:
"""The transcript ledger's ``T+mm:ss`` label when a ledger is attached."""
ledger = getattr(ctx, "transcript_ledger", None) if ctx is not None else None
elapsed_label = getattr(ledger, "elapsed_label", None)
if ledger is None or not callable(elapsed_label):
return None
try:
label = elapsed_label()
except Exception:
return None
return label if isinstance(label, str) and label else None
class ScreenshotSimilarityPromptComponent(PromptComponent):
"""Background check: compare current screen against post-action screenshots of the last few steps.
@@ -824,7 +936,7 @@ class ScreenshotSimilarityPromptComponent(PromptComponent):
if matched_step_nums:
steps_str = ", ".join(matched_step_nums)
note_text = f"Note: the screen is unchanged since step {steps_str} (pixel-identical)."
builder.add_human_content(note_text)
builder.add_human_content(note_text, ephemeral=True)
def _count_differing_pixels(
self,
@@ -947,5 +1059,6 @@ class HistoricalStateHintPromptComponent(PromptComponent):
builder.add_human_content(
f"Historical state hint: current screen closely resembles the"
f" post-action screen from Step {best_step_number}. Use"
" search_history / replay_steps only if its details are needed."
" search_history / replay_steps only if its details are needed.",
ephemeral=True,
)
+1 -1
View File
@@ -21,7 +21,7 @@ The user explicitly requested to extract the following information:
# Instruction
1. **Assess Confidence & Verify**: Based on the execution history, final state, and the target information to extract, determine if you can confidently formulate the final answer. If the current evidence is insufficient or you require more granular proof, you **MUST** actively gather information and details using your tools first. If your tools fail (e.g., return errors or time out), or if the steps were skipped/not observed, you **MUST** report this lack of evidence honestly. Do not assume, guess, or fabricate any unobserved steps.
2. **Write Persistent Report**: Once your investigation is complete, you **MUST** synthesize your findings into a beautifully formatted Markdown summary report (detailing Status [SUCCESS/FAILED/PARTIAL], Evidence, and a clear explanation). You must write this report to persistent memory under the key "output" by calling:
2. **Write Persistent Report**: Once your investigation is complete, you **MUST** synthesize your findings into a Markdown summary report (detailing Status [SUCCESS/FAILED/PARTIAL], Evidence, and a clear explanation). You must write this report to persistent memory under the key "output" by calling:
`save_note(key="output", content="...your markdown report...")`
3. **Final Response**: Only after saving the report, write a direct, helpful, and jargon-free final summary response to the user. Do not use any XML tags, code block wrappers, or special delimiters—just state the conclusion clearly and naturally, ensuring any extracted target information is clearly presented.
+1 -1
View File
@@ -5,7 +5,7 @@
"capabilities_operator": "## Execution Capabilities\n\nThe execution agent primarily interacts with the mobile device and analyzes screen recordings and logs. It can perform the following types of actions:\n- **UI Interaction**: Tapping, swiping, and entering text on the screen.\n- **App Management**: Launching and stopping specific applications.\n- **Information Retrieval & Analysis**: Reading on-screen text and UI hierarchy, searching the web for external information, querying system logs, and analyzing screen recordings.\n- **Terminal & Background Operations**: Executing arbitrary ADB shell commands on the device for exploration. It is capable of running long-running ADB commands asynchronously in the background as tasks.\n\n### Capability Boundaries & Planning Rules\n- **Cross-App Operations**: When a task requires interacting with multiple applications, split the operations into distinct milestones for each app. Avoid mixing actions for different apps in a single milestone.",
"output_protocol_initial": "## Task Plan & Scratchpad Protocol\n\n1. **Articulate Thinking**: Begin by reasoning through the 5-step **Systematic Thinking Process** to establish your strategy, funnel approach, bounds, and pre-mortem checks.\n2. **Designate Notes in Milestones & Pre-generate Answer Cards**: If the task involves collecting findings, logging periodic monitoring events, recording candidates, or analytical summaries, designate clear, descriptive note names directly in the milestone descriptions (e.g., `candidates_list`, `monitoring_log`, `schedule_summary`). You can and should invoke the `save_note` tool to pre-generate these structured template files (answer card / scratchpad skeletons with Markdown headers, sections, or tables) before submitting `task_plan`. This provides downstream executors with a clear, ready-to-fill schema.\n3. **Submit Plan**: Submit your initial plan by invoking the `save_note` tool with `key='task_plan'`. Avoid writing the plan only in your text response, as it will only be recognized by the system if submitted via the tool.\n\n### Plan Formatting Rules\n- Format the plan strictly as a Markdown task list starting with `- [ ]` for uncompleted items.\n{{ plan_grammar }}\n- For linear macro-milestones: `- [ ] <Phase Description>`\n- For bounded repetitive, iterative, or periodic polling milestones, tag the item with `[Loop]` and declare its Exit Boundary and Interval (e.g., `- [ ] [Loop] <Overall Loop / Monitoring Goal> (Exit: <Condition>; Interval: <e.g., every 5 minutes>)`).\n- For unbounded continuous monitoring that only the user can stop, tag the item with `[Loop:continuous]` instead (e.g., `- [ ] [Loop:continuous] <Monitoring Goal> (Interval: <e.g., every 5 minutes>)`); never declare a self-serve exit condition for it.\n- **Strict Two-Tier Hierarchy & Loop Subtask Contract**:\n 1. **Subtask Nesting Rule**: ALL individual iteration instances, polling checks, candidate inspections, or baseline setups MUST be formatted as 2-space indented sub-tasks (` - [ ]`) directly nested under their parent `[Loop]` milestone.\n 2. **Mandatory First-Instance Initialization**: Every `[Loop]` milestone MUST be initialized with its first concrete child sub-task (e.g., ` - [ ] Inspect Candidate #1: ...` or ` - [ ] Polling Check #1 (Baseline & Initial Sync): ...`).\n 3. **No Flattened Cycles**: NEVER elevate individual checks, recurring polling cycles, or candidate inspections to top-level milestones outside the `[Loop]`. Top-level items are reserved exclusively for strategic macro-phases.\n- Ensure the content of the `task_plan` note contains only the task list, excluding introductions, explanations, conversational text, or headers.",
"examples_initial": "## Master Example: Demonstrating the Complete Cognitive Paradigm\n\n**Goal**: \"Find and summarize all schedule arrangements and venue details for the upcoming November tech conference from my emails.\"\n\n### Naive / Incorrect Approaches:\n1. *Flawed Semantics & Too Granular*:\n- [ ] Tap search icon in Gmail\n- [ ] Type \"November\"\n- [ ] Open first email and tap back\n*(Fails: Micromanages atomic UI taps; assumes emails for November arrangements must have \"November\" in the title or be received in November)*\n2. *Vague & Unbounded Trap*:\n- [ ] Open email and check all emails for conference details\n*(Fails: Unbounded scan trap; causes infinite scrolling and step budget exhaustion on mobile)*\n\n---\n\n### Correct Approach (Full Cognitive Thinking & Robust Plan):\n\n#### Step-by-Step Thinking:\n1. **Intent & Semantic Expansion**:\n - *Core Objective*: Extract confirmed schedule dates, sessions, and venue info for the November tech conference.\n - *Semantic Insight*: Conference communications are often sent months in advance (e.g., ticket confirmations or registration emails in August/September). The subject line may say \"Global Developer Summit\", \"Tech Expo Agenda\", or \"Registration Confirmed\" rather than literally \"November conference\".\n - *Task Composition*: Linear setup (open email app and search) -> Iterative candidate inspection loop (examine each promising email) -> Final consolidation.\n2. **Visual Grounding**:\n - The device screenshot shows the Home screen with the Gmail app icon visible. The plan must start by opening the email client.\n3. **Tiered Strategy, Funnel Approach & Safeguards**:\n - *Tier 1 (Optimal)*: Use in-app search with rich semantic keywords (\"conference\", \"summit\", \"schedule\", \"agenda\", \"venue\", \"November\").\n - *Tier 2 (Fallback)*: If search is unavailable or yields zero results, fallback to chronological scanning of recent 3 months.\n - *Funnel (Broad Gathering -> Deep Inspection)*: Gather candidate emails from search results; do not discard candidates based solely on list previews. Open each candidate thread to inspect full body details.\n - *Feasibility Boundary*: Cap the candidate email inspection pool at top 15 emails to prevent infinite scrolling.\n4. **Risk Pre-mortem**:\n - *Risks*: Promotional marketing spam; conflicting drafts with preliminary dates; cancelled sessions.\n - *Defensive Check*: Instruct downstream executor to filter out promotional newsletters, verify session timestamps, and confirm final venue address.\n5. **Milestone Granularity & Scratchpad Blueprint**:\n - Focus milestones on business outcomes (search, inspect candidates, consolidate), avoiding micromanaged click-by-click steps.\n - Use indented sub-tasks to guide the loop of inspecting each candidate email, initializing with Item #1 and adding subsequent items as they are identified. This prevents the agent from losing state or re-inspecting the same items.\n - Designate note `november_conference_summary` as the destination for recording findings and pre-generate the structured template note.\n\n#### Tool Invocation:\n1. Pre-generate designated answer card note template:\n`save_note(key='november_conference_summary', content='# November Tech Conference Summary\\n\\n## Key Details\\n- Conference Name: \\n- Dates: \\n- Venue / Address: \\n\\n## Daily Schedule & Sessions\\n(To be populated during email inspection)\\n')`\n\n2. Submit milestone plan:\n`save_note(key='task_plan', content='- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\\n - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\\n- [ ] Review and consolidate all extracted information in `november_conference_summary` into a clear, finalized conference itinerary')`\n\n#### Content of `task_plan` Note:\n- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\n - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\n- [ ] ReviLine truncated
"check_generation": "## Declaring Verification Standards (check lines)\nYou may attach check lines to the plan; an independent Checker judges them asynchronously.\n\n- **User-declared expectations become `- assert:` lines.** Whenever the user's goal states an explicit expectation (\"should show X\", \"must not crash\", \"the toast Y should appear\"), declare it as an assert line preserving the user's wording verbatim. Expectations about the whole run or the final state use the `@end` suffix (task-level lines at the end of the plan).\n- **`- verify:` lines only for milestones that genuinely need acceptance checks** — completion points of irreversible operations, or milestones that directly contribute to the final goal. Do NOT attach a verify line to every subgoal: restraint is an explicit requirement (checking cost scales with the number of check lines).\n- **Do not fabricate assertions** for subgoals that have nothing verifiable. Rewrite transient expectations as persistently probeable state wherever possible (e.g., \"alarm exists in the alarm list\" instead of \"a toast flashed\") — evidence for transient moments only exists if the execution history happened to capture it.\n- **Ordering requirements are post-hoc.** If the user demands \"A must be confirmed before B runs\", still generate the assertion preserving their semantics, but annotate in the plan that ordering is verified retroactively from history — the system performs post-hoc audits and CANNOT block B from executing. Never silently promise enforcement that does not exist.",
"check_generation": "## Declaring Verification Standards (check lines)\nYou may attach check lines to the plan. {% if midway_checks %}An independent Checker judges each line in the background when its milestone completes, and a failed verify reopens that milestone for repair{% if final_check %}; a final review at task exit then audits the whole run against the user's goal{% endif %}.{% else %}A final review at task exit judges every line once, from the recorded evidence around each milestone's completion; there is no midway repair loop.{% endif %}\n\n- **User-declared expectations become `- assert:` lines.** Whenever the user's goal states an explicit expectation (\"should show X\", \"must not crash\", \"the toast Y should appear\"), declare it as an assert line preserving the user's wording verbatim. Expectations about the whole run or the final state use the `@end` suffix (task-level lines at the end of the plan).\n- **`- verify:` lines only for milestones that genuinely need acceptance checks** — completion points of irreversible operations, or milestones that directly contribute to the final goal. Do NOT attach a verify line to every subgoal: restraint is an explicit requirement (checking cost scales with the number of check lines).\n- **Do not fabricate assertions** for subgoals that have nothing verifiable. Rewrite transient expectations as persistently probeable state wherever possible (e.g., \"alarm exists in the alarm list\" instead of \"a toast flashed\") — evidence for transient moments only exists if the execution history happened to capture it.\n- **Ordering requirements are post-hoc.** If the user demands \"A must be confirmed before B runs\", still generate the assertion preserving their semantics, but annotate in the plan that ordering is verified retroactively from history — the system performs post-hoc audits and CANNOT block B from executing. Never silently promise enforcement that does not exist.",
"check_audit": "- **Check Standard Protection**: Check lines (`- verify:` / `- assert:`) declared in the plan are the task's verification standards. FLAG any proposed plan that deletes, weakens, or rewords existing check lines — softening a declared standard is never an acceptable adaptation. Adding new check lines is allowed.",
"role_validator": "## Role\nYou are the Task Plan Reviewer, a fast advisory second opinion. The Operator has modified the top-level task plan and the change is already applied. Your job is to spot a concrete concern — goal drift, a broken loop contract, a weakened check standard, a logical dead end — and, if there is one, state it with its reason so the Operator can decide. You never block or roll back anything.",
"validator_guidelines": "## Validation Principles\n- **Goal Alignment**: Ensure the new plan ultimately accomplishes the `Initial Goal` requested by the user. If the Operator has abandoned the core objective or fundamentally altered it, flag the plan.\n- **Logical Coherence**: Verify that the proposed milestones form a logical, achievable sequence to reach the goal. Support valid ongoing iteration/polling subgoals and trigger branches without misconstruing periodic checks as infinite loops. Flag the plan if it introduces contradictory steps, unhandled deadlocks, or obvious logical fallacies.\n{{ plan_grammar }}\n- **Loop & Monitoring Protection**: If the existing plan contains an active `[Loop]` or `[Loop:continuous]` milestone, you MUST FLAG any proposed plan that deletes the milestone or removes its loop tag, as well as any plan that marks a `[Loop:continuous]` continuous monitoring milestone as completed (`[x]`) without an explicit external user stop signal.\n- **Adaptive Acceptance**: The Operator operates in a dynamic environment and may encounter unexpected UI states or dynamic event triggers. You should approve alternative strategies, extra intermediate steps, loop status tracking, or necessary workarounds as long as they serve the primary goal effectively.\n- **Advisory Verdict & Constructive Feedback**: Your verdict is advisory — the change stays applied whether or not you approve it. Approve unless you have a concrete, evidence-backed concern; a vague unease is not a concern. If you flag the plan, give one or two sentences naming the flaw or goal deviation and the correction you suggest, concrete enough for the Operator to weigh it against its own observations."
+23 -8
View File
@@ -82,11 +82,19 @@ def validate_plan_format(content: str) -> tuple[bool, str]:
return True, ""
def _checks_enabled(ctx: ArtemisContext) -> bool:
"""Config-driven prompt assembly gate: with both check gates disabled, no
check-related instruction reaches any planner prompt."""
def _check_gates(ctx: ArtemisContext) -> tuple[bool, bool]:
"""(midway_checks, final_check): the two check gates, read separately so
the planner is taught the judgment moment that actually applies. With both
disabled, no check-related instruction reaches any planner prompt."""
setup = getattr(ctx, "execution_setup", None)
return bool(setup and getattr(setup, "checks_enabled", False))
midway = bool(setup and getattr(setup, "midway_checks_enabled", False))
final = bool(setup and getattr(setup, "final_check_enabled", False))
return midway, final
def _checks_enabled(ctx: ArtemisContext) -> bool:
"""Config-driven prompt assembly gate (either check gate active)."""
return any(_check_gates(ctx))
def build_planner_system_blocks(prompts_data: dict, mode: str, include_checks: bool) -> list[str]:
@@ -152,12 +160,15 @@ async def run_async_planner_validation(
with open(prompts_path, encoding="utf-8") as f:
prompts_data = json.load(f)
include_checks = _checks_enabled(ctx)
midway, final = _check_gates(ctx)
include_checks = midway or final
mode_config = prompts_data["modes"]["validator"]
system_blocks = build_planner_system_blocks(prompts_data, "validator", include_checks)
system_content = "\n\n".join(prompts_data["blocks"][block] for block in system_blocks)
system_message = Template(system_content).render(
plan_grammar=render_plan_grammar_spec(include_checks)
plan_grammar=render_plan_grammar_spec(midway=midway, final=final),
midway_checks=midway,
final_check=final,
)
history_str = "No history available."
@@ -228,15 +239,19 @@ class PlannerNode:
mode = "initial_plan"
mode_config = prompts_data["modes"][mode]
include_checks = _checks_enabled(self.ctx)
midway, final = _check_gates(self.ctx)
include_checks = midway or final
# Assembly point (not in-template conditionals): check-generation
# instructions only mount while the checking feature is active, so a
# fully-disabled configuration yields a byte-identical planner prompt.
# Inside the mounted block, the judgment moment is worded per gate.
system_blocks = build_planner_system_blocks(prompts_data, mode, include_checks)
system_content = "\n\n".join(prompts_data["blocks"][block] for block in system_blocks)
system_message = Template(system_content).render(
plan_grammar=render_plan_grammar_spec(include_checks)
plan_grammar=render_plan_grammar_spec(midway=midway, final=final),
midway_checks=midway,
final_check=final,
)
# Get screenshot
-35
View File
@@ -1,35 +0,0 @@
# ROLE
You are the Summarizer Agent for a mobile automation system. Your task is to summarize the latest execution step, providing a high-information-density and succinct history summary of the current step without losing any information.
# OUTPUT FORMAT
- **Format Constraint**: You MUST output the summary as a **single, continuous paragraph**.
- **Prohibited Formats**: Do NOT use bullet points, numbered lists, markdown formatting, or multiple paragraphs.
- **Goal**: Maintain a consistent, easily parsable narrative for the Operator's historical record.
# CORE PHILOSOPHY & RULES
1. **Perspective**: You MUST write the summary from the Operator's first-person perspective using "I" (e.g., "I intended to...", "I clicked..."). NEVER use third-person terms like "The operator" or "The agent", nor second-person terms like "You" when referring to the Operator. For other agents (e.g., Explorer, Diagnoser), you can use their names.
2. **Zero Subjective Validation**: Absolutely avoid declaring semantic success, completion, or failure. In historical compression, any subjective validation will cause future planners to falsely believe a goal was definitively met or permanently broken.
- **BANNED WORDS**: "successfully", "completed", "entered", "navigated to", "failed", "unsuccessful", "could not", "achieved".
3. **Focus on Intent & Strategy**: The specific physical action (exact coordinates, swipes) is now recorded automatically. Focus primarily on the high-level Intent and reasoning. If you realize the Operator is on the wrong path and needs to backtrack or pivot, describe this as a shift in strategy.
4. **Intended Context Transitions**: Describe the intended navigation path ("From Where -> Intended Where"). Do not declare that the transition actually happened. (e.g., "From the Home screen, I intended to open the Network Settings view").
5. **Preserve Verifications & Iteration Progress**: If the Operator verified any state in this step, processed a specific candidate item/polling round, or explicitly mentioned what prerequisites or items are still missing, you MUST succinctly preserve these concrete progress details and verified items in the summary.
6. **Objective Incident Logs**: If the safety net intercepted the action, the device rejected it, or a fast-action burst aborted midway (an `[Result]` line starting with "Error:" / an execution incident), faithfully record what was blocked and the stated reason, without judging whether the Operator's next move will resolve it.
# LOOP & STAGNATION DETECTION
Use the `RECENT 10 STEPS HISTORY` to intelligently detect loops or stagnation:
1. **Physical Action Loops vs. Planned Iterations**:
- *Planned Iterations / Monitoring (Valid Progress)*: If the Operator is systematically processing different candidate items in a loop, or executing planned periodic polling cycles with wait intervals under a `[Loop]` milestone, record this objectively as planned progress (e.g., "Inspected Candidate #2", "Completed Polling Round #2 for new messages"). Do NOT flag planned iterations across different items or spaced polling as abnormal loops.
- *Unintended Action Loops (Anomalies)*: If the Operator is repeatedly clicking the exact same target, failing to progress on the same item, or oscillating blindly between identical states without updating progress notes, explicitly record the anomalous loop pattern and iteration count (e.g., "This is the 3rd cycle of failing to open the same folder").
2. **Cognitive Stagnation**: While you generally ignore internal background tool calls (e.g., reading logs, checking states) when counting physical action loops, if you detect continuous multiple steps of purely internal tool calls without any physical actions or note updates, objectively record this "stagnation" or "analysis loop".
# INPUT STRUCTURE
You will receive:
- **CURRENT PLAN & TASK PLAN**: The current task plan including milestones, active subgoals, and live iteration checklists to help you understand the strategic context and loop phase.
- **RECENT 10 STEPS HISTORY**: A brief history of the last 10 steps to help you detect loops and stagnation patterns.
- **CHRONOLOGICAL STEP TRACE (CURRENT STEP)**: The step-by-step chronological history of the current execution turn, including:
- `[Operator Monologue] / [Operator Native Thought]`: The operator's planning, intentions, progress counting, reasoning, verified states, and explicit mentions of missing prerequisites.
- `[Operator Tool Call] / [Operator Final Action]`: The actual action(s) planned and executed.
- `[Planned Fast-Action Burst]`: A multi-action turn executed back to back without the safety net, with each member's outcome.
- `[Pre-Execution Safety Net]`: Validation checks prior to action execution.
- `[Result]`: The Validator's execution outcome; an "Error:" line here is an execution incident the Operator must resolve.
+1 -1
View File
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Pro-graph step summarizer node (M2: visual-transition lens dispatch).
"""Pro step summarizer dispatching visual-transition summaries.
Per the history-module redesign §5 (single step-level lens, confirmed
2026-08-31) this node no longer generates a per-step text capsule with its own
+87 -14
View File
@@ -460,8 +460,20 @@ class MemoryTranscriptConfig(BaseModel):
default=3,
ge=1,
description=(
"Depth K of the screenshot scrub edge: the K-th most recent"
" historical screenshot is replaced by its visual summary."
"Depth K of the screenshot scrub edge from the start gate up: the"
" K-th most recent screenshot (the live one counts as depth 1) is"
" replaced by its visual summary."
),
)
image_scrub_depth_relaxed: int = Field(
default=6,
ge=1,
description=(
"Screenshot scrub depth while the operator's measured context is"
" below budget*start_ratio (nothing is being replaced yet, so"
" more raw screenshots stay in view). Must be >= image_scrub_depth;"
" set equal to it to disable the occupancy-driven depth. A change"
" of depth never rewrites an already-scrubbed message."
),
)
pending_grace_steps: int = Field(
@@ -473,13 +485,17 @@ class MemoryTranscriptConfig(BaseModel):
" image is replaced by a placeholder referencing the DataEngine step."
),
)
xml_scrub_depth: int = Field(
xml_scrub_depth: int | None = Field(
default=1,
ge=1,
description=(
"How many of the most recent observations keep their UI Element"
" list; older lists are stripped (depth 1 matches the legacy"
" prune_history_xml semantics)."
"Text edge of the scrub: once a message is deeper than this many"
" image-bearing observations (1 = as soon as a newer observation"
" exists) its UI element list, plan recitation and per-turn"
" ephemeral blocks are removed. Kept shallow on purpose: only the"
" live observation carries an indexed element list, so a stale"
" [n] index can never be picked up as a target. Screenshots follow"
" image_scrub_depth / image_scrub_depth_relaxed."
),
)
context_budget_tokens: int = Field(
@@ -491,14 +507,27 @@ class MemoryTranscriptConfig(BaseModel):
" ratios below are applied against this budget."
),
)
start_ratio: float = Field(
default=0.35,
ge=0.0,
lt=1.0,
description=(
"Start gate: below budget*start_ratio of the operator's measured"
" prompt size, closed segments only prepare their capsule in the"
" background — the original turns stay in the transcript. Ready"
" chunks replace their turns once the measured context reaches"
" this ratio (the hard threshold overrides it). 0 restores"
" swap-as-soon-as-ready."
),
)
soft_ratio: float = Field(
default=0.7,
gt=0.0,
lt=1.0,
description=(
"Soft threshold: when the last measured prompt size reaches"
" budget*soft_ratio, the oldest open segment is chunk-compressed"
" (L2)."
"Soft threshold: when the operator's last measured prompt size"
" reaches budget*soft_ratio, the oldest open segment is"
" chunk-compressed (L2)."
),
)
hard_ratio: float = Field(
@@ -506,9 +535,13 @@ class MemoryTranscriptConfig(BaseModel):
gt=0.0,
le=1.0,
description=(
"Hard threshold: when the last measured prompt size reaches"
" budget*hard_ratio, the frozen region collapses to the L3 session"
" snapshot (chunk headers merged; per-step index retained)."
"Hard threshold: when the operator's last measured prompt size"
" reaches budget*hard_ratio, ready chunks swap as L2 blocks first;"
" only if that would not bring the estimated context back under the"
" line does everything closed force-swap (pending chunks included)"
" and the whole frozen region fold into a recall-only era — L3: the"
" period paragraph with the per-step index. The fold is permanent;"
" later L2 chunks are appended after it."
),
)
min_active_steps: int = Field(
@@ -545,6 +578,23 @@ class MemoryTranscriptConfig(BaseModel):
)
model_config = {"extra": "allow"}
@model_validator(mode="after")
def _ratios_are_ordered(self) -> "MemoryTranscriptConfig":
"""The three thresholds form a ladder against the same budget."""
if not (0.0 <= self.start_ratio <= self.soft_ratio <= self.hard_ratio <= 1.0):
raise ValueError(
"memory.transcript ratios must satisfy"
" 0 <= start_ratio <= soft_ratio <= hard_ratio <= 1"
f" (got start_ratio={self.start_ratio}, soft_ratio={self.soft_ratio},"
f" hard_ratio={self.hard_ratio})."
)
if self.image_scrub_depth_relaxed < self.image_scrub_depth:
raise ValueError(
"memory.transcript.image_scrub_depth_relaxed must be >= image_scrub_depth"
f" (got {self.image_scrub_depth_relaxed} < {self.image_scrub_depth})."
)
return self
class MemoryRecallConfig(BaseModel):
"""search_history tool options (agent.memory.recall)."""
@@ -620,8 +670,21 @@ class MemoryChunkingConfig(BaseModel):
default=2000,
ge=128,
description=(
"Size-threshold trigger: estimated source tokens (char/4 over the"
" open segment's transcript text) that force a chunk compression."
"Size-threshold trigger: estimated source tokens (transcript"
" characters through the session-calibrated ratio, 4 chars/token"
" by default) over the open segment's eligible portion that force"
" a chunk compression."
),
)
min_steps: int = Field(
default=3,
ge=1,
description=(
"Minimum turns in a chunk closed by the size or pressure trigger:"
" below this the eligible turns keep accumulating instead of"
" becoming one-step chunks (each chunk costs a capsule call)."
" Milestone closes are exempt (a complete segment is a unit) and"
" the hard threshold waives it (emergency)."
),
)
model: str = Field(
@@ -646,6 +709,16 @@ class MemoryChunkingConfig(BaseModel):
" behavior."
),
)
@model_validator(mode="after")
def _min_steps_within_cap(self) -> "MemoryChunkingConfig":
if self.min_steps > self.max_steps:
raise ValueError(
"memory.chunking.min_steps must not exceed max_steps"
f" (got min_steps={self.min_steps}, max_steps={self.max_steps})."
)
return self
model_config = {"extra": "allow"}
+12
View File
@@ -221,6 +221,18 @@ class ArtemisContext(BaseModel):
"""Plan content currently under async planner validation; becomes the new
baseline when the validator approves it."""
guidance_unprotected_checks: set[tuple[str, str]] = Field(default_factory=set)
"""``(kind, text)`` of every check line that existed in the task plan when a
mid-run user instruction arrived. User guidance outranks the plan's declared
standards, so these lines lose their machine restoration for the rest of the
run (the Operator may drop or reword them at any later write); check lines
added afterwards stay protected until the next instruction."""
guidance_retired_checks: set[tuple[str, str]] = Field(default_factory=set)
"""``(kind, text)`` of the check lines the Operator actually dropped under
that waiver. A verdict recorded for such a line before it was retired is a
result for a requirement the user withdrew: the run outcome reports it as
retired instead of counting it as a pass or failure."""
_genai_client: Any | None = PrivateAttr(default=None)
_video_blackboard: Any | None = PrivateAttr(default=None)
_video_circuit_breaker: Any | None = PrivateAttr(default=None)
+18
View File
@@ -20,6 +20,7 @@ from artemis.core.diagnostics.adb_keys import (
heal_adb_keys,
inspect_adb_keys,
)
from artemis.core.diagnostics.device_smoke import smoke_test_device
from artemis.core.diagnostics.emulator_manager import (
EmulatorLaunchStage,
EmulatorLaunchState,
@@ -32,11 +33,20 @@ from artemis.core.diagnostics.probes.credentials_probe import (
LLMCredentialsProbe,
VisionOCRProbe,
)
from artemis.core.diagnostics.probes.host_probe import IntegrationHostProbe
from artemis.core.diagnostics.probes.runtime_probe import (
PythonRuntimeProbe,
SystemConfigProbe,
)
from artemis.core.diagnostics.probes.toolchain_probe import ToolchainProbe
from artemis.core.diagnostics.readiness import (
CHECK_ORDER,
Verdict,
adb_keys_corrupted,
base_verdict,
collect_readiness,
sort_by_fix_order,
)
from artemis.core.diagnostics.schema import (
DeviceInfo,
ProbeAction,
@@ -53,6 +63,7 @@ __all__ = [
"EmulatorLaunchStage",
"EmulatorLaunchState",
"BaseProbe",
"IntegrationHostProbe",
"PythonRuntimeProbe",
"SystemConfigProbe",
"AdbDeviceProbe",
@@ -69,4 +80,11 @@ __all__ = [
"get_adb_key_paths",
"inspect_adb_keys",
"heal_adb_keys",
"smoke_test_device",
"CHECK_ORDER",
"Verdict",
"collect_readiness",
"base_verdict",
"adb_keys_corrupted",
"sort_by_fix_order",
]
+359
View File
@@ -0,0 +1,359 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""End-to-end device observation smoke test.
The readiness ADB probe only proves that ``adb devices`` and ``dumpsys`` answer.
The most common "device shows Connected but ARTEMIS cannot observe it" failure
is UIAutomator not answering (screenshot / hierarchy dump hangs), which today
only surfaces as a failed task. :func:`smoke_test_device` performs exactly the
observation the ``mobile_get_device_state`` MCP tool performs -- build (or reuse)
the cached device controller through ``artemis.mcp.adb_server._get_controller``
and call ``controller.get_screen_data()`` -- and reports a structured verdict
with concrete repair guidance instead of raising.
Design notes
------------
* ``_get_controller`` is synchronous, caches one controller per serial for the
life of the process and never touches ``DeviceExecutionLock``. Its first call
for a serial blocks on ``UIAutomatorClient.get_screen_data()`` (three
``u2.connect`` attempts plus a screenshot and hierarchy dump), so it runs on a
worker thread here.
* ``AdbDriver.get_screen_data`` invokes the synchronous UIAutomator client on
the event-loop thread. A plain ``asyncio.wait_for`` around the coroutine can
therefore never fire while UIAutomator hangs. The coroutine is executed on a
dedicated daemon thread with its own event loop, and the deadline is enforced
on the join. A thread abandoned by a timeout finishes on its own once the
underlying adb / HTTP timeouts elapse; being a daemon it never delays
interpreter shutdown.
* ``u2.connect`` restarts the on-device UIAutomator server when it believes the
server is unhealthy. Running that against a device another ARTEMIS process is
driving would break the running task's session, so a device with a live
``DeviceExecutionLock`` owner is reported as busy instead of probed.
* Nothing is left behind: no screenshot is written to disk, no lock is taken,
and -- mirroring ``mobile_get_device_state`` -- the cached controller is kept
for reuse rather than closed.
"""
from __future__ import annotations
import asyncio
import base64
from collections.abc import Callable
import os
import threading
import time
from typing import Any, TypeVar
from artemis.runtime.device_lock import DeviceExecutionLock
from artemis.utils.logger import get_logger
logger = get_logger(__name__)
T = TypeVar("T")
#: On-device packages installed by uiautomator2 (``python -m uiautomator2 init``).
UIAUTOMATOR_PACKAGES: tuple[str, ...] = (
"com.github.uiautomator",
"com.github.uiautomator.test",
)
#: A real JPEG screenshot of any phone screen is tens of kilobytes; the Android
#: driver substitutes a 1x1 PNG (~70 bytes) when every capture path failed.
_MIN_PLAUSIBLE_SCREENSHOT_BYTES = 512
_SERIAL_PLACEHOLDER = "<serial>"
def _serial_or_placeholder(serial: str | None) -> str:
return serial or _SERIAL_PLACEHOLDER
def _uiautomator_fix(serial: str | None) -> list[str]:
s = _serial_or_placeholder(serial)
force_stops = ", then ".join(
f"adb -s {s} shell am force-stop {pkg}" for pkg in UIAUTOMATOR_PACKAGES
)
return [
"Unlock the phone and keep the screen on (UIAutomator cannot dump a locked or sleeping screen).",
f"Restart the on-device UIAutomator server, then retry: {force_stops}",
(
"If it keeps failing, reinstall the UIAutomator server APKs: "
f"python -m uiautomator2 purge --serial {s}, then python -m uiautomator2 init --serial {s} "
"(run with the ARTEMIS virtualenv Python), or reboot the device."
),
]
def fix_for_error(error: str | None, serial: str | None) -> list[str]:
"""Map an error string to concrete repair steps (substring match, case-insensitive)."""
if not error:
return []
text = error.lower()
s = _serial_or_placeholder(serial)
if "busy" in text or "another task" in text:
return [
"Wait for the running task to finish, or stop it with mobile_manage_task(action='stop', ...).",
"Then rerun the smoke test.",
]
if "unauthorized" in text:
return [
"Accept the 'Allow USB debugging' prompt on the phone (tick 'Always allow from this computer').",
f"If no prompt appears, re-plug the USB cable and run: adb -s {s} devices",
]
if "offline" in text:
return [
"Re-plug the USB cable (try another port or cable).",
"Run: adb reconnect (or adb kill-server, then adb start-server) and check adb devices -l.",
]
if "no android devices" in text or "no device" in text or "not found" in text:
return [
"Connect an Android device with USB debugging enabled, or start an emulator.",
"mobile_diagnose(launch_avd='<avd name>') boots a named AVD; adb devices -l lists what is attached.",
]
if (
"timeout" in text
or "timed out" in text
or "uiautomator" in text
or "not responding" in text
or "did not respond" in text
or "hierarchy" in text
or "screenshot" in text
or "http" in text
or "deadobject" in text
):
return _uiautomator_fix(serial)
return [
f"Check the raw hierarchy dump manually: adb -s {s} shell uiautomator dump, then adb -s {s} shell cat /sdcard/window_dump.xml",
f"Check the raw screenshot path: adb -s {s} exec-out screencap -p > screen.png",
]
#: Exceptions the smoke test must never swallow.
_PASSTHROUGH_EXCEPTIONS: tuple[type[BaseException], ...] = (
KeyboardInterrupt,
SystemExit,
asyncio.CancelledError,
)
def _describe_failure(exc: BaseException) -> str:
"""uiautomator2 errors derive from ``BaseException`` and often carry an empty message."""
text = str(exc).strip()
name = type(exc).__name__
if not text:
return name
return text if name in ("Exception", "RuntimeError") else f"{name}: {text}"
def _resolve_requested_serial(device_serial: str | None) -> str | None:
"""Mirror ``_get_controller``'s serial resolution (explicit arg, then env)."""
return (
device_serial or os.environ.get("ARTEMIS_DEVICE_ID") or os.environ.get("ADB_DEVICE_SERIAL")
)
def _find_busy_owner(target_serial: str | None) -> tuple[str, Any] | None:
"""Return ``(device_key, owner)`` when a live ARTEMIS process owns the target device.
With no target serial ``_get_controller`` would pick the first attached device,
which cannot be matched against lock records cheaply; any live owner is then
treated as busy to stay on the safe side.
"""
try:
owners = DeviceExecutionLock.get_active_owners()
except Exception as exc: # pylint: disable=broad-exception-caught
logger.debug(f"Device lock inspection failed, skipping busy guard: {exc}")
return None
if not owners:
return None
if target_serial is None:
key, owner = next(iter(owners.items()))
return key, owner
clean = DeviceExecutionLock._normalize_device_id(target_serial)
for key, owner in owners.items():
if key == clean or key.endswith(f"__{clean}"):
return key, owner
return None
async def _run_in_daemon_thread(fn: Callable[[], T], timeout: float, what: str) -> T:
"""Run ``fn`` on a daemon thread and wait at most ``timeout`` seconds.
Raises :class:`TimeoutError` on deadline. The thread keeps running until
``fn`` returns on its own, but the caller is released immediately.
"""
loop = asyncio.get_running_loop()
future: asyncio.Future[T] = loop.create_future()
def _deliver(setter: Callable[[], None]) -> None:
try:
loop.call_soon_threadsafe(setter)
except RuntimeError:
# Event loop already closed: the caller gave up on us.
pass
def _worker() -> None:
try:
value = fn()
except BaseException as exc: # pylint: disable=broad-exception-caught
error: BaseException = exc
def _fail() -> None:
if not future.done():
future.set_exception(error)
_deliver(_fail)
return
def _succeed() -> None:
if not future.done():
future.set_result(value)
_deliver(_succeed)
threading.Thread(target=_worker, name=f"artemis-smoke-{what}", daemon=True).start()
try:
return await asyncio.wait_for(future, timeout=timeout)
except TimeoutError:
logger.warning(f"Device smoke test: {what} did not finish within {timeout:.1f}s")
raise
def _run_coroutine_blocking(coro_factory: Callable[[], Any]) -> Any:
"""Execute a coroutine on a fresh event loop owned by the current thread."""
return asyncio.run(coro_factory())
def _count_elements(elements: Any) -> int | None:
if elements is None:
return None
if isinstance(elements, str):
return elements.count("<node")
if isinstance(elements, dict):
return 1
try:
return len(elements)
except TypeError:
return None
def _decoded_length(b64: Any) -> int | None:
if not isinstance(b64, str) or not b64:
return None
try:
return len(base64.b64decode(b64))
except (ValueError, TypeError):
return None
async def smoke_test_device(
device_serial: str | None = None, timeout_seconds: float = 20.0
) -> dict[str, Any]:
"""Observe the device exactly like ``mobile_get_device_state`` and report a verdict.
Never raises. Returns::
{"ok": bool, "serial": str | None, "elapsed_seconds": float,
"screenshot_bytes": int | None, "element_count": int | None,
"error": str | None, "fix": list[str]}
"""
started = time.monotonic()
requested_serial = _resolve_requested_serial(device_serial)
result: dict[str, Any] = {
"ok": False,
"serial": requested_serial,
"elapsed_seconds": 0.0,
"screenshot_bytes": None,
"element_count": None,
"error": None,
"fix": [],
}
def _finish(error: str | None = None, cause: str | None = None) -> dict[str, Any]:
"""Record the verdict; ``cause`` is the raw failure text used to pick the fix."""
result["elapsed_seconds"] = round(time.monotonic() - started, 3)
result["error"] = error
result["ok"] = error is None
result["fix"] = fix_for_error(cause if cause is not None else error, result["serial"])
return result
busy = _find_busy_owner(requested_serial)
if busy is not None:
key, owner = busy
description = getattr(owner, "description", "") or "unknown task"
pid = getattr(owner, "pid", "?")
result["serial"] = requested_serial or getattr(owner, "device_id", None)
return _finish(
f"Device {key} is busy: another task is running on it ({description}, pid {pid});"
" skipping the screen capture so the running task is not disturbed."
)
try:
from artemis.mcp import adb_server
controller = await _run_in_daemon_thread(
lambda: adb_server._get_controller(device_serial=device_serial),
timeout_seconds,
"controller-init",
)
except TimeoutError:
return _finish(
f"UIAutomator/controller initialization did not respond within {timeout_seconds:g}s"
)
except _PASSTHROUGH_EXCEPTIONS:
raise
except BaseException as exc: # pylint: disable=broad-exception-caught
cause = _describe_failure(exc)
return _finish(f"Failed to initialize Android device controller: {cause}", cause)
controller_ctx = getattr(controller, "ctx", None)
device = getattr(controller_ctx, "device", None)
device_id = getattr(device, "device_id", None)
if isinstance(device_id, str) and device_id:
result["serial"] = device_id
remaining = max(0.5, timeout_seconds - (time.monotonic() - started))
try:
device_data = await _run_in_daemon_thread(
lambda: _run_coroutine_blocking(controller.get_screen_data),
remaining,
"screen-data",
)
except TimeoutError:
return _finish(f"UIAutomator/screen capture did not respond within {timeout_seconds:g}s")
except _PASSTHROUGH_EXCEPTIONS:
raise
except BaseException as exc: # pylint: disable=broad-exception-caught
cause = _describe_failure(exc)
return _finish(f"Screen capture failed: {cause}", cause)
result["screenshot_bytes"] = _decoded_length(getattr(device_data, "base64", None))
result["element_count"] = _count_elements(getattr(device_data, "elements", None))
if result["screenshot_bytes"] is None:
return _finish("Screen capture returned no screenshot data")
if result["screenshot_bytes"] < _MIN_PLAUSIBLE_SCREENSHOT_BYTES:
return _finish(
f"Screenshot capture failed (driver returned a {result['screenshot_bytes']}-byte placeholder image)"
)
if not result["element_count"]:
return _finish(
"UIAutomator hierarchy dump returned no UI elements (screenshot worked, hierarchy did not)"
)
return _finish(None)
__all__ = ["UIAUTOMATOR_PACKAGES", "fix_for_error", "smoke_test_device"]
+80 -4
View File
@@ -15,6 +15,7 @@
"""System Readiness & Diagnostic Orchestration Engine."""
import asyncio
import os
import subprocess
import time
from typing import Any
@@ -33,6 +34,7 @@ from artemis.core.diagnostics.probes.runtime_probe import (
from artemis.core.diagnostics.probes.toolchain_probe import ToolchainProbe
from artemis.core.diagnostics.schema import (
DeviceInfo,
ProbeAction,
ProbeCategory,
ProbeResult,
ProbeStatus,
@@ -50,6 +52,13 @@ class ReadinessEngine:
_REPORT_CACHE_TTL_SECONDS = 2.0
#: Deadline for one probe inside a report. The ADB probe shells out to
#: ``adb`` without a deadline of its own, and a wedged ADB server would
#: otherwise hang every surface that runs the report (``artemis doctor``,
#: the console wizard, ``mobile_diagnose``). A probe that overruns is
#: reported as a FAIL with the recovery steps instead.
PROBE_TIMEOUT_SECONDS = 30.0
def __init__(self):
self._probes: dict[str, BaseProbe] = {}
self._python_probe = PythonRuntimeProbe()
@@ -178,11 +187,22 @@ class ReadinessEngine:
if categories is None or probe.category in categories
]
# Concurrently execute probes
results: list[ProbeResult] = await asyncio.gather(
*[probe.probe() for probe in target_probes],
return_exceptions=False,
# Concurrently execute probes. A probe that raises must not take the
# whole report down with it: it becomes a structured FAIL so the
# remaining probes still reach the user and the verdict stays honest.
outcomes = await asyncio.gather(
*[
asyncio.wait_for(probe.probe(), timeout=self.PROBE_TIMEOUT_SECONDS)
for probe in target_probes
],
return_exceptions=True,
)
results: list[ProbeResult] = [
outcome
if isinstance(outcome, ProbeResult)
else self._crashed_probe_result(probe, outcome)
for probe, outcome in zip(target_probes, outcomes, strict=True)
]
blockers = [r for r in results if r.is_blocker]
passed_blockers = [r for r in blockers if r.status == ProbeStatus.PASS]
@@ -211,6 +231,62 @@ class ReadinessEngine:
timestamp=time.time(),
)
@staticmethod
def _crashed_probe_result(probe: BaseProbe, exc: object) -> ProbeResult:
"""Turn an exception escaping ``probe.probe()`` into a FAIL result.
Cancellation and other non-``Exception`` errors are re-raised: they are
not a diagnosis of the host, they are the event loop shutting us down.
"""
if isinstance(exc, BaseException) and not isinstance(exc, Exception):
raise exc
if isinstance(exc, TimeoutError):
logger.warning(
f"[ReadinessEngine] Probe '{probe.probe_id}' did not finish within"
f" {ReadinessEngine.PROBE_TIMEOUT_SECONDS:.0f}s; reporting it as FAIL."
)
return ProbeResult(
id=probe.probe_id,
category=probe.category,
title=probe.probe_id.replace("_", " ").title(),
status=ProbeStatus.FAIL,
is_blocker=probe.is_blocker,
summary="Probe timed out",
description=(
f"The '{probe.probe_id}' check did not finish within"
f" {ReadinessEngine.PROBE_TIMEOUT_SECONDS:.0f}s. A hung ADB server is the"
" usual cause: restart it and run the diagnosis again."
),
actions=[
ProbeAction(
action_type="command",
label="Restart ADB",
payload="adb kill-server && adb start-server",
),
],
metadata={
"exception_type": "TimeoutError",
"timeout_seconds": ReadinessEngine.PROBE_TIMEOUT_SECONDS,
},
)
logger.warning(
f"[ReadinessEngine] Probe '{probe.probe_id}' crashed; reporting it as FAIL: "
f"{type(exc).__name__}: {exc}"
)
return ProbeResult(
id=probe.probe_id,
category=probe.category,
title=probe.probe_id.replace("_", " ").title(),
status=ProbeStatus.FAIL,
is_blocker=probe.is_blocker,
summary="Probe crashed",
description=(
f"The '{probe.probe_id}' check raised {type(exc).__name__}: {exc}. "
"This is a diagnostics bug or a host permission problem, not a device fault."
),
metadata={"exception_type": type(exc).__name__, "exception": str(exc)},
)
async def heal_adb_keys(self, force: bool = False) -> dict[str, Any]:
"""Auto-heal corrupted or 0-byte ADB authentication keys."""
from artemis.core.diagnostics.adb_keys import heal_adb_keys
@@ -284,7 +284,7 @@ class VisionOCRProbe(BaseProbe):
status=ProbeStatus.PASS,
is_blocker=False,
summary="Not Configured (Optional)",
description="OCR_API_KEY is not set. Perception operates seamlessly in pure UI XML hierarchy mode.",
description="OCR_API_KEY is not set. Perception uses the UI XML hierarchy without OCR.",
metadata={"configured": False},
actions=[
ProbeAction(
@@ -0,0 +1,427 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Integration-host probe: the process environment an IDE / MCP client launched us in.
The standard readiness probes answer "can Artemis drive a phone on this
machine?". This probe answers the question that precedes it when Artemis is
embedded in an AI coding assistant (Claude Code, Cursor, Codex, Antigravity,
...): "was the server launched the way the installer expects, and can it
spawn a background runner?". The failure modes it covers are the ones users
hit mostly through MCP, rarely from the CLI or the web console:
* the MCP client started the server with a different interpreter than the
project virtualenv, so the spawned runner imports a different package set;
* the ``.env`` file the settings loader reads is not where the user put the key;
* the traces directory is not writable from the client's sandbox;
* the daemon port is held by another process (a very common collision with
dev servers that also default to port 8000).
The probe is not registered in the default :class:`ReadinessEngine` probe set:
the web console has its own device wizard and would only render noise. The
``mobile_diagnose`` MCP tool and ``artemis doctor`` run it explicitly.
Only the unwritable-traces case blocks task execution; the result carries
``is_blocker`` accordingly, so the WARN findings degrade the verdict without
blocking it.
"""
from __future__ import annotations
import os
from pathlib import Path
import sys
import tempfile
from typing import Any
from artemis.core.diagnostics.probes.base import BaseProbe
from artemis.core.diagnostics.schema import (
ProbeAction,
ProbeCategory,
ProbeResult,
ProbeStatus,
)
def _same_file(a: str | os.PathLike[str], b: str | os.PathLike[str]) -> bool:
"""Compare two interpreter paths tolerantly (symlinks, case on Windows)."""
try:
return os.path.samefile(a, b)
except OSError:
return os.path.normcase(os.path.abspath(a)) == os.path.normcase(os.path.abspath(b))
#: Client names accepted by ``artemis mcp --install`` (see interfaces/cli/commands/mcp.py).
MCP_CLIENT_NAMES: tuple[str, ...] = (
"antigravity",
"claude",
"cursor",
"windsurf",
"vscode",
"cline",
"roo",
"openclaw",
"codex",
)
def _env_has_prefix(prefix: str, *, exclude: tuple[str, ...] = ()) -> bool:
return any(key.startswith(prefix) and key not in exclude for key in os.environ)
def detect_mcp_client() -> str | None:
"""Best-effort guess of which MCP client launched this process, from ``os.environ``.
Returns one of :data:`MCP_CLIENT_NAMES` or ``None`` when unsure. MCP hosts
spawn the server with their own process environment, so the variables the
IDE exports to its child processes leak through. Specific hosts are checked
before the generic VS Code markers because Cursor, Windsurf, Cline and Roo
are (or run inside) VS Code and inherit ``VSCODE_*`` / ``TERM_PROGRAM``.
"""
# Antigravity injects its language-server endpoint and CSRF token
# (ANTIGRAVITY_LS_ADDRESS / ANTIGRAVITY_CSRF_TOKEN) so child processes can
# call `agentapi`; mcp_server/notifiers/agentapi.py relies on the same pair.
if _env_has_prefix("ANTIGRAVITY_"):
return "antigravity"
# Claude Code marks every child with CLAUDECODE=1 plus CLAUDE_CODE_* details
# (entrypoint, session id, ...). Claude Desktop shares the same installer
# target, so both map to "claude".
if os.environ.get("CLAUDECODE") == "1" or _env_has_prefix("CLAUDE_CODE_"):
return "claude"
# Cursor tags its integrated terminal and spawned processes with
# CURSOR_TRACE_ID (and CURSOR_AGENT in agent mode).
if "CURSOR_TRACE_ID" in os.environ or _env_has_prefix("CURSOR_"):
return "cursor"
# Windsurf (Codeium) exports WINDSURF_* to its child processes.
if _env_has_prefix("WINDSURF_"):
return "windsurf"
# Codex CLI sets CODEX_SANDBOX* / CODEX_* for the processes it runs.
# CODEX_HOME alone is excluded: users set it by hand to relocate the Codex
# config directory, so on its own it does not prove Codex launched us.
if _env_has_prefix("CODEX_", exclude=("CODEX_HOME",)):
return "codex"
# OPENCLAW_WEBHOOK_URL is a notifier setting users put in any host's env
# block (see mcp_server/notifiers/webhook.py); only other OPENCLAW_* vars
# indicate the OpenClaw runtime itself.
if _env_has_prefix("OPENCLAW_", exclude=("OPENCLAW_WEBHOOK_URL",)):
return "openclaw"
# Cline / Roo Code run as VS Code extensions; their own prefixes are rare
# but unambiguous when present, and must win over the generic VS Code check.
if _env_has_prefix("CLINE_"):
return "cline"
if _env_has_prefix("ROO_"):
return "roo"
# Plain VS Code (Copilot agent mode) exports VSCODE_PID / VSCODE_IPC_HOOK
# and sets TERM_PROGRAM=vscode. Weakest signal, so it is checked last.
if (
"VSCODE_PID" in os.environ
or "VSCODE_IPC_HOOK" in os.environ
or os.environ.get("TERM_PROGRAM") == "vscode"
):
return "vscode"
return None
def mcp_install_actions(client: str | None) -> list[ProbeAction]:
"""Actions that regenerate the MCP client config for ``client`` (or ask which one)."""
if client:
return [
ProbeAction(
action_type="command",
label="Regenerate MCP Config",
payload=f"uv run artemis mcp --install {client}",
)
]
return [
ProbeAction(
action_type="hint",
label="Regenerate MCP Config",
payload=(
"Could not tell which MCP client launched this server. Ask the user which IDE"
" they are using, then run `uv run artemis mcp --install <client>` with"
" <client> being one of: " + ", ".join(MCP_CLIENT_NAMES) + "."
),
)
]
def project_venv_python(project_root: Path) -> Path:
"""Return the interpreter path the installer writes into MCP client configs."""
if sys.platform == "win32":
return project_root / ".venv" / "Scripts" / "python.exe"
return project_root / ".venv" / "bin" / "python"
def directory_is_writable(path: Path) -> tuple[bool, str | None]:
"""Try to create and delete a scratch file; return (ok, error)."""
try:
path.mkdir(parents=True, exist_ok=True)
with tempfile.NamedTemporaryFile(dir=path, prefix=".artemis-diag-", delete=True):
pass
return True, None
except OSError as exc:
return False, str(exc)
def intended_traces_path() -> Path:
"""Return the traces directory the settings loader *would* pick, without creating it.
Mirrors the selection order of :func:`artemis.config.paths.get_default_traces_path`
(``ARTEMIS_TRACES_DIR`` env, else the user app dir when the install mode
keeps mutable state there, else ``ROOT_DIR/traces``) minus its ``mkdir``.
Used only when that function raised, so the probe can still name the
offending directory in its FAIL result instead of crashing.
"""
from artemis.config import paths as artemis_paths
env_traces = os.getenv(artemis_paths.ENV_ARTEMIS_TRACES_DIR)
if env_traces:
return Path(env_traces)
if artemis_paths._use_user_app_dir():
return artemis_paths.get_app_dir() / "traces"
return Path(artemis_paths.ROOT_DIR) / "traces"
def resolve_traces_dir() -> tuple[Path, str | None]:
"""Return ``(traces_dir, error)``; ``error`` is set when the directory could not be created.
``get_default_traces_path`` creates the directory as a side effect, so on a
read-only parent it raises before :func:`directory_is_writable` ever runs.
Surface that as data rather than letting the probe itself crash.
"""
from artemis.config.paths import get_default_traces_path
try:
return Path(get_default_traces_path()), None
except OSError as exc:
return intended_traces_path(), str(exc)
class IntegrationHostProbe(BaseProbe):
"""Probe verifying the MCP / IDE host process can actually run Artemis tasks."""
@property
def probe_id(self) -> str:
return "integration_host"
@property
def category(self) -> ProbeCategory:
return ProbeCategory.RUNTIME
@property
def is_blocker(self) -> bool:
return True
async def probe(self) -> ProbeResult:
# Lazy imports keep this module importable in stripped test environments
# and avoid a core -> runtime import at package import time.
from artemis.config.paths import ROOT_DIR, get_env_file, is_source_checkout
from artemis.runtime.daemon_client import (
DEFAULT_DAEMON_HOST,
DEFAULT_DAEMON_PORT,
daemon_log_path,
is_artemis_daemon,
is_standalone_forced,
)
from artemis.runtime.server_lifecycle import is_port_in_use
project_root = Path(ROOT_DIR)
source_checkout = is_source_checkout()
server_python = Path(sys.executable)
# A project virtualenv is only meaningful for a source checkout. In a
# wheel install ROOT_DIR is site-packages, which never carries a .venv;
# the current interpreter also runs background tasks.
venv_python: Path | None = project_venv_python(project_root) if source_checkout else None
venv_exists = venv_python is not None and venv_python.is_file()
runner_python = venv_python if venv_exists else server_python
interpreter_matches_venv = venv_exists and _same_file(server_python, venv_python)
mcp_client = detect_mcp_client()
env_file = Path(get_env_file())
traces_dir, traces_error = resolve_traces_dir()
if traces_error is None:
traces_writable, traces_error = directory_is_writable(traces_dir)
else:
traces_writable = False
standalone = is_standalone_forced()
# Identity, not liveness: a dev server squatting on the port answers
# HTTP too, and must show up as a hijacked port, not as the daemon.
daemon_reachable = False if standalone else is_artemis_daemon()
port_in_use = is_port_in_use(DEFAULT_DAEMON_PORT, DEFAULT_DAEMON_HOST)
port_hijacked = port_in_use and not daemon_reachable and not standalone
metadata: dict[str, Any] = {
"os": sys.platform,
"mcp_client": mcp_client,
"project_root": str(project_root),
"is_source_checkout": source_checkout,
"server_python": str(server_python),
"venv_python": str(venv_python) if venv_python is not None else None,
"venv_exists": venv_exists,
"interpreter_matches_venv": interpreter_matches_venv,
"runner_python": str(runner_python),
"env_file": str(env_file),
"env_file_exists": env_file.is_file(),
"traces_dir": str(traces_dir),
"traces_dir_writable": traces_writable,
"traces_dir_error": traces_error,
"daemon": {
"host": DEFAULT_DAEMON_HOST,
"port": DEFAULT_DAEMON_PORT,
"standalone_forced": standalone,
"reachable": daemon_reachable,
"port_in_use": port_in_use,
"port_held_by_other_process": port_hijacked,
"log_path": str(daemon_log_path()),
},
}
problems: list[str] = []
actions: list[ProbeAction] = []
status = ProbeStatus.PASS
if not traces_writable:
status = ProbeStatus.FAIL
problems.append(f"traces directory {traces_dir} is not writable ({traces_error})")
actions.append(
ProbeAction(
action_type="hint",
label="Fix Traces Directory",
payload=(
f"Make {traces_dir} writable for the user running the MCP server, or point "
"ARTEMIS_TRACES_DIR at a writable directory in the MCP server's env block, "
"then restart the MCP server."
),
)
)
if port_hijacked:
status = ProbeStatus.WARN if status is ProbeStatus.PASS else status
problems.append(
f"port {DEFAULT_DAEMON_PORT} is held by a process that is not the Artemis daemon"
)
actions.append(
ProbeAction(
action_type="hint",
label="Free the Daemon Port",
payload=(
f"Another program is listening on {DEFAULT_DAEMON_HOST}:{DEFAULT_DAEMON_PORT}"
" (often a dev server). Stop it, or set ARTEMIS_DAEMON_PORT to a free port in"
" the MCP server's env block and restart the MCP server. Tasks still run in"
" standalone mode meanwhile, without the shared scheduler."
),
)
)
if not source_checkout:
# Wheel / frozen install: no virtualenv expectation, nothing to warn about.
pass
elif venv_exists and not interpreter_matches_venv:
status = ProbeStatus.WARN if status is ProbeStatus.PASS else status
problems.append(
f"MCP server runs on {server_python}, not the project virtualenv {venv_python}"
)
actions.extend(mcp_install_actions(mcp_client))
actions.append(
ProbeAction(
action_type="hint",
label="Interpreter Mismatch",
payload=(
"The MCP client config should point `command` at the .venv interpreter so the"
" server and the background runner share one package set. Restart the MCP"
" server after changing the config."
),
)
)
elif not venv_exists:
status = ProbeStatus.WARN if status is ProbeStatus.PASS else status
problems.append(
"no project virtualenv (.venv) found; runner falls back to the server interpreter"
)
actions.append(
ProbeAction(
action_type="command",
label="Create Virtualenv",
payload="uv sync",
)
)
if not env_file.is_file():
actions.append(
ProbeAction(
action_type="hint",
label="Env File Location",
payload=(
f"No .env file at {env_file}. API keys are read from that file or from the MCP"
" server's env block; the file is created by `artemis init`."
),
)
)
if not daemon_reachable and not port_hijacked:
actions.append(
ProbeAction(
action_type="hint",
label="Daemon Auto-Start",
payload=(
"Standalone mode is forced (ARTEMIS_STANDALONE=1); tasks run as detached"
" runner processes."
if standalone
else "The Artemis daemon is not running; mobile_run_task starts it on the"
f" first task. Its log is {daemon_log_path()}."
),
)
)
if status is ProbeStatus.PASS:
summary = "Host Ready"
description = (
f"MCP server runs on {server_python}; runner uses {runner_python}. "
f"Traces at {traces_dir}. "
+ (
"Daemon reachable."
if daemon_reachable
else "Daemon will auto-start on the first task."
)
)
actions.insert(
0,
ProbeAction(
action_type="hint",
label="Host OK",
payload="Interpreter, traces directory and daemon port look consistent.",
),
)
else:
summary = "Host Misconfigured" if status is ProbeStatus.FAIL else "Host Warning"
joined = "; ".join(problems)
description = joined[:1].upper() + joined[1:] + "."
# Only an unwritable traces directory stops tasks from running. The
# WARN findings (interpreter mismatch, missing venv, squatted port) are
# degradations the runner works around, so they must not turn the
# verdict into "blocked" — a pip/conda checkout without .venv would
# otherwise never leave that state.
return ProbeResult(
id=self.probe_id,
category=self.category,
title="MCP / IDE Integration Host",
status=status,
is_blocker=status is not ProbeStatus.WARN,
summary=summary,
description=description,
metadata=metadata,
actions=actions,
)
+83
View File
@@ -0,0 +1,83 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Readiness collection and verdict shared by every diagnosis entry point.
``artemis doctor`` and the ``mobile_diagnose`` MCP tool must agree on which
probes run, in which order they are fixed, and what "ready" means. This module
holds exactly that common core; each entry point layers its own extra
conditions, fix policy and rendering on top.
"""
from __future__ import annotations
import asyncio
from collections.abc import Sequence
from typing import Literal
from artemis.core.diagnostics.engine import readiness_engine
from artemis.core.diagnostics.probes.host_probe import IntegrationHostProbe
from artemis.core.diagnostics.schema import ProbeResult, ProbeStatus, SystemReadinessReport
Verdict = Literal["ready", "degraded", "blocked"]
#: Fix order. Runtime problems mask everything after them, credentials mask
#: device problems (a task cannot start without a model), and the toolchain is
#: optional.
CHECK_ORDER: dict[str, int] = {
"python_runtime": 0,
"system_config": 1,
"integration_host": 2,
"gemini_api_key": 3,
"android_adb": 4,
"toolchain": 5,
"vision_ocr_key": 6,
}
def sort_by_fix_order(results: Sequence[ProbeResult]) -> list[ProbeResult]:
"""Order probe results so the first problem listed is the one to fix first."""
return sorted(results, key=lambda r: CHECK_ORDER.get(r.id, len(CHECK_ORDER)))
async def collect_readiness() -> tuple[SystemReadinessReport, ProbeResult]:
"""Run the engine probes and the integration-host probe concurrently."""
report, host = await asyncio.gather(
readiness_engine.run_all(force_refresh=True),
IntegrationHostProbe().probe(),
)
return report, host
def base_verdict(results: Sequence[ProbeResult]) -> Verdict:
"""Verdict from probe results alone.
Blocked when there is no blocker at all (nothing was checked) or any
blocker is not PASS; degraded when only optional checks fail; ready
otherwise.
"""
blockers = [r for r in results if r.is_blocker]
if not blockers or any(r.status is not ProbeStatus.PASS for r in blockers):
return "blocked"
if any(r.status is not ProbeStatus.PASS for r in results):
return "degraded"
return "ready"
def adb_keys_corrupted(results: Sequence[ProbeResult]) -> bool:
"""Whether the ADB probe reported corrupted RSA keys (the one self-heal both entry points share)."""
adb = next((r for r in results if r.id == "android_adb"), None)
if adb is None:
return False
return bool((adb.metadata.get("adb_keys") or {}).get("is_corrupted"))
+9 -2
View File
@@ -666,6 +666,13 @@ class DataEngine:
if session and session.end_time is not None and session.status not in ("running", "paused"):
logger.debug(f"Session end already published for {session_id}; skipping duplicate")
return
# Session-level LLM usage line (cache-hit ratios per source), best-effort.
try:
from artemis.services.token_meter import log_session_summary
log_session_summary(session_id)
except Exception as e:
logger.debug(f"Session usage summary skipped: {e}")
if session:
session.end_time = end_time
session.status = status
@@ -877,7 +884,7 @@ class DataEngine:
if pre_image_name and post_image_name and pre_image_name == post_image_name:
post_image_name = None
# M5: persist the foreground app for the recall search surface (the
# Persist the foreground app for the recall search surface (the
# parameter was historically accepted but never stored). The explicit
# parameter wins; otherwise it is derived best-effort from the UI
# tree's package attributes — a pure string scan, no device call.
@@ -889,7 +896,7 @@ class DataEngine:
except (AttributeError, TypeError, ValueError):
pass
# M4: stamp perceptual hashes for the local screen-similarity hint
# Stamp perceptual hashes for the local screen-similarity hint
# (pure PIL dHash; best-effort, computed synchronously to keep the
# record immutable once handed to the background writer).
try:
+58 -6
View File
@@ -14,6 +14,7 @@
import base64
from collections.abc import Callable
import contextlib
import functools
import hashlib
import inspect
@@ -175,6 +176,20 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
def on_tool_error(self, error: Any, *, run_id: uuid.UUID, **kwargs: Any) -> Any:
pass
@staticmethod
def _detached_lens() -> bool:
"""True inside ``detached_trace('lens:...')``: a background lens call
(visual summary, segment capsule) that belongs to no agent step."""
node_name = CURRENT_NODE_NAME.get() or ""
return CURRENT_TRACE_ID.get() is None and node_name.startswith("lens:")
def _current_step_id(self):
"""The step a trace attaches to: the engine's current step, or None for
a detached lens call (its reply must never read as that step's work)."""
if self._detached_lens():
return None
return self.ctx.data_engine.current_step_id
def on_llm_start(
self,
serialized: dict[str, Any],
@@ -200,7 +215,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
trace_id=run_id,
parent_trace_id=self._resolve_parent_id(parent_run_id, kwargs),
status="running",
step_id=self.ctx.data_engine.current_step_id,
step_id=self._current_step_id(),
)
def on_chat_model_start(
@@ -233,7 +248,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
trace_id=run_id,
parent_trace_id=self._resolve_parent_id(parent_run_id, kwargs),
status="running",
step_id=self.ctx.data_engine.current_step_id,
step_id=self._current_step_id(),
)
def _safe_uuid5(self, parent_id, name: str) -> uuid.UUID:
@@ -253,6 +268,11 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
) -> Any:
"""Run when LLM ends running."""
if self.ctx.data_engine:
# A background lens call (see ``detached_trace``) belongs to no step:
# its reply is a compression product, never an agent's reasoning, so
# it is neither attached to the current step nor recorded as thinking.
detached = self._detached_lens()
step_id = self._current_step_id()
generations = getattr(response, "generations", [])
flat_generations = []
if generations:
@@ -308,7 +328,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
parent_trace_id=self._resolve_parent_id(parent_run_id),
status="success",
duration=duration,
step_id=self.ctx.data_engine.current_step_id,
step_id=step_id,
)
native_thoughts = []
@@ -327,6 +347,8 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
elif gen_item.get("text") and str(gen_item.get("text")).strip():
raw_thoughts.append(str(gen_item.get("text")).strip())
if detached:
native_thoughts, raw_thoughts = [], []
if native_thoughts:
self.ctx.data_engine.record_trace(
type="thinking",
@@ -335,7 +357,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
trace_id=self._safe_uuid5(run_id, "thinking"),
parent_trace_id=run_id,
status="success",
step_id=self.ctx.data_engine.current_step_id,
step_id=step_id,
)
if raw_thoughts:
self.ctx.data_engine.record_trace(
@@ -345,7 +367,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
trace_id=self._safe_uuid5(run_id, "raw_thinking"),
parent_trace_id=run_id,
status="success",
step_id=self.ctx.data_engine.current_step_id,
step_id=step_id,
)
def on_llm_error(
@@ -379,7 +401,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
parent_trace_id=self._resolve_parent_id(parent_run_id),
status="failed",
duration=duration,
step_id=self.ctx.data_engine.current_step_id,
step_id=self._current_step_id(),
)
def _serialize_message(self, msg: BaseMessage) -> dict[str, Any]:
@@ -666,6 +688,36 @@ class TraceSpan:
)
@contextlib.contextmanager
def detached_trace(node_name: str | None = None):
"""Run a block outside the caller's trace span.
Background work (the step-memory lenses: visual-transition summaries,
segment capsules) is dispatched while an agent's step span is current, and
an ``asyncio`` task inherits that context. Every model call made inside
the block would then hang off the agent's span: the callback handler
records its reply as a ``thinking``/``raw_thinking`` child of that span, so
``replay_steps``/``search_history`` later show a capsule JSON as the
operator's own reasoning, and the stream deltas surface under the
operator's timeline entry.
Inside the block :data:`CURRENT_TRACE_ID` is cleared (the block's traces
have no parent span and can never be attributed to the caller's step) and
:data:`CURRENT_NODE_NAME` is replaced by ``node_name`` (when given), so
usage metering labels the calls after the background job rather than the
agent that happened to dispatch it. Both are restored on exit, exceptions
included. Only the current task's context is touched.
"""
token = CURRENT_TRACE_ID.set(None)
token_name = CURRENT_NODE_NAME.set(node_name) if node_name is not None else None
try:
yield
finally:
CURRENT_TRACE_ID.reset(token)
if token_name is not None:
CURRENT_NODE_NAME.reset(token_name)
def trace_langchain_tool(tool: BaseTool, ctx: ArtemisContext) -> BaseTool:
"""Wraps a LangChain tool to trace its execution.
+1 -1
View File
@@ -70,7 +70,7 @@ def _escape_for_adb_text(s: str) -> str:
class AndroidAdbDriver(BaseDeviceDriver):
"""Production-grade Android Device Driver powered by ADB and UIAutomator2."""
"""Android device driver using ADB and UIAutomator2."""
def __init__(
self,
+162 -14
View File
@@ -135,6 +135,10 @@ class TestSummary(BaseModel):
inconclusive: int = 0
unchecked: int = 0
failed_items: list[dict] = Field(default_factory=list)
#: Check lines the user retired mid-run (dropped from the plan under user
#: guidance): their earlier verdicts are reported here, never as failures.
retired: int = 0
retired_items: list[dict] = Field(default_factory=list)
@property
def total(self) -> int:
@@ -506,6 +510,112 @@ def _finding_registry(ctx) -> dict:
return registry
def _finding_items(ctx) -> dict:
"""checkpoint_id -> ``{(kind, item_text), ...}`` of the failed verify items
that produced the headline currently held in :func:`_finding_registry`.
Sibling map of the registry (same lifetime, same keys) so a headline can
be retired when user guidance withdraws every item behind it.
"""
items = getattr(ctx, "checker_finding_items", None)
if not isinstance(items, dict):
items = {}
try:
ctx.checker_finding_items = items
except (AttributeError, TypeError, ValueError):
pass
return items
def _register_finding(ctx, checkpoint_id: str, verify_failures: list) -> None:
_finding_registry(ctx)[checkpoint_id] = _render_finding_headline(checkpoint_id, verify_failures)
_finding_items(ctx)[checkpoint_id] = {
(str(getattr(v, "kind", "") or ""), str(getattr(v, "item_text", "") or ""))
for v in verify_failures
}
def _clear_finding(ctx, checkpoint_id: str) -> bool:
"""Drops a checkpoint's standing headline; True when one was registered."""
_finding_items(ctx).pop(checkpoint_id, None)
return _finding_registry(ctx).pop(checkpoint_id, None) is not None
def _retired_checks(ctx) -> set[tuple[str, str]]:
retired = getattr(ctx, "guidance_retired_checks", None)
return retired if isinstance(retired, set) else set()
def retire_check_items(ctx, items) -> None:
"""Records ``(kind, text)`` check items the Operator dropped under user
guidance and withdraws their already-materialized findings.
Recording feeds the harvest gate (:func:`_active_verdicts`) and the run
outcome. Cleanup: every standing headline produced *only* by now-retired
items is popped and the plan's finding lines are re-projected; a headline
that still names a live item stays untouched.
"""
incoming = {(str(kind), str(text)) for kind, text in items}
if not incoming:
return
retired = _retired_checks(ctx) | incoming
ctx.guidance_retired_checks = retired
registry = _finding_registry(ctx)
produced_by = _finding_items(ctx)
stale = [cid for cid in list(registry) if produced_by.get(cid) and produced_by[cid] <= retired]
for cid in stale:
_clear_finding(ctx, cid)
logger.info(
f"Checkpoint {cid}: standing finding headline withdrawn — every item"
" behind it was retired by user guidance."
)
if stale:
sync_finding_lines(ctx)
def reinstate_check_items(ctx, items) -> None:
"""Removes ``(kind, text)`` items from the retired set: the Operator has
declared them in the plan again, so their verdicts drive side effects
once more and the run outcome counts them normally. A no-op when none of
the given items is retired."""
retired = _retired_checks(ctx)
if not retired:
return
back = {(str(kind), str(text)) for kind, text in items} & retired
if not back:
return
ctx.guidance_retired_checks = retired - back
logger.info(
"Check line(s) retired under user guidance are declared again and"
f" reinstated: {sorted(back)}"
)
def _active_verdicts(ctx, checkpoint, verdicts: list) -> list:
"""The verdicts still allowed to drive side effects: those whose
``(kind, item_text)`` was not retired by user guidance. Ledger recording is
never filtered — this gate only protects execution state."""
retired = _retired_checks(ctx)
if not retired:
return list(verdicts)
active: list = []
dropped: list = []
for v in verdicts:
key = (str(getattr(v, "kind", "") or ""), str(getattr(v, "item_text", "") or ""))
(dropped if key in retired else active).append(v)
if dropped:
logger.info(
f"Checkpoint {checkpoint.checkpoint_id}: {len(dropped)} verdict(s)"
" recorded but dropped from side effects — their check items were"
" retired by user guidance: "
+ ", ".join(
f"[{k}] '{t}'"
for k, t in ((getattr(v, "kind", ""), getattr(v, "item_text", "")) for v in dropped)
)
)
return active
def _render_finding_headline(checkpoint_id: str, verify_failures: list) -> str:
first = verify_failures[0]
evidence = str(getattr(first, "evidence", "") or "").strip()
@@ -778,16 +888,26 @@ def harvest_run(
# Layer 3: the per-checkpoint repair log accumulates every booked attempt.
_log_checker_note(ctx, run, verdicts)
# Guidance gate: a verdict for a check item the user retired mid-run (the
# Operator dropped its line under a guidance waiver) is a result for a
# withdrawn requirement — booked above, never a repair/halt trigger.
active = _active_verdicts(ctx, checkpoint, verdicts)
all_retired = bool(verdicts) and not active
# Applicability gate for side effects: the verdict must belong to the
# current attempt of its checkpoint AND the anchored subgoal's text must be
# unchanged in the current plan. Stale/mismatched verdicts are ledger-only.
applicable = allow_side_effects and _subgoal_still_current(ctx, checkpoint.checkpoint_id)
# unchanged in the current plan AND at least one verdict must still be
# live. Stale/mismatched/fully retired verdicts are ledger-only.
applicable = (
allow_side_effects
and not all_retired
and _subgoal_still_current(ctx, checkpoint.checkpoint_id)
)
user_stopped = bool(state is not None and getattr(state, "user_stop_requested", False))
verify_failures = [v for v in verdicts if v.kind == "verify" and v.status == "failed"]
assert_failures = [v for v in verdicts if v.kind == "assert" and v.status == "failed"]
verify_failures = [v for v in active if v.kind == "verify" and v.status == "failed"]
assert_failures = [v for v in active if v.kind == "assert" and v.status == "failed"]
registry = _finding_registry(ctx)
registry_changed = False
reverted = False
@@ -799,9 +919,7 @@ def harvest_run(
reverted = revert_subgoal_status(ctx, checkpoint.checkpoint_id)
# Layer 2: register the standing plan headline for this unresolved
# finding; the projection below renders it under the subgoal.
registry[checkpoint.checkpoint_id] = _render_finding_headline(
checkpoint.checkpoint_id, verify_failures
)
_register_finding(ctx, checkpoint.checkpoint_id, verify_failures)
registry_changed = True
for v in verify_failures:
suggestion = getattr(v, "suggestion", "") or ""
@@ -820,16 +938,18 @@ def harvest_run(
# Quota exhausted: the run is settled for this checkpoint — the
# standing headline is retired (the ledger and the checker note
# keep the full record).
registry_changed = registry.pop(checkpoint.checkpoint_id, None) is not None
registry_changed = _clear_finding(ctx, checkpoint.checkpoint_id)
logger.warning(
f"Checkpoint {checkpoint.checkpoint_id} exhausted its repair"
f" quota ({max_repairs}); verdict stays failed, no further"
" repair loop."
)
elif not verify_failures:
elif not verify_failures and not all_retired:
# A booked attempt without failed verify criteria resolves (or
# releases) the checkpoint: its standing headline is retired.
registry_changed = registry.pop(checkpoint.checkpoint_id, None) is not None
# releases) the checkpoint: its standing headline is retired. (A fully
# retired report says nothing about the checkpoint — its headline, if
# any, was already handled by ``retire_check_items``.)
registry_changed = _clear_finding(ctx, checkpoint.checkpoint_id)
if registry_changed:
sync_finding_lines(ctx)
@@ -842,11 +962,21 @@ def harvest_run(
"Assert failure under 'halt' policy: latching halt flag for exit settlement."
)
# Event payload: ``verdicts`` carries every booked verdict (ledger view);
# ``applicable`` is False when nothing could take effect — including a
# report whose check items were all retired by user guidance; ``retired``
# lists the ``(kind, item_text)`` pairs of the verdicts dropped by that
# guidance gate (empty when none were).
_event(
"done",
verdict_payloads(verdicts),
applicable=bool(applicable),
reverted=bool(reverted),
retired=[
{"kind": p["kind"], "item_text": p["item_text"]}
for p in verdict_payloads(verdicts)
if (p["kind"], p["item_text"]) in _retired_checks(ctx)
],
repairs_used=int(ctx.checkpoint_repairs.get(checkpoint.checkpoint_id, 0)),
)
return findings
@@ -1006,8 +1136,20 @@ def resolve_item_status(kind: str, records: list[dict]) -> str:
return "unchecked"
def compute_test_summary(check_items: list[CheckItem], records: list[dict]) -> TestSummary:
def compute_test_summary(
check_items: list[CheckItem],
records: list[dict],
retired: set[tuple[str, str]] | None = None,
) -> TestSummary:
"""Folds the verdict ledger into pass/fail counts over the declared items.
``retired`` names check lines the Operator dropped under user guidance; a
line that is retired and no longer declared in the plan is reported under
``retired`` with its last recorded verdict, not as a pass or a failure —
the user withdrew that requirement.
"""
summary = TestSummary()
retired = retired or set()
by_item: dict[tuple[str, str], list[dict]] = {}
for r in records:
by_item.setdefault((str(r.get("kind")), str(r.get("item_text"))), []).append(r)
@@ -1026,8 +1168,13 @@ def compute_test_summary(check_items: list[CheckItem], records: list[dict]) -> T
seen.add(sig)
all_items.append(sig)
declared = {(ci.kind, ci.text) for ci in check_items}
for kind, text in all_items:
status = resolve_item_status(kind, by_item.get((kind, text), []))
if (kind, text) in retired and (kind, text) not in declared:
summary.retired += 1
summary.retired_items.append({"item_text": text, "kind": kind, "last_status": status})
continue
if status == "passed":
summary.passed += 1
elif status == "failed":
@@ -1052,6 +1199,7 @@ def compute_run_outcome(
records: list[dict],
*,
verify_blocked: bool,
retired: set[tuple[str, str]] | None = None,
) -> RunOutcome:
"""Assembles the machine-readable run outcome.
@@ -1060,7 +1208,7 @@ def compute_run_outcome(
completion. Assert failures never change ``task_status``: a finished task
with failed assertions is ``completed`` with ``tests.failed > 0``.
"""
tests = compute_test_summary(list(snapshot.all_check_items), records)
tests = compute_test_summary(list(snapshot.all_check_items), records, retired=retired)
if verify_blocked:
task_status: Literal["completed", "partial", "blocked"] = "blocked"
elif snapshot.all_top_level_done:
+66 -12
View File
@@ -16,6 +16,7 @@ import asyncio
import base64
import functools
import json
from collections import Counter
from typing import Annotated, Literal
from langchain_core.messages import ToolMessage
@@ -49,6 +50,8 @@ from artemis.graph.checkpoints import (
publish_checker_event,
queue_checkpoints,
read_ledger,
reinstate_check_items,
retire_check_items,
revert_subgoal_status,
settle_all_checkpoints,
spawn_pending_checkpoints,
@@ -87,8 +90,10 @@ from artemis.utils.notes import (
)
from artemis.utils.plan_grammar import (
milestones_changed,
missing_check_items,
new_top_level_completions,
parse_plan,
render_check_line,
restore_missing_check_items,
subgoal_hash,
unintended_milestone_edits,
@@ -432,7 +437,13 @@ async def exit_settlement_node(state: State, ctx: ArtemisContext):
# END path: assemble the machine-readable run outcome.
records = read_ledger(base_dir)
outcome = compute_run_outcome(snapshot, records, verify_blocked=verify_blocked)
retired = getattr(ctx, "guidance_retired_checks", None)
outcome = compute_run_outcome(
snapshot,
records,
verify_blocked=verify_blocked,
retired=retired if isinstance(retired, set) else None,
)
if snapshot.all_check_items or records or verify_blocked:
update["run_outcome"] = outcome.model_dump()
# BLOCKED/partial wrap-ups carry the last findings in the metadata file.
@@ -451,7 +462,8 @@ async def exit_settlement_node(state: State, ctx: ArtemisContext):
f"Run outcome: task_status={outcome.task_status},"
f" tests(passed={outcome.tests.passed}, failed={outcome.tests.failed},"
f" inconclusive={outcome.tests.inconclusive},"
f" unchecked={outcome.tests.unchecked})"
f" unchecked={outcome.tests.unchecked},"
f" retired={outcome.tests.retired})"
)
return update
@@ -634,17 +646,59 @@ async def _process_plan_write(
# deleted/rewritten check lines are merged back in a pure text operation.
# This is content-driven and independent of any switch — a resumed plan
# with check lines stays protected even when checking is disabled.
merged = restore_missing_check_items(content_before, content_after)
if merged is not None:
# The one exception is user guidance: it outranks the declared standards,
# so every check line that existed when an instruction arrived (recorded
# by perception in ``ctx.guidance_unprotected_checks``) may be dropped or
# reworded at any later write — the Operator usually re-plans over several
# surgical edits, turns after the guidance landed. Lines added after the
# instruction are protected again until the next one.
#
# ``missing_check_items`` diffs on the full signature (kind, when, text,
# parent hash), so rewording a milestone re-signs every check line under
# it. Such a line has merely moved: it is still declared, so it is neither
# merged back (that would duplicate it as a task-level ``@end`` orphan)
# nor retired (the ledger, the harvest gate and the run outcome all key a
# check item by ``(kind, text)``). Only a line whose ``(kind, text)`` is
# gone from the new plan counts as dropped.
before_snapshot = parse_plan(content_before)
after_snapshot = parse_plan(content_after)
missing = missing_check_items(before_snapshot, after_snapshot)
unprotected = getattr(ctx, "guidance_unprotected_checks", None)
if not isinstance(unprotected, set):
unprotected = set()
still_declared = Counter((ci.kind, ci.text) for ci in after_snapshot.check_items)
dropped: list = []
for ci in missing:
if still_declared.get((ci.kind, ci.text), 0) > 0:
still_declared[(ci.kind, ci.text)] -= 1
continue # moved under a reworded milestone / re-timed: still declared
dropped.append(ci)
waived = [ci for ci in dropped if (ci.kind, ci.text) in unprotected]
protected = [ci for ci in dropped if (ci.kind, ci.text) not in unprotected]
if waived:
logger.info(
"Plan write removed or rewrote declared check lines; merging them"
" back (check standards are protected deterministically)."
"Plan write dropped declared check lines because of user guidance"
f" (the change stands): {[render_check_line(ci).strip() for ci in waived]}"
)
content_after = merged
try:
task_plan_path.write_text(content_after, encoding="utf-8")
except Exception as e:
logger.error(f"Failed to write merged check lines: {e}")
# Records the retirement (harvest gate + run outcome) and withdraws any
# standing finding headline those lines had already produced.
retire_check_items(ctx, [(ci.kind, ci.text) for ci in waived])
# A retired line the Operator declares again is live again: the harvest
# gate and the run outcome must agree with the plan on disk.
reinstate_check_items(ctx, [(ci.kind, ci.text) for ci in after_snapshot.check_items])
if protected:
merged = restore_missing_check_items(content_before, content_after, items=protected)
if merged is not None:
logger.info(
"Plan write removed or rewrote declared check lines; merging them"
" back (check standards are protected deterministically):"
f" {[render_check_line(ci).strip() for ci in protected]}"
)
content_after = merged
try:
task_plan_path.write_text(content_after, encoding="utf-8")
except Exception as e:
logger.error(f"Failed to write merged check lines: {e}")
# Deterministic finding-line projection: unresolved verify findings are
# re-rendered from the checkpoint repair state on every plan write, so a
@@ -653,7 +707,7 @@ async def _process_plan_write(
# never influences the parses below.
sync_finding_lines(ctx)
before = parse_plan(content_before)
before = before_snapshot
after = parse_plan(content_after)
# Ratchet baseline: judge the current plan against the last *validated*
+31
View File
@@ -92,6 +92,29 @@ def _check_injected_instruction_file(base_dir: str) -> dict | None:
return None
def _unprotect_current_check_lines(ctx: ArtemisContext) -> None:
"""Record the plan's current check lines as editable under user guidance."""
from artemis.utils.notes import get_note_file_path
from artemis.utils.plan_grammar import parse_plan
unprotected = getattr(ctx, "guidance_unprotected_checks", None)
if not isinstance(unprotected, set):
unprotected = set()
try:
plan_path = get_note_file_path(ctx.data_engine.base_dir, "task_plan")
text = plan_path.read_text(encoding="utf-8") if plan_path.exists() else ""
except Exception as e:
logger.warning(f"Could not read task_plan for the guidance check-line waiver: {e}")
text = ""
added = {(ci.kind, ci.text) for ci in parse_plan(text).check_items}
unprotected |= added
ctx.guidance_unprotected_checks = unprotected
logger.info(
f"User guidance received: {len(added)} declared check line(s) may now be edited"
" by the Operator."
)
async def perception_node(state: State, ctx: ArtemisContext) -> dict:
"""Perception node (fast path): captures screenshot, extracts XML hierarchy, and performs OCR.
@@ -114,6 +137,14 @@ async def perception_node(state: State, ctx: ArtemisContext) -> dict:
if injected_payload:
injected_instruction = injected_payload.get("instruction")
user_stop_requested = bool(injected_payload.get("release_loop"))
if isinstance(injected_instruction, str) and injected_instruction.strip():
# User guidance outranks the plan's declared standards: every
# check line that exists right now loses its machine
# restoration for the rest of the run (kept on the run context,
# not the per-turn State), so the Operator can drop or reword
# it in whichever later write it gets to it. Lines added after
# this moment stay protected until the next instruction.
_unprotect_current_check_lines(ctx)
controller = UnifiedMobileController(ctx)
+1 -1
View File
@@ -116,7 +116,7 @@ NODE_VISIBILITY: dict[str, NodeVisibility] = {
"operator_native_thinking",
"user_stop_requested",
"initial_goal",
# M3: the turn's injected instruction is stamped verbatim onto the
# The turn's injected instruction is stamped verbatim onto the
# step record so the chunk ledger can preserve it (never-evict).
"injected_instruction",
},
+332 -306
View File
@@ -12,321 +12,155 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""System, environment, and device diagnostics (artemis doctor)."""
"""System, environment, and device diagnostics (artemis doctor).
The command renders the same readiness probes the web console's device wizard
and the ``mobile_diagnose`` MCP tool run (:mod:`artemis.core.diagnostics`), so
the terminal, the browser and the IDE agree on one verdict. Only two rows are
CLI-specific and not engine probes: Node.js/npm and the Showcase UI build.
"""
from __future__ import annotations
import asyncio
from dataclasses import dataclass
import json
import platform as sys_platform
import shutil
import subprocess
import sys
from typing import Any
from rich.console import Console
from rich.markup import escape
from rich.panel import Panel
from rich.table import Table
import typer
from artemis.config import (
parse_llm_config,
settings,
)
from artemis.config.paths import ROOT_DIR
from artemis.core.diagnostics.adb_keys import heal_adb_keys, inspect_adb_keys
from artemis.core.diagnostics.engine import readiness_engine
from artemis.core.diagnostics.readiness import (
Verdict,
adb_keys_corrupted,
base_verdict,
collect_readiness,
sort_by_fix_order,
)
from artemis.core.diagnostics.schema import ProbeResult, ProbeStatus
from artemis.platform import OSType, platform
from artemis.toolchain import toolchain
from artemis.utils.ocr_api import is_ocr_configured
from artemis.runtime.device_lock import DeviceExecutionLock
def doctor_command(
fix: bool = typer.Option(
False,
"--fix",
"-f",
help="Automatically repair fixable environment issues (such as corrupted ADB authentication keys).",
),
) -> None:
"""Run diagnostics to inspect system dependencies, device connectivity, and configuration."""
console = Console()
console.print()
os_name = sys_platform.system()
arch_name = sys_platform.machine()
console.print(
Panel(
f"[bold cyan]☕ Artemis System & Environment Doctor[/bold cyan]\n"
f"[dim]Platform: {os_name} ({arch_name}) | PAL: {platform.os_type.value}[/dim]",
expand=False,
@dataclass(frozen=True)
class ExtraRow:
"""A CLI-only diagnostic row that is not backed by an engine probe."""
key: str
title: str
status: str # "pass" | "missing"
status_markup: str
summary: str
detail: str
@dataclass(frozen=True)
class FixOutcome:
"""Result of one ``--fix`` repair step."""
fix: str
success: bool
message: str
# ---------------------------------------------------------------------------
# Collection
# ---------------------------------------------------------------------------
async def _collect() -> list[ProbeResult]:
report, host = await collect_readiness()
return sort_by_fix_order([*report.probes, host])
async def _apply_fixes(results: list[ProbeResult]) -> list[FixOutcome]:
"""Repair corrupted ADB keys and sweep stale device locks left by dead runners."""
outcomes: list[FixOutcome] = []
if adb_keys_corrupted(results):
result = await readiness_engine.heal_adb_keys()
outcomes.append(
FixOutcome(
fix="heal_adb_keys",
success=bool(result.get("success")),
message=str(result.get("message") or "")
or ("Corrupted RSA keys regenerated." if result.get("success") else "Heal failed."),
)
)
else:
outcomes.append(
FixOutcome(
fix="heal_adb_keys",
success=True,
message="ADB authentication keys are healthy; nothing to repair.",
)
)
removed = await asyncio.to_thread(DeviceExecutionLock.cleanup_stale_locks)
outcomes.append(
FixOutcome(
fix="cleanup_stale_locks",
success=True,
message=(
f"Removed {removed} stale device lock(s)/queue ticket(s)."
if removed
else "No stale device locks or queue tickets found."
),
)
)
return outcomes
table = Table(show_header=True, header_style="bold magenta")
table.add_column("Component", style="dim", width=22)
table.add_column("Status", width=12)
table.add_column("Details & Recommendations")
all_passed = True
async def _diagnose(fix: bool) -> tuple[list[ProbeResult], list[FixOutcome]]:
results = await _collect()
fixes: list[FixOutcome] = []
if fix:
fixes = await _apply_fixes(results)
results = await _collect()
return results, fixes
# 1. Python Version
py_ver = f"{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}"
if sys.version_info >= (3, 12):
table.add_row(
"Python Runtime",
"[bold green]✔ OK[/bold green]",
f"Python {py_ver} ({arch_name}, 64-bit)",
)
else:
all_passed = False
table.add_row(
"Python Runtime",
"[bold red]✖ Warning[/bold red]",
f"Python {py_ver} detected. Python >= 3.12 is strongly recommended.",
)
# 2. ADB Installation & Server
adb_path = toolchain.resolve("adb")
if adb_path:
try:
res = subprocess.run(
[adb_path, "version"],
capture_output=True,
text=True,
timeout=5,
encoding="utf-8",
errors="replace",
)
adb_info = res.stdout.splitlines()[0] if res.stdout else "Installed"
table.add_row(
"Android ADB", "[bold green]✔ OK[/bold green]", f"{adb_path} ({adb_info})"
)
except Exception:
table.add_row("Android ADB", "[bold green]✔ OK[/bold green]", f"{adb_path}")
else:
all_passed = False
hint = (
"Run: winget install Google.PlatformTools"
if platform.os_type == OSType.WINDOWS
else "Run: brew install android-platform-tools"
if platform.os_type == OSType.MACOS
else "Run: sudo apt-get install -y adb"
)
table.add_row(
"Android ADB",
"[bold red]✖ Missing[/bold red]",
f"adb not found in PATH or Android SDK. {hint}",
)
# ---------------------------------------------------------------------------
# CLI-only extra rows (not engine probes)
# ---------------------------------------------------------------------------
# 2b. ADB Authentication RSA Keys Check
key_status = inspect_adb_keys()
if key_status.is_corrupted:
if fix:
heal_result = heal_adb_keys(adb_path=adb_path)
if heal_result.get("success"):
table.add_row(
"ADB Auth Keys",
"[bold green]✔ Repaired[/bold green]",
"Corrupted RSA keys detected and automatically regenerated.",
)
else:
all_passed = False
table.add_row(
"ADB Auth Keys",
"[bold red]✖ Error[/bold red]",
f"Failed to auto-heal keys: {heal_result.get('message')}",
)
else:
all_passed = False
table.add_row(
"ADB Auth Keys",
"[bold red]✖ Corrupted[/bold red]",
f"{key_status.error_reason} Run [bold cyan]artemis doctor --fix[/bold cyan] to auto-repair.",
)
elif key_status.exists:
table.add_row(
"ADB Auth Keys",
"[bold green]✔ OK[/bold green]",
f"{key_status.key_path} ({key_status.key_size_bytes} bytes RSA)",
)
else:
table.add_row(
"ADB Auth Keys",
"[bold cyan]○ Ready[/bold cyan]",
"Keys will be generated automatically on first device connection.",
)
# 3. Connected Android Devices
devices_info: list[tuple[str, str, str]] = []
if adb_path:
try:
res = subprocess.run(
[adb_path, "devices", "-l"],
capture_output=True,
text=True,
timeout=5,
encoding="utf-8",
errors="replace",
)
lines = [line_str.strip() for line_str in res.stdout.splitlines() if line_str.strip()]
for line in lines[1:]: # skip 'List of devices attached'
parts = line.split()
if len(parts) >= 2:
serial = parts[0]
state = parts[1]
extra = " ".join(parts[2:])
devices_info.append((serial, state, extra))
except (OSError, subprocess.SubprocessError):
# adb probe failed or timed out; report as "no devices detected".
pass
if devices_info:
active_devices = [d for d in devices_info if d[1] == "device"]
if active_devices:
summary = "\n".join(
[
f"• [bold cyan]{d[0]}[/bold cyan] ({d[2] or 'Authorized'})"
for d in active_devices
]
)
table.add_row("Android Devices", "[bold green]✔ Connected[/bold green]", summary)
else:
all_passed = False
table.add_row(
"Android Devices",
"[bold yellow]⚠ Unauthorized[/bold yellow]",
"Device detected but unauthorized. Check phone screen for USB Debugging prompt.",
)
else:
table.add_row(
"Android Devices",
"[bold yellow]⚠ None Found[/bold yellow]",
"No Android device or emulator detected. Connect phone via USB or start emulator.",
)
# 4. LLM & Vision OCR Key Configuration
gemini_key = settings.get_api_key("google")
ocr_key = settings.get_api_key("ocr")
openai_key = settings.get_api_key("openai")
claude_key = settings.get_api_key("anthropic")
openrouter_key = settings.get_api_key("openrouter")
configured_keys = []
if gemini_key:
val = gemini_key.get_secret_value()
masked = f"{val[:6]}...{val[-4:]}" if len(val) > 10 else "***"
configured_keys.append(f"Gemini LLM ({masked})")
if openai_key:
configured_keys.append("OpenAI")
if claude_key:
configured_keys.append("Anthropic")
if openrouter_key:
configured_keys.append("OpenRouter")
if configured_keys:
table.add_row(
"LLM Providers", "[bold green]✔ Configured[/bold green]", ", ".join(configured_keys)
)
else:
all_passed = False
table.add_row(
"LLM Providers",
"[bold red]✖ Missing[/bold red]",
"No LLM key found in .env. Run [bold cyan]artemis init[/bold cyan] to configure.",
)
# 5. Vision OCR Key Check (Optional)
if is_ocr_configured() and ocr_key:
val = ocr_key.get_secret_value()
masked = f"{val[:6]}...{val[-4:]}" if len(val) > 10 else "***"
table.add_row(
"Vision OCR",
"[bold green]✔ Configured[/bold green]",
f"Google Cloud Vision API ({masked})",
)
else:
table.add_row(
"Vision OCR",
"[bold cyan]○ Optional[/bold cyan]",
"OCR_API_KEY is not set. Perception operates in pure XML layout mode.",
)
# 6. Configuration File Health
try:
cfg = parse_llm_config()
planner_model = cfg.planner.model
table.add_row(
"Configuration",
"[bold green]✔ Valid[/bold green]",
f"config/artemis.jsonc parsed (Planner: {planner_model})",
)
except Exception as e:
all_passed = False
table.add_row(
"Configuration",
"[bold red]✖ Error[/bold red]",
f"Failed to parse config/artemis.jsonc: {e}",
)
# 7. Video Recording Tools (FFmpeg & scrcpy)
ffmpeg_path = toolchain.resolve("ffmpeg")
scrcpy_path = toolchain.resolve("scrcpy")
if ffmpeg_path:
table.add_row(
"FFmpeg Tools", "[bold green]✔ Installed[/bold green]", f"{ffmpeg_path} (Video ready)"
)
else:
hint = (
"winget install Gyan.FFmpeg"
if platform.os_type == OSType.WINDOWS
else "brew install ffmpeg"
if platform.os_type == OSType.MACOS
else "sudo apt-get install -y ffmpeg"
)
table.add_row(
"FFmpeg Tools",
"[dim]⚪ Optional[/dim]",
f"Not found. ({hint})",
)
if scrcpy_path:
table.add_row(
"Screen Recorder",
"[bold green]✔ Installed[/bold green]",
f"{scrcpy_path} (Live stream ready)",
)
else:
hint = (
"winget install Genymobile.scrcpy"
if platform.os_type == OSType.WINDOWS
else "brew install scrcpy"
if platform.os_type == OSType.MACOS
else "sudo apt-get install -y scrcpy"
)
table.add_row(
"Screen Recorder",
"[dim]⚪ Optional[/dim]",
f"scrcpy not found. ({hint})",
)
# 8. Node.js & npm (for UI compilation)
def _npm_row() -> ExtraRow:
npm_path = shutil.which("npm")
if npm_path:
table.add_row(
"Node.js / npm",
"[bold green]✔ Installed[/bold green]",
f"{npm_path} (Ready to build Showcase UI)",
)
else:
hint = (
"winget install OpenJS.NodeJS.LTS"
if platform.os_type == OSType.WINDOWS
else "brew install node"
if platform.os_type == OSType.MACOS
else "sudo apt-get install -y nodejs npm"
)
table.add_row(
"Node.js / npm",
"[dim]⚪ Optional[/dim]",
f"Not found in PATH. ({hint})",
return ExtraRow(
key="nodejs_npm",
title="Node.js / npm",
status="pass",
status_markup="[bold green]✔ Installed[/bold green]",
summary="Installed",
detail=f"{npm_path} (Ready to build Showcase UI)",
)
hint = (
"winget install OpenJS.NodeJS.LTS"
if platform.os_type == OSType.WINDOWS
else "brew install node"
if platform.os_type == OSType.MACOS
else "sudo apt-get install -y nodejs npm"
)
return ExtraRow(
key="nodejs_npm",
title="Node.js / npm",
status="missing",
status_markup="[dim]⚪ Optional[/dim]",
summary="Not Found",
detail=f"Not found in PATH. ({hint})",
)
# 9. Showcase UI Build Status
def _showcase_row() -> ExtraRow:
from artemis.resources import get_bundled_showcase_dist
base_dist = ROOT_DIR / "apps" / "showcase_ui" / "dist"
@@ -342,22 +176,98 @@ def doctor_command(
if bundled_showcase is not None:
found_showcase = bundled_showcase / "index.html"
if found_showcase:
table.add_row(
"Showcase UI",
"[bold green]✔ Compiled[/bold green]",
f"{found_showcase.parent} (Angular ready)",
)
else:
table.add_row(
"Showcase UI",
"[bold yellow]○ Not Compiled[/bold yellow]",
"Run ./start.sh or artemis ui to auto-compile.",
return ExtraRow(
key="showcase_ui",
title="Showcase UI",
status="pass",
status_markup="[bold green]✔ Compiled[/bold green]",
summary="Compiled",
detail=f"{found_showcase.parent} (Angular ready)",
)
return ExtraRow(
key="showcase_ui",
title="Showcase UI",
status="missing",
status_markup="[bold yellow]○ Not Compiled[/bold yellow]",
summary="Not Compiled",
detail="Run ./start.sh or artemis ui to auto-compile.",
)
console.print(table)
# ---------------------------------------------------------------------------
# Rendering
# ---------------------------------------------------------------------------
def _status_markup(result: ProbeResult) -> str:
status = result.status
if status is ProbeStatus.PASS:
return "[bold green]✔ OK[/bold green]"
if status is ProbeStatus.SKIPPED:
return "[dim]○ Skipped[/dim]"
summary = escape(result.summary)
if status is ProbeStatus.WARN:
# A warning keeps its summary whether or not it blocks (the host
# probe's warnings degrade the verdict without blocking it).
return f"[bold yellow]⚠ {summary}[/bold yellow]"
if not result.is_blocker:
return "[dim]⚪ Optional[/dim]"
return f"[bold red]✖ {summary}[/bold red]"
def _action_lines(result: ProbeResult) -> list[str]:
"""One plain-text line per remediation; ``&&`` chains split for PowerShell 5.1."""
lines: list[str] = []
for action in result.actions:
payload = action.payload.strip()
if not payload:
continue
if action.action_type == "command":
for part in payload.split("&&"):
if part.strip():
lines.append(f"Run: {part.strip()}")
else:
lines.append(payload)
return lines
def _details_markup(result: ProbeResult) -> str:
# Only summary/description/actions are ever rendered. Probe metadata can
# carry raw credentials (the console prefills its settings form from it)
# and must never reach a terminal or a log.
parts = [escape(result.description)]
if result.status is not ProbeStatus.PASS:
for line in _action_lines(result):
if line.startswith("Run: "):
parts.append(f"[bold cyan]Run:[/bold cyan] {escape(line[len('Run: ') :])}")
else:
parts.append(escape(line))
return "\n".join(parts)
def _render_table(results: list[ProbeResult], extras: list[ExtraRow]) -> Table:
table = Table(show_header=True, header_style="bold magenta")
table.add_column("Component", style="dim", min_width=22, max_width=30)
table.add_column("Status", min_width=12, max_width=28)
table.add_column("Details & Recommendations", overflow="fold")
for result in results:
table.add_row(escape(result.title), _status_markup(result), _details_markup(result))
for extra in extras:
table.add_row(extra.title, extra.status_markup, escape(extra.detail))
return table
def _render_fixes(console: Console, fixes: list[FixOutcome]) -> None:
lines = []
for outcome in fixes:
mark = "[bold green]✔[/bold green]" if outcome.success else "[bold red]✖[/bold red]"
lines.append(f"{mark} {escape(outcome.fix)}: {escape(outcome.message)}")
console.print(Panel("\n".join(lines), title="Repairs (--fix)", expand=False))
console.print()
if all_passed:
def _render_footer(console: Console, verdict: Verdict, results: list[ProbeResult]) -> None:
if verdict == "ready":
console.print(
Panel(
"[bold green]🎉 All system checks passed![/bold green]\n\n"
@@ -367,12 +277,128 @@ def doctor_command(
expand=False,
)
)
else:
return
if verdict == "degraded":
console.print(
Panel(
"[bold yellow]💡 Tip:[/bold yellow] Run [bold cyan]artemis init[/bold cyan] to quickly set up your API key and device.",
title="Action Required",
"[bold green]Required checks passed.[/bold green] Optional components are "
"missing; tasks can run, but some features (video replay, OCR) are off.\n\n"
"Run your first automation task now:\n"
' [bold cyan]artemis run "Open Settings and check Battery level"[/bold cyan]',
title="Status: Ready (optional gaps)",
expand=False,
)
)
console.print()
return
failing = [r.title for r in results if r.is_blocker and r.status is not ProbeStatus.PASS]
tips = [
"[bold yellow]💡 Blocking issues:[/bold yellow] " + escape(", ".join(failing)),
"",
"• API keys & device setup: run [bold cyan]artemis init[/bold cyan].",
"• Follow the [bold cyan]Run:[/bold cyan] lines above, then re-run "
"[bold cyan]artemis doctor[/bold cyan].",
"• Using an IDE (Claude Code, Cursor, ...)? Ask it to call "
"[bold cyan]mobile_diagnose[/bold cyan] - it runs these same checks and can "
"apply safe fixes from the assistant.",
]
if adb_keys_corrupted(results):
tips.append(
"• Corrupted ADB keys detected: run [bold cyan]artemis doctor --fix[/bold cyan]."
)
console.print(Panel("\n".join(tips), title="Action Required", expand=False))
def _json_document(
results: list[ProbeResult],
extras: list[ExtraRow],
fixes: list[FixOutcome],
verdict: Verdict,
) -> dict[str, Any]:
# Metadata is intentionally omitted: it may hold raw API keys.
checks = [
{
"id": r.id,
"title": r.title,
"status": r.status.value,
"required": r.is_blocker,
"summary": r.summary,
"detail": r.description,
"fix": []
if r.status is ProbeStatus.PASS
else [
{"type": a.action_type, "label": a.label, "payload": a.payload} for a in r.actions
],
}
for r in results
]
return {
"verdict": verdict,
"checks": checks,
"extras": {
e.key: {
"title": e.title,
"status": e.status,
"summary": e.summary,
"detail": e.detail,
}
for e in extras
},
"fixes": [{"fix": f.fix, "success": f.success, "message": f.message} for f in fixes],
}
# ---------------------------------------------------------------------------
# Command
# ---------------------------------------------------------------------------
def doctor_command(
fix: bool = typer.Option(
False,
"--fix",
"-f",
help=(
"Automatically repair fixable environment issues (corrupted ADB authentication "
"keys, stale device locks left by crashed runners)."
),
),
json_output: bool = typer.Option(
False,
"--json",
help="Print the diagnosis as a JSON document instead of a table.",
),
) -> None:
"""Run diagnostics to inspect system dependencies, device connectivity, and configuration."""
results, fixes = asyncio.run(_diagnose(fix))
extras = [_npm_row(), _showcase_row()]
verdict = base_verdict(results)
if json_output:
typer.echo(
json.dumps(
_json_document(results, extras, fixes, verdict), indent=2, ensure_ascii=False
)
)
else:
console = Console()
console.print()
os_name = sys_platform.system()
arch_name = sys_platform.machine()
console.print(
Panel(
f"[bold cyan]☕ Artemis System & Environment Doctor[/bold cyan]\n"
f"[dim]Platform: {os_name} ({arch_name}) | PAL: {platform.os_type.value}[/dim]",
expand=False,
)
)
if fixes:
_render_fixes(console, fixes)
console.print(_render_table(results, extras))
console.print()
_render_footer(console, verdict, results)
console.print()
if verdict == "blocked":
raise typer.Exit(code=1)
+1
View File
@@ -41,6 +41,7 @@ ARTEMIS_MCP_TOOLS = (
"mobile_manage_task",
"mobile_get_device_state",
"mobile_inspect_trace",
"mobile_diagnose",
)
+182 -35
View File
@@ -15,8 +15,16 @@
"""MCP-backed action executor for the FlashRunner.
Device actions use the in-process MCP session. Argument normalization,
coordinate descriptions, smart swipes, post-action observations, state updates,
tracing, and helper tools are handled here on the agent side.
element-index resolution, coordinate descriptions, smart swipes, post-action
observations, state updates, tracing, and helper tools are handled here on the
agent side.
Flash binds the same agent dialect as the Pro Operator: a ``click``,
``long_press`` or ``input_text`` target is either an element index into the
indexed ``--- Visible UI Elements ---`` list (resolved here to the element's
center, its observed text/bounds/id recorded) or a normalized ``[x, y]`` pair
carrying the model's own ``target_description``. The wire only ever sees
coordinates.
Device action status comes from ``ActionResult.ok``; helper tools report failures
through ``ToolFailure`` or an explicit status.
@@ -66,6 +74,9 @@ AGENT_TOOL_NAMES: frozenset[str] = (
frozenset({"read_note", "list_notes", "ask_explorer", "video_analyzer"}) | HISTORY_TOOL_NAMES
)
#: Actions whose ``target`` is a single point: an element index or an [x, y] pair.
_POINT_TARGET_ACTIONS: frozenset[str] = frozenset({"click", "long_press", "input_text"})
class _ArgError(ValueError):
"""Argument-translation failure whose message is already fully formatted."""
@@ -108,15 +119,28 @@ class McpActionExecutor:
args: dict[str, Any],
tool_call_id: str,
state: Any,
*,
index_elements: list[dict[str, Any]] | None = None,
) -> ToolExecutionResult:
"""Executes the named tool and wraps the outcome for the calling agent."""
"""Executes the named tool and wraps the outcome for the calling agent.
``index_elements`` is the element list an index target (``click(3)``)
resolves against; ``None`` means ``state.indexed_elements`` at call
time. A multi-action turn passes the list from the observation the
model decided on, so its later indices are not resolved against the
list refreshed by its earlier actions (``state.indexed_elements`` is
still refreshed after every action: it is what the next observation
shows).
"""
raw_name = name.split(":")[-1] if ":" in name else name
if raw_name in AGENT_TOOL_NAMES:
return await self._execute_agent_tool(raw_name, name, args, tool_call_id, state)
if raw_name in self.action_tool_names:
return await self._execute_device_action(raw_name, name, args, tool_call_id, state)
return await self._execute_device_action(
raw_name, name, args, tool_call_id, state, index_elements=index_elements
)
return ToolExecutionResult(
tool_call_id=tool_call_id,
@@ -134,6 +158,8 @@ class McpActionExecutor:
args: dict[str, Any],
tool_call_id: str,
state: Any,
*,
index_elements: list[dict[str, Any]] | None = None,
) -> ToolExecutionResult:
span = TraceSpan(name=raw_name, trace_type="action", ctx=self.ctx)
span.payload = {"args": args}
@@ -144,12 +170,18 @@ class McpActionExecutor:
shot_path: str | None = None
xml_list: str | None = None
target_semantics: dict[str, Any] = {}
target_coordinates: list[int] | None = None
with span:
try:
session = await self._session_or_start()
wire_name, wire_args, finalize, target_semantics = self._translate(
raw_name, args, state
raw_name, args, state, index_elements=index_elements
)
if raw_name in _POINT_TARGET_ACTIONS:
# The normalized point the action was sent to: the recorded
# coordinates of an index target (the model named an index,
# not a point).
target_coordinates = wire_args.get("target")
extension = raw_name not in (REQUIRED_ACTIONS | OPTIONAL_ACTIONS)
if extension:
@@ -214,6 +246,7 @@ class McpActionExecutor:
"code": res.code.value,
"normalized_coordinates": res.normalized_coordinates,
"target_semantics": target_semantics,
"target_coordinates": target_coordinates,
},
)
@@ -232,24 +265,35 @@ class McpActionExecutor:
# --- Argument translation --------------------------------------------------------
def _translate(
self, raw_name: str, args: dict[str, Any], state: Any
self,
raw_name: str,
args: dict[str, Any],
state: Any,
*,
index_elements: list[dict[str, Any]] | None = None,
) -> tuple[str, dict[str, Any], Any, dict[str, Any]]:
"""Maps agent-facing args to wire args; returns (name, args, finalize, recorded).
``finalize(res)`` adds client-side context to swipe result messages.
``recorded`` contains validated target descriptions in the Pro Operator's
record format. Focused input and directional swipes have no coordinate
target, so they do not record a description.
``recorded`` holds the target semantics in the Pro Operator's record
format: the observed element fields of an index target, or the validated
``target_description`` of a coordinate target. Focused input and
directional swipes have no target, so they record nothing.
``index_elements`` overrides the list index targets resolve against
(see :meth:`execute`).
"""
if raw_name == "click":
target = self._require_pair(
args.get("target") or args.get("coordinates") or [], "click"
target, recorded = self._resolve_target(
args.get("target") or args.get("coordinates"),
"click",
args,
state,
index_elements=index_elements,
)
recorded = self._require_description(args, "click")
return (
"click",
{
"target": list(target),
"target": target,
"times": args.get("times", 1),
"delay_ms": args.get("delay_ms", 100),
},
@@ -258,14 +302,17 @@ class McpActionExecutor:
)
if raw_name == "long_press":
target = self._require_pair(
args.get("target") or args.get("coordinates") or [], "long press"
target, recorded = self._resolve_target(
args.get("target") or args.get("coordinates"),
"long press",
args,
state,
index_elements=index_elements,
)
recorded = self._require_description(args, "long press")
return (
"long_press",
{
"target": list(target),
"target": target,
"duration_ms": args.get("duration_ms", args.get("duration", 1000)),
},
None,
@@ -277,8 +324,9 @@ class McpActionExecutor:
target = None
recorded = {}
if raw_target:
target = list(self._require_pair(raw_target, "input text"))
recorded = self._require_description(args, "input text")
target, recorded = self._resolve_target(
raw_target, "input text", args, state, index_elements=index_elements
)
return (
"input_text",
{
@@ -304,7 +352,7 @@ class McpActionExecutor:
)
if raw_name == "swipe":
return self._translate_swipe(args, state)
return self._translate_swipe(args, state, index_elements=index_elements)
if raw_name == "press_key":
return "press_key", {"key": args.get("key", "BACK")}, None, {}
@@ -338,17 +386,101 @@ class McpActionExecutor:
# Backend extension: pass the arguments straight through.
return raw_name, dict(args), None, {}
@staticmethod
def _require_pair(raw: Any, label: str) -> tuple[int, int]:
def _screen_size(self) -> tuple[int, int]:
"""The device resolution the indexed element list was built against."""
device = getattr(self.ctx, "device", None)
width = getattr(device, "device_width", None) if device else None
height = getattr(device, "device_height", None) if device else None
return (
width if isinstance(width, int) and width > 0 else 1080,
height if isinstance(height, int) and height > 0 else 2400,
)
def _resolve_target(
self,
raw: Any,
label: str,
args: dict[str, Any],
state: Any,
*,
index_elements: list[dict[str, Any]] | None = None,
) -> tuple[list[int], dict[str, Any]]:
"""Resolves a point target to ``([x, y] normalized, recorded semantics)``.
An element index carries observed semantics (text, bounds, id, class
straight from the indexed list); a coordinate pair carries only the
model's own ``target_description``. The two never mix (Pro parity).
"""
target = normalize_coordinate_target(raw)
if not isinstance(target, (list, tuple)) or len(target) != 2:
prefix = {
"click": "Error during click",
"long press": "Error during long press",
"input text": "Error during input text",
}[label]
raise _ArgError(f"{prefix}: Invalid target format: {raw}")
return int(target[0]), int(target[1])
if isinstance(target, bool):
target = None
if isinstance(target, (int, float)):
return self._resolve_index(int(target), label, state, index_elements=index_elements)
if isinstance(target, (list, tuple)) and len(target) == 2:
recorded = self._require_description(args, label)
return [int(target[0]), int(target[1])], recorded
raise _ArgError(
f"Error during {label}: Invalid target format: {raw!r}. Use an element"
" index from the Visible UI Elements list (e.g. 3) or normalized [x, y]"
" coordinates with a target_description."
)
def _resolve_index(
self,
index: int,
label: str,
state: Any,
*,
index_elements: list[dict[str, Any]] | None = None,
) -> tuple[list[int], dict[str, Any]]:
"""Maps an element index onto the element's center (normalized 0-1000).
The recorded fields are what the Operator records for an index target,
with bounds normalized like every other Flash coordinate. The index is
resolved against ``index_elements`` when given, else against the
state's current list.
"""
elements = self._index_elements(state, index_elements)
if not 1 <= index <= len(elements):
hint = (
" The list is empty on this screen."
if not elements
else f" Active index range is 1 to {len(elements)}."
)
raise _ArgError(
f"Error during {label}: Invalid target index {index}.{hint} Use an index"
" shown in the Visible UI Elements list, or normalized [x, y]"
" coordinates with a target_description (ask_explorer can locate an"
" element that is visible but not listed)."
)
element = elements[index - 1]
width, height = self._screen_size()
def _norm_x(v: Any) -> int:
return int(max(0, min(1000, round(float(v) * 1000 / max(1, width)))))
def _norm_y(v: Any) -> int:
return int(max(0, min(1000, round(float(v) * 1000 / max(1, height)))))
center = element.get("center")
if not (isinstance(center, (list, tuple)) and len(center) == 2):
raise _ArgError(f"Error during {label}: element [{index}] has no usable center.")
bounds = element.get("bounds")
norm_bounds = None
if isinstance(bounds, (list, tuple)) and len(bounds) == 4:
norm_bounds = [
_norm_x(bounds[0]),
_norm_y(bounds[1]),
_norm_x(bounds[2]),
_norm_y(bounds[3]),
]
recorded = {
"target_text": element.get("text"),
"target_bounds": norm_bounds,
"target_resource_id": element.get("resource_id"),
"target_class": element.get("class"),
}
return [_norm_x(center[0]), _norm_y(center[1])], recorded
@staticmethod
def _require_description(args: dict[str, Any], label: str) -> dict[str, Any]:
@@ -414,11 +546,23 @@ class McpActionExecutor:
resolved.append([nx, ny])
return resolved
@staticmethod
def _index_elements(
state: Any, index_elements: list[dict[str, Any]] | None
) -> list[dict[str, Any]]:
"""The element list index targets resolve against (``None`` -> state)."""
if index_elements is not None:
return index_elements
return getattr(state, "indexed_elements", None) or []
def _translate_swipe(
self, args: dict[str, Any], state: Any
self,
args: dict[str, Any],
state: Any,
*,
index_elements: list[dict[str, Any]] | None = None,
) -> tuple[str, dict[str, Any], Any, dict[str, Any]]:
width = getattr(self.ctx.device, "device_width", 1080) if self.ctx.device else 1080
height = getattr(self.ctx.device, "device_height", 2400) if self.ctx.device else 2400
width, height = self._screen_size()
default_duration = args.get("duration", 400)
kind, target, final_duration = parse_swipe_parameters(
dict(args), default_duration=default_duration
@@ -429,7 +573,7 @@ class McpActionExecutor:
x1, y1, x2, y2, smart_dur = compute_smart_swipe_coordinates(
direction=target,
target=args.get("target"),
indexed_elements=getattr(state, "indexed_elements", None) if state else None,
indexed_elements=self._index_elements(state, index_elements) if state else None,
ui_hierarchy=getattr(state, "latest_ui_hierarchy", None) if state else None,
width=width,
height=height,
@@ -608,6 +752,9 @@ class McpActionExecutor:
self.ctx, state, query, context_feedback or "", agent_name=self.agent_name
)
registered = register_candidates(self.ctx, state, outcome)
return render_text(query, outcome, registered), not outcome.error
# Candidates join ``state.indexed_elements``, and this executor's
# ``click`` resolves an index against that list (Pro parity), so the
# answer teaches the index syntax next to the coordinates.
return render_text(query, outcome, registered, index_targets=True), not outcome.error
except Exception as e:
return f"Error executing ask_explorer: {e}", False
+142 -342
View File
@@ -21,49 +21,46 @@ the declarations drifted apart (``click.target`` took an element index in one pl
and only coordinates in another). This module is now the single place all three are
*defined*; the historical sites import their surface from here, generated.
An action has up to three dialects, and the differences between them are **deliberate
and declared**, not accidental:
An action has one agent-facing dialect and one wire dialect:
* ``operator`` -- the LangChain shell tools the Operator binds. The Operator receives
an indexed "Visible UI Elements" list and its prompt teaches index-first targeting,
so target parameters accept ``int`` element indices as well as normalized
coordinates. The shells are declaration-only ("Action Recorded"): the Operator's
translate step lowers them into structured decisions.
* ``declaration`` -- the JSON ``ToolDeclaration`` bound by the Validator's
FlashRunner. These agents act on coordinates directly (the
element list they see carries coordinates), so target parameters are coordinate
pairs and the wording teaches the act-then-observe loop ("The screen after X will
be returned automatically").
* ``operator`` -- the agent dialect. Both profiles receive the same indexed
"Visible UI Elements" list, so target parameters accept ``int`` element indices
as well as normalized coordinates. The Pro Operator binds it as a LangChain shell
(declaration-only, "Action Recorded": the Operator's translate step lowers the
call into structured decisions); the FlashRunner binds the very same description
and parameters projected onto a JSON ``ToolDeclaration`` (:func:`tool_declaration`),
and ``McpActionExecutor`` resolves an index against the current element list.
One definition, two bindings: the two profiles can no longer drift apart.
* ``declaration`` -- a JSON-only agent dialect, used solely for actions that have
no operator shell (``click_sequence`` is declared to Flash alone).
* ``wire`` -- what the action MCP server serves, mirroring the actuator protocol
one-to-one: normalized coordinates, millisecond durations, no addressing sugar.
The client-side executor (``McpActionExecutor``) lowers the
agent dialects onto it.
The client-side executor (``McpActionExecutor``) lowers the agent dialect onto it.
A coordinate target names nothing by itself, so both agent dialects pair it with
A coordinate target names nothing by itself, so the agent dialect pairs it with
``target_description`` (``target_descriptions`` for a ``click_sequence``): the
model's own statement of what it is aiming at. It is required whenever the target
is a coordinate (always, in the declaration dialect), recorded verbatim on the
action, and never sent to the wire. An element index needs none -- its text,
bounds and id are read from the indexed list -- and the recorded fields stay
separate (``target_text``/``target_bounds``/... for observed elements,
``target_description`` for described coordinates) so no reader can mistake the
model's belief for an observation.
is a coordinate, recorded verbatim on the action, and never sent to the wire. An
element index needs none -- its text, bounds and id are read from the indexed
list -- and the recorded fields stay separate (``target_text``/``target_bounds``/...
for observed elements, ``target_description`` for described coordinates) so no
reader can mistake the model's belief for an observation.
Each ``ActionSpec.param_bridge`` maps operator parameter names onto declaration
parameter names, so every rename (``duration`` -> ``duration_ms``, ``gesture`` ->
``action``) is written down exactly once. ``tests/unit/mcp/test_action_specs.py``
re-derives the bridge from the generated schemas, so an undeclared divergence --
today's definition of drift -- fails CI instead of shipping.
``tests/unit/mcp/test_action_specs.py`` pins the generated schema of every model
surface and checks that the Flash declaration of each shared action is exactly the
operator shell's schema, so a divergence between what one profile sees and what the
other sees -- the historical definition of drift -- fails CI instead of shipping.
The classification of actions (required/optional/internal) stays in
``action_manifest``; this module holds their schemas and teaching text.
"""
from collections.abc import Awaitable, Callable
from dataclasses import dataclass, field
from dataclasses import dataclass
from functools import cache
import inspect
from typing import Annotated, Any, Literal
import types
from typing import Annotated, Any, Literal, Union, get_args, get_origin
from langchain_core.tools import StructuredTool
from mcp.types import CallToolResult
@@ -105,7 +102,11 @@ class ParamSpec:
@dataclass(frozen=True)
class OperatorDialect:
"""The Operator's LangChain shell: index-capable targets, declaration-only body."""
"""The agent dialect: index-capable targets, one wording for both profiles.
Bound by the Pro Operator as a declaration-only LangChain shell and by the
FlashRunner as the JSON projection of the same parameters.
"""
description: str
params: tuple[ParamSpec, ...]
@@ -113,7 +114,7 @@ class OperatorDialect:
@dataclass(frozen=True)
class DeclarationDialect:
"""The Validator/Flash JSON declaration, wording preserved."""
"""A JSON-only agent dialect, for actions without an operator shell."""
description: str
parameters: dict[str, Any]
@@ -139,57 +140,43 @@ class ActionSpec:
Attributes:
name: The canonical tool name every dialect shares.
operator: Operator shell dialect, or ``None`` when the Operator does not bind
this action.
declaration: Validator/Flash dialect, or ``None`` when no JSON declaration
exists (an action can be wire-only, reachable through executors but never
declared to a model directly).
operator: The agent dialect (Operator shell and, projected, the Flash
declaration), or ``None`` when neither profile binds this action as a
shell.
declaration: A JSON-only agent dialect for actions without an operator
shell, or ``None`` (an action can be wire-only, reachable through
executors but never declared to a model directly).
wire: Action-server dialect; ``None`` only for purely virtual actions.
param_bridge: operator param name -> declaration param name, for every
operator param, when both dialects exist. Identical names map to
themselves; renames are the deliberate dialect differences.
differences: Human-readable record of every deliberate cross-dialect
divergence (semantics, not just spelling). Empty means the dialects
align modulo the documented dialect philosophy above.
differences: Human-readable record of every deliberate divergence between
the agent dialect and the wire (semantics, not just spelling). Empty
means the dialects align modulo the documented dialect philosophy above.
"""
name: str
operator: OperatorDialect | None = None
declaration: DeclarationDialect | None = None
wire: WireDialect | None = None
param_bridge: dict[str, str] = field(default_factory=dict)
differences: str = ""
# --- Shared type aliases -------------------------------------------------------------
#: Operator-dialect target: element index into the indexed UI list, or a normalized
#: [x, y] pair. Only the Operator sees this union; see the module docstring.
#: Agent-dialect target: element index into the indexed UI list, or a normalized
#: [x, y] pair.
IndexOrCoords = int | list[int]
Direction = Literal["up", "down", "left", "right"]
_COORD_PAIR_JSON = {
"type": "array",
"items": {"type": "integer"},
}
#: Operator-dialect wording for the description that a coordinate target must carry.
#: An element index already names its element (its text, id and bounds are recorded
#: from the indexed list); a bare coordinate names nothing, so the model states what
#: it is aiming at and that statement is recorded as ``target_description``.
_OPERATOR_TARGET_DESCRIPTION = (
#: Wording for the description that a coordinate target must carry. An element
#: index already names its element (its text, id and bounds are recorded from the
#: indexed list); a bare coordinate names nothing, so the model states what it is
#: aiming at and that statement is recorded as ``target_description``.
_TARGET_DESCRIPTION = (
"What the target is, in a few words (e.g. 'play button', 'search input',"
" 'video body'). REQUIRED when target is a coordinate pair; ignored for an"
" element index."
)
#: Declaration-dialect wording (coordinates-only dialect: always required).
_DECLARATION_TARGET_DESCRIPTION = (
"What the target is, in a few words (e.g. 'play button', 'search input',"
" 'video body'). Required: a coordinate names nothing by itself."
)
# --- Wire bindings -------------------------------------------------------------------
# Each binding reproduces the exact conversion of the historical hand-written FastMCP
@@ -274,7 +261,7 @@ _SPECS: tuple[ActionSpec, ...] = (
ParamSpec(
"target_description",
str | None,
_OPERATOR_TARGET_DESCRIPTION,
_TARGET_DESCRIPTION,
required=False,
default=None,
),
@@ -296,39 +283,6 @@ _SPECS: tuple[ActionSpec, ...] = (
),
),
),
declaration=DeclarationDialect(
description=(
"[ACTION] Tap/click on the target coordinate on the screen. The screen"
" after click will be returned automatically. For buttons, checkboxes,"
" tabs, icons, items in a list, tap ON the element. For text fields /"
" search bars / input boxes, tap INSIDE the input box to focus it."
),
parameters={
"type": "object",
"properties": {
"target": {
"type": "array",
"items": {"type": "integer"},
"description": "Normalized coordinates [x, y] in 0-1000 scale.",
},
"target_description": {
"type": "string",
"description": _DECLARATION_TARGET_DESCRIPTION,
},
"times": {
"type": "integer",
"description": (
"Number of taps to perform (default 1). Set to 2 for double click."
),
},
"delay_ms": {
"type": "integer",
"description": "Delay between taps in milliseconds (default 100).",
},
},
"required": ["target", "target_description"],
},
),
wire=WireDialect(
description="Tap at a 0-1000 normalized [x, y] coordinate.",
params=(
@@ -338,17 +292,11 @@ _SPECS: tuple[ActionSpec, ...] = (
),
bind=_wire_click,
),
param_bridge={
"target": "target",
"target_description": "target_description",
"times": "times",
"delay_ms": "delay_ms",
},
differences=(
"target: operator accepts an element index or a coordinate pair; the"
" declaration and wire dialects take coordinates only."
" target_description: recorded, never sent to the wire; the operator"
" requires it only for coordinate targets, the declaration always."
"target: the agent dialect accepts an element index or a coordinate pair"
" (an index is resolved client-side against the indexed element list); the"
" wire takes coordinates only. target_description: recorded, never sent"
" to the wire; required only for coordinate targets."
),
),
ActionSpec(
@@ -407,6 +355,9 @@ _SPECS: tuple[ActionSpec, ...] = (
" burst instead); sequence entries are normalized coordinate pairs in both"
" dialects (an element index is refused, never resolved client-side)."
" target_descriptions is recorded per entry and never sent to the wire."
" Unlike click, a burst is not resolved against the element list: its"
" whole point is to outrun a transient UI, and the list was captured"
" before the trigger tap."
),
),
ActionSpec(
@@ -427,7 +378,7 @@ _SPECS: tuple[ActionSpec, ...] = (
ParamSpec(
"target_description",
str | None,
_OPERATOR_TARGET_DESCRIPTION,
_TARGET_DESCRIPTION,
required=False,
default=None,
),
@@ -440,31 +391,6 @@ _SPECS: tuple[ActionSpec, ...] = (
),
),
),
declaration=DeclarationDialect(
description=(
"[ACTION] Long press on a target coordinate on the screen. The screen"
" after long pressing will be returned automatically."
),
parameters={
"type": "object",
"properties": {
"target": {
"type": "array",
"items": {"type": "integer"},
"description": "Normalized coordinates [x, y] in 0-1000 scale.",
},
"target_description": {
"type": "string",
"description": _DECLARATION_TARGET_DESCRIPTION,
},
"duration_ms": {
"type": "integer",
"description": "Duration of press in milliseconds (default 1000ms).",
},
},
"required": ["target", "target_description"],
},
),
wire=WireDialect(
description="Long-press at a 0-1000 normalized [x, y] coordinate.",
params=(
@@ -473,19 +399,13 @@ _SPECS: tuple[ActionSpec, ...] = (
),
bind=_wire_long_press,
),
param_bridge={
"target": "target",
"target_description": "target_description",
"duration": "duration_ms",
},
differences=(
"target: operator accepts an element index or a coordinate pair."
" target_description: recorded, never sent to the wire; the operator"
" requires it only for coordinate targets, the declaration always."
" duration: the operator spells the duration parameter `duration` (its"
" structured decisions and recorded traces carry that key); the"
" declaration and wire dialects spell it `duration_ms`. Executors accept"
" both spellings."
"target: the agent dialect accepts an element index or a coordinate pair;"
" the wire takes coordinates only. target_description: recorded, never"
" sent to the wire; required only for coordinate targets."
" duration: the agent dialect spells the duration parameter `duration`"
" (structured decisions and recorded traces carry that key); the wire"
" spells it `duration_ms`. Executors accept both spellings."
),
),
ActionSpec(
@@ -512,7 +432,7 @@ _SPECS: tuple[ActionSpec, ...] = (
ParamSpec(
"target_description",
str | None,
_OPERATOR_TARGET_DESCRIPTION,
_TARGET_DESCRIPTION,
required=False,
default=None,
),
@@ -527,39 +447,6 @@ _SPECS: tuple[ActionSpec, ...] = (
),
),
),
declaration=DeclarationDialect(
description=(
"[ACTION] Type text into an input field on the screen. The screen after"
" typing will be returned automatically. Automatically taps inside the"
" input box at target [x, y] to focus before typing."
),
parameters={
"type": "object",
"properties": {
"text": {
"type": "string",
"description": "Text to type into the focused input field.",
},
"target": {
"type": "array",
"items": {"type": "integer"},
"description": "Coordinates [x, y] of the input box in 0-1000 scale.",
},
"target_description": {
"type": "string",
"description": _DECLARATION_TARGET_DESCRIPTION,
},
"clear_exist": {
"type": "boolean",
"description": (
"Whether to clear existing text in the input box before typing"
" (default True)."
),
},
},
"required": ["text", "target", "target_description"],
},
),
wire=WireDialect(
description=("Type text, optionally focusing a 0-1000 normalized [x, y] target first."),
params=(
@@ -569,18 +456,11 @@ _SPECS: tuple[ActionSpec, ...] = (
),
bind=_wire_input_text,
),
param_bridge={
"target": "target",
"target_description": "target_description",
"text": "text",
"clear_exist": "clear_exist",
},
differences=(
"target: operator accepts an element index or a coordinate pair, and"
" requires a target; the wire dialect allows omitting the target to type"
" into the already-focused field. target_description: recorded, never"
" sent to the wire; the operator requires it only for coordinate targets,"
" the declaration always."
"target: the agent dialect accepts an element index or a coordinate pair,"
" and requires a target; the wire allows omitting the target to type into"
" the already-focused field. target_description: recorded, never sent to"
" the wire; required only for coordinate targets."
),
),
ActionSpec(
@@ -662,70 +542,6 @@ _SPECS: tuple[ActionSpec, ...] = (
),
),
),
declaration=DeclarationDialect(
description=(
"[ACTION] Perform a swipe, drag, or slider-adjustment gesture on the screen. The screen and UI hierarchy after swipe will be returned automatically.\n\n"
"• Directional Scrolling ('direction' or 'action'): Recommended for general browsing and standard page scrolling in most scenarios. Automatically computes safe swipe vectors and adaptive duration, retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings. Supports scoping to a sub-container via 'target'. If it fails on certain custom layouts, fall back to specifying exact coordinates ('start' and 'end') directly.\n"
"• Precise Coordinate Gestures ('start', 'end' or coordinates list): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars), drag-and-drop / list reordering, or as a reliable fallback when directional scrolling fails on specific containers. Always drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), swipe fully to the extreme boundary."
),
parameters={
"type": "object",
"properties": {
"direction": {
"type": "string",
"enum": ["up", "down", "left", "right"],
"description": (
"Direction for scrolling and swiping: 'up' (drags bottom-to-top, scrolling down to reveal content below),"
" 'down' (drags top-to-bottom, scrolling up to reveal content above),"
" 'left' (drags right-to-left, scrolling right),"
" 'right' (drags left-to-right, scrolling left)."
),
},
"start": {
"type": "array",
"items": {"type": "integer"},
"description": (
"Start normalized coordinates [start_x, start_y] in 0-1000 scale."
),
},
"end": {
"type": "array",
"items": {"type": "integer"},
"description": (
"End normalized coordinates [end_x, end_y] in 0-1000 scale."
),
},
"target": {
"description": (
"Optional target element index (e.g. 2) or container bounds"
" [left, top, right, bottom] to scope the directional swipe within."
),
},
"action": {
"description": (
"Backward-compatible swipe gesture: smart direction string ('up', 'down', 'left', 'right')"
" OR precise custom coordinates [start_x, start_y, end_x, end_y] in 0-1000 scale."
),
},
"target_description": {
"type": "string",
"description": (
"What is being dragged, in a few words (e.g. 'brightness slider"
" knob'). Required for coordinate gestures ('start'/'end' or a"
" coordinates list); ignored for directional scrolling."
),
},
"duration": {
"type": "integer",
"description": (
"Optional swipe/drag duration in milliseconds (default 800). For drag-and-drop,"
" list reordering, or sliding/adjusting sliders (e.g., volume, brightness, SeekBars),"
" set duration >= 1000 (e.g. 1500). If omitted for directional swipe, duration is computed automatically."
),
},
},
},
),
wire=WireDialect(
description="Swipe between two 0-1000 normalized [x, y] points.",
params=(
@@ -735,23 +551,12 @@ _SPECS: tuple[ActionSpec, ...] = (
),
bind=_wire_swipe,
),
param_bridge={
"direction": "direction",
"start": "start",
"end": "end",
"target": "target",
"gesture": "action",
"target_description": "target_description",
"duration": "duration",
},
differences=(
"target_description: recorded, never sent to the wire; required by both"
" agent dialects for coordinate gestures only."
" gesture/action: the legacy combined direction-or-coordinates parameter is"
" spelled `gesture` in the operator dialect and `action` in the"
" declaration dialect (`action` collides with the operator's structured"
" decision verb field). Smart directional swipes exist only in the agent"
" dialects; the wire takes a resolved start/end pair, and direction"
"target_description: recorded, never sent to the wire; required for"
" coordinate gestures only. gesture: the legacy combined"
" direction-or-coordinates parameter (executors also accept the older"
" `action` spelling). Smart directional swipes exist only in the agent"
" dialect; the wire takes a resolved start/end pair, and direction"
" resolution happens client-side against the live UI tree."
),
),
@@ -770,24 +575,6 @@ _SPECS: tuple[ActionSpec, ...] = (
),
),
),
declaration=DeclarationDialect(
description=(
"[ACTION] Press a physical or virtual system button. The screen after"
" pressing will be returned automatically."
),
parameters={
"type": "object",
"properties": {
"key": {
"type": "string",
"description": (
"Standard Android system button name (ENTER, BACK, HOME, APP_SWITCH)."
),
}
},
"required": ["key"],
},
),
wire=WireDialect(
description=(
"Press a device key (home, back, enter, delete, tab, search, menu, app_switch)."
@@ -795,13 +582,11 @@ _SPECS: tuple[ActionSpec, ...] = (
params=(ParamSpec("key", str),),
bind=_wire_press_key,
),
param_bridge={"key": "key"},
differences=(
"key vocabulary: the operator dialect is a closed uppercase enum (its"
" translate step emits Android KEYCODE_* names); the declaration dialect"
" is an open string taught the same uppercase names; the wire takes the"
" actuator's bare lowercase key words. `to_canonical_call` and the"
" executors normalize between them."
"key vocabulary: the agent dialect is a closed uppercase enum (the"
" Operator's translate step emits Android KEYCODE_* names); the wire"
" takes the actuator's bare lowercase key words. `to_canonical_call` and"
" the executors normalize between them."
),
),
ActionSpec(
@@ -817,26 +602,6 @@ _SPECS: tuple[ActionSpec, ...] = (
),
),
),
declaration=DeclarationDialect(
description=(
"[ACTION] Launch or force stop a specified application. The screen"
" after launch/stop will be returned automatically."
),
parameters={
"type": "object",
"properties": {
"action": {
"type": "string",
"description": "The action type ('launch' or 'stop').",
},
"app_name": {
"type": "string",
"description": "Display name or package name of the application.",
},
},
"required": ["action", "app_name"],
},
),
wire=WireDialect(
description="Launch or stop an app by human-readable name or package.",
params=(
@@ -845,7 +610,6 @@ _SPECS: tuple[ActionSpec, ...] = (
),
bind=_wire_manage_app,
),
param_bridge={"action": "action", "app_name": "app_name"},
),
ActionSpec(
name="wait_for_delay",
@@ -867,32 +631,11 @@ _SPECS: tuple[ActionSpec, ...] = (
),
),
),
declaration=DeclarationDialect(
description=(
"[ACTION] Pause execution and wait for a specified duration in milliseconds."
" The screen after pause will be returned automatically."
),
parameters={
"type": "object",
"properties": {
"time_in_ms": {
"type": "integer",
"description": (
"The exact duration to pause in milliseconds (e.g., 2000 for 2s,"
" 60000 for 1 min, 180000 for 3 mins, 300000 for 5 mins)."
" Convert any required waiting duration into milliseconds."
),
}
},
"required": ["time_in_ms"],
},
),
wire=WireDialect(
description="Wait for a fixed number of milliseconds.",
params=(ParamSpec("time_in_ms", int),),
bind=_wire_wait_for_delay,
),
param_bridge={"time_in_ms": "time_in_ms"},
),
ActionSpec(
name="wait_for_text",
@@ -1007,16 +750,73 @@ def operator_shell_tool(name: str) -> StructuredTool:
)
_SCALAR_JSON: dict[Any, dict[str, str]] = {
bool: {"type": "boolean"},
int: {"type": "integer"},
float: {"type": "number"},
str: {"type": "string"},
}
def _json_schema(annotation: Any) -> dict[str, Any]:
"""Projects one Python-typed parameter annotation onto JSON schema.
``None`` members of a union only make the parameter optional (its
``required`` flag says so); they never appear in the projected schema, so
the declaration carries no ``null`` alternatives or ``default`` noise.
"""
origin = get_origin(annotation)
if origin in (Union, types.UnionType):
members = [a for a in get_args(annotation) if a is not type(None)]
schemas = [_json_schema(m) for m in members]
return schemas[0] if len(schemas) == 1 else {"anyOf": schemas}
if origin is Literal:
return {"type": "string", "enum": list(get_args(annotation))}
if origin is list:
(item,) = get_args(annotation) or (str,)
return {"type": "array", "items": _json_schema(item)}
try:
return dict(_SCALAR_JSON[annotation])
except KeyError:
raise TypeError(f"No JSON projection for parameter annotation {annotation!r}.") from None
def _projected_parameters(dialect: OperatorDialect) -> dict[str, Any]:
"""The operator shell's parameters as a JSON-schema ``parameters`` object."""
properties: dict[str, Any] = {}
required: list[str] = []
for p in dialect.params:
schema = _json_schema(p.annotation)
if p.description:
schema["description"] = p.description
properties[p.name] = schema
if p.required:
required.append(p.name)
return {"type": "object", "properties": properties, "required": required}
@cache
def tool_declaration(name: str) -> ToolDeclaration:
"""Builds the Validator/Flash ``ToolDeclaration`` for one action."""
dialect = ACTION_SPECS[name].declaration
if dialect is None:
raise ValueError(f"Action '{name}' has no declaration dialect.")
return ToolDeclaration(
name=name,
description=dialect.description,
parameters=dialect.parameters,
)
"""Builds the Flash ``ToolDeclaration`` for one action.
A shared action is the operator shell projected onto JSON (same description,
same parameters, same targeting semantics); a JSON-only action uses its own
``DeclarationDialect``.
"""
spec = ACTION_SPECS[name]
if spec.declaration is not None:
return ToolDeclaration(
name=name,
description=spec.declaration.description,
parameters=spec.declaration.parameters,
)
if spec.operator is not None:
return ToolDeclaration(
name=name,
description=spec.operator.description,
parameters=_projected_parameters(spec.operator),
)
raise ValueError(f"Action '{name}' has no agent dialect to declare.")
def wire_dialects() -> tuple[ActionSpec, ...]:
+321 -139
View File
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""History compression into segment chunks, eras, and a session snapshot.
"""History compression into segment chunks, eras, and recall-only periods.
Each chunk keeps three bands: synopsis and effects, interval summaries, and a
per-step action ledger. Capsule generation reads the operator's turn transcripts
@@ -20,12 +20,18 @@ alongside the recorded step facts. Milestone changes, segment size, and token
limits determine chunk boundaries.
Original messages remain in context until their capsule is ready. Failed capsule
jobs retain their source text for retry. Only the hard token limit forces pending
chunks into the session snapshot. Older chunks fold into eras and then period
summaries; full step records remain available through the history tools.
jobs retain their source text for retry. At the hard token limit, ready chunks
swap as ordinary L2 blocks first; only when that cannot bring the context back
under the line does everything closed force-swap (pending chunks included) and
the whole frozen region fold into a recall-only era — L3: the period paragraph
plus the per-step minimal index (step number, session offset, action phrase).
The fold is monotonic: later L2 swaps append new chunk blocks after the folded
eras and never restore the folded chunks. Older chunks also fold into eras and
then recall-only periods by count overflow; full step records remain available
through the history tools.
Checkpoint annotations persist independently of capsule generation. Frozen
context is rebuilt only when a capsule or snapshot replaces the source turns.
context is rebuilt only when a compression event replaces or folds turns.
"""
import asyncio
@@ -61,6 +67,29 @@ _NOTE_TOOLS = ("save_note", "update_note", "append_note")
# Reuse one tool trace per chunk across running, success, and failed states.
COMPRESSION_TRACE_NAME = "compress_history"
#: Plain-language phases carried in the trace payload (``args.phase``) so the
#: timeline can say what the compression is *doing* independently of the
#: trace ``status`` (running/success/failed), which other code keys on.
#: summarizing — the segment closed and its capsule is generating (running)
#: ready — the capsule is on hand but the start gate holds the swap (running)
#: applied — the compressed block replaced the raw turns (success)
#: failed — capsule generation gave up; the full record is kept (failed)
COMPRESSION_PHASE_SUMMARIZING = "summarizing"
COMPRESSION_PHASE_READY = "ready"
COMPRESSION_PHASE_APPLIED = "applied"
COMPRESSION_PHASE_FAILED = "failed"
COMPRESSION_PHASES = (
COMPRESSION_PHASE_SUMMARIZING,
COMPRESSION_PHASE_READY,
COMPRESSION_PHASE_APPLIED,
COMPRESSION_PHASE_FAILED,
)
_STATUS_TO_PHASE = {
"running": COMPRESSION_PHASE_SUMMARIZING,
"success": COMPRESSION_PHASE_APPLIED,
"failed": COMPRESSION_PHASE_FAILED,
}
# ---------------------------------------------------------------------------
# Band ③ — mechanical per-step action ledger
@@ -161,7 +190,7 @@ def extract_note_writes(step: dict) -> list[dict[str, str]]:
def validate_interval_coverage(intervals: Any, start_step: int, end_step: int) -> bool:
"""Band ② hard constraint: the interval union covers [start, end] seamlessly."""
"""Band ② hard constraint: the interval union covers [start, end] without gaps or overlaps."""
if not isinstance(intervals, list) or not intervals:
return False
expected = start_step
@@ -289,7 +318,7 @@ class StepCapsuleLens(StepLens):
" keys doing/did/effect/entry_state/exit_state/verified_facts/"
"unresolved/failed_paths/important_entities/intervals. Never use"
" verdict words (successfully, completed, failed, ...); intervals"
" must cover the step range seamlessly. Return only JSON."
" must cover the full step range without gaps or overlaps. Return only JSON."
)
def _get_llm(self):
@@ -396,8 +425,9 @@ class StepCapsuleLens(StepLens):
def _cap_transcript(cls, transcript: str) -> str:
"""Cap one turn transcript in place, keeping its head and its tail.
The tail is kept intact because the validator result message closes
every turn and outranks everything above it (fact priority); the cut
The tail is kept intact because the turn's last messages (the final
action's tool result, or the failure result message that closes a
failed turn) outrank everything above them (fact priority); the cut
lands in the middle and is announced with its size.
"""
cap = cls.MAX_TURN_TRANSCRIPT_CHARS
@@ -475,8 +505,9 @@ class StepCapsuleLens(StepLens):
async def _invoke(self, llm: Any, messages: list[BaseMessage], model_label: str):
response = await asyncio.wait_for(llm.ainvoke(messages), timeout=90.0)
# Raw-model bypass metering (gateway-wrapped models meter themselves);
# capsule prompts are small and must not clobber the session's
# last_prompt_tokens (the compaction thresholds' live context base).
# capsule prompts are small and are kept out of the session meter's
# last_prompt_tokens (the compaction base itself is operator-owned on
# the transcript ledger and never reads the session meter).
try:
from artemis.services.llm import RobustChatModelWrapper
from artemis.services.token_meter import record_llm_usage
@@ -588,7 +619,8 @@ class ChunkState:
trace_id: Any = None
trace_step_id: Any = None
announced_at: float | None = None
# Measure source at close and summary at swap; estimate tokens as chars // 4.
# Measure source at close and summary at swap; tokens are derived through
# the ledger's calibrated chars-per-token ratio (chars // 4 by default).
# Forced swaps without a capsule leave summary_chars unset.
source_chars: int = 0
summary_chars: int | None = None
@@ -762,8 +794,13 @@ def render_era_period_paragraph(era: EraState) -> str:
def render_era_block(era: EraState) -> str:
"""Era block: ① merged headers, ② degraded to titles, ③ per-chunk ledgers.
A recall-only era includes the step range and session offsets for video
alignment, a synopsis, recall guidance, and preserved user instructions.
A recall-only era (L3) includes the step range and session offsets for
video alignment, a synopsis, recall guidance, and the per-step minimal
index of every member chunk (step number, session offset, action phrase
— results only via search_history). Pending chunks contribute their
index too. User-injected instruction lines are never evicted: they sit
inside the index at their step, or are appended when a chunk carries no
index text.
"""
if era.recall_only:
lines = [
@@ -774,9 +811,12 @@ def render_era_block(era: EraState) -> str:
),
RECALL_GUIDANCE_TEMPLATE.format(start=era.start_step_number, end=era.end_step_number),
]
# Never-evict: user-injected instruction lines survive even recall-only.
for chunk in era.chunks:
lines.extend(chunk.user_lines)
if chunk.minimal_index:
lines.append(chunk.minimal_index)
rendered = "\n".join(lines)
for chunk in era.chunks:
lines.extend(line for line in chunk.user_lines if line not in rendered)
return "\n".join(lines)
merged = merge_structured_fields([c.band1 for c in era.chunks if c.band1])
@@ -816,48 +856,6 @@ def render_era_block(era: EraState) -> str:
return "\n".join(parts)
def render_l3_snapshot(
eras: list[EraState],
chunks: list[ChunkState],
*,
goal: str | None,
plan_text: str | None,
) -> str:
"""L3 emergency snapshot: merged knowledge plane + minimal per-step index.
The chunk headers are mechanically set-merged (no summary-of-summary); the
per-step skeleton survives at minimal width so step-level time perception
never breaks.
"""
all_chunks = [c for era in eras for c in era.chunks] + list(chunks)
merged = merge_structured_fields([c.band1 for c in all_chunks if c.band1])
exit_state = ""
for chunk in reversed(all_chunks):
if chunk.band1:
exit_state = chunk.band1.get("exit_state") or ""
break
start = all_chunks[0].start_step_number if all_chunks else 0
end = all_chunks[-1].end_step_number if all_chunks else 0
parts = [
f"[Session snapshot | Steps {start}–{end} | hard-threshold fallback]",
f"Overall goal: {goal or '-'}",
"Plan state:",
(plan_text or "-").rstrip(),
f"Verified facts: {'; '.join(merged['verified_facts']) or '-'}",
f"Unresolved: {'; '.join(merged['unresolved']) or '-'}",
f"Failed paths: {'; '.join(merged['failed_paths']) or '-'}",
f"Important entities: {'; '.join(merged['important_entities']) or '-'}",
f"Device/app state: {exit_state or '-'}",
"",
"--- Step index (minimal width, per chunk) ---",
]
for chunk in all_chunks:
parts.append(f"[{chunk.step_range_label}]")
parts.append(chunk.minimal_index)
return "\n".join(parts)
# ---------------------------------------------------------------------------
# HistoryChunkManager — triggers, compression events, F-region rendering
# ---------------------------------------------------------------------------
@@ -885,10 +883,13 @@ class HistoryChunkManager:
):
self._engine = engine
self._ctx = ctx
# Accepted for the callers' wiring; the frozen region no longer
# recites the goal (the prompt carries it elsewhere).
self._goal = goal
cc = chunking_config
self._max_steps = int(getattr(cc, "max_steps", 12) or 12)
self._min_steps = max(1, min(self._max_steps, int(getattr(cc, "min_steps", 3) or 3)))
self._target_source_tokens = int(getattr(cc, "target_source_tokens", 2000) or 2000)
self._model_name = getattr(cc, "model", None) or "gemini-3.8-flash"
self._max_chunks = int(getattr(cc, "max_chunks", 8) or 8)
@@ -897,6 +898,8 @@ class HistoryChunkManager:
tc = transcript_config
self._budget = int(getattr(tc, "context_budget_tokens", 80000) or 80000)
start_ratio = getattr(tc, "start_ratio", None)
self._start_ratio = float(0.35 if start_ratio is None else start_ratio)
self._soft_ratio = float(getattr(tc, "soft_ratio", 0.7) or 0.7)
self._hard_ratio = float(getattr(tc, "hard_ratio", 0.9) or 0.9)
self._min_active_steps = int(getattr(tc, "min_active_steps", 5) or 5)
@@ -916,7 +919,7 @@ class HistoryChunkManager:
# whose original turns still live in the transcript until their
# capsule header is ready (ready-gated swap). An entry's ``chunk`` is
# None when the segment had no step records — it can then only leave
# the queue through the hard-threshold emergency swap.
# the queue through the hard-threshold force-swap.
self._chunks: list[ChunkState] = []
self._eras: list[EraState] = []
self._awaiting: list[dict[str, Any]] = []
@@ -1065,11 +1068,11 @@ class HistoryChunkManager:
logger.error(f"History chunk compression event failed: {e}")
def _on_render_inner(self, ledger) -> None:
base_tokens = self._context_base_tokens()
base_tokens = self._context_base_tokens(ledger)
soft = base_tokens is not None and base_tokens >= self._budget * self._soft_ratio
hard = base_tokens is not None and base_tokens >= self._budget * self._hard_ratio
closed_any = self._close_new_segments(ledger, soft)
closed_any = self._close_new_segments(ledger, soft, hard)
if closed_any or soft or hard:
# Failure rung of the degradation ladder: exhausted capsules are
# re-dispatched (the lens retries a fallback model per attempt);
@@ -1079,12 +1082,20 @@ class HistoryChunkManager:
self._harvest_capsules()
self._swap_ready_segments(ledger, hard=hard, base_tokens=base_tokens)
def _close_new_segments(self, ledger, soft: bool) -> bool:
def _close_new_segments(self, ledger, soft: bool, hard: bool = False) -> bool:
"""Trigger evaluation: close due segments and dispatch their capsules.
Closing NEVER freezes turns (ready-gated swap): the segment's original
messages stay live in the transcript; a closed segment joins the
``_awaiting`` queue until its capsule header is ready.
Minimum chunk length: the size and pressure triggers only close once
at least ``min_steps`` eligible turns have accumulated in the open
segment — turns age past the sliding-window floor one per render, so
without it a heavy session closes a one-turn chunk (one capsule call)
every render. A milestone close is exempt (a complete segment is a
unit, however short) and the hard threshold waives it (emergency:
whatever is eligible closes so the swap has material).
"""
turns = ledger.unchunked_turns()
if len(turns) <= self._min_active_steps:
@@ -1128,17 +1139,19 @@ class HistoryChunkManager:
if tail_segment is not None:
tail_portion = tail_segment[1][: max(0, remaining_eligible - consumed_prefix)]
long_enough = hard or len(tail_portion) >= self._min_steps
size_event = False
if tail_segment is not None and tail_portion:
if tail_segment is not None and tail_portion and long_enough:
tail_chars = ledger.turn_text_chars(tail_portion)
size_event = (
len(tail_segment[1]) >= self._max_steps
or (tail_chars // 4) >= self._target_source_tokens
or self._chars_to_tokens(tail_chars, ledger) >= self._target_source_tokens
)
if size_event:
selected.append((tail_segment[0], tail_portion, "size"))
elif not milestone_event and soft and tail_portion:
elif not milestone_event and soft and tail_portion and long_enough:
# Soft threshold with nothing else due: close the oldest open
# segment's eligible portion (bounded by the chunk size cap).
selected.append((tail_segment[0], tail_portion[: self._max_steps], "pressure"))
@@ -1176,6 +1189,10 @@ class HistoryChunkManager:
" retained until a capsule lands."
)
self._capsule_service.submit(key, payload)
# A fresh attempt: if the chunk readies again while the start
# gate still holds it, the timeline gets a fresh ``held`` note
# instead of staying on ``retrying``.
entry["held_announced"] = False
self._announce(chunk, "running", note="retrying")
except Exception as e:
logger.error(f"Capsule re-dispatch for {key} failed: {e}")
@@ -1185,41 +1202,91 @@ class HistoryChunkManager:
Swaps consume the transcript's oldest unchunked turns, so only a
contiguous READY prefix may swap — an older pending segment keeps its
(and every younger segment's) original text live. The hard threshold
is the sole exception: everything closed force-swaps, pending chunks
included, and the frozen region renders as the L3 snapshot.
(and every younger segment's) original text live.
Start gate: closing a segment (milestone/size triggers) only *prepares*
its capsule in the background; the original text is replaced only once
the operator's measured context has reached ``budget * start_ratio``.
Below that the ready chunks are held in the awaiting queue, so at low
occupancy the model keeps reading the full transcript while capsules
are already on hand for when pressure arrives. An unknown base (no
measured operator call yet, or a provider without usage metadata)
cannot gate and falls through to swap-on-ready.
Hard threshold ladder (L2 first, L3 last): at ``budget * hard_ratio``
the ready prefix swaps as ordinary L2 chunk blocks when the estimated
context after that swap drops back under the hard line (rung 1). Only
when no ready prefix exists, or the L2 swap would still leave the
context at or above the line, does everything closed force-swap —
pending chunks included — and the whole frozen region fold into a
recall-only era (rung 2: L3 = period paragraph + per-step minimal
index). With nothing awaiting at all, the last resort is folding the
existing frozen region the same way without consuming turns (rung 3).
The fold is a monotonic state change: later L2 swaps append new chunk
blocks after the folded eras and never restore the folded chunks;
repeated hard renders with nothing left to fold leave the frozen
blocks byte-identical.
``base_tokens`` is the prompt size before the swap, used to estimate
the remaining context on the last swapped chunk's trace.
"""
hard_line = self._budget * self._hard_ratio
if not self._awaiting:
if hard and self._collapse_frozen_region():
ledger.freeze_turns(0, self._render_frozen_blocks())
logger.info(
"History compression: hard threshold with nothing awaiting;"
" frozen region folded into the recall-only period."
)
return
# An entry without a chunk (turns with no step records: nothing to
# summarize, nothing to render) is consumable at once — it must not
# pin the queue, or no L2 swap ever happens and the hard line jumps
# straight to the fold.
ready_prefix: list[dict[str, Any]] = []
for entry in self._awaiting:
chunk = entry["chunk"]
if chunk is not None and chunk.status != "ready":
break
ready_prefix.append(entry)
swap: list[dict[str, Any]] = []
fold = False
if hard:
swap, self._awaiting = self._awaiting, []
estimate = self._estimate_context_after(base_tokens, ready_prefix, ledger)
if ready_prefix and estimate is not None and estimate < hard_line:
swap = ready_prefix # rung 1: L2 is enough
else:
swap = list(self._awaiting) # rung 2: force everything, fold
fold = True
elif base_tokens is not None and base_tokens < self._budget * self._start_ratio:
self._announce_held(base_tokens, ledger)
return
else:
while self._awaiting:
chunk = self._awaiting[0]["chunk"]
if chunk is None or chunk.status != "ready":
break
swap.append(self._awaiting.pop(0))
swap = ready_prefix
if not swap:
return
self._awaiting = self._awaiting[len(swap) :]
for entry in swap:
if entry["chunk"] is not None:
self._chunks.append(entry["chunk"])
self._fold_eras()
blocks = self._render_frozen_blocks(hard=hard)
if fold:
self._collapse_frozen_region()
else:
self._fold_eras()
blocks = self._render_frozen_blocks()
consumed = sum(len(e["turns"]) for e in swap)
ledger.freeze_turns(consumed, blocks)
logger.info(
f"History compression swap: froze {consumed} turns"
f" ({len(self._chunks)} chunks, {len(self._eras)} eras,"
f" {len(self._awaiting)} still awaiting, hard={hard})."
f" {len(self._awaiting)} still awaiting, hard={hard}, folded={fold})."
)
# Pending chunks contribute only their minimal index to the L3 snapshot.
# Include that size in the context estimate, without a compression ratio.
# A folded chunk contributes only its minimal index (its ① fields are
# merged into the period paragraph); a pending chunk has no summary
# size to report, so its trace carries no compression ratio.
swapped = [e for e in swap if e["chunk"] is not None]
source_tokens_total = 0
replacement_tokens_total = 0
@@ -1227,12 +1294,12 @@ class HistoryChunkManager:
chunk = entry["chunk"]
if chunk.status == "ready":
chunk.summary_chars = len(render_chunk_block(chunk))
replacement_chars = chunk.summary_chars
else:
chunk.summary_chars = None
replacement_chars = len(chunk.minimal_index or "")
source_tokens_total += chunk.source_chars // 4
replacement_tokens_total += replacement_chars // 4
source_tokens_total += self._chars_to_tokens(chunk.source_chars, ledger)
replacement_tokens_total += self._chars_to_tokens(
self._replacement_chars(chunk, folded=fold), ledger
)
context_tokens: int | None = None
if base_tokens is not None:
context_tokens = max(
@@ -1242,19 +1309,26 @@ class HistoryChunkManager:
for index, entry in enumerate(swapped):
chunk = entry["chunk"]
turns = len(entry["turns"])
forced = hard and chunk.status != "ready"
if not forced:
result = f"Replaced {turns} turn{'s' if turns != 1 else ''} with the summary."
else:
plural = "s" if turns != 1 else ""
forced = fold and chunk.status != "ready"
if forced:
result = (
"Context budget reached before the summary was ready;"
f" replaced {turns} turn{'s' if turns != 1 else ''} with a brief"
" snapshot."
f" {turns} turn{plural} folded into the period summary."
)
elif fold:
result = (
f"Replaced {turns} turn{plural} with the summary, folded into"
" the period summary (context budget reached)."
)
else:
result = f"Replaced {turns} turn{plural} with the summary."
extra: dict[str, Any] = {
"source_tokens": chunk.source_chars // 4,
"source_tokens": self._chars_to_tokens(chunk.source_chars, ledger),
"summary_tokens": (
chunk.summary_chars // 4 if chunk.summary_chars is not None else None
self._chars_to_tokens(chunk.summary_chars, ledger)
if chunk.summary_chars is not None
else None
),
}
# Show the context estimate once per swap, on its last trace.
@@ -1264,21 +1338,119 @@ class HistoryChunkManager:
extra["context_estimated"] = True
self._announce(chunk, "success", result=result, forced=forced, extra=extra)
def _collapse_frozen_region(self) -> bool:
"""Fold the whole frozen region into recall-only eras (hard-threshold
L3). Every existing era becomes recall-only and the loose chunks fold
into one new recall-only era (a later fold appends another era, so
ordinals and step order stay monotonic). Returns whether anything
changed — False means the region is already fully folded, so the
caller can leave the frozen blocks untouched (byte-stable)."""
changed = False
for era in self._eras:
if not era.recall_only:
era.recall_only = True
changed = True
if self._chunks:
self._era_counter += 1
self._eras.append(
EraState(ordinal=self._era_counter, chunks=self._chunks, recall_only=True)
)
self._chunks = []
changed = True
return changed
@staticmethod
def _chars_to_tokens(chars: int, ledger: Any = None) -> int:
"""Characters → tokens through the ledger's session-calibrated ratio
(:meth:`TranscriptLedger.chars_to_tokens`); ``chars // 4`` without a
ledger. Identical to ``// 4`` at the default ratio."""
convert = getattr(ledger, "chars_to_tokens", None)
if callable(convert):
try:
return int(convert(chars))
except (TypeError, ValueError, ZeroDivisionError):
pass
return max(0, int(chars)) // 4
@staticmethod
def _replacement_chars(chunk: ChunkState, *, folded: bool = False) -> int:
"""Characters a swapped chunk contributes in place of its turns: the
full block when ready and rendered as a chunk block, only the minimal
index when still pending or folded into a recall-only era."""
if chunk.status == "ready" and not folded:
return len(render_chunk_block(chunk))
return len(chunk.minimal_index or "")
def _estimate_context_after(
self, base_tokens: int | None, entries: list[dict[str, Any]], ledger: Any = None
) -> int | None:
"""Estimated prompt size after swapping ``entries`` as L2 blocks."""
if base_tokens is None:
return None
delta = 0
for entry in entries:
chunk = entry["chunk"]
if chunk is None:
continue
delta += self._chars_to_tokens(
self._replacement_chars(chunk), ledger
) - self._chars_to_tokens(chunk.source_chars, ledger)
return max(0, int(base_tokens) + delta)
def _announce_held(self, base_tokens: int, ledger: Any = None) -> None:
"""Once per chunk: its capsule is ready but the start gate holds the
original text live. The timeline line stays ``running`` with a
``held`` note so the UI can say the summary is ready and waiting."""
threshold = int(self._budget * self._start_ratio)
for entry in self._awaiting:
chunk = entry["chunk"]
if chunk is None or chunk.status != "ready" or entry.get("held_announced"):
continue
entry["held_announced"] = True
self._announce(
chunk,
"running",
phase=COMPRESSION_PHASE_READY,
note="held",
extra={
"summary_tokens": self._chars_to_tokens(len(render_chunk_block(chunk)), ledger),
"context_tokens": int(base_tokens),
"context_budget": self._budget,
"swap_at_tokens": threshold,
},
)
def _partition(self, eligible: list[dict]) -> list[tuple[str | None, list[dict]]]:
"""Split eligible turns into consecutive same-hash segments."""
"""Split eligible turns into consecutive same-hash segments.
A turn without a stamp (no recorded step — a reply without a tool
call, a helper-only turn — or a step not stamped yet) joins the
running segment. Leading unstamped turns have no running segment to
join: they are held and become the head of the first stamped segment
instead of forming a stampless segment of their own, which would
close as a chunk with no step records (``_create_chunk`` returns
``None`` for it) and sit at the head of the awaiting queue.
"""
segments: list[tuple[str | None, list[dict]]] = []
current_hash: str | None = None
current: list[dict] = []
started = False
for turn in eligible:
key = turn.get("step_key")
stamped = self._step_hashes.get(str(key)) if key is not None else None
if stamped is None and not started:
current.append(turn) # leading unstamped turns wait for a segment
continue
if stamped is None:
stamped = current_hash # unknown stamps join the running segment
if current and stamped == current_hash:
if not started:
started = True
current_hash = stamped
current.append(turn)
elif stamped == current_hash:
current.append(turn)
else:
if current:
segments.append((current_hash, current))
segments.append((current_hash, current))
current_hash = stamped
current = [turn]
if current:
@@ -1290,21 +1462,26 @@ class HistoryChunkManager:
seg_turns[i : i + self._max_steps] for i in range(0, len(seg_turns), self._max_steps)
]
def _context_base_tokens(self) -> int | None:
def _context_base_tokens(self, ledger=None) -> int | None:
"""The live context base for the start/soft/hard thresholds.
An injected ``meter_getter`` wins (tests / custom wiring); otherwise
the base is the owning operator's last measured prompt size recorded
on the ledger (:meth:`TranscriptLedger.record_prompt_tokens`). The
session-wide token meter is deliberately NOT consulted: its
``last_prompt_tokens`` is overwritten by whichever agent called last
(Planner, Validator, Checker, sub-agents, lenses), so it does not
describe the operator's context at all.
"""
if self._meter_getter is not None:
try:
return self._meter_getter()
except Exception:
return None
try:
from artemis.services.token_meter import get_meter
session_id = getattr(self._engine, "current_session_id", None)
if not session_id:
return None
last = get_meter(session_id).last_prompt_tokens
return last or None
except Exception:
last = getattr(ledger, "last_prompt_tokens", None)
return int(last) if last else None
except (TypeError, ValueError):
return None
# ------------------------------------------------------------------
@@ -1395,7 +1572,11 @@ class HistoryChunkManager:
chunk.source_chars = int(ledger.turn_text_chars(slice_turns))
except Exception as e:
logger.debug(f"Chunk source size unavailable for {chunk.capsule_key}: {e}")
self._announce(chunk, "running", extra={"source_tokens": chunk.source_chars // 4})
self._announce(
chunk,
"running",
extra={"source_tokens": self._chars_to_tokens(chunk.source_chars, ledger)},
)
return chunk
def _announce(
@@ -1408,19 +1589,29 @@ class HistoryChunkManager:
note: str | None = None,
forced: bool = False,
extra: dict[str, Any] | None = None,
phase: str | None = None,
) -> None:
"""Update the chunk's timeline trace without interrupting compression."""
"""Update the chunk's timeline trace without interrupting compression.
``phase`` is the machine-readable plain-language phase written to
``args.phase`` (one of ``COMPRESSION_PHASES``). It defaults from the
trace ``status`` — running → summarizing, success → applied,
failed → failed — and the held path passes ``ready`` explicitly.
"""
if chunk.trace_id is None:
return
engine = self._engine
if engine is None or not hasattr(engine, "record_trace"):
return
if phase is None:
phase = _STATUS_TO_PHASE.get(status, COMPRESSION_PHASE_SUMMARIZING)
args: dict[str, Any] = {
"start_step": chunk.start_step_number,
"end_step": chunk.end_step_number,
"steps": len(chunk.source_step_ids),
"milestone": chunk.milestone_label,
"trigger": chunk.trigger,
"phase": phase,
}
if note:
args["note"] = note
@@ -1450,9 +1641,15 @@ class HistoryChunkManager:
logger.debug(f"Compression announcement skipped for {chunk.capsule_key}: {e}")
def _on_capsule_failed(self, key: JobKey) -> None:
"""Mark the trace failed; keep the chunk pending for a later retry."""
for chunk in self._all_chunks():
if chunk.status != "pending" or chunk.capsule_key != key:
"""Mark the trace failed; keep the chunk pending for a later retry.
Only a chunk still awaiting can be retried (:meth:`_redispatch_failed_capsules`
walks the awaiting queue); a pending chunk that was already force-swapped
keeps its swap trace, since no retry follows for it.
"""
for entry in self._awaiting:
chunk = entry["chunk"]
if chunk is None or chunk.status != "pending" or chunk.capsule_key != key:
continue
self._announce(
chunk,
@@ -1584,31 +1781,28 @@ class HistoryChunkManager:
# ------------------------------------------------------------------
def _fold_eras(self) -> None:
"""Count overflow: the oldest loose chunks merge into an era; the
oldest ledger-bearing eras beyond ``max_eras`` become recall-only
(eras already folded by a hard event do not count against the cap)."""
overflow = len(self._chunks) - self._max_chunks
if overflow > 0:
folded, self._chunks = self._chunks[:overflow], self._chunks[overflow:]
self._era_counter += 1
self._eras.append(EraState(ordinal=self._era_counter, chunks=folded))
era_overflow = len(self._eras) - self._max_eras
with_ledger = [era for era in self._eras if not era.recall_only]
era_overflow = len(with_ledger) - self._max_eras
if era_overflow > 0:
for era in self._eras[:era_overflow]:
if not era.recall_only:
era.recall_only = True
logger.info(
f"Era {era.ordinal} overflowed to the recall-only period"
f" paragraph (Steps {era.start_step_number}–"
f"{era.end_step_number})."
)
for era in with_ledger[:era_overflow]:
era.recall_only = True
logger.info(
f"Era {era.ordinal} overflowed to the recall-only period"
f" paragraph (Steps {era.start_step_number}–"
f"{era.end_step_number})."
)
def _render_frozen_blocks(self, *, hard: bool) -> list[BaseMessage]:
if not self._chunks and not self._eras:
return []
if hard:
plan_text = self._read_plan_text()
text = render_l3_snapshot(
self._eras, self._chunks, goal=self._goal, plan_text=plan_text
)
return [HumanMessage(content=[{"type": "text", "text": text}])]
def _render_frozen_blocks(self) -> list[BaseMessage]:
"""Frozen region, oldest first: era blocks (recall-only or merged),
then the loose chunk blocks."""
blocks: list[BaseMessage] = []
for era in self._eras:
blocks.append(HumanMessage(content=[{"type": "text", "text": render_era_block(era)}]))
@@ -1618,18 +1812,6 @@ class HistoryChunkManager:
)
return blocks
def _read_plan_text(self) -> str | None:
try:
from artemis.utils.notes import get_note_file_path
base_dir = getattr(self._engine, "base_dir", None)
if not base_dir:
return None
path = get_note_file_path(base_dir, "task_plan")
return path.read_text(encoding="utf-8") if path.exists() else None
except Exception:
return None
# ------------------------------------------------------------------
# Draining
# ------------------------------------------------------------------
+3 -3
View File
@@ -2,14 +2,14 @@
You are the Segment Capsule writer for a mobile automation agent's history compression. You receive one contiguous segment of executed steps and produce the two LLM-authored bands of the segment's history chunk:
- Band ① — Synopsis & effects: what this segment was doing, what was actually done, and what effects/artifacts it left behind, plus structured fields.
- Band ② — Compressed step summary: an interval narrative ("Steps xx–xx did ..., Step xx did ...") that seamlessly covers every step of the segment.
- Band ② — Compressed step summary: an interval narrative ("Steps xx–xx did ..., Step xx did ...") covering every step of the segment.
The mechanical per-step action ledger (band ③) is assembled outside of you; do NOT reproduce it.
# INPUT SHAPE
The segment arrives turn by turn, in order. Each turn block has:
- `Recorded steps:` — the mechanical facts of the step(s) executed in that turn: step number, session offset, the exact action, the controller/validator result, the visual transition summary, the notes written (tool → note key: gist), and any user-injected instruction.
- `Transcript as seen by the operator during this turn:` — the turn exactly as it stood in the operator's own context: the observation text, the operator's reasoning (including `(thinking)` blocks), every `[tool call]` with its arguments, every `[tool result]` returned to the operator (explorer answers, recalled history, note reads, ADB output, screenshot placeholders), and the `--- Action Execution Result ---` message. Your capsule REPLACES this transcript in the operator's context — everything the operator learned from it that still matters must survive in your output.
- `Transcript as seen by the operator during this turn:` — the turn exactly as it stood in the operator's own context: the observation text, the operator's reasoning (including `(thinking)` blocks), every `[tool call]` with its arguments, every `[tool result]` returned to the operator (explorer answers, recalled history, note reads, ADB output, screenshot placeholders), and the `--- Action Execution Result ---` message that closes the turn (every turn on the Pro path; only when an action failed on the Flash path). Your capsule REPLACES this transcript in the operator's context — everything the operator learned from it that still matters must survive in your output.
A turn block without a transcript falls back to a reasoning excerpt; treat it the same way with less evidence.
@@ -45,7 +45,7 @@ Field `intervals`: an ordered array of `{"start_step": N, "end_step": M, "text":
- **NEVER narrate step by step** ("Step 5: I tapped X. Step 6: I tapped Y."). One interval is a behavior plus the observed outcome, e.g. "Opened the 'Network & internet' section and returned to the main list; the section showed Wi‑Fi, mobile network and VPN rows."
- **Merge homogeneous consecutive actions into one interval** (repeated scrolls, a sweep through several sections, a polling loop, retries of the same target) — describe the run and its count/result once. A step that does something of a different nature gets its own single-step interval (`start_step == end_step`).
- Every line's text must reference concrete behavior for those steps (the step numbers come from the `Recorded steps:` lines): the target's visible text, the screen or state observed afterwards, values read, errors shown.
- **The union of intervals MUST cover the segment's step range seamlessly** — no gaps, no overlaps, in ascending order. This is machine-checked; a gap forces regeneration.
- **The union of intervals MUST cover the segment's full step range** — no gaps, no overlaps, in ascending order. This is machine-checked; a gap forces regeneration.
- Keep the neutral-wording contract in every line.
# LENGTH TARGET (SOFT)
+28 -1
View File
@@ -20,12 +20,13 @@ with alias resolution (tool_call_id / legacy ordinals), and status queries.
What a summary *contains* (the lens semantics) lives in subclasses — the
service only guarantees how jobs are scheduled, retried, keyed, and drained.
Both Flash and (from M2 on) Pro profiles share this runtime.
Flash and Pro profiles share this runtime.
"""
import asyncio
from typing import Any
from artemis.data_engine.trace import detached_trace
from artemis.utils.logger import get_logger
logger = get_logger(__name__)
@@ -67,12 +68,20 @@ class StepMemoryService:
where available; tool_call_ids (and legacy ordinals) remain queryable
through an alias map so message-level consumers need no migration.
5. Bounded concurrency: at most ``max_concurrency`` attempts run at once.
6. Detached tracing: every job runs inside
:func:`~artemis.data_engine.trace.detached_trace`, so a lens's model
calls never attach to the trace span (the operator's step) that was
current when the job was submitted — no lens needs to know.
Subclasses implement :meth:`_attempt` (one summarization attempt for a
payload) and may override :meth:`_on_status` to persist pending/failed
status transitions.
"""
#: Trace node label of a lens-less subclass's jobs (``lens:<name>``); a
#: configured :class:`StepLens` contributes its own ``name`` instead.
lens_name: str | None = None
def __init__(
self,
ctx: Any,
@@ -131,7 +140,25 @@ class StepMemoryService:
# Scheduling core
# ------------------------------------------------------------------
@property
def trace_node_name(self) -> str:
"""Node label the job's traces and usage records carry (``lens:<name>``)."""
name = getattr(self._lens, "name", None) or self.lens_name or type(self).__name__.lower()
return f"lens:{name}"
async def _run_until_ready(self, key: JobKey) -> None:
"""Run one job to completion, detached from the submitter's trace span.
The task was created while the submitting agent's step span was
current and inherits that context; the whole job — every attempt,
retry sleeps included — runs under :func:`detached_trace` so nothing
a lens records (model calls, their thinking/raw_thinking children,
stream deltas, usage) can be attributed to the submitter's step.
"""
with detached_trace(self.trace_node_name):
await self._run_attempts(key)
async def _run_attempts(self, key: JobKey) -> None:
"""Retry one job independently, bounded by the configured retry limit.
Attempts at most ``1 + retry_limit`` times; on exhaustion the job
+341 -13
View File
@@ -22,11 +22,16 @@ Messages are organized into four regions:
- **A (active window)**: raw per-turn messages, append-only. Each committed
turn contributes its tail observation HumanMessage, the operator AIMessages
(tool_calls and native thinking preserved by reference), the in-turn
ToolMessages, and the turn's validator result message. The scrub edge
(:class:`~artemis.agents.flash.context_compressor.ScrubEdgeCompressor`) strips
old UI lists and plan recitations at depth 1 and resolves screenshots to
visual summaries at depth K — messages are never
removed or reordered, so tool-call/response pairs are never split.
ToolMessages, and the turn's result message (every turn on the Pro path;
only when an action failed on the Flash path). The
scrub edge (:class:`~artemis.agents.flash.context_compressor.ScrubEdgeCompressor`)
has two depths: a shallow text edge (UI list, plan recitation and ephemeral
per-turn blocks leave as soon as a newer observation exists) and a
screenshot edge that follows the occupancy tier (relaxed below the start
gate, tightened past it), where the screenshot is replaced in place by its
visual summary. Every rewrite happens once and is never undone; everything
newer is byte-identical to what the model was first shown. Messages are
never removed or reordered, so tool-call/response pairs are never split.
- **T (current tail)**: built fresh every turn by the operator; passed to
:meth:`render` and only enters A when the turn is committed.
@@ -47,7 +52,7 @@ import json
import time
from typing import Any, Callable
from langchain_core.messages import BaseMessage, HumanMessage
from langchain_core.messages import AIMessage, BaseMessage, HumanMessage
from artemis.memory.step_memory import StepMemoryService
from artemis.utils.logger import get_logger
@@ -67,6 +72,28 @@ EXECUTION_RESULT_MARKER = "--- Action Execution Result"
#: Header prefix of the cold-start restored-history block.
RESTORED_HISTORY_HEADER = "[Restored history]"
#: ``HumanMessage.additional_kwargs`` key listing the indices of content blocks that
#: are only meaningful for the turn they were built for (reminders, hints, user
#: guidance, per-turn notices). The scrub edge deletes them at its text edge, so
#: they never reach the frozen region or a chunk capsule. The indices always
#: describe the message's *current* block layout: a rewrite that removes blocks
#: before the text edge (the screenshot swap when the image edge is shallower)
#: remaps the surviving indices in place, and the key is dropped once the blocks
#: are deleted. ``additional_kwargs`` is never sent to the provider, so the
#: marking costs the model nothing.
EPHEMERAL_BLOCKS_KEY = "ephemeral_blocks"
def mark_ephemeral(message: BaseMessage, indices) -> None:
"""Flag content blocks of ``message`` (by index) as ephemeral, see
:data:`EPHEMERAL_BLOCKS_KEY`."""
existing = list(message.additional_kwargs.get(EPHEMERAL_BLOCKS_KEY, []))
for index in indices:
index = int(index)
if index not in existing:
existing.append(index)
message.additional_kwargs[EPHEMERAL_BLOCKS_KEY] = existing
def format_session_offset(seconds: float) -> str:
"""Render a session-start offset as ``T+mm:ss`` (minutes never wrap)."""
@@ -135,6 +162,8 @@ def render_turn_transcript(messages: list[BaseMessage]) -> str:
names_by_id: dict[str, str] = {}
for msg in messages:
kind = str(getattr(msg, "type", "") or "message")
if isinstance(msg, AIMessage):
kind = "ai" # streamed replies report type "AIMessageChunk"
role = _ROLE_LABELS.get(kind, kind)
if kind == "tool":
call_id = getattr(msg, "tool_call_id", None)
@@ -157,6 +186,71 @@ def render_turn_transcript(messages: list[BaseMessage]) -> str:
return "\n".join(lines)
#: Tokens charged per image block in the provider-less prompt estimate
#: (Gemini's flat per-image cost; the Flash runner uses the same figure).
IMAGE_BLOCK_TOKENS = 258
#: Characters per token assumed before any calibrated measurement.
DEFAULT_CHARS_PER_TOKEN = 4.0
#: Bounds of the calibrated chars-per-token ratio: ~1 for CJK-heavy prompts,
#: ~4 for English prose; anything outside is a measurement artefact.
_CHARS_PER_TOKEN_BOUNDS = (1.0, 6.0)
#: Smallest text growth (characters) between two consecutive measured prompts
#: for their difference to count as a calibration sample (see
#: :meth:`TranscriptLedger.record_prompt_tokens`). Below this the provider's
#: rounding and a few hundred tokens of turn-to-turn jitter dominate the
#: quotient.
MIN_CALIBRATION_DELTA_CHARS = 2000
def measure_prompt_content(messages: list[Any]) -> tuple[int, int]:
"""Count ``(text_chars, image_count)`` over a prompt's messages.
The walk covers what the provider tokenizes as text: string content in
full; in list content the ``text`` blocks by their text, ``image_url`` /
``image`` blocks as one image each, any other block by its ``str()``; and
each message's ``tool_calls`` by the JSON of their arguments (an
AIMessage's calls are sent back to the model verbatim, and the ledger's
turn transcripts size them the same way).
"""
text_chars = 0
image_count = 0
for msg in messages:
content = getattr(msg, "content", None)
if isinstance(content, str):
text_chars += len(content)
elif isinstance(content, list):
for block in content:
if isinstance(block, dict):
if block.get("type") in ("image_url", "image") or "image_url" in block:
image_count += 1
elif block.get("type") == "text":
text_chars += len(str(block.get("text", "")))
else:
text_chars += len(str(block))
else:
text_chars += len(str(block))
elif content is not None:
text_chars += len(str(content))
for call in getattr(msg, "tool_calls", None) or []:
if isinstance(call, dict):
text_chars += len(
json.dumps(call.get("args") or {}, ensure_ascii=False, default=str)
)
return text_chars, image_count
def estimate_prompt_tokens(messages: list[Any]) -> int:
"""Provider-less prompt size estimate: ``text_chars // 4`` plus
:data:`IMAGE_BLOCK_TOKENS` per image block (see
:func:`measure_prompt_content` for what counts). Used as the context base
when a provider reports no usage metadata; never below 1."""
text_chars, image_count = measure_prompt_content(messages)
return max(1, text_chars // 4 + image_count * IMAGE_BLOCK_TOKENS)
class TranscriptLedger:
"""Append-only four-region message ledger for one Pro session.
@@ -172,7 +266,10 @@ class TranscriptLedger:
prune_history_xml: bool = True,
image_scrub_depth: int = 3,
pending_grace_steps: int = 3,
xml_scrub_depth: int = 1,
xml_scrub_depth: int | None = None,
image_scrub_depth_relaxed: int | None = None,
context_budget_tokens: int | None = None,
start_ratio: float | None = None,
clock: Callable[[], float] | None = None,
session_start: float | None = None,
):
@@ -204,6 +301,51 @@ class TranscriptLedger:
self._frozen_blocks: list[BaseMessage] = []
self._chunker: Any | None = None
# The owning operator's last measured prompt size (tokens). This is
# the live context base the chunker's start/soft/hard thresholds read;
# only the operator's own calls write it, so Planner/Checker/lens
# calls sharing the session can never masquerade as the operator's
# context (see HistoryChunkManager._context_base_tokens).
self._last_prompt_tokens: int | None = None
# Session-calibrated characters-per-token ratio (see
# record_prompt_tokens): the chunker converts transcript characters
# to tokens through it, so CJK-heavy sessions (~1 char/token on
# Gemini) are not under-estimated 4x by the fixed //4 default.
# Calibrated differentially: ``_calibration_sample`` is the previous
# measured ``(text_chars, image_count, prompt_tokens)`` and the
# pooled deltas between consecutive same-image-count prompts give
# the session's single best ratio.
self._chars_per_token: float = DEFAULT_CHARS_PER_TOKEN
self._calibration_sample: tuple[int, int, int] | None = None
self._pooled_delta_chars = 0
self._pooled_delta_tokens = 0
# Whether the last committed turn carried no visible reasoning text (bare
# tool calls / thinking only). Set by ``commit_staged``; both runners read
# it to decide whether to attach the shared reasoning reminder.
self._last_turn_silent: bool = False
# Text edge (default 1): UI list, plan recitation and ephemeral blocks
# leave as soon as a newer observation exists; screenshots follow K.
self._xml_scrub_depth = xml_scrub_depth
# Occupancy-driven screenshot depth. Below the start gate
# (``last_prompt_tokens < budget * start_ratio``, or no measurement
# yet) the scrub edge sits at the relaxed depth so the model keeps
# more raw screenshots while nothing is being replaced; from the
# start gate up it tightens to ``image_scrub_depth``. Scrubbing is
# irreversible, so a change of depth never rewrites a message:
# tightening scrubs the extra depths once, relaxing lets new images
# live longer. See :attr:`effective_image_scrub_depth`.
self._image_scrub_depth = max(1, int(image_scrub_depth))
self._image_scrub_depth_relaxed = (
max(self._image_scrub_depth, int(image_scrub_depth_relaxed))
if image_scrub_depth_relaxed is not None
else self._image_scrub_depth
)
self._context_budget_tokens = int(context_budget_tokens) if context_budget_tokens else None
self._start_ratio = float(start_ratio) if start_ratio is not None else None
# id(message) -> summary-job key (DataEngine step id). A side map keeps
# the key out of the serialized message payload entirely.
self._step_keys: dict[int, str] = {}
@@ -235,6 +377,186 @@ class TranscriptLedger:
"""The current session offset as ``T+mm:ss``."""
return format_session_offset(self.elapsed_seconds())
# ------------------------------------------------------------------
# Context base (operator-measured prompt size)
# ------------------------------------------------------------------
@property
def last_prompt_tokens(self) -> int | None:
"""The operator's last measured prompt size, or None before the
first measured call (or when the provider reports no usage)."""
return self._last_prompt_tokens
@property
def last_turn_silent(self) -> bool:
"""True when the last committed turn had no visible reasoning text.
A turn is silent when it carries at least one AI message and none of
them shows text: string content that is empty or whitespace, or list
content without a non-empty ``text`` block (native ``thinking`` blocks
are not visible text). A turn with no AI message at all is not silent.
"""
return self._last_turn_silent
@staticmethod
def _ai_message_has_visible_text(message: BaseMessage) -> bool:
content = getattr(message, "content", None)
if isinstance(content, str):
return bool(content.strip())
if isinstance(content, list):
for block in content:
if isinstance(block, str):
if block.strip():
return True
elif isinstance(block, dict) and block.get("type") == "text":
if str(block.get("text") or "").strip():
return True
return False
return False
@classmethod
def _turn_is_silent(cls, messages: list[BaseMessage]) -> bool:
# Streamed replies are ``AIMessageChunk`` (type "AIMessageChunk"), so
# the class, not the type string, identifies the model's messages.
ai_messages = [m for m in messages if isinstance(m, AIMessage)]
if not ai_messages:
return False
return not any(cls._ai_message_has_visible_text(m) for m in ai_messages)
@property
def chars_per_token(self) -> float:
"""Calibrated characters-per-token ratio (4.0 until a measured call
with its messages is recorded)."""
return self._chars_per_token
def chars_to_tokens(self, chars: int) -> int:
"""Convert transcript characters to tokens through the calibrated
ratio (identical to ``chars // 4`` at the default ratio)."""
return int(max(0, int(chars)) / self._chars_per_token)
@property
def occupancy(self) -> float | None:
"""``last_prompt_tokens / context_budget_tokens``, or None when either
is unknown."""
if self._last_prompt_tokens is None or not self._context_budget_tokens:
return None
return self._last_prompt_tokens / self._context_budget_tokens
@property
def below_start_gate(self) -> bool:
"""Whether the measured occupancy is under the start gate. An unknown
occupancy (no measured call yet) counts as below: a session starts
empty."""
if self._start_ratio is None:
return False
ratio = self.occupancy
return ratio is None or ratio < self._start_ratio
@property
def effective_image_scrub_depth(self) -> int:
"""The screenshot depth the scrub edge uses at the next render: the
relaxed depth below the start gate, the tight depth from there up."""
if self.below_start_gate:
return self._image_scrub_depth_relaxed
return self._image_scrub_depth
def record_prompt_tokens(
self, prompt_tokens: int | None, messages: list[Any] | None = None
) -> None:
"""Record the operator's measured prompt size for one call.
Called by the ledger's owning operator (Flash runner / Pro operator)
after each of its own LLM calls; within a multi-call turn the last
(largest) call wins. Non-positive or missing values are ignored so a
provider without usage metadata leaves the base unknown rather than
zero (and the calibration state below is left untouched).
When ``messages`` (the exact list sent to the model) accompanies a
measured size, the call also calibrates :attr:`chars_per_token`,
**differentially**: the ledger remembers the previous measured
sample ``(text_chars, image_count, prompt_tokens)`` and, when the new
sample carries the same number of images, grows the text by at least
:data:`MIN_CALIBRATION_DELTA_CHARS` and costs more tokens, it pools
the pair's differences (``pooled_chars += Δchars``,
``pooled_tokens += Δtokens``) and sets the ratio to
``pooled_chars / pooled_tokens``, clamped to [1.0, 6.0]. The pooled
quotient is the maximum-likelihood single ratio for the session (no
smoothing constant to tune, no last-pair noise). The reference is
replaced only when it was consumed, when the image count changed, or
when the prompt shrank in tokens (a compaction or scrub reset the
baseline); a sample that moved too little keeps the reference, so
several small turns accumulate into one usable difference instead of
being discarded one by one (real sessions often grow < 2000 chars
per turn and would otherwise never calibrate).
Why differences: the provider's ``prompt_tokens`` also counts terms
that are not in ``messages`` or not proportional to their text — the
bound tool/function schemas, and full-size screenshots that tile to
~1100–1800 tokens rather than the flat :data:`IMAGE_BLOCK_TOKENS`.
A whole-prompt quotient charges all of that to the text, and it
also averages over the static prefix, whereas the chunker applies
the ratio to the transcript *tail*, whose density differs. On real
Gemini Operator traces (10 sessions, 2026-09) the two disagree by
5-30% with a session-dependent sign, so the size-based chunk
trigger fired early in some sessions and late in others. Between
two prompts with the same image count the constant terms cancel and
the difference measures the tail's own rate. Until the first
usable difference exists the whole-prompt quotient
``text_chars / (prompt_tokens - images * IMAGE_BLOCK_TOKENS)`` is
used as a fallback; once a pooled difference exists the pooled ratio
is authoritative and the fallback is no longer applied.
"""
try:
value = int(prompt_tokens) if prompt_tokens is not None else 0
except (TypeError, ValueError):
return
if value <= 0:
return
self._last_prompt_tokens = value
if messages is None:
return
try:
text_chars, image_count = measure_prompt_content(messages)
except Exception as exc:
logger.debug(f"Prompt calibration skipped: {exc}", exc_info=True)
return
if text_chars <= 0:
return
low, high = _CHARS_PER_TOKEN_BOUNDS
previous = self._calibration_sample
sample = (text_chars, image_count, value)
if previous is None:
self._calibration_sample = sample
else:
prev_chars, prev_images, prev_tokens = previous
delta_chars = text_chars - prev_chars
delta_tokens = value - prev_tokens
if image_count != prev_images or delta_tokens < 0:
# Not comparable (image set changed) or the prompt actually
# shrank (compaction / scrub): start a fresh baseline here.
self._calibration_sample = sample
elif delta_chars >= MIN_CALIBRATION_DELTA_CHARS:
if delta_tokens > 0:
self._pooled_delta_chars += delta_chars
self._pooled_delta_tokens += delta_tokens
pooled = self._pooled_delta_chars / self._pooled_delta_tokens
self._chars_per_token = min(high, max(low, pooled))
self._calibration_sample = sample
return
self._calibration_sample = sample # text grew at no cost: unusable
# else: moved too little to measure yet (in either direction) —
# keep the reference so the movement accumulates against it.
if self._pooled_delta_tokens > 0:
# A pooled difference already exists: it is authoritative, and
# the whole-prompt quotient (which mis-charges schemas and image
# tiling to the text) is never applied again.
return
text_tokens = max(1, value - image_count * IMAGE_BLOCK_TOKENS)
self._chars_per_token = min(high, max(low, text_chars / text_tokens))
# ------------------------------------------------------------------
# S region
# ------------------------------------------------------------------
@@ -320,6 +642,7 @@ class TranscriptLedger:
return
staged = self._staged
self._staged = None
self._last_turn_silent = self._turn_is_silent(staged)
if step_key is not None:
observation = self._first_image_message(staged)
@@ -449,18 +772,23 @@ class TranscriptLedger:
def render(self, tail: list[BaseMessage]) -> list[BaseMessage]:
"""Advance compression and return ``S + F + A + tail``.
Order per turn: the chunker runs first (a triggered compression event
deep-mutates the frozen region and advances the F/A boundary), then
the scrub edge advances over the active window. The returned list is a
fresh container: appending to it (the operator's in-turn tool loop)
never mutates the ledger regions.
Order per turn: the occupancy band is read once (from the operator's
last measured prompt), the scrub edge advances over the active window
at that band's screenshot depth, then the chunker runs (a triggered
compression event deep-mutates the frozen region and advances the F/A
boundary). Scrub-before-chunk guarantees that the turns a chunk
closes over — all older than the sliding-window floor — have already
had their screenshots resolved when the relaxed depth is no deeper
than the floor plus one. The returned list is a fresh container:
appending to it (the operator's in-turn tool loop) never mutates the
ledger regions.
"""
self._compressor.compress(self._active, image_scrub_depth=self.effective_image_scrub_depth)
if self._chunker is not None:
try:
self._chunker.on_render(self)
except Exception as e:
logger.error(f"History chunker render hook failed: {e}")
self._compressor.compress(self._active)
return [
*self._static,
*self._restored,
+25 -3
View File
@@ -192,7 +192,7 @@
// ==========================================================================
// 🌐 4.3 Shared Capabilities & Global Overrides
// ==========================================================================
// 🧠 Unified Step-Memory Runtime (history-module redesign v3, M0–M5)
// Shared step memory and history compression
// Transcript ledger + layered compression for the Pro operator.
"memory": {
// Background summarization runtime (visual/capsule lenses)
@@ -204,15 +204,33 @@
"transcript": {
// ROLLBACK SWITCH: set false to restore the legacy per-turn 2-message
// prompt path byte-for-byte (L2/L3 chunk compression rides this same
// flag). Default on since M5 (2026-09-01) per the on-device A/B:
// turn latency −77% P50, cache-hit calls 43%→88%, output tokens −74%.
// flag). See services/token_meter.py for measured session token usage
// and cache-hit ratios.
"enabled": true,
"context_budget_tokens": 80000,
// Below start_ratio of the operator's own measured prompt size, closed
// segments only prepare their summary; the original turns are kept in
// context and replaced once occupancy reaches this ratio. Measurements
// come from the operator; other agents' calls do not affect this gate.
"start_ratio": 0.35,
"soft_ratio": 0.7,
"hard_ratio": 0.9,
"min_active_steps": 5,
// Screenshot edge: 6 while the operator's measured context is below
// start_ratio (nothing is being replaced yet), 3 from there up. A
// screenshot reaching the edge is replaced in place by its visual
// summary (the 'Current Screenshot' label leaves with it) and never
// restored; the depth only decides how long new ones stay.
"image_scrub_depth": 3,
"image_scrub_depth_relaxed": 6,
"pending_grace_steps": 3,
// Text edge: once a newer observation exists (depth 1) the UI element
// list, the plan recitation and the per-turn ephemeral blocks
// (reminders, hints, guidance wrappers) leave the older observation.
// User instruction bodies remain in history. The text edge stays
// shallow on purpose: only the live observation carries an indexed
// element list, so a stale [n] index can never be picked up as a
// target.
"xml_scrub_depth": 1,
// Local historical-state hint (no model call); distance calibrated
// 2026-09-01 from 460 dHash-stamped on-device steps (same screen <=4,
@@ -223,6 +241,10 @@
// Segment (L2) chunk compression: three-band chunks, era folding
"chunking": {
"max_steps": 12,
// Size/pressure closes wait until at least this many turns are
// eligible, so heavy turns do not become one-step chunks (one capsule
// call each); milestone closes are exempt, the hard line waives it.
"min_steps": 3,
"target_source_tokens": 2000,
"model": "gemini-3.8-flash",
"max_chunks": 8
+2
View File
@@ -33,6 +33,7 @@ from artemis.runtime.daemon_client import (
ensure_daemon_running,
get_daemon_session,
get_daemon_status,
is_artemis_daemon,
is_daemon_running,
stop_task_on_daemon,
submit_batch_to_daemon,
@@ -84,6 +85,7 @@ __all__ = [
"get_daemon_session",
"get_daemon_status",
"get_server_status",
"is_artemis_daemon",
"is_daemon_running",
"is_cancel_requested",
"is_port_in_use",
+26
View File
@@ -17,6 +17,7 @@
from __future__ import annotations
import json
from http.client import HTTPException
import os
from pathlib import Path
import subprocess
@@ -62,6 +63,31 @@ def is_daemon_running(
return False
def is_artemis_daemon(
host: str = DEFAULT_DAEMON_HOST,
port: int = DEFAULT_DAEMON_PORT,
timeout: float = 0.5,
) -> bool:
"""Identity probe: is the process on ``host:port`` the Artemis daemon?
:func:`is_daemon_running` is a liveness probe that accepts any HTTP
listener (a dev server on port 8000 answers ``200`` on ``/``). Diagnostics
need to tell the daemon apart from a port squatter, so this asks the
daemon's lightweight ``/api/system/server-status`` route and requires the
JSON shape only Artemis serves.
"""
url = f"http://{host}:{port}/api/system/server-status"
try:
req = urllib.request.Request(url, headers={"User-Agent": "Artemis-Daemon-Probe"})
with urllib.request.urlopen(req, timeout=timeout) as response:
if response.status != 200:
return False
payload = json.loads(response.read().decode("utf-8", errors="replace") or "null")
except (OSError, ValueError, HTTPException):
return False
return isinstance(payload, dict) and "current_pid" in payload and "port" in payload
def daemon_log_path(port: int = DEFAULT_DAEMON_PORT) -> Path:
"""Return the log file path used by a background-spawned Daemon on the given port."""
from artemis.config.paths import get_app_dir
+111 -16
View File
@@ -12,20 +12,26 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Session-level LLM token metering (M0: pure recording, no decisions).
"""Session-level LLM token usage and cache-hit metering.
Every gateway-managed LLM call reports its real ``usage_metadata`` here. The
meter accumulates per-session totals and records one ``llm_usage`` trace per
call into the DataEngine, carrying:
- the call's measured ``prompt_tokens`` — the last call's measured input is the
best available estimate of the *current context base* for the next call
(the compaction thresholds of M2/M3 will consume this, never char-count
heuristics);
- the call's measured ``prompt_tokens`` (``context_base_tokens`` in the trace
payload, kept for the timeline). Note: the session meter's
``last_prompt_tokens`` is whichever call happened last in the session —
Planner, Validator, Checker, sub-agents included — so the compaction
thresholds do NOT read it; the operator records its own prompt size on the
transcript ledger (``TranscriptLedger.record_prompt_tokens``);
- provider-reported cache hits (Gemini ``cached_content_token_count`` /
LangChain ``input_token_details.cache_read``) to establish the cache-hit-rate
baseline;
- running per-session totals.
- running per-session totals, including ``session_cached_ratio``
(cached / prompt tokens over the session) and the same ratio for the call's
own ``source`` (``source_cached_ratio``), so a session whose operator calls
never hit the prompt cache is visible on its ``llm_usage`` traces and in the
session summary line (:func:`log_session_summary`).
This module never raises into the LLM call path and makes no decisions.
"""
@@ -71,8 +77,15 @@ def extract_usage(response: Any) -> dict[str, int] | None:
}
def cached_ratio(cached_tokens: int, prompt_tokens: int) -> float:
"""``cached / prompt`` rounded to three decimals; 0.0 with no prompt tokens."""
if prompt_tokens <= 0:
return 0.0
return round(cached_tokens / prompt_tokens, 3)
class SessionTokenMeter:
"""Accumulates measured LLM usage for one session."""
"""Accumulates measured LLM usage for one session, in total and per source."""
def __init__(self, session_id: str):
self.session_id = session_id
@@ -83,36 +96,116 @@ class SessionTokenMeter:
self.cached_tokens = 0
self.cache_hit_calls = 0
self.last_prompt_tokens = 0
# Per-source totals (source = gateway endpoint key or lens label):
# {"llm_calls", "prompt_tokens", "cached_tokens", "cache_hit_calls"}.
self.sources: dict[str, dict[str, int]] = {}
def record(self, usage: dict[str, int], *, update_last_prompt: bool = True) -> dict[str, int]:
def record(
self,
usage: dict[str, int],
*,
update_last_prompt: bool = True,
source: str | None = None,
) -> dict[str, Any]:
"""Accumulates one call's usage; returns a session snapshot.
``update_last_prompt=False`` accumulates the totals without touching
``last_prompt_tokens`` — background lens calls carry tiny prompts that
must not masquerade as the live context base consumed by the L2/L3
compaction thresholds.
compaction thresholds. With a ``source`` the call also joins that
source's totals and the snapshot carries ``source_cached_ratio``.
"""
with self._lock:
self.llm_calls += 1
self.prompt_tokens += usage.get("prompt_tokens", 0)
self.completion_tokens += usage.get("completion_tokens", 0)
prompt = usage.get("prompt_tokens", 0)
cached = usage.get("cached_tokens", 0)
self.llm_calls += 1
self.prompt_tokens += prompt
self.completion_tokens += usage.get("completion_tokens", 0)
self.cached_tokens += cached
if cached > 0:
self.cache_hit_calls += 1
if update_last_prompt:
self.last_prompt_tokens = usage.get("prompt_tokens", 0)
return self.snapshot()
self.last_prompt_tokens = prompt
snapshot = self._snapshot_locked()
if source:
totals = self.sources.setdefault(
source,
{"llm_calls": 0, "prompt_tokens": 0, "cached_tokens": 0, "cache_hit_calls": 0},
)
totals["llm_calls"] += 1
totals["prompt_tokens"] += prompt
totals["cached_tokens"] += cached
if cached > 0:
totals["cache_hit_calls"] += 1
snapshot["source_cached_ratio"] = cached_ratio(
totals["cached_tokens"], totals["prompt_tokens"]
)
return snapshot
def snapshot(self) -> dict[str, int]:
def _snapshot_locked(self) -> dict[str, Any]:
return {
"session_llm_calls": self.llm_calls,
"session_prompt_tokens": self.prompt_tokens,
"session_completion_tokens": self.completion_tokens,
"session_cached_tokens": self.cached_tokens,
"session_cache_hit_calls": self.cache_hit_calls,
"session_cached_ratio": cached_ratio(self.cached_tokens, self.prompt_tokens),
}
def snapshot(self) -> dict[str, Any]:
with self._lock:
return self._snapshot_locked()
def summary(self) -> dict[str, Any]:
"""Session totals plus every source's totals with its ``cached_ratio``."""
with self._lock:
summary: dict[str, Any] = {"session_id": self.session_id, **self._snapshot_locked()}
summary["sources"] = {
source: {
**totals,
"cached_ratio": cached_ratio(totals["cached_tokens"], totals["prompt_tokens"]),
}
for source, totals in self.sources.items()
}
return summary
def format_session_summary(session_id: Any) -> str:
"""One log line: session cache ratio, then each source's calls and ratio."""
summary = get_meter(session_id).summary()
parts = [
f"LLM usage for session {summary['session_id']}:"
f" {summary['session_llm_calls']} calls,"
f" {summary['session_prompt_tokens']} prompt tokens,"
f" {summary['session_cached_tokens']} cached"
f" (cached_ratio={summary['session_cached_ratio']:.3f},"
f" cache-hit calls {summary['session_cache_hit_calls']})"
]
for source, totals in summary["sources"].items():
parts.append(
f"{source}: {totals['llm_calls']} calls, cached_ratio={totals['cached_ratio']:.3f}"
)
return "; ".join(parts)
def log_session_summary(session_id: Any) -> dict[str, Any] | None:
"""Log the session's usage summary (INFO) at session end, best-effort.
Returns the summary dict, or None when the session never metered a call
(nothing is logged then). Never raises.
"""
try:
key = str(session_id)
with _LOCK:
meter = _METERS.get(key)
if meter is None or meter.llm_calls == 0:
return None
logger.info(format_session_summary(key))
return meter.summary()
except Exception as exc:
logger.debug(f"Token meter session summary skipped: {exc}")
return None
def get_meter(session_id: Any) -> SessionTokenMeter:
"""Returns (creating on first use) the meter for a session id."""
@@ -147,7 +240,9 @@ def record_llm_usage(
if not session_id or usage is None:
return None
snapshot = get_meter(session_id).record(usage, update_last_prompt=update_last_prompt)
snapshot = get_meter(session_id).record(
usage, update_last_prompt=update_last_prompt, source=source
)
payload: dict[str, Any] = {
**usage,
"context_base_tokens": usage["prompt_tokens"],
+49 -19
View File
@@ -125,23 +125,52 @@ def _adb_binary() -> str:
return "adb"
#: Inline display cap for command output, in characters. Beyond it the full
#: text is cached under a TaskId and only the tail is shown to the model.
_LONG_OUTPUT_CHARS = 50000
#: How many trailing lines of an over-long output stay visible inline.
_LONG_OUTPUT_TAIL_LINES = 200
def _is_output_long(output: str) -> bool:
"""Checks if command output exceeds the max threshold for inline display."""
return len(output.splitlines()) > 500 or len(output) > 50000
"""Whether command output is too long to show inline.
The trigger is size alone: more than ``_LONG_OUTPUT_CHARS`` characters.
A line count by itself never trips it, so a 1000-line / 4 KB listing
(``seq 1 1000``) is shown in full instead of being cut to its tail.
"""
return len(output) > _LONG_OUTPUT_CHARS
def _format_long_output_response(_task_id: str, output: str, intro: str) -> str:
def _format_long_output_response(task_id: str, output: str, intro: str) -> str:
"""Formats truncated output with guidance to use the analyzer tool."""
lines = output.splitlines()
last_200 = "\n".join(lines[-200:])
tail = "\n".join(lines[-_LONG_OUTPUT_TAIL_LINES:])
return (
f"{intro}\nWarning: Output is too long ({len(lines)} lines,"
f" {len(output)} chars) and has been truncated.\n--- Last 200 lines of"
f" output ---\n{last_200}\n------------------------\nYou can use the"
" 'analyze_task_output' tool to get more information."
f" {len(output)} chars) and has been truncated.\n--- Last"
f" {_LONG_OUTPUT_TAIL_LINES} lines of output ---\n{tail}\n"
f"------------------------\nThe full output is kept under TaskId"
f" {task_id}: use analyze_task_output with TaskId {task_id} to query it."
)
def _render_output_body(output: str) -> str:
"""Renders command output for the model; whitespace-only output reads as empty."""
return output if output.strip() else "(empty output)"
def _abbreviate(text: str, limit: int = 40) -> str:
"""Cuts ``text`` to ``limit`` characters, marking the cut only when one happened."""
return text if len(text) <= limit else f"{text[:limit]}..."
def _terminal_label(terminal_id: str | None) -> str:
"""Renders a terminal id for the model; a task without one shows as ``-``."""
return terminal_id or "-"
def _filter_persistent_env(env: dict[str, str]) -> dict[str, str]:
"""Drops the platform-provided variables from a captured shell environment."""
kept: dict[str, str] = {}
@@ -700,8 +729,7 @@ class RunAdbCommandTool(ArtemisTool):
intro += "."
if run_persistent:
intro += f" TerminalID: {terminal_id}."
body = clean_output if clean_output.strip() else "(empty output)"
return f"{intro}\nOutput:\n{body}"
return f"{intro}\nOutput:\n{_render_output_body(clean_output)}"
# We spawn the host subprocess: adb -s <serial> shell <script>.
# The script travels as an argument so the remote shell exits with it;
@@ -779,7 +807,7 @@ class RunAdbCommandTool(ArtemisTool):
registry.finished[task_id]["notified"] = True
return _format_long_output_response(task_id, clean_output, intro)
return f"{intro}\nOutput:\n{clean_output}"
return f"{intro}\nOutput:\n{_render_output_body(clean_output)}"
# pylint: disable=too-many-arguments,too-many-positional-arguments
async def _hand_off_to_background(
@@ -896,8 +924,8 @@ class ManageTaskTool(ArtemisTool):
active_tasks = []
for tid, t in registry.background.items():
active_tasks.append(
f"- {tid}: Command='{t.command[:40]}...', Phone"
f" Cwd='{t.cwd}', TerminalID='{t.terminal_id}',"
f"- {tid}: Command='{_abbreviate(t.command)}', Phone"
f" Cwd='{t.cwd}', TerminalID={_terminal_label(t.terminal_id)},"
f" Interactive={t.interactive}"
)
if not active_tasks:
@@ -908,7 +936,7 @@ class ManageTaskTool(ArtemisTool):
finished_tasks = []
for tid, t in registry.finished.items():
finished_tasks.append(
f"- {tid}: Command='{t['command'][:40]}...', Status='{t['status']}'"
f"- {tid}: Command='{_abbreviate(t['command'])}', Status='{t['status']}'"
)
if not finished_tasks:
return "No recently finished tasks."
@@ -924,7 +952,7 @@ class ManageTaskTool(ArtemisTool):
if action == "status":
task_info = _get_task_info(task_id, ctx)
if not task_info:
return f"Task {task_id} not found."
return ToolFailure(f"Error: Task {task_id} not found.")
output_text = task_info.get("output", "")
if _ENV_START_MARKER in output_text:
@@ -935,7 +963,7 @@ class ManageTaskTool(ArtemisTool):
f"Status: {task_info.get('status')}\n"
f"Command: {task_info.get('command')}\n"
f"Cwd: {task_info.get('cwd')}\n"
f"TerminalID: {task_info.get('terminal_id')}"
f"TerminalID: {_terminal_label(task_info.get('terminal_id'))}"
)
exit_code = task_info.get("exit_code")
if exit_code is not None:
@@ -948,16 +976,18 @@ class ManageTaskTool(ArtemisTool):
if action == "kill":
if not task:
return f"Task {task_id} is not active or already finished."
return ToolFailure(f"Error: Task {task_id} is not active or already finished.")
try:
await task.stop(ctx, status="killed")
return f"Task {task_id} successfully terminated."
except Exception as e: # pylint: disable=broad-exception-caught
return ToolFailure(f"Failed to terminate task {task_id}: {e}")
if task.exit_code is not None:
return f"Terminated task {task_id} (exit code {task.exit_code})."
return f"Terminated task {task_id}."
if action == "send_input":
if not task:
return f"Task {task_id} is not active."
return ToolFailure(f"Error: Task {task_id} is not active.")
if not input_str:
return ToolFailure("Error: Input is required for send_input action.")
if not task.interactive or task.process.stdin is None:
@@ -968,7 +998,7 @@ class ManageTaskTool(ArtemisTool):
try:
task.process.stdin.write(input_str.encode())
await task.process.stdin.drain()
return f"Input successfully sent to task {task_id}."
return f"Sent input to task {task_id}."
except Exception as e: # pylint: disable=broad-exception-caught
return ToolFailure(f"Failed to send input: {e}")
+33 -10
View File
@@ -209,6 +209,9 @@ class RegisteredCandidate:
bounds: tuple[int, int, int, int] | None = None
#: True when the candidate mapped onto an element that was already indexed.
reused: bool = False
#: The text the agent's own indexed list shows for the reused element, so
#: the answer can name it the way the agent already knows it.
existing_text: str = ""
#: A bare-point candidate this close to an existing indexed center (in
@@ -331,6 +334,7 @@ def register_candidates(
description=cand.description,
bounds=_element_pixel_bounds(existing),
reused=True,
existing_text=str(existing.get("text") or ""),
)
)
continue
@@ -371,8 +375,20 @@ def register_candidates(
# --------------------------------------------------------------------------- #
def render_text(query: str, outcome: ExplorerOutcome, registered: list[RegisteredCandidate]) -> str:
"""Plain-text answer for the calling agent."""
def render_text(
query: str,
outcome: ExplorerOutcome,
registered: list[RegisteredCandidate],
*,
index_targets: bool = False,
) -> str:
"""Plain-text answer for the calling agent.
``index_targets`` selects the targeting syntax the caller's ``click``
accepts: the Pro Operator and the Flash / MCP action executor take a bare
integer index next to coordinates; the Validator's text entry takes
coordinates only.
"""
if registered:
lines = [
f"Explorer located {len(registered)} candidate(s) for '{query}' in the"
@@ -381,12 +397,18 @@ def render_text(query: str, outcome: ExplorerOutcome, registered: list[Registere
for c in registered:
line = f"- [{c.index}] '{c.description}' at normalized [{c.coords[0]}, {c.coords[1]}]"
if c.reused:
line += " (already in your indexed list)"
line += f" (matches your indexed element [{c.index}] '{c.existing_text}')"
lines.append(line)
lines.append(
"Act on a candidate by its index (e.g. target=[index]) or by the"
" normalized coordinate. Candidates are ranked by confidence."
)
if index_targets:
lines.append(
"Act on a candidate by its index (target=3, a bare integer) or by its"
" normalized coordinate. Candidates are ranked by confidence."
)
else:
lines.append(
"Act on a candidate by its normalized coordinate (target=[x, y]) with a"
" target_description. Candidates are ranked by confidence."
)
if outcome.message:
lines.append(f"Explorer notes: {outcome.message}")
return "\n".join(lines)
@@ -444,7 +466,7 @@ def render_operator_blocks(
registered: list[RegisteredCandidate],
) -> str | list[dict[str, Any]]:
"""Multimodal answer for the Operator: text plus the annotated screenshot."""
text = render_text(query, outcome, registered)
text = render_text(query, outcome, registered, index_targets=True)
screenshot_path = getattr(state, "latest_screenshot", None)
image_path = (
render_annotated_image(ctx, screenshot_path, registered) if screenshot_path else None
@@ -475,10 +497,11 @@ async def ask_explorer_text(
*,
agent_name: str = "validator",
) -> str:
"""Text-only entry used by the Validator / Flash runner and MCP executors."""
"""Text-only, coordinate-only entry used by the Validator (Flash and the MCP
executor reach the explorer through ``McpActionExecutor`` with index targets)."""
outcome = await locate(ctx, state, query, context_feedback, agent_name=agent_name)
registered = register_candidates(ctx, state, outcome)
return render_text(query, outcome, registered)
return render_text(query, outcome, registered, index_targets=False)
class AskExplorerTool(ArtemisTool):
+8 -4
View File
@@ -15,9 +15,9 @@
"""replay_steps: full replay of a step range, exactly as the executing agent saw it.
Each step is rendered with :func:`~artemis.utils.task_tree.render_step_replay`
(``[Screen]`` description, reasoning, every non-internal tool call with name /
arguments / result, the planned action, safety-net interception and the
execution result). Two loose bounds keep one tool result finite (config
(``T+mm:ss`` step header, ``[Screen]`` description, reasoning, every
non-internal tool call with name / arguments / result, the executed action,
safety-net interception and the execution result). Two loose bounds keep one tool result finite (config
``agent.memory.replay``): at most ``max_steps`` steps per call, and a
``max_tokens`` budget applied by dropping *whole trailing steps* — a replayed
step is never cut mid-sentence. Screenshots are deliberately not part of the
@@ -79,6 +79,10 @@ def replay_steps_text(
return ToolFailure(f"Error: no recorded steps in range {start}–{end}.")
budget_tokens = max(1, int(max_tokens or DEFAULT_MAX_TOKENS))
# Step headers carry the session clock (``T+mm:ss``) every agent prompt
# and search_history use; without a session start they fall back to the
# stored relative time.
session_start = getattr(reader, "session_start_time", None)
rendered: list[str] = []
shown: list[dict] = []
used = 0
@@ -88,7 +92,7 @@ def replay_steps_text(
dropped += 1
continue
try:
text = render_step_replay(step)
text = render_step_replay(step, session_start=session_start)
except Exception as e:
return ToolFailure(f"Error rendering steps {start}–{end}: {e}")
cost = _estimate_tokens(text) + 1
+21 -10
View File
@@ -29,6 +29,7 @@ from pathlib import Path
import sqlite3
from typing import Any
from artemis.core.tool_failure import ToolFailure
from artemis.utils.logger import get_logger
from artemis.utils.visualization import overlay_action_on_screenshot
@@ -47,10 +48,18 @@ class ScreenshotResult:
image_bytes: bytes | None = None
#: True when ``which == "overlay"`` and the action was actually drawn.
overlay_drawn: bool = False
#: True when the lookup could not serve the request (bad arguments, no
#: history, unknown step, unreadable file). A step that simply has no
#: screenshot of the requested kind is an answer, not an error.
is_error: bool = False
def to_content_blocks(self) -> list[dict[str, Any]] | str:
"""The tool's return value: content blocks with the image, or the
plain-text explanation when there is no image to attach."""
"""The tool's return value: content blocks with the image, the
plain-text answer when there is no image to attach, or a
:class:`ToolFailure` (status ``error`` downstream) when the lookup
failed."""
if self.is_error:
return ToolFailure(self.description)
if not self.image_bytes:
return self.description
encoded = base64.b64encode(self.image_bytes).decode("utf-8")
@@ -60,41 +69,43 @@ class ScreenshotResult:
]
def _failure(step_number: int, which: str, description: str) -> ScreenshotResult:
return ScreenshotResult(step_number, which, description, is_error=True)
def load_step_screenshot(reader: Any, step_number: Any, which: str = "pre") -> ScreenshotResult:
"""Loads (and for ``overlay`` annotates) one step screenshot from ``reader``."""
which = str(which or "pre").lower()
try:
number = int(step_number)
except (TypeError, ValueError):
return ScreenshotResult(
-1, which, f"Error: step_number must be an integer, got {step_number!r}."
)
return _failure(-1, which, f"Error: step_number must be an integer, got {step_number!r}.")
if which not in WHICH_CHOICES:
return ScreenshotResult(
return _failure(
number, which, f"Error: 'which' must be 'pre', 'post' or 'overlay', got '{which}'."
)
if reader is None:
return ScreenshotResult(number, which, "Error: no execution history available.")
return _failure(number, which, "Error: no execution history available.")
source = "pre" if which == "overlay" else which
try:
path = reader.get_step_image_path(number, source)
except (sqlite3.Error, OSError, ValueError) as e:
return ScreenshotResult(number, which, f"Error looking up step {number}: {e}")
return _failure(number, which, f"Error looking up step {number}: {e}")
if path is None:
try:
record = reader.get_step_record(number)
except (sqlite3.Error, ValueError):
record = None
if record is None:
return ScreenshotResult(number, which, f"Error: step {number} not found.")
return _failure(number, which, f"Error: step {number} not found.")
return ScreenshotResult(
number, which, f"No {source}-action screenshot recorded for step {number}."
)
try:
image_bytes = Path(path).read_bytes()
except OSError as e:
return ScreenshotResult(number, which, f"Error reading screenshot: {e}")
return _failure(number, which, f"Error reading screenshot: {e}")
if which != "overlay":
return ScreenshotResult(
+165 -14
View File
@@ -30,6 +30,12 @@ Hard boundaries (config ``agent.memory.recall``):
- every result carries a step number / step id;
- large raw sources return only excerpts around the match plus a reference.
Two surfaces per step: the *scoring* haystack keeps the raw record (action
JSON, execution result JSON, UI-tree XML) so resource ids, bounds and package
names stay searchable; the *excerpt* shown as ``Match:`` is built from the
readable twin (the ledger action phrase, result status/error words, element
text / content descriptions / OCR text) so the model never reads JSON.
A ``step_range`` additionally returns the full-width ``build_action_ledger``
rows of that range — the re-entry point for compressed-history marker lines.
"""
@@ -104,6 +110,15 @@ def result_search_text(result: Any) -> str:
the haystack via :func:`_screen_text`), so base64 never enters the search."""
if result is None:
return ""
# A content-block list stored as its JSON string (trace serialization)
# is unwrapped first so its text blocks render, not the JSON.
if isinstance(result, str) and result.lstrip().startswith("[{"):
try:
parsed = json.loads(result)
except ValueError:
parsed = None
if isinstance(parsed, list) and any(isinstance(b, dict) and "type" in b for b in parsed):
return result_search_text(parsed)
if isinstance(result, list):
texts = [
str(block.get("text") or "")
@@ -135,23 +150,32 @@ def _events_text(step: dict) -> str:
return "\n".join(parts)
def _action_text(action_taken: Any) -> str:
"""Combine the rendered action with raw fields for search.
Rendered excerpts identify self-described targets. Raw JSON also makes
resource IDs, bounds, and other fields searchable.
"""
def _rendered_action(action_taken: Any) -> str:
"""The shared ledger phrase of a step's action (``format_actions_clean``)."""
if not action_taken:
return ""
try:
rendered = format_actions_clean(action_taken)
return format_actions_clean(action_taken)
except Exception:
rendered = ""
return ""
def _action_text(action_taken: Any) -> str:
"""Combine the rendered action with raw fields for *scoring*.
Rendered text identifies self-described targets. Raw JSON also makes
resource IDs, bounds, and other fields searchable. Never shown as-is:
the excerpt comes from :func:`_step_display_text`.
"""
if not action_taken:
return ""
rendered = _rendered_action(action_taken)
raw = json.dumps(action_taken, ensure_ascii=False, default=str)
return f"{rendered}\n{raw}" if rendered else raw
def _step_haystack(step: dict) -> str:
"""The scoring surface of a step: every recorded field, raw JSON included."""
extra = step.get("extra_metadata") or {}
fields = [
str(step.get("summary") or ""),
@@ -170,6 +194,70 @@ def _step_haystack(step: dict) -> str:
return "\n".join(f for f in fields if f)
def _result_display_text(result: Any) -> str:
"""The execution result as words: its status and result/error strings
(an execution incident renders as its one-line phrase), never JSON."""
if not result:
return ""
if not isinstance(result, dict):
return str(result)
parts: list[str] = []
status = result.get("status")
if status:
parts.append(f"status {status}")
try:
from artemis.utils.task_tree import format_result_clean
incident_or_error = format_result_clean(result)
except Exception:
incident_or_error = None
if incident_or_error:
parts.append(str(incident_or_error))
else:
for key in ("result", "error", "error_msg", "message"):
value = result.get(key)
if isinstance(value, str) and value.strip():
parts.append(value.strip())
return " | ".join(parts)
def _events_display_text(step: dict) -> str:
"""Tool calls and reasoning of a step as the replay shows them (name,
arguments, result text), without the raw event JSON."""
parts: list[str] = []
for event in step.get("interleaved_events") or []:
etype = event.get("type")
if etype == "tool_call":
name = str(event.get("name") or "")
args = event.get("args") or {}
result_text = result_search_text(event.get("result"))
try:
call = f"{name}({json.dumps(args, ensure_ascii=False, default=str)})"
except (TypeError, ValueError):
call = name
parts.append(f"{call} -> {result_text}" if result_text else call)
elif etype and "thought" in etype:
parts.append(str(event.get("content") or ""))
return "\n".join(p for p in parts if p)
def _step_display_text(step: dict) -> str:
"""The *excerpt* surface of a step: the same fields as the haystack, as the
agent would read them. The summary is left out — it already heads the
result as the ``Screen:`` line."""
extra = step.get("extra_metadata") or {}
fields = [
_rendered_action(step.get("action_taken")),
_result_display_text(step.get("last_execution_result")),
str(step.get("operator_raw_thinking") or ""),
_events_display_text(step),
str(extra.get("injected_instruction") or ""),
str(extra.get("foreground_app") or ""),
]
# Excerpts flatten whitespace, so the field boundary must survive as text.
return " | ".join(f.strip() for f in fields if f and f.strip())
def _image_text(reader: Any, image_name: str | None) -> str:
"""Raw XML/OCR text stored for one screenshot — the textual description
of that image (package/activity strings live inside the XML blob)."""
@@ -212,9 +300,37 @@ def referenced_image_names(step: dict) -> list[str]:
return names
def _screen_text(reader: Any, step: dict) -> str:
"""Description text of the screenshots behind a step: its own pre- and
post-action screenshots plus any screenshot a tool result embedded."""
#: UI-tree node attributes whose values are human-readable screen content.
_UI_NODE_TEXT_KEYS = ("text", "content-desc", "content_desc", "resource-id", "resource_id")
def _collect_screen_strings(value: Any, out: list[str], depth: int = 0) -> None:
"""Walks a stored UI tree / OCR blob and appends its readable values
(element text, content description, resource id, OCR text) in order."""
if depth > 50:
return
if isinstance(value, str):
text = " ".join(value.split())
if text and text not in out:
out.append(text)
elif isinstance(value, dict):
for key in _UI_NODE_TEXT_KEYS:
text = value.get(key)
if isinstance(text, str):
text = " ".join(text.split())
if text and text not in out:
out.append(text)
for child_key in ("children", "nodes", "elements"):
children = value.get(child_key)
if isinstance(children, (list, tuple)):
for child in children:
_collect_screen_strings(child, out, depth + 1)
elif isinstance(value, (list, tuple)):
for child in value:
_collect_screen_strings(child, out, depth + 1)
def _screen_names(step: dict) -> list[str]:
names: list[str] = []
for name in (
step.get("pre_image_name"),
@@ -223,7 +339,32 @@ def _screen_text(reader: Any, step: dict) -> str:
):
if name and name not in names:
names.append(name)
return "\n".join(t for t in (_image_text(reader, n) for n in names) if t)
return names
def _screen_text(reader: Any, step: dict) -> str:
"""Description text of the screenshots behind a step (its own pre- and
post-action screenshots plus any screenshot a tool result embedded) —
the raw XML/OCR blobs, for *scoring* only."""
return "\n".join(t for t in (_image_text(reader, n) for n in _screen_names(step)) if t)
def _screen_display_text(reader: Any, step: dict) -> str:
"""The readable content of the same screenshots for the *excerpt*: element
text / content descriptions / resource ids and OCR text joined by
``" | "``, never the UI-tree JSON."""
strings: list[str] = []
for name in _screen_names(step):
try:
record = reader.storage.get_image(name)
except (sqlite3.Error, ValueError, AttributeError):
continue
if record is None:
continue
for blob in (record.ui_tree, record.ocr_result):
if blob:
_collect_screen_strings(blob, strings)
return " | ".join(strings)
def _step_label(step: dict, session_start: float | None) -> str:
@@ -321,12 +462,22 @@ def search_history_text(
total = score + screen_score
if total <= 0:
continue
source = haystack if score >= screen_score else screen
lines = [f"[{_step_label(step, session_start)} | id {step.get('step_id')}]"]
summary = step.get("summary")
if summary:
lines.append(f" Screen: {_clamp(str(summary), EXCERPT_CHARS)}")
lines.append(f" Match: {_excerpt(source, terms, EXCERPT_CHARS)}")
# The excerpt comes from the readable twin of whichever surface
# scored higher; the raw JSON only ever scores. When the hit sits
# in the summary alone, the Screen line already shows it.
display = (
_step_display_text(step)
if score >= screen_score
else _screen_display_text(reader, step)
)
display_hit = _score(display, terms) > 0
summary_hit = bool(summary) and _score(str(summary), terms) > 0
if display and (display_hit or not summary_hit):
lines.append(f" Match: {_excerpt(display, terms, EXCERPT_CHARS)}")
number = step.get("step_number") or 0
results.append((total, number, "\n".join(lines)))
+24 -1
View File
@@ -29,6 +29,29 @@ from artemis.utils.logger import get_logger
logger = get_logger(__name__)
#: How many line ranges a merged block's header lists before it is cut.
_MAX_HEADER_RANGES = 8
def _compress_line_numbers(indices: list[int]) -> str:
"""Renders sorted line numbers as ranges: ``6``, ``69-79``, ``699-798``.
Consecutive runs collapse into ``start-end``; at most
``_MAX_HEADER_RANGES`` ranges are listed, the rest replaced by ``...``.
"""
runs: list[str] = []
start = prev = indices[0]
for idx in indices[1:]:
if idx == prev + 1:
prev = idx
continue
runs.append(str(start) if start == prev else f"{start}-{prev}")
start = prev = idx
runs.append(str(start) if start == prev else f"{start}-{prev}")
if len(runs) > _MAX_HEADER_RANGES:
return ", ".join(runs[:_MAX_HEADER_RANGES]) + ", ..."
return ", ".join(runs)
# pylint: disable=too-many-branches,too-many-locals,too-many-statements
def search_and_merge_logs(
@@ -99,7 +122,7 @@ def search_and_merge_logs(
truncated = True
break
match_str = ", ".join(map(str, match_indices))
match_str = _compress_line_numbers(match_indices)
matches.append(f"--- Match at line(s) {match_str} ---")
for idx in range(start, end):
line_content = log_lines[idx]
+11 -11
View File
@@ -148,7 +148,7 @@ class SaveNoteTool(ArtemisTool):
else "."
)
save_note_content(base_dir, k, c)
agent_outcome = f"Successfully saved note to {k}.md."
agent_outcome = f"Saved note '{k}'."
status = "success"
except Exception as e: # pylint: disable=broad-exception-caught
agent_outcome = f"Failed to save note {k}.md: {e}"
@@ -227,7 +227,7 @@ class AppendNoteTool(ArtemisTool):
else "."
)
append_note_content(base_dir, k, c)
agent_outcome = f"Successfully appended note to {k}.md."
agent_outcome = f"Appended to note '{k}'."
status = "success"
except Exception as e: # pylint: disable=broad-exception-caught
agent_outcome = f"Failed to append note {k}.md: {e}"
@@ -309,9 +309,9 @@ class UpdateNoteTool(ArtemisTool):
)
warning = update_note_content(base_dir, k, t, r)
if warning:
agent_outcome = f"Successfully updated note '{k}'.\nWARNING: {warning}"
agent_outcome = f"Updated note '{k}'.\nWARNING: {warning}"
else:
agent_outcome = f"Successfully updated note '{k}'."
agent_outcome = f"Updated note '{k}'."
status = "success"
except Exception as e: # pylint: disable=broad-exception-caught
agent_outcome = f"Failed to update note '{k}': {e}"
@@ -642,7 +642,7 @@ class SaveNotePureTool(ArtemisTool):
else "."
)
save_note_content(base_dir, k, c)
return f"Successfully saved note to {k}.md."
return f"Saved note '{k}'."
except Exception as e: # pylint: disable=broad-exception-caught
return ToolFailure(f"Failed to save note {k}.md: {e}")
@@ -709,8 +709,8 @@ class UpdateNotePureTool(ArtemisTool):
)
warning = update_note_content(base_dir, k, t, r)
if warning:
return f"Successfully updated note '{k}'.\nWARNING: {warning}"
return f"Successfully updated note '{k}'."
return f"Updated note '{k}'.\nWARNING: {warning}"
return f"Updated note '{k}'."
except Exception as e: # pylint: disable=broad-exception-caught
return ToolFailure(f"Failed to update note '{k}': {e}")
@@ -774,7 +774,7 @@ class AppendNotePureTool(ArtemisTool):
else "."
)
append_note_content(base_dir, k, c)
return f"Successfully appended note to {k}.md."
return f"Appended to note '{k}'."
except Exception as e: # pylint: disable=broad-exception-caught
return ToolFailure(f"Failed to append note {k}.md: {e}")
@@ -791,7 +791,7 @@ def get_append_note_tool_pure(ctx: ArtemisContext) -> BaseTool:
save_note_wrapper = ToolWrapper(
tool_fn_getter=get_save_note_tool,
on_success_fn=lambda key: f"Successfully saved note '{key}'.",
on_success_fn=lambda key: f"Saved note '{key}'.",
on_failure_fn=lambda key: f"Failed to save note '{key}'.",
)
@@ -810,13 +810,13 @@ list_notes_wrapper = ToolWrapper(
update_note_wrapper = ToolWrapper(
tool_fn_getter=get_update_note_tool,
on_success_fn=lambda key: f"Successfully updated note '{key}'.",
on_success_fn=lambda key: f"Updated note '{key}'.",
on_failure_fn=lambda key: f"Failed to update note '{key}'.",
)
append_note_wrapper = ToolWrapper(
tool_fn_getter=get_append_note_tool,
on_success_fn=lambda key: f"Successfully appended note '{key}'.",
on_success_fn=lambda key: f"Appended to note '{key}'.",
on_failure_fn=lambda key: f"Failed to append note '{key}'.",
)
+1 -1
View File
@@ -63,7 +63,7 @@ class VideoAnalyzerArgs(BaseModel):
OPERATOR_VIDEO_ANALYZER_DOCSTRING = (
"[VIDEO] Use this tool to analyze any specific time range of the video"
" recording. This tool delegates to a powerful video-analyzing subagent"
" recording. This tool delegates to a video analysis subagent"
" with full access to the continuous screen recording of your entire"
" execution session.\n\n"
"- This tool overcomes your limitation of only receiving discrete"
+182 -44
View File
@@ -15,6 +15,8 @@
"""Utilities for handling app locking and initial app launch logic."""
import asyncio
from dataclasses import dataclass, field
import re
from artemis.context import AppLaunchResult, ArtemisContext
from artemis.controllers.platform_specific_commands_controller import (
@@ -28,43 +30,180 @@ from artemis.utils.logger import get_logger
logger = get_logger(__name__)
SYSTEM_OVERLAYS_VALID_FOR_LAUNCH = {
"com.google.android.permissioncontroller",
"com.android.permissioncontroller",
"com.google.android.packageinstaller",
"com.android.packageinstaller",
}
# ``ActivityRecord{<hash> u<user> <package>/<activity> t<task id>}`` as printed by
# ``dumpsys activity activities`` on every line that names an activity.
_ACTIVITY_RECORD_RE = re.compile(
r"ActivityRecord\{[0-9a-f]+ u\d+ (?P<package>[\w.]+)/(?P<activity>[\w.$]+) t(?P<task>\d+)\}"
)
# ``* Task{<hash> #<id> type=... A=<uid>:<affinity> ... visible=true ...}`` task headers
# (``* TaskRecord{<hash> #<id> A=<affinity> ...}`` on Android 11 and older).
_TASK_HEADER_RE = re.compile(r"^\s*\* Task(?:Record)?\{[0-9a-f]+ #(?P<task>\d+)(?P<attrs>[^}]*)\}")
# Device-side pre-filter: the full dump runs to hundreds of KB on a busy device;
# only the task headers, history entries and resumed-activity lines are parsed.
_FOREGROUND_DUMP_CMD = (
"dumpsys activity activities | grep -E"
r" '^ *\* (Task|TaskRecord)\{|^ *\* Hist |ResumedActivity'"
)
# ``* Hist #<index>: ActivityRecord{...}`` entries; index 0 is the task's base activity.
_HIST_RE = re.compile(r"^\s*\* Hist\s+#(?P<index>\d+): (?P<record>ActivityRecord\{[^}]*\})")
# ``topResumedActivity=ActivityRecord{...}`` (per task, top task first) and the global
# ``ResumedActivity: ActivityRecord{...}`` line; the first one seen is the foreground one.
_RESUMED_RE = re.compile(r"ResumedActivity[:=]\s*(?P<record>ActivityRecord\{[^}]*\})")
def get_focused_task_package(ctx: ArtemisContext) -> str | None:
"""Inspect the active/focused task stack to find the package affinity of the top task.
@dataclass(frozen=True)
class ForegroundTask:
"""What ``dumpsys activity activities`` says is on top of the display.
Uses adb shell dumpsys activity activities to extract this info.
``resumed_package`` is the package of the resumed activity (what the user sees).
``base_package`` is the package of the task's root activity (``Hist #0``), and
``packages`` holds every package that has an activity in that task, so an app whose
task currently shows a helper activity from another package (Settings search,
permission dialogs, account pickers, system update pages, ...) is still
recognised as the foreground app.
"""
task_id: int | None = None
affinity: str | None = None
resumed_component: str | None = None
resumed_package: str | None = None
base_package: str | None = None
packages: frozenset[str] = field(default_factory=frozenset)
def owns(self, app_package: str) -> bool:
"""Whether ``app_package`` is the app the foreground task belongs to.
The task affinity is the app's own claim on the task (by default the
package name, or a package-prefixed name such as
``com.android.settings.root``); it stays with the task even when the
app's root activity has been destroyed and only a companion activity
(e.g. the Settings search UI) remains in the history."""
affinity = self.affinity or ""
return (
app_package == self.resumed_package
or app_package == self.base_package
or app_package in self.packages
or affinity == app_package
or affinity.startswith(app_package + ".")
)
def describe(self) -> str:
task = f"task #{self.task_id}" if self.task_id is not None else "task ?"
details = [f"base={self.base_package}", f"resumed={self.resumed_component}"]
if self.affinity:
details.append(f"affinity={self.affinity}")
others = sorted(self.packages - {self.base_package, self.resumed_package, None})
if others:
details.append(f"also={','.join(others)}")
return f"{task} ({', '.join(details)})"
def parse_foreground_task(dump: str) -> ForegroundTask | None:
"""Parse ``dumpsys activity activities`` output into the foreground task summary.
The foreground task is the one holding the resumed activity; when no activity is
resumed (mid-transition) the first visible task with history entries is used.
Returns None when the dump contains no task information at all.
"""
task_attrs: dict[int, str] = {}
task_order: list[int] = []
task_activities: dict[int, list[tuple[int, str, str]]] = {}
resumed: re.Match[str] | None = None
for line in dump.splitlines():
header = _TASK_HEADER_RE.match(line)
if header:
task_id = int(header.group("task"))
if task_id not in task_attrs:
task_attrs[task_id] = header.group("attrs")
task_order.append(task_id)
continue
hist = _HIST_RE.match(line)
if hist:
record = _ACTIVITY_RECORD_RE.search(hist.group("record"))
if record:
task_activities.setdefault(int(record.group("task")), []).append(
(int(hist.group("index")), record.group("package"), record.group("activity"))
)
continue
if resumed is None:
match = _RESUMED_RE.search(line)
if match:
resumed = _ACTIVITY_RECORD_RE.search(match.group("record"))
if not task_attrs and resumed is None:
return None
task_id: int | None = None
if resumed is not None:
task_id = int(resumed.group("task"))
else:
for candidate in task_order:
if "visible=true" in task_attrs[candidate] and candidate in task_activities:
task_id = candidate
break
activities = sorted(task_activities.get(task_id, []))
affinity_match = re.search(r"\bA=(?:\d+:)?(\S+)", task_attrs.get(task_id, ""))
return ForegroundTask(
task_id=task_id,
affinity=affinity_match.group(1) if affinity_match else None,
resumed_component=(
f"{resumed.group('package')}/{resumed.group('activity')}" if resumed else None
),
resumed_package=resumed.group("package") if resumed else None,
base_package=activities[0][1] if activities else None,
packages=frozenset(package for _, package, _ in activities),
)
def get_foreground_task(ctx: ArtemisContext) -> ForegroundTask | None:
"""Read the foreground task from ``dumpsys activity activities`` on the device.
Blocking (a synchronous ADB shell round-trip); async callers go through
:func:`get_foreground_task_async` so the event loop keeps serving.
"""
try:
device = get_adb_device(ctx)
output = str(device.shell("dumpsys activity activities | grep topDisplayFocusedRootTask"))
if "topDisplayFocusedRootTask=" in output:
segment = output.split("topDisplayFocusedRootTask=")[-1]
if "A=" in segment:
affinity = segment.split("A=")[-1].split("}")[0].split()[0].strip()
if ":" in affinity:
return affinity.split(":")[-1]
return affinity
if "{" in segment and "}" in segment:
braces_content = segment.split("{")[-1].split("}")[0]
tokens = braces_content.split()
for token in tokens:
if "." in token and not token.startswith("#") and not token.isdigit():
if ":" in token:
return token.split(":")[-1]
return token
return None
if device is None:
return None
dump = str(device.shell(_FOREGROUND_DUMP_CMD))
if "Task" not in dump:
# No grep on the device (or nothing matched): parse the full dump.
dump = str(device.shell("dumpsys activity activities"))
return parse_foreground_task(dump)
except Exception as e:
logger.debug(f"Failed to retrieve focused task package via dumpsys: {e}")
logger.debug(f"Failed to retrieve foreground task via dumpsys: {e}")
return None
async def get_foreground_task_async(ctx: ArtemisContext) -> ForegroundTask | None:
""":func:`get_foreground_task` off the event loop."""
return await asyncio.to_thread(get_foreground_task, ctx)
async def _observe_foreground(
ctx: ArtemisContext, app_package: str
) -> tuple[bool, str | None, ForegroundTask | None]:
"""Report whether ``app_package`` is in the foreground.
Fast path: the focused window's package equals the target. Otherwise the app also
counts as foreground when the top task belongs to it (see ``ForegroundTask``).
A ``None`` focused package means the window manager is mid-transition and is
reported as "not yet" without consulting the task stack.
Returns (is_foreground, focused_package, foreground_task); the task is only read
when the fast path fails on a non-null focused package.
"""
current_package = await get_current_foreground_package_async(ctx)
if current_package == app_package:
return True, current_package, None
if current_package is None:
return False, None, None
task = await get_foreground_task_async(ctx)
return task is not None and task.owns(app_package), current_package, task
async def _poll_for_app_ready(
ctx: ArtemisContext,
app_package: str,
@@ -88,38 +227,37 @@ async def _poll_for_app_ready(
polls = int(max_poll_seconds / poll_interval)
for i in range(polls):
current_package = await get_current_foreground_package_async(ctx)
ready, current_package, task = await _observe_foreground(ctx, app_package)
if current_package == app_package:
logger.success(f"App {app_package} is ready (took ~{i * poll_interval:.1f}s)")
return True, None
if current_package in SYSTEM_OVERLAYS_VALID_FOR_LAUNCH:
task_package = get_focused_task_package(ctx)
if task_package == app_package:
if ready:
if current_package == app_package:
logger.success(f"App {app_package} is ready (took ~{i * poll_interval:.1f}s)")
else:
logger.success(
f"App {app_package} launch succeeded (system overlay"
f" '{current_package}' focused, but top task stack affinity"
f" matches expected app package '{task_package}', took"
f" ~{i * poll_interval:.1f}s)"
f"App {app_package} is ready (focused window belongs to"
f" '{current_package}', but the foreground {task.describe()} is owned"
f" by {app_package}, took ~{i * poll_interval:.1f}s)"
)
return True, None
return True, None
if current_package is None:
logger.debug(f"Poll {i + 1}/{polls}: App loading (mCurrentFocus=null)...")
else:
logger.debug(
f"Poll {i + 1}/{polls}: Wrong app in foreground (expected"
f" '{app_package}', got '{current_package}'). Still waiting..."
f" '{app_package}', got '{current_package}', foreground"
f" {task.describe() if task else 'task unknown'}). Still waiting..."
)
if i < polls - 1:
await asyncio.sleep(poll_interval)
current_package = await get_current_foreground_package_async(ctx)
task = await get_foreground_task_async(ctx)
error_msg = (
f"Timeout waiting for {app_package} to load after {max_poll_seconds}s. "
f"Current foreground: {current_package}"
f"Current foreground: {current_package}; foreground"
f" {task.describe() if task else 'task unknown'}"
)
logger.error(error_msg)
return False, error_msg
@@ -225,10 +363,10 @@ async def _handle_initial_app_launch(
logger.info(f"Starting initial app launch for package: {locked_app_package}")
try:
current_package = await get_current_foreground_package_async(ctx)
already_foreground, current_package, _ = await _observe_foreground(ctx, locked_app_package)
logger.info(f"Current foreground app: {current_package}")
if current_package == locked_app_package:
if already_foreground:
logger.info(f"App {locked_app_package} is already in foreground")
return AppLaunchResult(
locked_app_package=locked_app_package,
+8 -4
View File
@@ -93,8 +93,10 @@ async def validate_api_key(
async with httpx.AsyncClient(timeout=timeout) as client:
if is_google_family_provider(clean_provider):
# Test Google Gemini via generativelanguage API
url = f"https://generativelanguage.googleapis.com/v1beta/models?key={clean_key}&pageSize=1"
resp = await client.get(url)
# The key travels in a header, never in the URL: httpx logs request
# URLs at INFO level and those logs end up in files.
url = "https://generativelanguage.googleapis.com/v1beta/models?pageSize=1"
resp = await client.get(url, headers={"x-goog-api-key": clean_key})
if resp.status_code == 200:
return True, "Google Gemini API key verified successfully."
err_msg = _extract_error_message(resp)
@@ -102,7 +104,7 @@ async def validate_api_key(
elif clean_provider in ("ocr", "vision", "google_vision"):
# Test Google Cloud Vision OCR API
url = f"https://vision.googleapis.com/v1/images:annotate?key={clean_key}"
url = "https://vision.googleapis.com/v1/images:annotate"
payload = {
"requests": [
{
@@ -112,7 +114,9 @@ async def validate_api_key(
]
}
resp = await client.post(
url, json=payload, headers={"Content-Type": "application/json"}
url,
json=payload,
headers={"Content-Type": "application/json", "x-goog-api-key": clean_key},
)
if resp.status_code == 200:
return True, "Google Cloud Vision OCR API key verified successfully."
+6 -5
View File
@@ -376,17 +376,18 @@ def format_read_note_success(
start_line: int | None = None,
end_line: int | None = None,
) -> str:
"""Formats the success message for reading a note."""
range_str = ""
"""The read result: a one-line header naming the note and its extent, then the content."""
if start_line is not None or end_line is not None:
range_str = f" (lines {start_line or 1} to {end_line or 'end'})"
return f"Successfully read note '{key}'{range_str}. '{key}' note content:\n{content}"
extent = f"lines {start_line or 1} to {end_line or 'end'}"
else:
extent = f"{len(content.splitlines())} lines"
return f"Note '{key}' ({extent}):\n{content}"
def format_read_note_failure(key: str, error: str) -> str:
"""Formats the failure message for reading a note."""
if "not found" in error.lower():
return ToolFailure(f"Note '{key}' not found in scratchpad.")
return ToolFailure(f"Note '{key}' not found. Use list_notes to see the saved keys.")
return ToolFailure(f"Failed to read note '{key}': {error}")
+65 -15
View File
@@ -314,18 +314,23 @@ def render_check_line(item: CheckItem, indent: int = 2) -> str:
return f"{prefix}- {item.kind}{suffix}: {item.text}"
def restore_missing_check_items(content_before: str, content_after: str) -> str | None:
def restore_missing_check_items(
content_before: str,
content_after: str,
items: list[CheckItem] | None = None,
) -> str | None:
"""Deterministic guard: merge check lines deleted/rewritten by a plan write
back into the new content.
Returns the merged plan text, or ``None`` when nothing is missing. Lines
whose parent subgoal still exists are re-anchored directly under it; lines
whose parent subgoal was removed are converted to task-level ``@end`` items
appended at the end of the plan.
appended at the end of the plan. ``items`` restricts the merge to the given
missing items (the caller may waive some of them).
"""
before = parse_plan(content_before)
after = parse_plan(content_after)
missing = missing_check_items(before, after)
missing = missing_check_items(before, after) if items is None else list(items)
if not missing:
return None
@@ -438,23 +443,68 @@ Every checklist line MUST match `<indent>- [<status>] <text>`:
- `{CONTINUOUS_LOOP_TAG}` tags an UNBOUNDED continuous-monitoring milestone: it must stay `[{STATUS_ACTIVE}]` and can never be marked `[{STATUS_DONE}]`, deleted, or untagged by you — the system mechanically rejects such edits. Only an explicit external stop signal injected by the user unlocks its completion; that signal is an external interruption delivered by the system, never something inferred from the screen or the plan.
- The task terminates only when every top-level milestone is `[{STATUS_DONE}]`."""
_CHECK_GRAMMAR_EXTENSION = """### Check Line Grammar (declared verification standards)
A top-level milestone may carry indented check lines, and the plan may end with task-level check lines at zero indentation:
- ` - verify: <expected state>` — an acceptance criterion for its parent milestone. It is judged by an independent Checker at the moment the milestone is marked completed; a failed verify triggers a repair loop.
- ` - assert: <expected observation>` — a test assertion. It is judged by the independent Checker and a failure is recorded verbatim as a legitimate test result; assertions are NEVER repaired, worked around, or satisfied by constructing state.
- `- verify@end: ...` / `- assert@end: ...` — the `@end` suffix defers judgment to task exit, using the final device state. Items without `@end` are judged with the evidence available when their anchored milestone completes.
- Capability boundary (declare honestly, never promise more): all checks are POST-HOC audits over recorded evidence. They detect and record violations after the fact; they CANNOT block an action from happening, so "B must not run before A is confirmed" ordering rules are verified retroactively, not enforced. Transient prompts that were never captured in the execution history cannot be recovered — prefer expressing expectations as persistently probeable state.
- ` - finding: ...` — a SYSTEM-authored standing headline pinned under a milestone whose verify criterion failed. You never author these lines: the system re-renders them on every plan write (deleting one only makes it reappear) and removes them automatically once the verify passes again or its repair budget is exhausted. Each finding names a `checker-...` note holding the full details and repair log — read it with `read_note`; note keys with the `checker-` prefix are reserved for the system and are read-only for you."""
_CHECK_GRAMMAR_HEADER = """### Check Line Grammar (declared verification standards)
A top-level milestone may carry indented check lines, and the plan may end with task-level check lines at zero indentation:"""
#: Judgment-moment wording of the check lines, keyed by the midway gate.
_VERIFY_JUDGED_MIDWAY = (
"It is judged by an independent Checker at the moment the milestone is marked"
" completed; a failed verify reopens the milestone for repair."
)
_VERIFY_JUDGED_AT_EXIT = (
"It is judged once at task exit from the recorded evidence around the"
" milestone's completion; there is no midway repair loop."
)
_ASSERT_JUDGED_MIDWAY = (
"It is judged by the independent Checker when its milestone completes (at task exit for `@end`)"
)
_ASSERT_JUDGED_AT_EXIT = "It is judged once at task exit from the recorded evidence"
_CHECK_GRAMMAR_COMMON = """- `- verify@end: ...` / `- assert@end: ...` — the `@end` suffix defers judgment to task exit, using the final device state. Items without `@end` are judged from the evidence recorded around their anchored milestone's completion.
- Capability boundary (declare honestly, never promise more): all checks are POST-HOC audits over recorded evidence. They detect and record violations after the fact; they CANNOT block an action from happening, so "B must not run before A is confirmed" ordering rules are verified retroactively, not enforced. Transient prompts that were never captured in the execution history cannot be recovered — prefer expressing expectations as persistently probeable state."""
_FINDING_GRAMMAR = """- ` - finding: ...` — a SYSTEM-authored standing headline pinned under a milestone whose verify criterion failed. You never author these lines: the system re-renders them on every plan write (deleting one only makes it reappear) and removes them automatically once the verify passes again or its repair budget is exhausted. Each finding names a `checker-...` note holding the full details and repair log — read it with `read_note`; note keys with the `checker-` prefix are reserved for the system and are read-only for you."""
def render_plan_grammar_spec(include_checks: bool) -> str:
def _render_check_grammar(*, midway: bool, final: bool) -> str:
"""The check-line grammar worded for the active gates.
Midway on: verify lines are judged at milestone completion and can reopen
the milestone; finding lines exist. Midway off (final only): every line is
judged once at task exit, and no repair loop or finding line exists, so
neither is taught.
"""
verify_when = _VERIFY_JUDGED_MIDWAY if midway else _VERIFY_JUDGED_AT_EXIT
assert_when = _ASSERT_JUDGED_MIDWAY if midway else _ASSERT_JUDGED_AT_EXIT
lines = [
_CHECK_GRAMMAR_HEADER,
"- ` - verify: <expected state>` — an acceptance criterion for its parent"
f" milestone. {verify_when}",
f"- ` - assert: <expected observation>` — a test assertion. {assert_when} and a"
" failure is recorded verbatim as a legitimate test result; assertions are"
" NEVER repaired, worked around, or satisfied by constructing state.",
_CHECK_GRAMMAR_COMMON,
]
if midway:
lines.append(_FINDING_GRAMMAR)
return "\n".join(lines)
def render_plan_grammar_spec(
include_checks: bool | None = None, *, midway: bool = False, final: bool = False
) -> str:
"""Renders the grammar spec taught to agents.
``include_checks=False`` (both check gates disabled) returns exactly the
base grammar — zero context pollution from the checking feature.
``midway`` / ``final`` are the two check gates
(``ExecutionSetup.midway_checks_enabled`` / ``final_check_enabled``); the
check-line grammar is worded for whichever is active. Both off returns
exactly the base grammar — zero context pollution from the checking
feature. ``include_checks`` is a compatibility shim meaning "both gates".
"""
if include_checks:
return _PLAN_GRAMMAR_BASE + "\n\n" + _CHECK_GRAMMAR_EXTENSION
if include_checks is not None:
midway = final = bool(include_checks)
if midway or final:
return _PLAN_GRAMMAR_BASE + "\n\n" + _render_check_grammar(midway=midway, final=final)
return _PLAN_GRAMMAR_BASE
+134 -24
View File
@@ -119,8 +119,6 @@ def format_action_clean(action_obj) -> str:
direction = action_obj.get("intent")
duration = action_obj.get("duration") or action_obj.get("duration_ms")
duration_str = f" over {duration}ms" if duration else ""
if direction:
return f"Swiped {direction}{duration_str}"
start_coords = action_obj.get("start_coordinates")
end_coords = action_obj.get("end_coordinates")
@@ -129,6 +127,14 @@ def format_action_clean(action_obj) -> str:
if isinstance(coords_list, list) and len(coords_list) == 4:
start_coords = coords_list[:2]
end_coords = coords_list[2:]
if not start_coords or not end_coords:
start_coords = action_obj.get("normalized_start_coordinates")
end_coords = action_obj.get("normalized_end_coordinates")
if direction:
# A directional swipe is what the agent issued; the recorded path
# (when any) is the detail, not the headline.
path = f" (from {start_coords} to {end_coords})" if start_coords and end_coords else ""
return f"Swiped {direction}{path}{duration_str}"
dragged = _target_label(action_obj, fallback="")
dragged_str = f" {dragged}" if dragged else ""
return f"Swiped{dragged_str} from {start_coords} to {end_coords}{duration_str}"
@@ -176,8 +182,32 @@ def format_action_clean(action_obj) -> str:
)
return f"Tapped sequence of targets: {labelled}"
return f"Tapped sequence of targets: {seq}"
elif act_type == "wait_for_text":
state = str(action_obj.get("wait_state") or "appear")
timeout = action_obj.get("timeout_ms")
timeout_str = f" (timeout {timeout}ms)" if timeout else ""
return f"Waited for text '{action_obj.get('text', '')}' to {state}{timeout_str}"
elif act_type == "open_link":
return f"Opened link '{action_obj.get('url', '')}'"
elif act_type == "erase_one_char":
return "Erased one character"
elif act_type == "focus_and_clear_text":
return f"Cleared text at {coords}"
else:
return f"Action: {act_type} with args: {json.dumps(action_obj, ensure_ascii=False)}"
# Unknown action: show only the agent-facing arguments, never the
# bookkeeping keys the record carries for other consumers.
internal = {
"action",
"name",
"intent",
"coordinate_space",
"normalized_coordinates",
"normalized_start_coordinates",
"normalized_end_coordinates",
}
shown = {k: v for k, v in action_obj.items() if k not in internal and v is not None}
args_str = f" with args: {json.dumps(shown, ensure_ascii=False)}" if shown else ""
return f"Action: {act_type}{args_str}"
def format_actions_clean(actions) -> str:
@@ -192,6 +222,48 @@ def format_actions_clean(actions) -> str:
return format_action_clean(actions)
#: Past-tense outcome prefix (``format_action_clean``) -> intent-form prefix.
#: Longest prefixes first so "Double tapped" / "Tapped sequence" win over "Tapped".
_INTENT_PREFIXES: tuple[tuple[str, str], ...] = (
("Tapped sequence of targets: ", "tap sequence of targets: "),
("Double tapped ", "double tap "),
("Long pressed ", "long press "),
("Tapped ", "tap "),
("Inputted ", "type "),
("Swiped ", "swipe "),
("Pressed key ", "press key "),
("Launched app ", "launch app "),
("Stopped app ", "stop app "),
("Managed app ", "manage app "),
("Waited for text ", "wait for text "),
("Waited for ", "wait "),
("Opened link ", "open link "),
("Erased one character", "erase one character"),
("Cleared text ", "clear text "),
)
def action_intent_phrase(description: str) -> str:
"""The intent form of a rendered action phrase: ``Tapped 'Wi-Fi row' at
[500, 520]`` -> ``tap 'Wi-Fi row' at [500, 520]``.
``format_action_clean`` renders actions as past-tense outcomes, which
reads as success; a failure or incident context must describe what the
agent *tried* to do instead. Phrases without a known prefix (bursts,
unknown action types) are returned unchanged.
"""
text = str(description or "")
for outcome, intent in _INTENT_PREFIXES:
if text.startswith(outcome):
return intent + text[len(outcome) :]
return text
def format_action_intent(action_obj) -> str:
"""Intent-form rendering of one action (``launch app 'X'``, ``swipe up``)."""
return action_intent_phrase(format_action_clean(action_obj))
def _is_terminal_attempt_failure(attempts) -> bool:
if not attempts:
return False
@@ -216,11 +288,17 @@ def failed_execution_error(result_obj) -> str | None:
def format_incident_clean(incident: dict) -> str:
"""One-line rendering of an execution incident for history and result lines."""
"""One-line rendering of an execution incident for history and result lines.
The action is quoted in intent form (``launch app 'X'``): it did not
happen, so the past-tense outcome phrase would misreport it as done.
"""
kind = incident.get("kind") or "exec_error"
category = incident.get("category") or "general"
consecutive = incident.get("consecutive_failures") or 1
description = incident.get("action_description") or format_action_clean(incident.get("action"))
description = action_intent_phrase(
incident.get("action_description") or format_action_clean(incident.get("action"))
)
reason = str(incident.get("reason") or "").strip()
burst_size = int(incident.get("burst_size") or 1)
index = int(incident.get("action_index") or 0)
@@ -296,6 +374,21 @@ def safe_parse_validation_result(result: Any) -> list:
return []
#: Block labels of a detailed / replayed step. Role-neutral on purpose: the
#: same renderer replays a Flash turn (no Operator) and a Pro Operator turn,
#: and every Pro agent (Checker, Diagnoser, Outputter) reads them.
REASONING_BLOCK_LABEL = "[Reasoning & tool calls]"
ACTION_BLOCK_LABEL = "[Action]"
BURST_BLOCK_LABEL = "[Fast-Action Burst]"
#: Continuation lines of a " - " bullet stay inside the bullet.
_BULLET_CONTINUATION = "\n "
def _indent_continuation(text: str) -> str:
return text.replace("\n", _BULLET_CONTINUATION)
#: Limit tool-result text in the live context, which is rebuilt every turn.
LIVE_RESULT_CHARS = 120
@@ -544,9 +637,27 @@ def _burst_member_status(result: Any, index: int) -> str:
return f" (FAILED: {last.strip()})"
def replay_time_label(step: dict, session_start: Any) -> str:
"""The step-header clock of a replay: the session-relative ``T+mm:ss``
offset every agent prompt and ``search_history`` use, or the legacy
``Start: 12.3s`` text when no session start / step timestamp is known."""
ts = step.get("timestamp")
if (
isinstance(session_start, (int, float))
and not isinstance(session_start, bool)
and isinstance(ts, (int, float))
and not isinstance(ts, bool)
):
from artemis.memory.chunking import step_offset_label
return step_offset_label(step, float(session_start))
return f"Start: {step.get('relative_time') or 'N/A'}"
def render_step_replay(
step: dict,
*,
session_start: Any = None,
result_chars: int = REPLAY_RESULT_CHARS,
include_summary: bool = True,
) -> str:
@@ -556,6 +667,9 @@ def render_step_replay(
/ ``get_agent_friendly_steps``): coordinates already normalized, traces
already expanded into ``interleaved_events``. Used by ``replay_steps`` and
the MCP trace inspector. Screenshots are fetched separately.
``session_start`` (the reader's ``session_start_time``) makes the header
carry the ``T+mm:ss`` session clock; see :func:`replay_time_label`.
"""
return _render_step_detailed(
step=step,
@@ -566,6 +680,7 @@ def render_step_replay(
is_most_recent=False,
result_chars=result_chars,
include_summary=include_summary,
time_label=replay_time_label(step, session_start),
)
@@ -597,15 +712,17 @@ def _render_step_detailed(
*,
result_chars: int = LIVE_RESULT_CHARS,
include_summary: bool = False,
time_label: str | None = None,
) -> str:
# 1. Step Header
# 1. Step Header (``time_label`` overrides the live window's "Start: 12.3s")
status_str = "Most Recent Step, " if is_most_recent else ""
clock = time_label or f"Start: {relative_time}"
# In the live window the summary is omitted from the detailed view header
# as it is already fully detailed below; replay callers opt in because the
# summary is the Operator's own claim about the step, which an auditor
# compares against the evidence that follows.
step_line = f"- **Step {step['step_number']} ({status_str}Start: {relative_time})**"
step_line = f"- **Step {step['step_number']} ({status_str}{clock})**"
if include_summary:
screen_line = _screen_description_line(step, summary)
if screen_line:
@@ -695,9 +812,10 @@ def _render_step_detailed(
if raw_thinking and raw_thinking.strip() and not thought_exists(raw_thinking):
operator_events.append({"type": "thought", "content": raw_thinking.strip()})
# 4. Render Operator Loop
# 4. Render the acting model's reasoning and tool calls (role-neutral: the
# same block replays a Flash turn and a Pro Operator turn)
if operator_events:
step_line += "\n * [Operator Decision Loop]:"
step_line += f"\n * {REASONING_BLOCK_LABEL}:"
ACTION_TOOLS = {
"click",
"swipe",
@@ -722,7 +840,7 @@ def _render_step_detailed(
)
cleaned = re.sub(r"</?thought>", "", cleaned, flags=re.DOTALL).strip()
if cleaned:
step_line += f"\n - {cleaned}"
step_line += f"\n - {_indent_continuation(cleaned)}"
elif e_type == "tool_call":
if name in ACTION_TOOLS:
continue
@@ -735,17 +853,13 @@ def _render_step_detailed(
if formatted:
step_line += f"\n - [Tool Call]: {formatted}"
# Check if the execution failed
exec_error = None
if isinstance(result, dict) and result.get("status") in ("failed", "error"):
exec_error = failed_execution_error(result)
# 5. Render Planned Action (single vetted action or fast-action burst)
# 5. Render the action the step executed (single vetted action or a burst).
# A failed action's error is printed once, on the [Result] line below.
if action:
actions = action if isinstance(action, list) else [action]
if len(actions) > 1:
step_line += (
f"\n * [Planned Fast-Action Burst]: {len(actions)} actions fired back to"
f"\n * {BURST_BLOCK_LABEL}: {len(actions)} actions fired back to"
" back without the safety net"
)
for i, member in enumerate(actions):
@@ -756,15 +870,11 @@ def _render_step_detailed(
action_clean = format_action_clean(actions[0])
if intercepted:
step_line += (
f"\n * [Planned Action]: {action_clean}"
f"\n * {ACTION_BLOCK_LABEL}: {action_clean}"
" (Intercepted by Pre-Execution Safety Net)"
)
elif exec_error:
step_line += (
f"\n * [Planned Action]: {action_clean} -> (Execution failed: {exec_error})"
)
else:
step_line += f"\n * [Planned Action]: {action_clean}"
step_line += f"\n * {ACTION_BLOCK_LABEL}: {action_clean}"
# 6. Render Pre-Execution Safety Net
if intercepted:
@@ -892,7 +1002,7 @@ def build_plan_and_history(
# Ensure the last step is always in detailed_step_ids
detailed_step_ids.add(full_info_step_id)
# M4: chunk blocks replace the per-step lines of already-chunked
# Chunk blocks replace the per-step lines of already-chunked
# ranges. Chunks only ever cover frozen (old) turns, so they are
# emitted first, in chronological order; steps that must render
# detailed (recent window / most-recent step) are never suppressed.