feat: unify history tools and improve execution reliability
This commit is contained in:
somew1nd committed
1 parent
d23a4f6472
commit
2fea9865ee
116 files changed
+8236
-2448
No files matched your search
@@ -273,8 +273,8 @@ Evaluated on [AndroidWorld](https://github.com/google-research/android_world)
|
||||
|
||||
ARTEMIS supports two execution profiles tailored for different automation requirements:
|
||||
|
||||
* **Flash Profile (`--profile flash`)**: Fast and token-efficient reactive loop (~3–5s per step): one model observes the live screen, thinks, and acts, with no graph orchestration. Ideal for routine, deterministic UI tasks. The loop is unbounded by default (`agent.flash.max_turns`, 0 = unlimited) because history is compressed rather than capped: Flash shares the Pro session transcript ledger (session-relative `T+mm:ss` clock, screenshots folded into visual summaries, older steps chunked into eras and recallable on demand via `recall_history`) and can query the session recording through `video_analyzer`. Transient UI (auto-fading control bars, toasts) is handled by chaining taps into one `click_sequence`. *Limitations*: No task plan or notes, no pre-execution safety net, no checkpoint verification or final report, and no ADB shell.
|
||||
* **Pro Profile (`--profile pro`)**: Our most capable architecture (~15–40s per step), built as a multi-agent graph. A **Planner** maintains a living Markdown task plan with milestones and `verify` / `assert` check items; the **Operator** executes it with the full toolset (Explorer grounding whose `flash` / `pro` / `ultra` tier is a user setting per profile — `pro.explorer.mode` / `flash.explorer_mode` in `config/artemis.jsonc` or `--explorer-pro-mode` — never chosen by the agent; notes, history recall, video analysis, ADB diagnostics). Every single action passes a pre-execution **Safety Net** (XML-first, pixel fallback), while multi-action **fast-action bursts** fire back to back to beat turn latency on transient UI. A blocked or failed action opens an **execution incident** that stays in the Operator's context until a later action succeeds, so recovery is handled by the Operator itself with no separate repair agent. A read-only **Checker** verifies plan checkpoints and runs an exit final review against the original goal (`--verification-level`: `off` / `final` (default) / `checkpoints` / `strict`), and plan milestone edits get an advisory review. Handles 100+ step long-horizon workflows, `[Loop:continuous]` monitoring, and an optional written report.
|
||||
* **Flash Profile (`--profile flash`)**: Fast and token-efficient reactive loop (~3–5s per step): one model observes the live screen, thinks, and acts, with no graph orchestration. Ideal for routine, deterministic UI tasks. The loop is unbounded by default (`agent.flash.max_turns`, 0 = unlimited) because history is compressed rather than capped: Flash shares the Pro session transcript ledger (session-relative `T+mm:ss` clock, screenshots folded into visual summaries, older steps chunked into eras and recallable on demand via `search_history` / `replay_steps`) and can query the session recording through `video_analyzer`. Transient UI (auto-fading control bars, toasts) is handled by chaining taps into one `click_sequence`. *Limitations*: No task plan or notes, no pre-execution safety net, no checkpoint verification or final report, and no ADB shell.
|
||||
* **Pro Profile (`--profile pro`)**: Our most capable architecture (~15–40s per step), built as a multi-agent graph. A **Planner** maintains a living Markdown task plan with milestones and `verify` / `assert` check items; the **Operator** executes it with the full toolset (Explorer grounding whose `flash` / `pro` / `ultra` tier is a user setting per profile — `pro.explorer.mode` / `flash.explorer_mode` in `config/artemis.jsonc` or `--explorer-pro-mode` — never chosen by the agent; notes, history recall, video analysis, ADB diagnostics). Every single action passes a pre-execution **Safety Net** (XML-first, pixel fallback), while multi-action **fast-action bursts** vet only their first member and fire the rest back to back to beat turn latency on transient UI. A blocked or failed action opens an **execution incident** that stays in the Operator's context until a later action succeeds, so recovery is handled by the Operator itself with no separate repair agent. A read-only **Checker** verifies plan checkpoints and runs an exit final review against the original goal (`--verification-level`: `off` / `final` (default) / `checkpoints` / `strict`), and plan milestone edits get an advisory review. Handles 100+ step long-horizon workflows, `[Loop:continuous]` monitoring, and an optional written report.
|
||||
|
||||
## Roadmap
|
||||
|
||||
|
||||
+2
-2
@@ -275,8 +275,8 @@ if __name__ == "__main__":
|
||||
|
||||
ARTEMIS 提供两种运行模式以适应不同的自动化需求：
|
||||
|
||||
* **Flash 模式 (`--profile flash`)**：轻量极速且省 Token 的响应式循环（单步约 3–5 秒）：单个模型观察实时屏幕、思考、执行，不经过图编排，适合常规确定性 UI 操作。默认不限步数（`agent.flash.max_turns`，0 表示不限），因为历史是被压缩而不是被截断：Flash 与 Pro 共用同一套会话记录账本（相对测试时间 `T+mm:ss`、截图折叠为视觉摘要、更早的步骤分块归档并可通过 `recall_history` 按需召回），并可调用 `video_analyzer` 分析整段会话录屏；对自动消失的控制栏、toast 等瞬态控件，用 `click_sequence` 把多次点击串成一个原子序列。*局限性*：没有任务计划与笔记、没有执行前安全网、没有检查点校验与最终报告、不能执行 ADB 命令。
|
||||
* **Pro 模式 (`--profile pro`)**：最强大的架构（单步约 15–40 秒），由多智能体图编排：**Planner** 维护一份带里程碑和 `verify` / `assert` 检查项的 Markdown 任务计划；**Operator** 按计划执行，拥有全套工具（Explorer 元素定位，其 `flash` / `pro` / `ultra` 三档感知深度是按运行模式设置的用户配置——`config/artemis.jsonc` 里的 `pro.explorer.mode` / `flash.explorer_mode` 或 `--explorer-pro-mode`——智能体自身不会选择档位；笔记、历史召回、视频分析、ADB 诊断）。单个动作执行前都经过 **Safety Net** 校验（XML 优先、像素兜底），而一轮里的多个动作则作为**快速动作序列 (fast-action burst)** 连发，赶在瞬态控件消失前完成操作。动作被拦截或失败时会打开一条**执行事故 (execution incident)**，持续留在 Operator 的上下文里直到后续动作执行成功，恢复由 Operator 自己完成，不再有独立的修复智能体。只读的 **Checker** 校验计划中的检查点，并在退出前对照原始目标做终审（`--verification-level`：`off` / `final`（默认）/ `checkpoints` / `strict`），计划里程碑的改动会得到建议式复核。支持 100+ 步的长程复杂业务流、`[Loop:continuous]` 持续监控以及可选的书面报告。
|
||||
* **Flash 模式 (`--profile flash`)**：轻量极速且省 Token 的响应式循环（单步约 3–5 秒）：单个模型观察实时屏幕、思考、执行，不经过图编排，适合常规确定性 UI 操作。默认不限步数（`agent.flash.max_turns`，0 表示不限），因为历史是被压缩而不是被截断：Flash 与 Pro 共用同一套会话记录账本（相对测试时间 `T+mm:ss`、截图折叠为视觉摘要、更早的步骤分块归档并可通过 `search_history` / `replay_steps` 按需召回），并可调用 `video_analyzer` 分析整段会话录屏；对自动消失的控制栏、toast 等瞬态控件，用 `click_sequence` 把多次点击串成一个原子序列。*局限性*：没有任务计划与笔记、没有执行前安全网、没有检查点校验与最终报告、不能执行 ADB 命令。
|
||||
* **Pro 模式 (`--profile pro`)**：最强大的架构（单步约 15–40 秒），由多智能体图编排：**Planner** 维护一份带里程碑和 `verify` / `assert` 检查项的 Markdown 任务计划；**Operator** 按计划执行，拥有全套工具（Explorer 元素定位，其 `flash` / `pro` / `ultra` 三档感知深度是按运行模式设置的用户配置——`config/artemis.jsonc` 里的 `pro.explorer.mode` / `flash.explorer_mode` 或 `--explorer-pro-mode`——智能体自身不会选择档位；笔记、历史召回、视频分析、ADB 诊断）。单个动作执行前都经过 **Safety Net** 校验（XML 优先、像素兜底），而一轮里的多个动作则作为**快速动作序列 (fast-action burst)** 连发：只校验第一个动作，其余动作不经校验背靠背执行，赶在瞬态控件消失前完成操作。动作被拦截或失败时会打开一条**执行事故 (execution incident)**，持续留在 Operator 的上下文里直到后续动作执行成功，恢复由 Operator 自己完成，不再有独立的修复智能体。只读的 **Checker** 校验计划中的检查点，并在退出前对照原始目标做终审（`--verification-level`：`off` / `final`（默认）/ `checkpoints` / `strict`），计划里程碑的改动会得到建议式复核。支持 100+ 步的长程复杂业务流、`[Loop:continuous]` 持续监控以及可选的书面报告。
|
||||
|
||||
## 路线图
|
||||
|
||||
|
||||
@@ -12,6 +12,7 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sqlite3
|
||||
@@ -301,6 +302,89 @@ class SessionRepository:
|
||||
result.setdefault(session_id, []).append(name)
|
||||
return result
|
||||
|
||||
def get_session_usage(self, session_id: str) -> dict[str, Any]:
|
||||
"""Aggregate measured LLM usage for one session.
|
||||
|
||||
Sums every ``llm_usage`` trace the token meter recorded (all agents,
|
||||
Flash and Pro alike) and reports the prompt size of the latest
|
||||
Operator/FlashRunner call as the live executor context. Also echoes
|
||||
the profile and Pro tuning the run was started with, read from the
|
||||
session's stored ``device_info``.
|
||||
"""
|
||||
from artemis.services.token_meter import (
|
||||
OPERATOR_CONTEXT_WINDOW_TOKENS,
|
||||
OPERATOR_NODE_NAMES,
|
||||
)
|
||||
|
||||
def _int(value: Any) -> int:
|
||||
try:
|
||||
return int(value or 0)
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
llm_calls = prompt = completion = total = cached = 0
|
||||
operator_context: int | None = None
|
||||
operator_context_at: float | None = None
|
||||
with db_session(self.db_path) as conn:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute(
|
||||
"SELECT payload, timestamp FROM traces WHERE session_id = ?"
|
||||
" AND type = 'llm_call' AND name = 'llm_usage' ORDER BY timestamp ASC",
|
||||
(session_id,),
|
||||
)
|
||||
for row in cursor.fetchall():
|
||||
raw = row["payload"]
|
||||
try:
|
||||
payload = json.loads(raw) if isinstance(raw, str) else raw
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
if not isinstance(payload, dict):
|
||||
continue
|
||||
p = _int(payload.get("prompt_tokens"))
|
||||
c = _int(payload.get("completion_tokens"))
|
||||
t = _int(payload.get("total_tokens")) or (p + c)
|
||||
if t <= 0:
|
||||
continue
|
||||
llm_calls += 1
|
||||
prompt += p
|
||||
completion += c
|
||||
total += t
|
||||
cached += _int(payload.get("cached_tokens"))
|
||||
node = str(payload.get("node") or "").strip().lower()
|
||||
if node in OPERATOR_NODE_NAMES:
|
||||
operator_context = _int(payload.get("context_base_tokens")) or p
|
||||
operator_context_at = row["timestamp"]
|
||||
|
||||
cursor.execute("SELECT device_info FROM sessions WHERE session_id = ?", (session_id,))
|
||||
session_row = cursor.fetchone()
|
||||
|
||||
profile: str | None = None
|
||||
run_tuning: dict[str, Any] | None = None
|
||||
if session_row and session_row["device_info"]:
|
||||
try:
|
||||
d_info = json.loads(session_row["device_info"])
|
||||
except (ValueError, TypeError):
|
||||
d_info = None
|
||||
if isinstance(d_info, dict):
|
||||
profile = d_info.get("profile") or None
|
||||
tuning = d_info.get("run_tuning")
|
||||
if isinstance(tuning, dict) and tuning:
|
||||
run_tuning = dict(tuning)
|
||||
|
||||
return {
|
||||
"session_id": session_id,
|
||||
"llm_calls": llm_calls,
|
||||
"prompt_tokens": prompt,
|
||||
"completion_tokens": completion,
|
||||
"total_tokens": total,
|
||||
"cached_tokens": cached,
|
||||
"operator_context_tokens": operator_context,
|
||||
"operator_context_window_tokens": OPERATOR_CONTEXT_WINDOW_TOKENS,
|
||||
"operator_context_updated_at": operator_context_at,
|
||||
"profile": profile,
|
||||
"run_tuning": run_tuning,
|
||||
}
|
||||
|
||||
def get_session_by_id(self, session_id: str) -> dict[str, Any] | None:
|
||||
try:
|
||||
with db_session(self.db_path) as conn:
|
||||
|
||||
@@ -176,6 +176,15 @@ async def get_session_details(session_id: str):
|
||||
return dict(row)
|
||||
|
||||
|
||||
@router.get("/api/sessions/{session_id}/usage")
|
||||
async def get_session_usage(session_id: str):
|
||||
"""Session-wide LLM token totals, live executor context size and run tuning."""
|
||||
try:
|
||||
return await asyncio.to_thread(session_repo.get_session_usage, session_id)
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/api/sessions/{session_id}/tree")
|
||||
async def get_tree(session_id: str):
|
||||
try:
|
||||
|
||||
@@ -397,15 +397,19 @@ class MediaService:
|
||||
or not segment_path.is_file()
|
||||
):
|
||||
continue
|
||||
segments.append(
|
||||
{
|
||||
segment = {
|
||||
"url": cls.path_to_video_url(segment_path),
|
||||
"start": float(item.get("start", 0)),
|
||||
"duration": float(item.get("duration", 0)),
|
||||
"width": int(item.get("width", 0)),
|
||||
"height": int(item.get("height", 0)),
|
||||
}
|
||||
)
|
||||
# Manifest v2 carries the real session-relative timeline so the
|
||||
# UI can seek by step time across scrcpy restart gaps.
|
||||
if item.get("offset_ms") is not None and item.get("duration_ms") is not None:
|
||||
segment["offset_ms"] = int(item["offset_ms"])
|
||||
segment["duration_ms"] = int(item["duration_ms"])
|
||||
segments.append(segment)
|
||||
return segments
|
||||
except (OSError, KeyError, TypeError, ValueError, json.JSONDecodeError):
|
||||
return []
|
||||
@@ -536,18 +540,12 @@ class MediaService:
|
||||
return "No task plan created yet."
|
||||
|
||||
@staticmethod
|
||||
def get_session_checks(session_id: str) -> dict[str, Any]:
|
||||
"""Checker material of one session for UI backfill: the append-only
|
||||
verdict ledger (``check_ledger.jsonl``) and the machine-readable run
|
||||
outcome (``run_outcome.json``), both written by
|
||||
``artemis.graph.checkpoints``. Missing files mean "no checks ran".
|
||||
"""
|
||||
session_dir = TRACES_PATH / session_id
|
||||
def _read_jsonl_records(path: Path, label: str, session_id: str) -> list[dict[str, Any]]:
|
||||
records: list[dict[str, Any]] = []
|
||||
ledger_path = session_dir / "check_ledger.jsonl"
|
||||
if ledger_path.is_file():
|
||||
if not path.is_file():
|
||||
return records
|
||||
try:
|
||||
for line in ledger_path.read_text(encoding="utf-8").splitlines():
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
@@ -558,7 +556,26 @@ class MediaService:
|
||||
if isinstance(rec, dict):
|
||||
records.append(rec)
|
||||
except (OSError, ValueError) as e:
|
||||
logger.warning(f"Failed to read check ledger for {session_id}: {e}")
|
||||
logger.warning(f"Failed to read {label} for {session_id}: {e}")
|
||||
return records
|
||||
|
||||
@staticmethod
|
||||
def get_session_checks(session_id: str) -> dict[str, Any]:
|
||||
"""Checker material of one session for UI backfill: the append-only
|
||||
verdict ledger (``check_ledger.jsonl``), the per-attempt transcripts of
|
||||
the Checker's streamed reasoning (``check_streams.jsonl``: one record
|
||||
per attempt with timestamped ``segments``) and the machine-readable run
|
||||
outcome (``run_outcome.json``), all written by the Checker /
|
||||
``artemis.graph.checkpoints``. Missing files mean "no checks ran"
|
||||
(sessions recorded before transcripts existed simply have no streams).
|
||||
"""
|
||||
session_dir = TRACES_PATH / session_id
|
||||
records = MediaService._read_jsonl_records(
|
||||
session_dir / "check_ledger.jsonl", "check ledger", session_id
|
||||
)
|
||||
streams = MediaService._read_jsonl_records(
|
||||
session_dir / "check_streams.jsonl", "check streams", session_id
|
||||
)
|
||||
|
||||
run_outcome: dict[str, Any] | None = None
|
||||
outcome_path = session_dir / "run_outcome.json"
|
||||
@@ -570,7 +587,7 @@ class MediaService:
|
||||
except (OSError, ValueError) as e:
|
||||
logger.warning(f"Failed to read run outcome for {session_id}: {e}")
|
||||
|
||||
return {"records": records, "run_outcome": run_outcome}
|
||||
return {"records": records, "streams": streams, "run_outcome": run_outcome}
|
||||
|
||||
@staticmethod
|
||||
def get_session_notes_content(session_id: str) -> dict[str, str]:
|
||||
|
||||
Generated
+1
-1
@@ -224,7 +224,7 @@
|
||||
},
|
||||
"node_modules/@angular/cdk": {
|
||||
"version": "22.1.5",
|
||||
"resolved": "http://airlock-proxy.uplink.goog:999/npm/artifact-foundry-prod/ah-3p-staging-npm/@angular/cdk/-/cdk-22.1.5.tgz",
|
||||
"resolved": "https://registry.npmjs.org/@angular/cdk/-/cdk-22.1.5.tgz",
|
||||
"integrity": "sha512-1tYAeI5KKIqNGNvyDuMM0Drs2FU3M4B25Q1BlAnwtZe3wvrpVETAGXgHaBX3Hs/pvWfgMNwgbcHkxlKVUBNXMA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
|
||||
@@ -387,17 +387,95 @@ limitations under the License.
|
||||
<!-- Floating Top-Right Badges & Toolbar (Pinned overlay, hidden in empty/initial state) -->
|
||||
@if (consolidatedBlocks().length > 0 || showPlanningLoader()) {
|
||||
<div class="stream-floating-toolbar">
|
||||
<!-- Active Architecture Info Chip (Informational Metadata, Not a Button) -->
|
||||
@if (agentService.activeModel(); as model) {
|
||||
<div
|
||||
<!-- Active Architecture Chip: click opens the run info popover -->
|
||||
@if (agentService.viewedModel(); as model) {
|
||||
<button
|
||||
type="button"
|
||||
#runInfoOrigin="cdkOverlayOrigin"
|
||||
cdkOverlayOrigin
|
||||
class="model-info-chip"
|
||||
[ngClass]="getModelClass(model.name)"
|
||||
[class.active]="isRunInfoOpen()"
|
||||
[title]="getArchitectureTooltip(model)"
|
||||
aria-haspopup="dialog"
|
||||
[attr.aria-expanded]="isRunInfoOpen()"
|
||||
(click)="toggleRunInfo($event)"
|
||||
>
|
||||
<span class="material-symbols-outlined model-icon">{{ getModelIcon(model.name) }}</span>
|
||||
<span class="model-label">Architecture:</span>
|
||||
<span class="model-name">{{ getModelDisplayName(model.name) }}</span>
|
||||
</button>
|
||||
|
||||
<ng-template
|
||||
cdkConnectedOverlay
|
||||
[cdkConnectedOverlayOrigin]="runInfoOrigin"
|
||||
[cdkConnectedOverlayOpen]="isRunInfoOpen()"
|
||||
[cdkConnectedOverlayHasBackdrop]="true"
|
||||
cdkConnectedOverlayBackdropClass="cdk-overlay-transparent-backdrop"
|
||||
(backdropClick)="isRunInfoOpen.set(false)"
|
||||
[cdkConnectedOverlayPositions]="dropdownPositions"
|
||||
>
|
||||
<div class="vscode-dropdown-menu run-info-dropdown" role="dialog" aria-label="Run info">
|
||||
<div class="run-info-header">
|
||||
<span class="material-symbols-outlined" [class.is-pro]="runInfoIsPro()">{{ getModelIcon(model.name) }}</span>
|
||||
<span class="run-info-title">{{ getModelDisplayName(model.name) }}</span>
|
||||
@if (model.id) {
|
||||
<span class="run-info-model" [title]="model.id">{{ model.id }}</span>
|
||||
}
|
||||
</div>
|
||||
|
||||
<div class="run-info-row">
|
||||
<span class="run-info-label">Elapsed</span>
|
||||
<span class="run-info-value">{{ runInfoElapsed() }}</span>
|
||||
</div>
|
||||
|
||||
<div class="run-info-row">
|
||||
<span class="run-info-label">Tokens</span>
|
||||
@if (runUsage(); as usage) {
|
||||
<span class="run-info-value">
|
||||
{{ formatRunTokens(usage.total_tokens) }}
|
||||
<span class="run-info-sub">{{ formatRunTokens(usage.prompt_tokens) }} in · {{ formatRunTokens(usage.completion_tokens) }} out</span>
|
||||
</span>
|
||||
} @else {
|
||||
<span class="run-info-value is-muted">—</span>
|
||||
}
|
||||
</div>
|
||||
|
||||
<div class="run-info-row is-stacked">
|
||||
<div class="run-info-line">
|
||||
<span class="run-info-label">Context</span>
|
||||
@if (runInfoContextPercent() !== null) {
|
||||
<span class="run-info-value">
|
||||
{{ runInfoContextPercent() }}%
|
||||
<span class="run-info-sub-inline">of {{ formatRunTokens(runUsage()?.operator_context_window_tokens) }}</span>
|
||||
</span>
|
||||
} @else {
|
||||
<span class="run-info-value is-muted">—</span>
|
||||
}
|
||||
</div>
|
||||
<div
|
||||
class="run-info-meter"
|
||||
[class.is-high]="(runInfoContextPercent() ?? 0) >= 70"
|
||||
[class.is-critical]="(runInfoContextPercent() ?? 0) >= 90"
|
||||
>
|
||||
<div class="run-info-meter-fill" [style.width.%]="runInfoContextPercent() ?? 0"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@if (runInfoIsPro()) {
|
||||
<div class="run-info-row">
|
||||
<span class="run-info-label">Result check</span>
|
||||
<span class="run-info-value">{{ runTuningLabel('verify', runInfoTuning()?.verification_level) }}</span>
|
||||
</div>
|
||||
<div class="run-info-row">
|
||||
<span class="run-info-label">Screen reading</span>
|
||||
<span class="run-info-value">{{ runTuningLabel('explore', runInfoTuning()?.explorer_mode) }}</span>
|
||||
</div>
|
||||
} @else {
|
||||
<div class="run-info-note">No extra settings for Flash.</div>
|
||||
}
|
||||
</div>
|
||||
</ng-template>
|
||||
}
|
||||
<!-- Screen Recording Action Button (Interactive Button) -->
|
||||
<button
|
||||
|
||||
@@ -224,21 +224,44 @@ $stream-row-height: 25px;
|
||||
top: 72px;
|
||||
}
|
||||
|
||||
/* Model Info Chip: Informational Tag / Metadata (Not clickable) */
|
||||
/* Model Info Chip: click opens the run info popover */
|
||||
.model-info-chip {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 5px;
|
||||
height: 30px;
|
||||
padding: 0 11px;
|
||||
margin: 0;
|
||||
box-sizing: border-box;
|
||||
background: rgba(255, 255, 255, 0.72);
|
||||
backdrop-filter: blur(16px) saturate(180%);
|
||||
-webkit-backdrop-filter: blur(16px) saturate(180%);
|
||||
border: 1px solid rgba(226, 232, 240, 0.85);
|
||||
border-radius: 15px;
|
||||
cursor: default;
|
||||
cursor: pointer;
|
||||
user-select: none;
|
||||
appearance: none;
|
||||
-webkit-appearance: none;
|
||||
|
||||
&:hover {
|
||||
background: rgba(255, 255, 255, 0.92);
|
||||
border-color: rgba(203, 213, 225, 0.95);
|
||||
}
|
||||
|
||||
&.active {
|
||||
background: #eff6ff;
|
||||
border-color: #bfdbfe;
|
||||
|
||||
.model-icon,
|
||||
.model-name {
|
||||
color: #1d4ed8;
|
||||
}
|
||||
}
|
||||
|
||||
&:focus-visible {
|
||||
outline: 2px solid #93c5fd;
|
||||
outline-offset: 1px;
|
||||
}
|
||||
box-shadow: 0 2px 6px -1px rgba(15, 23, 42, 0.05), inset 0 1px 1px rgba(255, 255, 255, 0.9);
|
||||
font-family: 'Google Sans', 'Google Sans Text', 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1);
|
||||
@@ -3044,6 +3067,137 @@ $stream-row-height: 25px;
|
||||
}
|
||||
}
|
||||
|
||||
/* Run info popover: elapsed, tokens, context share, Pro tuning */
|
||||
&.run-info-dropdown {
|
||||
width: 288px;
|
||||
gap: 0;
|
||||
padding: 12px 14px 10px;
|
||||
|
||||
.run-info-header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 7px;
|
||||
padding-bottom: 9px;
|
||||
border-bottom: 1px solid #e2e8f0;
|
||||
|
||||
.material-symbols-outlined {
|
||||
font-size: 18px;
|
||||
color: #64748b;
|
||||
|
||||
&.is-pro {
|
||||
color: #2563eb;
|
||||
}
|
||||
}
|
||||
|
||||
.run-info-title {
|
||||
font-size: 13px;
|
||||
font-weight: 700;
|
||||
color: #0f172a;
|
||||
}
|
||||
|
||||
.run-info-model {
|
||||
margin-left: auto;
|
||||
max-width: 150px;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
font-family: 'Fira Code', 'Roboto Mono', Consolas, monospace;
|
||||
font-size: 10.5px;
|
||||
color: #94a3b8;
|
||||
}
|
||||
}
|
||||
|
||||
.run-info-row {
|
||||
display: flex;
|
||||
align-items: baseline;
|
||||
justify-content: space-between;
|
||||
gap: 12px;
|
||||
padding: 9px 0;
|
||||
border-bottom: 1px solid #f1f5f9;
|
||||
|
||||
&:last-child {
|
||||
border-bottom: none;
|
||||
}
|
||||
|
||||
&.is-stacked {
|
||||
flex-direction: column;
|
||||
align-items: stretch;
|
||||
gap: 0;
|
||||
}
|
||||
}
|
||||
|
||||
.run-info-line {
|
||||
display: flex;
|
||||
align-items: baseline;
|
||||
justify-content: space-between;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.run-info-label {
|
||||
font-size: 12px;
|
||||
font-weight: 500;
|
||||
color: #64748b;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.run-info-value {
|
||||
font-size: 12.5px;
|
||||
font-weight: 600;
|
||||
color: #0f172a;
|
||||
text-align: right;
|
||||
font-variant-numeric: tabular-nums;
|
||||
|
||||
&.is-muted {
|
||||
color: #cbd5e1;
|
||||
font-weight: 500;
|
||||
}
|
||||
}
|
||||
|
||||
.run-info-sub {
|
||||
display: block;
|
||||
margin-top: 2px;
|
||||
font-size: 11px;
|
||||
font-weight: 400;
|
||||
color: #94a3b8;
|
||||
}
|
||||
|
||||
.run-info-sub-inline {
|
||||
margin-left: 3px;
|
||||
font-size: 11px;
|
||||
font-weight: 400;
|
||||
color: #94a3b8;
|
||||
}
|
||||
|
||||
.run-info-meter {
|
||||
height: 4px;
|
||||
margin-top: 7px;
|
||||
border-radius: 2px;
|
||||
background: #e2e8f0;
|
||||
overflow: hidden;
|
||||
|
||||
.run-info-meter-fill {
|
||||
height: 100%;
|
||||
border-radius: 2px;
|
||||
background: #2563eb;
|
||||
transition: width 0.3s ease;
|
||||
}
|
||||
|
||||
&.is-high .run-info-meter-fill {
|
||||
background: #f59e0b;
|
||||
}
|
||||
|
||||
&.is-critical .run-info-meter-fill {
|
||||
background: #ef4444;
|
||||
}
|
||||
}
|
||||
|
||||
.run-info-note {
|
||||
padding-top: 9px;
|
||||
font-size: 11.5px;
|
||||
color: #94a3b8;
|
||||
}
|
||||
}
|
||||
|
||||
&.notes-dropdown {
|
||||
width: 380px;
|
||||
max-height: 520px;
|
||||
|
||||
@@ -20,10 +20,18 @@ import { FormsModule } from '@angular/forms';
|
||||
import { RouterLink } from '@angular/router';
|
||||
import { HttpClient } from '@angular/common/http';
|
||||
import { AgentService, StartupProgressEvent } from '../../services/agent.service';
|
||||
import { Session, ModelInfo } from '../../core/models/session.model';
|
||||
import { Session, ModelInfo, SessionUsage } from '../../core/models/session.model';
|
||||
import { MarkdownSegment, MarkdownLine, NoteMilestone, ParsedNote } from '../../core/models/markdown.model';
|
||||
import { OverlayModule, ConnectedPosition } from '@angular/cdk/overlay';
|
||||
import { StepBlock, PhaseBlock, StepEvent, ActionParam, CheckerResult, StreamResetNotice, DEFAULT_STREAM_RESET_MESSAGE } from '../../core/models/stream.model';
|
||||
import {
|
||||
contextPercent,
|
||||
formatCompactTokens,
|
||||
formatElapsed,
|
||||
parseRunTuning,
|
||||
sessionElapsedSeconds,
|
||||
tuningLabel
|
||||
} from '../../utils/run-info.util';
|
||||
|
||||
export const PLANNING_LOADER_PHRASES: string[] = [
|
||||
'Planning next step...',
|
||||
@@ -235,6 +243,34 @@ export class AgentStreamComponent implements AfterViewInit {
|
||||
public isTaskDropdownOpen = signal<boolean>(false);
|
||||
public isNotesDropdownOpen = signal<boolean>(false);
|
||||
|
||||
// Run info popover (opened from the Flash / Pro chip): elapsed time, token
|
||||
// usage, executor context share and the Pro tuning the run started with.
|
||||
public isRunInfoOpen = signal<boolean>(false);
|
||||
public runUsage = signal<SessionUsage | null>(null);
|
||||
private runInfoClock = signal<number>(Date.now());
|
||||
|
||||
public runInfoIsPro = computed(() => this.getModelClass(this.agentService.viewedModel()?.name) === 'is-pro');
|
||||
|
||||
public runInfoElapsed = computed(() => {
|
||||
const session = this.agentService.currentSession();
|
||||
const running = this.agentService.isCurrentSessionRunning();
|
||||
return formatElapsed(sessionElapsedSeconds(session, running, this.runInfoClock()));
|
||||
});
|
||||
|
||||
public runInfoTuning = computed(() => {
|
||||
const usage = this.runUsage();
|
||||
if (usage?.run_tuning && (usage.run_tuning.verification_level || usage.run_tuning.explorer_mode)) {
|
||||
return usage.run_tuning;
|
||||
}
|
||||
return parseRunTuning(this.agentService.currentSession()?.device_info);
|
||||
});
|
||||
|
||||
public runInfoContextPercent = computed(() => {
|
||||
const usage = this.runUsage();
|
||||
if (!usage) return null;
|
||||
return contextPercent(usage.operator_context_tokens, usage.operator_context_window_tokens);
|
||||
});
|
||||
|
||||
// CDK Connected Overlay Positioning Strategy
|
||||
public readonly dropdownPositions: ConnectedPosition[] = [
|
||||
{
|
||||
@@ -476,6 +512,28 @@ export class AgentStreamComponent implements AfterViewInit {
|
||||
}
|
||||
|
||||
constructor() {
|
||||
// Run info popover: fetch usage when opened or when the viewed session
|
||||
// changes, and keep refreshing while that session is still running.
|
||||
effect((onCleanup) => {
|
||||
const open = this.isRunInfoOpen();
|
||||
const sessionId = this.agentService.currentSessionId();
|
||||
const running = this.agentService.isCurrentSessionRunning();
|
||||
if (!open || !sessionId) return;
|
||||
if (untracked(() => this.runUsage())?.session_id !== sessionId) {
|
||||
this.runUsage.set(null);
|
||||
}
|
||||
this.loadRunUsage(sessionId);
|
||||
if (!running) return;
|
||||
const usageInterval = setInterval(() => this.loadRunUsage(sessionId), 3000);
|
||||
onCleanup(() => clearInterval(usageInterval));
|
||||
});
|
||||
effect((onCleanup) => {
|
||||
if (!this.isRunInfoOpen() || !this.agentService.isCurrentSessionRunning()) return;
|
||||
this.runInfoClock.set(Date.now());
|
||||
const clockInterval = setInterval(() => this.runInfoClock.set(Date.now()), 1000);
|
||||
onCleanup(() => clearInterval(clockInterval));
|
||||
});
|
||||
|
||||
// The retry clock only needs to tick while something on screen is timing
|
||||
// (startup progress, LLM retry countdowns). An idle page stays quiet.
|
||||
this.zone.runOutsideAngular(() => {
|
||||
@@ -758,6 +816,38 @@ export class AgentStreamComponent implements AfterViewInit {
|
||||
public closeAllDropdowns(): void {
|
||||
this.isTaskDropdownOpen.set(false);
|
||||
this.isNotesDropdownOpen.set(false);
|
||||
this.isRunInfoOpen.set(false);
|
||||
}
|
||||
|
||||
public toggleRunInfo(event: Event): void {
|
||||
event.stopPropagation();
|
||||
this.isRunInfoOpen.update((v) => !v);
|
||||
if (this.isRunInfoOpen()) {
|
||||
this.isTaskDropdownOpen.set(false);
|
||||
this.isNotesDropdownOpen.set(false);
|
||||
}
|
||||
}
|
||||
|
||||
private loadRunUsage(sessionId: string): void {
|
||||
this.agentService.getSessionUsage(sessionId).subscribe({
|
||||
next: (usage) => {
|
||||
if (this.agentService.currentSessionId() !== sessionId) return;
|
||||
this.runUsage.set(usage);
|
||||
},
|
||||
error: (err) => console.error('Failed to load session usage:', err)
|
||||
});
|
||||
}
|
||||
|
||||
public formatRunElapsed(seconds: number): string {
|
||||
return formatElapsed(seconds);
|
||||
}
|
||||
|
||||
public formatRunTokens(tokens: number | null | undefined): string {
|
||||
return formatCompactTokens(tokens);
|
||||
}
|
||||
|
||||
public runTuningLabel(kind: 'verify' | 'explore', id: string | null | undefined): string {
|
||||
return tuningLabel(kind, id);
|
||||
}
|
||||
|
||||
public getTaskStatus(session: Session): 'running' | 'paused' | 'completed' | 'pending' | 'failed' | 'cancelled' {
|
||||
|
||||
+18
-6
@@ -33,6 +33,7 @@ import { AgentService } from '../../services/agent.service';
|
||||
import { StepReplayFrame } from '../../core/models/stream.model';
|
||||
import { drawActionCoordinatesOnOverlay } from '../../utils/image-overlay.util';
|
||||
import { getActionIcon } from '../../utils/action-formatter.util';
|
||||
import { locateTimelineTime, sessionTimeToTimelineTime } from '../../utils/recording-timeline.util';
|
||||
|
||||
@Component({
|
||||
selector: 'app-floating-video-player',
|
||||
@@ -179,7 +180,7 @@ export class FloatingVideoPlayerComponent implements OnDestroy {
|
||||
if (video && video.readyState >= 1) {
|
||||
const target = this.pendingAbsoluteSeek;
|
||||
this.pendingAbsoluteSeek = null;
|
||||
this.seek(target, true);
|
||||
this.seekToSessionTime(target, true);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -302,7 +303,7 @@ export class FloatingVideoPlayerComponent implements OnDestroy {
|
||||
if (this.pendingAbsoluteSeek !== null) {
|
||||
const target = this.pendingAbsoluteSeek;
|
||||
this.pendingAbsoluteSeek = null;
|
||||
this.seek(target, true);
|
||||
this.seekToSessionTime(target, true);
|
||||
// A seek into another finalized segment changes the video source. Let
|
||||
// that segment's metadata event apply pendingLocalTime.
|
||||
if (this.pendingLocalTime !== null) return;
|
||||
@@ -382,19 +383,30 @@ export class FloatingVideoPlayerComponent implements OnDestroy {
|
||||
this.seek(0, true);
|
||||
}
|
||||
|
||||
/**
|
||||
* Seek to a session-relative time (seconds since the task session started,
|
||||
* the axis step timestamps and analyzer ranges use). Manifest v2 segments
|
||||
* carry real session offsets, so the time lands in the right segment even
|
||||
* across scrcpy restart gaps; legacy playlists fall back to the timeline axis.
|
||||
*/
|
||||
public seekToSessionTime(sessionSeconds: number, autoplay = false): void {
|
||||
const segments = this.agentService.activeVideoSegments();
|
||||
this.seek(sessionTimeToTimelineTime(segments, sessionSeconds), autoplay);
|
||||
}
|
||||
|
||||
/** Seek on the back-to-back playlist axis used by the scrubber. */
|
||||
public seek(seconds: number, autoplay = false): void {
|
||||
const v = this.videoRef?.nativeElement;
|
||||
if (!v) return;
|
||||
const target = Math.max(0, Math.min(this.duration(), seconds));
|
||||
const segments = this.agentService.activeVideoSegments();
|
||||
if (!segments.length) {
|
||||
const location = locateTimelineTime(segments, target);
|
||||
if (!location) {
|
||||
v.currentTime = target;
|
||||
if (autoplay) v.play().catch(() => {});
|
||||
return;
|
||||
}
|
||||
let index = segments.findIndex(segment => target < segment.start + segment.duration);
|
||||
if (index < 0) index = segments.length - 1;
|
||||
const localTime = Math.max(0, target - segments[index].start);
|
||||
const { index, localTime } = location;
|
||||
if (index === this.activeSegmentIndex()) {
|
||||
v.currentTime = localTime;
|
||||
if (autoplay) v.play().catch(() => {});
|
||||
|
||||
@@ -57,6 +57,28 @@ export interface AgentStatusResponse {
|
||||
paused_error?: string | null;
|
||||
}
|
||||
|
||||
/** Per-run Pro tuning persisted with the session (`device_info.run_tuning`). */
|
||||
export interface SessionRunTuning {
|
||||
verification_level?: string | null;
|
||||
explorer_mode?: string | null;
|
||||
}
|
||||
|
||||
/** `GET /api/sessions/{id}/usage`: session-wide LLM usage and the live executor context. */
|
||||
export interface SessionUsage {
|
||||
session_id: string;
|
||||
llm_calls: number;
|
||||
prompt_tokens: number;
|
||||
completion_tokens: number;
|
||||
total_tokens: number;
|
||||
cached_tokens: number;
|
||||
/** Prompt size of the latest Operator / Flash runner call; null before the first call. */
|
||||
operator_context_tokens: number | null;
|
||||
operator_context_window_tokens: number;
|
||||
operator_context_updated_at?: number | null;
|
||||
profile?: string | null;
|
||||
run_tuning?: SessionRunTuning | null;
|
||||
}
|
||||
|
||||
export type AgentStatus = 'idle' | 'running' | 'completed' | 'offline' | string;
|
||||
|
||||
export type TaskStatus = 'running' | 'paused' | 'completed' | 'pending' | 'failed' | 'cancelled';
|
||||
|
||||
@@ -137,7 +137,9 @@ export interface CheckerCheckItem {
|
||||
/**
|
||||
* One streamed LLM turn of a multi-turn agent (the Checker's tool loop):
|
||||
* the text of one execution id with the time its first chunk arrived, so the
|
||||
* timeline can interleave it with the tool calls that followed.
|
||||
* timeline can interleave it with the tool calls that followed. Live sessions
|
||||
* build these from `llm_stream` chunks; historical sessions rebuild them from
|
||||
* the persisted transcript (`PersistedCheckerStream`).
|
||||
*/
|
||||
export interface StreamSegment {
|
||||
execution_id: string;
|
||||
@@ -149,6 +151,32 @@ export interface StreamSegment {
|
||||
resetMessage?: string;
|
||||
}
|
||||
|
||||
/** One persisted segment of a Checker attempt's streamed reasoning (backend `check_streams.jsonl`). */
|
||||
export interface PersistedStreamSegment {
|
||||
execution_id: string;
|
||||
/** `thought` is the model's reasoning stream, `answer` its visible text. */
|
||||
role: 'thought' | 'answer' | string;
|
||||
/** Unix seconds of the segment's first chunk. */
|
||||
when: number;
|
||||
text: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* One Checker attempt's persisted transcript as served by
|
||||
* `GET /api/sessions/{id}/checks` (`streams[]`). Text is bounded server-side;
|
||||
* `truncated` says the middle was cut out.
|
||||
*/
|
||||
export interface PersistedCheckerStream {
|
||||
attempt_id: string;
|
||||
checkpoint_id?: string;
|
||||
phase?: string;
|
||||
trace_id?: string | null;
|
||||
ts?: number;
|
||||
truncated?: boolean;
|
||||
dropped_chars?: number;
|
||||
segments: PersistedStreamSegment[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Data of a `checker` timeline block: one Checker attempt (a midway checkpoint
|
||||
* of a completed subgoal or the exit final review), or the run outcome
|
||||
@@ -182,7 +210,7 @@ export interface CheckerBlockData {
|
||||
duration?: number;
|
||||
isCompleted?: boolean;
|
||||
generic_tools?: any[];
|
||||
/** Per-turn stream segments (live sessions only); the flat fields below hold the joined text. */
|
||||
/** Per-turn stream segments (live chunks or the persisted transcript); the flat fields below hold the joined text. */
|
||||
stream_segments?: StreamSegment[];
|
||||
operator_native_thinking?: string;
|
||||
operator_raw_thinking?: string;
|
||||
|
||||
@@ -125,6 +125,42 @@ describe('AgentService live LLM retry timeline', () => {
|
||||
expect(service.isSessionContentLoading()).toBeFalse();
|
||||
});
|
||||
|
||||
it('attaches the persisted checker transcript to the attempt it belongs to', () => {
|
||||
const service = createServiceWithoutPolling();
|
||||
(service as any).currentSessionId = signal<string | null>('session-1');
|
||||
(service as any).http = {
|
||||
get: () => of({
|
||||
records: [
|
||||
{ attempt_id: 'abc#1', checkpoint_id: 'abc', subgoal_text: 'Create the alarm', trace_id: 't-1', item_text: 'alarm exists', kind: 'verify', status: 'passed', evidence: 'seen', ts: 10 },
|
||||
{ attempt_id: 'final#1', checkpoint_id: 'final', trace_id: 't-2', item_text: 'alarm exists', kind: 'verify', status: 'passed', evidence: 'seen', ts: 20 }
|
||||
],
|
||||
streams: [
|
||||
{
|
||||
attempt_id: 'abc#1',
|
||||
trace_id: 't-1',
|
||||
ts: 11,
|
||||
segments: [
|
||||
{ execution_id: 'e-1', role: 'thought', when: 8, text: 'Looking' },
|
||||
{ execution_id: 'e-1', role: 'answer', when: 9, text: 'Seen it' }
|
||||
]
|
||||
}
|
||||
],
|
||||
run_outcome: null
|
||||
})
|
||||
};
|
||||
service.sessionLogs.set([{ type: 'checker_event', checks_snapshot: true, data: { attempt_id: 'stale' } }]);
|
||||
|
||||
service.fetchChecks('session-1');
|
||||
|
||||
const logs = service.sessionLogs();
|
||||
expect(logs.map((l) => l.data.attempt_id)).toEqual(['abc#1', 'final#1']);
|
||||
expect(logs[0].data.stream_segments).toEqual([
|
||||
{ execution_id: 'e-1', stream_type: 'thinking', text: 'Looking', timestamp: new Date(8000).toISOString(), isCompleted: true },
|
||||
{ execution_id: 'e-1', stream_type: 'text', text: 'Seen it', timestamp: new Date(9000).toISOString(), isCompleted: true }
|
||||
]);
|
||||
expect('stream_segments' in logs[1].data).toBeFalse();
|
||||
});
|
||||
|
||||
it('follows a just-started task even when status polling saw it first', () => {
|
||||
const service = createServiceWithoutPolling();
|
||||
(service as any).http = {
|
||||
|
||||
@@ -18,20 +18,25 @@ import { Injectable, signal, inject, computed, DestroyRef, NgZone } from '@angul
|
||||
import { HttpClient } from '@angular/common/http';
|
||||
import { Observable } from 'rxjs';
|
||||
|
||||
import { Session, ModelInfo, TaskQueueItem, AgentStatusResponse } from '../core/models/session.model';
|
||||
import { Session, ModelInfo, TaskQueueItem, AgentStatusResponse, SessionUsage } from '../core/models/session.model';
|
||||
import { ProTuningDefaults, ProTuningOptions } from '../core/models/pro-tuning.model';
|
||||
import { StepItemData, StepReplayFrame, LLMStreamResetEventData, StreamResetNotice, DEFAULT_STREAM_RESET_MESSAGE } from '../core/models/stream.model';
|
||||
import { StepItemData, StepReplayFrame, LLMStreamResetEventData, StreamResetNotice, DEFAULT_STREAM_RESET_MESSAGE, PersistedCheckerStream, StreamSegment } from '../core/models/stream.model';
|
||||
import { extractStepReplayFrames } from '../utils/action-formatter.util';
|
||||
import { persistedStreamToSegments } from '../utils/stream-aggregator.util';
|
||||
export type { Session, ModelInfo, TaskQueueItem, AgentStatusResponse, StepItemData, StepReplayFrame, LLMStreamResetEventData, StreamResetNotice };
|
||||
|
||||
const SESSION_CACHE_KEY = 'artemis.sessions.v1';
|
||||
|
||||
export interface VideoSegment {
|
||||
url: string;
|
||||
/** Back-to-back playlist timeline start (seconds); gaps between segments are not represented. */
|
||||
start: number;
|
||||
duration: number;
|
||||
width: number;
|
||||
height: number;
|
||||
/** Session-relative first-frame offset in milliseconds (manifest v2). */
|
||||
offset_ms?: number;
|
||||
duration_ms?: number;
|
||||
}
|
||||
|
||||
export type RecordingPlaybackStatus =
|
||||
@@ -202,6 +207,7 @@ export class AgentService {
|
||||
public pausedError = signal<string | null>(null);
|
||||
public isRetrying = signal<boolean>(false);
|
||||
public retryMessage = signal<string | null>(null);
|
||||
/** Architecture / LLM of the task the status poll reports as running. */
|
||||
public activeModel = signal<{ name: string; id: string; provider: string } | null>(null);
|
||||
public userPinnedSessionId = signal<string | null>(null); // Pinned session if user explicitly selected a non-running task
|
||||
public startupProgressBySession = signal<Record<string, StartupProgressEvent[]>>({});
|
||||
@@ -300,6 +306,19 @@ export class AgentService {
|
||||
return this.sessions().find(s => s.session_id === curId) || null;
|
||||
});
|
||||
|
||||
/**
|
||||
* Architecture / LLM of the session being viewed.
|
||||
*
|
||||
* Each session row carries its own `model_info` (resolved by the backend),
|
||||
* so a Pro session stays "Pro" while a Flash task runs elsewhere. Only a
|
||||
* session that has not reached the list yet falls back to the running model.
|
||||
*/
|
||||
public viewedModel = computed<ModelInfo | null>(() => {
|
||||
const current = this.currentSession();
|
||||
if (current?.model_info) return current.model_info;
|
||||
return this.activeModel();
|
||||
});
|
||||
|
||||
/**
|
||||
* Computed whether the currently viewed session is actively running or paused.
|
||||
* Returns false when viewing completed/failed/cancelled tasks, or when no session is selected.
|
||||
@@ -388,6 +407,11 @@ export class AgentService {
|
||||
return this.http.get<ProTuningDefaults>('/api/run/defaults');
|
||||
}
|
||||
|
||||
/** Session-wide token totals, live executor context size and the run's tuning. */
|
||||
public getSessionUsage(sessionId: string): Observable<SessionUsage> {
|
||||
return this.http.get<SessionUsage>(`/api/sessions/${encodeURIComponent(sessionId)}/usage`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Run a new task by submitting to backend queue
|
||||
*/
|
||||
@@ -585,7 +609,6 @@ export class AgentService {
|
||||
next: (data) => {
|
||||
this.rawSessions.set(data);
|
||||
this.persistSessionsCache(data);
|
||||
this.updateModelForCurrentView(data);
|
||||
// On initial load, if nothing is selected, not pinned, not running, and sessions exist, select latest
|
||||
if (!this.currentSessionId() && !this.userPinnedSessionId() && this.agentStatus() !== 'running' && data.length > 0) {
|
||||
this.selectSession(data[0].session_id, false);
|
||||
@@ -720,7 +743,6 @@ export class AgentService {
|
||||
this.pausedError.set(null);
|
||||
this.fetchNotes(sessionId);
|
||||
this.fetchChecks(sessionId);
|
||||
this.updateModelForCurrentView();
|
||||
|
||||
// If video window is currently open, dynamically sync/refresh video for new session
|
||||
if (this.isVideoWindowOpen()) {
|
||||
@@ -876,7 +898,6 @@ export class AgentService {
|
||||
this.beginRecordingFinalization(endedId);
|
||||
}
|
||||
}
|
||||
this.updateModelForCurrentView();
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1576,22 +1597,6 @@ export class AgentService {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Update active model badge according to currently viewed session
|
||||
*/
|
||||
private updateModelForCurrentView(sessionsList?: Session[]): void {
|
||||
const curId = this.currentSessionId();
|
||||
const allSessions = sessionsList || this.sessions();
|
||||
|
||||
if (curId) {
|
||||
const selected = allSessions.find(s => s.session_id === curId);
|
||||
if (selected && selected.model_info) {
|
||||
this.activeModel.set(selected.model_info);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Clean up event source connection and timers on service destruction
|
||||
*/
|
||||
@@ -1623,10 +1628,12 @@ export class AgentService {
|
||||
*/
|
||||
public fetchChecks(sessionId: string): void {
|
||||
if (!sessionId) return;
|
||||
this.http.get<{ records?: any[]; run_outcome?: any }>(`/api/sessions/${sessionId}/checks`).subscribe({
|
||||
this.http.get<{ records?: any[]; streams?: PersistedCheckerStream[]; run_outcome?: any }>(`/api/sessions/${sessionId}/checks`).subscribe({
|
||||
next: (res) => {
|
||||
if (this.currentSessionId() !== sessionId) return;
|
||||
const snapshot = this.buildCheckerSnapshotLogs(sessionId, res?.records || [], res?.run_outcome || null);
|
||||
const snapshot = this.buildCheckerSnapshotLogs(
|
||||
sessionId, res?.records || [], res?.run_outcome || null, res?.streams || []
|
||||
);
|
||||
this.sessionLogs.update((logs) => [
|
||||
...logs.filter((log) => !log.checks_snapshot),
|
||||
...snapshot
|
||||
@@ -1638,8 +1645,26 @@ export class AgentService {
|
||||
});
|
||||
}
|
||||
|
||||
private buildCheckerSnapshotLogs(sessionId: string, records: any[], runOutcome: any): any[] {
|
||||
/**
|
||||
* `streams` are the attempts' persisted transcripts (what the Checker
|
||||
* streamed while reaching its verdict): they become the attempt's
|
||||
* `stream_segments`, the same shape the live `llm_stream` chunks build, so
|
||||
* a reopened session interleaves Thought/Work text with tool rows exactly
|
||||
* like the live view did.
|
||||
*/
|
||||
private buildCheckerSnapshotLogs(
|
||||
sessionId: string,
|
||||
records: any[],
|
||||
runOutcome: any,
|
||||
streams: PersistedCheckerStream[] = []
|
||||
): any[] {
|
||||
const byAttempt = new Map<string, any>();
|
||||
const segmentsByAttempt = new Map<string, StreamSegment[]>();
|
||||
for (const stream of streams || []) {
|
||||
if (!stream || !stream.attempt_id) continue;
|
||||
const segments = persistedStreamToSegments(stream);
|
||||
if (segments.length > 0) segmentsByAttempt.set(String(stream.attempt_id), segments);
|
||||
}
|
||||
for (const rec of records) {
|
||||
if (!rec || !rec.attempt_id) continue;
|
||||
const attemptId = String(rec.attempt_id);
|
||||
@@ -1689,6 +1714,8 @@ export class AgentService {
|
||||
}
|
||||
const ts = attempt.ts ?? Date.now() / 1000;
|
||||
attempt.timestamp = ts;
|
||||
const segments = segmentsByAttempt.get(attempt.attempt_id);
|
||||
if (segments) attempt.stream_segments = segments;
|
||||
logs.push({
|
||||
type: 'checker_event',
|
||||
session_id: sessionId,
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
import {
|
||||
hasSessionOffsets,
|
||||
locateSessionTime,
|
||||
locateTimelineTime,
|
||||
sessionTimeToTimelineTime,
|
||||
TimelineSegment
|
||||
} from './recording-timeline.util';
|
||||
|
||||
describe('recording-timeline.util', () => {
|
||||
// Recording started 1.2s after the session began. Segment 0 ran 10s, then
|
||||
// scrcpy restarted for rotation (1.5s gap), segment 1 ran 5s, another 0.8s
|
||||
// gap, and segment 2 ran 4s.
|
||||
const v2Segments: TimelineSegment[] = [
|
||||
{ start: 0, duration: 10, offset_ms: 1200, duration_ms: 10000 },
|
||||
{ start: 10, duration: 5, offset_ms: 12700, duration_ms: 5000 },
|
||||
{ start: 15, duration: 4, offset_ms: 18500, duration_ms: 4000 }
|
||||
];
|
||||
|
||||
const legacySegments: TimelineSegment[] = [
|
||||
{ start: 0, duration: 10 },
|
||||
{ start: 10, duration: 5 }
|
||||
];
|
||||
|
||||
describe('hasSessionOffsets', () => {
|
||||
it('detects manifest v2 segments and rejects legacy or mixed lists', () => {
|
||||
expect(hasSessionOffsets(v2Segments)).toBeTrue();
|
||||
expect(hasSessionOffsets(legacySegments)).toBeFalse();
|
||||
expect(hasSessionOffsets([v2Segments[0], legacySegments[0]])).toBeFalse();
|
||||
expect(hasSessionOffsets([])).toBeFalse();
|
||||
});
|
||||
});
|
||||
|
||||
describe('locateTimelineTime', () => {
|
||||
it('maps back-to-back playlist times into segments', () => {
|
||||
expect(locateTimelineTime(legacySegments, 3)).toEqual({ index: 0, localTime: 3, timelineTime: 3 });
|
||||
expect(locateTimelineTime(legacySegments, 10)).toEqual({ index: 1, localTime: 0, timelineTime: 10 });
|
||||
expect(locateTimelineTime(legacySegments, 12.5)).toEqual({ index: 1, localTime: 2.5, timelineTime: 12.5 });
|
||||
});
|
||||
|
||||
it('clamps out-of-range times into the first or last segment', () => {
|
||||
expect(locateTimelineTime(legacySegments, -4)).toEqual({ index: 0, localTime: 0, timelineTime: 0 });
|
||||
expect(locateTimelineTime(legacySegments, 99)).toEqual({ index: 1, localTime: 5, timelineTime: 15 });
|
||||
expect(locateTimelineTime([], 3)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('locateSessionTime', () => {
|
||||
it('maps a session time inside a segment using its real offset', () => {
|
||||
// T+4.2s is 3.0s into segment 0 (which started at T+1.2s).
|
||||
const inFirst = locateSessionTime(v2Segments, 4.2)!;
|
||||
expect(inFirst.index).toBe(0);
|
||||
expect(inFirst.localTime).toBeCloseTo(3.0, 6);
|
||||
expect(inFirst.timelineTime).toBeCloseTo(3.0, 6);
|
||||
|
||||
// T+14.7s is 2.0s into segment 1 (started at T+12.7s), which sits at
|
||||
// timeline 10s, so it must NOT drift to timeline 14.7s.
|
||||
const inSecond = locateSessionTime(v2Segments, 14.7)!;
|
||||
expect(inSecond.index).toBe(1);
|
||||
expect(inSecond.localTime).toBeCloseTo(2.0, 6);
|
||||
expect(inSecond.timelineTime).toBeCloseTo(12.0, 6);
|
||||
|
||||
const inThird = locateSessionTime(v2Segments, 20.5)!;
|
||||
expect(inThird.index).toBe(2);
|
||||
expect(inThird.localTime).toBeCloseTo(2.0, 6);
|
||||
expect(inThird.timelineTime).toBeCloseTo(17.0, 6);
|
||||
});
|
||||
|
||||
it('snaps a time inside a restart gap to the start of the next segment', () => {
|
||||
// Segment 0 ends at T+11.2s, segment 1 starts at T+12.7s.
|
||||
expect(locateSessionTime(v2Segments, 11.9)).toEqual({ index: 1, localTime: 0, timelineTime: 10 });
|
||||
// Segment 1 ends at T+17.7s, segment 2 starts at T+18.5s.
|
||||
expect(locateSessionTime(v2Segments, 18.0)).toEqual({ index: 2, localTime: 0, timelineTime: 15 });
|
||||
});
|
||||
|
||||
it('treats segment boundaries as inclusive of the end frame', () => {
|
||||
const atEnd = locateSessionTime(v2Segments, 11.2)!;
|
||||
expect(atEnd.index).toBe(0);
|
||||
expect(atEnd.localTime).toBeCloseTo(10, 6);
|
||||
expect(locateSessionTime(v2Segments, 12.7)).toEqual({ index: 1, localTime: 0, timelineTime: 10 });
|
||||
});
|
||||
|
||||
it('snaps times before the first frame and after the last frame', () => {
|
||||
expect(locateSessionTime(v2Segments, 0)).toEqual({ index: 0, localTime: 0, timelineTime: 0 });
|
||||
expect(locateSessionTime(v2Segments, 0.5)).toEqual({ index: 0, localTime: 0, timelineTime: 0 });
|
||||
expect(locateSessionTime(v2Segments, 60)).toEqual({ index: 2, localTime: 4, timelineTime: 19 });
|
||||
expect(locateSessionTime(v2Segments, Number.NaN)).toEqual({ index: 0, localTime: 0, timelineTime: 0 });
|
||||
});
|
||||
|
||||
it('falls back to back-to-back timeline behaviour for legacy manifests', () => {
|
||||
expect(locateSessionTime(legacySegments, 12.5)).toEqual({ index: 1, localTime: 2.5, timelineTime: 12.5 });
|
||||
expect(locateSessionTime([], 5)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('sessionTimeToTimelineTime', () => {
|
||||
it('returns the playlist position for a session time', () => {
|
||||
expect(sessionTimeToTimelineTime(v2Segments, 14.7)).toBeCloseTo(12.0, 6);
|
||||
expect(sessionTimeToTimelineTime(v2Segments, 11.9)).toBe(10);
|
||||
expect(sessionTimeToTimelineTime(legacySegments, 7)).toBe(7);
|
||||
});
|
||||
|
||||
it('passes the time through when no playlist is available', () => {
|
||||
expect(sessionTimeToTimelineTime([], 7)).toBe(7);
|
||||
expect(sessionTimeToTimelineTime([], -2)).toBe(0);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,123 @@
|
||||
/**
|
||||
* Copyright 2026 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
/**
|
||||
* Pure helpers that map times onto the multi-segment screen recording.
|
||||
*
|
||||
* A recording is split into segments whenever scrcpy restarts (device
|
||||
* rotation, crash recovery). The player plays them back to back on a
|
||||
* "timeline" axis (`start`/`duration`, no gaps), while step timestamps live
|
||||
* on the "session" axis (seconds since the DataEngine session start). Manifest
|
||||
* v2 segments carry `offset_ms`/`duration_ms` on the session axis, which lets
|
||||
* a session time land in the right segment even after each restart gap.
|
||||
*/
|
||||
|
||||
export interface TimelineSegment {
|
||||
/** Back-to-back playlist start in seconds. */
|
||||
start: number;
|
||||
/** Segment length in seconds. */
|
||||
duration: number;
|
||||
/** Session-relative first-frame offset in milliseconds (manifest v2). */
|
||||
offset_ms?: number;
|
||||
/** Segment length in milliseconds (manifest v2). */
|
||||
duration_ms?: number;
|
||||
}
|
||||
|
||||
export interface SegmentLocation {
|
||||
/** Index into the segment list. */
|
||||
index: number;
|
||||
/** Seconds into that segment's own media. */
|
||||
localTime: number;
|
||||
/** Equivalent position on the back-to-back playlist axis, in seconds. */
|
||||
timelineTime: number;
|
||||
}
|
||||
|
||||
const clamp = (value: number, min: number, max: number): number => Math.max(min, Math.min(max, value));
|
||||
|
||||
/** True when every segment carries the session-relative v2 fields. */
|
||||
export function hasSessionOffsets(segments: readonly TimelineSegment[]): boolean {
|
||||
return segments.length > 0 && segments.every(
|
||||
(segment) => Number.isFinite(segment.offset_ms) && Number.isFinite(segment.duration_ms)
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Locate a position on the back-to-back playlist axis (what the scrubber uses).
|
||||
* Times past the end clamp into the last segment.
|
||||
*/
|
||||
export function locateTimelineTime(
|
||||
segments: readonly TimelineSegment[],
|
||||
timelineSeconds: number
|
||||
): SegmentLocation | null {
|
||||
if (!segments.length) return null;
|
||||
const total = segments.reduce((sum, segment) => sum + Math.max(0, segment.duration), 0);
|
||||
const target = clamp(Number.isFinite(timelineSeconds) ? timelineSeconds : 0, 0, total);
|
||||
let index = segments.findIndex((segment) => target < segment.start + segment.duration);
|
||||
if (index < 0) index = segments.length - 1;
|
||||
const segment = segments[index];
|
||||
const localTime = clamp(target - segment.start, 0, Math.max(0, segment.duration));
|
||||
return { index, localTime, timelineTime: segment.start + localTime };
|
||||
}
|
||||
|
||||
/**
|
||||
* Locate a session-relative time (seconds since session start, the axis step
|
||||
* timestamps use). A time inside `[offset, offset + duration]` of a segment maps
|
||||
* into that segment; a time in the gap between two segments (scrcpy restart)
|
||||
* snaps to the start of the next segment; a time before the first frame snaps
|
||||
* to the first frame; a time after the last frame clamps to the last frame.
|
||||
*
|
||||
* Segments without session offsets (legacy manifests) are treated as if the
|
||||
* session axis were the back-to-back timeline axis.
|
||||
*/
|
||||
export function locateSessionTime(
|
||||
segments: readonly TimelineSegment[],
|
||||
sessionSeconds: number
|
||||
): SegmentLocation | null {
|
||||
if (!segments.length) return null;
|
||||
if (!hasSessionOffsets(segments)) {
|
||||
return locateTimelineTime(segments, sessionSeconds);
|
||||
}
|
||||
const targetMs = Number.isFinite(sessionSeconds) ? sessionSeconds * 1000 : 0;
|
||||
for (let index = 0; index < segments.length; index++) {
|
||||
const segment = segments[index];
|
||||
const offsetMs = segment.offset_ms as number;
|
||||
const endMs = offsetMs + (segment.duration_ms as number);
|
||||
if (targetMs < offsetMs) {
|
||||
// Before this segment's first frame: either before the recording began
|
||||
// or inside the restart gap after the previous segment. Snap forward.
|
||||
return { index, localTime: 0, timelineTime: segment.start };
|
||||
}
|
||||
if (targetMs <= endMs) {
|
||||
const localTime = clamp((targetMs - offsetMs) / 1000, 0, Math.max(0, segment.duration));
|
||||
return { index, localTime, timelineTime: segment.start + localTime };
|
||||
}
|
||||
}
|
||||
const lastIndex = segments.length - 1;
|
||||
const last = segments[lastIndex];
|
||||
const localTime = Math.max(0, last.duration);
|
||||
return { index: lastIndex, localTime, timelineTime: last.start + localTime };
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a session-relative time into the back-to-back playlist time the
|
||||
* player scrubber understands. Returns 0 when there are no segments.
|
||||
*/
|
||||
export function sessionTimeToTimelineTime(
|
||||
segments: readonly TimelineSegment[],
|
||||
sessionSeconds: number
|
||||
): number {
|
||||
return locateSessionTime(segments, sessionSeconds)?.timelineTime ?? Math.max(0, sessionSeconds || 0);
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
import {
|
||||
contextPercent,
|
||||
formatCompactTokens,
|
||||
formatElapsed,
|
||||
parseRunTuning,
|
||||
sessionElapsedSeconds,
|
||||
tuningLabel
|
||||
} from './run-info.util';
|
||||
|
||||
describe('run-info.util', () => {
|
||||
describe('formatElapsed', () => {
|
||||
it('formats seconds, minutes and hours with zero padding', () => {
|
||||
expect(formatElapsed(0)).toBe('0s');
|
||||
expect(formatElapsed(42)).toBe('42s');
|
||||
expect(formatElapsed(65)).toBe('1m 05s');
|
||||
expect(formatElapsed(3725)).toBe('1h 02m 05s');
|
||||
});
|
||||
|
||||
it('treats invalid or negative input as zero', () => {
|
||||
expect(formatElapsed(-5)).toBe('0s');
|
||||
expect(formatElapsed(undefined)).toBe('0s');
|
||||
expect(formatElapsed(Number.NaN)).toBe('0s');
|
||||
});
|
||||
});
|
||||
|
||||
describe('formatCompactTokens', () => {
|
||||
it('uses plain, k and M units', () => {
|
||||
expect(formatCompactTokens(842)).toBe('842');
|
||||
expect(formatCompactTokens(76_400)).toBe('76.4k');
|
||||
expect(formatCompactTokens(1_000)).toBe('1k');
|
||||
expect(formatCompactTokens(1_250_000)).toBe('1.25M');
|
||||
expect(formatCompactTokens(null)).toBe('0');
|
||||
});
|
||||
});
|
||||
|
||||
describe('contextPercent', () => {
|
||||
it('reports one-decimal share of the window and clamps at 100', () => {
|
||||
expect(contextPercent(150_000, 1_000_000)).toBe(15);
|
||||
expect(contextPercent(123_456, 1_000_000)).toBe(12.3);
|
||||
expect(contextPercent(2_000_000, 1_000_000)).toBe(100);
|
||||
});
|
||||
|
||||
it('falls back to the 1M window and is null when usage is unknown', () => {
|
||||
expect(contextPercent(250_000, undefined)).toBe(25);
|
||||
expect(contextPercent(null, 1_000_000)).toBeNull();
|
||||
expect(contextPercent(undefined, 1_000_000)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('sessionElapsedSeconds', () => {
|
||||
it('counts to now while running and to end_time once finished', () => {
|
||||
const session = { start_time: 1_000, end_time: 1_090 };
|
||||
expect(sessionElapsedSeconds(session, true, 1_200_000)).toBe(200);
|
||||
expect(sessionElapsedSeconds(session, false, 1_200_000)).toBe(90);
|
||||
});
|
||||
|
||||
it('is zero without a start time or session', () => {
|
||||
expect(sessionElapsedSeconds(null, true, 5_000)).toBe(0);
|
||||
expect(sessionElapsedSeconds({ start_time: 0 }, true, 5_000)).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('tuningLabel', () => {
|
||||
it('maps ids to the launcher labels and defaults unknown ids', () => {
|
||||
expect(tuningLabel('verify', 'checkpoints')).toBe('Every step');
|
||||
expect(tuningLabel('verify', 'nonsense')).toBe('At the end');
|
||||
expect(tuningLabel('explore', 'ultra')).toBe('Close-up');
|
||||
expect(tuningLabel('explore', null)).toBe('Quick glance');
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseRunTuning', () => {
|
||||
it('reads run_tuning from a JSON string or an object', () => {
|
||||
const asString = JSON.stringify({
|
||||
profile: 'pro',
|
||||
run_tuning: { verification_level: 'strict', explorer_mode: 'pro' }
|
||||
});
|
||||
expect(parseRunTuning(asString)).toEqual({ verification_level: 'strict', explorer_mode: 'pro' });
|
||||
expect(parseRunTuning({ run_tuning: { verification_level: 'off' } })).toEqual({
|
||||
verification_level: 'off',
|
||||
explorer_mode: null
|
||||
});
|
||||
});
|
||||
|
||||
it('is null for flash sessions, malformed JSON and empty tuning', () => {
|
||||
expect(parseRunTuning(JSON.stringify({ profile: 'flash' }))).toBeNull();
|
||||
expect(parseRunTuning('{not json')).toBeNull();
|
||||
expect(parseRunTuning({ run_tuning: {} })).toBeNull();
|
||||
expect(parseRunTuning(null)).toBeNull();
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,116 @@
|
||||
/**
|
||||
* Copyright 2026 Google LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
/**
|
||||
* Pure helpers behind the run info popover (the Flash / Pro chip in the
|
||||
* task stream): elapsed time, compact token counts, context-window share
|
||||
* and the plain-language labels of the Pro tuning a run was started with.
|
||||
*/
|
||||
|
||||
import {
|
||||
EXPLORER_MODES,
|
||||
VERIFICATION_LEVELS,
|
||||
levelIndex,
|
||||
DEFAULT_EXPLORER_MODE,
|
||||
DEFAULT_VERIFICATION_LEVEL
|
||||
} from '../core/models/pro-tuning.model';
|
||||
import { SessionRunTuning } from '../core/models/session.model';
|
||||
|
||||
/** Denominator for "context used" when the backend does not report one. */
|
||||
export const OPERATOR_CONTEXT_WINDOW_TOKENS = 1_000_000;
|
||||
|
||||
/** `42s`, `12m 05s`, `1h 02m 05s`. Negative or invalid input reads as `0s`. */
|
||||
export function formatElapsed(totalSeconds: number | null | undefined): string {
|
||||
const secs = Math.max(0, Math.floor(Number(totalSeconds) || 0));
|
||||
const hours = Math.floor(secs / 3600);
|
||||
const minutes = Math.floor((secs % 3600) / 60);
|
||||
const seconds = secs % 60;
|
||||
const pad = (n: number) => String(n).padStart(2, '0');
|
||||
if (hours > 0) return `${hours}h ${pad(minutes)}m ${pad(seconds)}s`;
|
||||
if (minutes > 0) return `${minutes}m ${pad(seconds)}s`;
|
||||
return `${seconds}s`;
|
||||
}
|
||||
|
||||
/** `842`, `76.4k`, `1.2M` (no unit suffix; the caller labels the row). */
|
||||
export function formatCompactTokens(tokens: number | null | undefined): string {
|
||||
const n = Math.max(0, Math.floor(Number(tokens) || 0));
|
||||
if (n < 1000) return String(n);
|
||||
if (n < 1_000_000) return `${(n / 1000).toFixed(1).replace(/\.0$/, '')}k`;
|
||||
return `${(n / 1_000_000).toFixed(2).replace(/\.?0+$/, '')}M`;
|
||||
}
|
||||
|
||||
/** Share of the context window in percent (one decimal), or null when unknown. */
|
||||
export function contextPercent(
|
||||
usedTokens: number | null | undefined,
|
||||
windowTokens: number | null | undefined
|
||||
): number | null {
|
||||
const used = Number(usedTokens);
|
||||
const window = Number(windowTokens) || OPERATOR_CONTEXT_WINDOW_TOKENS;
|
||||
if (!Number.isFinite(used) || used < 0 || usedTokens === null || usedTokens === undefined) {
|
||||
return null;
|
||||
}
|
||||
const pct = (used / window) * 100;
|
||||
return Math.min(100, Math.round(pct * 10) / 10);
|
||||
}
|
||||
|
||||
/**
|
||||
* Seconds a session has been (or was) running.
|
||||
*
|
||||
* Live sessions count from `start_time` to `nowMs`; finished sessions use the
|
||||
* recorded `end_time`. A session with no `start_time` reads as 0.
|
||||
*/
|
||||
export function sessionElapsedSeconds(
|
||||
session: { start_time?: number; end_time?: number | null } | null | undefined,
|
||||
isRunning: boolean,
|
||||
nowMs: number
|
||||
): number {
|
||||
if (!session?.start_time) return 0;
|
||||
const startMs = session.start_time * 1000;
|
||||
const endMs = !isRunning && session.end_time ? session.end_time * 1000 : nowMs;
|
||||
return Math.max(0, Math.round((endMs - startMs) / 1000));
|
||||
}
|
||||
|
||||
/** Plain-language label for a tuning id, falling back to the ladder default. */
|
||||
export function tuningLabel(kind: 'verify' | 'explore', id: string | null | undefined): string {
|
||||
if (kind === 'verify') {
|
||||
return VERIFICATION_LEVELS[levelIndex(VERIFICATION_LEVELS, id, DEFAULT_VERIFICATION_LEVEL)].label;
|
||||
}
|
||||
return EXPLORER_MODES[levelIndex(EXPLORER_MODES, id, DEFAULT_EXPLORER_MODE)].label;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads `run_tuning` out of a session's `device_info` (which the sessions API
|
||||
* returns either as a JSON string or an already-parsed object).
|
||||
*/
|
||||
export function parseRunTuning(deviceInfo: unknown): SessionRunTuning | null {
|
||||
let info: unknown = deviceInfo;
|
||||
if (typeof info === 'string') {
|
||||
try {
|
||||
info = JSON.parse(info);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
if (!info || typeof info !== 'object') return null;
|
||||
const tuning = (info as { run_tuning?: unknown }).run_tuning;
|
||||
if (!tuning || typeof tuning !== 'object') return null;
|
||||
const t = tuning as SessionRunTuning;
|
||||
if (!t.verification_level && !t.explorer_mode) return null;
|
||||
return {
|
||||
verification_level: t.verification_level ?? null,
|
||||
explorer_mode: t.explorer_mode ?? null
|
||||
};
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
import { consolidateLogsToBlocks, getSortedStepEvents } from './stream-aggregator.util';
|
||||
import { consolidateLogsToBlocks, getSortedStepEvents, persistedStreamToSegments } from './stream-aggregator.util';
|
||||
|
||||
describe('stream aggregator timeline ordering', () => {
|
||||
it('interleaves text, tools, and actions by timestamp', () => {
|
||||
@@ -203,7 +203,7 @@ describe('stream aggregator checker lane', () => {
|
||||
parent_trace_id: 'trace-checker-1',
|
||||
agent_name: 'checker',
|
||||
type: 'tool',
|
||||
name: 'get_step_detail',
|
||||
name: 'replay_steps',
|
||||
status: 'success',
|
||||
timestamp: '2026-08-25T20:00:07.000Z'
|
||||
}
|
||||
@@ -223,6 +223,114 @@ describe('stream aggregator checker lane', () => {
|
||||
expect(events[2].data.text).toBe('The alarm is there.');
|
||||
});
|
||||
|
||||
it('rebuilds the persisted transcript into the same interleaved segments as the live stream', () => {
|
||||
const transcript = {
|
||||
attempt_id: 'abc#1',
|
||||
checkpoint_id: 'abc',
|
||||
phase: 'checkpoint',
|
||||
trace_id: 'trace-checker-1',
|
||||
ts: 1787688009,
|
||||
truncated: false,
|
||||
segments: [
|
||||
// 1787688006 = 2026-08-25T20:00:06Z
|
||||
{ execution_id: 'exec-1', role: 'thought', when: 1787688006, text: 'Inspecting the alarm list' },
|
||||
{ execution_id: 'exec-1', role: 'answer', when: 1787688006.5, text: 'Let me look at step 2.' },
|
||||
{ execution_id: 'exec-2', role: 'answer', when: 1787688008, text: 'The alarm is there.' },
|
||||
{ execution_id: 'exec-3', role: 'answer', when: 1787688008.5, text: '' }
|
||||
]
|
||||
};
|
||||
const segments = persistedStreamToSegments(transcript);
|
||||
expect(segments).toEqual([
|
||||
{ execution_id: 'exec-1', stream_type: 'thinking', text: 'Inspecting the alarm list', timestamp: '2026-08-25T20:00:06.000Z', isCompleted: true },
|
||||
{ execution_id: 'exec-1', stream_type: 'text', text: 'Let me look at step 2.', timestamp: '2026-08-25T20:00:06.500Z', isCompleted: true },
|
||||
{ execution_id: 'exec-2', stream_type: 'text', text: 'The alarm is there.', timestamp: '2026-08-25T20:00:08.000Z', isCompleted: true }
|
||||
]);
|
||||
expect(persistedStreamToSegments(null)).toEqual([]);
|
||||
|
||||
// Historical session: only the ledger backfill (attempt_finished carrying
|
||||
// the rebuilt segments) and the relocated tool trace exist.
|
||||
const tool = {
|
||||
type: 'trace_recorded',
|
||||
timestamp: '2026-08-25T20:00:07.000Z',
|
||||
data: {
|
||||
trace_id: 'tool-detail',
|
||||
parent_trace_id: 'trace-checker-1',
|
||||
agent_name: 'checker',
|
||||
type: 'tool',
|
||||
name: 'replay_steps',
|
||||
status: 'success',
|
||||
timestamp: '2026-08-25T20:00:07.000Z'
|
||||
}
|
||||
};
|
||||
const historical = consolidateLogsToBlocks([
|
||||
{
|
||||
type: 'checker_event',
|
||||
timestamp: '2026-08-25T20:00:09.000Z',
|
||||
checks_snapshot: true,
|
||||
data: {
|
||||
event: 'attempt_finished',
|
||||
phase: 'checkpoint',
|
||||
attempt_id: 'abc#1',
|
||||
checkpoint_id: 'abc',
|
||||
trace_id: 'trace-checker-1',
|
||||
status: 'done',
|
||||
verdicts: [{ item_text: 'alarm exists', kind: 'verify', status: 'passed', evidence: 'seen' }],
|
||||
findings: [],
|
||||
ts: 1756152009,
|
||||
stream_segments: segments
|
||||
}
|
||||
},
|
||||
tool
|
||||
]);
|
||||
const block = historical.find((b) => b.id === 'checker-abc#1')!;
|
||||
expect(block.data.stream_segments).toEqual(segments);
|
||||
expect(block.data.operator_native_thinking).toBe('Inspecting the alarm list');
|
||||
expect(block.data.operator_raw_thinking).toBe('Let me look at step 2.\n\nThe alarm is there.');
|
||||
expect(block.data.operator_native_thinking_timestamp).toBe('2026-08-25T20:00:06.000Z');
|
||||
expect(block.data.operator_raw_thinking_timestamp).toBe('2026-08-25T20:00:06.500Z');
|
||||
const events = getSortedStepEvents(block.data);
|
||||
expect(events.map((e) => e.type)).toEqual(['thinking', 'text', 'tool', 'text']);
|
||||
expect(events.map((e) => e.data.text ?? e.data.name)).toEqual([
|
||||
'Inspecting the alarm list',
|
||||
'Let me look at step 2.',
|
||||
'replay_steps',
|
||||
'The alarm is there.'
|
||||
]);
|
||||
|
||||
// Live session: the same turns arrived as llm_stream chunks; a later
|
||||
// ledger backfill must neither duplicate nor reorder them.
|
||||
const chunk = (execId: string, streamType: string, timestamp: string, text: string) => ({
|
||||
type: 'llm_stream',
|
||||
timestamp,
|
||||
data: { execution_id: execId, parent_trace_id: 'trace-checker-1', stream_type: streamType, text, isCompleted: true }
|
||||
});
|
||||
const live = consolidateLogsToBlocks([
|
||||
started,
|
||||
chunk('exec-1', 'thinking', '2026-08-25T20:00:06.000Z', 'Inspecting the alarm list'),
|
||||
chunk('exec-1', 'text', '2026-08-25T20:00:06.500Z', 'Let me look at step 2.'),
|
||||
tool,
|
||||
chunk('exec-2', 'text', '2026-08-25T20:00:08.000Z', 'The alarm is there.'),
|
||||
{
|
||||
type: 'checker_event',
|
||||
timestamp: '2026-08-25T20:00:09.000Z',
|
||||
data: { event: 'attempt_finished', attempt_id: 'abc#1', status: 'done', verdicts: [], findings: [], ts: 1756152009 }
|
||||
},
|
||||
{
|
||||
type: 'checker_event',
|
||||
timestamp: '2026-08-25T20:00:09.000Z',
|
||||
checks_snapshot: true,
|
||||
data: { event: 'attempt_finished', attempt_id: 'abc#1', status: 'done', ts: 1756152009, stream_segments: segments }
|
||||
}
|
||||
]);
|
||||
const liveBlock = live.find((b) => b.id === 'checker-abc#1')!;
|
||||
expect(liveBlock.data.stream_segments.map((s: any) => [s.execution_id, s.stream_type, s.text])).toEqual(
|
||||
segments.map((s) => [s.execution_id, s.stream_type, s.text])
|
||||
);
|
||||
expect(getSortedStepEvents(liveBlock.data).map((e) => e.data.text ?? e.data.name)).toEqual(
|
||||
events.map((e) => e.data.text ?? e.data.name)
|
||||
);
|
||||
});
|
||||
|
||||
it('builds the run outcome block once and keeps it idempotent', () => {
|
||||
const outcome = {
|
||||
type: 'checker_event',
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
import { StepBlock, PhaseBlock, StepEvent, StreamSegment, StreamResetNotice, DEFAULT_STREAM_RESET_MESSAGE } from '../core/models/stream.model';
|
||||
import { StepBlock, PhaseBlock, StepEvent, StreamSegment, StreamResetNotice, DEFAULT_STREAM_RESET_MESSAGE, PersistedCheckerStream } from '../core/models/stream.model';
|
||||
import { isAndroidAction, isReportStatusAction, getReportStatusExplanation, getReportStatusValue, getActionObject } from './action-formatter.util';
|
||||
import { getUniqueGenericTools, isInternalPlumbingTool } from './tool-formatter.util';
|
||||
|
||||
@@ -52,6 +52,76 @@ function findRunningCheckerBlock(blocks: StepBlock[]): number {
|
||||
return -1;
|
||||
}
|
||||
|
||||
/**
|
||||
* Rebuild the live `StreamSegment` list of a Checker attempt from its
|
||||
* persisted transcript, so a reopened session renders the same interleaved
|
||||
* Thought/Work segments as the live stream did.
|
||||
*/
|
||||
export function persistedStreamToSegments(record: PersistedCheckerStream | null | undefined): StreamSegment[] {
|
||||
if (!record || !Array.isArray(record.segments)) return [];
|
||||
const segments: StreamSegment[] = [];
|
||||
for (const seg of record.segments) {
|
||||
if (!seg || typeof seg.text !== 'string' || !seg.text) continue;
|
||||
const when = typeof seg.when === 'number' && seg.when > 0 ? seg.when : (record.ts || 0);
|
||||
segments.push({
|
||||
execution_id: String(seg.execution_id || ''),
|
||||
stream_type: seg.role === 'thought' ? 'thinking' : 'text',
|
||||
text: seg.text,
|
||||
timestamp: new Date(when * 1000).toISOString(),
|
||||
isCompleted: true
|
||||
});
|
||||
}
|
||||
return segments;
|
||||
}
|
||||
|
||||
function segmentKey(segment: StreamSegment): string {
|
||||
return `${segment.execution_id}|${segment.stream_type}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Merge persisted segments into the ones a block already holds. Live chunks
|
||||
* (complete by the time the transcript exists) win over the backfill copy of
|
||||
* the same turn; turns only the transcript knows are added, then everything
|
||||
* is kept in first-chunk order.
|
||||
*/
|
||||
function mergeStreamSegments(existing: any, incoming: any): StreamSegment[] | undefined {
|
||||
const base: StreamSegment[] = Array.isArray(existing) ? [...existing] : [];
|
||||
const extra: StreamSegment[] = Array.isArray(incoming) ? incoming : [];
|
||||
if (extra.length === 0) return base.length > 0 ? base : undefined;
|
||||
const seen = new Set(base.map(segmentKey));
|
||||
let added = false;
|
||||
for (const segment of extra) {
|
||||
const key = segmentKey(segment);
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
base.push(segment);
|
||||
added = true;
|
||||
}
|
||||
if (added) {
|
||||
base.sort((a, b) => getItemTimestamp(a.timestamp, 0) - getItemTimestamp(b.timestamp, 0));
|
||||
}
|
||||
return base;
|
||||
}
|
||||
|
||||
/**
|
||||
* Keep the flat `operator_*_thinking` fields (joined text + first-seen time)
|
||||
* in step with the segment list for consumers that only know steps.
|
||||
*/
|
||||
function applyFlatStreamFields(data: any, segments: StreamSegment[]): void {
|
||||
const of = (type: string) => segments.filter((s) => s.stream_type === type && s.text.trim());
|
||||
const joined = (type: string) => of(type).map((s) => s.text).join('\n\n');
|
||||
data.operator_native_thinking = joined('thinking');
|
||||
data.operator_raw_thinking = joined('text');
|
||||
const firstThinking = of('thinking')[0];
|
||||
const firstText = of('text')[0];
|
||||
if (firstThinking && !data.operator_native_thinking_timestamp) {
|
||||
data.operator_native_thinking_timestamp = firstThinking.timestamp;
|
||||
}
|
||||
if (firstText && !data.operator_raw_thinking_timestamp) {
|
||||
data.operator_raw_thinking_timestamp = firstText.timestamp;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply one `checker_event` (attempt_started / attempt_finished / run_outcome)
|
||||
* to the block list. Attempts are keyed by attempt id so live events and the
|
||||
@@ -106,7 +176,10 @@ function applyCheckerLog(blocks: StepBlock[], log: any): void {
|
||||
const items = existing?.data?.items?.length
|
||||
? existing.data.items
|
||||
: verdicts.map((v: any) => ({ kind: v.kind, text: v.item_text, when: v.when }));
|
||||
const data = {
|
||||
// The ledger backfill carries the attempt's persisted transcript as
|
||||
// ready-made segments; a live block keeps the ones it streamed.
|
||||
const streamSegments = mergeStreamSegments(existing?.data?.stream_segments, d.stream_segments);
|
||||
const data: any = {
|
||||
...(existing?.data || {}),
|
||||
...d,
|
||||
status: d.status || 'done',
|
||||
@@ -114,11 +187,13 @@ function applyCheckerLog(blocks: StepBlock[], log: any): void {
|
||||
items,
|
||||
findings: Array.isArray(d.findings) ? d.findings : (existing?.data?.findings || []),
|
||||
generic_tools: existing?.data?.generic_tools || [],
|
||||
stream_segments: streamSegments,
|
||||
started_at: startedAt,
|
||||
finished_at: ts,
|
||||
duration: startedAt !== undefined && ts !== undefined ? Math.max(0, ts - startedAt) : undefined,
|
||||
isCompleted: true
|
||||
};
|
||||
if (streamSegments) applyFlatStreamFields(data, streamSegments);
|
||||
const block: StepBlock = { id, type: 'checker', timestamp: existing?.timestamp || log.timestamp, data };
|
||||
if (existing) blocks[idx] = block; else blocks.push(block);
|
||||
}
|
||||
@@ -224,12 +299,6 @@ export function consolidateLogsToBlocks(rawLogs: any[]): StepBlock[] {
|
||||
if (segmentIndex > -1) segments[segmentIndex] = segment; else segments.push(segment);
|
||||
checkerData.stream_segments = segments;
|
||||
// Flat fields stay as the joined text for consumers that only know steps.
|
||||
const joined = (type: string) => segments
|
||||
.filter((s) => s.stream_type === type && s.text.trim())
|
||||
.map((s) => s.text)
|
||||
.join('\n\n');
|
||||
checkerData.operator_native_thinking = joined('thinking');
|
||||
checkerData.operator_raw_thinking = joined('text');
|
||||
if (streamType === 'thinking') {
|
||||
checkerData.operator_native_thinking_timestamp =
|
||||
checkerData.operator_native_thinking_timestamp || log.timestamp;
|
||||
@@ -237,6 +306,7 @@ export function consolidateLogsToBlocks(rawLogs: any[]): StepBlock[] {
|
||||
checkerData.operator_raw_thinking_timestamp =
|
||||
checkerData.operator_raw_thinking_timestamp || log.timestamp;
|
||||
}
|
||||
applyFlatStreamFields(checkerData, segments);
|
||||
if (execId && !checkerData.execution_id) {
|
||||
checkerData.execution_id = execId;
|
||||
}
|
||||
|
||||
@@ -662,24 +662,34 @@ export function getToolDisplayLabel(tool: any, isFirstSaveNote: boolean = false)
|
||||
const q = args.specific_query || args.query || '';
|
||||
return q ? `Analyzing audio track: "${q}"` : 'Analyzing recording audio track';
|
||||
}
|
||||
case 'get_step_detail':
|
||||
case 'get_step_details': {
|
||||
const n = args.step_no ?? args.step_number ?? args.step;
|
||||
case 'search_history': {
|
||||
const q = args.query || '';
|
||||
const range = Array.isArray(args.step_range) && args.step_range.length
|
||||
? ` in steps ${args.step_range[0]}–${args.step_range[args.step_range.length - 1]}`
|
||||
: '';
|
||||
return q ? `Searching execution history for "${q}"${range}` : `Searching execution history${range}`;
|
||||
}
|
||||
case 'replay_steps': {
|
||||
const n = args.start_step;
|
||||
const end = args.end_step;
|
||||
if (n !== undefined && n !== '' && end !== undefined && end !== null && end !== '' && String(end) !== String(n)) {
|
||||
return `Reviewing steps ${n}–${end}`;
|
||||
}
|
||||
return n !== undefined && n !== '' ? `Reviewing step ${n}` : 'Reviewing step details';
|
||||
}
|
||||
case 'get_step_screenshot': {
|
||||
const n = args.step_no ?? args.step_number ?? args.step;
|
||||
const which = String(args.which || '').toLowerCase() === 'post' ? 'after' : 'before';
|
||||
const n = args.step_number;
|
||||
const variant = String(args.which || '').toLowerCase();
|
||||
if (variant === 'overlay') {
|
||||
return n !== undefined && n !== '' ? `Looking at where step ${n}'s action landed` : 'Looking at where an action landed';
|
||||
}
|
||||
const which = variant === 'post' ? 'after' : 'before';
|
||||
return n !== undefined && n !== '' ? `Looking at the screen ${which} step ${n}` : 'Looking at a step screenshot';
|
||||
}
|
||||
case 'probe_device': {
|
||||
const kind = args.kind ? String(args.kind).replace(/_/g, ' ') : '';
|
||||
return kind ? `Reading ${kind} from the device` : 'Reading device state';
|
||||
}
|
||||
case 'search_history_for_text': {
|
||||
const q = args.text || args.query || '';
|
||||
return q ? `Searching execution trace for "${q}"` : 'Searching execution trace';
|
||||
}
|
||||
case 'outputter':
|
||||
case 'output_synthesis':
|
||||
return 'Synthesizing output report';
|
||||
@@ -751,15 +761,14 @@ export function getToolIcon(tool: any): string {
|
||||
return 'smart_toy';
|
||||
case 'analyze_audio_only':
|
||||
return 'graphic_eq';
|
||||
case 'get_step_detail':
|
||||
case 'get_step_details':
|
||||
case 'search_history':
|
||||
return 'find_in_page';
|
||||
case 'replay_steps':
|
||||
return 'manage_search';
|
||||
case 'get_step_screenshot':
|
||||
return 'image_search';
|
||||
case 'probe_device':
|
||||
return 'sensors';
|
||||
case 'search_history_for_text':
|
||||
return 'find_in_page';
|
||||
case 'outputter':
|
||||
case 'output_synthesis':
|
||||
return 'assignment_turned_in';
|
||||
|
||||
+149
-111
@@ -32,9 +32,9 @@ The release decision is computed by the caller from the verdicts
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal
|
||||
@@ -46,12 +46,15 @@ from pydantic import BaseModel, Field, ValidationError
|
||||
|
||||
from artemis.constants import CHECKER_MAX_ITERATIONS
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.data_engine.context_vars import CURRENT_TRACE_ID
|
||||
from artemis.data_engine.trace import trace, trace_langchain_tool
|
||||
from artemis.services.llm import acomplete, get_llm, invoke_llm_with_timeout_message
|
||||
from artemis.tools.history import get_history_tools
|
||||
from artemis.tools.scratchpad import get_read_note_tool_pure
|
||||
from artemis.tools.tool_wrapper import (
|
||||
get_tool_result_content,
|
||||
invoke_tool_with_injection,
|
||||
split_multimodal_result,
|
||||
tool_result_messages,
|
||||
)
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.ocr_api import is_ocr_configured, perform_ocr
|
||||
@@ -233,77 +236,6 @@ def get_probe_tool(ctx: ArtemisContext) -> BaseTool:
|
||||
return probe_device
|
||||
|
||||
|
||||
# --- History / step evidence tools ---------------------------------------------------
|
||||
|
||||
|
||||
def get_step_detail_tool(ctx: ArtemisContext) -> BaseTool:
|
||||
@tool
|
||||
def get_step_detail(step_no: int) -> str:
|
||||
"""Returns the recorded details of one execution step (action taken,
|
||||
summary, metadata) by its step number."""
|
||||
if not ctx.data_engine:
|
||||
return "Error: no execution history available."
|
||||
record = ctx.data_engine.get_step_record(step_no)
|
||||
if record is None:
|
||||
return f"Error: step {step_no} not found."
|
||||
payload = {
|
||||
"step_number": record.step_number,
|
||||
"timestamp": record.timestamp,
|
||||
"summary": record.summary,
|
||||
"action_taken": record.action_taken,
|
||||
"last_execution_result": record.last_execution_result,
|
||||
"extra_metadata": record.extra_metadata,
|
||||
"has_pre_screenshot": bool(record.pre_image_name),
|
||||
"has_post_screenshot": bool(record.post_image_name),
|
||||
}
|
||||
try:
|
||||
return json.dumps(payload, ensure_ascii=False, default=str)
|
||||
except Exception as e:
|
||||
return f"Error serializing step: {e}"
|
||||
|
||||
return get_step_detail
|
||||
|
||||
|
||||
class _ScreenshotRequest:
|
||||
"""Sentinel result carrying an image to splice into the conversation."""
|
||||
|
||||
def __init__(self, description: str, image_b64: str | None):
|
||||
self.description = description
|
||||
self.image_b64 = image_b64
|
||||
|
||||
|
||||
def get_step_screenshot_tool(ctx: ArtemisContext) -> BaseTool:
|
||||
@tool
|
||||
def get_step_screenshot(step_no: int, which: str = "pre") -> str:
|
||||
"""Attaches the recorded screenshot of a step to the conversation.
|
||||
`which` is 'pre' (state observed at the start of the step) or 'post'
|
||||
(state after the step's action)."""
|
||||
# The real work happens in the loop's interception; this body only
|
||||
# documents the contract for schema generation.
|
||||
return f"screenshot request: step {step_no} ({which})"
|
||||
|
||||
return get_step_screenshot
|
||||
|
||||
|
||||
def _load_step_screenshot(ctx: ArtemisContext, step_no: int, which: str) -> _ScreenshotRequest:
|
||||
if which not in ("pre", "post"):
|
||||
return _ScreenshotRequest(f"Error: 'which' must be 'pre' or 'post', got '{which}'.", None)
|
||||
if not ctx.data_engine:
|
||||
return _ScreenshotRequest("Error: no execution history available.", None)
|
||||
path = ctx.data_engine.get_step_image_path(step_no, which)
|
||||
if path is None:
|
||||
return _ScreenshotRequest(
|
||||
f"No {which}-action screenshot recorded for step {step_no}.", None
|
||||
)
|
||||
try:
|
||||
image_b64 = base64.b64encode(Path(path).read_bytes()).decode("utf-8")
|
||||
except Exception as e:
|
||||
return _ScreenshotRequest(f"Error reading screenshot: {e}", None)
|
||||
return _ScreenshotRequest(
|
||||
f"Screenshot of step {step_no} ({which}-action) is attached below.", image_b64
|
||||
)
|
||||
|
||||
|
||||
# --- Prompt assembly (segments x entry x content x config) ---------------------------
|
||||
|
||||
|
||||
@@ -365,10 +297,12 @@ def build_checker_tools(
|
||||
the final screen through its prompt (not a tool), so the table is identical
|
||||
for both entries; the difference lives in what the prompt provides.
|
||||
Device actions, note writes, and sub-agents are registered for neither.
|
||||
|
||||
The history tools are the shared ``search_history`` / ``replay_steps`` /
|
||||
``get_step_screenshot`` definitions (read-only, deterministic).
|
||||
"""
|
||||
tools: list[BaseTool] = [
|
||||
get_step_detail_tool(ctx),
|
||||
get_step_screenshot_tool(ctx),
|
||||
*get_history_tools(ctx),
|
||||
get_read_note_tool_pure(ctx),
|
||||
]
|
||||
if probes_enabled(ctx):
|
||||
@@ -507,37 +441,6 @@ async def _run_check_loop(
|
||||
for tc in response.tool_calls:
|
||||
tool_name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
|
||||
args = dict(tc["args"])
|
||||
if tool_name == "get_step_screenshot":
|
||||
shot = _load_step_screenshot(
|
||||
ctx, int(args.get("step_no", -1)), str(args.get("which", "pre"))
|
||||
)
|
||||
# The image is spliced in below; invoking the (traced) tool as
|
||||
# well records the lookup as a tool trace so the timeline shows
|
||||
# it in order with the rest of the check.
|
||||
screenshot_tool = next((t for t in traced_tools if t.name == tool_name), None)
|
||||
if screenshot_tool is not None:
|
||||
try:
|
||||
await invoke_tool_with_injection(
|
||||
tool=screenshot_tool, args=args, tool_call_id=tc["id"]
|
||||
)
|
||||
except Exception as e:
|
||||
logger.debug(f"Screenshot lookup trace not recorded: {e}")
|
||||
messages.append(ToolMessage(tool_call_id=tc["id"], content=shot.description))
|
||||
if shot.image_b64:
|
||||
messages.append(
|
||||
HumanMessage(
|
||||
content=[
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": f"data:image/jpeg;base64,{shot.image_b64}"
|
||||
},
|
||||
}
|
||||
]
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
tool_to_run = next((t for t in traced_tools if t.name == tool_name), None)
|
||||
if tool_to_run is None:
|
||||
messages.append(
|
||||
@@ -552,18 +455,136 @@ async def _run_check_loop(
|
||||
result_obj = await invoke_tool_with_injection(
|
||||
tool=tool_to_run, args=args, tool_call_id=tc["id"]
|
||||
)
|
||||
result_str = get_tool_result_content(result_obj)
|
||||
status = "error" if str(result_str).startswith("Error") else "success"
|
||||
text, _ = split_multimodal_result(result_obj)
|
||||
status = "error" if text.startswith("Error") else "success"
|
||||
except Exception as e:
|
||||
result_str = f"Error running tool {tool_name}: {e}"
|
||||
result_obj = f"Error running tool {tool_name}: {e}"
|
||||
status = "error"
|
||||
messages.append(ToolMessage(tool_call_id=tc["id"], content=result_str, status=status))
|
||||
# Screenshots (get_step_screenshot) enter the conversation in the
|
||||
# carrier the model's provider accepts; text results stay a plain
|
||||
# ToolMessage.
|
||||
messages.extend(
|
||||
tool_result_messages(tc["id"], result_obj, name=tool_name, status=status, llm=llm)
|
||||
)
|
||||
|
||||
if report is None:
|
||||
report = CheckReport(verdicts=[])
|
||||
return _normalize_report(report, check_items)
|
||||
|
||||
|
||||
# --- Streamed-reasoning transcript -----------------------------------------------------
|
||||
|
||||
|
||||
class CheckerStreamCapture:
|
||||
"""Collects the Checker's own streamed LLM output for replay.
|
||||
|
||||
The LLM gateway forwards every provider delta to the DataEngine bus as an
|
||||
``llm_stream`` event carrying the emitting agent's trace id
|
||||
(``parent_trace_id``). Inside the Checker's traced scope that id is the
|
||||
Checker's own, so filtering the bus by it yields exactly the text the live
|
||||
UI rendered for this attempt — one segment per (LLM turn, stream kind),
|
||||
stamped with the time its first chunk arrived. A mid-stream reset
|
||||
(``llm_stream_reset`` / ``discard``) drops that turn's partial text, the
|
||||
same way the live UI does.
|
||||
"""
|
||||
|
||||
def __init__(self, ctx: ArtemisContext):
|
||||
self._engine = getattr(ctx, "data_engine", None)
|
||||
try:
|
||||
trace_id = CURRENT_TRACE_ID.get()
|
||||
except LookupError:
|
||||
trace_id = None
|
||||
self.trace_id: str | None = str(trace_id) if trace_id else None
|
||||
self._segments: dict[tuple[str, str], dict] = {}
|
||||
self._subscribed = False
|
||||
|
||||
def __enter__(self) -> CheckerStreamCapture:
|
||||
subscribe = getattr(self._engine, "subscribe", None)
|
||||
if self.trace_id and callable(subscribe):
|
||||
try:
|
||||
subscribe(self._on_event)
|
||||
self._subscribed = True
|
||||
except Exception as e:
|
||||
logger.debug(f"Checker stream capture not attached: {e}")
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc) -> None:
|
||||
if not self._subscribed:
|
||||
return
|
||||
self._subscribed = False
|
||||
try:
|
||||
self._engine.unsubscribe(self._on_event)
|
||||
except Exception as e:
|
||||
logger.debug(f"Checker stream capture not detached: {e}")
|
||||
|
||||
def _on_event(self, event_type: str, data: Any) -> None:
|
||||
if not isinstance(data, dict):
|
||||
return
|
||||
if event_type == "llm_stream":
|
||||
if str(data.get("parent_trace_id") or "") != self.trace_id:
|
||||
return
|
||||
exec_id = str(data.get("execution_id") or "")
|
||||
chunk = data.get("chunk")
|
||||
if not exec_id or not isinstance(chunk, str) or not chunk:
|
||||
return
|
||||
role = "thought" if data.get("stream_type") == "thinking" else "answer"
|
||||
segment = self._segments.get((exec_id, role))
|
||||
if segment is None:
|
||||
self._segments[(exec_id, role)] = {
|
||||
"execution_id": exec_id,
|
||||
"role": role,
|
||||
"when": time.time(),
|
||||
"text": chunk,
|
||||
}
|
||||
else:
|
||||
segment["text"] += chunk
|
||||
elif event_type == "llm_stream_reset" and data.get("action") == "discard":
|
||||
exec_id = str(data.get("stream_exec_id") or data.get("stream_execution_id") or "")
|
||||
if exec_id:
|
||||
for key in [k for k in self._segments if k[0] == exec_id]:
|
||||
del self._segments[key]
|
||||
|
||||
@property
|
||||
def segments(self) -> list[dict]:
|
||||
"""Segments in the order their first chunk arrived."""
|
||||
return [dict(s) for s in sorted(self._segments.values(), key=lambda s: s["when"])]
|
||||
|
||||
|
||||
def _persist_stream(
|
||||
ctx: ArtemisContext,
|
||||
capture: CheckerStreamCapture,
|
||||
*,
|
||||
attempt_id: str | None,
|
||||
phase: str,
|
||||
checkpoint_id: str,
|
||||
) -> None:
|
||||
"""Writes the attempt's transcript next to the ledger (best-effort).
|
||||
|
||||
Runs when the attempt ends for any reason — verdict, exception, timeout or
|
||||
supersession — so a cancelled attempt still leaves what it said. A
|
||||
transcript is observability, not execution state: writing it never
|
||||
influences booking or release (the ledger stays the source of truth).
|
||||
"""
|
||||
segments = capture.segments
|
||||
if not attempt_id or not segments:
|
||||
return
|
||||
try:
|
||||
from artemis.graph.checkpoints import append_attempt_stream_record
|
||||
|
||||
append_attempt_stream_record(
|
||||
ctx.data_engine.base_dir,
|
||||
{
|
||||
"attempt_id": attempt_id,
|
||||
"checkpoint_id": checkpoint_id,
|
||||
"phase": phase,
|
||||
"trace_id": capture.trace_id,
|
||||
"segments": segments,
|
||||
},
|
||||
)
|
||||
except Exception as e:
|
||||
logger.debug(f"Checker stream transcript for {attempt_id} not persisted: {e}")
|
||||
|
||||
|
||||
# --- Entry points --------------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -593,12 +614,13 @@ async def run_checkpoint_check(
|
||||
state) and persistent-state probes are the only valid sources.
|
||||
"""
|
||||
items = list(check_items)
|
||||
checkpoint_id = attempt_id.split("#", 1)[0] if attempt_id else ""
|
||||
logger.info(f"Starting checkpoint check for '{subgoal_text}' ({len(items)} item(s))")
|
||||
_announce(
|
||||
ctx,
|
||||
attempt_id=attempt_id,
|
||||
phase="checkpoint",
|
||||
checkpoint_id=attempt_id.split("#", 1)[0] if attempt_id else "",
|
||||
checkpoint_id=checkpoint_id,
|
||||
subgoal_text=subgoal_text,
|
||||
anchor_step_id=getattr(anchor, "anchor_step_id", None),
|
||||
check_items=items,
|
||||
@@ -638,7 +660,17 @@ async def run_checkpoint_check(
|
||||
SystemMessage(content=system_content),
|
||||
HumanMessage(content=human_text),
|
||||
]
|
||||
with CheckerStreamCapture(ctx) as capture:
|
||||
try:
|
||||
return await _run_check_loop(ctx, messages, tools, items)
|
||||
finally:
|
||||
_persist_stream(
|
||||
ctx,
|
||||
capture,
|
||||
attempt_id=attempt_id,
|
||||
phase="checkpoint",
|
||||
checkpoint_id=checkpoint_id,
|
||||
)
|
||||
|
||||
|
||||
async def _capture_final_screen(ctx: ArtemisContext) -> tuple[str | None, str]:
|
||||
@@ -748,4 +780,10 @@ async def run_final_check(
|
||||
SystemMessage(content=system_content),
|
||||
HumanMessage(content=content),
|
||||
]
|
||||
with CheckerStreamCapture(ctx) as capture:
|
||||
try:
|
||||
return await _run_check_loop(ctx, messages, tools, items)
|
||||
finally:
|
||||
_persist_stream(
|
||||
ctx, capture, attempt_id=attempt_id, phase="final", checkpoint_id="final"
|
||||
)
|
||||
@@ -33,6 +33,7 @@ from artemis.graph.state import State
|
||||
from artemis.services.llm import acomplete, get_llm, invoke_llm_with_timeout_message
|
||||
from artemis.tools.command_tool import get_run_short_adb_command_tool
|
||||
from artemis.tools.diagnoser_submit_answer_tool import get_submit_answer_tool
|
||||
from artemis.tools.history import get_history_tools
|
||||
from artemis.tools.index import get_tool_by_name
|
||||
from artemis.tools.log_tool import get_analyze_logs_tool
|
||||
from artemis.tools.mobile.read_hierarchy import get_ui_hierarchy_tool
|
||||
@@ -40,6 +41,8 @@ from artemis.tools.scratchpad import get_list_notes_tool, get_read_note_tool
|
||||
from artemis.tools.tool_wrapper import (
|
||||
get_tool_result_content,
|
||||
invoke_tool_with_injection,
|
||||
split_multimodal_result,
|
||||
tool_result_messages,
|
||||
)
|
||||
from artemis.tools.video_tool import get_video_analyzer_tool
|
||||
from artemis.tools.wait_tool import get_wait_tool
|
||||
@@ -63,6 +66,8 @@ class Diagnoser:
|
||||
def __init__(self, ctx: ArtemisContext):
|
||||
self.ctx = ctx
|
||||
self.is_device_online = self._is_device_online()
|
||||
#: The model of the current run; decides how screenshot results travel.
|
||||
self._llm = None
|
||||
|
||||
def _is_device_online(self) -> bool:
|
||||
try:
|
||||
@@ -101,6 +106,7 @@ class Diagnoser:
|
||||
get_run_short_adb_command_tool(self.ctx),
|
||||
get_wait_tool(self.ctx),
|
||||
get_ui_hierarchy_tool(self.ctx),
|
||||
*get_history_tools(self.ctx),
|
||||
get_submit_answer_tool(self.ctx),
|
||||
]
|
||||
if not self.is_device_online:
|
||||
@@ -452,8 +458,13 @@ class Diagnoser:
|
||||
status="success",
|
||||
)
|
||||
|
||||
async def _run_tool(self, tc: dict, traced_tools: list, state: State) -> ToolMessage:
|
||||
"""Executes one requested tool call and wraps its result as a tool message."""
|
||||
async def _run_tool(self, tc: dict, traced_tools: list, state: State) -> list[BaseMessage]:
|
||||
"""Executes one requested tool call and wraps its result as messages.
|
||||
|
||||
Text results are one tool message; a step screenshot travels in the
|
||||
carrier the model's provider accepts (``tool_result_messages``), so
|
||||
base64 never lands in a text field.
|
||||
"""
|
||||
tool_name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
|
||||
logger.info(f"Diagnoser requested tool: {tool_name}")
|
||||
|
||||
@@ -461,7 +472,7 @@ class Diagnoser:
|
||||
status = "success"
|
||||
try:
|
||||
if tool_name == "video_analyzer":
|
||||
return self._start_video_job(tc, traced_tools, state)
|
||||
return [self._start_video_job(tc, traced_tools, state)]
|
||||
|
||||
if ":" in tool_name:
|
||||
tool_to_run = get_tool_by_name(tool_name, traced_tools)
|
||||
@@ -482,12 +493,7 @@ class Diagnoser:
|
||||
state=state,
|
||||
)
|
||||
result = get_tool_result_content(result_obj)
|
||||
if isinstance(result, list):
|
||||
result_str = "\n".join(map(str, result))
|
||||
elif not isinstance(result, str):
|
||||
result_str = str(result)
|
||||
else:
|
||||
result_str = result
|
||||
result_str, _ = split_multimodal_result(result)
|
||||
span.result = result
|
||||
if result_str.startswith("Error"):
|
||||
status = "error"
|
||||
@@ -506,26 +512,21 @@ class Diagnoser:
|
||||
result = f"Error running tool {tool_name}: {e}"
|
||||
status = "error"
|
||||
|
||||
return ToolMessage(
|
||||
tool_call_id=tc["id"],
|
||||
content=result,
|
||||
status=status,
|
||||
)
|
||||
return tool_result_messages(tc["id"], result, name=tool_name, status=status, llm=self._llm)
|
||||
|
||||
async def _execute_tool_calls(
|
||||
self, tool_calls: list, traced_tools: list, state: State
|
||||
) -> list[ToolMessage]:
|
||||
) -> list[BaseMessage]:
|
||||
"""Executes all requested tools in parallel, skipping ``google_search``."""
|
||||
active_tool_calls = [
|
||||
tc
|
||||
for tc in tool_calls
|
||||
if (tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]) != "google_search"
|
||||
]
|
||||
return list(
|
||||
await asyncio.gather(
|
||||
batches = await asyncio.gather(
|
||||
*(self._run_tool(tc, traced_tools, state) for tc in active_tool_calls)
|
||||
)
|
||||
)
|
||||
return [message for batch in batches for message in batch]
|
||||
|
||||
@trace(type="agent", name="diagnoser")
|
||||
async def run(self, prompt: str, state: State) -> str:
|
||||
@@ -534,6 +535,7 @@ class Diagnoser:
|
||||
system_prompt, checker_feedback_template = self._load_prompt_config()
|
||||
|
||||
base_llm = get_llm(ctx=self.ctx, name="diagnoser")
|
||||
self._llm = base_llm
|
||||
traced_tools = self._build_traced_tools()
|
||||
|
||||
plan_and_history = self._build_plan_and_history()
|
||||
|
||||
@@ -12,19 +12,12 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Universal Context Compressor for Artemis Flash profile.
|
||||
"""Context compression shared by Flash and Pro.
|
||||
|
||||
:class:`ScrubEdgeCompressor` implements the scrub-edge (擦洗沿) discipline
|
||||
from the history-module redesign §3.2: each message is mutated a bounded
|
||||
number of times near the tail (XML stripped at depth 1, screenshot resolved
|
||||
at depth K), then frozen forever. A late summary never backfills a frozen
|
||||
:class:`ScrubEdgeCompressor` strips XML at depth 1 and resolves screenshots
|
||||
at depth K, then freezes the message. A late summary never backfills a frozen
|
||||
message. FlashRunner uses this implementation; the Pro transcript ledger
|
||||
reuses it over its active region (see the class docstring).
|
||||
|
||||
The legacy per-turn full-rescan ``compress_flash_messages`` was removed in M5
|
||||
(2026-09-01) after serving as the M1/M2 semantic reference — its surviving
|
||||
semantics are asserted directly against ``ScrubEdgeCompressor`` in
|
||||
``tests/unit/agents/test_flash_scrub_edge.py``.
|
||||
"""
|
||||
|
||||
from typing import Any, Callable
|
||||
@@ -36,7 +29,10 @@ from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
_HISTORY_SUMMARY_PREFIX = "--- Historical Visual Transition ---\n"
|
||||
#: Header written above a resolved visual summary (public: the chunk capsule
|
||||
#: lens checks it to avoid repeating a summary already present verbatim).
|
||||
HISTORY_SUMMARY_PREFIX = "--- Historical Visual Transition ---\n"
|
||||
_HISTORY_SUMMARY_PREFIX = HISTORY_SUMMARY_PREFIX
|
||||
_UI_LIST_MARKER = "--- UI Element List ---"
|
||||
|
||||
|
||||
@@ -53,7 +49,7 @@ def _without_marked_suffix(text: str, markers: tuple[str, ...] = (_UI_LIST_MARKE
|
||||
|
||||
|
||||
class ScrubEdgeCompressor:
|
||||
"""Scrub-edge context compressor (history-module redesign §3.2, M1).
|
||||
"""Replace older observation details with summaries near the message tail.
|
||||
|
||||
Mutation discipline — every message is touched at most twice, always near
|
||||
the tail, and is then frozen forever:
|
||||
@@ -75,8 +71,8 @@ class ScrubEdgeCompressor:
|
||||
touches the few messages near the scrub edge instead of rescanning the
|
||||
whole history.
|
||||
|
||||
Profile generalization (M2): the Pro transcript ledger reuses this exact
|
||||
discipline over its active region. Its observation messages carry no
|
||||
The Pro transcript ledger uses this discipline over its active region.
|
||||
Its observation messages carry no
|
||||
``tool_call_id``, so ``summary_key_getter`` supplies the summary-job key
|
||||
(the DataEngine step id) per message; ``strip_markers`` extends the
|
||||
depth-1 strip to the Pro tail blocks (UI element list + plan recitation);
|
||||
|
||||
@@ -25,7 +25,9 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
|
||||
# 3. READING YOUR HISTORY & THE SESSION CLOCK
|
||||
- **Conversation = execution history**: The earlier turns of this conversation are your raw execution history. Each observation opens with `# CURRENT OBSERVATION [T+mm:ss]`, and each turn that executed a device action ends with an `--- Action Execution Result (T+mm:ss) ---` message (`Status: success` means the action was dispatched and observed; `Status: failed` carries the `Error:` line). Every timestamp is a session-relative `T+mm:ss` offset from the start of this session; it is the only clock you have, for pacing waits as well as for reasoning about elapsed time.
|
||||
- **Compression**: Screenshots older than a few turns are replaced by short `--- Historical Visual Transition ---` summaries and old `--- Visible UI Elements ---` lists are dropped. Long spans render as `[Era N | Steps a–b | T+.. → T+..]` blocks (a merged synopsis, segment titles, and a per-step action ledger). Because of this, if you spot critical text that you will need later (e.g., verification codes, specific titles, search lists), **you must explicitly verbalize and write those details into your text reasoning** so they persist across turns.
|
||||
{% if "recall_history" in available_tools %}- **Cold-History Recall (`recall_history`)**: Compressed history keeps the step numbers and `T+` offsets but not every detail. When you need exactly what you did or saw at an earlier step (a value you read, the result of an action, which screen you were on), call `recall_history` with keywords and/or the `[start, end]` step range shown in a chunk block; it searches the full stored history (summaries, actions, results, reasoning, on-screen text). It is a pre-decision helper: read its answer, then act.{% endif %}
|
||||
{% if "search_history" in available_tools %}- **History Lookup (`search_history`)**: Compressed history keeps the step numbers and `T+` offsets but not every detail. When you need exactly what you did or saw at an earlier step (a value you read, the result of an action, which screen you were on), call `search_history` with keywords and/or the `[start, end]` step range shown in a chunk block; it searches the full stored history (screen descriptions, actions, results, reasoning, tool calls, notes, on-screen text) and every hit carries its step number. It is a pre-decision helper: read its answer, then act.{% endif %}
|
||||
{% if "replay_steps" in available_tools %}- **Step Replay (`replay_steps`)**: Replays one earlier step (or a short range) exactly as your context showed it — screen description, reasoning, tool calls with their results, the action and its execution result. Use the step numbers shown in the compressed history{% if "search_history" in available_tools %} or returned by `search_history`{% endif %}.{% endif %}
|
||||
{% if "get_step_screenshot" in available_tools %}- **Step Screenshot (`get_step_screenshot`)**: Attaches the stored screenshot of an earlier step (which: pre, post, or overlay — the pre screenshot with the action drawn on it) when you need to see that screen again rather than read about it.{% endif %}
|
||||
{% if "video_analyzer" in available_tools %}- **Video Recall (`video_analyzer`)**: The whole session is continuously screen-recorded on the same `T+` clock. When a static screenshot is not enough — a video or animation that played, something that changed between two observations, a transient toast, a fast-scrolling list — call `video_analyzer` with a time range expressed on that clock (e.g., `from T+01:20 to T+01:35`, or `from 80s to 95s`) and a precise question. Prefer narrow ranges; ask for audio explicitly when only audio matters. It is a pre-decision helper: read its answer, then act.{% endif %}
|
||||
|
||||
---
|
||||
|
||||
@@ -74,7 +74,13 @@ from artemis.services.llm import (
|
||||
get_llm,
|
||||
invoke_llm_with_timeout_message,
|
||||
)
|
||||
from artemis.utils.coordinates import parse_swipe_parameters
|
||||
from artemis.tools.history import history_tool_declarations
|
||||
from artemis.tools.tool_wrapper import tool_result_messages
|
||||
from artemis.utils.coordinates import (
|
||||
COORDINATE_SPACE_KEY,
|
||||
COORDINATE_SPACE_NORMALIZED,
|
||||
parse_swipe_parameters,
|
||||
)
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
@@ -178,25 +184,14 @@ class FlashRunner:
|
||||
setup = getattr(self.ctx, "execution_setup", None)
|
||||
return getattr(setup, "video_recording_tools_enabled", False) is True
|
||||
|
||||
def _recall_enabled(self) -> bool:
|
||||
"""Same gate as the Pro operator: a DataEngine session and the recall config."""
|
||||
from artemis.tools.history_recall import _recall_available
|
||||
|
||||
try:
|
||||
return bool(_recall_available(self.ctx))
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def _get_tools(self) -> list:
|
||||
tools = list(VALIDATOR_TOOLS_DECLARATION)
|
||||
tools.insert(1, CLICK_SEQUENCE_TOOL)
|
||||
tools.append(ASK_EXPLORER_TOOL)
|
||||
# Helper tools shared with the Pro operator, declared next to their
|
||||
# LangChain tools so each contract has one owner.
|
||||
if self._recall_enabled():
|
||||
from artemis.tools.history_recall import RECALL_HISTORY_TOOL
|
||||
|
||||
tools.append(RECALL_HISTORY_TOOL)
|
||||
# Helper tools shared with the Pro operator: the history tools are
|
||||
# declared from the very same args schemas the LangChain tools bind
|
||||
# (same availability gates: a DataEngine session, recall config).
|
||||
tools.extend(history_tool_declarations(self.ctx))
|
||||
if self._video_tools_enabled():
|
||||
from artemis.tools.video_tool import VIDEO_ANALYZER_TOOL
|
||||
|
||||
@@ -261,6 +256,9 @@ class FlashRunner:
|
||||
pass
|
||||
return ledger
|
||||
|
||||
#: The model of the current run; decides how screenshot tool results travel.
|
||||
_llm = None
|
||||
|
||||
def _init_llm(self):
|
||||
"""Initializes the Universal LLM via the Service Layer."""
|
||||
try:
|
||||
@@ -579,6 +577,9 @@ class FlashRunner:
|
||||
|
||||
norm_coords, norm_start, norm_end = self._extract_normalized_coordinates(name, args)
|
||||
|
||||
# Flash records the model's own 0–1000 coordinates verbatim; the
|
||||
# explicit space marker keeps every later normalization pass
|
||||
# (agent-friendly steps, MCP inspector) a no-op on this record.
|
||||
action_dict = {
|
||||
"action": name,
|
||||
"coordinates": (
|
||||
@@ -587,6 +588,7 @@ class FlashRunner:
|
||||
or args.get("sequence")
|
||||
or norm_coords
|
||||
),
|
||||
COORDINATE_SPACE_KEY: COORDINATE_SPACE_NORMALIZED,
|
||||
"args": args,
|
||||
}
|
||||
if norm_coords:
|
||||
@@ -709,21 +711,19 @@ class FlashRunner:
|
||||
if exec_result.ui_elements_text:
|
||||
xml_list = exec_result.ui_elements_text
|
||||
|
||||
# Helper tools may return multimodal blocks (recalled screenshots);
|
||||
# Helper tools may return multimodal blocks (a step screenshot);
|
||||
# device actions always report text — their post screenshot is the
|
||||
# next observation tail, never a tool-message image.
|
||||
# next observation tail, never a tool-message image. The image
|
||||
# carrier follows the model's provider (tool_result_messages).
|
||||
raw_blocks = exec_result.raw_result if name not in action_names else None
|
||||
content = (
|
||||
raw_blocks
|
||||
if isinstance(raw_blocks, list) and raw_blocks
|
||||
else exec_result.text_summary or f"Action '{name}' completed."
|
||||
)
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc_id,
|
||||
name=name,
|
||||
content=content,
|
||||
status=exec_result.status,
|
||||
messages.extend(
|
||||
tool_result_messages(
|
||||
tc_id, content, name=name, status=exec_result.status, llm=self._llm
|
||||
)
|
||||
)
|
||||
|
||||
@@ -842,6 +842,7 @@ class FlashRunner:
|
||||
|
||||
# 1. Initialize Universal LLM via Service Layer
|
||||
llm = self._init_llm()
|
||||
self._llm = llm
|
||||
|
||||
tools_declaration = self._get_tools()
|
||||
report_only_tools = [t for t in tools_declaration if t.name == "report_task_status"]
|
||||
|
||||
@@ -3,6 +3,6 @@ You are a **History Analyzer**, expert in analyzing multi-agent session executio
|
||||
|
||||
# MANDATE
|
||||
1. You are provided with the **Task Plan and Execution History** summarizing the high-level plan and history steps.
|
||||
2. For queries about specific step details (such as precise actions, coordinates, `operator_raw_thinking`, or `last_execution_result`), you should call the `get_step_details` tool.
|
||||
3. Never assume, guess, or invent details not visible in the Task Plan and Execution History. Always query `get_step_details` for specifics. If details are unavailable, state it clearly.
|
||||
2. For queries about specific step details (such as precise actions, coordinates, the reasoning behind a step, or its execution result), call `replay_steps` for the step or range in question; use `search_history` to locate the right steps by keyword when the step numbers are not obvious, and `get_step_screenshot` when you need to see what the screen showed.
|
||||
3. Never assume, guess, or invent details not visible in the Task Plan and Execution History. Always query `replay_steps` for specifics. If details are unavailable, state it clearly.
|
||||
4. Make good use of note tools to see what notes are available. When analyzing history records, open and cross-reference relevant notes to ensure you have grasped all necessary information.
|
||||
@@ -12,83 +12,58 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import json
|
||||
"""History Analyzer: answers natural-language questions about the session history.
|
||||
|
||||
Mounts the shared history tools (``search_history`` / ``replay_steps`` /
|
||||
``get_step_screenshot``) plus the read-only note tools; tool results enter the
|
||||
conversation through ``tool_result_messages`` so a fetched screenshot travels
|
||||
in the carrier the model's provider accepts.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.messages import BaseMessage, HumanMessage, SystemMessage
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
from langchain_core.messages import (
|
||||
BaseMessage,
|
||||
HumanMessage,
|
||||
SystemMessage,
|
||||
ToolMessage,
|
||||
)
|
||||
from langchain_core.tools import BaseTool, tool
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.data_engine.trace import CURRENT_TRACE_ID, TraceSpan, trace
|
||||
from artemis.data_engine.trace import TraceSpan, trace
|
||||
from artemis.memory.context_policy import build_history_for
|
||||
from artemis.services.llm import acomplete, get_llm, invoke_llm_with_timeout_message
|
||||
from artemis.tools.history import get_history_tools
|
||||
from artemis.tools.scratchpad import (
|
||||
get_list_notes_tool_pure,
|
||||
get_read_note_tool_pure,
|
||||
)
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.notes import (
|
||||
get_note_file_path,
|
||||
get_notes_dir,
|
||||
from artemis.tools.tool_wrapper import (
|
||||
invoke_tool_with_injection,
|
||||
split_multimodal_result,
|
||||
tool_result_messages,
|
||||
)
|
||||
from artemis.memory.context_policy import build_history_for
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.notes import get_note_file_path
|
||||
from artemis.utils.task_tree import get_active_subgoal_hashes
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
_FALLBACK_SYSTEM_PROMPT = (
|
||||
"You are a History Analyzer. Your role is to analyze the execution history"
|
||||
" of a session and answer user queries in natural language.\nIf you need"
|
||||
" the full record of specific steps, use the `replay_steps` tool; use"
|
||||
" `search_history` to locate steps by keyword and `get_step_screenshot` to"
|
||||
" look at a step's screen."
|
||||
)
|
||||
|
||||
|
||||
class HistoryAnalyzer:
|
||||
def __init__(self, ctx: ArtemisContext):
|
||||
self.ctx = ctx
|
||||
|
||||
def exec_get_step_details(self, start_step: int, end_step: int) -> str:
|
||||
"""Retrieve the detailed, full-granularity information for a range of steps (inclusive).
|
||||
|
||||
Use this when you need to inspect specific actions taken, operator
|
||||
thinking, or execution results.
|
||||
"""
|
||||
try:
|
||||
s_step = int(start_step)
|
||||
e_step = int(end_step)
|
||||
except (ValueError, TypeError):
|
||||
return (
|
||||
f"Error: start_step and end_step must be integers, got {start_step} and {end_step}."
|
||||
)
|
||||
|
||||
matched_steps = []
|
||||
for s in getattr(self, "history_steps", []):
|
||||
step_num = s.get("step_number")
|
||||
if step_num is not None and s_step <= step_num <= e_step:
|
||||
details = {
|
||||
"step_number": s.get("step_number"),
|
||||
"relative_time": s.get("relative_time"),
|
||||
"summary": s.get("summary"),
|
||||
"action_taken": s.get("action_taken"),
|
||||
"operator_raw_thinking": s.get("operator_raw_thinking"),
|
||||
"last_execution_result": s.get("last_execution_result"),
|
||||
}
|
||||
matched_steps.append(details)
|
||||
if not matched_steps:
|
||||
return f"No steps found in range [{s_step}, {e_step}]."
|
||||
return json.dumps(matched_steps, indent=2, ensure_ascii=False)
|
||||
|
||||
def _get_step_details_tool(self, history_steps: list[dict[str, Any]]) -> BaseTool:
|
||||
self.history_steps = history_steps
|
||||
|
||||
@tool
|
||||
def get_step_details(start_step: int, end_step: int) -> str:
|
||||
"""Retrieve the detailed, full-granularity information for a range of steps (inclusive).
|
||||
|
||||
Use this when you need to inspect specific actions taken, operator
|
||||
thinking, or execution results.
|
||||
"""
|
||||
return self.exec_get_step_details(start_step, end_step)
|
||||
|
||||
return get_step_details
|
||||
def _build_tools(self) -> list[BaseTool]:
|
||||
return [
|
||||
*get_history_tools(self.ctx),
|
||||
get_list_notes_tool_pure(self.ctx),
|
||||
get_read_note_tool_pure(self.ctx),
|
||||
]
|
||||
|
||||
@trace(type="agent", name="history_analyzer")
|
||||
async def run(self, query: str) -> str:
|
||||
@@ -96,14 +71,13 @@ class HistoryAnalyzer:
|
||||
return "Error: DataEngine is not available to retrieve history."
|
||||
|
||||
# 1. Fetch all steps from Data Engine
|
||||
self.history_steps = self.ctx.data_engine.get_agent_friendly_steps()
|
||||
if not self.history_steps:
|
||||
history_steps = self.ctx.data_engine.get_agent_friendly_steps()
|
||||
if not history_steps:
|
||||
return "No history recorded for this session yet."
|
||||
|
||||
# 2. Build the plan and history or step list
|
||||
plan_and_history = ""
|
||||
try:
|
||||
notes_dir = get_notes_dir(self.ctx.data_engine.base_dir)
|
||||
current_plan = ""
|
||||
current_path = get_note_file_path(self.ctx.data_engine.base_dir, "task_plan")
|
||||
if current_path.exists():
|
||||
@@ -114,7 +88,7 @@ class HistoryAnalyzer:
|
||||
plan_and_history = build_history_for(
|
||||
"history_analyzer",
|
||||
current_plan,
|
||||
self.history_steps,
|
||||
history_steps,
|
||||
active_subgoal_hash,
|
||||
engine=self.ctx.data_engine,
|
||||
)
|
||||
@@ -127,7 +101,7 @@ class HistoryAnalyzer:
|
||||
[
|
||||
f"- Step {s.get('step_number')} ({s.get('relative_time')}):"
|
||||
f" {s.get('summary') or 'No summary'}"
|
||||
for s in self.history_steps
|
||||
for s in history_steps
|
||||
]
|
||||
)
|
||||
|
||||
@@ -136,12 +110,7 @@ class HistoryAnalyzer:
|
||||
if prompt_path.exists():
|
||||
system_prompt_template = prompt_path.read_text(encoding="utf-8")
|
||||
else:
|
||||
system_prompt_template = (
|
||||
"You are a History Analyzer. Your role is to analyze the"
|
||||
" execution history of a session and answer user queries in"
|
||||
" natural language.\nIf you need specific details for a range"
|
||||
" of steps, use the `get_step_details` tool."
|
||||
)
|
||||
system_prompt_template = _FALLBACK_SYSTEM_PROMPT
|
||||
|
||||
system_message_content = (
|
||||
system_prompt_template + f"\n\n### Task Plan and Execution History:\n{plan_and_history}"
|
||||
@@ -149,11 +118,8 @@ class HistoryAnalyzer:
|
||||
|
||||
# 4. Prepare LLM and bind tools
|
||||
llm = get_llm(ctx=self.ctx, name="history_analyzer")
|
||||
get_step_details = self._get_step_details_tool(self.history_steps)
|
||||
|
||||
list_notes_tool = get_list_notes_tool_pure(self.ctx)
|
||||
read_note_tool = get_read_note_tool_pure(self.ctx)
|
||||
llm = llm.bind_tools(tools=[get_step_details, list_notes_tool, read_note_tool])
|
||||
tools = self._build_tools()
|
||||
llm = llm.bind_tools(tools=tools)
|
||||
|
||||
# 5. ReAct Loop
|
||||
messages: list[BaseMessage] = [
|
||||
@@ -178,62 +144,38 @@ class HistoryAnalyzer:
|
||||
# Process tool calls
|
||||
for tc in response.tool_calls:
|
||||
tool_name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
|
||||
args = tc["args"]
|
||||
args = dict(tc.get("args") or {})
|
||||
tool = next((t for t in tools if t.name == tool_name), None)
|
||||
|
||||
with TraceSpan(name=tool_name, ctx=self.ctx) as span:
|
||||
span.payload = {"args": args}
|
||||
if tool_name == "get_step_details":
|
||||
logger.info(
|
||||
f"HistoryAnalyzer executing tool get_step_details with args: {args}"
|
||||
)
|
||||
try:
|
||||
start_step = int(args.get("start_step", 0))
|
||||
end_step = int(args.get("end_step", 0))
|
||||
result = get_step_details.invoke(
|
||||
{"start_step": start_step, "end_step": end_step}
|
||||
)
|
||||
span.result = result
|
||||
status = "success"
|
||||
except Exception as e:
|
||||
span.status = "failed"
|
||||
span.error = str(e)
|
||||
result = f"Error running get_step_details: {e}"
|
||||
status = "error"
|
||||
elif tool_name == "list_notes":
|
||||
logger.info("HistoryAnalyzer executing tool list_notes")
|
||||
try:
|
||||
result = list_notes_tool.invoke({})
|
||||
span.result = result
|
||||
status = "success"
|
||||
except Exception as e:
|
||||
span.status = "failed"
|
||||
span.error = str(e)
|
||||
result = f"Error running list_notes: {e}"
|
||||
status = "error"
|
||||
elif tool_name == "read_note":
|
||||
logger.info(f"HistoryAnalyzer executing tool read_note with args: {args}")
|
||||
try:
|
||||
key = args.get("key", "")
|
||||
result = read_note_tool.invoke({"key": key})
|
||||
span.result = result
|
||||
status = "success"
|
||||
except Exception as e:
|
||||
span.status = "failed"
|
||||
span.error = str(e)
|
||||
result = f"Error running read_note: {e}"
|
||||
status = "error"
|
||||
else:
|
||||
if tool is None:
|
||||
result = f"Error: Tool {tool_name} is not supported."
|
||||
status = "error"
|
||||
span.status = "failed"
|
||||
span.error = result
|
||||
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc["id"],
|
||||
content=result,
|
||||
status=status,
|
||||
else:
|
||||
logger.info(f"HistoryAnalyzer executing tool {tool_name} with args: {args}")
|
||||
try:
|
||||
result = await invoke_tool_with_injection(
|
||||
tool=tool, args=args, tool_call_id=tc["id"]
|
||||
)
|
||||
text, _ = split_multimodal_result(result)
|
||||
status = "error" if text.startswith("Error") else "success"
|
||||
span.result = text
|
||||
if status == "error":
|
||||
span.status = "failed"
|
||||
span.error = text
|
||||
except Exception as e:
|
||||
span.status = "failed"
|
||||
span.error = str(e)
|
||||
result = f"Error running {tool_name}: {e}"
|
||||
status = "error"
|
||||
|
||||
# A step screenshot travels in the carrier the model's provider
|
||||
# accepts; text results stay a single ToolMessage.
|
||||
messages.extend(
|
||||
tool_result_messages(tc["id"], result, name=tool_name, status=status, llm=llm)
|
||||
)
|
||||
|
||||
return "Error: HistoryAnalyzer failed to resolve the query within maximum turns."
|
||||
@@ -1,3 +1,3 @@
|
||||
{
|
||||
"main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n[% if \"wait_for_delay\" in available_tools %]- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n[% endif %]- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.\n 2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading state. This boundary probe must remain unidirectional; never reverse direction merely to prove completion. If the task declares an earlier exit condition (for example, find one verified match or process N items), that explicit condition may end the traversal without reaching the list boundary.\n 3. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n 4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content.\n- **Dynamic Monitoring & Reactive Loop Protocol (Monitor-Trigger-Action)**: When executing an active milestone matching `[Loop] <Goal> into note <TargetNote> (Exit: <ExitCondition>; Interval: <Interval>)` or `[Loop:continuous] <Goal> into note <TargetNote> (Interval: <Interval>)` (the milestone grammar itself is defined under Task Plan Grammar below), run this 5-step cyclic state machine:\n - **Step 1 (Baseline Sync)**: On the initial cycle, observe all visible entries/metrics in the primary monitoring view and save the full snaLine truncated
|
||||
"main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n[% if \"wait_for_delay\" in available_tools %]- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n[% endif %]- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.\n 2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading state. This boundary probe must remain unidirectional; never reverse direction merely to prove completion. If the task declares an earlier exit condition (for example, find one verified match or process N items), that explicit condition may end the traversal without reaching the list boundary.\n 3. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n 4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content.\n- **Dynamic Monitoring & Reactive Loop Protocol (Monitor-Trigger-Action)**: When executing an active milestone matching `[Loop] <Goal> into note <TargetNote> (Exit: <ExitCondition>; Interval: <Interval>)` or `[Loop:continuous] <Goal> into note <TargetNote> (Interval: <Interval>)` (the milestone grammar itself is defined under Task Plan Grammar below), run this 5-step cyclic state machine:\n - **Step 1 (Baseline Sync)**: On the initial cycle, observe all visible entries/metrics in the primary monitoring view and save the full snaLine truncated
|
||||
}
|
||||
@@ -37,8 +37,11 @@ from artemis.tools.index import get_tool_by_name
|
||||
from artemis.tools.tool_wrapper import (
|
||||
get_tool_result_content,
|
||||
invoke_tool_with_injection,
|
||||
tool_result_messages,
|
||||
)
|
||||
from artemis.utils.coordinates import (
|
||||
COORDINATE_SPACE_KEY,
|
||||
COORDINATE_SPACE_PIXEL,
|
||||
compute_smart_swipe_coordinates,
|
||||
parse_swipe_parameters,
|
||||
)
|
||||
@@ -58,7 +61,9 @@ DEFERRING_TOOLS = {
|
||||
"video_analyzer",
|
||||
"read_note",
|
||||
"list_notes",
|
||||
"recall_history",
|
||||
"search_history",
|
||||
"replay_steps",
|
||||
"get_step_screenshot",
|
||||
"run_adb_command",
|
||||
"manage_task",
|
||||
"analyze_task_output",
|
||||
@@ -639,11 +644,15 @@ class OperatorNode:
|
||||
results.append((f"Error: Tool {tool_name} not supported", "error"))
|
||||
|
||||
for tc, (content, status) in zip(other_calls, results):
|
||||
tool_outputs.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc["id"],
|
||||
content=content,
|
||||
# A step screenshot (content blocks) travels in the carrier
|
||||
# this model's provider accepts; text stays a ToolMessage.
|
||||
tool_outputs.extend(
|
||||
tool_result_messages(
|
||||
tc["id"],
|
||||
content,
|
||||
name=normalize_name(tc["name"]),
|
||||
status=status,
|
||||
llm=base_llm,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -732,8 +741,8 @@ class OperatorNode:
|
||||
)
|
||||
action_calls_to_translate = []
|
||||
elif len(action_calls) > burst_cap:
|
||||
# A multi-action turn is a fast-action burst that the Validator
|
||||
# fires without the safety net; cap its length before it runs.
|
||||
# A multi-action turn is a fast-action burst: the Validator vets its first
|
||||
# member, fires the rest unvetted; cap its length before it runs.
|
||||
validation_errors = True
|
||||
for tc in action_calls:
|
||||
tool_outputs.append(
|
||||
@@ -1168,6 +1177,7 @@ class OperatorNode:
|
||||
{
|
||||
"action": "tap",
|
||||
"coordinates": el["center"],
|
||||
COORDINATE_SPACE_KEY: COORDINATE_SPACE_PIXEL,
|
||||
"normalized_coordinates": norm_c,
|
||||
"times": times,
|
||||
"delay_ms": delay_ms,
|
||||
@@ -1193,6 +1203,7 @@ class OperatorNode:
|
||||
{
|
||||
"action": "long_press_on",
|
||||
"coordinates": el["center"],
|
||||
COORDINATE_SPACE_KEY: COORDINATE_SPACE_PIXEL,
|
||||
"normalized_coordinates": norm_c,
|
||||
"duration": duration,
|
||||
"target_text": el.get("text"),
|
||||
@@ -1229,6 +1240,7 @@ class OperatorNode:
|
||||
{
|
||||
"action": "focus_and_input_text",
|
||||
"coordinates": el["center"],
|
||||
COORDINATE_SPACE_KEY: COORDINATE_SPACE_PIXEL,
|
||||
"normalized_coordinates": norm_c,
|
||||
"text": text,
|
||||
"clear_before_input": clear_exist,
|
||||
@@ -1310,6 +1322,7 @@ class OperatorNode:
|
||||
{
|
||||
"action": "swipe",
|
||||
"coordinates": [x1, y1, x2, y2],
|
||||
COORDINATE_SPACE_KEY: COORDINATE_SPACE_PIXEL,
|
||||
"normalized_coordinates": [nx1, ny1, nx2, ny2],
|
||||
"normalized_start_coordinates": [nx1, ny1],
|
||||
"normalized_end_coordinates": [nx2, ny2],
|
||||
|
||||
@@ -20,6 +20,8 @@ from pathlib import Path
|
||||
|
||||
from jinja2 import Environment, StrictUndefined, Template
|
||||
from langchain_core.messages import HumanMessage, SystemMessage
|
||||
|
||||
from artemis.tools.history import HISTORY_TOOL_NAMES
|
||||
from PIL import Image
|
||||
|
||||
from artemis.context import ArtemisContext
|
||||
@@ -62,7 +64,13 @@ def load_operator_prompts() -> dict[str, str]:
|
||||
|
||||
_PRE_DECISION_HELPER_TOOLS = ("ask_explorer", "ask_diagnoser", "video_analyzer")
|
||||
_PRE_DECISION_ADB_TOOLS = ("run_adb_command", "manage_task")
|
||||
_PRE_DECISION_MEMORY_TOOLS = ("read_note", "list_notes", "recall_history")
|
||||
_PRE_DECISION_MEMORY_TOOLS = (
|
||||
"read_note",
|
||||
"list_notes",
|
||||
"search_history",
|
||||
"replay_steps",
|
||||
"get_step_screenshot",
|
||||
)
|
||||
_PRE_DECISION_ALL_TOOLS = (
|
||||
_PRE_DECISION_HELPER_TOOLS + _PRE_DECISION_ADB_TOOLS + _PRE_DECISION_MEMORY_TOOLS
|
||||
)
|
||||
@@ -188,10 +196,12 @@ def resolve_operator_prompt_tools(ctx: ArtemisContext) -> frozenset[str]:
|
||||
if not (setup and getattr(setup, "video_recording_tools_enabled", False)):
|
||||
available.discard("video_analyzer")
|
||||
|
||||
# recall_history (M4) needs a DataEngine to search and is config-gated;
|
||||
# the prompt must not advertise it when the tool is not bound this run.
|
||||
if getattr(ctx, "data_engine", None) is None or not _recall_enabled():
|
||||
available.discard("recall_history")
|
||||
# The history tools need a DataEngine to read, and search_history is also
|
||||
# config-gated; the prompt must not advertise a tool that is not bound.
|
||||
if getattr(ctx, "data_engine", None) is None:
|
||||
available.difference_update(HISTORY_TOOL_NAMES)
|
||||
elif not _recall_enabled():
|
||||
available.discard("search_history")
|
||||
|
||||
# Device actions the installed actuator backend does not implement disappear
|
||||
# from the prompt in lockstep with their tool declarations.
|
||||
@@ -438,14 +448,13 @@ def _last_successful_action_before(steps: list, step_number: int | None) -> tupl
|
||||
|
||||
def _incident_target_label(incident: dict) -> str:
|
||||
action = incident.get("action") or {}
|
||||
# The Operator only ever sees 0–1000 coordinates: the recorded pixel
|
||||
# target is the controller's business and is never rendered here.
|
||||
normalized = action.get("normalized_coordinates")
|
||||
pixels = action.get("coordinates")
|
||||
text = action.get("target_text")
|
||||
parts = []
|
||||
if normalized:
|
||||
parts.append(f"normalized {list(normalized)}")
|
||||
elif pixels:
|
||||
parts.append(f"pixel {list(pixels)}")
|
||||
if text:
|
||||
parts.append(f'"{text}"')
|
||||
return " ".join(parts) if parts else "the recorded target"
|
||||
@@ -844,7 +853,7 @@ class ScreenshotSimilarityPromptComponent(PromptComponent):
|
||||
|
||||
|
||||
def _recall_enabled() -> bool:
|
||||
"""Whether the recall_history tool is enabled by configuration (M4)."""
|
||||
"""Whether the search_history tool is enabled by configuration."""
|
||||
try:
|
||||
from artemis.config import load_agent_config
|
||||
|
||||
@@ -861,7 +870,7 @@ class HistoricalStateHintPromptComponent(PromptComponent):
|
||||
including steps already chunk-compressed out of the visible transcript) —
|
||||
an O(n) integer scan, no model call, no historical image bytes. On a
|
||||
close match to a step *older* than the recent window it injects a one-line
|
||||
hint pointing at ``recall_history``.
|
||||
hint pointing at ``search_history``.
|
||||
|
||||
Division of labor with :class:`ScreenshotSimilarityPromptComponent`: that
|
||||
component's pixel-exact 3-step look-back covers "stuck on the same
|
||||
@@ -936,5 +945,5 @@ class HistoricalStateHintPromptComponent(PromptComponent):
|
||||
builder.add_human_content(
|
||||
f"Historical state hint: current screen closely resembles the"
|
||||
f" post-action screen from Step {best_step_number}. Use"
|
||||
" recall_history only if its details are needed."
|
||||
" search_history / replay_steps only if its details are needed."
|
||||
)
|
||||
@@ -16,12 +16,7 @@ import json
|
||||
from pathlib import Path
|
||||
|
||||
from jinja2 import Template
|
||||
from langchain_core.messages import BaseMessage, HumanMessage, SystemMessage, ToolMessage
|
||||
from artemis.agents.outputter.tools import (
|
||||
get_search_history_tool,
|
||||
get_step_details_tool,
|
||||
get_step_screenshot_tool,
|
||||
)
|
||||
from langchain_core.messages import BaseMessage, HumanMessage, SystemMessage
|
||||
from artemis.config import OutputConfig
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.data_engine.trace import trace
|
||||
@@ -36,7 +31,12 @@ from artemis.tools.scratchpad import (
|
||||
get_update_note_tool_pure,
|
||||
)
|
||||
from artemis.tools.video_tool import get_video_analyzer_tool_pure
|
||||
from artemis.tools.tool_wrapper import invoke_tool_with_injection
|
||||
from artemis.tools.history import get_history_tools
|
||||
from artemis.tools.tool_wrapper import (
|
||||
invoke_tool_with_injection,
|
||||
split_multimodal_result,
|
||||
tool_result_messages,
|
||||
)
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.memory.context_policy import build_history_for
|
||||
from artemis.utils.task_tree import get_active_subgoal_hashes
|
||||
@@ -112,10 +112,11 @@ async def outputter(
|
||||
" LAZY VERIFICATION: If the execution history and the final screenshot"
|
||||
" already in your context provide undeniable proof of the outcome, do"
|
||||
" not call tools. Directly output your conclusion.\n- ACTIVE RETRIEVAL:"
|
||||
" You should only call tools (e.g., `get_step_details`,"
|
||||
" `get_step_screenshot`, `search_history_for_text`) if you need to"
|
||||
" extract specific hidden data (like verification codes, tracking"
|
||||
" numbers) or if the final state is ambiguous.\n- VIDEO ANALYSIS COST:"
|
||||
" You should only call tools (e.g., `search_history` to locate the"
|
||||
" step, `replay_steps` for its full record, `get_step_screenshot` for"
|
||||
" its image) if you need to extract specific hidden data (like"
|
||||
" verification codes, tracking numbers) or if the final state is"
|
||||
" ambiguous.\n- VIDEO ANALYSIS COST:"
|
||||
" The `video_analyzer` tool is expensive. Use it only when necessary"
|
||||
" (e.g., to verify video playback or motion).\n- PERSISTENT WRITE: If"
|
||||
" you extract important information requested by the user (such as a"
|
||||
@@ -196,14 +197,8 @@ async def outputter(
|
||||
HumanMessage(content=human_message_content),
|
||||
]
|
||||
|
||||
# Setup tools
|
||||
history_steps = []
|
||||
if ctx.data_engine:
|
||||
history_steps = ctx.data_engine.get_agent_friendly_steps()
|
||||
|
||||
details_tool = get_step_details_tool(history_steps)
|
||||
screenshot_tool = get_step_screenshot_tool(ctx, history_steps)
|
||||
search_tool = get_search_history_tool(ctx)
|
||||
# Setup tools (the shared history tools plus notes and video)
|
||||
search_tool, replay_tool, screenshot_tool = get_history_tools(ctx)
|
||||
list_notes_tool = get_list_notes_tool_pure(ctx)
|
||||
read_note_tool = get_read_note_tool_pure(ctx)
|
||||
save_note_tool = get_save_note_tool_pure(ctx)
|
||||
@@ -212,9 +207,9 @@ async def outputter(
|
||||
video_analyzer_tool = get_video_analyzer_tool_pure(ctx)
|
||||
|
||||
tools = [
|
||||
details_tool,
|
||||
screenshot_tool,
|
||||
search_tool,
|
||||
replay_tool,
|
||||
screenshot_tool,
|
||||
list_notes_tool,
|
||||
read_note_tool,
|
||||
save_note_tool,
|
||||
@@ -254,9 +249,9 @@ async def outputter(
|
||||
|
||||
logger.info(f"Outputter executing tool {tool_name} with args: {args}")
|
||||
tool_map = {
|
||||
"get_step_details": details_tool,
|
||||
"search_history": search_tool,
|
||||
"replay_steps": replay_tool,
|
||||
"get_step_screenshot": screenshot_tool,
|
||||
"search_history_for_text": search_tool,
|
||||
"list_notes": list_notes_tool,
|
||||
"read_note": read_note_tool,
|
||||
"save_note": save_note_tool,
|
||||
@@ -277,22 +272,17 @@ async def outputter(
|
||||
)
|
||||
else:
|
||||
result = f"Error: Tool {tool_name} is not supported."
|
||||
status = (
|
||||
"success"
|
||||
if not (isinstance(result, str) and result.startswith("Error"))
|
||||
else "error"
|
||||
)
|
||||
text, _ = split_multimodal_result(result)
|
||||
status = "error" if text.startswith("Error") else "success"
|
||||
except Exception as e:
|
||||
logger.error(f"Error running tool {tool_name}: {e}")
|
||||
result = f"Error running tool {tool_name}: {e}"
|
||||
status = "error"
|
||||
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc["id"],
|
||||
content=result if isinstance(result, (str, list)) else str(result),
|
||||
status=status,
|
||||
)
|
||||
# Step screenshots enter the conversation in the carrier the
|
||||
# model's provider accepts; text results stay a plain ToolMessage.
|
||||
messages.extend(
|
||||
tool_result_messages(tc["id"], result, name=tool_name, status=status, llm=llm)
|
||||
)
|
||||
|
||||
if raw_answer is None:
|
||||
|
||||
@@ -1,170 +0,0 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import base64
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.tools import BaseTool, tool
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def get_step_details_tool(history_steps: list[dict[str, Any]]) -> BaseTool:
|
||||
@tool
|
||||
def get_step_details(
|
||||
start_step: int,
|
||||
end_step: int,
|
||||
) -> str:
|
||||
"""Retrieve the detailed, full-granularity information for a range of steps (inclusive).
|
||||
|
||||
Use this when you need to inspect specific actions taken, operator
|
||||
thinking, or execution results.
|
||||
"""
|
||||
try:
|
||||
s_step = int(start_step)
|
||||
e_step = int(end_step)
|
||||
except (ValueError, TypeError):
|
||||
return (
|
||||
f"Error: start_step and end_step must be integers, got {start_step} and {end_step}."
|
||||
)
|
||||
|
||||
matched_steps = []
|
||||
for s in history_steps:
|
||||
step_num = s.get("step_number")
|
||||
if step_num is not None and s_step <= step_num <= e_step:
|
||||
details = {
|
||||
"step_number": s.get("step_number"),
|
||||
"relative_time": s.get("relative_time"),
|
||||
"summary": s.get("summary"),
|
||||
"action_taken": s.get("action_taken"),
|
||||
"operator_raw_thinking": s.get("operator_raw_thinking"),
|
||||
"operator_native_thinking": s.get("operator_native_thinking"),
|
||||
"last_execution_result": s.get("last_execution_result"),
|
||||
"interleaved_events": s.get("interleaved_events"),
|
||||
}
|
||||
matched_steps.append(details)
|
||||
if not matched_steps:
|
||||
return f"No steps found in range [{s_step}, {e_step}]."
|
||||
return json.dumps(matched_steps, indent=2, ensure_ascii=False)
|
||||
|
||||
return get_step_details
|
||||
|
||||
|
||||
def get_step_screenshot_tool(ctx: ArtemisContext, history_steps: list[dict[str, Any]]) -> BaseTool:
|
||||
@tool
|
||||
def get_step_screenshot(step_number: int) -> list[dict[str, Any]] | str:
|
||||
"""Retrieve the screenshot for a specific step.
|
||||
|
||||
Returns a multimodal content block containing the image.
|
||||
"""
|
||||
if not ctx.data_engine:
|
||||
return "Error: DataEngine not available."
|
||||
|
||||
try:
|
||||
step_num = int(step_number)
|
||||
except (ValueError, TypeError):
|
||||
return f"Error: step_number must be an integer, got {step_number}."
|
||||
|
||||
# Find the step to get the image name
|
||||
step_record = next((s for s in history_steps if s.get("step_number") == step_num), None)
|
||||
if not step_record:
|
||||
return f"Error: Step {step_num} not found in history."
|
||||
|
||||
image_name = step_record.get("pre_image_name") or step_record.get("post_image_name")
|
||||
if not image_name:
|
||||
return f"Error: No screenshot recorded for step {step_num}."
|
||||
|
||||
images_dir = Path(ctx.data_engine.global_base_dir) / "images"
|
||||
image_path = images_dir / f"{image_name}.jpg"
|
||||
|
||||
if not image_path.exists():
|
||||
return f"Error: Screenshot file for step {step_num} does not exist on disk."
|
||||
|
||||
try:
|
||||
with open(image_path, "rb") as f:
|
||||
img_base64 = base64.b64encode(f.read()).decode("utf-8")
|
||||
except Exception as e:
|
||||
return f"Error reading screenshot file: {e}"
|
||||
|
||||
# Return a LangChain multimodal content structure
|
||||
return [
|
||||
{"type": "text", "text": f"Screenshot for step {step_num}:"},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/jpeg;base64,{img_base64}"},
|
||||
},
|
||||
]
|
||||
|
||||
return get_step_screenshot
|
||||
|
||||
|
||||
def get_search_history_tool(ctx: ArtemisContext) -> BaseTool:
|
||||
@tool
|
||||
def search_history_for_text(query: str) -> str:
|
||||
"""Search for a specific text query within the OCR and UI hierarchy of all historical steps."""
|
||||
if not ctx.data_engine:
|
||||
return "Error: DataEngine not available."
|
||||
|
||||
session_id = ctx.data_engine.current_session_id
|
||||
if not session_id:
|
||||
return "Error: No active session."
|
||||
|
||||
try:
|
||||
steps = ctx.data_engine.storage.get_steps(session_id)
|
||||
except Exception as e:
|
||||
return f"Error retrieving steps from storage: {e}"
|
||||
|
||||
results = []
|
||||
for step in steps:
|
||||
step_matches = []
|
||||
# Search pre-action image
|
||||
if step.pre_image_name:
|
||||
try:
|
||||
matches = ctx.data_engine.storage.search_ui_by_hash(step.pre_image_name, query)
|
||||
if matches:
|
||||
step_matches.extend(matches)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to search UI by hash {step.pre_image_name}: {e}")
|
||||
|
||||
# Search post-action image
|
||||
if step.post_image_name:
|
||||
try:
|
||||
matches = ctx.data_engine.storage.search_ui_by_hash(step.post_image_name, query)
|
||||
if matches:
|
||||
step_matches.extend(matches)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to search UI by hash {step.post_image_name}: {e}")
|
||||
|
||||
if step_matches:
|
||||
# Clean up match details to not bloat context, keep top 3
|
||||
clean_matches = []
|
||||
for m in step_matches[:3]:
|
||||
clean_matches.append(
|
||||
{
|
||||
"type": m.get("type"),
|
||||
"matched_text": m.get("matched_text"),
|
||||
"score": m.get("score"),
|
||||
}
|
||||
)
|
||||
results.append({"step_number": step.step_number, "matches": clean_matches})
|
||||
|
||||
if not results:
|
||||
return f"No matches found for '{query}' in history."
|
||||
return json.dumps(results, indent=2, ensure_ascii=False)
|
||||
|
||||
return search_history_for_text
|
||||
@@ -30,6 +30,6 @@ You will receive:
|
||||
- **CHRONOLOGICAL STEP TRACE (CURRENT STEP)**: The step-by-step chronological history of the current execution turn, including:
|
||||
- `[Operator Monologue] / [Operator Native Thought]`: The operator's planning, intentions, progress counting, reasoning, verified states, and explicit mentions of missing prerequisites.
|
||||
- `[Operator Tool Call] / [Operator Final Action]`: The actual action(s) planned and executed.
|
||||
- `[Planned Fast-Action Burst]`: A multi-action turn executed back to back without the safety net, with each member's outcome.
|
||||
- `[Planned Fast-Action Burst]`: A multi-action turn: the first member is vetted by the safety net, the rest fire back to back unvetted; each member's outcome is listed.
|
||||
- `[Pre-Execution Safety Net]`: Validation checks prior to action execution.
|
||||
- `[Result]`: The Validator's execution outcome; an "Error:" line here is an execution incident the Operator must resolve.
|
||||
@@ -80,7 +80,7 @@ class SummarizerNode:
|
||||
pre_bytes = self._read_step_image(step_number, "pre")
|
||||
post_bytes = self._read_step_image(step_number, "post")
|
||||
|
||||
action_name, action_args = self._extract_action(record, state)
|
||||
action_name, action_args = self._extract_action(record, state, ctx=self.ctx)
|
||||
result = getattr(state, "last_execution_result", None)
|
||||
# format_result_clean reports only errors/repairs; a clean run keeps
|
||||
# the controller's bare status word.
|
||||
@@ -116,7 +116,7 @@ class SummarizerNode:
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _extract_action(record: Any, state: State) -> tuple[str, dict]:
|
||||
def _extract_action(record: Any, state: State, *, ctx: Any = None) -> tuple[str, dict]:
|
||||
actions = getattr(record, "action_taken", None) if record is not None else None
|
||||
if not actions:
|
||||
decisions = getattr(state, "structured_decisions", None)
|
||||
@@ -129,6 +129,15 @@ class SummarizerNode:
|
||||
actions = [actions]
|
||||
if not isinstance(actions, list) or not actions or not isinstance(actions[0], dict):
|
||||
return "no_op", {}
|
||||
# The visual-transition lens is a model: it sees 0–1000 coordinates
|
||||
# only. Pro records pixels (space-marked), so convert once here; a
|
||||
# record already marked normalized passes through untouched.
|
||||
from artemis.utils.coordinates import normalize_any_structure
|
||||
|
||||
device = getattr(ctx, "device", None) if ctx is not None else None
|
||||
width = getattr(device, "device_width", None) or 1080
|
||||
height = getattr(device, "device_height", None) or 2400
|
||||
actions = normalize_any_structure(actions, width, height)
|
||||
first = actions[0]
|
||||
action_name = first.get("action") or "unknown"
|
||||
action_args = {k: v for k, v in first.items() if k != "action"}
|
||||
|
||||
@@ -18,12 +18,14 @@ Two execution tiers, selected by how many turn-ending actions the Operator
|
||||
emitted in the turn:
|
||||
|
||||
* **Vetted single action** (one action): the pre-execution Safety Net runs
|
||||
first (XML-first with pixel fallback), then the action executes locally with
|
||||
a short retry loop.
|
||||
* **Fast-action burst** (two or more actions): the actions fire back to back
|
||||
with no safety net, no retries and no screenshots in between. The burst is
|
||||
the Operator's tool against turn latency (transient menus, toasts, control
|
||||
bars); the Operator owns the risk. The first failure aborts the rest.
|
||||
first (XML-first; the pixel judge decides only when the XML verdict is
|
||||
weak), then the action executes locally with a short retry loop.
|
||||
* **Fast-action burst** (two or more actions): the first member is vetted by
|
||||
the same Safety Net (the screen it was decided on is still the live screen)
|
||||
and a refusal aborts the whole burst; the remaining members fire back to
|
||||
back with no safety net, no retries and no screenshots in between. The burst
|
||||
is the Operator's tool against turn latency (transient menus, toasts,
|
||||
control bars); the Operator owns the risk. The first failure aborts the rest.
|
||||
|
||||
Either way, a failure does not spawn a repair agent. The loop opens an
|
||||
*execution incident* (:mod:`artemis.agents.validator.incidents`) that rides in
|
||||
@@ -49,8 +51,10 @@ from artemis.agents.validator.incidents import (
|
||||
KIND_SAFETY_NET,
|
||||
open_incident,
|
||||
)
|
||||
from artemis.agents.validator.precondition_xml import XML_VERDICT_STRONG
|
||||
from artemis.data_engine.trace import CURRENT_TRACE_ID
|
||||
from artemis.graph.state import State
|
||||
from artemis.utils.coordinates import normalize_action_dict
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.task_tree import format_action_clean
|
||||
|
||||
@@ -72,10 +76,29 @@ class _ActionOutcome:
|
||||
post_screenshot_b64: str | None = None
|
||||
|
||||
|
||||
#: ``safety_net_evidence`` keys the incident carries through verbatim (how the
|
||||
#: verdict was reached), next to the normalized location/occupant facts.
|
||||
_VERDICT_EVIDENCE_KEYS = ("xml_verdict", "xml_element_count", "xml_weak_reason", "pixel_judge")
|
||||
|
||||
|
||||
def _note_pixel_judge(action_item: dict, note: str) -> None:
|
||||
"""Records on the action item how the pixel judge figured in the verdict."""
|
||||
action_item.setdefault("safety_net_evidence", {})["pixel_judge"] = note
|
||||
|
||||
|
||||
async def _run_precondition_gate(
|
||||
node, session, action_item: dict, state: State, pre_screenshot_b64: str, original_coords: list
|
||||
) -> tuple[bool, ValidationErrorCategory, str]:
|
||||
"""Runs the pre-execution Safety Net: XML-first with pixel fallback."""
|
||||
"""Runs the pre-execution Safety Net: XML-first, pixel judge for weak verdicts.
|
||||
|
||||
The pixel judge (a VLM call) is a fallback for when the hierarchy could not
|
||||
see the target, never a second opinion when it could. A STRONG XML failure
|
||||
(healthy, non-trivial tree; the target's identifiers were searched and not
|
||||
found) is final and blocks the action without consulting the judge. A WEAK
|
||||
one (XML bypassed/timed out, sparse tree, identifier-less target) hands the
|
||||
decision to the judge. The path taken is recorded in ``safety_net_evidence``
|
||||
and appended to the reason as one short clause.
|
||||
"""
|
||||
is_explorer_candidate = action_item.get("target_class") == "ExplorerCandidate"
|
||||
is_ocr_element = "[OCR]" in str(action_item.get("target_class") or "")
|
||||
has_index_metadata = bool(
|
||||
@@ -88,11 +111,17 @@ async def _run_precondition_gate(
|
||||
and not is_ocr_element
|
||||
)
|
||||
|
||||
if not has_index_metadata:
|
||||
# Fall back to Pixel-based validation for pure coordinate/non-index interactions
|
||||
return await node._validate_action_precondition_pixel(
|
||||
async def _pixel_judge(note: str) -> tuple[bool, ValidationErrorCategory, str]:
|
||||
passed, category, error = await node._validate_action_precondition_pixel(
|
||||
session, action_item, pre_screenshot_b64, original_coords, state=state
|
||||
)
|
||||
if not passed:
|
||||
_note_pixel_judge(action_item, note)
|
||||
return passed, category, error
|
||||
|
||||
if not has_index_metadata:
|
||||
# Pure coordinate/non-index interactions: the pixel judge is the only gate.
|
||||
return await _pixel_judge("consulted alone (target had no hierarchy metadata)")
|
||||
|
||||
# Index-based validation is fastest and safest when metadata is present
|
||||
(
|
||||
@@ -110,27 +139,33 @@ async def _run_precondition_gate(
|
||||
f" ({validation_error}). Falling back to"
|
||||
" Pixel-based validation..."
|
||||
)
|
||||
return await node._validate_action_precondition_pixel(
|
||||
session, action_item, pre_screenshot_b64, original_coords, state=state
|
||||
)
|
||||
return await _pixel_judge("consulted alone (XML bypassed)")
|
||||
|
||||
logger.info(
|
||||
"Pre-execution XML-based validation failed:"
|
||||
f" {validation_error}. Attempting Pixel-based"
|
||||
" validation fallback..."
|
||||
evidence = action_item.get("safety_net_evidence") or {}
|
||||
if evidence.get("xml_verdict") == XML_VERDICT_STRONG:
|
||||
clause = (
|
||||
f"XML verdict final: {evidence.get('xml_element_count')} nodes searched for the"
|
||||
" target's identifiers; pixel judge not consulted"
|
||||
)
|
||||
(
|
||||
pixel_passed,
|
||||
pixel_category,
|
||||
pixel_error,
|
||||
) = await node._validate_action_precondition_pixel(
|
||||
session, action_item, pre_screenshot_b64, original_coords, state=state
|
||||
_note_pixel_judge(action_item, "not consulted (strong XML verdict)")
|
||||
logger.info(f"Pre-execution XML-based validation failed: {validation_error}. {clause}.")
|
||||
return False, validation_category, f"{validation_error} [{clause}]"
|
||||
|
||||
weak_reason = evidence.get("xml_weak_reason") or "no verdict strength recorded"
|
||||
logger.info(
|
||||
f"Pre-execution XML-based validation failed: {validation_error}. XML evidence is"
|
||||
f" weak ({weak_reason}); asking the pixel judge..."
|
||||
)
|
||||
pixel_passed, pixel_category, _pixel_error = await _pixel_judge(
|
||||
"consulted (weak XML verdict); did not confirm the target"
|
||||
)
|
||||
if pixel_passed and pixel_category == ValidationErrorCategory.NONE:
|
||||
logger.success("Pixel-based validation fallback PASSED! Overriding XML validation failure.")
|
||||
logger.success("Pixel judge confirmed the target is present; overriding weak XML verdict.")
|
||||
action_item.pop("safety_net_evidence", None)
|
||||
return True, ValidationErrorCategory.NONE, ""
|
||||
|
||||
return validation_passed, validation_category, validation_error
|
||||
clause = f"XML evidence weak ({weak_reason}); pixel judge did not confirm the target"
|
||||
return validation_passed, validation_category, f"{validation_error} [{clause}]"
|
||||
|
||||
|
||||
async def _attempt_local_execution(
|
||||
@@ -180,14 +215,19 @@ async def _process_action(
|
||||
pre_screenshot_b64: str,
|
||||
*,
|
||||
burst: bool,
|
||||
vetted: bool = True,
|
||||
) -> _ActionOutcome:
|
||||
"""Runs precondition checks (vetted tier only) and execution for one action item."""
|
||||
"""Runs precondition checks (``vetted`` items only) and execution for one action item.
|
||||
|
||||
``vetted`` is True for a single action and for the first member of a burst;
|
||||
later burst members fire unvetted. ``burst`` members never retry.
|
||||
"""
|
||||
action_item = dict(action_item) # Make a copy to avoid mutating original state actions
|
||||
outcome = _ActionOutcome(action_item=action_item)
|
||||
attempts_log = []
|
||||
|
||||
validation_passed = True
|
||||
if not burst:
|
||||
if vetted:
|
||||
# 1. Pre-execution validation (Safety Net)
|
||||
original_coords = list(action_item.get("coordinates") or [])
|
||||
(
|
||||
@@ -324,6 +364,9 @@ def _incident_evidence(ctx, outcome: _ActionOutcome) -> dict:
|
||||
evidence["occupant"] = raw["occupant"]
|
||||
if raw.get("occupant_bounds") is not None:
|
||||
evidence["occupant_bounds"] = raw["occupant_bounds"]
|
||||
for key in _VERDICT_EVIDENCE_KEYS:
|
||||
if raw.get(key) is not None:
|
||||
evidence[key] = raw[key]
|
||||
return evidence
|
||||
|
||||
|
||||
@@ -337,6 +380,17 @@ def _current_step_number(ctx, state: State) -> int | None:
|
||||
return None
|
||||
|
||||
|
||||
def _model_facing_action(ctx, action_item: dict) -> dict:
|
||||
"""The action item in the Operator's own 0–1000 coordinate space.
|
||||
|
||||
The live item carries the controller's pixels; every phrase rendered for
|
||||
a model is normalized first, and the space marker makes that idempotent.
|
||||
"""
|
||||
width = getattr(getattr(ctx, "device", None), "device_width", None) or 1080
|
||||
height = getattr(getattr(ctx, "device", None), "device_height", None) or 2400
|
||||
return normalize_action_dict(action_item, width, height)
|
||||
|
||||
|
||||
def _build_incident(ctx, state: State, outcome: _ActionOutcome, *, index: int, total: int) -> dict:
|
||||
step_number = _current_step_number(ctx, state)
|
||||
return open_incident(
|
||||
@@ -345,7 +399,7 @@ def _build_incident(ctx, state: State, outcome: _ActionOutcome, *, index: int, t
|
||||
category=outcome.error_category,
|
||||
reason=outcome.error_msg,
|
||||
action_item=outcome.action_item,
|
||||
action_description=format_action_clean(outcome.action_item),
|
||||
action_description=format_action_clean(_model_facing_action(ctx, outcome.action_item)),
|
||||
action_index=index,
|
||||
burst_size=total,
|
||||
step_number=step_number,
|
||||
@@ -371,8 +425,8 @@ async def run_validation_loop(node, state: State) -> dict:
|
||||
burst = total_actions > 1
|
||||
if burst:
|
||||
logger.info(
|
||||
f"Fast-action burst: {total_actions} actions will fire back to back"
|
||||
" without the safety net."
|
||||
f"Fast-action burst: {total_actions} actions; the first is vetted by the"
|
||||
" safety net, the rest fire back to back unvetted."
|
||||
)
|
||||
|
||||
session = await node._get_mcp_session()
|
||||
@@ -407,7 +461,16 @@ async def run_validation_loop(node, state: State) -> dict:
|
||||
|
||||
try:
|
||||
outcome = await _process_action(
|
||||
node, state, session, action_item, action_name, last_screenshot_b64, burst=burst
|
||||
node,
|
||||
state,
|
||||
session,
|
||||
action_item,
|
||||
action_name,
|
||||
last_screenshot_b64,
|
||||
burst=burst,
|
||||
# The first burst member was decided on the live screen, so the
|
||||
# safety net still applies to it; later members fire unvetted.
|
||||
vetted=(not burst or index == 0),
|
||||
)
|
||||
success = outcome.success
|
||||
execution.append(outcome.action_item)
|
||||
|
||||
@@ -4,7 +4,7 @@ UI State Verification Expert. The Operator decided to act at the red dot while l
|
||||
Read the `[Target]` block first to determine which rule applies.
|
||||
|
||||
Rules
|
||||
1. `Kind: specific UI control` — the target has a label, resource id, class or bounds. Judge whether that exact control remains visible at the red dot in Image 2. Ignore video, map, animation or background changes beneath it. A control that has hidden, faded, collapsed, moved or become covered is not present, even if tapping the same spot could restore it.
|
||||
1. `Kind: specific UI control` — the target has a label, resource id, class or bounds. Judge whether that exact control remains visible at the red dot in Image 2. Ignore video, map, animation or background changes beneath it. A control that has hidden, faded, collapsed, moved or become covered is not present, even if tapping the same spot could restore it. Judge the red dot, not the screen: if a different control, or nothing, now sits under the dot, the target is not present even when the same control (or its label) survives elsewhere on the screen or has turned into a different state such as an active input field.
|
||||
2. `Kind: coordinates only` — no UI hierarchy element was found at the point. Identify what appears under the red dot in both images. If Image 1 shows a distinct button, icon, link or handle, apply Rule 1. Apply the surface rule only when Image 1 shows blank space, a backdrop, or the body of a video, animation, map or live stream.
|
||||
3. Surface rule. Typical intents: tap the video body to wake its hidden controls, tap the backdrop to dismiss a dialog. Surfaces are expected to change between frames: ignore natural frame changes and the appearance or fading of overlay controls. Output `{"is_present": true, "confidence": 1.0}` unless an unexpected blocking popup now covers the red dot.
|
||||
4. Use `[Planned Action & Original Thinking]` to resolve whether the target was a control or its underlying surface. Do not treat a missing control as present because another tap could restore it.
|
||||
|
||||
@@ -51,6 +51,15 @@ W_TEXT = 0.4
|
||||
W_BOUNDS = 0.3
|
||||
W_COORD = 0.3
|
||||
|
||||
#: Strength of a failed XML verdict, recorded in ``safety_net_evidence``.
|
||||
#: STRONG verdicts are final; WEAK ones hand the decision to the pixel judge.
|
||||
XML_VERDICT_STRONG = "strong"
|
||||
XML_VERDICT_WEAK = "weak"
|
||||
|
||||
#: A live hierarchy with fewer nodes than this is treated as sparse (a WebView,
|
||||
#: game canvas or half-drawn screen): its "not found" is weak evidence.
|
||||
XML_STRONG_MIN_ELEMENTS = 8
|
||||
|
||||
|
||||
async def validate_action_precondition(
|
||||
node, session, action_item: dict, state: State | None = None
|
||||
@@ -527,6 +536,45 @@ def _describe_occupation(
|
||||
return is_occupied, occupant_desc, occupant_bounds
|
||||
|
||||
|
||||
def _count_nodes(elements) -> int:
|
||||
"""Counts hierarchy nodes, descending into nested ``children`` lists."""
|
||||
total = 0
|
||||
for elem in elements or []:
|
||||
if isinstance(elem, dict):
|
||||
total += 1 + _count_nodes(elem.get("children") or [])
|
||||
return total
|
||||
|
||||
|
||||
def classify_xml_verdict(element_count: int, target_text, target_resource_id) -> tuple[str, str]:
|
||||
"""Grades a failed XML verdict as strong or weak.
|
||||
|
||||
STRONG: the hierarchy is non-trivial AND the target carried identifiers
|
||||
(resource-id / text / content-desc) that were searched and not matched.
|
||||
Anything else (sparse tree, identifier-less target such as bare bounds from
|
||||
a hit test) is WEAK, and the pixel judge gets the final say.
|
||||
|
||||
Returns ``(strength, weak_reason)``; ``weak_reason`` is empty for STRONG.
|
||||
"""
|
||||
if element_count < XML_STRONG_MIN_ELEMENTS:
|
||||
return XML_VERDICT_WEAK, f"sparse hierarchy ({element_count} nodes)"
|
||||
if not (target_text or target_resource_id):
|
||||
return XML_VERDICT_WEAK, "target had no resource-id/text to search for"
|
||||
return XML_VERDICT_STRONG, ""
|
||||
|
||||
|
||||
def _annotate_verdict_strength(
|
||||
action_item: dict, elements: list, target_text, target_resource_id
|
||||
) -> None:
|
||||
"""Records the verdict strength on the action item's ``safety_net_evidence``."""
|
||||
count = _count_nodes(elements)
|
||||
strength, weak_reason = classify_xml_verdict(count, target_text, target_resource_id)
|
||||
evidence = action_item.setdefault("safety_net_evidence", {})
|
||||
evidence["xml_verdict"] = strength
|
||||
evidence["xml_element_count"] = count
|
||||
if weak_reason:
|
||||
evidence["xml_weak_reason"] = weak_reason
|
||||
|
||||
|
||||
def _classify_failure(
|
||||
elements: list,
|
||||
action_item: dict,
|
||||
@@ -602,6 +650,9 @@ async def validate_action_precondition_single(
|
||||
target_bounds = action_item.get("target_bounds") # [l, t, r, b]
|
||||
target_resource_id = action_item.get("target_resource_id")
|
||||
|
||||
# Evidence belongs to one attempt; a retry must not inherit a stale verdict.
|
||||
action_item.pop("safety_net_evidence", None)
|
||||
|
||||
width, height = _resolve_screen_dims(ctx, state)
|
||||
|
||||
# Calculate diagonal-based scale factor for resolution-independent thresholds
|
||||
@@ -659,7 +710,7 @@ async def validate_action_precondition_single(
|
||||
if best["score"] >= threshold:
|
||||
return _handle_match_success(action_item, best, target_text, scale_factor)
|
||||
|
||||
return _classify_failure(
|
||||
verdict = _classify_failure(
|
||||
elements,
|
||||
action_item,
|
||||
best,
|
||||
@@ -670,3 +721,5 @@ async def validate_action_precondition_single(
|
||||
height=height,
|
||||
scale_factor=scale_factor,
|
||||
)
|
||||
_annotate_verdict_strength(action_item, elements, target_text, target_resource_id)
|
||||
return verdict
|
||||
@@ -23,6 +23,8 @@ from artemis.config.agent import (
|
||||
FlashProfileConfig,
|
||||
MemoryChunkingConfig,
|
||||
MemoryConfig,
|
||||
MemoryRecallConfig,
|
||||
MemoryReplayConfig,
|
||||
MemoryRuntimeConfig,
|
||||
MemoryTranscriptConfig,
|
||||
OutputterConfig,
|
||||
@@ -35,6 +37,7 @@ from artemis.config.agent import (
|
||||
VERIFICATION_LEVEL_PRESETS,
|
||||
VerificationLevel,
|
||||
checker_overrides_for_level,
|
||||
run_tuning_for_profile,
|
||||
verification_level_for_checker,
|
||||
)
|
||||
from artemis.config.constants import (
|
||||
@@ -213,6 +216,8 @@ __all__ = [
|
||||
"FlashProfileConfig",
|
||||
"MemoryChunkingConfig",
|
||||
"MemoryConfig",
|
||||
"MemoryRecallConfig",
|
||||
"MemoryReplayConfig",
|
||||
"MemoryRuntimeConfig",
|
||||
"MemoryTranscriptConfig",
|
||||
"OutputterConfig",
|
||||
@@ -225,6 +230,7 @@ __all__ = [
|
||||
"VERIFICATION_LEVEL_PRESETS",
|
||||
"VerificationLevel",
|
||||
"checker_overrides_for_level",
|
||||
"run_tuning_for_profile",
|
||||
"verification_level_for_checker",
|
||||
# Output Config
|
||||
"OutputConfig",
|
||||
|
||||
+60
-13
@@ -266,6 +266,26 @@ def verification_level_for_checker(checker: "CheckerConfig") -> VerificationLeve
|
||||
return "final"
|
||||
|
||||
|
||||
def run_tuning_for_profile(
|
||||
profile: str | None,
|
||||
*,
|
||||
checker: "CheckerConfig",
|
||||
explorer: "ExplorerConfig",
|
||||
explorer_versions: dict[str, str] | None = None,
|
||||
) -> dict[str, str] | None:
|
||||
"""Return Pro verification and explorer settings for session metadata.
|
||||
|
||||
Flash has no per-run tuning and returns None.
|
||||
"""
|
||||
key = str(profile or "").strip().lower()
|
||||
if key != "pro":
|
||||
return None
|
||||
return {
|
||||
"verification_level": verification_level_for_checker(checker),
|
||||
"explorer_mode": explorer.resolve(profile="pro", per_agent_overrides=explorer_versions),
|
||||
}
|
||||
|
||||
|
||||
class OutputterConfig(BaseModel):
|
||||
"""Configuration specific to Outputter post-execution report synthesis agent."""
|
||||
|
||||
@@ -502,36 +522,58 @@ class MemoryTranscriptConfig(BaseModel):
|
||||
|
||||
|
||||
class MemoryRecallConfig(BaseModel):
|
||||
"""History recall tool options (agent.memory.recall)."""
|
||||
"""search_history tool options (agent.memory.recall)."""
|
||||
|
||||
enabled: bool = Field(
|
||||
default=True,
|
||||
description=(
|
||||
"Whether the Pro operator gets the recall_history tool for"
|
||||
" deterministic lookups into cold (compressed/evicted) history."
|
||||
"Whether the operator (Pro and Flash) gets the search_history tool"
|
||||
" for deterministic lookups into cold (compressed/evicted) history."
|
||||
" replay_steps and get_step_screenshot are not gated by this flag."
|
||||
),
|
||||
)
|
||||
max_results: int = Field(
|
||||
default=5,
|
||||
ge=1,
|
||||
le=20,
|
||||
description="Upper bound on results per recall_history call.",
|
||||
description="Upper bound on results per search_history call.",
|
||||
)
|
||||
max_text_tokens: int = Field(
|
||||
default=2000,
|
||||
ge=128,
|
||||
description=(
|
||||
"Estimated-token cap (char/4) on one recall_history text response;"
|
||||
"Estimated-token cap (char/4) on one search_history response;"
|
||||
" excerpts are truncated to fit."
|
||||
),
|
||||
)
|
||||
max_image_steps: int = Field(
|
||||
default=1,
|
||||
screen_scan_steps: int = Field(
|
||||
default=150,
|
||||
ge=0,
|
||||
le=1,
|
||||
description=(
|
||||
"Maximum number of steps whose real stored screenshots may be"
|
||||
" returned as data URLs when include_images is requested."
|
||||
"How many most-recent steps have their screenshot OCR/UI-tree text"
|
||||
" scanned by search_history (per-image JSON loads are the expensive"
|
||||
" part of the sweep). Narrow with step_range to reach older steps."
|
||||
),
|
||||
)
|
||||
model_config = {"extra": "allow"}
|
||||
|
||||
|
||||
class MemoryReplayConfig(BaseModel):
|
||||
"""replay_steps tool options (agent.memory.replay)."""
|
||||
|
||||
max_steps: int = Field(
|
||||
default=5,
|
||||
ge=1,
|
||||
le=50,
|
||||
description="Maximum steps replayed per replay_steps call.",
|
||||
)
|
||||
max_tokens: int = Field(
|
||||
default=12000,
|
||||
ge=128,
|
||||
description=(
|
||||
"Estimated-token budget (char/4) of one replay_steps response."
|
||||
" When exceeded, whole trailing steps are dropped; individual"
|
||||
" steps are kept intact."
|
||||
),
|
||||
)
|
||||
model_config = {"extra": "allow"}
|
||||
@@ -574,7 +616,7 @@ class MemoryChunkingConfig(BaseModel):
|
||||
ge=1,
|
||||
description=(
|
||||
"Maximum eras kept with their per-segment ledgers; older eras"
|
||||
" collapse their ledgers to recall_history marker lines. None"
|
||||
" collapse their ledgers to search_history marker lines. None"
|
||||
" (default) follows max_chunks, preserving the pre-M5 equal-value"
|
||||
" behavior."
|
||||
),
|
||||
@@ -599,7 +641,11 @@ class MemoryConfig(BaseModel):
|
||||
)
|
||||
recall: MemoryRecallConfig = Field(
|
||||
default_factory=MemoryRecallConfig,
|
||||
description="History recall tool options.",
|
||||
description="search_history tool options.",
|
||||
)
|
||||
replay: MemoryReplayConfig = Field(
|
||||
default_factory=MemoryReplayConfig,
|
||||
description="replay_steps tool options.",
|
||||
)
|
||||
policies: dict[str, dict[str, Any]] = Field(
|
||||
default_factory=dict,
|
||||
@@ -659,7 +705,8 @@ class ExecutionConfig(BaseModel):
|
||||
le=10,
|
||||
description=(
|
||||
"Maximum turn-ending actions the Operator may chain into one fast-action"
|
||||
" burst (executed back to back without the safety net). A longer turn is"
|
||||
" burst (first member vetted by the safety net, the rest fire back to back"
|
||||
" unvetted). A longer turn is"
|
||||
" rejected before execution and fed back to the Operator."
|
||||
),
|
||||
)
|
||||
|
||||
@@ -499,6 +499,31 @@ class UnifiedMobileController:
|
||||
max(0.0, end_time - session.start_time),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _segment_session_offsets(
|
||||
session: RecordingSession, mp4_paths: list[Path]
|
||||
) -> dict[Path, float]:
|
||||
"""Map each finalized segment MP4 to its first-frame offset from session start.
|
||||
|
||||
``record["start"]`` is relative to the recording's own t=0 (the first
|
||||
frame of the first segment); shifting it by the recording/DataEngine
|
||||
anchor difference yields the session-relative offset the UI needs to
|
||||
align step timestamps with the playlist.
|
||||
"""
|
||||
anchor = session.data_engine_start_time
|
||||
recording_shift = (session.start_time - anchor) if anchor is not None else 0.0
|
||||
offsets: dict[Path, float] = {}
|
||||
for record in session.android_segment_records:
|
||||
output_path = Path(record.get("output_path") or record.get("path") or "")
|
||||
if not output_path.name:
|
||||
continue
|
||||
offsets[output_path] = max(0.0, float(record.get("start", 0.0)) + recording_shift)
|
||||
for path in mp4_paths:
|
||||
# Emergency fallback remux of the live file has no record; it is
|
||||
# the whole recording, so it starts at the recording anchor.
|
||||
offsets.setdefault(path, max(0.0, recording_shift))
|
||||
return offsets
|
||||
|
||||
def _record_recording_failure(self, session: RecordingSession, message: str) -> None:
|
||||
if self.ctx and self.ctx.data_engine:
|
||||
self.ctx.data_engine.record_video_failure(
|
||||
@@ -802,7 +827,9 @@ class UnifiedMobileController:
|
||||
|
||||
final_video_path = mp4_paths[0]
|
||||
output_dir = final_video_path.parent
|
||||
manifest_path = await write_recording_manifest(output_dir, mp4_paths)
|
||||
manifest_path = await write_recording_manifest(
|
||||
output_dir, mp4_paths, self._segment_session_offsets(session, mp4_paths)
|
||||
)
|
||||
|
||||
remove_active_session(device_id)
|
||||
|
||||
|
||||
+345
-186
@@ -19,6 +19,7 @@ import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import socket
|
||||
import threading
|
||||
import time
|
||||
@@ -88,6 +89,305 @@ def _derive_foreground_app(ui_tree: Any | None) -> str | None:
|
||||
return max(counts.items(), key=lambda kv: kv[1])[0]
|
||||
|
||||
|
||||
#: ``smart_serialize`` stores any image payload inside a trace as this
|
||||
#: reference; the hash is the SHA-256 of the decoded image bytes — the same
|
||||
#: scheme ``record_step`` uses for ``image_name`` — so a referenced image can
|
||||
#: be resolved back to the step screenshot it came from.
|
||||
_IMAGE_REF_RE = re.compile(r"<ImageRef: sha256=([0-9a-fA-F]{64})")
|
||||
|
||||
#: Text stand-in for an image whose origin is unknown (e.g. an annotated
|
||||
#: Explorer overlay that was never stored as a step screenshot).
|
||||
GENERIC_IMAGE_LABEL = "[image attached]"
|
||||
|
||||
|
||||
def image_ref_hash(value: Any) -> str | None:
|
||||
"""The SHA-256 named by an ``<ImageRef: …>`` placeholder, if any."""
|
||||
if not isinstance(value, str):
|
||||
return None
|
||||
match = _IMAGE_REF_RE.search(value)
|
||||
return match.group(1).lower() if match else None
|
||||
|
||||
|
||||
def build_image_describer(steps: list[Any]) -> Callable[[str | None], dict[str, Any]]:
|
||||
"""Describer that resolves an image hash to the step screenshot it is.
|
||||
|
||||
Returns a callable producing a text block: ``[screenshot: pre-action of
|
||||
Step 3]`` (with the ``image_name`` attached so search can pull that
|
||||
screenshot's OCR/XML as the image's description) or the generic label
|
||||
when the hash matches no recorded step screenshot.
|
||||
"""
|
||||
index: dict[str, tuple[int, str]] = {}
|
||||
for step in steps:
|
||||
number = getattr(step, "step_number", None)
|
||||
if number is None and isinstance(step, dict):
|
||||
number = step.get("step_number")
|
||||
for which in ("pre", "post"):
|
||||
name = getattr(step, f"{which}_image_name", None)
|
||||
if name is None and isinstance(step, dict):
|
||||
name = step.get(f"{which}_image_name")
|
||||
if name and name not in index:
|
||||
index[str(name).lower()] = (number, which)
|
||||
|
||||
def describe(image_name: str | None) -> dict[str, Any]:
|
||||
if image_name and image_name.lower() in index:
|
||||
number, which = index[image_name.lower()]
|
||||
return {
|
||||
"type": "text",
|
||||
"text": f"[screenshot: {which}-action of Step {number}]",
|
||||
"image_name": image_name.lower(),
|
||||
}
|
||||
return {"type": "text", "text": GENERIC_IMAGE_LABEL}
|
||||
|
||||
return describe
|
||||
|
||||
|
||||
def describe_result_images(result: Any, describer: Callable[[str | None], dict[str, Any]]) -> Any:
|
||||
"""Rewrites image blocks inside a stored tool result into text blocks.
|
||||
|
||||
Images are never dropped silently from the text views: a block becomes
|
||||
the description the describer gives it (which step screenshot it is, or
|
||||
a generic label), so replay and search both see that an image was there
|
||||
and, when known, what it showed.
|
||||
"""
|
||||
if isinstance(result, list):
|
||||
out = []
|
||||
for block in result:
|
||||
if isinstance(block, dict) and block.get("type") in ("image_url", "image"):
|
||||
url = block.get("image_url")
|
||||
if isinstance(url, dict):
|
||||
url = url.get("url")
|
||||
if url is None:
|
||||
url = block.get("source") or block.get("data")
|
||||
out.append(describer(image_ref_hash(url)))
|
||||
else:
|
||||
out.append(describe_result_images(block, describer))
|
||||
return out
|
||||
if isinstance(result, dict):
|
||||
return {k: describe_result_images(v, describer) for k, v in result.items()}
|
||||
return result
|
||||
|
||||
|
||||
def build_interleaved_events(
|
||||
traces: list[Any],
|
||||
image_describer: Callable[[str | None], dict[str, Any]] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Turns one step's raw trace rows into the agent-facing event stream.
|
||||
|
||||
Operator LLM calls become ``thought`` / ``native_thought`` events; top-level
|
||||
tool and sub-agent traces become ``tool_call`` events carrying ``name``,
|
||||
``args`` and ``result``. Nested calls made *inside* a kept tool/agent trace
|
||||
(e.g. the Explorer's own perception calls) are dropped so only the
|
||||
conclusion the Operator actually saw is replayed. Shared by the live
|
||||
DataEngine and the offline MCP trace inspector so both replay the same
|
||||
events.
|
||||
|
||||
Image blocks inside tool results are rewritten into text descriptions via
|
||||
``image_describer`` (see ``build_image_describer``); without one they
|
||||
become the generic label.
|
||||
"""
|
||||
describer = image_describer or build_image_describer([])
|
||||
interleaved_events: list[dict[str, Any]] = []
|
||||
denylist = (
|
||||
"operator",
|
||||
"perception",
|
||||
"planner",
|
||||
"validator",
|
||||
"summarizer",
|
||||
"checker",
|
||||
)
|
||||
|
||||
relevant_traces = []
|
||||
for t in traces:
|
||||
if t.status not in ("success", "failed"):
|
||||
continue
|
||||
|
||||
is_relevant_llm = False
|
||||
if t.type == "llm_call":
|
||||
curr_parent = t.parent_trace_id
|
||||
visited = set()
|
||||
while curr_parent:
|
||||
if curr_parent in visited:
|
||||
break
|
||||
visited.add(curr_parent)
|
||||
parent_trace = next(
|
||||
(x for x in traces if x.trace_id == curr_parent),
|
||||
None,
|
||||
)
|
||||
if parent_trace:
|
||||
if parent_trace.name == "operator":
|
||||
is_relevant_llm = True
|
||||
t.name = parent_trace.name
|
||||
break
|
||||
curr_parent = parent_trace.parent_trace_id
|
||||
else:
|
||||
break
|
||||
|
||||
if is_relevant_llm:
|
||||
relevant_traces.append(t)
|
||||
elif t.type in ("tool", "agent") and t.name not in denylist:
|
||||
relevant_traces.append(t)
|
||||
|
||||
# Sort by timestamp to ensure exact chronological order
|
||||
relevant_traces.sort(key=lambda x: x.timestamp)
|
||||
|
||||
# Separate set of candidate tool/agent IDs to do top-level filtering
|
||||
# (only keep top-level tool calls)
|
||||
candidate_ids = {t.trace_id for t in relevant_traces if t.type in ("tool", "agent")}
|
||||
|
||||
for t in relevant_traces:
|
||||
if t.type == "llm_call":
|
||||
payload = t.payload or {}
|
||||
response_list = payload.get("response") or []
|
||||
thought_text = ""
|
||||
has_structured_blocks = False
|
||||
if response_list:
|
||||
first_gen = response_list[0]
|
||||
if isinstance(first_gen, dict):
|
||||
content_val = first_gen.get("content")
|
||||
if isinstance(content_val, list):
|
||||
has_structured_blocks = True
|
||||
for block in content_val:
|
||||
if isinstance(block, dict):
|
||||
if (
|
||||
block.get("type") == "thinking"
|
||||
and block.get("thinking", "").strip()
|
||||
):
|
||||
interleaved_events.append(
|
||||
{
|
||||
"type": "native_thought",
|
||||
"content": (block["thinking"].strip()),
|
||||
}
|
||||
)
|
||||
elif block.get("type") == "text" and block.get("text", "").strip():
|
||||
interleaved_events.append(
|
||||
{
|
||||
"type": "thought",
|
||||
"content": (block["text"].strip()),
|
||||
}
|
||||
)
|
||||
if not has_structured_blocks:
|
||||
thought_text = first_gen.get("content") or first_gen.get("text") or ""
|
||||
else:
|
||||
thought_text = str(first_gen)
|
||||
|
||||
if not has_structured_blocks:
|
||||
if isinstance(thought_text, list):
|
||||
thought_text = safe_extract_text(thought_text)
|
||||
|
||||
if thought_text and thought_text.strip():
|
||||
interleaved_events.append(
|
||||
{
|
||||
"type": "thought",
|
||||
"content": thought_text.strip(),
|
||||
}
|
||||
)
|
||||
elif t.type in ("tool", "agent"):
|
||||
# Keep only top-level candidate traces (no ancestor in candidates)
|
||||
is_sub_call = False
|
||||
curr_parent = t.parent_trace_id
|
||||
visited = set()
|
||||
while curr_parent:
|
||||
if curr_parent in visited:
|
||||
break
|
||||
visited.add(curr_parent)
|
||||
if curr_parent in candidate_ids:
|
||||
is_sub_call = True
|
||||
break
|
||||
parent_trace = next(
|
||||
(x for x in traces if x.trace_id == curr_parent),
|
||||
None,
|
||||
)
|
||||
if parent_trace:
|
||||
curr_parent = parent_trace.parent_trace_id
|
||||
else:
|
||||
break
|
||||
if is_sub_call:
|
||||
continue
|
||||
|
||||
payload = t.payload or {}
|
||||
tc_args = payload.get("args") or {}
|
||||
|
||||
if isinstance(tc_args, dict):
|
||||
filtered_args = {
|
||||
k: v for k, v in tc_args.items() if k not in ("state", "tool_call_id")
|
||||
}
|
||||
else:
|
||||
filtered_args = tc_args
|
||||
|
||||
tc_result = payload.get("result") or payload.get("error") or "No result"
|
||||
tc_result = describe_result_images(tc_result, describer)
|
||||
|
||||
interleaved_events.append(
|
||||
{
|
||||
"type": "tool_call",
|
||||
"name": t.name,
|
||||
"args": filtered_args,
|
||||
"result": tc_result,
|
||||
}
|
||||
)
|
||||
|
||||
return interleaved_events
|
||||
|
||||
|
||||
def relative_time_label(timestamp: float | None, session_start_time: float | None) -> str:
|
||||
"""Session-relative ``"12.3s"`` label of a step timestamp (``"0.0s"`` when
|
||||
the session clock is unknown, matching ``DataEngine.get_relative_time``)."""
|
||||
if session_start_time is None or timestamp is None:
|
||||
return "0.0s"
|
||||
return f"{timestamp - session_start_time:.1f}s"
|
||||
|
||||
|
||||
def friendly_step(
|
||||
step: StepRecord,
|
||||
traces: list[TraceRecord],
|
||||
image_describer: Callable[[str | None], dict[str, Any]] | None = None,
|
||||
*,
|
||||
session_start_time: float | None = None,
|
||||
default_width: int = 1080,
|
||||
default_height: int = 2400,
|
||||
) -> dict[str, Any]:
|
||||
"""Agent-friendly record of one stored step.
|
||||
|
||||
JSON columns are already parsed (``StepRecord``), the step's traces are
|
||||
expanded into ``interleaved_events`` (see ``build_interleaved_events``),
|
||||
``tool_calls`` lists the tool events, ``relative_time`` is the
|
||||
session-relative label and physical coordinates are normalized using the
|
||||
dimensions stamped on the step (falling back to the defaults). Shared by
|
||||
the live ``DataEngine`` and the offline ``OfflineHistoryReader`` so both
|
||||
replay the same record.
|
||||
"""
|
||||
step_dict = step.model_dump()
|
||||
step_dict["relative_time"] = relative_time_label(step.timestamp, session_start_time)
|
||||
|
||||
try:
|
||||
step_dict["interleaved_events"] = build_interleaved_events(traces, image_describer)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to retrieve traces for step {step.step_id}: {e}")
|
||||
step_dict["interleaved_events"] = []
|
||||
step_dict["tool_calls"] = [
|
||||
e for e in step_dict["interleaved_events"] if e["type"] == "tool_call"
|
||||
]
|
||||
|
||||
# Normalize physical coordinates to normalized ones for agent consumption
|
||||
try:
|
||||
extra = step_dict.get("extra_metadata") or {}
|
||||
width = extra.get("width") or default_width
|
||||
height = extra.get("height") or default_height
|
||||
|
||||
if step_dict.get("action_taken"):
|
||||
step_dict["action_taken"] = normalize_any_structure(
|
||||
step_dict["action_taken"], width, height
|
||||
)
|
||||
if step_dict.get("last_execution_result"):
|
||||
step_dict["last_execution_result"] = normalize_any_structure(
|
||||
step_dict["last_execution_result"], width, height
|
||||
)
|
||||
except Exception as norm_err:
|
||||
logger.error(f"Failed to normalize coordinates in friendly steps history: {norm_err}")
|
||||
|
||||
return step_dict
|
||||
|
||||
|
||||
class DataEngine:
|
||||
"""Core engine for managing runtime data, storage, and streaming."""
|
||||
|
||||
@@ -1191,197 +1491,56 @@ class DataEngine:
|
||||
if not self.current_session_id:
|
||||
return []
|
||||
steps_with_traces = self.storage.get_steps_with_traces(self.current_session_id)
|
||||
result = []
|
||||
for step, traces in steps_with_traces:
|
||||
step_dict = step.model_dump()
|
||||
step_dict["relative_time"] = self.get_relative_time(step.timestamp)
|
||||
describer = build_image_describer([step for step, _ in steps_with_traces])
|
||||
return [self._friendly_step(step, traces, describer) for step, traces in steps_with_traces]
|
||||
|
||||
interleaved_events = []
|
||||
def get_agent_friendly_steps_in_range(
|
||||
self, start_step: int, end_step: int | None = None
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Agent-friendly records of steps ``start_step``..``end_step`` (inclusive,
|
||||
1-based) without loading the traces of the whole session. ``end_step``
|
||||
defaults to ``start_step`` (a single step)."""
|
||||
if not self.current_session_id:
|
||||
return []
|
||||
end_step = start_step if end_step is None else end_step
|
||||
if start_step > end_step:
|
||||
start_step, end_step = end_step, start_step
|
||||
try:
|
||||
denylist = (
|
||||
"operator",
|
||||
"perception",
|
||||
"planner",
|
||||
"validator",
|
||||
"summarizer",
|
||||
"checker",
|
||||
)
|
||||
|
||||
relevant_traces = []
|
||||
for t in traces:
|
||||
if t.status not in ("success", "failed"):
|
||||
continue
|
||||
|
||||
is_relevant_llm = False
|
||||
if t.type == "llm_call":
|
||||
curr_parent = t.parent_trace_id
|
||||
visited = set()
|
||||
while curr_parent:
|
||||
if curr_parent in visited:
|
||||
break
|
||||
visited.add(curr_parent)
|
||||
parent_trace = next(
|
||||
(x for x in traces if x.trace_id == curr_parent),
|
||||
None,
|
||||
)
|
||||
if parent_trace:
|
||||
if parent_trace.name == "operator":
|
||||
is_relevant_llm = True
|
||||
t.name = parent_trace.name
|
||||
break
|
||||
curr_parent = parent_trace.parent_trace_id
|
||||
else:
|
||||
break
|
||||
|
||||
if is_relevant_llm:
|
||||
relevant_traces.append(t)
|
||||
elif t.type in ("tool", "agent") and t.name not in denylist:
|
||||
relevant_traces.append(t)
|
||||
|
||||
# Sort by timestamp to ensure exact chronological order
|
||||
relevant_traces.sort(key=lambda x: x.timestamp)
|
||||
|
||||
# Separate set of candidate tool/agent IDs to do top-level filtering
|
||||
# (only keep top-level tool calls)
|
||||
candidate_ids = {t.trace_id for t in relevant_traces if t.type in ("tool", "agent")}
|
||||
|
||||
for t in relevant_traces:
|
||||
if t.type == "llm_call":
|
||||
payload = t.payload or {}
|
||||
response_list = payload.get("response") or []
|
||||
thought_text = ""
|
||||
has_structured_blocks = False
|
||||
if response_list:
|
||||
first_gen = response_list[0]
|
||||
if isinstance(first_gen, dict):
|
||||
content_val = first_gen.get("content")
|
||||
if isinstance(content_val, list):
|
||||
has_structured_blocks = True
|
||||
for block in content_val:
|
||||
if isinstance(block, dict):
|
||||
if (
|
||||
block.get("type") == "thinking"
|
||||
and block.get("thinking", "").strip()
|
||||
):
|
||||
interleaved_events.append(
|
||||
{
|
||||
"type": "native_thought",
|
||||
"content": (block["thinking"].strip()),
|
||||
}
|
||||
)
|
||||
elif (
|
||||
block.get("type") == "text"
|
||||
and block.get("text", "").strip()
|
||||
):
|
||||
interleaved_events.append(
|
||||
{
|
||||
"type": "thought",
|
||||
"content": (block["text"].strip()),
|
||||
}
|
||||
)
|
||||
if not has_structured_blocks:
|
||||
thought_text = (
|
||||
first_gen.get("content") or first_gen.get("text") or ""
|
||||
)
|
||||
else:
|
||||
thought_text = str(first_gen)
|
||||
|
||||
if not has_structured_blocks:
|
||||
if isinstance(thought_text, list):
|
||||
thought_text = safe_extract_text(thought_text)
|
||||
|
||||
if thought_text and thought_text.strip():
|
||||
interleaved_events.append(
|
||||
{
|
||||
"type": "thought",
|
||||
"content": thought_text.strip(),
|
||||
}
|
||||
)
|
||||
elif t.type in ("tool", "agent"):
|
||||
# Keep only top-level candidate traces (no ancestor in candidates)
|
||||
is_sub_call = False
|
||||
curr_parent = t.parent_trace_id
|
||||
visited = set()
|
||||
while curr_parent:
|
||||
if curr_parent in visited:
|
||||
break
|
||||
visited.add(curr_parent)
|
||||
if curr_parent in candidate_ids:
|
||||
is_sub_call = True
|
||||
break
|
||||
parent_trace = next(
|
||||
(x for x in traces if x.trace_id == curr_parent),
|
||||
None,
|
||||
)
|
||||
if parent_trace:
|
||||
curr_parent = parent_trace.parent_trace_id
|
||||
else:
|
||||
break
|
||||
if is_sub_call:
|
||||
continue
|
||||
|
||||
payload = t.payload or {}
|
||||
tc_args = payload.get("args") or {}
|
||||
|
||||
if isinstance(tc_args, dict):
|
||||
filtered_args = {
|
||||
k: v
|
||||
for k, v in tc_args.items()
|
||||
if k not in ("state", "tool_call_id")
|
||||
}
|
||||
else:
|
||||
filtered_args = tc_args
|
||||
|
||||
tc_result = payload.get("result") or payload.get("error") or "No result"
|
||||
|
||||
interleaved_events.append(
|
||||
{
|
||||
"type": "tool_call",
|
||||
"name": t.name,
|
||||
"args": filtered_args,
|
||||
"result": tc_result,
|
||||
}
|
||||
)
|
||||
|
||||
step_dict["interleaved_events"] = interleaved_events
|
||||
step_dict["tool_calls"] = [
|
||||
e for e in interleaved_events if e["type"] == "tool_call"
|
||||
]
|
||||
steps = self.storage.get_steps(self.current_session_id) or []
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to retrieve traces for step {step.step_id}: {e}")
|
||||
step_dict["interleaved_events"] = []
|
||||
step_dict["tool_calls"] = []
|
||||
|
||||
# Normalize physical coordinates to normalized ones for agent consumption
|
||||
try:
|
||||
extra = step_dict.get("extra_metadata") or {}
|
||||
width = extra.get("width") or (
|
||||
getattr(self.ctx.device, "device_width", 1080)
|
||||
if self.ctx and self.ctx.device
|
||||
else 1080
|
||||
)
|
||||
height = extra.get("height") or (
|
||||
getattr(self.ctx.device, "device_height", 2400)
|
||||
if self.ctx and self.ctx.device
|
||||
else 2400
|
||||
)
|
||||
|
||||
if step_dict.get("action_taken"):
|
||||
step_dict["action_taken"] = normalize_any_structure(
|
||||
step_dict["action_taken"], width, height
|
||||
)
|
||||
if step_dict.get("last_execution_result"):
|
||||
step_dict["last_execution_result"] = normalize_any_structure(
|
||||
step_dict["last_execution_result"], width, height
|
||||
)
|
||||
except Exception as norm_err:
|
||||
logger.error(
|
||||
f"Failed to normalize coordinates in friendly steps history: {norm_err}"
|
||||
)
|
||||
|
||||
result.append(step_dict)
|
||||
logger.error(f"Failed to fetch steps for range lookup: {e}")
|
||||
return []
|
||||
describer = build_image_describer(steps)
|
||||
result = []
|
||||
for step in steps:
|
||||
if start_step <= step.step_number <= end_step:
|
||||
traces = self.storage.get_traces_for_step(step.step_id)
|
||||
result.append(self._friendly_step(step, traces, describer))
|
||||
return result
|
||||
|
||||
def get_agent_friendly_step(self, step_number: int) -> dict[str, Any] | None:
|
||||
"""Agent-friendly record of one step by its 1-based number, or None."""
|
||||
rows = self.get_agent_friendly_steps_in_range(step_number, step_number)
|
||||
return rows[0] if rows else None
|
||||
|
||||
def _friendly_step(
|
||||
self,
|
||||
step: StepRecord,
|
||||
traces: list[TraceRecord],
|
||||
image_describer: Callable[[str | None], dict[str, Any]] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Live twin of :func:`friendly_step`: the session clock and the device
|
||||
dimensions come from this engine's context."""
|
||||
device = self.ctx.device if self.ctx else None
|
||||
return friendly_step(
|
||||
step,
|
||||
traces,
|
||||
image_describer,
|
||||
session_start_time=self.session_start_time,
|
||||
default_width=getattr(device, "device_width", 1080) if device else 1080,
|
||||
default_height=getattr(device, "device_height", 2400) if device else 2400,
|
||||
)
|
||||
|
||||
def register_background_task(self, task_id: str, summary: str, trace_id: UUID | None = None):
|
||||
if not self.current_session_id:
|
||||
return
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Read-only access to live and stored session history.
|
||||
|
||||
DataEngine implements HistoryReader for active sessions. OfflineHistoryReader
|
||||
opens an existing database and uses the same step renderer for trace inspection.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
from typing import Any, Protocol, runtime_checkable
|
||||
|
||||
from artemis.data_engine.engine import build_image_describer, friendly_step
|
||||
from artemis.data_engine.models import HistoryChunkRecord, StepRecord
|
||||
from artemis.data_engine.storage import StorageManager
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
__all__ = ["HistoryReader", "OfflineHistoryReader", "friendly_step"]
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class HistoryReader(Protocol):
|
||||
"""Everything the history tools need from a session's stored history."""
|
||||
|
||||
session_start_time: float | None
|
||||
base_dir: Path
|
||||
storage: StorageManager
|
||||
|
||||
def get_agent_friendly_steps(self) -> list[dict[str, Any]]: ...
|
||||
|
||||
def get_agent_friendly_steps_in_range(
|
||||
self, start_step: int, end_step: int | None = None
|
||||
) -> list[dict[str, Any]]: ...
|
||||
|
||||
def get_agent_friendly_step(self, step_number: int) -> dict[str, Any] | None: ...
|
||||
|
||||
def get_step_record(self, step_number: int) -> StepRecord | None: ...
|
||||
|
||||
def get_step_image_path(self, step_number: int, which: str = "pre") -> Path | None: ...
|
||||
|
||||
def get_image_path(self, image_name: str) -> Path: ...
|
||||
|
||||
def get_history_chunks(self) -> list[HistoryChunkRecord]: ...
|
||||
|
||||
|
||||
class OfflineHistoryReader:
|
||||
"""Read-only history of one session straight from the traces database.
|
||||
|
||||
``traces_dir`` is the traces root (``data_engine.db`` and ``images/`` live
|
||||
there); ``base_dir`` resolves to the session's own directory, where its
|
||||
notes are stored — the same layout the live ``DataEngine`` writes.
|
||||
"""
|
||||
|
||||
def __init__(self, db_path: str | Path, traces_dir: str | Path, session_id: Any):
|
||||
self.global_base_dir = Path(traces_dir)
|
||||
self.session_id = str(session_id)
|
||||
self.storage = StorageManager(db_path, traces_dir, read_only=True)
|
||||
self.base_dir = self.global_base_dir / self.session_id
|
||||
self.session_start_time: float | None = None
|
||||
try:
|
||||
session = self.storage.get_session(self.session_id)
|
||||
self.session_start_time = session.start_time if session else None
|
||||
except (sqlite3.Error, ValueError) as e:
|
||||
logger.debug(f"Session clock unavailable for {self.session_id}: {e}")
|
||||
|
||||
# --- Images ----------------------------------------------------------------------
|
||||
|
||||
def get_image_path(self, image_name: str) -> Path:
|
||||
return self.global_base_dir / "images" / f"{image_name}.jpg"
|
||||
|
||||
def get_step_image_path(self, step_number: int, which: str = "pre") -> Path | None:
|
||||
step = self.get_step_record(step_number)
|
||||
if step is None:
|
||||
return None
|
||||
image_name = step.pre_image_name if which == "pre" else step.post_image_name
|
||||
if not image_name:
|
||||
return None
|
||||
path = self.get_image_path(image_name)
|
||||
return path if path.exists() else None
|
||||
|
||||
# --- Steps -----------------------------------------------------------------------
|
||||
|
||||
def _steps(self) -> list[StepRecord]:
|
||||
return self.storage.get_steps(self.session_id) or []
|
||||
|
||||
def _friendly(self, step: StepRecord, describer) -> dict[str, Any]:
|
||||
traces = self.storage.get_traces_for_step(step.step_id)
|
||||
return friendly_step(step, traces, describer, session_start_time=self.session_start_time)
|
||||
|
||||
def get_agent_friendly_steps(self) -> list[dict[str, Any]]:
|
||||
steps = self._steps()
|
||||
describer = build_image_describer(steps)
|
||||
return [self._friendly(step, describer) for step in steps]
|
||||
|
||||
def get_agent_friendly_steps_in_range(
|
||||
self, start_step: int, end_step: int | None = None
|
||||
) -> list[dict[str, Any]]:
|
||||
end_step = start_step if end_step is None else end_step
|
||||
if start_step > end_step:
|
||||
start_step, end_step = end_step, start_step
|
||||
steps = self._steps()
|
||||
describer = build_image_describer(steps)
|
||||
return [
|
||||
self._friendly(step, describer)
|
||||
for step in steps
|
||||
if start_step <= step.step_number <= end_step
|
||||
]
|
||||
|
||||
def get_agent_friendly_step(self, step_number: int) -> dict[str, Any] | None:
|
||||
rows = self.get_agent_friendly_steps_in_range(step_number, step_number)
|
||||
return rows[0] if rows else None
|
||||
|
||||
def get_step_record(self, step_number: int) -> StepRecord | None:
|
||||
for step in self._steps():
|
||||
if step.step_number == step_number:
|
||||
return step
|
||||
return None
|
||||
|
||||
# --- Chunks ----------------------------------------------------------------------
|
||||
|
||||
def get_history_chunks(self) -> list[HistoryChunkRecord]:
|
||||
try:
|
||||
return self.storage.get_history_chunks(self.session_id)
|
||||
except (sqlite3.Error, ValueError) as e:
|
||||
logger.debug(f"History chunks unavailable for {self.session_id}: {e}")
|
||||
return []
|
||||
@@ -64,10 +64,18 @@ logger = get_logger(__name__)
|
||||
class StorageManager:
|
||||
"""Manages persistence for Data Engine using SQLite and File System."""
|
||||
|
||||
def __init__(self, db_path: str | Path, base_trace_dir: str | Path):
|
||||
def __init__(self, db_path: str | Path, base_trace_dir: str | Path, *, read_only: bool = False):
|
||||
self.db_path = Path(db_path)
|
||||
self.base_trace_dir = Path(base_trace_dir)
|
||||
self._lock = threading.RLock()
|
||||
self.read_only = read_only
|
||||
|
||||
if read_only:
|
||||
# Offline readers (MCP trace inspection) never create directories,
|
||||
# tables or WAL files: they open the existing database read-only.
|
||||
if not self.db_path.exists():
|
||||
raise FileNotFoundError(f"Data engine database not found: {self.db_path}")
|
||||
return
|
||||
|
||||
# Ensure directories exist
|
||||
self.db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
@@ -78,6 +86,11 @@ class StorageManager:
|
||||
@contextmanager
|
||||
def _get_connection(self) -> Iterator[sqlite3.Connection]:
|
||||
with self._lock:
|
||||
if self.read_only:
|
||||
conn = sqlite3.connect(
|
||||
f"{self.db_path.resolve().as_uri()}?mode=ro", uri=True, timeout=30.0
|
||||
)
|
||||
else:
|
||||
conn = sqlite3.connect(self.db_path, timeout=30.0)
|
||||
conn.row_factory = sqlite3.Row
|
||||
try:
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
"""Plan-driven checkpoint scheduling, the append-only verdict ledger, and exit
|
||||
settlement bookkeeping.
|
||||
|
||||
Three invariants from the redesign govern everything here:
|
||||
Checkpoint processing preserves three invariants:
|
||||
|
||||
1. **Append-only history**: the Operator's context is never rolled back or
|
||||
switched; verdicts influence the future (status flips forward, findings
|
||||
@@ -57,6 +57,15 @@ logger = get_logger(__name__)
|
||||
|
||||
LEDGER_FILENAME = "check_ledger.jsonl"
|
||||
RUN_OUTCOME_FILENAME = "run_outcome.json"
|
||||
#: Per-attempt transcript of the Checker's own streamed reasoning (one record
|
||||
#: per attempt, appended by the Checker when its run ends). Companion of the
|
||||
#: ledger for UI replay: the ledger books verdicts, this file keeps what the
|
||||
#: Checker said while reaching them.
|
||||
STREAMS_FILENAME = "check_streams.jsonl"
|
||||
#: Upper bound on the persisted text of one attempt (all segments together);
|
||||
#: the middle is cut out with a marker so both the opening and the conclusion
|
||||
#: survive.
|
||||
STREAM_TEXT_LIMIT = 20_000
|
||||
|
||||
#: Event type published on the DataEngine bus (and forwarded over IPC/SSE to the
|
||||
#: admin console) for every visible step of the Checker's process.
|
||||
@@ -297,6 +306,103 @@ def has_ledger_records(base_dir: str | Path) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
# --- Attempt stream transcripts (append-only, one record per attempt) ----------------
|
||||
|
||||
|
||||
def streams_path(base_dir: str | Path) -> Path:
|
||||
return Path(base_dir) / STREAMS_FILENAME
|
||||
|
||||
|
||||
def clamp_stream_segments(
|
||||
segments: list[dict], limit: int = STREAM_TEXT_LIMIT
|
||||
) -> tuple[list[dict], int]:
|
||||
"""Bounds the total text of an attempt's stream segments to ``limit`` chars.
|
||||
|
||||
The head and the tail of the attempt (in stream order) are kept and the
|
||||
middle is replaced by one ``…[N chars truncated]…`` marker, attached to the
|
||||
segment where the cut happens. Returns ``(segments, dropped_chars)``;
|
||||
segments left with no text are removed.
|
||||
"""
|
||||
texts = [str(s.get("text") or "") for s in segments]
|
||||
total = sum(len(t) for t in texts)
|
||||
if total <= limit:
|
||||
return [dict(s) for s in segments], 0
|
||||
|
||||
head_end = limit // 2
|
||||
tail_start = total - (limit - head_end)
|
||||
dropped = tail_start - head_end
|
||||
marker = f"\n…[{dropped} chars truncated]…\n"
|
||||
out: list[dict] = []
|
||||
pos = 0
|
||||
marker_done = False
|
||||
for seg, text in zip(segments, texts):
|
||||
start, end = pos, pos + len(text)
|
||||
pos = end
|
||||
piece = ""
|
||||
if start < head_end:
|
||||
piece += text[: max(0, min(len(text), head_end - start))]
|
||||
if end > tail_start:
|
||||
if not marker_done:
|
||||
piece += marker
|
||||
marker_done = True
|
||||
piece += text[max(0, tail_start - start) :]
|
||||
elif start < head_end < end and not marker_done:
|
||||
piece += marker
|
||||
marker_done = True
|
||||
if not piece:
|
||||
continue
|
||||
out.append({**seg, "text": piece})
|
||||
return out, dropped
|
||||
|
||||
|
||||
def append_attempt_stream_record(
|
||||
base_dir: str | Path, record: dict, limit: int = STREAM_TEXT_LIMIT
|
||||
) -> None:
|
||||
"""Appends one attempt's streamed-reasoning transcript.
|
||||
|
||||
``record["segments"]`` is a list of ``{execution_id, role, when, text}``
|
||||
dicts (``role`` is ``thought`` for the model's reasoning stream and
|
||||
``answer`` for its visible text) in the order the turns started. The text
|
||||
is clamped to ``limit`` chars before writing; the record then carries
|
||||
``truncated`` / ``dropped_chars``.
|
||||
"""
|
||||
record = dict(record)
|
||||
record.setdefault("ts", time.time())
|
||||
segments, dropped = clamp_stream_segments(list(record.get("segments") or []), limit)
|
||||
record["segments"] = segments
|
||||
record["truncated"] = dropped > 0
|
||||
record["dropped_chars"] = dropped
|
||||
path = streams_path(base_dir)
|
||||
try:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to append check stream record: {e}")
|
||||
|
||||
|
||||
def read_attempt_streams(base_dir: str | Path) -> list[dict]:
|
||||
path = streams_path(base_dir)
|
||||
if not path.exists():
|
||||
return []
|
||||
records: list[dict] = []
|
||||
try:
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
loaded = json.loads(line)
|
||||
except Exception:
|
||||
logger.warning(f"Skipping malformed check stream line: {line[:120]}")
|
||||
continue
|
||||
if isinstance(loaded, dict):
|
||||
records.append(loaded)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to read check streams: {e}")
|
||||
return records
|
||||
|
||||
|
||||
# --- Plan helpers --------------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -355,7 +461,7 @@ def revert_subgoal_status(ctx, subgoal_key: str) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
# --- Verify-finding persistence (four layers; redesign "verify finding") -------------
|
||||
# --- Verify-finding persistence (four layers) -------------
|
||||
#
|
||||
# Layer 1: the append-only ledger above (single source of truth — unchanged).
|
||||
# Layer 2: `- finding:` plan lines — a single-direction projection rendered by
|
||||
@@ -444,7 +550,7 @@ def _log_checker_note(ctx, run: CheckpointRun, verdicts: list) -> None:
|
||||
including the eventually passing one — as a persistent repair log. It is
|
||||
written through the notes storage API directly (never the model tool
|
||||
path), stays on disk after resolution, and is discoverable via read_note /
|
||||
recall_history under the key returned by :func:`checker_note_key`.
|
||||
search_history under the key returned by :func:`checker_note_key`.
|
||||
"""
|
||||
try:
|
||||
base_dir = ctx.data_engine.base_dir
|
||||
@@ -518,7 +624,7 @@ def queue_checkpoints(
|
||||
logger.info(f"Queued checkpoint for completed subgoal: {text}")
|
||||
|
||||
|
||||
# --- Harvest (§5.2) ------------------------------------------------------------------
|
||||
# --- Harvest ------------------------------------------------------------------
|
||||
|
||||
|
||||
def _record_verdicts(
|
||||
@@ -549,7 +655,7 @@ def _record_verdicts(
|
||||
|
||||
|
||||
def _annotate_history_chunks(ctx, run: CheckpointRun, verdicts) -> None:
|
||||
"""Post-hoc chunk annotation (M3 §9): a verdict harvested after its subgoal's
|
||||
"""Annotate an existing history chunk: a verdict harvested after its subgoal's
|
||||
segment was chunk-compressed is appended to that chunk and bumps its
|
||||
version. Best-effort — history annotation never gates checkpoint booking."""
|
||||
try:
|
||||
@@ -760,7 +866,7 @@ def harvest_finished_checkpoints(ctx, state) -> list[str]:
|
||||
return findings
|
||||
|
||||
|
||||
# --- Spawn (§5.1: after record_step) -------------------------------------------------
|
||||
# --- Spawn after record_step -------------------------------------------------
|
||||
|
||||
|
||||
async def spawn_pending_checkpoints(ctx, state, anchor_step_id: str | None) -> list[str]:
|
||||
@@ -834,7 +940,7 @@ async def spawn_pending_checkpoints(ctx, state, anchor_step_id: str | None) -> l
|
||||
return findings
|
||||
|
||||
|
||||
# --- Exit settlement barrier (§5.3 phase 1) ------------------------------------------
|
||||
# --- Exit settlement barrier ------------------------------------------
|
||||
|
||||
|
||||
async def settle_all_checkpoints(ctx, state) -> None:
|
||||
|
||||
@@ -930,15 +930,15 @@ async def get_graph(ctx: ArtemisContext) -> CompiledStateGraph:
|
||||
else:
|
||||
operator_native_tools.append(t)
|
||||
|
||||
# Get specialized tools for Operator
|
||||
from artemis.tools.history_recall import recall_history_wrapper
|
||||
# Get specialized tools for Operator (history tools: shared definitions)
|
||||
from artemis.tools.history import HISTORY_TOOL_WRAPPERS
|
||||
|
||||
operator_specialized_wrappers = [
|
||||
ask_diagnoser_wrapper,
|
||||
run_adb_command_wrapper,
|
||||
manage_task_wrapper,
|
||||
ask_explorer_wrapper,
|
||||
recall_history_wrapper,
|
||||
*HISTORY_TOOL_WRAPPERS,
|
||||
]
|
||||
# ask_committee does not follow the Operator's pre-decision / turn-ending contract.
|
||||
|
||||
|
||||
@@ -16,6 +16,7 @@
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
@@ -26,6 +27,7 @@ import urllib.request
|
||||
import webbrowser
|
||||
|
||||
from rich.console import Console
|
||||
from rich.markup import escape
|
||||
from rich.panel import Panel
|
||||
import typer
|
||||
|
||||
@@ -87,6 +89,106 @@ def _showcase_build_required(showcase_dir: Path) -> bool:
|
||||
return any(path.exists() and path.stat().st_mtime > build_time for path in input_paths)
|
||||
|
||||
|
||||
def _npm_install_required(showcase_dir: Path) -> bool:
|
||||
"""Return whether ``node_modules`` is missing or older than the package manifests.
|
||||
|
||||
npm rewrites ``node_modules/.package-lock.json`` on every successful
|
||||
install, so its mtime is the "dependencies last synced" stamp.
|
||||
"""
|
||||
stamp = showcase_dir / "node_modules" / ".package-lock.json"
|
||||
if not stamp.exists():
|
||||
return True
|
||||
synced_at = stamp.stat().st_mtime
|
||||
manifests = (showcase_dir / "package.json", showcase_dir / "package-lock.json")
|
||||
return any(path.exists() and path.stat().st_mtime > synced_at for path in manifests)
|
||||
|
||||
|
||||
_ANSI_ESCAPE_RE = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]")
|
||||
_HEARTBEAT_INTERVAL_SECONDS = 15.0
|
||||
|
||||
|
||||
def _format_elapsed(seconds: float) -> str:
|
||||
"""``42s`` / ``3m 51s``."""
|
||||
whole = int(seconds)
|
||||
if whole < 60:
|
||||
return f"{whole}s"
|
||||
return f"{whole // 60}m {whole % 60:02d}s"
|
||||
|
||||
|
||||
def _run_build_step(console: Console, label: str, cmd: list[str], cwd: Path) -> None:
|
||||
"""Run one build step, relaying its output live and raising on failure.
|
||||
|
||||
The child's stdout/stderr are forwarded line by line (ANSI stripped,
|
||||
indented, dimmed) so npm summaries, ``ng build`` progress and any error
|
||||
reach the terminal as they happen. On an interactive terminal a spinner
|
||||
with the elapsed time keeps ticking through the long silent phases
|
||||
(downloads, bundling); without a TTY a plain heartbeat line is printed
|
||||
every 15 seconds instead so logs are not flooded with spinner frames.
|
||||
"""
|
||||
started = time.monotonic()
|
||||
console.print(f" [bold]{label}[/bold]")
|
||||
process = subprocess.Popen(
|
||||
cmd,
|
||||
cwd=cwd,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
)
|
||||
assert process.stdout is not None
|
||||
|
||||
def relay_line(raw: str) -> None:
|
||||
line = _ANSI_ESCAPE_RE.sub("", raw).rstrip()
|
||||
if line.strip():
|
||||
console.print(f" {escape(line)}", style="dim", highlight=False)
|
||||
|
||||
def stream_output(status=None) -> None:
|
||||
last_heartbeat = started
|
||||
for raw in process.stdout:
|
||||
relay_line(raw)
|
||||
now = time.monotonic()
|
||||
if status is not None:
|
||||
status.update(f"{label} … {_format_elapsed(now - started)}")
|
||||
elif now - last_heartbeat >= _HEARTBEAT_INTERVAL_SECONDS:
|
||||
console.print(f" still running… {_format_elapsed(now - started)}")
|
||||
last_heartbeat = now
|
||||
|
||||
if console.is_terminal:
|
||||
with console.status(f"{label} … 0s", spinner="dots") as status:
|
||||
ticker_stop = threading.Event()
|
||||
|
||||
def tick() -> None:
|
||||
while not ticker_stop.wait(1.0):
|
||||
status.update(f"{label} … {_format_elapsed(time.monotonic() - started)}")
|
||||
|
||||
ticker = threading.Thread(target=tick, daemon=True)
|
||||
ticker.start()
|
||||
try:
|
||||
stream_output(status)
|
||||
finally:
|
||||
ticker_stop.set()
|
||||
ticker.join(timeout=2.0)
|
||||
else:
|
||||
heartbeat_stop = threading.Event()
|
||||
|
||||
def heartbeat() -> None:
|
||||
while not heartbeat_stop.wait(_HEARTBEAT_INTERVAL_SECONDS):
|
||||
console.print(f" still running… {_format_elapsed(time.monotonic() - started)}")
|
||||
|
||||
beat = threading.Thread(target=heartbeat, daemon=True)
|
||||
beat.start()
|
||||
try:
|
||||
stream_output()
|
||||
finally:
|
||||
heartbeat_stop.set()
|
||||
beat.join(timeout=2.0)
|
||||
|
||||
returncode = process.wait()
|
||||
if returncode != 0:
|
||||
raise subprocess.CalledProcessError(returncode, cmd)
|
||||
|
||||
|
||||
def ensure_showcase_built(console: Console) -> None:
|
||||
"""Rebuild the Angular Showcase UI when its sources are newer than the dist build."""
|
||||
from artemis.config.paths import ROOT_DIR
|
||||
@@ -105,12 +207,26 @@ def ensure_showcase_built(console: Console) -> None:
|
||||
console.print(
|
||||
" [yellow]🎨 Showcase UI sources changed. Compiling Angular Showcase UI...[/yellow]"
|
||||
)
|
||||
started = time.monotonic()
|
||||
try:
|
||||
subprocess.run([npm_executable, "install", "--silent"], cwd=showcase_dir, check=True)
|
||||
subprocess.run([npm_executable, "run", "build"], cwd=showcase_dir, check=True)
|
||||
console.print(" [green]✓ Showcase UI built successfully.[/green]\n")
|
||||
if _npm_install_required(showcase_dir):
|
||||
_run_build_step(
|
||||
console,
|
||||
"① npm install (dependencies changed)",
|
||||
[npm_executable, "install", "--no-audit", "--no-fund", "--loglevel=warn"],
|
||||
showcase_dir,
|
||||
)
|
||||
_run_build_step(console, "② ng build", [npm_executable, "run", "build"], showcase_dir)
|
||||
console.print(
|
||||
f" [green]✓ Showcase UI built in {_format_elapsed(time.monotonic() - started)}."
|
||||
"[/green]\n"
|
||||
)
|
||||
except Exception as e:
|
||||
console.print(f" [red]⚠ Failed to auto-build Showcase UI: {e}[/red]\n")
|
||||
console.print(
|
||||
f" [red]⚠ Failed to auto-build Showcase UI: {e}[/red]\n"
|
||||
" [dim]Build it by hand with: cd apps/showcase_ui && npm install && "
|
||||
"npm run build[/dim]\n"
|
||||
)
|
||||
|
||||
|
||||
def load_session_reconciler() -> None:
|
||||
|
||||
@@ -23,7 +23,8 @@ Kept agent-side (by design, not omission): argument normalization, element-index
|
||||
resolution against LangGraph ``State``, smart-swipe resolution, act-then-observe
|
||||
capture and ``State`` write-back, tracing (a ``ContextVar`` cannot cross the server's
|
||||
task boundary -- see the plan's R2), and the non-device tools (``read_note``,
|
||||
``list_notes``, ``ask_explorer``, ``video_analyzer``, ``recall_history``).
|
||||
``list_notes``, ``ask_explorer``, ``video_analyzer`` and the shared history tools
|
||||
``search_history`` / ``replay_steps`` / ``get_step_screenshot``).
|
||||
|
||||
Status is carried by ``ActionResult.ok`` -- the historical substring sniffing on
|
||||
``"Error"``/``"Failed"`` is gone, so typing the literal text "Failed" into a field no
|
||||
@@ -49,6 +50,8 @@ from artemis.agents.validator.tool_declarations import (
|
||||
ToolExecutionResult,
|
||||
normalize_coordinate_target,
|
||||
)
|
||||
from artemis.tools.history import HISTORY_TOOL_NAMES, history_tool_by_name
|
||||
from artemis.tools.tool_wrapper import split_multimodal_result
|
||||
from artemis.utils.coordinates import (
|
||||
compute_smart_swipe_coordinates,
|
||||
parse_swipe_parameters,
|
||||
@@ -69,8 +72,8 @@ __all__ = ["McpActionExecutor"]
|
||||
|
||||
|
||||
#: Non-device tools executed agent-side (never behind the action server).
|
||||
AGENT_TOOL_NAMES: frozenset[str] = frozenset(
|
||||
{"read_note", "list_notes", "ask_explorer", "video_analyzer", "recall_history"}
|
||||
AGENT_TOOL_NAMES: frozenset[str] = (
|
||||
frozenset({"read_note", "list_notes", "ask_explorer", "video_analyzer"}) | HISTORY_TOOL_NAMES
|
||||
)
|
||||
|
||||
|
||||
@@ -532,8 +535,8 @@ class McpActionExecutor:
|
||||
args.get("time_description") or args.get("TimeDescription") or "",
|
||||
args.get("purpose") or args.get("Purpose") or "",
|
||||
)
|
||||
elif raw_name == "recall_history":
|
||||
text, blocks = await self._recall_history(args)
|
||||
elif raw_name in HISTORY_TOOL_NAMES:
|
||||
text, blocks = await self._history_tool(raw_name, args)
|
||||
ok = True
|
||||
else:
|
||||
text, ok = await self._ask_explorer(
|
||||
@@ -551,7 +554,7 @@ class McpActionExecutor:
|
||||
# Note texts come from our own failure formatters, so the historical
|
||||
# "Error" marker check is reliable there (unlike free-form device
|
||||
# output); the Explorer and the video analyzer report their status
|
||||
# explicitly and a history recall is a lookup whose "no match" answer
|
||||
# explicitly and a history lookup's "no match" / "not found" answer
|
||||
# is not an error.
|
||||
if ok is None:
|
||||
ok = "Error" not in (text or "")
|
||||
@@ -560,7 +563,7 @@ class McpActionExecutor:
|
||||
tool_name=name,
|
||||
status="success" if ok else "error",
|
||||
text_summary=text,
|
||||
# Content blocks (recalled screenshots) ride along for callers that
|
||||
# Content blocks (a step screenshot) ride along for callers that
|
||||
# forward multimodal tool results; the text summary stays complete.
|
||||
raw_result=blocks,
|
||||
)
|
||||
@@ -583,26 +586,28 @@ class McpActionExecutor:
|
||||
except Exception as e:
|
||||
return format_list_notes_failure(str(e))
|
||||
|
||||
async def _recall_history(self, args: dict[str, Any]) -> tuple[str, list[dict] | None]:
|
||||
"""Deterministic cold-history lookup (same tool the Pro operator binds).
|
||||
async def _history_tool(
|
||||
self, raw_name: str, args: dict[str, Any]
|
||||
) -> tuple[str, list[dict] | None]:
|
||||
"""Runs one of the shared history tools (the same instances the Pro
|
||||
agents bind), dispatched generically by name.
|
||||
|
||||
Returns the text answer and, when screenshots were requested, the
|
||||
multimodal content blocks the tool produced.
|
||||
Returns the text answer and, for a screenshot, the multimodal content
|
||||
blocks the tool produced.
|
||||
"""
|
||||
from artemis.tools.history_recall import RecallHistoryArgs, recall_history
|
||||
|
||||
accepted = {k: v for k, v in args.items() if k in RecallHistoryArgs.model_fields}
|
||||
tool = history_tool_by_name(raw_name)
|
||||
if tool is None:
|
||||
return f"Error: Tool '{raw_name}' not supported.", None
|
||||
accepted = {k: v for k, v in args.items() if k in tool.args_schema.model_fields}
|
||||
try:
|
||||
result = await recall_history.execute(ctx=self.ctx, **accepted)
|
||||
result = await tool.execute(ctx=self.ctx, **accepted)
|
||||
except Exception as e:
|
||||
logger.error(f"Error running recall_history: {e}")
|
||||
return f"recall_history failed: {e}", None
|
||||
if isinstance(result, list):
|
||||
texts = [
|
||||
b.get("text", "") for b in result if isinstance(b, dict) and b.get("type") == "text"
|
||||
]
|
||||
return "\n".join(t for t in texts if t) or "Recalled history.", result
|
||||
return str(result), None
|
||||
logger.error(f"Error running {raw_name}: {e}")
|
||||
return f"{raw_name} failed: {e}", None
|
||||
text, images = split_multimodal_result(result)
|
||||
if images:
|
||||
return text or "Screenshot attached.", result
|
||||
return text, None
|
||||
|
||||
async def _video_analyzer(self, time_description: str, purpose: str) -> tuple[str, bool]:
|
||||
"""Runs the video-analyzing subagent over the session recording.
|
||||
|
||||
@@ -124,7 +124,9 @@ BACKEND_INDEPENDENT_TOOLS: frozenset[str] = frozenset(
|
||||
"run_adb_command",
|
||||
"manage_task",
|
||||
"analyze_task_output",
|
||||
"recall_history",
|
||||
"search_history",
|
||||
"replay_steps",
|
||||
"get_step_screenshot",
|
||||
"report_task_status",
|
||||
}
|
||||
)
|
||||
|
||||
+232
-103
@@ -12,58 +12,20 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""L2/L3 history compression: segment chunks, eras, and the emergency snapshot.
|
||||
"""History compression into segment chunks, eras, and a session snapshot.
|
||||
|
||||
Implements §3.3 of the history-module redesign (M3). A **HistoryChunk** is a
|
||||
*segmented ledger*, never a fused capsule: compression shrinks each step's
|
||||
width, never the step count. Every chunk renders as three bands:
|
||||
Each chunk keeps three bands: synopsis and effects, interval summaries, and a
|
||||
per-step action ledger. Capsule generation reads the operator's turn transcripts
|
||||
alongside the recorded step facts. Milestone changes, segment size, and token
|
||||
limits determine chunk boundaries.
|
||||
|
||||
- **① Synopsis & effects** (LLM, :class:`StepCapsuleLens`): three-question
|
||||
prose (doing / did / effect — the effect must name the notes left behind)
|
||||
plus structured fields (verified_facts / unresolved / failed_paths /
|
||||
important_entities / entry_state / exit_state).
|
||||
- **② Compressed step summary** (same LLM call): interval narrative whose
|
||||
union must cover the segment's step range seamlessly (machine-checked; a
|
||||
gap forces regeneration, bounded retries degrade to pending).
|
||||
- **③ Per-step action ledger** (mechanical, zero-distortion,
|
||||
:func:`build_action_ledger`): one line per step — step number, ``T+mm:ss``
|
||||
session offset, ``format_actions_clean`` semantic action (a fast-action
|
||||
burst lists every member), controller/validator result phrase (an execution
|
||||
incident renders as the ``Error:`` phrase); user-injected instructions
|
||||
verbatim on their own never-evicted lines.
|
||||
Original messages remain in context until their capsule is ready. Failed capsule
|
||||
jobs retain their source text for retry. Only the hard token limit forces pending
|
||||
chunks into the session snapshot. Older chunks fold into eras and then period
|
||||
summaries; full step records remain available through the history tools.
|
||||
|
||||
Triggers (:class:`HistoryChunkManager`): milestone switch (sole fact source:
|
||||
the *stamped* ``subgoal_hash`` changing between consecutive recorded steps;
|
||||
plan-write completions only queue an unconfirmed boundary hint), segment size
|
||||
(steps / estimated source tokens), and the measured-token soft threshold.
|
||||
The hard threshold collapses the frozen region to the L3 session snapshot
|
||||
(chunk headers set-merged; a minimal per-step index survives). Chunk-count
|
||||
overflow folds the oldest chunks into eras (① set-merged, ② degraded to
|
||||
segment title lines, ③ kept); era overflow compresses the oldest eras into
|
||||
the extreme-layer **period paragraph** (§10 decision 5, final user review
|
||||
2026-09-01): one header line carrying the step range and start/end session
|
||||
offsets plus a mechanically assembled one-paragraph synopsis (from the member
|
||||
chunks' band-① fields, no extra LLM call), a ``recall_history`` guidance
|
||||
line, and the verbatim never-evict user-injection lines. Step-level
|
||||
addressability degrades to *period* addressability at this layer only; the
|
||||
full per-step ledger stays recallable from the DataEngine.
|
||||
|
||||
Async discipline — **ready-gated swap** (user decision 2026-09-01, revising
|
||||
the original M3 freeze-then-harvest form): a boundary/size/soft trigger only
|
||||
*closes* a segment and dispatches its capsule; the original messages stay in
|
||||
the transcript verbatim (lossless-pending, carried from the image layer to the
|
||||
segment layer). The swap — freezing the segment's turns and rendering the
|
||||
chunk block — happens at the first render at or after the capsule is ready.
|
||||
Rationale: a header-less chunk has no ``verified_facts``, and a model that
|
||||
sees "what happened" without "what was established" re-doubts verified facts
|
||||
and loops. Degradation ladder: capsule generation failure (bounded retries
|
||||
exhausted) keeps the original text and re-dispatches (the lens itself falls
|
||||
back to a secondary model per attempt); soft-threshold pressure alone still
|
||||
keeps the original text (correctness over tokens); only the hard threshold
|
||||
force-swaps everything closed — pending chunks included — into the L3
|
||||
snapshot. Checkpoint annotations are independent of the header (they bump the
|
||||
DB version immediately) and render in pending blocks too; frozen text still
|
||||
only re-renders at swap events, so it stays byte-stable in between.
|
||||
Checkpoint annotations persist independently of capsule generation. Frozen
|
||||
context is rebuilt only when a capsule or snapshot replaces the source turns.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
@@ -82,13 +44,13 @@ from artemis.utils.logger import get_logger
|
||||
logger = get_logger(__name__)
|
||||
|
||||
#: Recall guidance line rendered under an extreme-layer period paragraph
|
||||
#: (an era whose per-step ledger overflowed to recall-only; §3.3).
|
||||
RECALL_GUIDANCE_TEMPLATE = " (Step-level ledger via recall_history for steps {start}–{end})"
|
||||
#: (an era whose per-step ledger overflowed to recall-only).
|
||||
RECALL_GUIDANCE_TEMPLATE = " (Step-level ledger via search_history for steps {start}–{end})"
|
||||
|
||||
CHUNK_PENDING_NOTE = (
|
||||
"①/② capsule pending (background generation); the mechanical"
|
||||
" ledger below is complete. Original step records remain recallable"
|
||||
" via recall_history."
|
||||
" via search_history / replay_steps."
|
||||
)
|
||||
|
||||
_NOTE_TOOLS = ("save_note", "update_note", "append_note")
|
||||
@@ -216,6 +178,23 @@ def validate_interval_coverage(intervals: Any, start_step: int, end_step: int) -
|
||||
_CAPSULE_REQUIRED_KEYS = ("doing", "did", "effect", "intervals")
|
||||
_CAPSULE_LIST_KEYS = ("verified_facts", "unresolved", "failed_paths", "important_entities")
|
||||
|
||||
#: Coordinate literals belong in the action ledger (band 3), not the
|
||||
#: interval summary (band 2).
|
||||
_COORDINATE_LITERAL_RE = re.compile(
|
||||
r"\[\s*-?\d+(?:\.\d+)?\s*,\s*-?\d+(?:\.\d+)?(?:\s*,\s*-?\d+(?:\.\d+)?){0,2}\s*\]"
|
||||
)
|
||||
|
||||
|
||||
def band2_intervals_carry_coordinates(intervals: Any) -> bool:
|
||||
"""True when any band-② interval text contains a coordinate literal."""
|
||||
if not isinstance(intervals, list):
|
||||
return False
|
||||
return any(
|
||||
isinstance(interval, dict)
|
||||
and _COORDINATE_LITERAL_RE.search(str(interval.get("text") or ""))
|
||||
for interval in intervals
|
||||
)
|
||||
|
||||
|
||||
def collect_note_keys(payload: dict[str, Any]) -> list[str]:
|
||||
"""Deduplicated note keys written during the segment (payload order)."""
|
||||
@@ -236,21 +215,47 @@ def _strip_code_fences(text: str) -> str:
|
||||
return match.group(1) if match else text
|
||||
|
||||
|
||||
class StepCapsuleLens(StepLens):
|
||||
"""Chunk-level lens producing bands ①+② in one call (§5).
|
||||
def _visual_summary_marker() -> str:
|
||||
"""Header the scrub edge writes above a resolved visual summary.
|
||||
|
||||
The payload carries the segment's per-step visual summaries, exact
|
||||
actions, validator results, reasoning excerpts, notes writes, and
|
||||
injected instructions. The rendered summary string is the JSON capsule;
|
||||
an attempt whose interval union fails the mechanical coverage check
|
||||
returns ``None`` so the bounded service retry regenerates it, and on
|
||||
exhaustion the chunk simply stays pending (③ is independently usable).
|
||||
Imported lazily: ``context_compressor`` imports ``artemis.memory``, so a
|
||||
module-level import here would close an import cycle.
|
||||
"""
|
||||
from artemis.agents.flash.context_compressor import HISTORY_SUMMARY_PREFIX
|
||||
|
||||
return HISTORY_SUMMARY_PREFIX.strip()
|
||||
|
||||
|
||||
class StepCapsuleLens(StepLens):
|
||||
"""Chunk-level lens producing bands ①+② in one call.
|
||||
|
||||
The payload carries the segment turn by turn: for each committed turn
|
||||
the mechanical step lines (step number, session offset, action,
|
||||
controller result, visual transition, notes written, injected
|
||||
instruction) followed by the turn's transcript exactly as the operator
|
||||
saw it (:func:`artemis.memory.transcript.render_turn_transcript`). The
|
||||
flat ``steps`` list is kept beside ``turns`` for the machine checks and
|
||||
as the rendering fallback when a segment carries no transcript (e.g. a
|
||||
chunk built without a live ledger). The rendered summary string is the
|
||||
JSON capsule; an attempt whose interval union fails the mechanical
|
||||
coverage check returns ``None`` so the bounded service retry regenerates
|
||||
it, and on exhaustion the chunk simply stays pending (③ is independently
|
||||
usable).
|
||||
"""
|
||||
|
||||
name = "step_capsule"
|
||||
|
||||
_PROMPT_PATH = Path(__file__).parent / "step_capsule.md"
|
||||
|
||||
#: Per-turn transcript cap inside one capsule request — a runaway tool
|
||||
#: result must not sink the whole segment. The ledger and the DataEngine
|
||||
#: keep the full text; the cut is announced in place.
|
||||
MAX_TURN_TRANSCRIPT_CHARS = 16000
|
||||
|
||||
#: Tail of a capped transcript kept intact (the validator result closes
|
||||
#: every turn); bounded to a quarter of the cap.
|
||||
TRANSCRIPT_TAIL_KEEP_CHARS = 2000
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model_name: str | None = None,
|
||||
@@ -316,27 +321,101 @@ class StepCapsuleLens(StepLens):
|
||||
header.append(f"MILESTONE: {label}")
|
||||
|
||||
blocks: list[str] = ["\n".join(header)]
|
||||
turns = payload.get("turns")
|
||||
if turns:
|
||||
for turn in turns:
|
||||
blocks.append(self._render_turn_block(turn))
|
||||
else:
|
||||
for step in payload.get("steps", []):
|
||||
part = [
|
||||
f"## Step {step.get('step_number')} ({step.get('offset')})",
|
||||
f"Action: {step.get('action')}",
|
||||
f"Recorded result: {step.get('outcome')}",
|
||||
]
|
||||
if step.get("visual_summary"):
|
||||
part.append(f"Visual transition: {step['visual_summary']}")
|
||||
if step.get("thinking_excerpt"):
|
||||
part.append(f"Reasoning excerpt: {step['thinking_excerpt']}")
|
||||
for write in step.get("note_writes", []):
|
||||
part.append(f"Note written ({write['tool']} -> {write['key']}): {write['gist']}")
|
||||
if step.get("injected_instruction"):
|
||||
part.append(f'User injected instruction: "{step["injected_instruction"]}"')
|
||||
blocks.append("\n".join(part))
|
||||
blocks.append(self._render_step_block(step))
|
||||
|
||||
return [
|
||||
SystemMessage(content=self._prompt),
|
||||
HumanMessage(content="\n\n".join(blocks)),
|
||||
]
|
||||
|
||||
@staticmethod
|
||||
def _step_record_lines(step: dict[str, Any], *, include_visual: bool = True) -> list[str]:
|
||||
"""Mechanical facts of one step (the same facts band ③ is built from).
|
||||
|
||||
``include_visual=False`` skips the visual transition line when the
|
||||
turn transcript already carries the resolved summary verbatim.
|
||||
"""
|
||||
lines = [
|
||||
f"- Step {step.get('step_number')} ({step.get('offset')}):"
|
||||
f" {step.get('action')} -> {step.get('outcome')}"
|
||||
]
|
||||
if include_visual and step.get("visual_summary"):
|
||||
lines.append(f" Visual transition: {step['visual_summary']}")
|
||||
for write in step.get("note_writes") or []:
|
||||
lines.append(f" Note written ({write['tool']} -> {write['key']}): {write['gist']}")
|
||||
if step.get("injected_instruction"):
|
||||
lines.append(f' User injected instruction: "{step["injected_instruction"]}"')
|
||||
return lines
|
||||
|
||||
@classmethod
|
||||
def _render_turn_block(cls, turn: dict[str, Any]) -> str:
|
||||
"""One committed turn: its recorded step lines, then the transcript the
|
||||
operator saw during that turn (verbatim, capped in place)."""
|
||||
steps = turn.get("steps") or []
|
||||
if not steps:
|
||||
header = "## Turn without a recorded step"
|
||||
elif len(steps) == 1:
|
||||
header = f"## Step {steps[0].get('step_number')} ({steps[0].get('offset')})"
|
||||
else:
|
||||
header = (
|
||||
f"## Steps {steps[0].get('step_number')}–{steps[-1].get('step_number')}"
|
||||
f" ({steps[0].get('offset')} → {steps[-1].get('offset')})"
|
||||
)
|
||||
lines = [header]
|
||||
transcript = str(turn.get("transcript") or "").strip()
|
||||
# The resolved visual summary is already verbatim in the transcript
|
||||
# unless the screenshot was still inside the pending-grace window.
|
||||
include_visual = not transcript or _visual_summary_marker() not in transcript
|
||||
if steps:
|
||||
lines.append("Recorded steps:")
|
||||
for step in steps:
|
||||
lines.extend(cls._step_record_lines(step, include_visual=include_visual))
|
||||
if transcript:
|
||||
lines.append("Transcript as seen by the operator during this turn:")
|
||||
lines.append(cls._cap_transcript(transcript))
|
||||
else:
|
||||
for step in steps:
|
||||
if step.get("thinking_excerpt"):
|
||||
lines.append(
|
||||
f"Reasoning excerpt (Step {step.get('step_number')}):"
|
||||
f" {step['thinking_excerpt']}"
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
@classmethod
|
||||
def _cap_transcript(cls, transcript: str) -> str:
|
||||
"""Cap one turn transcript in place, keeping its head and its tail.
|
||||
|
||||
The tail is kept intact because the validator result message closes
|
||||
every turn and outranks everything above it (fact priority); the cut
|
||||
lands in the middle and is announced with its size.
|
||||
"""
|
||||
cap = cls.MAX_TURN_TRANSCRIPT_CHARS
|
||||
if len(transcript) <= cap:
|
||||
return transcript
|
||||
tail_keep = min(cls.TRANSCRIPT_TAIL_KEEP_CHARS, cap // 4)
|
||||
head, tail = transcript[: cap - tail_keep], transcript[-tail_keep:]
|
||||
cut = len(transcript) - len(head) - len(tail)
|
||||
return (
|
||||
f"{head}\n[… {cut} characters cut here; the full step records remain"
|
||||
f" recallable via replay_steps …]\n{tail}"
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def _render_step_block(cls, step: dict[str, Any]) -> str:
|
||||
"""Fallback rendering for a payload without turn transcripts."""
|
||||
lines = [f"## Step {step.get('step_number')} ({step.get('offset')})"]
|
||||
lines.extend(cls._step_record_lines(step))
|
||||
if step.get("thinking_excerpt"):
|
||||
lines.append(f"Reasoning excerpt: {step['thinking_excerpt']}")
|
||||
return "\n".join(lines)
|
||||
|
||||
def parse_capsule(self, text: str, payload: dict[str, Any]) -> dict[str, Any] | None:
|
||||
"""Parse + machine-check one capsule; None on any violation."""
|
||||
try:
|
||||
@@ -361,9 +440,16 @@ class StepCapsuleLens(StepLens):
|
||||
f" {payload['start_step']}–{payload['end_step']}; regenerating."
|
||||
)
|
||||
return None
|
||||
# Notes linkage is a machine check, same rank as ② coverage (user
|
||||
# decision 2026-09-01): every note key written during the segment must
|
||||
# surface in the band-① text, or the capsule regenerates.
|
||||
# Band ② must not restate band ③: a coordinate literal in an interval
|
||||
# text is the ledger written twice, and it fails the attempt the same
|
||||
# way a coverage gap does.
|
||||
if band2_intervals_carry_coordinates(parsed.get("intervals")):
|
||||
logger.warning(
|
||||
"StepCapsuleLens: band ② interval text carries coordinates for Steps"
|
||||
f" {payload['start_step']}–{payload['end_step']}; regenerating."
|
||||
)
|
||||
return None
|
||||
# Regenerate the capsule if band 1 omits any note key written in the segment.
|
||||
note_keys = collect_note_keys(payload)
|
||||
if note_keys:
|
||||
searchable = " ".join(
|
||||
@@ -427,7 +513,14 @@ class StepCapsuleLens(StepLens):
|
||||
parsed = self.parse_capsule(str(content), payload)
|
||||
if parsed is None:
|
||||
return None
|
||||
return json.dumps(parsed, ensure_ascii=False)
|
||||
capsule = json.dumps(parsed, ensure_ascii=False)
|
||||
# Measure the capsule size against the transcript it replaces.
|
||||
source_chars = len(str(messages[-1].content))
|
||||
logger.info(
|
||||
f"Chunk capsule {key}: source {source_chars} chars -> capsule"
|
||||
f" {len(capsule)} chars ({len(capsule) / max(1, source_chars):.0%})."
|
||||
)
|
||||
return capsule
|
||||
|
||||
|
||||
class ChunkCapsuleService(StepMemoryService):
|
||||
@@ -436,6 +529,15 @@ class ChunkCapsuleService(StepMemoryService):
|
||||
def __init__(self, ctx: Any, lens: StepCapsuleLens, **kwargs):
|
||||
super().__init__(ctx, lens=lens, **kwargs)
|
||||
|
||||
def _on_ready(self, key: JobKey, summary: str) -> None:
|
||||
# The turn transcripts only serve generation. A ready capsule is never
|
||||
# regenerated (only failed ones are re-dispatched), so release the
|
||||
# bulk of the payload instead of holding every segment's source text
|
||||
# for the rest of the session.
|
||||
payload = self._step_inputs.get(key)
|
||||
if payload:
|
||||
payload.pop("turns", None)
|
||||
|
||||
@property
|
||||
def lens(self) -> StepCapsuleLens:
|
||||
return self._lens # type: ignore[return-value]
|
||||
@@ -533,7 +635,7 @@ def _render_annotations(chunk: ChunkState) -> list[str]:
|
||||
|
||||
|
||||
def render_chunk_block(chunk: ChunkState) -> str:
|
||||
"""Full three-band chunk block, strictly in §3.3 order (① → ② → ③)."""
|
||||
"""Full three-band chunk block, in order (① → ② → ③)."""
|
||||
milestone = f' | Milestone "{chunk.milestone_label}"' if chunk.milestone_label else ""
|
||||
header = (
|
||||
f"[Chunk {chunk.ordinal}{milestone} | {chunk.step_range_label}"
|
||||
@@ -582,7 +684,7 @@ def _first_sentence(text: str) -> str:
|
||||
|
||||
|
||||
def render_era_period_paragraph(era: EraState) -> str:
|
||||
"""One-paragraph period synopsis for a recall-only era (§3.3 extreme layer).
|
||||
"""One-paragraph period synopsis for a recall-only era.
|
||||
|
||||
Mechanically assembled from the member chunks' band-① fields — never an
|
||||
extra LLM call: the first non-empty ``doing``, each chunk's first ``did``
|
||||
@@ -638,11 +740,8 @@ def render_era_period_paragraph(era: EraState) -> str:
|
||||
def render_era_block(era: EraState) -> str:
|
||||
"""Era block: ① merged headers, ② degraded to titles, ③ per-chunk ledgers.
|
||||
|
||||
A recall-only era instead renders as the extreme-layer *period paragraph*
|
||||
(§10 decision 5, final review 2026-09-01): step range + start/end session
|
||||
offsets in the header (period addressability — the coarse foreign key for
|
||||
video alignment), a mechanically assembled synopsis, the recall guidance
|
||||
line, and the verbatim never-evict user-injection lines.
|
||||
A recall-only era includes the step range and session offsets for video
|
||||
alignment, a synopsis, recall guidance, and preserved user instructions.
|
||||
"""
|
||||
if era.recall_only:
|
||||
lines = [
|
||||
@@ -706,7 +805,7 @@ def render_l3_snapshot(
|
||||
|
||||
The chunk headers are mechanically set-merged (no summary-of-summary); the
|
||||
per-step skeleton survives at minimal width so step-level time perception
|
||||
never breaks (§3.3 L3).
|
||||
never breaks.
|
||||
"""
|
||||
all_chunks = [c for era in eras for c in era.chunks] + list(chunks)
|
||||
merged = merge_structured_fields([c.band1 for c in all_chunks if c.band1])
|
||||
@@ -771,7 +870,7 @@ class HistoryChunkManager:
|
||||
self._target_source_tokens = int(getattr(cc, "target_source_tokens", 2000) or 2000)
|
||||
self._model_name = getattr(cc, "model", None) or "gemini-3.8-flash"
|
||||
self._max_chunks = int(getattr(cc, "max_chunks", 8) or 8)
|
||||
# Independent era cap (M5): None follows max_chunks (pre-M5 behavior).
|
||||
# None uses max_chunks as the era cap.
|
||||
self._max_eras = int(getattr(cc, "max_eras", None) or self._max_chunks)
|
||||
|
||||
tc = transcript_config
|
||||
@@ -874,7 +973,7 @@ class HistoryChunkManager:
|
||||
"""A plan write completed a top-level milestone: queue an *unconfirmed*
|
||||
boundary. Only the next stamped step confirms it (a hash change); a
|
||||
stamp without a hash change (e.g. the write was vetoed and rolled
|
||||
back) discards the hint — pseudo-switch protection (§3.3)."""
|
||||
back) discards the hint — pseudo-switch protection."""
|
||||
self._boundary_hint_pending = True
|
||||
|
||||
def on_step_stamped(self, step_id: str, subgoal_hash: str | None) -> None:
|
||||
@@ -980,7 +1079,7 @@ class HistoryChunkManager:
|
||||
# Milestone trigger: a *complete* closed segment — one that ended (a
|
||||
# later segment follows, or its hash differs from the current stamp)
|
||||
# and has fully aged past the sliding-window floor — closes whole
|
||||
# (§3.3: the previous segment becomes ONE HistoryChunk; the floor only
|
||||
# (the previous segment becomes one HistoryChunk; the floor only
|
||||
# delays the event, it never splits the segment). The trailing segment
|
||||
# is subject to the size/soft triggers over its eligible portion only.
|
||||
selected: list[tuple[str | None, list[dict]]] = []
|
||||
@@ -1026,7 +1125,7 @@ class HistoryChunkManager:
|
||||
steps_by_id = self._load_steps_by_id()
|
||||
for seg_hash, seg_turns in selected:
|
||||
for slice_turns in self._slices(seg_turns):
|
||||
chunk = self._create_chunk(slice_turns, seg_hash, steps_by_id)
|
||||
chunk = self._create_chunk(slice_turns, seg_hash, steps_by_id, ledger)
|
||||
self._awaiting.append({"chunk": chunk, "turns": list(slice_turns)})
|
||||
logger.info(
|
||||
f"History segments closed (ready-gated): {len(self._awaiting)} awaiting"
|
||||
@@ -1154,15 +1253,29 @@ class HistoryChunkManager:
|
||||
slice_turns: list[dict],
|
||||
seg_hash: str | None,
|
||||
steps_by_id: dict[str, dict],
|
||||
ledger: Any = None,
|
||||
) -> ChunkState | None:
|
||||
# A turn carries every step id it recorded (``step_keys``, Flash
|
||||
# multi-action turns); older ledgers only expose ``step_key``.
|
||||
# multi-action turns); older ledgers only expose ``step_key``. Each
|
||||
# turn's post-scrub transcript (what the operator saw) rides along
|
||||
# into the capsule payload when a live ledger is available.
|
||||
step_keys: list[str] = []
|
||||
turn_records: list[dict[str, Any]] = []
|
||||
for turn in slice_turns:
|
||||
keys = turn.get("step_keys") or ([turn.get("step_key")] if turn.get("step_key") else [])
|
||||
turn_keys: list[str] = []
|
||||
for key in keys:
|
||||
if key and str(key) not in step_keys:
|
||||
step_keys.append(str(key))
|
||||
if key and str(key) in steps_by_id and str(key) not in turn_keys:
|
||||
turn_keys.append(str(key))
|
||||
transcript = None
|
||||
if ledger is not None:
|
||||
try:
|
||||
transcript = ledger.turn_transcript(turn)
|
||||
except Exception as e:
|
||||
logger.warning(f"Turn transcript unavailable for chunk capsule: {e}")
|
||||
turn_records.append({"step_ids": turn_keys, "transcript": transcript})
|
||||
steps = [steps_by_id[k] for k in step_keys if k in steps_by_id]
|
||||
steps.sort(key=lambda s: s.get("step_number") or 0)
|
||||
if not steps:
|
||||
@@ -1194,28 +1307,42 @@ class HistoryChunkManager:
|
||||
# enters self._chunks (and the frozen region) once its capsule is
|
||||
# ready, or through the hard-threshold emergency swap.
|
||||
self._persist(chunk)
|
||||
self._dispatch_capsule(chunk, steps, session_start)
|
||||
self._dispatch_capsule(chunk, steps, session_start, turn_records)
|
||||
return chunk
|
||||
|
||||
def _dispatch_capsule(
|
||||
self, chunk: ChunkState, steps: list[dict], session_start: float | None
|
||||
) -> None:
|
||||
payload_steps = []
|
||||
for step in steps:
|
||||
payload_steps.append(
|
||||
{
|
||||
@staticmethod
|
||||
def _step_payload(step: dict, session_start: float | None) -> dict[str, Any]:
|
||||
"""Mechanical per-step facts handed to the capsule lens."""
|
||||
return {
|
||||
"step_number": step.get("step_number"),
|
||||
"offset": step_offset_label(step, session_start),
|
||||
"action": _action_phrase(step),
|
||||
"outcome": _result_phrase(step.get("last_execution_result")),
|
||||
"visual_summary": step.get("summary"),
|
||||
# Fallback only: rendered when a turn carries no transcript.
|
||||
"thinking_excerpt": (step.get("operator_raw_thinking") or "")[:1200] or None,
|
||||
"note_writes": extract_note_writes(step),
|
||||
"injected_instruction": (step.get("extra_metadata") or {}).get(
|
||||
"injected_instruction"
|
||||
),
|
||||
"injected_instruction": (step.get("extra_metadata") or {}).get("injected_instruction"),
|
||||
}
|
||||
)
|
||||
|
||||
def _dispatch_capsule(
|
||||
self,
|
||||
chunk: ChunkState,
|
||||
steps: list[dict],
|
||||
session_start: float | None,
|
||||
turn_records: list[dict[str, Any]] | None = None,
|
||||
) -> None:
|
||||
"""Submit the capsule job: flat step facts plus, per committed turn,
|
||||
the transcript the operator saw (the text this chunk will replace)."""
|
||||
payload_steps = [self._step_payload(step, session_start) for step in steps]
|
||||
by_id = {str(step.get("step_id")): entry for step, entry in zip(steps, payload_steps)}
|
||||
payload_turns: list[dict[str, Any]] = []
|
||||
for record in turn_records or []:
|
||||
turn_steps = [by_id[k] for k in record.get("step_ids") or [] if k in by_id]
|
||||
transcript = record.get("transcript")
|
||||
if not turn_steps and not transcript:
|
||||
continue
|
||||
payload_turns.append({"steps": turn_steps, "transcript": transcript})
|
||||
payload = {
|
||||
"start_step": chunk.start_step_number,
|
||||
"end_step": chunk.end_step_number,
|
||||
@@ -1223,6 +1350,8 @@ class HistoryChunkManager:
|
||||
"milestone_label": chunk.milestone_label,
|
||||
"steps": payload_steps,
|
||||
}
|
||||
if any(t.get("transcript") for t in payload_turns):
|
||||
payload["turns"] = payload_turns
|
||||
try:
|
||||
self._capsule_service.submit(chunk.capsule_key, payload)
|
||||
except Exception as e:
|
||||
|
||||
@@ -12,22 +12,18 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""ContextPolicy: the declarative per-agent compiled-history policy table (M4).
|
||||
"""Per-agent policies for rendering compiled history.
|
||||
|
||||
Every production caller of :func:`artemis.utils.task_tree.build_plan_and_history`
|
||||
used to hard-code its visibility kwargs at the call site. This module gathers
|
||||
those call sites into one declaration per agent — the rendered output for each
|
||||
agent is byte-identical to the previous hard-coded call (pinned by the golden
|
||||
tests in ``tests/unit/memory/test_context_policy.py``) — and adds the M4 chunk
|
||||
hand-off: when the transcript flag is on and history chunks exist, the
|
||||
compiled view replaces chunked-away step lines with chunk blocks.
|
||||
Each policy sets the visibility options for
|
||||
:func:`artemis.utils.task_tree.build_plan_and_history`. When transcript mode
|
||||
is enabled, compressed steps are rendered as chunk blocks.
|
||||
|
||||
Chunk view per policy (§10 decision 4):
|
||||
Chunk views:
|
||||
|
||||
- ``"full"`` (outputter / history_analyzer): the chunk's stored three-band
|
||||
rendering including the ③ full-width per-step ledger;
|
||||
- ``"digest"`` (everything else): bands ① + ② with the ③ ledger replaced by
|
||||
a ``ledger via recall_history`` marker line.
|
||||
a ``ledger via search_history`` marker line.
|
||||
|
||||
Because chunks only exist when ``agent.memory.transcript.enabled`` is on, the
|
||||
flag-off output of every agent stays byte-for-byte unchanged.
|
||||
@@ -46,7 +42,7 @@ logger = get_logger(__name__)
|
||||
|
||||
#: Marker appended to a digest-view chunk block in place of the ③ ledger.
|
||||
DIGEST_LEDGER_MARKER_TEMPLATE = (
|
||||
"③ Step action ledger: available via recall_history (Steps {start}–{end})"
|
||||
"③ Step action ledger: available via search_history (Steps {start}–{end})"
|
||||
)
|
||||
|
||||
#: The band-③ heading inside a chunk's stored ``rendered_text``.
|
||||
|
||||
@@ -1,40 +1,56 @@
|
||||
# ROLE
|
||||
You are the Segment Capsule writer for a mobile automation agent's history compression. You receive one contiguous segment of executed steps (per-step visual transition summaries, exact actions, controller/validator results, operator reasoning excerpts, notes written, and user-injected instructions) and produce the two LLM-authored bands of the segment's history chunk:
|
||||
You are the Segment Capsule writer for a mobile automation agent's history compression. You receive one contiguous segment of executed steps and produce the two LLM-authored bands of the segment's history chunk:
|
||||
|
||||
- Band ① — Synopsis & effects: what this segment was doing, what was actually done, and what effects/artifacts it left behind, plus structured fields.
|
||||
- Band ② — Compressed step summary: an interval narrative ("Steps xx–xx did ..., Step xx did ...") that seamlessly covers every step of the segment.
|
||||
|
||||
The mechanical per-step action ledger (band ③) is assembled outside of you; do NOT reproduce it.
|
||||
|
||||
# INPUT SHAPE
|
||||
The segment arrives turn by turn, in order. Each turn block has:
|
||||
- `Recorded steps:` — the mechanical facts of the step(s) executed in that turn: step number, session offset, the exact action, the controller/validator result, the visual transition summary, the notes written (tool → note key: gist), and any user-injected instruction.
|
||||
- `Transcript as seen by the operator during this turn:` — the turn exactly as it stood in the operator's own context: the observation text, the operator's reasoning (including `(thinking)` blocks), every `[tool call]` with its arguments, every `[tool result]` returned to the operator (explorer answers, recalled history, note reads, ADB output, screenshot placeholders), and the `--- Action Execution Result ---` message. Your capsule REPLACES this transcript in the operator's context — everything the operator learned from it that still matters must survive in your output.
|
||||
|
||||
A turn block without a transcript falls back to a reasoning excerpt; treat it the same way with less evidence.
|
||||
|
||||
# NEUTRAL-WORDING CONTRACT (STRICT)
|
||||
1. **Zero subjective validation**: never declare semantic success, completion, or final failure. Subjective verdicts in compressed history cause future turns to falsely believe a goal was definitively met or permanently broken.
|
||||
- **BANNED WORDS**: "successfully", "completed", "achieved", "entered", "navigated to", "arrived at", "failed", "unsuccessful", "could not", "impossible".
|
||||
- Describe observable transitions and recorded results instead (e.g. "the settings page was displayed", "the controller reported an execution error").
|
||||
- Tool results in the transcript may themselves use verdict words; restate them as recorded results ("the explorer reported ...", "the tool returned ...").
|
||||
2. **First-person perspective**: write from the agent's perspective using "I". Never "the agent" / "the operator".
|
||||
3. **Planned iterations are not anomalies**: systematic processing of different candidate items, or periodic polling cycles under a `[Loop]` / `[Loop:continuous]` milestone, is planned progress — record it objectively (e.g. "polling round #2"). Only flag a loop when the exact same target is repeated without progress or states oscillate blindly; then record the pattern and count factually.
|
||||
4. **Preserve, don't abstract**: be deliberately detailed and stay close to the source wording. Prefer longer output over abstracting facts beyond recognition. Faithfully carry visible text, codes, prices, account names, file names, error banners.
|
||||
5. **Fact priority**: user instructions > validator/controller recorded results > UI/OCR deltas > sourced visual summaries > reasoning excerpts.
|
||||
4. **Preserve, don't abstract**: be deliberately detailed and stay close to the source wording. Prefer longer output over abstracting facts beyond recognition. Faithfully carry visible text, codes, prices, account names, file names, error banners — and the concrete content of tool results the operator relied on (an explorer's answer, a recalled step, a note's contents, a command's output).
|
||||
5. **Fact priority**: user instructions > validator/controller recorded results > tool results returned to the operator (explorer / recall / notes / ADB) > UI/OCR deltas > visual transition summaries > the operator's reasoning.
|
||||
|
||||
# BAND ① REQUIREMENTS
|
||||
Answer three questions in detailed prose (field `doing` / `did` / `effect`):
|
||||
- `doing`: what this segment was trying to do and why (strategic context within the plan).
|
||||
- `did`: what was actually carried out, including detours, retries, recoveries, and pivots.
|
||||
- `effect`: what state/effects/artifacts the segment left behind. **This MUST include every note left during the segment**: for each notes write in the input, name the target note file and the gist of what was recorded. Every note's target key is machine-checked against your effect field — an omitted note key forces regeneration.
|
||||
Band ① is the segment-level account; band ② below it is the interval-level account. The two bands must not say the same thing twice: ① never re-lists what the intervals of ② already narrate, and it should come out shorter than ② (the `doing` + `did` + `effect` prose together — not counting the structured fields — shorter than the ② intervals together).
|
||||
|
||||
Structured fields (arrays of short strings, may be empty):
|
||||
- `verified_facts`: states/values actually observed or verified on screen.
|
||||
Answer three questions in prose (field `doing` / `did` / `effect`):
|
||||
- `doing`: what this segment was trying to do and why (strategic context within the plan). One or two sentences.
|
||||
- `did`: the route the segment actually took, at segment level: the plan-relevant outcome, plus every detour, retry, recovery, pivot, and tool consultation (what I asked, what it returned) when that shaped the next action. Do NOT walk through the steps one by one — ② does that. A segment that went straight through gets one sentence.
|
||||
- `effect`: what the segment left behind beyond the screen state already in `exit_state`: persistent state changes, artifacts, and notes. **This MUST include every note left during the segment**: for each notes write in the input, name the target note file and the gist of what was recorded. Every note's target key is machine-checked against your effect field — an omitted note key forces regeneration. If nothing was left behind besides the screen state, say so in a few words.
|
||||
|
||||
Structured fields (arrays of short strings, may be empty; each entry one short clause, not a sentence):
|
||||
- `verified_facts`: states/values actually observed on screen or returned by a tool that later steps may need (name the source: "on screen", "explorer", "recall", "adb"). Values, names, versions, counts, codes, prices — not a description of every screen visited.
|
||||
- `unresolved`: open questions, unverified assumptions, missing prerequisites.
|
||||
- `failed_paths`: approaches attempted in this segment whose recorded results were errors (state the recorded error factually).
|
||||
- `important_entities`: accounts, files, order/tracking numbers, package names, key UI entry points.
|
||||
- `entry_state` / `exit_state` (single strings): the observable screen/app state at segment start and end.
|
||||
- `entry_state` / `exit_state` (single strings): the observable screen/app state at segment start and end, a short clause each.
|
||||
|
||||
# BAND ② REQUIREMENTS
|
||||
Field `intervals`: an ordered array of `{"start_step": N, "end_step": M, "text": "..."}` objects.
|
||||
- Every line's text must reference concrete behavior for those steps.
|
||||
- Band ② answers only "what were these steps doing, and what did I see as a result". The action itself (tap / swipe / input, its target and its coordinates) is ALREADY carried step by step by the mechanical ledger (band ③) that sits right below your output — do not write it twice.
|
||||
- **NEVER include coordinates** — no `[x, y]` pairs, no pixel or normalized positions, no bounds. This is machine-checked; any `[number, number]` in an interval text forces regeneration.
|
||||
- **NEVER narrate step by step** ("Step 5: I tapped X. Step 6: I tapped Y."). One interval is a behavior plus the observed outcome, e.g. "Opened the 'Network & internet' section and returned to the main list; the section showed Wi‑Fi, mobile network and VPN rows."
|
||||
- **Merge homogeneous consecutive actions into one interval** (repeated scrolls, a sweep through several sections, a polling loop, retries of the same target) — describe the run and its count/result once. A step that does something of a different nature gets its own single-step interval (`start_step == end_step`).
|
||||
- Every line's text must reference concrete behavior for those steps (the step numbers come from the `Recorded steps:` lines): the target's visible text, the screen or state observed afterwards, values read, errors shown.
|
||||
- **The union of intervals MUST cover the segment's step range seamlessly** — no gaps, no overlaps, in ascending order. This is machine-checked; a gap forces regeneration.
|
||||
- Group only homogeneous consecutive actions into one interval; a heterogeneous action gets its own single-step interval (`start_step == end_step`).
|
||||
- Keep the neutral-wording contract in every line.
|
||||
|
||||
# LENGTH TARGET (SOFT)
|
||||
Bands ① and ② together should stay within roughly one third of the source text you were given. Reach that by not repeating what band ③ already records (actions, targets, coordinates) and by merging homogeneous runs — NEVER by abstracting away concrete facts (visible text, values read, tool answers, error banners, note contents must all survive verbatim).
|
||||
|
||||
# OUTPUT FORMAT
|
||||
Return ONLY one JSON object (no markdown fences, no commentary):
|
||||
|
||||
|
||||
+103
-22
@@ -12,21 +12,20 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Session-level message transcript ledger for the Pro operator (M2).
|
||||
"""Session message ledger shared by the Pro operator and Flash runner.
|
||||
|
||||
Implements the four-region discipline of the history-module redesign §3.2:
|
||||
Messages are organized into four regions:
|
||||
|
||||
- **S (stable prefix)**: the static system message — byte-identical for the
|
||||
whole session (the plan+history section is moved out by the template split).
|
||||
- **F (frozen history)**: in M2 only the cold-start restored-history block; L2
|
||||
chunks arrive in M3.
|
||||
- **F (frozen history)**: restored history and compressed chunks.
|
||||
- **A (active window)**: raw per-turn messages, append-only. Each committed
|
||||
turn contributes its tail observation HumanMessage, the operator AIMessages
|
||||
(tool_calls and native thinking preserved by reference), the in-turn
|
||||
ToolMessages, and the turn's validator result message. The scrub edge
|
||||
(:class:`~artemis.agents.flash.context_compressor.ScrubEdgeCompressor`,
|
||||
generalized in M2) strips old UI lists / plan recitations at depth 1 and
|
||||
resolves screenshots to visual summaries at depth K — messages are never
|
||||
(:class:`~artemis.agents.flash.context_compressor.ScrubEdgeCompressor`) strips
|
||||
old UI lists and plan recitations at depth 1 and resolves screenshots to
|
||||
visual summaries at depth K — messages are never
|
||||
removed or reordered, so tool-call/response pairs are never split.
|
||||
- **T (current tail)**: built fresh every turn by the operator; passed to
|
||||
:meth:`render` and only enters A when the turn is committed.
|
||||
@@ -75,6 +74,89 @@ def format_session_offset(seconds: float) -> str:
|
||||
return f"T+{total // 60:02d}:{total % 60:02d}"
|
||||
|
||||
|
||||
#: Placeholder for an image block that is still inside the scrub edge's
|
||||
#: pending-grace window when a turn transcript is rendered.
|
||||
SCREENSHOT_PLACEHOLDER = "[screenshot]"
|
||||
|
||||
_ROLE_LABELS = {
|
||||
"human": "observation",
|
||||
"ai": "operator",
|
||||
"tool": "tool result",
|
||||
"system": "system",
|
||||
}
|
||||
|
||||
|
||||
def _render_content_text(content: Any) -> str:
|
||||
"""Flatten message content to plain text.
|
||||
|
||||
Text blocks pass through verbatim, native thinking blocks are labelled,
|
||||
image blocks become :data:`SCREENSHOT_PLACEHOLDER`.
|
||||
"""
|
||||
if content is None:
|
||||
return ""
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if not isinstance(content, list):
|
||||
return str(content)
|
||||
parts: list[str] = []
|
||||
for block in content:
|
||||
if isinstance(block, str):
|
||||
parts.append(block)
|
||||
elif not isinstance(block, dict):
|
||||
parts.append(str(block))
|
||||
elif block.get("type") == "text":
|
||||
parts.append(str(block.get("text", "")))
|
||||
elif block.get("type") == "thinking":
|
||||
parts.append(f"(thinking) {block.get('thinking', '')}")
|
||||
elif block.get("type") in ("image_url", "image"):
|
||||
parts.append(SCREENSHOT_PLACEHOLDER)
|
||||
else:
|
||||
parts.append(json.dumps(block, ensure_ascii=False, default=str))
|
||||
return "\n".join(part for part in parts if part)
|
||||
|
||||
|
||||
def render_turn_transcript(messages: list[BaseMessage]) -> str:
|
||||
"""Mechanical plain-text rendering of one committed turn's messages.
|
||||
|
||||
Lossless with respect to what the operator model was shown: every text
|
||||
block, native thinking block, tool call (name and arguments) and tool
|
||||
result is carried verbatim, in message order. Image blocks — present only
|
||||
while a screenshot is still inside the scrub edge's pending-grace window —
|
||||
become a placeholder; a resolved visual-transition summary is already
|
||||
plain text. The segment capsule uses this transcript as its source when
|
||||
compressing the turn.
|
||||
|
||||
A tool result is labelled with its tool's name, resolved through the
|
||||
turn's own preceding tool calls when the message carries no name (the Pro
|
||||
operator's ToolMessages don't); the opaque call id is shown only when no
|
||||
name can be resolved.
|
||||
"""
|
||||
lines: list[str] = []
|
||||
names_by_id: dict[str, str] = {}
|
||||
for msg in messages:
|
||||
kind = str(getattr(msg, "type", "") or "message")
|
||||
role = _ROLE_LABELS.get(kind, kind)
|
||||
if kind == "tool":
|
||||
call_id = getattr(msg, "tool_call_id", None)
|
||||
name = getattr(msg, "name", None) or names_by_id.get(str(call_id)) or call_id
|
||||
if name:
|
||||
role = f"{role} {name}"
|
||||
status = getattr(msg, "status", None)
|
||||
if status and status != "success":
|
||||
role = f"{role} ({status})"
|
||||
body = _render_content_text(getattr(msg, "content", None))
|
||||
if body:
|
||||
lines.append(f"[{role}]\n{body}")
|
||||
for call in getattr(msg, "tool_calls", None) or []:
|
||||
if not isinstance(call, dict):
|
||||
continue
|
||||
if call.get("id") and call.get("name"):
|
||||
names_by_id[str(call["id"])] = str(call["name"])
|
||||
args = json.dumps(call.get("args") or {}, ensure_ascii=False, default=str)
|
||||
lines.append(f"[tool call] {call.get('name')}({args})")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
class TranscriptLedger:
|
||||
"""Append-only four-region message ledger for one Pro session.
|
||||
|
||||
@@ -111,7 +193,7 @@ class TranscriptLedger:
|
||||
self._staged: list[BaseMessage] | None = None
|
||||
self._turn_count = 0
|
||||
|
||||
# L2/L3 chunk compression (M3). ``_turns`` records each committed
|
||||
# L2/L3 chunk compression. ``_turns`` records each committed
|
||||
# turn's message span in ``_active``; a compression event advances
|
||||
# ``_active_start`` past whole turns (never splitting tool-call pairs)
|
||||
# and replaces the frozen chunk blocks wholesale. The underlying
|
||||
@@ -168,7 +250,7 @@ class TranscriptLedger:
|
||||
self._static = list(messages)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# F region (M2: cold-start restored history only)
|
||||
# F region: restored history
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@property
|
||||
@@ -308,7 +390,7 @@ class TranscriptLedger:
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# F region (M3): chunk compression primitives
|
||||
# F region: chunk compression
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def attach_chunker(self, chunker: Any) -> None:
|
||||
@@ -328,18 +410,17 @@ class TranscriptLedger:
|
||||
return [dict(t) for t in self._turns[self._chunked_turn_count :]]
|
||||
|
||||
def turn_text_chars(self, turns: list[dict]) -> int:
|
||||
"""Total text characters of the given turns' messages (size trigger)."""
|
||||
total = 0
|
||||
for turn in turns:
|
||||
for msg in self._active[turn["start"] : turn["end"]]:
|
||||
content = getattr(msg, "content", None)
|
||||
if isinstance(content, str):
|
||||
total += len(content)
|
||||
elif isinstance(content, list):
|
||||
for block in content:
|
||||
if isinstance(block, dict) and block.get("type") == "text":
|
||||
total += len(block.get("text", ""))
|
||||
return total
|
||||
"""Total characters of the given turns' rendered transcripts (size
|
||||
trigger). Measured on the very text the chunk capsule receives, so
|
||||
the trigger sizes exactly what gets compressed."""
|
||||
return sum(len(self.turn_transcript(turn)) for turn in turns)
|
||||
|
||||
def turn_transcript(self, turn: dict) -> str:
|
||||
"""Plain-text rendering of one committed turn exactly as it currently
|
||||
stands in the active region (i.e. after the scrub edge), see
|
||||
:func:`render_turn_transcript`. Chunk compression feeds this to the
|
||||
segment capsule so the capsule digests precisely what it replaces."""
|
||||
return render_turn_transcript(self._active[turn["start"] : turn["end"]])
|
||||
|
||||
def freeze_turns(self, turn_count: int, frozen_blocks: list[BaseMessage]) -> None:
|
||||
"""Compression event: consume the oldest ``turn_count`` unchunked turns
|
||||
|
||||
@@ -227,12 +227,17 @@
|
||||
"max_chunks": 8
|
||||
// "max_eras": 8 // independent era cap; omitted = follows max_chunks
|
||||
},
|
||||
// Deterministic cold-history recall tool (Pro operator)
|
||||
// search_history: deterministic keyword / step-range lookup over stored history
|
||||
"recall": {
|
||||
"enabled": true,
|
||||
"max_results": 5,
|
||||
"max_text_tokens": 2000,
|
||||
"max_image_steps": 1
|
||||
"screen_scan_steps": 150
|
||||
},
|
||||
// replay_steps: full step replays (whole trailing steps are dropped over budget)
|
||||
"replay": {
|
||||
"max_steps": 5,
|
||||
"max_tokens": 12000
|
||||
}
|
||||
},
|
||||
// 🎬 Screen Recording & Video Analyzer Configuration
|
||||
|
||||
+30
-1
@@ -42,7 +42,13 @@ from pydantic import BaseModel
|
||||
from artemis.agents.flash.runner import FlashRunner
|
||||
from artemis.agents.outputter.outputter import outputter
|
||||
from artemis.clients.ui_automator_client import UIAutomatorClient
|
||||
from artemis.config import OutputConfig, record_events, settings
|
||||
from artemis.config import (
|
||||
CheckerConfig,
|
||||
OutputConfig,
|
||||
record_events,
|
||||
run_tuning_for_profile,
|
||||
settings,
|
||||
)
|
||||
from artemis.constants import RECURSION_LIMIT
|
||||
from artemis.context import (
|
||||
ArtemisContext,
|
||||
@@ -93,6 +99,26 @@ TOutput = TypeVar("TOutput", bound=BaseModel | None)
|
||||
load_dotenv()
|
||||
|
||||
|
||||
def run_tuning_summary(config: AgentConfig, profile: str | None) -> dict[str, str] | None:
|
||||
"""Per-run Pro tuning (verification level, explorer mode) for an SDK config.
|
||||
|
||||
The SDK config carries the Checker switches as flat flags; fold them back
|
||||
into a :class:`CheckerConfig` so the ladder classification stays single-
|
||||
sourced in ``artemis.config``. Returns ``None`` for Flash runs.
|
||||
"""
|
||||
return run_tuning_for_profile(
|
||||
profile,
|
||||
checker=CheckerConfig(
|
||||
enabled=not config.disable_checker,
|
||||
midway_checks=not config.disable_midway_checks,
|
||||
final_check=not config.disable_final_check,
|
||||
assert_failure_policy=config.assert_failure_policy,
|
||||
),
|
||||
explorer=config.explorer,
|
||||
explorer_versions=dict(config.explorer_versions or {}),
|
||||
)
|
||||
|
||||
|
||||
class Agent:
|
||||
_config: AgentConfig
|
||||
_tasks: list[Task] = []
|
||||
@@ -1108,6 +1134,9 @@ class Agent:
|
||||
device_data = context.device.model_dump() if context.device else {}
|
||||
if task.request.profile:
|
||||
device_data["profile"] = task.request.profile
|
||||
run_tuning = run_tuning_summary(self._config, task.request.profile)
|
||||
if run_tuning:
|
||||
device_data["run_tuning"] = run_tuning
|
||||
|
||||
target_sid = (
|
||||
self._session_id
|
||||
|
||||
@@ -33,6 +33,7 @@ This module never raises into the LLM call path and makes no decisions.
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
from artemis.data_engine.context_vars import CURRENT_NODE_NAME
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
@@ -40,6 +41,16 @@ logger = get_logger(__name__)
|
||||
_LOCK = threading.Lock()
|
||||
_METERS: dict[str, "SessionTokenMeter"] = {}
|
||||
|
||||
# Size of the model context window the live Operator/FlashRunner prompt is
|
||||
# measured against when the UI reports "context used". The prompt size of the
|
||||
# last executor call (``context_base_tokens`` on its ``llm_usage`` trace) is
|
||||
# the numerator; this is the denominator.
|
||||
OPERATOR_CONTEXT_WINDOW_TOKENS = 1_000_000
|
||||
|
||||
# Traced node names whose LLM calls carry the live executor context. Both the
|
||||
# Pro Operator and the Flash runner are "the operator" from the user's view.
|
||||
OPERATOR_NODE_NAMES = frozenset({"operator", "flashrunner"})
|
||||
|
||||
|
||||
def extract_usage(response: Any) -> dict[str, int] | None:
|
||||
"""Extracts unified token usage from an LLM response message, best-effort.
|
||||
@@ -184,6 +195,9 @@ def record_llm_usage(
|
||||
}
|
||||
if source:
|
||||
payload["source"] = source
|
||||
node = CURRENT_NODE_NAME.get()
|
||||
if node:
|
||||
payload["node"] = str(node)
|
||||
|
||||
engine.record_trace(
|
||||
type="llm_call",
|
||||
|
||||
+77
-44
@@ -30,6 +30,7 @@ from langchain_core.tools import BaseTool, StructuredTool
|
||||
from langgraph.prebuilt import InjectedState
|
||||
from pydantic import BaseModel
|
||||
|
||||
from artemis.core.tool_declaration import ToolDeclaration
|
||||
from artemis.drivers.base import BaseDeviceDriver
|
||||
from artemis.drivers.factory import get_driver
|
||||
from artemis.utils.logger import get_logger
|
||||
@@ -133,59 +134,37 @@ class ArtemisTool:
|
||||
args_schema=self.args_schema,
|
||||
)
|
||||
|
||||
def _flat_schema(self) -> tuple[dict[str, dict[str, Any]], list[str]]:
|
||||
if not self.args_schema:
|
||||
return {}, []
|
||||
return _flatten_json_schema_properties(self.args_schema.model_json_schema())
|
||||
|
||||
def to_tool_declaration(self) -> ToolDeclaration:
|
||||
"""Build the Flash declaration from the args_schema used by LangChain."""
|
||||
properties, required = self._flat_schema()
|
||||
return ToolDeclaration(
|
||||
name=self.name,
|
||||
description=self.description,
|
||||
parameters={"type": "object", "properties": properties, "required": required},
|
||||
)
|
||||
|
||||
def to_genai_declaration(self) -> genai_types.FunctionDeclaration:
|
||||
"""Exports this tool to a Google GenAI FunctionDeclaration schema."""
|
||||
flat_properties, required = self._flat_schema()
|
||||
properties = {}
|
||||
required = []
|
||||
|
||||
if self.args_schema:
|
||||
schema = self.args_schema.model_json_schema()
|
||||
props = schema.get("properties", {})
|
||||
required = schema.get("required", [])
|
||||
|
||||
for prop_name, prop_data in props.items():
|
||||
p_type_str = prop_data.get("type")
|
||||
items_data = prop_data.get("items")
|
||||
if not p_type_str and "anyOf" in prop_data:
|
||||
for sub_schema in prop_data["anyOf"]:
|
||||
sub_type = sub_schema.get("type")
|
||||
if sub_type and sub_type != "null":
|
||||
p_type_str = sub_type
|
||||
if "items" in sub_schema and not items_data:
|
||||
items_data = sub_schema["items"]
|
||||
break
|
||||
if not p_type_str:
|
||||
p_type_str = "string"
|
||||
|
||||
p_desc = prop_data.get("description", "")
|
||||
|
||||
genai_type = genai_types.Type.STRING
|
||||
if p_type_str == "integer":
|
||||
genai_type = genai_types.Type.INTEGER
|
||||
elif p_type_str == "number":
|
||||
genai_type = genai_types.Type.NUMBER
|
||||
elif p_type_str == "boolean":
|
||||
genai_type = genai_types.Type.BOOLEAN
|
||||
elif p_type_str == "array":
|
||||
genai_type = genai_types.Type.ARRAY
|
||||
elif p_type_str == "object":
|
||||
genai_type = genai_types.Type.OBJECT
|
||||
|
||||
# Handle array items if present
|
||||
for prop_name, prop in flat_properties.items():
|
||||
genai_type = _GENAI_TYPES.get(prop["type"], genai_types.Type.STRING)
|
||||
items_schema = None
|
||||
if genai_type == genai_types.Type.ARRAY:
|
||||
item_type_str = (items_data or {}).get("type", "string")
|
||||
item_genai_type = (
|
||||
genai_types.Type.INTEGER
|
||||
if item_type_str == "integer"
|
||||
else genai_types.Type.STRING
|
||||
item_type = (prop.get("items") or {}).get("type", "string")
|
||||
items_schema = genai_types.Schema(
|
||||
type=_GENAI_TYPES.get(item_type, genai_types.Type.STRING)
|
||||
)
|
||||
items_schema = genai_types.Schema(type=item_genai_type)
|
||||
|
||||
properties[prop_name] = genai_types.Schema(
|
||||
type=genai_type,
|
||||
description=p_desc,
|
||||
description=prop.get("description", ""),
|
||||
items=items_schema,
|
||||
enum=[str(v) for v in prop["enum"]] if prop.get("enum") else None,
|
||||
)
|
||||
|
||||
return genai_types.FunctionDeclaration(
|
||||
@@ -197,3 +176,57 @@ class ArtemisTool:
|
||||
required=required,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
_GENAI_TYPES = {
|
||||
"string": genai_types.Type.STRING,
|
||||
"integer": genai_types.Type.INTEGER,
|
||||
"number": genai_types.Type.NUMBER,
|
||||
"boolean": genai_types.Type.BOOLEAN,
|
||||
"array": genai_types.Type.ARRAY,
|
||||
"object": genai_types.Type.OBJECT,
|
||||
}
|
||||
|
||||
|
||||
def _flatten_property(prop_data: dict[str, Any]) -> dict[str, Any]:
|
||||
"""One Pydantic JSON-schema property as a flat single-type entry.
|
||||
|
||||
``anyOf`` unions (``X | None``) collapse to their first non-null member;
|
||||
``items`` (array element type) and ``enum`` (``Literal`` choices) are
|
||||
carried over; unknown shapes fall back to ``string``.
|
||||
"""
|
||||
p_type = prop_data.get("type")
|
||||
items = prop_data.get("items")
|
||||
enum = prop_data.get("enum")
|
||||
if not p_type and "anyOf" in prop_data:
|
||||
for sub_schema in prop_data["anyOf"]:
|
||||
sub_type = sub_schema.get("type")
|
||||
if sub_type and sub_type != "null":
|
||||
p_type = sub_type
|
||||
items = items or sub_schema.get("items")
|
||||
enum = enum or sub_schema.get("enum")
|
||||
break
|
||||
if not p_type:
|
||||
p_type = "string"
|
||||
entry: dict[str, Any] = {"type": p_type}
|
||||
if p_type == "array":
|
||||
entry["items"] = {"type": (items or {}).get("type") or "string"}
|
||||
if enum:
|
||||
entry["enum"] = list(enum)
|
||||
return entry
|
||||
|
||||
|
||||
def _flatten_json_schema_properties(
|
||||
schema: dict[str, Any],
|
||||
) -> tuple[dict[str, dict[str, Any]], list[str]]:
|
||||
"""Flattens a Pydantic model JSON schema into plain JSON-schema properties
|
||||
plus the required-field list. Descriptions are copied verbatim; non-null
|
||||
defaults are kept under ``default``."""
|
||||
properties: dict[str, dict[str, Any]] = {}
|
||||
for prop_name, prop_data in (schema.get("properties") or {}).items():
|
||||
entry = _flatten_property(prop_data)
|
||||
entry["description"] = prop_data.get("description", "")
|
||||
if prop_data.get("default") is not None:
|
||||
entry["default"] = prop_data["default"]
|
||||
properties[prop_name] = entry
|
||||
return properties, list(schema.get("required") or [])
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"planner_avatar": "You are the **Planner Avatar** (Meeting Host) in a committee resolving a task plan failure. Your role is to lead the meeting and advocate for a robust and correct high-level plan. You are aware of:\n- **History Analyzer**: The trajectory specialist who can analyze execution history and read notes. They have tools like `get_step_details` and `read_note`.\n- **Diagnoser**: The forensics specialist who can analyze logs and videos to find why things failed. They have tools like `video_analyzer` and `analyze_logs`.\nYour master (Master Planner) has given you a specific directive for this meeting, which is at the top of the blackboard.\n**Your Task**:\n1. In the first round, you must open the meeting, state the problem based on the master's directive, and call on the other experts (Diagnoser or History Analyzer) to provide their analysis.\n2. In subsequent rounds, evaluate their suggestions and push for a concrete, revised plan of subgoals.\n3. In the final round, summarize the discussion and output the final recommended plan revisions and actions.",
|
||||
"history_analyzer_expert": "You are the **History Analyzer** (Trajectory Specialist) in a committee resolving a task failure. Your role is to analyze the execution history of the current session, understand what has been tried, what succeeded, and what failed, and check persistent notes. You are aware of:\n- **Planner Avatar**: The meeting host who needs a concrete plan.\n- **Diagnoser**: The forensics specialist who analyzes logs and videos.\nYou have tools to retrieve full-granularity step details and read notes. Wait for the Planner Avatar to call on you or respond to the current state of the blackboard. Be concise.",
|
||||
"planner_avatar": "You are the **Planner Avatar** (Meeting Host) in a committee resolving a task plan failure. Your role is to lead the meeting and advocate for a robust and correct high-level plan. You are aware of:\n- **History Analyzer**: The trajectory specialist who can analyze execution history and read notes. They have tools like `search_history`, `replay_steps`, `get_step_screenshot` and `read_note`.\n- **Diagnoser**: The forensics specialist who can analyze logs and videos to find why things failed. They have tools like `video_analyzer` and `analyze_logs`.\nYour master (Master Planner) has given you a specific directive for this meeting, which is at the top of the blackboard.\n**Your Task**:\n1. In the first round, you must open the meeting, state the problem based on the master's directive, and call on the other experts (Diagnoser or History Analyzer) to provide their analysis.\n2. In subsequent rounds, evaluate their suggestions and push for a concrete, revised plan of subgoals.\n3. In the final round, summarize the discussion and output the final recommended plan revisions and actions.",
|
||||
"history_analyzer_expert": "You are the **History Analyzer** (Trajectory Specialist) in a committee resolving a task failure. Your role is to analyze the execution history of the current session, understand what has been tried, what succeeded, and what failed, and check persistent notes. You are aware of:\n- **Planner Avatar**: The meeting host who needs a concrete plan.\n- **Diagnoser**: The forensics specialist who analyzes logs and videos.\nYou have tools to search the execution history (`search_history`), replay steps in full (`replay_steps`), look at a step's stored screenshot (`get_step_screenshot`) and read notes. Wait for the Planner Avatar to call on you or respond to the current state of the blackboard. Be concise.",
|
||||
"diagnoser_expert": "You are the **Diagnoser** (Forensics Specialist) in a committee resolving a UI automation failure. Your role is to analyze what went wrong in the past, look for error patterns in logs, and understand system state transitions. You are aware of:\n- **Planner Avatar**: The meeting host who needs a concrete plan.\n- **History Analyzer**: The trajectory specialist who analyzes history and notes.\nYou have tools to analyze video and logs. Wait for the Planner Avatar to call on you or respond to the current state of the blackboard. Be concise."
|
||||
}
|
||||
@@ -22,11 +22,10 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from langchain_core.messages import HumanMessage, SystemMessage, ToolMessage
|
||||
from langchain_core.messages import HumanMessage, SystemMessage
|
||||
from langchain_core.tools import BaseTool
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from artemis.agents.history_analyzer.history_analyzer import HistoryAnalyzer
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.data_engine.trace import (
|
||||
CURRENT_TRACE_ID,
|
||||
@@ -37,12 +36,15 @@ from artemis.drivers.base import BaseDeviceDriver
|
||||
from artemis.graph.state import State
|
||||
from artemis.services.llm import get_llm, invoke_llm_with_timeout_message
|
||||
from artemis.tools.base import ArtemisTool, ToolCategory
|
||||
from artemis.tools.history import get_history_tools
|
||||
from artemis.tools.log_tool import get_analyze_logs_tool
|
||||
from artemis.tools.scratchpad import get_list_notes_tool, get_read_note_tool
|
||||
from artemis.tools.tool_wrapper import (
|
||||
ToolWrapper,
|
||||
get_tool_result_content,
|
||||
invoke_tool_with_injection,
|
||||
split_multimodal_result,
|
||||
tool_result_messages,
|
||||
)
|
||||
from artemis.tools.types import CyFunctionDetector
|
||||
from artemis.tools.video_tool import get_video_analyzer_tool
|
||||
@@ -232,16 +234,12 @@ async def _execute_committee(
|
||||
|
||||
# 4. Prepare Specialized Tools for Members
|
||||
|
||||
# History Analyzer Tools
|
||||
analyzer_inst = HistoryAnalyzer(ctx)
|
||||
history_steps = ctx.data_engine.get_agent_friendly_steps() if ctx.data_engine else []
|
||||
# pylint: disable=protected-access
|
||||
details_tool = analyzer_inst._get_step_details_tool(history_steps)
|
||||
# History Analyzer Tools: the shared history tools plus the note readers
|
||||
read_note_tool_hist = get_read_note_tool(ctx)
|
||||
list_notes_tool_hist = get_list_notes_tool(ctx)
|
||||
|
||||
history_analyzer_tools = [
|
||||
trace_langchain_tool(details_tool, ctx),
|
||||
*get_history_tools(ctx),
|
||||
trace_langchain_tool(read_note_tool_hist, ctx),
|
||||
trace_langchain_tool(list_notes_tool_hist, ctx),
|
||||
]
|
||||
@@ -328,11 +326,16 @@ async def _execute_committee(
|
||||
tool_outputs = await asyncio.gather(*(run_tool(tc) for tc in active_tool_calls))
|
||||
|
||||
for tc, result in zip(active_tool_calls, tool_outputs):
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc["id"],
|
||||
content=result,
|
||||
status="success" if not result.startswith("Error") else "error",
|
||||
# A step screenshot (content blocks) travels in the carrier
|
||||
# the member's provider accepts; text stays a ToolMessage.
|
||||
text, _ = split_multimodal_result(result)
|
||||
messages.extend(
|
||||
tool_result_messages(
|
||||
tc["id"],
|
||||
result,
|
||||
name=tc["name"],
|
||||
status="error" if text.startswith("Error") else "success",
|
||||
llm=llm,
|
||||
)
|
||||
)
|
||||
return "Error: Reached max iterations in turn."
|
||||
|
||||
@@ -0,0 +1,397 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Shared history search, step replay, and screenshot tools.
|
||||
|
||||
Each tool's args_schema defines both its LangChain interface for Pro and its
|
||||
ToolDeclaration for Flash. The MCP trace inspector calls the underlying
|
||||
functions with an OfflineHistoryReader.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Literal
|
||||
|
||||
from langchain_core.tools import BaseTool
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from artemis.core.tool_declaration import ToolDeclaration
|
||||
from artemis.data_engine.trace import trace_langchain_tool
|
||||
from artemis.tools.base import ArtemisTool
|
||||
from artemis.tools.history.replay import (
|
||||
DEFAULT_MAX_STEPS as DEFAULT_REPLAY_MAX_STEPS,
|
||||
)
|
||||
from artemis.tools.history.replay import (
|
||||
DEFAULT_MAX_TOKENS as DEFAULT_REPLAY_MAX_TOKENS,
|
||||
)
|
||||
from artemis.tools.history.replay import replay_steps_text
|
||||
from artemis.tools.history.screenshot import ScreenshotResult, load_step_screenshot
|
||||
from artemis.tools.history.search import search_history_text
|
||||
from artemis.tools.tool_wrapper import ToolWrapper
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
__all__ = [
|
||||
"HISTORY_TOOLS",
|
||||
"HISTORY_TOOL_DECLARATIONS",
|
||||
"HISTORY_TOOL_NAMES",
|
||||
"HISTORY_TOOL_WRAPPERS",
|
||||
"GET_STEP_SCREENSHOT_TOOL",
|
||||
"REPLAY_STEPS_TOOL",
|
||||
"SEARCH_HISTORY_TOOL",
|
||||
"GetStepScreenshotArgs",
|
||||
"GetStepScreenshotTool",
|
||||
"ReplayStepsArgs",
|
||||
"ReplayStepsTool",
|
||||
"ScreenshotResult",
|
||||
"SearchHistoryArgs",
|
||||
"SearchHistoryTool",
|
||||
"get_history_tools",
|
||||
"get_replay_steps_tool",
|
||||
"get_search_history_tool",
|
||||
"get_step_screenshot",
|
||||
"get_step_screenshot_tool",
|
||||
"history_available",
|
||||
"history_tool_by_name",
|
||||
"history_tool_declarations",
|
||||
"load_step_screenshot",
|
||||
"replay_steps",
|
||||
"replay_steps_text",
|
||||
"search_history",
|
||||
"search_history_available",
|
||||
"search_history_text",
|
||||
]
|
||||
|
||||
|
||||
# --- Configuration -----------------------------------------------------------------------
|
||||
|
||||
|
||||
def _memory_config(warn: bool = True) -> Any:
|
||||
try:
|
||||
from artemis.config import load_agent_config
|
||||
|
||||
return load_agent_config().memory
|
||||
except Exception as e:
|
||||
if warn:
|
||||
logger.debug(f"memory config unavailable, using defaults: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def _recall_config(warn: bool = True) -> Any:
|
||||
return getattr(_memory_config(warn), "recall", None)
|
||||
|
||||
|
||||
def _replay_config(warn: bool = True) -> Any:
|
||||
return getattr(_memory_config(warn), "replay", None)
|
||||
|
||||
|
||||
def history_available(ctx: Any) -> bool:
|
||||
"""History tools need a stored history to read (a DataEngine session)."""
|
||||
return ctx is not None and getattr(ctx, "data_engine", None) is not None
|
||||
|
||||
|
||||
def search_history_available(ctx: Any) -> bool:
|
||||
"""``search_history`` is additionally gated by ``agent.memory.recall.enabled``."""
|
||||
if not history_available(ctx):
|
||||
return False
|
||||
return bool(getattr(_recall_config(warn=False), "enabled", True))
|
||||
|
||||
|
||||
# --- search_history ----------------------------------------------------------------------
|
||||
|
||||
|
||||
class SearchHistoryArgs(BaseModel):
|
||||
"""Arguments schema for the search_history tool."""
|
||||
|
||||
query: str = Field(
|
||||
default="",
|
||||
description=(
|
||||
"Keywords to search for across the whole execution history"
|
||||
" (screen descriptions, actions, results, reasoning, tool calls,"
|
||||
" notes, on-screen text, package/activity names). Case-insensitive;"
|
||||
" every whitespace-separated term is matched independently. May be"
|
||||
" empty when step_range is given."
|
||||
),
|
||||
)
|
||||
step_range: list[int] | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Optional [start, end] step-number range (inclusive) to restrict the"
|
||||
" search — e.g. the range shown in a compressed-history block. Also"
|
||||
" returns the full per-step action ledger of the range."
|
||||
),
|
||||
)
|
||||
max_results: int = Field(
|
||||
default=5,
|
||||
ge=1,
|
||||
description="Maximum results to return (server-side cap still applies).",
|
||||
)
|
||||
|
||||
|
||||
SEARCH_HISTORY_DESCRIPTION = (
|
||||
"Deterministic keyword / step-range lookup over the full stored execution"
|
||||
" history: step summaries, exact actions and results, reasoning, tool"
|
||||
" calls, notes, on-screen text (OCR/UI tree, incl. package/activity names)"
|
||||
" and compressed-history ledgers. Every hit carries its step number; use"
|
||||
" `replay_steps` for the full record of a step and `get_step_screenshot`"
|
||||
" for its image."
|
||||
)
|
||||
|
||||
|
||||
class SearchHistoryTool(ArtemisTool):
|
||||
"""Deterministic search over the stored execution history."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__(
|
||||
name="search_history",
|
||||
description=SEARCH_HISTORY_DESCRIPTION,
|
||||
args_schema=SearchHistoryArgs,
|
||||
category="memory",
|
||||
)
|
||||
|
||||
def is_available(self, ctx: Any = None) -> bool:
|
||||
return search_history_available(ctx)
|
||||
|
||||
# pylint: disable=too-many-arguments
|
||||
async def execute(
|
||||
self,
|
||||
driver: Any = None, # pylint: disable=unused-argument
|
||||
ctx: Any = None,
|
||||
query: str = "",
|
||||
step_range: list[int] | None = None,
|
||||
max_results: int = 5,
|
||||
**kwargs: Any,
|
||||
) -> str:
|
||||
reader = getattr(ctx, "data_engine", None) if ctx else None
|
||||
if reader is None:
|
||||
return "search_history unavailable: no active execution history."
|
||||
if not query and not step_range:
|
||||
return (
|
||||
"search_history needs a query and/or a step_range — e.g."
|
||||
' search_history(query="login timeout") or'
|
||||
' search_history(query="", step_range=[1, 40]).'
|
||||
)
|
||||
try:
|
||||
return search_history_text(
|
||||
reader,
|
||||
query=query or "",
|
||||
step_range=step_range,
|
||||
max_results=int(max_results or 5),
|
||||
recall_config=_recall_config(),
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"search_history failed: {e}")
|
||||
return f"search_history failed: {e}"
|
||||
|
||||
|
||||
# --- replay_steps ------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ReplayStepsArgs(BaseModel):
|
||||
"""Arguments schema for the replay_steps tool."""
|
||||
|
||||
start_step: int = Field(..., description="First step to replay (1-based step number).")
|
||||
end_step: int | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Last step of the inclusive range to replay; omit to replay the single"
|
||||
" start_step. Reversed bounds are swapped."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
REPLAY_STEPS_DESCRIPTION = (
|
||||
"Replays recorded steps exactly as the executing agent's own context showed"
|
||||
" them: what the screen showed, its reasoning, every tool call (name,"
|
||||
" arguments, result), the planned action, any pre-execution interception"
|
||||
" and the execution result. Coordinates normalized [x, y]. Up to"
|
||||
f" {DEFAULT_REPLAY_MAX_STEPS} steps per call. Screenshots are not included"
|
||||
" — fetch them with `get_step_screenshot`."
|
||||
)
|
||||
|
||||
|
||||
class ReplayStepsTool(ArtemisTool):
|
||||
"""Full replay of a range of recorded steps."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__(
|
||||
name="replay_steps",
|
||||
description=REPLAY_STEPS_DESCRIPTION,
|
||||
args_schema=ReplayStepsArgs,
|
||||
category="memory",
|
||||
)
|
||||
|
||||
def is_available(self, ctx: Any = None) -> bool:
|
||||
return history_available(ctx)
|
||||
|
||||
async def execute(
|
||||
self,
|
||||
driver: Any = None, # pylint: disable=unused-argument
|
||||
ctx: Any = None,
|
||||
start_step: Any = None,
|
||||
end_step: Any = None,
|
||||
**kwargs: Any,
|
||||
) -> str:
|
||||
reader = getattr(ctx, "data_engine", None) if ctx else None
|
||||
if reader is None:
|
||||
return "Error: no execution history available."
|
||||
cfg = _replay_config()
|
||||
try:
|
||||
return replay_steps_text(
|
||||
reader,
|
||||
start_step,
|
||||
end_step,
|
||||
max_steps=int(getattr(cfg, "max_steps", None) or DEFAULT_REPLAY_MAX_STEPS),
|
||||
max_tokens=int(getattr(cfg, "max_tokens", None) or DEFAULT_REPLAY_MAX_TOKENS),
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"replay_steps failed: {e}")
|
||||
return f"replay_steps failed: {e}"
|
||||
|
||||
|
||||
# --- get_step_screenshot -----------------------------------------------------------------
|
||||
|
||||
|
||||
class GetStepScreenshotArgs(BaseModel):
|
||||
"""Arguments schema for the get_step_screenshot tool."""
|
||||
|
||||
step_number: int = Field(..., description="The step whose screenshot to attach (1-based).")
|
||||
which: Literal["pre", "post", "overlay"] = Field(
|
||||
default="pre",
|
||||
description=(
|
||||
"'pre' = screen observed at the start of the step; 'post' = screen"
|
||||
" after the action (when recorded); 'overlay' = the pre screenshot"
|
||||
" with the step's action drawn on it."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
GET_STEP_SCREENSHOT_DESCRIPTION = (
|
||||
"Attaches one recorded screenshot of a step: 'pre' (screen observed at the"
|
||||
" start of the step), 'post' (screen after the action, when recorded) or"
|
||||
" 'overlay' (the pre screenshot with the step's action drawn on it — tap"
|
||||
" point / swipe path — to check the action landed on the intended element)."
|
||||
)
|
||||
|
||||
|
||||
class GetStepScreenshotTool(ArtemisTool):
|
||||
"""One recorded step screenshot, attached to the conversation."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__(
|
||||
name="get_step_screenshot",
|
||||
description=GET_STEP_SCREENSHOT_DESCRIPTION,
|
||||
args_schema=GetStepScreenshotArgs,
|
||||
category="memory",
|
||||
)
|
||||
|
||||
def is_available(self, ctx: Any = None) -> bool:
|
||||
return history_available(ctx)
|
||||
|
||||
async def execute(
|
||||
self,
|
||||
driver: Any = None, # pylint: disable=unused-argument
|
||||
ctx: Any = None,
|
||||
step_number: Any = None,
|
||||
which: str = "pre",
|
||||
**kwargs: Any,
|
||||
) -> list[dict[str, Any]] | str:
|
||||
reader = getattr(ctx, "data_engine", None) if ctx else None
|
||||
if reader is None:
|
||||
return "Error: no execution history available."
|
||||
try:
|
||||
return load_step_screenshot(reader, step_number, which).to_content_blocks()
|
||||
except Exception as e:
|
||||
logger.error(f"get_step_screenshot failed: {e}")
|
||||
return f"get_step_screenshot failed: {e}"
|
||||
|
||||
|
||||
# --- Instances, declarations, exports ------------------------------------------------------
|
||||
|
||||
search_history = SearchHistoryTool()
|
||||
replay_steps = ReplayStepsTool()
|
||||
get_step_screenshot = GetStepScreenshotTool()
|
||||
|
||||
HISTORY_TOOLS: tuple[ArtemisTool, ...] = (search_history, replay_steps, get_step_screenshot)
|
||||
HISTORY_TOOL_NAMES: frozenset[str] = frozenset(t.name for t in HISTORY_TOOLS)
|
||||
|
||||
#: Flash-profile declarations, derived from the very same args schemas.
|
||||
SEARCH_HISTORY_TOOL: ToolDeclaration = search_history.to_tool_declaration()
|
||||
REPLAY_STEPS_TOOL: ToolDeclaration = replay_steps.to_tool_declaration()
|
||||
GET_STEP_SCREENSHOT_TOOL: ToolDeclaration = get_step_screenshot.to_tool_declaration()
|
||||
HISTORY_TOOL_DECLARATIONS: tuple[ToolDeclaration, ...] = (
|
||||
SEARCH_HISTORY_TOOL,
|
||||
REPLAY_STEPS_TOOL,
|
||||
GET_STEP_SCREENSHOT_TOOL,
|
||||
)
|
||||
|
||||
|
||||
def history_tool_by_name(name: str) -> ArtemisTool | None:
|
||||
"""The shared tool instance behind a history tool name (Flash dispatch)."""
|
||||
raw = name.split(":")[-1] if ":" in name else name
|
||||
return next((t for t in HISTORY_TOOLS if t.name == raw), None)
|
||||
|
||||
|
||||
def history_tool_declarations(ctx: Any) -> list[ToolDeclaration]:
|
||||
"""Declarations of the history tools available for ``ctx`` (Flash runner)."""
|
||||
return [tool.to_tool_declaration() for tool in HISTORY_TOOLS if tool.is_available(ctx)]
|
||||
|
||||
|
||||
def get_search_history_tool(ctx: Any) -> BaseTool:
|
||||
"""Exports search_history as a traced LangChain BaseTool."""
|
||||
return trace_langchain_tool(search_history.to_langchain_tool(ctx), ctx)
|
||||
|
||||
|
||||
def get_replay_steps_tool(ctx: Any) -> BaseTool:
|
||||
"""Exports replay_steps as a traced LangChain BaseTool."""
|
||||
return trace_langchain_tool(replay_steps.to_langchain_tool(ctx), ctx)
|
||||
|
||||
|
||||
def get_step_screenshot_tool(ctx: Any) -> BaseTool:
|
||||
"""Exports get_step_screenshot as a traced LangChain BaseTool."""
|
||||
return trace_langchain_tool(get_step_screenshot.to_langchain_tool(ctx), ctx)
|
||||
|
||||
|
||||
def get_history_tools(ctx: Any) -> list[BaseTool]:
|
||||
"""All three history tools as LangChain tools (read-only agents mount them as a set)."""
|
||||
return [get_search_history_tool(ctx), get_replay_steps_tool(ctx), get_step_screenshot_tool(ctx)]
|
||||
|
||||
|
||||
search_history_wrapper = ToolWrapper(
|
||||
tool_fn_getter=get_search_history_tool,
|
||||
on_success_fn=lambda *a, **k: "Searched history",
|
||||
on_failure_fn=lambda err: f"search_history failed: {err}",
|
||||
is_available_fn=search_history_available,
|
||||
)
|
||||
|
||||
replay_steps_wrapper = ToolWrapper(
|
||||
tool_fn_getter=get_replay_steps_tool,
|
||||
on_success_fn=lambda *a, **k: "Replayed steps",
|
||||
on_failure_fn=lambda err: f"replay_steps failed: {err}",
|
||||
is_available_fn=history_available,
|
||||
)
|
||||
|
||||
get_step_screenshot_wrapper = ToolWrapper(
|
||||
tool_fn_getter=get_step_screenshot_tool,
|
||||
on_success_fn=lambda *a, **k: "Fetched step screenshot",
|
||||
on_failure_fn=lambda err: f"get_step_screenshot failed: {err}",
|
||||
is_available_fn=history_available,
|
||||
)
|
||||
|
||||
HISTORY_TOOL_WRAPPERS: tuple[ToolWrapper, ...] = (
|
||||
search_history_wrapper,
|
||||
replay_steps_wrapper,
|
||||
get_step_screenshot_wrapper,
|
||||
)
|
||||
@@ -0,0 +1,115 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""replay_steps: full replay of a step range, exactly as the executing agent saw it.
|
||||
|
||||
Each step is rendered with :func:`~artemis.utils.task_tree.render_step_replay`
|
||||
(``[Screen]`` description, reasoning, every non-internal tool call with name /
|
||||
arguments / result, the planned action, safety-net interception and the
|
||||
execution result). Two loose bounds keep one tool result finite (config
|
||||
``agent.memory.replay``): at most ``max_steps`` steps per call, and a
|
||||
``max_tokens`` budget applied by dropping *whole trailing steps* — a replayed
|
||||
step is never cut mid-sentence. Screenshots are deliberately not part of the
|
||||
replay; ``get_step_screenshot`` fetches them one at a time.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from artemis.utils.task_tree import render_step_replay
|
||||
|
||||
DEFAULT_MAX_STEPS = 5
|
||||
DEFAULT_MAX_TOKENS = 12000
|
||||
|
||||
|
||||
def _estimate_tokens(text: str) -> int:
|
||||
return len(text) // 4
|
||||
|
||||
|
||||
def replay_steps_text(
|
||||
reader: Any,
|
||||
start_step: Any,
|
||||
end_step: Any = None,
|
||||
*,
|
||||
max_steps: int = DEFAULT_MAX_STEPS,
|
||||
max_tokens: int = DEFAULT_MAX_TOKENS,
|
||||
) -> str:
|
||||
"""Replays steps ``start_step``..``end_step`` (inclusive, 1-based) from ``reader``."""
|
||||
if reader is None:
|
||||
return "Error: no execution history available."
|
||||
try:
|
||||
start = int(start_step)
|
||||
end = start if end_step is None else int(end_step)
|
||||
except (TypeError, ValueError):
|
||||
return (
|
||||
f"Error: start_step and end_step must be integers, got {start_step!r} and {end_step!r}."
|
||||
)
|
||||
if start > end:
|
||||
start, end = end, start
|
||||
|
||||
max_steps = max(1, int(max_steps or DEFAULT_MAX_STEPS))
|
||||
notes: list[str] = []
|
||||
if end - start + 1 > max_steps:
|
||||
end = start + max_steps - 1
|
||||
notes.append(
|
||||
f"(Only {max_steps} steps are replayed per call: showing Steps {start}–{end}."
|
||||
f" Call again from Step {end + 1} for the rest.)"
|
||||
)
|
||||
|
||||
try:
|
||||
steps: list[dict] = reader.get_agent_friendly_steps_in_range(start, end) or []
|
||||
except Exception as e:
|
||||
return f"Error loading steps {start}–{end}: {e}"
|
||||
if not steps:
|
||||
if start == end:
|
||||
return f"Error: step {start} not found."
|
||||
return f"Error: no recorded steps in range {start}–{end}."
|
||||
|
||||
budget_tokens = max(1, int(max_tokens or DEFAULT_MAX_TOKENS))
|
||||
rendered: list[str] = []
|
||||
shown: list[dict] = []
|
||||
used = 0
|
||||
dropped = 0
|
||||
for step in steps:
|
||||
if dropped:
|
||||
dropped += 1
|
||||
continue
|
||||
try:
|
||||
text = render_step_replay(step)
|
||||
except Exception as e:
|
||||
return f"Error rendering steps {start}–{end}: {e}"
|
||||
cost = _estimate_tokens(text) + 1
|
||||
# The first step is always replayed; afterwards the budget drops whole
|
||||
# trailing steps rather than truncating a step in the middle.
|
||||
if rendered and used + cost > budget_tokens:
|
||||
dropped = 1
|
||||
continue
|
||||
rendered.append(text)
|
||||
shown.append(step)
|
||||
used += cost
|
||||
|
||||
if dropped:
|
||||
last_shown = shown[-1].get("step_number", start)
|
||||
first_shown = shown[0].get("step_number", start)
|
||||
notes.append(
|
||||
f"(Token budget reached: Steps {first_shown}–{last_shown} replayed;"
|
||||
f" {dropped} more step(s) in the range not shown."
|
||||
f" Call again from Step {last_shown + 1}.)"
|
||||
)
|
||||
|
||||
text = "\n\n".join(rendered)
|
||||
if notes:
|
||||
text += "\n\n" + "\n".join(notes)
|
||||
return text
|
||||
@@ -0,0 +1,128 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""get_step_screenshot: one recorded screenshot of one step, on demand.
|
||||
|
||||
``pre`` is the screen observed at the start of the step, ``post`` the screen
|
||||
after its action (when recorded), ``overlay`` the pre screenshot with the
|
||||
step's action drawn on it (tap point / swipe path) so a reviewer can check
|
||||
that the action landed on the intended element. The overlay is drawn from the
|
||||
step's *raw* stored action (physical pixels), never from the normalized twin.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
from typing import Any
|
||||
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.visualization import overlay_action_on_screenshot
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
WHICH_CHOICES = ("pre", "post", "overlay")
|
||||
|
||||
|
||||
@dataclass
|
||||
class ScreenshotResult:
|
||||
"""One screenshot lookup: a caption plus the JPEG bytes when a file exists."""
|
||||
|
||||
step_number: int
|
||||
which: str
|
||||
description: str
|
||||
image_bytes: bytes | None = None
|
||||
#: True when ``which == "overlay"`` and the action was actually drawn.
|
||||
overlay_drawn: bool = False
|
||||
|
||||
def to_content_blocks(self) -> list[dict[str, Any]] | str:
|
||||
"""The tool's return value: content blocks with the image, or the
|
||||
plain-text explanation when there is no image to attach."""
|
||||
if not self.image_bytes:
|
||||
return self.description
|
||||
encoded = base64.b64encode(self.image_bytes).decode("utf-8")
|
||||
return [
|
||||
{"type": "text", "text": self.description},
|
||||
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{encoded}"}},
|
||||
]
|
||||
|
||||
|
||||
def load_step_screenshot(reader: Any, step_number: Any, which: str = "pre") -> ScreenshotResult:
|
||||
"""Loads (and for ``overlay`` annotates) one step screenshot from ``reader``."""
|
||||
which = str(which or "pre").lower()
|
||||
try:
|
||||
number = int(step_number)
|
||||
except (TypeError, ValueError):
|
||||
return ScreenshotResult(
|
||||
-1, which, f"Error: step_number must be an integer, got {step_number!r}."
|
||||
)
|
||||
if which not in WHICH_CHOICES:
|
||||
return ScreenshotResult(
|
||||
number, which, f"Error: 'which' must be 'pre', 'post' or 'overlay', got '{which}'."
|
||||
)
|
||||
if reader is None:
|
||||
return ScreenshotResult(number, which, "Error: no execution history available.")
|
||||
|
||||
source = "pre" if which == "overlay" else which
|
||||
try:
|
||||
path = reader.get_step_image_path(number, source)
|
||||
except (sqlite3.Error, OSError, ValueError) as e:
|
||||
return ScreenshotResult(number, which, f"Error looking up step {number}: {e}")
|
||||
if path is None:
|
||||
try:
|
||||
record = reader.get_step_record(number)
|
||||
except (sqlite3.Error, ValueError):
|
||||
record = None
|
||||
if record is None:
|
||||
return ScreenshotResult(number, which, f"Error: step {number} not found.")
|
||||
return ScreenshotResult(
|
||||
number, which, f"No {source}-action screenshot recorded for step {number}."
|
||||
)
|
||||
try:
|
||||
image_bytes = Path(path).read_bytes()
|
||||
except OSError as e:
|
||||
return ScreenshotResult(number, which, f"Error reading screenshot: {e}")
|
||||
|
||||
if which != "overlay":
|
||||
return ScreenshotResult(
|
||||
number, which, f"Screenshot of step {number} ({which}-action) is attached.", image_bytes
|
||||
)
|
||||
|
||||
try:
|
||||
record = reader.get_step_record(number)
|
||||
except (sqlite3.Error, ValueError):
|
||||
record = None
|
||||
action = getattr(record, "action_taken", None) if record is not None else None
|
||||
try:
|
||||
annotated = overlay_action_on_screenshot(image_bytes, action)
|
||||
except (TypeError, ValueError, OSError) as e:
|
||||
logger.debug(f"Action overlay for step {number} failed: {e}")
|
||||
annotated = None
|
||||
if annotated is None:
|
||||
return ScreenshotResult(
|
||||
number,
|
||||
which,
|
||||
f"Screenshot of step {number} (pre-action; the step's action could not be"
|
||||
" drawn, plain screenshot) is attached.",
|
||||
image_bytes,
|
||||
)
|
||||
return ScreenshotResult(
|
||||
number,
|
||||
which,
|
||||
f"Screenshot of step {number} (pre-action with the step's action drawn on it) is attached.",
|
||||
annotated,
|
||||
overlay_drawn=True,
|
||||
)
|
||||
@@ -0,0 +1,396 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""search_history: deterministic keyword / step-range lookup over stored history.
|
||||
|
||||
One search surface over a :class:`~artemis.data_engine.history_reader.HistoryReader`'s
|
||||
raw records — step summaries, history chunks, exact actions and execution
|
||||
results, reasoning, the tool calls a step made (name, arguments, result
|
||||
text), notes, the OCR/UI-tree text of the step's pre and post screenshots
|
||||
(which also carries package/activity strings) plus any screenshot a tool
|
||||
result embedded, and the foreground-app stamp. Keyword + step-range filtering
|
||||
in Python; no vector database, no model call.
|
||||
|
||||
Hard boundaries (config ``agent.memory.recall``):
|
||||
|
||||
- at most ``max_results`` results (default 5);
|
||||
- one response is capped at ``max_text_tokens`` estimated tokens (char/4);
|
||||
- screen text is scanned for the most recent ``screen_scan_steps`` steps;
|
||||
- every result carries a step number / step id;
|
||||
- large raw sources return only excerpts around the match plus a reference.
|
||||
|
||||
A ``step_range`` additionally returns the full-width ``build_action_ledger``
|
||||
rows of that range — the re-entry point for compressed-history marker lines.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from typing import Any
|
||||
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
#: Character clamp for one matched excerpt.
|
||||
EXCERPT_CHARS = 240
|
||||
|
||||
#: Per-tool-result clamp on the *search* surface (scoring/excerpts only; the
|
||||
#: replay rendering has its own, looser clamp).
|
||||
RESULT_SEARCH_CHARS = 8000
|
||||
|
||||
DEFAULT_MAX_RESULTS = 5
|
||||
DEFAULT_MAX_TEXT_TOKENS = 2000
|
||||
DEFAULT_SCREEN_SCAN_STEPS = 150
|
||||
|
||||
|
||||
def _terms(query: str) -> list[str]:
|
||||
return [t for t in (query or "").lower().split() if t]
|
||||
|
||||
|
||||
def _estimate_tokens(text: str) -> int:
|
||||
return len(text) // 4
|
||||
|
||||
|
||||
def _clamp(text: str, limit: int) -> str:
|
||||
text = (text or "").strip()
|
||||
if len(text) <= limit:
|
||||
return text
|
||||
return text[: limit - 1].rstrip() + "…"
|
||||
|
||||
|
||||
def _excerpt(text: str, terms: list[str], limit: int) -> str:
|
||||
"""A window of ``text`` around the first term hit (flattened whitespace)."""
|
||||
flat = " ".join((text or "").split())
|
||||
lower = flat.lower()
|
||||
pos = -1
|
||||
for term in terms:
|
||||
pos = lower.find(term)
|
||||
if pos >= 0:
|
||||
break
|
||||
if pos < 0:
|
||||
return _clamp(flat, limit)
|
||||
start = max(0, pos - limit // 3)
|
||||
window = flat[start : start + limit]
|
||||
prefix = "…" if start > 0 else ""
|
||||
suffix = "…" if start + limit < len(flat) else ""
|
||||
return f"{prefix}{window.strip()}{suffix}"
|
||||
|
||||
|
||||
def _score(haystack: str, terms: list[str]) -> int:
|
||||
lower = haystack.lower()
|
||||
return sum(1 for term in terms if term in lower)
|
||||
|
||||
|
||||
def result_search_text(result: Any) -> str:
|
||||
"""Searchable text of a tool result. Content-block lists contribute their
|
||||
text blocks — image blocks have already been rewritten into
|
||||
``[screenshot: …]`` descriptions by the DataEngine; any raw image block
|
||||
that slipped through contributes nothing (its OCR/XML description joins
|
||||
the haystack via :func:`_screen_text`), so base64 never enters the search."""
|
||||
if result is None:
|
||||
return ""
|
||||
if isinstance(result, list):
|
||||
texts = [
|
||||
str(block.get("text") or "")
|
||||
for block in result
|
||||
if isinstance(block, dict) and block.get("type") == "text"
|
||||
]
|
||||
if texts or any(
|
||||
isinstance(block, dict) and block.get("type") in ("image_url", "image")
|
||||
for block in result
|
||||
):
|
||||
return "\n".join(texts)[:RESULT_SEARCH_CHARS]
|
||||
if isinstance(result, dict):
|
||||
return json.dumps(result, ensure_ascii=False, default=str)[:RESULT_SEARCH_CHARS]
|
||||
return str(result)[:RESULT_SEARCH_CHARS]
|
||||
|
||||
|
||||
def _events_text(step: dict) -> str:
|
||||
parts: list[str] = []
|
||||
for event in step.get("interleaved_events") or []:
|
||||
etype = event.get("type")
|
||||
if etype == "tool_call":
|
||||
parts.append(str(event.get("name") or ""))
|
||||
parts.append(json.dumps(event.get("args") or {}, ensure_ascii=False, default=str))
|
||||
result_text = result_search_text(event.get("result"))
|
||||
if result_text:
|
||||
parts.append(result_text)
|
||||
elif etype and "thought" in etype:
|
||||
parts.append(str(event.get("content") or ""))
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def _step_haystack(step: dict) -> str:
|
||||
extra = step.get("extra_metadata") or {}
|
||||
fields = [
|
||||
str(step.get("summary") or ""),
|
||||
json.dumps(step.get("action_taken"), ensure_ascii=False, default=str)
|
||||
if step.get("action_taken")
|
||||
else "",
|
||||
json.dumps(step.get("last_execution_result"), ensure_ascii=False, default=str)
|
||||
if step.get("last_execution_result")
|
||||
else "",
|
||||
str(step.get("operator_raw_thinking") or ""),
|
||||
_events_text(step),
|
||||
str(extra.get("injected_instruction") or ""),
|
||||
# The foreground app package stamped by record_step joins the search
|
||||
# surface, so package-name queries never depend on the screen-text
|
||||
# scan (capped to recent steps) alone.
|
||||
str(extra.get("foreground_app") or ""),
|
||||
]
|
||||
return "\n".join(f for f in fields if f)
|
||||
|
||||
|
||||
def _image_text(reader: Any, image_name: str | None) -> str:
|
||||
"""Raw XML/OCR text stored for one screenshot — the textual description
|
||||
of that image (package/activity strings live inside the XML blob)."""
|
||||
if not image_name:
|
||||
return ""
|
||||
try:
|
||||
record = reader.storage.get_image(image_name)
|
||||
except (sqlite3.Error, ValueError):
|
||||
return ""
|
||||
if record is None:
|
||||
return ""
|
||||
parts = []
|
||||
for blob in (record.ui_tree, record.ocr_result):
|
||||
if blob:
|
||||
parts.append(
|
||||
blob if isinstance(blob, str) else json.dumps(blob, ensure_ascii=False, default=str)
|
||||
)
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def referenced_image_names(step: dict) -> list[str]:
|
||||
"""Screenshots that tool results of this step embedded (rewritten by the
|
||||
DataEngine into ``[screenshot: …]`` text blocks carrying ``image_name``)."""
|
||||
names: list[str] = []
|
||||
|
||||
def walk(value: Any) -> None:
|
||||
if isinstance(value, dict):
|
||||
name = value.get("image_name")
|
||||
if value.get("type") == "text" and name and name not in names:
|
||||
names.append(str(name))
|
||||
for child in value.values():
|
||||
walk(child)
|
||||
elif isinstance(value, list):
|
||||
for child in value:
|
||||
walk(child)
|
||||
|
||||
for event in step.get("interleaved_events") or []:
|
||||
if event.get("type") == "tool_call":
|
||||
walk(event.get("result"))
|
||||
return names
|
||||
|
||||
|
||||
def _screen_text(reader: Any, step: dict) -> str:
|
||||
"""Description text of the screenshots behind a step: its own pre- and
|
||||
post-action screenshots plus any screenshot a tool result embedded."""
|
||||
names: list[str] = []
|
||||
for name in (
|
||||
step.get("pre_image_name"),
|
||||
step.get("post_image_name"),
|
||||
*referenced_image_names(step),
|
||||
):
|
||||
if name and name not in names:
|
||||
names.append(name)
|
||||
return "\n".join(t for t in (_image_text(reader, n) for n in names) if t)
|
||||
|
||||
|
||||
def _step_label(step: dict, session_start: float | None) -> str:
|
||||
from artemis.memory.chunking import step_offset_label
|
||||
|
||||
return f"Step {step.get('step_number')} ({step_offset_label(step, session_start)})"
|
||||
|
||||
|
||||
def _note_writer_steps(steps: list[dict], key: str) -> list[int]:
|
||||
writers: list[int] = []
|
||||
for step in steps:
|
||||
for event in step.get("interleaved_events") or []:
|
||||
if (
|
||||
event.get("type") == "tool_call"
|
||||
and event.get("name") in ("save_note", "update_note", "append_note")
|
||||
and (event.get("args") or {}).get("key") == key
|
||||
):
|
||||
number = step.get("step_number")
|
||||
if isinstance(number, int):
|
||||
writers.append(number)
|
||||
return writers
|
||||
|
||||
|
||||
def _parse_range(step_range: list[int] | None) -> tuple[int | None, int | None]:
|
||||
if not step_range:
|
||||
return None, None
|
||||
try:
|
||||
start, end = int(step_range[0]), int(step_range[-1])
|
||||
except (TypeError, ValueError, IndexError):
|
||||
return None, None
|
||||
if start > end:
|
||||
start, end = end, start
|
||||
return start, end
|
||||
|
||||
|
||||
# pylint: disable=too-many-branches,too-many-locals,too-many-statements
|
||||
def search_history_text(
|
||||
reader: Any,
|
||||
*,
|
||||
query: str = "",
|
||||
step_range: list[int] | None = None,
|
||||
max_results: int = DEFAULT_MAX_RESULTS,
|
||||
recall_config: Any = None,
|
||||
) -> str:
|
||||
"""Deterministic history search over ``reader``; always returns text."""
|
||||
cfg_max_results = int(getattr(recall_config, "max_results", None) or DEFAULT_MAX_RESULTS)
|
||||
cfg_max_tokens = int(getattr(recall_config, "max_text_tokens", None) or DEFAULT_MAX_TEXT_TOKENS)
|
||||
scan_steps = int(getattr(recall_config, "screen_scan_steps", None) or DEFAULT_SCREEN_SCAN_STEPS)
|
||||
max_results = max(1, min(int(max_results or cfg_max_results), cfg_max_results))
|
||||
char_budget = cfg_max_tokens * 4
|
||||
|
||||
try:
|
||||
steps: list[dict] = reader.get_agent_friendly_steps() or []
|
||||
except Exception as e:
|
||||
return f"search_history failed to load history: {e}"
|
||||
|
||||
range_start, range_end = _parse_range(step_range)
|
||||
|
||||
def in_range(number: Any) -> bool:
|
||||
if range_start is None:
|
||||
return True
|
||||
return isinstance(number, int) and range_start <= number <= range_end
|
||||
|
||||
scoped_steps = [s for s in steps if in_range(s.get("step_number"))]
|
||||
terms = _terms(query)
|
||||
session_start = getattr(reader, "session_start_time", None)
|
||||
|
||||
if not steps:
|
||||
return "search_history: no recorded steps yet."
|
||||
if step_range and not scoped_steps:
|
||||
return f"search_history: no recorded steps in range {range_start}–{range_end}."
|
||||
|
||||
sections: list[str] = []
|
||||
|
||||
# --- Range ledger: the explicit re-entry point for compressed-history
|
||||
# marker lines — full-width build_action_ledger rows.
|
||||
if range_start is not None and scoped_steps:
|
||||
from artemis.memory.chunking import build_action_ledger
|
||||
|
||||
ledger = build_action_ledger(scoped_steps, session_start)
|
||||
sections.append(
|
||||
f"Action ledger for Steps {range_start}–{range_end}"
|
||||
f" ({len(scoped_steps)} recorded steps):\n{ledger}"
|
||||
)
|
||||
|
||||
# --- Keyword search over steps.
|
||||
results: list[tuple[int, int, str]] = [] # (score, step_number, rendered)
|
||||
if terms:
|
||||
screen_eligible = {id(s) for s in scoped_steps[-scan_steps:]} if scan_steps > 0 else set()
|
||||
for step in scoped_steps:
|
||||
haystack = _step_haystack(step)
|
||||
screen = _screen_text(reader, step) if id(step) in screen_eligible else ""
|
||||
score = _score(haystack, terms)
|
||||
screen_score = _score(screen, terms) if screen else 0
|
||||
total = score + screen_score
|
||||
if total <= 0:
|
||||
continue
|
||||
source = haystack if score >= screen_score else screen
|
||||
lines = [f"[{_step_label(step, session_start)} | id {step.get('step_id')}]"]
|
||||
summary = step.get("summary")
|
||||
if summary:
|
||||
lines.append(f" Screen: {_clamp(str(summary), EXCERPT_CHARS)}")
|
||||
lines.append(f" Match: {_excerpt(source, terms, EXCERPT_CHARS)}")
|
||||
number = step.get("step_number") or 0
|
||||
results.append((total, number, "\n".join(lines)))
|
||||
|
||||
# --- History chunks (bands ①②③ / rendered text).
|
||||
try:
|
||||
chunk_rows = reader.get_history_chunks() or []
|
||||
except (sqlite3.Error, ValueError):
|
||||
chunk_rows = []
|
||||
for row in chunk_rows:
|
||||
start_n = getattr(row, "start_step_number", None)
|
||||
end_n = getattr(row, "end_step_number", None)
|
||||
if range_start is not None and not (
|
||||
isinstance(start_n, int)
|
||||
and isinstance(end_n, int)
|
||||
and start_n <= range_end
|
||||
and end_n >= range_start
|
||||
):
|
||||
continue
|
||||
text = getattr(row, "rendered_text", None) or "\n".join(
|
||||
filter(None, [getattr(row, "band2", None), getattr(row, "band3", None)])
|
||||
)
|
||||
score = _score(text or "", terms)
|
||||
if score <= 0:
|
||||
continue
|
||||
rendered = (
|
||||
f"[History chunk | Steps {start_n}–{end_n} | status"
|
||||
f" {getattr(row, 'status', '?')}]\n Match: {_excerpt(text, terms, EXCERPT_CHARS)}"
|
||||
)
|
||||
results.append((score, int(end_n or 0), rendered))
|
||||
|
||||
# --- Notes (filesystem; attributed to their writing steps).
|
||||
try:
|
||||
from artemis.utils.notes import list_notes_keys, read_note_content
|
||||
|
||||
base_dir = getattr(reader, "base_dir", None)
|
||||
note_keys = list_notes_keys(base_dir) if base_dir else []
|
||||
except OSError:
|
||||
note_keys = []
|
||||
for key in note_keys:
|
||||
try:
|
||||
content = read_note_content(base_dir, key)
|
||||
except (OSError, UnicodeError):
|
||||
continue
|
||||
score = _score(content, terms) + _score(key, terms)
|
||||
if score <= 0:
|
||||
continue
|
||||
writers = _note_writer_steps(steps, key)
|
||||
if (
|
||||
range_start is not None
|
||||
and writers
|
||||
and not any(range_start <= n <= range_end for n in writers)
|
||||
):
|
||||
continue
|
||||
anchor = writers[-1] if writers else (steps[-1].get("step_number") or 0)
|
||||
origin = f"last written at Step {writers[-1]}" if writers else f"as of Step {anchor}"
|
||||
results.append(
|
||||
(
|
||||
score,
|
||||
int(anchor),
|
||||
f"[Note '{key}' | {origin}]\n Match: {_excerpt(content, terms, EXCERPT_CHARS)}",
|
||||
)
|
||||
)
|
||||
|
||||
results.sort(key=lambda r: (-r[0], -r[1]))
|
||||
kept = results[:max_results]
|
||||
dropped = len(results) - len(kept)
|
||||
|
||||
if kept:
|
||||
header = f"Top {len(kept)} match(es) for {terms}:"
|
||||
if dropped > 0:
|
||||
header += f" ({dropped} more not shown; narrow the query or step_range)"
|
||||
sections.append(header + "\n\n" + "\n".join(r[2] for r in kept))
|
||||
elif terms:
|
||||
sections.append(f"No matches for {terms} in the searched history.")
|
||||
|
||||
text = "\n\n".join(sections)
|
||||
if _estimate_tokens(text) > cfg_max_tokens:
|
||||
text = (
|
||||
text[:char_budget].rstrip()
|
||||
+ "\n… (response truncated at the search token budget; narrow the"
|
||||
" query or step_range)"
|
||||
)
|
||||
return text
|
||||
@@ -1,590 +0,0 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""recall_history: deterministic lossless lookup into cold history (M4).
|
||||
|
||||
Implements the sister design's §10 as adopted by the history-module redesign
|
||||
§3.5: one unified tool over the DataEngine's raw records — step summaries,
|
||||
history chunks (bands ①②③), exact actions and validator results, operator
|
||||
thinking, notes, XML/OCR text (which also carries package/activity strings),
|
||||
Failure-Analyzer traces, and image references. Phase one is keyword + step
|
||||
range filtering over SQLite-backed records in Python — no vector database, no
|
||||
model call.
|
||||
|
||||
Hard boundaries (config ``agent.memory.recall``):
|
||||
|
||||
- at most ``max_results`` results (default 5);
|
||||
- one response is capped at ``max_text_tokens`` estimated tokens (char/4);
|
||||
- every result carries a step number / step id;
|
||||
- ``include_details`` output is clamped per field;
|
||||
- ``include_images`` returns only the screenshots that actually exist on disk
|
||||
for a single step (data URLs), never more than ``max_image_steps`` steps;
|
||||
- large raw sources return only excerpts around the match plus a reference.
|
||||
|
||||
An era whose ledger overflowed to a ``[Era N | Steps a–b: ledger via
|
||||
recall_history]`` marker line is re-entered here: recalling that step range
|
||||
returns the full-width ``build_action_ledger`` rows for it.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from artemis.core.tool_declaration import ToolDeclaration
|
||||
from artemis.data_engine.trace import trace_langchain_tool
|
||||
from artemis.tools.base import ArtemisTool
|
||||
from artemis.tools.tool_wrapper import ToolWrapper
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
#: How many most-recent steps get their screenshot XML/OCR blobs scanned
|
||||
#: (per-image JSON loads are the expensive part of the sweep).
|
||||
XML_SCAN_CAP = 150
|
||||
|
||||
#: Character clamp for one matched excerpt.
|
||||
EXCERPT_CHARS = 240
|
||||
DETAIL_EXCERPT_CHARS = 600
|
||||
|
||||
|
||||
class RecallHistoryArgs(BaseModel):
|
||||
"""Arguments schema for the recall_history tool."""
|
||||
|
||||
query: str = Field(
|
||||
...,
|
||||
description=(
|
||||
"Keywords to search for across the whole execution history"
|
||||
" (summaries, actions, validator results, reasoning, notes,"
|
||||
" on-screen XML/OCR text, package/activity names). Case-"
|
||||
"insensitive; every whitespace-separated term is matched"
|
||||
" independently. May be empty when step_range is given (returns"
|
||||
" the per-step action ledger of that range)."
|
||||
),
|
||||
)
|
||||
step_range: list[int] | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Optional [start, end] step-number range (inclusive) to restrict"
|
||||
" the search — e.g. the range shown in a '[... ledger via"
|
||||
" recall_history]' marker line. Also returns the full per-step"
|
||||
" action ledger of the range."
|
||||
),
|
||||
)
|
||||
include_details: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Include clamped per-result details (exact action, validator"
|
||||
" result, reasoning excerpt) instead of the one-line form."
|
||||
),
|
||||
)
|
||||
include_images: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Return the stored screenshots of the single best-matching step"
|
||||
" (only images that actually exist; at most one step)."
|
||||
),
|
||||
)
|
||||
max_results: int = Field(
|
||||
default=5,
|
||||
ge=1,
|
||||
description="Maximum results to return (server-side cap still applies).",
|
||||
)
|
||||
|
||||
|
||||
def _terms(query: str) -> list[str]:
|
||||
return [t for t in (query or "").lower().split() if t]
|
||||
|
||||
|
||||
def _estimate_tokens(text: str) -> int:
|
||||
return len(text) // 4
|
||||
|
||||
|
||||
def _clamp(text: str, limit: int) -> str:
|
||||
text = (text or "").strip()
|
||||
if len(text) <= limit:
|
||||
return text
|
||||
return text[: limit - 1].rstrip() + "…"
|
||||
|
||||
|
||||
def _excerpt(text: str, terms: list[str], limit: int) -> str:
|
||||
"""A window of ``text`` around the first term hit (flattened whitespace)."""
|
||||
flat = " ".join((text or "").split())
|
||||
lower = flat.lower()
|
||||
pos = -1
|
||||
for term in terms:
|
||||
pos = lower.find(term)
|
||||
if pos >= 0:
|
||||
break
|
||||
if pos < 0:
|
||||
return _clamp(flat, limit)
|
||||
start = max(0, pos - limit // 3)
|
||||
window = flat[start : start + limit]
|
||||
prefix = "…" if start > 0 else ""
|
||||
suffix = "…" if start + limit < len(flat) else ""
|
||||
return f"{prefix}{window.strip()}{suffix}"
|
||||
|
||||
|
||||
def _score(haystack: str, terms: list[str]) -> int:
|
||||
lower = haystack.lower()
|
||||
return sum(1 for term in terms if term in lower)
|
||||
|
||||
|
||||
def _events_text(step: dict) -> str:
|
||||
parts: list[str] = []
|
||||
for event in step.get("interleaved_events") or []:
|
||||
etype = event.get("type")
|
||||
if etype == "tool_call":
|
||||
parts.append(str(event.get("name") or ""))
|
||||
parts.append(json.dumps(event.get("args") or {}, ensure_ascii=False, default=str))
|
||||
result = event.get("result")
|
||||
if result is not None:
|
||||
parts.append(str(result)[:2000])
|
||||
elif etype and "thought" in etype:
|
||||
parts.append(str(event.get("content") or ""))
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def _step_haystack(step: dict) -> str:
|
||||
fields = [
|
||||
str(step.get("summary") or ""),
|
||||
json.dumps(step.get("action_taken"), ensure_ascii=False, default=str)
|
||||
if step.get("action_taken")
|
||||
else "",
|
||||
json.dumps(step.get("last_execution_result"), ensure_ascii=False, default=str)
|
||||
if step.get("last_execution_result")
|
||||
else "",
|
||||
str(step.get("operator_raw_thinking") or ""),
|
||||
_events_text(step),
|
||||
str((step.get("extra_metadata") or {}).get("injected_instruction") or ""),
|
||||
# M5: the foreground app package stamped by record_step joins the
|
||||
# search surface, so package-name queries no longer depend on the
|
||||
# XML-blob scan (capped to recent steps) alone.
|
||||
str((step.get("extra_metadata") or {}).get("foreground_app") or ""),
|
||||
]
|
||||
return "\n".join(f for f in fields if f)
|
||||
|
||||
|
||||
def _screen_text(engine: Any, step: dict) -> str:
|
||||
"""Raw XML/OCR text of a step's pre-action screenshot (package/activity
|
||||
strings live inside the XML blob)."""
|
||||
image_name = step.get("pre_image_name") or step.get("post_image_name")
|
||||
if not image_name:
|
||||
return ""
|
||||
try:
|
||||
record = engine.storage.get_image(image_name)
|
||||
except Exception:
|
||||
return ""
|
||||
if record is None:
|
||||
return ""
|
||||
parts = []
|
||||
for blob in (record.ui_tree, record.ocr_result):
|
||||
if blob:
|
||||
parts.append(
|
||||
blob if isinstance(blob, str) else json.dumps(blob, ensure_ascii=False, default=str)
|
||||
)
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def _session_start(engine: Any) -> float | None:
|
||||
return getattr(engine, "session_start_time", None)
|
||||
|
||||
|
||||
def _step_label(step: dict, session_start: float | None) -> str:
|
||||
from artemis.memory.chunking import step_offset_label
|
||||
|
||||
return f"Step {step.get('step_number')} ({step_offset_label(step, session_start)})"
|
||||
|
||||
|
||||
def _note_writer_steps(steps: list[dict], key: str) -> list[int]:
|
||||
writers: list[int] = []
|
||||
for step in steps:
|
||||
for event in step.get("interleaved_events") or []:
|
||||
if (
|
||||
event.get("type") == "tool_call"
|
||||
and event.get("name") in ("save_note", "update_note", "append_note")
|
||||
and (event.get("args") or {}).get("key") == key
|
||||
):
|
||||
number = step.get("step_number")
|
||||
if isinstance(number, int):
|
||||
writers.append(number)
|
||||
return writers
|
||||
|
||||
|
||||
def search_history(
|
||||
engine: Any,
|
||||
*,
|
||||
query: str,
|
||||
step_range: list[int] | None = None,
|
||||
include_details: bool = False,
|
||||
include_images: bool = False,
|
||||
max_results: int = 5,
|
||||
recall_config: Any = None,
|
||||
) -> str | list[dict]:
|
||||
"""Deterministic history search; returns text or content blocks (images)."""
|
||||
cfg_max_results = int(getattr(recall_config, "max_results", 5) or 5)
|
||||
cfg_max_tokens = int(getattr(recall_config, "max_text_tokens", 2000) or 2000)
|
||||
cfg_max_image_steps = int(getattr(recall_config, "max_image_steps", 1))
|
||||
|
||||
max_results = max(1, min(int(max_results or cfg_max_results), cfg_max_results))
|
||||
char_budget = cfg_max_tokens * 4
|
||||
|
||||
try:
|
||||
steps: list[dict] = engine.get_agent_friendly_steps() or []
|
||||
except Exception as e:
|
||||
return f"recall_history failed to load history: {e}"
|
||||
|
||||
range_start = range_end = None
|
||||
if step_range:
|
||||
try:
|
||||
range_start, range_end = int(step_range[0]), int(step_range[-1])
|
||||
if range_start > range_end:
|
||||
range_start, range_end = range_end, range_start
|
||||
except (TypeError, ValueError, IndexError):
|
||||
range_start = range_end = None
|
||||
|
||||
def in_range(number: Any) -> bool:
|
||||
if range_start is None:
|
||||
return True
|
||||
return isinstance(number, int) and range_start <= number <= range_end
|
||||
|
||||
scoped_steps = [s for s in steps if in_range(s.get("step_number"))]
|
||||
terms = _terms(query)
|
||||
session_start = _session_start(engine)
|
||||
|
||||
if not steps:
|
||||
return "recall_history: no recorded steps yet."
|
||||
if step_range and not scoped_steps:
|
||||
return f"recall_history: no recorded steps in range {range_start}–{range_end}."
|
||||
|
||||
sections: list[str] = []
|
||||
|
||||
# --- Range ledger: the explicit re-entry point for 'ledger via
|
||||
# recall_history' guidance lines and recall-only era period paragraphs —
|
||||
# full-width build_action_ledger rows.
|
||||
if range_start is not None and scoped_steps:
|
||||
from artemis.memory.chunking import build_action_ledger
|
||||
|
||||
ledger = build_action_ledger(scoped_steps, session_start)
|
||||
sections.append(
|
||||
f"Action ledger for Steps {range_start}–{range_end}"
|
||||
f" ({len(scoped_steps)} recorded steps):\n{ledger}"
|
||||
)
|
||||
|
||||
# --- Keyword search over steps.
|
||||
results: list[tuple[int, int, str]] = [] # (score, step_number, rendered)
|
||||
if terms:
|
||||
xml_eligible = {id(s) for s in scoped_steps[-XML_SCAN_CAP:]}
|
||||
for step in scoped_steps:
|
||||
haystack = _step_haystack(step)
|
||||
screen = _screen_text(engine, step) if id(step) in xml_eligible else ""
|
||||
score = _score(haystack, terms)
|
||||
screen_score = _score(screen, terms) if screen else 0
|
||||
total = score + screen_score
|
||||
if total <= 0:
|
||||
continue
|
||||
excerpt_limit = DETAIL_EXCERPT_CHARS if include_details else EXCERPT_CHARS
|
||||
source = haystack if score >= screen_score else screen
|
||||
body = _excerpt(source, terms, excerpt_limit)
|
||||
lines = [f"[{_step_label(step, session_start)} | id {step.get('step_id')}]"]
|
||||
summary = step.get("summary")
|
||||
if summary:
|
||||
lines.append(f" Summary: {_clamp(str(summary), EXCERPT_CHARS)}")
|
||||
lines.append(f" Match: {body}")
|
||||
if include_details:
|
||||
action = step.get("action_taken")
|
||||
if action:
|
||||
lines.append(
|
||||
" Action: "
|
||||
+ _clamp(
|
||||
json.dumps(action, ensure_ascii=False, default=str),
|
||||
DETAIL_EXCERPT_CHARS,
|
||||
)
|
||||
)
|
||||
result = step.get("last_execution_result")
|
||||
if result:
|
||||
lines.append(
|
||||
" Result: "
|
||||
+ _clamp(
|
||||
json.dumps(result, ensure_ascii=False, default=str),
|
||||
DETAIL_EXCERPT_CHARS,
|
||||
)
|
||||
)
|
||||
thinking = step.get("operator_raw_thinking")
|
||||
if thinking:
|
||||
lines.append(f" Reasoning: {_clamp(str(thinking), DETAIL_EXCERPT_CHARS)}")
|
||||
number = step.get("step_number") or 0
|
||||
results.append((total, number, "\n".join(lines)))
|
||||
|
||||
# --- History chunks (bands ①②③ / rendered text).
|
||||
try:
|
||||
chunk_rows = engine.get_history_chunks() or []
|
||||
except Exception:
|
||||
chunk_rows = []
|
||||
for row in chunk_rows:
|
||||
start_n = getattr(row, "start_step_number", None)
|
||||
end_n = getattr(row, "end_step_number", None)
|
||||
if range_start is not None and not (
|
||||
isinstance(start_n, int)
|
||||
and isinstance(end_n, int)
|
||||
and start_n <= range_end
|
||||
and end_n >= range_start
|
||||
):
|
||||
continue
|
||||
text = getattr(row, "rendered_text", None) or "\n".join(
|
||||
filter(None, [getattr(row, "band2", None), getattr(row, "band3", None)])
|
||||
)
|
||||
score = _score(text or "", terms)
|
||||
if score <= 0:
|
||||
continue
|
||||
body = _excerpt(text, terms, DETAIL_EXCERPT_CHARS if include_details else EXCERPT_CHARS)
|
||||
rendered = (
|
||||
f"[History chunk | Steps {start_n}–{end_n} | status"
|
||||
f" {getattr(row, 'status', '?')}]\n Match: {body}"
|
||||
)
|
||||
results.append((score, int(end_n or 0), rendered))
|
||||
|
||||
# --- Notes (filesystem; attributed to their writing steps).
|
||||
try:
|
||||
from artemis.utils.notes import list_notes_keys, read_note_content
|
||||
|
||||
base_dir = getattr(engine, "base_dir", None)
|
||||
note_keys = list_notes_keys(base_dir) if base_dir else []
|
||||
except Exception:
|
||||
note_keys = []
|
||||
for key in note_keys:
|
||||
try:
|
||||
content = read_note_content(base_dir, key)
|
||||
except Exception:
|
||||
continue
|
||||
score = _score(content, terms) + _score(key, terms)
|
||||
if score <= 0:
|
||||
continue
|
||||
writers = _note_writer_steps(steps, key)
|
||||
if (
|
||||
range_start is not None
|
||||
and writers
|
||||
and not any(range_start <= n <= range_end for n in writers)
|
||||
):
|
||||
continue
|
||||
anchor = writers[-1] if writers else (steps[-1].get("step_number") or 0)
|
||||
origin = f"last written at Step {writers[-1]}" if writers else f"as of Step {anchor}"
|
||||
body = _excerpt(
|
||||
content, terms, DETAIL_EXCERPT_CHARS if include_details else EXCERPT_CHARS
|
||||
)
|
||||
results.append((score, int(anchor), f"[Note '{key}' | {origin}]\n Match: {body}"))
|
||||
|
||||
results.sort(key=lambda r: (-r[0], -r[1]))
|
||||
kept = results[:max_results]
|
||||
dropped = len(results) - len(kept)
|
||||
|
||||
if kept:
|
||||
header = f"Top {len(kept)} match(es) for {terms}:"
|
||||
if dropped > 0:
|
||||
header += f" ({dropped} more not shown; narrow the query or step_range)"
|
||||
sections.append(header + "\n\n" + "\n".join(r[2] for r in kept))
|
||||
elif terms:
|
||||
sections.append(f"No matches for {terms} in the searched history.")
|
||||
|
||||
text = "\n\n".join(sections)
|
||||
if _estimate_tokens(text) > cfg_max_tokens:
|
||||
text = (
|
||||
text[:char_budget].rstrip()
|
||||
+ "\n… (response truncated at the recall token budget; narrow the"
|
||||
" query or step_range)"
|
||||
)
|
||||
|
||||
if not include_images or cfg_max_image_steps < 1:
|
||||
return text
|
||||
|
||||
# --- Image recall: real stored screenshots of ONE step only.
|
||||
target_step: dict | None = None
|
||||
if kept:
|
||||
# The best step-shaped result; fall back to the newest scoped step.
|
||||
best_numbers = [r[1] for r in kept]
|
||||
for number in best_numbers:
|
||||
target_step = next((s for s in scoped_steps if s.get("step_number") == number), None)
|
||||
if target_step:
|
||||
break
|
||||
if target_step is None and scoped_steps:
|
||||
target_step = scoped_steps[-1]
|
||||
if target_step is None:
|
||||
return text
|
||||
|
||||
blocks: list[dict] = [{"type": "text", "text": text}]
|
||||
added = 0
|
||||
for which, name in (
|
||||
("pre-action", target_step.get("pre_image_name")),
|
||||
("post-action", target_step.get("post_image_name")),
|
||||
):
|
||||
if not name:
|
||||
continue
|
||||
try:
|
||||
path = engine.get_image_path(name)
|
||||
if not path or not path.exists():
|
||||
continue
|
||||
encoded = base64.b64encode(path.read_bytes()).decode("utf-8")
|
||||
except Exception:
|
||||
continue
|
||||
blocks.append(
|
||||
{
|
||||
"type": "text",
|
||||
"text": (f"--- {which} screenshot of Step {target_step.get('step_number')} ---"),
|
||||
}
|
||||
)
|
||||
blocks.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/jpeg;base64,{encoded}"},
|
||||
}
|
||||
)
|
||||
added += 1
|
||||
if added == 0:
|
||||
blocks.append(
|
||||
{
|
||||
"type": "text",
|
||||
"text": (
|
||||
f"(no stored screenshots exist for Step {target_step.get('step_number')})"
|
||||
),
|
||||
}
|
||||
)
|
||||
return blocks
|
||||
|
||||
|
||||
RECALL_HISTORY_DOCSTRING = (
|
||||
"[SHELL] Deterministic lookup into older execution history that has been"
|
||||
" compressed out of the visible context. Searches step summaries, exact"
|
||||
" actions and validator results, reasoning, notes, on-screen XML/OCR text"
|
||||
" and history-chunk ledgers by keywords and/or a step-number range;"
|
||||
" a step_range also returns that range's full per-step action ledger"
|
||||
" (the entry point for 'ledger via recall_history' guidance/marker lines"
|
||||
" and for recall-only '[Era ...]' period paragraphs)."
|
||||
" Results are bounded and every result carries its step number.\n"
|
||||
"Call it when: the current screen closely resembles a much older state;"
|
||||
" you need an exact old value (text, id, amount, path); two consecutive"
|
||||
" steps made no progress; a history summary conflicts with what you"
|
||||
" observe; or the user asks to return to an earlier state. A"
|
||||
" '(Step-level ledger via recall_history for steps a–b)' line under an"
|
||||
" '[Era ...]' period paragraph is an explicit entry point — query that"
|
||||
" step range for the full per-step ledger. Do not call it speculatively"
|
||||
" on ordinary steps: the visible history is normally sufficient."
|
||||
)
|
||||
|
||||
|
||||
class RecallHistoryTool(ArtemisTool):
|
||||
"""Unified cold-history recall over the DataEngine (design §10)."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__(
|
||||
name="recall_history",
|
||||
description=RECALL_HISTORY_DOCSTRING,
|
||||
args_schema=RecallHistoryArgs,
|
||||
category="memory",
|
||||
)
|
||||
|
||||
def is_available(self, ctx: Any = None) -> bool:
|
||||
if ctx is None or getattr(ctx, "data_engine", None) is None:
|
||||
return False
|
||||
return bool(getattr(_recall_config(warn=False), "enabled", True))
|
||||
|
||||
# pylint: disable=too-many-arguments
|
||||
async def execute(
|
||||
self,
|
||||
driver: Any = None, # pylint: disable=unused-argument
|
||||
ctx: Any = None,
|
||||
query: str = "",
|
||||
step_range: list[int] | None = None,
|
||||
include_details: bool = False,
|
||||
include_images: bool = False,
|
||||
max_results: int = 5,
|
||||
**kwargs: Any,
|
||||
) -> Any:
|
||||
engine = getattr(ctx, "data_engine", None) if ctx else None
|
||||
if engine is None:
|
||||
return "recall_history unavailable: no active DataEngine session."
|
||||
if not query and not step_range:
|
||||
return (
|
||||
"recall_history needs a query and/or a step_range — e.g."
|
||||
' recall_history(query="login timeout") or'
|
||||
' recall_history(query="", step_range=[1, 40]).'
|
||||
)
|
||||
try:
|
||||
return search_history(
|
||||
engine,
|
||||
query=query or "",
|
||||
step_range=step_range,
|
||||
include_details=bool(include_details),
|
||||
include_images=bool(include_images),
|
||||
max_results=int(max_results or 5),
|
||||
recall_config=_recall_config(),
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"recall_history failed: {e}")
|
||||
return f"recall_history failed: {e}"
|
||||
|
||||
|
||||
def _recall_config(warn: bool = True) -> Any:
|
||||
try:
|
||||
from artemis.config import load_agent_config
|
||||
|
||||
return load_agent_config().memory.recall
|
||||
except Exception as e:
|
||||
if warn:
|
||||
logger.debug(f"recall config unavailable, using defaults: {e}")
|
||||
return None
|
||||
|
||||
|
||||
recall_history = RecallHistoryTool()
|
||||
|
||||
_DECLARATION_TYPES = {
|
||||
"query": {"type": "string"},
|
||||
"step_range": {"type": "array", "items": {"type": "integer"}},
|
||||
"include_details": {"type": "boolean"},
|
||||
"include_images": {"type": "boolean"},
|
||||
"max_results": {"type": "integer"},
|
||||
}
|
||||
|
||||
#: Flash-profile declaration of the same tool (FlashRunner binds JSON-schema
|
||||
#: declarations rather than LangChain tools). Descriptions come from
|
||||
#: ``RecallHistoryArgs`` so the two contracts never drift.
|
||||
RECALL_HISTORY_TOOL = ToolDeclaration(
|
||||
name="recall_history",
|
||||
description=RECALL_HISTORY_DOCSTRING,
|
||||
parameters={
|
||||
"type": "object",
|
||||
"properties": {
|
||||
name: {**_DECLARATION_TYPES[name], "description": info.description or ""}
|
||||
for name, info in RecallHistoryArgs.model_fields.items()
|
||||
},
|
||||
"required": ["query"],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def get_recall_history_tool(ctx: Any):
|
||||
"""Exports recall_history as a traced LangChain BaseTool."""
|
||||
return trace_langchain_tool(recall_history.to_langchain_tool(ctx), ctx)
|
||||
|
||||
|
||||
def _recall_available(ctx: Any) -> bool:
|
||||
return recall_history.is_available(ctx)
|
||||
|
||||
|
||||
recall_history_wrapper = ToolWrapper(
|
||||
tool_fn_getter=get_recall_history_tool,
|
||||
on_success_fn=lambda *a, **k: "Recalled history",
|
||||
on_failure_fn=lambda err: f"recall_history failed: {err}",
|
||||
is_available_fn=_recall_available,
|
||||
)
|
||||
@@ -15,11 +15,12 @@
|
||||
from collections.abc import Callable
|
||||
import functools
|
||||
import inspect
|
||||
import re
|
||||
import time
|
||||
from typing import Annotated, Any, get_args, get_origin
|
||||
from typing import Annotated, Any, Literal, get_args, get_origin
|
||||
import uuid
|
||||
|
||||
from langchain_core.messages import ToolMessage
|
||||
from langchain_core.messages import BaseMessage, HumanMessage, ToolMessage
|
||||
from langchain_core.tools import BaseTool
|
||||
from langchain_core.tools.base import InjectedToolCallId
|
||||
from langgraph.prebuilt import InjectedState
|
||||
@@ -168,3 +169,104 @@ def get_tool_result_content(result: Any) -> Any:
|
||||
if isinstance(result, ToolMessage):
|
||||
return result.content
|
||||
return result
|
||||
|
||||
|
||||
# --- Multimodal tool results ------------------------------------------------------------
|
||||
|
||||
ImageCarrier = Literal["tool", "human"]
|
||||
|
||||
#: Providers whose tool-result messages may carry image parts directly.
|
||||
_TOOL_IMAGE_PROVIDERS = frozenset({"google", "anthropic"})
|
||||
|
||||
_STEP_RE = re.compile(r"\bstep\s+(\d+)", re.IGNORECASE)
|
||||
|
||||
|
||||
def split_multimodal_result(result: Any) -> tuple[str, list[dict[str, Any]]]:
|
||||
"""Splits a tool result into its text and its image content blocks.
|
||||
|
||||
Plain strings have no images; content-block lists contribute their text
|
||||
blocks (joined by newlines) and their ``image_url`` / ``image`` blocks.
|
||||
``ToolMessage`` results are unwrapped first.
|
||||
"""
|
||||
content = get_tool_result_content(result)
|
||||
if content is None:
|
||||
return "", []
|
||||
if isinstance(content, list):
|
||||
texts: list[str] = []
|
||||
images: list[dict[str, Any]] = []
|
||||
for block in content:
|
||||
if isinstance(block, dict) and block.get("type") in ("image_url", "image"):
|
||||
images.append(block)
|
||||
elif isinstance(block, dict) and block.get("type") == "text":
|
||||
texts.append(str(block.get("text") or ""))
|
||||
else:
|
||||
texts.append(str(block))
|
||||
return "\n".join(t for t in texts if t), images
|
||||
return content if isinstance(content, str) else str(content), []
|
||||
|
||||
|
||||
def resolve_image_carrier(llm_or_provider: Any) -> ImageCarrier:
|
||||
"""Which message carries images returned by a tool for this LLM.
|
||||
|
||||
Gemini (direct API) and Anthropic accept image parts inside the tool
|
||||
result message itself (``"tool"``). Vertex AI and the OpenAI-compatible
|
||||
providers (openai / openrouter / xai / ...) reject them there, so the
|
||||
image travels in a ``HumanMessage`` that immediately follows the textual
|
||||
tool result (``"human"``). Accepts a provider name, a provider enum, or an
|
||||
LLM whose ``endpoint.provider`` names the provider that actually issues
|
||||
the request (a fallback model is judged by its own provider).
|
||||
"""
|
||||
provider: Any = llm_or_provider
|
||||
if provider is not None and not isinstance(provider, str):
|
||||
# An LLM (RobustChatModelWrapper) names its provider on ``endpoint``;
|
||||
# anything without an endpoint is taken as the provider itself.
|
||||
endpoint = getattr(provider, "endpoint", None)
|
||||
if endpoint is not None:
|
||||
provider = getattr(endpoint, "provider", None)
|
||||
value = str(getattr(provider, "value", provider) or "").lower()
|
||||
return "tool" if value in _TOOL_IMAGE_PROVIDERS else "human"
|
||||
|
||||
|
||||
def tool_result_messages(
|
||||
tool_call_id: str,
|
||||
result: Any,
|
||||
*,
|
||||
name: str | None = None,
|
||||
status: str = "success",
|
||||
image_carrier: ImageCarrier | None = None,
|
||||
llm: Any = None,
|
||||
) -> list[BaseMessage]:
|
||||
"""The conversation messages that deliver one tool result to the model.
|
||||
|
||||
Text-only results become a single ``ToolMessage``. Results carrying images
|
||||
(e.g. ``get_step_screenshot``) are delivered according to the image
|
||||
carrier — given explicitly or resolved from ``llm`` — so every agent loop
|
||||
handles multimodal tool results the same way and no loop ever pastes
|
||||
base64 into a text field. The tool call id, name and status are preserved.
|
||||
"""
|
||||
text, images = split_multimodal_result(result)
|
||||
extra = {"name": name} if name else {}
|
||||
if not images:
|
||||
content = get_tool_result_content(result)
|
||||
if not isinstance(content, (str, list)):
|
||||
content = text
|
||||
return [ToolMessage(tool_call_id=tool_call_id, content=content, status=status, **extra)]
|
||||
|
||||
carrier = image_carrier or resolve_image_carrier(llm)
|
||||
if carrier == "tool":
|
||||
blocks: list[dict[str, Any]] = [{"type": "text", "text": text}] if text else []
|
||||
return [
|
||||
ToolMessage(
|
||||
tool_call_id=tool_call_id, content=[*blocks, *images], status=status, **extra
|
||||
)
|
||||
]
|
||||
|
||||
match = _STEP_RE.search(text)
|
||||
caption = f"[Screenshot returned by {name or 'the tool'}"
|
||||
if match:
|
||||
caption += f" for step {match.group(1)}"
|
||||
caption += "]"
|
||||
return [
|
||||
ToolMessage(tool_call_id=tool_call_id, content=text, status=status, **extra),
|
||||
HumanMessage(content=[{"type": "text", "text": caption}, *images]),
|
||||
]
|
||||
@@ -328,39 +328,69 @@ def denormalize_point(nx: int, ny: int, width: int, height: int) -> list[int]:
|
||||
return [max(0, min(width - 1, x)), max(0, min(height - 1, y))]
|
||||
|
||||
|
||||
#: Pro records physical pixels; Flash records normalized 0–1000 coordinates.
|
||||
#: The marker prevents normalization from being applied twice during rendering.
|
||||
COORDINATE_SPACE_KEY = "coordinate_space"
|
||||
COORDINATE_SPACE_PIXEL = "pixel"
|
||||
COORDINATE_SPACE_NORMALIZED = "normalized"
|
||||
|
||||
_COORDINATE_FIELDS = ("coordinates", "start_coordinates", "end_coordinates", "target")
|
||||
|
||||
|
||||
def normalize_action_dict(action: dict[str, Any], width: int, height: int) -> dict[str, Any]:
|
||||
"""Recursively search and normalize coordinate fields inside an action dictionary."""
|
||||
"""Normalize the coordinate fields of one recorded action dict (idempotent).
|
||||
|
||||
A dict already stamped ``coordinate_space="normalized"`` is returned as a
|
||||
copy without any conversion; a pixel-space (or legacy unstamped) dict has
|
||||
its ``coordinates`` / ``start_coordinates`` / ``end_coordinates`` /
|
||||
``target`` fields converted once and comes back stamped ``normalized``, so
|
||||
a second pass over the same output is a no-op.
|
||||
"""
|
||||
if not isinstance(action, dict):
|
||||
return action
|
||||
|
||||
action_copy = copy.deepcopy(action)
|
||||
if action_copy.get(COORDINATE_SPACE_KEY) == COORDINATE_SPACE_NORMALIZED:
|
||||
return action_copy
|
||||
|
||||
converted = False
|
||||
# Normalize 'coordinates' if present and valid
|
||||
coords = action_copy.get("coordinates")
|
||||
if coords and isinstance(coords, list):
|
||||
if len(coords) == 2:
|
||||
action_copy["coordinates"] = normalize_point(coords[0], coords[1], width, height)
|
||||
converted = True
|
||||
elif len(coords) == 4:
|
||||
p1 = normalize_point(coords[0], coords[1], width, height)
|
||||
p2 = normalize_point(coords[2], coords[3], width, height)
|
||||
action_copy["coordinates"] = p1 + p2
|
||||
converted = True
|
||||
|
||||
# Normalize 'start_coordinates' and 'end_coordinates' for generic formats
|
||||
for key in ["start_coordinates", "end_coordinates", "target"]:
|
||||
for key in _COORDINATE_FIELDS[1:]:
|
||||
val = action_copy.get(key)
|
||||
if val and isinstance(val, list) and len(val) == 2:
|
||||
action_copy[key] = normalize_point(val[0], val[1], width, height)
|
||||
converted = True
|
||||
|
||||
if converted or COORDINATE_SPACE_KEY in action_copy:
|
||||
action_copy[COORDINATE_SPACE_KEY] = COORDINATE_SPACE_NORMALIZED
|
||||
return action_copy
|
||||
|
||||
|
||||
def _is_action_item(obj: dict[str, Any]) -> bool:
|
||||
"""An action item names its action; a container that merely holds an
|
||||
``action`` dict (an execution incident) is recursed into instead."""
|
||||
return isinstance(obj.get("action"), str) or isinstance(obj.get("name"), str)
|
||||
|
||||
|
||||
def normalize_any_structure(obj: Any, width: int, height: int) -> Any:
|
||||
"""Recursively searches lists and dicts to normalize any action coordinate schemas."""
|
||||
if isinstance(obj, list):
|
||||
return [normalize_any_structure(item, width, height) for item in obj]
|
||||
elif isinstance(obj, dict):
|
||||
# If it looks like a single action item
|
||||
if "action" in obj or "name" in obj:
|
||||
if _is_action_item(obj):
|
||||
return normalize_action_dict(obj, width, height)
|
||||
# Otherwise recurse down keys
|
||||
return {k: normalize_any_structure(v, width, height) for k, v in obj.items()}
|
||||
|
||||
@@ -16,70 +16,162 @@ import ast
|
||||
import logging
|
||||
from pathlib import Path
|
||||
import queue
|
||||
import subprocess
|
||||
import time
|
||||
from typing import IO
|
||||
|
||||
import jupyter_client
|
||||
from jupyter_client.blocking import BlockingKernelClient
|
||||
from zmq import ZMQError
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Allow extra startup time when the host is busy.
|
||||
KERNEL_STARTUP_TIMEOUT_S = 60.0
|
||||
EXECUTION_TIMEOUT_S = 120.0
|
||||
# Poll frequently enough to detect a dead kernel or an expired deadline.
|
||||
IOPUB_POLL_INTERVAL_S = 1.0
|
||||
INTERRUPT_DRAIN_TIMEOUT_S = 5.0
|
||||
|
||||
def validate_coder_script(code_str: str):
|
||||
"""Validates that the generated python code does not violate constraints.
|
||||
|
||||
Specifically checks for usage of cv2.resize and numpy slicing which could
|
||||
mess up the transformation tracking.
|
||||
"""
|
||||
def validate_coder_script(code_str: str) -> bool:
|
||||
"""Reject resize() calls that bypass canvas transformation tracking."""
|
||||
try:
|
||||
tree = ast.parse(code_str)
|
||||
except SyntaxError as e:
|
||||
raise ValueError(f"Syntax error in code: {e}")
|
||||
raise ValueError(f"Syntax error in code: {e}") from e
|
||||
|
||||
for node in ast.walk(tree):
|
||||
# Prevent cv2.resize
|
||||
if isinstance(node, ast.Attribute):
|
||||
if node.attr == "resize":
|
||||
# Quick check if it might be cv2.resize
|
||||
if isinstance(node, ast.Attribute) and node.attr == "resize":
|
||||
raise ValueError(
|
||||
"SECURITY/VALIDATION ERROR: Direct usage of resize() is"
|
||||
" forbidden. You MUST use canvas.resize_by_factor(factor)."
|
||||
)
|
||||
|
||||
# Prevent slicing (very aggressive, but forces them to use canvas.crop)
|
||||
# Note: We cannot easily block all slicing because they might slice other arrays.
|
||||
# But we can look for specific variable names or just warn.
|
||||
# Let's be lenient on slicing but strict in the prompt.
|
||||
|
||||
return True
|
||||
|
||||
|
||||
class PythonExecutor:
|
||||
def __init__(self, session_dir: Path | None = None):
|
||||
self.session_dir = session_dir
|
||||
"""Run snippets in a dedicated Jupyter kernel with execution deadlines.
|
||||
|
||||
Kernel output goes to a session log or DEVNULL so a lingering kernel
|
||||
cannot keep the parent's output pipes open.
|
||||
"""
|
||||
|
||||
def __init__(self, session_dir: Path | None = None) -> None:
|
||||
self.session_dir = Path(session_dir) if session_dir is not None else None
|
||||
self.km: jupyter_client.KernelManager | None = None
|
||||
self.kc: BlockingKernelClient | None = None
|
||||
self._kernel_log: IO[bytes] | None = None
|
||||
self._start()
|
||||
|
||||
def _start(self) -> None:
|
||||
try:
|
||||
if self.session_dir is not None:
|
||||
self.session_dir.mkdir(parents=True, exist_ok=True)
|
||||
self._kernel_log = (self.session_dir / "kernel.log").open("ab")
|
||||
kernel_output = self._kernel_log if self._kernel_log is not None else subprocess.DEVNULL
|
||||
|
||||
self.km = jupyter_client.KernelManager(kernel_name="python3")
|
||||
self.km.start_kernel()
|
||||
self.km.start_kernel(stdout=kernel_output, stderr=kernel_output)
|
||||
self.kc = self.km.client()
|
||||
self.kc.start_channels()
|
||||
self.kc.wait_for_ready(timeout=10)
|
||||
self.kc.wait_for_ready(timeout=KERNEL_STARTUP_TIMEOUT_S)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start Jupyter kernel: {e}")
|
||||
raise RuntimeError(f"Kernel startup failed: {e}")
|
||||
logger.error("Failed to start Jupyter kernel: %s", e)
|
||||
# Startup may fail after creating the process or opening the log.
|
||||
self.close()
|
||||
raise RuntimeError(f"Kernel startup failed: {e}") from e
|
||||
|
||||
def _restart(self) -> None:
|
||||
"""Replace an unresponsive kernel so the next cell can execute.
|
||||
|
||||
Image intermediates are stored on disk and survive the restart.
|
||||
"""
|
||||
logger.warning("Restarting Python kernel after an execution timeout")
|
||||
self.close()
|
||||
self._start()
|
||||
|
||||
def _kernel_alive(self) -> bool:
|
||||
if self.km is None or self.kc is None:
|
||||
return False
|
||||
try:
|
||||
return bool(self.km.is_alive())
|
||||
except (OSError, RuntimeError) as e:
|
||||
logger.error("Failed to query kernel liveness: %s", e)
|
||||
return False
|
||||
|
||||
def _interrupt_and_drain(self, msg_id: str) -> bool:
|
||||
"""Return whether the interrupted request becomes idle before the deadline.
|
||||
|
||||
Restart the kernel if it does not acknowledge the interrupt.
|
||||
"""
|
||||
assert self.km is not None and self.kc is not None
|
||||
try:
|
||||
self.km.interrupt_kernel()
|
||||
except (OSError, RuntimeError, ZMQError) as e:
|
||||
logger.error("Failed to interrupt kernel after timeout: %s", e)
|
||||
return False
|
||||
deadline = time.monotonic() + INTERRUPT_DRAIN_TIMEOUT_S
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
msg = self.kc.get_iopub_msg(timeout=IOPUB_POLL_INTERVAL_S)
|
||||
except queue.Empty:
|
||||
if not self._kernel_alive():
|
||||
return False
|
||||
continue
|
||||
if (
|
||||
msg["parent_header"].get("msg_id") == msg_id
|
||||
and msg["header"]["msg_type"] == "status"
|
||||
and msg["content"].get("execution_state") == "idle"
|
||||
):
|
||||
return True
|
||||
return False
|
||||
|
||||
def execute(self, code: str) -> str:
|
||||
# First validate code
|
||||
try:
|
||||
validate_coder_script(code)
|
||||
except ValueError as ve:
|
||||
return str(ve)
|
||||
|
||||
start_time = time.time()
|
||||
self.kc.execute(code)
|
||||
if not self._kernel_alive():
|
||||
return "[Kernel Error] The Python kernel is not running."
|
||||
assert self.kc is not None
|
||||
|
||||
msg_id = self.kc.execute(code)
|
||||
deadline = time.monotonic() + EXECUTION_TIMEOUT_S
|
||||
output_text = []
|
||||
|
||||
while True:
|
||||
# Check before reading: a cell may produce output continuously.
|
||||
if time.monotonic() >= deadline:
|
||||
output_text.append(
|
||||
"[Execution Timeout Error] Execution exceeded"
|
||||
f" {EXECUTION_TIMEOUT_S:.0f}s and was stopped."
|
||||
)
|
||||
if not self._interrupt_and_drain(msg_id):
|
||||
try:
|
||||
msg = self.kc.get_iopub_msg(timeout=15)
|
||||
msg_type = msg["header"]["msg_type"]
|
||||
self._restart()
|
||||
except RuntimeError as e:
|
||||
output_text.append(f"\n[Kernel Error] {e}")
|
||||
break
|
||||
|
||||
try:
|
||||
msg = self.kc.get_iopub_msg(timeout=IOPUB_POLL_INTERVAL_S)
|
||||
except queue.Empty:
|
||||
if not self._kernel_alive():
|
||||
output_text.append(
|
||||
"[Kernel Error] The Python kernel died while executing the code."
|
||||
)
|
||||
break
|
||||
continue
|
||||
|
||||
# Only consume messages that belong to this request; anything else
|
||||
# is a leftover from kernel startup or an earlier interrupted cell.
|
||||
if msg["parent_header"].get("msg_id") != msg_id:
|
||||
continue
|
||||
|
||||
msg_type = msg["header"]["msg_type"]
|
||||
if msg_type == "stream":
|
||||
output_text.append(msg["content"]["text"])
|
||||
elif msg_type == "error":
|
||||
@@ -91,15 +183,31 @@ class PythonExecutor:
|
||||
output_text.append(data["text/plain"])
|
||||
elif msg_type == "status" and msg["content"]["execution_state"] == "idle":
|
||||
break
|
||||
except queue.Empty:
|
||||
output_text.append("[Execution Timeout Error]")
|
||||
break
|
||||
|
||||
return "".join(output_text)
|
||||
|
||||
def close(self):
|
||||
def close(self) -> None:
|
||||
kc, km, kernel_log = self.kc, self.km, self._kernel_log
|
||||
self.kc = None
|
||||
self.km = None
|
||||
self._kernel_log = None
|
||||
|
||||
if kc is not None:
|
||||
try:
|
||||
self.kc.stop_channels()
|
||||
self.km.shutdown_kernel()
|
||||
kc.stop_channels()
|
||||
except (OSError, RuntimeError, ZMQError) as e:
|
||||
logger.error("Failed to stop kernel channels: %s", e)
|
||||
|
||||
if km is not None:
|
||||
try:
|
||||
if km.has_kernel:
|
||||
# Do not wait for a busy cell to finish before shutting down.
|
||||
km.shutdown_kernel(now=True)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to cleanly shutdown kernel: {e}")
|
||||
logger.error("Failed to shut down kernel: %s", e)
|
||||
|
||||
if kernel_log is not None:
|
||||
try:
|
||||
kernel_log.close()
|
||||
except OSError as e:
|
||||
logger.debug("Failed to close kernel log: %s", e)
|
||||
+106
-12
@@ -39,6 +39,16 @@ def format_action_clean(action_obj) -> str:
|
||||
if not act_type:
|
||||
return json.dumps(action_obj, ensure_ascii=False)
|
||||
|
||||
# Flash records the tool call verbatim: the action name at the top level
|
||||
# and every argument (app_name, key, text, direction, ...) under ``args``.
|
||||
# Read those as fallbacks so both record shapes render the same phrase.
|
||||
raw_args = action_obj.get("args")
|
||||
if isinstance(raw_args, dict):
|
||||
merged = {k: v for k, v in raw_args.items() if k != "action"}
|
||||
merged.update({k: v for k, v in action_obj.items() if v is not None and k != "args"})
|
||||
merged.setdefault("intent", raw_args.get("action"))
|
||||
action_obj = merged
|
||||
|
||||
target_text = action_obj.get("target_text") or action_obj.get("text")
|
||||
coords = action_obj.get("coordinates") or action_obj.get("target")
|
||||
app_name = (
|
||||
@@ -72,13 +82,22 @@ def format_action_clean(action_obj) -> str:
|
||||
else:
|
||||
return f"Tapped {label} at {coords}"
|
||||
elif act_type in ("input_text", "focus_and_input_text"):
|
||||
label = f"'{target_text}'" if target_text else "field"
|
||||
# The field label is the resolved target's text only — the typed text
|
||||
# is not the field's name.
|
||||
field_text = action_obj.get("target_text")
|
||||
label = f"'{field_text}'" if field_text else "field"
|
||||
text_val = action_obj.get("text")
|
||||
clear_exist = action_obj.get("clear_exist") or action_obj.get("clear_before_input")
|
||||
clear_str = " (without clearing)" if clear_exist is False else ""
|
||||
return f"Inputted '{text_val}' into {label} at {coords}{clear_str}"
|
||||
elif act_type == "swipe":
|
||||
direction = action_obj.get("action_direction") or action_obj.get("direction")
|
||||
if not direction and isinstance(action_obj.get("gesture"), str):
|
||||
# Flash swipe-by-direction records the direction as ``gesture``
|
||||
# (or the FA ``action`` word, already mapped to ``intent``).
|
||||
direction = action_obj.get("gesture")
|
||||
if not direction and isinstance(action_obj.get("intent"), str):
|
||||
direction = action_obj.get("intent")
|
||||
duration = action_obj.get("duration") or action_obj.get("duration_ms")
|
||||
duration_str = f" over {duration}ms" if duration else ""
|
||||
if direction:
|
||||
@@ -100,6 +119,10 @@ def format_action_clean(action_obj) -> str:
|
||||
return f"Stopped app '{app_name}'"
|
||||
elif act_type == "manage_app":
|
||||
intent = action_obj.get("intent")
|
||||
if intent == "launch":
|
||||
return f"Launched app '{app_name}'"
|
||||
if intent == "stop":
|
||||
return f"Stopped app '{app_name}'"
|
||||
return f"Managed app '{app_name}' (action: {intent})"
|
||||
elif act_type == "wait_for_delay":
|
||||
delay_ms = (
|
||||
@@ -235,7 +258,22 @@ def safe_parse_validation_result(result: Any) -> list:
|
||||
return []
|
||||
|
||||
|
||||
def format_tool_call_clean(name, args, result) -> str | None:
|
||||
#: Limit tool-result text in the live context, which is rebuilt every turn.
|
||||
LIVE_RESULT_CHARS = 120
|
||||
|
||||
#: Replay allows longer tool results, but still caps large OCR and hierarchy dumps.
|
||||
REPLAY_RESULT_CHARS = 6000
|
||||
|
||||
|
||||
def _clamp_text(text: str, limit: int) -> str:
|
||||
if limit and len(text) > limit:
|
||||
return text[:limit] + "..."
|
||||
return text
|
||||
|
||||
|
||||
def format_tool_call_clean(
|
||||
name, args, result, *, result_chars: int = LIVE_RESULT_CHARS
|
||||
) -> str | None:
|
||||
ACTION_TOOLS = {
|
||||
"click",
|
||||
"click_sequence",
|
||||
@@ -300,6 +338,10 @@ def format_tool_call_clean(name, args, result) -> str | None:
|
||||
"action_direction": args.get("action_direction"),
|
||||
}
|
||||
|
||||
if action_name in ("input_text", "focus_and_input_text"):
|
||||
# The typed text is not the field's label.
|
||||
action_obj["target_text"] = args.get("target_text") or args.get("target_text_or_desc")
|
||||
|
||||
# Handle special FA swipe "action" parameter
|
||||
if action_name == "swipe" and "action" in args:
|
||||
action_val = args["action"]
|
||||
@@ -377,21 +419,17 @@ def format_tool_call_clean(name, args, result) -> str | None:
|
||||
elif block.get("type") == "image_url":
|
||||
pass
|
||||
if text_blocks:
|
||||
res_str = "\n".join(text_blocks)
|
||||
res_str = _clamp_text("\n".join(text_blocks), result_chars)
|
||||
else:
|
||||
res_str = str(result)
|
||||
res_str = _clamp_text(str(result), result_chars)
|
||||
elif isinstance(result, dict):
|
||||
error = result.get("error")
|
||||
if error:
|
||||
res_str = f"Error: {error}"
|
||||
else:
|
||||
res_str = json.dumps(result, ensure_ascii=False)
|
||||
if len(res_str) > 120:
|
||||
res_str = res_str[:120] + "..."
|
||||
res_str = _clamp_text(json.dumps(result, ensure_ascii=False), result_chars)
|
||||
else:
|
||||
res_str = str(result)
|
||||
if len(res_str) > 120:
|
||||
res_str = res_str[:120] + "..."
|
||||
res_str = _clamp_text(str(result), result_chars)
|
||||
|
||||
return f"`{name}({args_str})` -> {res_str}"
|
||||
|
||||
@@ -466,6 +504,49 @@ def _burst_member_status(result: Any, index: int) -> str:
|
||||
return f" (FAILED: {last.strip()})"
|
||||
|
||||
|
||||
def render_step_replay(
|
||||
step: dict,
|
||||
*,
|
||||
result_chars: int = REPLAY_RESULT_CHARS,
|
||||
include_summary: bool = True,
|
||||
) -> str:
|
||||
"""Render a step's reasoning, tool calls, action, and execution result.
|
||||
|
||||
``step`` is an agent-friendly record (``DataEngine.get_agent_friendly_step``
|
||||
/ ``get_agent_friendly_steps``): coordinates already normalized, traces
|
||||
already expanded into ``interleaved_events``. Used by ``replay_steps`` and
|
||||
the MCP trace inspector. Screenshots are fetched separately.
|
||||
"""
|
||||
return _render_step_detailed(
|
||||
step=step,
|
||||
relative_time=str(step.get("relative_time") or "N/A"),
|
||||
summary=step.get("summary"),
|
||||
action=step.get("action_taken"),
|
||||
result=step.get("last_execution_result"),
|
||||
is_most_recent=False,
|
||||
result_chars=result_chars,
|
||||
include_summary=include_summary,
|
||||
)
|
||||
|
||||
|
||||
def _screen_description_line(step: dict, summary: Any) -> str | None:
|
||||
"""The replay's ``[Screen]`` line: the step's stored screen description
|
||||
(the visual-transition summary), or its status when no text exists yet.
|
||||
|
||||
``extra_metadata.summary_status`` is ``pending`` while the background
|
||||
summarizer has not finished, ``failed`` when it gave up; without a status
|
||||
the line is omitted altogether.
|
||||
"""
|
||||
if summary and str(summary).strip():
|
||||
return f"[Screen]: {str(summary).strip()}"
|
||||
status = str((step.get("extra_metadata") or {}).get("summary_status") or "").lower()
|
||||
if status == "pending":
|
||||
return "[Screen]: (screen description pending)"
|
||||
if status in ("failed", "unavailable"):
|
||||
return "[Screen]: (screen description unavailable)"
|
||||
return None
|
||||
|
||||
|
||||
def _render_step_detailed(
|
||||
step: dict,
|
||||
relative_time: str,
|
||||
@@ -473,12 +554,22 @@ def _render_step_detailed(
|
||||
action: Any,
|
||||
result: Any,
|
||||
is_most_recent: bool,
|
||||
*,
|
||||
result_chars: int = LIVE_RESULT_CHARS,
|
||||
include_summary: bool = False,
|
||||
) -> str:
|
||||
# 1. Step Header
|
||||
status_str = "Most Recent Step, " if is_most_recent else ""
|
||||
|
||||
# Summary is omitted from the detailed view header as it is already fully detailed below.
|
||||
# In the live window the summary is omitted from the detailed view header
|
||||
# as it is already fully detailed below; replay callers opt in because the
|
||||
# summary is the Operator's own claim about the step, which an auditor
|
||||
# compares against the evidence that follows.
|
||||
step_line = f"- **Step {step['step_number']} ({status_str}Start: {relative_time})**"
|
||||
if include_summary:
|
||||
screen_line = _screen_description_line(step, summary)
|
||||
if screen_line:
|
||||
step_line += f"\n * {screen_line}"
|
||||
|
||||
interleaved = step.get("interleaved_events") or []
|
||||
|
||||
@@ -596,7 +687,10 @@ def _render_step_detailed(
|
||||
if name in ACTION_TOOLS:
|
||||
continue
|
||||
formatted = format_tool_call_clean(
|
||||
name, event.get("args") or {}, event.get("result")
|
||||
name,
|
||||
event.get("args") or {},
|
||||
event.get("result"),
|
||||
result_chars=result_chars,
|
||||
)
|
||||
if formatted:
|
||||
step_line += f"\n - [Tool Call]: {formatted}"
|
||||
|
||||
+40
-3
@@ -453,10 +453,31 @@ async def get_android_display_state(device_id: str) -> tuple[int, int, int] | No
|
||||
return None
|
||||
|
||||
|
||||
async def write_recording_manifest(output_dir: Path, mp4_paths: list[Path]) -> Path | None:
|
||||
"""Write the browser playlist used for orientation-aware playback."""
|
||||
RECORDING_MANIFEST_VERSION = 2
|
||||
|
||||
|
||||
async def write_recording_manifest(
|
||||
output_dir: Path,
|
||||
mp4_paths: list[Path],
|
||||
segment_offsets: dict[Path, float] | None = None,
|
||||
) -> Path | None:
|
||||
"""Write the browser playlist used for orientation-aware playback.
|
||||
|
||||
``segment_offsets`` maps each MP4 path to the wall-clock offset (seconds)
|
||||
of its first frame relative to the DataEngine session start. Segments are
|
||||
still listed back to back via ``start``/``duration`` for the legacy player
|
||||
timeline, and additionally carry ``offset_ms``/``duration_ms`` so the UI
|
||||
can map a session-relative step time onto the right segment even though
|
||||
scrcpy restarts (rotation, crash recovery) leave gaps between segments.
|
||||
A segment without a known offset is assumed to follow the previous one
|
||||
without a gap.
|
||||
"""
|
||||
segments = []
|
||||
timeline = 0.0
|
||||
session_cursor_ms = 0
|
||||
offsets = {
|
||||
Path(path).resolve(): float(offset) for path, offset in (segment_offsets or {}).items()
|
||||
}
|
||||
for path in mp4_paths:
|
||||
if not path.exists():
|
||||
continue
|
||||
@@ -467,23 +488,39 @@ async def write_recording_manifest(output_dir: Path, mp4_paths: list[Path]) -> P
|
||||
if duration <= 0 or width <= 0 or height <= 0:
|
||||
logger.warning(f"Recording segment skipped due to invalid metadata: {path}")
|
||||
continue
|
||||
duration_ms = int(round(duration * 1000))
|
||||
known_offset = offsets.get(path.resolve())
|
||||
offset_ms = (
|
||||
max(0, int(round(known_offset * 1000)))
|
||||
if known_offset is not None
|
||||
else session_cursor_ms
|
||||
)
|
||||
segments.append(
|
||||
{
|
||||
"file": path.name,
|
||||
"start": round(timeline, 3),
|
||||
"duration": round(duration, 3),
|
||||
"offset_ms": offset_ms,
|
||||
"duration_ms": duration_ms,
|
||||
"width": width,
|
||||
"height": height,
|
||||
}
|
||||
)
|
||||
timeline += duration
|
||||
session_cursor_ms = offset_ms + duration_ms
|
||||
if not segments:
|
||||
return None
|
||||
manifest_path = output_dir / "recording.json"
|
||||
temporary_manifest_path = output_dir / "recording.part.json"
|
||||
temporary_manifest_path.write_text(
|
||||
json.dumps(
|
||||
{"version": 1, "duration": round(timeline, 3), "segments": segments},
|
||||
{
|
||||
"version": RECORDING_MANIFEST_VERSION,
|
||||
"duration": round(timeline, 3),
|
||||
"session_offset_ms": segments[0]["offset_ms"],
|
||||
"session_end_ms": session_cursor_ms,
|
||||
"segments": segments,
|
||||
},
|
||||
indent=2,
|
||||
),
|
||||
encoding="utf-8",
|
||||
|
||||
@@ -482,6 +482,8 @@ def crop_and_annotate_target(
|
||||
|
||||
out_buf = io.BytesIO()
|
||||
if crop_size is None:
|
||||
if cropped_img.mode != "RGB":
|
||||
cropped_img = cropped_img.convert("RGB")
|
||||
cropped_img.save(out_buf, format="JPEG", quality=85)
|
||||
else:
|
||||
cropped_img.save(out_buf, format="PNG")
|
||||
@@ -746,3 +748,42 @@ def draw_action_overlay_on_image(
|
||||
|
||||
except Exception:
|
||||
return image_bytes
|
||||
|
||||
|
||||
def overlay_action_on_screenshot(image_bytes: bytes, action_obj: Any) -> bytes | None:
|
||||
"""Draws a recorded step action (as stored in ``action_taken``) onto the
|
||||
step's before-action screenshot.
|
||||
|
||||
Accepts the raw stored action (a dict, or a burst list whose first member
|
||||
is used), maps the usual coordinate keys into the drawing arguments and
|
||||
returns the annotated JPEG bytes. Returns ``None`` when there is nothing to
|
||||
draw (no action, no coordinates, or the drawing left the image unchanged).
|
||||
Shared by the Checker's ``get_step_screenshot(which="overlay")`` and the
|
||||
MCP trace inspector so both produce the same picture.
|
||||
"""
|
||||
if not image_bytes:
|
||||
return None
|
||||
action = action_obj[0] if isinstance(action_obj, list) and action_obj else action_obj
|
||||
if not isinstance(action, dict):
|
||||
return None
|
||||
act_name = action.get("action") or action.get("name") or ""
|
||||
if not act_name:
|
||||
return None
|
||||
action_args = dict(action.get("args") or {})
|
||||
for key in (
|
||||
"target",
|
||||
"coordinates",
|
||||
"point",
|
||||
"direction",
|
||||
"sequence",
|
||||
"targets",
|
||||
"normalized_coordinates",
|
||||
):
|
||||
if key in action and key not in action_args:
|
||||
action_args[key] = action[key]
|
||||
annotated = draw_action_overlay_on_image(
|
||||
image_bytes=image_bytes, action_name=act_name, action_args=action_args
|
||||
)
|
||||
if not annotated or annotated == image_bytes:
|
||||
return None
|
||||
return annotated
|
||||
@@ -227,12 +227,17 @@
|
||||
"max_chunks": 8
|
||||
// "max_eras": 8 // independent era cap; omitted = follows max_chunks
|
||||
},
|
||||
// Deterministic cold-history recall tool (Pro operator)
|
||||
// search_history: deterministic keyword / step-range lookup over stored history
|
||||
"recall": {
|
||||
"enabled": true,
|
||||
"max_results": 5,
|
||||
"max_text_tokens": 2000,
|
||||
"max_image_steps": 1
|
||||
"screen_scan_steps": 150
|
||||
},
|
||||
// replay_steps: full step replays (whole trailing steps are dropped over budget)
|
||||
"replay": {
|
||||
"max_steps": 5,
|
||||
"max_tokens": 12000
|
||||
}
|
||||
},
|
||||
// 🎬 Screen Recording & Video Analyzer Configuration
|
||||
|
||||
+1
-1
@@ -19,7 +19,7 @@ Deeply understand and select between ARTEMIS's dual execution models (**ARTEMIS
|
||||
|
||||
- **ARTEMIS Flash (Fast / Reactive Model)**:
|
||||
- **Applicable Scenarios**: Designed for simple, highly deterministic, lightweight UI operations or direct automation workflows. There is no step cap by default (history is compressed rather than truncated), so task length alone is not a reason to choose Pro; choose Pro when the task needs a persistent plan, verified checkpoints, notes or a written report, ADB / log diagnostics, or multi-branch exploration.
|
||||
- **Execution Mechanism (Reactive Loop)**: Does not enter the LangGraph multi-node orchestration: there is no Planner, no pre-execution safety net, and no Checker. `FlashRunner` receives the user's goal, the latest UI element list, and the live screenshot, performs "Observe-Think-Act" with a single LLM, and calls action tools until it reports the task status. It chains taps into one `click_sequence` to catch transient UI (auto-fading control bars, toasts) before they expire, and it can call `ask_explorer` for element grounding, `recall_history` to look up compressed earlier steps, and `video_analyzer` over the session recording. History lives in the same session transcript ledger as Pro (session-relative `T+mm:ss` clock, screenshots folded into visual summaries, older steps chunked into eras).
|
||||
- **Execution Mechanism (Reactive Loop)**: Does not enter the LangGraph multi-node orchestration: there is no Planner, no pre-execution safety net, and no Checker. `FlashRunner` receives the user's goal, the latest UI element list, and the live screenshot, performs "Observe-Think-Act" with a single LLM, and calls action tools until it reports the task status. It chains taps into one `click_sequence` to catch transient UI (auto-fading control bars, toasts) before they expire, and it can call `ask_explorer` for element grounding, `search_history` / `replay_steps` / `get_step_screenshot` to look up compressed earlier steps, and `video_analyzer` over the session recording. History lives in the same session transcript ledger as Pro (session-relative `T+mm:ss` clock, screenshots folded into visual summaries, older steps chunked into eras).
|
||||
|
||||
- **ARTEMIS Pro (Deep / Multi-Agent Graph-Driven Closed-Loop Model)**:
|
||||
- **Applicable Scenarios**: Designed for long-range, highly complex, dynamic multi-branch tasks; continuous monitoring / polling loops; and tasks requiring deep system diagnostics (ADB, logs, video), verified checkpoints, or a detailed written report.
|
||||
|
||||
+108
-160
@@ -15,66 +15,49 @@
|
||||
"""MCP Tool: mobile_inspect_trace."""
|
||||
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
from typing import Any
|
||||
from PIL import Image, ImageDraw
|
||||
|
||||
from mcp_server.base import mcp
|
||||
from mcp_server.utils import env_utils
|
||||
from artemis.data_engine.history_reader import OfflineHistoryReader
|
||||
from artemis.runtime import trace_store
|
||||
from artemis.utils.visualization import draw_action_overlay_on_image
|
||||
from artemis.utils.task_tree import (
|
||||
_render_step_detailed,
|
||||
build_plan_and_history,
|
||||
)
|
||||
from artemis.tools.history import load_step_screenshot, replay_steps_text, search_history_text
|
||||
from artemis.utils.task_tree import build_plan_and_history
|
||||
|
||||
|
||||
def _draw_action_overlay(bg_image_path: str, action_obj: dict, output_path: str) -> bool:
|
||||
"""Draws the action overlay on the image using Artemis's native visualization engine."""
|
||||
def _write_overlay(annotated_bytes: bytes, output_path: str) -> bool:
|
||||
"""Persists an already-drawn action overlay next to the trace."""
|
||||
try:
|
||||
if not os.path.exists(bg_image_path):
|
||||
return False
|
||||
|
||||
with open(bg_image_path, "rb") as f:
|
||||
raw_bytes = f.read()
|
||||
|
||||
act_name = action_obj.get("action") or action_obj.get("name") or ""
|
||||
action_args = dict(action_obj.get("args") or {})
|
||||
for k in (
|
||||
"target",
|
||||
"coordinates",
|
||||
"point",
|
||||
"direction",
|
||||
"sequence",
|
||||
"targets",
|
||||
"normalized_coordinates",
|
||||
):
|
||||
if k in action_obj and k not in action_args:
|
||||
action_args[k] = action_obj[k]
|
||||
|
||||
annotated_bytes = draw_action_overlay_on_image(
|
||||
image_bytes=raw_bytes,
|
||||
action_name=act_name,
|
||||
action_args=action_args,
|
||||
)
|
||||
|
||||
if not annotated_bytes or annotated_bytes == raw_bytes:
|
||||
return False
|
||||
|
||||
os.makedirs(os.path.dirname(output_path), exist_ok=True)
|
||||
with open(output_path, "wb") as f:
|
||||
f.write(annotated_bytes)
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"Error drawing action overlay: {e}", file=sys.stderr)
|
||||
except OSError as e:
|
||||
print(f"Error writing action overlay: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
|
||||
def _recall_config() -> Any:
|
||||
try:
|
||||
from artemis.config import load_agent_config
|
||||
|
||||
return load_agent_config().memory.recall
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def mobile_inspect_trace(action: str, trace_id: str, step_number: int | None = None) -> Any:
|
||||
async def mobile_inspect_trace(
|
||||
action: str,
|
||||
trace_id: str,
|
||||
step_number: int | None = None,
|
||||
query: str | None = None,
|
||||
step_range: list[int] | None = None,
|
||||
max_results: int = 5,
|
||||
) -> Any:
|
||||
"""Retrieves execution details of a mobile automation task (running or finished).
|
||||
|
||||
Use it to monitor progress, verify answers, and diagnose agent errors for
|
||||
@@ -84,21 +67,35 @@ async def mobile_inspect_trace(action: str, trace_id: str, step_number: int | No
|
||||
- **'view_summary'**: High-level execution summary across all steps.
|
||||
Pro: hierarchical task plan with per-step status; Flash: full execution
|
||||
chain with each step's reasoning and action.
|
||||
- **'search'**: Deterministic keyword / step-range lookup over the full
|
||||
stored history (screen descriptions, exact actions and results,
|
||||
reasoning, tool calls, notes, on-screen OCR/UI-tree text incl.
|
||||
package/activity names, compressed-history ledgers). Every hit carries
|
||||
its step number; a `step_range` also returns that range's per-step
|
||||
action ledger. Requires `query` and/or `step_range`.
|
||||
- **'view_step_screenshots'**: Local file paths of one step's screenshots:
|
||||
`before_screenshot` (what the agent saw), `after_screenshot` (only set
|
||||
when the action failed/was intercepted, else null), and
|
||||
`action_overlay_screenshot` (the action visually marked — e.g. red circle
|
||||
for taps — key for verifying the agent tapped the right element).
|
||||
- **'view_step_details'**: Deep per-step diagnostics (raw VLM thoughts,
|
||||
exact actions). Pro only — DO NOT use for Flash tasks ('view_summary'
|
||||
already carries their full execution chain).
|
||||
- **'view_step_details'**: Full replay of one step exactly as the agent's
|
||||
own context showed it: what the screen showed, its reasoning, every tool
|
||||
call it made (name, arguments, result — e.g. what the explorer or OCR
|
||||
reported), the planned action, any safety-net interception and the
|
||||
execution result. Coordinates are normalized [x, y]. Works for Flash and
|
||||
Pro tasks.
|
||||
|
||||
Args:
|
||||
action: `"view_summary"`, `"view_step_screenshots"`, or
|
||||
action: `"view_summary"`, `"search"`, `"view_step_screenshots"`, or
|
||||
`"view_step_details"`.
|
||||
trace_id: The task's session identifier from `mobile_run_task`.
|
||||
step_number: 1-indexed step to query; required for the two per-step
|
||||
actions, omit for `view_summary`.
|
||||
actions, omit otherwise.
|
||||
query: Keywords for `"search"` (case-insensitive; every
|
||||
whitespace-separated term is matched independently).
|
||||
step_range: Optional `[start, end]` step range (inclusive) for
|
||||
`"search"`.
|
||||
max_results: Maximum `"search"` hits to return (server-side cap applies).
|
||||
"""
|
||||
project_root = env_utils.get_project_root()
|
||||
db_path = os.path.join(trace_store.TRACES_DIR, "data_engine.db")
|
||||
@@ -278,20 +275,45 @@ async def mobile_inspect_trace(action: str, trace_id: str, step_number: int | No
|
||||
"message": f"Failed to format plan and history using Artemis task tree: {e}",
|
||||
}
|
||||
|
||||
# 2. Handle Actions requiring step_number
|
||||
elif action in ("view_step_screenshots", "view_step_details"):
|
||||
# 2. Actions served by the offline history reader (same records as the live engine)
|
||||
elif action in ("search", "view_step_screenshots", "view_step_details"):
|
||||
status_data = trace_store.read_status(trace_id)
|
||||
is_flash = status_data and status_data.get("model", "").lower() == "flash"
|
||||
device_serial = status_data.get("device_serial") if status_data else None
|
||||
|
||||
if action == "view_step_details" and is_flash:
|
||||
try:
|
||||
reader = OfflineHistoryReader(db_path, os.path.dirname(db_path), trace_id)
|
||||
except Exception as e:
|
||||
return {
|
||||
"error": "Not applicable for Flash model",
|
||||
"message": (
|
||||
"Action 'view_step_details' is not needed for Flash model runs. "
|
||||
"For Flash tasks, 'view_summary' already outputs the complete "
|
||||
"step-by-step reasoning (thoughts) and executed actions for the entire run."
|
||||
),
|
||||
"error": "Database error",
|
||||
"message": f"Failed to open the trace database: {e}",
|
||||
}
|
||||
|
||||
# 3. Handle Action: "search"
|
||||
if action == "search":
|
||||
if not query and not step_range:
|
||||
return {
|
||||
"error": "Missing parameter",
|
||||
"message": "Action 'search' needs 'query' and/or 'step_range'.",
|
||||
}
|
||||
try:
|
||||
results = search_history_text(
|
||||
reader,
|
||||
query=query or "",
|
||||
step_range=step_range,
|
||||
max_results=int(max_results or 5),
|
||||
recall_config=_recall_config(),
|
||||
)
|
||||
except Exception as e:
|
||||
return {
|
||||
"error": "Search error",
|
||||
"message": f"Failed to search trace '{trace_id}': {e}",
|
||||
}
|
||||
return {
|
||||
"trace_id": trace_id,
|
||||
"device_serial": device_serial,
|
||||
"query": query or "",
|
||||
"step_range": step_range,
|
||||
"results": results,
|
||||
}
|
||||
|
||||
if step_number is None:
|
||||
@@ -301,138 +323,64 @@ async def mobile_inspect_trace(action: str, trace_id: str, step_number: int | No
|
||||
}
|
||||
|
||||
try:
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.row_factory = sqlite3.Row
|
||||
cursor = conn.cursor()
|
||||
cursor.execute(
|
||||
"SELECT * FROM steps WHERE session_id = ? AND step_number = ?",
|
||||
(trace_id, step_number),
|
||||
)
|
||||
row = cursor.fetchone()
|
||||
|
||||
if not row:
|
||||
conn.close()
|
||||
return {
|
||||
"error": "Step not found",
|
||||
"message": f"Step number {step_number} not found for trace '{trace_id}'.",
|
||||
}
|
||||
|
||||
step_dict = dict(row)
|
||||
for col in ("action_taken", "last_execution_result", "extra_metadata"):
|
||||
if step_dict.get(col):
|
||||
try:
|
||||
step_dict[col] = json.loads(step_dict[col])
|
||||
except (ValueError, TypeError):
|
||||
# Non-JSON column value: keep the raw string.
|
||||
pass
|
||||
|
||||
cursor.execute(
|
||||
"SELECT start_time FROM sessions WHERE session_id = ?",
|
||||
(trace_id,),
|
||||
)
|
||||
session_row = cursor.fetchone()
|
||||
session_start_time = session_row["start_time"] if session_row else None
|
||||
|
||||
cursor.execute(
|
||||
"SELECT type, name, status, timestamp, duration, payload FROM traces WHERE step_id = ? ORDER BY timestamp ASC",
|
||||
(step_dict["step_id"],),
|
||||
)
|
||||
trace_rows = cursor.fetchall()
|
||||
events = []
|
||||
for tr in trace_rows:
|
||||
t_dict = dict(tr)
|
||||
if t_dict.get("payload"):
|
||||
try:
|
||||
t_dict["payload"] = json.loads(t_dict["payload"])
|
||||
except (ValueError, TypeError):
|
||||
# Non-JSON payload: keep the raw string.
|
||||
pass
|
||||
events.append(t_dict)
|
||||
step_dict["interleaved_events"] = events
|
||||
conn.close()
|
||||
record = reader.get_step_record(step_number)
|
||||
except Exception as e:
|
||||
return {
|
||||
"error": "Database error",
|
||||
"message": f"Database query failed: {e}",
|
||||
}
|
||||
if record is None:
|
||||
return {
|
||||
"error": "Step not found",
|
||||
"message": f"Step number {step_number} not found for trace '{trace_id}'.",
|
||||
}
|
||||
|
||||
# 3. Handle Action: "view_step_screenshots"
|
||||
# 4. Handle Action: "view_step_screenshots"
|
||||
if action == "view_step_screenshots":
|
||||
images_dir = os.path.join(trace_store.TRACES_DIR, "images")
|
||||
if not os.path.exists(images_dir):
|
||||
images_dir = os.path.join(project_root, "traces", "images")
|
||||
|
||||
pre_image = None
|
||||
post_image = None
|
||||
overlay_image = None
|
||||
|
||||
if step_dict.get("pre_image_name"):
|
||||
name = step_dict["pre_image_name"]
|
||||
if not name.endswith(".jpg"):
|
||||
name += ".jpg"
|
||||
path = os.path.join(images_dir, name)
|
||||
if os.path.exists(path):
|
||||
pre_image = f"file://{path}"
|
||||
pre_path = reader.get_step_image_path(step_number, "pre")
|
||||
if pre_path is not None:
|
||||
pre_image = f"file://{pre_path}"
|
||||
|
||||
overlay_dir = trace_store.get_trace_dir(trace_id)
|
||||
os.makedirs(overlay_dir, exist_ok=True)
|
||||
overlay_filename = f"step_{step_dict['step_number']}_overlay.jpg"
|
||||
overlay_abs_path = os.path.join(overlay_dir, overlay_filename)
|
||||
|
||||
action_obj = step_dict.get("action_taken")
|
||||
action_parsed = (
|
||||
action_obj[0] if isinstance(action_obj, list) and action_obj else action_obj
|
||||
)
|
||||
|
||||
if isinstance(action_parsed, dict) and (
|
||||
action_parsed.get("action") or action_parsed.get("name")
|
||||
):
|
||||
if not os.path.exists(overlay_abs_path):
|
||||
success_draw = _draw_action_overlay(
|
||||
path, action_parsed, overlay_abs_path
|
||||
)
|
||||
if success_draw:
|
||||
overlay_abs_path = os.path.join(overlay_dir, f"step_{step_number}_overlay.jpg")
|
||||
if os.path.exists(overlay_abs_path):
|
||||
overlay_image = f"file://{overlay_abs_path}"
|
||||
else:
|
||||
# The overlay is drawn from the raw stored action (physical
|
||||
# pixels) by the same loader the agents' tool uses.
|
||||
shot = load_step_screenshot(reader, step_number, "overlay")
|
||||
if shot.overlay_drawn and shot.image_bytes:
|
||||
if _write_overlay(shot.image_bytes, overlay_abs_path):
|
||||
overlay_image = f"file://{overlay_abs_path}"
|
||||
|
||||
if step_dict.get("post_image_name"):
|
||||
name = step_dict["post_image_name"]
|
||||
if not name.endswith(".jpg"):
|
||||
name += ".jpg"
|
||||
path = os.path.join(images_dir, name)
|
||||
if os.path.exists(path):
|
||||
post_image = f"file://{path}"
|
||||
post_path = reader.get_step_image_path(step_number, "post")
|
||||
if post_path is not None:
|
||||
post_image = f"file://{post_path}"
|
||||
|
||||
return {
|
||||
"trace_id": trace_id,
|
||||
"device_serial": device_serial,
|
||||
"step_number": step_dict["step_number"],
|
||||
"step_number": record.step_number,
|
||||
"before_screenshot": pre_image,
|
||||
"after_screenshot": post_image,
|
||||
"action_overlay_screenshot": overlay_image,
|
||||
}
|
||||
|
||||
# 4. Handle Action: "view_step_details"
|
||||
elif action == "view_step_details":
|
||||
relative_time_str = "N/A"
|
||||
if session_start_time and step_dict.get("timestamp"):
|
||||
relative_time_val = step_dict["timestamp"] - session_start_time
|
||||
relative_time_str = f"{relative_time_val:.1f}s"
|
||||
|
||||
rendered_text = _render_step_detailed(
|
||||
step=step_dict,
|
||||
relative_time=relative_time_str,
|
||||
summary=step_dict.get("summary"),
|
||||
action=step_dict.get("action_taken"),
|
||||
result=step_dict.get("last_execution_result"),
|
||||
is_most_recent=False,
|
||||
)
|
||||
|
||||
# 5. Handle Action: "view_step_details"
|
||||
rendered_text = replay_steps_text(reader, step_number, max_steps=1)
|
||||
if rendered_text.startswith("Error"):
|
||||
return {
|
||||
"error": "Formatting error",
|
||||
"message": f"Failed to render step {step_number}: {rendered_text}",
|
||||
}
|
||||
return {
|
||||
"trace_id": trace_id,
|
||||
"device_serial": device_serial,
|
||||
"step_number": step_dict["step_number"],
|
||||
"step_number": record.step_number,
|
||||
"details": rendered_text,
|
||||
}
|
||||
|
||||
@@ -441,6 +389,6 @@ async def mobile_inspect_trace(action: str, trace_id: str, step_number: int | No
|
||||
"error": "Invalid action",
|
||||
"message": (
|
||||
f"Action '{action}' is not supported. Supported actions: "
|
||||
"'view_summary', 'view_step_screenshots', 'view_step_details'."
|
||||
"'view_summary', 'search', 'view_step_screenshots', 'view_step_details'."
|
||||
),
|
||||
}
|
||||
+47
-11
@@ -13,30 +13,51 @@
|
||||
# limitations under the License.
|
||||
|
||||
import asyncio
|
||||
import faulthandler
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
from unittest.mock import patch
|
||||
import time
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
from langchain_core.messages import AIMessage
|
||||
import pytest
|
||||
|
||||
from artemis.config import settings
|
||||
from artemis.agents.image_processor.image_processor import ImageProcessor
|
||||
from artemis.config import settings
|
||||
from artemis.utils.python_executor import PythonExecutor
|
||||
from tests.integration.agents.explorer.test_explorer_all_tools.helpers import (
|
||||
create_mock_context,
|
||||
get_or_create_test_screenshot,
|
||||
)
|
||||
from artemis.utils.python_executor import PythonExecutor
|
||||
import pytest
|
||||
|
||||
# Setup logging
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger("test_explorer_ask_image_processor_jupyter")
|
||||
|
||||
# Hard ceiling for the whole test. Kernel startup, one cell and shutdown take a
|
||||
# few seconds even on a saturated host; if anything blocks past this point,
|
||||
# faulthandler prints every thread's stack to stderr and terminates the process
|
||||
# instead of leaving the run hanging.
|
||||
TEST_WALL_CLOCK_LIMIT_S = 300
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_explorer_ask_image_processor_jupyter():
|
||||
test_name = "test_explorer_ask_image_processor_jupyter"
|
||||
started_at = time.monotonic()
|
||||
faulthandler.dump_traceback_later(TEST_WALL_CLOCK_LIMIT_S, exit=True)
|
||||
original_traces_path = settings.TRACES_PATH
|
||||
try:
|
||||
await _run_scenario(test_name)
|
||||
finally:
|
||||
settings.TRACES_PATH = original_traces_path
|
||||
faulthandler.cancel_dump_traceback_later()
|
||||
logger.info(f"Test wall-clock time: {time.monotonic() - started_at:.1f}s")
|
||||
|
||||
|
||||
async def _run_scenario(test_name: str) -> None:
|
||||
# 1. Setup outputs directory
|
||||
outputs_dir = Path(__file__).resolve().parent / "outputs" / test_name
|
||||
outputs_dir.mkdir(parents=True, exist_ok=True)
|
||||
@@ -90,21 +111,22 @@ async def test_explorer_ask_image_processor_jupyter():
|
||||
executed_turns.append({"code": code, "output": result})
|
||||
return result
|
||||
|
||||
from langchain_core.messages import AIMessage
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
sample_code = """
|
||||
# The code runs in a separate kernel process, which does not see this
|
||||
# test's settings.TRACES_PATH override, so the intermediates location is
|
||||
# passed in as a literal path.
|
||||
image_processor_dir = outputs_dir / "images" / "image_processor"
|
||||
sample_code = f"""
|
||||
from artemis.utils.cv_canvas import ImageCanvas
|
||||
import glob
|
||||
from pathlib import Path
|
||||
from artemis.config import settings
|
||||
|
||||
dirs = sorted(glob.glob(str(Path(settings.TRACES_PATH) / "images" / "image_processor" / "intermediates_*")))
|
||||
intermediate_dir = dirs[-1] if dirs else "."
|
||||
dirs = sorted(glob.glob(str(Path({str(image_processor_dir)!r}) / "intermediates_*")))
|
||||
intermediate_dir = dirs[-1]
|
||||
|
||||
canvas = ImageCanvas("img_0", intermediate_dir)
|
||||
canvas.crop(10, 10, 50, 50)
|
||||
canvas.save(final=True)
|
||||
print("saved crop from", intermediate_dir)
|
||||
"""
|
||||
|
||||
mock_llm = MagicMock()
|
||||
@@ -223,8 +245,22 @@ canvas.save(final=True)
|
||||
|
||||
# 6. Assertions
|
||||
assert "error" not in result, f"Image Processor failed: {result.get('error')}"
|
||||
assert len(executed_turns) == 1, "Exactly one execute_python turn was expected"
|
||||
cell_output = executed_turns[0]["output"]
|
||||
assert "Traceback" not in cell_output, f"Kernel raised while cropping:\n{cell_output}"
|
||||
assert "Error]" not in cell_output, f"Executor reported an error:\n{cell_output}"
|
||||
assert "saved crop from" in cell_output, f"Unexpected cell output:\n{cell_output}"
|
||||
assert "outputs" in result, "Image Processor did not return outputs"
|
||||
assert len(result["outputs"]) > 0, "Image Processor did not return any output images"
|
||||
final_entry = result["outputs"][-1]
|
||||
assert final_entry.get("is_output") is True, (
|
||||
"Final image was not produced by the executed code (fell back to the input image)"
|
||||
)
|
||||
assert final_entry["image_id"] != "img_0", (
|
||||
"Final image must be the cropped result, not the input"
|
||||
)
|
||||
assert Path(final_entry["path"]).is_file(), f"Cropped image missing: {final_entry['path']}"
|
||||
assert result.get("summary") == "Cropped target G-Logo region."
|
||||
logger.info("Test completed successfully! All assertions passed.")
|
||||
|
||||
|
||||
|
||||
@@ -12,51 +12,15 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import json
|
||||
from artemis.agents.history_analyzer.history_analyzer import HistoryAnalyzer
|
||||
from artemis.config import parse_llm_config
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_exec_get_step_details(artemis_context):
|
||||
"""Test the exec_get_step_details method with manually configured steps."""
|
||||
analyzer = HistoryAnalyzer(artemis_context)
|
||||
analyzer.history_steps = [
|
||||
{
|
||||
"step_number": 0,
|
||||
"relative_time": 0.0,
|
||||
"summary": "Initial step",
|
||||
"action_taken": "none",
|
||||
"operator_raw_thinking": "Starting up",
|
||||
"last_execution_result": "Success",
|
||||
},
|
||||
{
|
||||
"step_number": 1,
|
||||
"relative_time": 5.0,
|
||||
"summary": "Second step",
|
||||
"action_taken": "click",
|
||||
"operator_raw_thinking": "Clicking button",
|
||||
"last_execution_result": "Success",
|
||||
},
|
||||
]
|
||||
|
||||
first_step_num = 0
|
||||
result = analyzer.exec_get_step_details(first_step_num, first_step_num)
|
||||
|
||||
assert "Error:" not in result
|
||||
assert "No steps found" not in result
|
||||
|
||||
parsed = json.loads(result)
|
||||
assert isinstance(parsed, list)
|
||||
assert len(parsed) == 1
|
||||
assert parsed[0]["step_number"] == first_step_num
|
||||
assert "summary" in parsed[0]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_step_details_tool_with_fixture(artemis_context):
|
||||
"""Test _get_step_details_tool against real steps loaded from the inputs/data_engine.db fixture."""
|
||||
async def test_replay_steps_tool_with_fixture(artemis_context):
|
||||
"""The analyzer's replay_steps tool against real steps loaded from the
|
||||
inputs/data_engine.db fixture."""
|
||||
if not artemis_context.data_engine:
|
||||
pytest.skip("DataEngine fixture not initialized")
|
||||
|
||||
@@ -65,18 +29,21 @@ async def test_get_step_details_tool_with_fixture(artemis_context):
|
||||
assert len(steps) > 0
|
||||
|
||||
analyzer = HistoryAnalyzer(artemis_context)
|
||||
tool = analyzer._get_step_details_tool(steps)
|
||||
tools = {t.name: t for t in analyzer._build_tools()}
|
||||
assert {
|
||||
"search_history",
|
||||
"replay_steps",
|
||||
"get_step_screenshot",
|
||||
"list_notes",
|
||||
"read_note",
|
||||
} <= set(tools)
|
||||
|
||||
result = tool.invoke({"start_step": 1, "end_step": 3})
|
||||
result = tools["replay_steps"].invoke({"start_step": 1, "end_step": 3})
|
||||
assert isinstance(result, str)
|
||||
assert "Error:" not in result
|
||||
assert "No steps found" not in result
|
||||
|
||||
parsed = json.loads(result)
|
||||
assert isinstance(parsed, list)
|
||||
assert len(parsed) == 3
|
||||
assert parsed[0]["step_number"] == 1
|
||||
assert "YouTube" in parsed[0]["summary"]
|
||||
assert not result.startswith("Error")
|
||||
assert "- **Step 1 (" in result
|
||||
assert "- **Step 3 (" in result
|
||||
assert "YouTube" in result
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -12,36 +12,32 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from artemis.agents.outputter.tools import (
|
||||
get_search_history_tool,
|
||||
get_step_details_tool,
|
||||
get_step_screenshot_tool,
|
||||
)
|
||||
from artemis.context import ArtemisContext
|
||||
"""The outputter mounts the shared history tools; smoke-test them against the
|
||||
fixture database."""
|
||||
|
||||
import pytest
|
||||
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.tools.history import get_history_tools
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def history_steps(artemis_context: ArtemisContext):
|
||||
if artemis_context.data_engine:
|
||||
return artemis_context.data_engine.get_agent_friendly_steps()
|
||||
return []
|
||||
def history_tools(artemis_context: ArtemisContext):
|
||||
search, replay, screenshot = get_history_tools(artemis_context)
|
||||
return {"search_history": search, "replay_steps": replay, "get_step_screenshot": screenshot}
|
||||
|
||||
|
||||
def test_get_step_details_tool(history_steps):
|
||||
"""Test get_step_details_tool to ensure it runs without errors."""
|
||||
tool = get_step_details_tool(history_steps)
|
||||
result = tool.invoke({"start_step": 1, "end_step": 5})
|
||||
def test_replay_steps_tool(history_tools):
|
||||
"""replay_steps runs against the fixture history without errors."""
|
||||
result = history_tools["replay_steps"].invoke({"start_step": 1, "end_step": 5})
|
||||
|
||||
assert isinstance(result, str)
|
||||
assert len(result) > 0
|
||||
|
||||
|
||||
def test_get_step_screenshot_tool(artemis_context: ArtemisContext, history_steps):
|
||||
"""Test get_step_screenshot_tool to ensure it handles requests gracefully."""
|
||||
tool = get_step_screenshot_tool(artemis_context, history_steps)
|
||||
|
||||
result = tool.invoke({"step_number": 1})
|
||||
def test_get_step_screenshot_tool(history_tools):
|
||||
"""get_step_screenshot returns content blocks with an image, or a plain explanation."""
|
||||
result = history_tools["get_step_screenshot"].invoke({"step_number": 1})
|
||||
|
||||
if isinstance(result, list):
|
||||
assert len(result) == 2
|
||||
@@ -52,10 +48,9 @@ def test_get_step_screenshot_tool(artemis_context: ArtemisContext, history_steps
|
||||
assert len(result) > 0
|
||||
|
||||
|
||||
def test_get_search_history_tool(artemis_context: ArtemisContext):
|
||||
"""Test get_search_history_tool to ensure it can search through history."""
|
||||
tool = get_search_history_tool(artemis_context)
|
||||
result = tool.invoke({"query": "test"})
|
||||
def test_search_history_tool(history_tools):
|
||||
"""search_history searches the fixture history and always answers in text."""
|
||||
result = history_tools["search_history"].invoke({"query": "test"})
|
||||
|
||||
assert isinstance(result, str)
|
||||
assert len(result) > 0
|
||||
@@ -14,6 +14,7 @@
|
||||
|
||||
"""Recording files of cancelled / killed workers must survive and be served whole."""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import time
|
||||
@@ -171,3 +172,61 @@ def test_recover_orphaned_recording_returns_none_without_files(tmp_path):
|
||||
missing = tmp_path / "missing" / "recording.mkv"
|
||||
assert MediaService.recover_orphaned_recording(str(missing)) is None
|
||||
assert MediaService.recover_orphaned_recording(None) is None
|
||||
|
||||
|
||||
def test_resolve_video_segments_passes_through_session_offsets(tmp_path, monkeypatch):
|
||||
folder = tmp_path / "artemis-traces" / "web_9_done"
|
||||
folder.mkdir(parents=True)
|
||||
(folder / "recording.mp4").write_bytes(b"seg0")
|
||||
(folder / "recording_001.mp4").write_bytes(b"seg1")
|
||||
(folder / "recording.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"version": 2,
|
||||
"duration": 15.0,
|
||||
"segments": [
|
||||
{
|
||||
"file": "recording.mp4",
|
||||
"start": 0.0,
|
||||
"duration": 10.0,
|
||||
"offset_ms": 1200,
|
||||
"duration_ms": 10000,
|
||||
"width": 1080,
|
||||
"height": 1920,
|
||||
},
|
||||
{
|
||||
"file": "recording_001.mp4",
|
||||
"start": 10.0,
|
||||
"duration": 5.0,
|
||||
"offset_ms": 12700,
|
||||
"duration_ms": 5000,
|
||||
"width": 1920,
|
||||
"height": 1080,
|
||||
},
|
||||
{"file": "legacy.mp4", "start": 15.0, "duration": 1.0, "width": 1, "height": 1},
|
||||
],
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(folder / "legacy.mp4").write_bytes(b"legacy")
|
||||
monkeypatch.setattr(media_module, "WORKSPACE_ROOT", tmp_path)
|
||||
|
||||
segments = MediaService.resolve_video_segments(
|
||||
"/videos/artemis-traces/web_9_done/recording.mp4"
|
||||
)
|
||||
|
||||
assert [s["url"] for s in segments] == [
|
||||
"/videos/artemis-traces/web_9_done/recording.mp4",
|
||||
"/videos/artemis-traces/web_9_done/recording_001.mp4",
|
||||
"/videos/artemis-traces/web_9_done/legacy.mp4",
|
||||
]
|
||||
assert [(s["start"], s["duration"]) for s in segments] == [
|
||||
(0.0, 10.0),
|
||||
(10.0, 5.0),
|
||||
(15.0, 1.0),
|
||||
]
|
||||
assert segments[0]["offset_ms"] == 1200 and segments[0]["duration_ms"] == 10000
|
||||
assert segments[1]["offset_ms"] == 12700 and segments[1]["duration_ms"] == 5000
|
||||
# Legacy entries without offsets stay offset-free so the UI falls back to back-to-back.
|
||||
assert "offset_ms" not in segments[2]
|
||||
@@ -45,20 +45,38 @@ def test_get_session_checks_reads_ledger_and_outcome(tmp_path, monkeypatch):
|
||||
json.dumps({"task_status": "completed", "tests": {"passed": 2, "failed": 0}}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
stream = {
|
||||
"attempt_id": "abc#1",
|
||||
"checkpoint_id": "abc",
|
||||
"phase": "checkpoint",
|
||||
"trace_id": "trace-1",
|
||||
"ts": 1.5,
|
||||
"truncated": False,
|
||||
"dropped_chars": 0,
|
||||
"segments": [
|
||||
{"execution_id": "e1", "role": "thought", "when": 1.1, "text": "Looking"},
|
||||
{"execution_id": "e1", "role": "answer", "when": 1.2, "text": "Seen"},
|
||||
],
|
||||
}
|
||||
(session_dir / "check_streams.jsonl").write_text(
|
||||
json.dumps(stream) + "\nnot json\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
result = MediaService.get_session_checks("sess-1")
|
||||
|
||||
assert [r["attempt_id"] for r in result["records"]] == ["abc#1", "final#1"]
|
||||
assert result["records"][0]["subgoal_text"] == "Create the alarm"
|
||||
assert result["run_outcome"]["task_status"] == "completed"
|
||||
assert result["streams"] == [stream]
|
||||
|
||||
|
||||
def test_get_session_checks_without_material(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(media_service_module, "TRACES_PATH", tmp_path)
|
||||
(tmp_path / "sess-2").mkdir()
|
||||
|
||||
assert MediaService.get_session_checks("sess-2") == {"records": [], "run_outcome": None}
|
||||
assert MediaService.get_session_checks("missing") == {"records": [], "run_outcome": None}
|
||||
empty = {"records": [], "streams": [], "run_outcome": None}
|
||||
assert MediaService.get_session_checks("sess-2") == empty
|
||||
assert MediaService.get_session_checks("missing") == empty
|
||||
|
||||
|
||||
def test_checks_route_is_registered():
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""``/api/sessions/{id}/usage``: token totals, live context and run tuning."""
|
||||
|
||||
import json
|
||||
|
||||
from apps.admin_console.database.connection import db_session
|
||||
from apps.admin_console.database.repositories.session_repository import SessionRepository
|
||||
|
||||
|
||||
def _insert_session(db_path, session_id, device_info):
|
||||
with db_session(db_path) as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO sessions (session_id, initial_goal, start_time, status, device_info)"
|
||||
" VALUES (?, ?, ?, ?, ?)",
|
||||
(session_id, "goal", 1.0, "completed", json.dumps(device_info)),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def _insert_usage(db_path, session_id, trace_id, ts, payload, *, name="llm_usage"):
|
||||
with db_session(db_path) as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO traces (trace_id, session_id, type, name, timestamp, status, payload)"
|
||||
" VALUES (?, ?, 'llm_call', ?, ?, 'success', ?)",
|
||||
(trace_id, session_id, name, ts, json.dumps(payload)),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def _usage(prompt, completion, node=None, cached=0):
|
||||
payload = {
|
||||
"prompt_tokens": prompt,
|
||||
"completion_tokens": completion,
|
||||
"total_tokens": prompt + completion,
|
||||
"cached_tokens": cached,
|
||||
"context_base_tokens": prompt,
|
||||
}
|
||||
if node:
|
||||
payload["node"] = node
|
||||
return payload
|
||||
|
||||
|
||||
def test_usage_sums_all_agents_and_tracks_latest_operator_context(tmp_path):
|
||||
db_path = tmp_path / "sessions.db"
|
||||
repo = SessionRepository(db_path)
|
||||
_insert_session(
|
||||
db_path,
|
||||
"s1",
|
||||
{
|
||||
"profile": "pro",
|
||||
"run_tuning": {"verification_level": "checkpoints", "explorer_mode": "pro"},
|
||||
},
|
||||
)
|
||||
_insert_usage(db_path, "s1", "t1", 10.0, _usage(30_000, 500, node="planner"))
|
||||
_insert_usage(db_path, "s1", "t2", 11.0, _usage(120_000, 800, node="operator", cached=90_000))
|
||||
_insert_usage(db_path, "s1", "t3", 12.0, _usage(5_000, 100, node="checker"))
|
||||
_insert_usage(db_path, "s1", "t4", 13.0, _usage(150_000, 700, node="operator"))
|
||||
# A retry trace shares the type but is not a usage record.
|
||||
_insert_usage(db_path, "s1", "t5", 14.0, {"error": "503"}, name="llm_retry")
|
||||
|
||||
usage = repo.get_session_usage("s1")
|
||||
|
||||
assert usage["llm_calls"] == 4
|
||||
assert usage["prompt_tokens"] == 305_000
|
||||
assert usage["completion_tokens"] == 2_100
|
||||
assert usage["total_tokens"] == 307_100
|
||||
assert usage["cached_tokens"] == 90_000
|
||||
assert usage["operator_context_tokens"] == 150_000
|
||||
assert usage["operator_context_updated_at"] == 13.0
|
||||
assert usage["operator_context_window_tokens"] == 1_000_000
|
||||
assert usage["profile"] == "pro"
|
||||
assert usage["run_tuning"] == {"verification_level": "checkpoints", "explorer_mode": "pro"}
|
||||
|
||||
|
||||
def test_usage_flash_runner_counts_as_operator_and_has_no_tuning(tmp_path):
|
||||
db_path = tmp_path / "sessions.db"
|
||||
repo = SessionRepository(db_path)
|
||||
_insert_session(db_path, "s2", {"profile": "flash"})
|
||||
_insert_usage(db_path, "s2", "t1", 1.0, _usage(20_000, 200, node="FlashRunner"))
|
||||
|
||||
usage = repo.get_session_usage("s2")
|
||||
|
||||
assert usage["operator_context_tokens"] == 20_000
|
||||
assert usage["total_tokens"] == 20_200
|
||||
assert usage["profile"] == "flash"
|
||||
assert usage["run_tuning"] is None
|
||||
|
||||
|
||||
def test_usage_for_unknown_or_empty_session_is_zeroed(tmp_path):
|
||||
db_path = tmp_path / "sessions.db"
|
||||
repo = SessionRepository(db_path)
|
||||
|
||||
usage = repo.get_session_usage("missing")
|
||||
|
||||
assert usage["llm_calls"] == 0
|
||||
assert usage["total_tokens"] == 0
|
||||
assert usage["operator_context_tokens"] is None
|
||||
assert usage["profile"] is None
|
||||
assert usage["run_tuning"] is None
|
||||
|
||||
|
||||
def test_usage_route_is_registered():
|
||||
from apps.admin_console.routers import sessions
|
||||
|
||||
paths = {getattr(r, "path", None) for r in sessions.router.routes}
|
||||
assert "/api/sessions/{session_id}/usage" in paths
|
||||
@@ -173,8 +173,8 @@ def test_tool_table_probe_gate():
|
||||
names = {t.name for t in build_checker_tools(ctx, "checkpoint")}
|
||||
assert "probe_device" in names
|
||||
assert "read_note" in names
|
||||
assert "get_step_detail" in names
|
||||
assert "get_step_screenshot" in names
|
||||
# The shared history tools, mounted as a set.
|
||||
assert {"search_history", "replay_steps", "get_step_screenshot"} <= names
|
||||
# Never any device action, note write, or sub-agent
|
||||
assert not names & {
|
||||
"save_note",
|
||||
@@ -263,3 +263,309 @@ async def test_final_entry_captures_live_screen():
|
||||
)
|
||||
|
||||
capture_mock.assert_awaited_once()
|
||||
|
||||
|
||||
# --- Shared history tools in the check loop ------------------------------------------
|
||||
|
||||
|
||||
def _jpeg_file(tmp_path, color="white"):
|
||||
from PIL import Image
|
||||
|
||||
path = tmp_path / "pre.jpg"
|
||||
Image.new("RGB", (200, 400), color).save(path, format="JPEG")
|
||||
return path
|
||||
|
||||
|
||||
def _screenshot_then_verdict(provider, tool_call=None):
|
||||
"""LLM double: first turn asks for a step screenshot (or ``tool_call``),
|
||||
second turn answers."""
|
||||
from types import SimpleNamespace
|
||||
|
||||
report = CheckReport(
|
||||
verdicts=[CheckVerdict(item_text="x", kind="verify", status="passed", evidence="e")]
|
||||
)
|
||||
structured = MagicMock()
|
||||
structured.ainvoke = AsyncMock(return_value=report)
|
||||
llm = MagicMock()
|
||||
llm.endpoint = SimpleNamespace(provider=provider)
|
||||
llm.with_structured_output.return_value = structured
|
||||
|
||||
ask = MagicMock()
|
||||
ask.tool_calls = [
|
||||
tool_call
|
||||
or {
|
||||
"name": "get_step_screenshot",
|
||||
"args": {"step_number": 2, "which": "pre"},
|
||||
"id": "tc-shot",
|
||||
}
|
||||
]
|
||||
done = MagicMock()
|
||||
done.tool_calls = []
|
||||
return llm, AsyncMock(side_effect=[ask, done])
|
||||
|
||||
|
||||
async def _run_with_screenshot(tmp_path, provider):
|
||||
path = _jpeg_file(tmp_path)
|
||||
ctx = _mock_ctx()
|
||||
ctx.data_engine.get_step_image_path.return_value = path
|
||||
llm, acomplete = _screenshot_then_verdict(provider)
|
||||
|
||||
with (
|
||||
patch("artemis.agents.checker.checker.get_llm", return_value=llm),
|
||||
patch("artemis.agents.checker.checker.acomplete", new=acomplete),
|
||||
):
|
||||
report = await run_checkpoint_check(
|
||||
ctx,
|
||||
check_items=[_ci(text="x")],
|
||||
anchor=EvidenceAnchor(anchor_step_id="sid", trigger_ts=0.0, plan_text="- [x] G"),
|
||||
goal="the goal",
|
||||
subgoal_text="G",
|
||||
)
|
||||
ctx.data_engine.get_step_image_path.assert_called_once_with(2, "pre")
|
||||
# The message list handed to the second model call carries the tool result.
|
||||
messages = acomplete.call_args_list[1][0][1]
|
||||
return report, messages
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_step_screenshot_rides_inside_the_tool_message_for_gemini(tmp_path):
|
||||
from langchain_core.messages import HumanMessage, ToolMessage
|
||||
|
||||
report, messages = await _run_with_screenshot(tmp_path, "google")
|
||||
assert report.verdicts[0].status == "passed"
|
||||
|
||||
tool_msgs = [m for m in messages if isinstance(m, ToolMessage)]
|
||||
assert len(tool_msgs) == 1
|
||||
assert tool_msgs[0].tool_call_id == "tc-shot"
|
||||
assert tool_msgs[0].name == "get_step_screenshot"
|
||||
assert tool_msgs[0].status == "success"
|
||||
blocks = tool_msgs[0].content
|
||||
assert blocks[0]["text"] == "Screenshot of step 2 (pre-action) is attached."
|
||||
assert blocks[1]["image_url"]["url"].startswith("data:image/jpeg;base64,")
|
||||
# No extra human turn was spliced in for this provider.
|
||||
assert sum(isinstance(m, HumanMessage) for m in messages) == 1
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_step_screenshot_follows_as_a_human_message_for_openai(tmp_path):
|
||||
from langchain_core.messages import HumanMessage, ToolMessage
|
||||
|
||||
_, messages = await _run_with_screenshot(tmp_path, "openai")
|
||||
tool_msgs = [m for m in messages if isinstance(m, ToolMessage)]
|
||||
assert len(tool_msgs) == 1
|
||||
assert tool_msgs[0].content == "Screenshot of step 2 (pre-action) is attached."
|
||||
follower = messages[messages.index(tool_msgs[0]) + 1]
|
||||
assert isinstance(follower, HumanMessage)
|
||||
assert follower.content[0]["text"] == "[Screenshot returned by get_step_screenshot for step 2]"
|
||||
assert follower.content[1]["image_url"]["url"].startswith("data:image/jpeg;base64,")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_replay_steps_result_is_a_plain_tool_message(tmp_path):
|
||||
from langchain_core.messages import ToolMessage
|
||||
|
||||
ctx = _mock_ctx()
|
||||
ctx.data_engine.get_agent_friendly_steps_in_range.return_value = [
|
||||
{
|
||||
"step_id": "s3",
|
||||
"step_number": 3,
|
||||
"relative_time": "9.0s",
|
||||
"summary": "Saved the alarm.",
|
||||
"action_taken": {"action": "click", "target": [0.5, 0.5], "target_text": "Save"},
|
||||
"last_execution_result": {"status": "success"},
|
||||
"interleaved_events": [],
|
||||
}
|
||||
]
|
||||
llm, acomplete = _screenshot_then_verdict(
|
||||
"google", {"name": "replay_steps", "args": {"start_step": 3}, "id": "tc-replay"}
|
||||
)
|
||||
with (
|
||||
patch("artemis.agents.checker.checker.get_llm", return_value=llm),
|
||||
patch("artemis.agents.checker.checker.acomplete", new=acomplete),
|
||||
):
|
||||
await run_checkpoint_check(
|
||||
ctx,
|
||||
check_items=[_ci(text="x")],
|
||||
anchor=EvidenceAnchor(anchor_step_id="sid", trigger_ts=0.0, plan_text="- [x] G"),
|
||||
goal="the goal",
|
||||
subgoal_text="G",
|
||||
)
|
||||
ctx.data_engine.get_agent_friendly_steps_in_range.assert_called_once_with(3, 3)
|
||||
messages = acomplete.call_args_list[1][0][1]
|
||||
tool_msg = next(m for m in messages if isinstance(m, ToolMessage))
|
||||
assert tool_msg.name == "replay_steps"
|
||||
assert "- **Step 3 (Start: 9.0s)**" in tool_msg.content
|
||||
assert "[Screen]: Saved the alarm." in tool_msg.content
|
||||
assert "[Planned Action]: Tapped 'Save' at [0.5, 0.5]" in tool_msg.content
|
||||
|
||||
|
||||
# --- Streamed-reasoning transcript (replay of the live Thought/Work text) -------------
|
||||
|
||||
|
||||
class _FakeBus:
|
||||
"""Minimal DataEngine bus: subscribers receive every published event."""
|
||||
|
||||
def __init__(self):
|
||||
self.subscribers = []
|
||||
|
||||
def subscribe(self, cb):
|
||||
self.subscribers.append(cb)
|
||||
|
||||
def unsubscribe(self, cb):
|
||||
self.subscribers.remove(cb)
|
||||
|
||||
def _publish(self, event_type, data):
|
||||
for cb in list(self.subscribers):
|
||||
cb(event_type, data)
|
||||
|
||||
|
||||
def _bus_ctx(tmp_path):
|
||||
"""Mock engine whose bus actually delivers events (the rest stays mocked)."""
|
||||
ctx = _mock_ctx()
|
||||
ctx.data_engine.base_dir = tmp_path
|
||||
bus = _FakeBus()
|
||||
ctx.data_engine.subscribers = bus.subscribers
|
||||
ctx.data_engine.subscribe = bus.subscribe
|
||||
ctx.data_engine.unsubscribe = bus.unsubscribe
|
||||
ctx.data_engine._publish = bus._publish
|
||||
return ctx
|
||||
|
||||
|
||||
def _stream(ctx, exec_id, chunk, *, stream_type="text", parent=None):
|
||||
from artemis.data_engine.context_vars import CURRENT_TRACE_ID
|
||||
|
||||
ctx.data_engine._publish(
|
||||
"llm_stream",
|
||||
{
|
||||
"execution_id": exec_id,
|
||||
"parent_trace_id": parent or str(CURRENT_TRACE_ID.get()),
|
||||
"chunk": chunk,
|
||||
"stream_type": stream_type,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _read_streams(tmp_path):
|
||||
from artemis.graph.checkpoints import read_attempt_streams
|
||||
|
||||
return read_attempt_streams(tmp_path)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_checker_persists_its_streamed_turns_as_timestamped_segments(tmp_path):
|
||||
"""Every llm_stream chunk emitted under the Checker's own trace is kept as
|
||||
one segment per (turn, kind) in first-chunk order; other agents' streams
|
||||
and a discarded (mid-stream reset) turn are left out."""
|
||||
ctx = _bus_ctx(tmp_path)
|
||||
|
||||
async def loop(ctx, messages, tools, items):
|
||||
_stream(ctx, "turn-1", "Looking at ", stream_type="thinking")
|
||||
_stream(ctx, "turn-1", "step 2.", stream_type="thinking")
|
||||
_stream(ctx, "turn-1", "Let me replay it.")
|
||||
_stream(ctx, "operator-turn", "not mine", parent="someone-else")
|
||||
_stream(ctx, "turn-2", "partial ")
|
||||
ctx.data_engine._publish(
|
||||
"llm_stream_reset", {"stream_exec_id": "turn-2", "action": "discard"}
|
||||
)
|
||||
_stream(ctx, "turn-3", "The alarm is there.")
|
||||
return CheckReport(verdicts=[])
|
||||
|
||||
with (
|
||||
patch("artemis.agents.checker.checker._run_check_loop", side_effect=loop),
|
||||
patch("artemis.agents.checker.checker.build_checker_tools", return_value=[]),
|
||||
):
|
||||
await run_checkpoint_check(
|
||||
ctx,
|
||||
check_items=[_ci(text="x")],
|
||||
anchor=EvidenceAnchor(anchor_step_id="sid", trigger_ts=0.0, plan_text="- [x] G"),
|
||||
goal="the goal",
|
||||
subgoal_text="G",
|
||||
attempt_id="abc#1",
|
||||
)
|
||||
|
||||
assert ctx.data_engine.subscribers == [] # detached after the run
|
||||
records = _read_streams(tmp_path)
|
||||
assert len(records) == 1
|
||||
rec = records[0]
|
||||
assert rec["attempt_id"] == "abc#1"
|
||||
assert rec["checkpoint_id"] == "abc"
|
||||
assert rec["phase"] == "checkpoint"
|
||||
assert rec["trace_id"]
|
||||
assert rec["truncated"] is False and rec["dropped_chars"] == 0
|
||||
assert isinstance(rec["ts"], float)
|
||||
segs = rec["segments"]
|
||||
assert [(s["execution_id"], s["role"], s["text"]) for s in segs] == [
|
||||
("turn-1", "thought", "Looking at step 2."),
|
||||
("turn-1", "answer", "Let me replay it."),
|
||||
("turn-3", "answer", "The alarm is there."),
|
||||
]
|
||||
assert all(isinstance(s["when"], float) for s in segs)
|
||||
assert segs[0]["when"] <= segs[1]["when"] <= segs[2]["when"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_final_entry_persists_transcript_even_when_the_loop_fails(tmp_path):
|
||||
"""The transcript is written in a ``finally``: an attempt that times out,
|
||||
raises or is superseded still leaves what it said."""
|
||||
ctx = _bus_ctx(tmp_path)
|
||||
|
||||
async def loop(ctx, messages, tools, items):
|
||||
_stream(ctx, "turn-1", "Checking the final screen…")
|
||||
raise RuntimeError("provider down")
|
||||
|
||||
with (
|
||||
patch("artemis.agents.checker.checker._run_check_loop", side_effect=loop),
|
||||
patch("artemis.agents.checker.checker.build_checker_tools", return_value=[]),
|
||||
patch(
|
||||
"artemis.agents.checker.checker._capture_final_screen",
|
||||
AsyncMock(return_value=(None, "")),
|
||||
),
|
||||
pytest.raises(RuntimeError),
|
||||
):
|
||||
await run_final_check(
|
||||
ctx,
|
||||
goal="g",
|
||||
plan_text="- [x] G",
|
||||
ledger=[],
|
||||
check_items=[_ci(text="x")],
|
||||
attempt_id="final#1",
|
||||
)
|
||||
|
||||
records = _read_streams(tmp_path)
|
||||
assert [r["attempt_id"] for r in records] == ["final#1"]
|
||||
assert records[0]["phase"] == "final" and records[0]["checkpoint_id"] == "final"
|
||||
assert records[0]["segments"][0]["text"] == "Checking the final screen…"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_transcript_without_attempt_id_or_streamed_text(tmp_path):
|
||||
ctx = _bus_ctx(tmp_path)
|
||||
|
||||
async def silent(ctx, messages, tools, items):
|
||||
return CheckReport(verdicts=[])
|
||||
|
||||
async def talkative(ctx, messages, tools, items):
|
||||
_stream(ctx, "turn-1", "text")
|
||||
return CheckReport(verdicts=[])
|
||||
|
||||
with patch("artemis.agents.checker.checker.build_checker_tools", return_value=[]):
|
||||
with patch("artemis.agents.checker.checker._run_check_loop", side_effect=talkative):
|
||||
await run_checkpoint_check(
|
||||
ctx,
|
||||
check_items=[_ci(text="x")],
|
||||
anchor=EvidenceAnchor(anchor_step_id="sid", trigger_ts=0.0, plan_text=""),
|
||||
goal="g",
|
||||
subgoal_text="G",
|
||||
)
|
||||
with patch("artemis.agents.checker.checker._run_check_loop", side_effect=silent):
|
||||
await run_checkpoint_check(
|
||||
ctx,
|
||||
check_items=[_ci(text="x")],
|
||||
anchor=EvidenceAnchor(anchor_step_id="sid", trigger_ts=0.0, plan_text=""),
|
||||
goal="g",
|
||||
subgoal_text="G",
|
||||
attempt_id="abc#2",
|
||||
)
|
||||
|
||||
assert _read_streams(tmp_path) == []
|
||||
assert not (tmp_path / "check_streams.jsonl").exists()
|
||||
@@ -138,3 +138,70 @@ def test_flash_runner_screenshot_pruning(mock_context):
|
||||
assert messages[2].content[0]["type"] == "text"
|
||||
assert messages[2].content[1]["type"] == "image_url"
|
||||
assert messages[2].content[1]["image_url"]["url"] == "data:image/jpeg;base64,SCREENSHOT_3"
|
||||
|
||||
|
||||
# --- Recorded action shape (coordinate space + ledger rendering) ---------------------
|
||||
|
||||
|
||||
def _record(runner, name, args):
|
||||
from types import SimpleNamespace
|
||||
|
||||
runner.ctx.data_engine = Mock()
|
||||
runner.ctx.data_engine.current_step_id = None
|
||||
runner.ctx.data_engine.record_step.return_value = "step-1"
|
||||
exec_result = SimpleNamespace(
|
||||
status="success", text_summary=f"{name} executed", metadata={}, ui_elements_text="ui"
|
||||
)
|
||||
step_id = runner._record_action_step(
|
||||
name, args, exec_result, "thought", {}, b"pre", ["xml"], b"post"
|
||||
)
|
||||
assert step_id == "step-1"
|
||||
return runner.ctx.data_engine.record_step.call_args.kwargs["action_taken"]
|
||||
|
||||
|
||||
def test_record_action_step_stamps_the_normalized_coordinate_space(mock_context):
|
||||
"""Flash stores the model's own 0–1000 target verbatim, marked as
|
||||
normalized, so the agent-friendly view never re-normalizes it."""
|
||||
from artemis.utils.coordinates import (
|
||||
COORDINATE_SPACE_KEY,
|
||||
COORDINATE_SPACE_NORMALIZED,
|
||||
normalize_any_structure,
|
||||
)
|
||||
|
||||
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||
runner = FlashRunner(mock_context, goal="g")
|
||||
recorded = _record(runner, "click", {"target": [320, 399]})
|
||||
assert recorded["coordinates"] == [320, 399]
|
||||
assert recorded["normalized_coordinates"] == [320, 399]
|
||||
assert recorded[COORDINATE_SPACE_KEY] == COORDINATE_SPACE_NORMALIZED
|
||||
assert normalize_any_structure(recorded, 1080, 2400)["coordinates"] == [320, 399]
|
||||
|
||||
|
||||
def test_flash_records_render_in_the_ledger_with_their_arguments(mock_context):
|
||||
"""The band-③ phrase reads the tool arguments Flash keeps under ``args``:
|
||||
app launches name the package, key presses the key, inputs the text,
|
||||
direction swipes the direction — never ``'None'``."""
|
||||
from artemis.utils.task_tree import format_actions_clean
|
||||
|
||||
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||
runner = FlashRunner(mock_context, goal="g")
|
||||
|
||||
launch = _record(runner, "manage_app", {"action": "launch", "app_name": "com.android.settings"})
|
||||
assert format_actions_clean(launch) == "Launched app 'com.android.settings'"
|
||||
stop = _record(runner, "manage_app", {"action": "stop", "app_name": "com.android.settings"})
|
||||
assert format_actions_clean(stop) == "Stopped app 'com.android.settings'"
|
||||
direct = _record(runner, "launch_app", {"app_name": "com.android.settings"})
|
||||
assert format_actions_clean(direct) == "Launched app 'com.android.settings'"
|
||||
|
||||
key = _record(runner, "press_key", {"key": "BACK"})
|
||||
assert format_actions_clean(key) == "Pressed key 'BACK'"
|
||||
|
||||
typed = _record(runner, "input_text", {"target": [500, 600], "text": "hello"})
|
||||
assert format_actions_clean(typed) == "Inputted 'hello' into field at [500, 600]"
|
||||
|
||||
swipe_dir = _record(runner, "swipe", {"direction": "up"})
|
||||
assert format_actions_clean(swipe_dir) == "Swiped up"
|
||||
swipe_fa = _record(runner, "swipe", {"action": "down"})
|
||||
assert format_actions_clean(swipe_fa) == "Swiped down"
|
||||
swipe_coords = _record(runner, "swipe", {"coordinates": [556, 289, 556, 124]})
|
||||
assert format_actions_clean(swipe_coords) == "Swiped from [556, 289] to [556, 124]"
|
||||
@@ -87,28 +87,55 @@ def test_video_analyzer_bound_only_when_recording_tools_enabled(mock_context):
|
||||
assert "video_analyzer" not in names
|
||||
|
||||
|
||||
def test_recall_history_bound_only_with_a_data_engine_session(mock_context):
|
||||
HISTORY_TOOLS = {"search_history", "replay_steps", "get_step_screenshot"}
|
||||
|
||||
|
||||
def test_history_tools_bound_only_with_a_data_engine_session(mock_context):
|
||||
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||
# No DataEngine: nothing to search, the tool stays out (as in Pro).
|
||||
# No DataEngine: nothing to read, the tools stay out (as in Pro).
|
||||
names = [t.name for t in FlashRunner(mock_context, goal="g")._get_tools()]
|
||||
assert "recall_history" not in names
|
||||
assert not (HISTORY_TOOLS & set(names))
|
||||
prompt = FlashRunner(mock_context, goal="g")._render_system_prompt(
|
||||
FlashRunner(mock_context, goal="g")._get_tools()
|
||||
)
|
||||
assert "search_history" not in prompt and "replay_steps" not in prompt
|
||||
|
||||
mock_context.data_engine = Mock()
|
||||
with patch("artemis.tools.history_recall._recall_config", return_value=None):
|
||||
with patch("artemis.tools.history._recall_config", return_value=None):
|
||||
runner = FlashRunner(mock_context, goal="g")
|
||||
tools = runner._get_tools()
|
||||
names = [t.name for t in tools]
|
||||
assert "recall_history" in names
|
||||
declaration = next(t for t in tools if t.name == "recall_history")
|
||||
assert set(declaration.parameters["properties"]) == {
|
||||
"query",
|
||||
"step_range",
|
||||
"include_details",
|
||||
"include_images",
|
||||
"max_results",
|
||||
}
|
||||
assert declaration.parameters["required"] == ["query"]
|
||||
assert "`recall_history`" in runner._render_system_prompt(tools)
|
||||
assert HISTORY_TOOLS <= set(names)
|
||||
assert names[-1] == "report_task_status"
|
||||
|
||||
# Same declarations as the LangChain tools: derived from one args schema.
|
||||
search = next(t for t in tools if t.name == "search_history")
|
||||
assert set(search.parameters["properties"]) == {"query", "step_range", "max_results"}
|
||||
assert search.parameters["required"] == []
|
||||
replay = next(t for t in tools if t.name == "replay_steps")
|
||||
assert set(replay.parameters["properties"]) == {"start_step", "end_step"}
|
||||
assert replay.parameters["required"] == ["start_step"]
|
||||
shot = next(t for t in tools if t.name == "get_step_screenshot")
|
||||
assert shot.parameters["properties"]["which"]["enum"] == ["pre", "post", "overlay"]
|
||||
|
||||
prompt = runner._render_system_prompt(tools)
|
||||
assert "`search_history`" in prompt
|
||||
assert "`replay_steps`" in prompt
|
||||
assert "`get_step_screenshot`" in prompt
|
||||
|
||||
|
||||
def test_search_history_alone_follows_the_recall_config_gate(mock_context):
|
||||
from types import SimpleNamespace
|
||||
|
||||
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||
mock_context.data_engine = Mock()
|
||||
with patch(
|
||||
"artemis.tools.history._recall_config",
|
||||
return_value=SimpleNamespace(enabled=False),
|
||||
):
|
||||
names = [t.name for t in FlashRunner(mock_context, goal="g")._get_tools()]
|
||||
assert "search_history" not in names
|
||||
assert {"replay_steps", "get_step_screenshot"} <= set(names)
|
||||
|
||||
|
||||
def test_video_analyzer_prompt_segment_follows_availability(mock_context):
|
||||
|
||||
@@ -95,7 +95,7 @@ async def test_hint_fires_on_old_matching_step():
|
||||
assert len(parts) == 1
|
||||
assert "Historical state hint" in parts[0]
|
||||
assert "Step 2" in parts[0]
|
||||
assert "recall_history" in parts[0]
|
||||
assert "search_history" in parts[0]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -12,7 +12,6 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
@@ -108,12 +107,15 @@ async def test_history_analyzer_detailed_query_with_tool_call():
|
||||
},
|
||||
]
|
||||
mock_ctx.data_engine.get_agent_friendly_steps.return_value = steps
|
||||
mock_ctx.data_engine.get_agent_friendly_steps_in_range.side_effect = lambda s, e: [
|
||||
st for st in steps if s <= st["step_number"] <= e
|
||||
]
|
||||
|
||||
# Turn 1 LLM response: Tool call to get_step_details for step 2
|
||||
# Turn 1 LLM response: Tool call to replay_steps for step 2
|
||||
mock_response_turn_1 = MagicMock()
|
||||
mock_response_turn_1.content = ""
|
||||
mock_tool_call = {
|
||||
"name": "get_step_details",
|
||||
"name": "replay_steps",
|
||||
"args": {"start_step": 2, "end_step": 2},
|
||||
"id": "call_123456",
|
||||
}
|
||||
@@ -160,16 +162,14 @@ async def test_history_analyzer_detailed_query_with_tool_call():
|
||||
] # ToolMessage is second to last after final response is appended
|
||||
assert isinstance(tool_msg, ToolMessage)
|
||||
assert tool_msg.tool_call_id == "call_123456"
|
||||
assert tool_msg.name == "replay_steps"
|
||||
|
||||
# Parse the JSON in the ToolMessage content
|
||||
tool_result = json.loads(tool_msg.content)
|
||||
assert len(tool_result) == 1
|
||||
assert tool_result[0]["step_number"] == 2
|
||||
assert tool_result[0]["operator_raw_thinking"] == "Toggle the switch to turn wifi on"
|
||||
assert tool_result[0]["action_taken"] == {
|
||||
"action": "tap",
|
||||
"coordinates": [100, 200],
|
||||
}
|
||||
# The shared replay renderer: step header, screen description, thinking.
|
||||
mock_ctx.data_engine.get_agent_friendly_steps_in_range.assert_called_once_with(2, 2)
|
||||
assert "- **Step 2 (Start: 4.5s)**" in tool_msg.content
|
||||
assert "[Screen]: Toggled the wifi switch" in tool_msg.content
|
||||
assert "Toggle the switch to turn wifi on" in tool_msg.content
|
||||
assert "**Step 1" not in tool_msg.content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -282,13 +282,14 @@ def test_history_analyzer_robust_tools_behavior():
|
||||
]
|
||||
|
||||
analyzer = HistoryAnalyzer(mock_ctx)
|
||||
mock_ctx.data_engine.get_agent_friendly_steps_in_range.return_value = steps
|
||||
|
||||
# 1. Test get_step_details with string inputs
|
||||
details_tool = analyzer._get_step_details_tool(steps)
|
||||
result_details = details_tool.invoke({"start_step": "1", "end_step": "1"})
|
||||
parsed_result = json.loads(result_details)
|
||||
assert len(parsed_result) == 1
|
||||
assert parsed_result[0]["step_number"] == 1
|
||||
# 1. The shared replay tool coerces string step numbers
|
||||
replay_tool = next(t for t in analyzer._build_tools() if t.name == "replay_steps")
|
||||
result_details = replay_tool.invoke({"start_step": "1", "end_step": "1"})
|
||||
mock_ctx.data_engine.get_agent_friendly_steps_in_range.assert_called_once_with(1, 1)
|
||||
assert "- **Step 1 (Start: 1.0s)**" in result_details
|
||||
assert "[Screen]: Action 1" in result_details
|
||||
|
||||
# 2. Test read_note with .md suffix
|
||||
from artemis.tools.scratchpad import get_read_note_tool_pure
|
||||
|
||||
@@ -113,6 +113,7 @@ async def test_perform_action_validation():
|
||||
{
|
||||
"action": "tap",
|
||||
"coordinates": [200, 300],
|
||||
"coordinate_space": "pixel",
|
||||
"normalized_coordinates": [185, 125],
|
||||
"times": 1,
|
||||
"delay_ms": 100,
|
||||
@@ -143,6 +144,7 @@ async def test_perform_action_validation():
|
||||
assert actions[0] == {
|
||||
"action": "focus_and_input_text",
|
||||
"coordinates": [400, 500],
|
||||
"coordinate_space": "pixel",
|
||||
"normalized_coordinates": [370, 208],
|
||||
"text": "hello",
|
||||
"clear_before_input": True,
|
||||
@@ -171,6 +173,7 @@ async def test_perform_action_validation():
|
||||
{
|
||||
"action": "tap",
|
||||
"coordinates": [400, 500],
|
||||
"coordinate_space": "pixel",
|
||||
"normalized_coordinates": [370, 208],
|
||||
"times": 1,
|
||||
"delay_ms": 100,
|
||||
@@ -263,6 +266,7 @@ async def test_bare_coordinate_click_enriched_by_hit_test():
|
||||
{
|
||||
"action": "tap",
|
||||
"coordinates": [540, 1440],
|
||||
"coordinate_space": "pixel",
|
||||
"normalized_coordinates": [500, 600],
|
||||
"times": 1,
|
||||
"delay_ms": 100,
|
||||
@@ -784,6 +788,7 @@ async def test_long_press_action_translation():
|
||||
{
|
||||
"action": "long_press_on",
|
||||
"coordinates": [200, 300],
|
||||
"coordinate_space": "pixel",
|
||||
"normalized_coordinates": [185, 125],
|
||||
"duration": 1500,
|
||||
"target_text": "Item 1",
|
||||
@@ -806,6 +811,7 @@ async def test_long_press_action_translation():
|
||||
{
|
||||
"action": "long_press_on",
|
||||
"coordinates": [540, 1440],
|
||||
"coordinate_space": "pixel",
|
||||
"normalized_coordinates": [500, 600],
|
||||
"duration": 1000,
|
||||
"target_text": None,
|
||||
|
||||
@@ -32,8 +32,12 @@ def test_operator_prompt_contract_matches_note_runtime_semantics():
|
||||
)
|
||||
assert "--- Execution Incident (OPEN) ---" in prompt
|
||||
assert "Failure Analyzer" not in prompt
|
||||
assert "(`read_note`, `list_notes`, `recall_history`)" in prompt
|
||||
assert "`save_note`, and `recall_history`" not in prompt
|
||||
assert (
|
||||
"(`read_note`, `list_notes`, `search_history`, `replay_steps`, `get_step_screenshot`)"
|
||||
in prompt
|
||||
)
|
||||
assert "recall_history" not in prompt
|
||||
assert "`save_note`, and `search_history`" not in prompt
|
||||
assert "Do NOT submit a Turn-Ending Action at the same time" not in prompt
|
||||
assert "memory note tools (`read_note`, `list_notes`, `save_note`, `update_note`" not in prompt
|
||||
|
||||
|
||||
@@ -12,11 +12,11 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Operator dual prompt path (M2): legacy golden parity and transcript mode.
|
||||
"""Operator prompt snapshots and transcript mode.
|
||||
|
||||
Flag off (``agent.memory.transcript.enabled=false``): the legacy 2-message
|
||||
build must stay byte-for-byte identical — pinned by SHA-256 goldens recorded
|
||||
against the pre-split implementation.
|
||||
for the current template.
|
||||
|
||||
Flag on: ``_build_prompt`` renders ``S + F + A + tail`` from the session
|
||||
transcript ledger; the turn's messages are staged after the tool loop and
|
||||
@@ -51,20 +51,10 @@ from artemis.memory.transcript import (
|
||||
TranscriptLedger,
|
||||
)
|
||||
|
||||
# SHA-256 of the legacy system message rendered by TemplatePromptComponent
|
||||
# with the fixed inputs below. Originally recorded BEFORE the M2 template
|
||||
# split; re-recorded in M5 (2026-09-01) after the Short-Term Memory block was
|
||||
# removed, and again on 2026-09-02 when the Failure Analyzer was retired: the
|
||||
# operator.json template gained the Time-Sensitive Tasks principle, the two
|
||||
# execution tiers (vetted single action / fast-action burst), the Execution
|
||||
# Incident workflow section and the Phase 3 burst-discipline rule; and again
|
||||
# on 2026-09-02 after the prompt consolidation (every rule stated once, the
|
||||
# transcript/legacy history-marker split, the recited tool budget, the
|
||||
# recommended diagnosis triggers); and again on 2026-09-03 for the nested
|
||||
# plan-ledger contract and its validation gate. Any other drift means the
|
||||
# flag-off path is no longer byte-identical.
|
||||
GOLDEN_EMPTY_PLAN = "2ed35be503e3872f008ae5a12616af6f90cde65f23b63adbb6da782079ae4e7c"
|
||||
GOLDEN_SENTINEL_PLAN = "8bd6aa60cf3fad8d21cf5c284a01f586c57d209fcd1d2b4a517a791736f9b1de"
|
||||
# SHA-256 snapshots of the legacy system message with the fixed inputs below.
|
||||
# Update these when an intentional template change alters the rendered prompt.
|
||||
GOLDEN_EMPTY_PLAN = "dfcb15a164fa32172cf4bd2ab7b39409229a05dd1d5995eae26992784cfaa4f6"
|
||||
GOLDEN_SENTINEL_PLAN = "4b49a3c1ace6387df05a6f1f163dda007e4c2f8ff3d544dac12520317f17d21b"
|
||||
|
||||
SCREENSHOT_B64 = base64.b64encode(b"fake-jpeg-bytes").decode("utf-8")
|
||||
|
||||
@@ -166,13 +156,8 @@ def _no_action_llm(captured: list):
|
||||
@pytest.mark.asyncio
|
||||
async def test_transcript_mode_two_turns_build_four_regions(tmp_path):
|
||||
ctx = _transcript_ctx()
|
||||
# Regression data shape (M2 cold-start bug, on-device A/B 2026-09-01):
|
||||
# turn 1 runs before any step record exists; by turn 2 the first step HAS
|
||||
# been recorded, so `steps` is non-empty while the previous turn is still
|
||||
# staged (turn_count == 0). The cold-start check must not read that
|
||||
# live-session state as a process restart — seeding would be rejected by
|
||||
# the empty-ledger invariant and silently demote every turn to the legacy
|
||||
# 2-message path.
|
||||
# Turn 2 has a recorded step but no committed ledger turn yet. This is
|
||||
# an active session, so it must not trigger cold-start history seeding.
|
||||
ctx.data_engine = MagicMock()
|
||||
ctx.data_engine.base_dir = str(tmp_path)
|
||||
ctx.data_engine.global_base_dir = str(tmp_path)
|
||||
|
||||
@@ -1,102 +0,0 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import json
|
||||
from unittest.mock import MagicMock, mock_open, patch
|
||||
|
||||
from artemis.agents.outputter.tools import (
|
||||
get_search_history_tool,
|
||||
get_step_details_tool,
|
||||
get_step_screenshot_tool,
|
||||
)
|
||||
from artemis.context import ArtemisContext
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_context():
|
||||
ctx = MagicMock(spec=ArtemisContext)
|
||||
ctx.data_engine = MagicMock()
|
||||
ctx.data_engine.global_base_dir = "/tmp/fake_traces"
|
||||
return ctx
|
||||
|
||||
|
||||
def test_get_step_details_tool():
|
||||
steps = [
|
||||
{
|
||||
"step_number": 1,
|
||||
"relative_time": "1s",
|
||||
"summary": "Step 1",
|
||||
"action_taken": "tap",
|
||||
},
|
||||
{
|
||||
"step_number": 2,
|
||||
"relative_time": "2s",
|
||||
"summary": "Step 2",
|
||||
"action_taken": "swipe",
|
||||
},
|
||||
]
|
||||
tool = get_step_details_tool(steps)
|
||||
|
||||
# Test valid range
|
||||
result = tool.invoke({"start_step": 1, "end_step": 1})
|
||||
parsed = json.loads(result)
|
||||
assert len(parsed) == 1
|
||||
assert parsed[0]["step_number"] == 1
|
||||
assert parsed[0]["action_taken"] == "tap"
|
||||
|
||||
# Test invalid range
|
||||
result_invalid = tool.invoke({"start_step": 3, "end_step": 4})
|
||||
assert "No steps found" in result_invalid
|
||||
|
||||
|
||||
def test_get_step_screenshot_tool(mock_context):
|
||||
steps = [
|
||||
{"step_number": 1, "pre_image_name": "img1", "post_image_name": "img2"},
|
||||
]
|
||||
tool = get_step_screenshot_tool(mock_context, steps)
|
||||
|
||||
# Mock open and exists
|
||||
with (
|
||||
patch("pathlib.Path.exists", return_value=True),
|
||||
patch("builtins.open", mock_open(read_data=b"fake_image_bytes")),
|
||||
):
|
||||
result = tool.invoke({"step_number": 1})
|
||||
assert isinstance(result, list)
|
||||
assert result[0]["type"] == "text"
|
||||
assert result[1]["type"] == "image_url"
|
||||
assert "data:image/jpeg;base64" in result[1]["image_url"]["url"]
|
||||
|
||||
|
||||
def test_search_history_tool(mock_context):
|
||||
tool = get_search_history_tool(mock_context)
|
||||
|
||||
# Mock steps returned by storage
|
||||
mock_step_1 = MagicMock()
|
||||
mock_step_1.step_number = 1
|
||||
mock_step_1.pre_image_name = "img1"
|
||||
mock_step_1.post_image_name = None
|
||||
|
||||
mock_context.data_engine.storage.get_steps.return_value = [mock_step_1]
|
||||
|
||||
# Mock search_ui_by_hash
|
||||
mock_context.data_engine.storage.search_ui_by_hash.return_value = [
|
||||
{"type": "ocr", "matched_text": "verification code: 1234", "score": 0.9}
|
||||
]
|
||||
|
||||
result = tool.invoke({"query": "verification"})
|
||||
parsed = json.loads(result)
|
||||
assert len(parsed) == 1
|
||||
assert parsed[0]["step_number"] == 1
|
||||
assert parsed[0]["matches"][0]["matched_text"] == "verification code: 1234"
|
||||
@@ -98,3 +98,6 @@ def test_prompt_rules_are_keyed_on_target_kind():
|
||||
assert "`Kind: coordinates only`" in prompt
|
||||
assert "hidden" in prompt
|
||||
assert "Identify what appears under the red dot" in prompt
|
||||
# Rule 1 judges the dot, not the screen: a control surviving elsewhere (or as
|
||||
# an active input field) does not make the tapped spot valid.
|
||||
assert "Judge the red dot, not the screen" in prompt
|
||||
@@ -85,7 +85,11 @@ def test_full_toolset_is_the_default():
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("tool", OPERATOR_DEVICE_TOOLS + ("video_analyzer", "recall_history"))
|
||||
@pytest.mark.parametrize(
|
||||
"tool",
|
||||
OPERATOR_DEVICE_TOOLS
|
||||
+ ("video_analyzer", "search_history", "replay_steps", "get_step_screenshot"),
|
||||
)
|
||||
def test_removed_tool_leaves_no_reference(tool):
|
||||
"""The executable definition of 'an absent tool costs the model nothing'."""
|
||||
for name, template in load_operator_prompts().items():
|
||||
@@ -148,7 +152,10 @@ def test_full_set_enum_slots_render_verbatim():
|
||||
)
|
||||
assert "Helper/Subagent tools (`ask_explorer`, `ask_diagnoser`, `video_analyzer`)" in out
|
||||
assert "ADB/task tools (`run_adb_command`, `manage_task`)" in out
|
||||
assert "(`read_note`, `list_notes`, `recall_history`)" in out
|
||||
assert (
|
||||
"(`read_note`, `list_notes`, `search_history`, `replay_steps`, `get_step_screenshot`)"
|
||||
in out
|
||||
)
|
||||
# save_note is a write-through tool, never a result-dependent one; the on-demand
|
||||
# analyze_task_output is not advertised.
|
||||
assert "`save_note`, and" not in out
|
||||
|
||||
@@ -60,7 +60,14 @@ def mock_context(tmp_path):
|
||||
engine = Mock()
|
||||
engine.get_step_number.return_value = 7
|
||||
engine.get_step_record.return_value = SimpleNamespace(
|
||||
action_taken=[{"action": "tap", "coordinates": [500, 600], "target_text": "Search"}]
|
||||
action_taken=[
|
||||
{
|
||||
"action": "tap",
|
||||
"coordinates": [540, 1440],
|
||||
"coordinate_space": "pixel",
|
||||
"target_text": "Search",
|
||||
}
|
||||
]
|
||||
)
|
||||
engine.get_step_image_path.side_effect = lambda step_number, which="pre": (
|
||||
pre_path if which == "pre" else post_path
|
||||
@@ -84,7 +91,11 @@ async def test_summarizer_dispatches_visual_lens(mock_context):
|
||||
mock_context.step_memory.dispatch.assert_called_once_with(
|
||||
step_number=7,
|
||||
action_name="tap",
|
||||
action_args={"coordinates": [500, 600], "target_text": "Search"},
|
||||
action_args={
|
||||
"coordinates": [500, 600],
|
||||
"coordinate_space": "normalized",
|
||||
"target_text": "Search",
|
||||
},
|
||||
pre_img_bytes=b"PRE_IMAGE",
|
||||
post_img_bytes=b"POST_IMAGE",
|
||||
exec_outcome="success",
|
||||
@@ -123,8 +134,9 @@ async def test_summarizer_falls_back_to_structured_decisions(mock_context):
|
||||
|
||||
state = DummyState(
|
||||
structured_decisions=(
|
||||
'[{"action": "swipe", "coordinates": [1, 2, 3, 4]},'
|
||||
' {"action": "tap", "coordinates": [9, 9]}]'
|
||||
'[{"action": "swipe", "coordinates": [540, 1800, 540, 600],'
|
||||
' "coordinate_space": "pixel"},'
|
||||
' {"action": "tap", "coordinates": [540, 1200], "coordinate_space": "pixel"}]'
|
||||
),
|
||||
last_execution_result={"status": "success"},
|
||||
current_step_id="12345678-1234-5678-1234-567812345678",
|
||||
@@ -135,8 +147,10 @@ async def test_summarizer_falls_back_to_structured_decisions(mock_context):
|
||||
|
||||
kwargs = mock_context.step_memory.dispatch.call_args.kwargs
|
||||
assert kwargs["action_name"] == "swipe"
|
||||
assert kwargs["action_args"]["coordinates"] == [1, 2, 3, 4]
|
||||
assert kwargs["action_args"]["additional_actions"] == [{"action": "tap", "coordinates": [9, 9]}]
|
||||
assert kwargs["action_args"]["coordinates"] == [500, 750, 500, 250]
|
||||
assert kwargs["action_args"]["additional_actions"] == [
|
||||
{"action": "tap", "coordinates": [500, 500], "coordinate_space": "normalized"}
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
|
||||
@@ -28,6 +28,13 @@ from langchain_core.messages import AIMessage
|
||||
import pytest
|
||||
|
||||
from artemis.agents.validator.categories import ValidationErrorCategory
|
||||
from artemis.agents.validator.execution_loop import BURST_SKIPPED_MARKER
|
||||
from artemis.agents.validator.precondition_xml import (
|
||||
XML_STRONG_MIN_ELEMENTS,
|
||||
XML_VERDICT_STRONG,
|
||||
XML_VERDICT_WEAK,
|
||||
classify_xml_verdict,
|
||||
)
|
||||
from artemis.agents.validator.validator import ValidatorNode
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.mcp.action_types import ActionCode, ActionResult
|
||||
@@ -107,6 +114,28 @@ def temp_screenshot(tmp_path):
|
||||
return str(p)
|
||||
|
||||
|
||||
def _unrelated_hierarchy(count: int) -> list[dict]:
|
||||
"""``count`` labelled rows far below the target region (nothing matches it)."""
|
||||
return [
|
||||
{
|
||||
"text": f"Item {i}",
|
||||
"resource-id": f"row_{i}",
|
||||
"bounds": f"[0,{1000 + i * 100}][1080,{1080 + i * 100}]",
|
||||
"clickable": "true",
|
||||
}
|
||||
for i in range(count)
|
||||
]
|
||||
|
||||
|
||||
_LOGIN_TAP = {
|
||||
"action": "tap",
|
||||
"coordinates": [200, 250],
|
||||
"target_text": "Login",
|
||||
"target_bounds": [100, 200, 300, 300],
|
||||
"target_resource_id": "btn_login",
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_mcp():
|
||||
fake = FakeActionSession()
|
||||
@@ -175,17 +204,20 @@ async def test_validator_exec_error_opens_incident(mock_mcp, mock_context, temp_
|
||||
assert incident["reason"] == "Error: Element not found"
|
||||
assert incident["consecutive_failures"] == 1
|
||||
assert incident["burst_size"] == 1
|
||||
assert incident["action_description"] == "Tapped element at [105, 205]"
|
||||
# The Operator reads the incident: the pixel target [105, 205] is rendered
|
||||
# in its own 0-1000 space (1080x2400 frame).
|
||||
assert incident["action_description"] == "Tapped element at [97, 85]"
|
||||
# The same record rides in graph state for the Operator prompt.
|
||||
assert result["open_incident"] == incident
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_validator_burst_skips_safety_net_and_aborts_on_first_failure(
|
||||
async def test_validator_burst_vets_first_member_only_and_aborts_on_first_failure(
|
||||
mock_mcp, mock_context, temp_screenshot
|
||||
):
|
||||
"""Two or more actions form a fast-action burst: no precondition gate, a
|
||||
single dispatch per member, and the first failure aborts the rest."""
|
||||
"""Two or more actions form a fast-action burst: the precondition gate runs
|
||||
for the first member only, every member is dispatched once, and the first
|
||||
failure aborts the rest."""
|
||||
calls = []
|
||||
|
||||
def handler(name, args):
|
||||
@@ -213,9 +245,12 @@ async def test_validator_burst_skips_safety_net_and_aborts_on_first_failure(
|
||||
) as mock_gate,
|
||||
patch("artemis.utils.image_diff.check_ui_change", return_value=False),
|
||||
):
|
||||
mock_gate.return_value = (True, ValidationErrorCategory.NONE, "")
|
||||
result = await node(state)
|
||||
|
||||
mock_gate.assert_not_called()
|
||||
# Only the first member (decided on the still-live screen) is vetted.
|
||||
mock_gate.assert_called_once()
|
||||
assert mock_gate.call_args.args[2]["coordinates"] == [105, 205]
|
||||
|
||||
report = result["last_execution_result"]
|
||||
assert report["burst"] is True
|
||||
@@ -237,6 +272,42 @@ async def test_validator_burst_skips_safety_net_and_aborts_on_first_failure(
|
||||
assert result["open_incident"] == incident
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_validator_burst_first_member_blocked_aborts_whole_burst(
|
||||
mock_mcp, mock_context, temp_screenshot
|
||||
):
|
||||
"""A refused first member opens a safety-net incident and skips the rest
|
||||
of the burst; nothing is dispatched to the device."""
|
||||
mock_mcp.hierarchy = _unrelated_hierarchy(XML_STRONG_MIN_ELEMENTS + 2)
|
||||
decisions = json.dumps([dict(_LOGIN_TAP), {"action": "press_key", "keycode": "ENTER"}])
|
||||
state = DummyState(structured_decisions=decisions, latest_screenshot=temp_screenshot)
|
||||
|
||||
node = ValidatorNode(mock_context)
|
||||
with (
|
||||
patch("artemis.utils.image_diff.check_ui_change", return_value=False),
|
||||
patch.object(
|
||||
ValidatorNode, "_validate_action_precondition_pixel", new_callable=AsyncMock
|
||||
) as mock_pixel,
|
||||
):
|
||||
result = await node(state)
|
||||
|
||||
report = result["last_execution_result"]
|
||||
assert report["burst"] is True
|
||||
assert report["status"] == "failed"
|
||||
assert report["execution"][0]["attempts"][0].startswith("Pre-execution validation failed")
|
||||
assert report["execution"][1]["attempts"] == [BURST_SKIPPED_MARKER]
|
||||
assert mock_mcp.calls_for("click") == []
|
||||
assert mock_mcp.calls_for("press_key") == []
|
||||
mock_pixel.assert_not_called()
|
||||
|
||||
incident = report["incident"]
|
||||
assert incident["kind"] == "safety_net"
|
||||
assert incident["category"] == ValidationErrorCategory.TARGET_DISAPPEARED.value
|
||||
assert incident["action_index"] == 0
|
||||
assert incident["burst_size"] == 2
|
||||
assert result["open_incident"] == incident
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_validator_burst_success_executes_every_member(
|
||||
mock_mcp, mock_context, temp_screenshot
|
||||
@@ -572,6 +643,126 @@ async def test_validator_pre_execution_validation_anonymous_occupant(
|
||||
)
|
||||
|
||||
|
||||
def test_classify_xml_verdict_grades_strength():
|
||||
strong, why = classify_xml_verdict(XML_STRONG_MIN_ELEMENTS, "Login", None)
|
||||
assert (strong, why) == (XML_VERDICT_STRONG, "")
|
||||
assert classify_xml_verdict(XML_STRONG_MIN_ELEMENTS, None, "btn_login")[0] == XML_VERDICT_STRONG
|
||||
|
||||
weak, why = classify_xml_verdict(XML_STRONG_MIN_ELEMENTS - 1, "Login", "btn_login")
|
||||
assert weak == XML_VERDICT_WEAK and "sparse hierarchy" in why
|
||||
|
||||
weak, why = classify_xml_verdict(XML_STRONG_MIN_ELEMENTS, None, None)
|
||||
assert weak == XML_VERDICT_WEAK and "no resource-id/text" in why
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_validator_strong_xml_missing_verdict_is_final_without_pixel_judge(
|
||||
mock_mcp, mock_context, temp_screenshot
|
||||
):
|
||||
"""A strong XML verdict blocks the action without consulting the pixel judge."""
|
||||
mock_mcp.hierarchy = _unrelated_hierarchy(XML_STRONG_MIN_ELEMENTS + 2)
|
||||
state = DummyState(
|
||||
structured_decisions=json.dumps([dict(_LOGIN_TAP)]), latest_screenshot=temp_screenshot
|
||||
)
|
||||
node = ValidatorNode(mock_context)
|
||||
|
||||
with (
|
||||
patch("artemis.utils.image_diff.check_ui_change", return_value=True),
|
||||
patch.object(
|
||||
ValidatorNode, "_validate_action_precondition_pixel", new_callable=AsyncMock
|
||||
) as mock_pixel,
|
||||
):
|
||||
# A positive pixel verdict must not override strong XML evidence.
|
||||
mock_pixel.return_value = (True, ValidationErrorCategory.NONE, "")
|
||||
result = await node(state)
|
||||
|
||||
mock_pixel.assert_not_called()
|
||||
assert mock_mcp.calls_for("click") == []
|
||||
|
||||
incident = result["last_execution_result"]["incident"]
|
||||
assert incident["kind"] == "safety_net"
|
||||
assert incident["category"] == ValidationErrorCategory.TARGET_DISAPPEARED.value
|
||||
assert "was not found on the screen" in incident["reason"]
|
||||
assert "pixel judge not consulted" in incident["reason"]
|
||||
assert incident["evidence"]["xml_verdict"] == XML_VERDICT_STRONG
|
||||
assert incident["evidence"]["xml_element_count"] == XML_STRONG_MIN_ELEMENTS + 2
|
||||
assert incident["evidence"]["pixel_judge"] == "not consulted (strong XML verdict)"
|
||||
# The clause also rides in the action record the traces keep.
|
||||
assert (
|
||||
"pixel judge not consulted"
|
||||
in result["last_execution_result"]["execution"][0]["attempts"][0]
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_validator_weak_xml_sparse_tree_lets_pixel_judge_allow(
|
||||
mock_mcp, mock_context, temp_screenshot
|
||||
):
|
||||
"""A sparse hierarchy (WebView/game-like) cannot vouch for a missing target;
|
||||
the pixel judge decides, and a confident "present" lets the action run."""
|
||||
mock_mcp.hierarchy = _unrelated_hierarchy(2)
|
||||
state = DummyState(
|
||||
structured_decisions=json.dumps([dict(_LOGIN_TAP)]), latest_screenshot=temp_screenshot
|
||||
)
|
||||
node = ValidatorNode(mock_context)
|
||||
|
||||
with (
|
||||
patch("artemis.utils.image_diff.check_ui_change", return_value=True),
|
||||
patch.object(
|
||||
ValidatorNode, "_validate_action_precondition_pixel", new_callable=AsyncMock
|
||||
) as mock_pixel,
|
||||
):
|
||||
mock_pixel.return_value = (True, ValidationErrorCategory.NONE, "")
|
||||
result = await node(state)
|
||||
|
||||
mock_pixel.assert_called_once()
|
||||
report = result["last_execution_result"]
|
||||
assert report["status"] == "success"
|
||||
assert report["incident"] is None
|
||||
assert len(mock_mcp.calls_for("click")) == 1
|
||||
assert "safety_net_evidence" not in report["execution"][0]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_validator_weak_xml_identifierless_target_pixel_judge_blocks(
|
||||
mock_mcp, mock_context, temp_screenshot
|
||||
):
|
||||
"""Bare bounds from a hit test carry no identifiers to search for, so the
|
||||
XML "missing" is weak; the pixel judge is consulted and its refusal is
|
||||
recorded next to the XML verdict."""
|
||||
mock_mcp.hierarchy = _unrelated_hierarchy(XML_STRONG_MIN_ELEMENTS + 2)
|
||||
item = {
|
||||
"action": "tap",
|
||||
"coordinates": [200, 250],
|
||||
"target_class": "android.widget.ImageView",
|
||||
"target_bounds": [100, 200, 300, 300],
|
||||
"target_label_source": "hit_test",
|
||||
}
|
||||
state = DummyState(structured_decisions=json.dumps([item]), latest_screenshot=temp_screenshot)
|
||||
node = ValidatorNode(mock_context)
|
||||
|
||||
with (
|
||||
patch("artemis.utils.image_diff.check_ui_change", return_value=True),
|
||||
patch.object(
|
||||
ValidatorNode, "_validate_action_precondition_pixel", new_callable=AsyncMock
|
||||
) as mock_pixel,
|
||||
):
|
||||
mock_pixel.return_value = (
|
||||
False,
|
||||
ValidationErrorCategory.PIXEL_TARGET_DISAPPEARED,
|
||||
"Pixel check failed",
|
||||
)
|
||||
result = await node(state)
|
||||
|
||||
mock_pixel.assert_called_once()
|
||||
incident = result["last_execution_result"]["incident"]
|
||||
assert incident["category"] == ValidationErrorCategory.TARGET_DISAPPEARED.value
|
||||
assert "XML evidence weak (target had no resource-id/text" in incident["reason"]
|
||||
assert "pixel judge did not confirm the target" in incident["reason"]
|
||||
assert incident["evidence"]["xml_verdict"] == XML_VERDICT_WEAK
|
||||
assert incident["evidence"]["pixel_judge"].startswith("consulted (weak XML verdict)")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_validator_pixel_validation_success(mock_mcp, mock_context, temp_screenshot):
|
||||
"""Test that the pixel safety net succeeds when Gemini reports target is present."""
|
||||
|
||||
@@ -438,3 +438,194 @@ def test_record_step_without_app_data_stamps_nothing(tmp_path):
|
||||
|
||||
step = engine.storage.get_steps(engine.current_session_id)[0]
|
||||
assert "foreground_app" not in (step.extra_metadata or {})
|
||||
|
||||
|
||||
# --- Agent-friendly single-step / range lookups (step replay backing) ----------------
|
||||
|
||||
|
||||
def _replay_engine(tmp_path):
|
||||
mock_ctx = MagicMock(spec=ArtemisContext)
|
||||
mock_execution_setup = MagicMock()
|
||||
mock_execution_setup.traces_path = str(tmp_path)
|
||||
mock_ctx.execution_setup = mock_execution_setup
|
||||
mock_ctx.device = None
|
||||
engine = DataEngine(mock_ctx)
|
||||
engine.start_session("replay lookup session")
|
||||
return engine
|
||||
|
||||
|
||||
def _join_pending(engine):
|
||||
for t in list(engine._pending_threads):
|
||||
t.join()
|
||||
|
||||
|
||||
def test_friendly_step_range_expands_tool_traces_and_matches_full_listing(tmp_path):
|
||||
engine = _replay_engine(tmp_path)
|
||||
engine.record_step(
|
||||
summary="Asked the explorer about the toggle.",
|
||||
action_taken={"action": "click", "target": [540, 1200], "target_text": "Save"},
|
||||
last_execution_result={"status": "success"},
|
||||
)
|
||||
engine.record_trace(
|
||||
type="tool",
|
||||
name="ask_explorer",
|
||||
payload={"args": {"question": "toggle?"}, "result": "The toggle is ON"},
|
||||
step_id=engine.last_recorded_step_id,
|
||||
)
|
||||
engine.record_step(summary="Second step.", action_taken={"action": "swipe"})
|
||||
engine.record_step(summary="Third step.", action_taken={"action": "press_key", "key": "BACK"})
|
||||
_join_pending(engine)
|
||||
|
||||
rows = engine.get_agent_friendly_steps_in_range(1, 2)
|
||||
assert [r["step_number"] for r in rows] == [1, 2]
|
||||
calls = rows[0]["tool_calls"]
|
||||
assert calls and calls[0]["name"] == "ask_explorer"
|
||||
assert calls[0]["args"] == {"question": "toggle?"}
|
||||
assert calls[0]["result"] == "The toggle is ON"
|
||||
assert any(e["type"] == "tool_call" for e in rows[0]["interleaved_events"])
|
||||
|
||||
# The single-step path renders exactly what the whole-session listing does
|
||||
# (same trace expansion, same coordinate normalization).
|
||||
full = engine.get_agent_friendly_steps()
|
||||
assert rows[0]["action_taken"] == full[0]["action_taken"]
|
||||
assert rows[0]["interleaved_events"] == full[0]["interleaved_events"]
|
||||
|
||||
# Reversed bounds are tolerated; a single step defaults end to start.
|
||||
assert [r["step_number"] for r in engine.get_agent_friendly_steps_in_range(3, 2)] == [2, 3]
|
||||
assert [r["step_number"] for r in engine.get_agent_friendly_steps_in_range(3)] == [3]
|
||||
|
||||
one = engine.get_agent_friendly_step(3)
|
||||
assert one is not None and one["summary"] == "Third step."
|
||||
assert engine.get_agent_friendly_step(99) is None
|
||||
|
||||
|
||||
def test_friendly_step_lookups_without_session_are_empty(tmp_path):
|
||||
mock_ctx = MagicMock(spec=ArtemisContext)
|
||||
mock_execution_setup = MagicMock()
|
||||
mock_execution_setup.traces_path = str(tmp_path)
|
||||
mock_ctx.execution_setup = mock_execution_setup
|
||||
mock_ctx.device = None
|
||||
engine = DataEngine(mock_ctx)
|
||||
assert engine.get_agent_friendly_steps_in_range(1, 5) == []
|
||||
assert engine.get_agent_friendly_step(1) is None
|
||||
|
||||
|
||||
def test_tool_result_images_are_described_as_step_screenshots(tmp_path):
|
||||
"""An image a tool result embedded (stored as an <ImageRef: sha256=…>
|
||||
placeholder) is rewritten into the description of the step screenshot it
|
||||
is, never dropped silently; unknown images get the generic label."""
|
||||
import hashlib
|
||||
import io
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from artemis.data_engine.engine import GENERIC_IMAGE_LABEL
|
||||
|
||||
buf = io.BytesIO()
|
||||
Image.new("RGB", (32, 64), "red").save(buf, format="JPEG")
|
||||
pre_bytes = buf.getvalue()
|
||||
pre_hash = hashlib.sha256(pre_bytes).hexdigest()
|
||||
|
||||
engine = _replay_engine(tmp_path)
|
||||
engine.record_step(
|
||||
pre_screenshot_bytes=pre_bytes,
|
||||
summary="Opened the alarms list.",
|
||||
action_taken={"action": "click"},
|
||||
ocr_result=[{"text": "Alarm 07:30 needleocr", "bounds": [0, 0, 10, 10]}],
|
||||
)
|
||||
engine.record_step(summary="Recalled the earlier screen.", action_taken={"action": "click"})
|
||||
engine.record_trace(
|
||||
type="tool",
|
||||
name="get_step_screenshot",
|
||||
payload={
|
||||
"args": {"step_number": 1, "which": "pre"},
|
||||
"result": [
|
||||
{"type": "text", "text": "Screenshot of step 1 (pre-action) is attached."},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"<ImageRef: sha256={pre_hash} length=99999>"},
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": "<ImageRef: sha256=" + "0" * 64 + " length=5>"},
|
||||
},
|
||||
],
|
||||
},
|
||||
step_id=engine.last_recorded_step_id,
|
||||
)
|
||||
_join_pending(engine)
|
||||
|
||||
step2 = engine.get_agent_friendly_step(2)
|
||||
result = step2["tool_calls"][0]["result"]
|
||||
assert all(block["type"] == "text" for block in result)
|
||||
assert result[1]["text"] == "[screenshot: pre-action of Step 1]"
|
||||
assert result[1]["image_name"] == pre_hash
|
||||
assert result[2]["text"] == GENERIC_IMAGE_LABEL
|
||||
# Whole-session listing describes the same way.
|
||||
assert engine.get_agent_friendly_steps()[1]["tool_calls"][0]["result"] == result
|
||||
|
||||
|
||||
# --- Coordinate space of agent-friendly steps -------------------------
|
||||
|
||||
|
||||
def test_flash_normalized_record_survives_agent_friendly_steps_unchanged(tmp_path):
|
||||
"""Flash records the model's 0–1000 target with an explicit space marker;
|
||||
the agent-friendly view must not normalize it a second time."""
|
||||
from artemis.utils.coordinates import COORDINATE_SPACE_KEY, COORDINATE_SPACE_NORMALIZED
|
||||
|
||||
engine = _replay_engine(tmp_path)
|
||||
engine.record_step(
|
||||
action_taken={
|
||||
"action": "click",
|
||||
"coordinates": [320, 399],
|
||||
"normalized_coordinates": [320, 399],
|
||||
COORDINATE_SPACE_KEY: COORDINATE_SPACE_NORMALIZED,
|
||||
"args": {"target": [320, 399]},
|
||||
},
|
||||
last_execution_result={"status": "success", "result": "click executed"},
|
||||
)
|
||||
_join_pending(engine)
|
||||
|
||||
row = engine.get_agent_friendly_steps()[0]
|
||||
assert row["action_taken"]["coordinates"] == [320, 399]
|
||||
assert row["action_taken"][COORDINATE_SPACE_KEY] == COORDINATE_SPACE_NORMALIZED
|
||||
assert engine.get_agent_friendly_step(1)["action_taken"]["coordinates"] == [320, 399]
|
||||
|
||||
|
||||
def test_pro_pixel_record_is_normalized_exactly_once(tmp_path):
|
||||
"""Pro records physical pixels (with the frame size in extra_metadata);
|
||||
the friendly view converts them once, and the result is stamped so a
|
||||
downstream pass (MCP inspector, replay) leaves it alone."""
|
||||
from artemis.utils.coordinates import (
|
||||
COORDINATE_SPACE_KEY,
|
||||
COORDINATE_SPACE_NORMALIZED,
|
||||
COORDINATE_SPACE_PIXEL,
|
||||
normalize_any_structure,
|
||||
)
|
||||
|
||||
engine = _replay_engine(tmp_path)
|
||||
action = {
|
||||
"action": "tap",
|
||||
"coordinates": [540, 1200],
|
||||
"normalized_coordinates": [500, 500],
|
||||
COORDINATE_SPACE_KEY: COORDINATE_SPACE_PIXEL,
|
||||
"target_text": "Battery",
|
||||
}
|
||||
engine.record_step(
|
||||
action_taken=[action],
|
||||
last_execution_result={
|
||||
"status": "success",
|
||||
"execution": [dict(action, attempts=["Success"])],
|
||||
},
|
||||
extra_metadata={"width": 1080, "height": 2400},
|
||||
)
|
||||
_join_pending(engine)
|
||||
|
||||
row = engine.get_agent_friendly_steps()[0]
|
||||
friendly = row["action_taken"][0]
|
||||
assert friendly["coordinates"] == [500, 500]
|
||||
assert friendly[COORDINATE_SPACE_KEY] == COORDINATE_SPACE_NORMALIZED
|
||||
assert row["last_execution_result"]["execution"][0]["coordinates"] == [500, 500]
|
||||
|
||||
again = normalize_any_structure(row["action_taken"], 1080, 2400)
|
||||
assert again == row["action_taken"]
|
||||
@@ -0,0 +1,139 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""OfflineHistoryReader: the MCP inspector's read-only twin of the live engine.
|
||||
|
||||
The offline reader must produce exactly the records the live ``DataEngine``
|
||||
produces (same ``friendly_step``), open the database read-only, and satisfy
|
||||
the ``HistoryReader`` protocol the history tools are written against.
|
||||
"""
|
||||
|
||||
import io
|
||||
import sqlite3
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
from PIL import Image
|
||||
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.data_engine.engine import DataEngine
|
||||
from artemis.data_engine.history_reader import HistoryReader, OfflineHistoryReader
|
||||
|
||||
|
||||
def _jpeg(color: str) -> bytes:
|
||||
buf = io.BytesIO()
|
||||
Image.new("RGB", (64, 128), color).save(buf, format="JPEG")
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
def _engine(tmp_path):
|
||||
ctx = MagicMock(spec=ArtemisContext)
|
||||
ctx.execution_setup = MagicMock()
|
||||
ctx.execution_setup.traces_path = str(tmp_path)
|
||||
ctx.device = None
|
||||
engine = DataEngine(ctx)
|
||||
engine.start_session("offline reader test")
|
||||
return engine
|
||||
|
||||
|
||||
def _flush(engine):
|
||||
for t in list(engine._pending_threads):
|
||||
t.join()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def recorded(tmp_path):
|
||||
engine = _engine(tmp_path)
|
||||
engine.record_step(
|
||||
pre_screenshot_bytes=_jpeg("red"),
|
||||
post_screenshot_bytes=_jpeg("blue"),
|
||||
summary="Tapped Save after asking the explorer.",
|
||||
action_taken={"action": "click", "target": [540, 1200], "target_text": "Save"},
|
||||
last_execution_result={"status": "success"},
|
||||
operator_raw_thinking="Confirm the toggle first.",
|
||||
extra_metadata={"width": 1080, "height": 2400},
|
||||
)
|
||||
engine.record_trace(
|
||||
type="tool",
|
||||
name="ask_explorer",
|
||||
payload={"args": {"question": "toggle?"}, "result": "The toggle is ON."},
|
||||
step_id=engine.last_recorded_step_id,
|
||||
)
|
||||
engine.record_step(
|
||||
summary="Went back.",
|
||||
action_taken={"action": "press_key", "key": "back"},
|
||||
last_execution_result={"status": "success"},
|
||||
)
|
||||
engine.record_history_chunk(
|
||||
start_step_number=1,
|
||||
end_step_number=2,
|
||||
version=1,
|
||||
status="ready",
|
||||
band2=" - Steps 1–2: saved the toggle",
|
||||
band3="- Step 1 (T+00:00): click -> executed",
|
||||
rendered_text="[Chunk 1 | Steps 1–2]",
|
||||
)
|
||||
_flush(engine)
|
||||
return engine
|
||||
|
||||
|
||||
def _reader(engine) -> OfflineHistoryReader:
|
||||
return OfflineHistoryReader(engine.db_path, engine.global_base_dir, engine.current_session_id)
|
||||
|
||||
|
||||
def test_offline_reader_matches_live_friendly_steps(recorded):
|
||||
reader = _reader(recorded)
|
||||
assert isinstance(reader, HistoryReader)
|
||||
assert reader.session_start_time == recorded.session_start_time
|
||||
assert reader.base_dir == recorded.base_dir
|
||||
|
||||
live = recorded.get_agent_friendly_steps()
|
||||
offline = reader.get_agent_friendly_steps()
|
||||
assert offline == live
|
||||
assert offline[0]["tool_calls"][0]["name"] == "ask_explorer"
|
||||
# Coordinates normalized exactly like the live engine.
|
||||
assert offline[0]["action_taken"]["target"] == live[0]["action_taken"]["target"]
|
||||
assert offline[0]["relative_time"] == live[0]["relative_time"]
|
||||
|
||||
assert reader.get_agent_friendly_steps_in_range(2, 1) == live
|
||||
assert reader.get_agent_friendly_step(2) == live[1]
|
||||
assert reader.get_agent_friendly_step(9) is None
|
||||
|
||||
|
||||
def test_offline_reader_step_records_images_and_chunks(recorded):
|
||||
reader = _reader(recorded)
|
||||
record = reader.get_step_record(1)
|
||||
assert record is not None and record.step_number == 1
|
||||
assert reader.get_step_record(7) is None
|
||||
|
||||
pre = reader.get_step_image_path(1, "pre")
|
||||
post = reader.get_step_image_path(1, "post")
|
||||
assert pre is not None and pre.exists()
|
||||
assert post is not None and post.exists()
|
||||
assert pre == recorded.get_image_path(record.pre_image_name)
|
||||
assert reader.get_step_image_path(2, "pre") is None
|
||||
|
||||
chunks = reader.get_history_chunks()
|
||||
assert len(chunks) == 1
|
||||
assert chunks[0].start_step_number == 1
|
||||
|
||||
|
||||
def test_offline_reader_is_read_only(recorded):
|
||||
reader = _reader(recorded)
|
||||
with reader.storage._get_connection() as conn:
|
||||
with pytest.raises(sqlite3.OperationalError):
|
||||
conn.execute("DELETE FROM steps")
|
||||
# No directories or tables are created for a missing database.
|
||||
with pytest.raises(FileNotFoundError):
|
||||
OfflineHistoryReader(recorded.global_base_dir / "nope.db", recorded.global_base_dir, "x")
|
||||
@@ -702,3 +702,95 @@ async def test_exception_fail_open_releases_but_books_inconclusive(tmp_path):
|
||||
for r in read_ledger(tmp_path):
|
||||
assert r["status"] == "inconclusive"
|
||||
assert "checker blew up" in r["evidence"]
|
||||
|
||||
|
||||
# --- Attempt stream transcripts: bounded, append-only, replayable ------------------------
|
||||
|
||||
|
||||
def _seg(exec_id, text, role="answer", when=1.0):
|
||||
return {"execution_id": exec_id, "role": role, "when": when, "text": text}
|
||||
|
||||
|
||||
def test_clamp_keeps_short_transcripts_verbatim():
|
||||
from artemis.graph.checkpoints import clamp_stream_segments
|
||||
|
||||
segs = [_seg("a", "hello", "thought"), _seg("b", "world")]
|
||||
out, dropped = clamp_stream_segments(segs, limit=20)
|
||||
assert out == segs and dropped == 0
|
||||
assert out[0] is not segs[0] # copies, the input is never mutated
|
||||
|
||||
|
||||
def test_clamp_cuts_the_middle_once_and_keeps_head_and_tail():
|
||||
from artemis.graph.checkpoints import clamp_stream_segments
|
||||
|
||||
segs = [
|
||||
_seg("a", "A" * 40, "thought", when=1.0),
|
||||
_seg("b", "B" * 40, when=2.0),
|
||||
_seg("c", "C" * 40, when=3.0),
|
||||
_seg("d", "D" * 40, when=4.0),
|
||||
]
|
||||
out, dropped = clamp_stream_segments(segs, limit=100)
|
||||
|
||||
assert dropped == 60
|
||||
# head: 50 chars = all of a + 10 of b; tail: 50 chars = 10 of c + all of d
|
||||
assert [s["execution_id"] for s in out] == ["a", "b", "c", "d"]
|
||||
assert out[0]["text"] == "A" * 40 and out[0]["role"] == "thought"
|
||||
assert out[1]["text"] == "B" * 10 + "\n…[60 chars truncated]…\n"
|
||||
assert out[2]["text"] == "C" * 10
|
||||
assert out[3]["text"] == "D" * 40
|
||||
joined = "".join(s["text"] for s in out)
|
||||
assert joined.count("chars truncated") == 1
|
||||
assert len(joined) == 100 + len("\n…[60 chars truncated]…\n")
|
||||
# Timestamps and the other fields ride along untouched.
|
||||
assert [s["when"] for s in out] == [1.0, 2.0, 3.0, 4.0]
|
||||
|
||||
|
||||
def test_clamp_drops_segments_that_fall_entirely_into_the_cut():
|
||||
from artemis.graph.checkpoints import clamp_stream_segments
|
||||
|
||||
segs = [_seg("a", "A" * 10), _seg("mid", "M" * 100), _seg("z", "Z" * 10)]
|
||||
out, dropped = clamp_stream_segments(segs, limit=20)
|
||||
assert dropped == 100
|
||||
assert [s["execution_id"] for s in out] == ["a", "z"]
|
||||
assert out[0]["text"] == "A" * 10
|
||||
assert out[1]["text"] == "\n…[100 chars truncated]…\n" + "Z" * 10
|
||||
|
||||
|
||||
def test_clamp_single_huge_segment_keeps_both_ends():
|
||||
from artemis.graph.checkpoints import clamp_stream_segments
|
||||
|
||||
out, dropped = clamp_stream_segments([_seg("a", "H" * 30 + "M" * 100 + "T" * 30)], limit=60)
|
||||
assert dropped == 100
|
||||
assert len(out) == 1
|
||||
assert out[0]["text"] == "H" * 30 + "\n…[100 chars truncated]…\n" + "T" * 30
|
||||
|
||||
|
||||
def test_stream_records_round_trip_and_carry_truncation_flags(tmp_path):
|
||||
from artemis.graph.checkpoints import (
|
||||
STREAM_TEXT_LIMIT,
|
||||
append_attempt_stream_record,
|
||||
read_attempt_streams,
|
||||
streams_path,
|
||||
)
|
||||
|
||||
assert STREAM_TEXT_LIMIT == 20_000
|
||||
append_attempt_stream_record(
|
||||
tmp_path,
|
||||
{"attempt_id": "abc#1", "trace_id": "t1", "segments": [_seg("a", "short")]},
|
||||
)
|
||||
append_attempt_stream_record(
|
||||
tmp_path,
|
||||
{"attempt_id": "abc#2", "trace_id": "t2", "segments": [_seg("b", "x" * 50)]},
|
||||
limit=10,
|
||||
)
|
||||
with streams_path(tmp_path).open("a", encoding="utf-8") as f:
|
||||
f.write("not json\n\n")
|
||||
|
||||
records = read_attempt_streams(tmp_path)
|
||||
assert [r["attempt_id"] for r in records] == ["abc#1", "abc#2"]
|
||||
assert records[0]["truncated"] is False and records[0]["dropped_chars"] == 0
|
||||
assert records[0]["segments"] == [_seg("a", "short")]
|
||||
assert isinstance(records[0]["ts"], float)
|
||||
assert records[1]["truncated"] is True and records[1]["dropped_chars"] == 40
|
||||
assert records[1]["segments"][0]["text"] == "x" * 5 + "\n…[40 chars truncated]…\n" + "x" * 5
|
||||
assert read_attempt_streams(tmp_path / "nowhere") == []
|
||||
@@ -1,7 +1,7 @@
|
||||
"""Helper-tool routing in McpActionExecutor (Flash profile).
|
||||
|
||||
The Flash runner binds ``ask_explorer``, ``video_analyzer`` and
|
||||
``recall_history`` declarations next to its device actions; the executor must
|
||||
The Flash runner binds ``ask_explorer``, ``video_analyzer`` and the history
|
||||
tools (``search_history`` / ``replay_steps`` / ``get_step_screenshot``) next to its device actions; the executor must
|
||||
route them to the same implementations the Pro operator's LangChain tools
|
||||
delegate to, as agent-side text tools (no screenshot capture, no device
|
||||
action), and report their status explicitly instead of sniffing the text for
|
||||
@@ -38,7 +38,13 @@ def _state():
|
||||
|
||||
def test_helper_tools_are_not_device_actions():
|
||||
executor = _make_executor()
|
||||
assert {"ask_explorer", "video_analyzer", "recall_history"} <= AGENT_TOOL_NAMES
|
||||
assert {
|
||||
"ask_explorer",
|
||||
"video_analyzer",
|
||||
"search_history",
|
||||
"replay_steps",
|
||||
"get_step_screenshot",
|
||||
} <= AGENT_TOOL_NAMES
|
||||
assert not (AGENT_TOOL_NAMES & executor.action_tool_names)
|
||||
|
||||
|
||||
@@ -231,22 +237,32 @@ async def test_video_analyzer_exception_is_contained():
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# recall_history
|
||||
# history tools (search_history / replay_steps / get_step_screenshot)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _history_tool_double(return_value=None, side_effect=None):
|
||||
from artemis.tools.history import SearchHistoryArgs
|
||||
|
||||
tool = Mock()
|
||||
tool.args_schema = SearchHistoryArgs
|
||||
tool.execute = AsyncMock(return_value=return_value, side_effect=side_effect)
|
||||
return tool
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_recall_history_routes_to_the_shared_tool_with_text_result():
|
||||
async def test_history_tool_routes_to_the_shared_tool_with_text_result():
|
||||
executor = _make_executor()
|
||||
with patch("artemis.tools.history_recall.recall_history") as tool:
|
||||
tool.execute = AsyncMock(return_value="- Step 3 (T+00:30): tap btn3 -> executed")
|
||||
tool = _history_tool_double(return_value="- Step 3 (T+00:30): tap btn3 -> executed")
|
||||
with patch("artemis.mcp.action_executor.history_tool_by_name", return_value=tool) as by_name:
|
||||
result = await executor.execute(
|
||||
"recall_history",
|
||||
"search_history",
|
||||
{"query": "login Error timeout", "step_range": [1, 5], "unknown_arg": 1},
|
||||
"tc-recall",
|
||||
"tc-search",
|
||||
None,
|
||||
)
|
||||
|
||||
by_name.assert_called_once_with("search_history")
|
||||
tool.execute.assert_awaited_once_with(
|
||||
ctx=executor.ctx, query="login Error timeout", step_range=[1, 5]
|
||||
)
|
||||
@@ -257,27 +273,40 @@ async def test_recall_history_routes_to_the_shared_tool_with_text_result():
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_recall_history_forwards_multimodal_blocks():
|
||||
async def test_history_tool_forwards_multimodal_blocks():
|
||||
executor = _make_executor()
|
||||
blocks = [
|
||||
{"type": "text", "text": "Step 4 matched."},
|
||||
{"type": "text", "text": "Screenshot of step 4 (pre-action) is attached."},
|
||||
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,AAA"}},
|
||||
]
|
||||
with patch("artemis.tools.history_recall.recall_history") as tool:
|
||||
tool.execute = AsyncMock(return_value=blocks)
|
||||
result = await executor.execute(
|
||||
"recall_history", {"query": "settings", "include_images": True}, "tc", None
|
||||
)
|
||||
tool = _history_tool_double(return_value=blocks)
|
||||
with patch("artemis.mcp.action_executor.history_tool_by_name", return_value=tool):
|
||||
result = await executor.execute("get_step_screenshot", {"query": "unused"}, "tc", None)
|
||||
assert result.status == "success"
|
||||
assert result.text_summary == "Step 4 matched."
|
||||
assert result.text_summary == "Screenshot of step 4 (pre-action) is attached."
|
||||
assert result.raw_result == blocks
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_recall_history_exception_is_contained():
|
||||
async def test_history_tool_exception_is_contained():
|
||||
executor = _make_executor()
|
||||
with patch("artemis.tools.history_recall.recall_history") as tool:
|
||||
tool.execute = AsyncMock(side_effect=RuntimeError("db gone"))
|
||||
result = await executor.execute("recall_history", {"query": "x"}, "tc", None)
|
||||
tool = _history_tool_double(side_effect=RuntimeError("db gone"))
|
||||
with patch("artemis.mcp.action_executor.history_tool_by_name", return_value=tool):
|
||||
result = await executor.execute("replay_steps", {"query": "x"}, "tc", None)
|
||||
assert result.status == "success" # contained: the answer explains the failure
|
||||
assert result.text_summary == "recall_history failed: db gone"
|
||||
assert result.text_summary == "replay_steps failed: db gone"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_history_tools_dispatch_to_the_real_shared_instances():
|
||||
"""No per-tool branch: every history tool name resolves through the package."""
|
||||
executor = _make_executor() # ctx.data_engine is None -> tools degrade to text
|
||||
for name in ("search_history", "replay_steps", "get_step_screenshot"):
|
||||
result = await executor.execute(
|
||||
name, {"query": "x", "start_step": 1, "step_number": 1}, "tc", None
|
||||
)
|
||||
assert result.status == "success"
|
||||
assert (
|
||||
"no active execution history" in result.text_summary
|
||||
or "no execution history" in result.text_summary
|
||||
)
|
||||
@@ -77,6 +77,10 @@ def test_tool_signatures():
|
||||
assert "action" in sig_inspect.parameters
|
||||
assert "trace_id" in sig_inspect.parameters
|
||||
assert "step_number" in sig_inspect.parameters
|
||||
assert "query" in sig_inspect.parameters
|
||||
assert "step_range" in sig_inspect.parameters
|
||||
assert sig_inspect.parameters["max_results"].default == 5
|
||||
assert "'search'" in (mobile_inspect_trace.__doc__ or "")
|
||||
|
||||
|
||||
def test_mobile_run_task_invalid_model():
|
||||
@@ -501,3 +505,152 @@ def test_mobile_manage_task_stop_via_daemon(temp_trace_env, monkeypatch):
|
||||
mock_stop.assert_called_once_with(trace_id)
|
||||
assert res["status"] == "cancelled"
|
||||
assert trace_store.read_status(trace_id)["status"] == "cancelled"
|
||||
|
||||
|
||||
def _seed_trace(temp_dir: str, trace_id: str, *, with_image: bool = True) -> str:
|
||||
"""Writes one Flash-style step (with a tool trace and a pre screenshot)
|
||||
through the real storage layer, exactly as a live run would."""
|
||||
import io
|
||||
|
||||
from PIL import Image
|
||||
|
||||
from artemis.data_engine.models import (
|
||||
ImageRecord,
|
||||
SessionMetadata,
|
||||
StepRecord,
|
||||
TraceRecord,
|
||||
)
|
||||
from artemis.data_engine.storage import StorageManager
|
||||
|
||||
storage = StorageManager(os.path.join(temp_dir, "data_engine.db"), temp_dir)
|
||||
storage.create_session(
|
||||
SessionMetadata(session_id=trace_id, initial_goal="Replay test", start_time=1000.0)
|
||||
)
|
||||
image_name = None
|
||||
if with_image:
|
||||
image_name = "a" * 64
|
||||
images_dir = os.path.join(temp_dir, "images")
|
||||
os.makedirs(images_dir, exist_ok=True)
|
||||
buf = io.BytesIO()
|
||||
Image.new("RGB", (1080, 2400), "white").save(buf, format="JPEG")
|
||||
with open(os.path.join(images_dir, f"{image_name}.jpg"), "wb") as f:
|
||||
f.write(buf.getvalue())
|
||||
storage.create_image(
|
||||
ImageRecord(
|
||||
image_name=image_name,
|
||||
ocr_result=[{"text": "Save ocrneedle", "bounds": [0, 0, 10, 10]}],
|
||||
)
|
||||
)
|
||||
step_id = str(uuid.uuid4())
|
||||
storage.create_step(
|
||||
StepRecord(
|
||||
step_id=step_id,
|
||||
session_id=trace_id,
|
||||
step_number=1,
|
||||
timestamp=1012.0,
|
||||
pre_image_name=image_name,
|
||||
summary="Asked the explorer, then tapped Save.",
|
||||
action_taken={"action": "click", "target": [540, 1200], "target_text": "Save"},
|
||||
operator_raw_thinking="Confirm the toggle first.",
|
||||
last_execution_result={"status": "success"},
|
||||
extra_metadata={"width": 1080, "height": 2400},
|
||||
)
|
||||
)
|
||||
storage.create_trace(
|
||||
TraceRecord(
|
||||
trace_id=str(uuid.uuid4()),
|
||||
session_id=trace_id,
|
||||
step_id=step_id,
|
||||
type="tool",
|
||||
name="ask_explorer",
|
||||
status="success",
|
||||
timestamp=1011.0,
|
||||
duration=0.8,
|
||||
payload={"args": {"question": "toggle?"}, "result": "The toggle is ON. " * 100},
|
||||
)
|
||||
)
|
||||
return step_id
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mobile_inspect_trace_step_details_replays_tool_calls_for_flash(temp_trace_env):
|
||||
"""view_step_details works for Flash runs too and replays the recorded
|
||||
tool calls through the same friendly-step path the live DataEngine uses."""
|
||||
trace_id = str(uuid.uuid4())
|
||||
trace_store.init_trace(trace_id, "Replay test", "Flash", "conv-1", device_serial="pixel-7")
|
||||
_seed_trace(temp_trace_env, trace_id)
|
||||
|
||||
res = await mobile_inspect_trace(action="view_step_details", trace_id=trace_id, step_number=1)
|
||||
assert "error" not in res
|
||||
assert res["device_serial"] == "pixel-7"
|
||||
details = res["details"]
|
||||
assert "- **Step 1 (Start: 12.0s)**" in details
|
||||
assert "[Screen]: Asked the explorer, then tapped Save." in details
|
||||
assert "Confirm the toggle first." in details
|
||||
assert "`ask_explorer(" in details
|
||||
assert "The toggle is ON. " * 100 in details
|
||||
# Coordinates are replayed normalized, not as stored pixels.
|
||||
assert "[540, 1200]" not in details
|
||||
assert "Tapped 'Save'" in details
|
||||
|
||||
missing = await mobile_inspect_trace(
|
||||
action="view_step_details", trace_id=trace_id, step_number=9
|
||||
)
|
||||
assert missing["error"] == "Step not found"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mobile_inspect_trace_step_screenshots_draw_the_overlay(temp_trace_env):
|
||||
trace_id = str(uuid.uuid4())
|
||||
trace_store.init_trace(trace_id, "Shot test", "Flash", "conv-1", device_serial="pixel-7")
|
||||
_seed_trace(temp_trace_env, trace_id)
|
||||
|
||||
res = await mobile_inspect_trace(
|
||||
action="view_step_screenshots", trace_id=trace_id, step_number=1
|
||||
)
|
||||
assert "error" not in res
|
||||
assert res["before_screenshot"].startswith("file://")
|
||||
assert res["before_screenshot"].endswith(f"{'a' * 64}.jpg")
|
||||
assert res["after_screenshot"] is None
|
||||
overlay = res["action_overlay_screenshot"]
|
||||
assert overlay is not None
|
||||
overlay_path = overlay[len("file://") :]
|
||||
assert os.path.exists(overlay_path)
|
||||
assert overlay_path.endswith("step_1_overlay.jpg")
|
||||
# The overlay differs from the plain pre screenshot: the action was drawn.
|
||||
with open(overlay_path, "rb") as f, open(res["before_screenshot"][7:], "rb") as g:
|
||||
assert f.read() != g.read()
|
||||
|
||||
# Second call reuses the file it already wrote.
|
||||
again = await mobile_inspect_trace(
|
||||
action="view_step_screenshots", trace_id=trace_id, step_number=1
|
||||
)
|
||||
assert again["action_overlay_screenshot"] == overlay
|
||||
|
||||
no_step = await mobile_inspect_trace(
|
||||
action="view_step_screenshots", trace_id=trace_id, step_number=None
|
||||
)
|
||||
assert no_step["error"] == "Missing parameter"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mobile_inspect_trace_search_uses_the_shared_history_search(temp_trace_env):
|
||||
trace_id = str(uuid.uuid4())
|
||||
trace_store.init_trace(trace_id, "Search test", "Flash", "conv-1", device_serial="pixel-7")
|
||||
_seed_trace(temp_trace_env, trace_id)
|
||||
|
||||
res = await mobile_inspect_trace(action="search", trace_id=trace_id, query="toggle")
|
||||
assert "error" not in res
|
||||
assert res["device_serial"] == "pixel-7"
|
||||
assert "[Step 1 " in res["results"]
|
||||
assert "Screen: Asked the explorer, then tapped Save." in res["results"]
|
||||
|
||||
# On-screen OCR text of the step's screenshot is part of the search surface.
|
||||
res = await mobile_inspect_trace(action="search", trace_id=trace_id, query="ocrneedle")
|
||||
assert "[Step 1 " in res["results"]
|
||||
|
||||
ledger = await mobile_inspect_trace(action="search", trace_id=trace_id, step_range=[1, 1])
|
||||
assert "Action ledger for Steps 1–1" in ledger["results"]
|
||||
|
||||
empty = await mobile_inspect_trace(action="search", trace_id=trace_id)
|
||||
assert empty["error"] == "Missing parameter"
|
||||
@@ -358,7 +358,7 @@ def _hashes(boundary_after: int):
|
||||
|
||||
|
||||
def test_milestone_switch_closes_previous_segment_whole_and_gates_swap():
|
||||
"""Ready-gated swap (user decision 2026-09-01): a milestone switch closes
|
||||
"""Ready-gated swap: a milestone switch closes
|
||||
the previous segment whole and dispatches its capsule, but the original
|
||||
turns stay in the transcript until the capsule header is ready; the swap
|
||||
happens at the first render after readiness."""
|
||||
@@ -726,7 +726,7 @@ def test_era_overflow_compresses_oldest_era_to_period_paragraph():
|
||||
assert "Verified: " in match.group(1)
|
||||
# Recall guidance line is retained.
|
||||
assert (
|
||||
f" (Step-level ledger via recall_history for steps"
|
||||
f" (Step-level ledger via search_history for steps"
|
||||
f" {era.start_step_number}–{era.end_step_number})"
|
||||
) in frozen
|
||||
# Never-evict: the user injection survives even the period paragraph.
|
||||
@@ -799,7 +799,7 @@ def test_period_paragraph_degrades_to_milestone_list_when_all_band1_missing():
|
||||
|
||||
block = render_era_block(era)
|
||||
assert block.startswith(f"[Era 3 | Steps 1–8 | T+01:00 → T+08:59] {paragraph}")
|
||||
assert "(Step-level ledger via recall_history for steps 1–8)" in block
|
||||
assert "(Step-level ledger via search_history for steps 1–8)" in block
|
||||
assert 'User @ Step 6: "stay logged in"' in block
|
||||
|
||||
|
||||
@@ -1034,7 +1034,7 @@ async def test_flush_harvests_late_capsule_and_persists_ready_version():
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Ready-gated swap (user decision 2026-09-01): degradation ladder
|
||||
# Ready-gated swap: degradation ladder
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -1098,7 +1098,7 @@ def test_hard_threshold_force_swaps_pending_chunks_into_l3():
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Band ① note-linkage machine check (user decision 2026-09-01)
|
||||
# Band ① note-linkage machine check
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -1176,3 +1176,299 @@ def test_chunk_lists_every_step_of_a_multi_action_turn():
|
||||
assert chunk.source_step_ids == ["s1", "s2", "s3"]
|
||||
assert chunk.start_step_number == 1 and chunk.end_step_number == 3
|
||||
assert "- Step 2 (T+00:20)" in chunk.band3
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Capsule source = the operator's own transcript
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _rich_turn(i: int) -> list:
|
||||
"""A turn that asks the explorer before acting, with a UI list and a
|
||||
native thinking block — the parts the old per-step projection dropped."""
|
||||
from artemis.memory.transcript import PRO_UI_LIST_MARKER
|
||||
|
||||
return [
|
||||
HumanMessage(
|
||||
content=[
|
||||
{"type": "text", "text": f"# CURRENT OBSERVATION [T+00:{i % 60:02d}]"},
|
||||
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,IMG_{i}"}},
|
||||
{"type": "text", "text": f"{PRO_UI_LIST_MARKER}\n[0] Buy button"},
|
||||
]
|
||||
),
|
||||
AIMessage(
|
||||
content=[
|
||||
{"type": "thinking", "thinking": "need the price first"},
|
||||
{"type": "text", "text": "Ask the explorer for the price."},
|
||||
],
|
||||
tool_calls=[
|
||||
{
|
||||
"name": "ask_explorer",
|
||||
"args": {"question": "price?"},
|
||||
"id": f"tcx{i}",
|
||||
"type": "tool_call",
|
||||
}
|
||||
],
|
||||
),
|
||||
ToolMessage(tool_call_id=f"tcx{i}", content="Price shown: ¥39"),
|
||||
AIMessage(
|
||||
content=f"thinking {i}",
|
||||
tool_calls=[
|
||||
{"name": "click", "args": {"target": i}, "id": f"tc{i}", "type": "tool_call"}
|
||||
],
|
||||
),
|
||||
ToolMessage(tool_call_id=f"tc{i}", content="Action Recorded"),
|
||||
]
|
||||
|
||||
|
||||
def _payload_with_step(capsule: StubCapsuleService, step_number: int) -> dict:
|
||||
for payload in capsule._step_inputs.values():
|
||||
if any(s.get("step_number") == step_number for s in payload.get("steps", [])):
|
||||
return payload
|
||||
raise AssertionError(f"no capsule payload carries step {step_number}")
|
||||
|
||||
|
||||
def test_capsule_payload_carries_each_turn_transcript_as_the_operator_saw_it():
|
||||
"""The segment capsule digests exactly the text it replaces: every turn's
|
||||
post-scrub transcript — reasoning (native thinking included), every tool
|
||||
call with its arguments and returned result, the validator result — not
|
||||
a hand-picked projection. Scrubbed parts (UI list, screenshot) are gone
|
||||
from the source too, because the operator no longer saw them either."""
|
||||
steps = [_step(i) for i in range(1, 7)]
|
||||
ledger, chunker, _engine, capsule = _make(steps, min_active=2, max_steps=3)
|
||||
for i in range(1, 7):
|
||||
ledger.commit_staged(
|
||||
step_key=f"s{i - 1}" if i > 1 else None,
|
||||
validator_result={"status": "success"} if i > 1 else None,
|
||||
)
|
||||
ledger.render([_observation(i)])
|
||||
ledger.stage_turn(_rich_turn(i) if i == 2 else _turn(i))
|
||||
chunker.on_step_stamped(f"s{i}", "hash-a")
|
||||
|
||||
payload = _payload_with_step(capsule, 2)
|
||||
turn = next(t for t in payload["turns"] if t["steps"][0]["step_number"] == 2)
|
||||
transcript = turn["transcript"]
|
||||
|
||||
# What the operator saw during that turn, verbatim and in order.
|
||||
assert "(thinking) need the price first" in transcript
|
||||
assert "Ask the explorer for the price." in transcript
|
||||
assert '[tool call] ask_explorer({"question": "price?"})' in transcript
|
||||
assert "[tool result ask_explorer]\nPrice shown: ¥39" in transcript
|
||||
assert '[tool call] click({"target": 2})' in transcript
|
||||
assert "--- Action Execution Result" in transcript and "Status: success" in transcript
|
||||
assert transcript.index("Price shown") < transcript.index("click(")
|
||||
# What the scrub edge already removed from the operator's view.
|
||||
assert "Buy button" not in transcript and "IMG_2" not in transcript
|
||||
|
||||
# The lens request renders the turn block: step facts, then the transcript.
|
||||
lens = StepCapsuleLens(model_name="test", llm=object())
|
||||
text = lens.build_messages(payload)[1].content
|
||||
assert "## Step 2 (T+00:20)" in text
|
||||
assert "- Step 2 (T+00:20): " in text and "Visual transition of step 2." in text
|
||||
assert "Transcript as seen by the operator during this turn:" in text
|
||||
assert text.index("Recorded steps:") < text.index("Price shown: ¥39")
|
||||
assert "Reasoning excerpt" not in text # the transcript supersedes the excerpt
|
||||
|
||||
|
||||
def test_chunk_without_ledger_falls_back_to_per_step_projection():
|
||||
"""A chunk built without a live ledger (no transcripts) keeps the
|
||||
per-step fallback rendering so the capsule can still be produced."""
|
||||
steps = [_step(i) for i in range(1, 3)]
|
||||
_ledger, chunker, _engine, capsule = _make(steps)
|
||||
turns = [{"step_key": f"s{i}", "step_keys": [f"s{i}"], "start": 0, "end": 3} for i in (1, 2)]
|
||||
chunk = chunker._create_chunk(turns, None, {s["step_id"]: s for s in steps})
|
||||
assert chunk is not None
|
||||
payload = capsule._step_inputs[chunk.capsule_key]
|
||||
assert "turns" not in payload
|
||||
|
||||
text = StepCapsuleLens(model_name="test", llm=object()).build_messages(payload)[1].content
|
||||
assert "## Step 1 (T+00:10)" in text and "## Step 2 (T+00:20)" in text
|
||||
assert "Reasoning excerpt: thinking 1" in text
|
||||
assert "Transcript as seen by the operator" not in text
|
||||
|
||||
|
||||
def test_turn_transcript_is_capped_in_place_inside_the_capsule_request():
|
||||
lens = StepCapsuleLens(model_name="test", llm=object())
|
||||
cap = StepCapsuleLens.MAX_TURN_TRANSCRIPT_CHARS
|
||||
step = {"step_number": 4, "offset": "T+00:40", "action": "tap", "outcome": "executed"}
|
||||
payload = {
|
||||
"start_step": 4,
|
||||
"end_step": 4,
|
||||
"steps": [step],
|
||||
"turns": [{"steps": [step], "transcript": "x" * (cap + 500)}],
|
||||
}
|
||||
text = lens.build_messages(payload)[1].content
|
||||
assert "x" * (cap + 1) not in text
|
||||
assert "500 characters cut here" in text
|
||||
|
||||
|
||||
def test_capped_transcript_keeps_its_tail_so_the_validator_result_survives():
|
||||
"""The cut lands in the middle: the head (observation + reasoning) and the
|
||||
tail (the validator result that closes every turn) both stay verbatim."""
|
||||
cap = StepCapsuleLens.MAX_TURN_TRANSCRIPT_CHARS
|
||||
tail = "\n[observation]\n--- Action Execution Result (T+09:59) ---\nStatus: failed\nError: boom"
|
||||
transcript = "HEAD-START " + "h" * (cap + 3000) + tail
|
||||
capped = StepCapsuleLens._cap_transcript(transcript)
|
||||
assert capped.startswith("HEAD-START ")
|
||||
assert capped.endswith(tail)
|
||||
assert "characters cut here" in capped
|
||||
assert len(capped) <= cap + 200 # cap plus the announcement line
|
||||
|
||||
|
||||
def test_visual_transition_line_is_skipped_when_the_transcript_already_has_it():
|
||||
"""The resolved visual summary sits verbatim in the transcript once the
|
||||
scrub edge replaced the screenshot; only an unresolved turn (placeholder
|
||||
still there) gets the DataEngine summary as a separate line."""
|
||||
from artemis.agents.flash.context_compressor import HISTORY_SUMMARY_PREFIX
|
||||
|
||||
step = {
|
||||
"step_number": 4,
|
||||
"offset": "T+00:40",
|
||||
"action": "tap",
|
||||
"outcome": "executed",
|
||||
"visual_summary": "The cart badge changed from 0 to 1.",
|
||||
}
|
||||
resolved = f"[observation]\n{HISTORY_SUMMARY_PREFIX}The cart badge changed from 0 to 1."
|
||||
unresolved = "[observation]\n[screenshot]"
|
||||
for transcript, expected_count in ((resolved, 1), (unresolved, 1), ("", 1)):
|
||||
payload = {"start_step": 4, "end_step": 4, "steps": [step]}
|
||||
if transcript:
|
||||
payload["turns"] = [{"steps": [step], "transcript": transcript}]
|
||||
text = StepCapsuleLens(model_name="test", llm=object()).build_messages(payload)[1].content
|
||||
assert text.count("The cart badge changed from 0 to 1.") == expected_count, transcript
|
||||
resolved_text = (
|
||||
StepCapsuleLens(model_name="test", llm=object())
|
||||
.build_messages(
|
||||
{
|
||||
"start_step": 4,
|
||||
"end_step": 4,
|
||||
"steps": [step],
|
||||
"turns": [{"steps": [step], "transcript": resolved}],
|
||||
}
|
||||
)[1]
|
||||
.content
|
||||
)
|
||||
assert "Visual transition:" not in resolved_text
|
||||
unresolved_text = (
|
||||
StepCapsuleLens(model_name="test", llm=object())
|
||||
.build_messages(
|
||||
{
|
||||
"start_step": 4,
|
||||
"end_step": 4,
|
||||
"steps": [step],
|
||||
"turns": [{"steps": [step], "transcript": unresolved}],
|
||||
}
|
||||
)[1]
|
||||
.content
|
||||
)
|
||||
assert "Visual transition: The cart badge" in unresolved_text
|
||||
|
||||
|
||||
def test_size_trigger_measures_the_rendered_transcript_the_capsule_receives():
|
||||
"""``turn_text_chars`` sizes the exact text the lens gets — tool-call
|
||||
arguments included — so the source-token trigger and the capsule payload
|
||||
never disagree about how big a segment is."""
|
||||
steps = [_step(i) for i in range(1, 5)]
|
||||
ledger, chunker, _engine, capsule = _make(steps, min_active=1, target_tokens=10**9)
|
||||
_run_turns(ledger, chunker, 1, 4)
|
||||
turns = ledger.unchunked_turns()
|
||||
assert ledger.turn_text_chars(turns) == sum(len(ledger.turn_transcript(t)) for t in turns)
|
||||
assert ledger.turn_text_chars(turns[:1]) == len(ledger.turn_transcript(turns[0]))
|
||||
assert '[tool call] click({"target": 1})' in ledger.turn_transcript(turns[0])
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ready_capsule_releases_its_turn_transcripts():
|
||||
"""Once a capsule landed the job is never re-run, so the service drops the
|
||||
bulky ``turns`` from the retained payload (the flat step facts stay)."""
|
||||
from artemis.memory.chunking import ChunkCapsuleService
|
||||
|
||||
class EchoLens(StepCapsuleLens):
|
||||
async def render(self, key, payload):
|
||||
return json.dumps(_capsule(1, 1))
|
||||
|
||||
service = ChunkCapsuleService(None, EchoLens(model_name="test", llm=object()))
|
||||
payload = {
|
||||
"start_step": 1,
|
||||
"end_step": 1,
|
||||
"step_number": 1,
|
||||
"steps": [{"step_number": 1}],
|
||||
"turns": [{"steps": [{"step_number": 1}], "transcript": "big text"}],
|
||||
}
|
||||
service.submit("chunk:1-1", payload)
|
||||
await service.flush()
|
||||
assert service.has_summary("chunk:1-1")
|
||||
assert "turns" not in service.get_job_payload("chunk:1-1")
|
||||
assert service.get_job_payload("chunk:1-1")["steps"] == [{"step_number": 1}]
|
||||
|
||||
|
||||
def test_multi_action_turn_renders_one_block_with_every_recorded_step():
|
||||
lens = StepCapsuleLens(model_name="test", llm=object())
|
||||
steps = [
|
||||
{"step_number": n, "offset": f"T+00:{n}0", "action": f"tap btn{n}", "outcome": "executed"}
|
||||
for n in (7, 8)
|
||||
]
|
||||
payload = {"start_step": 7, "end_step": 8, "steps": steps}
|
||||
payload["turns"] = [{"steps": steps, "transcript": "[operator]\nfired two taps"}]
|
||||
text = lens.build_messages(payload)[1].content
|
||||
assert "## Steps 7–8 (T+00:70 → T+00:80)" in text
|
||||
assert "- Step 7 (T+00:70): tap btn7 -> executed" in text
|
||||
assert "- Step 8 (T+00:80): tap btn8 -> executed" in text
|
||||
assert "fired two taps" in text
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Band ② must not restate band ③
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_band2_coordinates_are_machine_checked():
|
||||
"""A coordinate literal in any band-② interval text fails the attempt
|
||||
(regenerate), same rank as the coverage check: band ③ already carries
|
||||
every action's target, so ② repeating it is the ledger written twice."""
|
||||
from artemis.memory.chunking import band2_intervals_carry_coordinates
|
||||
|
||||
lens = StepCapsuleLens(model_name="test", llm=object())
|
||||
payload = {"start_step": 3, "end_step": 6, "steps": []}
|
||||
|
||||
with_coords = dict(
|
||||
_capsule(3, 6),
|
||||
intervals=[
|
||||
{"start_step": 3, "end_step": 5, "text": "opened three sections in turn"},
|
||||
{"start_step": 6, "end_step": 6, "text": "tapped 'Battery' at [320, 399]"},
|
||||
],
|
||||
)
|
||||
assert band2_intervals_carry_coordinates(with_coords["intervals"])
|
||||
assert lens.parse_capsule(json.dumps(with_coords), payload) is None
|
||||
|
||||
swipe_literal = dict(
|
||||
_capsule(3, 6),
|
||||
intervals=[{"start_step": 3, "end_step": 6, "text": "swiped [556, 289, 556, 124]"}],
|
||||
)
|
||||
assert lens.parse_capsule(json.dumps(swipe_literal), payload) is None
|
||||
|
||||
behavior_only = dict(
|
||||
_capsule(3, 6),
|
||||
intervals=[
|
||||
{
|
||||
"start_step": 3,
|
||||
"end_step": 6,
|
||||
"text": "Opened 'Apps' and 'Battery' and returned to the main list each time;"
|
||||
" the Battery page showed 100% and 3 rows.",
|
||||
}
|
||||
],
|
||||
)
|
||||
assert not band2_intervals_carry_coordinates(behavior_only["intervals"])
|
||||
assert lens.parse_capsule(json.dumps(behavior_only), payload) is not None
|
||||
|
||||
|
||||
def test_capsule_prompt_forbids_band2_coordinates_and_sets_a_length_target():
|
||||
"""The prompt carries the ② contract (behavior + observed result, no
|
||||
coordinates, homogeneous runs merged) and the ①+② soft length target,
|
||||
while keeping the preserve-don't-abstract rule."""
|
||||
prompt = StepCapsuleLens(model_name="test", llm=object())._prompt
|
||||
assert "NEVER include coordinates" in prompt
|
||||
assert "NEVER narrate step by step" in prompt
|
||||
assert "Merge homogeneous consecutive actions" in prompt
|
||||
assert "one third of the source text" in prompt
|
||||
assert "Preserve, don't abstract" in prompt
|
||||
@@ -320,3 +320,101 @@ def test_commit_records_every_step_key_of_a_multi_action_turn():
|
||||
ledger.stage_turn(_turn(2))
|
||||
ledger.commit_staged(step_key=None)
|
||||
assert ledger.unchunked_turns()[1]["step_keys"] == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Turn transcript rendering (chunk-capsule source)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_render_turn_transcript_is_lossless_over_text_thinking_tools_and_results():
|
||||
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage, ToolMessage
|
||||
|
||||
from artemis.memory.transcript import SCREENSHOT_PLACEHOLDER, render_turn_transcript
|
||||
|
||||
messages = [
|
||||
HumanMessage(
|
||||
content=[
|
||||
{"type": "text", "text": "# CURRENT OBSERVATION [T+01:00]"},
|
||||
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,IMG"}},
|
||||
]
|
||||
),
|
||||
AIMessage(
|
||||
content=[
|
||||
{"type": "thinking", "thinking": "compare the two prices"},
|
||||
{"type": "text", "text": "I will recall the earlier price."},
|
||||
],
|
||||
tool_calls=[
|
||||
{
|
||||
"name": "search_history",
|
||||
"args": {"query": "price"},
|
||||
"id": "c1",
|
||||
"type": "tool_call",
|
||||
}
|
||||
],
|
||||
),
|
||||
ToolMessage(tool_call_id="c1", content="Step 3: price ¥39", status="success"),
|
||||
ToolMessage(
|
||||
tool_call_id="c2", name="adb_shell", content="permission denied", status="error"
|
||||
),
|
||||
ToolMessage(tool_call_id="c9", content="orphan result"),
|
||||
SystemMessage(content="You have reached the maximum number of tool calls"),
|
||||
AIMessage(
|
||||
content="tap now",
|
||||
tool_calls=[{"name": "click", "args": {"target": [0.5, 0.5]}, "id": "c3"}],
|
||||
),
|
||||
]
|
||||
text = render_turn_transcript(messages)
|
||||
assert text.splitlines()[0] == "[observation]"
|
||||
assert "# CURRENT OBSERVATION [T+01:00]" in text
|
||||
assert SCREENSHOT_PLACEHOLDER in text and "IMG" not in text
|
||||
assert "[operator]\n(thinking) compare the two prices\nI will recall the earlier price." in text
|
||||
assert '[tool call] search_history({"query": "price"})' in text
|
||||
# A Pro ToolMessage carries no name: resolved through the turn's own call.
|
||||
assert "[tool result search_history]\nStep 3: price ¥39" in text
|
||||
assert "[tool result adb_shell (error)]\npermission denied" in text
|
||||
assert "[tool result c9]\norphan result" in text # unresolvable: id shown
|
||||
assert "[system]\nYou have reached the maximum number of tool calls" in text
|
||||
assert '[tool call] click({"target": [0.5, 0.5]})' in text
|
||||
# Order is message order.
|
||||
assert text.index("search_history(") < text.index("price ¥39") < text.index("tap now")
|
||||
|
||||
|
||||
def test_ledger_turn_transcript_reflects_the_scrubbed_active_region():
|
||||
"""The transcript handed to chunk compression is the turn as it stands
|
||||
after the scrub edge: the UI list stripped, the validator result present."""
|
||||
from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
|
||||
|
||||
from artemis.memory.transcript import PRO_UI_LIST_MARKER, TranscriptLedger
|
||||
|
||||
ledger = TranscriptLedger(clock=lambda: 0.0)
|
||||
|
||||
def turn(i: int) -> list:
|
||||
return [
|
||||
HumanMessage(
|
||||
content=[
|
||||
{"type": "text", "text": f"# CURRENT OBSERVATION [T+00:0{i}]"},
|
||||
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,IMG{i}"}},
|
||||
{"type": "text", "text": f"{PRO_UI_LIST_MARKER}\n[0] Button {i}"},
|
||||
]
|
||||
),
|
||||
AIMessage(
|
||||
content=f"thought {i}",
|
||||
tool_calls=[{"name": "click", "args": {}, "id": f"t{i}"}],
|
||||
),
|
||||
ToolMessage(tool_call_id=f"t{i}", content="Action Recorded"),
|
||||
]
|
||||
|
||||
ledger.stage_turn(turn(1))
|
||||
ledger.commit_staged(step_key="s1", validator_result={"status": "success"})
|
||||
ledger.stage_turn(turn(2))
|
||||
ledger.commit_staged(step_key="s2", validator_result={"status": "failed", "error": "boom"})
|
||||
ledger.render([HumanMessage(content=[{"type": "text", "text": "tail"}])])
|
||||
|
||||
first, second = ledger.unchunked_turns()
|
||||
text = ledger.turn_transcript(first)
|
||||
assert "thought 1" in text and "[tool call] click({})" in text
|
||||
assert "Button 1" not in text # depth-1 strip already applied
|
||||
assert "--- Action Execution Result (T+00:00) ---\nStatus: success" in text
|
||||
assert "Status: failed" in ledger.turn_transcript(second)
|
||||
assert "thought 2" not in text # a turn transcript never leaks into another turn
|
||||
@@ -0,0 +1,41 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""The Pro tuning summary the SDK persists with every session's device_info."""
|
||||
|
||||
from artemis.sdk.agent import run_tuning_summary
|
||||
from artemis.sdk.builders.agent_config_builder import AgentConfigBuilder
|
||||
|
||||
|
||||
def test_flash_runs_persist_no_tuning():
|
||||
config = AgentConfigBuilder().build(validate_profiles=False)
|
||||
assert run_tuning_summary(config, "flash") is None
|
||||
|
||||
|
||||
def test_pro_runs_persist_the_two_launcher_sliders():
|
||||
config = (
|
||||
AgentConfigBuilder()
|
||||
.with_verification_level("checkpoints")
|
||||
.with_explorer(pro_mode="ultra")
|
||||
.build(validate_profiles=False)
|
||||
)
|
||||
assert run_tuning_summary(config, "pro") == {
|
||||
"verification_level": "checkpoints",
|
||||
"explorer_mode": "ultra",
|
||||
}
|
||||
|
||||
|
||||
def test_pro_summary_reflects_checker_switched_off():
|
||||
config = AgentConfigBuilder().with_verification_level("off").build(validate_profiles=False)
|
||||
assert run_tuning_summary(config, "pro")["verification_level"] == "off"
|
||||
Loaded 100 of 116 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user