refactor: unify agent history and harden runtime diagnostics

Preserve user guidance through history compression, stabilize Flash index targets, and retire withdrawn checks without applying stale verdicts.

Improve runtime diagnostics, tool responses, and history rendering. Simplify documentation and comments.

Validation: 2076 tests passed in the full suite; 381 affected tests and 26 frontend tests passed after cleanup. Ruff and the quality ratchet passed.
This commit is contained in:
somew1nd committed 2026-09-09 21:35:40 -07:00
1 parent e983180b5b
commit 5d92939d91
127 files changed
+13674 -1924

No files matched your search

+16 -15
View File
@@ -33,17 +33,17 @@
## Key Highlights ## Key Highlights
* **Cross-App Automation & Autonomous AI Assistant**: Operates not just as a robust testing framework, but as an autonomous agent capable of handling complex cross-app workflows and daily tasks via natural language; * **Cross-App Automation**: Executes testing workflows and everyday tasks on Android from natural language instructions.
* **Zero-Maintenance Test Automation**: Built upon a "Dynamic-First, Coordinate-Fallback" multimodal locating engine, eliminating fragile XPath/ID selector maintenance and remaining resilient to UI redesigns, system updates, and resolution drift; * **Multimodal Targeting**: Uses element indices when available, with coordinate and visual locating fallbacks for custom interfaces.
* **One-Click Bug Repro & Logcat Diagnostics in IDE**: Native **Model Context Protocol (MCP)** integration allows **Antigravity, Claude Code, and Windsurf** to drive physical test devices via natural language, automatically capturing crash stacks from **Logcat** and keyframe screenshots; * **IDE Diagnostics**: **Model Context Protocol (MCP)** integration lets **Antigravity, Claude Code, and Windsurf** drive test devices and collect **Logcat** output and screenshots.
* **Ultra-Fast Execution (3–5s per Step)**: Pioneered an **Optimistic Asynchronous Pipeline** that completely decouples UI interaction from heavy LLM reasoning, achieving rapid regression throughput in Flash mode; * **Flash Execution**: A reactive observe-and-act loop with asynchronous history summaries, typically **3–5s per step**.
* **Popup Self-Healing & 10+ Hour Exploration**: Proprietary **Safety Net** double-checks targets before action execution to intercept and clear interfering system popups; Pro mode supports **10+ hours** of continuous exploratory & monkey-plus stability testing; * **Pro Exploration**: Checks targets before individual actions and returns blocked actions to the Operator for recovery. Supports long-running exploratory and stability tests.
* **Industry-Leading SOTA**: Achieved **99%+ task completion** on Google Research's **AndroidWorld** benchmark (100+ complex multi-step tasks). * **AndroidWorld Results**: **99%+ task completion** on Google Research's **AndroidWorld** benchmark (100+ multi-step tasks).
<a id="workflow-showcase"></a> <a id="workflow-showcase"></a>
## Antigravity × ARTEMIS: Autonomous Testing Workflow ## Antigravity × ARTEMIS: Autonomous Testing Workflow
Experience seamless collaboration between **Antigravity** and **ARTEMIS** via native MCP integration — taking you from a natural language requirement to a production-grade diagnostic report in four automated steps: **Antigravity** uses **ARTEMIS** through MCP to turn a test request into a plan, device execution, and a diagnostic report:
<table width="100%"> <table width="100%">
<tr> <tr>
@@ -65,7 +65,7 @@ Experience seamless collaboration between **Antigravity** and **ARTEMIS** via na
<img src="./docs/assets/workflow-3-exec.png" width="100%" alt="Step 3: Autonomous Test Execution" /> <img src="./docs/assets/workflow-3-exec.png" width="100%" alt="Step 3: Autonomous Test Execution" />
</td> </td>
<td width="50%" align="center"> <td width="50%" align="center">
<b>4. Comprehensive Final Report</b><br> <b>4. Final Report</b><br>
<sub>Delivers structured audit findings, metric tables, and raw datasets</sub><br><br> <sub>Delivers structured audit findings, metric tables, and raw datasets</sub><br><br>
<img src="./docs/assets/workflow-4-report.png" width="100%" alt="Step 4: Final Report" /> <img src="./docs/assets/workflow-4-report.png" width="100%" alt="Step 4: Final Report" />
</td> </td>
@@ -159,7 +159,8 @@ PYTHONPATH = "/path/to/artemis"
"mobile_run_task": { "eager": true }, "mobile_run_task": { "eager": true },
"mobile_manage_task": { "eager": true }, "mobile_manage_task": { "eager": true },
"mobile_get_device_state": { "eager": true }, "mobile_get_device_state": { "eager": true },
"mobile_inspect_trace": { "eager": true } "mobile_inspect_trace": { "eager": true },
"mobile_diagnose": { "eager": true }
} }
} }
} }
@@ -246,14 +247,14 @@ if __name__ == "__main__":
</p> </p>
* **Web Visual Test Console (`uv run artemis ui`)**: Real-time screen projection and interactive panel, supporting natural language test dispatch, live reasoning telemetry, action trajectories, and execution replay; manage server lifecycle anytime from any terminal using `uv run artemis restart`, `uv run artemis stop`, and `uv run artemis status`; * **Web Visual Test Console (`uv run artemis ui`)**: Real-time screen projection and interactive panel, supporting natural language test dispatch, live reasoning telemetry, action trajectories, and execution replay; manage server lifecycle anytime from any terminal using `uv run artemis restart`, `uv run artemis stop`, and `uv run artemis status`;
* **Native MCP Protocol (IDE Collaboration)**: Operates as a standard MCP server seamlessly integrating with **Antigravity, Claude Code, Windsurf**, etc., directly driving real devices inside the IDE to verify bugs and run test cases; * **MCP Server**: Connects **Antigravity, Claude Code, Windsurf**, and other MCP clients to real devices for bug reproduction and test execution;
* **Developer CLI (`uv run artemis run`)**: Direct terminal execution for automated test cases, exploratory stability inspection, or AndroidWorld benchmarks with high-fidelity structured terminal output; * **Developer CLI (`uv run artemis run`)**: Direct terminal execution for automated test cases, exploratory stability inspection, or AndroidWorld benchmarks with high-fidelity structured terminal output;
* **Python SDK**: Integrates as a standard Python library into existing automated testing frameworks (e.g., pytest) or CI/CD pipelines with strongly typed Pydantic structured outputs and assertion support. * **Python SDK**: Integrates as a standard Python library into existing automated testing frameworks (e.g., pytest) or CI/CD pipelines with strongly typed Pydantic structured outputs and assertion support.
<a id="benchmarks"></a> <a id="benchmarks"></a>
## Benchmarks: AndroidWorld (SOTA 99%+) ## Benchmarks: AndroidWorld (SOTA 99%+)
Evaluated on [AndroidWorld](https://github.com/google-research/android_world) — Google Research's gold-standard benchmark spanning 20+ real apps and 100+ complex multi-step tasks: **Artemis demonstrated exceptional robustness across the entire benchmark suite, achieving a 99%+ completion rate.** Artemis achieved a **99%+ completion rate** on [AndroidWorld](https://github.com/google-research/android_world), Google Research's benchmark spanning 20+ apps and 100+ multi-step tasks.
<p align="center"> <p align="center">
<img src="./docs/assets/androidworld_leaderboard.png?v=2" alt="AndroidWorld Benchmark Comparison" width="100%" /> <img src="./docs/assets/androidworld_leaderboard.png?v=2" alt="AndroidWorld Benchmark Comparison" width="100%" />
@@ -261,9 +262,9 @@ Evaluated on [AndroidWorld](https://github.com/google-research/android_world)
## How ARTEMIS is Architected ## How ARTEMIS is Architected
* **Pre-Touch Pixel Gate & Speculative Chaining**: Eliminates "silent misclicks" from inference latency race conditions. Milliseconds before dispatch, a local UI guard intercepts unexpected dialogs (0 tokens, 0 cloud wait), a Micro-ROI gate verifies target stability, and speculative chained taps hit transient UI (e.g. auto-fading video controls) before they expire; * **Pre-Execution Checks and Action Bursts**: Pro checks the target against the live UI tree and pixels before dispatching an individual action. Action bursts handle transient controls without waiting for another model turn.
* **Three-Layer Progressive Grounding Engine**: Fuses local OCR with accessibility hierarchies (~150ms, 0 tokens) to drive 85%+ of standard actions via drift-free numeric indices, gracefully falling back to spatial vision models for custom Canvas/Compose/Flutter UI and sandboxed CV probing for subtle pixel states; * **Element Locating**: Combines accessibility hierarchies and OCR with visual models for custom Canvas, Compose, and Flutter interfaces.
* **Elastic Dual Engine with In-Flight Context Compactor**: Seamlessly toggles between high-throughput reactive CI loops (Flash Mode, 3–5s/step) and multi-step cognitive state graphs (Pro Mode), using background visual deltas and DOM pruning to slash token consumption by >70% for 10+ hours of continuous, unattended soak testing. * **Shared History Compression**: Flash and Pro replace older screenshots with visual summaries and compress completed steps into searchable history chunks. Context thresholds control when raw turns are replaced.
<p align="center"> <p align="center">
<img src="./docs/assets/artemis_architecture_diagram.png" alt="ARTEMIS System Architecture Diagram" width="100%" /> <img src="./docs/assets/artemis_architecture_diagram.png" alt="ARTEMIS System Architecture Diagram" width="100%" />
@@ -274,7 +275,7 @@ Evaluated on [AndroidWorld](https://github.com/google-research/android_world)
ARTEMIS supports two execution profiles tailored for different automation requirements: ARTEMIS supports two execution profiles tailored for different automation requirements:
* **Flash Profile (`--profile flash`)**: Fast and token-efficient reactive loop (~3–5s per step): one model observes the live screen, thinks, and acts, with no graph orchestration. Ideal for routine, deterministic UI tasks. The loop is unbounded by default (`agent.flash.max_turns`, 0 = unlimited) because history is compressed rather than capped: Flash shares the Pro session transcript ledger (session-relative `T+mm:ss` clock, screenshots folded into visual summaries, older steps chunked into eras and recallable on demand via `search_history` / `replay_steps`) and can query the session recording through `video_analyzer`. Transient UI (auto-fading control bars, toasts) is handled by chaining taps into one `click_sequence`. *Limitations*: No task plan or notes, no pre-execution safety net, no checkpoint verification or final report, and no ADB shell. * **Flash Profile (`--profile flash`)**: Fast and token-efficient reactive loop (~3–5s per step): one model observes the live screen, thinks, and acts, with no graph orchestration. Ideal for routine, deterministic UI tasks. The loop is unbounded by default (`agent.flash.max_turns`, 0 = unlimited) because history is compressed rather than capped: Flash shares the Pro session transcript ledger (session-relative `T+mm:ss` clock, screenshots folded into visual summaries, older steps chunked into eras and recallable on demand via `search_history` / `replay_steps`) and can query the session recording through `video_analyzer`. Transient UI (auto-fading control bars, toasts) is handled by chaining taps into one `click_sequence`. *Limitations*: No task plan or notes, no pre-execution safety net, no checkpoint verification or final report, and no ADB shell.
* **Pro Profile (`--profile pro`)**: Our most capable architecture (~15–40s per step), built as a multi-agent graph. A **Planner** maintains a living Markdown task plan with milestones and `verify` / `assert` check items; the **Operator** executes it with the full toolset (Explorer grounding whose `flash` / `pro` / `ultra` tier is a user setting per profile — `pro.explorer.mode` / `flash.explorer_mode` in `config/artemis.jsonc` or `--explorer-pro-mode` — never chosen by the agent; notes, history recall, video analysis, ADB diagnostics). Every single action passes a pre-execution **Safety Net** (XML-first, pixel fallback), while multi-action **fast-action bursts** fire back to back to beat turn latency on transient UI. A blocked or failed action opens an **execution incident** that stays in the Operator's context until a later action succeeds, so recovery is handled by the Operator itself with no separate repair agent. A read-only **Checker** verifies plan checkpoints and runs an exit final review against the original goal (`--verification-level`: `off` / `final` (default) / `checkpoints` / `strict`), and plan milestone edits get an advisory review. Handles 100+ step long-horizon workflows, `[Loop:continuous]` monitoring, and an optional written report. * **Pro Profile (`--profile pro`)**: A planning and verification workflow (~15–40s per step), built as a multi-agent graph. A **Planner** maintains a living Markdown task plan with milestones and `verify` / `assert` check items; the **Operator** executes it with the full toolset (Explorer grounding whose `flash` / `pro` / `ultra` tier is a user setting per profile — `pro.explorer.mode` / `flash.explorer_mode` in `config/artemis.jsonc` or `--explorer-pro-mode` — never chosen by the agent; notes, history recall, video analysis, ADB diagnostics). Every single action passes a pre-execution **Safety Net** (XML-first, pixel fallback), while multi-action **fast-action bursts** fire back to back to beat turn latency on transient UI. A blocked or failed action opens an **execution incident** that stays in the Operator's context until a later action succeeds, so recovery is handled by the Operator itself with no separate repair agent. A read-only **Checker** verifies plan checkpoints and runs an exit final review against the original goal (`--verification-level`: `off` / `final` (default) / `checkpoints` / `strict`), and plan milestone edits get an advisory review. Handles 100+ step long-horizon workflows, `[Loop:continuous]` monitoring, and an optional written report.
## Roadmap ## Roadmap
+17 -16
View File
@@ -33,18 +33,18 @@
## 核心亮点 ## 核心亮点
* **跨 App 复杂自动化与 AI 个人助理**:不仅能执行严格的自动化测试用例,更具备接管真机的自主决策与长流程操作能力,一句话处理跨应用复杂业务与任务; * **跨 App 自动化**:根据自然语言指令,在 Android 上执行测试流程和日常任务。
* **零脚本维护自动化测试**:基于“动态优先、坐标兜底”的多模态语义定位,彻底告别传统 XPath / 控件 ID 频繁失效的痛点,无惧 App 改版、分辨率差异与系统升级; * **多模态定位**:优先使用元素索引,对自绘界面提供坐标和视觉定位兜底。
* **IDE 内一键 Bug 复现与 Logcat 诊断**:原生支持 **MCP 协议**,可在 **Antigravity、Claude Code、Windsurf** 中直接用自然语言驱动真机复现缺陷,自动抓取关键帧截图与 **Logcat 崩溃堆栈**,完成研发测试闭环; * **IDE 内诊断**:通过 **MCP 协议**,让 **Antigravity、Claude Code、Windsurf** 操作测试设备,收集 **Logcat** 输出和截图。
* **极速执行吞吐(单步 3–5 秒)**:首创**全链路乐观异步流水线 (Optimistic Async Pipeline)**,解耦重度推理与页面操作,在 Flash 模式下实现丝滑的高频交互与快速回归验证; * **Flash 执行**:使用观察、执行循环和异步历史摘要,单步通常约 **3–5 秒**。
* **干扰弹窗自愈与超长程巡检**:独创 **Safety Net** 执行前校验机制,自动识别并清除系统权限弹窗、通知遮挡等异常;Pro 模式支持连续 **10+ 小时** 无人值守稳定性巡检与探索性测试 (Exploratory / Monkey-plus Testing); * **Pro 探索**:单个动作执行前校验目标,被拦截的动作交回 Operator 处理。支持长时间探索和稳定性测试。
* **业界顶尖 SOTA**:在 Google Research **AndroidWorld** 基准评测(100+ 复杂长程系统与应用交互任务)中取得 **99%+ 任务完成率**。 * **AndroidWorld 结果**:在 Google Research **AndroidWorld** 基准评测(100+ 多步任务)中取得 **99%+ 任务完成率**。
<a id="workflow-showcase"></a> <a id="workflow-showcase"></a>
<a id="全流程演示"></a> <a id="全流程演示"></a>
## Antigravity × ARTEMIS:全流程自主测试演示 ## Antigravity × ARTEMIS:全流程自主测试演示
通过原生 MCP 协议,**Antigravity** 与 **ARTEMIS** 深度协同——只需一句自然语言指令,即可自动完成从需求理解、测试规划、真机执行到深度报告输出的完整闭环: **Antigravity** 通过 MCP 调用 **ARTEMIS**,将测试需求转为计划、真机操作和诊断报告:
<table width="100%"> <table width="100%">
<tr> <tr>
@@ -161,7 +161,8 @@ PYTHONPATH = "/path/to/artemis"
"mobile_run_task": { "eager": true }, "mobile_run_task": { "eager": true },
"mobile_manage_task": { "eager": true }, "mobile_manage_task": { "eager": true },
"mobile_get_device_state": { "eager": true }, "mobile_get_device_state": { "eager": true },
"mobile_inspect_trace": { "eager": true } "mobile_inspect_trace": { "eager": true },
"mobile_diagnose": { "eager": true }
} }
} }
} }
@@ -247,15 +248,15 @@ if __name__ == "__main__":
</p> </p>
* **Web 可视化测试控制台 (`uv run artemis ui`)**:集成设备实时投屏与交互面板,支持通过自然语言下发测试用例,实时观测推理步骤、操作轨迹、截图留存与异常状态回放;支持在任意终端使用 `uv run artemis restart`、`uv run artemis stop`、`uv run artemis status` 一键重启、关停或查看服务状态; * **Web 可视化测试控制台 (`uv run artemis ui`)**:集成设备实时投屏与交互面板,支持通过自然语言下发测试用例,实时观测推理步骤、操作轨迹、截图留存与异常状态回放;支持在任意终端使用 `uv run artemis restart`、`uv run artemis stop`、`uv run artemis status` 一键重启、关停或查看服务状态;
* **原生 MCP 协议 (IDE 协同)**:作为标准 MCP 服务器无缝接入 **Antigravity、Claude Code、Windsurf** 等开发环境,在 IDE 中直接驱动真机完成自动化测试与 Bug 复现验证; * **原生 MCP 协议 (IDE 协同)**:作为标准 MCP 服务器接入 **Antigravity、Claude Code、Windsurf** 等开发环境,在 IDE 中直接驱动真机完成自动化测试与 Bug 复现验证;
* **开发者命令行 CLI (`uv run artemis run`)**:支持通过终端直接执行自动化测试用例、探索性稳定性巡检或 AndroidWorld 基准评测,提供高保真结构化终端输出; * **开发者命令行 CLI (`uv run artemis run`)**:支持通过终端直接执行自动化测试用例、探索性稳定性巡检或 AndroidWorld 基准评测,提供高保真结构化终端输出;
* **Python SDK**:作为标准 Python 库无缝集成至现有自动化测试框架(如 pytest)或 CI/CD 流水线,提供基于 Pydantic 的强类型结构化结果与断言支持。 * **Python SDK**:作为标准 Python 库集成至现有自动化测试框架(如 pytest)或 CI/CD 流水线,提供基于 Pydantic 的强类型结构化结果与断言支持。
<a id="benchmarks"></a> <a id="benchmarks"></a>
<a id="基准评测"></a> <a id="基准评测"></a>
## 基准评测:AndroidWorld (SOTA 99%+) ## 基准评测:AndroidWorld (SOTA 99%+)
在 Google Research 发布的业界基准评测 [AndroidWorld](https://github.com/google-research/android_world)(涵盖 20+ 款常用应用与 100+ 项复杂多步长程任务)中:**Artemis 在全套长程任务评测中展现了高鲁棒性,取得了超过 99% 的任务完成率。** Artemis 在 Google Research 发布的 [AndroidWorld](https://github.com/google-research/android_world) 基准评测中取得 **99%+ 任务完成率**。该评测涵盖 20+ 款应用与 100+ 项多步任务。
<p align="center"> <p align="center">
<img src="./docs/assets/androidworld_leaderboard.png?v=2" alt="AndroidWorld 评测基准对比" width="100%" /> <img src="./docs/assets/androidworld_leaderboard.png?v=2" alt="AndroidWorld 评测基准对比" width="100%" />
@@ -263,9 +264,9 @@ if __name__ == "__main__":
## ARTEMIS 是如何建构的 ## ARTEMIS 是如何建构的
* **Pre-Touch 触控前像素守门与预测性连点**:彻底杜绝大模型推理延迟造成的「静默误触」。动作下发前毫秒级拦截意外弹窗(0 Token、0 云端等待),Micro-ROI 局部校验目标稳定性;针对视频全屏等瞬态控件,结合历史先验毫秒级连击唤醒,在淡出窗口期内瞬时闭环; * **执行前校验与动作序列**:Pro 在下发单个动作前,通过实时 UI 树和像素校验目标。快速动作序列用于处理瞬态控件,无需等待下一轮模型响应。
* **三层递进式目标定位引擎 (Progressive Grounding)**:首层通过本地 OCR 与无障碍树几何融合(~150ms、0 Token),以无偏移数字索引承载 85%+ 常见交互;二层对 Flutter/Compose/Canvas 等自绘组件自动回退至空间视觉模型,三层辅以沙箱 CV 探针核验细微像素状态; * **元素定位**:结合无障碍树、OCR 和视觉模型,支持 Canvas、Compose、Flutter 等自绘界面。
* **弹性双执行引擎与运行时上下文动态压缩**:兼顾 3–5 秒秒级响应的 CI 高吞吐回归(Flash 模式)与深度长程认知状态图探索(Pro 模式),后台异步生成视觉增量并剪除冗余 DOM,长程任务 Token 消耗锐降 70%+,支持 10+ 小时无人值守稳定性压测。 * **共享历史压缩**:Flash 和 Pro 将较早的截图替换为视觉摘要,并把已完成步骤压缩为可检索的历史分块。上下文阈值决定何时替换原始记录。
<p align="center"> <p align="center">
<img src="./docs/assets/artemis_architecture_diagram.png" alt="ARTEMIS 架构系统示意图" width="100%" /> <img src="./docs/assets/artemis_architecture_diagram.png" alt="ARTEMIS 架构系统示意图" width="100%" />
@@ -275,8 +276,8 @@ if __name__ == "__main__":
ARTEMIS 提供两种运行模式以适应不同的自动化需求: ARTEMIS 提供两种运行模式以适应不同的自动化需求:
* **Flash 模式 (`--profile flash`)**:轻量极速且省 Token 的响应式循环(单步约 3–5 秒):单个模型观察实时屏幕、思考、执行,不经过图编排,适合常规确定性 UI 操作。默认不限步数(`agent.flash.max_turns`,0 表示不限),因为历史是被压缩而不是被截断:Flash 与 Pro 共用同一套会话记录账本(相对测试时间 `T+mm:ss`、截图折叠为视觉摘要、更早的步骤分块归档并可通过 `search_history` / `replay_steps` 按需召回),并可调用 `video_analyzer` 分析整段会话录屏;对自动消失的控制栏、toast 等瞬态控件,用 `click_sequence` 把多次点击串成一个原子序列。*局限性*:没有任务计划与笔记、没有执行前安全网、没有检查点校验与最终报告、不能执行 ADB 命令。 * **Flash 模式 (`--profile flash`)**:使用历史压缩的响应式循环(单步约 3–5 秒):单个模型观察实时屏幕、思考、执行,不经过图编排,适合常规确定性 UI 操作。默认不限步数(`agent.flash.max_turns`,0 表示不限),因为历史是被压缩而不是被截断:Flash 与 Pro 共用同一套会话记录账本(相对测试时间 `T+mm:ss`、截图折叠为视觉摘要、更早的步骤分块归档并可通过 `search_history` / `replay_steps` 按需召回),并可调用 `video_analyzer` 分析整段会话录屏;对自动消失的控制栏、toast 等瞬态控件,用 `click_sequence` 把多次点击串成一个原子序列。*局限性*:没有任务计划与笔记、没有执行前安全网、没有检查点校验与最终报告、不能执行 ADB 命令。
* **Pro 模式 (`--profile pro`)**:最强大的架构(单步约 15–40 秒),由多智能体图编排:**Planner** 维护一份带里程碑和 `verify` / `assert` 检查项的 Markdown 任务计划;**Operator** 按计划执行,拥有全套工具(Explorer 元素定位,其 `flash` / `pro` / `ultra` 三档感知深度是按运行模式设置的用户配置——`config/artemis.jsonc` 里的 `pro.explorer.mode` / `flash.explorer_mode` 或 `--explorer-pro-mode`——智能体自身不会选择档位;笔记、历史召回、视频分析、ADB 诊断)。单个动作执行前都经过 **Safety Net** 校验(XML 优先、像素兜底),而一轮里的多个动作则作为**快速动作序列 (fast-action burst)** 连发,赶在瞬态控件消失前完成操作。动作被拦截或失败时会打开一条**执行事故 (execution incident)**,持续留在 Operator 的上下文里直到后续动作执行成功,恢复由 Operator 自己完成,不再有独立的修复智能体。只读的 **Checker** 校验计划中的检查点,并在退出前对照原始目标做终审(`--verification-level`:`off` / `final`(默认)/ `checkpoints` / `strict`),计划里程碑的改动会得到建议式复核。支持 100+ 步的长程复杂业务流、`[Loop:continuous]` 持续监控以及可选的书面报告。 * **Pro 模式 (`--profile pro`)**:包含计划和校验的执行流程(单步约 15–40 秒),由多智能体图编排:**Planner** 维护一份带里程碑和 `verify` / `assert` 检查项的 Markdown 任务计划;**Operator** 按计划执行,拥有全套工具(Explorer 元素定位,其 `flash` / `pro` / `ultra` 三档感知深度是按运行模式设置的用户配置——`config/artemis.jsonc` 里的 `pro.explorer.mode` / `flash.explorer_mode` 或 `--explorer-pro-mode`——智能体自身不会选择档位;笔记、历史召回、视频分析、ADB 诊断)。单个动作执行前都经过 **Safety Net** 校验(XML 优先、像素兜底),而一轮里的多个动作则作为**快速动作序列 (fast-action burst)** 连发,赶在瞬态控件消失前完成操作。动作被拦截或失败时会打开一条**执行事故 (execution incident)**,持续留在 Operator 的上下文里直到后续动作执行成功,恢复由 Operator 自己完成,不再有独立的修复智能体。只读的 **Checker** 校验计划中的检查点,并在退出前对照原始目标做终审(`--verification-level`:`off` / `final`(默认)/ `checkpoints` / `strict`),计划里程碑的改动会得到建议式复核。支持 100+ 步的长程复杂业务流、`[Loop:continuous]` 持续监控以及可选的书面报告。
## 路线图 ## 路线图
@@ -1070,13 +1070,14 @@ limitations under the License.
<!-- Generic Tool Calls (Text format for note tools, video analyzer, and other sub-agents) --> <!-- Generic Tool Calls (Text format for note tools, video analyzer, and other sub-agents) -->
@if (shouldShowTool(item.data, block.data) && !isDeviceActionTool(item.data)) { @if (shouldShowTool(item.data, block.data) && !isDeviceActionTool(item.data)) {
<div class="step-generic-tool" <div class="step-generic-tool"
[class.is-running]="isItemRunning(item, block)" [class.is-running]="isToolLineRunning(item, block)"
[class.is-note-tool]="isNoteTool(item.data)" [class.is-note-tool]="isNoteTool(item.data)"
[class.is-video-tool]="isVideoTool(item.data)" [class.is-video-tool]="isVideoTool(item.data)"
[class.is-video-attention]="isVideoTool(item.data) && isVideoAnalysisAttention(item.data)" [class.is-video-attention]="isVideoTool(item.data) && isVideoAnalysisAttention(item.data)"
[attr.title]="getCompressionPhaseTitle(item.data)"
> >
<div class="tool-call-label"> <div class="tool-call-label">
<span class="stream-status-dot" [class.streaming]="isItemRunning(item, block)"></span> <span class="stream-status-dot" [class.streaming]="isToolLineRunning(item, block)"></span>
<div class="tool-call-text"> <div class="tool-call-text">
<span>{{ isVideoTool(item.data) ? getVideoAnalysisLabel(item.data) : getToolDisplayLabel(item.data) }}{{ ((isNoteTool(item.data) && getToolKey(item.data)) || isVideoTool(item.data)) ? ':' : '' }}</span> <span>{{ isVideoTool(item.data) ? getVideoAnalysisLabel(item.data) : getToolDisplayLabel(item.data) }}{{ ((isNoteTool(item.data) && getToolKey(item.data)) || isVideoTool(item.data)) ? ':' : '' }}</span>
<!-- Note Tool Pill --> <!-- Note Tool Pill -->
@@ -172,6 +172,9 @@ import {
getUniqueGenericTools, getUniqueGenericTools,
getToolKey, getToolKey,
getToolDisplayLabel, getToolDisplayLabel,
isCompressionTool,
isCompressionWaiting,
getCompressionPhaseLabel,
getToolIcon, getToolIcon,
getToolTitle, getToolTitle,
getToolTargetText, getToolTargetText,
@@ -1465,6 +1468,20 @@ export class AgentStreamComponent implements AfterViewInit {
return getToolDisplayLabel(tool, this.isFirstSaveNoteForKey(tool)); return getToolDisplayLabel(tool, this.isFirstSaveNoteForKey(tool));
} }
/** Short plain-language phase for a compress_history line (hover title). */
public getCompressionPhaseTitle(tool: any): string | null {
return isCompressionTool(tool) ? getCompressionPhaseLabel(tool) : null;
}
/**
* A generic tool line pulses only while work is in flight. A compression
* whose summary is ready but held back by the context gate is waiting, not
* running, even though its trace status is still `running`.
*/
public isToolLineRunning(item: any, block: any): boolean {
return this.isItemRunning(item, block) && !isCompressionWaiting(item?.data);
}
public isNoteTool(tool: any): boolean { public isNoteTool(tool: any): boolean {
return isNoteTool(tool); return isNoteTool(tool);
} }
@@ -29,7 +29,7 @@
<!-- Main Content Layout --> <!-- Main Content Layout -->
<main class="home-content"> <main class="home-content">
<!-- Hero Intent Section (Modern Linear / Claude Style) --> <!-- Task entry section -->
<section class="hero-section"> <section class="hero-section">
<h1 class="hero-title"> <h1 class="hero-title">
What would you like to do on your phone? What would you like to do on your phone?
@@ -1718,7 +1718,7 @@
</div> </div>
} }
<!-- Unified Intelligent Composer Surface (Apple / Linear / Claude Grade) --> <!-- Task composer -->
<div class="composer-card" [class.is-pro-active]="selectedProfile() === 'pro'"> <div class="composer-card" [class.is-pro-active]="selectedProfile() === 'pro'">
<!-- Smart Intent Recommendation Strip (if detected) --> <!-- Smart Intent Recommendation Strip (if detected) -->
@@ -1737,7 +1737,7 @@
</div> </div>
} }
<!-- Borderless Seamless Textarea Input Area --> <!-- Borderless task input -->
<div class="composer-input-area"> <div class="composer-input-area">
<textarea <textarea
id="task-input-field" id="task-input-field"
@@ -1768,7 +1768,7 @@
<p class="desc-line"> <p class="desc-line">
<span class="mode-name">Pro</span> <span class="mode-name">Pro</span>
<span class="bullet">·</span> <span class="bullet">·</span>
<span>Our most powerful mode. Plans first, checks its own results and recovers when a tap is blocked. Handles <strong>100+ step</strong> workflows and live phone monitoring (<strong>15–40s</strong> / step).</span> <span>Plans first, checks its results and recovers when a tap is blocked. Handles <strong>100+ step</strong> workflows and live phone monitoring (<strong>15–40s</strong> / step).</span>
</p> </p>
} }
</div> </div>
@@ -353,7 +353,7 @@ $text-light: #94a3b8;
flex-direction: column; flex-direction: column;
gap: 18px; gap: 18px;
/* Hero Section (Modern Linear / Claude Style) */ /* Hero section */
.hero-section { .hero-section {
text-align: center; text-align: center;
padding: 16px 0 6px; padding: 16px 0 6px;
@@ -3819,7 +3819,7 @@ $text-light: #94a3b8;
} }
} }
/* Unified Sculptured Composer (Linear / Claude / ChatGPT Grade) */ /* Task composer */
.composer-card { .composer-card {
background: rgba(255, 255, 255, 0.88); background: rgba(255, 255, 255, 0.88);
backdrop-filter: blur(16px) saturate(190%); backdrop-filter: blur(16px) saturate(190%);
@@ -2,6 +2,9 @@ import {
cleanErrorMessage, cleanErrorMessage,
extractToolExtraParams, extractToolExtraParams,
getCompressionLabel, getCompressionLabel,
getCompressionPhase,
getCompressionPhaseLabel,
isCompressionWaiting,
getToolDisplayLabel, getToolDisplayLabel,
getToolIcon, getToolIcon,
getToolTargetText, getToolTargetText,
@@ -35,6 +38,22 @@ describe('compress_history timeline line', () => {
.toBe('Condensing step 5 into a short memory to free up room…'); .toBe('Condensing step 5 into a short memory to free up room…');
}); });
it('says the summary is ready but held while working memory is still small', () => {
const held = trace('running', {
start_step: 1,
end_step: 4,
note: 'held',
context_tokens: 20000,
context_budget: 80000,
swap_at_tokens: 28000
});
expect(getCompressionLabel(held)).toBe(
'Short memory for steps 1–4 is ready; keeping the full record until working memory fills up · ≈ 20k of 28k tokens'
);
expect(getCompressionLabel(trace('running', { start_step: 1, end_step: 4, note: 'held' })))
.toBe('Short memory for steps 1–4 is ready; keeping the full record until working memory fills up');
});
it('reports the size reduction and the working memory once done', () => { it('reports the size reduction and the working memory once done', () => {
const done = trace('success', { const done = trace('success', {
start_step: 12, start_step: 12,
@@ -54,13 +73,53 @@ describe('compress_history timeline line', () => {
expect(getCompressionLabel(done)).toBe('Steps 1–4 condensed into a short memory · 3k → 1.5k tokens (2× smaller)'); expect(getCompressionLabel(done)).toBe('Steps 1–4 condensed into a short memory · 3k → 1.5k tokens (2× smaller)');
}); });
it('explains a forced snapshot and a failed attempt in plain words', () => { it('explains a forced recap and a failed attempt in plain words', () => {
const forced = trace('success', { start_step: 1, end_step: 4, forced: true, context_tokens: 70000, context_budget: 80000 }); const forced = trace('success', { start_step: 1, end_step: 4, forced: true, context_tokens: 70000, context_budget: 80000 });
expect(getCompressionLabel(forced)) expect(getCompressionLabel(forced))
.toBe('Steps 1–4 replaced by a brief snapshot (memory was nearly full) · working memory ≈ 70k of 80k tokens'); .toBe('Steps 1–4 condensed into a recap to free up memory · working memory ≈ 70k of 80k tokens');
expect(getCompressionLabel(trace('failed', { start_step: 1, end_step: 4 }))) expect(getCompressionLabel(trace('failed', { start_step: 1, end_step: 4 })))
.toBe("Couldn't condense steps 1–4 yet; keeping the full record and retrying later"); .toBe("Couldn't condense steps 1–4 yet; keeping the full record and retrying later");
}); });
describe('phase (args.phase from the backend)', () => {
it('reads the declared phase and maps it to one short plain-language label', () => {
const cases: Array<[string, string, string]> = [
['running', 'summarizing', 'Summarizing this stretch'],
['running', 'ready', 'Summary ready; kept in reserve until the context fills up'],
['success', 'applied', 'Replaced this stretch with its summary'],
['failed', 'failed', 'Summary failed; full record kept']
];
for (const [status, phase, label] of cases) {
const tool = trace(status, { start_step: 1, end_step: 4, phase });
expect(getCompressionPhase(tool)).toBe(phase as any);
expect(getCompressionPhaseLabel(tool)).toBe(label);
}
});
it('falls back to status and note on traces that carry no phase', () => {
expect(getCompressionPhase(trace('running', { start_step: 1, end_step: 4 }))).toBe('summarizing');
expect(getCompressionPhase(trace('running', { start_step: 1, end_step: 4, note: 'retrying' }))).toBe('summarizing');
expect(getCompressionPhase(trace('running', { start_step: 1, end_step: 4, note: 'held' }))).toBe('ready');
expect(getCompressionPhase(trace('success', { start_step: 1, end_step: 4 }))).toBe('applied');
expect(getCompressionPhase(trace('failed', { start_step: 1, end_step: 4 }))).toBe('failed');
expect(getCompressionPhase(trace('running', { start_step: 1, end_step: 4, phase: 'bogus' }))).toBe('summarizing');
});
it('drives the timeline line from the phase even without a note', () => {
expect(getCompressionLabel(trace('running', { start_step: 1, end_step: 4, phase: 'ready' })))
.toBe('Short memory for steps 1–4 is ready; keeping the full record until working memory fills up');
expect(getCompressionLabel(trace('running', { start_step: 1, end_step: 4, phase: 'summarizing' })))
.toBe('Condensing steps 1–4 into a short memory to free up room…');
});
it('treats a ready-but-held summary as waiting, not running', () => {
expect(isCompressionWaiting(trace('running', { start_step: 1, end_step: 4, phase: 'ready' }))).toBeTrue();
expect(isCompressionWaiting(trace('running', { start_step: 1, end_step: 4, note: 'held' }))).toBeTrue();
expect(isCompressionWaiting(trace('running', { start_step: 1, end_step: 4, phase: 'summarizing' }))).toBeFalse();
expect(isCompressionWaiting(trace('success', { start_step: 1, end_step: 4, phase: 'applied' }))).toBeFalse();
expect(isCompressionWaiting({ type: 'tool', name: 'save_note', status: 'running', payload: { args: { phase: 'ready' } } })).toBeFalse();
});
});
}); });
describe('cleanErrorMessage', () => { describe('cleanErrorMessage', () => {
@@ -725,21 +725,31 @@ export function getCompressionLabel(tool: any): string {
: 'earlier steps'; : 'earlier steps';
const rangeCapitalized = range.charAt(0).toUpperCase() + range.slice(1); const rangeCapitalized = range.charAt(0).toUpperCase() + range.slice(1);
const status = String(tool?.status || '').toLowerCase(); const status = String(tool?.status || '').toLowerCase();
const phase = getCompressionPhase(tool);
if (status === 'failed') { if (status === 'failed' || phase === 'failed') {
return `Couldn't condense ${range} yet; keeping the full record and retrying later`; return `Couldn't condense ${range} yet; keeping the full record and retrying later`;
} }
if (status !== 'success') { if (status !== 'success') {
return String(args.note || '').toLowerCase() === 'retrying' const note = String(args.note || '').toLowerCase();
? `Retrying the memory summary for ${range}…` if (note === 'retrying') return `Retrying the memory summary for ${range}…`;
: `Condensing ${range} into a short memory to free up room…`; if (note === 'held' || phase === 'ready') {
const swapAt = Number(args.swap_at_tokens);
const heldContext = Number(args.context_tokens);
const heldParts = [`Short memory for ${range} is ready; keeping the full record until working memory fills up`];
if (heldContext > 0 && swapAt > 0) {
heldParts.push(`≈ ${formatTokenFigure(heldContext)} of ${formatTokenFigure(swapAt)} tokens`);
}
return heldParts.join(' · ');
}
return `Condensing ${range} into a short memory to free up room…`;
} }
const parts: string[] = []; const parts: string[] = [];
const source = Number(args.source_tokens); const source = Number(args.source_tokens);
const summary = Number(args.summary_tokens); const summary = Number(args.summary_tokens);
if (args.forced) { if (args.forced) {
parts.push(`${rangeCapitalized} replaced by a brief snapshot (memory was nearly full)`); parts.push(`${rangeCapitalized} condensed into a recap to free up memory`);
} else { } else {
parts.push(`${rangeCapitalized} condensed into a short memory`); parts.push(`${rangeCapitalized} condensed into a short memory`);
if (source > 0 && summary > 0) { if (source > 0 && summary > 0) {
@@ -758,6 +768,57 @@ export function getCompressionLabel(tool: any): string {
return parts.join(' · '); return parts.join(' · ');
} }
/**
* Plain-language phases of one compress_history line. Mirrors
* COMPRESSION_PHASES in artemis/memory/chunking.py: the backend writes
* `args.phase`; the trace `status` (running/success/failed) is left alone.
*/
export type CompressionPhase = 'summarizing' | 'ready' | 'applied' | 'failed';
const COMPRESSION_PHASES: readonly CompressionPhase[] = ['summarizing', 'ready', 'applied', 'failed'];
export function isCompressionTool(tool: any): boolean {
const name = String(tool?.name || '').toLowerCase().replace(/^(_)?exec_/, '');
return name === 'compress_history';
}
/**
* Phase of a compress_history line. Prefers the backend's `args.phase`;
* older traces without it fall back to status + note (`held` ⇒ ready).
*/
export function getCompressionPhase(tool: any): CompressionPhase {
const args = getToolArgs(tool) || {};
const declared = String(args.phase || '').toLowerCase() as CompressionPhase;
if (COMPRESSION_PHASES.includes(declared)) return declared;
const status = String(tool?.status || '').toLowerCase();
if (status === 'failed') return 'failed';
if (status === 'success') return 'applied';
if (String(args.note || '').toLowerCase() === 'held') return 'ready';
return 'summarizing';
}
/** One short, non-technical label per compression phase. */
export function getCompressionPhaseLabel(tool: any): string {
switch (getCompressionPhase(tool)) {
case 'ready':
return 'Summary ready; kept in reserve until the context fills up';
case 'applied':
return 'Replaced this stretch with its summary';
case 'failed':
return 'Summary failed; full record kept';
default:
return 'Summarizing this stretch';
}
}
/**
* A ready-but-held summary is waiting, not working: the card must not keep
* its running pulse even though the trace status is still `running`.
*/
export function isCompressionWaiting(tool: any): boolean {
return isCompressionTool(tool) && getCompressionPhase(tool) === 'ready';
}
/** /**
* Get Material Symbol icon for tools * Get Material Symbol icon for tools
*/ */
+3 -3
View File
@@ -1,9 +1,9 @@
{ {
"system": "You are a precise, evidence-driven verification expert (Checker). You audit declared check items against recorded execution evidence and issue verdicts. You never operate the device, never modify any state, and never negotiate with the executing agent — you only observe, probe read-only state where allowed, and judge.", "system": "You are a precise, evidence-driven verification expert (Checker). You audit declared check items against recorded execution evidence and issue verdicts. You never operate the device, never modify any state, and never negotiate with the executing agent — you only observe, probe read-only state where allowed, and judge.",
"base_rules": "# Verdict Rules (always apply)\n- Quote every check item verbatim in `item_text` — never paraphrase.\n- Every verdict MUST carry concrete evidence: a specific probe output, a specific note excerpt, or a specific on-screen/screenshot observation. A `failed` verdict without concrete evidence is invalid and will be downgraded.\n- `inconclusive` means the evidence needed for a judgment was never captured or is genuinely ambiguous. When evidence was never collected, output `inconclusive` and state explicitly that the evidence is missing — do NOT output `failed` for lack of evidence.\n- Judge only what the check item declares. Do not invent additional criteria and do not relax the declared ones.", "base_rules": "# Verdict Rules (always apply)\n- Quote every check item verbatim in `item_text` — never paraphrase.\n- Every verdict MUST carry concrete evidence: a specific probe output, a specific note excerpt, or a specific on-screen/screenshot observation. A `failed` verdict without concrete evidence is invalid and will be downgraded.\n- `inconclusive` means the evidence needed for a judgment was never captured or is genuinely ambiguous. When evidence was never collected, output `inconclusive` and state explicitly that the evidence is missing — do NOT output `failed` for lack of evidence.\n- Judge only what the check item declares. Do not invent additional criteria and do not relax the declared ones.",
"verify_semantics": "# `verify` items (completion acceptance criteria)\n- A `verify` item asks: was the anchored subgoal actually accomplished to this standard?\n- `failed` verify verdicts trigger a repair loop, so a failed verdict MUST include a `suggestion`: a concrete, actionable hint telling the executor what is missing or wrong.\n- The release decision (whether execution proceeds) is computed outside of you and is fail-open on errors; your job is only the truthful verdict value.", "verify_semantics": "# `verify` items (completion acceptance criteria)\n- A `verify` item asks: was the anchored subgoal actually accomplished to this standard?\n- A `failed` verify verdict can send its milestone back for repair, so it must include a `suggestion`: a concrete, actionable hint telling the executor what is missing or wrong.\n- The release decision (whether execution proceeds) is computed outside of you and is fail-open on errors; your job is only the truthful verdict value.",
"assert_semantics": "# `assert` items (test assertions)\n- An `assert` item is a test expectation, not an instruction. A failed assertion is a legitimate, valuable test result: record it truthfully with evidence. Nothing will (or should) try to repair it.\n- Never mark an assert `failed` merely because evidence is absent — absence of captured evidence is `inconclusive` with an explicit note that the evidence is missing.\n- `suggestion` is not needed for assert verdicts.", "assert_semantics": "# `assert` items (test assertions)\n- An `assert` item is a test expectation, not an instruction. A failed assertion is a legitimate, valuable test result: record it truthfully with evidence. Nothing will (or should) try to repair it.\n- Never mark an assert `failed` merely because evidence is absent — absence of captured evidence is `inconclusive` with an explicit note that the evidence is missing.\n- Precondition first: before judging an assert, confirm from the execution history (step replay, notes, anchor evidence) that the action(s) meant to trigger the asserted behavior were actually and correctly performed by the executor (the toggle was really tapped, the intended screen was really reached).\n- If that precondition is not established — the executor never performed the trigger, performed the wrong action, or the evidence cannot show it — output `inconclusive` and state explicitly that the precondition failed / the triggering action is not confirmed as executed. Never output `failed` here: a failed assert is attributed to the device under test, and an execution problem must not be reported as a device defect.\n- Output `failed` only when the precondition is established AND the device state, probe output, or recorded evidence still contradicts the expectation. The evidence must cite the concrete step(s) that performed the trigger plus the probe output / observation that contradicts the expectation, so the verdict reads as a device-defect diagnosis.\n- `suggestion` is not needed for assert verdicts.",
"anchor_guide": "# Evidence anchoring (checkpoint audit)\n- You are auditing a subgoal at its completion moment. The authoritative evidence is the execution history around the anchor step (the anchor step's pre-action screenshot and the preceding step's post-action state) plus persistently probeable device state.\n- For transient expectations (toasts, dialogs, momentary states) and ordering expectations, you MUST inspect the recorded history near the anchor step — the live device has already moved on and is NOT valid evidence for them.\n- Prefer persistent state probes and recorded notes over inference from summaries.", "anchor_guide": "# Evidence anchoring (checkpoint audit)\n- You are auditing a subgoal at its completion moment. The authoritative evidence is the execution history around the anchor step (the anchor step's pre-action screenshot and the preceding step's post-action state) plus persistently probeable device state.\n- For transient expectations (toasts, dialogs, momentary states) and ordering expectations, you MUST inspect the recorded history near the anchor step — the live device has already moved on and is NOT valid evidence for them.\n- Prefer persistent state probes and recorded notes over inference from summaries.",
"final_guide": "# Final review (task exit audit)\n- Audit against the USER'S ORIGINAL GOAL, not merely the plan's checkboxes.\n- `@end` items: judge them using the FINAL device state provided to you.\n- `on_complete` items: cite the verdict ledger — an already-recorded PASS/FAIL from the item's checkpoint attempt is authoritative (an assert that ever failed stays failed). For items never checked, judge from the historical evidence near their subgoal's completion; if that evidence was never captured, output `inconclusive` stating the evidence is missing. NEVER re-judge an `on_complete` item from the current screen: the current screen only proves the present, not the past moment the item was declared for.\n- `unmet_subgoals`: list, verbatim, any plan subgoal marked complete whose verify criteria are demonstrably not met. Leave it empty otherwise; failed asserts alone do NOT make a subgoal unmet.", "final_guide": "# Final review (task exit audit)\n- Audit against the user's original goal as amended by the user guidance received during the run (listed in the prompt), not merely the plan's checkboxes. Guidance outranks the plan and its check lines: judge a check line the guidance retired or changed by the guidance, not by its original wording.\n- `@end` items: judge them using the FINAL device state provided to you.\n- `on_complete` items: cite the verdict ledger — an already-recorded PASS/FAIL from the item's checkpoint attempt is authoritative (an assert that ever failed stays failed). For items never checked, judge from the historical evidence near their subgoal's completion; if that evidence was never captured, output `inconclusive` stating the evidence is missing. NEVER re-judge an `on_complete` item from the current screen: the current screen only proves the present, not the past moment the item was declared for.\n- `unmet_subgoals`: list, verbatim, any plan subgoal marked complete whose verify criteria are demonstrably not met. Leave it empty otherwise; failed asserts alone do NOT make a subgoal unmet.",
"probe_guide": "# Device probes (read-only)\n- `probe_device` runs a fixed, enumerated set of read-only state probes. Prefer probing persistent state over inferring it: alarms (`alarms`), battery status (`battery`), foreground activities (`foreground`), notifications (`notifications`), a settings value (`setting` with `namespace` + `key`), a content provider query (`content` with `uri`), installed packages (`packages`), a system property (`prop` with `key`).\n- Probes outside this table do not exist; you cannot run arbitrary commands." "probe_guide": "# Device probes (read-only)\n- `probe_device` runs a fixed, enumerated set of read-only state probes. Prefer probing persistent state over inferring it: alarms (`alarms`), battery status (`battery`), foreground activities (`foreground`), notifications (`notifications`), a settings value (`setting` with `namespace` + `key`), a content provider query (`content` with `uri`), installed packages (`packages`), a system property (`prop` with `key`).\n- Probes outside this table do not exist; you cannot run arbitrary commands."
} }
+59 -7
View File
@@ -319,21 +319,54 @@ def build_checker_tools(
# --- History formatting -------------------------------------------------------------- # --- History formatting --------------------------------------------------------------
def _format_history(ctx: ArtemisContext, limit: int = 60) -> str: def _load_steps(ctx: ArtemisContext) -> list[dict] | str:
"""The agent-friendly step records, or the reason they are unavailable."""
if not ctx.data_engine: if not ctx.data_engine:
return "No execution history available." return "No execution history available."
try: try:
steps = ctx.data_engine.get_agent_friendly_steps() or [] return list(ctx.data_engine.get_agent_friendly_steps() or [])
except Exception as e: except Exception as e:
return f"Failed to load execution history: {e}" return f"Failed to load execution history: {e}"
def _step_offset_label(step: dict, session_start: float | None) -> str:
"""Session-relative ``T+mm:ss`` label of a step: the clock every agent shares.
The recorded timestamp against the session start is authoritative; a
record that only carries the engine's ``"97.7s"`` label is converted.
"""
from artemis.memory.transcript import format_session_offset
ts = step.get("timestamp")
if isinstance(session_start, (int, float)) and isinstance(ts, (int, float)):
return format_session_offset(float(ts) - float(session_start))
rel = str(step.get("relative_time") or "").strip()
if rel.startswith("T+"):
return rel
if rel.endswith("s"):
try:
return format_session_offset(float(rel[:-1]))
except ValueError:
pass
return "T+??:??"
def _format_history(ctx: ArtemisContext, limit: int = 60, steps: list[dict] | None = None) -> str:
if steps is None:
steps = _load_steps(ctx)
if isinstance(steps, str):
return steps
if not steps: if not steps:
return "No execution history available." return "No execution history available."
session_start = getattr(ctx.data_engine, "session_start_time", None)
lines = [] lines = []
for s in steps[-limit:]: for s in steps[-limit:]:
num = s.get("step_number") num = s.get("step_number")
rel = s.get("relative_time", "") summary = (s.get("summary") or "").strip()
summary = s.get("summary") or ""
action = s.get("action_taken") action = s.get("action_taken")
if not summary and not action:
# Nothing to audit on this step: an empty row is only noise.
continue
action_str = "" action_str = ""
if action and not summary: if action and not summary:
# The shared renderer keeps the target's provenance visible: a # The shared renderer keeps the target's provenance visible: a
@@ -343,8 +376,25 @@ def _format_history(ctx: ArtemisContext, limit: int = 60) -> str:
action_str = format_actions_clean(action)[:160] action_str = format_actions_clean(action)[:160]
except Exception: except Exception:
action_str = str(action)[:160] action_str = str(action)[:160]
lines.append(f"- Step {num} ({rel}): {summary or action_str}".rstrip()) label = _step_offset_label(s, session_start)
return "\n".join(lines) lines.append(f"- Step {num} ({label}): {summary or action_str}".rstrip())
return "\n".join(lines) or "No execution history available."
def _format_user_guidance(ctx: ArtemisContext, steps: list[dict] | None = None) -> str:
"""Every mid-run user instruction, read from the step it was stamped on."""
if steps is None:
steps = _load_steps(ctx)
if isinstance(steps, str) or not steps:
return "(none)"
session_start = getattr(ctx.data_engine, "session_start_time", None)
lines = []
for s in steps:
instruction = (s.get("extra_metadata") or {}).get("injected_instruction")
if isinstance(instruction, str) and instruction.strip():
label = _step_offset_label(s, session_start)
lines.append(f'- Step {s.get("step_number")} ({label}): "{instruction.strip()}"')
return "\n".join(lines) or "(none)"
def _format_check_items(check_items: list) -> str: def _format_check_items(check_items: list) -> str:
@@ -773,14 +823,16 @@ async def run_final_check(
) )
screenshot_b64, minimal_list = await _capture_final_screen(ctx) screenshot_b64, minimal_list = await _capture_final_screen(ctx)
steps = _load_steps(ctx)
human_text = ( human_text = (
f"# User's Original Goal\n{goal}\n\n" f"# User's Original Goal\n{goal}\n\n"
f"# User guidance received during the run\n{_format_user_guidance(ctx, steps)}\n\n"
f"# Final Task Plan\n{plan_text or '(no plan)'}\n\n" f"# Final Task Plan\n{plan_text or '(no plan)'}\n\n"
f"# Declared Check Items\n{_format_check_items(items)}\n\n" f"# Declared Check Items\n{_format_check_items(items)}\n\n"
f"# Verdict Ledger (checkpoint results, authoritative for on_complete" f"# Verdict Ledger (checkpoint results, authoritative for on_complete"
f" items)\n{_format_ledger(ledger)}\n\n" f" items)\n{_format_ledger(ledger)}\n\n"
f"# Execution History (summaries)\n{_format_history(ctx)}\n\n" f"# Execution History (summaries)\n{_format_history(ctx, steps=steps)}\n\n"
f"# Final Screen Elements\n{minimal_list}" f"# Final Screen Elements\n{minimal_list}"
) )
+4 -3
View File
@@ -356,9 +356,10 @@ class Diagnoser:
if submit_tool: if submit_tool:
submit_tool.invoke(args) submit_tool.invoke(args)
outcome = f"Analysis: {analysis} | Actionable Steps: " outcome = f"Analysis: {analysis}"
for step in steps: if steps:
outcome += f"{step} " numbered = "\n".join(f"{i}. {step}" for i, step in enumerate(steps, start=1))
outcome += f"\nActionable steps:\n{numbered}"
return outcome return outcome
def _max_iteration_outcome(self, response, outcome: str) -> str: def _max_iteration_outcome(self, response, outcome: str) -> str:
+2 -2
View File
@@ -61,7 +61,7 @@ UNIVERSAL_EXPLORER_TOOLS = [
"function": { "function": {
"name": "detect_objects", "name": "detect_objects",
"description": ( "description": (
"[Perception] Locates elements, shapes and more using powerful VLM." "[Perception] Locates elements, shapes and more using a vision-language model."
" Returns their normalized coordinates (0-1000 scale) with label dots." " Returns their normalized coordinates (0-1000 scale) with label dots."
), ),
"parameters": { "parameters": {
@@ -235,7 +235,7 @@ NATIVE_EXPLORER_TOOL_DECLARATIONS = [
types.FunctionDeclaration( types.FunctionDeclaration(
name="detect_objects", name="detect_objects",
description=( description=(
"[Perception] Locates elements, shapes and more using powerful" "[Perception] Locates elements, shapes and more using a"
" VLM. Returns their normalized coordinates (in a [0, 1000]" " VLM. Returns their normalized coordinates (in a [0, 1000]"
" scale) labeled with [D1], [D2] etc. in the text output, along" " scale) labeled with [D1], [D2] etc. in the text output, along"
" with an annotated image containing corresponding visual label" " with an annotated image containing corresponding visual label"
+243 -112
View File
@@ -14,10 +14,14 @@
"""Context compression shared by Flash and Pro. """Context compression shared by Flash and Pro.
:class:`ScrubEdgeCompressor` strips XML at depth 1 and resolves screenshots :class:`ScrubEdgeCompressor` maintains two edges over the observation history:
at depth K, then freezes the message. A late summary never backfills a frozen a shallow text edge (``xml_scrub_depth``, default 1: as soon as a newer
message. FlashRunner uses this implementation; the Pro transcript ledger observation exists) where the heavy text blocks and the per-turn ephemeral
reuses it over its active region (see the class docstring). blocks leave, and the screenshot edge at depth K where the image is resolved
in place into its visual summary. A message past both edges is frozen; a late
summary never backfills a frozen message. FlashRunner uses this implementation;
the Pro transcript ledger reuses it over its active region (see the class
docstring).
""" """
from typing import Any, Callable from typing import Any, Callable
@@ -25,6 +29,7 @@ from typing import Any, Callable
from langchain_core.messages import BaseMessage from langchain_core.messages import BaseMessage
from artemis.memory.step_memory import StepMemoryService from artemis.memory.step_memory import StepMemoryService
from artemis.memory.transcript import EPHEMERAL_BLOCKS_KEY
from artemis.utils.logger import get_logger from artemis.utils.logger import get_logger
logger = get_logger(__name__) logger = get_logger(__name__)
@@ -35,6 +40,15 @@ HISTORY_SUMMARY_PREFIX = "--- Historical Visual Transition ---\n"
_HISTORY_SUMMARY_PREFIX = HISTORY_SUMMARY_PREFIX _HISTORY_SUMMARY_PREFIX = HISTORY_SUMMARY_PREFIX
_UI_LIST_MARKER = "--- UI Element List ---" _UI_LIST_MARKER = "--- UI Element List ---"
#: Text of the label block both observation shapes place directly above the
#: screenshot. It only makes sense next to an image, so it leaves together
#: with the image it labels (the visual summary carries its own header).
SCREENSHOT_LABEL = "--- Current Screenshot ---"
#: Sentinel: the screenshot stays for now (summary pending inside the grace
#: window); the message is revisited for the image swap only.
_KEEP_IMAGE = object()
def _without_marked_suffix(text: str, markers: tuple[str, ...] = (_UI_LIST_MARKER,)) -> str: def _without_marked_suffix(text: str, markers: tuple[str, ...] = (_UI_LIST_MARKER,)) -> str:
"""Remove a marked heavy block without discarding text before its marker.""" """Remove a marked heavy block without discarding text before its marker."""
@@ -49,22 +63,32 @@ def _without_marked_suffix(text: str, markers: tuple[str, ...] = (_UI_LIST_MARKE
class ScrubEdgeCompressor: class ScrubEdgeCompressor:
"""Replace older observation details with summaries near the message tail. """Replace older observation details with summaries at two scrub edges.
Mutation discipline — every message is touched at most twice, always near Depth 1 == the most recent image-bearing message. Messages newer than an
the tail, and is then frozen forever: edge are byte-identical to what the model was first shown. Each message
is edited at most twice and is then frozen:
1. Depth 1 (XML): once a newer observation exists, historical UI Element 1. Text edge (depth > ``xml_scrub_depth``, default 1, i.e. as soon as a
lists are stripped (same semantics as the legacy ``prune_history_xml``). newer observation exists), one pass, exactly once per message: the
2. Depth K (image, ``image_scrub_depth``): the K-th most recent heavy blocks behind ``strip_markers`` (the UI element list; for Pro
image-bearing message has its screenshot resolved — replaced by the also the plan recitation) are cut at their marker with the text before
ready visual summary; if the summary is still pending the image is the marker kept, and every content block whose index is listed under
retained for up to ``pending_grace_steps`` further image-depths, after :data:`~artemis.memory.transcript.EPHEMERAL_BLOCKS_KEY` in the
which it is replaced by a pending placeholder. A failed summary is message's ``additional_kwargs`` (reminders, hints, per-turn notices)
replaced by an unavailable placeholder immediately. Both placeholders is removed. Only the live observation therefore ever carries an
carry the DataEngine step reference. indexed element list: a stale ``[n]`` index from an older screen can
3. Frozen: after resolution the message is never read or written again — never be picked up as a target.
a late summary never backfills a frozen message. 2. Screenshot edge (depth >= ``image_scrub_depth``), IN PLACE: the ready
visual summary block takes the image block's position and the
:data:`SCREENSHOT_LABEL` block above it is removed. A failed summary
becomes an unavailable placeholder immediately; a summary still
pending keeps the image (and its label) for up to
``pending_grace_steps`` further depths, after which a pending
placeholder replaces it. Both placeholders carry the DataEngine step
reference. An image with no summary job at all is dropped silently.
3. Frozen: after the image is resolved the message is never read or
written again; a late summary never backfills a frozen message.
Contract: the message list is append-only (FlashRunner never removes or Contract: the message list is append-only (FlashRunner never removes or
reorders entries), so bookkeeping is index-based and each pass only reorders entries), so bookkeeping is index-based and each pass only
@@ -72,15 +96,14 @@ class ScrubEdgeCompressor:
whole history. whole history.
The Pro transcript ledger uses this discipline over its active region. The Pro transcript ledger uses this discipline over its active region.
Its observation messages carry no Its observation messages carry no ``tool_call_id``, so
``tool_call_id``, so ``summary_key_getter`` supplies the summary-job key ``summary_key_getter`` supplies the summary-job key (the DataEngine step
(the DataEngine step id) per message; ``strip_markers`` extends the id) per message; ``strip_markers`` extends the strip to the Pro tail
depth-1 strip to the Pro tail blocks (UI element list + plan recitation); blocks (UI element list + plan recitation); and ``tail_offset=1``
and ``tail_offset=1`` accounts for the live observation living outside accounts for the live observation living outside the tracked list (the
the tracked list (the ledger renders it as a separate tail), so depth ledger renders it as a separate tail), so depth arithmetic stays aligned
arithmetic and the keep-window stay aligned with the Flash semantics with the Flash semantics where the live observation is *inside* the
where the live observation is *inside* the list. Defaults leave the list.
Flash behavior byte-identical.
""" """
def __init__( def __init__(
@@ -90,7 +113,7 @@ class ScrubEdgeCompressor:
prune_history_xml: bool = True, prune_history_xml: bool = True,
image_scrub_depth: int = 3, image_scrub_depth: int = 3,
pending_grace_steps: int = 3, pending_grace_steps: int = 3,
xml_scrub_depth: int = 1, xml_scrub_depth: int | None = 1,
summary_key_getter: Callable[[BaseMessage], str | None] | None = None, summary_key_getter: Callable[[BaseMessage], str | None] | None = None,
strip_markers: tuple[str, ...] = (_UI_LIST_MARKER,), strip_markers: tuple[str, ...] = (_UI_LIST_MARKER,),
tail_offset: int = 0, tail_offset: int = 0,
@@ -99,7 +122,9 @@ class ScrubEdgeCompressor:
self._prune_history_xml = prune_history_xml self._prune_history_xml = prune_history_xml
self._image_scrub_depth = max(1, image_scrub_depth) self._image_scrub_depth = max(1, image_scrub_depth)
self._pending_grace_steps = max(0, pending_grace_steps) self._pending_grace_steps = max(0, pending_grace_steps)
self._xml_scrub_depth = max(1, xml_scrub_depth) # Text edge: the heavy text blocks and ephemeral blocks leave once a
# message is deeper than this (None keeps the default of 1).
self._xml_scrub_depth = max(1, int(xml_scrub_depth)) if xml_scrub_depth is not None else 1
self._summary_key_getter = summary_key_getter self._summary_key_getter = summary_key_getter
self._strip_markers = tuple(strip_markers) or (_UI_LIST_MARKER,) self._strip_markers = tuple(strip_markers) or (_UI_LIST_MARKER,)
self._tail_offset = max(0, tail_offset) self._tail_offset = max(0, tail_offset)
@@ -110,15 +135,28 @@ class ScrubEdgeCompressor:
# {"idx", "key" (str tool_call_id | None), "legacy" (ordinal | None), "is_tool"} # {"idx", "key" (str tool_call_id | None), "legacy" (ordinal | None), "is_tool"}
self._tracked: list[dict[str, Any]] = [] self._tracked: list[dict[str, Any]] = []
self._frozen: set[int] = set() self._frozen: set[int] = set()
# Messages whose one-time text edits (strip + ephemeral deletion) are
# done; a message inside the pending-grace window sits here unfrozen.
self._text_scrubbed: set[int] = set()
# Strip-marker carriers without an image (e.g. a step whose screenshot
# failed); they are stripped when the edge passes their position.
self._xml_candidates: list[int] = [] self._xml_candidates: list[int] = []
def compress(self, messages: list[BaseMessage]) -> None: def compress(
"""Advance the scrub edge over newly appended messages in-place.""" self, messages: list[BaseMessage], *, image_scrub_depth: int | None = None
) -> None:
"""Advance the scrub edge over newly appended messages in-place.
``image_scrub_depth`` overrides the constructor depth for this pass
(the transcript ledger passes its occupancy-driven depth). Because a
scrubbed message is frozen, a deeper pass after a shallower one never
backfills anything: relaxing the depth only lets *new* images live
longer, tightening it scrubs the extra depths once.
"""
self._discover(messages) self._discover(messages)
if not self._tracked: if not self._tracked and not self._xml_candidates:
return return
self._scrub_xml(messages) self._scrub_edge(messages, image_scrub_depth)
self._scrub_images(messages)
# ------------------------------------------------------------------ # ------------------------------------------------------------------
# Discovery (incremental; frozen prefix is never rescanned) # Discovery (incremental; frozen prefix is never rescanned)
@@ -144,8 +182,6 @@ class ScrubEdgeCompressor:
elif block.get("type") == "text" and self._has_strip_marker(block.get("text", "")): elif block.get("type") == "text" and self._has_strip_marker(block.get("text", "")):
has_xml = True has_xml = True
if has_xml:
self._xml_candidates.append(idx)
if has_image: if has_image:
key = str(tool_call_id) if tool_call_id is not None else None key = str(tool_call_id) if tool_call_id is not None else None
if key is None and self._summary_key_getter is not None: if key is None and self._summary_key_getter is not None:
@@ -163,62 +199,24 @@ class ScrubEdgeCompressor:
"is_tool": tool_call_id is not None, "is_tool": tool_call_id is not None,
} }
) )
elif has_xml:
self._xml_candidates.append(idx)
self._scanned_until = len(messages) self._scanned_until = len(messages)
def _has_strip_marker(self, text: str) -> bool: def _has_strip_marker(self, text: str) -> bool:
return any(marker in text for marker in self._strip_markers) return any(marker in text for marker in self._strip_markers)
# ------------------------------------------------------------------ # ------------------------------------------------------------------
# Depth-1 XML scrub # The edges: text strip + ephemeral deletion (shallow), image swap (K)
# ------------------------------------------------------------------ # ------------------------------------------------------------------
def _scrub_xml(self, messages: list[BaseMessage]) -> None: def _scrub_edge(
if not self._prune_history_xml: self, messages: list[BaseMessage], image_scrub_depth: int | None = None
return ) -> None:
# Keep the UI list only in the newest xml_scrub_depth observation
# messages; everything older (or non-image XML carriers) is stripped.
# With a live tail outside the tracked list (tail_offset > 0) the tail
# itself occupies the newest slots of the keep window.
keep_count = max(0, self._xml_scrub_depth - self._tail_offset)
keep_indices = {rec["idx"] for rec in self._tracked[-keep_count:]} if keep_count else set()
still_pending: list[int] = []
for idx in self._xml_candidates:
if idx in keep_indices:
still_pending.append(idx)
continue
if idx in self._frozen:
continue
self._strip_xml_blocks(messages[idx])
self._xml_candidates = still_pending
def _strip_xml_blocks(self, msg: BaseMessage) -> None:
content = getattr(msg, "content", None)
if not isinstance(content, list):
return
new_blocks: list[Any] = []
for block in content:
if (
isinstance(block, dict)
and block.get("type") == "text"
and self._has_strip_marker(block.get("text", ""))
):
retained_text = _without_marked_suffix(block.get("text", ""), self._strip_markers)
if retained_text:
retained_block = dict(block)
retained_block["text"] = retained_text
new_blocks.append(retained_block)
continue
new_blocks.append(block)
msg.content = new_blocks
# ------------------------------------------------------------------
# Depth-K image scrub
# ------------------------------------------------------------------
def _scrub_images(self, messages: list[BaseMessage]) -> None:
total = len(self._tracked) total = len(self._tracked)
scrub_depth = self._image_scrub_depth scrub_depth = max(1, int(image_scrub_depth or self._image_scrub_depth))
grace_limit = scrub_depth + self._pending_grace_steps grace_limit = scrub_depth + self._pending_grace_steps
text_depth = self._xml_scrub_depth
for pos, rec in enumerate(self._tracked): for pos, rec in enumerate(self._tracked):
if rec["idx"] in self._frozen: if rec["idx"] in self._frozen:
@@ -226,12 +224,25 @@ class ScrubEdgeCompressor:
# 1 == most recent image-bearing message; a live tail outside the # 1 == most recent image-bearing message; a live tail outside the
# tracked list (tail_offset > 0) counts as the newest depths. # tracked list (tail_offset > 0) counts as the newest depths.
depth = total - pos + self._tail_offset depth = total - pos + self._tail_offset
if depth < scrub_depth:
continue # active window
if self._tail_offset == 0 and pos == total - 1: if self._tail_offset == 0 and pos == total - 1:
continue # the live observation is never scrubbed continue # the live observation is never scrubbed
past_text_edge = depth > text_depth
past_image_edge = depth >= scrub_depth
if not past_text_edge and not past_image_edge:
continue # newer than both edges: byte-identical
if not past_image_edge:
# Past the text edge only: strip once, keep the screenshot.
if rec["idx"] not in self._text_scrubbed:
self._rewrite(
messages[rec["idx"]],
rec["idx"],
replacement=_KEEP_IMAGE,
is_tool=rec["is_tool"],
)
continue
summary, pending, failed, step_no = self._lookup(rec) summary, pending, failed, step_no = self._lookup(rec)
replacement: Any
if summary is not None: if summary is not None:
replacement = { replacement = {
"type": "text", "type": "text",
@@ -243,7 +254,7 @@ class ScrubEdgeCompressor:
"text": f"[visual summary unavailable; evidence at DataEngine step {step_no}]", "text": f"[visual summary unavailable; evidence at DataEngine step {step_no}]",
} }
elif pending and depth <= grace_limit: elif pending and depth <= grace_limit:
continue # grace period: retain the image, revisit next pass replacement = _KEEP_IMAGE # grace: text edits now, image swap later
elif pending: elif pending:
replacement = { replacement = {
"type": "text", "type": "text",
@@ -252,7 +263,26 @@ class ScrubEdgeCompressor:
else: else:
replacement = None # no summary job exists; drop the image silently replacement = None # no summary job exists; drop the image silently
self._resolve(messages, rec, replacement) self._resolve(messages, rec, replacement, text_edits=past_text_edge)
if self._xml_candidates:
# Imageless strip-marker carriers (a step whose screenshot failed)
# follow the same text edge, counted over every observation —
# image-bearing or not — plus the live tail: with the default
# depth 1 the list leaves as soon as any newer observation exists,
# so only the live observation ever carries an indexed element
# list. A carrier that is itself the newest observation stays.
observations = sorted({rec["idx"] for rec in self._tracked} | set(self._xml_candidates))
remaining: list[int] = []
for idx in self._xml_candidates:
newer = sum(1 for other in observations if other > idx) + self._tail_offset
if newer < text_depth:
remaining.append(idx)
continue
if idx not in self._frozen:
self._rewrite(messages[idx], idx, replacement=None, is_tool=False)
self._frozen.add(idx)
self._xml_candidates = remaining
def _lookup(self, rec: dict[str, Any]) -> tuple[str | None, bool, bool, int | None]: def _lookup(self, rec: dict[str, Any]) -> tuple[str | None, bool, bool, int | None]:
"""Mirror the legacy keying: tool_call_id first, ordinal fallback.""" """Mirror the legacy keying: tool_call_id first, ordinal fallback."""
@@ -280,42 +310,143 @@ class ScrubEdgeCompressor:
self, self,
messages: list[BaseMessage], messages: list[BaseMessage],
rec: dict[str, Any], rec: dict[str, Any],
replacement: dict[str, Any] | None, replacement: Any,
*,
text_edits: bool = True,
) -> None: ) -> None:
"""Replace the message's image blocks and freeze it permanently.""" """Rewrite one image-bearing message at the screenshot edge; freeze it
msg = messages[rec["idx"]] unless the screenshot is kept for the pending-grace window. With a text
content = getattr(msg, "content", None) edge deeper than the screenshot edge the text edits wait for their own
new_blocks: list[Any] = [] edge (``text_edits=False``); the message is then frozen only once both
if isinstance(content, list): have happened."""
for block in content: idx = rec["idx"]
if isinstance(block, dict) and block.get("type") in ("image_url", "image"): self._rewrite(
continue messages[idx],
if ( idx,
self._prune_history_xml replacement=replacement,
and isinstance(block, dict) is_tool=rec["is_tool"],
and block.get("type") == "text" text_edits=text_edits,
and self._has_strip_marker(block.get("text", ""))
):
# Freezing implies fully scrubbed: no stale UI list survives
# even when xml_scrub_depth exceeds the image depth.
retained_text = _without_marked_suffix(
block.get("text", ""), self._strip_markers
) )
if not text_edits:
return # text edge still ahead: revisit for the strip
if replacement is not _KEEP_IMAGE:
self._frozen.add(idx)
def _rewrite(
self,
msg: BaseMessage,
idx: int,
*,
replacement: Any,
is_tool: bool,
text_edits: bool = True,
) -> None:
"""The single-pass edit of one message (see the class docstring).
``replacement`` is the block that takes the first image's place (a
summary or placeholder), ``None`` to drop the image, or
:data:`_KEEP_IMAGE` to leave the image and its label in place. The
text edits (strip + ephemeral deletion) run only the first time the
message is rewritten and consume the ephemeral indices exactly once.
Any rewrite that happens *before* that first edit (the image swap when
the screenshot edge is shallower than the text edge, or a summary
arriving inside the grace window) can remove the label and the image
and so shift every later block; the surviving ephemeral indices are
then remapped to the new layout in ``additional_kwargs`` so the later
text edit still deletes the blocks that were marked, never a
neighbour that slid into their old position.
"""
content = getattr(msg, "content", None)
if not isinstance(content, list):
self._text_scrubbed.add(idx)
return
first_edit = idx not in self._text_scrubbed and text_edits
ephemeral: set[int] = set()
if first_edit:
ephemeral = self._ephemeral_indices(msg)
keep_image = replacement is _KEEP_IMAGE
new_blocks: list[Any] = []
# old position -> new position of every block that survives as itself
# (a replacement block takes the image's slot but is not the image).
position_map: dict[int, int] = {}
image_placed = False
for position, block in enumerate(content):
if position in ephemeral:
continue
if isinstance(block, dict):
kind = block.get("type")
if kind in ("image_url", "image"):
if keep_image:
position_map[position] = len(new_blocks)
new_blocks.append(block)
elif replacement is not None and not image_placed:
new_blocks.append(replacement) # in place of the image
image_placed = True
continue
if kind == "text":
text = block.get("text", "")
if not keep_image and text.strip() == SCREENSHOT_LABEL:
continue # the label leaves with the image it labelled
if first_edit and self._prune_history_xml and self._has_strip_marker(text):
retained_text = _without_marked_suffix(text, self._strip_markers)
if retained_text: if retained_text:
retained_block = dict(block) retained_block = dict(block)
retained_block["text"] = retained_text retained_block["text"] = retained_text
position_map[position] = len(new_blocks)
new_blocks.append(retained_block) new_blocks.append(retained_block)
continue continue
position_map[position] = len(new_blocks)
new_blocks.append(block) new_blocks.append(block)
if replacement is not None:
new_blocks.append(replacement)
# Defensive fallback: never leave ToolMessage or HumanMessage content empty # Defensive fallback: never leave ToolMessage or HumanMessage content empty
if not new_blocks: if not new_blocks:
new_blocks = [{"type": "text", "text": "Action dispatched." if rec["is_tool"] else ""}] new_blocks = [{"type": "text", "text": "Action dispatched." if is_tool else ""}]
position_map = {}
msg.content = new_blocks msg.content = new_blocks
self._frozen.add(rec["idx"]) if first_edit:
if rec["idx"] in self._xml_candidates: self._text_scrubbed.add(idx)
self._xml_candidates.remove(rec["idx"]) if ephemeral:
# Consumed: the indices no longer describe the block layout.
msg.additional_kwargs.pop(EPHEMERAL_BLOCKS_KEY, None)
else:
self._remap_ephemeral(msg, position_map)
@staticmethod
def _remap_ephemeral(msg: BaseMessage, position_map: dict[int, int]) -> None:
"""Rewrite the not-yet-consumed ephemeral indices of ``msg`` through
``position_map`` (old block position -> new block position); indices of
blocks that did not survive are dropped and the key is removed when
nothing survives. Only ``additional_kwargs`` is touched, the content
blocks sent to the provider stay exactly as assembled."""
kwargs = getattr(msg, "additional_kwargs", None)
if not isinstance(kwargs, dict) or EPHEMERAL_BLOCKS_KEY not in kwargs:
return
remapped: list[int] = []
for value in kwargs.get(EPHEMERAL_BLOCKS_KEY) or []:
try:
old_position = int(value)
except (TypeError, ValueError):
continue
new_position = position_map.get(old_position)
if new_position is not None and new_position not in remapped:
remapped.append(new_position)
if remapped:
kwargs[EPHEMERAL_BLOCKS_KEY] = remapped
else:
kwargs.pop(EPHEMERAL_BLOCKS_KEY, None)
@staticmethod
def _ephemeral_indices(msg: BaseMessage) -> set[int]:
kwargs = getattr(msg, "additional_kwargs", None)
if not isinstance(kwargs, dict):
return set()
indices: set[int] = set()
for value in kwargs.get(EPHEMERAL_BLOCKS_KEY) or []:
try:
indices.add(int(value))
except (TypeError, ValueError):
continue
return indices
+8 -7
View File
@@ -6,9 +6,9 @@ You are an autonomous and highly efficient Android Device Execution Agent. Your
--- ---
# 1. COGNITIVE PROTOCOL # 1. COGNITIVE PROTOCOL
You operate in a dynamic observation-reasoning-execution loop. **CRITICAL RULE: YOU MUST NEVER INVOKE A TOOL SILENTLY!** In every single turn, you **MUST FIRST output a natural language reasoning paragraph BEFORE generating any tool call.** In every turn, think through observation, planning and action in that order. Before calling an action tool or `report_task_status` (the Turn-Ending Action), your reply carries one short paragraph of connected prose reflecting those three phases; that written text is kept in the execution history and shown to the user.
- **Turn 1 Only (Initial Planning)**: Briefly decompose the overall `{{ goal }}` into high-level visual **Milestones** in your text output. Do not repeat this macro breakdown in subsequent turns. - **Turn 1 Only (Initial Planning)**: Briefly decompose the objective into high-level visual **Milestones** in your text output. Do not repeat this macro breakdown in subsequent turns.
--- ---
@@ -23,7 +23,7 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
--- ---
# 3. READING YOUR HISTORY & THE SESSION CLOCK # 3. READING YOUR HISTORY & THE SESSION CLOCK
- **Conversation = execution history**: The earlier turns of this conversation are your raw execution history. Each observation opens with `# CURRENT OBSERVATION [T+mm:ss]`, and each turn that executed a device action ends with an `--- Action Execution Result (T+mm:ss) ---` message (`Status: dispatched` means the device accepted the action; whether it had the intended effect is yours to judge from the observation that follows. `Status: failed` carries the `Error:` line). Every timestamp is a session-relative `T+mm:ss` offset from the start of this session; it is the only clock you have, for pacing waits as well as for reasoning about elapsed time. - **Conversation = execution history**: The earlier turns of this conversation are your raw execution history. Each observation opens with `# CURRENT OBSERVATION [T+mm:ss]`. Every device action runs immediately and its tool result states what the device did (e.g. `Swiped up.`); whether it had the intended effect is yours to judge from the observation that follows. Only a turn whose action failed ends with an `--- Action Execution Result (T+mm:ss) ---` message (`Status: failed` with the `Error:` line). Every timestamp is a session-relative `T+mm:ss` offset from the start of this session; it is the only clock you have, for pacing waits as well as for reasoning about elapsed time.
- **Compression**: Screenshots older than a few turns are replaced by short `--- Historical Visual Transition ---` summaries and old `--- Visible UI Elements ---` lists are dropped. Long spans render as `[Era N | Steps a–b | T+.. → T+..]` blocks (a merged synopsis, segment titles, and a per-step action ledger). Because of this, if you spot critical text that you will need later (e.g., verification codes, specific titles, search lists), **you must explicitly verbalize and write those details into your text reasoning** so they persist across turns. - **Compression**: Screenshots older than a few turns are replaced by short `--- Historical Visual Transition ---` summaries and old `--- Visible UI Elements ---` lists are dropped. Long spans render as `[Era N | Steps a–b | T+.. → T+..]` blocks (a merged synopsis, segment titles, and a per-step action ledger). Because of this, if you spot critical text that you will need later (e.g., verification codes, specific titles, search lists), **you must explicitly verbalize and write those details into your text reasoning** so they persist across turns.
{% if "search_history" in available_tools %}- **History Lookup (`search_history`)**: Compressed history keeps the step numbers and `T+` offsets but not every detail. When you need exactly what you did or saw at an earlier step (a value you read, the result of an action, which screen you were on), call `search_history` with keywords and/or the `[start, end]` step range shown in a chunk block; it searches the full stored history (screen descriptions, actions, results, reasoning, tool calls, notes, on-screen text) and every hit carries its step number. It is a pre-decision helper: read its answer, then act.{% endif %} {% if "search_history" in available_tools %}- **History Lookup (`search_history`)**: Compressed history keeps the step numbers and `T+` offsets but not every detail. When you need exactly what you did or saw at an earlier step (a value you read, the result of an action, which screen you were on), call `search_history` with keywords and/or the `[start, end]` step range shown in a chunk block; it searches the full stored history (screen descriptions, actions, results, reasoning, tool calls, notes, on-screen text) and every hit carries its step number. It is a pre-decision helper: read its answer, then act.{% endif %}
{% if "replay_steps" in available_tools %}- **Step Replay (`replay_steps`)**: Replays one earlier step (or a short range) exactly as your context showed it — screen description, reasoning, tool calls with their results, the action and its execution result. Use the step numbers shown in the compressed history{% if "search_history" in available_tools %} or returned by `search_history`{% endif %}.{% endif %} {% if "replay_steps" in available_tools %}- **Step Replay (`replay_steps`)**: Replays one earlier step (or a short range) exactly as your context showed it — screen description, reasoning, tool calls with their results, the action and its execution result. Use the step numbers shown in the compressed history{% if "search_history" in available_tools %} or returned by `search_history`{% endif %}.{% endif %}
@@ -39,12 +39,13 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
--- ---
# 5. ACTION RULES # 5. ACTION RULES
1. **Normalized Coordinates (`0-1000` Scale)**: Always use normalized `[x, y]` coordinates in `0-1000` scale for any location-based target (e.g., `target=[500, 600]` or `click_sequence=[[300, 400], [500, 600]]`). Every coordinate target must carry `target_description` (one entry per point for `click_sequence`): a short, meaningful name for what you are aiming at, as a user would call it (e.g. 'play button', 'video body'), never a generic 'element' or 'button'. Nothing is inferred on your behalf: it is recorded as your own statement, and your history shows it marked as self-described. 1. **Prefer the element index.** The `--- Visible UI Elements ---` list numbers every element it found (`[3] Text: 'Wi-Fi' | Bounds: ...`). When your target is listed, address it by that index (`target=3`, a bare integer) with `click`, `long_press` or `input_text`: the tap lands on the element's center and the element's own text and bounds are recorded with the action. Heed the overlap warnings in the list; an overlapped element may not receive the tap. Indices belong to the screen they were listed on: swiping, scrolling or a page update renumbers them, so only ever use an index from the list in the current observation, never one remembered from an earlier turn. Within one turn, every index you call (even after an earlier action in the same turn) refers to the list in the current observation, not to the screen your earlier action produced.
2. **Do not determine element coordinates based on guesswork.** If the element is visible in the screenshot but its coordinates are missing from the provided `--- Visible UI Elements ---` list, please use the `ask_explorer` tool to locate the element first. 2. **Normalized Coordinates (`0-1000` Scale) otherwise.** When the target is visible but not listed, use normalized `[x, y]` coordinates in `0-1000` scale (e.g., `target=[500, 600]`; `click_sequence` always takes coordinate pairs, e.g. `sequence=[[300, 400], [500, 600]]`). Every coordinate target must carry `target_description` (one entry per point for `click_sequence`): a short, meaningful name for what you are aiming at, as a user would call it (e.g. 'play button', 'video body'), never a generic 'element' or 'button'. Nothing is inferred on your behalf: it is recorded as your own statement, and your history shows it marked as self-described.
{% if "manage_app" in available_tools %}3. **App Launching**: To start or reset an application, always use `manage_app(action="launch", app_name="...")` or `action="stop"`. Do not manually swipe across home screens.{% endif %} 3. **Do not determine element coordinates based on guesswork.** If the element is visible in the screenshot but neither listed nor safely locatable from the screenshot, use the `ask_explorer` tool to locate it first; its candidates join the element list and can be addressed by index or by coordinate.
{% if "manage_app" in available_tools %}4. **App Launching**: To start or reset an application, always use `manage_app(action="launch", app_name="...")` or `action="stop"`. Do not manually swipe across home screens.{% endif %}
--- ---
# 6. TERMINATION PROTOCOL (`report_task_status`) # 6. TERMINATION PROTOCOL (`report_task_status`)
- **Completion (`status="completed"`)**: ONLY call `report_task_status(status="completed", ...)` after you have **visually verified** from the latest screen that the user's objective (`{{ goal }}`) is fully achieved. - **Completion (`status="completed"`)**: ONLY call `report_task_status(status="completed", ...)` after you have **visually verified** from the latest screen that the objective is fully achieved.
- **Failure (`status="failed"`)**: If the task is genuinely impossible due to paywalls, unrecoverable crashes, or hitting an absolute dead end after sensible fallbacks, call `report_task_status(status="failed", ...)` with the exact blocking reason. - **Failure (`status="failed"`)**: If the task is genuinely impossible due to paywalls, unrecoverable crashes, or hitting an absolute dead end after sensible fallbacks, call `report_task_status(status="failed", ...)` with the exact blocking reason.
+299 -96
View File
@@ -22,10 +22,19 @@ the prompt is built every turn from a session :class:`TranscriptLedger` —
a byte-stable system prefix, the committed earlier turns (append-only; old UI a byte-stable system prefix, the committed earlier turns (append-only; old UI
lists stripped at depth 1, screenshots resolved to visual summaries at depth lists stripped at depth 1, screenshots resolved to visual summaries at depth
K, long spans chunk-compressed), and a fresh tail carrying the current K, long spans chunk-compressed), and a fresh tail carrying the current
observation under a ``# CURRENT OBSERVATION [T+mm:ss]`` header. Each observation under a ``# CURRENT OBSERVATION [T+mm:ss]`` header. Actions run
committed turn ends with an ``--- Action Execution Result (T+mm:ss) ---`` synchronously and their tool messages already carry the outcome; only a turn
message, so every timestamp the model sees is a session-relative offset — the whose action failed ends with an ``--- Action Execution Result (T+mm:ss) ---``
same clock the video analyzer uses for the session recording. message (it carries the error). Every timestamp the model sees is a
session-relative offset — the same clock the video analyzer uses for the
session recording.
Per-turn notices on the tail (the reasoning reminder after a silent turn, the
no-tool-call nudge, the final-turn warning, the user-guidance wrapper) are
marked ephemeral: the ledger's scrub edge deletes them once the message leaves
the recent window, so they never reach the frozen history or a chunk capsule.
The verbatim body of a user instruction is the exception: it stays in the
active window until its turn is chunked, whose user lines then carry it on.
""" """
import asyncio import asyncio
@@ -45,9 +54,12 @@ from langchain_core.messages import (
) )
from artemis.agents.flash.summarizer import VisualStepSummarizer, build_focus_context from artemis.agents.flash.summarizer import VisualStepSummarizer, build_focus_context
from artemis.agents.operator.prompts import REASONING_REMINDER, UserGuidance, render_user_guidance
from artemis.agents.validator.tool_declarations import ( from artemis.agents.validator.tool_declarations import (
ASK_EXPLORER_TOOL, ASK_EXPLORER_TOOL,
CLICK_SEQUENCE_TOOL, CLICK_SEQUENCE_TOOL,
LIST_NOTES_TOOL,
READ_NOTE_TOOL,
REPORT_TASK_STATUS_TOOL, REPORT_TASK_STATUS_TOOL,
VALIDATOR_TOOLS_DECLARATION, VALIDATOR_TOOLS_DECLARATION,
capture_screenshot_and_parse_ui, capture_screenshot_and_parse_ui,
@@ -66,7 +78,8 @@ from artemis.graph.perception import _check_injected_instruction_file
from artemis.graph.state import State from artemis.graph.state import State
from artemis.llm.structured import ParseFailure, parse_structured from artemis.llm.structured import ParseFailure, parse_structured
from artemis.mcp.action_executor import McpActionExecutor from artemis.mcp.action_executor import McpActionExecutor
from artemis.memory.transcript import PRO_UI_LIST_MARKER, TranscriptLedger from artemis.mcp.observation import observe
from artemis.memory.transcript import PRO_UI_LIST_MARKER, TranscriptLedger, mark_ephemeral
from artemis.services.llm import ( from artemis.services.llm import (
RobustChatModelWrapper, RobustChatModelWrapper,
acomplete, acomplete,
@@ -85,12 +98,6 @@ from artemis.utils.logger import get_logger
logger = get_logger(__name__) logger = get_logger(__name__)
#: Reminder carried by the first observation only (the system prompt states the rule).
_REASONING_FIRST_REMINDER = (
"CRITICAL RULE: In every single turn, you MUST FIRST output a natural"
" language reasoning/explanation paragraph BEFORE invoking any tool call."
)
_NO_TOOL_CALL_NOTICE = ( _NO_TOOL_CALL_NOTICE = (
"You did not call any tools last turn. Please make progress by calling an" "You did not call any tools last turn. Please make progress by calling an"
" action tool or 'report_task_status'." " action tool or 'report_task_status'."
@@ -116,17 +123,17 @@ class _TurnRecord:
def result(self) -> dict | None: def result(self) -> dict | None:
"""The turn's execution result in the validator-report shape. """The turn's execution result in the validator-report shape.
``None`` when no device action ran (helper-only turns have no result ``None`` unless an action failed: Flash executes synchronously and each
message, as in Pro). action's tool message already carries its outcome, so a result message
would only repeat it. A failure is worth its own message because it
carries the error the turn ended on.
""" """
if not self.actions:
return None
for name, status, text in self.actions: for name, status, text in self.actions:
if status != "success": if status != "success":
# Executor messages name the action themselves ("Error # Executor messages name the action themselves ("Error
# executing click: ..."); prefix only the ones that don't. # executing click: ..."); prefix only the ones that don't.
return {"status": "failed", "error": text if name in text else f"{name}: {text}"} return {"status": "failed", "error": text if name in text else f"{name}: {text}"}
return {"status": "dispatched"} return None
class FlashRunner: class FlashRunner:
@@ -164,7 +171,7 @@ class FlashRunner:
if self.step_summarizer_cfg.enabled if self.step_summarizer_cfg.enabled
else None else None
) )
# Publish the service on the composition root (ctx.step_memory, M2) so # Publish the service on the composition root (ctx.step_memory) so
# any co-resident consumer shares this run's summary runtime. # any co-resident consumer shares this run's summary runtime.
if self.summarizer is not None and getattr(ctx, "step_memory", None) is None: if self.summarizer is not None and getattr(ctx, "step_memory", None) is None:
try: try:
@@ -187,7 +194,13 @@ class FlashRunner:
return getattr(setup, "video_recording_tools_enabled", False) is True return getattr(setup, "video_recording_tools_enabled", False) is True
def _get_tools(self) -> list: def _get_tools(self) -> list:
tools = list(VALIDATOR_TOOLS_DECLARATION) # Flash has no tool that writes notes and its prompt never teaches
# them, so the Validator set's note readers are left out here.
tools = [
t
for t in VALIDATOR_TOOLS_DECLARATION
if t is not READ_NOTE_TOOL and t is not LIST_NOTES_TOOL
]
tools.insert(1, CLICK_SEQUENCE_TOOL) tools.insert(1, CLICK_SEQUENCE_TOOL)
tools.append(ASK_EXPLORER_TOOL) tools.append(ASK_EXPLORER_TOOL)
# Helper tools shared with the Pro operator: the history tools are # Helper tools shared with the Pro operator: the history tools are
@@ -235,6 +248,9 @@ class FlashRunner:
image_scrub_depth=getattr(cfg, "image_scrub_depth", 3), image_scrub_depth=getattr(cfg, "image_scrub_depth", 3),
pending_grace_steps=getattr(cfg, "pending_grace_steps", 3), pending_grace_steps=getattr(cfg, "pending_grace_steps", 3),
xml_scrub_depth=getattr(cfg, "xml_scrub_depth", 1), xml_scrub_depth=getattr(cfg, "xml_scrub_depth", 1),
image_scrub_depth_relaxed=getattr(cfg, "image_scrub_depth_relaxed", None),
context_budget_tokens=getattr(cfg, "context_budget_tokens", None),
start_ratio=getattr(cfg, "start_ratio", None),
session_start=session_start if isinstance(session_start, (int, float)) else None, session_start=session_start if isinstance(session_start, (int, float)) else None,
) )
if engine is not None: if engine is not None:
@@ -286,14 +302,19 @@ class FlashRunner:
# Per-turn helpers (observe / think) # Per-turn helpers (observe / think)
# ------------------------------------------------------------------ # ------------------------------------------------------------------
async def _read_injected_instruction(self) -> tuple[str | None, str | None]: async def _read_injected_instruction(
"""Consumes this turn's injected instruction: ``(instruction, notice)``. self, offset_label: str | None = None
) -> tuple[str | None, UserGuidance | None]:
"""Consumes this turn's injected instruction: ``(instruction, guidance)``.
``instruction`` is the user's text verbatim — what the step record, ``instruction`` is the user's text verbatim — what the step record,
the chunk ledger and the visual-transition lens receive (the same the chunk ledger and the visual-transition lens receive (the same
shape Pro stamps). ``notice`` wraps it with the operator-facing shape Pro stamps). ``guidance`` is the shared user-guidance pair
directive and rides only on the observation tail. Both are None (:func:`render_user_guidance`, the Pro rendering without a plan to
when nothing was injected. edit): the ephemeral ``--- User Guidance ---`` wrapper and the
persistent verbatim body, both riding on the observation tail.
``offset_label`` stamps the body with the session offset. Both are
None when nothing was injected.
""" """
if not (self.ctx.data_engine and self.ctx.data_engine.base_dir): if not (self.ctx.data_engine and self.ctx.data_engine.base_dir):
return None, None return None, None
@@ -308,18 +329,13 @@ class FlashRunner:
if not (injected_payload and injected_payload.get("instruction")): if not (injected_payload and injected_payload.get("instruction")):
return None, None return None, None
instruction = str(injected_payload["instruction"]) instruction = str(injected_payload["instruction"])
notice = ( guidance = render_user_guidance(
"[REAL-TIME INJECTED INSTRUCTION from user]:" instruction,
f" {instruction}\nYou MUST immediately" release_loop=bool(injected_payload.get("release_loop")),
" follow this instruction and adjust your plan/actions." has_plan=False,
offset_label=offset_label,
) )
if injected_payload.get("release_loop"): return instruction, guidance
notice += (
"\nThe user has explicitly authorized stopping any"
" ongoing monitoring loop; you may now wrap up and"
" complete the task."
)
return instruction, notice
def _build_tail( def _build_tail(
self, self,
@@ -328,19 +344,28 @@ class FlashRunner:
img_bytes, img_bytes,
xml_list, xml_list,
*, *,
injected: str | None = None, injected: UserGuidance | None = None,
notices: list[str] | None = None, notices: list[str] | None = None,
is_final: bool = False, is_final: bool = False,
previous_turn_silent: bool = False,
) -> HumanMessage: ) -> HumanMessage:
"""Builds this turn's observation tail (Pro observation shape). """Builds this turn's observation tail (Pro observation shape).
Header, screenshot and UI list carry the same markers as the Pro Header, screenshot and UI list carry the same markers as the Pro
operator's tail so the ledger's scrub edge treats both alike. operator's tail so the ledger's scrub edge treats both alike. The
objective is stated once, in the system prompt; the tail never repeats
it. Every per-turn notice (nudges, the user-guidance wrapper, the
final-turn warning, the reasoning reminder after a silent turn) is
marked ephemeral so it leaves the transcript with the recent window.
The verbatim body of an injected instruction is the one exception: it
is a regular block, so a standing instruction stays in the active
window until the turn is chunked (the chunk's user lines carry it on).
""" """
blocks: list[dict] = [] # ``turns`` is kept for call-site symmetry: the tail has the same shape
if turns == 1: # on every turn.
blocks.append({"type": "text", "text": f"Your objective is: {self.goal}"}) blocks: list[dict] = [
blocks.append({"type": "text", "text": f"# CURRENT OBSERVATION [{ledger.elapsed_label()}]"}) {"type": "text", "text": f"# CURRENT OBSERVATION [{ledger.elapsed_label()}]"}
]
if img_bytes: if img_bytes:
img_b64 = base64.b64encode(img_bytes).decode("utf-8") img_b64 = base64.b64encode(img_bytes).decode("utf-8")
blocks.append({"type": "text", "text": "--- Current Screenshot ---"}) blocks.append({"type": "text", "text": "--- Current Screenshot ---"})
@@ -349,15 +374,26 @@ class FlashRunner:
) )
if xml_list: if xml_list:
blocks.append({"type": "text", "text": f"{PRO_UI_LIST_MARKER}\n{xml_list}"}) blocks.append({"type": "text", "text": f"{PRO_UI_LIST_MARKER}\n{xml_list}"})
for notice in notices or []: ephemeral: list[int] = []
blocks.append({"type": "text", "text": notice}) # (text, ephemeral) in tail order.
per_turn: list[tuple[str, bool]] = [(text, True) for text in (notices or [])]
if injected: if injected:
blocks.append({"type": "text", "text": injected}) # The wrapper is per-turn; the verbatim body right after it is a
# regular block that stays until the turn is chunked.
per_turn.append((injected.wrapper, True))
per_turn.append((injected.body, False))
if is_final: if is_final:
blocks.append({"type": "text", "text": _FINAL_TURN_WARNING}) per_turn.append((_FINAL_TURN_WARNING, True))
if turns == 1: if previous_turn_silent:
blocks.append({"type": "text", "text": _REASONING_FIRST_REMINDER}) per_turn.append((REASONING_REMINDER, True))
return HumanMessage(content=blocks) for text, is_ephemeral in per_turn:
if is_ephemeral:
ephemeral.append(len(blocks))
blocks.append({"type": "text", "text": text})
tail = HumanMessage(content=blocks)
if ephemeral:
mark_ephemeral(tail, ephemeral)
return tail
def _extract_response_text(self, response) -> str: def _extract_response_text(self, response) -> str:
"""Extracts the natural-language thought text from the model response.""" """Extracts the natural-language thought text from the model response."""
@@ -368,6 +404,22 @@ class FlashRunner:
) )
return raw_text return raw_text
def _extract_response_thinking(self, response) -> str:
"""Extracts the model's native thinking blocks (thought summaries), if any.
Providers running with ``include_thoughts`` return them as ``thinking``
content blocks next to the ``text`` blocks; they are recorded on the step
as ``operator_native_thinking`` and never fed back as reply text.
"""
if not isinstance(response.content, list):
return ""
parts = [
str(b.get("thinking") or "")
for b in response.content
if isinstance(b, dict) and b.get("type") == "thinking"
]
return "\n".join(p for p in parts if p.strip())
def _token_usage_from_response(self, response) -> dict | None: def _token_usage_from_response(self, response) -> dict | None:
"""Extracts token usage metadata from the model response, if present.""" """Extracts token usage metadata from the model response, if present."""
step_token_usage = None step_token_usage = None
@@ -435,12 +487,20 @@ class FlashRunner:
"total_tokens": max(1, prompt_tokens + completion_tokens), "total_tokens": max(1, prompt_tokens + completion_tokens),
} }
def _resolve_token_usage(self, response, messages: list[BaseMessage], raw_text: str) -> dict: def _resolve_token_usage(
"""Extracts token usage from response metadata, estimating as fallback.""" self, response, messages: list[BaseMessage], raw_text: str
) -> tuple[dict, bool]:
"""Token usage for the call and whether the provider measured it.
Falls back to an estimate when the response carries no usage
metadata; the flag lets the caller keep an estimate out of the
ledger's chars-per-token calibration (an estimate is itself derived
from character counts, so calibrating on it would be circular).
"""
step_token_usage = self._token_usage_from_response(response) step_token_usage = self._token_usage_from_response(response)
if not step_token_usage or step_token_usage.get("total_tokens", 0) <= 0: if step_token_usage and step_token_usage.get("total_tokens", 0) > 0:
step_token_usage = self._estimate_token_usage(messages, raw_text) return step_token_usage, True
return step_token_usage return self._estimate_token_usage(messages, raw_text), False
def _record_llm_trace(self, step_token_usage: dict, raw_text: str) -> None: def _record_llm_trace(self, step_token_usage: dict, raw_text: str) -> None:
"""Records the llm_call trace for this turn in the DataEngine.""" """Records the llm_call trace for this turn in the DataEngine."""
@@ -490,6 +550,7 @@ class FlashRunner:
pre_screenshot_bytes, pre_screenshot_bytes,
xml_list, xml_list,
messages: list[BaseMessage], messages: list[BaseMessage],
native_text: str | None = None,
) -> dict: ) -> dict:
"""Records the final step, acknowledges the tool call, and flushes.""" """Records the final step, acknowledges the tool call, and flushes."""
final_report = args final_report = args
@@ -502,6 +563,7 @@ class FlashRunner:
ui_tree=xml_list, ui_tree=xml_list,
action_taken={"action": "report_task_status", "args": args}, action_taken={"action": "report_task_status", "args": args},
operator_raw_thinking=raw_text, operator_raw_thinking=raw_text,
operator_native_thinking=native_text or None,
last_execution_result={"result": "Task completed with final report."}, last_execution_result={"result": "Task completed with final report."},
extra_metadata={"token_usage": step_token_usage}, extra_metadata={"token_usage": step_token_usage},
) )
@@ -519,25 +581,56 @@ class FlashRunner:
await self.summarizer.flush() await self.summarizer.flush()
return final_report return final_report
async def _capture_post_screenshot(self, exec_result, name: str, action_names): async def _capture_post_screenshot(
"""Returns the post-action screenshot, capturing a fallback if needed.""" self, exec_result, name: str, action_names, state: State | None = None
):
"""Returns the post-action screenshot, capturing a fallback if needed.
When a device action produced no observation (a failed action, an
unknown tool, ...) the screen is observed through the very same
pipeline the executor's success path uses (:func:`observe`), so the
next observation tail carries the indexed minimal UI list and
``state`` gets the same refreshed screenshot path and indexed
elements. Nothing is captured for non-action tools.
"""
post_img_bytes = exec_result.screenshot_bytes post_img_bytes = exec_result.screenshot_bytes
if not post_img_bytes and name in action_names: if post_img_bytes or name not in action_names:
return post_img_bytes
try: try:
controller = UnifiedMobileController(self.ctx) obs, screenshot_bytes = await observe(self.ctx, self.controller, settle_ms=0)
screen_data = await controller.get_screen_data()
post_img_bytes = base64.b64decode(screen_data.base64)
if not exec_result.ui_elements_text:
exec_result.ui_elements_text = screen_data.elements
except Exception as shot_err: except Exception as shot_err:
logger.warning(f"Failed to capture fallback screenshot in FlashRunner: {shot_err}") logger.warning(f"Failed to capture fallback screenshot in FlashRunner: {shot_err}")
return post_img_bytes return post_img_bytes
if not obs.ok:
logger.warning(f"Failed to capture fallback screenshot in FlashRunner: {obs.message}")
return post_img_bytes
post_img_bytes = screenshot_bytes
if not exec_result.ui_elements_text:
exec_result.ui_elements_text = obs.elements_text
if not exec_result.screenshot_path and obs.screenshot_path:
exec_result.screenshot_path = obs.screenshot_path
if state is not None:
if obs.hierarchy_ok:
state.indexed_points = [el["center"] for el in obs.elements]
state.indexed_elements = obs.elements
if obs.screenshot_path:
state.latest_screenshot = obs.screenshot_path
return post_img_bytes
def _extract_normalized_coordinates(self, name: str, args: dict): def _extract_normalized_coordinates(self, name: str, args: dict, resolved=None):
"""Extracts and enriches coordinate metadata for the recorded action.""" """Extracts and enriches coordinate metadata for the recorded action.
``resolved`` is the normalized point the executor sent a point action
to; it is the coordinate of record when the model addressed an element
by index (the arguments then carry no point at all).
"""
norm_coords = None norm_coords = None
norm_start = None norm_start = None
norm_end = None norm_end = None
if name in ("click", "tap", "long_press", "input_text") and (
isinstance(resolved, (list, tuple)) and len(resolved) == 2
):
return [int(resolved[0]), int(resolved[1])], None, None
if name == "swipe": if name == "swipe":
kind, target_val, _ = parse_swipe_parameters(args) kind, target_val, _ = parse_swipe_parameters(args)
if kind == "coords" and isinstance(target_val, list) and len(target_val) == 4: if kind == "coords" and isinstance(target_val, list) and len(target_val) == 4:
@@ -567,6 +660,40 @@ class FlashRunner:
norm_coords = [int(float(nums[0])), int(float(nums[1]))] norm_coords = [int(float(nums[0])), int(float(nums[1]))]
return norm_coords, norm_start, norm_end return norm_coords, norm_start, norm_end
def _build_action_record(self, name: str, args: dict, exec_result) -> dict:
"""The action as the step record and the visual lens see it.
Flash records the model's own 0–1000 coordinates verbatim (an index
target records the point the executor resolved it to); the explicit
space marker keeps every later normalization pass (agent-friendly
steps, MCP inspector) a no-op on this record. The tool arguments ride
along under ``args``, and the target semantics the executor validated
(observed element fields for an index, the model's own
``target_description`` for a point) land top level in the shape Pro
records.
"""
metadata = getattr(exec_result, "metadata", None) or {}
resolved = metadata.get("target_coordinates")
norm_coords, norm_start, norm_end = self._extract_normalized_coordinates(
name, args, resolved=resolved
)
given = args.get("target") or args.get("coordinates") or args.get("sequence")
if isinstance(given, (int, float)) and not isinstance(given, bool):
given = None # An element index is not a coordinate.
action_dict = {
"action": name,
"coordinates": given or norm_coords,
COORDINATE_SPACE_KEY: COORDINATE_SPACE_NORMALIZED,
"args": args,
}
if norm_coords:
action_dict["normalized_coordinates"] = norm_coords
if norm_start and norm_end:
action_dict["normalized_start_coordinates"] = norm_start
action_dict["normalized_end_coordinates"] = norm_end
action_dict.update(metadata.get("target_semantics") or {})
return action_dict
def _record_action_step( def _record_action_step(
self, self,
name: str, name: str,
@@ -578,6 +705,8 @@ class FlashRunner:
xml_list, xml_list,
post_img_bytes, post_img_bytes,
injected: str | None = None, injected: str | None = None,
native_text: str | None = None,
action_dict: dict | None = None,
): ):
"""Records telemetry / step in DataEngine; returns the step id or None.""" """Records telemetry / step in DataEngine; returns the step id or None."""
recorded_step_id = None recorded_step_id = None
@@ -585,31 +714,8 @@ class FlashRunner:
if self.ctx.data_engine.current_step_id is None: if self.ctx.data_engine.current_step_id is None:
self.ctx.data_engine.allocate_step_id() self.ctx.data_engine.allocate_step_id()
norm_coords, norm_start, norm_end = self._extract_normalized_coordinates(name, args) if action_dict is None:
action_dict = self._build_action_record(name, args, exec_result)
# Flash records the model's own 0–1000 coordinates verbatim; the
# explicit space marker keeps every later normalization pass
# (agent-friendly steps, MCP inspector) a no-op on this record.
action_dict = {
"action": name,
"coordinates": (
args.get("target")
or args.get("coordinates")
or args.get("sequence")
or norm_coords
),
COORDINATE_SPACE_KEY: COORDINATE_SPACE_NORMALIZED,
"args": args,
}
if norm_coords:
action_dict["normalized_coordinates"] = norm_coords
if norm_start and norm_end:
action_dict["normalized_start_coordinates"] = norm_start
action_dict["normalized_end_coordinates"] = norm_end
# The model's own statement of what it aimed at (target_description),
# validated and shaped by the executor into the fields Pro records.
action_dict.update((exec_result.metadata or {}).get("target_semantics") or {})
succeeded = exec_result.status == "success" succeeded = exec_result.status == "success"
last_execution_result = { last_execution_result = {
@@ -631,6 +737,7 @@ class FlashRunner:
ui_tree=(exec_result.ui_elements_text or xml_list), ui_tree=(exec_result.ui_elements_text or xml_list),
action_taken=action_dict, action_taken=action_dict,
operator_raw_thinking=raw_text, operator_raw_thinking=raw_text,
operator_native_thinking=native_text or None,
last_execution_result=last_execution_result, last_execution_result=last_execution_result,
extra_metadata=extra_metadata, extra_metadata=extra_metadata,
) )
@@ -664,21 +771,35 @@ class FlashRunner:
action_sequence: int, action_sequence: int,
turn: _TurnRecord, turn: _TurnRecord,
injected: str | None = None, injected: str | None = None,
native_text: str | None = None,
index_elements: list[dict] | None = None,
): ):
"""Executes one tool call, records it, and updates loop state. """Executes one tool call, records it, and updates loop state.
The tool message carries the outcome text only; the post-action The tool message carries the outcome text only; the post-action
screenshot and UI list become the next turn's observation tail. screenshot and UI list become the next turn's observation tail.
Returns the (possibly updated) pre_screenshot_bytes, xml_list, and ``index_elements`` is the turn's element snapshot index targets
action_sequence for the next iteration. resolve against (see :meth:`_process_tool_calls`); ``None`` resolves
against the state's current list. Returns the (possibly updated)
pre_screenshot_bytes, xml_list, and action_sequence for the next
iteration.
""" """
# Dynamic dispatch set: manifest device actions plus any backend # Dynamic dispatch set: manifest device actions plus any backend
# extension tools, so extension steps are recorded like actions. # extension tools, so extension steps are recorded like actions.
action_names = self.executor.action_tool_names action_names = self.executor.action_tool_names
try: try:
exec_result = await self.executor.execute(name, args, tc_id, state) exec_result = await self.executor.execute(
name, args, tc_id, state, index_elements=index_elements
)
post_img_bytes = await self._capture_post_screenshot(exec_result, name, action_names) post_img_bytes = await self._capture_post_screenshot(
exec_result, name, action_names, state
)
# One record shape for the DataEngine step and the visual lens.
action_dict = (
self._build_action_record(name, args, exec_result) if name in action_names else None
)
# Record telemetry / step in DataEngine # Record telemetry / step in DataEngine
recorded_step_id = None recorded_step_id = None
@@ -693,6 +814,8 @@ class FlashRunner:
xml_list, xml_list,
post_img_bytes, post_img_bytes,
injected=injected, injected=injected,
native_text=native_text,
action_dict=action_dict,
) )
if name in action_names: if name in action_names:
@@ -705,7 +828,9 @@ class FlashRunner:
self.summarizer.dispatch( self.summarizer.dispatch(
step_number=action_sequence, step_number=action_sequence,
action_name=name, action_name=name,
action_args=args, # The recorded action minus its verb: the same shape the Pro
# summarizer hands the lens, so the phrase renders alike.
action_args={k: v for k, v in action_dict.items() if k != "action"},
pre_img_bytes=pre_screenshot_bytes, pre_img_bytes=pre_screenshot_bytes,
post_img_bytes=post_img_bytes, post_img_bytes=post_img_bytes,
exec_outcome=exec_result.text_summary, exec_outcome=exec_result.text_summary,
@@ -780,12 +905,33 @@ class FlashRunner:
action_sequence: int, action_sequence: int,
turn: _TurnRecord, turn: _TurnRecord,
injected: str | None = None, injected: str | None = None,
native_text: str | None = None,
): ):
"""Dispatches the turn's tool calls. """Dispatches the turn's tool calls.
Every index target of the turn resolves against the element list the
model decided on: the list of the observation this turn was built
from, snapshotted here before the first action. Each action still
refreshes ``state.indexed_elements`` (that refreshed list is what the
next observation shows), so without the snapshot ``click(5)`` after
``click(3)`` would be resolved against the screen after the first
click.
The snapshot is the very list object the observation left on the
state (not a copy), and observations replace the state's list rather
than mutating it, so ``ask_explorer`` candidates registered before
any action land in the snapshot by construction. When a helper tool
appends candidates after an action has already replaced the state's
list, the explorer numbered them against that replaced list, so the
snapshot is re-taken from it: indices the explorer just handed out
resolve correctly, at the cost of the pre-action numbering (mixing
the two in one turn is the model's error, not a supported case).
Returns (final_report, pre_screenshot_bytes, xml_list, action_sequence); Returns (final_report, pre_screenshot_bytes, xml_list, action_sequence);
final_report is non-None only when 'report_task_status' was called. final_report is non-None only when 'report_task_status' was called.
""" """
action_names = self.executor.action_tool_names
index_elements = self._turn_index_snapshot(state)
for tc in tool_calls: for tc in tool_calls:
name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"] name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
args = tc.get("args") or {} args = tc.get("args") or {}
@@ -802,9 +948,12 @@ class FlashRunner:
pre_screenshot_bytes, pre_screenshot_bytes,
xml_list, xml_list,
messages, messages,
native_text=native_text,
) )
return final_report, pre_screenshot_bytes, xml_list, action_sequence return final_report, pre_screenshot_bytes, xml_list, action_sequence
elements_before = getattr(state, "indexed_elements", None)
before_len = len(elements_before) if isinstance(elements_before, list) else 0
pre_screenshot_bytes, xml_list, action_sequence = await self._execute_and_record_action( pre_screenshot_bytes, xml_list, action_sequence = await self._execute_and_record_action(
name, name,
args, args,
@@ -818,9 +967,46 @@ class FlashRunner:
action_sequence, action_sequence,
turn, turn,
injected=injected, injected=injected,
native_text=native_text,
index_elements=index_elements,
) )
if name not in action_names:
index_elements = self._extend_index_snapshot(state, index_elements, before_len)
return None, pre_screenshot_bytes, xml_list, action_sequence return None, pre_screenshot_bytes, xml_list, action_sequence
@staticmethod
def _turn_index_snapshot(state: State) -> list[dict]:
"""The element list this turn's index targets resolve against.
The state's list object itself, so in-place appends (explorer
candidates) before any action join the snapshot; an empty list is
installed on the state when there is none, for the same reason.
"""
elements = getattr(state, "indexed_elements", None)
if not isinstance(elements, list):
elements = []
try:
state.indexed_elements = elements
except (AttributeError, TypeError, ValueError):
pass
return elements
@staticmethod
def _extend_index_snapshot(state: State, snapshot: list[dict], before_len: int) -> list[dict]:
"""Re-takes the snapshot after a helper tool appended candidates to a
list that is no longer the snapshot (an action replaced it first).
Appends into the snapshot itself need nothing: the explorer numbered
them against the snapshot. Only growth of a different list object
moves the snapshot, since that is the list the new numbers refer to.
"""
current = getattr(state, "indexed_elements", None)
if not isinstance(current, list) or current is snapshot:
return snapshot
if len(current) > before_len:
return current
return snapshot
async def _prepare_conversation(self, state: State, tools_declaration: list): async def _prepare_conversation(self, state: State, tools_declaration: list):
"""Installs the static prefix and captures the initial device state.""" """Installs the static prefix and captures the initial device state."""
ledger = self._build_ledger() ledger = self._build_ledger()
@@ -883,8 +1069,10 @@ class FlashRunner:
previous_turn = None previous_turn = None
# Check for real-time injected instructions: the verbatim text # Check for real-time injected instructions: the verbatim text
# is stamped on the step, the wrapped notice goes to the tail. # is stamped on the step, the guidance pair goes to the tail.
injected, injected_notice = await self._read_injected_instruction() injected, injected_guidance = await self._read_injected_instruction(
ledger.elapsed_label()
)
# Tool restriction on the final turn (bounded loops only) # Tool restriction on the final turn (bounded loops only)
is_final = limit is not None and turns == limit is_final = limit is not None and turns == limit
@@ -893,9 +1081,12 @@ class FlashRunner:
turns, turns,
current_pre_screenshot_bytes, current_pre_screenshot_bytes,
current_xml_list, current_xml_list,
injected=injected_notice, injected=injected_guidance,
notices=pending_notices, notices=pending_notices,
is_final=is_final, is_final=is_final,
# The ledger judged the turn it just committed: a bare tool
# call (or thinking only) earns a one-line reminder, no bounce.
previous_turn_silent=bool(getattr(ledger, "last_turn_silent", False)),
) )
pending_notices = [] pending_notices = []
@@ -913,9 +1104,20 @@ class FlashRunner:
# Extract thought and tool calls # Extract thought and tool calls
raw_text = self._extract_response_text(response) raw_text = self._extract_response_text(response)
native_text = self._extract_response_thinking(response)
# Extract token usage metadata from response # Extract token usage metadata from response
step_token_usage = self._resolve_token_usage(response, messages, raw_text) step_token_usage, measured = self._resolve_token_usage(response, messages, raw_text)
# The ledger's compaction thresholds read the runner's own prompt
# size, never the session-wide meter (which any agent overwrites).
# The prompt as sent (``messages`` minus the just-appended reply)
# rides along so the ledger can calibrate its chars-per-token
# ratio — only from a provider measurement, never from the
# estimate (same as the Pro operator).
ledger.record_prompt_tokens(
step_token_usage.get("prompt_tokens"),
messages=messages[:-1] if measured else None,
)
self._record_llm_trace(step_token_usage, raw_text) self._record_llm_trace(step_token_usage, raw_text)
@@ -952,6 +1154,7 @@ class FlashRunner:
action_sequence, action_sequence,
turn, turn,
injected=injected, injected=injected,
native_text=native_text,
) )
ledger.stage_turn(messages[turn_base:]) ledger.stage_turn(messages[turn_base:])
previous_turn = turn previous_turn = turn
+73 -11
View File
@@ -260,24 +260,83 @@ class VisualStepSummarizer(StepMemoryService):
def _action_phrase(action_name: str, action_args: dict[str, Any] | None) -> str: def _action_phrase(action_name: str, action_args: dict[str, Any] | None) -> str:
"""The action as every other history reader sees it (``format_actions_clean``). """The action as every other history reader sees it (``format_actions_clean``).
A described coordinate target renders as ``'play button' (self-described)``, ``action_args`` is the recorded action minus its verb (what the Pro
an observed element as ``'Play'``; a Pro burst lists every member. summarizer extracts from ``action_taken`` and what the Flash runner
builds from the same record shape). The action name always wins over
an argument of the same key: ``manage_app``'s own ``action="launch"``
argument becomes the ``intent`` the renderer reads, never the verb. A
described coordinate target renders as ``'play button'
(self-described)``, an observed element as ``'Play'``; a Pro burst
lists every member.
""" """
args = dict(action_args or {}) args = dict(action_args or {})
extra = args.pop("additional_actions", None) extra = args.pop("additional_actions", None)
own_verb = args.pop("action", None)
if own_verb is not None and own_verb != action_name:
args.setdefault("intent", own_verb)
# Flash's coordinate swipe names its points ``start``/``end``; the # Flash's coordinate swipe names its points ``start``/``end``; the
# history renderer reads ``start_coordinates``/``end_coordinates``. # history renderer reads ``start_coordinates``/``end_coordinates``.
if action_name == "swipe" and "start" in args and "end" in args: if action_name == "swipe":
args.setdefault("start_coordinates", args["start"]) raw = args.get("args") if isinstance(args.get("args"), dict) else {}
args.setdefault("end_coordinates", args["end"]) start = args.get("start", raw.get("start"))
end = args.get("end", raw.get("end"))
if start is not None and end is not None:
args.setdefault("start_coordinates", start)
args.setdefault("end_coordinates", end)
try: try:
first = {"action": action_name, **args} first = {**args, "action": action_name}
actions = [first, *extra] if isinstance(extra, list) and extra else first actions = [first, *extra] if isinstance(extra, list) and extra else first
return format_actions_clean(actions) return format_actions_clean(actions)
except Exception as exc: except Exception as exc:
logger.debug(f"Action phrase rendering fell back to raw args: {exc}", exc_info=True) logger.debug(f"Action phrase rendering fell back to raw args: {exc}", exc_info=True)
return f"{action_name}({action_args})" return f"{action_name}({action_args})"
@staticmethod
def _verbatim_args(action_args: dict[str, Any] | None) -> dict[str, Any]:
"""The agent-facing arguments of the action.
A Flash record keeps the tool call's arguments under ``args``; a Pro
record is the arguments themselves. Bookkeeping keys never count.
"""
args = action_args or {}
raw = args.get("args")
if isinstance(raw, dict):
return dict(raw)
internal = {
"additional_actions",
"coordinate_space",
"normalized_coordinates",
"normalized_start_coordinates",
"normalized_end_coordinates",
}
return {k: v for k, v in args.items() if k not in internal}
@staticmethod
def _phrase_carries(phrase: str, args: dict[str, Any]) -> bool:
"""Whether every argument value is visible in the rendered phrase.
Booleans and ``None`` never count (the phrase spells their effect, not
their value); everything else must appear verbatim, so an index
target, a repeat count or a duration the phrase folded away keeps the
verbatim argument line.
"""
haystack = phrase.lower()
def _shown(value: Any) -> bool:
if isinstance(value, (list, tuple)):
# A point renders whole (``at [880, 410]``); a sequence or a
# description list renders member by member.
return str(list(value)).lower() in haystack or (
bool(value) and all(_shown(v) for v in value)
)
return str(value).lower() in haystack
return all(
_shown(value)
for value in args.values()
if value is not None and not isinstance(value, bool)
)
def _meter_lens_call(self, response: Any) -> None: def _meter_lens_call(self, response: Any) -> None:
"""Meter one raw-model lens call as an ``llm_usage`` trace, best-effort. """Meter one raw-model lens call as an ``llm_usage`` trace, best-effort.
@@ -338,11 +397,14 @@ class VisualStepSummarizer(StepMemoryService):
dual = bool(pre_bytes) and bool(post_bytes) dual = bool(pre_bytes) and bool(post_bytes)
template = self._prompt_template if dual else self._single_prompt_template template = self._prompt_template if dual else self._single_prompt_template
rendered_prompt = Template(template).render(step_number=step_number) rendered_prompt = Template(template).render(step_number=step_number)
lead_lines = [ phrase = self._action_phrase(action_name, action_args)
f"Step {step_number} Physical Action: {self._action_phrase(action_name, action_args)}", lead_lines = [f"Step {step_number} Physical Action: {phrase}"]
f"Action arguments (verbatim): {action_name}({action_args})", # The verbatim arguments are worth a line only when the phrase
f"Controller Outcome: {exec_outcome}", # folded some of them away (an index target, a repeat count, ...).
] verbatim = self._verbatim_args(action_args)
if verbatim and not self._phrase_carries(phrase, verbatim):
lead_lines.append(f"Action arguments (verbatim): {action_name}({verbatim})")
lead_lines.append(f"Controller Outcome: {exec_outcome}")
focus = input_data.get("focus") focus = input_data.get("focus")
if isinstance(focus, dict) and focus: if isinstance(focus, dict) and focus:
lead_lines.extend(["", render_focus_block(focus, dual=dual)]) lead_lines.extend(["", render_focus_block(focus, dual=dual)])
+1 -1
View File
@@ -1,3 +1,3 @@
{ {
"main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n[% if \"wait_for_delay\" in available_tools %]- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n[% endif %]- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.\n 2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading state. This boundary probe must remain unidirectional; never reverse direction merely to prove completion. If the task declares an earlier exit condition (for example, find one verified match or process N items), that explicit condition may end the traversal without reaching the list boundary.\n 3. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n 4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content.\n- **Dynamic Monitoring & Reactive Loop Protocol (Monitor-Trigger-Action)**: When executing an active milestone matching `[Loop] <Goal> into note <TargetNote> (Exit: <ExitCondition>; Interval: <Interval>)` or `[Loop:continuous] <Goal> into note <TargetNote> (Interval: <Interval>)` (the milestone grammar itself is defined under Task Plan Grammar below), run this 5-step cyclic state machine:\n - **Step 1 (Baseline Sync)**: On the initial cycle, observe all visible entries/metrics in the primary monitoring view and save the full snaLine truncated "main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n[% if \"wait_for_delay\" in available_tools %]- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n[% endif %]- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.\n 2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading state. This boundary probe must remain unidirectional; never reverse direction merely to prove completion. If the task declares an earlier exit condition (for example, find one verified match or process N items), that explicit condition may end the traversal without reaching the list boundary.\n 3. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n 4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content.\n- **Dynamic Monitoring & Reactive Loop Protocol (Monitor-Trigger-Action)**: When executing an active milestone matching `[Loop] <Goal> into note <TargetNote> (Exit: <ExitCondition>; Interval: <Interval>)` or `[Loop:continuous] <Goal> into note <TargetNote> (Interval: <Interval>)` (the milestone grammar itself is defined under Task Plan Grammar below), run this 5-step cyclic state machine:\n - **Step 1 (Baseline Sync)**: On the initial cycle, observe all visible entries/metrics in the primary monitoring view and save the full snaLine truncated
} }
+112 -35
View File
@@ -71,6 +71,9 @@ DEFERRING_TOOLS = {
"ask_explorer", "ask_explorer",
} }
# Bare key names press_key accepts (case-insensitive, optional KEYCODE_ prefix).
SUPPORTED_PRESS_KEYS = ("ENTER", "BACK", "HOME", "APP_SWITCH")
from artemis.agents.operator.prompts import ( from artemis.agents.operator.prompts import (
OPERATOR_MAX_TOOL_ITERATIONS, OPERATOR_MAX_TOOL_ITERATIONS,
PromptBuilder, PromptBuilder,
@@ -82,9 +85,9 @@ from artemis.agents.operator.prompts import (
HistoricalStateHintPromptComponent, HistoricalStateHintPromptComponent,
FeedbackPromptComponent, FeedbackPromptComponent,
ExecutionIncidentPromptComponent, ExecutionIncidentPromptComponent,
CheckItemsExplainerPromptComponent,
BackgroundTasksPromptComponent, BackgroundTasksPromptComponent,
render_plan_ledger_bounce, render_plan_ledger_bounce,
ReasoningReminderPromptComponent,
unwritten_action_streak, unwritten_action_streak,
ToolLimitWarningPromptComponent, ToolLimitWarningPromptComponent,
InjectedInstructionPromptComponent, InjectedInstructionPromptComponent,
@@ -137,6 +140,24 @@ class OperatorNode:
self._transcript_turn_base: int | None = None self._transcript_turn_base: int | None = None
self._turn_plan_baseline: str | None = None self._turn_plan_baseline: str | None = None
self._turn_ledger_streak: int = 0 self._turn_ledger_streak: int = 0
# Whether the previous turn's replies carried no visible reasoning text;
# the next observation tail then carries a one-line reminder (no extra
# model call). Legacy-path fallback only: with a transcript ledger the
# ledger's own ``last_turn_silent`` is authoritative.
self._previous_turn_silent: bool = False
def _previous_turn_was_silent(self) -> bool:
"""Whether the previous turn carried no visible reasoning text.
The transcript ledger judges its committed turns itself; the local flag
only serves the legacy path, which has no ledger.
"""
from artemis.memory import TranscriptLedger
ledger = getattr(self.ctx, "transcript_ledger", None)
if isinstance(ledger, TranscriptLedger):
return bool(ledger.last_turn_silent)
return self._previous_turn_silent
def _available_device_actions(self) -> frozenset[str]: def _available_device_actions(self) -> frozenset[str]:
"""Device actions the installed actuator backend provides. """Device actions the installed actuator backend provides.
@@ -225,7 +246,6 @@ class OperatorNode:
TemplatePromptComponent(), TemplatePromptComponent(),
{"template_name": "main_template"}, {"template_name": "main_template"},
), ),
(CheckItemsExplainerPromptComponent(), {}),
(ExecutionIncidentPromptComponent(), {}), (ExecutionIncidentPromptComponent(), {}),
(ObservationPromptComponent(), {}), (ObservationPromptComponent(), {}),
(ScreenshotSimilarityPromptComponent(), {}), (ScreenshotSimilarityPromptComponent(), {}),
@@ -233,10 +253,14 @@ class OperatorNode:
(InjectedInstructionPromptComponent(), {}), (InjectedInstructionPromptComponent(), {}),
(BackgroundTasksPromptComponent(), {}), (BackgroundTasksPromptComponent(), {}),
(FeedbackPromptComponent(), {}), (FeedbackPromptComponent(), {}),
(ReasoningReminderPromptComponent(), {}),
(ToolLimitWarningPromptComponent(), {}), (ToolLimitWarningPromptComponent(), {}),
] ]
else: else:
components = [(c, {}) for c in components] + [(ToolLimitWarningPromptComponent(), {})] components = [(c, {}) for c in components] + [
(ReasoningReminderPromptComponent(), {}),
(ToolLimitWarningPromptComponent(), {}),
]
# Active and finished tasks are now passed as arguments # Active and finished tasks are now passed as arguments
active_tasks = active_background_tasks or [] active_tasks = active_background_tasks or []
@@ -252,6 +276,7 @@ class OperatorNode:
"current_step_num": current_step_num, "current_step_num": current_step_num,
"active_background_tasks": active_tasks, "active_background_tasks": active_tasks,
"newly_finished_tasks": newly_finished, "newly_finished_tasks": newly_finished,
"previous_turn_silent": self._previous_turn_was_silent(),
"steps": steps, "steps": steps,
} }
kwargs_to_pass.update(extra_kwargs) kwargs_to_pass.update(extra_kwargs)
@@ -280,6 +305,9 @@ class OperatorNode:
image_scrub_depth=getattr(cfg, "image_scrub_depth", 3), image_scrub_depth=getattr(cfg, "image_scrub_depth", 3),
pending_grace_steps=getattr(cfg, "pending_grace_steps", 3), pending_grace_steps=getattr(cfg, "pending_grace_steps", 3),
xml_scrub_depth=getattr(cfg, "xml_scrub_depth", 1), xml_scrub_depth=getattr(cfg, "xml_scrub_depth", 1),
image_scrub_depth_relaxed=getattr(cfg, "image_scrub_depth_relaxed", None),
context_budget_tokens=getattr(cfg, "context_budget_tokens", None),
start_ratio=getattr(cfg, "start_ratio", None),
session_start=session_start if isinstance(session_start, (int, float)) else None, session_start=session_start if isinstance(session_start, (int, float)) else None,
) )
# L2/L3 chunk compression (M3) rides the same flag; without a # L2/L3 chunk compression (M3) rides the same flag; without a
@@ -337,6 +365,8 @@ class OperatorNode:
""" """
from langchain_core.messages import HumanMessage from langchain_core.messages import HumanMessage
from artemis.memory.transcript import mark_ephemeral
ledger = self._ensure_transcript_ledger(state) ledger = self._ensure_transcript_ledger(state)
# 1. S region: rendered exactly once per session. # 1. S region: rendered exactly once per session.
@@ -387,7 +417,6 @@ class OperatorNode:
builder.add_human_content(f"# CURRENT OBSERVATION [{ledger.elapsed_label()}]") builder.add_human_content(f"# CURRENT OBSERVATION [{ledger.elapsed_label()}]")
components = [ components = [
(PlanRecitationPromptComponent(), {}), (PlanRecitationPromptComponent(), {}),
(CheckItemsExplainerPromptComponent(), {}),
(ExecutionIncidentPromptComponent(), {}), (ExecutionIncidentPromptComponent(), {}),
(ObservationPromptComponent(), {}), (ObservationPromptComponent(), {}),
(ScreenshotSimilarityPromptComponent(), {}), (ScreenshotSimilarityPromptComponent(), {}),
@@ -395,6 +424,7 @@ class OperatorNode:
(InjectedInstructionPromptComponent(), {}), (InjectedInstructionPromptComponent(), {}),
(BackgroundTasksPromptComponent(), {}), (BackgroundTasksPromptComponent(), {}),
(FeedbackPromptComponent(), {}), (FeedbackPromptComponent(), {}),
(ReasoningReminderPromptComponent(), {}),
(ToolLimitWarningPromptComponent(), {}), (ToolLimitWarningPromptComponent(), {}),
] ]
for component, extra_kwargs in components: for component, extra_kwargs in components:
@@ -407,6 +437,7 @@ class OperatorNode:
"active_background_tasks": active_background_tasks or [], "active_background_tasks": active_background_tasks or [],
"newly_finished_tasks": newly_finished_tasks or [], "newly_finished_tasks": newly_finished_tasks or [],
"steps": steps, "steps": steps,
"previous_turn_silent": self._previous_turn_was_silent(),
} }
kwargs_to_pass.update(extra_kwargs) kwargs_to_pass.update(extra_kwargs)
await component(builder, state, self.ctx, **kwargs_to_pass) await component(builder, state, self.ctx, **kwargs_to_pass)
@@ -418,6 +449,10 @@ class OperatorNode:
else: else:
tail_content.append(part) tail_content.append(part)
tail = HumanMessage(content=tail_content) tail = HumanMessage(content=tail_content)
# Per-turn notices (reminders, hints, guidance, incident blocks) are
# deleted by the scrub edge once the turn leaves the recent window.
if builder.ephemeral_indices:
mark_ephemeral(tail, builder.ephemeral_indices)
messages = ledger.render([tail]) messages = ledger.render([tail])
self._transcript_turn_base = len(messages) - 1 self._transcript_turn_base = len(messages) - 1
@@ -471,12 +506,27 @@ class OperatorNode:
response = await invoke_llm_with_timeout_message(acomplete(bound_llm, current_messages)) response = await invoke_llm_with_timeout_message(acomplete(bound_llm, current_messages))
usage = usage_from_message(response) usage = usage_from_message(response)
# The ledger's compaction thresholds read the operator's own
# prompt size, never the session-wide meter (which the
# Planner/Checker/sub-agent calls overwrite). The messages ride
# along so the ledger can calibrate its chars-per-token ratio.
transcript_ledger = getattr(self.ctx, "transcript_ledger", None)
if usage is not None: if usage is not None:
logger.info( logger.info(
f"LLM usage: prompt_tokens={usage['prompt_tokens']}" f"LLM usage: prompt_tokens={usage['prompt_tokens']}"
f" (cached={usage['cached_tokens']})," f" (cached={usage['cached_tokens']}),"
f" completion_tokens={usage['completion_tokens']}" f" completion_tokens={usage['completion_tokens']}"
) )
if hasattr(transcript_ledger, "record_prompt_tokens"):
transcript_ledger.record_prompt_tokens(
usage.get("prompt_tokens"), messages=current_messages
)
elif hasattr(transcript_ledger, "record_prompt_tokens"):
# No usage metadata from the provider: record an estimate so
# the start gate and thresholds still see a context base.
from artemis.memory.transcript import estimate_prompt_tokens
transcript_ledger.record_prompt_tokens(estimate_prompt_tokens(current_messages))
if response.content: if response.content:
if isinstance(response.content, str): if isinstance(response.content, str):
@@ -638,18 +688,22 @@ class OperatorNode:
"Scratchpad/helper tool call failed. Deferring" "Scratchpad/helper tool call failed. Deferring"
f" screen actions: {other_tool_failure_msg}" f" screen actions: {other_tool_failure_msg}"
) )
for tc in action_calls: # Every rejected call needs its own ToolMessage, but the
# full explanation travels once; the rest point at it.
for idx, tc in enumerate(action_calls):
tool_outputs.append( tool_outputs.append(
ToolMessage( ToolMessage(
tool_call_id=tc["id"], tool_call_id=tc["id"],
name=normalize_name(tc["name"]),
content=( content=(
"This screen action was rejected" "Not executed: this screen action was"
" because an accompanying tool call" " rejected because an accompanying tool"
" failed. Error:" f" call failed. Error: {other_tool_failure_msg}."
f" {other_tool_failure_msg}. Currently," " No screen action ran this turn. Correct"
" no screen actions have been" " the error, then re-issue the action."
" executed. Please correct the error" if idx == 0
" and try again." else "Not executed: see the accompanying"
" tool-call error above."
), ),
status="error", status="error",
) )
@@ -667,25 +721,23 @@ class OperatorNode:
" Deferring terminal actions." " Deferring terminal actions."
) )
for tc in action_calls: for idx, tc in enumerate(action_calls):
tool_outputs.append( tool_outputs.append(
ToolMessage( ToolMessage(
tool_call_id=tc["id"], tool_call_id=tc["id"],
name=normalize_name(tc["name"]),
content=( content=(
"Your pre-decision tools have been" "Not executed: this screen action was"
" successfully processed. However, the" " deferred because the same turn also"
" screen action you attempted to" " called a result-dependent pre-decision"
" execute was rejected because you are" " tool. No screen action ran this turn."
" not allowed to simultaneously" " Read the tool results above, then"
" use result-dependent pre-decision" " re-issue the action."
" tools and execute physical actions in" if idx == 0
" the same turn." else "Not executed: deferred for the same"
" Currently, no screen actions have" " reason as the screen action above."
" been executed. Please review your"
" updated context and re-output your"
" intended screen action."
), ),
status="success", status="error",
) )
) )
@@ -713,24 +765,36 @@ class OperatorNode:
logger.info(f"Plan ledger gate: bouncing action. {bounce}") logger.info(f"Plan ledger gate: bouncing action. {bounce}")
for tc in action_calls: for tc in action_calls:
tool_outputs.append( tool_outputs.append(
ToolMessage(tool_call_id=tc["id"], content=bounce, status="error") ToolMessage(
tool_call_id=tc["id"],
name=normalize_name(tc["name"]),
content=bounce,
status="error",
)
) )
action_calls_to_translate = [] action_calls_to_translate = []
elif len(action_calls) > burst_cap: elif len(action_calls) > burst_cap:
# A multi-action turn is a fast-action burst that the Validator # A multi-action turn is a fast-action burst that the Validator
# fires without the safety net; cap its length before it runs. # fires without the safety net; cap its length before it runs.
validation_errors = True validation_errors = True
for tc in action_calls: burst_error = (
tool_outputs.append(
ToolMessage(
tool_call_id=tc["id"],
content=(
f"Error: {len(action_calls)} Turn-Ending Actions in one" f"Error: {len(action_calls)} Turn-Ending Actions in one"
" turn exceed the fast-action burst limit of" " turn exceed the fast-action burst limit of"
f" {burst_cap}. Nothing was executed. Re-issue at most" f" {burst_cap}. Nothing was executed. Re-issue at most"
f" {burst_cap} actions (a burst is for sub-second" f" {burst_cap} actions (a burst is for sub-second"
" sequences on transient UI, not for batching normal" " sequences on transient UI, not for batching normal"
" steps), or a single vetted action." " steps), or a single vetted action."
)
# One ToolMessage per rejected call, full text only on the first.
for idx, tc in enumerate(action_calls):
tool_outputs.append(
ToolMessage(
tool_call_id=tc["id"],
name=normalize_name(tc["name"]),
content=(
burst_error
if idx == 0
else "Not executed: see the burst-limit error above."
), ),
status="error", status="error",
) )
@@ -750,6 +814,7 @@ class OperatorNode:
tool_outputs.append( tool_outputs.append(
ToolMessage( ToolMessage(
tool_call_id=tc["id"], tool_call_id=tc["id"],
name=normalize_name(tc["name"]),
content=error, content=error,
status="error", status="error",
) )
@@ -759,6 +824,7 @@ class OperatorNode:
tool_outputs.append( tool_outputs.append(
ToolMessage( ToolMessage(
tool_call_id=tc["id"], tool_call_id=tc["id"],
name=normalize_name(tc["name"]),
content="Action Recorded", content="Action Recorded",
status="success", status="success",
) )
@@ -896,6 +962,7 @@ class OperatorNode:
new_subagent_calls=new_subagent_calls, new_subagent_calls=new_subagent_calls,
state=state, state=state,
) )
self._previous_turn_silent = not (raw_thinking and raw_thinking.strip())
# 8b. Transcript path: hold this turn's messages (observation tail + # 8b. Transcript path: hold this turn's messages (observation tail +
# tool-loop products) for commit at the next build, when the turn's # tool-loop products) for commit at the next build, when the turn's
@@ -1276,6 +1343,10 @@ class OperatorNode:
"normalized_start_coordinates": [nx1, ny1], "normalized_start_coordinates": [nx1, ny1],
"normalized_end_coordinates": [nx2, ny2], "normalized_end_coordinates": [nx2, ny2],
"duration": duration, "duration": duration,
# A directional scroll keeps the direction the agent issued so
# history and replays render "Swiped up (...)" rather than a
# bare coordinate path.
**({"direction": str(target)} if kind == "direction" else {}),
**({"target_description": described} if described else {}), **({"target_description": described} if described else {}),
} }
], None ], None
@@ -1290,9 +1361,15 @@ class OperatorNode:
f"Error: 'key' must be a string, got {type(key).__name__}.", f"Error: 'key' must be a string, got {type(key).__name__}.",
) )
normalized_key = key[8:] if key.startswith("KEYCODE_") else key normalized_key = key.strip().upper()
if normalized_key not in ["ENTER", "BACK", "HOME", "APP_SWITCH"]: if normalized_key.startswith("KEYCODE_"):
return [], f"Error: Unsupported key '{key}'." normalized_key = normalized_key[len("KEYCODE_") :]
if normalized_key not in SUPPORTED_PRESS_KEYS:
return (
[],
f"Error: Unsupported key '{key}'. Supported keys:"
f" {', '.join(SUPPORTED_PRESS_KEYS)}.",
)
return [{"action": "press_key", "keycode": f"KEYCODE_{normalized_key}"}], None return [{"action": "press_key", "keycode": f"KEYCODE_{normalized_key}"}], None
elif tool_name == "manage_app": elif tool_name == "manage_app":
+200 -87
View File
@@ -17,6 +17,7 @@ from functools import lru_cache
import io import io
import json import json
from pathlib import Path from pathlib import Path
from typing import NamedTuple
from jinja2 import Environment, StrictUndefined, Template from jinja2 import Environment, StrictUndefined, Template
from langchain_core.messages import HumanMessage, SystemMessage from langchain_core.messages import HumanMessage, SystemMessage
@@ -31,9 +32,8 @@ from artemis.tools.command_tool import (
_is_output_long, _is_output_long,
) )
from artemis.utils.logger import get_logger from artemis.utils.logger import get_logger
from artemis.utils.notes import get_note_file_path
from artemis.utils.plan_grammar import parse_plan, render_plan_grammar_spec from artemis.utils.plan_grammar import parse_plan, render_plan_grammar_spec
from artemis.utils.task_tree import SELF_DESCRIBED_MARKER from artemis.utils.task_tree import SELF_DESCRIBED_MARKER, action_intent_phrase
logger = get_logger(__name__) logger = get_logger(__name__)
@@ -149,11 +149,19 @@ class PromptBuilder:
self.system_parts = [] self.system_parts = []
self.human_parts = [] self.human_parts = []
self.human_footer = None self.human_footer = None
#: Indices into ``human_parts`` of blocks that only matter this turn (see
#: ``artemis.memory.transcript.EPHEMERAL_BLOCKS_KEY``).
self.ephemeral_indices: list[int] = []
def add_system_text(self, text: str): def add_system_text(self, text: str):
self.system_parts.append(text) self.system_parts.append(text)
def add_human_content(self, content: str | dict): def add_human_content(self, content: str | dict, *, ephemeral: bool = False):
"""Append a block to the observation. ``ephemeral=True`` marks it as valid
for this turn only: the transcript scrub edge deletes it at depth K, so it
never reaches the frozen history or a chunk capsule."""
if ephemeral:
self.ephemeral_indices.append(len(self.human_parts))
self.human_parts.append(content) self.human_parts.append(content)
def set_human_footer(self, content: str): def set_human_footer(self, content: str):
@@ -171,10 +179,12 @@ class PromptBuilder:
if self.human_footer: if self.human_footer:
human_content.append({"type": "text", "text": self.human_footer}) human_content.append({"type": "text", "text": self.human_footer})
return [ human = HumanMessage(content=human_content)
SystemMessage(content=system_content), if self.ephemeral_indices:
HumanMessage(content=human_content), from artemis.memory.transcript import mark_ephemeral
]
mark_ephemeral(human, self.ephemeral_indices)
return [SystemMessage(content=system_content), human]
class PromptComponent: class PromptComponent:
@@ -221,18 +231,42 @@ def resolve_operator_prompt_tools(ctx: ArtemisContext) -> frozenset[str]:
return frozenset(available) return frozenset(available)
def _operator_grammar_flags(ctx: ArtemisContext) -> tuple[bool, bool]: def _operator_grammar_flags(ctx: ArtemisContext) -> tuple[bool, bool, bool]:
"""(include_checks, verification_active) for the template render.""" """(midway_checks, final_check, verification_active) for the template render.
The two check gates are read separately so the grammar and the check-line
guidance are worded for what actually runs: with midway checks off there is
no repair loop to teach, and with both off the check-line grammar never
enters any prompt.
"""
setup = getattr(ctx, "execution_setup", None) setup = getattr(ctx, "execution_setup", None)
# Grammar spec assembly is a function of configuration: with both check midway = bool(setup and getattr(setup, "midway_checks_enabled", False))
# gates disabled, the check-line grammar never enters any prompt. final = bool(setup and getattr(setup, "final_check_enabled", False))
include_checks = bool(setup and getattr(setup, "checks_enabled", False))
# The rejection/finding diagnosis trigger only exists while a mechanism # The rejection/finding diagnosis trigger only exists while a mechanism
# that can produce rejections or findings is active. # that can produce rejections or findings is active.
verification_active = include_checks or bool( verification_active = (
setup and not getattr(setup, "disable_planner_validation", True) midway or final or bool(setup and not getattr(setup, "disable_planner_validation", True))
) )
return include_checks, verification_active return midway, final, verification_active
def _checkpoint_max_repairs(ctx: ArtemisContext) -> int:
"""The midway repair budget recited in the static prompt."""
setup = getattr(ctx, "execution_setup", None)
max_repairs = getattr(setup, "checkpoint_max_repairs", None)
return max_repairs if isinstance(max_repairs, int) else 2
def _grammar_render_context(ctx: ArtemisContext) -> dict:
"""Template variables shared by the static and legacy system renders."""
midway, final, verification_active = _operator_grammar_flags(ctx)
return {
"plan_grammar": render_plan_grammar_spec(midway=midway, final=final),
"verification_active": verification_active,
"checks_active": midway or final,
"midway_checks_active": midway,
"checkpoint_max_repairs": _checkpoint_max_repairs(ctx),
}
# --- M2 template split ----------------------------------------------------------- # --- M2 template split -----------------------------------------------------------
@@ -282,18 +316,15 @@ def render_transcript_static_system(
available = resolve_operator_prompt_tools(ctx) available = resolve_operator_prompt_tools(ctx)
static_template = apply_operator_prompt_contract(static_template, available_tools=available) static_template = apply_operator_prompt_contract(static_template, available_tools=available)
include_checks, verification_active = _operator_grammar_flags(ctx)
return Template(static_template).render( return Template(static_template).render(
initial_goal=state.initial_goal, initial_goal=state.initial_goal,
subgoals_status="", subgoals_status="",
plan_and_history="", plan_and_history="",
unified_history="", unified_history="",
plan_grammar=render_plan_grammar_spec(include_checks),
verification_active=verification_active,
checks_active=include_checks,
transcript_history=True, transcript_history=True,
max_burst_actions=_max_burst_actions_for_prompt(), max_burst_actions=_max_burst_actions_for_prompt(),
max_tool_calls=OPERATOR_MAX_TOOL_ITERATIONS, max_tool_calls=OPERATOR_MAX_TOOL_ITERATIONS,
**_grammar_render_context(ctx),
) )
@@ -326,19 +357,15 @@ class TemplatePromptComponent(PromptComponent):
plan_and_history = kwargs.get("plan_and_history", "No plan or history yet.") plan_and_history = kwargs.get("plan_and_history", "No plan or history yet.")
include_checks, verification_active = _operator_grammar_flags(ctx)
full_prompt = Template(prompt_template).render( full_prompt = Template(prompt_template).render(
initial_goal=state.initial_goal, initial_goal=state.initial_goal,
subgoals_status="", subgoals_status="",
plan_and_history=plan_and_history, plan_and_history=plan_and_history,
unified_history="", unified_history="",
plan_grammar=render_plan_grammar_spec(include_checks),
verification_active=verification_active,
checks_active=include_checks,
transcript_history=False, transcript_history=False,
max_burst_actions=_max_burst_actions_for_prompt(), max_burst_actions=_max_burst_actions_for_prompt(),
max_tool_calls=OPERATOR_MAX_TOOL_ITERATIONS, max_tool_calls=OPERATOR_MAX_TOOL_ITERATIONS,
**_grammar_render_context(ctx),
) )
parts = full_prompt.split("# CURRENT OBSERVATION") parts = full_prompt.split("# CURRENT OBSERVATION")
@@ -400,7 +427,8 @@ class FeedbackPromptComponent(PromptComponent):
" address any reverted subgoal accordingly; do not re-litigate" " address any reverted subgoal accordingly; do not re-litigate"
" them. [planner] findings are advisory: a lightweight reviewer" " them. [planner] findings are advisory: a lightweight reviewer"
" had a concern about a plan change that stayed applied — weigh" " had a concern about a plan change that stayed applied — weigh"
" the reason against your own observations." " the reason against your own observations.",
ephemeral=True,
) )
@@ -472,7 +500,9 @@ def render_execution_incident(incident: dict, steps: list) -> str:
kind = incident.get("kind") or "exec_error" kind = incident.get("kind") or "exec_error"
category = str(incident.get("category") or "general") category = str(incident.get("category") or "general")
consecutive = int(incident.get("consecutive_failures") or 1) consecutive = int(incident.get("consecutive_failures") or 1)
description = incident.get("action_description") or "the planned action" # Intent form ("launch app 'X'"): the action did not happen, so the
# past-tense outcome phrase would misreport it as done.
description = action_intent_phrase(incident.get("action_description") or "the planned action")
reason = str(incident.get("reason") or "").strip() reason = str(incident.get("reason") or "").strip()
burst_size = int(incident.get("burst_size") or 1) burst_size = int(incident.get("burst_size") or 1)
index = int(incident.get("action_index") or 0) index = int(incident.get("action_index") or 0)
@@ -505,8 +535,8 @@ def render_execution_incident(incident: dict, steps: list) -> str:
) )
else: else:
lines.append( lines.append(
f"What happened: your planned action `{description}` was dispatched, but the" f"What happened: your planned action `{description}` could"
f" device/executor reported: {reason}" f" not be executed; the device/executor reported: {reason}"
) )
# --- Category-specific evidence ------------------------------------------------- # --- Category-specific evidence -------------------------------------------------
@@ -547,7 +577,7 @@ def render_closed_incident(closed: dict) -> str:
"""One-turn notice after an incident closes: settle the original intent.""" """One-turn notice after an incident closes: settle the original intent."""
opened = closed.get("step_number") opened = closed.get("step_number")
closed_at = closed.get("closed_at_step") closed_at = closed.get("closed_at_step")
description = closed.get("action_description") or "the blocked action" description = action_intent_phrase(closed.get("action_description") or "the blocked action")
header = ( header = (
f"--- Execution Incident (CLOSED at Step {closed_at}) ---" f"--- Execution Incident (CLOSED at Step {closed_at}) ---"
if closed_at if closed_at
@@ -567,61 +597,22 @@ class ExecutionIncidentPromptComponent(PromptComponent):
then a one-turn CLOSED notice asking the Operator to settle the intent. then a one-turn CLOSED notice asking the Operator to settle the intent.
Reads ``state.open_incident`` / ``state.last_closed_incident`` (both written Reads ``state.open_incident`` / ``state.last_closed_incident`` (both written
by the Validator). The block is its own text part of the observation tail, by the Validator). Both blocks are ephemeral: the open block is re-rendered
so the transcript ledger keeps it verbatim: the whole resolution effort every turn the incident stays open, and the step record already carries the
stays legible across turns. failed execution result, so older copies add nothing to the history.
""" """
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs): async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
incident = getattr(state, "open_incident", None) incident = getattr(state, "open_incident", None)
if isinstance(incident, dict) and incident.get("reason"): if isinstance(incident, dict) and incident.get("reason"):
builder.add_human_content( builder.add_human_content(
render_execution_incident(incident, kwargs.get("steps") or []) render_execution_incident(incident, kwargs.get("steps") or []),
ephemeral=True,
) )
return return
closed = getattr(state, "last_closed_incident", None) closed = getattr(state, "last_closed_incident", None)
if isinstance(closed, dict) and closed.get("kind"): if isinstance(closed, dict) and closed.get("kind"):
builder.add_human_content(render_closed_incident(closed)) builder.add_human_content(render_closed_incident(closed), ephemeral=True)
class CheckItemsExplainerPromptComponent(PromptComponent):
"""Behavioral guidance for check lines, rendered iff the CURRENT plan
actually contains check lines (content-driven, not switch-driven: a resumed
plan carrying check lines still gets the explanation)."""
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
if not ctx or not getattr(ctx, "data_engine", None):
return
try:
task_plan_path = get_note_file_path(ctx.data_engine.base_dir, "task_plan")
if not task_plan_path.exists():
return
snapshot = parse_plan(task_plan_path.read_text(encoding="utf-8"))
except Exception as e:
logger.error(f"Failed to parse plan for check-items explainer: {e}")
return
if not snapshot.all_check_items:
return
setup = getattr(ctx, "execution_setup", None)
max_repairs = getattr(setup, "checkpoint_max_repairs", None)
if not isinstance(max_repairs, int):
max_repairs = 2
builder.add_human_content(
"--- About the plan's check lines ---\n"
"The task plan declares `- verify:` / `- assert:` check lines (see Task"
" Plan Grammar). Use `verify:` lines to confirm your work is complete,"
" but never declare a check passed yourself or record conclusions on"
" its behalf. Take NO extra actions for `assert:` lines and never"
" construct or fake state to satisfy one. Check lines must not be"
" deleted or reworded (deletions are automatically restored by the"
" system); keep them verbatim when rewriting the plan — adding new ones"
" is allowed. Simply mark completions per the plan grammar as usual;"
" checking runs asynchronously in the background and does not block you."
f" A failed `verify:` reopens its milestone at most {max_repairs} time(s);"
" once that repair budget is exhausted, its standing `finding:` line"
" disappears and the failure stands as the recorded result — do not"
" keep repairing it."
)
class BackgroundTasksPromptComponent(PromptComponent): class BackgroundTasksPromptComponent(PromptComponent):
@@ -700,6 +691,26 @@ _LEDGER_BOUNCE_TAIL = (
) )
#: One-line tail notice for the turn after a turn whose replies carried no visible
#: reasoning text. Thinking-mode models satisfy "reason first" inside their thought
#: channel and emit bare tool calls; the reminder costs no extra model call.
REASONING_REMINDER = (
"--- Reminder ---\n"
"Your previous turn carried no visible reasoning text (internal thinking is not"
" shown). This turn, write one short paragraph of plain prose before calling a"
" Turn-Ending Action: what the screen shows, whether the last action worked, and"
" what you do next and why."
)
class ReasoningReminderPromptComponent(PromptComponent):
"""Renders :data:`REASONING_REMINDER` when the previous turn was silent."""
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
if kwargs.get("previous_turn_silent"):
builder.add_human_content(REASONING_REMINDER, ephemeral=True)
def render_plan_ledger_bounce(task_plan: str, streak: int, stale_turns: int) -> str | None: def render_plan_ledger_bounce(task_plan: str, streak: int, stale_turns: int) -> str | None:
"""Return a plan-ledger validation error, if any.""" """Return a plan-ledger validation error, if any."""
snapshot = parse_plan(task_plan or "") snapshot = parse_plan(task_plan or "")
@@ -735,27 +746,128 @@ class ToolLimitWarningPromptComponent(PromptComponent):
"\nWarning: your last turn used up its tool-call budget without a" "\nWarning: your last turn used up its tool-call budget without a"
" Turn-Ending Action. Re-examine the task goal and your plan;" " Turn-Ending Action. Re-examine the task goal and your plan;"
" the current approach may not be the right path. If the cause" " the current approach may not be the right path. If the cause"
" is unclear, ask_diagnoser can help." " is unclear, ask_diagnoser can help.",
ephemeral=True,
) )
#: Header of the user-guidance block (shared by the Flash runner and the Pro
#: operator; the operator's static prompt explains the channel once).
USER_GUIDANCE_MARKER = "--- User Guidance ---"
#: Prefix of the persistent user-instruction line (see :func:`render_user_guidance`).
USER_INSTRUCTION_PREFIX = "User instruction"
class UserGuidance(NamedTuple):
"""The two observation blocks delivering a mid-run user instruction.
``wrapper`` is the per-turn ``--- User Guidance ---`` framing (relay
notice, rank, release-loop wording): it is marked ephemeral and leaves
with the recent window. ``body`` is the verbatim instruction as one
self-contained line; it is added as a regular block so it stays in the
active window until the turn is chunk-compressed, at which point the
chunk's ``user_lines`` carry it verbatim. Standing instructions remain
visible after the turn in which they arrived.
"""
body: str
wrapper: str
def render_user_instruction(instruction: str, *, offset_label: str | None = None) -> str:
"""The persistent one-line body: ``User instruction (T+mm:ss): "..."``.
``offset_label`` is the session-relative time the instruction was
delivered when the call site has it; omitted otherwise.
"""
when = f" ({offset_label})" if offset_label else ""
return f'{USER_INSTRUCTION_PREFIX}{when}: "{instruction}"'
def render_user_guidance(
instruction: str,
*,
release_loop: bool = False,
has_plan: bool = True,
offset_label: str | None = None,
) -> UserGuidance:
"""The observation blocks delivering a mid-run user instruction.
One rendering for both profiles: the instruction is stated as what it is (an
external interruption from the user, relayed by the system) rather than as
advice to be weighed, and it explicitly outranks the plan and its check lines.
Returns ``(body, wrapper)``; callers add ``wrapper`` with ``ephemeral=True``
and ``body`` without (see :class:`UserGuidance`).
"""
rank = (
" It outranks the task plan and its check lines: edit the plan to match,"
" check lines included."
if has_plan
else " It outranks your current milestones: adjust them to match."
)
lines = [
USER_GUIDANCE_MARKER,
"The user watching this run sent the instruction quoted below; the system"
" relays it as an external interruption, like a stop signal." + rank,
"Act on it from this turn: reconcile it with the current screen and recent"
" history, then choose the next action. The quoted line stays in your"
" history as a standing instruction until the user sends another.",
]
if release_loop:
lines.append(
"The user has explicitly authorized stopping any ongoing monitoring loop;"
" you may now wrap up and complete the task."
)
return UserGuidance(
body=render_user_instruction(instruction, offset_label=offset_label),
wrapper=chr(10).join(lines),
)
def add_user_guidance(builder: PromptBuilder, guidance: UserGuidance) -> None:
"""Adds the two guidance blocks: the wrapper ephemeral, the body persistent."""
builder.add_human_content(guidance.wrapper, ephemeral=True)
builder.add_human_content(guidance.body)
class InjectedInstructionPromptComponent(PromptComponent): class InjectedInstructionPromptComponent(PromptComponent):
"""The mid-run user instruction: an ephemeral wrapper plus the persistent
verbatim body (:class:`UserGuidance`).
The step record also keeps the instruction verbatim (``extra_metadata``),
which is what the chunk ledger renders once the turn is compressed.
"""
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs): async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
injected = getattr(state, "injected_instruction", None) injected = getattr(state, "injected_instruction", None)
if injected: if injected:
builder.add_human_content( release_loop = bool(getattr(state, "user_stop_requested", False))
f"\n--- User Guidance ---\n" add_user_guidance(
f"The user observing your progress has provided the following" builder,
f" feedback or correction:\n" render_user_guidance(
f'"{injected}"\n\n' injected,
f"Please review this guidance, evaluate it against your current" release_loop=release_loop,
f" screen state and recent history, " has_plan=True,
f"and integrate it into your reasoning. Use it to refine your" offset_label=_session_offset_label(ctx),
f" task plan and determine the " ),
f"most appropriate next action."
) )
def _session_offset_label(ctx: ArtemisContext | None) -> str | None:
"""The transcript ledger's ``T+mm:ss`` label when a ledger is attached."""
ledger = getattr(ctx, "transcript_ledger", None) if ctx is not None else None
elapsed_label = getattr(ledger, "elapsed_label", None)
if ledger is None or not callable(elapsed_label):
return None
try:
label = elapsed_label()
except Exception:
return None
return label if isinstance(label, str) and label else None
class ScreenshotSimilarityPromptComponent(PromptComponent): class ScreenshotSimilarityPromptComponent(PromptComponent):
"""Background check: compare current screen against post-action screenshots of the last few steps. """Background check: compare current screen against post-action screenshots of the last few steps.
@@ -824,7 +936,7 @@ class ScreenshotSimilarityPromptComponent(PromptComponent):
if matched_step_nums: if matched_step_nums:
steps_str = ", ".join(matched_step_nums) steps_str = ", ".join(matched_step_nums)
note_text = f"Note: the screen is unchanged since step {steps_str} (pixel-identical)." note_text = f"Note: the screen is unchanged since step {steps_str} (pixel-identical)."
builder.add_human_content(note_text) builder.add_human_content(note_text, ephemeral=True)
def _count_differing_pixels( def _count_differing_pixels(
self, self,
@@ -947,5 +1059,6 @@ class HistoricalStateHintPromptComponent(PromptComponent):
builder.add_human_content( builder.add_human_content(
f"Historical state hint: current screen closely resembles the" f"Historical state hint: current screen closely resembles the"
f" post-action screen from Step {best_step_number}. Use" f" post-action screen from Step {best_step_number}. Use"
" search_history / replay_steps only if its details are needed." " search_history / replay_steps only if its details are needed.",
ephemeral=True,
) )
+1 -1
View File
@@ -21,7 +21,7 @@ The user explicitly requested to extract the following information:
# Instruction # Instruction
1. **Assess Confidence & Verify**: Based on the execution history, final state, and the target information to extract, determine if you can confidently formulate the final answer. If the current evidence is insufficient or you require more granular proof, you **MUST** actively gather information and details using your tools first. If your tools fail (e.g., return errors or time out), or if the steps were skipped/not observed, you **MUST** report this lack of evidence honestly. Do not assume, guess, or fabricate any unobserved steps. 1. **Assess Confidence & Verify**: Based on the execution history, final state, and the target information to extract, determine if you can confidently formulate the final answer. If the current evidence is insufficient or you require more granular proof, you **MUST** actively gather information and details using your tools first. If your tools fail (e.g., return errors or time out), or if the steps were skipped/not observed, you **MUST** report this lack of evidence honestly. Do not assume, guess, or fabricate any unobserved steps.
2. **Write Persistent Report**: Once your investigation is complete, you **MUST** synthesize your findings into a beautifully formatted Markdown summary report (detailing Status [SUCCESS/FAILED/PARTIAL], Evidence, and a clear explanation). You must write this report to persistent memory under the key "output" by calling: 2. **Write Persistent Report**: Once your investigation is complete, you **MUST** synthesize your findings into a Markdown summary report (detailing Status [SUCCESS/FAILED/PARTIAL], Evidence, and a clear explanation). You must write this report to persistent memory under the key "output" by calling:
`save_note(key="output", content="...your markdown report...")` `save_note(key="output", content="...your markdown report...")`
3. **Final Response**: Only after saving the report, write a direct, helpful, and jargon-free final summary response to the user. Do not use any XML tags, code block wrappers, or special delimiters—just state the conclusion clearly and naturally, ensuring any extracted target information is clearly presented. 3. **Final Response**: Only after saving the report, write a direct, helpful, and jargon-free final summary response to the user. Do not use any XML tags, code block wrappers, or special delimiters—just state the conclusion clearly and naturally, ensuring any extracted target information is clearly presented.
+1 -1
View File
@@ -5,7 +5,7 @@
"capabilities_operator": "## Execution Capabilities\n\nThe execution agent primarily interacts with the mobile device and analyzes screen recordings and logs. It can perform the following types of actions:\n- **UI Interaction**: Tapping, swiping, and entering text on the screen.\n- **App Management**: Launching and stopping specific applications.\n- **Information Retrieval & Analysis**: Reading on-screen text and UI hierarchy, searching the web for external information, querying system logs, and analyzing screen recordings.\n- **Terminal & Background Operations**: Executing arbitrary ADB shell commands on the device for exploration. It is capable of running long-running ADB commands asynchronously in the background as tasks.\n\n### Capability Boundaries & Planning Rules\n- **Cross-App Operations**: When a task requires interacting with multiple applications, split the operations into distinct milestones for each app. Avoid mixing actions for different apps in a single milestone.", "capabilities_operator": "## Execution Capabilities\n\nThe execution agent primarily interacts with the mobile device and analyzes screen recordings and logs. It can perform the following types of actions:\n- **UI Interaction**: Tapping, swiping, and entering text on the screen.\n- **App Management**: Launching and stopping specific applications.\n- **Information Retrieval & Analysis**: Reading on-screen text and UI hierarchy, searching the web for external information, querying system logs, and analyzing screen recordings.\n- **Terminal & Background Operations**: Executing arbitrary ADB shell commands on the device for exploration. It is capable of running long-running ADB commands asynchronously in the background as tasks.\n\n### Capability Boundaries & Planning Rules\n- **Cross-App Operations**: When a task requires interacting with multiple applications, split the operations into distinct milestones for each app. Avoid mixing actions for different apps in a single milestone.",
"output_protocol_initial": "## Task Plan & Scratchpad Protocol\n\n1. **Articulate Thinking**: Begin by reasoning through the 5-step **Systematic Thinking Process** to establish your strategy, funnel approach, bounds, and pre-mortem checks.\n2. **Designate Notes in Milestones & Pre-generate Answer Cards**: If the task involves collecting findings, logging periodic monitoring events, recording candidates, or analytical summaries, designate clear, descriptive note names directly in the milestone descriptions (e.g., `candidates_list`, `monitoring_log`, `schedule_summary`). You can and should invoke the `save_note` tool to pre-generate these structured template files (answer card / scratchpad skeletons with Markdown headers, sections, or tables) before submitting `task_plan`. This provides downstream executors with a clear, ready-to-fill schema.\n3. **Submit Plan**: Submit your initial plan by invoking the `save_note` tool with `key='task_plan'`. Avoid writing the plan only in your text response, as it will only be recognized by the system if submitted via the tool.\n\n### Plan Formatting Rules\n- Format the plan strictly as a Markdown task list starting with `- [ ]` for uncompleted items.\n{{ plan_grammar }}\n- For linear macro-milestones: `- [ ] <Phase Description>`\n- For bounded repetitive, iterative, or periodic polling milestones, tag the item with `[Loop]` and declare its Exit Boundary and Interval (e.g., `- [ ] [Loop] <Overall Loop / Monitoring Goal> (Exit: <Condition>; Interval: <e.g., every 5 minutes>)`).\n- For unbounded continuous monitoring that only the user can stop, tag the item with `[Loop:continuous]` instead (e.g., `- [ ] [Loop:continuous] <Monitoring Goal> (Interval: <e.g., every 5 minutes>)`); never declare a self-serve exit condition for it.\n- **Strict Two-Tier Hierarchy & Loop Subtask Contract**:\n 1. **Subtask Nesting Rule**: ALL individual iteration instances, polling checks, candidate inspections, or baseline setups MUST be formatted as 2-space indented sub-tasks (` - [ ]`) directly nested under their parent `[Loop]` milestone.\n 2. **Mandatory First-Instance Initialization**: Every `[Loop]` milestone MUST be initialized with its first concrete child sub-task (e.g., ` - [ ] Inspect Candidate #1: ...` or ` - [ ] Polling Check #1 (Baseline & Initial Sync): ...`).\n 3. **No Flattened Cycles**: NEVER elevate individual checks, recurring polling cycles, or candidate inspections to top-level milestones outside the `[Loop]`. Top-level items are reserved exclusively for strategic macro-phases.\n- Ensure the content of the `task_plan` note contains only the task list, excluding introductions, explanations, conversational text, or headers.", "output_protocol_initial": "## Task Plan & Scratchpad Protocol\n\n1. **Articulate Thinking**: Begin by reasoning through the 5-step **Systematic Thinking Process** to establish your strategy, funnel approach, bounds, and pre-mortem checks.\n2. **Designate Notes in Milestones & Pre-generate Answer Cards**: If the task involves collecting findings, logging periodic monitoring events, recording candidates, or analytical summaries, designate clear, descriptive note names directly in the milestone descriptions (e.g., `candidates_list`, `monitoring_log`, `schedule_summary`). You can and should invoke the `save_note` tool to pre-generate these structured template files (answer card / scratchpad skeletons with Markdown headers, sections, or tables) before submitting `task_plan`. This provides downstream executors with a clear, ready-to-fill schema.\n3. **Submit Plan**: Submit your initial plan by invoking the `save_note` tool with `key='task_plan'`. Avoid writing the plan only in your text response, as it will only be recognized by the system if submitted via the tool.\n\n### Plan Formatting Rules\n- Format the plan strictly as a Markdown task list starting with `- [ ]` for uncompleted items.\n{{ plan_grammar }}\n- For linear macro-milestones: `- [ ] <Phase Description>`\n- For bounded repetitive, iterative, or periodic polling milestones, tag the item with `[Loop]` and declare its Exit Boundary and Interval (e.g., `- [ ] [Loop] <Overall Loop / Monitoring Goal> (Exit: <Condition>; Interval: <e.g., every 5 minutes>)`).\n- For unbounded continuous monitoring that only the user can stop, tag the item with `[Loop:continuous]` instead (e.g., `- [ ] [Loop:continuous] <Monitoring Goal> (Interval: <e.g., every 5 minutes>)`); never declare a self-serve exit condition for it.\n- **Strict Two-Tier Hierarchy & Loop Subtask Contract**:\n 1. **Subtask Nesting Rule**: ALL individual iteration instances, polling checks, candidate inspections, or baseline setups MUST be formatted as 2-space indented sub-tasks (` - [ ]`) directly nested under their parent `[Loop]` milestone.\n 2. **Mandatory First-Instance Initialization**: Every `[Loop]` milestone MUST be initialized with its first concrete child sub-task (e.g., ` - [ ] Inspect Candidate #1: ...` or ` - [ ] Polling Check #1 (Baseline & Initial Sync): ...`).\n 3. **No Flattened Cycles**: NEVER elevate individual checks, recurring polling cycles, or candidate inspections to top-level milestones outside the `[Loop]`. Top-level items are reserved exclusively for strategic macro-phases.\n- Ensure the content of the `task_plan` note contains only the task list, excluding introductions, explanations, conversational text, or headers.",
"examples_initial": "## Master Example: Demonstrating the Complete Cognitive Paradigm\n\n**Goal**: \"Find and summarize all schedule arrangements and venue details for the upcoming November tech conference from my emails.\"\n\n### Naive / Incorrect Approaches:\n1. *Flawed Semantics & Too Granular*:\n- [ ] Tap search icon in Gmail\n- [ ] Type \"November\"\n- [ ] Open first email and tap back\n*(Fails: Micromanages atomic UI taps; assumes emails for November arrangements must have \"November\" in the title or be received in November)*\n2. *Vague & Unbounded Trap*:\n- [ ] Open email and check all emails for conference details\n*(Fails: Unbounded scan trap; causes infinite scrolling and step budget exhaustion on mobile)*\n\n---\n\n### Correct Approach (Full Cognitive Thinking & Robust Plan):\n\n#### Step-by-Step Thinking:\n1. **Intent & Semantic Expansion**:\n - *Core Objective*: Extract confirmed schedule dates, sessions, and venue info for the November tech conference.\n - *Semantic Insight*: Conference communications are often sent months in advance (e.g., ticket confirmations or registration emails in August/September). The subject line may say \"Global Developer Summit\", \"Tech Expo Agenda\", or \"Registration Confirmed\" rather than literally \"November conference\".\n - *Task Composition*: Linear setup (open email app and search) -> Iterative candidate inspection loop (examine each promising email) -> Final consolidation.\n2. **Visual Grounding**:\n - The device screenshot shows the Home screen with the Gmail app icon visible. The plan must start by opening the email client.\n3. **Tiered Strategy, Funnel Approach & Safeguards**:\n - *Tier 1 (Optimal)*: Use in-app search with rich semantic keywords (\"conference\", \"summit\", \"schedule\", \"agenda\", \"venue\", \"November\").\n - *Tier 2 (Fallback)*: If search is unavailable or yields zero results, fallback to chronological scanning of recent 3 months.\n - *Funnel (Broad Gathering -> Deep Inspection)*: Gather candidate emails from search results; do not discard candidates based solely on list previews. Open each candidate thread to inspect full body details.\n - *Feasibility Boundary*: Cap the candidate email inspection pool at top 15 emails to prevent infinite scrolling.\n4. **Risk Pre-mortem**:\n - *Risks*: Promotional marketing spam; conflicting drafts with preliminary dates; cancelled sessions.\n - *Defensive Check*: Instruct downstream executor to filter out promotional newsletters, verify session timestamps, and confirm final venue address.\n5. **Milestone Granularity & Scratchpad Blueprint**:\n - Focus milestones on business outcomes (search, inspect candidates, consolidate), avoiding micromanaged click-by-click steps.\n - Use indented sub-tasks to guide the loop of inspecting each candidate email, initializing with Item #1 and adding subsequent items as they are identified. This prevents the agent from losing state or re-inspecting the same items.\n - Designate note `november_conference_summary` as the destination for recording findings and pre-generate the structured template note.\n\n#### Tool Invocation:\n1. Pre-generate designated answer card note template:\n`save_note(key='november_conference_summary', content='# November Tech Conference Summary\\n\\n## Key Details\\n- Conference Name: \\n- Dates: \\n- Venue / Address: \\n\\n## Daily Schedule & Sessions\\n(To be populated during email inspection)\\n')`\n\n2. Submit milestone plan:\n`save_note(key='task_plan', content='- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\\n - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\\n- [ ] Review and consolidate all extracted information in `november_conference_summary` into a clear, finalized conference itinerary')`\n\n#### Content of `task_plan` Note:\n- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\n - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\n- [ ] ReviLine truncated "examples_initial": "## Master Example: Demonstrating the Complete Cognitive Paradigm\n\n**Goal**: \"Find and summarize all schedule arrangements and venue details for the upcoming November tech conference from my emails.\"\n\n### Naive / Incorrect Approaches:\n1. *Flawed Semantics & Too Granular*:\n- [ ] Tap search icon in Gmail\n- [ ] Type \"November\"\n- [ ] Open first email and tap back\n*(Fails: Micromanages atomic UI taps; assumes emails for November arrangements must have \"November\" in the title or be received in November)*\n2. *Vague & Unbounded Trap*:\n- [ ] Open email and check all emails for conference details\n*(Fails: Unbounded scan trap; causes infinite scrolling and step budget exhaustion on mobile)*\n\n---\n\n### Correct Approach (Full Cognitive Thinking & Robust Plan):\n\n#### Step-by-Step Thinking:\n1. **Intent & Semantic Expansion**:\n - *Core Objective*: Extract confirmed schedule dates, sessions, and venue info for the November tech conference.\n - *Semantic Insight*: Conference communications are often sent months in advance (e.g., ticket confirmations or registration emails in August/September). The subject line may say \"Global Developer Summit\", \"Tech Expo Agenda\", or \"Registration Confirmed\" rather than literally \"November conference\".\n - *Task Composition*: Linear setup (open email app and search) -> Iterative candidate inspection loop (examine each promising email) -> Final consolidation.\n2. **Visual Grounding**:\n - The device screenshot shows the Home screen with the Gmail app icon visible. The plan must start by opening the email client.\n3. **Tiered Strategy, Funnel Approach & Safeguards**:\n - *Tier 1 (Optimal)*: Use in-app search with rich semantic keywords (\"conference\", \"summit\", \"schedule\", \"agenda\", \"venue\", \"November\").\n - *Tier 2 (Fallback)*: If search is unavailable or yields zero results, fallback to chronological scanning of recent 3 months.\n - *Funnel (Broad Gathering -> Deep Inspection)*: Gather candidate emails from search results; do not discard candidates based solely on list previews. Open each candidate thread to inspect full body details.\n - *Feasibility Boundary*: Cap the candidate email inspection pool at top 15 emails to prevent infinite scrolling.\n4. **Risk Pre-mortem**:\n - *Risks*: Promotional marketing spam; conflicting drafts with preliminary dates; cancelled sessions.\n - *Defensive Check*: Instruct downstream executor to filter out promotional newsletters, verify session timestamps, and confirm final venue address.\n5. **Milestone Granularity & Scratchpad Blueprint**:\n - Focus milestones on business outcomes (search, inspect candidates, consolidate), avoiding micromanaged click-by-click steps.\n - Use indented sub-tasks to guide the loop of inspecting each candidate email, initializing with Item #1 and adding subsequent items as they are identified. This prevents the agent from losing state or re-inspecting the same items.\n - Designate note `november_conference_summary` as the destination for recording findings and pre-generate the structured template note.\n\n#### Tool Invocation:\n1. Pre-generate designated answer card note template:\n`save_note(key='november_conference_summary', content='# November Tech Conference Summary\\n\\n## Key Details\\n- Conference Name: \\n- Dates: \\n- Venue / Address: \\n\\n## Daily Schedule & Sessions\\n(To be populated during email inspection)\\n')`\n\n2. Submit milestone plan:\n`save_note(key='task_plan', content='- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\\n - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\\n- [ ] Review and consolidate all extracted information in `november_conference_summary` into a clear, finalized conference itinerary')`\n\n#### Content of `task_plan` Note:\n- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\n - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\n- [ ] ReviLine truncated
"check_generation": "## Declaring Verification Standards (check lines)\nYou may attach check lines to the plan; an independent Checker judges them asynchronously.\n\n- **User-declared expectations become `- assert:` lines.** Whenever the user's goal states an explicit expectation (\"should show X\", \"must not crash\", \"the toast Y should appear\"), declare it as an assert line preserving the user's wording verbatim. Expectations about the whole run or the final state use the `@end` suffix (task-level lines at the end of the plan).\n- **`- verify:` lines only for milestones that genuinely need acceptance checks** — completion points of irreversible operations, or milestones that directly contribute to the final goal. Do NOT attach a verify line to every subgoal: restraint is an explicit requirement (checking cost scales with the number of check lines).\n- **Do not fabricate assertions** for subgoals that have nothing verifiable. Rewrite transient expectations as persistently probeable state wherever possible (e.g., \"alarm exists in the alarm list\" instead of \"a toast flashed\") — evidence for transient moments only exists if the execution history happened to capture it.\n- **Ordering requirements are post-hoc.** If the user demands \"A must be confirmed before B runs\", still generate the assertion preserving their semantics, but annotate in the plan that ordering is verified retroactively from history — the system performs post-hoc audits and CANNOT block B from executing. Never silently promise enforcement that does not exist.", "check_generation": "## Declaring Verification Standards (check lines)\nYou may attach check lines to the plan. {% if midway_checks %}An independent Checker judges each line in the background when its milestone completes, and a failed verify reopens that milestone for repair{% if final_check %}; a final review at task exit then audits the whole run against the user's goal{% endif %}.{% else %}A final review at task exit judges every line once, from the recorded evidence around each milestone's completion; there is no midway repair loop.{% endif %}\n\n- **User-declared expectations become `- assert:` lines.** Whenever the user's goal states an explicit expectation (\"should show X\", \"must not crash\", \"the toast Y should appear\"), declare it as an assert line preserving the user's wording verbatim. Expectations about the whole run or the final state use the `@end` suffix (task-level lines at the end of the plan).\n- **`- verify:` lines only for milestones that genuinely need acceptance checks** — completion points of irreversible operations, or milestones that directly contribute to the final goal. Do NOT attach a verify line to every subgoal: restraint is an explicit requirement (checking cost scales with the number of check lines).\n- **Do not fabricate assertions** for subgoals that have nothing verifiable. Rewrite transient expectations as persistently probeable state wherever possible (e.g., \"alarm exists in the alarm list\" instead of \"a toast flashed\") — evidence for transient moments only exists if the execution history happened to capture it.\n- **Ordering requirements are post-hoc.** If the user demands \"A must be confirmed before B runs\", still generate the assertion preserving their semantics, but annotate in the plan that ordering is verified retroactively from history — the system performs post-hoc audits and CANNOT block B from executing. Never silently promise enforcement that does not exist.",
"check_audit": "- **Check Standard Protection**: Check lines (`- verify:` / `- assert:`) declared in the plan are the task's verification standards. FLAG any proposed plan that deletes, weakens, or rewords existing check lines — softening a declared standard is never an acceptable adaptation. Adding new check lines is allowed.", "check_audit": "- **Check Standard Protection**: Check lines (`- verify:` / `- assert:`) declared in the plan are the task's verification standards. FLAG any proposed plan that deletes, weakens, or rewords existing check lines — softening a declared standard is never an acceptable adaptation. Adding new check lines is allowed.",
"role_validator": "## Role\nYou are the Task Plan Reviewer, a fast advisory second opinion. The Operator has modified the top-level task plan and the change is already applied. Your job is to spot a concrete concern — goal drift, a broken loop contract, a weakened check standard, a logical dead end — and, if there is one, state it with its reason so the Operator can decide. You never block or roll back anything.", "role_validator": "## Role\nYou are the Task Plan Reviewer, a fast advisory second opinion. The Operator has modified the top-level task plan and the change is already applied. Your job is to spot a concrete concern — goal drift, a broken loop contract, a weakened check standard, a logical dead end — and, if there is one, state it with its reason so the Operator can decide. You never block or roll back anything.",
"validator_guidelines": "## Validation Principles\n- **Goal Alignment**: Ensure the new plan ultimately accomplishes the `Initial Goal` requested by the user. If the Operator has abandoned the core objective or fundamentally altered it, flag the plan.\n- **Logical Coherence**: Verify that the proposed milestones form a logical, achievable sequence to reach the goal. Support valid ongoing iteration/polling subgoals and trigger branches without misconstruing periodic checks as infinite loops. Flag the plan if it introduces contradictory steps, unhandled deadlocks, or obvious logical fallacies.\n{{ plan_grammar }}\n- **Loop & Monitoring Protection**: If the existing plan contains an active `[Loop]` or `[Loop:continuous]` milestone, you MUST FLAG any proposed plan that deletes the milestone or removes its loop tag, as well as any plan that marks a `[Loop:continuous]` continuous monitoring milestone as completed (`[x]`) without an explicit external user stop signal.\n- **Adaptive Acceptance**: The Operator operates in a dynamic environment and may encounter unexpected UI states or dynamic event triggers. You should approve alternative strategies, extra intermediate steps, loop status tracking, or necessary workarounds as long as they serve the primary goal effectively.\n- **Advisory Verdict & Constructive Feedback**: Your verdict is advisory — the change stays applied whether or not you approve it. Approve unless you have a concrete, evidence-backed concern; a vague unease is not a concern. If you flag the plan, give one or two sentences naming the flaw or goal deviation and the correction you suggest, concrete enough for the Operator to weigh it against its own observations." "validator_guidelines": "## Validation Principles\n- **Goal Alignment**: Ensure the new plan ultimately accomplishes the `Initial Goal` requested by the user. If the Operator has abandoned the core objective or fundamentally altered it, flag the plan.\n- **Logical Coherence**: Verify that the proposed milestones form a logical, achievable sequence to reach the goal. Support valid ongoing iteration/polling subgoals and trigger branches without misconstruing periodic checks as infinite loops. Flag the plan if it introduces contradictory steps, unhandled deadlocks, or obvious logical fallacies.\n{{ plan_grammar }}\n- **Loop & Monitoring Protection**: If the existing plan contains an active `[Loop]` or `[Loop:continuous]` milestone, you MUST FLAG any proposed plan that deletes the milestone or removes its loop tag, as well as any plan that marks a `[Loop:continuous]` continuous monitoring milestone as completed (`[x]`) without an explicit external user stop signal.\n- **Adaptive Acceptance**: The Operator operates in a dynamic environment and may encounter unexpected UI states or dynamic event triggers. You should approve alternative strategies, extra intermediate steps, loop status tracking, or necessary workarounds as long as they serve the primary goal effectively.\n- **Advisory Verdict & Constructive Feedback**: Your verdict is advisory — the change stays applied whether or not you approve it. Approve unless you have a concrete, evidence-backed concern; a vague unease is not a concern. If you flag the plan, give one or two sentences naming the flaw or goal deviation and the correction you suggest, concrete enough for the Operator to weigh it against its own observations."
+23 -8
View File
@@ -82,11 +82,19 @@ def validate_plan_format(content: str) -> tuple[bool, str]:
return True, "" return True, ""
def _checks_enabled(ctx: ArtemisContext) -> bool: def _check_gates(ctx: ArtemisContext) -> tuple[bool, bool]:
"""Config-driven prompt assembly gate: with both check gates disabled, no """(midway_checks, final_check): the two check gates, read separately so
check-related instruction reaches any planner prompt.""" the planner is taught the judgment moment that actually applies. With both
disabled, no check-related instruction reaches any planner prompt."""
setup = getattr(ctx, "execution_setup", None) setup = getattr(ctx, "execution_setup", None)
return bool(setup and getattr(setup, "checks_enabled", False)) midway = bool(setup and getattr(setup, "midway_checks_enabled", False))
final = bool(setup and getattr(setup, "final_check_enabled", False))
return midway, final
def _checks_enabled(ctx: ArtemisContext) -> bool:
"""Config-driven prompt assembly gate (either check gate active)."""
return any(_check_gates(ctx))
def build_planner_system_blocks(prompts_data: dict, mode: str, include_checks: bool) -> list[str]: def build_planner_system_blocks(prompts_data: dict, mode: str, include_checks: bool) -> list[str]:
@@ -152,12 +160,15 @@ async def run_async_planner_validation(
with open(prompts_path, encoding="utf-8") as f: with open(prompts_path, encoding="utf-8") as f:
prompts_data = json.load(f) prompts_data = json.load(f)
include_checks = _checks_enabled(ctx) midway, final = _check_gates(ctx)
include_checks = midway or final
mode_config = prompts_data["modes"]["validator"] mode_config = prompts_data["modes"]["validator"]
system_blocks = build_planner_system_blocks(prompts_data, "validator", include_checks) system_blocks = build_planner_system_blocks(prompts_data, "validator", include_checks)
system_content = "\n\n".join(prompts_data["blocks"][block] for block in system_blocks) system_content = "\n\n".join(prompts_data["blocks"][block] for block in system_blocks)
system_message = Template(system_content).render( system_message = Template(system_content).render(
plan_grammar=render_plan_grammar_spec(include_checks) plan_grammar=render_plan_grammar_spec(midway=midway, final=final),
midway_checks=midway,
final_check=final,
) )
history_str = "No history available." history_str = "No history available."
@@ -228,15 +239,19 @@ class PlannerNode:
mode = "initial_plan" mode = "initial_plan"
mode_config = prompts_data["modes"][mode] mode_config = prompts_data["modes"][mode]
include_checks = _checks_enabled(self.ctx) midway, final = _check_gates(self.ctx)
include_checks = midway or final
# Assembly point (not in-template conditionals): check-generation # Assembly point (not in-template conditionals): check-generation
# instructions only mount while the checking feature is active, so a # instructions only mount while the checking feature is active, so a
# fully-disabled configuration yields a byte-identical planner prompt. # fully-disabled configuration yields a byte-identical planner prompt.
# Inside the mounted block, the judgment moment is worded per gate.
system_blocks = build_planner_system_blocks(prompts_data, mode, include_checks) system_blocks = build_planner_system_blocks(prompts_data, mode, include_checks)
system_content = "\n\n".join(prompts_data["blocks"][block] for block in system_blocks) system_content = "\n\n".join(prompts_data["blocks"][block] for block in system_blocks)
system_message = Template(system_content).render( system_message = Template(system_content).render(
plan_grammar=render_plan_grammar_spec(include_checks) plan_grammar=render_plan_grammar_spec(midway=midway, final=final),
midway_checks=midway,
final_check=final,
) )
# Get screenshot # Get screenshot
-35
View File
@@ -1,35 +0,0 @@
# ROLE
You are the Summarizer Agent for a mobile automation system. Your task is to summarize the latest execution step, providing a high-information-density and succinct history summary of the current step without losing any information.
# OUTPUT FORMAT
- **Format Constraint**: You MUST output the summary as a **single, continuous paragraph**.
- **Prohibited Formats**: Do NOT use bullet points, numbered lists, markdown formatting, or multiple paragraphs.
- **Goal**: Maintain a consistent, easily parsable narrative for the Operator's historical record.
# CORE PHILOSOPHY & RULES
1. **Perspective**: You MUST write the summary from the Operator's first-person perspective using "I" (e.g., "I intended to...", "I clicked..."). NEVER use third-person terms like "The operator" or "The agent", nor second-person terms like "You" when referring to the Operator. For other agents (e.g., Explorer, Diagnoser), you can use their names.
2. **Zero Subjective Validation**: Absolutely avoid declaring semantic success, completion, or failure. In historical compression, any subjective validation will cause future planners to falsely believe a goal was definitively met or permanently broken.
- **BANNED WORDS**: "successfully", "completed", "entered", "navigated to", "failed", "unsuccessful", "could not", "achieved".
3. **Focus on Intent & Strategy**: The specific physical action (exact coordinates, swipes) is now recorded automatically. Focus primarily on the high-level Intent and reasoning. If you realize the Operator is on the wrong path and needs to backtrack or pivot, describe this as a shift in strategy.
4. **Intended Context Transitions**: Describe the intended navigation path ("From Where -> Intended Where"). Do not declare that the transition actually happened. (e.g., "From the Home screen, I intended to open the Network Settings view").
5. **Preserve Verifications & Iteration Progress**: If the Operator verified any state in this step, processed a specific candidate item/polling round, or explicitly mentioned what prerequisites or items are still missing, you MUST succinctly preserve these concrete progress details and verified items in the summary.
6. **Objective Incident Logs**: If the safety net intercepted the action, the device rejected it, or a fast-action burst aborted midway (an `[Result]` line starting with "Error:" / an execution incident), faithfully record what was blocked and the stated reason, without judging whether the Operator's next move will resolve it.
# LOOP & STAGNATION DETECTION
Use the `RECENT 10 STEPS HISTORY` to intelligently detect loops or stagnation:
1. **Physical Action Loops vs. Planned Iterations**:
- *Planned Iterations / Monitoring (Valid Progress)*: If the Operator is systematically processing different candidate items in a loop, or executing planned periodic polling cycles with wait intervals under a `[Loop]` milestone, record this objectively as planned progress (e.g., "Inspected Candidate #2", "Completed Polling Round #2 for new messages"). Do NOT flag planned iterations across different items or spaced polling as abnormal loops.
- *Unintended Action Loops (Anomalies)*: If the Operator is repeatedly clicking the exact same target, failing to progress on the same item, or oscillating blindly between identical states without updating progress notes, explicitly record the anomalous loop pattern and iteration count (e.g., "This is the 3rd cycle of failing to open the same folder").
2. **Cognitive Stagnation**: While you generally ignore internal background tool calls (e.g., reading logs, checking states) when counting physical action loops, if you detect continuous multiple steps of purely internal tool calls without any physical actions or note updates, objectively record this "stagnation" or "analysis loop".
# INPUT STRUCTURE
You will receive:
- **CURRENT PLAN & TASK PLAN**: The current task plan including milestones, active subgoals, and live iteration checklists to help you understand the strategic context and loop phase.
- **RECENT 10 STEPS HISTORY**: A brief history of the last 10 steps to help you detect loops and stagnation patterns.
- **CHRONOLOGICAL STEP TRACE (CURRENT STEP)**: The step-by-step chronological history of the current execution turn, including:
- `[Operator Monologue] / [Operator Native Thought]`: The operator's planning, intentions, progress counting, reasoning, verified states, and explicit mentions of missing prerequisites.
- `[Operator Tool Call] / [Operator Final Action]`: The actual action(s) planned and executed.
- `[Planned Fast-Action Burst]`: A multi-action turn executed back to back without the safety net, with each member's outcome.
- `[Pre-Execution Safety Net]`: Validation checks prior to action execution.
- `[Result]`: The Validator's execution outcome; an "Error:" line here is an execution incident the Operator must resolve.
+1 -1
View File
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
"""Pro-graph step summarizer node (M2: visual-transition lens dispatch). """Pro step summarizer dispatching visual-transition summaries.
Per the history-module redesign §5 (single step-level lens, confirmed Per the history-module redesign §5 (single step-level lens, confirmed
2026-08-31) this node no longer generates a per-step text capsule with its own 2026-08-31) this node no longer generates a per-step text capsule with its own
+87 -14
View File
@@ -460,8 +460,20 @@ class MemoryTranscriptConfig(BaseModel):
default=3, default=3,
ge=1, ge=1,
description=( description=(
"Depth K of the screenshot scrub edge: the K-th most recent" "Depth K of the screenshot scrub edge from the start gate up: the"
" historical screenshot is replaced by its visual summary." " K-th most recent screenshot (the live one counts as depth 1) is"
" replaced by its visual summary."
),
)
image_scrub_depth_relaxed: int = Field(
default=6,
ge=1,
description=(
"Screenshot scrub depth while the operator's measured context is"
" below budget*start_ratio (nothing is being replaced yet, so"
" more raw screenshots stay in view). Must be >= image_scrub_depth;"
" set equal to it to disable the occupancy-driven depth. A change"
" of depth never rewrites an already-scrubbed message."
), ),
) )
pending_grace_steps: int = Field( pending_grace_steps: int = Field(
@@ -473,13 +485,17 @@ class MemoryTranscriptConfig(BaseModel):
" image is replaced by a placeholder referencing the DataEngine step." " image is replaced by a placeholder referencing the DataEngine step."
), ),
) )
xml_scrub_depth: int = Field( xml_scrub_depth: int | None = Field(
default=1, default=1,
ge=1, ge=1,
description=( description=(
"How many of the most recent observations keep their UI Element" "Text edge of the scrub: once a message is deeper than this many"
" list; older lists are stripped (depth 1 matches the legacy" " image-bearing observations (1 = as soon as a newer observation"
" prune_history_xml semantics)." " exists) its UI element list, plan recitation and per-turn"
" ephemeral blocks are removed. Kept shallow on purpose: only the"
" live observation carries an indexed element list, so a stale"
" [n] index can never be picked up as a target. Screenshots follow"
" image_scrub_depth / image_scrub_depth_relaxed."
), ),
) )
context_budget_tokens: int = Field( context_budget_tokens: int = Field(
@@ -491,14 +507,27 @@ class MemoryTranscriptConfig(BaseModel):
" ratios below are applied against this budget." " ratios below are applied against this budget."
), ),
) )
start_ratio: float = Field(
default=0.35,
ge=0.0,
lt=1.0,
description=(
"Start gate: below budget*start_ratio of the operator's measured"
" prompt size, closed segments only prepare their capsule in the"
" background — the original turns stay in the transcript. Ready"
" chunks replace their turns once the measured context reaches"
" this ratio (the hard threshold overrides it). 0 restores"
" swap-as-soon-as-ready."
),
)
soft_ratio: float = Field( soft_ratio: float = Field(
default=0.7, default=0.7,
gt=0.0, gt=0.0,
lt=1.0, lt=1.0,
description=( description=(
"Soft threshold: when the last measured prompt size reaches" "Soft threshold: when the operator's last measured prompt size"
" budget*soft_ratio, the oldest open segment is chunk-compressed" " reaches budget*soft_ratio, the oldest open segment is"
" (L2)." " chunk-compressed (L2)."
), ),
) )
hard_ratio: float = Field( hard_ratio: float = Field(
@@ -506,9 +535,13 @@ class MemoryTranscriptConfig(BaseModel):
gt=0.0, gt=0.0,
le=1.0, le=1.0,
description=( description=(
"Hard threshold: when the last measured prompt size reaches" "Hard threshold: when the operator's last measured prompt size"
" budget*hard_ratio, the frozen region collapses to the L3 session" " reaches budget*hard_ratio, ready chunks swap as L2 blocks first;"
" snapshot (chunk headers merged; per-step index retained)." " only if that would not bring the estimated context back under the"
" line does everything closed force-swap (pending chunks included)"
" and the whole frozen region fold into a recall-only era — L3: the"
" period paragraph with the per-step index. The fold is permanent;"
" later L2 chunks are appended after it."
), ),
) )
min_active_steps: int = Field( min_active_steps: int = Field(
@@ -545,6 +578,23 @@ class MemoryTranscriptConfig(BaseModel):
) )
model_config = {"extra": "allow"} model_config = {"extra": "allow"}
@model_validator(mode="after")
def _ratios_are_ordered(self) -> "MemoryTranscriptConfig":
"""The three thresholds form a ladder against the same budget."""
if not (0.0 <= self.start_ratio <= self.soft_ratio <= self.hard_ratio <= 1.0):
raise ValueError(
"memory.transcript ratios must satisfy"
" 0 <= start_ratio <= soft_ratio <= hard_ratio <= 1"
f" (got start_ratio={self.start_ratio}, soft_ratio={self.soft_ratio},"
f" hard_ratio={self.hard_ratio})."
)
if self.image_scrub_depth_relaxed < self.image_scrub_depth:
raise ValueError(
"memory.transcript.image_scrub_depth_relaxed must be >= image_scrub_depth"
f" (got {self.image_scrub_depth_relaxed} < {self.image_scrub_depth})."
)
return self
class MemoryRecallConfig(BaseModel): class MemoryRecallConfig(BaseModel):
"""search_history tool options (agent.memory.recall).""" """search_history tool options (agent.memory.recall)."""
@@ -620,8 +670,21 @@ class MemoryChunkingConfig(BaseModel):
default=2000, default=2000,
ge=128, ge=128,
description=( description=(
"Size-threshold trigger: estimated source tokens (char/4 over the" "Size-threshold trigger: estimated source tokens (transcript"
" open segment's transcript text) that force a chunk compression." " characters through the session-calibrated ratio, 4 chars/token"
" by default) over the open segment's eligible portion that force"
" a chunk compression."
),
)
min_steps: int = Field(
default=3,
ge=1,
description=(
"Minimum turns in a chunk closed by the size or pressure trigger:"
" below this the eligible turns keep accumulating instead of"
" becoming one-step chunks (each chunk costs a capsule call)."
" Milestone closes are exempt (a complete segment is a unit) and"
" the hard threshold waives it (emergency)."
), ),
) )
model: str = Field( model: str = Field(
@@ -646,6 +709,16 @@ class MemoryChunkingConfig(BaseModel):
" behavior." " behavior."
), ),
) )
@model_validator(mode="after")
def _min_steps_within_cap(self) -> "MemoryChunkingConfig":
if self.min_steps > self.max_steps:
raise ValueError(
"memory.chunking.min_steps must not exceed max_steps"
f" (got min_steps={self.min_steps}, max_steps={self.max_steps})."
)
return self
model_config = {"extra": "allow"} model_config = {"extra": "allow"}
+12
View File
@@ -221,6 +221,18 @@ class ArtemisContext(BaseModel):
"""Plan content currently under async planner validation; becomes the new """Plan content currently under async planner validation; becomes the new
baseline when the validator approves it.""" baseline when the validator approves it."""
guidance_unprotected_checks: set[tuple[str, str]] = Field(default_factory=set)
"""``(kind, text)`` of every check line that existed in the task plan when a
mid-run user instruction arrived. User guidance outranks the plan's declared
standards, so these lines lose their machine restoration for the rest of the
run (the Operator may drop or reword them at any later write); check lines
added afterwards stay protected until the next instruction."""
guidance_retired_checks: set[tuple[str, str]] = Field(default_factory=set)
"""``(kind, text)`` of the check lines the Operator actually dropped under
that waiver. A verdict recorded for such a line before it was retired is a
result for a requirement the user withdrew: the run outcome reports it as
retired instead of counting it as a pass or failure."""
_genai_client: Any | None = PrivateAttr(default=None) _genai_client: Any | None = PrivateAttr(default=None)
_video_blackboard: Any | None = PrivateAttr(default=None) _video_blackboard: Any | None = PrivateAttr(default=None)
_video_circuit_breaker: Any | None = PrivateAttr(default=None) _video_circuit_breaker: Any | None = PrivateAttr(default=None)
+18
View File
@@ -20,6 +20,7 @@ from artemis.core.diagnostics.adb_keys import (
heal_adb_keys, heal_adb_keys,
inspect_adb_keys, inspect_adb_keys,
) )
from artemis.core.diagnostics.device_smoke import smoke_test_device
from artemis.core.diagnostics.emulator_manager import ( from artemis.core.diagnostics.emulator_manager import (
EmulatorLaunchStage, EmulatorLaunchStage,
EmulatorLaunchState, EmulatorLaunchState,
@@ -32,11 +33,20 @@ from artemis.core.diagnostics.probes.credentials_probe import (
LLMCredentialsProbe, LLMCredentialsProbe,
VisionOCRProbe, VisionOCRProbe,
) )
from artemis.core.diagnostics.probes.host_probe import IntegrationHostProbe
from artemis.core.diagnostics.probes.runtime_probe import ( from artemis.core.diagnostics.probes.runtime_probe import (
PythonRuntimeProbe, PythonRuntimeProbe,
SystemConfigProbe, SystemConfigProbe,
) )
from artemis.core.diagnostics.probes.toolchain_probe import ToolchainProbe from artemis.core.diagnostics.probes.toolchain_probe import ToolchainProbe
from artemis.core.diagnostics.readiness import (
CHECK_ORDER,
Verdict,
adb_keys_corrupted,
base_verdict,
collect_readiness,
sort_by_fix_order,
)
from artemis.core.diagnostics.schema import ( from artemis.core.diagnostics.schema import (
DeviceInfo, DeviceInfo,
ProbeAction, ProbeAction,
@@ -53,6 +63,7 @@ __all__ = [
"EmulatorLaunchStage", "EmulatorLaunchStage",
"EmulatorLaunchState", "EmulatorLaunchState",
"BaseProbe", "BaseProbe",
"IntegrationHostProbe",
"PythonRuntimeProbe", "PythonRuntimeProbe",
"SystemConfigProbe", "SystemConfigProbe",
"AdbDeviceProbe", "AdbDeviceProbe",
@@ -69,4 +80,11 @@ __all__ = [
"get_adb_key_paths", "get_adb_key_paths",
"inspect_adb_keys", "inspect_adb_keys",
"heal_adb_keys", "heal_adb_keys",
"smoke_test_device",
"CHECK_ORDER",
"Verdict",
"collect_readiness",
"base_verdict",
"adb_keys_corrupted",
"sort_by_fix_order",
] ]
+359
View File
@@ -0,0 +1,359 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""End-to-end device observation smoke test.
The readiness ADB probe only proves that ``adb devices`` and ``dumpsys`` answer.
The most common "device shows Connected but ARTEMIS cannot observe it" failure
is UIAutomator not answering (screenshot / hierarchy dump hangs), which today
only surfaces as a failed task. :func:`smoke_test_device` performs exactly the
observation the ``mobile_get_device_state`` MCP tool performs -- build (or reuse)
the cached device controller through ``artemis.mcp.adb_server._get_controller``
and call ``controller.get_screen_data()`` -- and reports a structured verdict
with concrete repair guidance instead of raising.
Design notes
------------
* ``_get_controller`` is synchronous, caches one controller per serial for the
life of the process and never touches ``DeviceExecutionLock``. Its first call
for a serial blocks on ``UIAutomatorClient.get_screen_data()`` (three
``u2.connect`` attempts plus a screenshot and hierarchy dump), so it runs on a
worker thread here.
* ``AdbDriver.get_screen_data`` invokes the synchronous UIAutomator client on
the event-loop thread. A plain ``asyncio.wait_for`` around the coroutine can
therefore never fire while UIAutomator hangs. The coroutine is executed on a
dedicated daemon thread with its own event loop, and the deadline is enforced
on the join. A thread abandoned by a timeout finishes on its own once the
underlying adb / HTTP timeouts elapse; being a daemon it never delays
interpreter shutdown.
* ``u2.connect`` restarts the on-device UIAutomator server when it believes the
server is unhealthy. Running that against a device another ARTEMIS process is
driving would break the running task's session, so a device with a live
``DeviceExecutionLock`` owner is reported as busy instead of probed.
* Nothing is left behind: no screenshot is written to disk, no lock is taken,
and -- mirroring ``mobile_get_device_state`` -- the cached controller is kept
for reuse rather than closed.
"""
from __future__ import annotations
import asyncio
import base64
from collections.abc import Callable
import os
import threading
import time
from typing import Any, TypeVar
from artemis.runtime.device_lock import DeviceExecutionLock
from artemis.utils.logger import get_logger
logger = get_logger(__name__)
T = TypeVar("T")
#: On-device packages installed by uiautomator2 (``python -m uiautomator2 init``).
UIAUTOMATOR_PACKAGES: tuple[str, ...] = (
"com.github.uiautomator",
"com.github.uiautomator.test",
)
#: A real JPEG screenshot of any phone screen is tens of kilobytes; the Android
#: driver substitutes a 1x1 PNG (~70 bytes) when every capture path failed.
_MIN_PLAUSIBLE_SCREENSHOT_BYTES = 512
_SERIAL_PLACEHOLDER = "<serial>"
def _serial_or_placeholder(serial: str | None) -> str:
return serial or _SERIAL_PLACEHOLDER
def _uiautomator_fix(serial: str | None) -> list[str]:
s = _serial_or_placeholder(serial)
force_stops = ", then ".join(
f"adb -s {s} shell am force-stop {pkg}" for pkg in UIAUTOMATOR_PACKAGES
)
return [
"Unlock the phone and keep the screen on (UIAutomator cannot dump a locked or sleeping screen).",
f"Restart the on-device UIAutomator server, then retry: {force_stops}",
(
"If it keeps failing, reinstall the UIAutomator server APKs: "
f"python -m uiautomator2 purge --serial {s}, then python -m uiautomator2 init --serial {s} "
"(run with the ARTEMIS virtualenv Python), or reboot the device."
),
]
def fix_for_error(error: str | None, serial: str | None) -> list[str]:
"""Map an error string to concrete repair steps (substring match, case-insensitive)."""
if not error:
return []
text = error.lower()
s = _serial_or_placeholder(serial)
if "busy" in text or "another task" in text:
return [
"Wait for the running task to finish, or stop it with mobile_manage_task(action='stop', ...).",
"Then rerun the smoke test.",
]
if "unauthorized" in text:
return [
"Accept the 'Allow USB debugging' prompt on the phone (tick 'Always allow from this computer').",
f"If no prompt appears, re-plug the USB cable and run: adb -s {s} devices",
]
if "offline" in text:
return [
"Re-plug the USB cable (try another port or cable).",
"Run: adb reconnect (or adb kill-server, then adb start-server) and check adb devices -l.",
]
if "no android devices" in text or "no device" in text or "not found" in text:
return [
"Connect an Android device with USB debugging enabled, or start an emulator.",
"mobile_diagnose(launch_avd='<avd name>') boots a named AVD; adb devices -l lists what is attached.",
]
if (
"timeout" in text
or "timed out" in text
or "uiautomator" in text
or "not responding" in text
or "did not respond" in text
or "hierarchy" in text
or "screenshot" in text
or "http" in text
or "deadobject" in text
):
return _uiautomator_fix(serial)
return [
f"Check the raw hierarchy dump manually: adb -s {s} shell uiautomator dump, then adb -s {s} shell cat /sdcard/window_dump.xml",
f"Check the raw screenshot path: adb -s {s} exec-out screencap -p > screen.png",
]
#: Exceptions the smoke test must never swallow.
_PASSTHROUGH_EXCEPTIONS: tuple[type[BaseException], ...] = (
KeyboardInterrupt,
SystemExit,
asyncio.CancelledError,
)
def _describe_failure(exc: BaseException) -> str:
"""uiautomator2 errors derive from ``BaseException`` and often carry an empty message."""
text = str(exc).strip()
name = type(exc).__name__
if not text:
return name
return text if name in ("Exception", "RuntimeError") else f"{name}: {text}"
def _resolve_requested_serial(device_serial: str | None) -> str | None:
"""Mirror ``_get_controller``'s serial resolution (explicit arg, then env)."""
return (
device_serial or os.environ.get("ARTEMIS_DEVICE_ID") or os.environ.get("ADB_DEVICE_SERIAL")
)
def _find_busy_owner(target_serial: str | None) -> tuple[str, Any] | None:
"""Return ``(device_key, owner)`` when a live ARTEMIS process owns the target device.
With no target serial ``_get_controller`` would pick the first attached device,
which cannot be matched against lock records cheaply; any live owner is then
treated as busy to stay on the safe side.
"""
try:
owners = DeviceExecutionLock.get_active_owners()
except Exception as exc: # pylint: disable=broad-exception-caught
logger.debug(f"Device lock inspection failed, skipping busy guard: {exc}")
return None
if not owners:
return None
if target_serial is None:
key, owner = next(iter(owners.items()))
return key, owner
clean = DeviceExecutionLock._normalize_device_id(target_serial)
for key, owner in owners.items():
if key == clean or key.endswith(f"__{clean}"):
return key, owner
return None
async def _run_in_daemon_thread(fn: Callable[[], T], timeout: float, what: str) -> T:
"""Run ``fn`` on a daemon thread and wait at most ``timeout`` seconds.
Raises :class:`TimeoutError` on deadline. The thread keeps running until
``fn`` returns on its own, but the caller is released immediately.
"""
loop = asyncio.get_running_loop()
future: asyncio.Future[T] = loop.create_future()
def _deliver(setter: Callable[[], None]) -> None:
try:
loop.call_soon_threadsafe(setter)
except RuntimeError:
# Event loop already closed: the caller gave up on us.
pass
def _worker() -> None:
try:
value = fn()
except BaseException as exc: # pylint: disable=broad-exception-caught
error: BaseException = exc
def _fail() -> None:
if not future.done():
future.set_exception(error)
_deliver(_fail)
return
def _succeed() -> None:
if not future.done():
future.set_result(value)
_deliver(_succeed)
threading.Thread(target=_worker, name=f"artemis-smoke-{what}", daemon=True).start()
try:
return await asyncio.wait_for(future, timeout=timeout)
except TimeoutError:
logger.warning(f"Device smoke test: {what} did not finish within {timeout:.1f}s")
raise
def _run_coroutine_blocking(coro_factory: Callable[[], Any]) -> Any:
"""Execute a coroutine on a fresh event loop owned by the current thread."""
return asyncio.run(coro_factory())
def _count_elements(elements: Any) -> int | None:
if elements is None:
return None
if isinstance(elements, str):
return elements.count("<node")
if isinstance(elements, dict):
return 1
try:
return len(elements)
except TypeError:
return None
def _decoded_length(b64: Any) -> int | None:
if not isinstance(b64, str) or not b64:
return None
try:
return len(base64.b64decode(b64))
except (ValueError, TypeError):
return None
async def smoke_test_device(
device_serial: str | None = None, timeout_seconds: float = 20.0
) -> dict[str, Any]:
"""Observe the device exactly like ``mobile_get_device_state`` and report a verdict.
Never raises. Returns::
{"ok": bool, "serial": str | None, "elapsed_seconds": float,
"screenshot_bytes": int | None, "element_count": int | None,
"error": str | None, "fix": list[str]}
"""
started = time.monotonic()
requested_serial = _resolve_requested_serial(device_serial)
result: dict[str, Any] = {
"ok": False,
"serial": requested_serial,
"elapsed_seconds": 0.0,
"screenshot_bytes": None,
"element_count": None,
"error": None,
"fix": [],
}
def _finish(error: str | None = None, cause: str | None = None) -> dict[str, Any]:
"""Record the verdict; ``cause`` is the raw failure text used to pick the fix."""
result["elapsed_seconds"] = round(time.monotonic() - started, 3)
result["error"] = error
result["ok"] = error is None
result["fix"] = fix_for_error(cause if cause is not None else error, result["serial"])
return result
busy = _find_busy_owner(requested_serial)
if busy is not None:
key, owner = busy
description = getattr(owner, "description", "") or "unknown task"
pid = getattr(owner, "pid", "?")
result["serial"] = requested_serial or getattr(owner, "device_id", None)
return _finish(
f"Device {key} is busy: another task is running on it ({description}, pid {pid});"
" skipping the screen capture so the running task is not disturbed."
)
try:
from artemis.mcp import adb_server
controller = await _run_in_daemon_thread(
lambda: adb_server._get_controller(device_serial=device_serial),
timeout_seconds,
"controller-init",
)
except TimeoutError:
return _finish(
f"UIAutomator/controller initialization did not respond within {timeout_seconds:g}s"
)
except _PASSTHROUGH_EXCEPTIONS:
raise
except BaseException as exc: # pylint: disable=broad-exception-caught
cause = _describe_failure(exc)
return _finish(f"Failed to initialize Android device controller: {cause}", cause)
controller_ctx = getattr(controller, "ctx", None)
device = getattr(controller_ctx, "device", None)
device_id = getattr(device, "device_id", None)
if isinstance(device_id, str) and device_id:
result["serial"] = device_id
remaining = max(0.5, timeout_seconds - (time.monotonic() - started))
try:
device_data = await _run_in_daemon_thread(
lambda: _run_coroutine_blocking(controller.get_screen_data),
remaining,
"screen-data",
)
except TimeoutError:
return _finish(f"UIAutomator/screen capture did not respond within {timeout_seconds:g}s")
except _PASSTHROUGH_EXCEPTIONS:
raise
except BaseException as exc: # pylint: disable=broad-exception-caught
cause = _describe_failure(exc)
return _finish(f"Screen capture failed: {cause}", cause)
result["screenshot_bytes"] = _decoded_length(getattr(device_data, "base64", None))
result["element_count"] = _count_elements(getattr(device_data, "elements", None))
if result["screenshot_bytes"] is None:
return _finish("Screen capture returned no screenshot data")
if result["screenshot_bytes"] < _MIN_PLAUSIBLE_SCREENSHOT_BYTES:
return _finish(
f"Screenshot capture failed (driver returned a {result['screenshot_bytes']}-byte placeholder image)"
)
if not result["element_count"]:
return _finish(
"UIAutomator hierarchy dump returned no UI elements (screenshot worked, hierarchy did not)"
)
return _finish(None)
__all__ = ["UIAUTOMATOR_PACKAGES", "fix_for_error", "smoke_test_device"]
+80 -4
View File
@@ -15,6 +15,7 @@
"""System Readiness & Diagnostic Orchestration Engine.""" """System Readiness & Diagnostic Orchestration Engine."""
import asyncio import asyncio
import os
import subprocess import subprocess
import time import time
from typing import Any from typing import Any
@@ -33,6 +34,7 @@ from artemis.core.diagnostics.probes.runtime_probe import (
from artemis.core.diagnostics.probes.toolchain_probe import ToolchainProbe from artemis.core.diagnostics.probes.toolchain_probe import ToolchainProbe
from artemis.core.diagnostics.schema import ( from artemis.core.diagnostics.schema import (
DeviceInfo, DeviceInfo,
ProbeAction,
ProbeCategory, ProbeCategory,
ProbeResult, ProbeResult,
ProbeStatus, ProbeStatus,
@@ -50,6 +52,13 @@ class ReadinessEngine:
_REPORT_CACHE_TTL_SECONDS = 2.0 _REPORT_CACHE_TTL_SECONDS = 2.0
#: Deadline for one probe inside a report. The ADB probe shells out to
#: ``adb`` without a deadline of its own, and a wedged ADB server would
#: otherwise hang every surface that runs the report (``artemis doctor``,
#: the console wizard, ``mobile_diagnose``). A probe that overruns is
#: reported as a FAIL with the recovery steps instead.
PROBE_TIMEOUT_SECONDS = 30.0
def __init__(self): def __init__(self):
self._probes: dict[str, BaseProbe] = {} self._probes: dict[str, BaseProbe] = {}
self._python_probe = PythonRuntimeProbe() self._python_probe = PythonRuntimeProbe()
@@ -178,11 +187,22 @@ class ReadinessEngine:
if categories is None or probe.category in categories if categories is None or probe.category in categories
] ]
# Concurrently execute probes # Concurrently execute probes. A probe that raises must not take the
results: list[ProbeResult] = await asyncio.gather( # whole report down with it: it becomes a structured FAIL so the
*[probe.probe() for probe in target_probes], # remaining probes still reach the user and the verdict stays honest.
return_exceptions=False, outcomes = await asyncio.gather(
*[
asyncio.wait_for(probe.probe(), timeout=self.PROBE_TIMEOUT_SECONDS)
for probe in target_probes
],
return_exceptions=True,
) )
results: list[ProbeResult] = [
outcome
if isinstance(outcome, ProbeResult)
else self._crashed_probe_result(probe, outcome)
for probe, outcome in zip(target_probes, outcomes, strict=True)
]
blockers = [r for r in results if r.is_blocker] blockers = [r for r in results if r.is_blocker]
passed_blockers = [r for r in blockers if r.status == ProbeStatus.PASS] passed_blockers = [r for r in blockers if r.status == ProbeStatus.PASS]
@@ -211,6 +231,62 @@ class ReadinessEngine:
timestamp=time.time(), timestamp=time.time(),
) )
@staticmethod
def _crashed_probe_result(probe: BaseProbe, exc: object) -> ProbeResult:
"""Turn an exception escaping ``probe.probe()`` into a FAIL result.
Cancellation and other non-``Exception`` errors are re-raised: they are
not a diagnosis of the host, they are the event loop shutting us down.
"""
if isinstance(exc, BaseException) and not isinstance(exc, Exception):
raise exc
if isinstance(exc, TimeoutError):
logger.warning(
f"[ReadinessEngine] Probe '{probe.probe_id}' did not finish within"
f" {ReadinessEngine.PROBE_TIMEOUT_SECONDS:.0f}s; reporting it as FAIL."
)
return ProbeResult(
id=probe.probe_id,
category=probe.category,
title=probe.probe_id.replace("_", " ").title(),
status=ProbeStatus.FAIL,
is_blocker=probe.is_blocker,
summary="Probe timed out",
description=(
f"The '{probe.probe_id}' check did not finish within"
f" {ReadinessEngine.PROBE_TIMEOUT_SECONDS:.0f}s. A hung ADB server is the"
" usual cause: restart it and run the diagnosis again."
),
actions=[
ProbeAction(
action_type="command",
label="Restart ADB",
payload="adb kill-server && adb start-server",
),
],
metadata={
"exception_type": "TimeoutError",
"timeout_seconds": ReadinessEngine.PROBE_TIMEOUT_SECONDS,
},
)
logger.warning(
f"[ReadinessEngine] Probe '{probe.probe_id}' crashed; reporting it as FAIL: "
f"{type(exc).__name__}: {exc}"
)
return ProbeResult(
id=probe.probe_id,
category=probe.category,
title=probe.probe_id.replace("_", " ").title(),
status=ProbeStatus.FAIL,
is_blocker=probe.is_blocker,
summary="Probe crashed",
description=(
f"The '{probe.probe_id}' check raised {type(exc).__name__}: {exc}. "
"This is a diagnostics bug or a host permission problem, not a device fault."
),
metadata={"exception_type": type(exc).__name__, "exception": str(exc)},
)
async def heal_adb_keys(self, force: bool = False) -> dict[str, Any]: async def heal_adb_keys(self, force: bool = False) -> dict[str, Any]:
"""Auto-heal corrupted or 0-byte ADB authentication keys.""" """Auto-heal corrupted or 0-byte ADB authentication keys."""
from artemis.core.diagnostics.adb_keys import heal_adb_keys from artemis.core.diagnostics.adb_keys import heal_adb_keys
@@ -284,7 +284,7 @@ class VisionOCRProbe(BaseProbe):
status=ProbeStatus.PASS, status=ProbeStatus.PASS,
is_blocker=False, is_blocker=False,
summary="Not Configured (Optional)", summary="Not Configured (Optional)",
description="OCR_API_KEY is not set. Perception operates seamlessly in pure UI XML hierarchy mode.", description="OCR_API_KEY is not set. Perception uses the UI XML hierarchy without OCR.",
metadata={"configured": False}, metadata={"configured": False},
actions=[ actions=[
ProbeAction( ProbeAction(
@@ -0,0 +1,427 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Integration-host probe: the process environment an IDE / MCP client launched us in.
The standard readiness probes answer "can Artemis drive a phone on this
machine?". This probe answers the question that precedes it when Artemis is
embedded in an AI coding assistant (Claude Code, Cursor, Codex, Antigravity,
...): "was the server launched the way the installer expects, and can it
spawn a background runner?". The failure modes it covers are the ones users
hit mostly through MCP, rarely from the CLI or the web console:
* the MCP client started the server with a different interpreter than the
project virtualenv, so the spawned runner imports a different package set;
* the ``.env`` file the settings loader reads is not where the user put the key;
* the traces directory is not writable from the client's sandbox;
* the daemon port is held by another process (a very common collision with
dev servers that also default to port 8000).
The probe is not registered in the default :class:`ReadinessEngine` probe set:
the web console has its own device wizard and would only render noise. The
``mobile_diagnose`` MCP tool and ``artemis doctor`` run it explicitly.
Only the unwritable-traces case blocks task execution; the result carries
``is_blocker`` accordingly, so the WARN findings degrade the verdict without
blocking it.
"""
from __future__ import annotations
import os
from pathlib import Path
import sys
import tempfile
from typing import Any
from artemis.core.diagnostics.probes.base import BaseProbe
from artemis.core.diagnostics.schema import (
ProbeAction,
ProbeCategory,
ProbeResult,
ProbeStatus,
)
def _same_file(a: str | os.PathLike[str], b: str | os.PathLike[str]) -> bool:
"""Compare two interpreter paths tolerantly (symlinks, case on Windows)."""
try:
return os.path.samefile(a, b)
except OSError:
return os.path.normcase(os.path.abspath(a)) == os.path.normcase(os.path.abspath(b))
#: Client names accepted by ``artemis mcp --install`` (see interfaces/cli/commands/mcp.py).
MCP_CLIENT_NAMES: tuple[str, ...] = (
"antigravity",
"claude",
"cursor",
"windsurf",
"vscode",
"cline",
"roo",
"openclaw",
"codex",
)
def _env_has_prefix(prefix: str, *, exclude: tuple[str, ...] = ()) -> bool:
return any(key.startswith(prefix) and key not in exclude for key in os.environ)
def detect_mcp_client() -> str | None:
"""Best-effort guess of which MCP client launched this process, from ``os.environ``.
Returns one of :data:`MCP_CLIENT_NAMES` or ``None`` when unsure. MCP hosts
spawn the server with their own process environment, so the variables the
IDE exports to its child processes leak through. Specific hosts are checked
before the generic VS Code markers because Cursor, Windsurf, Cline and Roo
are (or run inside) VS Code and inherit ``VSCODE_*`` / ``TERM_PROGRAM``.
"""
# Antigravity injects its language-server endpoint and CSRF token
# (ANTIGRAVITY_LS_ADDRESS / ANTIGRAVITY_CSRF_TOKEN) so child processes can
# call `agentapi`; mcp_server/notifiers/agentapi.py relies on the same pair.
if _env_has_prefix("ANTIGRAVITY_"):
return "antigravity"
# Claude Code marks every child with CLAUDECODE=1 plus CLAUDE_CODE_* details
# (entrypoint, session id, ...). Claude Desktop shares the same installer
# target, so both map to "claude".
if os.environ.get("CLAUDECODE") == "1" or _env_has_prefix("CLAUDE_CODE_"):
return "claude"
# Cursor tags its integrated terminal and spawned processes with
# CURSOR_TRACE_ID (and CURSOR_AGENT in agent mode).
if "CURSOR_TRACE_ID" in os.environ or _env_has_prefix("CURSOR_"):
return "cursor"
# Windsurf (Codeium) exports WINDSURF_* to its child processes.
if _env_has_prefix("WINDSURF_"):
return "windsurf"
# Codex CLI sets CODEX_SANDBOX* / CODEX_* for the processes it runs.
# CODEX_HOME alone is excluded: users set it by hand to relocate the Codex
# config directory, so on its own it does not prove Codex launched us.
if _env_has_prefix("CODEX_", exclude=("CODEX_HOME",)):
return "codex"
# OPENCLAW_WEBHOOK_URL is a notifier setting users put in any host's env
# block (see mcp_server/notifiers/webhook.py); only other OPENCLAW_* vars
# indicate the OpenClaw runtime itself.
if _env_has_prefix("OPENCLAW_", exclude=("OPENCLAW_WEBHOOK_URL",)):
return "openclaw"
# Cline / Roo Code run as VS Code extensions; their own prefixes are rare
# but unambiguous when present, and must win over the generic VS Code check.
if _env_has_prefix("CLINE_"):
return "cline"
if _env_has_prefix("ROO_"):
return "roo"
# Plain VS Code (Copilot agent mode) exports VSCODE_PID / VSCODE_IPC_HOOK
# and sets TERM_PROGRAM=vscode. Weakest signal, so it is checked last.
if (
"VSCODE_PID" in os.environ
or "VSCODE_IPC_HOOK" in os.environ
or os.environ.get("TERM_PROGRAM") == "vscode"
):
return "vscode"
return None
def mcp_install_actions(client: str | None) -> list[ProbeAction]:
"""Actions that regenerate the MCP client config for ``client`` (or ask which one)."""
if client:
return [
ProbeAction(
action_type="command",
label="Regenerate MCP Config",
payload=f"uv run artemis mcp --install {client}",
)
]
return [
ProbeAction(
action_type="hint",
label="Regenerate MCP Config",
payload=(
"Could not tell which MCP client launched this server. Ask the user which IDE"
" they are using, then run `uv run artemis mcp --install <client>` with"
" <client> being one of: " + ", ".join(MCP_CLIENT_NAMES) + "."
),
)
]
def project_venv_python(project_root: Path) -> Path:
"""Return the interpreter path the installer writes into MCP client configs."""
if sys.platform == "win32":
return project_root / ".venv" / "Scripts" / "python.exe"
return project_root / ".venv" / "bin" / "python"
def directory_is_writable(path: Path) -> tuple[bool, str | None]:
"""Try to create and delete a scratch file; return (ok, error)."""
try:
path.mkdir(parents=True, exist_ok=True)
with tempfile.NamedTemporaryFile(dir=path, prefix=".artemis-diag-", delete=True):
pass
return True, None
except OSError as exc:
return False, str(exc)
def intended_traces_path() -> Path:
"""Return the traces directory the settings loader *would* pick, without creating it.
Mirrors the selection order of :func:`artemis.config.paths.get_default_traces_path`
(``ARTEMIS_TRACES_DIR`` env, else the user app dir when the install mode
keeps mutable state there, else ``ROOT_DIR/traces``) minus its ``mkdir``.
Used only when that function raised, so the probe can still name the
offending directory in its FAIL result instead of crashing.
"""
from artemis.config import paths as artemis_paths
env_traces = os.getenv(artemis_paths.ENV_ARTEMIS_TRACES_DIR)
if env_traces:
return Path(env_traces)
if artemis_paths._use_user_app_dir():
return artemis_paths.get_app_dir() / "traces"
return Path(artemis_paths.ROOT_DIR) / "traces"
def resolve_traces_dir() -> tuple[Path, str | None]:
"""Return ``(traces_dir, error)``; ``error`` is set when the directory could not be created.
``get_default_traces_path`` creates the directory as a side effect, so on a
read-only parent it raises before :func:`directory_is_writable` ever runs.
Surface that as data rather than letting the probe itself crash.
"""
from artemis.config.paths import get_default_traces_path
try:
return Path(get_default_traces_path()), None
except OSError as exc:
return intended_traces_path(), str(exc)
class IntegrationHostProbe(BaseProbe):
"""Probe verifying the MCP / IDE host process can actually run Artemis tasks."""
@property
def probe_id(self) -> str:
return "integration_host"
@property
def category(self) -> ProbeCategory:
return ProbeCategory.RUNTIME
@property
def is_blocker(self) -> bool:
return True
async def probe(self) -> ProbeResult:
# Lazy imports keep this module importable in stripped test environments
# and avoid a core -> runtime import at package import time.
from artemis.config.paths import ROOT_DIR, get_env_file, is_source_checkout
from artemis.runtime.daemon_client import (
DEFAULT_DAEMON_HOST,
DEFAULT_DAEMON_PORT,
daemon_log_path,
is_artemis_daemon,
is_standalone_forced,
)
from artemis.runtime.server_lifecycle import is_port_in_use
project_root = Path(ROOT_DIR)
source_checkout = is_source_checkout()
server_python = Path(sys.executable)
# A project virtualenv is only meaningful for a source checkout. In a
# wheel install ROOT_DIR is site-packages, which never carries a .venv;
# the current interpreter also runs background tasks.
venv_python: Path | None = project_venv_python(project_root) if source_checkout else None
venv_exists = venv_python is not None and venv_python.is_file()
runner_python = venv_python if venv_exists else server_python
interpreter_matches_venv = venv_exists and _same_file(server_python, venv_python)
mcp_client = detect_mcp_client()
env_file = Path(get_env_file())
traces_dir, traces_error = resolve_traces_dir()
if traces_error is None:
traces_writable, traces_error = directory_is_writable(traces_dir)
else:
traces_writable = False
standalone = is_standalone_forced()
# Identity, not liveness: a dev server squatting on the port answers
# HTTP too, and must show up as a hijacked port, not as the daemon.
daemon_reachable = False if standalone else is_artemis_daemon()
port_in_use = is_port_in_use(DEFAULT_DAEMON_PORT, DEFAULT_DAEMON_HOST)
port_hijacked = port_in_use and not daemon_reachable and not standalone
metadata: dict[str, Any] = {
"os": sys.platform,
"mcp_client": mcp_client,
"project_root": str(project_root),
"is_source_checkout": source_checkout,
"server_python": str(server_python),
"venv_python": str(venv_python) if venv_python is not None else None,
"venv_exists": venv_exists,
"interpreter_matches_venv": interpreter_matches_venv,
"runner_python": str(runner_python),
"env_file": str(env_file),
"env_file_exists": env_file.is_file(),
"traces_dir": str(traces_dir),
"traces_dir_writable": traces_writable,
"traces_dir_error": traces_error,
"daemon": {
"host": DEFAULT_DAEMON_HOST,
"port": DEFAULT_DAEMON_PORT,
"standalone_forced": standalone,
"reachable": daemon_reachable,
"port_in_use": port_in_use,
"port_held_by_other_process": port_hijacked,
"log_path": str(daemon_log_path()),
},
}
problems: list[str] = []
actions: list[ProbeAction] = []
status = ProbeStatus.PASS
if not traces_writable:
status = ProbeStatus.FAIL
problems.append(f"traces directory {traces_dir} is not writable ({traces_error})")
actions.append(
ProbeAction(
action_type="hint",
label="Fix Traces Directory",
payload=(
f"Make {traces_dir} writable for the user running the MCP server, or point "
"ARTEMIS_TRACES_DIR at a writable directory in the MCP server's env block, "
"then restart the MCP server."
),
)
)
if port_hijacked:
status = ProbeStatus.WARN if status is ProbeStatus.PASS else status
problems.append(
f"port {DEFAULT_DAEMON_PORT} is held by a process that is not the Artemis daemon"
)
actions.append(
ProbeAction(
action_type="hint",
label="Free the Daemon Port",
payload=(
f"Another program is listening on {DEFAULT_DAEMON_HOST}:{DEFAULT_DAEMON_PORT}"
" (often a dev server). Stop it, or set ARTEMIS_DAEMON_PORT to a free port in"
" the MCP server's env block and restart the MCP server. Tasks still run in"
" standalone mode meanwhile, without the shared scheduler."
),
)
)
if not source_checkout:
# Wheel / frozen install: no virtualenv expectation, nothing to warn about.
pass
elif venv_exists and not interpreter_matches_venv:
status = ProbeStatus.WARN if status is ProbeStatus.PASS else status
problems.append(
f"MCP server runs on {server_python}, not the project virtualenv {venv_python}"
)
actions.extend(mcp_install_actions(mcp_client))
actions.append(
ProbeAction(
action_type="hint",
label="Interpreter Mismatch",
payload=(
"The MCP client config should point `command` at the .venv interpreter so the"
" server and the background runner share one package set. Restart the MCP"
" server after changing the config."
),
)
)
elif not venv_exists:
status = ProbeStatus.WARN if status is ProbeStatus.PASS else status
problems.append(
"no project virtualenv (.venv) found; runner falls back to the server interpreter"
)
actions.append(
ProbeAction(
action_type="command",
label="Create Virtualenv",
payload="uv sync",
)
)
if not env_file.is_file():
actions.append(
ProbeAction(
action_type="hint",
label="Env File Location",
payload=(
f"No .env file at {env_file}. API keys are read from that file or from the MCP"
" server's env block; the file is created by `artemis init`."
),
)
)
if not daemon_reachable and not port_hijacked:
actions.append(
ProbeAction(
action_type="hint",
label="Daemon Auto-Start",
payload=(
"Standalone mode is forced (ARTEMIS_STANDALONE=1); tasks run as detached"
" runner processes."
if standalone
else "The Artemis daemon is not running; mobile_run_task starts it on the"
f" first task. Its log is {daemon_log_path()}."
),
)
)
if status is ProbeStatus.PASS:
summary = "Host Ready"
description = (
f"MCP server runs on {server_python}; runner uses {runner_python}. "
f"Traces at {traces_dir}. "
+ (
"Daemon reachable."
if daemon_reachable
else "Daemon will auto-start on the first task."
)
)
actions.insert(
0,
ProbeAction(
action_type="hint",
label="Host OK",
payload="Interpreter, traces directory and daemon port look consistent.",
),
)
else:
summary = "Host Misconfigured" if status is ProbeStatus.FAIL else "Host Warning"
joined = "; ".join(problems)
description = joined[:1].upper() + joined[1:] + "."
# Only an unwritable traces directory stops tasks from running. The
# WARN findings (interpreter mismatch, missing venv, squatted port) are
# degradations the runner works around, so they must not turn the
# verdict into "blocked" — a pip/conda checkout without .venv would
# otherwise never leave that state.
return ProbeResult(
id=self.probe_id,
category=self.category,
title="MCP / IDE Integration Host",
status=status,
is_blocker=status is not ProbeStatus.WARN,
summary=summary,
description=description,
metadata=metadata,
actions=actions,
)
+83
View File
@@ -0,0 +1,83 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Readiness collection and verdict shared by every diagnosis entry point.
``artemis doctor`` and the ``mobile_diagnose`` MCP tool must agree on which
probes run, in which order they are fixed, and what "ready" means. This module
holds exactly that common core; each entry point layers its own extra
conditions, fix policy and rendering on top.
"""
from __future__ import annotations
import asyncio
from collections.abc import Sequence
from typing import Literal
from artemis.core.diagnostics.engine import readiness_engine
from artemis.core.diagnostics.probes.host_probe import IntegrationHostProbe
from artemis.core.diagnostics.schema import ProbeResult, ProbeStatus, SystemReadinessReport
Verdict = Literal["ready", "degraded", "blocked"]
#: Fix order. Runtime problems mask everything after them, credentials mask
#: device problems (a task cannot start without a model), and the toolchain is
#: optional.
CHECK_ORDER: dict[str, int] = {
"python_runtime": 0,
"system_config": 1,
"integration_host": 2,
"gemini_api_key": 3,
"android_adb": 4,
"toolchain": 5,
"vision_ocr_key": 6,
}
def sort_by_fix_order(results: Sequence[ProbeResult]) -> list[ProbeResult]:
"""Order probe results so the first problem listed is the one to fix first."""
return sorted(results, key=lambda r: CHECK_ORDER.get(r.id, len(CHECK_ORDER)))
async def collect_readiness() -> tuple[SystemReadinessReport, ProbeResult]:
"""Run the engine probes and the integration-host probe concurrently."""
report, host = await asyncio.gather(
readiness_engine.run_all(force_refresh=True),
IntegrationHostProbe().probe(),
)
return report, host
def base_verdict(results: Sequence[ProbeResult]) -> Verdict:
"""Verdict from probe results alone.
Blocked when there is no blocker at all (nothing was checked) or any
blocker is not PASS; degraded when only optional checks fail; ready
otherwise.
"""
blockers = [r for r in results if r.is_blocker]
if not blockers or any(r.status is not ProbeStatus.PASS for r in blockers):
return "blocked"
if any(r.status is not ProbeStatus.PASS for r in results):
return "degraded"
return "ready"
def adb_keys_corrupted(results: Sequence[ProbeResult]) -> bool:
"""Whether the ADB probe reported corrupted RSA keys (the one self-heal both entry points share)."""
adb = next((r for r in results if r.id == "android_adb"), None)
if adb is None:
return False
return bool((adb.metadata.get("adb_keys") or {}).get("is_corrupted"))
+9 -2
View File
@@ -666,6 +666,13 @@ class DataEngine:
if session and session.end_time is not None and session.status not in ("running", "paused"): if session and session.end_time is not None and session.status not in ("running", "paused"):
logger.debug(f"Session end already published for {session_id}; skipping duplicate") logger.debug(f"Session end already published for {session_id}; skipping duplicate")
return return
# Session-level LLM usage line (cache-hit ratios per source), best-effort.
try:
from artemis.services.token_meter import log_session_summary
log_session_summary(session_id)
except Exception as e:
logger.debug(f"Session usage summary skipped: {e}")
if session: if session:
session.end_time = end_time session.end_time = end_time
session.status = status session.status = status
@@ -877,7 +884,7 @@ class DataEngine:
if pre_image_name and post_image_name and pre_image_name == post_image_name: if pre_image_name and post_image_name and pre_image_name == post_image_name:
post_image_name = None post_image_name = None
# M5: persist the foreground app for the recall search surface (the # Persist the foreground app for the recall search surface (the
# parameter was historically accepted but never stored). The explicit # parameter was historically accepted but never stored). The explicit
# parameter wins; otherwise it is derived best-effort from the UI # parameter wins; otherwise it is derived best-effort from the UI
# tree's package attributes — a pure string scan, no device call. # tree's package attributes — a pure string scan, no device call.
@@ -889,7 +896,7 @@ class DataEngine:
except (AttributeError, TypeError, ValueError): except (AttributeError, TypeError, ValueError):
pass pass
# M4: stamp perceptual hashes for the local screen-similarity hint # Stamp perceptual hashes for the local screen-similarity hint
# (pure PIL dHash; best-effort, computed synchronously to keep the # (pure PIL dHash; best-effort, computed synchronously to keep the
# record immutable once handed to the background writer). # record immutable once handed to the background writer).
try: try:
+58 -6
View File
@@ -14,6 +14,7 @@
import base64 import base64
from collections.abc import Callable from collections.abc import Callable
import contextlib
import functools import functools
import hashlib import hashlib
import inspect import inspect
@@ -175,6 +176,20 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
def on_tool_error(self, error: Any, *, run_id: uuid.UUID, **kwargs: Any) -> Any: def on_tool_error(self, error: Any, *, run_id: uuid.UUID, **kwargs: Any) -> Any:
pass pass
@staticmethod
def _detached_lens() -> bool:
"""True inside ``detached_trace('lens:...')``: a background lens call
(visual summary, segment capsule) that belongs to no agent step."""
node_name = CURRENT_NODE_NAME.get() or ""
return CURRENT_TRACE_ID.get() is None and node_name.startswith("lens:")
def _current_step_id(self):
"""The step a trace attaches to: the engine's current step, or None for
a detached lens call (its reply must never read as that step's work)."""
if self._detached_lens():
return None
return self.ctx.data_engine.current_step_id
def on_llm_start( def on_llm_start(
self, self,
serialized: dict[str, Any], serialized: dict[str, Any],
@@ -200,7 +215,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
trace_id=run_id, trace_id=run_id,
parent_trace_id=self._resolve_parent_id(parent_run_id, kwargs), parent_trace_id=self._resolve_parent_id(parent_run_id, kwargs),
status="running", status="running",
step_id=self.ctx.data_engine.current_step_id, step_id=self._current_step_id(),
) )
def on_chat_model_start( def on_chat_model_start(
@@ -233,7 +248,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
trace_id=run_id, trace_id=run_id,
parent_trace_id=self._resolve_parent_id(parent_run_id, kwargs), parent_trace_id=self._resolve_parent_id(parent_run_id, kwargs),
status="running", status="running",
step_id=self.ctx.data_engine.current_step_id, step_id=self._current_step_id(),
) )
def _safe_uuid5(self, parent_id, name: str) -> uuid.UUID: def _safe_uuid5(self, parent_id, name: str) -> uuid.UUID:
@@ -253,6 +268,11 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
) -> Any: ) -> Any:
"""Run when LLM ends running.""" """Run when LLM ends running."""
if self.ctx.data_engine: if self.ctx.data_engine:
# A background lens call (see ``detached_trace``) belongs to no step:
# its reply is a compression product, never an agent's reasoning, so
# it is neither attached to the current step nor recorded as thinking.
detached = self._detached_lens()
step_id = self._current_step_id()
generations = getattr(response, "generations", []) generations = getattr(response, "generations", [])
flat_generations = [] flat_generations = []
if generations: if generations:
@@ -308,7 +328,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
parent_trace_id=self._resolve_parent_id(parent_run_id), parent_trace_id=self._resolve_parent_id(parent_run_id),
status="success", status="success",
duration=duration, duration=duration,
step_id=self.ctx.data_engine.current_step_id, step_id=step_id,
) )
native_thoughts = [] native_thoughts = []
@@ -327,6 +347,8 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
elif gen_item.get("text") and str(gen_item.get("text")).strip(): elif gen_item.get("text") and str(gen_item.get("text")).strip():
raw_thoughts.append(str(gen_item.get("text")).strip()) raw_thoughts.append(str(gen_item.get("text")).strip())
if detached:
native_thoughts, raw_thoughts = [], []
if native_thoughts: if native_thoughts:
self.ctx.data_engine.record_trace( self.ctx.data_engine.record_trace(
type="thinking", type="thinking",
@@ -335,7 +357,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
trace_id=self._safe_uuid5(run_id, "thinking"), trace_id=self._safe_uuid5(run_id, "thinking"),
parent_trace_id=run_id, parent_trace_id=run_id,
status="success", status="success",
step_id=self.ctx.data_engine.current_step_id, step_id=step_id,
) )
if raw_thoughts: if raw_thoughts:
self.ctx.data_engine.record_trace( self.ctx.data_engine.record_trace(
@@ -345,7 +367,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
trace_id=self._safe_uuid5(run_id, "raw_thinking"), trace_id=self._safe_uuid5(run_id, "raw_thinking"),
parent_trace_id=run_id, parent_trace_id=run_id,
status="success", status="success",
step_id=self.ctx.data_engine.current_step_id, step_id=step_id,
) )
def on_llm_error( def on_llm_error(
@@ -379,7 +401,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
parent_trace_id=self._resolve_parent_id(parent_run_id), parent_trace_id=self._resolve_parent_id(parent_run_id),
status="failed", status="failed",
duration=duration, duration=duration,
step_id=self.ctx.data_engine.current_step_id, step_id=self._current_step_id(),
) )
def _serialize_message(self, msg: BaseMessage) -> dict[str, Any]: def _serialize_message(self, msg: BaseMessage) -> dict[str, Any]:
@@ -666,6 +688,36 @@ class TraceSpan:
) )
@contextlib.contextmanager
def detached_trace(node_name: str | None = None):
"""Run a block outside the caller's trace span.
Background work (the step-memory lenses: visual-transition summaries,
segment capsules) is dispatched while an agent's step span is current, and
an ``asyncio`` task inherits that context. Every model call made inside
the block would then hang off the agent's span: the callback handler
records its reply as a ``thinking``/``raw_thinking`` child of that span, so
``replay_steps``/``search_history`` later show a capsule JSON as the
operator's own reasoning, and the stream deltas surface under the
operator's timeline entry.
Inside the block :data:`CURRENT_TRACE_ID` is cleared (the block's traces
have no parent span and can never be attributed to the caller's step) and
:data:`CURRENT_NODE_NAME` is replaced by ``node_name`` (when given), so
usage metering labels the calls after the background job rather than the
agent that happened to dispatch it. Both are restored on exit, exceptions
included. Only the current task's context is touched.
"""
token = CURRENT_TRACE_ID.set(None)
token_name = CURRENT_NODE_NAME.set(node_name) if node_name is not None else None
try:
yield
finally:
CURRENT_TRACE_ID.reset(token)
if token_name is not None:
CURRENT_NODE_NAME.reset(token_name)
def trace_langchain_tool(tool: BaseTool, ctx: ArtemisContext) -> BaseTool: def trace_langchain_tool(tool: BaseTool, ctx: ArtemisContext) -> BaseTool:
"""Wraps a LangChain tool to trace its execution. """Wraps a LangChain tool to trace its execution.
+1 -1
View File
@@ -70,7 +70,7 @@ def _escape_for_adb_text(s: str) -> str:
class AndroidAdbDriver(BaseDeviceDriver): class AndroidAdbDriver(BaseDeviceDriver):
"""Production-grade Android Device Driver powered by ADB and UIAutomator2.""" """Android device driver using ADB and UIAutomator2."""
def __init__( def __init__(
self, self,
+162 -14
View File
@@ -135,6 +135,10 @@ class TestSummary(BaseModel):
inconclusive: int = 0 inconclusive: int = 0
unchecked: int = 0 unchecked: int = 0
failed_items: list[dict] = Field(default_factory=list) failed_items: list[dict] = Field(default_factory=list)
#: Check lines the user retired mid-run (dropped from the plan under user
#: guidance): their earlier verdicts are reported here, never as failures.
retired: int = 0
retired_items: list[dict] = Field(default_factory=list)
@property @property
def total(self) -> int: def total(self) -> int:
@@ -506,6 +510,112 @@ def _finding_registry(ctx) -> dict:
return registry return registry
def _finding_items(ctx) -> dict:
"""checkpoint_id -> ``{(kind, item_text), ...}`` of the failed verify items
that produced the headline currently held in :func:`_finding_registry`.
Sibling map of the registry (same lifetime, same keys) so a headline can
be retired when user guidance withdraws every item behind it.
"""
items = getattr(ctx, "checker_finding_items", None)
if not isinstance(items, dict):
items = {}
try:
ctx.checker_finding_items = items
except (AttributeError, TypeError, ValueError):
pass
return items
def _register_finding(ctx, checkpoint_id: str, verify_failures: list) -> None:
_finding_registry(ctx)[checkpoint_id] = _render_finding_headline(checkpoint_id, verify_failures)
_finding_items(ctx)[checkpoint_id] = {
(str(getattr(v, "kind", "") or ""), str(getattr(v, "item_text", "") or ""))
for v in verify_failures
}
def _clear_finding(ctx, checkpoint_id: str) -> bool:
"""Drops a checkpoint's standing headline; True when one was registered."""
_finding_items(ctx).pop(checkpoint_id, None)
return _finding_registry(ctx).pop(checkpoint_id, None) is not None
def _retired_checks(ctx) -> set[tuple[str, str]]:
retired = getattr(ctx, "guidance_retired_checks", None)
return retired if isinstance(retired, set) else set()
def retire_check_items(ctx, items) -> None:
"""Records ``(kind, text)`` check items the Operator dropped under user
guidance and withdraws their already-materialized findings.
Recording feeds the harvest gate (:func:`_active_verdicts`) and the run
outcome. Cleanup: every standing headline produced *only* by now-retired
items is popped and the plan's finding lines are re-projected; a headline
that still names a live item stays untouched.
"""
incoming = {(str(kind), str(text)) for kind, text in items}
if not incoming:
return
retired = _retired_checks(ctx) | incoming
ctx.guidance_retired_checks = retired
registry = _finding_registry(ctx)
produced_by = _finding_items(ctx)
stale = [cid for cid in list(registry) if produced_by.get(cid) and produced_by[cid] <= retired]
for cid in stale:
_clear_finding(ctx, cid)
logger.info(
f"Checkpoint {cid}: standing finding headline withdrawn — every item"
" behind it was retired by user guidance."
)
if stale:
sync_finding_lines(ctx)
def reinstate_check_items(ctx, items) -> None:
"""Removes ``(kind, text)`` items from the retired set: the Operator has
declared them in the plan again, so their verdicts drive side effects
once more and the run outcome counts them normally. A no-op when none of
the given items is retired."""
retired = _retired_checks(ctx)
if not retired:
return
back = {(str(kind), str(text)) for kind, text in items} & retired
if not back:
return
ctx.guidance_retired_checks = retired - back
logger.info(
"Check line(s) retired under user guidance are declared again and"
f" reinstated: {sorted(back)}"
)
def _active_verdicts(ctx, checkpoint, verdicts: list) -> list:
"""The verdicts still allowed to drive side effects: those whose
``(kind, item_text)`` was not retired by user guidance. Ledger recording is
never filtered — this gate only protects execution state."""
retired = _retired_checks(ctx)
if not retired:
return list(verdicts)
active: list = []
dropped: list = []
for v in verdicts:
key = (str(getattr(v, "kind", "") or ""), str(getattr(v, "item_text", "") or ""))
(dropped if key in retired else active).append(v)
if dropped:
logger.info(
f"Checkpoint {checkpoint.checkpoint_id}: {len(dropped)} verdict(s)"
" recorded but dropped from side effects — their check items were"
" retired by user guidance: "
+ ", ".join(
f"[{k}] '{t}'"
for k, t in ((getattr(v, "kind", ""), getattr(v, "item_text", "")) for v in dropped)
)
)
return active
def _render_finding_headline(checkpoint_id: str, verify_failures: list) -> str: def _render_finding_headline(checkpoint_id: str, verify_failures: list) -> str:
first = verify_failures[0] first = verify_failures[0]
evidence = str(getattr(first, "evidence", "") or "").strip() evidence = str(getattr(first, "evidence", "") or "").strip()
@@ -778,16 +888,26 @@ def harvest_run(
# Layer 3: the per-checkpoint repair log accumulates every booked attempt. # Layer 3: the per-checkpoint repair log accumulates every booked attempt.
_log_checker_note(ctx, run, verdicts) _log_checker_note(ctx, run, verdicts)
# Guidance gate: a verdict for a check item the user retired mid-run (the
# Operator dropped its line under a guidance waiver) is a result for a
# withdrawn requirement — booked above, never a repair/halt trigger.
active = _active_verdicts(ctx, checkpoint, verdicts)
all_retired = bool(verdicts) and not active
# Applicability gate for side effects: the verdict must belong to the # Applicability gate for side effects: the verdict must belong to the
# current attempt of its checkpoint AND the anchored subgoal's text must be # current attempt of its checkpoint AND the anchored subgoal's text must be
# unchanged in the current plan. Stale/mismatched verdicts are ledger-only. # unchanged in the current plan AND at least one verdict must still be
applicable = allow_side_effects and _subgoal_still_current(ctx, checkpoint.checkpoint_id) # live. Stale/mismatched/fully retired verdicts are ledger-only.
applicable = (
allow_side_effects
and not all_retired
and _subgoal_still_current(ctx, checkpoint.checkpoint_id)
)
user_stopped = bool(state is not None and getattr(state, "user_stop_requested", False)) user_stopped = bool(state is not None and getattr(state, "user_stop_requested", False))
verify_failures = [v for v in verdicts if v.kind == "verify" and v.status == "failed"] verify_failures = [v for v in active if v.kind == "verify" and v.status == "failed"]
assert_failures = [v for v in verdicts if v.kind == "assert" and v.status == "failed"] assert_failures = [v for v in active if v.kind == "assert" and v.status == "failed"]
registry = _finding_registry(ctx)
registry_changed = False registry_changed = False
reverted = False reverted = False
@@ -799,9 +919,7 @@ def harvest_run(
reverted = revert_subgoal_status(ctx, checkpoint.checkpoint_id) reverted = revert_subgoal_status(ctx, checkpoint.checkpoint_id)
# Layer 2: register the standing plan headline for this unresolved # Layer 2: register the standing plan headline for this unresolved
# finding; the projection below renders it under the subgoal. # finding; the projection below renders it under the subgoal.
registry[checkpoint.checkpoint_id] = _render_finding_headline( _register_finding(ctx, checkpoint.checkpoint_id, verify_failures)
checkpoint.checkpoint_id, verify_failures
)
registry_changed = True registry_changed = True
for v in verify_failures: for v in verify_failures:
suggestion = getattr(v, "suggestion", "") or "" suggestion = getattr(v, "suggestion", "") or ""
@@ -820,16 +938,18 @@ def harvest_run(
# Quota exhausted: the run is settled for this checkpoint — the # Quota exhausted: the run is settled for this checkpoint — the
# standing headline is retired (the ledger and the checker note # standing headline is retired (the ledger and the checker note
# keep the full record). # keep the full record).
registry_changed = registry.pop(checkpoint.checkpoint_id, None) is not None registry_changed = _clear_finding(ctx, checkpoint.checkpoint_id)
logger.warning( logger.warning(
f"Checkpoint {checkpoint.checkpoint_id} exhausted its repair" f"Checkpoint {checkpoint.checkpoint_id} exhausted its repair"
f" quota ({max_repairs}); verdict stays failed, no further" f" quota ({max_repairs}); verdict stays failed, no further"
" repair loop." " repair loop."
) )
elif not verify_failures: elif not verify_failures and not all_retired:
# A booked attempt without failed verify criteria resolves (or # A booked attempt without failed verify criteria resolves (or
# releases) the checkpoint: its standing headline is retired. # releases) the checkpoint: its standing headline is retired. (A fully
registry_changed = registry.pop(checkpoint.checkpoint_id, None) is not None # retired report says nothing about the checkpoint — its headline, if
# any, was already handled by ``retire_check_items``.)
registry_changed = _clear_finding(ctx, checkpoint.checkpoint_id)
if registry_changed: if registry_changed:
sync_finding_lines(ctx) sync_finding_lines(ctx)
@@ -842,11 +962,21 @@ def harvest_run(
"Assert failure under 'halt' policy: latching halt flag for exit settlement." "Assert failure under 'halt' policy: latching halt flag for exit settlement."
) )
# Event payload: ``verdicts`` carries every booked verdict (ledger view);
# ``applicable`` is False when nothing could take effect — including a
# report whose check items were all retired by user guidance; ``retired``
# lists the ``(kind, item_text)`` pairs of the verdicts dropped by that
# guidance gate (empty when none were).
_event( _event(
"done", "done",
verdict_payloads(verdicts), verdict_payloads(verdicts),
applicable=bool(applicable), applicable=bool(applicable),
reverted=bool(reverted), reverted=bool(reverted),
retired=[
{"kind": p["kind"], "item_text": p["item_text"]}
for p in verdict_payloads(verdicts)
if (p["kind"], p["item_text"]) in _retired_checks(ctx)
],
repairs_used=int(ctx.checkpoint_repairs.get(checkpoint.checkpoint_id, 0)), repairs_used=int(ctx.checkpoint_repairs.get(checkpoint.checkpoint_id, 0)),
) )
return findings return findings
@@ -1006,8 +1136,20 @@ def resolve_item_status(kind: str, records: list[dict]) -> str:
return "unchecked" return "unchecked"
def compute_test_summary(check_items: list[CheckItem], records: list[dict]) -> TestSummary: def compute_test_summary(
check_items: list[CheckItem],
records: list[dict],
retired: set[tuple[str, str]] | None = None,
) -> TestSummary:
"""Folds the verdict ledger into pass/fail counts over the declared items.
``retired`` names check lines the Operator dropped under user guidance; a
line that is retired and no longer declared in the plan is reported under
``retired`` with its last recorded verdict, not as a pass or a failure —
the user withdrew that requirement.
"""
summary = TestSummary() summary = TestSummary()
retired = retired or set()
by_item: dict[tuple[str, str], list[dict]] = {} by_item: dict[tuple[str, str], list[dict]] = {}
for r in records: for r in records:
by_item.setdefault((str(r.get("kind")), str(r.get("item_text"))), []).append(r) by_item.setdefault((str(r.get("kind")), str(r.get("item_text"))), []).append(r)
@@ -1026,8 +1168,13 @@ def compute_test_summary(check_items: list[CheckItem], records: list[dict]) -> T
seen.add(sig) seen.add(sig)
all_items.append(sig) all_items.append(sig)
declared = {(ci.kind, ci.text) for ci in check_items}
for kind, text in all_items: for kind, text in all_items:
status = resolve_item_status(kind, by_item.get((kind, text), [])) status = resolve_item_status(kind, by_item.get((kind, text), []))
if (kind, text) in retired and (kind, text) not in declared:
summary.retired += 1
summary.retired_items.append({"item_text": text, "kind": kind, "last_status": status})
continue
if status == "passed": if status == "passed":
summary.passed += 1 summary.passed += 1
elif status == "failed": elif status == "failed":
@@ -1052,6 +1199,7 @@ def compute_run_outcome(
records: list[dict], records: list[dict],
*, *,
verify_blocked: bool, verify_blocked: bool,
retired: set[tuple[str, str]] | None = None,
) -> RunOutcome: ) -> RunOutcome:
"""Assembles the machine-readable run outcome. """Assembles the machine-readable run outcome.
@@ -1060,7 +1208,7 @@ def compute_run_outcome(
completion. Assert failures never change ``task_status``: a finished task completion. Assert failures never change ``task_status``: a finished task
with failed assertions is ``completed`` with ``tests.failed > 0``. with failed assertions is ``completed`` with ``tests.failed > 0``.
""" """
tests = compute_test_summary(list(snapshot.all_check_items), records) tests = compute_test_summary(list(snapshot.all_check_items), records, retired=retired)
if verify_blocked: if verify_blocked:
task_status: Literal["completed", "partial", "blocked"] = "blocked" task_status: Literal["completed", "partial", "blocked"] = "blocked"
elif snapshot.all_top_level_done: elif snapshot.all_top_level_done:
+59 -5
View File
@@ -16,6 +16,7 @@ import asyncio
import base64 import base64
import functools import functools
import json import json
from collections import Counter
from typing import Annotated, Literal from typing import Annotated, Literal
from langchain_core.messages import ToolMessage from langchain_core.messages import ToolMessage
@@ -49,6 +50,8 @@ from artemis.graph.checkpoints import (
publish_checker_event, publish_checker_event,
queue_checkpoints, queue_checkpoints,
read_ledger, read_ledger,
reinstate_check_items,
retire_check_items,
revert_subgoal_status, revert_subgoal_status,
settle_all_checkpoints, settle_all_checkpoints,
spawn_pending_checkpoints, spawn_pending_checkpoints,
@@ -87,8 +90,10 @@ from artemis.utils.notes import (
) )
from artemis.utils.plan_grammar import ( from artemis.utils.plan_grammar import (
milestones_changed, milestones_changed,
missing_check_items,
new_top_level_completions, new_top_level_completions,
parse_plan, parse_plan,
render_check_line,
restore_missing_check_items, restore_missing_check_items,
subgoal_hash, subgoal_hash,
unintended_milestone_edits, unintended_milestone_edits,
@@ -432,7 +437,13 @@ async def exit_settlement_node(state: State, ctx: ArtemisContext):
# END path: assemble the machine-readable run outcome. # END path: assemble the machine-readable run outcome.
records = read_ledger(base_dir) records = read_ledger(base_dir)
outcome = compute_run_outcome(snapshot, records, verify_blocked=verify_blocked) retired = getattr(ctx, "guidance_retired_checks", None)
outcome = compute_run_outcome(
snapshot,
records,
verify_blocked=verify_blocked,
retired=retired if isinstance(retired, set) else None,
)
if snapshot.all_check_items or records or verify_blocked: if snapshot.all_check_items or records or verify_blocked:
update["run_outcome"] = outcome.model_dump() update["run_outcome"] = outcome.model_dump()
# BLOCKED/partial wrap-ups carry the last findings in the metadata file. # BLOCKED/partial wrap-ups carry the last findings in the metadata file.
@@ -451,7 +462,8 @@ async def exit_settlement_node(state: State, ctx: ArtemisContext):
f"Run outcome: task_status={outcome.task_status}," f"Run outcome: task_status={outcome.task_status},"
f" tests(passed={outcome.tests.passed}, failed={outcome.tests.failed}," f" tests(passed={outcome.tests.passed}, failed={outcome.tests.failed},"
f" inconclusive={outcome.tests.inconclusive}," f" inconclusive={outcome.tests.inconclusive},"
f" unchecked={outcome.tests.unchecked})" f" unchecked={outcome.tests.unchecked},"
f" retired={outcome.tests.retired})"
) )
return update return update
@@ -634,11 +646,53 @@ async def _process_plan_write(
# deleted/rewritten check lines are merged back in a pure text operation. # deleted/rewritten check lines are merged back in a pure text operation.
# This is content-driven and independent of any switch — a resumed plan # This is content-driven and independent of any switch — a resumed plan
# with check lines stays protected even when checking is disabled. # with check lines stays protected even when checking is disabled.
merged = restore_missing_check_items(content_before, content_after) # The one exception is user guidance: it outranks the declared standards,
# so every check line that existed when an instruction arrived (recorded
# by perception in ``ctx.guidance_unprotected_checks``) may be dropped or
# reworded at any later write — the Operator usually re-plans over several
# surgical edits, turns after the guidance landed. Lines added after the
# instruction are protected again until the next one.
#
# ``missing_check_items`` diffs on the full signature (kind, when, text,
# parent hash), so rewording a milestone re-signs every check line under
# it. Such a line has merely moved: it is still declared, so it is neither
# merged back (that would duplicate it as a task-level ``@end`` orphan)
# nor retired (the ledger, the harvest gate and the run outcome all key a
# check item by ``(kind, text)``). Only a line whose ``(kind, text)`` is
# gone from the new plan counts as dropped.
before_snapshot = parse_plan(content_before)
after_snapshot = parse_plan(content_after)
missing = missing_check_items(before_snapshot, after_snapshot)
unprotected = getattr(ctx, "guidance_unprotected_checks", None)
if not isinstance(unprotected, set):
unprotected = set()
still_declared = Counter((ci.kind, ci.text) for ci in after_snapshot.check_items)
dropped: list = []
for ci in missing:
if still_declared.get((ci.kind, ci.text), 0) > 0:
still_declared[(ci.kind, ci.text)] -= 1
continue # moved under a reworded milestone / re-timed: still declared
dropped.append(ci)
waived = [ci for ci in dropped if (ci.kind, ci.text) in unprotected]
protected = [ci for ci in dropped if (ci.kind, ci.text) not in unprotected]
if waived:
logger.info(
"Plan write dropped declared check lines because of user guidance"
f" (the change stands): {[render_check_line(ci).strip() for ci in waived]}"
)
# Records the retirement (harvest gate + run outcome) and withdraws any
# standing finding headline those lines had already produced.
retire_check_items(ctx, [(ci.kind, ci.text) for ci in waived])
# A retired line the Operator declares again is live again: the harvest
# gate and the run outcome must agree with the plan on disk.
reinstate_check_items(ctx, [(ci.kind, ci.text) for ci in after_snapshot.check_items])
if protected:
merged = restore_missing_check_items(content_before, content_after, items=protected)
if merged is not None: if merged is not None:
logger.info( logger.info(
"Plan write removed or rewrote declared check lines; merging them" "Plan write removed or rewrote declared check lines; merging them"
" back (check standards are protected deterministically)." " back (check standards are protected deterministically):"
f" {[render_check_line(ci).strip() for ci in protected]}"
) )
content_after = merged content_after = merged
try: try:
@@ -653,7 +707,7 @@ async def _process_plan_write(
# never influences the parses below. # never influences the parses below.
sync_finding_lines(ctx) sync_finding_lines(ctx)
before = parse_plan(content_before) before = before_snapshot
after = parse_plan(content_after) after = parse_plan(content_after)
# Ratchet baseline: judge the current plan against the last *validated* # Ratchet baseline: judge the current plan against the last *validated*
+31
View File
@@ -92,6 +92,29 @@ def _check_injected_instruction_file(base_dir: str) -> dict | None:
return None return None
def _unprotect_current_check_lines(ctx: ArtemisContext) -> None:
"""Record the plan's current check lines as editable under user guidance."""
from artemis.utils.notes import get_note_file_path
from artemis.utils.plan_grammar import parse_plan
unprotected = getattr(ctx, "guidance_unprotected_checks", None)
if not isinstance(unprotected, set):
unprotected = set()
try:
plan_path = get_note_file_path(ctx.data_engine.base_dir, "task_plan")
text = plan_path.read_text(encoding="utf-8") if plan_path.exists() else ""
except Exception as e:
logger.warning(f"Could not read task_plan for the guidance check-line waiver: {e}")
text = ""
added = {(ci.kind, ci.text) for ci in parse_plan(text).check_items}
unprotected |= added
ctx.guidance_unprotected_checks = unprotected
logger.info(
f"User guidance received: {len(added)} declared check line(s) may now be edited"
" by the Operator."
)
async def perception_node(state: State, ctx: ArtemisContext) -> dict: async def perception_node(state: State, ctx: ArtemisContext) -> dict:
"""Perception node (fast path): captures screenshot, extracts XML hierarchy, and performs OCR. """Perception node (fast path): captures screenshot, extracts XML hierarchy, and performs OCR.
@@ -114,6 +137,14 @@ async def perception_node(state: State, ctx: ArtemisContext) -> dict:
if injected_payload: if injected_payload:
injected_instruction = injected_payload.get("instruction") injected_instruction = injected_payload.get("instruction")
user_stop_requested = bool(injected_payload.get("release_loop")) user_stop_requested = bool(injected_payload.get("release_loop"))
if isinstance(injected_instruction, str) and injected_instruction.strip():
# User guidance outranks the plan's declared standards: every
# check line that exists right now loses its machine
# restoration for the rest of the run (kept on the run context,
# not the per-turn State), so the Operator can drop or reword
# it in whichever later write it gets to it. Lines added after
# this moment stay protected until the next instruction.
_unprotect_current_check_lines(ctx)
controller = UnifiedMobileController(ctx) controller = UnifiedMobileController(ctx)
+1 -1
View File
@@ -116,7 +116,7 @@ NODE_VISIBILITY: dict[str, NodeVisibility] = {
"operator_native_thinking", "operator_native_thinking",
"user_stop_requested", "user_stop_requested",
"initial_goal", "initial_goal",
# M3: the turn's injected instruction is stamped verbatim onto the # The turn's injected instruction is stamped verbatim onto the
# step record so the chunk ledger can preserve it (never-evict). # step record so the chunk ledger can preserve it (never-evict).
"injected_instruction", "injected_instruction",
}, },
+320 -294
View File
@@ -12,307 +12,137 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
"""System, environment, and device diagnostics (artemis doctor).""" """System, environment, and device diagnostics (artemis doctor).
The command renders the same readiness probes the web console's device wizard
and the ``mobile_diagnose`` MCP tool run (:mod:`artemis.core.diagnostics`), so
the terminal, the browser and the IDE agree on one verdict. Only two rows are
CLI-specific and not engine probes: Node.js/npm and the Showcase UI build.
"""
from __future__ import annotations
import asyncio
from dataclasses import dataclass
import json
import platform as sys_platform import platform as sys_platform
import shutil import shutil
import subprocess from typing import Any
import sys
from rich.console import Console from rich.console import Console
from rich.markup import escape
from rich.panel import Panel from rich.panel import Panel
from rich.table import Table from rich.table import Table
import typer import typer
from artemis.config import (
parse_llm_config,
settings,
)
from artemis.config.paths import ROOT_DIR from artemis.config.paths import ROOT_DIR
from artemis.core.diagnostics.adb_keys import heal_adb_keys, inspect_adb_keys from artemis.core.diagnostics.engine import readiness_engine
from artemis.core.diagnostics.readiness import (
Verdict,
adb_keys_corrupted,
base_verdict,
collect_readiness,
sort_by_fix_order,
)
from artemis.core.diagnostics.schema import ProbeResult, ProbeStatus
from artemis.platform import OSType, platform from artemis.platform import OSType, platform
from artemis.toolchain import toolchain from artemis.runtime.device_lock import DeviceExecutionLock
from artemis.utils.ocr_api import is_ocr_configured
def doctor_command( @dataclass(frozen=True)
fix: bool = typer.Option( class ExtraRow:
False, """A CLI-only diagnostic row that is not backed by an engine probe."""
"--fix",
"-f", key: str
help="Automatically repair fixable environment issues (such as corrupted ADB authentication keys).", title: str
status: str # "pass" | "missing"
status_markup: str
summary: str
detail: str
@dataclass(frozen=True)
class FixOutcome:
"""Result of one ``--fix`` repair step."""
fix: str
success: bool
message: str
# ---------------------------------------------------------------------------
# Collection
# ---------------------------------------------------------------------------
async def _collect() -> list[ProbeResult]:
report, host = await collect_readiness()
return sort_by_fix_order([*report.probes, host])
async def _apply_fixes(results: list[ProbeResult]) -> list[FixOutcome]:
"""Repair corrupted ADB keys and sweep stale device locks left by dead runners."""
outcomes: list[FixOutcome] = []
if adb_keys_corrupted(results):
result = await readiness_engine.heal_adb_keys()
outcomes.append(
FixOutcome(
fix="heal_adb_keys",
success=bool(result.get("success")),
message=str(result.get("message") or "")
or ("Corrupted RSA keys regenerated." if result.get("success") else "Heal failed."),
)
)
else:
outcomes.append(
FixOutcome(
fix="heal_adb_keys",
success=True,
message="ADB authentication keys are healthy; nothing to repair.",
)
)
removed = await asyncio.to_thread(DeviceExecutionLock.cleanup_stale_locks)
outcomes.append(
FixOutcome(
fix="cleanup_stale_locks",
success=True,
message=(
f"Removed {removed} stale device lock(s)/queue ticket(s)."
if removed
else "No stale device locks or queue tickets found."
), ),
) -> None:
"""Run diagnostics to inspect system dependencies, device connectivity, and configuration."""
console = Console()
console.print()
os_name = sys_platform.system()
arch_name = sys_platform.machine()
console.print(
Panel(
f"[bold cyan]☕ Artemis System & Environment Doctor[/bold cyan]\n"
f"[dim]Platform: {os_name} ({arch_name}) | PAL: {platform.os_type.value}[/dim]",
expand=False,
) )
) )
return outcomes
table = Table(show_header=True, header_style="bold magenta")
table.add_column("Component", style="dim", width=22)
table.add_column("Status", width=12)
table.add_column("Details & Recommendations")
all_passed = True async def _diagnose(fix: bool) -> tuple[list[ProbeResult], list[FixOutcome]]:
results = await _collect()
# 1. Python Version fixes: list[FixOutcome] = []
py_ver = f"{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}"
if sys.version_info >= (3, 12):
table.add_row(
"Python Runtime",
"[bold green]✔ OK[/bold green]",
f"Python {py_ver} ({arch_name}, 64-bit)",
)
else:
all_passed = False
table.add_row(
"Python Runtime",
"[bold red]✖ Warning[/bold red]",
f"Python {py_ver} detected. Python >= 3.12 is strongly recommended.",
)
# 2. ADB Installation & Server
adb_path = toolchain.resolve("adb")
if adb_path:
try:
res = subprocess.run(
[adb_path, "version"],
capture_output=True,
text=True,
timeout=5,
encoding="utf-8",
errors="replace",
)
adb_info = res.stdout.splitlines()[0] if res.stdout else "Installed"
table.add_row(
"Android ADB", "[bold green]✔ OK[/bold green]", f"{adb_path} ({adb_info})"
)
except Exception:
table.add_row("Android ADB", "[bold green]✔ OK[/bold green]", f"{adb_path}")
else:
all_passed = False
hint = (
"Run: winget install Google.PlatformTools"
if platform.os_type == OSType.WINDOWS
else "Run: brew install android-platform-tools"
if platform.os_type == OSType.MACOS
else "Run: sudo apt-get install -y adb"
)
table.add_row(
"Android ADB",
"[bold red]✖ Missing[/bold red]",
f"adb not found in PATH or Android SDK. {hint}",
)
# 2b. ADB Authentication RSA Keys Check
key_status = inspect_adb_keys()
if key_status.is_corrupted:
if fix: if fix:
heal_result = heal_adb_keys(adb_path=adb_path) fixes = await _apply_fixes(results)
if heal_result.get("success"): results = await _collect()
table.add_row( return results, fixes
"ADB Auth Keys",
"[bold green]✔ Repaired[/bold green]",
"Corrupted RSA keys detected and automatically regenerated.",
)
else:
all_passed = False
table.add_row(
"ADB Auth Keys",
"[bold red]✖ Error[/bold red]",
f"Failed to auto-heal keys: {heal_result.get('message')}",
)
else:
all_passed = False
table.add_row(
"ADB Auth Keys",
"[bold red]✖ Corrupted[/bold red]",
f"{key_status.error_reason} Run [bold cyan]artemis doctor --fix[/bold cyan] to auto-repair.",
)
elif key_status.exists:
table.add_row(
"ADB Auth Keys",
"[bold green]✔ OK[/bold green]",
f"{key_status.key_path} ({key_status.key_size_bytes} bytes RSA)",
)
else:
table.add_row(
"ADB Auth Keys",
"[bold cyan]○ Ready[/bold cyan]",
"Keys will be generated automatically on first device connection.",
)
# 3. Connected Android Devices
devices_info: list[tuple[str, str, str]] = []
if adb_path:
try:
res = subprocess.run(
[adb_path, "devices", "-l"],
capture_output=True,
text=True,
timeout=5,
encoding="utf-8",
errors="replace",
)
lines = [line_str.strip() for line_str in res.stdout.splitlines() if line_str.strip()]
for line in lines[1:]: # skip 'List of devices attached'
parts = line.split()
if len(parts) >= 2:
serial = parts[0]
state = parts[1]
extra = " ".join(parts[2:])
devices_info.append((serial, state, extra))
except (OSError, subprocess.SubprocessError):
# adb probe failed or timed out; report as "no devices detected".
pass
if devices_info: # ---------------------------------------------------------------------------
active_devices = [d for d in devices_info if d[1] == "device"] # CLI-only extra rows (not engine probes)
if active_devices: # ---------------------------------------------------------------------------
summary = "\n".join(
[
f"• [bold cyan]{d[0]}[/bold cyan] ({d[2] or 'Authorized'})"
for d in active_devices
]
)
table.add_row("Android Devices", "[bold green]✔ Connected[/bold green]", summary)
else:
all_passed = False
table.add_row(
"Android Devices",
"[bold yellow]⚠ Unauthorized[/bold yellow]",
"Device detected but unauthorized. Check phone screen for USB Debugging prompt.",
)
else:
table.add_row(
"Android Devices",
"[bold yellow]⚠ None Found[/bold yellow]",
"No Android device or emulator detected. Connect phone via USB or start emulator.",
)
# 4. LLM & Vision OCR Key Configuration
gemini_key = settings.get_api_key("google")
ocr_key = settings.get_api_key("ocr")
openai_key = settings.get_api_key("openai")
claude_key = settings.get_api_key("anthropic")
openrouter_key = settings.get_api_key("openrouter")
configured_keys = [] def _npm_row() -> ExtraRow:
if gemini_key:
val = gemini_key.get_secret_value()
masked = f"{val[:6]}...{val[-4:]}" if len(val) > 10 else "***"
configured_keys.append(f"Gemini LLM ({masked})")
if openai_key:
configured_keys.append("OpenAI")
if claude_key:
configured_keys.append("Anthropic")
if openrouter_key:
configured_keys.append("OpenRouter")
if configured_keys:
table.add_row(
"LLM Providers", "[bold green]✔ Configured[/bold green]", ", ".join(configured_keys)
)
else:
all_passed = False
table.add_row(
"LLM Providers",
"[bold red]✖ Missing[/bold red]",
"No LLM key found in .env. Run [bold cyan]artemis init[/bold cyan] to configure.",
)
# 5. Vision OCR Key Check (Optional)
if is_ocr_configured() and ocr_key:
val = ocr_key.get_secret_value()
masked = f"{val[:6]}...{val[-4:]}" if len(val) > 10 else "***"
table.add_row(
"Vision OCR",
"[bold green]✔ Configured[/bold green]",
f"Google Cloud Vision API ({masked})",
)
else:
table.add_row(
"Vision OCR",
"[bold cyan]○ Optional[/bold cyan]",
"OCR_API_KEY is not set. Perception operates in pure XML layout mode.",
)
# 6. Configuration File Health
try:
cfg = parse_llm_config()
planner_model = cfg.planner.model
table.add_row(
"Configuration",
"[bold green]✔ Valid[/bold green]",
f"config/artemis.jsonc parsed (Planner: {planner_model})",
)
except Exception as e:
all_passed = False
table.add_row(
"Configuration",
"[bold red]✖ Error[/bold red]",
f"Failed to parse config/artemis.jsonc: {e}",
)
# 7. Video Recording Tools (FFmpeg & scrcpy)
ffmpeg_path = toolchain.resolve("ffmpeg")
scrcpy_path = toolchain.resolve("scrcpy")
if ffmpeg_path:
table.add_row(
"FFmpeg Tools", "[bold green]✔ Installed[/bold green]", f"{ffmpeg_path} (Video ready)"
)
else:
hint = (
"winget install Gyan.FFmpeg"
if platform.os_type == OSType.WINDOWS
else "brew install ffmpeg"
if platform.os_type == OSType.MACOS
else "sudo apt-get install -y ffmpeg"
)
table.add_row(
"FFmpeg Tools",
"[dim]⚪ Optional[/dim]",
f"Not found. ({hint})",
)
if scrcpy_path:
table.add_row(
"Screen Recorder",
"[bold green]✔ Installed[/bold green]",
f"{scrcpy_path} (Live stream ready)",
)
else:
hint = (
"winget install Genymobile.scrcpy"
if platform.os_type == OSType.WINDOWS
else "brew install scrcpy"
if platform.os_type == OSType.MACOS
else "sudo apt-get install -y scrcpy"
)
table.add_row(
"Screen Recorder",
"[dim]⚪ Optional[/dim]",
f"scrcpy not found. ({hint})",
)
# 8. Node.js & npm (for UI compilation)
npm_path = shutil.which("npm") npm_path = shutil.which("npm")
if npm_path: if npm_path:
table.add_row( return ExtraRow(
"Node.js / npm", key="nodejs_npm",
"[bold green]✔ Installed[/bold green]", title="Node.js / npm",
f"{npm_path} (Ready to build Showcase UI)", status="pass",
status_markup="[bold green]✔ Installed[/bold green]",
summary="Installed",
detail=f"{npm_path} (Ready to build Showcase UI)",
) )
else:
hint = ( hint = (
"winget install OpenJS.NodeJS.LTS" "winget install OpenJS.NodeJS.LTS"
if platform.os_type == OSType.WINDOWS if platform.os_type == OSType.WINDOWS
@@ -320,13 +150,17 @@ def doctor_command(
if platform.os_type == OSType.MACOS if platform.os_type == OSType.MACOS
else "sudo apt-get install -y nodejs npm" else "sudo apt-get install -y nodejs npm"
) )
table.add_row( return ExtraRow(
"Node.js / npm", key="nodejs_npm",
"[dim]⚪ Optional[/dim]", title="Node.js / npm",
f"Not found in PATH. ({hint})", status="missing",
status_markup="[dim]⚪ Optional[/dim]",
summary="Not Found",
detail=f"Not found in PATH. ({hint})",
) )
# 9. Showcase UI Build Status
def _showcase_row() -> ExtraRow:
from artemis.resources import get_bundled_showcase_dist from artemis.resources import get_bundled_showcase_dist
base_dist = ROOT_DIR / "apps" / "showcase_ui" / "dist" base_dist = ROOT_DIR / "apps" / "showcase_ui" / "dist"
@@ -342,22 +176,98 @@ def doctor_command(
if bundled_showcase is not None: if bundled_showcase is not None:
found_showcase = bundled_showcase / "index.html" found_showcase = bundled_showcase / "index.html"
if found_showcase: if found_showcase:
table.add_row( return ExtraRow(
"Showcase UI", key="showcase_ui",
"[bold green]✔ Compiled[/bold green]", title="Showcase UI",
f"{found_showcase.parent} (Angular ready)", status="pass",
status_markup="[bold green]✔ Compiled[/bold green]",
summary="Compiled",
detail=f"{found_showcase.parent} (Angular ready)",
) )
else: return ExtraRow(
table.add_row( key="showcase_ui",
"Showcase UI", title="Showcase UI",
"[bold yellow]○ Not Compiled[/bold yellow]", status="missing",
"Run ./start.sh or artemis ui to auto-compile.", status_markup="[bold yellow]○ Not Compiled[/bold yellow]",
summary="Not Compiled",
detail="Run ./start.sh or artemis ui to auto-compile.",
) )
console.print(table)
# ---------------------------------------------------------------------------
# Rendering
# ---------------------------------------------------------------------------
def _status_markup(result: ProbeResult) -> str:
status = result.status
if status is ProbeStatus.PASS:
return "[bold green]✔ OK[/bold green]"
if status is ProbeStatus.SKIPPED:
return "[dim]○ Skipped[/dim]"
summary = escape(result.summary)
if status is ProbeStatus.WARN:
# A warning keeps its summary whether or not it blocks (the host
# probe's warnings degrade the verdict without blocking it).
return f"[bold yellow]⚠ {summary}[/bold yellow]"
if not result.is_blocker:
return "[dim]⚪ Optional[/dim]"
return f"[bold red]✖ {summary}[/bold red]"
def _action_lines(result: ProbeResult) -> list[str]:
"""One plain-text line per remediation; ``&&`` chains split for PowerShell 5.1."""
lines: list[str] = []
for action in result.actions:
payload = action.payload.strip()
if not payload:
continue
if action.action_type == "command":
for part in payload.split("&&"):
if part.strip():
lines.append(f"Run: {part.strip()}")
else:
lines.append(payload)
return lines
def _details_markup(result: ProbeResult) -> str:
# Only summary/description/actions are ever rendered. Probe metadata can
# carry raw credentials (the console prefills its settings form from it)
# and must never reach a terminal or a log.
parts = [escape(result.description)]
if result.status is not ProbeStatus.PASS:
for line in _action_lines(result):
if line.startswith("Run: "):
parts.append(f"[bold cyan]Run:[/bold cyan] {escape(line[len('Run: ') :])}")
else:
parts.append(escape(line))
return "\n".join(parts)
def _render_table(results: list[ProbeResult], extras: list[ExtraRow]) -> Table:
table = Table(show_header=True, header_style="bold magenta")
table.add_column("Component", style="dim", min_width=22, max_width=30)
table.add_column("Status", min_width=12, max_width=28)
table.add_column("Details & Recommendations", overflow="fold")
for result in results:
table.add_row(escape(result.title), _status_markup(result), _details_markup(result))
for extra in extras:
table.add_row(extra.title, extra.status_markup, escape(extra.detail))
return table
def _render_fixes(console: Console, fixes: list[FixOutcome]) -> None:
lines = []
for outcome in fixes:
mark = "[bold green]✔[/bold green]" if outcome.success else "[bold red]✖[/bold red]"
lines.append(f"{mark} {escape(outcome.fix)}: {escape(outcome.message)}")
console.print(Panel("\n".join(lines), title="Repairs (--fix)", expand=False))
console.print() console.print()
if all_passed:
def _render_footer(console: Console, verdict: Verdict, results: list[ProbeResult]) -> None:
if verdict == "ready":
console.print( console.print(
Panel( Panel(
"[bold green]🎉 All system checks passed![/bold green]\n\n" "[bold green]🎉 All system checks passed![/bold green]\n\n"
@@ -367,12 +277,128 @@ def doctor_command(
expand=False, expand=False,
) )
) )
else: return
if verdict == "degraded":
console.print( console.print(
Panel( Panel(
"[bold yellow]💡 Tip:[/bold yellow] Run [bold cyan]artemis init[/bold cyan] to quickly set up your API key and device.", "[bold green]Required checks passed.[/bold green] Optional components are "
title="Action Required", "missing; tasks can run, but some features (video replay, OCR) are off.\n\n"
"Run your first automation task now:\n"
' [bold cyan]artemis run "Open Settings and check Battery level"[/bold cyan]',
title="Status: Ready (optional gaps)",
expand=False, expand=False,
) )
) )
return
failing = [r.title for r in results if r.is_blocker and r.status is not ProbeStatus.PASS]
tips = [
"[bold yellow]💡 Blocking issues:[/bold yellow] " + escape(", ".join(failing)),
"",
"• API keys & device setup: run [bold cyan]artemis init[/bold cyan].",
"• Follow the [bold cyan]Run:[/bold cyan] lines above, then re-run "
"[bold cyan]artemis doctor[/bold cyan].",
"• Using an IDE (Claude Code, Cursor, ...)? Ask it to call "
"[bold cyan]mobile_diagnose[/bold cyan] - it runs these same checks and can "
"apply safe fixes from the assistant.",
]
if adb_keys_corrupted(results):
tips.append(
"• Corrupted ADB keys detected: run [bold cyan]artemis doctor --fix[/bold cyan]."
)
console.print(Panel("\n".join(tips), title="Action Required", expand=False))
def _json_document(
results: list[ProbeResult],
extras: list[ExtraRow],
fixes: list[FixOutcome],
verdict: Verdict,
) -> dict[str, Any]:
# Metadata is intentionally omitted: it may hold raw API keys.
checks = [
{
"id": r.id,
"title": r.title,
"status": r.status.value,
"required": r.is_blocker,
"summary": r.summary,
"detail": r.description,
"fix": []
if r.status is ProbeStatus.PASS
else [
{"type": a.action_type, "label": a.label, "payload": a.payload} for a in r.actions
],
}
for r in results
]
return {
"verdict": verdict,
"checks": checks,
"extras": {
e.key: {
"title": e.title,
"status": e.status,
"summary": e.summary,
"detail": e.detail,
}
for e in extras
},
"fixes": [{"fix": f.fix, "success": f.success, "message": f.message} for f in fixes],
}
# ---------------------------------------------------------------------------
# Command
# ---------------------------------------------------------------------------
def doctor_command(
fix: bool = typer.Option(
False,
"--fix",
"-f",
help=(
"Automatically repair fixable environment issues (corrupted ADB authentication "
"keys, stale device locks left by crashed runners)."
),
),
json_output: bool = typer.Option(
False,
"--json",
help="Print the diagnosis as a JSON document instead of a table.",
),
) -> None:
"""Run diagnostics to inspect system dependencies, device connectivity, and configuration."""
results, fixes = asyncio.run(_diagnose(fix))
extras = [_npm_row(), _showcase_row()]
verdict = base_verdict(results)
if json_output:
typer.echo(
json.dumps(
_json_document(results, extras, fixes, verdict), indent=2, ensure_ascii=False
)
)
else:
console = Console()
console.print() console.print()
os_name = sys_platform.system()
arch_name = sys_platform.machine()
console.print(
Panel(
f"[bold cyan]☕ Artemis System & Environment Doctor[/bold cyan]\n"
f"[dim]Platform: {os_name} ({arch_name}) | PAL: {platform.os_type.value}[/dim]",
expand=False,
)
)
if fixes:
_render_fixes(console, fixes)
console.print(_render_table(results, extras))
console.print()
_render_footer(console, verdict, results)
console.print()
if verdict == "blocked":
raise typer.Exit(code=1)
+1
View File
@@ -41,6 +41,7 @@ ARTEMIS_MCP_TOOLS = (
"mobile_manage_task", "mobile_manage_task",
"mobile_get_device_state", "mobile_get_device_state",
"mobile_inspect_trace", "mobile_inspect_trace",
"mobile_diagnose",
) )
+182 -35
View File
@@ -15,8 +15,16 @@
"""MCP-backed action executor for the FlashRunner. """MCP-backed action executor for the FlashRunner.
Device actions use the in-process MCP session. Argument normalization, Device actions use the in-process MCP session. Argument normalization,
coordinate descriptions, smart swipes, post-action observations, state updates, element-index resolution, coordinate descriptions, smart swipes, post-action
tracing, and helper tools are handled here on the agent side. observations, state updates, tracing, and helper tools are handled here on the
agent side.
Flash binds the same agent dialect as the Pro Operator: a ``click``,
``long_press`` or ``input_text`` target is either an element index into the
indexed ``--- Visible UI Elements ---`` list (resolved here to the element's
center, its observed text/bounds/id recorded) or a normalized ``[x, y]`` pair
carrying the model's own ``target_description``. The wire only ever sees
coordinates.
Device action status comes from ``ActionResult.ok``; helper tools report failures Device action status comes from ``ActionResult.ok``; helper tools report failures
through ``ToolFailure`` or an explicit status. through ``ToolFailure`` or an explicit status.
@@ -66,6 +74,9 @@ AGENT_TOOL_NAMES: frozenset[str] = (
frozenset({"read_note", "list_notes", "ask_explorer", "video_analyzer"}) | HISTORY_TOOL_NAMES frozenset({"read_note", "list_notes", "ask_explorer", "video_analyzer"}) | HISTORY_TOOL_NAMES
) )
#: Actions whose ``target`` is a single point: an element index or an [x, y] pair.
_POINT_TARGET_ACTIONS: frozenset[str] = frozenset({"click", "long_press", "input_text"})
class _ArgError(ValueError): class _ArgError(ValueError):
"""Argument-translation failure whose message is already fully formatted.""" """Argument-translation failure whose message is already fully formatted."""
@@ -108,15 +119,28 @@ class McpActionExecutor:
args: dict[str, Any], args: dict[str, Any],
tool_call_id: str, tool_call_id: str,
state: Any, state: Any,
*,
index_elements: list[dict[str, Any]] | None = None,
) -> ToolExecutionResult: ) -> ToolExecutionResult:
"""Executes the named tool and wraps the outcome for the calling agent.""" """Executes the named tool and wraps the outcome for the calling agent.
``index_elements`` is the element list an index target (``click(3)``)
resolves against; ``None`` means ``state.indexed_elements`` at call
time. A multi-action turn passes the list from the observation the
model decided on, so its later indices are not resolved against the
list refreshed by its earlier actions (``state.indexed_elements`` is
still refreshed after every action: it is what the next observation
shows).
"""
raw_name = name.split(":")[-1] if ":" in name else name raw_name = name.split(":")[-1] if ":" in name else name
if raw_name in AGENT_TOOL_NAMES: if raw_name in AGENT_TOOL_NAMES:
return await self._execute_agent_tool(raw_name, name, args, tool_call_id, state) return await self._execute_agent_tool(raw_name, name, args, tool_call_id, state)
if raw_name in self.action_tool_names: if raw_name in self.action_tool_names:
return await self._execute_device_action(raw_name, name, args, tool_call_id, state) return await self._execute_device_action(
raw_name, name, args, tool_call_id, state, index_elements=index_elements
)
return ToolExecutionResult( return ToolExecutionResult(
tool_call_id=tool_call_id, tool_call_id=tool_call_id,
@@ -134,6 +158,8 @@ class McpActionExecutor:
args: dict[str, Any], args: dict[str, Any],
tool_call_id: str, tool_call_id: str,
state: Any, state: Any,
*,
index_elements: list[dict[str, Any]] | None = None,
) -> ToolExecutionResult: ) -> ToolExecutionResult:
span = TraceSpan(name=raw_name, trace_type="action", ctx=self.ctx) span = TraceSpan(name=raw_name, trace_type="action", ctx=self.ctx)
span.payload = {"args": args} span.payload = {"args": args}
@@ -144,12 +170,18 @@ class McpActionExecutor:
shot_path: str | None = None shot_path: str | None = None
xml_list: str | None = None xml_list: str | None = None
target_semantics: dict[str, Any] = {} target_semantics: dict[str, Any] = {}
target_coordinates: list[int] | None = None
with span: with span:
try: try:
session = await self._session_or_start() session = await self._session_or_start()
wire_name, wire_args, finalize, target_semantics = self._translate( wire_name, wire_args, finalize, target_semantics = self._translate(
raw_name, args, state raw_name, args, state, index_elements=index_elements
) )
if raw_name in _POINT_TARGET_ACTIONS:
# The normalized point the action was sent to: the recorded
# coordinates of an index target (the model named an index,
# not a point).
target_coordinates = wire_args.get("target")
extension = raw_name not in (REQUIRED_ACTIONS | OPTIONAL_ACTIONS) extension = raw_name not in (REQUIRED_ACTIONS | OPTIONAL_ACTIONS)
if extension: if extension:
@@ -214,6 +246,7 @@ class McpActionExecutor:
"code": res.code.value, "code": res.code.value,
"normalized_coordinates": res.normalized_coordinates, "normalized_coordinates": res.normalized_coordinates,
"target_semantics": target_semantics, "target_semantics": target_semantics,
"target_coordinates": target_coordinates,
}, },
) )
@@ -232,24 +265,35 @@ class McpActionExecutor:
# --- Argument translation -------------------------------------------------------- # --- Argument translation --------------------------------------------------------
def _translate( def _translate(
self, raw_name: str, args: dict[str, Any], state: Any self,
raw_name: str,
args: dict[str, Any],
state: Any,
*,
index_elements: list[dict[str, Any]] | None = None,
) -> tuple[str, dict[str, Any], Any, dict[str, Any]]: ) -> tuple[str, dict[str, Any], Any, dict[str, Any]]:
"""Maps agent-facing args to wire args; returns (name, args, finalize, recorded). """Maps agent-facing args to wire args; returns (name, args, finalize, recorded).
``finalize(res)`` adds client-side context to swipe result messages. ``finalize(res)`` adds client-side context to swipe result messages.
``recorded`` contains validated target descriptions in the Pro Operator's ``recorded`` holds the target semantics in the Pro Operator's record
record format. Focused input and directional swipes have no coordinate format: the observed element fields of an index target, or the validated
target, so they do not record a description. ``target_description`` of a coordinate target. Focused input and
directional swipes have no target, so they record nothing.
``index_elements`` overrides the list index targets resolve against
(see :meth:`execute`).
""" """
if raw_name == "click": if raw_name == "click":
target = self._require_pair( target, recorded = self._resolve_target(
args.get("target") or args.get("coordinates") or [], "click" args.get("target") or args.get("coordinates"),
"click",
args,
state,
index_elements=index_elements,
) )
recorded = self._require_description(args, "click")
return ( return (
"click", "click",
{ {
"target": list(target), "target": target,
"times": args.get("times", 1), "times": args.get("times", 1),
"delay_ms": args.get("delay_ms", 100), "delay_ms": args.get("delay_ms", 100),
}, },
@@ -258,14 +302,17 @@ class McpActionExecutor:
) )
if raw_name == "long_press": if raw_name == "long_press":
target = self._require_pair( target, recorded = self._resolve_target(
args.get("target") or args.get("coordinates") or [], "long press" args.get("target") or args.get("coordinates"),
"long press",
args,
state,
index_elements=index_elements,
) )
recorded = self._require_description(args, "long press")
return ( return (
"long_press", "long_press",
{ {
"target": list(target), "target": target,
"duration_ms": args.get("duration_ms", args.get("duration", 1000)), "duration_ms": args.get("duration_ms", args.get("duration", 1000)),
}, },
None, None,
@@ -277,8 +324,9 @@ class McpActionExecutor:
target = None target = None
recorded = {} recorded = {}
if raw_target: if raw_target:
target = list(self._require_pair(raw_target, "input text")) target, recorded = self._resolve_target(
recorded = self._require_description(args, "input text") raw_target, "input text", args, state, index_elements=index_elements
)
return ( return (
"input_text", "input_text",
{ {
@@ -304,7 +352,7 @@ class McpActionExecutor:
) )
if raw_name == "swipe": if raw_name == "swipe":
return self._translate_swipe(args, state) return self._translate_swipe(args, state, index_elements=index_elements)
if raw_name == "press_key": if raw_name == "press_key":
return "press_key", {"key": args.get("key", "BACK")}, None, {} return "press_key", {"key": args.get("key", "BACK")}, None, {}
@@ -338,17 +386,101 @@ class McpActionExecutor:
# Backend extension: pass the arguments straight through. # Backend extension: pass the arguments straight through.
return raw_name, dict(args), None, {} return raw_name, dict(args), None, {}
@staticmethod def _screen_size(self) -> tuple[int, int]:
def _require_pair(raw: Any, label: str) -> tuple[int, int]: """The device resolution the indexed element list was built against."""
device = getattr(self.ctx, "device", None)
width = getattr(device, "device_width", None) if device else None
height = getattr(device, "device_height", None) if device else None
return (
width if isinstance(width, int) and width > 0 else 1080,
height if isinstance(height, int) and height > 0 else 2400,
)
def _resolve_target(
self,
raw: Any,
label: str,
args: dict[str, Any],
state: Any,
*,
index_elements: list[dict[str, Any]] | None = None,
) -> tuple[list[int], dict[str, Any]]:
"""Resolves a point target to ``([x, y] normalized, recorded semantics)``.
An element index carries observed semantics (text, bounds, id, class
straight from the indexed list); a coordinate pair carries only the
model's own ``target_description``. The two never mix (Pro parity).
"""
target = normalize_coordinate_target(raw) target = normalize_coordinate_target(raw)
if not isinstance(target, (list, tuple)) or len(target) != 2: if isinstance(target, bool):
prefix = { target = None
"click": "Error during click", if isinstance(target, (int, float)):
"long press": "Error during long press", return self._resolve_index(int(target), label, state, index_elements=index_elements)
"input text": "Error during input text", if isinstance(target, (list, tuple)) and len(target) == 2:
}[label] recorded = self._require_description(args, label)
raise _ArgError(f"{prefix}: Invalid target format: {raw}") return [int(target[0]), int(target[1])], recorded
return int(target[0]), int(target[1]) raise _ArgError(
f"Error during {label}: Invalid target format: {raw!r}. Use an element"
" index from the Visible UI Elements list (e.g. 3) or normalized [x, y]"
" coordinates with a target_description."
)
def _resolve_index(
self,
index: int,
label: str,
state: Any,
*,
index_elements: list[dict[str, Any]] | None = None,
) -> tuple[list[int], dict[str, Any]]:
"""Maps an element index onto the element's center (normalized 0-1000).
The recorded fields are what the Operator records for an index target,
with bounds normalized like every other Flash coordinate. The index is
resolved against ``index_elements`` when given, else against the
state's current list.
"""
elements = self._index_elements(state, index_elements)
if not 1 <= index <= len(elements):
hint = (
" The list is empty on this screen."
if not elements
else f" Active index range is 1 to {len(elements)}."
)
raise _ArgError(
f"Error during {label}: Invalid target index {index}.{hint} Use an index"
" shown in the Visible UI Elements list, or normalized [x, y]"
" coordinates with a target_description (ask_explorer can locate an"
" element that is visible but not listed)."
)
element = elements[index - 1]
width, height = self._screen_size()
def _norm_x(v: Any) -> int:
return int(max(0, min(1000, round(float(v) * 1000 / max(1, width)))))
def _norm_y(v: Any) -> int:
return int(max(0, min(1000, round(float(v) * 1000 / max(1, height)))))
center = element.get("center")
if not (isinstance(center, (list, tuple)) and len(center) == 2):
raise _ArgError(f"Error during {label}: element [{index}] has no usable center.")
bounds = element.get("bounds")
norm_bounds = None
if isinstance(bounds, (list, tuple)) and len(bounds) == 4:
norm_bounds = [
_norm_x(bounds[0]),
_norm_y(bounds[1]),
_norm_x(bounds[2]),
_norm_y(bounds[3]),
]
recorded = {
"target_text": element.get("text"),
"target_bounds": norm_bounds,
"target_resource_id": element.get("resource_id"),
"target_class": element.get("class"),
}
return [_norm_x(center[0]), _norm_y(center[1])], recorded
@staticmethod @staticmethod
def _require_description(args: dict[str, Any], label: str) -> dict[str, Any]: def _require_description(args: dict[str, Any], label: str) -> dict[str, Any]:
@@ -414,11 +546,23 @@ class McpActionExecutor:
resolved.append([nx, ny]) resolved.append([nx, ny])
return resolved return resolved
@staticmethod
def _index_elements(
state: Any, index_elements: list[dict[str, Any]] | None
) -> list[dict[str, Any]]:
"""The element list index targets resolve against (``None`` -> state)."""
if index_elements is not None:
return index_elements
return getattr(state, "indexed_elements", None) or []
def _translate_swipe( def _translate_swipe(
self, args: dict[str, Any], state: Any self,
args: dict[str, Any],
state: Any,
*,
index_elements: list[dict[str, Any]] | None = None,
) -> tuple[str, dict[str, Any], Any, dict[str, Any]]: ) -> tuple[str, dict[str, Any], Any, dict[str, Any]]:
width = getattr(self.ctx.device, "device_width", 1080) if self.ctx.device else 1080 width, height = self._screen_size()
height = getattr(self.ctx.device, "device_height", 2400) if self.ctx.device else 2400
default_duration = args.get("duration", 400) default_duration = args.get("duration", 400)
kind, target, final_duration = parse_swipe_parameters( kind, target, final_duration = parse_swipe_parameters(
dict(args), default_duration=default_duration dict(args), default_duration=default_duration
@@ -429,7 +573,7 @@ class McpActionExecutor:
x1, y1, x2, y2, smart_dur = compute_smart_swipe_coordinates( x1, y1, x2, y2, smart_dur = compute_smart_swipe_coordinates(
direction=target, direction=target,
target=args.get("target"), target=args.get("target"),
indexed_elements=getattr(state, "indexed_elements", None) if state else None, indexed_elements=self._index_elements(state, index_elements) if state else None,
ui_hierarchy=getattr(state, "latest_ui_hierarchy", None) if state else None, ui_hierarchy=getattr(state, "latest_ui_hierarchy", None) if state else None,
width=width, width=width,
height=height, height=height,
@@ -608,6 +752,9 @@ class McpActionExecutor:
self.ctx, state, query, context_feedback or "", agent_name=self.agent_name self.ctx, state, query, context_feedback or "", agent_name=self.agent_name
) )
registered = register_candidates(self.ctx, state, outcome) registered = register_candidates(self.ctx, state, outcome)
return render_text(query, outcome, registered), not outcome.error # Candidates join ``state.indexed_elements``, and this executor's
# ``click`` resolves an index against that list (Pro parity), so the
# answer teaches the index syntax next to the coordinates.
return render_text(query, outcome, registered, index_targets=True), not outcome.error
except Exception as e: except Exception as e:
return f"Error executing ask_explorer: {e}", False return f"Error executing ask_explorer: {e}", False
+139 -339
View File
@@ -21,49 +21,46 @@ the declarations drifted apart (``click.target`` took an element index in one pl
and only coordinates in another). This module is now the single place all three are and only coordinates in another). This module is now the single place all three are
*defined*; the historical sites import their surface from here, generated. *defined*; the historical sites import their surface from here, generated.
An action has up to three dialects, and the differences between them are **deliberate An action has one agent-facing dialect and one wire dialect:
and declared**, not accidental:
* ``operator`` -- the LangChain shell tools the Operator binds. The Operator receives * ``operator`` -- the agent dialect. Both profiles receive the same indexed
an indexed "Visible UI Elements" list and its prompt teaches index-first targeting, "Visible UI Elements" list, so target parameters accept ``int`` element indices
so target parameters accept ``int`` element indices as well as normalized as well as normalized coordinates. The Pro Operator binds it as a LangChain shell
coordinates. The shells are declaration-only ("Action Recorded"): the Operator's (declaration-only, "Action Recorded": the Operator's translate step lowers the
translate step lowers them into structured decisions. call into structured decisions); the FlashRunner binds the very same description
* ``declaration`` -- the JSON ``ToolDeclaration`` bound by the Validator's and parameters projected onto a JSON ``ToolDeclaration`` (:func:`tool_declaration`),
FlashRunner. These agents act on coordinates directly (the and ``McpActionExecutor`` resolves an index against the current element list.
element list they see carries coordinates), so target parameters are coordinate One definition, two bindings: the two profiles can no longer drift apart.
pairs and the wording teaches the act-then-observe loop ("The screen after X will * ``declaration`` -- a JSON-only agent dialect, used solely for actions that have
be returned automatically"). no operator shell (``click_sequence`` is declared to Flash alone).
* ``wire`` -- what the action MCP server serves, mirroring the actuator protocol * ``wire`` -- what the action MCP server serves, mirroring the actuator protocol
one-to-one: normalized coordinates, millisecond durations, no addressing sugar. one-to-one: normalized coordinates, millisecond durations, no addressing sugar.
The client-side executor (``McpActionExecutor``) lowers the The client-side executor (``McpActionExecutor``) lowers the agent dialect onto it.
agent dialects onto it.
A coordinate target names nothing by itself, so both agent dialects pair it with A coordinate target names nothing by itself, so the agent dialect pairs it with
``target_description`` (``target_descriptions`` for a ``click_sequence``): the ``target_description`` (``target_descriptions`` for a ``click_sequence``): the
model's own statement of what it is aiming at. It is required whenever the target model's own statement of what it is aiming at. It is required whenever the target
is a coordinate (always, in the declaration dialect), recorded verbatim on the is a coordinate, recorded verbatim on the action, and never sent to the wire. An
action, and never sent to the wire. An element index needs none -- its text, element index needs none -- its text, bounds and id are read from the indexed
bounds and id are read from the indexed list -- and the recorded fields stay list -- and the recorded fields stay separate (``target_text``/``target_bounds``/...
separate (``target_text``/``target_bounds``/... for observed elements, for observed elements, ``target_description`` for described coordinates) so no
``target_description`` for described coordinates) so no reader can mistake the reader can mistake the model's belief for an observation.
model's belief for an observation.
Each ``ActionSpec.param_bridge`` maps operator parameter names onto declaration ``tests/unit/mcp/test_action_specs.py`` pins the generated schema of every model
parameter names, so every rename (``duration`` -> ``duration_ms``, ``gesture`` -> surface and checks that the Flash declaration of each shared action is exactly the
``action``) is written down exactly once. ``tests/unit/mcp/test_action_specs.py`` operator shell's schema, so a divergence between what one profile sees and what the
re-derives the bridge from the generated schemas, so an undeclared divergence -- other sees -- the historical definition of drift -- fails CI instead of shipping.
today's definition of drift -- fails CI instead of shipping.
The classification of actions (required/optional/internal) stays in The classification of actions (required/optional/internal) stays in
``action_manifest``; this module holds their schemas and teaching text. ``action_manifest``; this module holds their schemas and teaching text.
""" """
from collections.abc import Awaitable, Callable from collections.abc import Awaitable, Callable
from dataclasses import dataclass, field from dataclasses import dataclass
from functools import cache from functools import cache
import inspect import inspect
from typing import Annotated, Any, Literal import types
from typing import Annotated, Any, Literal, Union, get_args, get_origin
from langchain_core.tools import StructuredTool from langchain_core.tools import StructuredTool
from mcp.types import CallToolResult from mcp.types import CallToolResult
@@ -105,7 +102,11 @@ class ParamSpec:
@dataclass(frozen=True) @dataclass(frozen=True)
class OperatorDialect: class OperatorDialect:
"""The Operator's LangChain shell: index-capable targets, declaration-only body.""" """The agent dialect: index-capable targets, one wording for both profiles.
Bound by the Pro Operator as a declaration-only LangChain shell and by the
FlashRunner as the JSON projection of the same parameters.
"""
description: str description: str
params: tuple[ParamSpec, ...] params: tuple[ParamSpec, ...]
@@ -113,7 +114,7 @@ class OperatorDialect:
@dataclass(frozen=True) @dataclass(frozen=True)
class DeclarationDialect: class DeclarationDialect:
"""The Validator/Flash JSON declaration, wording preserved.""" """A JSON-only agent dialect, for actions without an operator shell."""
description: str description: str
parameters: dict[str, Any] parameters: dict[str, Any]
@@ -139,57 +140,43 @@ class ActionSpec:
Attributes: Attributes:
name: The canonical tool name every dialect shares. name: The canonical tool name every dialect shares.
operator: Operator shell dialect, or ``None`` when the Operator does not bind operator: The agent dialect (Operator shell and, projected, the Flash
this action. declaration), or ``None`` when neither profile binds this action as a
declaration: Validator/Flash dialect, or ``None`` when no JSON declaration shell.
exists (an action can be wire-only, reachable through executors but never declaration: A JSON-only agent dialect for actions without an operator
declared to a model directly). shell, or ``None`` (an action can be wire-only, reachable through
executors but never declared to a model directly).
wire: Action-server dialect; ``None`` only for purely virtual actions. wire: Action-server dialect; ``None`` only for purely virtual actions.
param_bridge: operator param name -> declaration param name, for every differences: Human-readable record of every deliberate divergence between
operator param, when both dialects exist. Identical names map to the agent dialect and the wire (semantics, not just spelling). Empty
themselves; renames are the deliberate dialect differences. means the dialects align modulo the documented dialect philosophy above.
differences: Human-readable record of every deliberate cross-dialect
divergence (semantics, not just spelling). Empty means the dialects
align modulo the documented dialect philosophy above.
""" """
name: str name: str
operator: OperatorDialect | None = None operator: OperatorDialect | None = None
declaration: DeclarationDialect | None = None declaration: DeclarationDialect | None = None
wire: WireDialect | None = None wire: WireDialect | None = None
param_bridge: dict[str, str] = field(default_factory=dict)
differences: str = "" differences: str = ""
# --- Shared type aliases ------------------------------------------------------------- # --- Shared type aliases -------------------------------------------------------------
#: Operator-dialect target: element index into the indexed UI list, or a normalized #: Agent-dialect target: element index into the indexed UI list, or a normalized
#: [x, y] pair. Only the Operator sees this union; see the module docstring. #: [x, y] pair.
IndexOrCoords = int | list[int] IndexOrCoords = int | list[int]
Direction = Literal["up", "down", "left", "right"] Direction = Literal["up", "down", "left", "right"]
_COORD_PAIR_JSON = { #: Wording for the description that a coordinate target must carry. An element
"type": "array", #: index already names its element (its text, id and bounds are recorded from the
"items": {"type": "integer"}, #: indexed list); a bare coordinate names nothing, so the model states what it is
} #: aiming at and that statement is recorded as ``target_description``.
_TARGET_DESCRIPTION = (
#: Operator-dialect wording for the description that a coordinate target must carry.
#: An element index already names its element (its text, id and bounds are recorded
#: from the indexed list); a bare coordinate names nothing, so the model states what
#: it is aiming at and that statement is recorded as ``target_description``.
_OPERATOR_TARGET_DESCRIPTION = (
"What the target is, in a few words (e.g. 'play button', 'search input'," "What the target is, in a few words (e.g. 'play button', 'search input',"
" 'video body'). REQUIRED when target is a coordinate pair; ignored for an" " 'video body'). REQUIRED when target is a coordinate pair; ignored for an"
" element index." " element index."
) )
#: Declaration-dialect wording (coordinates-only dialect: always required).
_DECLARATION_TARGET_DESCRIPTION = (
"What the target is, in a few words (e.g. 'play button', 'search input',"
" 'video body'). Required: a coordinate names nothing by itself."
)
# --- Wire bindings ------------------------------------------------------------------- # --- Wire bindings -------------------------------------------------------------------
# Each binding reproduces the exact conversion of the historical hand-written FastMCP # Each binding reproduces the exact conversion of the historical hand-written FastMCP
@@ -274,7 +261,7 @@ _SPECS: tuple[ActionSpec, ...] = (
ParamSpec( ParamSpec(
"target_description", "target_description",
str | None, str | None,
_OPERATOR_TARGET_DESCRIPTION, _TARGET_DESCRIPTION,
required=False, required=False,
default=None, default=None,
), ),
@@ -296,39 +283,6 @@ _SPECS: tuple[ActionSpec, ...] = (
), ),
), ),
), ),
declaration=DeclarationDialect(
description=(
"[ACTION] Tap/click on the target coordinate on the screen. The screen"
" after click will be returned automatically. For buttons, checkboxes,"
" tabs, icons, items in a list, tap ON the element. For text fields /"
" search bars / input boxes, tap INSIDE the input box to focus it."
),
parameters={
"type": "object",
"properties": {
"target": {
"type": "array",
"items": {"type": "integer"},
"description": "Normalized coordinates [x, y] in 0-1000 scale.",
},
"target_description": {
"type": "string",
"description": _DECLARATION_TARGET_DESCRIPTION,
},
"times": {
"type": "integer",
"description": (
"Number of taps to perform (default 1). Set to 2 for double click."
),
},
"delay_ms": {
"type": "integer",
"description": "Delay between taps in milliseconds (default 100).",
},
},
"required": ["target", "target_description"],
},
),
wire=WireDialect( wire=WireDialect(
description="Tap at a 0-1000 normalized [x, y] coordinate.", description="Tap at a 0-1000 normalized [x, y] coordinate.",
params=( params=(
@@ -338,17 +292,11 @@ _SPECS: tuple[ActionSpec, ...] = (
), ),
bind=_wire_click, bind=_wire_click,
), ),
param_bridge={
"target": "target",
"target_description": "target_description",
"times": "times",
"delay_ms": "delay_ms",
},
differences=( differences=(
"target: operator accepts an element index or a coordinate pair; the" "target: the agent dialect accepts an element index or a coordinate pair"
" declaration and wire dialects take coordinates only." " (an index is resolved client-side against the indexed element list); the"
" target_description: recorded, never sent to the wire; the operator" " wire takes coordinates only. target_description: recorded, never sent"
" requires it only for coordinate targets, the declaration always." " to the wire; required only for coordinate targets."
), ),
), ),
ActionSpec( ActionSpec(
@@ -407,6 +355,9 @@ _SPECS: tuple[ActionSpec, ...] = (
" burst instead); sequence entries are normalized coordinate pairs in both" " burst instead); sequence entries are normalized coordinate pairs in both"
" dialects (an element index is refused, never resolved client-side)." " dialects (an element index is refused, never resolved client-side)."
" target_descriptions is recorded per entry and never sent to the wire." " target_descriptions is recorded per entry and never sent to the wire."
" Unlike click, a burst is not resolved against the element list: its"
" whole point is to outrun a transient UI, and the list was captured"
" before the trigger tap."
), ),
), ),
ActionSpec( ActionSpec(
@@ -427,7 +378,7 @@ _SPECS: tuple[ActionSpec, ...] = (
ParamSpec( ParamSpec(
"target_description", "target_description",
str | None, str | None,
_OPERATOR_TARGET_DESCRIPTION, _TARGET_DESCRIPTION,
required=False, required=False,
default=None, default=None,
), ),
@@ -440,31 +391,6 @@ _SPECS: tuple[ActionSpec, ...] = (
), ),
), ),
), ),
declaration=DeclarationDialect(
description=(
"[ACTION] Long press on a target coordinate on the screen. The screen"
" after long pressing will be returned automatically."
),
parameters={
"type": "object",
"properties": {
"target": {
"type": "array",
"items": {"type": "integer"},
"description": "Normalized coordinates [x, y] in 0-1000 scale.",
},
"target_description": {
"type": "string",
"description": _DECLARATION_TARGET_DESCRIPTION,
},
"duration_ms": {
"type": "integer",
"description": "Duration of press in milliseconds (default 1000ms).",
},
},
"required": ["target", "target_description"],
},
),
wire=WireDialect( wire=WireDialect(
description="Long-press at a 0-1000 normalized [x, y] coordinate.", description="Long-press at a 0-1000 normalized [x, y] coordinate.",
params=( params=(
@@ -473,19 +399,13 @@ _SPECS: tuple[ActionSpec, ...] = (
), ),
bind=_wire_long_press, bind=_wire_long_press,
), ),
param_bridge={
"target": "target",
"target_description": "target_description",
"duration": "duration_ms",
},
differences=( differences=(
"target: operator accepts an element index or a coordinate pair." "target: the agent dialect accepts an element index or a coordinate pair;"
" target_description: recorded, never sent to the wire; the operator" " the wire takes coordinates only. target_description: recorded, never"
" requires it only for coordinate targets, the declaration always." " sent to the wire; required only for coordinate targets."
" duration: the operator spells the duration parameter `duration` (its" " duration: the agent dialect spells the duration parameter `duration`"
" structured decisions and recorded traces carry that key); the" " (structured decisions and recorded traces carry that key); the wire"
" declaration and wire dialects spell it `duration_ms`. Executors accept" " spells it `duration_ms`. Executors accept both spellings."
" both spellings."
), ),
), ),
ActionSpec( ActionSpec(
@@ -512,7 +432,7 @@ _SPECS: tuple[ActionSpec, ...] = (
ParamSpec( ParamSpec(
"target_description", "target_description",
str | None, str | None,
_OPERATOR_TARGET_DESCRIPTION, _TARGET_DESCRIPTION,
required=False, required=False,
default=None, default=None,
), ),
@@ -527,39 +447,6 @@ _SPECS: tuple[ActionSpec, ...] = (
), ),
), ),
), ),
declaration=DeclarationDialect(
description=(
"[ACTION] Type text into an input field on the screen. The screen after"
" typing will be returned automatically. Automatically taps inside the"
" input box at target [x, y] to focus before typing."
),
parameters={
"type": "object",
"properties": {
"text": {
"type": "string",
"description": "Text to type into the focused input field.",
},
"target": {
"type": "array",
"items": {"type": "integer"},
"description": "Coordinates [x, y] of the input box in 0-1000 scale.",
},
"target_description": {
"type": "string",
"description": _DECLARATION_TARGET_DESCRIPTION,
},
"clear_exist": {
"type": "boolean",
"description": (
"Whether to clear existing text in the input box before typing"
" (default True)."
),
},
},
"required": ["text", "target", "target_description"],
},
),
wire=WireDialect( wire=WireDialect(
description=("Type text, optionally focusing a 0-1000 normalized [x, y] target first."), description=("Type text, optionally focusing a 0-1000 normalized [x, y] target first."),
params=( params=(
@@ -569,18 +456,11 @@ _SPECS: tuple[ActionSpec, ...] = (
), ),
bind=_wire_input_text, bind=_wire_input_text,
), ),
param_bridge={
"target": "target",
"target_description": "target_description",
"text": "text",
"clear_exist": "clear_exist",
},
differences=( differences=(
"target: operator accepts an element index or a coordinate pair, and" "target: the agent dialect accepts an element index or a coordinate pair,"
" requires a target; the wire dialect allows omitting the target to type" " and requires a target; the wire allows omitting the target to type into"
" into the already-focused field. target_description: recorded, never" " the already-focused field. target_description: recorded, never sent to"
" sent to the wire; the operator requires it only for coordinate targets," " the wire; required only for coordinate targets."
" the declaration always."
), ),
), ),
ActionSpec( ActionSpec(
@@ -662,70 +542,6 @@ _SPECS: tuple[ActionSpec, ...] = (
), ),
), ),
), ),
declaration=DeclarationDialect(
description=(
"[ACTION] Perform a swipe, drag, or slider-adjustment gesture on the screen. The screen and UI hierarchy after swipe will be returned automatically.\n\n"
"• Directional Scrolling ('direction' or 'action'): Recommended for general browsing and standard page scrolling in most scenarios. Automatically computes safe swipe vectors and adaptive duration, retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings. Supports scoping to a sub-container via 'target'. If it fails on certain custom layouts, fall back to specifying exact coordinates ('start' and 'end') directly.\n"
"• Precise Coordinate Gestures ('start', 'end' or coordinates list): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars), drag-and-drop / list reordering, or as a reliable fallback when directional scrolling fails on specific containers. Always drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), swipe fully to the extreme boundary."
),
parameters={
"type": "object",
"properties": {
"direction": {
"type": "string",
"enum": ["up", "down", "left", "right"],
"description": (
"Direction for scrolling and swiping: 'up' (drags bottom-to-top, scrolling down to reveal content below),"
" 'down' (drags top-to-bottom, scrolling up to reveal content above),"
" 'left' (drags right-to-left, scrolling right),"
" 'right' (drags left-to-right, scrolling left)."
),
},
"start": {
"type": "array",
"items": {"type": "integer"},
"description": (
"Start normalized coordinates [start_x, start_y] in 0-1000 scale."
),
},
"end": {
"type": "array",
"items": {"type": "integer"},
"description": (
"End normalized coordinates [end_x, end_y] in 0-1000 scale."
),
},
"target": {
"description": (
"Optional target element index (e.g. 2) or container bounds"
" [left, top, right, bottom] to scope the directional swipe within."
),
},
"action": {
"description": (
"Backward-compatible swipe gesture: smart direction string ('up', 'down', 'left', 'right')"
" OR precise custom coordinates [start_x, start_y, end_x, end_y] in 0-1000 scale."
),
},
"target_description": {
"type": "string",
"description": (
"What is being dragged, in a few words (e.g. 'brightness slider"
" knob'). Required for coordinate gestures ('start'/'end' or a"
" coordinates list); ignored for directional scrolling."
),
},
"duration": {
"type": "integer",
"description": (
"Optional swipe/drag duration in milliseconds (default 800). For drag-and-drop,"
" list reordering, or sliding/adjusting sliders (e.g., volume, brightness, SeekBars),"
" set duration >= 1000 (e.g. 1500). If omitted for directional swipe, duration is computed automatically."
),
},
},
},
),
wire=WireDialect( wire=WireDialect(
description="Swipe between two 0-1000 normalized [x, y] points.", description="Swipe between two 0-1000 normalized [x, y] points.",
params=( params=(
@@ -735,23 +551,12 @@ _SPECS: tuple[ActionSpec, ...] = (
), ),
bind=_wire_swipe, bind=_wire_swipe,
), ),
param_bridge={
"direction": "direction",
"start": "start",
"end": "end",
"target": "target",
"gesture": "action",
"target_description": "target_description",
"duration": "duration",
},
differences=( differences=(
"target_description: recorded, never sent to the wire; required by both" "target_description: recorded, never sent to the wire; required for"
" agent dialects for coordinate gestures only." " coordinate gestures only. gesture: the legacy combined"
" gesture/action: the legacy combined direction-or-coordinates parameter is" " direction-or-coordinates parameter (executors also accept the older"
" spelled `gesture` in the operator dialect and `action` in the" " `action` spelling). Smart directional swipes exist only in the agent"
" declaration dialect (`action` collides with the operator's structured" " dialect; the wire takes a resolved start/end pair, and direction"
" decision verb field). Smart directional swipes exist only in the agent"
" dialects; the wire takes a resolved start/end pair, and direction"
" resolution happens client-side against the live UI tree." " resolution happens client-side against the live UI tree."
), ),
), ),
@@ -770,24 +575,6 @@ _SPECS: tuple[ActionSpec, ...] = (
), ),
), ),
), ),
declaration=DeclarationDialect(
description=(
"[ACTION] Press a physical or virtual system button. The screen after"
" pressing will be returned automatically."
),
parameters={
"type": "object",
"properties": {
"key": {
"type": "string",
"description": (
"Standard Android system button name (ENTER, BACK, HOME, APP_SWITCH)."
),
}
},
"required": ["key"],
},
),
wire=WireDialect( wire=WireDialect(
description=( description=(
"Press a device key (home, back, enter, delete, tab, search, menu, app_switch)." "Press a device key (home, back, enter, delete, tab, search, menu, app_switch)."
@@ -795,13 +582,11 @@ _SPECS: tuple[ActionSpec, ...] = (
params=(ParamSpec("key", str),), params=(ParamSpec("key", str),),
bind=_wire_press_key, bind=_wire_press_key,
), ),
param_bridge={"key": "key"},
differences=( differences=(
"key vocabulary: the operator dialect is a closed uppercase enum (its" "key vocabulary: the agent dialect is a closed uppercase enum (the"
" translate step emits Android KEYCODE_* names); the declaration dialect" " Operator's translate step emits Android KEYCODE_* names); the wire"
" is an open string taught the same uppercase names; the wire takes the" " takes the actuator's bare lowercase key words. `to_canonical_call` and"
" actuator's bare lowercase key words. `to_canonical_call` and the" " the executors normalize between them."
" executors normalize between them."
), ),
), ),
ActionSpec( ActionSpec(
@@ -817,26 +602,6 @@ _SPECS: tuple[ActionSpec, ...] = (
), ),
), ),
), ),
declaration=DeclarationDialect(
description=(
"[ACTION] Launch or force stop a specified application. The screen"
" after launch/stop will be returned automatically."
),
parameters={
"type": "object",
"properties": {
"action": {
"type": "string",
"description": "The action type ('launch' or 'stop').",
},
"app_name": {
"type": "string",
"description": "Display name or package name of the application.",
},
},
"required": ["action", "app_name"],
},
),
wire=WireDialect( wire=WireDialect(
description="Launch or stop an app by human-readable name or package.", description="Launch or stop an app by human-readable name or package.",
params=( params=(
@@ -845,7 +610,6 @@ _SPECS: tuple[ActionSpec, ...] = (
), ),
bind=_wire_manage_app, bind=_wire_manage_app,
), ),
param_bridge={"action": "action", "app_name": "app_name"},
), ),
ActionSpec( ActionSpec(
name="wait_for_delay", name="wait_for_delay",
@@ -867,32 +631,11 @@ _SPECS: tuple[ActionSpec, ...] = (
), ),
), ),
), ),
declaration=DeclarationDialect(
description=(
"[ACTION] Pause execution and wait for a specified duration in milliseconds."
" The screen after pause will be returned automatically."
),
parameters={
"type": "object",
"properties": {
"time_in_ms": {
"type": "integer",
"description": (
"The exact duration to pause in milliseconds (e.g., 2000 for 2s,"
" 60000 for 1 min, 180000 for 3 mins, 300000 for 5 mins)."
" Convert any required waiting duration into milliseconds."
),
}
},
"required": ["time_in_ms"],
},
),
wire=WireDialect( wire=WireDialect(
description="Wait for a fixed number of milliseconds.", description="Wait for a fixed number of milliseconds.",
params=(ParamSpec("time_in_ms", int),), params=(ParamSpec("time_in_ms", int),),
bind=_wire_wait_for_delay, bind=_wire_wait_for_delay,
), ),
param_bridge={"time_in_ms": "time_in_ms"},
), ),
ActionSpec( ActionSpec(
name="wait_for_text", name="wait_for_text",
@@ -1007,16 +750,73 @@ def operator_shell_tool(name: str) -> StructuredTool:
) )
_SCALAR_JSON: dict[Any, dict[str, str]] = {
bool: {"type": "boolean"},
int: {"type": "integer"},
float: {"type": "number"},
str: {"type": "string"},
}
def _json_schema(annotation: Any) -> dict[str, Any]:
"""Projects one Python-typed parameter annotation onto JSON schema.
``None`` members of a union only make the parameter optional (its
``required`` flag says so); they never appear in the projected schema, so
the declaration carries no ``null`` alternatives or ``default`` noise.
"""
origin = get_origin(annotation)
if origin in (Union, types.UnionType):
members = [a for a in get_args(annotation) if a is not type(None)]
schemas = [_json_schema(m) for m in members]
return schemas[0] if len(schemas) == 1 else {"anyOf": schemas}
if origin is Literal:
return {"type": "string", "enum": list(get_args(annotation))}
if origin is list:
(item,) = get_args(annotation) or (str,)
return {"type": "array", "items": _json_schema(item)}
try:
return dict(_SCALAR_JSON[annotation])
except KeyError:
raise TypeError(f"No JSON projection for parameter annotation {annotation!r}.") from None
def _projected_parameters(dialect: OperatorDialect) -> dict[str, Any]:
"""The operator shell's parameters as a JSON-schema ``parameters`` object."""
properties: dict[str, Any] = {}
required: list[str] = []
for p in dialect.params:
schema = _json_schema(p.annotation)
if p.description:
schema["description"] = p.description
properties[p.name] = schema
if p.required:
required.append(p.name)
return {"type": "object", "properties": properties, "required": required}
@cache
def tool_declaration(name: str) -> ToolDeclaration: def tool_declaration(name: str) -> ToolDeclaration:
"""Builds the Validator/Flash ``ToolDeclaration`` for one action.""" """Builds the Flash ``ToolDeclaration`` for one action.
dialect = ACTION_SPECS[name].declaration
if dialect is None: A shared action is the operator shell projected onto JSON (same description,
raise ValueError(f"Action '{name}' has no declaration dialect.") same parameters, same targeting semantics); a JSON-only action uses its own
``DeclarationDialect``.
"""
spec = ACTION_SPECS[name]
if spec.declaration is not None:
return ToolDeclaration( return ToolDeclaration(
name=name, name=name,
description=dialect.description, description=spec.declaration.description,
parameters=dialect.parameters, parameters=spec.declaration.parameters,
) )
if spec.operator is not None:
return ToolDeclaration(
name=name,
description=spec.operator.description,
parameters=_projected_parameters(spec.operator),
)
raise ValueError(f"Action '{name}' has no agent dialect to declare.")
def wire_dialects() -> tuple[ActionSpec, ...]: def wire_dialects() -> tuple[ActionSpec, ...]:
+315 -133
View File
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
"""History compression into segment chunks, eras, and a session snapshot. """History compression into segment chunks, eras, and recall-only periods.
Each chunk keeps three bands: synopsis and effects, interval summaries, and a Each chunk keeps three bands: synopsis and effects, interval summaries, and a
per-step action ledger. Capsule generation reads the operator's turn transcripts per-step action ledger. Capsule generation reads the operator's turn transcripts
@@ -20,12 +20,18 @@ alongside the recorded step facts. Milestone changes, segment size, and token
limits determine chunk boundaries. limits determine chunk boundaries.
Original messages remain in context until their capsule is ready. Failed capsule Original messages remain in context until their capsule is ready. Failed capsule
jobs retain their source text for retry. Only the hard token limit forces pending jobs retain their source text for retry. At the hard token limit, ready chunks
chunks into the session snapshot. Older chunks fold into eras and then period swap as ordinary L2 blocks first; only when that cannot bring the context back
summaries; full step records remain available through the history tools. under the line does everything closed force-swap (pending chunks included) and
the whole frozen region fold into a recall-only era — L3: the period paragraph
plus the per-step minimal index (step number, session offset, action phrase).
The fold is monotonic: later L2 swaps append new chunk blocks after the folded
eras and never restore the folded chunks. Older chunks also fold into eras and
then recall-only periods by count overflow; full step records remain available
through the history tools.
Checkpoint annotations persist independently of capsule generation. Frozen Checkpoint annotations persist independently of capsule generation. Frozen
context is rebuilt only when a capsule or snapshot replaces the source turns. context is rebuilt only when a compression event replaces or folds turns.
""" """
import asyncio import asyncio
@@ -61,6 +67,29 @@ _NOTE_TOOLS = ("save_note", "update_note", "append_note")
# Reuse one tool trace per chunk across running, success, and failed states. # Reuse one tool trace per chunk across running, success, and failed states.
COMPRESSION_TRACE_NAME = "compress_history" COMPRESSION_TRACE_NAME = "compress_history"
#: Plain-language phases carried in the trace payload (``args.phase``) so the
#: timeline can say what the compression is *doing* independently of the
#: trace ``status`` (running/success/failed), which other code keys on.
#: summarizing — the segment closed and its capsule is generating (running)
#: ready — the capsule is on hand but the start gate holds the swap (running)
#: applied — the compressed block replaced the raw turns (success)
#: failed — capsule generation gave up; the full record is kept (failed)
COMPRESSION_PHASE_SUMMARIZING = "summarizing"
COMPRESSION_PHASE_READY = "ready"
COMPRESSION_PHASE_APPLIED = "applied"
COMPRESSION_PHASE_FAILED = "failed"
COMPRESSION_PHASES = (
COMPRESSION_PHASE_SUMMARIZING,
COMPRESSION_PHASE_READY,
COMPRESSION_PHASE_APPLIED,
COMPRESSION_PHASE_FAILED,
)
_STATUS_TO_PHASE = {
"running": COMPRESSION_PHASE_SUMMARIZING,
"success": COMPRESSION_PHASE_APPLIED,
"failed": COMPRESSION_PHASE_FAILED,
}
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Band ③ — mechanical per-step action ledger # Band ③ — mechanical per-step action ledger
@@ -161,7 +190,7 @@ def extract_note_writes(step: dict) -> list[dict[str, str]]:
def validate_interval_coverage(intervals: Any, start_step: int, end_step: int) -> bool: def validate_interval_coverage(intervals: Any, start_step: int, end_step: int) -> bool:
"""Band ② hard constraint: the interval union covers [start, end] seamlessly.""" """Band ② hard constraint: the interval union covers [start, end] without gaps or overlaps."""
if not isinstance(intervals, list) or not intervals: if not isinstance(intervals, list) or not intervals:
return False return False
expected = start_step expected = start_step
@@ -289,7 +318,7 @@ class StepCapsuleLens(StepLens):
" keys doing/did/effect/entry_state/exit_state/verified_facts/" " keys doing/did/effect/entry_state/exit_state/verified_facts/"
"unresolved/failed_paths/important_entities/intervals. Never use" "unresolved/failed_paths/important_entities/intervals. Never use"
" verdict words (successfully, completed, failed, ...); intervals" " verdict words (successfully, completed, failed, ...); intervals"
" must cover the step range seamlessly. Return only JSON." " must cover the full step range without gaps or overlaps. Return only JSON."
) )
def _get_llm(self): def _get_llm(self):
@@ -396,8 +425,9 @@ class StepCapsuleLens(StepLens):
def _cap_transcript(cls, transcript: str) -> str: def _cap_transcript(cls, transcript: str) -> str:
"""Cap one turn transcript in place, keeping its head and its tail. """Cap one turn transcript in place, keeping its head and its tail.
The tail is kept intact because the validator result message closes The tail is kept intact because the turn's last messages (the final
every turn and outranks everything above it (fact priority); the cut action's tool result, or the failure result message that closes a
failed turn) outrank everything above them (fact priority); the cut
lands in the middle and is announced with its size. lands in the middle and is announced with its size.
""" """
cap = cls.MAX_TURN_TRANSCRIPT_CHARS cap = cls.MAX_TURN_TRANSCRIPT_CHARS
@@ -475,8 +505,9 @@ class StepCapsuleLens(StepLens):
async def _invoke(self, llm: Any, messages: list[BaseMessage], model_label: str): async def _invoke(self, llm: Any, messages: list[BaseMessage], model_label: str):
response = await asyncio.wait_for(llm.ainvoke(messages), timeout=90.0) response = await asyncio.wait_for(llm.ainvoke(messages), timeout=90.0)
# Raw-model bypass metering (gateway-wrapped models meter themselves); # Raw-model bypass metering (gateway-wrapped models meter themselves);
# capsule prompts are small and must not clobber the session's # capsule prompts are small and are kept out of the session meter's
# last_prompt_tokens (the compaction thresholds' live context base). # last_prompt_tokens (the compaction base itself is operator-owned on
# the transcript ledger and never reads the session meter).
try: try:
from artemis.services.llm import RobustChatModelWrapper from artemis.services.llm import RobustChatModelWrapper
from artemis.services.token_meter import record_llm_usage from artemis.services.token_meter import record_llm_usage
@@ -588,7 +619,8 @@ class ChunkState:
trace_id: Any = None trace_id: Any = None
trace_step_id: Any = None trace_step_id: Any = None
announced_at: float | None = None announced_at: float | None = None
# Measure source at close and summary at swap; estimate tokens as chars // 4. # Measure source at close and summary at swap; tokens are derived through
# the ledger's calibrated chars-per-token ratio (chars // 4 by default).
# Forced swaps without a capsule leave summary_chars unset. # Forced swaps without a capsule leave summary_chars unset.
source_chars: int = 0 source_chars: int = 0
summary_chars: int | None = None summary_chars: int | None = None
@@ -762,8 +794,13 @@ def render_era_period_paragraph(era: EraState) -> str:
def render_era_block(era: EraState) -> str: def render_era_block(era: EraState) -> str:
"""Era block: ① merged headers, ② degraded to titles, ③ per-chunk ledgers. """Era block: ① merged headers, ② degraded to titles, ③ per-chunk ledgers.
A recall-only era includes the step range and session offsets for video A recall-only era (L3) includes the step range and session offsets for
alignment, a synopsis, recall guidance, and preserved user instructions. video alignment, a synopsis, recall guidance, and the per-step minimal
index of every member chunk (step number, session offset, action phrase
— results only via search_history). Pending chunks contribute their
index too. User-injected instruction lines are never evicted: they sit
inside the index at their step, or are appended when a chunk carries no
index text.
""" """
if era.recall_only: if era.recall_only:
lines = [ lines = [
@@ -774,9 +811,12 @@ def render_era_block(era: EraState) -> str:
), ),
RECALL_GUIDANCE_TEMPLATE.format(start=era.start_step_number, end=era.end_step_number), RECALL_GUIDANCE_TEMPLATE.format(start=era.start_step_number, end=era.end_step_number),
] ]
# Never-evict: user-injected instruction lines survive even recall-only.
for chunk in era.chunks: for chunk in era.chunks:
lines.extend(chunk.user_lines) if chunk.minimal_index:
lines.append(chunk.minimal_index)
rendered = "\n".join(lines)
for chunk in era.chunks:
lines.extend(line for line in chunk.user_lines if line not in rendered)
return "\n".join(lines) return "\n".join(lines)
merged = merge_structured_fields([c.band1 for c in era.chunks if c.band1]) merged = merge_structured_fields([c.band1 for c in era.chunks if c.band1])
@@ -816,48 +856,6 @@ def render_era_block(era: EraState) -> str:
return "\n".join(parts) return "\n".join(parts)
def render_l3_snapshot(
eras: list[EraState],
chunks: list[ChunkState],
*,
goal: str | None,
plan_text: str | None,
) -> str:
"""L3 emergency snapshot: merged knowledge plane + minimal per-step index.
The chunk headers are mechanically set-merged (no summary-of-summary); the
per-step skeleton survives at minimal width so step-level time perception
never breaks.
"""
all_chunks = [c for era in eras for c in era.chunks] + list(chunks)
merged = merge_structured_fields([c.band1 for c in all_chunks if c.band1])
exit_state = ""
for chunk in reversed(all_chunks):
if chunk.band1:
exit_state = chunk.band1.get("exit_state") or ""
break
start = all_chunks[0].start_step_number if all_chunks else 0
end = all_chunks[-1].end_step_number if all_chunks else 0
parts = [
f"[Session snapshot | Steps {start}–{end} | hard-threshold fallback]",
f"Overall goal: {goal or '-'}",
"Plan state:",
(plan_text or "-").rstrip(),
f"Verified facts: {'; '.join(merged['verified_facts']) or '-'}",
f"Unresolved: {'; '.join(merged['unresolved']) or '-'}",
f"Failed paths: {'; '.join(merged['failed_paths']) or '-'}",
f"Important entities: {'; '.join(merged['important_entities']) or '-'}",
f"Device/app state: {exit_state or '-'}",
"",
"--- Step index (minimal width, per chunk) ---",
]
for chunk in all_chunks:
parts.append(f"[{chunk.step_range_label}]")
parts.append(chunk.minimal_index)
return "\n".join(parts)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# HistoryChunkManager — triggers, compression events, F-region rendering # HistoryChunkManager — triggers, compression events, F-region rendering
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -885,10 +883,13 @@ class HistoryChunkManager:
): ):
self._engine = engine self._engine = engine
self._ctx = ctx self._ctx = ctx
# Accepted for the callers' wiring; the frozen region no longer
# recites the goal (the prompt carries it elsewhere).
self._goal = goal self._goal = goal
cc = chunking_config cc = chunking_config
self._max_steps = int(getattr(cc, "max_steps", 12) or 12) self._max_steps = int(getattr(cc, "max_steps", 12) or 12)
self._min_steps = max(1, min(self._max_steps, int(getattr(cc, "min_steps", 3) or 3)))
self._target_source_tokens = int(getattr(cc, "target_source_tokens", 2000) or 2000) self._target_source_tokens = int(getattr(cc, "target_source_tokens", 2000) or 2000)
self._model_name = getattr(cc, "model", None) or "gemini-3.8-flash" self._model_name = getattr(cc, "model", None) or "gemini-3.8-flash"
self._max_chunks = int(getattr(cc, "max_chunks", 8) or 8) self._max_chunks = int(getattr(cc, "max_chunks", 8) or 8)
@@ -897,6 +898,8 @@ class HistoryChunkManager:
tc = transcript_config tc = transcript_config
self._budget = int(getattr(tc, "context_budget_tokens", 80000) or 80000) self._budget = int(getattr(tc, "context_budget_tokens", 80000) or 80000)
start_ratio = getattr(tc, "start_ratio", None)
self._start_ratio = float(0.35 if start_ratio is None else start_ratio)
self._soft_ratio = float(getattr(tc, "soft_ratio", 0.7) or 0.7) self._soft_ratio = float(getattr(tc, "soft_ratio", 0.7) or 0.7)
self._hard_ratio = float(getattr(tc, "hard_ratio", 0.9) or 0.9) self._hard_ratio = float(getattr(tc, "hard_ratio", 0.9) or 0.9)
self._min_active_steps = int(getattr(tc, "min_active_steps", 5) or 5) self._min_active_steps = int(getattr(tc, "min_active_steps", 5) or 5)
@@ -916,7 +919,7 @@ class HistoryChunkManager:
# whose original turns still live in the transcript until their # whose original turns still live in the transcript until their
# capsule header is ready (ready-gated swap). An entry's ``chunk`` is # capsule header is ready (ready-gated swap). An entry's ``chunk`` is
# None when the segment had no step records — it can then only leave # None when the segment had no step records — it can then only leave
# the queue through the hard-threshold emergency swap. # the queue through the hard-threshold force-swap.
self._chunks: list[ChunkState] = [] self._chunks: list[ChunkState] = []
self._eras: list[EraState] = [] self._eras: list[EraState] = []
self._awaiting: list[dict[str, Any]] = [] self._awaiting: list[dict[str, Any]] = []
@@ -1065,11 +1068,11 @@ class HistoryChunkManager:
logger.error(f"History chunk compression event failed: {e}") logger.error(f"History chunk compression event failed: {e}")
def _on_render_inner(self, ledger) -> None: def _on_render_inner(self, ledger) -> None:
base_tokens = self._context_base_tokens() base_tokens = self._context_base_tokens(ledger)
soft = base_tokens is not None and base_tokens >= self._budget * self._soft_ratio soft = base_tokens is not None and base_tokens >= self._budget * self._soft_ratio
hard = base_tokens is not None and base_tokens >= self._budget * self._hard_ratio hard = base_tokens is not None and base_tokens >= self._budget * self._hard_ratio
closed_any = self._close_new_segments(ledger, soft) closed_any = self._close_new_segments(ledger, soft, hard)
if closed_any or soft or hard: if closed_any or soft or hard:
# Failure rung of the degradation ladder: exhausted capsules are # Failure rung of the degradation ladder: exhausted capsules are
# re-dispatched (the lens retries a fallback model per attempt); # re-dispatched (the lens retries a fallback model per attempt);
@@ -1079,12 +1082,20 @@ class HistoryChunkManager:
self._harvest_capsules() self._harvest_capsules()
self._swap_ready_segments(ledger, hard=hard, base_tokens=base_tokens) self._swap_ready_segments(ledger, hard=hard, base_tokens=base_tokens)
def _close_new_segments(self, ledger, soft: bool) -> bool: def _close_new_segments(self, ledger, soft: bool, hard: bool = False) -> bool:
"""Trigger evaluation: close due segments and dispatch their capsules. """Trigger evaluation: close due segments and dispatch their capsules.
Closing NEVER freezes turns (ready-gated swap): the segment's original Closing NEVER freezes turns (ready-gated swap): the segment's original
messages stay live in the transcript; a closed segment joins the messages stay live in the transcript; a closed segment joins the
``_awaiting`` queue until its capsule header is ready. ``_awaiting`` queue until its capsule header is ready.
Minimum chunk length: the size and pressure triggers only close once
at least ``min_steps`` eligible turns have accumulated in the open
segment — turns age past the sliding-window floor one per render, so
without it a heavy session closes a one-turn chunk (one capsule call)
every render. A milestone close is exempt (a complete segment is a
unit, however short) and the hard threshold waives it (emergency:
whatever is eligible closes so the swap has material).
""" """
turns = ledger.unchunked_turns() turns = ledger.unchunked_turns()
if len(turns) <= self._min_active_steps: if len(turns) <= self._min_active_steps:
@@ -1128,17 +1139,19 @@ class HistoryChunkManager:
if tail_segment is not None: if tail_segment is not None:
tail_portion = tail_segment[1][: max(0, remaining_eligible - consumed_prefix)] tail_portion = tail_segment[1][: max(0, remaining_eligible - consumed_prefix)]
long_enough = hard or len(tail_portion) >= self._min_steps
size_event = False size_event = False
if tail_segment is not None and tail_portion: if tail_segment is not None and tail_portion and long_enough:
tail_chars = ledger.turn_text_chars(tail_portion) tail_chars = ledger.turn_text_chars(tail_portion)
size_event = ( size_event = (
len(tail_segment[1]) >= self._max_steps len(tail_segment[1]) >= self._max_steps
or (tail_chars // 4) >= self._target_source_tokens or self._chars_to_tokens(tail_chars, ledger) >= self._target_source_tokens
) )
if size_event: if size_event:
selected.append((tail_segment[0], tail_portion, "size")) selected.append((tail_segment[0], tail_portion, "size"))
elif not milestone_event and soft and tail_portion: elif not milestone_event and soft and tail_portion and long_enough:
# Soft threshold with nothing else due: close the oldest open # Soft threshold with nothing else due: close the oldest open
# segment's eligible portion (bounded by the chunk size cap). # segment's eligible portion (bounded by the chunk size cap).
selected.append((tail_segment[0], tail_portion[: self._max_steps], "pressure")) selected.append((tail_segment[0], tail_portion[: self._max_steps], "pressure"))
@@ -1176,6 +1189,10 @@ class HistoryChunkManager:
" retained until a capsule lands." " retained until a capsule lands."
) )
self._capsule_service.submit(key, payload) self._capsule_service.submit(key, payload)
# A fresh attempt: if the chunk readies again while the start
# gate still holds it, the timeline gets a fresh ``held`` note
# instead of staying on ``retrying``.
entry["held_announced"] = False
self._announce(chunk, "running", note="retrying") self._announce(chunk, "running", note="retrying")
except Exception as e: except Exception as e:
logger.error(f"Capsule re-dispatch for {key} failed: {e}") logger.error(f"Capsule re-dispatch for {key} failed: {e}")
@@ -1185,41 +1202,91 @@ class HistoryChunkManager:
Swaps consume the transcript's oldest unchunked turns, so only a Swaps consume the transcript's oldest unchunked turns, so only a
contiguous READY prefix may swap — an older pending segment keeps its contiguous READY prefix may swap — an older pending segment keeps its
(and every younger segment's) original text live. The hard threshold (and every younger segment's) original text live.
is the sole exception: everything closed force-swaps, pending chunks
included, and the frozen region renders as the L3 snapshot. Start gate: closing a segment (milestone/size triggers) only *prepares*
its capsule in the background; the original text is replaced only once
the operator's measured context has reached ``budget * start_ratio``.
Below that the ready chunks are held in the awaiting queue, so at low
occupancy the model keeps reading the full transcript while capsules
are already on hand for when pressure arrives. An unknown base (no
measured operator call yet, or a provider without usage metadata)
cannot gate and falls through to swap-on-ready.
Hard threshold ladder (L2 first, L3 last): at ``budget * hard_ratio``
the ready prefix swaps as ordinary L2 chunk blocks when the estimated
context after that swap drops back under the hard line (rung 1). Only
when no ready prefix exists, or the L2 swap would still leave the
context at or above the line, does everything closed force-swap —
pending chunks included — and the whole frozen region fold into a
recall-only era (rung 2: L3 = period paragraph + per-step minimal
index). With nothing awaiting at all, the last resort is folding the
existing frozen region the same way without consuming turns (rung 3).
The fold is a monotonic state change: later L2 swaps append new chunk
blocks after the folded eras and never restore the folded chunks;
repeated hard renders with nothing left to fold leave the frozen
blocks byte-identical.
``base_tokens`` is the prompt size before the swap, used to estimate ``base_tokens`` is the prompt size before the swap, used to estimate
the remaining context on the last swapped chunk's trace. the remaining context on the last swapped chunk's trace.
""" """
hard_line = self._budget * self._hard_ratio
if not self._awaiting: if not self._awaiting:
if hard and self._collapse_frozen_region():
ledger.freeze_turns(0, self._render_frozen_blocks())
logger.info(
"History compression: hard threshold with nothing awaiting;"
" frozen region folded into the recall-only period."
)
return return
swap: list[dict[str, Any]] = []
if hard: # An entry without a chunk (turns with no step records: nothing to
swap, self._awaiting = self._awaiting, [] # summarize, nothing to render) is consumable at once — it must not
else: # pin the queue, or no L2 swap ever happens and the hard line jumps
while self._awaiting: # straight to the fold.
chunk = self._awaiting[0]["chunk"] ready_prefix: list[dict[str, Any]] = []
if chunk is None or chunk.status != "ready": for entry in self._awaiting:
chunk = entry["chunk"]
if chunk is not None and chunk.status != "ready":
break break
swap.append(self._awaiting.pop(0)) ready_prefix.append(entry)
swap: list[dict[str, Any]] = []
fold = False
if hard:
estimate = self._estimate_context_after(base_tokens, ready_prefix, ledger)
if ready_prefix and estimate is not None and estimate < hard_line:
swap = ready_prefix # rung 1: L2 is enough
else:
swap = list(self._awaiting) # rung 2: force everything, fold
fold = True
elif base_tokens is not None and base_tokens < self._budget * self._start_ratio:
self._announce_held(base_tokens, ledger)
return
else:
swap = ready_prefix
if not swap: if not swap:
return return
self._awaiting = self._awaiting[len(swap) :]
for entry in swap: for entry in swap:
if entry["chunk"] is not None: if entry["chunk"] is not None:
self._chunks.append(entry["chunk"]) self._chunks.append(entry["chunk"])
if fold:
self._collapse_frozen_region()
else:
self._fold_eras() self._fold_eras()
blocks = self._render_frozen_blocks(hard=hard) blocks = self._render_frozen_blocks()
consumed = sum(len(e["turns"]) for e in swap) consumed = sum(len(e["turns"]) for e in swap)
ledger.freeze_turns(consumed, blocks) ledger.freeze_turns(consumed, blocks)
logger.info( logger.info(
f"History compression swap: froze {consumed} turns" f"History compression swap: froze {consumed} turns"
f" ({len(self._chunks)} chunks, {len(self._eras)} eras," f" ({len(self._chunks)} chunks, {len(self._eras)} eras,"
f" {len(self._awaiting)} still awaiting, hard={hard})." f" {len(self._awaiting)} still awaiting, hard={hard}, folded={fold})."
) )
# Pending chunks contribute only their minimal index to the L3 snapshot. # A folded chunk contributes only its minimal index (its ① fields are
# Include that size in the context estimate, without a compression ratio. # merged into the period paragraph); a pending chunk has no summary
# size to report, so its trace carries no compression ratio.
swapped = [e for e in swap if e["chunk"] is not None] swapped = [e for e in swap if e["chunk"] is not None]
source_tokens_total = 0 source_tokens_total = 0
replacement_tokens_total = 0 replacement_tokens_total = 0
@@ -1227,12 +1294,12 @@ class HistoryChunkManager:
chunk = entry["chunk"] chunk = entry["chunk"]
if chunk.status == "ready": if chunk.status == "ready":
chunk.summary_chars = len(render_chunk_block(chunk)) chunk.summary_chars = len(render_chunk_block(chunk))
replacement_chars = chunk.summary_chars
else: else:
chunk.summary_chars = None chunk.summary_chars = None
replacement_chars = len(chunk.minimal_index or "") source_tokens_total += self._chars_to_tokens(chunk.source_chars, ledger)
source_tokens_total += chunk.source_chars // 4 replacement_tokens_total += self._chars_to_tokens(
replacement_tokens_total += replacement_chars // 4 self._replacement_chars(chunk, folded=fold), ledger
)
context_tokens: int | None = None context_tokens: int | None = None
if base_tokens is not None: if base_tokens is not None:
context_tokens = max( context_tokens = max(
@@ -1242,19 +1309,26 @@ class HistoryChunkManager:
for index, entry in enumerate(swapped): for index, entry in enumerate(swapped):
chunk = entry["chunk"] chunk = entry["chunk"]
turns = len(entry["turns"]) turns = len(entry["turns"])
forced = hard and chunk.status != "ready" plural = "s" if turns != 1 else ""
if not forced: forced = fold and chunk.status != "ready"
result = f"Replaced {turns} turn{'s' if turns != 1 else ''} with the summary." if forced:
else:
result = ( result = (
"Context budget reached before the summary was ready;" "Context budget reached before the summary was ready;"
f" replaced {turns} turn{'s' if turns != 1 else ''} with a brief" f" {turns} turn{plural} folded into the period summary."
" snapshot."
) )
elif fold:
result = (
f"Replaced {turns} turn{plural} with the summary, folded into"
" the period summary (context budget reached)."
)
else:
result = f"Replaced {turns} turn{plural} with the summary."
extra: dict[str, Any] = { extra: dict[str, Any] = {
"source_tokens": chunk.source_chars // 4, "source_tokens": self._chars_to_tokens(chunk.source_chars, ledger),
"summary_tokens": ( "summary_tokens": (
chunk.summary_chars // 4 if chunk.summary_chars is not None else None self._chars_to_tokens(chunk.summary_chars, ledger)
if chunk.summary_chars is not None
else None
), ),
} }
# Show the context estimate once per swap, on its last trace. # Show the context estimate once per swap, on its last trace.
@@ -1264,20 +1338,118 @@ class HistoryChunkManager:
extra["context_estimated"] = True extra["context_estimated"] = True
self._announce(chunk, "success", result=result, forced=forced, extra=extra) self._announce(chunk, "success", result=result, forced=forced, extra=extra)
def _collapse_frozen_region(self) -> bool:
"""Fold the whole frozen region into recall-only eras (hard-threshold
L3). Every existing era becomes recall-only and the loose chunks fold
into one new recall-only era (a later fold appends another era, so
ordinals and step order stay monotonic). Returns whether anything
changed — False means the region is already fully folded, so the
caller can leave the frozen blocks untouched (byte-stable)."""
changed = False
for era in self._eras:
if not era.recall_only:
era.recall_only = True
changed = True
if self._chunks:
self._era_counter += 1
self._eras.append(
EraState(ordinal=self._era_counter, chunks=self._chunks, recall_only=True)
)
self._chunks = []
changed = True
return changed
@staticmethod
def _chars_to_tokens(chars: int, ledger: Any = None) -> int:
"""Characters → tokens through the ledger's session-calibrated ratio
(:meth:`TranscriptLedger.chars_to_tokens`); ``chars // 4`` without a
ledger. Identical to ``// 4`` at the default ratio."""
convert = getattr(ledger, "chars_to_tokens", None)
if callable(convert):
try:
return int(convert(chars))
except (TypeError, ValueError, ZeroDivisionError):
pass
return max(0, int(chars)) // 4
@staticmethod
def _replacement_chars(chunk: ChunkState, *, folded: bool = False) -> int:
"""Characters a swapped chunk contributes in place of its turns: the
full block when ready and rendered as a chunk block, only the minimal
index when still pending or folded into a recall-only era."""
if chunk.status == "ready" and not folded:
return len(render_chunk_block(chunk))
return len(chunk.minimal_index or "")
def _estimate_context_after(
self, base_tokens: int | None, entries: list[dict[str, Any]], ledger: Any = None
) -> int | None:
"""Estimated prompt size after swapping ``entries`` as L2 blocks."""
if base_tokens is None:
return None
delta = 0
for entry in entries:
chunk = entry["chunk"]
if chunk is None:
continue
delta += self._chars_to_tokens(
self._replacement_chars(chunk), ledger
) - self._chars_to_tokens(chunk.source_chars, ledger)
return max(0, int(base_tokens) + delta)
def _announce_held(self, base_tokens: int, ledger: Any = None) -> None:
"""Once per chunk: its capsule is ready but the start gate holds the
original text live. The timeline line stays ``running`` with a
``held`` note so the UI can say the summary is ready and waiting."""
threshold = int(self._budget * self._start_ratio)
for entry in self._awaiting:
chunk = entry["chunk"]
if chunk is None or chunk.status != "ready" or entry.get("held_announced"):
continue
entry["held_announced"] = True
self._announce(
chunk,
"running",
phase=COMPRESSION_PHASE_READY,
note="held",
extra={
"summary_tokens": self._chars_to_tokens(len(render_chunk_block(chunk)), ledger),
"context_tokens": int(base_tokens),
"context_budget": self._budget,
"swap_at_tokens": threshold,
},
)
def _partition(self, eligible: list[dict]) -> list[tuple[str | None, list[dict]]]: def _partition(self, eligible: list[dict]) -> list[tuple[str | None, list[dict]]]:
"""Split eligible turns into consecutive same-hash segments.""" """Split eligible turns into consecutive same-hash segments.
A turn without a stamp (no recorded step — a reply without a tool
call, a helper-only turn — or a step not stamped yet) joins the
running segment. Leading unstamped turns have no running segment to
join: they are held and become the head of the first stamped segment
instead of forming a stampless segment of their own, which would
close as a chunk with no step records (``_create_chunk`` returns
``None`` for it) and sit at the head of the awaiting queue.
"""
segments: list[tuple[str | None, list[dict]]] = [] segments: list[tuple[str | None, list[dict]]] = []
current_hash: str | None = None current_hash: str | None = None
current: list[dict] = [] current: list[dict] = []
started = False
for turn in eligible: for turn in eligible:
key = turn.get("step_key") key = turn.get("step_key")
stamped = self._step_hashes.get(str(key)) if key is not None else None stamped = self._step_hashes.get(str(key)) if key is not None else None
if stamped is None and not started:
current.append(turn) # leading unstamped turns wait for a segment
continue
if stamped is None: if stamped is None:
stamped = current_hash # unknown stamps join the running segment stamped = current_hash # unknown stamps join the running segment
if current and stamped == current_hash: if not started:
started = True
current_hash = stamped
current.append(turn)
elif stamped == current_hash:
current.append(turn) current.append(turn)
else: else:
if current:
segments.append((current_hash, current)) segments.append((current_hash, current))
current_hash = stamped current_hash = stamped
current = [turn] current = [turn]
@@ -1290,21 +1462,26 @@ class HistoryChunkManager:
seg_turns[i : i + self._max_steps] for i in range(0, len(seg_turns), self._max_steps) seg_turns[i : i + self._max_steps] for i in range(0, len(seg_turns), self._max_steps)
] ]
def _context_base_tokens(self) -> int | None: def _context_base_tokens(self, ledger=None) -> int | None:
"""The live context base for the start/soft/hard thresholds.
An injected ``meter_getter`` wins (tests / custom wiring); otherwise
the base is the owning operator's last measured prompt size recorded
on the ledger (:meth:`TranscriptLedger.record_prompt_tokens`). The
session-wide token meter is deliberately NOT consulted: its
``last_prompt_tokens`` is overwritten by whichever agent called last
(Planner, Validator, Checker, sub-agents, lenses), so it does not
describe the operator's context at all.
"""
if self._meter_getter is not None: if self._meter_getter is not None:
try: try:
return self._meter_getter() return self._meter_getter()
except Exception: except Exception:
return None return None
try: try:
from artemis.services.token_meter import get_meter last = getattr(ledger, "last_prompt_tokens", None)
return int(last) if last else None
session_id = getattr(self._engine, "current_session_id", None) except (TypeError, ValueError):
if not session_id:
return None
last = get_meter(session_id).last_prompt_tokens
return last or None
except Exception:
return None return None
# ------------------------------------------------------------------ # ------------------------------------------------------------------
@@ -1395,7 +1572,11 @@ class HistoryChunkManager:
chunk.source_chars = int(ledger.turn_text_chars(slice_turns)) chunk.source_chars = int(ledger.turn_text_chars(slice_turns))
except Exception as e: except Exception as e:
logger.debug(f"Chunk source size unavailable for {chunk.capsule_key}: {e}") logger.debug(f"Chunk source size unavailable for {chunk.capsule_key}: {e}")
self._announce(chunk, "running", extra={"source_tokens": chunk.source_chars // 4}) self._announce(
chunk,
"running",
extra={"source_tokens": self._chars_to_tokens(chunk.source_chars, ledger)},
)
return chunk return chunk
def _announce( def _announce(
@@ -1408,19 +1589,29 @@ class HistoryChunkManager:
note: str | None = None, note: str | None = None,
forced: bool = False, forced: bool = False,
extra: dict[str, Any] | None = None, extra: dict[str, Any] | None = None,
phase: str | None = None,
) -> None: ) -> None:
"""Update the chunk's timeline trace without interrupting compression.""" """Update the chunk's timeline trace without interrupting compression.
``phase`` is the machine-readable plain-language phase written to
``args.phase`` (one of ``COMPRESSION_PHASES``). It defaults from the
trace ``status`` — running → summarizing, success → applied,
failed → failed — and the held path passes ``ready`` explicitly.
"""
if chunk.trace_id is None: if chunk.trace_id is None:
return return
engine = self._engine engine = self._engine
if engine is None or not hasattr(engine, "record_trace"): if engine is None or not hasattr(engine, "record_trace"):
return return
if phase is None:
phase = _STATUS_TO_PHASE.get(status, COMPRESSION_PHASE_SUMMARIZING)
args: dict[str, Any] = { args: dict[str, Any] = {
"start_step": chunk.start_step_number, "start_step": chunk.start_step_number,
"end_step": chunk.end_step_number, "end_step": chunk.end_step_number,
"steps": len(chunk.source_step_ids), "steps": len(chunk.source_step_ids),
"milestone": chunk.milestone_label, "milestone": chunk.milestone_label,
"trigger": chunk.trigger, "trigger": chunk.trigger,
"phase": phase,
} }
if note: if note:
args["note"] = note args["note"] = note
@@ -1450,9 +1641,15 @@ class HistoryChunkManager:
logger.debug(f"Compression announcement skipped for {chunk.capsule_key}: {e}") logger.debug(f"Compression announcement skipped for {chunk.capsule_key}: {e}")
def _on_capsule_failed(self, key: JobKey) -> None: def _on_capsule_failed(self, key: JobKey) -> None:
"""Mark the trace failed; keep the chunk pending for a later retry.""" """Mark the trace failed; keep the chunk pending for a later retry.
for chunk in self._all_chunks():
if chunk.status != "pending" or chunk.capsule_key != key: Only a chunk still awaiting can be retried (:meth:`_redispatch_failed_capsules`
walks the awaiting queue); a pending chunk that was already force-swapped
keeps its swap trace, since no retry follows for it.
"""
for entry in self._awaiting:
chunk = entry["chunk"]
if chunk is None or chunk.status != "pending" or chunk.capsule_key != key:
continue continue
self._announce( self._announce(
chunk, chunk,
@@ -1584,15 +1781,18 @@ class HistoryChunkManager:
# ------------------------------------------------------------------ # ------------------------------------------------------------------
def _fold_eras(self) -> None: def _fold_eras(self) -> None:
"""Count overflow: the oldest loose chunks merge into an era; the
oldest ledger-bearing eras beyond ``max_eras`` become recall-only
(eras already folded by a hard event do not count against the cap)."""
overflow = len(self._chunks) - self._max_chunks overflow = len(self._chunks) - self._max_chunks
if overflow > 0: if overflow > 0:
folded, self._chunks = self._chunks[:overflow], self._chunks[overflow:] folded, self._chunks = self._chunks[:overflow], self._chunks[overflow:]
self._era_counter += 1 self._era_counter += 1
self._eras.append(EraState(ordinal=self._era_counter, chunks=folded)) self._eras.append(EraState(ordinal=self._era_counter, chunks=folded))
era_overflow = len(self._eras) - self._max_eras with_ledger = [era for era in self._eras if not era.recall_only]
era_overflow = len(with_ledger) - self._max_eras
if era_overflow > 0: if era_overflow > 0:
for era in self._eras[:era_overflow]: for era in with_ledger[:era_overflow]:
if not era.recall_only:
era.recall_only = True era.recall_only = True
logger.info( logger.info(
f"Era {era.ordinal} overflowed to the recall-only period" f"Era {era.ordinal} overflowed to the recall-only period"
@@ -1600,15 +1800,9 @@ class HistoryChunkManager:
f"{era.end_step_number})." f"{era.end_step_number})."
) )
def _render_frozen_blocks(self, *, hard: bool) -> list[BaseMessage]: def _render_frozen_blocks(self) -> list[BaseMessage]:
if not self._chunks and not self._eras: """Frozen region, oldest first: era blocks (recall-only or merged),
return [] then the loose chunk blocks."""
if hard:
plan_text = self._read_plan_text()
text = render_l3_snapshot(
self._eras, self._chunks, goal=self._goal, plan_text=plan_text
)
return [HumanMessage(content=[{"type": "text", "text": text}])]
blocks: list[BaseMessage] = [] blocks: list[BaseMessage] = []
for era in self._eras: for era in self._eras:
blocks.append(HumanMessage(content=[{"type": "text", "text": render_era_block(era)}])) blocks.append(HumanMessage(content=[{"type": "text", "text": render_era_block(era)}]))
@@ -1618,18 +1812,6 @@ class HistoryChunkManager:
) )
return blocks return blocks
def _read_plan_text(self) -> str | None:
try:
from artemis.utils.notes import get_note_file_path
base_dir = getattr(self._engine, "base_dir", None)
if not base_dir:
return None
path = get_note_file_path(base_dir, "task_plan")
return path.read_text(encoding="utf-8") if path.exists() else None
except Exception:
return None
# ------------------------------------------------------------------ # ------------------------------------------------------------------
# Draining # Draining
# ------------------------------------------------------------------ # ------------------------------------------------------------------
+3 -3
View File
@@ -2,14 +2,14 @@
You are the Segment Capsule writer for a mobile automation agent's history compression. You receive one contiguous segment of executed steps and produce the two LLM-authored bands of the segment's history chunk: You are the Segment Capsule writer for a mobile automation agent's history compression. You receive one contiguous segment of executed steps and produce the two LLM-authored bands of the segment's history chunk:
- Band ① — Synopsis & effects: what this segment was doing, what was actually done, and what effects/artifacts it left behind, plus structured fields. - Band ① — Synopsis & effects: what this segment was doing, what was actually done, and what effects/artifacts it left behind, plus structured fields.
- Band ② — Compressed step summary: an interval narrative ("Steps xx–xx did ..., Step xx did ...") that seamlessly covers every step of the segment. - Band ② — Compressed step summary: an interval narrative ("Steps xx–xx did ..., Step xx did ...") covering every step of the segment.
The mechanical per-step action ledger (band ③) is assembled outside of you; do NOT reproduce it. The mechanical per-step action ledger (band ③) is assembled outside of you; do NOT reproduce it.
# INPUT SHAPE # INPUT SHAPE
The segment arrives turn by turn, in order. Each turn block has: The segment arrives turn by turn, in order. Each turn block has:
- `Recorded steps:` — the mechanical facts of the step(s) executed in that turn: step number, session offset, the exact action, the controller/validator result, the visual transition summary, the notes written (tool → note key: gist), and any user-injected instruction. - `Recorded steps:` — the mechanical facts of the step(s) executed in that turn: step number, session offset, the exact action, the controller/validator result, the visual transition summary, the notes written (tool → note key: gist), and any user-injected instruction.
- `Transcript as seen by the operator during this turn:` — the turn exactly as it stood in the operator's own context: the observation text, the operator's reasoning (including `(thinking)` blocks), every `[tool call]` with its arguments, every `[tool result]` returned to the operator (explorer answers, recalled history, note reads, ADB output, screenshot placeholders), and the `--- Action Execution Result ---` message. Your capsule REPLACES this transcript in the operator's context — everything the operator learned from it that still matters must survive in your output. - `Transcript as seen by the operator during this turn:` — the turn exactly as it stood in the operator's own context: the observation text, the operator's reasoning (including `(thinking)` blocks), every `[tool call]` with its arguments, every `[tool result]` returned to the operator (explorer answers, recalled history, note reads, ADB output, screenshot placeholders), and the `--- Action Execution Result ---` message that closes the turn (every turn on the Pro path; only when an action failed on the Flash path). Your capsule REPLACES this transcript in the operator's context — everything the operator learned from it that still matters must survive in your output.
A turn block without a transcript falls back to a reasoning excerpt; treat it the same way with less evidence. A turn block without a transcript falls back to a reasoning excerpt; treat it the same way with less evidence.
@@ -45,7 +45,7 @@ Field `intervals`: an ordered array of `{"start_step": N, "end_step": M, "text":
- **NEVER narrate step by step** ("Step 5: I tapped X. Step 6: I tapped Y."). One interval is a behavior plus the observed outcome, e.g. "Opened the 'Network & internet' section and returned to the main list; the section showed Wi‑Fi, mobile network and VPN rows." - **NEVER narrate step by step** ("Step 5: I tapped X. Step 6: I tapped Y."). One interval is a behavior plus the observed outcome, e.g. "Opened the 'Network & internet' section and returned to the main list; the section showed Wi‑Fi, mobile network and VPN rows."
- **Merge homogeneous consecutive actions into one interval** (repeated scrolls, a sweep through several sections, a polling loop, retries of the same target) — describe the run and its count/result once. A step that does something of a different nature gets its own single-step interval (`start_step == end_step`). - **Merge homogeneous consecutive actions into one interval** (repeated scrolls, a sweep through several sections, a polling loop, retries of the same target) — describe the run and its count/result once. A step that does something of a different nature gets its own single-step interval (`start_step == end_step`).
- Every line's text must reference concrete behavior for those steps (the step numbers come from the `Recorded steps:` lines): the target's visible text, the screen or state observed afterwards, values read, errors shown. - Every line's text must reference concrete behavior for those steps (the step numbers come from the `Recorded steps:` lines): the target's visible text, the screen or state observed afterwards, values read, errors shown.
- **The union of intervals MUST cover the segment's step range seamlessly** — no gaps, no overlaps, in ascending order. This is machine-checked; a gap forces regeneration. - **The union of intervals MUST cover the segment's full step range** — no gaps, no overlaps, in ascending order. This is machine-checked; a gap forces regeneration.
- Keep the neutral-wording contract in every line. - Keep the neutral-wording contract in every line.
# LENGTH TARGET (SOFT) # LENGTH TARGET (SOFT)
+28 -1
View File
@@ -20,12 +20,13 @@ with alias resolution (tool_call_id / legacy ordinals), and status queries.
What a summary *contains* (the lens semantics) lives in subclasses — the What a summary *contains* (the lens semantics) lives in subclasses — the
service only guarantees how jobs are scheduled, retried, keyed, and drained. service only guarantees how jobs are scheduled, retried, keyed, and drained.
Both Flash and (from M2 on) Pro profiles share this runtime. Flash and Pro profiles share this runtime.
""" """
import asyncio import asyncio
from typing import Any from typing import Any
from artemis.data_engine.trace import detached_trace
from artemis.utils.logger import get_logger from artemis.utils.logger import get_logger
logger = get_logger(__name__) logger = get_logger(__name__)
@@ -67,12 +68,20 @@ class StepMemoryService:
where available; tool_call_ids (and legacy ordinals) remain queryable where available; tool_call_ids (and legacy ordinals) remain queryable
through an alias map so message-level consumers need no migration. through an alias map so message-level consumers need no migration.
5. Bounded concurrency: at most ``max_concurrency`` attempts run at once. 5. Bounded concurrency: at most ``max_concurrency`` attempts run at once.
6. Detached tracing: every job runs inside
:func:`~artemis.data_engine.trace.detached_trace`, so a lens's model
calls never attach to the trace span (the operator's step) that was
current when the job was submitted — no lens needs to know.
Subclasses implement :meth:`_attempt` (one summarization attempt for a Subclasses implement :meth:`_attempt` (one summarization attempt for a
payload) and may override :meth:`_on_status` to persist pending/failed payload) and may override :meth:`_on_status` to persist pending/failed
status transitions. status transitions.
""" """
#: Trace node label of a lens-less subclass's jobs (``lens:<name>``); a
#: configured :class:`StepLens` contributes its own ``name`` instead.
lens_name: str | None = None
def __init__( def __init__(
self, self,
ctx: Any, ctx: Any,
@@ -131,7 +140,25 @@ class StepMemoryService:
# Scheduling core # Scheduling core
# ------------------------------------------------------------------ # ------------------------------------------------------------------
@property
def trace_node_name(self) -> str:
"""Node label the job's traces and usage records carry (``lens:<name>``)."""
name = getattr(self._lens, "name", None) or self.lens_name or type(self).__name__.lower()
return f"lens:{name}"
async def _run_until_ready(self, key: JobKey) -> None: async def _run_until_ready(self, key: JobKey) -> None:
"""Run one job to completion, detached from the submitter's trace span.
The task was created while the submitting agent's step span was
current and inherits that context; the whole job — every attempt,
retry sleeps included — runs under :func:`detached_trace` so nothing
a lens records (model calls, their thinking/raw_thinking children,
stream deltas, usage) can be attributed to the submitter's step.
"""
with detached_trace(self.trace_node_name):
await self._run_attempts(key)
async def _run_attempts(self, key: JobKey) -> None:
"""Retry one job independently, bounded by the configured retry limit. """Retry one job independently, bounded by the configured retry limit.
Attempts at most ``1 + retry_limit`` times; on exhaustion the job Attempts at most ``1 + retry_limit`` times; on exhaustion the job
+341 -13
View File
@@ -22,11 +22,16 @@ Messages are organized into four regions:
- **A (active window)**: raw per-turn messages, append-only. Each committed - **A (active window)**: raw per-turn messages, append-only. Each committed
turn contributes its tail observation HumanMessage, the operator AIMessages turn contributes its tail observation HumanMessage, the operator AIMessages
(tool_calls and native thinking preserved by reference), the in-turn (tool_calls and native thinking preserved by reference), the in-turn
ToolMessages, and the turn's validator result message. The scrub edge ToolMessages, and the turn's result message (every turn on the Pro path;
(:class:`~artemis.agents.flash.context_compressor.ScrubEdgeCompressor`) strips only when an action failed on the Flash path). The
old UI lists and plan recitations at depth 1 and resolves screenshots to scrub edge (:class:`~artemis.agents.flash.context_compressor.ScrubEdgeCompressor`)
visual summaries at depth K — messages are never has two depths: a shallow text edge (UI list, plan recitation and ephemeral
removed or reordered, so tool-call/response pairs are never split. per-turn blocks leave as soon as a newer observation exists) and a
screenshot edge that follows the occupancy tier (relaxed below the start
gate, tightened past it), where the screenshot is replaced in place by its
visual summary. Every rewrite happens once and is never undone; everything
newer is byte-identical to what the model was first shown. Messages are
never removed or reordered, so tool-call/response pairs are never split.
- **T (current tail)**: built fresh every turn by the operator; passed to - **T (current tail)**: built fresh every turn by the operator; passed to
:meth:`render` and only enters A when the turn is committed. :meth:`render` and only enters A when the turn is committed.
@@ -47,7 +52,7 @@ import json
import time import time
from typing import Any, Callable from typing import Any, Callable
from langchain_core.messages import BaseMessage, HumanMessage from langchain_core.messages import AIMessage, BaseMessage, HumanMessage
from artemis.memory.step_memory import StepMemoryService from artemis.memory.step_memory import StepMemoryService
from artemis.utils.logger import get_logger from artemis.utils.logger import get_logger
@@ -67,6 +72,28 @@ EXECUTION_RESULT_MARKER = "--- Action Execution Result"
#: Header prefix of the cold-start restored-history block. #: Header prefix of the cold-start restored-history block.
RESTORED_HISTORY_HEADER = "[Restored history]" RESTORED_HISTORY_HEADER = "[Restored history]"
#: ``HumanMessage.additional_kwargs`` key listing the indices of content blocks that
#: are only meaningful for the turn they were built for (reminders, hints, user
#: guidance, per-turn notices). The scrub edge deletes them at its text edge, so
#: they never reach the frozen region or a chunk capsule. The indices always
#: describe the message's *current* block layout: a rewrite that removes blocks
#: before the text edge (the screenshot swap when the image edge is shallower)
#: remaps the surviving indices in place, and the key is dropped once the blocks
#: are deleted. ``additional_kwargs`` is never sent to the provider, so the
#: marking costs the model nothing.
EPHEMERAL_BLOCKS_KEY = "ephemeral_blocks"
def mark_ephemeral(message: BaseMessage, indices) -> None:
"""Flag content blocks of ``message`` (by index) as ephemeral, see
:data:`EPHEMERAL_BLOCKS_KEY`."""
existing = list(message.additional_kwargs.get(EPHEMERAL_BLOCKS_KEY, []))
for index in indices:
index = int(index)
if index not in existing:
existing.append(index)
message.additional_kwargs[EPHEMERAL_BLOCKS_KEY] = existing
def format_session_offset(seconds: float) -> str: def format_session_offset(seconds: float) -> str:
"""Render a session-start offset as ``T+mm:ss`` (minutes never wrap).""" """Render a session-start offset as ``T+mm:ss`` (minutes never wrap)."""
@@ -135,6 +162,8 @@ def render_turn_transcript(messages: list[BaseMessage]) -> str:
names_by_id: dict[str, str] = {} names_by_id: dict[str, str] = {}
for msg in messages: for msg in messages:
kind = str(getattr(msg, "type", "") or "message") kind = str(getattr(msg, "type", "") or "message")
if isinstance(msg, AIMessage):
kind = "ai" # streamed replies report type "AIMessageChunk"
role = _ROLE_LABELS.get(kind, kind) role = _ROLE_LABELS.get(kind, kind)
if kind == "tool": if kind == "tool":
call_id = getattr(msg, "tool_call_id", None) call_id = getattr(msg, "tool_call_id", None)
@@ -157,6 +186,71 @@ def render_turn_transcript(messages: list[BaseMessage]) -> str:
return "\n".join(lines) return "\n".join(lines)
#: Tokens charged per image block in the provider-less prompt estimate
#: (Gemini's flat per-image cost; the Flash runner uses the same figure).
IMAGE_BLOCK_TOKENS = 258
#: Characters per token assumed before any calibrated measurement.
DEFAULT_CHARS_PER_TOKEN = 4.0
#: Bounds of the calibrated chars-per-token ratio: ~1 for CJK-heavy prompts,
#: ~4 for English prose; anything outside is a measurement artefact.
_CHARS_PER_TOKEN_BOUNDS = (1.0, 6.0)
#: Smallest text growth (characters) between two consecutive measured prompts
#: for their difference to count as a calibration sample (see
#: :meth:`TranscriptLedger.record_prompt_tokens`). Below this the provider's
#: rounding and a few hundred tokens of turn-to-turn jitter dominate the
#: quotient.
MIN_CALIBRATION_DELTA_CHARS = 2000
def measure_prompt_content(messages: list[Any]) -> tuple[int, int]:
"""Count ``(text_chars, image_count)`` over a prompt's messages.
The walk covers what the provider tokenizes as text: string content in
full; in list content the ``text`` blocks by their text, ``image_url`` /
``image`` blocks as one image each, any other block by its ``str()``; and
each message's ``tool_calls`` by the JSON of their arguments (an
AIMessage's calls are sent back to the model verbatim, and the ledger's
turn transcripts size them the same way).
"""
text_chars = 0
image_count = 0
for msg in messages:
content = getattr(msg, "content", None)
if isinstance(content, str):
text_chars += len(content)
elif isinstance(content, list):
for block in content:
if isinstance(block, dict):
if block.get("type") in ("image_url", "image") or "image_url" in block:
image_count += 1
elif block.get("type") == "text":
text_chars += len(str(block.get("text", "")))
else:
text_chars += len(str(block))
else:
text_chars += len(str(block))
elif content is not None:
text_chars += len(str(content))
for call in getattr(msg, "tool_calls", None) or []:
if isinstance(call, dict):
text_chars += len(
json.dumps(call.get("args") or {}, ensure_ascii=False, default=str)
)
return text_chars, image_count
def estimate_prompt_tokens(messages: list[Any]) -> int:
"""Provider-less prompt size estimate: ``text_chars // 4`` plus
:data:`IMAGE_BLOCK_TOKENS` per image block (see
:func:`measure_prompt_content` for what counts). Used as the context base
when a provider reports no usage metadata; never below 1."""
text_chars, image_count = measure_prompt_content(messages)
return max(1, text_chars // 4 + image_count * IMAGE_BLOCK_TOKENS)
class TranscriptLedger: class TranscriptLedger:
"""Append-only four-region message ledger for one Pro session. """Append-only four-region message ledger for one Pro session.
@@ -172,7 +266,10 @@ class TranscriptLedger:
prune_history_xml: bool = True, prune_history_xml: bool = True,
image_scrub_depth: int = 3, image_scrub_depth: int = 3,
pending_grace_steps: int = 3, pending_grace_steps: int = 3,
xml_scrub_depth: int = 1, xml_scrub_depth: int | None = None,
image_scrub_depth_relaxed: int | None = None,
context_budget_tokens: int | None = None,
start_ratio: float | None = None,
clock: Callable[[], float] | None = None, clock: Callable[[], float] | None = None,
session_start: float | None = None, session_start: float | None = None,
): ):
@@ -204,6 +301,51 @@ class TranscriptLedger:
self._frozen_blocks: list[BaseMessage] = [] self._frozen_blocks: list[BaseMessage] = []
self._chunker: Any | None = None self._chunker: Any | None = None
# The owning operator's last measured prompt size (tokens). This is
# the live context base the chunker's start/soft/hard thresholds read;
# only the operator's own calls write it, so Planner/Checker/lens
# calls sharing the session can never masquerade as the operator's
# context (see HistoryChunkManager._context_base_tokens).
self._last_prompt_tokens: int | None = None
# Session-calibrated characters-per-token ratio (see
# record_prompt_tokens): the chunker converts transcript characters
# to tokens through it, so CJK-heavy sessions (~1 char/token on
# Gemini) are not under-estimated 4x by the fixed //4 default.
# Calibrated differentially: ``_calibration_sample`` is the previous
# measured ``(text_chars, image_count, prompt_tokens)`` and the
# pooled deltas between consecutive same-image-count prompts give
# the session's single best ratio.
self._chars_per_token: float = DEFAULT_CHARS_PER_TOKEN
self._calibration_sample: tuple[int, int, int] | None = None
self._pooled_delta_chars = 0
self._pooled_delta_tokens = 0
# Whether the last committed turn carried no visible reasoning text (bare
# tool calls / thinking only). Set by ``commit_staged``; both runners read
# it to decide whether to attach the shared reasoning reminder.
self._last_turn_silent: bool = False
# Text edge (default 1): UI list, plan recitation and ephemeral blocks
# leave as soon as a newer observation exists; screenshots follow K.
self._xml_scrub_depth = xml_scrub_depth
# Occupancy-driven screenshot depth. Below the start gate
# (``last_prompt_tokens < budget * start_ratio``, or no measurement
# yet) the scrub edge sits at the relaxed depth so the model keeps
# more raw screenshots while nothing is being replaced; from the
# start gate up it tightens to ``image_scrub_depth``. Scrubbing is
# irreversible, so a change of depth never rewrites a message:
# tightening scrubs the extra depths once, relaxing lets new images
# live longer. See :attr:`effective_image_scrub_depth`.
self._image_scrub_depth = max(1, int(image_scrub_depth))
self._image_scrub_depth_relaxed = (
max(self._image_scrub_depth, int(image_scrub_depth_relaxed))
if image_scrub_depth_relaxed is not None
else self._image_scrub_depth
)
self._context_budget_tokens = int(context_budget_tokens) if context_budget_tokens else None
self._start_ratio = float(start_ratio) if start_ratio is not None else None
# id(message) -> summary-job key (DataEngine step id). A side map keeps # id(message) -> summary-job key (DataEngine step id). A side map keeps
# the key out of the serialized message payload entirely. # the key out of the serialized message payload entirely.
self._step_keys: dict[int, str] = {} self._step_keys: dict[int, str] = {}
@@ -235,6 +377,186 @@ class TranscriptLedger:
"""The current session offset as ``T+mm:ss``.""" """The current session offset as ``T+mm:ss``."""
return format_session_offset(self.elapsed_seconds()) return format_session_offset(self.elapsed_seconds())
# ------------------------------------------------------------------
# Context base (operator-measured prompt size)
# ------------------------------------------------------------------
@property
def last_prompt_tokens(self) -> int | None:
"""The operator's last measured prompt size, or None before the
first measured call (or when the provider reports no usage)."""
return self._last_prompt_tokens
@property
def last_turn_silent(self) -> bool:
"""True when the last committed turn had no visible reasoning text.
A turn is silent when it carries at least one AI message and none of
them shows text: string content that is empty or whitespace, or list
content without a non-empty ``text`` block (native ``thinking`` blocks
are not visible text). A turn with no AI message at all is not silent.
"""
return self._last_turn_silent
@staticmethod
def _ai_message_has_visible_text(message: BaseMessage) -> bool:
content = getattr(message, "content", None)
if isinstance(content, str):
return bool(content.strip())
if isinstance(content, list):
for block in content:
if isinstance(block, str):
if block.strip():
return True
elif isinstance(block, dict) and block.get("type") == "text":
if str(block.get("text") or "").strip():
return True
return False
return False
@classmethod
def _turn_is_silent(cls, messages: list[BaseMessage]) -> bool:
# Streamed replies are ``AIMessageChunk`` (type "AIMessageChunk"), so
# the class, not the type string, identifies the model's messages.
ai_messages = [m for m in messages if isinstance(m, AIMessage)]
if not ai_messages:
return False
return not any(cls._ai_message_has_visible_text(m) for m in ai_messages)
@property
def chars_per_token(self) -> float:
"""Calibrated characters-per-token ratio (4.0 until a measured call
with its messages is recorded)."""
return self._chars_per_token
def chars_to_tokens(self, chars: int) -> int:
"""Convert transcript characters to tokens through the calibrated
ratio (identical to ``chars // 4`` at the default ratio)."""
return int(max(0, int(chars)) / self._chars_per_token)
@property
def occupancy(self) -> float | None:
"""``last_prompt_tokens / context_budget_tokens``, or None when either
is unknown."""
if self._last_prompt_tokens is None or not self._context_budget_tokens:
return None
return self._last_prompt_tokens / self._context_budget_tokens
@property
def below_start_gate(self) -> bool:
"""Whether the measured occupancy is under the start gate. An unknown
occupancy (no measured call yet) counts as below: a session starts
empty."""
if self._start_ratio is None:
return False
ratio = self.occupancy
return ratio is None or ratio < self._start_ratio
@property
def effective_image_scrub_depth(self) -> int:
"""The screenshot depth the scrub edge uses at the next render: the
relaxed depth below the start gate, the tight depth from there up."""
if self.below_start_gate:
return self._image_scrub_depth_relaxed
return self._image_scrub_depth
def record_prompt_tokens(
self, prompt_tokens: int | None, messages: list[Any] | None = None
) -> None:
"""Record the operator's measured prompt size for one call.
Called by the ledger's owning operator (Flash runner / Pro operator)
after each of its own LLM calls; within a multi-call turn the last
(largest) call wins. Non-positive or missing values are ignored so a
provider without usage metadata leaves the base unknown rather than
zero (and the calibration state below is left untouched).
When ``messages`` (the exact list sent to the model) accompanies a
measured size, the call also calibrates :attr:`chars_per_token`,
**differentially**: the ledger remembers the previous measured
sample ``(text_chars, image_count, prompt_tokens)`` and, when the new
sample carries the same number of images, grows the text by at least
:data:`MIN_CALIBRATION_DELTA_CHARS` and costs more tokens, it pools
the pair's differences (``pooled_chars += Δchars``,
``pooled_tokens += Δtokens``) and sets the ratio to
``pooled_chars / pooled_tokens``, clamped to [1.0, 6.0]. The pooled
quotient is the maximum-likelihood single ratio for the session (no
smoothing constant to tune, no last-pair noise). The reference is
replaced only when it was consumed, when the image count changed, or
when the prompt shrank in tokens (a compaction or scrub reset the
baseline); a sample that moved too little keeps the reference, so
several small turns accumulate into one usable difference instead of
being discarded one by one (real sessions often grow < 2000 chars
per turn and would otherwise never calibrate).
Why differences: the provider's ``prompt_tokens`` also counts terms
that are not in ``messages`` or not proportional to their text — the
bound tool/function schemas, and full-size screenshots that tile to
~1100–1800 tokens rather than the flat :data:`IMAGE_BLOCK_TOKENS`.
A whole-prompt quotient charges all of that to the text, and it
also averages over the static prefix, whereas the chunker applies
the ratio to the transcript *tail*, whose density differs. On real
Gemini Operator traces (10 sessions, 2026-09) the two disagree by
5-30% with a session-dependent sign, so the size-based chunk
trigger fired early in some sessions and late in others. Between
two prompts with the same image count the constant terms cancel and
the difference measures the tail's own rate. Until the first
usable difference exists the whole-prompt quotient
``text_chars / (prompt_tokens - images * IMAGE_BLOCK_TOKENS)`` is
used as a fallback; once a pooled difference exists the pooled ratio
is authoritative and the fallback is no longer applied.
"""
try:
value = int(prompt_tokens) if prompt_tokens is not None else 0
except (TypeError, ValueError):
return
if value <= 0:
return
self._last_prompt_tokens = value
if messages is None:
return
try:
text_chars, image_count = measure_prompt_content(messages)
except Exception as exc:
logger.debug(f"Prompt calibration skipped: {exc}", exc_info=True)
return
if text_chars <= 0:
return
low, high = _CHARS_PER_TOKEN_BOUNDS
previous = self._calibration_sample
sample = (text_chars, image_count, value)
if previous is None:
self._calibration_sample = sample
else:
prev_chars, prev_images, prev_tokens = previous
delta_chars = text_chars - prev_chars
delta_tokens = value - prev_tokens
if image_count != prev_images or delta_tokens < 0:
# Not comparable (image set changed) or the prompt actually
# shrank (compaction / scrub): start a fresh baseline here.
self._calibration_sample = sample
elif delta_chars >= MIN_CALIBRATION_DELTA_CHARS:
if delta_tokens > 0:
self._pooled_delta_chars += delta_chars
self._pooled_delta_tokens += delta_tokens
pooled = self._pooled_delta_chars / self._pooled_delta_tokens
self._chars_per_token = min(high, max(low, pooled))
self._calibration_sample = sample
return
self._calibration_sample = sample # text grew at no cost: unusable
# else: moved too little to measure yet (in either direction) —
# keep the reference so the movement accumulates against it.
if self._pooled_delta_tokens > 0:
# A pooled difference already exists: it is authoritative, and
# the whole-prompt quotient (which mis-charges schemas and image
# tiling to the text) is never applied again.
return
text_tokens = max(1, value - image_count * IMAGE_BLOCK_TOKENS)
self._chars_per_token = min(high, max(low, text_chars / text_tokens))
# ------------------------------------------------------------------ # ------------------------------------------------------------------
# S region # S region
# ------------------------------------------------------------------ # ------------------------------------------------------------------
@@ -320,6 +642,7 @@ class TranscriptLedger:
return return
staged = self._staged staged = self._staged
self._staged = None self._staged = None
self._last_turn_silent = self._turn_is_silent(staged)
if step_key is not None: if step_key is not None:
observation = self._first_image_message(staged) observation = self._first_image_message(staged)
@@ -449,18 +772,23 @@ class TranscriptLedger:
def render(self, tail: list[BaseMessage]) -> list[BaseMessage]: def render(self, tail: list[BaseMessage]) -> list[BaseMessage]:
"""Advance compression and return ``S + F + A + tail``. """Advance compression and return ``S + F + A + tail``.
Order per turn: the chunker runs first (a triggered compression event Order per turn: the occupancy band is read once (from the operator's
deep-mutates the frozen region and advances the F/A boundary), then last measured prompt), the scrub edge advances over the active window
the scrub edge advances over the active window. The returned list is a at that band's screenshot depth, then the chunker runs (a triggered
fresh container: appending to it (the operator's in-turn tool loop) compression event deep-mutates the frozen region and advances the F/A
never mutates the ledger regions. boundary). Scrub-before-chunk guarantees that the turns a chunk
closes over — all older than the sliding-window floor — have already
had their screenshots resolved when the relaxed depth is no deeper
than the floor plus one. The returned list is a fresh container:
appending to it (the operator's in-turn tool loop) never mutates the
ledger regions.
""" """
self._compressor.compress(self._active, image_scrub_depth=self.effective_image_scrub_depth)
if self._chunker is not None: if self._chunker is not None:
try: try:
self._chunker.on_render(self) self._chunker.on_render(self)
except Exception as e: except Exception as e:
logger.error(f"History chunker render hook failed: {e}") logger.error(f"History chunker render hook failed: {e}")
self._compressor.compress(self._active)
return [ return [
*self._static, *self._static,
*self._restored, *self._restored,
+25 -3
View File
@@ -192,7 +192,7 @@
// ========================================================================== // ==========================================================================
// 🌐 4.3 Shared Capabilities & Global Overrides // 🌐 4.3 Shared Capabilities & Global Overrides
// ========================================================================== // ==========================================================================
// 🧠 Unified Step-Memory Runtime (history-module redesign v3, M0–M5) // Shared step memory and history compression
// Transcript ledger + layered compression for the Pro operator. // Transcript ledger + layered compression for the Pro operator.
"memory": { "memory": {
// Background summarization runtime (visual/capsule lenses) // Background summarization runtime (visual/capsule lenses)
@@ -204,15 +204,33 @@
"transcript": { "transcript": {
// ROLLBACK SWITCH: set false to restore the legacy per-turn 2-message // ROLLBACK SWITCH: set false to restore the legacy per-turn 2-message
// prompt path byte-for-byte (L2/L3 chunk compression rides this same // prompt path byte-for-byte (L2/L3 chunk compression rides this same
// flag). Default on since M5 (2026-09-01) per the on-device A/B: // flag). See services/token_meter.py for measured session token usage
// turn latency −77% P50, cache-hit calls 43%→88%, output tokens −74%. // and cache-hit ratios.
"enabled": true, "enabled": true,
"context_budget_tokens": 80000, "context_budget_tokens": 80000,
// Below start_ratio of the operator's own measured prompt size, closed
// segments only prepare their summary; the original turns are kept in
// context and replaced once occupancy reaches this ratio. Measurements
// come from the operator; other agents' calls do not affect this gate.
"start_ratio": 0.35,
"soft_ratio": 0.7, "soft_ratio": 0.7,
"hard_ratio": 0.9, "hard_ratio": 0.9,
"min_active_steps": 5, "min_active_steps": 5,
// Screenshot edge: 6 while the operator's measured context is below
// start_ratio (nothing is being replaced yet), 3 from there up. A
// screenshot reaching the edge is replaced in place by its visual
// summary (the 'Current Screenshot' label leaves with it) and never
// restored; the depth only decides how long new ones stay.
"image_scrub_depth": 3, "image_scrub_depth": 3,
"image_scrub_depth_relaxed": 6,
"pending_grace_steps": 3, "pending_grace_steps": 3,
// Text edge: once a newer observation exists (depth 1) the UI element
// list, the plan recitation and the per-turn ephemeral blocks
// (reminders, hints, guidance wrappers) leave the older observation.
// User instruction bodies remain in history. The text edge stays
// shallow on purpose: only the live observation carries an indexed
// element list, so a stale [n] index can never be picked up as a
// target.
"xml_scrub_depth": 1, "xml_scrub_depth": 1,
// Local historical-state hint (no model call); distance calibrated // Local historical-state hint (no model call); distance calibrated
// 2026-09-01 from 460 dHash-stamped on-device steps (same screen <=4, // 2026-09-01 from 460 dHash-stamped on-device steps (same screen <=4,
@@ -223,6 +241,10 @@
// Segment (L2) chunk compression: three-band chunks, era folding // Segment (L2) chunk compression: three-band chunks, era folding
"chunking": { "chunking": {
"max_steps": 12, "max_steps": 12,
// Size/pressure closes wait until at least this many turns are
// eligible, so heavy turns do not become one-step chunks (one capsule
// call each); milestone closes are exempt, the hard line waives it.
"min_steps": 3,
"target_source_tokens": 2000, "target_source_tokens": 2000,
"model": "gemini-3.8-flash", "model": "gemini-3.8-flash",
"max_chunks": 8 "max_chunks": 8
+2
View File
@@ -33,6 +33,7 @@ from artemis.runtime.daemon_client import (
ensure_daemon_running, ensure_daemon_running,
get_daemon_session, get_daemon_session,
get_daemon_status, get_daemon_status,
is_artemis_daemon,
is_daemon_running, is_daemon_running,
stop_task_on_daemon, stop_task_on_daemon,
submit_batch_to_daemon, submit_batch_to_daemon,
@@ -84,6 +85,7 @@ __all__ = [
"get_daemon_session", "get_daemon_session",
"get_daemon_status", "get_daemon_status",
"get_server_status", "get_server_status",
"is_artemis_daemon",
"is_daemon_running", "is_daemon_running",
"is_cancel_requested", "is_cancel_requested",
"is_port_in_use", "is_port_in_use",
+26
View File
@@ -17,6 +17,7 @@
from __future__ import annotations from __future__ import annotations
import json import json
from http.client import HTTPException
import os import os
from pathlib import Path from pathlib import Path
import subprocess import subprocess
@@ -62,6 +63,31 @@ def is_daemon_running(
return False return False
def is_artemis_daemon(
host: str = DEFAULT_DAEMON_HOST,
port: int = DEFAULT_DAEMON_PORT,
timeout: float = 0.5,
) -> bool:
"""Identity probe: is the process on ``host:port`` the Artemis daemon?
:func:`is_daemon_running` is a liveness probe that accepts any HTTP
listener (a dev server on port 8000 answers ``200`` on ``/``). Diagnostics
need to tell the daemon apart from a port squatter, so this asks the
daemon's lightweight ``/api/system/server-status`` route and requires the
JSON shape only Artemis serves.
"""
url = f"http://{host}:{port}/api/system/server-status"
try:
req = urllib.request.Request(url, headers={"User-Agent": "Artemis-Daemon-Probe"})
with urllib.request.urlopen(req, timeout=timeout) as response:
if response.status != 200:
return False
payload = json.loads(response.read().decode("utf-8", errors="replace") or "null")
except (OSError, ValueError, HTTPException):
return False
return isinstance(payload, dict) and "current_pid" in payload and "port" in payload
def daemon_log_path(port: int = DEFAULT_DAEMON_PORT) -> Path: def daemon_log_path(port: int = DEFAULT_DAEMON_PORT) -> Path:
"""Return the log file path used by a background-spawned Daemon on the given port.""" """Return the log file path used by a background-spawned Daemon on the given port."""
from artemis.config.paths import get_app_dir from artemis.config.paths import get_app_dir
+111 -16
View File
@@ -12,20 +12,26 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
"""Session-level LLM token metering (M0: pure recording, no decisions). """Session-level LLM token usage and cache-hit metering.
Every gateway-managed LLM call reports its real ``usage_metadata`` here. The Every gateway-managed LLM call reports its real ``usage_metadata`` here. The
meter accumulates per-session totals and records one ``llm_usage`` trace per meter accumulates per-session totals and records one ``llm_usage`` trace per
call into the DataEngine, carrying: call into the DataEngine, carrying:
- the call's measured ``prompt_tokens`` — the last call's measured input is the - the call's measured ``prompt_tokens`` (``context_base_tokens`` in the trace
best available estimate of the *current context base* for the next call payload, kept for the timeline). Note: the session meter's
(the compaction thresholds of M2/M3 will consume this, never char-count ``last_prompt_tokens`` is whichever call happened last in the session —
heuristics); Planner, Validator, Checker, sub-agents included — so the compaction
thresholds do NOT read it; the operator records its own prompt size on the
transcript ledger (``TranscriptLedger.record_prompt_tokens``);
- provider-reported cache hits (Gemini ``cached_content_token_count`` / - provider-reported cache hits (Gemini ``cached_content_token_count`` /
LangChain ``input_token_details.cache_read``) to establish the cache-hit-rate LangChain ``input_token_details.cache_read``) to establish the cache-hit-rate
baseline; baseline;
- running per-session totals. - running per-session totals, including ``session_cached_ratio``
(cached / prompt tokens over the session) and the same ratio for the call's
own ``source`` (``source_cached_ratio``), so a session whose operator calls
never hit the prompt cache is visible on its ``llm_usage`` traces and in the
session summary line (:func:`log_session_summary`).
This module never raises into the LLM call path and makes no decisions. This module never raises into the LLM call path and makes no decisions.
""" """
@@ -71,8 +77,15 @@ def extract_usage(response: Any) -> dict[str, int] | None:
} }
def cached_ratio(cached_tokens: int, prompt_tokens: int) -> float:
"""``cached / prompt`` rounded to three decimals; 0.0 with no prompt tokens."""
if prompt_tokens <= 0:
return 0.0
return round(cached_tokens / prompt_tokens, 3)
class SessionTokenMeter: class SessionTokenMeter:
"""Accumulates measured LLM usage for one session.""" """Accumulates measured LLM usage for one session, in total and per source."""
def __init__(self, session_id: str): def __init__(self, session_id: str):
self.session_id = session_id self.session_id = session_id
@@ -83,36 +96,116 @@ class SessionTokenMeter:
self.cached_tokens = 0 self.cached_tokens = 0
self.cache_hit_calls = 0 self.cache_hit_calls = 0
self.last_prompt_tokens = 0 self.last_prompt_tokens = 0
# Per-source totals (source = gateway endpoint key or lens label):
# {"llm_calls", "prompt_tokens", "cached_tokens", "cache_hit_calls"}.
self.sources: dict[str, dict[str, int]] = {}
def record(self, usage: dict[str, int], *, update_last_prompt: bool = True) -> dict[str, int]: def record(
self,
usage: dict[str, int],
*,
update_last_prompt: bool = True,
source: str | None = None,
) -> dict[str, Any]:
"""Accumulates one call's usage; returns a session snapshot. """Accumulates one call's usage; returns a session snapshot.
``update_last_prompt=False`` accumulates the totals without touching ``update_last_prompt=False`` accumulates the totals without touching
``last_prompt_tokens`` — background lens calls carry tiny prompts that ``last_prompt_tokens`` — background lens calls carry tiny prompts that
must not masquerade as the live context base consumed by the L2/L3 must not masquerade as the live context base consumed by the L2/L3
compaction thresholds. compaction thresholds. With a ``source`` the call also joins that
source's totals and the snapshot carries ``source_cached_ratio``.
""" """
with self._lock: with self._lock:
self.llm_calls += 1 prompt = usage.get("prompt_tokens", 0)
self.prompt_tokens += usage.get("prompt_tokens", 0)
self.completion_tokens += usage.get("completion_tokens", 0)
cached = usage.get("cached_tokens", 0) cached = usage.get("cached_tokens", 0)
self.llm_calls += 1
self.prompt_tokens += prompt
self.completion_tokens += usage.get("completion_tokens", 0)
self.cached_tokens += cached self.cached_tokens += cached
if cached > 0: if cached > 0:
self.cache_hit_calls += 1 self.cache_hit_calls += 1
if update_last_prompt: if update_last_prompt:
self.last_prompt_tokens = usage.get("prompt_tokens", 0) self.last_prompt_tokens = prompt
return self.snapshot() snapshot = self._snapshot_locked()
if source:
totals = self.sources.setdefault(
source,
{"llm_calls": 0, "prompt_tokens": 0, "cached_tokens": 0, "cache_hit_calls": 0},
)
totals["llm_calls"] += 1
totals["prompt_tokens"] += prompt
totals["cached_tokens"] += cached
if cached > 0:
totals["cache_hit_calls"] += 1
snapshot["source_cached_ratio"] = cached_ratio(
totals["cached_tokens"], totals["prompt_tokens"]
)
return snapshot
def snapshot(self) -> dict[str, int]: def _snapshot_locked(self) -> dict[str, Any]:
return { return {
"session_llm_calls": self.llm_calls, "session_llm_calls": self.llm_calls,
"session_prompt_tokens": self.prompt_tokens, "session_prompt_tokens": self.prompt_tokens,
"session_completion_tokens": self.completion_tokens, "session_completion_tokens": self.completion_tokens,
"session_cached_tokens": self.cached_tokens, "session_cached_tokens": self.cached_tokens,
"session_cache_hit_calls": self.cache_hit_calls, "session_cache_hit_calls": self.cache_hit_calls,
"session_cached_ratio": cached_ratio(self.cached_tokens, self.prompt_tokens),
} }
def snapshot(self) -> dict[str, Any]:
with self._lock:
return self._snapshot_locked()
def summary(self) -> dict[str, Any]:
"""Session totals plus every source's totals with its ``cached_ratio``."""
with self._lock:
summary: dict[str, Any] = {"session_id": self.session_id, **self._snapshot_locked()}
summary["sources"] = {
source: {
**totals,
"cached_ratio": cached_ratio(totals["cached_tokens"], totals["prompt_tokens"]),
}
for source, totals in self.sources.items()
}
return summary
def format_session_summary(session_id: Any) -> str:
"""One log line: session cache ratio, then each source's calls and ratio."""
summary = get_meter(session_id).summary()
parts = [
f"LLM usage for session {summary['session_id']}:"
f" {summary['session_llm_calls']} calls,"
f" {summary['session_prompt_tokens']} prompt tokens,"
f" {summary['session_cached_tokens']} cached"
f" (cached_ratio={summary['session_cached_ratio']:.3f},"
f" cache-hit calls {summary['session_cache_hit_calls']})"
]
for source, totals in summary["sources"].items():
parts.append(
f"{source}: {totals['llm_calls']} calls, cached_ratio={totals['cached_ratio']:.3f}"
)
return "; ".join(parts)
def log_session_summary(session_id: Any) -> dict[str, Any] | None:
"""Log the session's usage summary (INFO) at session end, best-effort.
Returns the summary dict, or None when the session never metered a call
(nothing is logged then). Never raises.
"""
try:
key = str(session_id)
with _LOCK:
meter = _METERS.get(key)
if meter is None or meter.llm_calls == 0:
return None
logger.info(format_session_summary(key))
return meter.summary()
except Exception as exc:
logger.debug(f"Token meter session summary skipped: {exc}")
return None
def get_meter(session_id: Any) -> SessionTokenMeter: def get_meter(session_id: Any) -> SessionTokenMeter:
"""Returns (creating on first use) the meter for a session id.""" """Returns (creating on first use) the meter for a session id."""
@@ -147,7 +240,9 @@ def record_llm_usage(
if not session_id or usage is None: if not session_id or usage is None:
return None return None
snapshot = get_meter(session_id).record(usage, update_last_prompt=update_last_prompt) snapshot = get_meter(session_id).record(
usage, update_last_prompt=update_last_prompt, source=source
)
payload: dict[str, Any] = { payload: dict[str, Any] = {
**usage, **usage,
"context_base_tokens": usage["prompt_tokens"], "context_base_tokens": usage["prompt_tokens"],
+49 -19
View File
@@ -125,23 +125,52 @@ def _adb_binary() -> str:
return "adb" return "adb"
#: Inline display cap for command output, in characters. Beyond it the full
#: text is cached under a TaskId and only the tail is shown to the model.
_LONG_OUTPUT_CHARS = 50000
#: How many trailing lines of an over-long output stay visible inline.
_LONG_OUTPUT_TAIL_LINES = 200
def _is_output_long(output: str) -> bool: def _is_output_long(output: str) -> bool:
"""Checks if command output exceeds the max threshold for inline display.""" """Whether command output is too long to show inline.
return len(output.splitlines()) > 500 or len(output) > 50000
The trigger is size alone: more than ``_LONG_OUTPUT_CHARS`` characters.
A line count by itself never trips it, so a 1000-line / 4 KB listing
(``seq 1 1000``) is shown in full instead of being cut to its tail.
"""
return len(output) > _LONG_OUTPUT_CHARS
def _format_long_output_response(_task_id: str, output: str, intro: str) -> str: def _format_long_output_response(task_id: str, output: str, intro: str) -> str:
"""Formats truncated output with guidance to use the analyzer tool.""" """Formats truncated output with guidance to use the analyzer tool."""
lines = output.splitlines() lines = output.splitlines()
last_200 = "\n".join(lines[-200:]) tail = "\n".join(lines[-_LONG_OUTPUT_TAIL_LINES:])
return ( return (
f"{intro}\nWarning: Output is too long ({len(lines)} lines," f"{intro}\nWarning: Output is too long ({len(lines)} lines,"
f" {len(output)} chars) and has been truncated.\n--- Last 200 lines of" f" {len(output)} chars) and has been truncated.\n--- Last"
f" output ---\n{last_200}\n------------------------\nYou can use the" f" {_LONG_OUTPUT_TAIL_LINES} lines of output ---\n{tail}\n"
" 'analyze_task_output' tool to get more information." f"------------------------\nThe full output is kept under TaskId"
f" {task_id}: use analyze_task_output with TaskId {task_id} to query it."
) )
def _render_output_body(output: str) -> str:
"""Renders command output for the model; whitespace-only output reads as empty."""
return output if output.strip() else "(empty output)"
def _abbreviate(text: str, limit: int = 40) -> str:
"""Cuts ``text`` to ``limit`` characters, marking the cut only when one happened."""
return text if len(text) <= limit else f"{text[:limit]}..."
def _terminal_label(terminal_id: str | None) -> str:
"""Renders a terminal id for the model; a task without one shows as ``-``."""
return terminal_id or "-"
def _filter_persistent_env(env: dict[str, str]) -> dict[str, str]: def _filter_persistent_env(env: dict[str, str]) -> dict[str, str]:
"""Drops the platform-provided variables from a captured shell environment.""" """Drops the platform-provided variables from a captured shell environment."""
kept: dict[str, str] = {} kept: dict[str, str] = {}
@@ -700,8 +729,7 @@ class RunAdbCommandTool(ArtemisTool):
intro += "." intro += "."
if run_persistent: if run_persistent:
intro += f" TerminalID: {terminal_id}." intro += f" TerminalID: {terminal_id}."
body = clean_output if clean_output.strip() else "(empty output)" return f"{intro}\nOutput:\n{_render_output_body(clean_output)}"
return f"{intro}\nOutput:\n{body}"
# We spawn the host subprocess: adb -s <serial> shell <script>. # We spawn the host subprocess: adb -s <serial> shell <script>.
# The script travels as an argument so the remote shell exits with it; # The script travels as an argument so the remote shell exits with it;
@@ -779,7 +807,7 @@ class RunAdbCommandTool(ArtemisTool):
registry.finished[task_id]["notified"] = True registry.finished[task_id]["notified"] = True
return _format_long_output_response(task_id, clean_output, intro) return _format_long_output_response(task_id, clean_output, intro)
return f"{intro}\nOutput:\n{clean_output}" return f"{intro}\nOutput:\n{_render_output_body(clean_output)}"
# pylint: disable=too-many-arguments,too-many-positional-arguments # pylint: disable=too-many-arguments,too-many-positional-arguments
async def _hand_off_to_background( async def _hand_off_to_background(
@@ -896,8 +924,8 @@ class ManageTaskTool(ArtemisTool):
active_tasks = [] active_tasks = []
for tid, t in registry.background.items(): for tid, t in registry.background.items():
active_tasks.append( active_tasks.append(
f"- {tid}: Command='{t.command[:40]}...', Phone" f"- {tid}: Command='{_abbreviate(t.command)}', Phone"
f" Cwd='{t.cwd}', TerminalID='{t.terminal_id}'," f" Cwd='{t.cwd}', TerminalID={_terminal_label(t.terminal_id)},"
f" Interactive={t.interactive}" f" Interactive={t.interactive}"
) )
if not active_tasks: if not active_tasks:
@@ -908,7 +936,7 @@ class ManageTaskTool(ArtemisTool):
finished_tasks = [] finished_tasks = []
for tid, t in registry.finished.items(): for tid, t in registry.finished.items():
finished_tasks.append( finished_tasks.append(
f"- {tid}: Command='{t['command'][:40]}...', Status='{t['status']}'" f"- {tid}: Command='{_abbreviate(t['command'])}', Status='{t['status']}'"
) )
if not finished_tasks: if not finished_tasks:
return "No recently finished tasks." return "No recently finished tasks."
@@ -924,7 +952,7 @@ class ManageTaskTool(ArtemisTool):
if action == "status": if action == "status":
task_info = _get_task_info(task_id, ctx) task_info = _get_task_info(task_id, ctx)
if not task_info: if not task_info:
return f"Task {task_id} not found." return ToolFailure(f"Error: Task {task_id} not found.")
output_text = task_info.get("output", "") output_text = task_info.get("output", "")
if _ENV_START_MARKER in output_text: if _ENV_START_MARKER in output_text:
@@ -935,7 +963,7 @@ class ManageTaskTool(ArtemisTool):
f"Status: {task_info.get('status')}\n" f"Status: {task_info.get('status')}\n"
f"Command: {task_info.get('command')}\n" f"Command: {task_info.get('command')}\n"
f"Cwd: {task_info.get('cwd')}\n" f"Cwd: {task_info.get('cwd')}\n"
f"TerminalID: {task_info.get('terminal_id')}" f"TerminalID: {_terminal_label(task_info.get('terminal_id'))}"
) )
exit_code = task_info.get("exit_code") exit_code = task_info.get("exit_code")
if exit_code is not None: if exit_code is not None:
@@ -948,16 +976,18 @@ class ManageTaskTool(ArtemisTool):
if action == "kill": if action == "kill":
if not task: if not task:
return f"Task {task_id} is not active or already finished." return ToolFailure(f"Error: Task {task_id} is not active or already finished.")
try: try:
await task.stop(ctx, status="killed") await task.stop(ctx, status="killed")
return f"Task {task_id} successfully terminated."
except Exception as e: # pylint: disable=broad-exception-caught except Exception as e: # pylint: disable=broad-exception-caught
return ToolFailure(f"Failed to terminate task {task_id}: {e}") return ToolFailure(f"Failed to terminate task {task_id}: {e}")
if task.exit_code is not None:
return f"Terminated task {task_id} (exit code {task.exit_code})."
return f"Terminated task {task_id}."
if action == "send_input": if action == "send_input":
if not task: if not task:
return f"Task {task_id} is not active." return ToolFailure(f"Error: Task {task_id} is not active.")
if not input_str: if not input_str:
return ToolFailure("Error: Input is required for send_input action.") return ToolFailure("Error: Input is required for send_input action.")
if not task.interactive or task.process.stdin is None: if not task.interactive or task.process.stdin is None:
@@ -968,7 +998,7 @@ class ManageTaskTool(ArtemisTool):
try: try:
task.process.stdin.write(input_str.encode()) task.process.stdin.write(input_str.encode())
await task.process.stdin.drain() await task.process.stdin.drain()
return f"Input successfully sent to task {task_id}." return f"Sent input to task {task_id}."
except Exception as e: # pylint: disable=broad-exception-caught except Exception as e: # pylint: disable=broad-exception-caught
return ToolFailure(f"Failed to send input: {e}") return ToolFailure(f"Failed to send input: {e}")
+30 -7
View File
@@ -209,6 +209,9 @@ class RegisteredCandidate:
bounds: tuple[int, int, int, int] | None = None bounds: tuple[int, int, int, int] | None = None
#: True when the candidate mapped onto an element that was already indexed. #: True when the candidate mapped onto an element that was already indexed.
reused: bool = False reused: bool = False
#: The text the agent's own indexed list shows for the reused element, so
#: the answer can name it the way the agent already knows it.
existing_text: str = ""
#: A bare-point candidate this close to an existing indexed center (in #: A bare-point candidate this close to an existing indexed center (in
@@ -331,6 +334,7 @@ def register_candidates(
description=cand.description, description=cand.description,
bounds=_element_pixel_bounds(existing), bounds=_element_pixel_bounds(existing),
reused=True, reused=True,
existing_text=str(existing.get("text") or ""),
) )
) )
continue continue
@@ -371,8 +375,20 @@ def register_candidates(
# --------------------------------------------------------------------------- # # --------------------------------------------------------------------------- #
def render_text(query: str, outcome: ExplorerOutcome, registered: list[RegisteredCandidate]) -> str: def render_text(
"""Plain-text answer for the calling agent.""" query: str,
outcome: ExplorerOutcome,
registered: list[RegisteredCandidate],
*,
index_targets: bool = False,
) -> str:
"""Plain-text answer for the calling agent.
``index_targets`` selects the targeting syntax the caller's ``click``
accepts: the Pro Operator and the Flash / MCP action executor take a bare
integer index next to coordinates; the Validator's text entry takes
coordinates only.
"""
if registered: if registered:
lines = [ lines = [
f"Explorer located {len(registered)} candidate(s) for '{query}' in the" f"Explorer located {len(registered)} candidate(s) for '{query}' in the"
@@ -381,12 +397,18 @@ def render_text(query: str, outcome: ExplorerOutcome, registered: list[Registere
for c in registered: for c in registered:
line = f"- [{c.index}] '{c.description}' at normalized [{c.coords[0]}, {c.coords[1]}]" line = f"- [{c.index}] '{c.description}' at normalized [{c.coords[0]}, {c.coords[1]}]"
if c.reused: if c.reused:
line += " (already in your indexed list)" line += f" (matches your indexed element [{c.index}] '{c.existing_text}')"
lines.append(line) lines.append(line)
if index_targets:
lines.append( lines.append(
"Act on a candidate by its index (e.g. target=[index]) or by the" "Act on a candidate by its index (target=3, a bare integer) or by its"
" normalized coordinate. Candidates are ranked by confidence." " normalized coordinate. Candidates are ranked by confidence."
) )
else:
lines.append(
"Act on a candidate by its normalized coordinate (target=[x, y]) with a"
" target_description. Candidates are ranked by confidence."
)
if outcome.message: if outcome.message:
lines.append(f"Explorer notes: {outcome.message}") lines.append(f"Explorer notes: {outcome.message}")
return "\n".join(lines) return "\n".join(lines)
@@ -444,7 +466,7 @@ def render_operator_blocks(
registered: list[RegisteredCandidate], registered: list[RegisteredCandidate],
) -> str | list[dict[str, Any]]: ) -> str | list[dict[str, Any]]:
"""Multimodal answer for the Operator: text plus the annotated screenshot.""" """Multimodal answer for the Operator: text plus the annotated screenshot."""
text = render_text(query, outcome, registered) text = render_text(query, outcome, registered, index_targets=True)
screenshot_path = getattr(state, "latest_screenshot", None) screenshot_path = getattr(state, "latest_screenshot", None)
image_path = ( image_path = (
render_annotated_image(ctx, screenshot_path, registered) if screenshot_path else None render_annotated_image(ctx, screenshot_path, registered) if screenshot_path else None
@@ -475,10 +497,11 @@ async def ask_explorer_text(
*, *,
agent_name: str = "validator", agent_name: str = "validator",
) -> str: ) -> str:
"""Text-only entry used by the Validator / Flash runner and MCP executors.""" """Text-only, coordinate-only entry used by the Validator (Flash and the MCP
executor reach the explorer through ``McpActionExecutor`` with index targets)."""
outcome = await locate(ctx, state, query, context_feedback, agent_name=agent_name) outcome = await locate(ctx, state, query, context_feedback, agent_name=agent_name)
registered = register_candidates(ctx, state, outcome) registered = register_candidates(ctx, state, outcome)
return render_text(query, outcome, registered) return render_text(query, outcome, registered, index_targets=False)
class AskExplorerTool(ArtemisTool): class AskExplorerTool(ArtemisTool):
+8 -4
View File
@@ -15,9 +15,9 @@
"""replay_steps: full replay of a step range, exactly as the executing agent saw it. """replay_steps: full replay of a step range, exactly as the executing agent saw it.
Each step is rendered with :func:`~artemis.utils.task_tree.render_step_replay` Each step is rendered with :func:`~artemis.utils.task_tree.render_step_replay`
(``[Screen]`` description, reasoning, every non-internal tool call with name / (``T+mm:ss`` step header, ``[Screen]`` description, reasoning, every
arguments / result, the planned action, safety-net interception and the non-internal tool call with name / arguments / result, the executed action,
execution result). Two loose bounds keep one tool result finite (config safety-net interception and the execution result). Two loose bounds keep one tool result finite (config
``agent.memory.replay``): at most ``max_steps`` steps per call, and a ``agent.memory.replay``): at most ``max_steps`` steps per call, and a
``max_tokens`` budget applied by dropping *whole trailing steps* — a replayed ``max_tokens`` budget applied by dropping *whole trailing steps* — a replayed
step is never cut mid-sentence. Screenshots are deliberately not part of the step is never cut mid-sentence. Screenshots are deliberately not part of the
@@ -79,6 +79,10 @@ def replay_steps_text(
return ToolFailure(f"Error: no recorded steps in range {start}–{end}.") return ToolFailure(f"Error: no recorded steps in range {start}–{end}.")
budget_tokens = max(1, int(max_tokens or DEFAULT_MAX_TOKENS)) budget_tokens = max(1, int(max_tokens or DEFAULT_MAX_TOKENS))
# Step headers carry the session clock (``T+mm:ss``) every agent prompt
# and search_history use; without a session start they fall back to the
# stored relative time.
session_start = getattr(reader, "session_start_time", None)
rendered: list[str] = [] rendered: list[str] = []
shown: list[dict] = [] shown: list[dict] = []
used = 0 used = 0
@@ -88,7 +92,7 @@ def replay_steps_text(
dropped += 1 dropped += 1
continue continue
try: try:
text = render_step_replay(step) text = render_step_replay(step, session_start=session_start)
except Exception as e: except Exception as e:
return ToolFailure(f"Error rendering steps {start}–{end}: {e}") return ToolFailure(f"Error rendering steps {start}–{end}: {e}")
cost = _estimate_tokens(text) + 1 cost = _estimate_tokens(text) + 1
+21 -10
View File
@@ -29,6 +29,7 @@ from pathlib import Path
import sqlite3 import sqlite3
from typing import Any from typing import Any
from artemis.core.tool_failure import ToolFailure
from artemis.utils.logger import get_logger from artemis.utils.logger import get_logger
from artemis.utils.visualization import overlay_action_on_screenshot from artemis.utils.visualization import overlay_action_on_screenshot
@@ -47,10 +48,18 @@ class ScreenshotResult:
image_bytes: bytes | None = None image_bytes: bytes | None = None
#: True when ``which == "overlay"`` and the action was actually drawn. #: True when ``which == "overlay"`` and the action was actually drawn.
overlay_drawn: bool = False overlay_drawn: bool = False
#: True when the lookup could not serve the request (bad arguments, no
#: history, unknown step, unreadable file). A step that simply has no
#: screenshot of the requested kind is an answer, not an error.
is_error: bool = False
def to_content_blocks(self) -> list[dict[str, Any]] | str: def to_content_blocks(self) -> list[dict[str, Any]] | str:
"""The tool's return value: content blocks with the image, or the """The tool's return value: content blocks with the image, the
plain-text explanation when there is no image to attach.""" plain-text answer when there is no image to attach, or a
:class:`ToolFailure` (status ``error`` downstream) when the lookup
failed."""
if self.is_error:
return ToolFailure(self.description)
if not self.image_bytes: if not self.image_bytes:
return self.description return self.description
encoded = base64.b64encode(self.image_bytes).decode("utf-8") encoded = base64.b64encode(self.image_bytes).decode("utf-8")
@@ -60,41 +69,43 @@ class ScreenshotResult:
] ]
def _failure(step_number: int, which: str, description: str) -> ScreenshotResult:
return ScreenshotResult(step_number, which, description, is_error=True)
def load_step_screenshot(reader: Any, step_number: Any, which: str = "pre") -> ScreenshotResult: def load_step_screenshot(reader: Any, step_number: Any, which: str = "pre") -> ScreenshotResult:
"""Loads (and for ``overlay`` annotates) one step screenshot from ``reader``.""" """Loads (and for ``overlay`` annotates) one step screenshot from ``reader``."""
which = str(which or "pre").lower() which = str(which or "pre").lower()
try: try:
number = int(step_number) number = int(step_number)
except (TypeError, ValueError): except (TypeError, ValueError):
return ScreenshotResult( return _failure(-1, which, f"Error: step_number must be an integer, got {step_number!r}.")
-1, which, f"Error: step_number must be an integer, got {step_number!r}."
)
if which not in WHICH_CHOICES: if which not in WHICH_CHOICES:
return ScreenshotResult( return _failure(
number, which, f"Error: 'which' must be 'pre', 'post' or 'overlay', got '{which}'." number, which, f"Error: 'which' must be 'pre', 'post' or 'overlay', got '{which}'."
) )
if reader is None: if reader is None:
return ScreenshotResult(number, which, "Error: no execution history available.") return _failure(number, which, "Error: no execution history available.")
source = "pre" if which == "overlay" else which source = "pre" if which == "overlay" else which
try: try:
path = reader.get_step_image_path(number, source) path = reader.get_step_image_path(number, source)
except (sqlite3.Error, OSError, ValueError) as e: except (sqlite3.Error, OSError, ValueError) as e:
return ScreenshotResult(number, which, f"Error looking up step {number}: {e}") return _failure(number, which, f"Error looking up step {number}: {e}")
if path is None: if path is None:
try: try:
record = reader.get_step_record(number) record = reader.get_step_record(number)
except (sqlite3.Error, ValueError): except (sqlite3.Error, ValueError):
record = None record = None
if record is None: if record is None:
return ScreenshotResult(number, which, f"Error: step {number} not found.") return _failure(number, which, f"Error: step {number} not found.")
return ScreenshotResult( return ScreenshotResult(
number, which, f"No {source}-action screenshot recorded for step {number}." number, which, f"No {source}-action screenshot recorded for step {number}."
) )
try: try:
image_bytes = Path(path).read_bytes() image_bytes = Path(path).read_bytes()
except OSError as e: except OSError as e:
return ScreenshotResult(number, which, f"Error reading screenshot: {e}") return _failure(number, which, f"Error reading screenshot: {e}")
if which != "overlay": if which != "overlay":
return ScreenshotResult( return ScreenshotResult(
+165 -14
View File
@@ -30,6 +30,12 @@ Hard boundaries (config ``agent.memory.recall``):
- every result carries a step number / step id; - every result carries a step number / step id;
- large raw sources return only excerpts around the match plus a reference. - large raw sources return only excerpts around the match plus a reference.
Two surfaces per step: the *scoring* haystack keeps the raw record (action
JSON, execution result JSON, UI-tree XML) so resource ids, bounds and package
names stay searchable; the *excerpt* shown as ``Match:`` is built from the
readable twin (the ledger action phrase, result status/error words, element
text / content descriptions / OCR text) so the model never reads JSON.
A ``step_range`` additionally returns the full-width ``build_action_ledger`` A ``step_range`` additionally returns the full-width ``build_action_ledger``
rows of that range — the re-entry point for compressed-history marker lines. rows of that range — the re-entry point for compressed-history marker lines.
""" """
@@ -104,6 +110,15 @@ def result_search_text(result: Any) -> str:
the haystack via :func:`_screen_text`), so base64 never enters the search.""" the haystack via :func:`_screen_text`), so base64 never enters the search."""
if result is None: if result is None:
return "" return ""
# A content-block list stored as its JSON string (trace serialization)
# is unwrapped first so its text blocks render, not the JSON.
if isinstance(result, str) and result.lstrip().startswith("[{"):
try:
parsed = json.loads(result)
except ValueError:
parsed = None
if isinstance(parsed, list) and any(isinstance(b, dict) and "type" in b for b in parsed):
return result_search_text(parsed)
if isinstance(result, list): if isinstance(result, list):
texts = [ texts = [
str(block.get("text") or "") str(block.get("text") or "")
@@ -135,23 +150,32 @@ def _events_text(step: dict) -> str:
return "\n".join(parts) return "\n".join(parts)
def _action_text(action_taken: Any) -> str: def _rendered_action(action_taken: Any) -> str:
"""Combine the rendered action with raw fields for search. """The shared ledger phrase of a step's action (``format_actions_clean``)."""
Rendered excerpts identify self-described targets. Raw JSON also makes
resource IDs, bounds, and other fields searchable.
"""
if not action_taken: if not action_taken:
return "" return ""
try: try:
rendered = format_actions_clean(action_taken) return format_actions_clean(action_taken)
except Exception: except Exception:
rendered = "" return ""
def _action_text(action_taken: Any) -> str:
"""Combine the rendered action with raw fields for *scoring*.
Rendered text identifies self-described targets. Raw JSON also makes
resource IDs, bounds, and other fields searchable. Never shown as-is:
the excerpt comes from :func:`_step_display_text`.
"""
if not action_taken:
return ""
rendered = _rendered_action(action_taken)
raw = json.dumps(action_taken, ensure_ascii=False, default=str) raw = json.dumps(action_taken, ensure_ascii=False, default=str)
return f"{rendered}\n{raw}" if rendered else raw return f"{rendered}\n{raw}" if rendered else raw
def _step_haystack(step: dict) -> str: def _step_haystack(step: dict) -> str:
"""The scoring surface of a step: every recorded field, raw JSON included."""
extra = step.get("extra_metadata") or {} extra = step.get("extra_metadata") or {}
fields = [ fields = [
str(step.get("summary") or ""), str(step.get("summary") or ""),
@@ -170,6 +194,70 @@ def _step_haystack(step: dict) -> str:
return "\n".join(f for f in fields if f) return "\n".join(f for f in fields if f)
def _result_display_text(result: Any) -> str:
"""The execution result as words: its status and result/error strings
(an execution incident renders as its one-line phrase), never JSON."""
if not result:
return ""
if not isinstance(result, dict):
return str(result)
parts: list[str] = []
status = result.get("status")
if status:
parts.append(f"status {status}")
try:
from artemis.utils.task_tree import format_result_clean
incident_or_error = format_result_clean(result)
except Exception:
incident_or_error = None
if incident_or_error:
parts.append(str(incident_or_error))
else:
for key in ("result", "error", "error_msg", "message"):
value = result.get(key)
if isinstance(value, str) and value.strip():
parts.append(value.strip())
return " | ".join(parts)
def _events_display_text(step: dict) -> str:
"""Tool calls and reasoning of a step as the replay shows them (name,
arguments, result text), without the raw event JSON."""
parts: list[str] = []
for event in step.get("interleaved_events") or []:
etype = event.get("type")
if etype == "tool_call":
name = str(event.get("name") or "")
args = event.get("args") or {}
result_text = result_search_text(event.get("result"))
try:
call = f"{name}({json.dumps(args, ensure_ascii=False, default=str)})"
except (TypeError, ValueError):
call = name
parts.append(f"{call} -> {result_text}" if result_text else call)
elif etype and "thought" in etype:
parts.append(str(event.get("content") or ""))
return "\n".join(p for p in parts if p)
def _step_display_text(step: dict) -> str:
"""The *excerpt* surface of a step: the same fields as the haystack, as the
agent would read them. The summary is left out — it already heads the
result as the ``Screen:`` line."""
extra = step.get("extra_metadata") or {}
fields = [
_rendered_action(step.get("action_taken")),
_result_display_text(step.get("last_execution_result")),
str(step.get("operator_raw_thinking") or ""),
_events_display_text(step),
str(extra.get("injected_instruction") or ""),
str(extra.get("foreground_app") or ""),
]
# Excerpts flatten whitespace, so the field boundary must survive as text.
return " | ".join(f.strip() for f in fields if f and f.strip())
def _image_text(reader: Any, image_name: str | None) -> str: def _image_text(reader: Any, image_name: str | None) -> str:
"""Raw XML/OCR text stored for one screenshot — the textual description """Raw XML/OCR text stored for one screenshot — the textual description
of that image (package/activity strings live inside the XML blob).""" of that image (package/activity strings live inside the XML blob)."""
@@ -212,9 +300,37 @@ def referenced_image_names(step: dict) -> list[str]:
return names return names
def _screen_text(reader: Any, step: dict) -> str: #: UI-tree node attributes whose values are human-readable screen content.
"""Description text of the screenshots behind a step: its own pre- and _UI_NODE_TEXT_KEYS = ("text", "content-desc", "content_desc", "resource-id", "resource_id")
post-action screenshots plus any screenshot a tool result embedded."""
def _collect_screen_strings(value: Any, out: list[str], depth: int = 0) -> None:
"""Walks a stored UI tree / OCR blob and appends its readable values
(element text, content description, resource id, OCR text) in order."""
if depth > 50:
return
if isinstance(value, str):
text = " ".join(value.split())
if text and text not in out:
out.append(text)
elif isinstance(value, dict):
for key in _UI_NODE_TEXT_KEYS:
text = value.get(key)
if isinstance(text, str):
text = " ".join(text.split())
if text and text not in out:
out.append(text)
for child_key in ("children", "nodes", "elements"):
children = value.get(child_key)
if isinstance(children, (list, tuple)):
for child in children:
_collect_screen_strings(child, out, depth + 1)
elif isinstance(value, (list, tuple)):
for child in value:
_collect_screen_strings(child, out, depth + 1)
def _screen_names(step: dict) -> list[str]:
names: list[str] = [] names: list[str] = []
for name in ( for name in (
step.get("pre_image_name"), step.get("pre_image_name"),
@@ -223,7 +339,32 @@ def _screen_text(reader: Any, step: dict) -> str:
): ):
if name and name not in names: if name and name not in names:
names.append(name) names.append(name)
return "\n".join(t for t in (_image_text(reader, n) for n in names) if t) return names
def _screen_text(reader: Any, step: dict) -> str:
"""Description text of the screenshots behind a step (its own pre- and
post-action screenshots plus any screenshot a tool result embedded) —
the raw XML/OCR blobs, for *scoring* only."""
return "\n".join(t for t in (_image_text(reader, n) for n in _screen_names(step)) if t)
def _screen_display_text(reader: Any, step: dict) -> str:
"""The readable content of the same screenshots for the *excerpt*: element
text / content descriptions / resource ids and OCR text joined by
``" | "``, never the UI-tree JSON."""
strings: list[str] = []
for name in _screen_names(step):
try:
record = reader.storage.get_image(name)
except (sqlite3.Error, ValueError, AttributeError):
continue
if record is None:
continue
for blob in (record.ui_tree, record.ocr_result):
if blob:
_collect_screen_strings(blob, strings)
return " | ".join(strings)
def _step_label(step: dict, session_start: float | None) -> str: def _step_label(step: dict, session_start: float | None) -> str:
@@ -321,12 +462,22 @@ def search_history_text(
total = score + screen_score total = score + screen_score
if total <= 0: if total <= 0:
continue continue
source = haystack if score >= screen_score else screen
lines = [f"[{_step_label(step, session_start)} | id {step.get('step_id')}]"] lines = [f"[{_step_label(step, session_start)} | id {step.get('step_id')}]"]
summary = step.get("summary") summary = step.get("summary")
if summary: if summary:
lines.append(f" Screen: {_clamp(str(summary), EXCERPT_CHARS)}") lines.append(f" Screen: {_clamp(str(summary), EXCERPT_CHARS)}")
lines.append(f" Match: {_excerpt(source, terms, EXCERPT_CHARS)}") # The excerpt comes from the readable twin of whichever surface
# scored higher; the raw JSON only ever scores. When the hit sits
# in the summary alone, the Screen line already shows it.
display = (
_step_display_text(step)
if score >= screen_score
else _screen_display_text(reader, step)
)
display_hit = _score(display, terms) > 0
summary_hit = bool(summary) and _score(str(summary), terms) > 0
if display and (display_hit or not summary_hit):
lines.append(f" Match: {_excerpt(display, terms, EXCERPT_CHARS)}")
number = step.get("step_number") or 0 number = step.get("step_number") or 0
results.append((total, number, "\n".join(lines))) results.append((total, number, "\n".join(lines)))
+24 -1
View File
@@ -29,6 +29,29 @@ from artemis.utils.logger import get_logger
logger = get_logger(__name__) logger = get_logger(__name__)
#: How many line ranges a merged block's header lists before it is cut.
_MAX_HEADER_RANGES = 8
def _compress_line_numbers(indices: list[int]) -> str:
"""Renders sorted line numbers as ranges: ``6``, ``69-79``, ``699-798``.
Consecutive runs collapse into ``start-end``; at most
``_MAX_HEADER_RANGES`` ranges are listed, the rest replaced by ``...``.
"""
runs: list[str] = []
start = prev = indices[0]
for idx in indices[1:]:
if idx == prev + 1:
prev = idx
continue
runs.append(str(start) if start == prev else f"{start}-{prev}")
start = prev = idx
runs.append(str(start) if start == prev else f"{start}-{prev}")
if len(runs) > _MAX_HEADER_RANGES:
return ", ".join(runs[:_MAX_HEADER_RANGES]) + ", ..."
return ", ".join(runs)
# pylint: disable=too-many-branches,too-many-locals,too-many-statements # pylint: disable=too-many-branches,too-many-locals,too-many-statements
def search_and_merge_logs( def search_and_merge_logs(
@@ -99,7 +122,7 @@ def search_and_merge_logs(
truncated = True truncated = True
break break
match_str = ", ".join(map(str, match_indices)) match_str = _compress_line_numbers(match_indices)
matches.append(f"--- Match at line(s) {match_str} ---") matches.append(f"--- Match at line(s) {match_str} ---")
for idx in range(start, end): for idx in range(start, end):
line_content = log_lines[idx] line_content = log_lines[idx]
+11 -11
View File
@@ -148,7 +148,7 @@ class SaveNoteTool(ArtemisTool):
else "." else "."
) )
save_note_content(base_dir, k, c) save_note_content(base_dir, k, c)
agent_outcome = f"Successfully saved note to {k}.md." agent_outcome = f"Saved note '{k}'."
status = "success" status = "success"
except Exception as e: # pylint: disable=broad-exception-caught except Exception as e: # pylint: disable=broad-exception-caught
agent_outcome = f"Failed to save note {k}.md: {e}" agent_outcome = f"Failed to save note {k}.md: {e}"
@@ -227,7 +227,7 @@ class AppendNoteTool(ArtemisTool):
else "." else "."
) )
append_note_content(base_dir, k, c) append_note_content(base_dir, k, c)
agent_outcome = f"Successfully appended note to {k}.md." agent_outcome = f"Appended to note '{k}'."
status = "success" status = "success"
except Exception as e: # pylint: disable=broad-exception-caught except Exception as e: # pylint: disable=broad-exception-caught
agent_outcome = f"Failed to append note {k}.md: {e}" agent_outcome = f"Failed to append note {k}.md: {e}"
@@ -309,9 +309,9 @@ class UpdateNoteTool(ArtemisTool):
) )
warning = update_note_content(base_dir, k, t, r) warning = update_note_content(base_dir, k, t, r)
if warning: if warning:
agent_outcome = f"Successfully updated note '{k}'.\nWARNING: {warning}" agent_outcome = f"Updated note '{k}'.\nWARNING: {warning}"
else: else:
agent_outcome = f"Successfully updated note '{k}'." agent_outcome = f"Updated note '{k}'."
status = "success" status = "success"
except Exception as e: # pylint: disable=broad-exception-caught except Exception as e: # pylint: disable=broad-exception-caught
agent_outcome = f"Failed to update note '{k}': {e}" agent_outcome = f"Failed to update note '{k}': {e}"
@@ -642,7 +642,7 @@ class SaveNotePureTool(ArtemisTool):
else "." else "."
) )
save_note_content(base_dir, k, c) save_note_content(base_dir, k, c)
return f"Successfully saved note to {k}.md." return f"Saved note '{k}'."
except Exception as e: # pylint: disable=broad-exception-caught except Exception as e: # pylint: disable=broad-exception-caught
return ToolFailure(f"Failed to save note {k}.md: {e}") return ToolFailure(f"Failed to save note {k}.md: {e}")
@@ -709,8 +709,8 @@ class UpdateNotePureTool(ArtemisTool):
) )
warning = update_note_content(base_dir, k, t, r) warning = update_note_content(base_dir, k, t, r)
if warning: if warning:
return f"Successfully updated note '{k}'.\nWARNING: {warning}" return f"Updated note '{k}'.\nWARNING: {warning}"
return f"Successfully updated note '{k}'." return f"Updated note '{k}'."
except Exception as e: # pylint: disable=broad-exception-caught except Exception as e: # pylint: disable=broad-exception-caught
return ToolFailure(f"Failed to update note '{k}': {e}") return ToolFailure(f"Failed to update note '{k}': {e}")
@@ -774,7 +774,7 @@ class AppendNotePureTool(ArtemisTool):
else "." else "."
) )
append_note_content(base_dir, k, c) append_note_content(base_dir, k, c)
return f"Successfully appended note to {k}.md." return f"Appended to note '{k}'."
except Exception as e: # pylint: disable=broad-exception-caught except Exception as e: # pylint: disable=broad-exception-caught
return ToolFailure(f"Failed to append note {k}.md: {e}") return ToolFailure(f"Failed to append note {k}.md: {e}")
@@ -791,7 +791,7 @@ def get_append_note_tool_pure(ctx: ArtemisContext) -> BaseTool:
save_note_wrapper = ToolWrapper( save_note_wrapper = ToolWrapper(
tool_fn_getter=get_save_note_tool, tool_fn_getter=get_save_note_tool,
on_success_fn=lambda key: f"Successfully saved note '{key}'.", on_success_fn=lambda key: f"Saved note '{key}'.",
on_failure_fn=lambda key: f"Failed to save note '{key}'.", on_failure_fn=lambda key: f"Failed to save note '{key}'.",
) )
@@ -810,13 +810,13 @@ list_notes_wrapper = ToolWrapper(
update_note_wrapper = ToolWrapper( update_note_wrapper = ToolWrapper(
tool_fn_getter=get_update_note_tool, tool_fn_getter=get_update_note_tool,
on_success_fn=lambda key: f"Successfully updated note '{key}'.", on_success_fn=lambda key: f"Updated note '{key}'.",
on_failure_fn=lambda key: f"Failed to update note '{key}'.", on_failure_fn=lambda key: f"Failed to update note '{key}'.",
) )
append_note_wrapper = ToolWrapper( append_note_wrapper = ToolWrapper(
tool_fn_getter=get_append_note_tool, tool_fn_getter=get_append_note_tool,
on_success_fn=lambda key: f"Successfully appended note '{key}'.", on_success_fn=lambda key: f"Appended to note '{key}'.",
on_failure_fn=lambda key: f"Failed to append note '{key}'.", on_failure_fn=lambda key: f"Failed to append note '{key}'.",
) )
+1 -1
View File
@@ -63,7 +63,7 @@ class VideoAnalyzerArgs(BaseModel):
OPERATOR_VIDEO_ANALYZER_DOCSTRING = ( OPERATOR_VIDEO_ANALYZER_DOCSTRING = (
"[VIDEO] Use this tool to analyze any specific time range of the video" "[VIDEO] Use this tool to analyze any specific time range of the video"
" recording. This tool delegates to a powerful video-analyzing subagent" " recording. This tool delegates to a video analysis subagent"
" with full access to the continuous screen recording of your entire" " with full access to the continuous screen recording of your entire"
" execution session.\n\n" " execution session.\n\n"
"- This tool overcomes your limitation of only receiving discrete" "- This tool overcomes your limitation of only receiving discrete"
+178 -40
View File
@@ -15,6 +15,8 @@
"""Utilities for handling app locking and initial app launch logic.""" """Utilities for handling app locking and initial app launch logic."""
import asyncio import asyncio
from dataclasses import dataclass, field
import re
from artemis.context import AppLaunchResult, ArtemisContext from artemis.context import AppLaunchResult, ArtemisContext
from artemis.controllers.platform_specific_commands_controller import ( from artemis.controllers.platform_specific_commands_controller import (
@@ -28,43 +30,180 @@ from artemis.utils.logger import get_logger
logger = get_logger(__name__) logger = get_logger(__name__)
SYSTEM_OVERLAYS_VALID_FOR_LAUNCH = { # ``ActivityRecord{<hash> u<user> <package>/<activity> t<task id>}`` as printed by
"com.google.android.permissioncontroller", # ``dumpsys activity activities`` on every line that names an activity.
"com.android.permissioncontroller", _ACTIVITY_RECORD_RE = re.compile(
"com.google.android.packageinstaller", r"ActivityRecord\{[0-9a-f]+ u\d+ (?P<package>[\w.]+)/(?P<activity>[\w.$]+) t(?P<task>\d+)\}"
"com.android.packageinstaller", )
} # ``* Task{<hash> #<id> type=... A=<uid>:<affinity> ... visible=true ...}`` task headers
# (``* TaskRecord{<hash> #<id> A=<affinity> ...}`` on Android 11 and older).
_TASK_HEADER_RE = re.compile(r"^\s*\* Task(?:Record)?\{[0-9a-f]+ #(?P<task>\d+)(?P<attrs>[^}]*)\}")
# Device-side pre-filter: the full dump runs to hundreds of KB on a busy device;
# only the task headers, history entries and resumed-activity lines are parsed.
_FOREGROUND_DUMP_CMD = (
"dumpsys activity activities | grep -E"
r" '^ *\* (Task|TaskRecord)\{|^ *\* Hist |ResumedActivity'"
)
# ``* Hist #<index>: ActivityRecord{...}`` entries; index 0 is the task's base activity.
_HIST_RE = re.compile(r"^\s*\* Hist\s+#(?P<index>\d+): (?P<record>ActivityRecord\{[^}]*\})")
# ``topResumedActivity=ActivityRecord{...}`` (per task, top task first) and the global
# ``ResumedActivity: ActivityRecord{...}`` line; the first one seen is the foreground one.
_RESUMED_RE = re.compile(r"ResumedActivity[:=]\s*(?P<record>ActivityRecord\{[^}]*\})")
def get_focused_task_package(ctx: ArtemisContext) -> str | None: @dataclass(frozen=True)
"""Inspect the active/focused task stack to find the package affinity of the top task. class ForegroundTask:
"""What ``dumpsys activity activities`` says is on top of the display.
Uses adb shell dumpsys activity activities to extract this info. ``resumed_package`` is the package of the resumed activity (what the user sees).
``base_package`` is the package of the task's root activity (``Hist #0``), and
``packages`` holds every package that has an activity in that task, so an app whose
task currently shows a helper activity from another package (Settings search,
permission dialogs, account pickers, system update pages, ...) is still
recognised as the foreground app.
"""
task_id: int | None = None
affinity: str | None = None
resumed_component: str | None = None
resumed_package: str | None = None
base_package: str | None = None
packages: frozenset[str] = field(default_factory=frozenset)
def owns(self, app_package: str) -> bool:
"""Whether ``app_package`` is the app the foreground task belongs to.
The task affinity is the app's own claim on the task (by default the
package name, or a package-prefixed name such as
``com.android.settings.root``); it stays with the task even when the
app's root activity has been destroyed and only a companion activity
(e.g. the Settings search UI) remains in the history."""
affinity = self.affinity or ""
return (
app_package == self.resumed_package
or app_package == self.base_package
or app_package in self.packages
or affinity == app_package
or affinity.startswith(app_package + ".")
)
def describe(self) -> str:
task = f"task #{self.task_id}" if self.task_id is not None else "task ?"
details = [f"base={self.base_package}", f"resumed={self.resumed_component}"]
if self.affinity:
details.append(f"affinity={self.affinity}")
others = sorted(self.packages - {self.base_package, self.resumed_package, None})
if others:
details.append(f"also={','.join(others)}")
return f"{task} ({', '.join(details)})"
def parse_foreground_task(dump: str) -> ForegroundTask | None:
"""Parse ``dumpsys activity activities`` output into the foreground task summary.
The foreground task is the one holding the resumed activity; when no activity is
resumed (mid-transition) the first visible task with history entries is used.
Returns None when the dump contains no task information at all.
"""
task_attrs: dict[int, str] = {}
task_order: list[int] = []
task_activities: dict[int, list[tuple[int, str, str]]] = {}
resumed: re.Match[str] | None = None
for line in dump.splitlines():
header = _TASK_HEADER_RE.match(line)
if header:
task_id = int(header.group("task"))
if task_id not in task_attrs:
task_attrs[task_id] = header.group("attrs")
task_order.append(task_id)
continue
hist = _HIST_RE.match(line)
if hist:
record = _ACTIVITY_RECORD_RE.search(hist.group("record"))
if record:
task_activities.setdefault(int(record.group("task")), []).append(
(int(hist.group("index")), record.group("package"), record.group("activity"))
)
continue
if resumed is None:
match = _RESUMED_RE.search(line)
if match:
resumed = _ACTIVITY_RECORD_RE.search(match.group("record"))
if not task_attrs and resumed is None:
return None
task_id: int | None = None
if resumed is not None:
task_id = int(resumed.group("task"))
else:
for candidate in task_order:
if "visible=true" in task_attrs[candidate] and candidate in task_activities:
task_id = candidate
break
activities = sorted(task_activities.get(task_id, []))
affinity_match = re.search(r"\bA=(?:\d+:)?(\S+)", task_attrs.get(task_id, ""))
return ForegroundTask(
task_id=task_id,
affinity=affinity_match.group(1) if affinity_match else None,
resumed_component=(
f"{resumed.group('package')}/{resumed.group('activity')}" if resumed else None
),
resumed_package=resumed.group("package") if resumed else None,
base_package=activities[0][1] if activities else None,
packages=frozenset(package for _, package, _ in activities),
)
def get_foreground_task(ctx: ArtemisContext) -> ForegroundTask | None:
"""Read the foreground task from ``dumpsys activity activities`` on the device.
Blocking (a synchronous ADB shell round-trip); async callers go through
:func:`get_foreground_task_async` so the event loop keeps serving.
""" """
try: try:
device = get_adb_device(ctx) device = get_adb_device(ctx)
output = str(device.shell("dumpsys activity activities | grep topDisplayFocusedRootTask")) if device is None:
if "topDisplayFocusedRootTask=" in output:
segment = output.split("topDisplayFocusedRootTask=")[-1]
if "A=" in segment:
affinity = segment.split("A=")[-1].split("}")[0].split()[0].strip()
if ":" in affinity:
return affinity.split(":")[-1]
return affinity
if "{" in segment and "}" in segment:
braces_content = segment.split("{")[-1].split("}")[0]
tokens = braces_content.split()
for token in tokens:
if "." in token and not token.startswith("#") and not token.isdigit():
if ":" in token:
return token.split(":")[-1]
return token
return None return None
dump = str(device.shell(_FOREGROUND_DUMP_CMD))
if "Task" not in dump:
# No grep on the device (or nothing matched): parse the full dump.
dump = str(device.shell("dumpsys activity activities"))
return parse_foreground_task(dump)
except Exception as e: except Exception as e:
logger.debug(f"Failed to retrieve focused task package via dumpsys: {e}") logger.debug(f"Failed to retrieve foreground task via dumpsys: {e}")
return None return None
async def get_foreground_task_async(ctx: ArtemisContext) -> ForegroundTask | None:
""":func:`get_foreground_task` off the event loop."""
return await asyncio.to_thread(get_foreground_task, ctx)
async def _observe_foreground(
ctx: ArtemisContext, app_package: str
) -> tuple[bool, str | None, ForegroundTask | None]:
"""Report whether ``app_package`` is in the foreground.
Fast path: the focused window's package equals the target. Otherwise the app also
counts as foreground when the top task belongs to it (see ``ForegroundTask``).
A ``None`` focused package means the window manager is mid-transition and is
reported as "not yet" without consulting the task stack.
Returns (is_foreground, focused_package, foreground_task); the task is only read
when the fast path fails on a non-null focused package.
"""
current_package = await get_current_foreground_package_async(ctx)
if current_package == app_package:
return True, current_package, None
if current_package is None:
return False, None, None
task = await get_foreground_task_async(ctx)
return task is not None and task.owns(app_package), current_package, task
async def _poll_for_app_ready( async def _poll_for_app_ready(
ctx: ArtemisContext, ctx: ArtemisContext,
app_package: str, app_package: str,
@@ -88,20 +227,16 @@ async def _poll_for_app_ready(
polls = int(max_poll_seconds / poll_interval) polls = int(max_poll_seconds / poll_interval)
for i in range(polls): for i in range(polls):
current_package = await get_current_foreground_package_async(ctx) ready, current_package, task = await _observe_foreground(ctx, app_package)
if ready:
if current_package == app_package: if current_package == app_package:
logger.success(f"App {app_package} is ready (took ~{i * poll_interval:.1f}s)") logger.success(f"App {app_package} is ready (took ~{i * poll_interval:.1f}s)")
return True, None else:
if current_package in SYSTEM_OVERLAYS_VALID_FOR_LAUNCH:
task_package = get_focused_task_package(ctx)
if task_package == app_package:
logger.success( logger.success(
f"App {app_package} launch succeeded (system overlay" f"App {app_package} is ready (focused window belongs to"
f" '{current_package}' focused, but top task stack affinity" f" '{current_package}', but the foreground {task.describe()} is owned"
f" matches expected app package '{task_package}', took" f" by {app_package}, took ~{i * poll_interval:.1f}s)"
f" ~{i * poll_interval:.1f}s)"
) )
return True, None return True, None
@@ -110,16 +245,19 @@ async def _poll_for_app_ready(
else: else:
logger.debug( logger.debug(
f"Poll {i + 1}/{polls}: Wrong app in foreground (expected" f"Poll {i + 1}/{polls}: Wrong app in foreground (expected"
f" '{app_package}', got '{current_package}'). Still waiting..." f" '{app_package}', got '{current_package}', foreground"
f" {task.describe() if task else 'task unknown'}). Still waiting..."
) )
if i < polls - 1: if i < polls - 1:
await asyncio.sleep(poll_interval) await asyncio.sleep(poll_interval)
current_package = await get_current_foreground_package_async(ctx) current_package = await get_current_foreground_package_async(ctx)
task = await get_foreground_task_async(ctx)
error_msg = ( error_msg = (
f"Timeout waiting for {app_package} to load after {max_poll_seconds}s. " f"Timeout waiting for {app_package} to load after {max_poll_seconds}s. "
f"Current foreground: {current_package}" f"Current foreground: {current_package}; foreground"
f" {task.describe() if task else 'task unknown'}"
) )
logger.error(error_msg) logger.error(error_msg)
return False, error_msg return False, error_msg
@@ -225,10 +363,10 @@ async def _handle_initial_app_launch(
logger.info(f"Starting initial app launch for package: {locked_app_package}") logger.info(f"Starting initial app launch for package: {locked_app_package}")
try: try:
current_package = await get_current_foreground_package_async(ctx) already_foreground, current_package, _ = await _observe_foreground(ctx, locked_app_package)
logger.info(f"Current foreground app: {current_package}") logger.info(f"Current foreground app: {current_package}")
if current_package == locked_app_package: if already_foreground:
logger.info(f"App {locked_app_package} is already in foreground") logger.info(f"App {locked_app_package} is already in foreground")
return AppLaunchResult( return AppLaunchResult(
locked_app_package=locked_app_package, locked_app_package=locked_app_package,
+8 -4
View File
@@ -93,8 +93,10 @@ async def validate_api_key(
async with httpx.AsyncClient(timeout=timeout) as client: async with httpx.AsyncClient(timeout=timeout) as client:
if is_google_family_provider(clean_provider): if is_google_family_provider(clean_provider):
# Test Google Gemini via generativelanguage API # Test Google Gemini via generativelanguage API
url = f"https://generativelanguage.googleapis.com/v1beta/models?key={clean_key}&pageSize=1" # The key travels in a header, never in the URL: httpx logs request
resp = await client.get(url) # URLs at INFO level and those logs end up in files.
url = "https://generativelanguage.googleapis.com/v1beta/models?pageSize=1"
resp = await client.get(url, headers={"x-goog-api-key": clean_key})
if resp.status_code == 200: if resp.status_code == 200:
return True, "Google Gemini API key verified successfully." return True, "Google Gemini API key verified successfully."
err_msg = _extract_error_message(resp) err_msg = _extract_error_message(resp)
@@ -102,7 +104,7 @@ async def validate_api_key(
elif clean_provider in ("ocr", "vision", "google_vision"): elif clean_provider in ("ocr", "vision", "google_vision"):
# Test Google Cloud Vision OCR API # Test Google Cloud Vision OCR API
url = f"https://vision.googleapis.com/v1/images:annotate?key={clean_key}" url = "https://vision.googleapis.com/v1/images:annotate"
payload = { payload = {
"requests": [ "requests": [
{ {
@@ -112,7 +114,9 @@ async def validate_api_key(
] ]
} }
resp = await client.post( resp = await client.post(
url, json=payload, headers={"Content-Type": "application/json"} url,
json=payload,
headers={"Content-Type": "application/json", "x-goog-api-key": clean_key},
) )
if resp.status_code == 200: if resp.status_code == 200:
return True, "Google Cloud Vision OCR API key verified successfully." return True, "Google Cloud Vision OCR API key verified successfully."
+6 -5
View File
@@ -376,17 +376,18 @@ def format_read_note_success(
start_line: int | None = None, start_line: int | None = None,
end_line: int | None = None, end_line: int | None = None,
) -> str: ) -> str:
"""Formats the success message for reading a note.""" """The read result: a one-line header naming the note and its extent, then the content."""
range_str = ""
if start_line is not None or end_line is not None: if start_line is not None or end_line is not None:
range_str = f" (lines {start_line or 1} to {end_line or 'end'})" extent = f"lines {start_line or 1} to {end_line or 'end'}"
return f"Successfully read note '{key}'{range_str}. '{key}' note content:\n{content}" else:
extent = f"{len(content.splitlines())} lines"
return f"Note '{key}' ({extent}):\n{content}"
def format_read_note_failure(key: str, error: str) -> str: def format_read_note_failure(key: str, error: str) -> str:
"""Formats the failure message for reading a note.""" """Formats the failure message for reading a note."""
if "not found" in error.lower(): if "not found" in error.lower():
return ToolFailure(f"Note '{key}' not found in scratchpad.") return ToolFailure(f"Note '{key}' not found. Use list_notes to see the saved keys.")
return ToolFailure(f"Failed to read note '{key}': {error}") return ToolFailure(f"Failed to read note '{key}': {error}")
+65 -15
View File
@@ -314,18 +314,23 @@ def render_check_line(item: CheckItem, indent: int = 2) -> str:
return f"{prefix}- {item.kind}{suffix}: {item.text}" return f"{prefix}- {item.kind}{suffix}: {item.text}"
def restore_missing_check_items(content_before: str, content_after: str) -> str | None: def restore_missing_check_items(
content_before: str,
content_after: str,
items: list[CheckItem] | None = None,
) -> str | None:
"""Deterministic guard: merge check lines deleted/rewritten by a plan write """Deterministic guard: merge check lines deleted/rewritten by a plan write
back into the new content. back into the new content.
Returns the merged plan text, or ``None`` when nothing is missing. Lines Returns the merged plan text, or ``None`` when nothing is missing. Lines
whose parent subgoal still exists are re-anchored directly under it; lines whose parent subgoal still exists are re-anchored directly under it; lines
whose parent subgoal was removed are converted to task-level ``@end`` items whose parent subgoal was removed are converted to task-level ``@end`` items
appended at the end of the plan. appended at the end of the plan. ``items`` restricts the merge to the given
missing items (the caller may waive some of them).
""" """
before = parse_plan(content_before) before = parse_plan(content_before)
after = parse_plan(content_after) after = parse_plan(content_after)
missing = missing_check_items(before, after) missing = missing_check_items(before, after) if items is None else list(items)
if not missing: if not missing:
return None return None
@@ -438,23 +443,68 @@ Every checklist line MUST match `<indent>- [<status>] <text>`:
- `{CONTINUOUS_LOOP_TAG}` tags an UNBOUNDED continuous-monitoring milestone: it must stay `[{STATUS_ACTIVE}]` and can never be marked `[{STATUS_DONE}]`, deleted, or untagged by you — the system mechanically rejects such edits. Only an explicit external stop signal injected by the user unlocks its completion; that signal is an external interruption delivered by the system, never something inferred from the screen or the plan. - `{CONTINUOUS_LOOP_TAG}` tags an UNBOUNDED continuous-monitoring milestone: it must stay `[{STATUS_ACTIVE}]` and can never be marked `[{STATUS_DONE}]`, deleted, or untagged by you — the system mechanically rejects such edits. Only an explicit external stop signal injected by the user unlocks its completion; that signal is an external interruption delivered by the system, never something inferred from the screen or the plan.
- The task terminates only when every top-level milestone is `[{STATUS_DONE}]`.""" - The task terminates only when every top-level milestone is `[{STATUS_DONE}]`."""
_CHECK_GRAMMAR_EXTENSION = """### Check Line Grammar (declared verification standards) _CHECK_GRAMMAR_HEADER = """### Check Line Grammar (declared verification standards)
A top-level milestone may carry indented check lines, and the plan may end with task-level check lines at zero indentation: A top-level milestone may carry indented check lines, and the plan may end with task-level check lines at zero indentation:"""
- ` - verify: <expected state>` — an acceptance criterion for its parent milestone. It is judged by an independent Checker at the moment the milestone is marked completed; a failed verify triggers a repair loop.
- ` - assert: <expected observation>` — a test assertion. It is judged by the independent Checker and a failure is recorded verbatim as a legitimate test result; assertions are NEVER repaired, worked around, or satisfied by constructing state. #: Judgment-moment wording of the check lines, keyed by the midway gate.
- `- verify@end: ...` / `- assert@end: ...` — the `@end` suffix defers judgment to task exit, using the final device state. Items without `@end` are judged with the evidence available when their anchored milestone completes. _VERIFY_JUDGED_MIDWAY = (
- Capability boundary (declare honestly, never promise more): all checks are POST-HOC audits over recorded evidence. They detect and record violations after the fact; they CANNOT block an action from happening, so "B must not run before A is confirmed" ordering rules are verified retroactively, not enforced. Transient prompts that were never captured in the execution history cannot be recovered — prefer expressing expectations as persistently probeable state. "It is judged by an independent Checker at the moment the milestone is marked"
- ` - finding: ...` — a SYSTEM-authored standing headline pinned under a milestone whose verify criterion failed. You never author these lines: the system re-renders them on every plan write (deleting one only makes it reappear) and removes them automatically once the verify passes again or its repair budget is exhausted. Each finding names a `checker-...` note holding the full details and repair log — read it with `read_note`; note keys with the `checker-` prefix are reserved for the system and are read-only for you.""" " completed; a failed verify reopens the milestone for repair."
)
_VERIFY_JUDGED_AT_EXIT = (
"It is judged once at task exit from the recorded evidence around the"
" milestone's completion; there is no midway repair loop."
)
_ASSERT_JUDGED_MIDWAY = (
"It is judged by the independent Checker when its milestone completes (at task exit for `@end`)"
)
_ASSERT_JUDGED_AT_EXIT = "It is judged once at task exit from the recorded evidence"
_CHECK_GRAMMAR_COMMON = """- `- verify@end: ...` / `- assert@end: ...` — the `@end` suffix defers judgment to task exit, using the final device state. Items without `@end` are judged from the evidence recorded around their anchored milestone's completion.
- Capability boundary (declare honestly, never promise more): all checks are POST-HOC audits over recorded evidence. They detect and record violations after the fact; they CANNOT block an action from happening, so "B must not run before A is confirmed" ordering rules are verified retroactively, not enforced. Transient prompts that were never captured in the execution history cannot be recovered — prefer expressing expectations as persistently probeable state."""
_FINDING_GRAMMAR = """- ` - finding: ...` — a SYSTEM-authored standing headline pinned under a milestone whose verify criterion failed. You never author these lines: the system re-renders them on every plan write (deleting one only makes it reappear) and removes them automatically once the verify passes again or its repair budget is exhausted. Each finding names a `checker-...` note holding the full details and repair log — read it with `read_note`; note keys with the `checker-` prefix are reserved for the system and are read-only for you."""
def render_plan_grammar_spec(include_checks: bool) -> str: def _render_check_grammar(*, midway: bool, final: bool) -> str:
"""The check-line grammar worded for the active gates.
Midway on: verify lines are judged at milestone completion and can reopen
the milestone; finding lines exist. Midway off (final only): every line is
judged once at task exit, and no repair loop or finding line exists, so
neither is taught.
"""
verify_when = _VERIFY_JUDGED_MIDWAY if midway else _VERIFY_JUDGED_AT_EXIT
assert_when = _ASSERT_JUDGED_MIDWAY if midway else _ASSERT_JUDGED_AT_EXIT
lines = [
_CHECK_GRAMMAR_HEADER,
"- ` - verify: <expected state>` — an acceptance criterion for its parent"
f" milestone. {verify_when}",
f"- ` - assert: <expected observation>` — a test assertion. {assert_when} and a"
" failure is recorded verbatim as a legitimate test result; assertions are"
" NEVER repaired, worked around, or satisfied by constructing state.",
_CHECK_GRAMMAR_COMMON,
]
if midway:
lines.append(_FINDING_GRAMMAR)
return "\n".join(lines)
def render_plan_grammar_spec(
include_checks: bool | None = None, *, midway: bool = False, final: bool = False
) -> str:
"""Renders the grammar spec taught to agents. """Renders the grammar spec taught to agents.
``include_checks=False`` (both check gates disabled) returns exactly the ``midway`` / ``final`` are the two check gates
base grammar — zero context pollution from the checking feature. (``ExecutionSetup.midway_checks_enabled`` / ``final_check_enabled``); the
check-line grammar is worded for whichever is active. Both off returns
exactly the base grammar — zero context pollution from the checking
feature. ``include_checks`` is a compatibility shim meaning "both gates".
""" """
if include_checks: if include_checks is not None:
return _PLAN_GRAMMAR_BASE + "\n\n" + _CHECK_GRAMMAR_EXTENSION midway = final = bool(include_checks)
if midway or final:
return _PLAN_GRAMMAR_BASE + "\n\n" + _render_check_grammar(midway=midway, final=final)
return _PLAN_GRAMMAR_BASE return _PLAN_GRAMMAR_BASE
+134 -24
View File
@@ -119,8 +119,6 @@ def format_action_clean(action_obj) -> str:
direction = action_obj.get("intent") direction = action_obj.get("intent")
duration = action_obj.get("duration") or action_obj.get("duration_ms") duration = action_obj.get("duration") or action_obj.get("duration_ms")
duration_str = f" over {duration}ms" if duration else "" duration_str = f" over {duration}ms" if duration else ""
if direction:
return f"Swiped {direction}{duration_str}"
start_coords = action_obj.get("start_coordinates") start_coords = action_obj.get("start_coordinates")
end_coords = action_obj.get("end_coordinates") end_coords = action_obj.get("end_coordinates")
@@ -129,6 +127,14 @@ def format_action_clean(action_obj) -> str:
if isinstance(coords_list, list) and len(coords_list) == 4: if isinstance(coords_list, list) and len(coords_list) == 4:
start_coords = coords_list[:2] start_coords = coords_list[:2]
end_coords = coords_list[2:] end_coords = coords_list[2:]
if not start_coords or not end_coords:
start_coords = action_obj.get("normalized_start_coordinates")
end_coords = action_obj.get("normalized_end_coordinates")
if direction:
# A directional swipe is what the agent issued; the recorded path
# (when any) is the detail, not the headline.
path = f" (from {start_coords} to {end_coords})" if start_coords and end_coords else ""
return f"Swiped {direction}{path}{duration_str}"
dragged = _target_label(action_obj, fallback="") dragged = _target_label(action_obj, fallback="")
dragged_str = f" {dragged}" if dragged else "" dragged_str = f" {dragged}" if dragged else ""
return f"Swiped{dragged_str} from {start_coords} to {end_coords}{duration_str}" return f"Swiped{dragged_str} from {start_coords} to {end_coords}{duration_str}"
@@ -176,8 +182,32 @@ def format_action_clean(action_obj) -> str:
) )
return f"Tapped sequence of targets: {labelled}" return f"Tapped sequence of targets: {labelled}"
return f"Tapped sequence of targets: {seq}" return f"Tapped sequence of targets: {seq}"
elif act_type == "wait_for_text":
state = str(action_obj.get("wait_state") or "appear")
timeout = action_obj.get("timeout_ms")
timeout_str = f" (timeout {timeout}ms)" if timeout else ""
return f"Waited for text '{action_obj.get('text', '')}' to {state}{timeout_str}"
elif act_type == "open_link":
return f"Opened link '{action_obj.get('url', '')}'"
elif act_type == "erase_one_char":
return "Erased one character"
elif act_type == "focus_and_clear_text":
return f"Cleared text at {coords}"
else: else:
return f"Action: {act_type} with args: {json.dumps(action_obj, ensure_ascii=False)}" # Unknown action: show only the agent-facing arguments, never the
# bookkeeping keys the record carries for other consumers.
internal = {
"action",
"name",
"intent",
"coordinate_space",
"normalized_coordinates",
"normalized_start_coordinates",
"normalized_end_coordinates",
}
shown = {k: v for k, v in action_obj.items() if k not in internal and v is not None}
args_str = f" with args: {json.dumps(shown, ensure_ascii=False)}" if shown else ""
return f"Action: {act_type}{args_str}"
def format_actions_clean(actions) -> str: def format_actions_clean(actions) -> str:
@@ -192,6 +222,48 @@ def format_actions_clean(actions) -> str:
return format_action_clean(actions) return format_action_clean(actions)
#: Past-tense outcome prefix (``format_action_clean``) -> intent-form prefix.
#: Longest prefixes first so "Double tapped" / "Tapped sequence" win over "Tapped".
_INTENT_PREFIXES: tuple[tuple[str, str], ...] = (
("Tapped sequence of targets: ", "tap sequence of targets: "),
("Double tapped ", "double tap "),
("Long pressed ", "long press "),
("Tapped ", "tap "),
("Inputted ", "type "),
("Swiped ", "swipe "),
("Pressed key ", "press key "),
("Launched app ", "launch app "),
("Stopped app ", "stop app "),
("Managed app ", "manage app "),
("Waited for text ", "wait for text "),
("Waited for ", "wait "),
("Opened link ", "open link "),
("Erased one character", "erase one character"),
("Cleared text ", "clear text "),
)
def action_intent_phrase(description: str) -> str:
"""The intent form of a rendered action phrase: ``Tapped 'Wi-Fi row' at
[500, 520]`` -> ``tap 'Wi-Fi row' at [500, 520]``.
``format_action_clean`` renders actions as past-tense outcomes, which
reads as success; a failure or incident context must describe what the
agent *tried* to do instead. Phrases without a known prefix (bursts,
unknown action types) are returned unchanged.
"""
text = str(description or "")
for outcome, intent in _INTENT_PREFIXES:
if text.startswith(outcome):
return intent + text[len(outcome) :]
return text
def format_action_intent(action_obj) -> str:
"""Intent-form rendering of one action (``launch app 'X'``, ``swipe up``)."""
return action_intent_phrase(format_action_clean(action_obj))
def _is_terminal_attempt_failure(attempts) -> bool: def _is_terminal_attempt_failure(attempts) -> bool:
if not attempts: if not attempts:
return False return False
@@ -216,11 +288,17 @@ def failed_execution_error(result_obj) -> str | None:
def format_incident_clean(incident: dict) -> str: def format_incident_clean(incident: dict) -> str:
"""One-line rendering of an execution incident for history and result lines.""" """One-line rendering of an execution incident for history and result lines.
The action is quoted in intent form (``launch app 'X'``): it did not
happen, so the past-tense outcome phrase would misreport it as done.
"""
kind = incident.get("kind") or "exec_error" kind = incident.get("kind") or "exec_error"
category = incident.get("category") or "general" category = incident.get("category") or "general"
consecutive = incident.get("consecutive_failures") or 1 consecutive = incident.get("consecutive_failures") or 1
description = incident.get("action_description") or format_action_clean(incident.get("action")) description = action_intent_phrase(
incident.get("action_description") or format_action_clean(incident.get("action"))
)
reason = str(incident.get("reason") or "").strip() reason = str(incident.get("reason") or "").strip()
burst_size = int(incident.get("burst_size") or 1) burst_size = int(incident.get("burst_size") or 1)
index = int(incident.get("action_index") or 0) index = int(incident.get("action_index") or 0)
@@ -296,6 +374,21 @@ def safe_parse_validation_result(result: Any) -> list:
return [] return []
#: Block labels of a detailed / replayed step. Role-neutral on purpose: the
#: same renderer replays a Flash turn (no Operator) and a Pro Operator turn,
#: and every Pro agent (Checker, Diagnoser, Outputter) reads them.
REASONING_BLOCK_LABEL = "[Reasoning & tool calls]"
ACTION_BLOCK_LABEL = "[Action]"
BURST_BLOCK_LABEL = "[Fast-Action Burst]"
#: Continuation lines of a " - " bullet stay inside the bullet.
_BULLET_CONTINUATION = "\n "
def _indent_continuation(text: str) -> str:
return text.replace("\n", _BULLET_CONTINUATION)
#: Limit tool-result text in the live context, which is rebuilt every turn. #: Limit tool-result text in the live context, which is rebuilt every turn.
LIVE_RESULT_CHARS = 120 LIVE_RESULT_CHARS = 120
@@ -544,9 +637,27 @@ def _burst_member_status(result: Any, index: int) -> str:
return f" (FAILED: {last.strip()})" return f" (FAILED: {last.strip()})"
def replay_time_label(step: dict, session_start: Any) -> str:
"""The step-header clock of a replay: the session-relative ``T+mm:ss``
offset every agent prompt and ``search_history`` use, or the legacy
``Start: 12.3s`` text when no session start / step timestamp is known."""
ts = step.get("timestamp")
if (
isinstance(session_start, (int, float))
and not isinstance(session_start, bool)
and isinstance(ts, (int, float))
and not isinstance(ts, bool)
):
from artemis.memory.chunking import step_offset_label
return step_offset_label(step, float(session_start))
return f"Start: {step.get('relative_time') or 'N/A'}"
def render_step_replay( def render_step_replay(
step: dict, step: dict,
*, *,
session_start: Any = None,
result_chars: int = REPLAY_RESULT_CHARS, result_chars: int = REPLAY_RESULT_CHARS,
include_summary: bool = True, include_summary: bool = True,
) -> str: ) -> str:
@@ -556,6 +667,9 @@ def render_step_replay(
/ ``get_agent_friendly_steps``): coordinates already normalized, traces / ``get_agent_friendly_steps``): coordinates already normalized, traces
already expanded into ``interleaved_events``. Used by ``replay_steps`` and already expanded into ``interleaved_events``. Used by ``replay_steps`` and
the MCP trace inspector. Screenshots are fetched separately. the MCP trace inspector. Screenshots are fetched separately.
``session_start`` (the reader's ``session_start_time``) makes the header
carry the ``T+mm:ss`` session clock; see :func:`replay_time_label`.
""" """
return _render_step_detailed( return _render_step_detailed(
step=step, step=step,
@@ -566,6 +680,7 @@ def render_step_replay(
is_most_recent=False, is_most_recent=False,
result_chars=result_chars, result_chars=result_chars,
include_summary=include_summary, include_summary=include_summary,
time_label=replay_time_label(step, session_start),
) )
@@ -597,15 +712,17 @@ def _render_step_detailed(
*, *,
result_chars: int = LIVE_RESULT_CHARS, result_chars: int = LIVE_RESULT_CHARS,
include_summary: bool = False, include_summary: bool = False,
time_label: str | None = None,
) -> str: ) -> str:
# 1. Step Header # 1. Step Header (``time_label`` overrides the live window's "Start: 12.3s")
status_str = "Most Recent Step, " if is_most_recent else "" status_str = "Most Recent Step, " if is_most_recent else ""
clock = time_label or f"Start: {relative_time}"
# In the live window the summary is omitted from the detailed view header # In the live window the summary is omitted from the detailed view header
# as it is already fully detailed below; replay callers opt in because the # as it is already fully detailed below; replay callers opt in because the
# summary is the Operator's own claim about the step, which an auditor # summary is the Operator's own claim about the step, which an auditor
# compares against the evidence that follows. # compares against the evidence that follows.
step_line = f"- **Step {step['step_number']} ({status_str}Start: {relative_time})**" step_line = f"- **Step {step['step_number']} ({status_str}{clock})**"
if include_summary: if include_summary:
screen_line = _screen_description_line(step, summary) screen_line = _screen_description_line(step, summary)
if screen_line: if screen_line:
@@ -695,9 +812,10 @@ def _render_step_detailed(
if raw_thinking and raw_thinking.strip() and not thought_exists(raw_thinking): if raw_thinking and raw_thinking.strip() and not thought_exists(raw_thinking):
operator_events.append({"type": "thought", "content": raw_thinking.strip()}) operator_events.append({"type": "thought", "content": raw_thinking.strip()})
# 4. Render Operator Loop # 4. Render the acting model's reasoning and tool calls (role-neutral: the
# same block replays a Flash turn and a Pro Operator turn)
if operator_events: if operator_events:
step_line += "\n * [Operator Decision Loop]:" step_line += f"\n * {REASONING_BLOCK_LABEL}:"
ACTION_TOOLS = { ACTION_TOOLS = {
"click", "click",
"swipe", "swipe",
@@ -722,7 +840,7 @@ def _render_step_detailed(
) )
cleaned = re.sub(r"</?thought>", "", cleaned, flags=re.DOTALL).strip() cleaned = re.sub(r"</?thought>", "", cleaned, flags=re.DOTALL).strip()
if cleaned: if cleaned:
step_line += f"\n - {cleaned}" step_line += f"\n - {_indent_continuation(cleaned)}"
elif e_type == "tool_call": elif e_type == "tool_call":
if name in ACTION_TOOLS: if name in ACTION_TOOLS:
continue continue
@@ -735,17 +853,13 @@ def _render_step_detailed(
if formatted: if formatted:
step_line += f"\n - [Tool Call]: {formatted}" step_line += f"\n - [Tool Call]: {formatted}"
# Check if the execution failed # 5. Render the action the step executed (single vetted action or a burst).
exec_error = None # A failed action's error is printed once, on the [Result] line below.
if isinstance(result, dict) and result.get("status") in ("failed", "error"):
exec_error = failed_execution_error(result)
# 5. Render Planned Action (single vetted action or fast-action burst)
if action: if action:
actions = action if isinstance(action, list) else [action] actions = action if isinstance(action, list) else [action]
if len(actions) > 1: if len(actions) > 1:
step_line += ( step_line += (
f"\n * [Planned Fast-Action Burst]: {len(actions)} actions fired back to" f"\n * {BURST_BLOCK_LABEL}: {len(actions)} actions fired back to"
" back without the safety net" " back without the safety net"
) )
for i, member in enumerate(actions): for i, member in enumerate(actions):
@@ -756,15 +870,11 @@ def _render_step_detailed(
action_clean = format_action_clean(actions[0]) action_clean = format_action_clean(actions[0])
if intercepted: if intercepted:
step_line += ( step_line += (
f"\n * [Planned Action]: {action_clean}" f"\n * {ACTION_BLOCK_LABEL}: {action_clean}"
" (Intercepted by Pre-Execution Safety Net)" " (Intercepted by Pre-Execution Safety Net)"
) )
elif exec_error:
step_line += (
f"\n * [Planned Action]: {action_clean} -> (Execution failed: {exec_error})"
)
else: else:
step_line += f"\n * [Planned Action]: {action_clean}" step_line += f"\n * {ACTION_BLOCK_LABEL}: {action_clean}"
# 6. Render Pre-Execution Safety Net # 6. Render Pre-Execution Safety Net
if intercepted: if intercepted:
@@ -892,7 +1002,7 @@ def build_plan_and_history(
# Ensure the last step is always in detailed_step_ids # Ensure the last step is always in detailed_step_ids
detailed_step_ids.add(full_info_step_id) detailed_step_ids.add(full_info_step_id)
# M4: chunk blocks replace the per-step lines of already-chunked # Chunk blocks replace the per-step lines of already-chunked
# ranges. Chunks only ever cover frozen (old) turns, so they are # ranges. Chunks only ever cover frozen (old) turns, so they are
# emitted first, in chronological order; steps that must render # emitted first, in chronological order; steps that must render
# detailed (recent window / most-recent step) are never suppressed. # detailed (recent window / most-recent step) are never suppressed.
+25 -3
View File
@@ -192,7 +192,7 @@
// ========================================================================== // ==========================================================================
// 🌐 4.3 Shared Capabilities & Global Overrides // 🌐 4.3 Shared Capabilities & Global Overrides
// ========================================================================== // ==========================================================================
// 🧠 Unified Step-Memory Runtime (history-module redesign v3, M0–M5) // Shared step memory and history compression
// Transcript ledger + layered compression for the Pro operator. // Transcript ledger + layered compression for the Pro operator.
"memory": { "memory": {
// Background summarization runtime (visual/capsule lenses) // Background summarization runtime (visual/capsule lenses)
@@ -204,15 +204,33 @@
"transcript": { "transcript": {
// ROLLBACK SWITCH: set false to restore the legacy per-turn 2-message // ROLLBACK SWITCH: set false to restore the legacy per-turn 2-message
// prompt path byte-for-byte (L2/L3 chunk compression rides this same // prompt path byte-for-byte (L2/L3 chunk compression rides this same
// flag). Default on since M5 (2026-09-01) per the on-device A/B: // flag). See services/token_meter.py for measured session token usage
// turn latency −77% P50, cache-hit calls 43%→88%, output tokens −74%. // and cache-hit ratios.
"enabled": true, "enabled": true,
"context_budget_tokens": 80000, "context_budget_tokens": 80000,
// Below start_ratio of the operator's own measured prompt size, closed
// segments only prepare their summary; the original turns are kept in
// context and replaced once occupancy reaches this ratio. Measurements
// come from the operator; other agents' calls do not affect this gate.
"start_ratio": 0.35,
"soft_ratio": 0.7, "soft_ratio": 0.7,
"hard_ratio": 0.9, "hard_ratio": 0.9,
"min_active_steps": 5, "min_active_steps": 5,
// Screenshot edge: 6 while the operator's measured context is below
// start_ratio (nothing is being replaced yet), 3 from there up. A
// screenshot reaching the edge is replaced in place by its visual
// summary (the 'Current Screenshot' label leaves with it) and never
// restored; the depth only decides how long new ones stay.
"image_scrub_depth": 3, "image_scrub_depth": 3,
"image_scrub_depth_relaxed": 6,
"pending_grace_steps": 3, "pending_grace_steps": 3,
// Text edge: once a newer observation exists (depth 1) the UI element
// list, the plan recitation and the per-turn ephemeral blocks
// (reminders, hints, guidance wrappers) leave the older observation.
// User instruction bodies remain in history. The text edge stays
// shallow on purpose: only the live observation carries an indexed
// element list, so a stale [n] index can never be picked up as a
// target.
"xml_scrub_depth": 1, "xml_scrub_depth": 1,
// Local historical-state hint (no model call); distance calibrated // Local historical-state hint (no model call); distance calibrated
// 2026-09-01 from 460 dHash-stamped on-device steps (same screen <=4, // 2026-09-01 from 460 dHash-stamped on-device steps (same screen <=4,
@@ -223,6 +241,10 @@
// Segment (L2) chunk compression: three-band chunks, era folding // Segment (L2) chunk compression: three-band chunks, era folding
"chunking": { "chunking": {
"max_steps": 12, "max_steps": 12,
// Size/pressure closes wait until at least this many turns are
// eligible, so heavy turns do not become one-step chunks (one capsule
// call each); milestone closes are exempt, the hard line waives it.
"min_steps": 3,
"target_source_tokens": 2000, "target_source_tokens": 2000,
"model": "gemini-3.8-flash", "model": "gemini-3.8-flash",
"max_chunks": 8 "max_chunks": 8
+5 -2
View File
@@ -1,6 +1,6 @@
# 🔌 Universal MCP Server for ARTEMIS # 🔌 Universal MCP Server for ARTEMIS
This directory contains the universal **Model Context Protocol (MCP)** server for **ARTEMIS**, enabling seamless mobile device automation inside AI IDEs and agents like **Antigravity**, **Cursor**, **Claude Code**, **OpenClaw**, and **Windsurf**. This directory contains the **Model Context Protocol (MCP)** server for **ARTEMIS**. It exposes mobile device automation to clients such as **Antigravity**, **Cursor**, **Claude Code**, **OpenClaw**, and **Windsurf**.
## 🏗️ Architecture ## 🏗️ Architecture
@@ -24,7 +24,8 @@ mcp_server/
│ ├── task_runner.py # mobile_run_task │ ├── task_runner.py # mobile_run_task
│ ├── task_manager.py # mobile_manage_task │ ├── task_manager.py # mobile_manage_task
│ ├── device_state.py # mobile_get_device_state │ ├── device_state.py # mobile_get_device_state
│ └── inspect_trace.py # mobile_inspect_trace │ ├── inspect_trace.py # mobile_inspect_trace
│ └── diagnose.py # mobile_diagnose
└── utils/ # Environment and device utilities └── utils/ # Environment and device utilities
├── device_utils.py # Cross-platform ADB and emulator resolver ├── device_utils.py # Cross-platform ADB and emulator resolver
└── env_utils.py # Python interpreter and process manager └── env_utils.py # Python interpreter and process manager
@@ -56,6 +57,7 @@ The included [`rules.md`](./rules.md) file contains the **Mobile Testing Mindset
2. **Flash vs. Pro Routing Strategy**: Guides the AI to choose **Flash** for rapid, straightforward UI actions (no step cap by default) and **Pro** for tasks that need a persistent plan, verified checkpoints, polling loops, or ADB / video / log diagnosis. 2. **Flash vs. Pro Routing Strategy**: Guides the AI to choose **Flash** for rapid, straightforward UI actions (no step cap by default) and **Pro** for tasks that need a persistent plan, verified checkpoints, polling loops, or ADB / video / log diagnosis.
3. **Latency & Timing Compensation**: Clarifies the difference between AI exploratory latency (e.g., model decision intervals) and the deterministic timing requirements of final test code. 3. **Latency & Timing Compensation**: Clarifies the difference between AI exploratory latency (e.g., model decision intervals) and the deterministic timing requirements of final test code.
4. **"Dynamic-First, Coordinate-Fallback" Locator Pattern**: Teaches the AI to prioritize dynamic UI locators (Resource IDs, OCR text, semantics) for layout resilience, while implementing absolute coordinate fallbacks for maximum execution reliability. 4. **"Dynamic-First, Coordinate-Fallback" Locator Pattern**: Teaches the AI to prioritize dynamic UI locators (Resource IDs, OCR text, semantics) for layout resilience, while implementing absolute coordinate fallbacks for maximum execution reliability.
5. **Environment Self-Diagnosis**: Tells the AI to call `mobile_diagnose` first whenever a tool errors or the user says ARTEMIS is not working, and how to act on the returned fix list (run local commands itself, relay device prompts to the user, never move API keys through the chat, reload the MCP server after config changes).
### How Global Rules & MCP are Installed ### How Global Rules & MCP are Installed
When you run `uv run artemis mcp --install all` (or target a specific IDE like `--install cursor`, `--install claude`, etc.), the installer **automatically synchronizes both global MCP server configuration and global testing rules** into the corresponding user home directories (without creating project-level rule files in your workspace): When you run `uv run artemis mcp --install all` (or target a specific IDE like `--install cursor`, `--install claude`, etc.), the installer **automatically synchronizes both global MCP server configuration and global testing rules** into the corresponding user home directories (without creating project-level rule files in your workspace):
@@ -75,6 +77,7 @@ When you run `uv run artemis mcp --install all` (or target a specific IDE like `
* **`mobile_manage_task`**: Manages task lifecycle (`status`, `stop`, `inject_instruction`), returning task state and assigned `device_serial`. Pass `release_loop=True` with `inject_instruction` to gracefully end a `[Loop:continuous]` monitoring task — this explicit signal (not "please stop" wording) is what unlocks the loop milestone's completion. * **`mobile_manage_task`**: Manages task lifecycle (`status`, `stop`, `inject_instruction`), returning task state and assigned `device_serial`. Pass `release_loop=True` with `inject_instruction` to gracefully end a `[Loop:continuous]` monitoring task — this explicit signal (not "please stop" wording) is what unlocks the loop milestone's completion.
* **`mobile_get_device_state`**: Real-time observer (`screenshot` or OCR+XML `hierarchy`) with optional `device_serial`. * **`mobile_get_device_state`**: Real-time observer (`screenshot` or OCR+XML `hierarchy`) with optional `device_serial`.
* **`mobile_inspect_trace`**: Granular trace inspection, visual action overlays, agent reasoning, and `device_serial` tracking. * **`mobile_inspect_trace`**: Granular trace inspection, visual action overlays, agent reasoning, and `device_serial` tracking.
* **`mobile_diagnose`**: Environment doctor for the IDE. Reuses the readiness probes behind `artemis doctor` and the web console's device wizard (Python runtime, config, LLM credentials, ADB / devices / RSA keys / emulators, video toolchain) plus an MCP-host probe (server interpreter vs project `.venv`, detected MCP client, `.env` location, traces directory, daemon port collisions). Returns a `verdict` (`ready` | `degraded` | `blocked`), an ordered `next_steps` fix list the AI agent can act on (`Run:` one-command lines vs `Guidance:` for the user), scrubbed per-check details, `tasks` holding or waiting for devices, log paths, and the most recent failed task with its `recent_errors`. Optional deep checks: `verify_credentials=true` (live API-key validation, ~12s, result in `credentials`), `probe_device=true` (end-to-end screenshot + UIAutomator hierarchy, ~20s, result in `device_probe`), `launch_avd="<name>"` (boots an installed AVD in the background; re-run after ~60s). `attempt_fix=true` applies the safe self-heals (regenerate corrupted ADB keys, restart the ADB server, clear stale device locks / queue tickets left by crashed runners).
### 📱 Device Selection & Multi-Device Execution ### 📱 Device Selection & Multi-Device Execution
ARTEMIS supports parallel execution across multiple connected Android devices and emulators: ARTEMIS supports parallel execution across multiple connected Android devices and emulators:
+3 -1
View File
@@ -37,7 +37,9 @@ mcp = FastMCP(
"Use mobile_run_task to launch autonomous UI workflows on connected Android devices or emulators, " "Use mobile_run_task to launch autonomous UI workflows on connected Android devices or emulators, "
"mobile_manage_task to check status or steer execution, " "mobile_manage_task to check status or steer execution, "
"mobile_get_device_state to inspect real-time device screen/hierarchy, " "mobile_get_device_state to inspect real-time device screen/hierarchy, "
"and mobile_inspect_trace to inspect detailed execution steps and visual action overlays." "mobile_inspect_trace to inspect detailed execution steps and visual action overlays, "
"and mobile_diagnose whenever a tool errors, no device is found, or the user says "
"ARTEMIS is not working: it checks the environment and returns ordered fix steps."
), ),
) )
+14 -2
View File
@@ -1,9 +1,9 @@
# Mobile Testing Mindset (ARTEMIS Integration) # Mobile Testing Mindset (ARTEMIS Integration)
As a senior test engineer, you must ensure that all test code you write is **fully executable, stable, and exceptionally robust**. You must never guess or assume how an interaction behaves on a mobile device, no matter how simple the task may seem. Instead, you must leverage **ARTEMIS**, a powerful mobile UI automation framework, to discover and verify the exact execution flow, and author your test scripts based strictly on these concrete, verified steps. Furthermore, if you need to investigate or research specific software or hardware behaviors, you should utilize ARTEMIS to perform active explorations on the device, whether by mimicking human interactions or executing precise ADB commands. Write executable tests based on device behavior verified with **ARTEMIS**. Before authoring a test, explore the target application and confirm its screens, transitions, and interactions. Use ARTEMIS device actions or ADB commands to investigate software and hardware behavior instead of assuming how an interaction works.
### 1. The Runnable Code Principle & ARTEMIS Exploration ### 1. The Runnable Code Principle & ARTEMIS Exploration
- **When tasked with authoring tests**, your primary goal is to deliver **runnable, production-grade test code**. - **When tasked with authoring tests**, deliver runnable test code with verified interactions and explicit wait conditions.
- Before writing any test code, you must use the ARTEMIS MCP tools to interactively run and explore the target application. This allows you to discover the exact sequence of UI states, transitions, and required interactions. - Before writing any test code, you must use the ARTEMIS MCP tools to interactively run and explore the target application. This allows you to discover the exact sequence of UI states, transitions, and required interactions.
- Analyze the user's target testing framework to exploit its native capabilities and maximize test stability. - Analyze the user's target testing framework to exploit its native capabilities and maximize test stability.
- **Timing & Latency Management (Exploration vs. Execution)**: - **Timing & Latency Management (Exploration vs. Execution)**:
@@ -54,3 +54,15 @@ Deeply understand and select between ARTEMIS's dual execution models (**ARTEMIS
2. **Catch**: If the dynamic attempt fails, fall back to the precise absolute coordinates verified during your ARTEMIS exploration. 2. **Catch**: If the dynamic attempt fails, fall back to the precise absolute coordinates verified during your ARTEMIS exploration.
- **Coordinate-Only Frameworks**: If the framework only supports coordinates, ensure the coordinates are well-documented, and where possible, parameterized or made relative to screen boundaries to mitigate resolution differences. - **Coordinate-Only Frameworks**: If the framework only supports coordinates, ensure the coordinates are well-documented, and where possible, parameterized or made relative to screen boundaries to mitigate resolution differences.
- **Dynamic-Only Frameworks**: If the framework does not support coordinate-based clicks, focus entirely on generating highly robust dynamic locators, leveraging ARTEMIS's element descriptions and XML tree analysis. - **Dynamic-Only Frameworks**: If the framework does not support coordinate-based clicks, focus entirely on generating highly robust dynamic locators, leveraging ARTEMIS's element descriptions and XML tree analysis.
### 5. Environment Self-Diagnosis (`mobile_diagnose`)
- **Diagnose before guessing**: Whenever an ARTEMIS tool returns an error, a task fails to start or is rejected, no device is found, or the user reports that ARTEMIS "doesn't work" in the IDE, call `mobile_diagnose` first. Do not troubleshoot by re-running `mobile_run_task` on trial and error, and do not ask the user to run `adb` commands by hand before you have the report.
- **If `mobile_diagnose` is not in your tool list at all**, the MCP server itself failed to start: check the IDE's MCP server log and run `uv run artemis doctor` in the project directory (the CLI shares the same checks).
- **Follow `next_steps` in order**: The report lists fixes in dependency order (Python runtime → config → MCP host → LLM credentials → device → optional video toolchain). `Run:` lines are single, local, non-destructive shell commands (one command per line, no `&&`) you may execute yourself (ask before installing software with winget / brew / apt). `Guidance:` lines describe something only the user can do (unlock the phone, tap "Allow" on the USB-debugging prompt, plug in a cable, free a port, tell you which IDE they use) — relay them verbatim. `Docs:` lines are references.
- **Let ARTEMIS self-heal first**: Call `mobile_diagnose(attempt_fix=true)` before asking for manual ADB intervention; it regenerates corrupted ADB RSA keys, restarts the ADB server when that is safe (no device ready, no task holding a device), and removes stale device locks / queue tickets left behind by crashed runners.
- **Deeper checks on demand**: `verify_credentials=true` validates the configured API keys live (~12s; use when a key exists but tasks fail with auth / quota errors; result in `credentials`). `probe_device=true` runs a true end-to-end check — screenshot plus UI hierarchy through UIAutomator (~20s; use when the device shows connected but observation or tasks fail; result in `device_probe`).
- **Emulators**: `launch_avd="<name>"` starts an installed Android Virtual Device in the background via ARTEMIS's emulator manager and returns immediately; boot takes 1-3 minutes, so re-run `mobile_diagnose` after ~60s. Never run `emulator -avd` in the shell yourself — it is a foreground process and hangs the shell.
- **Busy devices**: `tasks: {active, queued}` lists tasks currently holding or waiting for devices. A busy device means wait, or stop the task with `mobile_manage_task(action="stop", trace_id=...)`.
- **Credentials never go through the chat**: If the LLM key is missing, ask the user to add it to the `.env` file at the path the report shows (`host.env_file`) or to the MCP server's `env` block, then to restart the MCP server. `artemis init` is interactive and cannot be driven from a tool call.
- **Restart after configuration changes**: Settings are read when the server starts. After editing `.env`, `config/artemis.jsonc`, or the MCP client configuration, reload the MCP server in the IDE before calling `mobile_diagnose` again; otherwise it reports the old state.
- **Failed tasks**: `logs.last_failed_task` gives the trace id, its `stderr_log`, and `recent_errors` (the error lines from that stderr, so you need not open the file); read them before re-running the task. Repeat `mobile_diagnose` until `verdict` is `ready` or `degraded` before delegating tasks again (`degraded` means tasks can run but something optional is off: video tools, the OCR key, a secondary credential, or a host warning such as an interpreter mismatch or a squatted daemon port; relay its `[OPTIONAL]` steps to the user, do not loop on them).
+2
View File
@@ -15,6 +15,7 @@
"""MCP Tools package for ARTEMIS.""" """MCP Tools package for ARTEMIS."""
from mcp_server.tools.device_state import mobile_get_device_state from mcp_server.tools.device_state import mobile_get_device_state
from mcp_server.tools.diagnose import mobile_diagnose
from mcp_server.tools.inspect_trace import mobile_inspect_trace from mcp_server.tools.inspect_trace import mobile_inspect_trace
from mcp_server.tools.task_manager import mobile_manage_task from mcp_server.tools.task_manager import mobile_manage_task
from mcp_server.tools.task_runner import mobile_run_task from mcp_server.tools.task_runner import mobile_run_task
@@ -24,4 +25,5 @@ __all__ = [
"mobile_manage_task", "mobile_manage_task",
"mobile_get_device_state", "mobile_get_device_state",
"mobile_inspect_trace", "mobile_inspect_trace",
"mobile_diagnose",
] ]
File diff suppressed because it is too large. Load diff
+1 -3
View File
@@ -284,9 +284,7 @@ async def run_part_4():
print_status_snapshot("Part 4 Running (Mixed Queueing)") print_status_snapshot("Part 4 Running (Mixed Queueing)")
await asyncio.gather(t4a, t4b, t4c) await asyncio.gather(t4a, t4b, t4c)
print( print("Part 4 passed: cross-device concurrency and single-device queueing.\n")
"🎉 Part 4 Success: Cross-device concurrency and single-device queueing work together seamlessly!\n"
)
async def main(): async def main():
+1 -1
View File
@@ -17,7 +17,7 @@
Fastest and simplest way to launch the Artemis Web UI across all platforms: Fastest and simplest way to launch the Artemis Web UI across all platforms:
- Automatically detects if the UI is already running and opens the browser directly. - Automatically detects if the UI is already running and opens the browser directly.
- Uses uv or existing virtualenv seamlessly. - Uses uv or an existing virtualenv.
- Zero-configuration one-click start. - Zero-configuration one-click start.
""" """
+96 -36
View File
@@ -3,33 +3,39 @@
"type": "function", "type": "function",
"function": { "function": {
"name": "click", "name": "click",
"description": "[ACTION] Tap/click on the target coordinate on the screen. The screen after click will be returned automatically. For buttons, checkboxes, tabs, icons, items in a list, tap ON the element. For text fields / search bars / input boxes, tap INSIDE the input box to focus it.", "description": "[ACTION] Click on the target location on the screen (supports element index or absolute normalized coordinates).",
"parameters": { "parameters": {
"type": "object", "type": "object",
"properties": { "properties": {
"target": { "target": {
"anyOf": [
{
"type": "integer"
},
{
"type": "array", "type": "array",
"items": { "items": {
"type": "integer" "type": "integer"
}, }
"description": "Normalized coordinates [x, y] in 0-1000 scale." }
],
"description": "Click target. Can be an element index number (int, e.g. 3) OR normalized coordinates (list of 2 integers, e.g. [500, 600])."
}, },
"target_description": { "target_description": {
"type": "string", "type": "string",
"description": "What the target is, in a few words (e.g. 'play button', 'search input', 'video body'). Required: a coordinate names nothing by itself." "description": "What the target is, in a few words (e.g. 'play button', 'search input', 'video body'). REQUIRED when target is a coordinate pair; ignored for an element index."
}, },
"times": { "times": {
"type": "integer", "type": "integer",
"description": "Number of taps to perform (default 1). Set to 2 for double click." "description": "Number of consecutive clicks on this target. Use this for double-clicks or multi-clicks (e.g. 7 to enter developer mode). Default is 1."
}, },
"delay_ms": { "delay_ms": {
"type": "integer", "type": "integer",
"description": "Delay between taps in milliseconds (default 100)." "description": "Delay in milliseconds between consecutive clicks. Default is 100."
} }
}, },
"required": [ "required": [
"target", "target"
"target_description"
] ]
} }
} }
@@ -76,29 +82,35 @@
"type": "function", "type": "function",
"function": { "function": {
"name": "long_press", "name": "long_press",
"description": "[ACTION] Long press on a target coordinate on the screen. The screen after long pressing will be returned automatically.", "description": "[ACTION] Long press on the target location on the screen (supports element index or absolute normalized coordinates).",
"parameters": { "parameters": {
"type": "object", "type": "object",
"properties": { "properties": {
"target": { "target": {
"anyOf": [
{
"type": "integer"
},
{
"type": "array", "type": "array",
"items": { "items": {
"type": "integer" "type": "integer"
}, }
"description": "Normalized coordinates [x, y] in 0-1000 scale." }
],
"description": "Long press target. Can be an element index number (int, e.g. 3) OR normalized coordinates (list of 2 integers, e.g. [500, 600])."
}, },
"target_description": { "target_description": {
"type": "string", "type": "string",
"description": "What the target is, in a few words (e.g. 'play button', 'search input', 'video body'). Required: a coordinate names nothing by itself." "description": "What the target is, in a few words (e.g. 'play button', 'search input', 'video body'). REQUIRED when target is a coordinate pair; ignored for an element index."
}, },
"duration_ms": { "duration": {
"type": "integer", "type": "integer",
"description": "Duration of press in milliseconds (default 1000ms)." "description": "Long press duration in milliseconds (default 1000)."
} }
}, },
"required": [ "required": [
"target", "target"
"target_description"
] ]
} }
} }
@@ -107,34 +119,40 @@
"type": "function", "type": "function",
"function": { "function": {
"name": "input_text", "name": "input_text",
"description": "[ACTION] Type text into an input field on the screen. The screen after typing will be returned automatically. Automatically taps inside the input box at target [x, y] to focus before typing.", "description": "[ACTION] Type text into the target input field (supports replacing whole text or appending to the end, and multi-line strings with '\\n').",
"parameters": { "parameters": {
"type": "object", "type": "object",
"properties": { "properties": {
"text": { "text": {
"type": "string", "type": "string",
"description": "Text to type into the focused input field." "description": "The text content to input. Supports multi-line content with '\\n'."
}, },
"target": { "target": {
"anyOf": [
{
"type": "integer"
},
{
"type": "array", "type": "array",
"items": { "items": {
"type": "integer" "type": "integer"
}, }
"description": "Coordinates [x, y] of the input box in 0-1000 scale." }
],
"description": "Input target field. Can be an input box element index number (int, e.g. 3) OR normalized coordinates (list of 2 integers, e.g. [500, 600])."
}, },
"target_description": { "target_description": {
"type": "string", "type": "string",
"description": "What the target is, in a few words (e.g. 'play button', 'search input', 'video body'). Required: a coordinate names nothing by itself." "description": "What the target is, in a few words (e.g. 'play button', 'search input', 'video body'). REQUIRED when target is a coordinate pair; ignored for an element index."
}, },
"clear_exist": { "clear_exist": {
"type": "boolean", "type": "boolean",
"description": "Whether to clear existing text in the input box before typing (default True)." "description": "Whether to clear existing text before typing. True (default): clear/replace entire text. False: append at the end of existing content."
} }
}, },
"required": [ "required": [
"text", "text",
"target", "target"
"target_description"
] ]
} }
} }
@@ -143,7 +161,7 @@
"type": "function", "type": "function",
"function": { "function": {
"name": "swipe", "name": "swipe",
"description": "[ACTION] Perform a swipe, drag, or slider-adjustment gesture on the screen. The screen and UI hierarchy after swipe will be returned automatically.\n\n• Directional Scrolling ('direction' or 'action'): Recommended for general browsing and standard page scrolling in most scenarios. Automatically computes safe swipe vectors and adaptive duration, retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings. Supports scoping to a sub-container via 'target'. If it fails on certain custom layouts, fall back to specifying exact coordinates ('start' and 'end') directly.\n• Precise Coordinate Gestures ('start', 'end' or coordinates list): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars), drag-and-drop / list reordering, or as a reliable fallback when directional scrolling fails on specific containers. Always drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), swipe fully to the extreme boundary.", "description": "[ACTION] Perform a swipe, drag, or slider-adjustment gesture on the screen.\n\n• Directional Scrolling ('direction'): Recommended for general browsing and standard page scrolling in most scenarios. Automatically computes safe swipe vectors and adaptive duration, retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings. Supports scoping to a sub-container via 'target'. If it fails on certain custom layouts, fall back to specifying exact coordinates ('start' and 'end') directly.\n• Precise Coordinate Gestures ('start', 'end'): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars), drag-and-drop / list reordering, or as a reliable fallback when directional scrolling fails on specific containers. Always drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), swipe fully to the extreme boundary.\n\nArgs:\n direction: Smart directional scrolling ('up', 'down', 'left', 'right'). Automatically computes safe swipe vectors, retaining 40% visual overlap: 'up' (reveal content below), 'down' (reveal content above), 'left', 'right'.\n start: Start normalized coordinates [start_x, start_y] in 0-1000 scale.\n end: End normalized coordinates [end_x, end_y] in 0-1000 scale.\n target: Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within.\n gesture: Backward-compatible parameter: direction string OR custom coordinates list [start_x, start_y, end_x, end_y] in 0-1000 scale.\n target_description: What is being dragged, in a few words (e.g. 'brightness slider knob'). REQUIRED for coordinate gestures ('start'/'end' or a coordinates list); ignored for directional scrolling.\n duration: Optional gesture duration in milliseconds (default 800).",
"parameters": { "parameters": {
"type": "object", "type": "object",
"properties": { "properties": {
@@ -162,30 +180,62 @@
"items": { "items": {
"type": "integer" "type": "integer"
}, },
"description": "Start normalized coordinates [start_x, start_y] in 0-1000 scale." "description": "Start normalized coordinates [start_x, start_y] in 0-1000 scale for precise, local interactions (e.g. adjusting sliders, SeekBars, fine range selection, or drag-and-drop)."
}, },
"end": { "end": {
"type": "array", "type": "array",
"items": { "items": {
"type": "integer" "type": "integer"
}, },
"description": "End normalized coordinates [end_x, end_y] in 0-1000 scale." "description": "End normalized coordinates [end_x, end_y] in 0-1000 scale for precise, local interactions (e.g. adjusting sliders, SeekBars, fine range selection, or drag-and-drop)."
}, },
"target": { "target": {
"anyOf": [
{
"type": "integer"
},
{
"type": "array",
"items": {
"type": "integer"
}
},
{
"type": "string"
}
],
"description": "Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within." "description": "Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within."
}, },
"action": { "gesture": {
"anyOf": [
{
"type": "string",
"enum": [
"up",
"down",
"left",
"right"
]
},
{
"type": "array",
"items": {
"type": "integer"
}
}
],
"description": "Backward-compatible swipe gesture: smart direction string ('up', 'down', 'left', 'right') OR precise custom coordinates [start_x, start_y, end_x, end_y] in 0-1000 scale." "description": "Backward-compatible swipe gesture: smart direction string ('up', 'down', 'left', 'right') OR precise custom coordinates [start_x, start_y, end_x, end_y] in 0-1000 scale."
}, },
"target_description": { "target_description": {
"type": "string", "type": "string",
"description": "What is being dragged, in a few words (e.g. 'brightness slider knob'). Required for coordinate gestures ('start'/'end' or a coordinates list); ignored for directional scrolling." "description": "What is being dragged, in a few words (e.g. 'brightness slider knob'). REQUIRED for coordinate gestures ('start'/'end' or a coordinates list); ignored for directional scrolling."
}, },
"duration": { "duration": {
"type": "integer", "type": "integer",
"description": "Optional swipe/drag duration in milliseconds (default 800). For drag-and-drop, list reordering, or sliding/adjusting sliders (e.g., volume, brightness, SeekBars), set duration >= 1000 (e.g. 1500). If omitted for directional swipe, duration is computed automatically." "description": "Optional swipe/drag duration in milliseconds (default 800; computed automatically for directional swipes)."
}
} }
},
"required": []
} }
} }
}, },
@@ -193,12 +243,18 @@
"type": "function", "type": "function",
"function": { "function": {
"name": "press_key", "name": "press_key",
"description": "[ACTION] Press a physical or virtual system button. The screen after pressing will be returned automatically.", "description": "[ACTION] Press a physical or virtual system button (e.g. ENTER, BACK, HOME, APP_SWITCH).",
"parameters": { "parameters": {
"type": "object", "type": "object",
"properties": { "properties": {
"key": { "key": {
"type": "string", "type": "string",
"enum": [
"ENTER",
"BACK",
"HOME",
"APP_SWITCH"
],
"description": "Standard Android system button name (ENTER, BACK, HOME, APP_SWITCH)." "description": "Standard Android system button name (ENTER, BACK, HOME, APP_SWITCH)."
} }
}, },
@@ -212,13 +268,17 @@
"type": "function", "type": "function",
"function": { "function": {
"name": "manage_app", "name": "manage_app",
"description": "[ACTION] Launch or force stop a specified application. The screen after launch/stop will be returned automatically.", "description": "[ACTION] Launch or force stop a specified application.",
"parameters": { "parameters": {
"type": "object", "type": "object",
"properties": { "properties": {
"action": { "action": {
"type": "string", "type": "string",
"description": "The action type ('launch' or 'stop')." "enum": [
"launch",
"stop"
],
"description": "The action type."
}, },
"app_name": { "app_name": {
"type": "string", "type": "string",
@@ -236,13 +296,13 @@
"type": "function", "type": "function",
"function": { "function": {
"name": "wait_for_delay", "name": "wait_for_delay",
"description": "[ACTION] Pause execution and wait for a specified duration in milliseconds. The screen after pause will be returned automatically.", "description": "[ACTION] Pause execution and wait for a specified duration in milliseconds.\n\nUse this whenever you need time to elapse—whether for UI loading, animations,\nscreen transitions, or longer scheduled delays and intervals specified in the task.",
"parameters": { "parameters": {
"type": "object", "type": "object",
"properties": { "properties": {
"time_in_ms": { "time_in_ms": {
"type": "integer", "type": "integer",
"description": "The exact duration to pause in milliseconds (e.g., 2000 for 2s, 60000 for 1 min, 180000 for 3 mins, 300000 for 5 mins). Convert any required waiting duration into milliseconds." "description": "The exact duration to wait in milliseconds. Accurately convert the required time duration into milliseconds based on your objective or plan (e.g., 2000 for 2s, 5000 for 5s, 60000 for 1 minute, 180000 for 3 minutes, 300000 for 5 minutes)."
} }
}, },
"required": [ "required": [
@@ -0,0 +1,302 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""A Flash multi-action turn resolves every index against one element list.
Defect: each action's post-observation replaced ``state.indexed_elements``
and the executor resolved index targets against the state at call time, so
``click(3)`` then ``click(5)`` decided on one screen sent the second tap to
whatever the screen after the first click listed as ``[5]``. The runner now
snapshots the list once per turn and threads it through
``McpActionExecutor.execute(index_elements=...)``; the state's list is still
refreshed after every action (that is what the next observation shows).
"""
from unittest.mock import AsyncMock, Mock, patch
import pytest
from langchain_core.messages import ToolMessage
from artemis.agents.flash.runner import FlashRunner, _TurnRecord
from artemis.context import ArtemisContext
from artemis.graph.state import State
from artemis.mcp.action_executor import McpActionExecutor, _ArgError
from artemis.mcp.action_types import ActionResult, ObserveResult
def _el(index: int, text: str, cx: int, cy: int) -> dict:
return {
"index": index,
"center": [cx, cy],
"text": text,
"bounds": [cx - 100, cy - 50, cx + 100, cy + 50],
"class": "android.widget.TextView",
"resource_id": f"id/{text.lower().replace(' ', '_')}",
"is_ocr": False,
}
# The list of the observation the model decided on (1080x2400 device).
ORIGINAL = [
_el(1, "Home", 540, 200),
_el(2, "Search", 540, 600),
_el(3, "Wi-Fi", 540, 1000),
_el(4, "Bluetooth", 540, 1400),
_el(5, "Display", 540, 1800),
]
# The screen after the first click: fewer elements, different numbering.
REFRESHED = [_el(1, "Back", 100, 100), _el(2, "Wi-Fi toggle", 900, 300)]
class _FakeSession:
"""An action session whose observations replay a scripted list sequence."""
started = True
def __init__(self, observations: list[list[dict]]):
self.calls: list[tuple[str, dict]] = []
self._observations = list(observations)
async def call(self, name, args):
self.calls.append((name, dict(args)))
return ActionResult.success(name, f"{name} dispatched")
async def observe(self, settle_ms=0):
elements = self._observations.pop(0) if self._observations else []
return ObserveResult(
ok=True,
elements_text="\n".join(f"[{e['index']}] {e['text']}" for e in elements),
elements=elements,
hierarchy_ok=True,
)
@pytest.fixture
def mock_context():
ctx = Mock(spec=ArtemisContext)
ctx.llm_config = Mock()
llm_cfg = Mock()
llm_cfg.model = "gemini-2.5-flash"
llm_cfg.temperature = 0.1
ctx.llm_config.get_agent.return_value = llm_cfg
ctx.device = Mock()
ctx.device.device_width = 1080
ctx.device.device_height = 2400
ctx.data_engine = None
ctx.adb_client = None
ctx.driver = Mock()
return ctx
def _runner(mock_context, observations):
with patch("artemis.controllers.unified_controller.get_driver"):
runner = FlashRunner(mock_context, goal="Open Wi-Fi and Display")
runner.summarizer = None
runner.executor._session = _FakeSession(observations)
# No device: the executor's observation already refreshed the state.
runner._capture_post_screenshot = AsyncMock(return_value=None)
return runner
def _state(elements):
state = State(initial_goal="g")
state.indexed_elements = list(elements)
state.indexed_points = [e["center"] for e in elements]
return state
def _click(index, tc_id):
return {"name": "click", "args": {"target": index}, "id": tc_id}
# --- Executor: the explicit list wins over the state ---------------------------------
def _executor():
ctx = Mock()
ctx.device.device_width = 1080
ctx.device.device_height = 2400
actuator = Mock()
actuator.controller = Mock()
return McpActionExecutor(ctx, actuator=actuator)
def test_resolve_index_uses_the_given_list_and_falls_back_to_the_state():
executor = _executor()
state = _state(REFRESHED)
# None -> the state's current list (Pro operator / validator / explorer callers).
point, recorded = executor._resolve_index(2, "click", state)
assert point == [833, 125] and recorded["target_text"] == "Wi-Fi toggle"
# An explicit list: index 5 exists only there.
point, recorded = executor._resolve_index(5, "click", state, index_elements=ORIGINAL)
assert point == [500, 750] and recorded["target_text"] == "Display"
assert recorded["target_resource_id"] == "id/display"
# The snapshot bounds the valid range, not the state.
with pytest.raises(_ArgError, match="Active index range is 1 to 2"):
executor._resolve_index(5, "click", state, index_elements=REFRESHED)
_, _, _, recorded = executor._translate(
"long_press", {"target": 4}, state, index_elements=ORIGINAL
)
assert recorded["target_text"] == "Bluetooth"
_, wire, _, recorded = executor._translate(
"input_text", {"target": 2, "text": "hi"}, state, index_elements=ORIGINAL
)
assert wire["target"] == [500, 250] and recorded["target_text"] == "Search"
# --- Runner: one snapshot per turn ------------------------------------------------------
@pytest.mark.asyncio
async def test_two_index_clicks_in_one_turn_resolve_against_the_original_list(mock_context):
"""``click(3)`` then ``click(5)``: the first click's observation lists only
two elements, yet the second index still lands on the original ``[5]``
(Display), with its semantics recorded from the original element."""
runner = _runner(mock_context, observations=[REFRESHED, [_el(1, "Display page", 1, 1)]])
state = _state(ORIGINAL)
messages: list = []
turn = _TurnRecord()
final, _, xml_list, _ = await runner._process_tool_calls(
[_click(3, "tc-1"), _click(5, "tc-2")],
state,
messages,
"thought",
{},
b"pre",
"xml",
0,
turn,
)
assert final is None
session = runner.executor._session
assert [args["target"] for _, args in session.calls] == [[500, 417], [500, 750]]
assert [status for _, status, _ in turn.actions] == ["success", "success"]
tool_msgs = [m for m in messages if isinstance(m, ToolMessage)]
assert [m.status for m in tool_msgs] == ["success", "success"]
# The state kept being refreshed: the NEXT observation shows the newest list.
assert [e["text"] for e in state.indexed_elements] == ["Display page"]
assert xml_list == "[1] Display page"
@pytest.mark.asyncio
async def test_second_index_beyond_the_refreshed_list_is_still_valid(mock_context):
"""Without the snapshot ``click(5)`` after the refresh was an
'Active index range is 1 to 2' error; with it the turn's own list is
authoritative and the recorded metadata comes from that list."""
runner = _runner(mock_context, observations=[REFRESHED, REFRESHED])
state = _state(ORIGINAL)
captured = []
real_execute = runner.executor.execute
async def spy(name, args, tc_id, st, **kwargs):
result = await real_execute(name, args, tc_id, st, **kwargs)
captured.append(result)
return result
runner.executor.execute = spy
await runner._process_tool_calls(
[_click(4, "tc-1"), _click(5, "tc-2")],
state,
[],
"thought",
{},
b"pre",
"xml",
0,
_TurnRecord(),
)
assert [r.status for r in captured] == ["success", "success"]
assert [r.metadata["target_coordinates"] for r in captured] == [[500, 583], [500, 750]]
assert [r.metadata["target_semantics"]["target_text"] for r in captured] == [
"Bluetooth",
"Display",
]
assert captured[1].metadata["target_semantics"]["target_bounds"] == [407, 729, 593, 771]
@pytest.mark.asyncio
async def test_explorer_candidates_registered_before_an_action_join_the_snapshot(mock_context):
"""``ask_explorer`` appends to the state's list in place; before any
action that list is the snapshot, so the new index resolves in the same
turn even though the action's observation then replaces the state list."""
runner = _runner(mock_context, observations=[REFRESHED])
state = _state(ORIGINAL)
async def fake_explorer(query, context_feedback, st):
st.indexed_points.append([540, 2200])
st.indexed_elements.append(_el(6, "Hidden gear", 540, 2200))
return "Found 'gear' at index 6.", True
runner.executor._ask_explorer = fake_explorer
await runner._process_tool_calls(
[
{"name": "ask_explorer", "args": {"query": "gear"}, "id": "tc-0"},
_click(6, "tc-1"),
],
state,
[],
"thought",
{},
b"pre",
"xml",
0,
_TurnRecord(),
)
session = runner.executor._session
assert [args["target"] for _, args in session.calls] == [[500, 917]]
assert [e["text"] for e in state.indexed_elements] == ["Back", "Wi-Fi toggle"]
@pytest.mark.asyncio
async def test_explorer_after_an_action_moves_the_snapshot_to_the_list_it_numbered(mock_context):
"""After an action replaced the state list, the explorer numbers its
candidates against that replaced list; the snapshot follows so the index
the explorer just handed out resolves correctly."""
runner = _runner(mock_context, observations=[REFRESHED, REFRESHED])
state = _state(ORIGINAL)
async def fake_explorer(query, context_feedback, st):
st.indexed_points.append([200, 2000])
st.indexed_elements.append(_el(len(st.indexed_elements) + 1, "Gear", 200, 2000))
return "Found.", True
runner.executor._ask_explorer = fake_explorer
await runner._process_tool_calls(
[
_click(3, "tc-1"),
{"name": "ask_explorer", "args": {"query": "gear"}, "id": "tc-2"},
_click(3, "tc-3"), # the explorer's index on the refreshed list
],
state,
[],
"thought",
{},
b"pre",
"xml",
0,
_TurnRecord(),
)
session = runner.executor._session
assert [args["target"] for _, args in session.calls] == [[500, 417], [185, 833]]
+39 -27
View File
@@ -13,8 +13,8 @@
# limitations under the License. # limitations under the License.
"""§1.3 assembly discipline: prompts are functions of configuration x scenario. """§1.3 assembly discipline: prompts are functions of configuration x scenario.
Both check gates off => zero context pollution; a plan carrying check lines Both check gates off => zero context pollution; with a gate on, the check-line
gets the behavior explainer regardless of switches.""" guidance lives once in the static system prompt, worded for that gate."""
import json import json
from pathlib import Path from pathlib import Path
@@ -23,10 +23,10 @@ from unittest.mock import MagicMock
import pytest import pytest
from artemis.agents.operator.prompts import ( from artemis.agents.operator.prompts import (
CheckItemsExplainerPromptComponent,
FeedbackPromptComponent, FeedbackPromptComponent,
PromptBuilder, PromptBuilder,
TemplatePromptComponent, TemplatePromptComponent,
render_transcript_static_system,
) )
from artemis.agents.planner.planner import build_planner_system_blocks from artemis.agents.planner.planner import build_planner_system_blocks
from artemis.context import ArtemisContext, ExecutionSetup from artemis.context import ArtemisContext, ExecutionSetup
@@ -95,35 +95,47 @@ def _state(**overrides):
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_explainer_renders_iff_plan_has_check_lines(tmp_path): async def test_check_line_guidance_lives_in_the_static_prompt_once():
component = CheckItemsExplainerPromptComponent() """The check-line guidance is a static-prompt section gated on the check
gates (never a per-turn observation block), worded for the active gate."""
off = await _render_main_template(disable_checker=True)
assert "*Check Lines*" not in off
assert "never declare a check passed" not in off
# Plan without check lines -> nothing rendered final_only = await _render_main_template(disable_midway_checks=True, disable_final_check=False)
builder = PromptBuilder() assert final_only.count("*Check Lines*") == 1
ctx = _mock_ctx(tmp_path, plan="- [ ] G\n") assert "never declare a check passed" in final_only
await component(builder, _state(), ctx) assert "restores deleted ones unless user guidance called for the change" in final_only
assert builder.human_parts == [] # No midway repair loop -> no repair budget to recite, and the grammar says so.
assert "reopens its milestone at most" not in final_only
assert "there is no midway repair loop" in final_only
# Plan with check lines -> explainer rendered midway = await _render_main_template(disable_midway_checks=False, checkpoint_max_repairs=3)
builder2 = PromptBuilder() assert midway.count("*Check Lines*") == 1
ctx2 = _mock_ctx(tmp_path, plan="- [ ] G\n - verify: V\n") assert "reopens its milestone at most 3 times" in midway
await component(builder2, _state(), ctx2) assert "- finding:" in midway
joined = "\n".join(p for p in builder2.human_parts if isinstance(p, str))
assert "check lines" in joined
assert "never declare a check passed yourself" in joined def test_transcript_static_system_recites_the_repair_budget_from_setup():
assert "automatically restored" in joined ctx = _mock_ctx(disable_midway_checks=False, checkpoint_max_repairs=5)
ctx.actuator = None
prompts = json.loads(OPERATOR_JSON.read_text(encoding="utf-8"))
text = render_transcript_static_system(prompts, ctx, _state(initial_goal="G"))
assert text.count("*Check Lines*") == 1
assert "reopens its milestone at most 5 times" in text
quiet = _mock_ctx(disable_checker=True)
quiet.actuator = None
assert "*Check Lines*" not in render_transcript_static_system(
prompts, quiet, _state(initial_goal="G")
)
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_explainer_is_content_driven_not_switch_driven(tmp_path): async def test_static_prompt_declares_the_user_guidance_channel_once():
"""A resumed old plan with check lines still gets the explanation even when text = await _render_main_template(disable_checker=True)
every check switch is off.""" assert text.count("--- User Guidance ---") == 1
component = CheckItemsExplainerPromptComponent() assert "outranks the task plan and its check lines" in text
builder = PromptBuilder()
ctx = _mock_ctx(tmp_path, plan="- [ ] G\n - assert: A\n", disable_checker=True)
await component(builder, _state(), ctx)
joined = "\n".join(p for p in builder.human_parts if isinstance(p, str))
assert "check lines" in joined
async def _render_main_template(**setup_kwargs) -> str: async def _render_main_template(**setup_kwargs) -> str:
+84 -1
View File
@@ -400,7 +400,7 @@ async def test_replay_steps_result_is_a_plain_tool_message(tmp_path):
assert tool_msg.name == "replay_steps" assert tool_msg.name == "replay_steps"
assert "- **Step 3 (Start: 9.0s)**" in tool_msg.content assert "- **Step 3 (Start: 9.0s)**" in tool_msg.content
assert "[Screen]: Saved the alarm." in tool_msg.content assert "[Screen]: Saved the alarm." in tool_msg.content
assert "[Planned Action]: Tapped 'Save' at [0.5, 0.5]" in tool_msg.content assert "[Action]: Tapped 'Save' at [0.5, 0.5]" in tool_msg.content
# --- Streamed-reasoning transcript (replay of the live Thought/Work text) ------------- # --- Streamed-reasoning transcript (replay of the live Thought/Work text) -------------
@@ -647,6 +647,89 @@ def test_format_history_marks_self_described_targets_only():
assert "target_resource_id" not in text assert "target_resource_id" not in text
def test_format_history_uses_session_offsets_and_skips_empty_steps():
"""Rows carry the session-relative ``T+mm:ss`` clock every agent shares
(from the step timestamp, else from the engine's ``97.7s`` label), and a
step with neither summary nor action renders no row."""
from artemis.agents.checker.checker import _format_history
ctx = _mock_ctx()
ctx.data_engine.session_start_time = 1000.0
ctx.data_engine.get_agent_friendly_steps.return_value = [
{"step_number": 1, "timestamp": 1097.7, "summary": "Opened the app"},
{"step_number": 2, "relative_time": "12.4s", "summary": "", "action_taken": None},
{
"step_number": 3,
"relative_time": "130.0s",
"summary": None,
"action_taken": {"action": "click", "coordinates": [1, 2], "target_text": "Login"},
},
]
lines = _format_history(ctx).splitlines()
assert len(lines) == 2
assert lines[0] == "- Step 1 (T+01:37): Opened the app"
assert lines[1].startswith("- Step 3 (T+02:10): ")
assert "'Login'" in lines[1]
async def _final_check_human_text(ctx) -> str:
report = CheckReport(verdicts=[])
structured = MagicMock()
structured.ainvoke = AsyncMock(return_value=report)
llm = MagicMock()
llm.with_structured_output.return_value = structured
response = MagicMock()
response.tool_calls = []
acomplete_mock = AsyncMock(return_value=response)
with (
patch("artemis.agents.checker.checker.get_llm", return_value=llm),
patch("artemis.agents.checker.checker.acomplete", new=acomplete_mock),
patch(
"artemis.agents.checker.checker._capture_final_screen",
new=AsyncMock(return_value=(None, "elements")),
),
):
await run_final_check(ctx, goal="the goal", plan_text="- [x] G", ledger=[], check_items=[])
messages = acomplete_mock.await_args.args[1]
human = messages[-1]
return "\n".join(
block["text"] for block in human.content if isinstance(block, dict) and "text" in block
)
@pytest.mark.asyncio
async def test_final_review_lists_user_guidance_from_the_step_records():
"""Every mid-run user instruction reaches the final review, read from the
step it was stamped on, so the goal is audited as amended by it."""
ctx = _mock_ctx()
ctx.data_engine.get_agent_friendly_steps.return_value = [
{"step_number": 1, "relative_time": "3.0s", "summary": "s1", "extra_metadata": {}},
{
"step_number": 2,
"relative_time": "65.0s",
"summary": "s2",
"extra_metadata": {"injected_instruction": "Skip the login"},
},
]
text = await _final_check_human_text(ctx)
section = text.split("# User guidance received during the run\n", 1)[1].split("\n\n", 1)[0]
assert section == '- Step 2 (T+01:05): "Skip the login"'
# The system prompt audits the goal as amended by that guidance.
from artemis.agents.checker.checker import _load_prompts
assert "as amended by the user guidance" in _load_prompts()["final_guide"]
@pytest.mark.asyncio
async def test_final_review_reports_no_guidance_explicitly():
ctx = _mock_ctx()
text = await _final_check_human_text(ctx)
assert "# User guidance received during the run\n(none)\n" in text
# --- tool results: status is structural, never sniffed from the words ----------------- # --- tool results: status is structural, never sniffed from the words -----------------
+36
View File
@@ -332,3 +332,39 @@ async def test_run_tool_message_status_is_structural(
assert messages[0].tool_call_id == "c1" assert messages[0].tool_call_id == "c1"
assert messages[0].status == expected_status assert messages[0].status == expected_status
assert messages[0].content == str(result) assert messages[0].content == str(result)
# --- submit_answer rendering ------------------------------------------------------------
def test_submit_answer_renders_numbered_steps(mock_context):
agent = Diagnoser(mock_context)
call = {
"name": "submit_answer",
"args": {
"analysis": "Airplane mode blocks the toggle.",
"actionable_steps": ["Turn off airplane mode", "Retry the toggle"],
},
}
outcome = agent._handle_submit_call(call, [])
assert outcome == (
"Analysis: Airplane mode blocks the toggle.\n"
"Actionable steps:\n"
"1. Turn off airplane mode\n"
"2. Retry the toggle"
)
def test_submit_answer_without_steps_omits_section(mock_context):
agent = Diagnoser(mock_context)
call = {
"name": "submit_answer",
"args": {"analysis": "Nothing to fix.", "actionable_steps": []},
}
outcome = agent._handle_submit_call(call, [])
assert outcome == "Analysis: Nothing to fix."
assert "Actionable" not in outcome
@@ -154,7 +154,14 @@ def test_render_burst_abort_and_exec_error():
plain = _disappeared(kind=KIND_EXEC_ERROR, category="general") plain = _disappeared(kind=KIND_EXEC_ERROR, category="general")
text = render_execution_incident(plain, STEPS) text = render_execution_incident(plain, STEPS)
assert "was dispatched, but the device/executor reported" in text # A failed action is quoted as an intent, never as a done outcome, and
# the sentence does not claim it was dispatched.
assert (
"your planned action `tap 'Skip' at [950, 288]` could not be executed; the"
" device/executor reported:" in text
)
assert "was dispatched" not in text
assert "`Tapped 'Skip'" not in text
def test_render_without_history_omits_the_trigger_hint(): def test_render_without_history_omits_the_trigger_hint():
@@ -179,15 +186,17 @@ async def test_component_renders_only_while_an_incident_is_open():
builder, SimpleNamespace(open_incident=_disappeared()), ctx, steps=STEPS builder, SimpleNamespace(open_incident=_disappeared()), ctx, steps=STEPS
) )
assert len(builder.human_parts) == 1 assert len(builder.human_parts) == 1
# Its own text part: the transcript scrub keeps it verbatim across turns.
assert builder.human_parts[0].startswith(EXECUTION_INCIDENT_MARKER) assert builder.human_parts[0].startswith(EXECUTION_INCIDENT_MARKER)
# Re-rendered every turn while open, so older copies leave the history.
assert builder.ephemeral_indices == [0]
def test_render_closed_notice_settles_the_intent(): def test_render_closed_notice_settles_the_intent():
closed = dict(_disappeared(), closed_at_step=15) closed = dict(_disappeared(), closed_at_step=15)
text = render_closed_incident(closed) text = render_closed_incident(closed)
assert text.startswith("--- Execution Incident (CLOSED at Step 15) ---") assert text.startswith("--- Execution Incident (CLOSED at Step 15) ---")
assert "opened at Step 12 on `Tapped 'Skip' at [950, 288]`" in text # The blocked action is quoted in intent form: it never happened.
assert "opened at Step 12 on `tap 'Skip' at [950, 288]`" in text
assert "already served, still pending, or no longer needed" in text assert "already served, still pending, or no longer needed" in text
@@ -201,6 +210,7 @@ async def test_component_renders_closed_notice_once_no_incident_is_open():
await ExecutionIncidentPromptComponent()(builder, state, ctx, steps=STEPS) await ExecutionIncidentPromptComponent()(builder, state, ctx, steps=STEPS)
assert len(builder.human_parts) == 1 assert len(builder.human_parts) == 1
assert builder.human_parts[0].startswith("--- Execution Incident (CLOSED at Step 15) ---") assert builder.human_parts[0].startswith("--- Execution Incident (CLOSED at Step 15) ---")
assert builder.ephemeral_indices == [0]
# An open incident takes precedence over a stale closed notice. # An open incident takes precedence over a stale closed notice.
builder = PromptBuilder() builder = PromptBuilder()
+314 -4
View File
@@ -14,13 +14,18 @@
"""Unit tests for Universal FlashRunner.""" """Unit tests for Universal FlashRunner."""
from unittest.mock import Mock, patch import base64
from types import SimpleNamespace
from unittest.mock import AsyncMock, Mock, patch
import pytest import pytest
from langchain_core.messages import HumanMessage, ToolMessage from langchain_core.messages import HumanMessage, ToolMessage
from artemis.agents.explorer.constants import ASK_EXPLORER_DESCRIPTION from artemis.agents.explorer.constants import ASK_EXPLORER_DESCRIPTION
from artemis.agents.flash.runner import FlashRunner from artemis.agents.flash.runner import FlashRunner
from artemis.agents.validator.tool_declarations import ASK_EXPLORER_TOOL from artemis.agents.validator.tool_declarations import (
ASK_EXPLORER_TOOL,
ToolExecutionResult,
)
from artemis.context import ArtemisContext from artemis.context import ArtemisContext
@@ -57,6 +62,138 @@ def test_flash_runner_tools_initialization(mock_context):
assert "report_task_status" in tool_names assert "report_task_status" in tool_names
# Validator's report_failure_analysis should NOT be in FlashRunner # Validator's report_failure_analysis should NOT be in FlashRunner
assert "report_failure_analysis" not in tool_names assert "report_failure_analysis" not in tool_names
# Flash has no note-writing tool and its prompt never teaches notes:
# the Validator set's note readers are not bound.
assert "read_note" not in tool_names
assert "list_notes" not in tool_names
assert tool_names[:2] == ["click", "click_sequence"]
# --- Post-action screenshot fallback (failed action / unknown tool) ----------------
def _screen(elements):
return SimpleNamespace(
base64=base64.b64encode(b"POST-IMG").decode("ascii"),
elements=elements,
width=1080,
height=2400,
)
def _failed_click(tc_id="tc1"):
return ToolExecutionResult(
tool_call_id=tc_id,
tool_name="click",
status="error",
text_summary="Error during click: Invalid target format: 3",
)
@pytest.mark.asyncio
async def test_fallback_observation_renders_the_indexed_ui_list(mock_context, tmp_path):
"""After an action that produced no observation, the fallback capture must
hand the next tail the same indexed minimal list the success path builds
(never the raw element dict repr) and refresh the state like it."""
with patch("artemis.controllers.unified_controller.get_driver"):
runner = FlashRunner(mock_context, goal="g")
runner.controller = Mock()
runner.controller.get_screen_data = AsyncMock(
return_value=_screen(
[
{
"text": "Settings",
"content-desc": "",
"bounds": "[40,120][1040,270]",
"class": "android.widget.TextView",
"resource-id": "android:id/title",
}
]
)
)
state = Mock()
state.latest_screenshot = "old.jpg"
exec_result = _failed_click()
with patch("artemis.mcp.observation.get_temp_dir", return_value=tmp_path):
post = await runner._capture_post_screenshot(
exec_result, "click", frozenset({"click"}), state
)
assert post == b"POST-IMG"
assert isinstance(exec_result.ui_elements_text, str)
assert exec_result.ui_elements_text == "[1] Text: 'Settings' | Bounds: [37,50][962,112]"
assert "{'text'" not in exec_result.ui_elements_text
# State refreshed exactly as the executor's success path does.
assert state.indexed_elements[0]["index"] == 1
assert state.indexed_elements[0]["text"] == "Settings"
assert state.indexed_points == [state.indexed_elements[0]["center"]]
assert state.latest_screenshot != "old.jpg"
assert state.latest_screenshot.startswith(str(tmp_path))
assert exec_result.screenshot_path == state.latest_screenshot
@pytest.mark.asyncio
async def test_fallback_observation_is_skipped_for_non_action_tools(mock_context):
with patch("artemis.controllers.unified_controller.get_driver"):
runner = FlashRunner(mock_context, goal="g")
runner.controller = Mock()
runner.controller.get_screen_data = AsyncMock(return_value=_screen([]))
state = Mock()
exec_result = ToolExecutionResult(
tool_call_id="tc", tool_name="ask_explorer", status="success", text_summary="found"
)
post = await runner._capture_post_screenshot(
exec_result, "ask_explorer", frozenset({"click"}), state
)
assert post is None
assert exec_result.ui_elements_text is None
runner.controller.get_screen_data.assert_not_awaited()
@pytest.mark.asyncio
async def test_fallback_observation_keeps_the_executors_ui_list(mock_context, tmp_path):
"""An action that reported a UI list but no screenshot keeps its own list."""
with patch("artemis.controllers.unified_controller.get_driver"):
runner = FlashRunner(mock_context, goal="g")
runner.controller = Mock()
runner.controller.get_screen_data = AsyncMock(
return_value=_screen([{"text": "Other", "bounds": "[0,0][100,100]"}])
)
exec_result = ToolExecutionResult(
tool_call_id="tc",
tool_name="click",
status="success",
text_summary="click executed",
ui_elements_text="[1] Text: 'Mine' | Bounds: [0,0][10,10]",
)
with patch("artemis.mcp.observation.get_temp_dir", return_value=tmp_path):
post = await runner._capture_post_screenshot(
exec_result, "click", frozenset({"click"}), None
)
assert post == b"POST-IMG"
assert exec_result.ui_elements_text == "[1] Text: 'Mine' | Bounds: [0,0][10,10]"
@pytest.mark.asyncio
async def test_fallback_observation_failure_is_contained(mock_context):
with patch("artemis.controllers.unified_controller.get_driver"):
runner = FlashRunner(mock_context, goal="g")
runner.controller = Mock()
runner.controller.get_screen_data = AsyncMock(side_effect=RuntimeError("adb gone"))
state = Mock()
state.latest_screenshot = "old.jpg"
exec_result = _failed_click()
post = await runner._capture_post_screenshot(exec_result, "click", frozenset({"click"}), state)
assert post is None
assert exec_result.ui_elements_text is None
assert state.latest_screenshot == "old.jpg"
def test_flash_runner_executor_follows_the_flash_explorer_tier(mock_context): def test_flash_runner_executor_follows_the_flash_explorer_tier(mock_context):
@@ -202,10 +339,12 @@ def test_flash_records_render_in_the_ledger_with_their_arguments(mock_context):
typed = _record(runner, "input_text", {"target": [500, 600], "text": "hello"}) typed = _record(runner, "input_text", {"target": [500, 600], "text": "hello"})
assert format_actions_clean(typed) == "Inputted 'hello' into field at [500, 600]" assert format_actions_clean(typed) == "Inputted 'hello' into field at [500, 600]"
# A directional swipe names the direction first; the recorded canonical
# path is the parenthesised detail.
swipe_dir = _record(runner, "swipe", {"direction": "up"}) swipe_dir = _record(runner, "swipe", {"direction": "up"})
assert format_actions_clean(swipe_dir) == "Swiped up" assert format_actions_clean(swipe_dir) == "Swiped up (from [600, 700] to [600, 300])"
swipe_fa = _record(runner, "swipe", {"action": "down"}) swipe_fa = _record(runner, "swipe", {"action": "down"})
assert format_actions_clean(swipe_fa) == "Swiped down" assert format_actions_clean(swipe_fa) == "Swiped down (from [600, 300] to [600, 700])"
swipe_coords = _record(runner, "swipe", {"coordinates": [556, 289, 556, 124]}) swipe_coords = _record(runner, "swipe", {"coordinates": [556, 289, 556, 124]})
assert format_actions_clean(swipe_coords) == "Swiped from [556, 289] to [556, 124]" assert format_actions_clean(swipe_coords) == "Swiped from [556, 289] to [556, 124]"
@@ -246,3 +385,174 @@ def test_flash_records_the_executors_target_semantics(mock_context):
bare = _record(runner, "click", {"target": [320, 399]}, metadata={"target_semantics": {}}) bare = _record(runner, "click", {"target": [320, 399]}, metadata={"target_semantics": {}})
assert "target_description" not in bare assert "target_description" not in bare
assert "target_text" not in bare assert "target_text" not in bare
def test_flash_records_an_index_target_as_the_resolved_point_with_observed_semantics(
mock_context,
):
"""An element index is not a coordinate: the record carries the normalized
point the executor resolved it to, plus the element's observed fields in
the shape Pro records (never a self-described label)."""
from artemis.utils.task_tree import format_actions_clean
with patch("artemis.controllers.unified_controller.get_driver"):
runner = FlashRunner(mock_context, goal="g")
click = _record(
runner,
"click",
{"target": 3},
metadata={
"target_coordinates": [500, 520],
"target_semantics": {
"target_text": "Wi-Fi",
"target_bounds": [37, 480, 962, 560],
"target_resource_id": "android:id/title",
"target_class": "android.widget.TextView",
},
},
)
assert click["coordinates"] == [500, 520]
assert click["normalized_coordinates"] == [500, 520]
assert click["target_text"] == "Wi-Fi"
assert click["target_bounds"] == [37, 480, 962, 560]
assert "target_description" not in click
assert click["args"] == {"target": 3} # the tool call itself, verbatim
assert format_actions_clean(click) == "Tapped 'Wi-Fi' at [500, 520]"
typed = _record(
runner,
"input_text",
{"target": 2, "text": "hello"},
metadata={"target_coordinates": [500, 300], "target_semantics": {"target_text": "Search"}},
)
assert typed["coordinates"] == [500, 300]
assert format_actions_clean(typed) == "Inputted 'hello' into 'Search' at [500, 300]"
@pytest.mark.asyncio
async def test_visual_lens_receives_the_recorded_action_shape(mock_context):
"""The lens gets the recorded action minus its verb (what the Pro
summarizer hands it), so ``manage_app``'s own ``action`` argument can no
longer overwrite the action name in the rendered phrase."""
from artemis.agents.flash.runner import _TurnRecord
from artemis.agents.flash.summarizer import VisualStepSummarizer
with patch("artemis.controllers.unified_controller.get_driver"):
runner = FlashRunner(mock_context, goal="g")
runner.summarizer = Mock()
runner.executor.execute = AsyncMock(
return_value=ToolExecutionResult(
tool_call_id="tc",
tool_name="manage_app",
status="success",
text_summary="Launched app 'Settings' (com.android.settings); foreground confirmed.",
screenshot_bytes=b"post",
metadata={"target_semantics": {}, "target_coordinates": None},
)
)
args = {"action": "launch", "app_name": "Settings"}
await runner._execute_and_record_action(
"manage_app", args, "tc", Mock(), [], "thought", {}, b"pre", "xml", 0, _TurnRecord()
)
runner.summarizer.dispatch.assert_called_once()
kwargs = runner.summarizer.dispatch.call_args.kwargs
assert kwargs["action_name"] == "manage_app"
assert "action" not in kwargs["action_args"]
assert kwargs["action_args"]["args"] == args
assert VisualStepSummarizer._action_phrase("manage_app", kwargs["action_args"]) == (
"Launched app 'Settings'"
)
# --- Native thinking (thought summaries) reach the step record ----------------------
def _thinking_response(content):
from types import SimpleNamespace
return SimpleNamespace(content=content, tool_calls=[])
def test_extract_response_thinking_reads_only_thinking_blocks(mock_context):
"""``include_thoughts`` replies carry ``thinking`` blocks next to ``text``
blocks; only the former are native thinking, and plain-string replies
carry none."""
with patch("artemis.controllers.unified_controller.get_driver"):
runner = FlashRunner(mock_context, goal="g")
blocks = [
{"type": "thinking", "thinking": "**Verifying**\n\nMaps is open."},
{"type": "text", "text": "Tapping the search bar."},
{"type": "thinking", "thinking": " "},
]
assert runner._extract_response_thinking(_thinking_response(blocks)) == (
"**Verifying**\n\nMaps is open."
)
assert runner._extract_response_text(_thinking_response(blocks)) == "Tapping the search bar."
assert runner._extract_response_thinking(_thinking_response("plain reply")) == ""
assert (
runner._extract_response_thinking(_thinking_response([{"type": "text", "text": "t"}])) == ""
)
def test_record_action_step_persists_native_thinking(mock_context):
from types import SimpleNamespace
with patch("artemis.controllers.unified_controller.get_driver"):
runner = FlashRunner(mock_context, goal="g")
runner.ctx.data_engine = Mock()
runner.ctx.data_engine.current_step_id = None
runner.ctx.data_engine.record_step.return_value = "step-1"
exec_result = SimpleNamespace(
status="success", text_summary="click executed", metadata={}, ui_elements_text="ui"
)
runner._record_action_step(
"click",
{"target": [1, 2]},
exec_result,
"thought",
{},
b"pre",
["xml"],
b"post",
native_text="native summary",
)
kwargs = runner.ctx.data_engine.record_step.call_args.kwargs
assert kwargs["operator_raw_thinking"] == "thought"
assert kwargs["operator_native_thinking"] == "native summary"
runner._record_action_step(
"click", {"target": [1, 2]}, exec_result, "thought", {}, b"pre", ["xml"], b"post"
)
assert runner.ctx.data_engine.record_step.call_args.kwargs["operator_native_thinking"] is None
@pytest.mark.asyncio
async def test_final_report_persists_native_thinking(mock_context):
with patch("artemis.controllers.unified_controller.get_driver"):
runner = FlashRunner(mock_context, goal="g")
runner.ctx.data_engine = Mock()
runner.ctx.data_engine.current_step_id = None
runner.summarizer = None
messages = []
report = await runner._finalize_task_report(
"report_task_status",
{"status": "completed"},
"tc",
"final text",
{},
b"pre",
["xml"],
messages,
native_text="native summary",
)
assert report == {"status": "completed"}
kwargs = runner.ctx.data_engine.record_step.call_args.kwargs
assert kwargs["operator_raw_thinking"] == "final text"
assert kwargs["operator_native_thinking"] == "native summary"
assert isinstance(messages[-1], ToolMessage)
+276 -37
View File
@@ -17,33 +17,42 @@
Covers: the ``video_analyzer`` availability gate (same as the Pro graph) and Covers: the ``video_analyzer`` availability gate (same as the Pro graph) and
its prompt segment, the unbounded-by-default turn limit, the Pro-shaped its prompt segment, the unbounded-by-default turn limit, the Pro-shaped
observation tail (``# CURRENT OBSERVATION [T+mm:ss]`` header, screenshot, observation tail (``# CURRENT OBSERVATION [T+mm:ss]`` header, screenshot,
``--- Visible UI Elements ---`` list), and the end-to-end loop: committed ``--- Visible UI Elements ---`` list, ephemeral per-turn notices, the
turns carry text-only tool messages plus an ``--- Action Execution Result reasoning reminder after a silent turn), and the end-to-end loop: committed
(T+mm:ss) ---`` message, every recorded step of a multi-action turn is turns carry text-only tool messages and no execution-result message unless
registered on the committed turn, the final-turn tool restriction only an action failed, every recorded step of a multi-action turn is registered
applies to bounded loops, and a no-tool-call turn is nudged, not terminated. on the committed turn, the final-turn tool restriction only applies to
bounded loops, and a no-tool-call turn is nudged, not terminated.
""" """
import base64 import base64
import re import re
from unittest.mock import AsyncMock, Mock, patch from unittest.mock import AsyncMock, Mock, PropertyMock, patch
import pytest import pytest
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage, ToolMessage from langchain_core.messages import AIMessage, HumanMessage, SystemMessage, ToolMessage
from artemis.agents.flash.runner import FlashRunner, _TurnRecord from artemis.agents.flash.runner import _FINAL_TURN_WARNING, FlashRunner, _TurnRecord
from artemis.agents.operator.prompts import (
REASONING_REMINDER,
USER_GUIDANCE_MARKER,
UserGuidance,
render_user_guidance,
render_user_instruction,
)
from artemis.agents.validator.tool_declarations import ToolExecutionResult from artemis.agents.validator.tool_declarations import ToolExecutionResult
from artemis.context import ArtemisContext from artemis.context import ArtemisContext
from artemis.graph.state import State from artemis.graph.state import State
from artemis.memory.transcript import ( from artemis.memory.transcript import (
EPHEMERAL_BLOCKS_KEY,
EXECUTION_RESULT_MARKER, EXECUTION_RESULT_MARKER,
PRO_UI_LIST_MARKER, PRO_UI_LIST_MARKER,
TranscriptLedger, TranscriptLedger,
) )
OBSERVATION_HEADER_RE = re.compile(r"^# CURRENT OBSERVATION \[T\+\d{2,}:\d{2}\]$") OBSERVATION_HEADER_RE = re.compile(r"^# CURRENT OBSERVATION \[T\+\d{2,}:\d{2}\]$")
RESULT_RE = re.compile( FAILED_RESULT_RE = re.compile(
rf"^{re.escape(EXECUTION_RESULT_MARKER)} \(T\+\d{{2,}}:\d{{2}}\) ---\nStatus: dispatched$" rf"^{re.escape(EXECUTION_RESULT_MARKER)} \(T\+\d{{2,}}:\d{{2}}\) ---\nStatus: failed\n"
) )
@@ -173,31 +182,139 @@ def test_turn_limit_semantics(mock_context):
def test_observation_tail_has_pro_shape(mock_context): def test_observation_tail_has_pro_shape(mock_context):
"""The objective lives in the system prompt only; the tail opens with the
observation header on turn 1 exactly as on every later turn, and carries
no reminder unless the previous turn was silent."""
with patch("artemis.controllers.unified_controller.get_driver"): with patch("artemis.controllers.unified_controller.get_driver"):
runner = FlashRunner(mock_context, goal="Open Settings") runner = FlashRunner(mock_context, goal="Open Settings")
ledger = TranscriptLedger() ledger = TranscriptLedger()
tail = runner._build_tail(ledger, 1, b"IMG", "[1] Settings") tail = runner._build_tail(ledger, 1, b"IMG", "[1] Settings")
texts = [b["text"] for b in tail.content if b["type"] == "text"] texts = [b["text"] for b in tail.content if b["type"] == "text"]
assert texts[0] == "Your objective is: Open Settings" assert OBSERVATION_HEADER_RE.match(texts[0])
assert OBSERVATION_HEADER_RE.match(texts[1]) assert texts[1] == "--- Current Screenshot ---"
assert texts[2] == "--- Current Screenshot ---" assert texts[2] == f"{PRO_UI_LIST_MARKER}\n[1] Settings"
assert texts[3] == f"{PRO_UI_LIST_MARKER}\n[1] Settings" assert len(texts) == 3
assert sum(1 for b in tail.content if b["type"] == "image_url") == 1 assert sum(1 for b in tail.content if b["type"] == "image_url") == 1
joined = "".join(texts)
assert "Open Settings" not in joined and "objective" not in joined.lower()
assert "CRITICAL RULE" not in joined and REASONING_REMINDER not in joined
assert EPHEMERAL_BLOCKS_KEY not in tail.additional_kwargs
later = runner._build_tail( later = runner._build_tail(
ledger, 5, None, None, injected="[INJECTED] stop", notices=["nudge"], is_final=True ledger,
5,
None,
None,
injected=UserGuidance(body='User instruction: "stop"', wrapper="[INJECTED]"),
notices=["nudge"],
is_final=True,
) )
texts = [b["text"] for b in later.content] texts = [b["text"] for b in later.content]
assert OBSERVATION_HEADER_RE.match(texts[0]) assert OBSERVATION_HEADER_RE.match(texts[0])
assert "Your objective is" not in "".join(texts) assert "objective" not in "".join(texts).lower()
assert texts[1:] == [ assert texts[1:] == [
"nudge", "nudge",
"[INJECTED] stop", "[INJECTED]",
"[WARNING] This is your final turn; only 'report_task_status' is available.", 'User instruction: "stop"',
_FINAL_TURN_WARNING,
] ]
def test_per_turn_notices_are_marked_ephemeral(mock_context):
"""Nudges, the user-guidance wrapper, the final-turn warning and the
reasoning reminder are only meaningful for the turn they were built for:
every one of them is flagged ephemeral (by block index) so the scrub edge
drops them before the message freezes; the observation blocks never are."""
with patch("artemis.controllers.unified_controller.get_driver"):
runner = FlashRunner(mock_context, goal="Open Settings")
ledger = TranscriptLedger()
guidance = render_user_guidance("stop", release_loop=False, has_plan=False)
tail = runner._build_tail(
ledger,
3,
b"IMG",
"[1] Settings",
notices=["nudge"],
injected=guidance,
is_final=True,
previous_turn_silent=True,
)
texts = [b.get("text") for b in tail.content]
# header, screenshot label, image, UI list, then the per-turn notices with
# the persistent instruction body right after its wrapper
assert texts[4:] == [
"nudge",
guidance.wrapper,
guidance.body,
_FINAL_TURN_WARNING,
REASONING_REMINDER,
]
assert tail.additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [4, 5, 7, 8]
def test_injected_instruction_body_outlives_its_wrapper(mock_context):
"""Defect: a standing instruction ("don't send messages from now on")
vanished after one turn because the whole guidance block was ephemeral.
Now the wrapper is ephemeral and the verbatim body is a regular block, so
it stays in the active window until the turn is chunked."""
with patch("artemis.controllers.unified_controller.get_driver"):
runner = FlashRunner(mock_context, goal="g")
ledger = TranscriptLedger()
guidance = render_user_guidance(
"don't send messages from now on", has_plan=False, offset_label="T+01:05"
)
tail = runner._build_tail(ledger, 2, b"IMG", "[1] Send", injected=guidance)
texts = [b.get("text") for b in tail.content]
wrapper_index = texts.index(guidance.wrapper)
body_index = texts.index(guidance.body)
ephemeral = tail.additional_kwargs[EPHEMERAL_BLOCKS_KEY]
assert wrapper_index in ephemeral
assert body_index not in ephemeral
assert body_index == wrapper_index + 1
assert guidance.body == 'User instruction (T+01:05): "don\'t send messages from now on"'
assert guidance.body == render_user_instruction(
"don't send messages from now on", offset_label="T+01:05"
)
# The wrapper frames the quoted line but no longer carries the words.
assert guidance.wrapper.startswith(USER_GUIDANCE_MARKER)
assert "don't send messages" not in guidance.wrapper
# Scrub edge parity: once the turn has left the live position (a newer
# committed observation exists) the wrapper is gone and the body is
# still in the committed turn.
ledger.stage_turn([tail, AIMessage(content="ok")])
ledger.commit_staged(step_key="s1")
ledger.stage_turn([runner._build_tail(ledger, 3, b"IMG", "[1] Send"), AIMessage(content="ok")])
ledger.commit_staged(step_key="s2")
rendered = ledger.render([runner._build_tail(ledger, 4, b"IMG", "[1] Send")])
committed = next(
m
for m in rendered
if isinstance(m, HumanMessage) and guidance.body in [b.get("text") for b in m.content]
)
committed_texts = [b.get("text", "") for b in committed.content if isinstance(b, dict)]
assert guidance.wrapper not in committed_texts
assert not any(t.startswith(USER_GUIDANCE_MARKER) for t in committed_texts)
def test_reasoning_reminder_follows_a_silent_turn_only(mock_context):
with patch("artemis.controllers.unified_controller.get_driver"):
runner = FlashRunner(mock_context, goal="g")
ledger = TranscriptLedger()
quiet = runner._build_tail(ledger, 2, None, "[1] x", previous_turn_silent=False)
assert REASONING_REMINDER not in [b["text"] for b in quiet.content]
reminded = runner._build_tail(ledger, 2, None, "[1] x", previous_turn_silent=True)
texts = [b["text"] for b in reminded.content]
assert texts[-1] == REASONING_REMINDER
assert texts.count(REASONING_REMINDER) == 1
assert reminded.additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [len(texts) - 1]
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Injected instruction: verbatim text for the record and the lens, wrapped # Injected instruction: verbatim text for the record and the lens, wrapped
# notice for the observation tail only # notice for the observation tail only
@@ -209,8 +326,9 @@ async def test_injected_instruction_splits_verbatim_text_from_operator_notice(
mock_context, tmp_path mock_context, tmp_path
): ):
"""The user's words are what the step record, the chunk ledger and the """The user's words are what the step record, the chunk ledger and the
visual lens receive (Pro parity); the 'You MUST immediately ...' directive visual lens receive (Pro parity); the shared ``--- User Guidance ---``
is operator-facing and rides on the observation tail alone.""" block (the Pro rendering, without a plan to edit) rides on the
observation tail alone."""
import json import json
(tmp_path / "injected_instruction.json").write_text( (tmp_path / "injected_instruction.json").write_text(
@@ -225,11 +343,15 @@ async def test_injected_instruction_splits_verbatim_text_from_operator_notice(
instruction, notice = await runner._read_injected_instruction() instruction, notice = await runner._read_injected_instruction()
assert instruction == "Skip the popup and log in" assert instruction == "Skip the popup and log in"
assert notice.startswith( assert notice == render_user_guidance(
"[REAL-TIME INJECTED INSTRUCTION from user]: Skip the popup and log in" "Skip the popup and log in", release_loop=True, has_plan=False
) )
assert "You MUST immediately follow this instruction" in notice assert notice.wrapper.startswith(USER_GUIDANCE_MARKER)
assert "authorized stopping any ongoing monitoring loop" in notice assert notice.body == 'User instruction: "Skip the popup and log in"'
assert "outranks your current milestones" in notice.wrapper
assert "task plan" not in notice.wrapper
assert "authorized stopping any ongoing monitoring loop" in notice.wrapper
assert "REAL-TIME INJECTED" not in notice.wrapper and "You MUST" not in notice.wrapper
assert not (tmp_path / "injected_instruction.json").exists() assert not (tmp_path / "injected_instruction.json").exists()
with patch("artemis.controllers.unified_controller.get_driver"): with patch("artemis.controllers.unified_controller.get_driver"):
@@ -267,7 +389,7 @@ def _make_runner(mock_context, responses, max_turns=None):
runner._invoke_model = AsyncMock(side_effect=_invoke) runner._invoke_model = AsyncMock(side_effect=_invoke)
runner.executor.execute = AsyncMock( runner.executor.execute = AsyncMock(
side_effect=lambda name, args, tc_id, state: _exec_result( side_effect=lambda name, args, tc_id, state, **_: _exec_result(
tc_id, name, f"IMG-{tc_id}".encode() tc_id, name, f"IMG-{tc_id}".encode()
) )
) )
@@ -289,9 +411,10 @@ _INITIAL_OBSERVATION = ("shot0.png", b"IMG0", "[1] Settings")
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_run_builds_prompt_from_ledger_with_session_offsets(mock_context): async def test_run_builds_prompt_from_ledger_with_session_offsets(mock_context):
"""Turn 2 must see: the static system prefix, the committed turn 1 (its """Turn 2 must see: the static system prefix, the committed turn 1 (its
tail, the AI message, text-only tool messages and an execution-result tail, the AI message, text-only tool messages that already carry each
message with a ``T+`` offset) and a fresh observation tail carrying the action's outcome — no execution-result message repeats them) and a fresh
last action's post screenshot — the Pro transcript shape, with no cap.""" observation tail carrying the last action's post screenshot — the Pro
transcript shape, with no cap."""
responses = [ responses = [
AIMessage( AIMessage(
content="I will tap then wait.", content="I will tap then wait.",
@@ -327,33 +450,146 @@ async def test_run_builds_prompt_from_ledger_with_session_offsets(mock_context):
tail1 = turn2_messages[1] tail1 = turn2_messages[1]
assert isinstance(tail1, HumanMessage) assert isinstance(tail1, HumanMessage)
tail1_texts = [b["text"] for b in tail1.content if b["type"] == "text"] tail1_texts = [b["text"] for b in tail1.content if b["type"] == "text"]
assert tail1_texts[0] == "Your objective is: Open Settings" assert OBSERVATION_HEADER_RE.match(tail1_texts[0])
assert OBSERVATION_HEADER_RE.match(tail1_texts[1]) assert "objective" not in "".join(tail1_texts).lower()
assert "Open Settings" in turn2_messages[0].content # stated once, in the prefix
assert isinstance(turn2_messages[2], AIMessage) assert isinstance(turn2_messages[2], AIMessage)
tool_msgs = [m for m in turn2_messages if isinstance(m, ToolMessage)] tool_msgs = [m for m in turn2_messages if isinstance(m, ToolMessage)]
assert [m.content for m in tool_msgs] == ["click executed", "wait_for_delay executed"] assert [m.content for m in tool_msgs] == ["click executed", "wait_for_delay executed"]
result_msg = turn2_messages[5] # Every action succeeded: the tool messages are the outcome, nothing repeats them.
assert isinstance(result_msg, HumanMessage) assert not any(
assert RESULT_RE.match(result_msg.content[0]["text"]) isinstance(m, HumanMessage)
and any(EXECUTION_RESULT_MARKER in b.get("text", "") for b in m.content)
for m in turn2_messages
)
tail2 = turn2_messages[6] tail2 = turn2_messages[5]
tail2_texts = [b["text"] for b in tail2.content if b["type"] == "text"] tail2_texts = [b["text"] for b in tail2.content if b["type"] == "text"]
assert OBSERVATION_HEADER_RE.match(tail2_texts[0]) assert OBSERVATION_HEADER_RE.match(tail2_texts[0])
assert "Your objective is" not in "".join(tail2_texts) assert "objective" not in "".join(tail2_texts).lower()
images = [b for b in tail2.content if b["type"] == "image_url"] images = [b for b in tail2.content if b["type"] == "image_url"]
assert len(images) == 1 assert len(images) == 1
assert images[0]["image_url"]["url"].endswith(base64.b64encode(b"IMG-tc2").decode()) assert images[0]["image_url"]["url"].endswith(base64.b64encode(b"IMG-tc2").decode())
assert tail2_texts[1] == "--- Current Screenshot ---" assert tail2_texts[1] == "--- Current Screenshot ---"
assert tail2_texts[2] == f"{PRO_UI_LIST_MARKER}\n[1] next" assert tail2_texts[2] == f"{PRO_UI_LIST_MARKER}\n[1] next"
assert len(turn2_messages) == 7 assert len(turn2_messages) == 6
# Both actions of the multi-action turn were registered on the committed turn. # Both actions of the multi-action turn were registered on the committed turn.
ledger = mock_context.transcript_ledger ledger = mock_context.transcript_ledger
assert ledger.unchunked_turns()[0]["step_keys"] == ["tc1", "tc2"] assert ledger.unchunked_turns()[0]["step_keys"] == ["tc1", "tc2"]
@pytest.mark.asyncio
async def test_run_failed_action_turn_ends_with_the_error_result(mock_context):
"""A failed action is the one case worth a result message: it carries the
error the turn ended on, after the tool message."""
responses = [
AIMessage(
content="Tapping.",
tool_calls=[{"name": "click", "args": {"target": 99}, "id": "tc1"}],
),
_report(),
]
with (
patch("artemis.controllers.unified_controller.get_driver"),
patch(
"artemis.agents.flash.runner.capture_screenshot_and_parse_ui",
AsyncMock(return_value=_INITIAL_OBSERVATION),
),
):
runner = _make_runner(mock_context, responses, max_turns=0)
runner.executor.execute = AsyncMock(
return_value=ToolExecutionResult(
tool_call_id="tc1",
tool_name="click",
status="error",
text_summary="Error during click: Invalid target index 99.",
)
)
runner._capture_post_screenshot = AsyncMock(return_value=None)
await runner.run(State(initial_goal="Open Settings"))
turn2_messages = runner.calls[1]["messages"]
result_msgs = [
m
for m in turn2_messages
if isinstance(m, HumanMessage)
and any(EXECUTION_RESULT_MARKER in b.get("text", "") for b in m.content)
]
assert len(result_msgs) == 1
text = result_msgs[0].content[0]["text"]
assert FAILED_RESULT_RE.match(text)
assert "Invalid target index 99" in text
assert isinstance(turn2_messages[turn2_messages.index(result_msgs[0]) - 1], ToolMessage)
@pytest.mark.asyncio
async def test_run_reminds_after_a_silent_turn_without_a_bounce(mock_context):
"""The ledger judges the committed turn (sibling logic, authoritative
here); the runner only reads ``last_turn_silent`` when it builds the
next tail: one reminder block, marked ephemeral, one model call per turn."""
responses = [
AIMessage(
content="", # bare tool call, no visible reasoning
tool_calls=[
{"name": "click", "args": {"target": 1}, "id": "tc1"},
],
),
_report(),
]
with (
patch("artemis.controllers.unified_controller.get_driver"),
patch(
"artemis.agents.flash.runner.capture_screenshot_and_parse_ui",
AsyncMock(return_value=_INITIAL_OBSERVATION),
),
patch.object(TranscriptLedger, "last_turn_silent", new_callable=PropertyMock) as silent,
):
silent.side_effect = lambda: silent.call_count > 1 # turn 1: False, later: True
runner = _make_runner(mock_context, responses, max_turns=0)
await runner.run(State(initial_goal="Open Settings"))
assert runner._invoke_model.await_count == 2
tail1 = runner.calls[0]["messages"][-1]
assert REASONING_REMINDER not in [b.get("text") for b in tail1.content]
tail2 = runner.calls[1]["messages"][-1]
texts = [b.get("text") for b in tail2.content]
assert texts.count(REASONING_REMINDER) == 1
assert tail2.additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [texts.index(REASONING_REMINDER)]
@pytest.mark.asyncio
async def test_run_calibrates_the_ledger_only_from_measured_usage(mock_context):
"""A provider-reported prompt size calibrates the chars-per-token ratio
with the sent messages; an estimated size (no usage metadata) records the
base but never calibrates — the estimate is itself derived from character
counts, so calibrating on it would be circular."""
measured = AIMessage(
content="Tapping.",
tool_calls=[{"name": "click", "args": {"target": 1}, "id": "tc1"}],
usage_metadata={"input_tokens": 1200, "output_tokens": 5, "total_tokens": 1205},
)
responses = [measured, _report()]
with (
patch("artemis.controllers.unified_controller.get_driver"),
patch(
"artemis.agents.flash.runner.capture_screenshot_and_parse_ui",
AsyncMock(return_value=_INITIAL_OBSERVATION),
),
patch.object(TranscriptLedger, "record_prompt_tokens") as record,
):
runner = _make_runner(mock_context, responses, max_turns=0)
await runner.run(State(initial_goal="Open Settings"))
assert record.call_count == 2
first_call, second_call = record.call_args_list
assert first_call.args[0] == 1200 and first_call.kwargs["messages"] is not None
# The report turn carries no usage metadata: estimated, recorded, not calibrated.
assert second_call.args[0] >= 1 and second_call.kwargs["messages"] is None
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_run_final_turn_restricts_tools_and_fails_without_report(mock_context): async def test_run_final_turn_restricts_tools_and_fails_without_report(mock_context):
responses = [AIMessage(content="I give up.", tool_calls=[])] responses = [AIMessage(content="I give up.", tool_calls=[])]
@@ -411,11 +647,14 @@ def test_turn_record_result_is_none_for_helper_only_turns():
assert _TurnRecord().result() is None assert _TurnRecord().result() is None
def test_turn_record_result_is_dispatched_when_every_action_was_accepted(): def test_turn_record_result_is_none_when_every_action_succeeded():
"""The tool messages already say ``Swiped up.``; a ``Status: dispatched``
line after them would only duplicate the outcome (and put two human
messages back to back)."""
record = _TurnRecord( record = _TurnRecord(
actions=[("click", "success", "Tapped at [500, 500]"), ("swipe", "success", "Swiped up.")] actions=[("click", "success", "Tapped at [500, 500]"), ("swipe", "success", "Swiped up.")]
) )
assert record.result() == {"status": "dispatched"} assert record.result() is None
def test_turn_record_failed_result_keeps_an_executor_message_that_names_the_action(): def test_turn_record_failed_result_keeps_an_executor_message_that_names_the_action():
+390 -15
View File
@@ -12,13 +12,11 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
"""Unit tests for the Flash scrub-edge compressor (history redesign §3.2, M1). """Tests for screenshot summaries, text cleanup, and frozen history.
Covers the M1 acceptance items: the legacy-equivalent success-path product Covers pending-summary recovery and expiry, failed-summary placeholders,
(asserted directly since the legacy ``compress_flash_messages`` reference was independent text and image depths, ephemeral-block remapping, and stable
removed in M5), the three race regressions (grace recovery, grace exhaustion content after both cleanup passes.
without backfill, failed-summary placeholder), the freeze invariant (bytes
behind the scrub edge never change), and depth-1 XML stripping equivalence.
""" """
import json import json
@@ -83,9 +81,10 @@ def test_success_path_product_behind_the_scrub_edge(mock_context):
by the ``--- Historical Visual Transition ---`` summary block. by the ``--- Historical Visual Transition ---`` summary block.
(Asserted directly; the legacy ``compress_flash_messages`` reference this (Asserted directly; the legacy ``compress_flash_messages`` reference this
was originally diffed against was removed in M5.) Intended M1 timing was originally diffed against was removed in M5.) The swap happens at
change: the swap happens at depth K, so the newest K-1 historical steps depth K, so the newest K-1 historical steps still carry their screenshots;
still carry their screenshots. their UI lists leave earlier, at the shallow text edge (depth 1), so only
the live observation ever carries an indexed element list.
""" """
summarizer = VisualStepSummarizer(mock_context) summarizer = VisualStepSummarizer(mock_context)
for i in range(1, 7): for i in range(1, 7):
@@ -120,8 +119,8 @@ def test_success_path_product_behind_the_scrub_edge(mock_context):
assert _has_image(new_messages[6]) assert _has_image(new_messages[6])
assert "Tree 6" in str(new_messages[6].content) assert "Tree 6" in str(new_messages[6].content)
# Intended timing change: step 5 (depth 2 < K) keeps its screenshot under # Step 5 (depth 2 < K) is past the text edge only: screenshot intact,
# the scrub edge; its XML was still stripped at depth 1. # UI list already gone, summary not yet swapped in.
assert _has_image(new_messages[5]) assert _has_image(new_messages[5])
assert "Visual summary 5." not in str(new_messages[5].content) assert "Visual summary 5." not in str(new_messages[5].content)
assert "Tree 5" not in str(new_messages[5].content) assert "Tree 5" not in str(new_messages[5].content)
@@ -249,10 +248,10 @@ def test_freeze_invariant_bytes_never_change(mock_context):
assert "Late summary 2." not in " ".join(str(m.content) for m in messages) assert "Late summary 2." not in " ".join(str(m.content) for m in messages)
def test_xml_depth1_strip_matches_legacy_semantics(mock_context): def test_strip_at_the_edge_keeps_combined_block_prefix_and_live_list(mock_context):
"""Depth-1 XML stripping keeps combined-block prefixes and the live list.""" """The XML strip at the edge keeps combined-block prefixes and the live list."""
messages: list = [] messages: list = []
compressor = ScrubEdgeCompressor(summarizer=None, prune_history_xml=True) compressor = ScrubEdgeCompressor(summarizer=None, prune_history_xml=True, image_scrub_depth=2)
combined = ToolMessage( combined = ToolMessage(
tool_call_id="tc1", tool_call_id="tc1",
@@ -267,13 +266,44 @@ def test_xml_depth1_strip_matches_legacy_semantics(mock_context):
) )
_simulate_turns(compressor, messages, [combined, _step(2)]) _simulate_turns(compressor, messages, [combined, _step(2)])
# Historical combined block: prefix retained, tree stripped (the exact # Combined block at the edge: prefix retained, tree stripped (the exact
# legacy ``compress_flash_messages`` product for this input). # legacy ``compress_flash_messages`` product for this input).
assert messages[0].content[0] == {"type": "text", "text": "Tapped Settings."} assert messages[0].content[0] == {"type": "text", "text": "Tapped Settings."}
# Live observation keeps its UI list untouched. # Live observation keeps its UI list untouched.
assert "Tree 2" in str(messages[1].content) assert "Tree 2" in str(messages[1].content)
def test_imageless_ui_list_carrier_is_stripped_as_soon_as_a_newer_observation_exists(
mock_context,
):
"""A step whose screenshot failed still carries a UI list; it follows the
text edge like any observation: while it is the newest observation it keeps
the list, and it is stripped as soon as a newer observation exists — even
when that newer one has no image either — so only the live observation ever
carries an indexed element list."""
messages: list = []
compressor = ScrubEdgeCompressor(summarizer=None, image_scrub_depth=2)
def imageless(n: int) -> ToolMessage:
return ToolMessage(
tool_call_id=f"tc{n}",
name="click",
content=[
{"type": "text", "text": f"Tapped {n}, screenshot failed."},
{"type": "text", "text": f"--- UI Element List ---\nTree {n}"},
],
)
_simulate_turns(compressor, messages, [imageless(1)])
assert "Tree 1" in str(messages[0].content) # newest observation: kept
_simulate_turns(compressor, messages, [imageless(2)]) # screenshots keep failing
assert messages[0].content == [{"type": "text", "text": "Tapped 1, screenshot failed."}]
assert "Tree 2" in str(messages[1].content)
_simulate_turns(compressor, messages, [_step(3)])
assert messages[1].content == [{"type": "text", "text": "Tapped 2, screenshot failed."}]
assert "Tree 3" in str(messages[2].content) # live observation untouched
def test_prune_history_xml_disabled_keeps_ui_lists(mock_context): def test_prune_history_xml_disabled_keeps_ui_lists(mock_context):
"""With the switch off, UI lists survive both depth-1 and freeze scrubs.""" """With the switch off, UI lists survive both depth-1 and freeze scrubs."""
summarizer = VisualStepSummarizer(mock_context) summarizer = VisualStepSummarizer(mock_context)
@@ -317,3 +347,348 @@ def test_untracked_messages_are_never_touched(mock_context):
assert messages[2].content == "thinking" assert messages[2].content == "thinking"
# Image-only tool message froze with the defensive fallback text. # Image-only tool message froze with the defensive fallback text.
assert messages[1].content == [{"type": "text", "text": "Action dispatched."}] assert messages[1].content == [{"type": "text", "text": "Action dispatched."}]
def test_depth_override_tightens_once_and_deeper_pass_never_backfills(mock_context):
"""A per-call depth override: a shallower pass scrubs the extra depths in
one go; a deeper pass afterwards touches nothing already frozen."""
summarizer = VisualStepSummarizer(mock_context)
for i in range(1, 9):
summarizer._step_inputs[f"tc{i}"] = {"step_number": i}
summarizer._summaries[f"tc{i}"] = f"Summary {i}."
messages: list = [_initial_observation()]
compressor = ScrubEdgeCompressor(summarizer=summarizer, image_scrub_depth=3)
for i in range(1, 9):
compressor.compress(messages, image_scrub_depth=6)
messages.append(_step(i))
compressor.compress(messages, image_scrub_depth=6)
with_images = [m for m in messages if _has_image(m)]
# Depth 6 with the live step at depth 1: tc4..tc7 (depths 5..2) keep
# their screenshots, tc3 (depth 6) is scrubbed.
assert [m.tool_call_id for m in with_images] == ["tc4", "tc5", "tc6", "tc7", "tc8"]
compressor.compress(messages, image_scrub_depth=3) # tighten: three more scrubbed at once
with_images = [m for m in messages if _has_image(m)]
assert [m.tool_call_id for m in with_images] == ["tc7", "tc8"]
frozen = {idx: _content_fingerprint(messages[idx]) for idx in compressor._frozen}
compressor.compress(messages, image_scrub_depth=6) # relax: nothing restored
assert [m.tool_call_id for m in messages if _has_image(m)] == ["tc7", "tc8"]
assert {idx: _content_fingerprint(messages[idx]) for idx in frozen} == frozen
# ---------------------------------------------------------------------------
# Screenshot replacement, ephemeral blocks, and frozen content
# ---------------------------------------------------------------------------
def _labelled_step(i: int, *, footer: str | None = None) -> ToolMessage:
"""A step in the Flash observation shape with the screenshot label block."""
blocks = [
{"type": "text", "text": f"Action {i} completed."},
{"type": "text", "text": "--- Current Screenshot ---"},
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,IMG_{i}"}},
{"type": "text", "text": f"--- UI Element List ---\nTree {i}"},
]
if footer is not None:
blocks.append({"type": "text", "text": footer})
return ToolMessage(tool_call_id=f"tc{i}", name="click", content=blocks)
def test_summary_takes_the_images_place_and_the_label_leaves_with_it(mock_context):
"""(i) The visual summary block sits exactly where the image block was —
not appended after whatever followed — and no ``--- Current Screenshot ---``
label is left dangling above it."""
summarizer = VisualStepSummarizer(mock_context)
summarizer._summaries["tc1"] = "Summary 1."
messages: list = []
compressor = ScrubEdgeCompressor(summarizer=summarizer, image_scrub_depth=2)
_simulate_turns(
compressor, messages, [_labelled_step(1, footer="Footer note 1"), _labelled_step(2)]
)
assert messages[0].content == [
{"type": "text", "text": "Action 1 completed."},
{"type": "text", "text": "--- Historical Visual Transition ---\nSummary 1."},
{"type": "text", "text": "Footer note 1"},
]
assert "--- Current Screenshot ---" not in str(messages[0].content)
# The live step keeps its label above its screenshot.
assert messages[1].content[1] == {"type": "text", "text": "--- Current Screenshot ---"}
def test_ephemeral_blocks_vanish_at_the_text_edge_and_not_before(mock_context):
"""(ii) Blocks listed under ``EPHEMERAL_BLOCKS_KEY`` survive only while
the message is the live observation; at the text edge (a newer observation
exists) they are deleted together with the UI list, the screenshot stays
until depth K; the consumed indices are cleared at the text edge."""
from artemis.memory.transcript import EPHEMERAL_BLOCKS_KEY, mark_ephemeral
summarizer = VisualStepSummarizer(mock_context)
summarizer._summaries["tc1"] = "Summary 1."
step1 = ToolMessage(
tool_call_id="tc1",
name="click",
content=[
{"type": "text", "text": "Action 1 completed."},
{"type": "text", "text": "[reminder: state your reasoning]"},
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,IMG_1"}},
{"type": "text", "text": "--- UI Element List ---\nTree 1"},
{"type": "text", "text": "[hint: valid this turn only]"},
],
)
mark_ephemeral(step1, [1, 4])
messages: list = []
compressor = ScrubEdgeCompressor(summarizer=summarizer, image_scrub_depth=3)
_simulate_turns(compressor, messages, [step1])
# Live observation: everything, ephemeral blocks included, still there.
assert "[reminder: state your reasoning]" in str(messages[0].content)
assert "[hint: valid this turn only]" in str(messages[0].content)
assert messages[0].additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [1, 4]
_simulate_turns(compressor, messages, [_step(2)]) # step 1 reaches depth 2
# Text edge: notices and UI list gone, the screenshot stays (depth 2 < K).
assert messages[0].content == [
{"type": "text", "text": "Action 1 completed."},
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,IMG_1"}},
]
assert EPHEMERAL_BLOCKS_KEY not in messages[0].additional_kwargs
_simulate_turns(compressor, messages, [_step(3)]) # step 1 reaches depth 3
assert messages[0].content == [
{"type": "text", "text": "Action 1 completed."},
{"type": "text", "text": "--- Historical Visual Transition ---\nSummary 1."},
]
assert EPHEMERAL_BLOCKS_KEY not in messages[0].additional_kwargs
def test_grace_does_the_text_edits_at_k_once_and_only_the_image_swap_waits(mock_context):
"""A pending summary keeps the screenshot (and its label) inside the grace
window, but the strip and the ephemeral deletion happen at K exactly once;
the later swap puts the summary where the image was and freezes."""
from artemis.memory.transcript import mark_ephemeral
summarizer = VisualStepSummarizer(mock_context)
summarizer._step_inputs["tc1"] = {"step_number": 1} # pending
step1 = _labelled_step(1, footer="[per-turn notice]")
mark_ephemeral(step1, [4])
messages: list = []
compressor = ScrubEdgeCompressor(
summarizer=summarizer, image_scrub_depth=2, pending_grace_steps=3
)
_simulate_turns(compressor, messages, [step1, _labelled_step(2)])
assert messages[0].content == [
{"type": "text", "text": "Action 1 completed."},
{"type": "text", "text": "--- Current Screenshot ---"},
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,IMG_1"}},
]
after_text_edits = _content_fingerprint(messages[0])
_simulate_turns(compressor, messages, [_labelled_step(3)]) # still pending, in grace
assert _content_fingerprint(messages[0]) == after_text_edits
summarizer._summaries["tc1"] = "Late but in time."
_simulate_turns(compressor, messages, [_labelled_step(4)])
assert messages[0].content == [
{"type": "text", "text": "Action 1 completed."},
{"type": "text", "text": "--- Historical Visual Transition ---\nLate but in time."},
]
assert 0 in compressor._frozen
def test_messages_past_the_edge_are_byte_identical_on_every_later_pass(mock_context):
"""(iii) Once a message is past the edge, compressing again — with new
turns appended, depths relaxed or tightened, summaries arriving late —
leaves it byte-identical. Each pass rewrites exactly the two messages
that reached an edge (the text edge at depth 2, the screenshot edge at
K) and nothing older."""
summarizer = VisualStepSummarizer(mock_context)
for i in range(1, 13):
summarizer._step_inputs[f"tc{i}"] = {"step_number": i}
summarizer._summaries[f"tc{i}"] = f"Summary {i}."
messages: list = []
compressor = ScrubEdgeCompressor(summarizer=summarizer, image_scrub_depth=3)
previous: list[str] = []
for i in range(1, 13):
messages.append(_labelled_step(i))
compressor.compress(messages)
current = [_content_fingerprint(m) for m in messages]
# Everything before the messages that reached an edge is unchanged.
changed = [idx for idx, fp in enumerate(previous) if current[idx] != fp]
# The message reaching depth 2 (text edge) is index len-1 of the previous
# pass's list, the one reaching depth 3 (screenshot edge) is len-2; the
# new live message is not in that list yet.
expected = [idx for idx in (len(previous) - 2, len(previous) - 1) if idx >= 0]
assert changed == expected, changed
previous = current
# No new messages: a repeated pass, relaxed or not, is a no-op byte for byte.
compressor.compress(messages)
compressor.compress(messages, image_scrub_depth=6)
assert [_content_fingerprint(m) for m in messages] == previous
# Tightening rewrites exactly the message at the new depth, nothing older.
compressor.compress(messages, image_scrub_depth=2)
changed = [i for i, fp in enumerate(previous) if _content_fingerprint(messages[i]) != fp]
assert changed == [len(messages) - 2]
def test_text_edge_deeper_than_screenshot_edge_keeps_the_edges_independent(mock_context):
"""xml_scrub_depth=4 with K=3: the screenshot is resolved at depth 3 while
the UI list stays until the message is deeper than 4; the message is only
frozen once both edits have happened."""
summarizer = VisualStepSummarizer(mock_context)
for i in range(1, 8):
summarizer._summaries[f"tc{i}"] = f"Summary {i}."
messages: list = []
compressor = ScrubEdgeCompressor(summarizer=summarizer, image_scrub_depth=3, xml_scrub_depth=4)
_simulate_turns(compressor, messages, [_labelled_step(i) for i in range(1, 4)])
# Step 1 at depth 3: image swapped, label gone, UI list still there.
assert "Summary 1." in str(messages[0].content)
assert not _has_image(messages[0])
assert "--- Current Screenshot ---" not in str(messages[0].content)
assert "Tree 1" in str(messages[0].content)
_simulate_turns(compressor, messages, [_labelled_step(4), _labelled_step(5)])
# Step 1 at depth 5 (> 4): UI list gone now; the summary is untouched.
assert "Tree 1" not in str(messages[0].content)
assert messages[0].content == [
{"type": "text", "text": "Action 1 completed."},
{"type": "text", "text": "--- Historical Visual Transition ---\nSummary 1."},
]
frozen = _content_fingerprint(messages[0])
_simulate_turns(compressor, messages, [_labelled_step(6), _labelled_step(7)])
assert _content_fingerprint(messages[0]) == frozen
# ---------------------------------------------------------------------------
# Ephemeral indices stay aligned when the image edge is shallower than the
# text edge (a rewrite before the text edit shifts the later blocks)
# ---------------------------------------------------------------------------
def _step_with_notices(i: int) -> ToolMessage:
"""Flash observation shape with one ephemeral notice on each side of the
screenshot and a body block *after* the notice that follows the image, so
a stale index would delete the body instead of the notice."""
from artemis.memory.transcript import mark_ephemeral
msg = ToolMessage(
tool_call_id=f"tc{i}",
name="click",
content=[
{"type": "text", "text": f"Action {i} completed."},
{"type": "text", "text": f"[hint {i}: before the image]"},
{"type": "text", "text": "--- Current Screenshot ---"},
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,IMG_{i}"}},
{"type": "text", "text": f"--- UI Element List ---\nTree {i}"},
{"type": "text", "text": f"[reminder {i}: after the image]"},
{"type": "text", "text": f"Body note {i}"},
],
)
mark_ephemeral(msg, [1, 5])
return msg
def test_image_edge_before_text_edge_remaps_ephemeral_indices_to_the_summary_layout(
mock_context,
):
"""K=3 with xml_scrub_depth=4: the summary swap at depth 3 removes the
label block, so the notice after the image moves up by one. The stored
indices must follow it; the later text pass then deletes both notices and
leaves the body block intact."""
from artemis.memory.transcript import EPHEMERAL_BLOCKS_KEY
summarizer = VisualStepSummarizer(mock_context)
for i in range(1, 6):
summarizer._summaries[f"tc{i}"] = f"Summary {i}."
messages: list = []
compressor = ScrubEdgeCompressor(summarizer=summarizer, image_scrub_depth=3, xml_scrub_depth=4)
_simulate_turns(compressor, messages, [_step_with_notices(i) for i in range(1, 4)])
# Image pass (depth 3): label + image -> summary; notices still present.
assert messages[0].content == [
{"type": "text", "text": "Action 1 completed."},
{"type": "text", "text": "[hint 1: before the image]"},
{"type": "text", "text": "--- Historical Visual Transition ---\nSummary 1."},
{"type": "text", "text": "--- UI Element List ---\nTree 1"},
{"type": "text", "text": "[reminder 1: after the image]"},
{"type": "text", "text": "Body note 1"},
]
assert messages[0].additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [1, 4]
_simulate_turns(compressor, messages, [_step_with_notices(4), _step_with_notices(5)])
# Text pass (depth 5 > 4): notices and UI list gone, body block intact.
assert messages[0].content == [
{"type": "text", "text": "Action 1 completed."},
{"type": "text", "text": "--- Historical Visual Transition ---\nSummary 1."},
{"type": "text", "text": "Body note 1"},
]
assert EPHEMERAL_BLOCKS_KEY not in messages[0].additional_kwargs
assert 0 in compressor._frozen
def test_image_edge_before_text_edge_remaps_when_the_image_is_dropped(mock_context):
"""Same ordering with no summary job at all: label and image both leave
(two blocks removed), the notice after the image moves up by two."""
from artemis.memory.transcript import EPHEMERAL_BLOCKS_KEY
messages: list = []
compressor = ScrubEdgeCompressor(summarizer=None, image_scrub_depth=3, xml_scrub_depth=4)
_simulate_turns(compressor, messages, [_step_with_notices(i) for i in range(1, 4)])
assert messages[0].content == [
{"type": "text", "text": "Action 1 completed."},
{"type": "text", "text": "[hint 1: before the image]"},
{"type": "text", "text": "--- UI Element List ---\nTree 1"},
{"type": "text", "text": "[reminder 1: after the image]"},
{"type": "text", "text": "Body note 1"},
]
assert messages[0].additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [1, 3]
_simulate_turns(compressor, messages, [_step_with_notices(4), _step_with_notices(5)])
assert messages[0].content == [
{"type": "text", "text": "Action 1 completed."},
{"type": "text", "text": "Body note 1"},
]
assert EPHEMERAL_BLOCKS_KEY not in messages[0].additional_kwargs
assert 0 in compressor._frozen
def test_grace_swap_before_the_text_edge_remaps_ephemeral_indices(mock_context):
"""Pending summary inside the grace window with the text edge still ahead:
the keep-image revisit changes nothing (indices untouched); the late swap
removes the label and remaps; the text pass then deletes the right blocks."""
from artemis.memory.transcript import EPHEMERAL_BLOCKS_KEY
summarizer = VisualStepSummarizer(mock_context)
summarizer._step_inputs["tc1"] = {"step_number": 1} # pending
messages: list = []
compressor = ScrubEdgeCompressor(
summarizer=summarizer, image_scrub_depth=3, xml_scrub_depth=5, pending_grace_steps=3
)
_simulate_turns(compressor, messages, [_step_with_notices(i) for i in range(1, 4)])
# Depth 3, pending, in grace: byte-identical, indices untouched.
assert _has_image(messages[0])
assert "--- Current Screenshot ---" in str(messages[0].content)
assert messages[0].additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [1, 5]
summarizer._summaries["tc1"] = "Late but in time."
_simulate_turns(compressor, messages, [_step_with_notices(4)]) # depth 4: swap
assert messages[0].content[2] == {
"type": "text",
"text": "--- Historical Visual Transition ---\nLate but in time.",
}
assert messages[0].additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [1, 4]
_simulate_turns(compressor, messages, [_step_with_notices(5), _step_with_notices(6)])
# Depth 6 > 5: text pass on the remapped layout.
assert messages[0].content == [
{"type": "text", "text": "Action 1 completed."},
{"type": "text", "text": "--- Historical Visual Transition ---\nLate but in time."},
{"type": "text", "text": "Body note 1"},
]
assert EPHEMERAL_BLOCKS_KEY not in messages[0].additional_kwargs
assert 0 in compressor._frozen
+101 -4
View File
@@ -569,8 +569,9 @@ async def test_raw_lens_call_meters_llm_usage_without_touching_context_base(mock
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_focus_block_and_provenance_marked_action_phrase_reach_the_lens(mock_context): async def test_focus_block_and_provenance_marked_action_phrase_reach_the_lens(mock_context):
"""The lens input opens with the action as every history reader sees it """The lens input opens with the action as every history reader sees it
(self-described target marked), the verbatim arguments, the outcome, and (self-described target marked), the outcome, and the operator focus
the operator focus block; the system prompt teaches how to use it.""" block; the phrase already carries every argument, so no verbatim
argument line repeats them. The system prompt teaches how to use it."""
summarizer = VisualStepSummarizer(mock_context) summarizer = VisualStepSummarizer(mock_context)
captured: list = [] captured: list = []
@@ -609,9 +610,9 @@ async def test_focus_block_and_provenance_marked_action_phrase_reach_the_lens(mo
assert "Absence is a fact" in system_text assert "Absence is a fact" in system_text
assert lead_text.startswith( assert lead_text.startswith(
"Step 3 Physical Action: Tapped 'Wi-Fi toggle' (self-described) at [880, 410]\n" "Step 3 Physical Action: Tapped 'Wi-Fi toggle' (self-described) at [880, 410]\n"
"Controller Outcome: Tapped at [880, 410] (normalized)."
) )
assert "Action arguments (verbatim): click({'target': [880, 410]" in lead_text assert "Action arguments (verbatim)" not in lead_text
assert "Controller Outcome: Tapped at [880, 410] (normalized)." in lead_text
assert FOCUS_BLOCK_HEADER in lead_text assert FOCUS_BLOCK_HEADER in lead_text
assert "Task goal: Turn on Wi-Fi" in lead_text assert "Task goal: Turn on Wi-Fi" in lead_text
assert "Active sub-goal: Enable Wi-Fi > Tap the Wi-Fi switch" in lead_text assert "Active sub-goal: Enable Wi-Fi > Tap the Wi-Fi switch" in lead_text
@@ -688,6 +689,102 @@ async def test_single_frame_focus_label_forbids_describing_the_outcome(mock_cont
assert summarizer.get_job_payload(2)["focus"] is None assert summarizer.get_job_payload(2)["focus"] is None
def test_action_phrase_action_name_wins_over_an_action_argument():
"""``manage_app``'s own ``action="launch"`` argument is the intent the
renderer reads, never the verb: the phrase is ``Launched app 'X'`` for
the Flash record shape, the Pro record shape and a raw argument dict."""
flash_record = {
"coordinates": None,
"coordinate_space": "normalized",
"args": {"action": "launch", "app_name": "Settings"},
}
assert VisualStepSummarizer._action_phrase("manage_app", flash_record) == (
"Launched app 'Settings'"
)
pro_record = {"intent": "stop", "app_name": "com.android.settings"}
assert VisualStepSummarizer._action_phrase("manage_app", pro_record) == (
"Stopped app 'com.android.settings'"
)
raw = {"action": "launch", "app_name": "Settings"}
assert VisualStepSummarizer._action_phrase("manage_app", raw) == "Launched app 'Settings'"
assert "Action: launch" not in VisualStepSummarizer._action_phrase("manage_app", raw)
# The Flash record shape renders every other action through the same path.
key = {"coordinates": None, "args": {"key": "BACK"}}
assert VisualStepSummarizer._action_phrase("press_key", key) == "Pressed key 'BACK'"
swipe = {
"coordinates": [600, 700, 600, 300],
"normalized_start_coordinates": [600, 700],
"normalized_end_coordinates": [600, 300],
"args": {"direction": "up"},
}
assert VisualStepSummarizer._action_phrase("swipe", swipe) == (
"Swiped up (from [600, 700] to [600, 300])"
)
index_click = {
"coordinates": [500, 520],
"normalized_coordinates": [500, 520],
"target_text": "Wi-Fi",
"args": {"target": 3},
}
assert VisualStepSummarizer._action_phrase("click", index_click) == (
"Tapped 'Wi-Fi' at [500, 520]"
)
@pytest.mark.asyncio
async def test_verbatim_argument_line_appears_only_when_the_phrase_folds_arguments(
mock_context,
):
"""``Launched app 'Settings'`` carries its arguments: no verbatim line. An
index target (``target=3``) is folded into the resolved point, so the
verbatim line keeps the index the model actually named."""
summarizer = VisualStepSummarizer(mock_context)
captured: list = []
async def mock_ainvoke(messages):
captured.append(messages)
return AIMessage(content="In this step, I acted and the screen showed the result.")
summarizer._llm = Mock()
summarizer._llm.ainvoke = AsyncMock(side_effect=mock_ainvoke)
summarizer.dispatch(
step_number=1,
action_name="manage_app",
action_args={"coordinates": None, "args": {"action": "launch", "app_name": "Settings"}},
pre_img_bytes=b"pre",
post_img_bytes=b"post",
exec_outcome="Launched app 'Settings' (com.android.settings); foreground confirmed.",
)
summarizer.dispatch(
step_number=2,
action_name="click",
action_args={
"coordinates": [500, 520],
"normalized_coordinates": [500, 520],
"target_text": "Wi-Fi",
"args": {"target": 3},
},
pre_img_bytes=b"pre",
post_img_bytes=b"post",
exec_outcome="Tapped at [500, 520] (normalized).",
)
await summarizer.flush()
lead_by_step = {}
for messages in captured:
text = messages[1].content[0]["text"]
lead_by_step[text.split(" ")[1]] = text
launch = lead_by_step["1"]
assert launch.startswith("Step 1 Physical Action: Launched app 'Settings'\nController Outcome:")
assert "Action arguments (verbatim)" not in launch
assert "Action: launch" not in launch
click = lead_by_step["2"]
assert click.startswith("Step 2 Physical Action: Tapped 'Wi-Fi' at [500, 520]\n")
assert "Action arguments (verbatim): click({'target': 3})" in click
def test_action_phrase_renders_flash_swipe_points_and_bursts(): def test_action_phrase_renders_flash_swipe_points_and_bursts():
"""Flash's ``start``/``end`` swipe points and a Pro burst's additional """Flash's ``start``/``end`` swipe points and a Pro burst's additional
actions render through the shared history phrase, never as None.""" actions render through the shared history phrase, never as None."""
+4 -7
View File
@@ -263,9 +263,9 @@ async def test_history_analyzer_read_note_tool_call():
] # Index 5 contains the ToolMessage for read_note ] # Index 5 contains the ToolMessage for read_note
assert isinstance(read_msg, ToolMessage) assert isinstance(read_msg, ToolMessage)
assert read_msg.tool_call_id == "call_read" assert read_msg.tool_call_id == "call_read"
assert ( assert read_msg.content == (
read_msg.content == "Successfully read note 'tactical_plan'. 'tactical_plan' note" f"Note 'tactical_plan' ({len(fake_plan_content.splitlines())} lines):"
f" content:\n{fake_plan_content}" f"\n{fake_plan_content}"
) )
@@ -306,10 +306,7 @@ def test_history_analyzer_robust_tools_behavior():
patch("pathlib.Path.read_text", return_value="Tactical plan content"), patch("pathlib.Path.read_text", return_value="Tactical plan content"),
): ):
result_read = read_tool.invoke({"key": "tactical_plan.md"}) result_read = read_tool.invoke({"key": "tactical_plan.md"})
assert ( assert result_read == "Note 'tactical_plan.md' (1 lines):\nTactical plan content"
result_read == "Successfully read note 'tactical_plan.md'. 'tactical_plan.md'"
" note content:\nTactical plan content"
)
@pytest.mark.asyncio @pytest.mark.asyncio
+118 -2
View File
@@ -399,6 +399,14 @@ async def test_operator_node_multiple_actions():
assert decisions[1]["action"] == "tap" assert decisions[1]["action"] == "tap"
assert decisions[1]["coordinates"] == [54, 120] assert decisions[1]["coordinates"] == [54, 120]
# Each action's ToolMessage names its tool, like helper-tool results do
# (the loop appends them to the same list it sent to the model).
from langchain_core.messages import ToolMessage
sent = mock_llm.ainvoke.call_args[0][0]
recorded = [(m.name, m.content) for m in sent if isinstance(m, ToolMessage)]
assert recorded == [("input_text", "Action Recorded"), ("click", "Action Recorded")]
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_operator_node_no_record_step(): async def test_operator_node_no_record_step():
@@ -1467,6 +1475,40 @@ async def test_operator_swipe_translation():
assert actions[0]["coordinates"] == [648, 720, 648, 1680] assert actions[0]["coordinates"] == [648, 720, 648, 1680]
@pytest.mark.asyncio
async def test_operator_press_key_translation_is_case_insensitive():
from artemis.agents.operator.operator import OperatorNode
from artemis.context import ArtemisContext
from unittest.mock import MagicMock
mock_ctx = MagicMock(spec=ArtemisContext)
mock_state = MagicMock()
mock_state.operator_raw_data = {"width": 1080, "height": 2400}
node = OperatorNode(mock_ctx, transcript_config=LEGACY_TRANSCRIPT)
for spelling in ("back", "Back", "BACK", "KEYCODE_BACK", "keycode_back", " back "):
actions, err = node._translate_and_validate_tool(
{"name": "press_key", "args": {"key": spelling}}, mock_state
)
assert err is None, spelling
assert actions == [{"action": "press_key", "keycode": "KEYCODE_BACK"}], spelling
actions, err = node._translate_and_validate_tool(
{"name": "press_key", "args": {"key": "app_switch"}}, mock_state
)
assert err is None
assert actions == [{"action": "press_key", "keycode": "KEYCODE_APP_SWITCH"}]
# Unsupported keys get a self-explanatory error that lists the accepted names.
actions, err = node._translate_and_validate_tool(
{"name": "press_key", "args": {"key": "volume_up"}}, mock_state
)
assert actions == []
assert err == (
"Error: Unsupported key 'volume_up'. Supported keys: ENTER, BACK, HOME, APP_SWITCH."
)
# --- helper tool results: status is structural, never sniffed from the words ---------- # --- helper tool results: status is structural, never sniffed from the words ----------
@@ -1537,5 +1579,79 @@ async def test_operator_helper_tool_message_status_is_structural(result, expecte
tool_msgs = {m.tool_call_id: m for m in second_turn_messages if isinstance(m, ToolMessage)} tool_msgs = {m.tool_call_id: m for m in second_turn_messages if isinstance(m, ToolMessage)}
assert tool_msgs["call_read"].status == expected_status assert tool_msgs["call_read"].status == expected_status
assert tool_msgs["call_read"].content == str(result) assert tool_msgs["call_read"].content == str(result)
# The accompanying click is rejected only when the helper actually failed. # The accompanying click never ran: rejected when the helper failed, deferred
assert tool_msgs["call_click"].status == ("error" if expected_status == "error" else "success") # otherwise. Both are reported as errors and never claim success.
assert tool_msgs["call_click"].status == "error"
assert tool_msgs["call_click"].content.startswith("Not executed:")
assert "success" not in tool_msgs["call_click"].content.lower()
if expected_status == "error":
assert str(result) in tool_msgs["call_click"].content
else:
assert "deferred" in tool_msgs["call_click"].content
@pytest.mark.asyncio
async def test_operator_burst_limit_error_text_is_not_repeated_per_call():
"""An over-long burst answers every tool_call id, but the full burst-limit
paragraph travels once; the remaining ToolMessages just point at it."""
from langchain_core.messages import ToolMessage
mock_ctx = MagicMock(spec=ArtemisContext)
mock_ctx.execution_setup = None
mock_ctx.data_engine = None
mock_state = MagicMock()
mock_state.subagent_calls = []
mock_state.initial_goal = "Test goal"
mock_state.open_incident = None
turn_1 = MagicMock()
turn_1.tool_calls = [
{
"name": "click",
"args": {"target": [50, 50], "target_description": f"button {i}"},
"id": f"call_click_{i}",
}
for i in range(6)
]
turn_2 = MagicMock()
turn_2.tool_calls = [
{
"name": "click",
"args": {"target": [50, 50], "target_description": "button"},
"id": "call_click_final",
}
]
mock_llm = MagicMock()
mock_llm.ainvoke = AsyncMock(side_effect=[turn_1, turn_2])
mock_llm.bind_tools.return_value = mock_llm
with (
patch("artemis.agents.operator.operator.get_llm", return_value=mock_llm),
patch.object(OperatorNode, "_max_burst_actions", return_value=4),
):
node = OperatorNode(mock_ctx, transcript_config=LEGACY_TRANSCRIPT)
node_update = await node(mock_state)
assert mock_llm.ainvoke.await_count == 2
second_turn_messages = mock_llm.ainvoke.call_args_list[1].args[0]
burst_ids = {f"call_click_{i}" for i in range(6)}
tool_msgs = {
m.tool_call_id: m
for m in second_turn_messages
if isinstance(m, ToolMessage) and m.tool_call_id in burst_ids
}
assert set(tool_msgs) == burst_ids
assert all(m.status == "error" for m in tool_msgs.values())
first = tool_msgs["call_click_0"].content
assert first.startswith("Error: 6 Turn-Ending Actions in one turn exceed")
assert "burst limit of 4" in first
for i in range(1, 6):
assert tool_msgs[f"call_click_{i}"].content == (
"Not executed: see the burst-limit error above."
)
# Nothing from the rejected burst ran; turn 2's single click did.
decisions = json.loads(node_update["structured_decisions"])
assert len(decisions) == 1 and decisions[0]["action"] == "tap"
@@ -0,0 +1,122 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""The Pro operator's context base on a provider without usage metadata.
The transcript ledger's start gate and soft/hard thresholds read the
operator's own measured prompt size; when the provider reports no usage the
operator records a provider-less estimate instead (Flash already does), so
the thresholds still see a base. A measured call also hands the ledger the
messages it was measured on, for the chars-per-token calibration.
"""
import base64
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from artemis.agents.operator.operator import OperatorNode
from artemis.config.agent import MemoryTranscriptConfig
from artemis.context import ArtemisContext
from artemis.memory.step_memory import StepMemoryService
from artemis.memory.transcript import TranscriptLedger, estimate_prompt_tokens
SCREENSHOT_B64 = base64.b64encode(b"fake-jpeg-bytes").decode("utf-8")
def _ctx():
ctx = MagicMock(spec=ArtemisContext)
ctx.execution_setup = None
ctx.actuator = None
ctx.data_engine = None
ctx.step_memory = StepMemoryService(ctx=None)
ctx.transcript_ledger = None
return ctx
def _state():
state = MagicMock()
state.subagent_calls = []
state.initial_goal = "Estimate goal"
state.operator_feedback = None
state.injected_instruction = None
state.operator_tool_limit_exceeded = False
state.structured_decisions = None
state.last_execution_result = None
state.current_step_id = None
state.latest_ui_hierarchy = []
state.operator_raw_data = {
"screenshot_b64": SCREENSHOT_B64,
"xml_hierarchy": [],
"ocr_results": None,
"width": 1080,
"height": 2400,
}
return state
def _llm(captured: list, usage_metadata=None):
mock_llm = MagicMock()
response = MagicMock()
response.tool_calls = []
response.content = "no action this turn"
# usage_from_message only trusts a non-empty dict; a MagicMock attribute
# (the default) reads as "no usage reported".
if usage_metadata is not None:
response.usage_metadata = usage_metadata
async def ainvoke(*args, **kwargs):
captured.append(list(args[0]))
return response
mock_llm.ainvoke = AsyncMock(side_effect=ainvoke)
mock_llm.bind_tools.return_value = mock_llm
return mock_llm
@pytest.mark.asyncio
async def test_operator_records_an_estimate_when_the_provider_reports_no_usage():
ctx = _ctx()
captured: list = []
with patch("artemis.agents.operator.operator.get_llm", return_value=_llm(captured)):
node = OperatorNode(ctx, transcript_config=MemoryTranscriptConfig(enabled=True))
await node(_state())
ledger = ctx.transcript_ledger
assert isinstance(ledger, TranscriptLedger)
assert captured, "the operator made no model call"
assert ledger.last_prompt_tokens == estimate_prompt_tokens(captured[-1])
assert ledger.last_prompt_tokens > 0
# No measurement, no calibration.
assert ledger.chars_per_token == 4.0
@pytest.mark.asyncio
async def test_operator_measured_usage_calibrates_the_ledger_ratio():
ctx = _ctx()
captured: list = []
llm = _llm(
captured,
usage_metadata={"input_tokens": 50_000, "output_tokens": 10, "total_tokens": 50_010},
)
with patch("artemis.agents.operator.operator.get_llm", return_value=llm):
node = OperatorNode(ctx, transcript_config=MemoryTranscriptConfig(enabled=True))
await node(_state())
ledger = ctx.transcript_ledger
assert ledger.last_prompt_tokens == 50_000
# The prompt is a few thousand chars of English plus one image; measured
# at 50k tokens the ratio clamps to the lower bound — proof the messages
# reached the calibration, not just the token figure.
assert ledger.chars_per_token == 1.0
@@ -0,0 +1,147 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Tests for the Operator reasoning reminder.
Thinking-mode models satisfy "reason before acting" inside their thought channel
and emit bare tool calls. After such a silent turn the next observation tail
carries a one-line reminder; nothing is bounced and no extra model call is made.
"""
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from artemis.agents.operator.operator import OperatorNode
from artemis.agents.operator.prompts import (
REASONING_REMINDER,
apply_operator_prompt_contract,
load_operator_prompts,
)
from artemis.config.agent import MemoryTranscriptConfig
from artemis.context import ArtemisContext
LEGACY_TRANSCRIPT = MemoryTranscriptConfig(enabled=False)
PLAN_WITH_LEAF = """- [x] Open Google Maps and search for SFO
- [/] Read the commute duration and record it into note `commute_eta_info`
- [/] Read the driving duration
- [/] Tap the "Driving" tab so the fastest route is highlighted
- [ ] Draft the ETA message
"""
def _make_node(tmp_path):
notes = tmp_path / "notes"
notes.mkdir(parents=True)
(notes / "task_plan.md").write_text(PLAN_WITH_LEAF, encoding="utf-8")
ctx = MagicMock(spec=ArtemisContext)
ctx.execution_setup = None
ctx.data_engine = MagicMock()
ctx.data_engine.base_dir = str(tmp_path)
ctx.data_engine.current_session_id = "s"
ctx.data_engine.get_agent_friendly_steps.return_value = []
state = MagicMock()
state.subagent_calls = []
state.initial_goal = "goal"
state.open_incident = None
return OperatorNode(ctx, transcript_config=LEGACY_TRANSCRIPT), state
def _click_response(content):
response = MagicMock()
response.content = content
response.tool_calls = [
{
"name": "click",
"args": {"target": [50, 50], "target_description": "button"},
"id": "call_0",
}
]
return response
def _texts(messages):
out = []
for m in messages:
content = getattr(m, "content", None)
if isinstance(content, str):
out.append(content)
elif isinstance(content, list):
out.extend(b.get("text", "") for b in content if isinstance(b, dict))
return "\n".join(out)
@pytest.mark.asyncio
async def test_reminder_follows_a_silent_turn_and_costs_no_extra_call(tmp_path):
node, state = _make_node(tmp_path)
seen = []
replies = iter(
[
_click_response(""), # turn 1: silent
_click_response("Tapping the Driving tab."), # turn 2: reminded, speaks
_click_response([{"type": "thinking", "thinking": "internal"}]), # turn 3: silent
_click_response("Done."), # turn 4
]
)
async def ainvoke(messages, *args, **kwargs):
seen.append(list(messages))
return next(replies)
llm = MagicMock()
llm.ainvoke = AsyncMock(side_effect=ainvoke)
llm.bind_tools.return_value = llm
with patch("artemis.agents.operator.operator.get_llm", return_value=llm):
for _ in range(4):
await node(state)
assert llm.ainvoke.await_count == 4, "one model call per turn, never a bounce"
assert REASONING_REMINDER not in _texts(seen[0]) # nothing to remind on turn 1
assert REASONING_REMINDER in _texts(seen[1]) # turn 1 was silent
assert REASONING_REMINDER not in _texts(seen[2]) # turn 2 spoke
assert REASONING_REMINDER in _texts(seen[3]) # thinking-only turn 3 counts as silent
assert _texts(seen[3]).count(REASONING_REMINDER) == 1
def test_ledger_silence_verdict_outranks_the_local_flag(tmp_path):
"""With a transcript ledger, the ledger's own judgment of the previous
turn is authoritative; the local flag only serves the legacy path."""
from artemis.memory import TranscriptLedger
node, _ = _make_node(tmp_path)
node._previous_turn_silent = True
assert node._previous_turn_was_silent() is True # no ledger: local flag
ledger = MagicMock(spec=TranscriptLedger)
ledger.last_turn_silent = False
node.ctx.transcript_ledger = ledger
assert node._previous_turn_was_silent() is False
ledger.last_turn_silent = True
node._previous_turn_silent = False
assert node._previous_turn_was_silent() is True
def test_prompt_asks_for_one_paragraph_not_phase_sections():
prompt = apply_operator_prompt_contract(load_operator_prompts()["main_template"])
assert prompt.count("one short paragraph of connected prose reflecting these three phases") == 1
assert "in your raw thinking" not in prompt
assert "in your written reply" not in prompt
assert (
"bounced"
not in prompt.split("# THE THREE-PHASE COGNITIVE PROTOCOL")[1].split("## Phase 1")[0]
)
+54 -3
View File
@@ -45,6 +45,7 @@ from artemis.config.agent import MemoryTranscriptConfig
from artemis.context import ArtemisContext from artemis.context import ArtemisContext
from artemis.memory.step_memory import StepMemoryService from artemis.memory.step_memory import StepMemoryService
from artemis.memory.transcript import ( from artemis.memory.transcript import (
EPHEMERAL_BLOCKS_KEY,
EXECUTION_RESULT_MARKER, EXECUTION_RESULT_MARKER,
PLAN_RECITATION_MARKER, PLAN_RECITATION_MARKER,
RESTORED_HISTORY_HEADER, RESTORED_HISTORY_HEADER,
@@ -53,8 +54,8 @@ from artemis.memory.transcript import (
# SHA-256 snapshots of the legacy system message with the fixed inputs below. # SHA-256 snapshots of the legacy system message with the fixed inputs below.
# Update these when an intentional template change alters the rendered prompt. # Update these when an intentional template change alters the rendered prompt.
GOLDEN_EMPTY_PLAN = "d8cf83565c766d32893b124c974060bbd98de2d862c3b9925e546ea0c946c324" GOLDEN_EMPTY_PLAN = "513e0341014b9a2342f608ad4dd173b7dcf5704ce53ee028ba302a9a1b696f96"
GOLDEN_SENTINEL_PLAN = "8f871b5e5700ba5ee07ed03b80a8f82664abe68b0c170d3185bf1ee35f7821ed" GOLDEN_SENTINEL_PLAN = "012fe1b08645266d5a7e518023e20fc6fbfa65d4f2d985847ebb06e0184c67ee"
SCREENSHOT_B64 = base64.b64encode(b"fake-jpeg-bytes").decode("utf-8") SCREENSHOT_B64 = base64.b64encode(b"fake-jpeg-bytes").decode("utf-8")
@@ -103,6 +104,8 @@ def test_transcript_static_system_is_stable_and_carries_no_history():
# ...and no unrendered slot or trailing observation header survives. # ...and no unrendered slot or trailing observation header survives.
assert "{{" not in first assert "{{" not in first
assert not first.endswith("# CURRENT OBSERVATION\n") assert not first.endswith("# CURRENT OBSERVATION\n")
# The history guide names the marker a resolved screenshot renders as.
assert "--- Historical Visual Transition ---" in first
def _transcript_ctx(): def _transcript_ctx():
@@ -214,7 +217,9 @@ async def test_transcript_mode_two_turns_build_four_regions(tmp_path):
# Committed turn-1 observation followed by its validator result. # Committed turn-1 observation followed by its validator result.
assert any(EXECUTION_RESULT_MARKER in str(m.content) for m in active) assert any(EXECUTION_RESULT_MARKER in str(m.content) for m in active)
committed_obs = active[0] committed_obs = active[0]
# Depth-1 scrub removed the old plan recitation copy. # Text edge at depth 1: the committed observation has lost its plan
# recitation as soon as the next tail exists (its screenshot stays
# until the screenshot edge).
assert PLAN_RECITATION_MARKER not in str(committed_obs.content) assert PLAN_RECITATION_MARKER not in str(committed_obs.content)
# The live tail still recites the plan. # The live tail still recites the plan.
assert PLAN_RECITATION_MARKER in str(turn2_messages[-1].content) assert PLAN_RECITATION_MARKER in str(turn2_messages[-1].content)
@@ -223,6 +228,52 @@ async def test_transcript_mode_two_turns_build_four_regions(tmp_path):
assert re.search(r"T\+\d{2,}:\d{2}", str(turn2_messages[-1].content)) assert re.search(r"T\+\d{2,}:\d{2}", str(turn2_messages[-1].content))
@pytest.mark.asyncio
async def test_transcript_tail_marks_per_turn_notices_ephemeral():
"""The hand-built tail carries the builder's ephemeral indices: per-turn
notices are deleted by the scrub edge; header, plan recitation and the
observation itself are not."""
from artemis.agents.operator.prompts import USER_GUIDANCE_MARKER
ctx = _transcript_ctx()
captured: list = []
mock_llm = _no_action_llm(captured)
with patch("artemis.agents.operator.operator.get_llm", return_value=mock_llm):
node = OperatorNode(ctx, transcript_config=MemoryTranscriptConfig(enabled=True))
await node(
_transcript_state(
injected_instruction="Skip the login step",
user_stop_requested=False,
operator_tool_limit_exceeded=True,
)
)
tail = captured[0][-1]
assert isinstance(tail, HumanMessage)
texts = {
i: block.get("text", "")
for i, block in enumerate(tail.content)
if isinstance(block, dict) and block.get("type") == "text"
}
guidance = next(i for i, t in texts.items() if t.startswith(USER_GUIDANCE_MARKER))
warning = next(i for i, t in texts.items() if "tool-call budget" in t)
assert set(tail.additional_kwargs[EPHEMERAL_BLOCKS_KEY]) == {guidance, warning}
# The user's stop bit is relayed on the guidance block itself.
assert "authorized stopping" not in texts[guidance]
# The verbatim instruction is its own persistent block right after the
# wrapper (stamped with the ledger's session offset), never ephemeral:
# a standing instruction must outlive the turn it arrived on.
body = guidance + 1
assert re.fullmatch(r'User instruction \(T\+\d{2,}:\d{2}\): "Skip the login step"', texts[body])
assert body not in tail.additional_kwargs[EPHEMERAL_BLOCKS_KEY]
assert "Skip the login step" not in texts[guidance]
# Header, plan recitation and the observation stay in the history.
assert texts[0].startswith("# CURRENT OBSERVATION")
plan = next(i for i, t in texts.items() if t.startswith(PLAN_RECITATION_MARKER))
screenshot = next(i for i, t in texts.items() if t == "--- Current Screenshot ---")
assert not {0, plan, screenshot} & set(tail.additional_kwargs[EPHEMERAL_BLOCKS_KEY])
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_transcript_actionless_turn_commits_without_validator_message(): async def test_transcript_actionless_turn_commits_without_validator_message():
ctx = _transcript_ctx() ctx = _transcript_ctx()
+1 -1
View File
@@ -287,7 +287,7 @@ async def test_outputter_builds_concise_history(mock_get_llm):
} }
] ]
mock_data_engine.get_agent_friendly_steps.return_value = steps mock_data_engine.get_agent_friendly_steps.return_value = steps
# M5: the transcript flag defaults on, so the outputter's compiled view # The transcript flag defaults on, so the outputter's compiled view
# consults the chunk store; no persisted chunks keeps the classic path. # consults the chunk store; no persisted chunks keeps the classic path.
mock_data_engine.get_history_chunks.return_value = [] mock_data_engine.get_history_chunks.return_value = []
mock_data_engine.base_dir = "/tmp/mock_session" mock_data_engine.base_dir = "/tmp/mock_session"
+1 -1
View File
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
"""SummarizerNode tests (M2: visual-transition lens dispatch). """Tests for SummarizerNode visual-transition dispatch.
Per the history redesign §5 the Pro SummarizerNode no longer runs its own LLM Per the history redesign §5 the Pro SummarizerNode no longer runs its own LLM
call to produce a text capsule: it dispatches the step to the shared call to produce a text capsule: it dispatches the step to the shared
+364
View File
@@ -0,0 +1,364 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Unit tests for the end-to-end device observation smoke test (no real device)."""
import asyncio
import base64
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock
import pytest
from artemis.core.diagnostics import smoke_test_device
from artemis.core.diagnostics import device_smoke
from artemis.runtime.device_lock import DeviceExecutionLock, DeviceLockOwner
SERIAL = "59100DLCR0033X"
FAKE_JPEG = b"\xff\xd8\xff" + bytes(range(256)) * 8 + b"\xff\xd9"
@pytest.fixture(autouse=True)
def idle_devices(monkeypatch):
"""No live lock owners unless a test installs some."""
monkeypatch.setattr(DeviceExecutionLock, "get_active_owners", classmethod(lambda cls: {}))
monkeypatch.delenv("ARTEMIS_DEVICE_ID", raising=False)
monkeypatch.delenv("ADB_DEVICE_SERIAL", raising=False)
@pytest.fixture
def patch_controller(monkeypatch):
"""Install a fake ``_get_controller`` and return the controller it hands out."""
def _install(get_screen_data=None, factory=None):
controller = MagicMock()
controller.ctx = SimpleNamespace(device=SimpleNamespace(device_id=SERIAL))
if get_screen_data is not None:
controller.get_screen_data = get_screen_data
calls: list[str | None] = []
def _fake_get_controller(device_serial=None):
calls.append(device_serial)
if factory is not None:
return factory(device_serial)
return controller
monkeypatch.setattr("artemis.mcp.adb_server._get_controller", _fake_get_controller)
controller.calls = calls
return controller
return _install
def _screen_data(elements, jpeg=FAKE_JPEG):
return SimpleNamespace(
base64=base64.b64encode(jpeg).decode("ascii"),
elements=elements,
width=1080,
height=2400,
)
@pytest.mark.asyncio
async def test_success_reports_bytes_elements_serial_and_elapsed(patch_controller):
elements = [{"class": "android.widget.FrameLayout"}, {"text": "Hello"}, {"text": "World"}]
controller = patch_controller(AsyncMock(return_value=_screen_data(elements)))
result = await smoke_test_device(device_serial=SERIAL, timeout_seconds=5)
assert result["ok"] is True
assert result["error"] is None
assert result["fix"] == []
assert result["serial"] == SERIAL
assert result["screenshot_bytes"] == len(FAKE_JPEG)
assert result["element_count"] == 3
assert isinstance(result["elapsed_seconds"], float)
assert result["elapsed_seconds"] >= 0.0
assert controller.calls == [SERIAL]
controller.get_screen_data.assert_awaited_once()
assert set(result) == {
"ok",
"serial",
"elapsed_seconds",
"screenshot_bytes",
"element_count",
"error",
"fix",
}
@pytest.mark.asyncio
async def test_serial_comes_from_controller_when_not_requested(patch_controller):
patch_controller(AsyncMock(return_value=_screen_data([{"text": "x"}])))
result = await smoke_test_device()
assert result["ok"] is True
assert result["serial"] == SERIAL
@pytest.mark.asyncio
async def test_xml_string_elements_are_counted_by_node(patch_controller):
xml = '<hierarchy><node class="a"><node class="b"/><node class="c"/></node></hierarchy>'
patch_controller(AsyncMock(return_value=_screen_data(xml)))
result = await smoke_test_device(device_serial=SERIAL)
assert result["ok"] is True
assert result["element_count"] == 3
@pytest.mark.asyncio
async def test_init_failure_no_devices_points_to_launch_avd(patch_controller):
def _boom(device_serial):
raise Exception("No Android devices found at localhost:5037")
patch_controller(factory=_boom)
result = await smoke_test_device(device_serial=SERIAL, timeout_seconds=5)
assert result["ok"] is False
assert result["serial"] == SERIAL
assert result["screenshot_bytes"] is None
assert result["element_count"] is None
assert "No Android devices found" in result["error"]
assert any("launch_avd" in step for step in result["fix"])
assert any("mobile_diagnose" in step for step in result["fix"])
@pytest.mark.asyncio
async def test_init_failure_target_not_found_points_to_launch_avd(patch_controller):
def _boom(device_serial):
raise Exception(
f"Target Android device '{device_serial}' not found at localhost:5037 among available devices."
)
patch_controller(factory=_boom)
result = await smoke_test_device(device_serial="emulator-5554")
assert result["ok"] is False
assert result["serial"] == "emulator-5554"
assert any("launch_avd" in step for step in result["fix"])
@pytest.mark.asyncio
async def test_screen_data_timeout_reports_timeout_and_force_stop(patch_controller):
async def _hang():
await asyncio.sleep(3)
return _screen_data([{"text": "late"}])
patch_controller(_hang)
result = await smoke_test_device(device_serial=SERIAL, timeout_seconds=0.3)
assert result["ok"] is False
assert result["serial"] == SERIAL
assert result["error"] == "UIAutomator/screen capture did not respond within 0.3s"
assert result["elapsed_seconds"] < 2.5
joined = "\n".join(result["fix"])
assert f"adb -s {SERIAL} shell am force-stop com.github.uiautomator" in joined
assert "uiautomator2 init" in joined
assert "screen on" in joined
@pytest.mark.asyncio
async def test_init_timeout_is_reported_without_blocking(patch_controller):
import time
def _slow(device_serial):
time.sleep(2)
return MagicMock()
patch_controller(factory=_slow)
result = await smoke_test_device(device_serial=SERIAL, timeout_seconds=0.3)
assert result["ok"] is False
assert "did not respond within 0.3s" in result["error"]
assert result["elapsed_seconds"] < 1.5
assert any("force-stop" in step for step in result["fix"])
@pytest.mark.asyncio
async def test_generic_screen_data_exception_falls_back_to_manual_dump(patch_controller):
patch_controller(AsyncMock(side_effect=RuntimeError("weird failure 42")))
result = await smoke_test_device(device_serial=SERIAL)
assert result["ok"] is False
assert result["error"] == "Screen capture failed: weird failure 42"
assert any("uiautomator dump" in step for step in result["fix"])
@pytest.mark.asyncio
async def test_base_exception_from_uiautomator2_is_captured_not_raised(patch_controller):
class HTTPTimeoutError(BaseException):
"""uiautomator2 errors derive from BaseException."""
patch_controller(
AsyncMock(side_effect=HTTPTimeoutError("HTTP request timeout: read timed out"))
)
result = await smoke_test_device(device_serial=SERIAL)
assert result["ok"] is False
assert "HTTPTimeoutError" in result["error"]
assert any("force-stop" in step for step in result["fix"])
@pytest.mark.asyncio
async def test_unauthorized_and_offline_guidance(patch_controller):
def _unauth(device_serial):
raise Exception("device unauthorized. This adb server's $ADB_VENDOR_KEYS is not set")
patch_controller(factory=_unauth)
result = await smoke_test_device(device_serial=SERIAL)
assert result["ok"] is False
assert any("USB debugging" in step for step in result["fix"])
def _offline(device_serial):
raise Exception("device offline")
patch_controller(factory=_offline)
result = await smoke_test_device(device_serial=SERIAL)
assert result["ok"] is False
assert any("adb reconnect" in step for step in result["fix"])
@pytest.mark.asyncio
async def test_empty_hierarchy_is_not_ok(patch_controller):
patch_controller(AsyncMock(return_value=_screen_data([])))
result = await smoke_test_device(device_serial=SERIAL)
assert result["ok"] is False
assert result["screenshot_bytes"] == len(FAKE_JPEG)
assert result["element_count"] == 0
assert "no UI elements" in result["error"]
assert any("force-stop" in step for step in result["fix"])
@pytest.mark.asyncio
async def test_placeholder_screenshot_is_not_ok(patch_controller):
one_pixel_png = base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
)
patch_controller(AsyncMock(return_value=_screen_data([{"text": "x"}], jpeg=one_pixel_png)))
result = await smoke_test_device(device_serial=SERIAL)
assert result["ok"] is False
assert result["screenshot_bytes"] == len(one_pixel_png)
assert "placeholder" in result["error"]
def _owner(device_id: str, description: str = "task-1") -> DeviceLockOwner:
return DeviceLockOwner(
pid=4242,
process_created_at=1.0,
token="tok",
device_id=device_id,
description=description,
acquired_at="2026-09-09T00:00:00Z",
)
@pytest.mark.asyncio
async def test_busy_device_is_reported_without_touching_the_controller(
patch_controller, monkeypatch
):
controller = patch_controller(AsyncMock(return_value=_screen_data([{"text": "x"}])))
monkeypatch.setattr(
DeviceExecutionLock,
"get_active_owners",
classmethod(lambda cls: {SERIAL: _owner(SERIAL, "youtube smoke")}),
)
result = await smoke_test_device(device_serial=SERIAL)
assert result["ok"] is False
assert result["serial"] == SERIAL
assert "busy" in result["error"]
assert "youtube smoke" in result["error"]
assert "4242" in result["error"]
assert any("mobile_manage_task" in step for step in result["fix"])
assert controller.calls == []
controller.get_screen_data.assert_not_awaited()
@pytest.mark.asyncio
async def test_busy_guard_matches_scoped_lock_keys(patch_controller, monkeypatch):
controller = patch_controller(AsyncMock(return_value=_screen_data([{"text": "x"}])))
monkeypatch.setattr(
DeviceExecutionLock,
"get_active_owners",
classmethod(lambda cls: {f"endpoint1__{SERIAL}": _owner(SERIAL)}),
)
result = await smoke_test_device(device_serial=SERIAL)
assert result["ok"] is False
assert "busy" in result["error"]
assert controller.calls == []
@pytest.mark.asyncio
async def test_busy_guard_ignores_other_devices(patch_controller, monkeypatch):
controller = patch_controller(AsyncMock(return_value=_screen_data([{"text": "x"}])))
monkeypatch.setattr(
DeviceExecutionLock,
"get_active_owners",
classmethod(lambda cls: {"emulator-5554": _owner("emulator-5554")}),
)
result = await smoke_test_device(device_serial=SERIAL)
assert result["ok"] is True
assert controller.calls == [SERIAL]
@pytest.mark.asyncio
async def test_busy_guard_without_target_serial_treats_any_owner_as_busy(
patch_controller, monkeypatch
):
controller = patch_controller(AsyncMock(return_value=_screen_data([{"text": "x"}])))
monkeypatch.setattr(
DeviceExecutionLock,
"get_active_owners",
classmethod(lambda cls: {"emulator-5554": _owner("emulator-5554")}),
)
result = await smoke_test_device()
assert result["ok"] is False
assert result["serial"] == "emulator-5554"
assert "busy" in result["error"]
assert controller.calls == []
def test_fix_for_error_generic_fallback_uses_serial():
fix = device_smoke.fix_for_error("something odd", SERIAL)
assert fix
assert all(SERIAL in step for step in fix)
assert any("uiautomator dump" in step for step in fix)
assert device_smoke.fix_for_error(None, SERIAL) == []
assert "<serial>" in device_smoke.fix_for_error("timeout", None)[1]
def test_smoke_test_is_exported_from_diagnostics_package():
import artemis.core.diagnostics as diagnostics
assert "smoke_test_device" in diagnostics.__all__
assert diagnostics.smoke_test_device is device_smoke.smoke_test_device
+84
View File
@@ -429,3 +429,87 @@ async def test_emulator_manager_lifecycle():
dismiss_res = manager.dismiss() dismiss_res = manager.dismiss()
assert dismiss_res["success"] is True assert dismiss_res["success"] is True
assert manager.get_status().status == EmulatorLaunchStage.IDLE assert manager.get_status().status == EmulatorLaunchStage.IDLE
@pytest.mark.asyncio
async def test_build_report_turns_crashing_probe_into_fail_result():
engine = ReadinessEngine()
healthy = ProbeResult(
id="healthy",
category=ProbeCategory.RUNTIME,
title="Healthy",
status=ProbeStatus.PASS,
is_blocker=True,
summary="Ready",
description="Ready",
)
good = Mock()
good.probe_id = "healthy"
good.category = ProbeCategory.RUNTIME
good.is_blocker = True
good.probe = AsyncMock(return_value=healthy)
bad = Mock()
bad.probe_id = "integration_host"
bad.category = ProbeCategory.RUNTIME
bad.is_blocker = True
bad.probe = AsyncMock(side_effect=PermissionError(13, "Permission denied", "/ro/traces"))
engine._probes = {"healthy": good, "integration_host": bad}
report = await engine._build_report()
assert report.overall_ready is False
assert report.blocker_count == 2
assert report.passed_blocker_count == 1
by_id = {r.id: r for r in report.probes}
assert by_id["healthy"].status is ProbeStatus.PASS
crashed = by_id["integration_host"]
assert crashed.status is ProbeStatus.FAIL
assert crashed.is_blocker is True
assert crashed.category is ProbeCategory.RUNTIME
assert crashed.summary == "Probe crashed"
assert "PermissionError" in crashed.description
assert "Permission denied" in crashed.description
assert crashed.metadata["exception_type"] == "PermissionError"
@pytest.mark.asyncio
async def test_build_report_turns_hung_probe_into_fail_result(monkeypatch):
"""A probe that never returns (wedged ADB server) is cut off at the
engine's per-probe deadline and reported as a FAIL with the restart step,
so no surface running the report can hang."""
monkeypatch.setattr(ReadinessEngine, "PROBE_TIMEOUT_SECONDS", 0.05)
engine = ReadinessEngine()
async def never_returns():
await asyncio.sleep(10)
hung = Mock()
hung.probe_id = "android_adb"
hung.category = ProbeCategory.DEVICE
hung.is_blocker = True
hung.probe = never_returns
engine._probes = {"android_adb": hung}
report = await engine._build_report()
result = report.probes[0]
assert result.status is ProbeStatus.FAIL
assert result.summary == "Probe timed out"
assert result.metadata["exception_type"] == "TimeoutError"
assert any("adb kill-server" in a.payload for a in result.actions)
assert report.overall_ready is False
@pytest.mark.asyncio
async def test_build_report_does_not_swallow_cancellation():
engine = ReadinessEngine()
probe = Mock()
probe.probe_id = "cancelled"
probe.category = ProbeCategory.RUNTIME
probe.is_blocker = True
probe.probe = AsyncMock(side_effect=asyncio.CancelledError())
engine._probes = {"cancelled": probe}
with pytest.raises(asyncio.CancelledError):
await engine._build_report()
+471
View File
@@ -0,0 +1,471 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Unit tests for the MCP / IDE integration-host readiness probe."""
import os
from pathlib import Path
import sys
import pytest
from artemis.core.diagnostics import IntegrationHostProbe, ReadinessEngine
from artemis.core.diagnostics.probes import host_probe
from artemis.core.diagnostics.schema import ProbeCategory, ProbeStatus
#: Environment keys / prefixes any IDE might export; scrubbed so the developer's
#: own host (this test suite is often run from inside Claude Code or Cursor) does
#: not leak into the detection tests.
_SIGNAL_PREFIXES = (
"ANTIGRAVITY_",
"CLAUDE_CODE_",
"CURSOR_",
"WINDSURF_",
"CODEX_",
"OPENCLAW_",
"CLINE_",
"ROO_",
"VSCODE_",
)
_SIGNAL_KEYS = ("CLAUDECODE", "TERM_PROGRAM")
@pytest.fixture
def clean_ide_env(monkeypatch):
for key in list(os.environ):
if key in _SIGNAL_KEYS or key.startswith(_SIGNAL_PREFIXES):
monkeypatch.delenv(key, raising=False)
return monkeypatch
@pytest.fixture
def host_env(tmp_path, monkeypatch, clean_ide_env):
"""A consistent, healthy host layout the individual tests then break."""
venv_python = host_probe.project_venv_python(tmp_path)
venv_python.parent.mkdir(parents=True)
venv_python.write_bytes(b"")
traces = tmp_path / "traces"
monkeypatch.setattr("artemis.config.paths.ROOT_DIR", tmp_path)
monkeypatch.setattr("artemis.config.paths.get_env_file", lambda: tmp_path / ".env")
monkeypatch.setattr("artemis.config.paths.get_default_traces_path", lambda: traces)
monkeypatch.setattr("artemis.config.paths.is_source_checkout", lambda: True)
monkeypatch.setattr(sys, "executable", str(venv_python))
monkeypatch.setattr("artemis.runtime.daemon_client.is_standalone_forced", lambda: False)
monkeypatch.setattr("artemis.runtime.daemon_client.is_artemis_daemon", lambda: True)
monkeypatch.setattr(
"artemis.runtime.daemon_client.daemon_log_path", lambda: tmp_path / "daemon.log"
)
monkeypatch.setattr("artemis.runtime.server_lifecycle.is_port_in_use", lambda *a, **k: True)
return tmp_path
def test_probe_is_not_part_of_the_default_console_report():
assert "integration_host" not in ReadinessEngine()._probes
probe = IntegrationHostProbe()
assert probe.probe_id == "integration_host"
assert probe.category is ProbeCategory.RUNTIME
assert probe.is_blocker is True
@pytest.mark.asyncio
async def test_consistent_host_passes(host_env):
result = await IntegrationHostProbe().probe()
assert result.status is ProbeStatus.PASS
assert result.is_blocker is True
assert result.summary == "Host Ready"
meta = result.metadata
assert meta["venv_exists"] is True
assert meta["interpreter_matches_venv"] is True
assert Path(meta["runner_python"]) == host_probe.project_venv_python(host_env)
assert meta["env_file_exists"] is False
assert meta["traces_dir_writable"] is True
assert (host_env / "traces").is_dir()
assert meta["daemon"] == {
"host": "127.0.0.1",
"port": 8000,
"standalone_forced": False,
"reachable": True,
"port_in_use": True,
"port_held_by_other_process": False,
"log_path": str(host_env / "daemon.log"),
}
labels = [a.label for a in result.actions]
assert labels[0] == "Host OK"
assert "Env File Location" in labels # informational even when passing
@pytest.mark.asyncio
async def test_daemon_port_held_by_other_process_warns(host_env, monkeypatch):
monkeypatch.setattr("artemis.runtime.daemon_client.is_artemis_daemon", lambda: False)
result = await IntegrationHostProbe().probe()
assert result.status is ProbeStatus.WARN
assert result.is_blocker is False # tasks still run standalone
assert result.metadata["daemon"]["port_held_by_other_process"] is True
assert "port 8000 is held" in result.description.lower()
port_fix = next(a for a in result.actions if a.label == "Free the Daemon Port")
assert "ARTEMIS_DAEMON_PORT" in port_fix.payload
@pytest.mark.asyncio
async def test_daemon_not_running_and_port_free_is_informational(host_env, monkeypatch):
monkeypatch.setattr("artemis.runtime.daemon_client.is_artemis_daemon", lambda: False)
monkeypatch.setattr("artemis.runtime.server_lifecycle.is_port_in_use", lambda *a, **k: False)
result = await IntegrationHostProbe().probe()
assert result.status is ProbeStatus.PASS
assert "auto-start" in result.description
auto = next(a for a in result.actions if a.label == "Daemon Auto-Start")
assert str(host_env / "daemon.log") in auto.payload
@pytest.mark.asyncio
async def test_standalone_mode_skips_daemon_probe(host_env, monkeypatch):
monkeypatch.setattr("artemis.runtime.daemon_client.is_standalone_forced", lambda: True)
monkeypatch.setattr("artemis.runtime.daemon_client.is_artemis_daemon", lambda: False)
result = await IntegrationHostProbe().probe()
assert result.status is ProbeStatus.PASS
assert result.metadata["daemon"]["standalone_forced"] is True
assert result.metadata["daemon"]["port_held_by_other_process"] is False
auto = next(a for a in result.actions if a.label == "Daemon Auto-Start")
assert "ARTEMIS_STANDALONE" in auto.payload
@pytest.mark.asyncio
async def test_interpreter_mismatch_warns_and_points_at_installer(host_env, monkeypatch):
other = host_env / "other-python.exe"
other.write_bytes(b"")
monkeypatch.setattr(sys, "executable", str(other))
result = await IntegrationHostProbe().probe()
assert result.status is ProbeStatus.WARN
assert result.metadata["interpreter_matches_venv"] is False
assert Path(result.metadata["runner_python"]) == host_probe.project_venv_python(host_env)
assert result.metadata["mcp_client"] is None
assert not [a for a in result.actions if a.action_type == "command"]
regen = next(a for a in result.actions if a.label == "Regenerate MCP Config")
assert regen.action_type == "hint"
assert "ask the user which ide" in regen.payload.lower()
assert "uv run artemis mcp --install <client>" in regen.payload
for name in host_probe.MCP_CLIENT_NAMES:
assert name in regen.payload
assert next(a for a in result.actions if a.label == "Interpreter Mismatch")
@pytest.mark.asyncio
async def test_interpreter_mismatch_emits_runnable_install_for_detected_client(
host_env, monkeypatch
):
monkeypatch.setenv("CLAUDECODE", "1")
other = host_env / "other-python.exe"
other.write_bytes(b"")
monkeypatch.setattr(sys, "executable", str(other))
result = await IntegrationHostProbe().probe()
assert result.status is ProbeStatus.WARN
assert result.metadata["mcp_client"] == "claude"
regen = next(a for a in result.actions if a.action_type == "command")
assert regen.label == "Regenerate MCP Config"
assert regen.payload == "uv run artemis mcp --install claude"
assert "<client>" not in regen.payload
@pytest.mark.asyncio
async def test_metadata_reports_client_even_when_host_is_healthy(host_env, monkeypatch):
monkeypatch.setenv("CURSOR_TRACE_ID", "abc")
result = await IntegrationHostProbe().probe()
assert result.status is ProbeStatus.PASS
assert result.metadata["mcp_client"] == "cursor"
def test_detect_mcp_client_unknown_in_clean_environment(clean_ide_env):
assert host_probe.detect_mcp_client() is None
@pytest.mark.parametrize(
("variables", "expected"),
[
({"ANTIGRAVITY_LS_ADDRESS": "127.0.0.1:1"}, "antigravity"),
({"ANTIGRAVITY_CSRF_TOKEN": "t"}, "antigravity"),
({"CLAUDECODE": "1"}, "claude"),
({"CLAUDE_CODE_ENTRYPOINT": "cli"}, "claude"),
({"CURSOR_TRACE_ID": "abc"}, "cursor"),
({"CURSOR_AGENT": "1"}, "cursor"),
({"WINDSURF_SESSION": "x"}, "windsurf"),
({"CODEX_SANDBOX": "seatbelt"}, "codex"),
({"CODEX_HOME": "/tmp/codex", "CODEX_SANDBOX_NETWORK_DISABLED": "1"}, "codex"),
({"OPENCLAW_SESSION_ID": "s"}, "openclaw"),
({"CLINE_TASK_ID": "1"}, "cline"),
({"ROO_CODE_TASK": "1"}, "roo"),
({"VSCODE_PID": "42"}, "vscode"),
({"TERM_PROGRAM": "vscode"}, "vscode"),
],
)
def test_detect_mcp_client_signals(clean_ide_env, variables, expected):
for key, value in variables.items():
clean_ide_env.setenv(key, value)
assert host_probe.detect_mcp_client() == expected
@pytest.mark.parametrize(
"variables",
[
{"CODEX_HOME": "/home/me/.codex"}, # user relocated the Codex config dir
{"OPENCLAW_WEBHOOK_URL": "https://hooks.example/x"}, # notifier setting, any host
{"TERM_PROGRAM": "iTerm.app"},
],
)
def test_detect_mcp_client_ignores_weak_or_user_set_variables(clean_ide_env, variables):
for key, value in variables.items():
clean_ide_env.setenv(key, value)
assert host_probe.detect_mcp_client() is None
@pytest.mark.parametrize(
("variables", "expected"),
[
# Claude Code or Codex running inside a VS Code / Cursor terminal: the
# agent that spawned the server wins over the editor around it.
({"CLAUDECODE": "1", "VSCODE_PID": "1", "TERM_PROGRAM": "vscode"}, "claude"),
({"CLAUDECODE": "1", "CURSOR_TRACE_ID": "x"}, "claude"),
({"CODEX_SANDBOX": "seatbelt", "VSCODE_PID": "1"}, "codex"),
# Cursor / Windsurf / Cline are VS Code forks or extensions and inherit
# the generic VS Code markers.
({"CURSOR_TRACE_ID": "x", "VSCODE_PID": "1", "TERM_PROGRAM": "vscode"}, "cursor"),
({"WINDSURF_SESSION": "x", "TERM_PROGRAM": "vscode"}, "windsurf"),
({"CLINE_TASK_ID": "1", "VSCODE_PID": "1"}, "cline"),
],
)
def test_detect_mcp_client_specific_host_beats_generic_vscode(clean_ide_env, variables, expected):
for key, value in variables.items():
clean_ide_env.setenv(key, value)
assert host_probe.detect_mcp_client() == expected
def test_mcp_install_actions_shapes():
(cmd,) = host_probe.mcp_install_actions("antigravity")
assert cmd.action_type == "command"
assert cmd.payload == "uv run artemis mcp --install antigravity"
(hint,) = host_probe.mcp_install_actions(None)
assert hint.action_type == "hint"
assert "antigravity, claude, cursor, windsurf, vscode, cline, roo, openclaw, codex" in (
hint.payload
)
@pytest.mark.asyncio
async def test_missing_venv_warns_with_uv_sync(host_env, monkeypatch):
host_probe.project_venv_python(host_env).unlink()
monkeypatch.setattr(sys, "executable", str(host_env / "system-python.exe"))
result = await IntegrationHostProbe().probe()
assert result.status is ProbeStatus.WARN
assert result.metadata["venv_exists"] is False
assert result.metadata["runner_python"] == str(host_env / "system-python.exe")
assert any(a.payload == "uv sync" for a in result.actions)
@pytest.mark.asyncio
async def test_unwritable_traces_dir_fails_even_with_other_warnings(host_env, monkeypatch):
monkeypatch.setattr(host_probe, "directory_is_writable", lambda _p: (False, "denied"))
monkeypatch.setattr("artemis.runtime.daemon_client.is_artemis_daemon", lambda: False)
result = await IntegrationHostProbe().probe()
assert result.status is ProbeStatus.FAIL
assert result.is_blocker is True
assert result.summary == "Host Misconfigured"
assert result.metadata["traces_dir_writable"] is False
assert result.metadata["traces_dir_error"] == "denied"
assert result.actions[0].label == "Fix Traces Directory"
assert "ARTEMIS_TRACES_DIR" in result.actions[0].payload
# The port problem is still reported alongside the blocking failure.
assert "port 8000 is held" in result.description.lower()
def test_directory_is_writable_reports_real_errors(tmp_path):
ok, error = host_probe.directory_is_writable(tmp_path / "new" / "nested")
assert ok is True and error is None
blocker = tmp_path / "file"
blocker.write_text("x")
ok, error = host_probe.directory_is_writable(blocker / "child")
assert ok is False and error
@pytest.mark.asyncio
async def test_wheel_install_without_venv_is_not_a_venv_warning(host_env, monkeypatch):
"""A wheel install has no .venv under site-packages; that is normal, not a warning."""
host_probe.project_venv_python(host_env).unlink()
system_python = host_env / "system-python.exe"
monkeypatch.setattr(sys, "executable", str(system_python))
monkeypatch.setattr("artemis.config.paths.is_source_checkout", lambda: False)
result = await IntegrationHostProbe().probe()
assert result.status is ProbeStatus.PASS
assert result.summary == "Host Ready"
meta = result.metadata
assert meta["is_source_checkout"] is False
assert meta["venv_python"] is None
assert meta["venv_exists"] is False
assert meta["interpreter_matches_venv"] is False
assert meta["runner_python"] == str(system_python)
assert "venv" not in result.description.lower()
assert not [a for a in result.actions if a.payload == "uv sync"]
assert not [a for a in result.actions if a.label == "Interpreter Mismatch"]
@pytest.mark.asyncio
async def test_wheel_install_ignores_stray_venv_next_to_site_packages(host_env, monkeypatch):
"""Even if a .venv happens to exist under ROOT_DIR, a non-source install never compares against it."""
other = host_env / "other-python.exe"
other.write_bytes(b"")
monkeypatch.setattr(sys, "executable", str(other))
monkeypatch.setattr("artemis.config.paths.is_source_checkout", lambda: False)
result = await IntegrationHostProbe().probe()
assert result.status is ProbeStatus.PASS
assert result.metadata["venv_python"] is None
assert result.metadata["runner_python"] == str(other)
assert not [a for a in result.actions if a.label == "Interpreter Mismatch"]
@pytest.mark.asyncio
async def test_source_checkout_without_venv_still_warns_with_uv_sync(host_env, monkeypatch):
host_probe.project_venv_python(host_env).unlink()
monkeypatch.setattr(sys, "executable", str(host_env / "system-python.exe"))
monkeypatch.setattr("artemis.config.paths.is_source_checkout", lambda: True)
result = await IntegrationHostProbe().probe()
assert result.status is ProbeStatus.WARN
assert result.metadata["is_source_checkout"] is True
assert result.metadata["venv_python"] == str(host_probe.project_venv_python(host_env))
assert result.metadata["venv_exists"] is False
assert "no project virtualenv" in result.description.lower()
assert any(a.payload == "uv sync" for a in result.actions)
@pytest.mark.asyncio
async def test_traces_path_creation_error_becomes_fail_not_crash(host_env, monkeypatch):
"""get_default_traces_path mkdirs as a side effect; a read-only parent must not crash the probe."""
def denied():
raise PermissionError(13, "Permission denied", str(host_env / "traces"))
monkeypatch.setattr("artemis.config.paths.get_default_traces_path", denied)
monkeypatch.setenv("ARTEMIS_TRACES_DIR", str(host_env / "ro" / "traces"))
result = await IntegrationHostProbe().probe()
assert result.status is ProbeStatus.FAIL
assert result.is_blocker is True
assert result.summary == "Host Misconfigured"
meta = result.metadata
assert meta["traces_dir_writable"] is False
assert "Permission denied" in meta["traces_dir_error"]
# The intended path is still named, resolved without touching the disk.
assert Path(meta["traces_dir"]) == host_env / "ro" / "traces"
assert not (host_env / "ro").exists()
assert result.actions[0].label == "Fix Traces Directory"
assert "ARTEMIS_TRACES_DIR" in result.actions[0].payload
assert "not writable" in result.description
def test_intended_traces_path_mirrors_selection_without_mkdir(tmp_path, monkeypatch):
monkeypatch.setattr("artemis.config.paths.ROOT_DIR", tmp_path)
monkeypatch.delenv("ARTEMIS_TRACES_DIR", raising=False)
monkeypatch.setattr("artemis.config.paths._use_user_app_dir", lambda: False)
assert host_probe.intended_traces_path() == tmp_path / "traces"
monkeypatch.setattr("artemis.config.paths._use_user_app_dir", lambda: True)
monkeypatch.setattr("artemis.config.paths.get_app_dir", lambda: tmp_path / "app")
assert host_probe.intended_traces_path() == tmp_path / "app" / "traces"
monkeypatch.setenv("ARTEMIS_TRACES_DIR", str(tmp_path / "env-traces"))
assert host_probe.intended_traces_path() == tmp_path / "env-traces"
assert not (tmp_path / "traces").exists()
assert not (tmp_path / "app").exists()
assert not (tmp_path / "env-traces").exists()
# --------------------------------------------------------------------------- #
# Daemon identity (a dev server on port 8000 is not the daemon)
# --------------------------------------------------------------------------- #
class _FakeResponse:
def __init__(self, status: int, body: bytes):
self.status = status
self._body = body
def read(self) -> bytes:
return self._body
def __enter__(self):
return self
def __exit__(self, *exc):
return False
@pytest.mark.parametrize(
"status, body, expected",
[
(200, b'{"running": true, "port": 8000, "current_pid": 42}', True),
(200, b"<!doctype html><title>Vite App</title>", False), # dev server on /
(200, b'{"hello": "world"}', False),
(404, b"", False),
],
)
def test_is_artemis_daemon_requires_the_server_status_shape(monkeypatch, status, body, expected):
from artemis.runtime import daemon_client
seen: dict[str, str] = {}
def fake_urlopen(req, timeout):
seen["url"] = req.full_url
return _FakeResponse(status, body)
monkeypatch.setattr(daemon_client.urllib.request, "urlopen", fake_urlopen)
assert daemon_client.is_artemis_daemon("127.0.0.1", 8000) is expected
assert seen["url"] == "http://127.0.0.1:8000/api/system/server-status"
def test_is_artemis_daemon_is_false_when_nothing_listens(monkeypatch):
from artemis.runtime import daemon_client
def refuse(req, timeout):
raise ConnectionRefusedError()
monkeypatch.setattr(daemon_client.urllib.request, "urlopen", refuse)
assert daemon_client.is_artemis_daemon("127.0.0.1", 8000) is False
+157
View File
@@ -0,0 +1,157 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Unit tests for the readiness core shared by ``artemis doctor`` and ``mobile_diagnose``."""
import asyncio
import time
from unittest.mock import AsyncMock
from artemis.core.diagnostics import readiness
from artemis.core.diagnostics.probes.host_probe import IntegrationHostProbe
from artemis.core.diagnostics.readiness import (
CHECK_ORDER,
adb_keys_corrupted,
base_verdict,
collect_readiness,
sort_by_fix_order,
)
from artemis.core.diagnostics.schema import (
ProbeCategory,
ProbeResult,
ProbeStatus,
SystemReadinessReport,
)
def _probe(
probe_id: str,
status: ProbeStatus = ProbeStatus.PASS,
*,
blocker: bool = True,
metadata: dict | None = None,
) -> ProbeResult:
return ProbeResult(
id=probe_id,
title=probe_id,
category=ProbeCategory.RUNTIME,
status=status,
summary=status.value,
description=f"{probe_id} is {status.value}",
is_blocker=blocker,
metadata=metadata or {},
)
# --------------------------------------------------------------------------- #
# base_verdict
# --------------------------------------------------------------------------- #
def test_base_verdict_without_any_blocker_is_blocked():
assert base_verdict([]) == "blocked"
assert base_verdict([_probe("toolchain", blocker=False)]) == "blocked"
def test_base_verdict_failing_blocker_is_blocked():
results = [
_probe("python_runtime"),
_probe("android_adb", ProbeStatus.FAIL),
_probe("toolchain", blocker=False),
]
assert base_verdict(results) == "blocked"
def test_base_verdict_passing_blockers_with_warning_optional_is_degraded():
results = [
_probe("python_runtime"),
_probe("android_adb"),
_probe("toolchain", ProbeStatus.WARN, blocker=False),
]
assert base_verdict(results) == "degraded"
def test_base_verdict_all_pass_is_ready():
results = [_probe("python_runtime"), _probe("android_adb"), _probe("toolchain", blocker=False)]
assert base_verdict(results) == "ready"
def test_base_verdict_warning_blocker_is_blocked_not_degraded():
results = [_probe("python_runtime"), _probe("integration_host", ProbeStatus.WARN)]
assert base_verdict(results) == "blocked"
# --------------------------------------------------------------------------- #
# adb_keys_corrupted
# --------------------------------------------------------------------------- #
def test_adb_keys_corrupted_reads_the_adb_probe_flag():
corrupted = _probe("android_adb", metadata={"adb_keys": {"is_corrupted": True}})
healthy = _probe("android_adb", metadata={"adb_keys": {"is_corrupted": False}})
assert adb_keys_corrupted([_probe("python_runtime"), corrupted]) is True
assert adb_keys_corrupted([_probe("python_runtime"), healthy]) is False
def test_adb_keys_corrupted_is_false_without_probe_or_metadata():
assert adb_keys_corrupted([]) is False
assert adb_keys_corrupted([_probe("python_runtime")]) is False
assert adb_keys_corrupted([_probe("android_adb")]) is False
assert adb_keys_corrupted([_probe("android_adb", metadata={"adb_keys": None})]) is False
# --------------------------------------------------------------------------- #
# ordering / collection
# --------------------------------------------------------------------------- #
def test_sort_by_fix_order_follows_check_order_and_appends_unknown_ids():
results = [
_probe("custom_probe"),
_probe("toolchain", blocker=False),
_probe("android_adb"),
_probe("python_runtime"),
]
ordered = [r.id for r in sort_by_fix_order(results)]
assert ordered == ["python_runtime", "android_adb", "toolchain", "custom_probe"]
assert list(CHECK_ORDER) == [
"python_runtime",
"system_config",
"integration_host",
"gemini_api_key",
"android_adb",
"toolchain",
"vision_ocr_key",
]
def test_collect_readiness_gathers_engine_report_and_host_probe(monkeypatch):
report = SystemReadinessReport(
overall_ready=True,
blocker_count=1,
passed_blocker_count=1,
probes=[_probe("python_runtime")],
os_type="windows",
timestamp=time.time(),
)
host = _probe("integration_host")
run_all = AsyncMock(return_value=report)
monkeypatch.setattr(readiness.readiness_engine, "run_all", run_all)
monkeypatch.setattr(IntegrationHostProbe, "probe", AsyncMock(return_value=host))
got_report, got_host = asyncio.run(collect_readiness())
assert got_report is report
assert got_host is host
run_all.assert_awaited_once_with(force_refresh=True)
+1 -1
View File
@@ -392,7 +392,7 @@ def _foreground_engine(tmp_path):
def test_record_step_persists_explicit_foreground_app(tmp_path): def test_record_step_persists_explicit_foreground_app(tmp_path):
"""M5: the historically dropped foreground_app parameter is persisted.""" """The foreground_app parameter is persisted."""
engine = _foreground_engine(tmp_path) engine = _foreground_engine(tmp_path)
engine.record_step( engine.record_step(
foreground_app="com.android.settings", foreground_app="com.android.settings",
@@ -0,0 +1,79 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""A detached lens call (segment capsule, visual summary) must never be
recorded as the current step's reasoning: the callback handler attaches
``thinking``/``raw_thinking`` traces by the DataEngine's global
``current_step_id``, which is independent of the trace context, so the
handler itself has to recognise the detached lens context."""
import uuid
from unittest.mock import Mock
from langchain_core.messages import AIMessage
from langchain_core.outputs import ChatGeneration, LLMResult
from artemis.data_engine.context_vars import CURRENT_NODE_NAME, CURRENT_TRACE_ID
from artemis.data_engine.trace import DataEngineCallbackHandler, detached_trace
CAPSULE = '{"doing": "scrolling", "did": "swiped", "entry_state": "home"}'
def _handler():
ctx = Mock()
ctx.data_engine = Mock()
ctx.data_engine.current_step_id = uuid.uuid4()
ctx.data_engine.record_trace = Mock()
return DataEngineCallbackHandler(ctx), ctx
def _result(text: str) -> LLMResult:
return LLMResult(generations=[[ChatGeneration(message=AIMessage(content=text))]])
def _recorded(ctx) -> dict[str, dict]:
return {c.kwargs["type"]: c.kwargs for c in ctx.data_engine.record_trace.call_args_list}
def test_agent_call_reply_is_recorded_as_raw_thinking_under_the_step():
handler, ctx = _handler()
span = uuid.uuid4()
token = CURRENT_TRACE_ID.set(span)
token_name = CURRENT_NODE_NAME.set("operator")
try:
handler.on_llm_end(_result("I see the settings list."), run_id=uuid.uuid4())
finally:
CURRENT_TRACE_ID.reset(token)
CURRENT_NODE_NAME.reset(token_name)
recorded = _recorded(ctx)
assert recorded["llm_call"]["step_id"] == ctx.data_engine.current_step_id
assert recorded["raw_thinking"]["step_id"] == ctx.data_engine.current_step_id
assert recorded["raw_thinking"]["payload"]["thought"] == "I see the settings list."
def test_detached_lens_reply_is_neither_thinking_nor_attached_to_the_step():
handler, ctx = _handler()
span = uuid.uuid4()
token = CURRENT_TRACE_ID.set(span)
token_name = CURRENT_NODE_NAME.set("operator")
try:
with detached_trace("lens:step_capsule"):
handler.on_llm_end(_result(CAPSULE), run_id=uuid.uuid4())
finally:
CURRENT_TRACE_ID.reset(token)
CURRENT_NODE_NAME.reset(token_name)
recorded = _recorded(ctx)
assert "raw_thinking" not in recorded
assert "thinking" not in recorded
assert recorded["llm_call"]["step_id"] is None
+353
View File
@@ -208,6 +208,219 @@ async def test_process_plan_write_only_queues(tmp_path):
assert ctx.checkpoint_tasks == {} assert ctx.checkpoint_tasks == {}
VERIFY_LINE = " - verify: the alarm list shows 7:30 AM\n"
ASSERT_LINE = " - assert: a toast appeared\n"
def _guidance_arrives(ctx, plan_text):
"""What perception_node does when a non-empty user instruction lands: the
check lines that exist at that moment lose their machine restoration."""
from artemis.utils.plan_grammar import parse_plan
current = getattr(ctx, "guidance_unprotected_checks", None)
if not isinstance(current, set):
current = set()
ctx.guidance_unprotected_checks = current | {
(ci.kind, ci.text) for ci in parse_plan(plan_text).check_items
}
async def _plan_write(tmp_path, ctx, before, after):
"""One accepted task_plan write through the shared post-write pipeline;
returns the plan text as it stands on disk afterwards."""
from artemis.graph.graph import _process_plan_write
task_plan_path = _write_plan(tmp_path, after)
await _process_plan_write(ctx, _make_state(), task_plan_path, before, after, "ok", True)
return task_plan_path.read_text(encoding="utf-8")
NEW_VERIFY_LINE = " - verify: the build number is recorded\n"
@pytest.mark.asyncio
async def test_pre_guidance_check_lines_may_be_dropped_at_any_later_write(tmp_path):
"""Without guidance a deleted check line grows back. Once user guidance has
arrived, the lines that existed at that moment stay editable for the rest
of the run: the Operator re-plans over several writes, turns later, and
none of them is undone."""
without_verify = PLAN_WITH_CHECKS.replace(VERIFY_LINE, "")
without_both = without_verify.replace(ASSERT_LINE, "")
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
ctx.guidance_unprotected_checks = set()
# Ordinary write (no guidance): the deleted line grows back.
assert VERIFY_LINE in await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, without_verify)
# Turn N: guidance arrives; the Operator only advances a milestone.
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
started_next = PLAN_WITH_CHECKS.replace("- [ ] Next milestone", "- [/] Next milestone")
assert await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, started_next) == started_next
# Turn N+2: the first check line goes, and stands.
dropped_verify = started_next.replace(VERIFY_LINE, "")
assert await _plan_write(tmp_path, ctx, started_next, dropped_verify) == dropped_verify
# Turn N+3: the second one goes too, and also stands (no single-write window).
dropped_both = dropped_verify.replace(ASSERT_LINE, "")
assert await _plan_write(tmp_path, ctx, dropped_verify, dropped_both) == dropped_both
@pytest.mark.asyncio
async def test_check_lines_added_after_guidance_stay_protected(tmp_path):
"""A check line the Operator adds after the instruction is a new standard:
deleting it later is restored, while the pre-guidance line still may go."""
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
ctx.guidance_unprotected_checks = set()
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
with_new = PLAN_WITH_CHECKS.replace(ASSERT_LINE, ASSERT_LINE + NEW_VERIFY_LINE)
assert await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, with_new) == with_new
dropped_new = with_new.replace(NEW_VERIFY_LINE, "")
on_disk = await _plan_write(tmp_path, ctx, with_new, dropped_new)
assert NEW_VERIFY_LINE in on_disk # restored: it postdates the guidance
dropped_old = with_new.replace(VERIFY_LINE, "")
on_disk = await _plan_write(tmp_path, ctx, with_new, dropped_old)
assert VERIFY_LINE not in on_disk # stands: it predates the guidance
assert NEW_VERIFY_LINE in on_disk
@pytest.mark.asyncio
async def test_second_guidance_waives_the_lines_that_exist_by_then(tmp_path):
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
ctx.guidance_unprotected_checks = set()
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
with_new = PLAN_WITH_CHECKS.replace(ASSERT_LINE, ASSERT_LINE + NEW_VERIFY_LINE)
assert await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, with_new) == with_new
dropped_new = with_new.replace(NEW_VERIFY_LINE, "")
assert NEW_VERIFY_LINE in await _plan_write(tmp_path, ctx, with_new, dropped_new)
_guidance_arrives(ctx, with_new) # the new line exists now: it is waived too
assert NEW_VERIFY_LINE not in await _plan_write(tmp_path, ctx, with_new, dropped_new)
@pytest.mark.asyncio
async def test_mixed_write_restores_only_the_protected_lines(tmp_path):
"""One write drops a pre-guidance line and a post-guidance line: only the
protected one is merged back."""
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
ctx.guidance_unprotected_checks = set()
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
with_new = PLAN_WITH_CHECKS.replace(ASSERT_LINE, ASSERT_LINE + NEW_VERIFY_LINE)
assert await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, with_new) == with_new
dropped_both = with_new.replace(VERIFY_LINE, "").replace(NEW_VERIFY_LINE, "")
on_disk = await _plan_write(tmp_path, ctx, with_new, dropped_both)
assert VERIFY_LINE not in on_disk
assert NEW_VERIFY_LINE in on_disk
@pytest.mark.asyncio
async def test_guided_drop_records_the_retired_lines_and_the_outcome_excludes_them(tmp_path):
"""A line dropped under guidance is remembered as retired; a verdict the
ledger recorded for it earlier is reported as retired, not as a failure,
while a line still declared keeps its verdict."""
from artemis.graph.checkpoints import compute_test_summary
from artemis.utils.plan_grammar import parse_plan
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
ctx.guidance_unprotected_checks = set()
ctx.guidance_retired_checks = set()
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
without_verify = PLAN_WITH_CHECKS.replace(VERIFY_LINE, "")
assert await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, without_verify) == without_verify
assert ctx.guidance_retired_checks == {("verify", "the alarm list shows 7:30 AM")}
records = [
{"kind": "verify", "item_text": "the alarm list shows 7:30 AM", "status": "failed"},
{"kind": "assert", "item_text": "a toast appeared", "status": "passed"},
]
summary = compute_test_summary(
list(parse_plan(without_verify).all_check_items),
records,
retired=ctx.guidance_retired_checks,
)
assert (summary.passed, summary.failed, summary.retired) == (1, 0, 1)
assert summary.retired_items == [
{"item_text": "the alarm list shows 7:30 AM", "kind": "verify", "last_status": "failed"}
]
# A retired line that is declared again (the Operator re-added it) counts normally.
summary = compute_test_summary(
list(parse_plan(PLAN_WITH_CHECKS).all_check_items),
records,
retired=ctx.guidance_retired_checks,
)
assert (summary.passed, summary.failed, summary.retired) == (1, 1, 0)
REWORDED_MILESTONE = PLAN_WITH_CHECKS.replace(f"- [x] {GOAL_TEXT}", "- [x] Create the 7:30 alarm")
@pytest.mark.asyncio
async def test_rewording_a_milestone_keeps_its_check_lines_in_place(tmp_path):
"""The check lines under a reworded milestone change signature (parent
hash) but stay declared: they are neither duplicated as task-level @end
orphans nor, under guidance, retired."""
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
ctx.guidance_unprotected_checks = set()
ctx.guidance_retired_checks = set()
# No guidance: nothing to restore, the plan stands as written.
on_disk = await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, REWORDED_MILESTONE)
assert on_disk == REWORDED_MILESTONE
assert "@end" not in on_disk
# Guidance armed: the same edit retires nothing.
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
on_disk = await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, REWORDED_MILESTONE)
assert on_disk == REWORDED_MILESTONE
assert ctx.guidance_retired_checks == set()
@pytest.mark.asyncio
async def test_redeclaring_a_retired_line_reinstates_it(tmp_path):
"""Dropped under guidance, then added back: the harvest gate must see the
line as live again, matching what compute_test_summary reports."""
from artemis.graph.checkpoints import reinstate_check_items
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
ctx.guidance_unprotected_checks = set()
ctx.guidance_retired_checks = set()
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
without_verify = PLAN_WITH_CHECKS.replace(VERIFY_LINE, "")
await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, without_verify)
assert ctx.guidance_retired_checks == {RETIRED_VERIFY}
await _plan_write(tmp_path, ctx, without_verify, PLAN_WITH_CHECKS)
assert ctx.guidance_retired_checks == set()
# Direct helper: unrelated items leave the set untouched.
ctx.guidance_retired_checks = {RETIRED_VERIFY}
reinstate_check_items(ctx, [RETIRED_ASSERT])
assert ctx.guidance_retired_checks == {RETIRED_VERIFY}
@pytest.mark.asyncio
async def test_rejected_plan_write_leaves_the_waiver_untouched(tmp_path):
from langchain_core.messages import ToolMessage
from artemis.graph.graph import _process_plan_write
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
ctx.guidance_unprotected_checks = set()
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
waived = set(ctx.guidance_unprotected_checks)
reworded = PLAN_WITH_CHECKS.replace("- [ ] Next milestone", "- [ ] The next milestone")
task_plan_path = _write_plan(tmp_path, reworded)
result = await _process_plan_write(
ctx, _make_state(), task_plan_path, PLAN_WITH_CHECKS, reworded, "ok", False
)
assert isinstance(result, ToolMessage)
assert task_plan_path.read_text(encoding="utf-8") == PLAN_WITH_CHECKS
assert ctx.guidance_unprotected_checks == waived
# --- §8.1 / §8.10: spawn after record_step; record_step unconditional ---------------- # --- §8.1 / §8.10: spawn after record_step; record_step unconditional ----------------
@@ -515,6 +728,146 @@ async def test_assert_fail_halt_policy_latches_halt(tmp_path):
assert ctx.assert_halt is True assert ctx.assert_halt is True
# --- Retired check items: late verdicts are ledger-only ------------------------------
RETIRED_VERIFY = ("verify", "the alarm list shows 7:30 AM")
RETIRED_ASSERT = ("assert", "a toast appeared")
def _last_done_event(ctx):
calls = [
c for c in ctx.data_engine._publish.call_args_list if c.args[1].get("status") == "done"
]
return calls[-1].args[1]
@pytest.mark.asyncio
async def test_late_failing_verify_for_retired_line_is_ledger_only(tmp_path):
"""The user retired the verify line but kept the milestone: a late failing
verdict for it neither reopens the subgoal nor produces findings/headline,
yet the ledger still books it and the done event reports it as retired."""
plan_path = _write_plan(tmp_path, PLAN_WITH_CHECKS.replace(VERIFY_LINE, ""))
ctx = _make_ctx(tmp_path)
ctx.guidance_retired_checks = {RETIRED_VERIFY}
run = _done_run(ctx, CheckReport(verdicts=[_verdict("verify", "failed")]))
findings = harvest_run(ctx, _make_state(), run, allow_side_effects=True)
assert findings == []
assert f"- [x] {GOAL_TEXT}" in plan_path.read_text(encoding="utf-8")
assert "- finding:" not in plan_path.read_text(encoding="utf-8")
assert ctx.checkpoint_repairs == {}
assert getattr(ctx, "checker_findings", {}) == {}
records = read_ledger(tmp_path)
assert [(r["kind"], r["status"]) for r in records] == [("verify", "failed")]
event = _last_done_event(ctx)
assert event["applicable"] is False
assert event["reverted"] is False
assert event["retired"] == [{"kind": "verify", "item_text": RETIRED_VERIFY[1]}]
assert [v["status"] for v in event["verdicts"]] == ["failed"]
@pytest.mark.asyncio
async def test_failed_assert_for_retired_line_never_latches_halt(tmp_path):
_write_plan(tmp_path, PLAN_WITH_CHECKS.replace(ASSERT_LINE, ""))
ctx = _make_ctx(tmp_path, assert_failure_policy="halt")
ctx.guidance_retired_checks = {RETIRED_ASSERT}
run = _done_run(ctx, CheckReport(verdicts=[_verdict("assert", "failed")]))
harvest_run(ctx, _make_state(), run, allow_side_effects=True)
assert ctx.assert_halt is False
assert read_ledger(tmp_path)[0]["status"] == "failed" # still booked
assert _last_done_event(ctx)["applicable"] is False
@pytest.mark.asyncio
async def test_non_retired_sibling_verdict_still_takes_effect(tmp_path):
"""Same report: the retired verify is dropped, the live assert still halts
and the live verify sibling still repairs."""
plan_path = _write_plan(tmp_path, PLAN_WITH_CHECKS.replace(VERIFY_LINE, ""))
ctx = _make_ctx(tmp_path, assert_failure_policy="halt")
ctx.guidance_retired_checks = {RETIRED_VERIFY}
run = _done_run(
ctx,
CheckReport(verdicts=[_verdict("verify", "failed"), _verdict("assert", "failed")]),
)
findings = harvest_run(ctx, _make_state(), run, allow_side_effects=True)
assert findings == [] # the retired verify produces no repair finding
assert f"- [x] {GOAL_TEXT}" in plan_path.read_text(encoding="utf-8")
assert ctx.assert_halt is True # the live assert sibling still counts
event = _last_done_event(ctx)
assert event["applicable"] is True
assert event["retired"] == [{"kind": "verify", "item_text": RETIRED_VERIFY[1]}]
# And a live verify sibling next to a retired one still drives the repair,
# with the headline naming only the live criterion.
live_line = " - verify: the alarm is enabled\n"
plan_path = _write_plan(tmp_path, PLAN_WITH_CHECKS.replace(VERIFY_LINE, live_line))
ctx = _make_ctx(tmp_path)
ctx.guidance_retired_checks = {RETIRED_VERIFY}
run = _done_run(
ctx,
CheckReport(
verdicts=[
_verdict("verify", "failed"),
_verdict("verify", "failed", text="the alarm is enabled", evidence="toggle off"),
]
),
)
findings = harvest_run(ctx, _make_state(), run, allow_side_effects=True)
content = plan_path.read_text(encoding="utf-8")
assert f"- [/] {GOAL_TEXT}" in content
assert "- finding: verify failed — 'the alarm is enabled'" in content
assert RETIRED_VERIFY[1] not in " ".join(findings)
assert any("the alarm is enabled" in f for f in findings)
assert ctx.checker_finding_items[GOAL_KEY] == {("verify", "the alarm is enabled")}
@pytest.mark.asyncio
async def test_retiring_an_item_withdraws_its_registered_headline(tmp_path):
"""A headline registered by an earlier failing attempt disappears from the
plan when the user guidance retires the item that produced it; a headline
that still names a live item stays."""
from artemis.graph.checkpoints import retire_check_items
plan_path = _write_plan(tmp_path)
ctx = _make_ctx(tmp_path)
run = _done_run(ctx, CheckReport(verdicts=[_verdict("verify", "failed")]))
harvest_run(ctx, _make_state(), run, allow_side_effects=True)
assert "- finding: verify failed" in plan_path.read_text(encoding="utf-8")
retire_check_items(ctx, [RETIRED_ASSERT]) # unrelated item: headline stays
assert "- finding: verify failed" in plan_path.read_text(encoding="utf-8")
assert ctx.guidance_retired_checks == {RETIRED_ASSERT}
retire_check_items(ctx, [RETIRED_VERIFY])
assert "- finding:" not in plan_path.read_text(encoding="utf-8")
assert ctx.checker_findings == {}
assert ctx.checker_finding_items == {}
assert ctx.guidance_retired_checks == {RETIRED_ASSERT, RETIRED_VERIFY}
# The plan-write path reaches the same helper: dropping a waived line
# after a headline was registered removes the headline on that write.
plan_path = _write_plan(tmp_path)
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
ctx.guidance_unprotected_checks = set()
ctx.guidance_retired_checks = set()
run = _done_run(ctx, CheckReport(verdicts=[_verdict("verify", "failed")]))
harvest_run(ctx, _make_state(), run, allow_side_effects=True)
with_finding = plan_path.read_text(encoding="utf-8")
assert "- finding: verify failed" in with_finding
_guidance_arrives(ctx, with_finding)
without_verify = with_finding.replace(VERIFY_LINE, "")
on_disk = await _plan_write(tmp_path, ctx, with_finding, without_verify)
assert VERIFY_LINE not in on_disk
assert "- finding:" not in on_disk
assert ctx.guidance_retired_checks == {RETIRED_VERIFY}
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_ledger_is_append_only_fail_then_pass_both_present(tmp_path): async def test_ledger_is_append_only_fail_then_pass_both_present(tmp_path):
_write_plan(tmp_path) _write_plan(tmp_path)
@@ -0,0 +1,95 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Perception side of the guidance waiver: a non-empty user instruction records
the plan's current check lines in ``ctx.guidance_unprotected_checks`` on the run
context (which outlives the per-turn State rebuild); stop-only or empty
payloads do not, and an ordinary turn never touches the set."""
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from artemis.graph.perception import perception_node
from artemis.graph.state import State
async def _run_perception(ctx, payload):
with (
patch("artemis.graph.perception.is_ocr_configured", return_value=False),
patch("artemis.graph.perception.UnifiedMobileController") as controller_cls,
patch("artemis.graph.perception._check_injected_instruction_file", return_value=payload),
):
device_data = MagicMock()
device_data.width = 1080
device_data.height = 2400
device_data.base64 = "ZHVtbXk="
device_data.elements = []
controller = MagicMock()
controller.get_screen_data = AsyncMock(return_value=device_data)
controller_cls.return_value = controller
state = MagicMock(spec=State)
state.structured_decisions = []
return await perception_node(state, ctx)
PLAN = "- [/] Set the alarm\n - verify: the alarm list shows 7:30 AM\n- [ ] Next milestone\n"
def _ctx(tmp_path):
from artemis.utils.notes import get_note_file_path
ctx = MagicMock()
ctx.data_engine = MagicMock()
ctx.data_engine.base_dir = tmp_path
ctx.data_engine.current_step_id = "step_1"
ctx.device = MagicMock()
ctx.guidance_unprotected_checks = set()
plan_path = get_note_file_path(tmp_path, "task_plan")
plan_path.parent.mkdir(parents=True, exist_ok=True)
plan_path.write_text(PLAN, encoding="utf-8")
return ctx
@pytest.mark.asyncio
async def test_instruction_waives_the_current_check_lines_and_still_reaches_state(tmp_path):
ctx = _ctx(tmp_path)
update = await _run_perception(ctx, {"instruction": "Skip the toast check"})
assert ctx.guidance_unprotected_checks == {("verify", "the alarm list shows 7:30 AM")}
assert update["injected_instruction"] == "Skip the toast check"
assert update["user_stop_requested"] is False
@pytest.mark.asyncio
async def test_stop_only_or_blank_payload_does_not_arm(tmp_path):
ctx = _ctx(tmp_path)
update = await _run_perception(ctx, {"release_loop": True})
assert update["user_stop_requested"] is True
assert ctx.guidance_unprotected_checks == set()
update = await _run_perception(ctx, {"instruction": " ", "release_loop": True})
assert update["user_stop_requested"] is True
assert ctx.guidance_unprotected_checks == set()
@pytest.mark.asyncio
async def test_ordinary_turn_leaves_the_waiver_as_it_was(tmp_path):
"""The per-turn reset applies to State (``injected_instruction`` goes back
to None); the waiver on the context lives for the rest of the run."""
ctx = _ctx(tmp_path)
ctx.guidance_unprotected_checks = {("verify", "earlier line")}
update = await _run_perception(ctx, None)
assert update["injected_instruction"] is None
assert ctx.guidance_unprotected_checks == {("verify", "earlier line")}
@@ -77,6 +77,11 @@ async def test_ask_explorer_runs_the_pipeline_for_the_executor_agent():
assert result.status == "success" assert result.status == "success"
assert "Explorer located 1 candidate(s) for 'blue Send button'" in result.text_summary assert "Explorer located 1 candidate(s) for 'blue Send button'" in result.text_summary
assert "[1] 'Send' at normalized [500, 500]" in result.text_summary assert "[1] 'Send' at normalized [500, 500]" in result.text_summary
# This executor's click resolves an index against the element list the
# candidates just joined, so the guidance teaches the index syntax (Pro
# parity) next to the coordinates.
assert "target=3, a bare integer" in result.text_summary
assert "normalized coordinate" in result.text_summary
# Candidates were registered on the state for index-based actions. # Candidates were registered on the state for index-based actions.
assert state.indexed_points == [[540, 1200]] assert state.indexed_points == [[540, 1200]]
assert state.indexed_elements[0]["index"] == 1 assert state.indexed_elements[0]["index"] == 1
@@ -12,12 +12,14 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
"""Coordinate targets must be described by the model, never inferred. """Targets are either observed (an element index) or described (coordinates).
The Flash/Validator dialect addresses every target by coordinates, so the Flash binds the Pro Operator's agent dialect: a click, long press or focused
executor refuses a click, long press, focused input, coordinate swipe or click input names an element index from the indexed UI list -- resolved here to the
sequence that does not say what it is aiming at. The description is recorded element's center with its observed text/bounds/id recorded -- or a coordinate
by the runner and never reaches the wire. pair, which the model must describe itself. A coordinate swipe or click
sequence must be described too. Nothing is inferred, the description never
reaches the wire, and the wire only ever sees coordinates.
""" """
from unittest.mock import Mock from unittest.mock import Mock
@@ -44,6 +46,123 @@ def _state():
return state return state
def _indexed_state():
"""Two observed elements (pixel centers, the shape ``observe()`` writes)."""
state = _state()
state.indexed_elements = [
{
"index": 1,
"center": [540, 1248],
"text": "Wi-Fi",
"bounds": [40, 1152, 1040, 1344],
"class": "android.widget.TextView",
"resource_id": "android:id/title",
"is_ocr": False,
},
{
"index": 2,
"center": [540, 720],
"text": "Search settings",
"bounds": [80, 660, 1000, 780],
"class": "android.widget.EditText",
"resource_id": "com.android.settings:id/search",
"is_ocr": False,
},
]
state.indexed_points = [el["center"] for el in state.indexed_elements]
return state
# --- Index targets: resolved against the element list, observed semantics recorded ---
@pytest.mark.parametrize("index", [1, "1", 1.0, [1]])
def test_click_index_resolves_to_the_elements_center_and_records_observed_fields(index):
executor = _make_executor()
wire_name, wire_args, _, recorded = executor._translate(
"click", {"target": index}, _indexed_state()
)
assert wire_name == "click"
# 540/1080 -> 500, 1248/2400 -> 520 (normalized 0-1000)
assert wire_args == {"target": [500, 520], "times": 1, "delay_ms": 100}
assert recorded == {
"target_text": "Wi-Fi",
"target_bounds": [37, 480, 963, 560],
"target_resource_id": "android:id/title",
"target_class": "android.widget.TextView",
}
assert "target_description" not in recorded
def test_index_target_needs_no_description_and_ignores_a_stray_one():
"""An index already names its element; the model's belief never mixes with
the observation on the record."""
executor = _make_executor()
_, wire_args, _, recorded = executor._translate(
"click", {"target": 2, "target_description": "the search box"}, _indexed_state()
)
assert wire_args["target"] == [500, 300]
assert recorded["target_text"] == "Search settings"
assert "target_description" not in recorded
def test_long_press_and_input_text_resolve_indices_too():
executor = _make_executor()
_, press_args, _, recorded = executor._translate(
"long_press", {"target": 1, "duration": 1500}, _indexed_state()
)
assert press_args == {"target": [500, 520], "duration_ms": 1500}
assert recorded["target_text"] == "Wi-Fi"
_, type_args, _, recorded = executor._translate(
"input_text", {"text": "wifi", "target": 2}, _indexed_state()
)
assert type_args == {"text": "wifi", "target": [500, 300], "clear_exist": True}
assert recorded["target_resource_id"] == "com.android.settings:id/search"
@pytest.mark.parametrize("name, label", [("click", "click"), ("long_press", "long press")])
def test_out_of_range_index_is_refused_with_the_active_range(name, label):
executor = _make_executor()
with pytest.raises(_ArgError) as excinfo:
executor._translate(name, {"target": 7}, _indexed_state())
message = str(excinfo.value)
assert message.startswith(f"Error during {label}: Invalid target index 7.")
assert "1 to 2" in message
assert "ask_explorer" in message
def test_index_against_an_empty_list_is_refused():
executor = _make_executor()
with pytest.raises(_ArgError) as excinfo:
executor._translate("click", {"target": 1}, _state())
message = str(excinfo.value)
assert "Invalid target index 1" in message
assert "empty" in message
def test_unusable_target_is_refused_with_both_syntaxes_named():
executor = _make_executor()
with pytest.raises(_ArgError) as excinfo:
executor._translate("click", {"target": "the button"}, _indexed_state())
message = str(excinfo.value)
assert message.startswith("Error during click: Invalid target format")
assert "element index" in message and "[x, y]" in message
def test_index_resolution_uses_the_live_device_resolution():
"""The element list is built against the device size ``observe()`` wrote
on the context; a different resolution maps the same pixels differently."""
executor = _make_executor(width=720, height=1600)
state = _indexed_state()
state.indexed_elements[0]["center"] = [360, 800]
_, wire_args, _, _ = executor._translate("click", {"target": 1}, state)
assert wire_args["target"] == [500, 500]
# --- Coordinate targets: the model describes what it aims at -------------------------
@pytest.mark.parametrize( @pytest.mark.parametrize(
"name, args, label", "name, args, label",
[ [
+59 -36
View File
@@ -17,12 +17,11 @@
Two kinds of protection: Two kinds of protection:
* **Fixture pins** freeze the exact schema each model surface receives (operator * **Fixture pins** freeze the exact schema each model surface receives (operator
shells, Validator/Flash declarations, the action server manifest). A schema change shells, Flash declarations, the action server manifest). A schema change must be a
must be a conscious act: regenerate the fixture in the same commit and say why. conscious act: regenerate the fixture in the same commit and say why.
* **Bridge derivation** re-derives every cross-dialect parameter difference from the * **Projection identity** checks that the Flash declaration of every shared action
generated schemas and requires it to be declared in ``ActionSpec.param_bridge``. is the operator shell's own description and parameters, so the two profiles can
Undeclared divergence between what one agent sees and what another sees -- the never see different tools under one name -- the historical definition of drift.
historical definition of drift -- fails here instead of shipping.
""" """
import json import json
@@ -130,47 +129,71 @@ def test_validator_declaration_order_is_stable():
] ]
# --- Bridge derivation: dialect differences must be declared ------------------------- # --- Projection identity: Flash binds the operator shell, projected ------------------
def _operator_param_names(name: str) -> list[str]: def _without_optional_noise(schema: dict) -> dict:
schema = convert_to_openai_tool(operator_shell_tool(name)) """The pydantic-generated property schema minus what only marks optionality.
return list(schema["function"]["parameters"].get("properties", {}))
``default`` and the ``null`` alternative say "optional"; the declaration says
that through ``required`` alone.
"""
out = {k: v for k, v in schema.items() if k not in ("anyOf", "default")}
if "anyOf" in schema:
members = [m for m in schema["anyOf"] if m.get("type") != "null"]
out.update(members[0] if len(members) == 1 else {"anyOf": members})
return out
def _declaration_param_names(name: str) -> list[str]: def test_flash_declarations_are_the_operator_shells_projected():
return list(tool_declaration(name).parameters.get("properties", {})) """One definition, two bindings: for every action with an operator shell, the
Flash ``ToolDeclaration`` carries the shell's description, parameter names,
descriptions, types (index-or-coordinates unions included) and required set."""
def test_param_bridge_is_exact():
"""Every operator<->declaration parameter difference is declared, none invented."""
for spec in ACTION_SPECS.values(): for spec in ACTION_SPECS.values():
if spec.operator is None or spec.declaration is None: if spec.operator is None:
assert spec.param_bridge == {}, (
f"'{spec.name}' declares a param_bridge without both dialects."
)
continue continue
operator_names = _operator_param_names(spec.name) shell = convert_to_openai_tool(operator_shell_tool(spec.name))["function"]
declaration_names = _declaration_param_names(spec.name) declaration = tool_declaration(spec.name)
assert set(spec.param_bridge) == set(operator_names), ( assert declaration.description == shell["description"], spec.name
f"'{spec.name}': param_bridge keys must cover exactly the operator" shell_props = shell["parameters"]["properties"]
f" params. bridge={sorted(spec.param_bridge)} operator={operator_names}" decl_props = declaration.parameters["properties"]
assert list(decl_props) == list(shell_props), spec.name
for param, schema in shell_props.items():
assert decl_props[param] == _without_optional_noise(schema), (
f"'{spec.name}.{param}' diverges between the operator shell and the"
" Flash declaration."
) )
assert set(spec.param_bridge.values()) == set(declaration_names), ( assert declaration.parameters["required"] == shell["parameters"].get("required", []), (
f"'{spec.name}': param_bridge values must cover exactly the declaration" spec.name
f" params. bridge={sorted(spec.param_bridge.values())}"
f" declaration={declaration_names}"
) )
def test_renamed_params_are_documented(): def test_point_targets_accept_an_index_or_coordinates_in_both_profiles():
"""A cross-dialect rename is a deliberate difference; it must carry prose.""" """click / long_press / input_text address an element by index or by
normalized coordinates; the description is required only for coordinates."""
for name in ("click", "long_press", "input_text"):
target = tool_declaration(name).parameters["properties"]["target"]
assert target["anyOf"] == [
{"type": "integer"},
{"type": "array", "items": {"type": "integer"}},
], name
assert "target_description" not in tool_declaration(name).parameters["required"], name
assert tool_declaration("click_sequence").parameters["required"] == [
"sequence",
"target_descriptions",
]
def test_shared_descriptions_are_identical_between_profiles():
for name in ("swipe", "press_key", "manage_app", "wait_for_delay"):
assert tool_declaration(name).description == operator_shell_tool(name).description
def test_documented_differences_concern_the_wire_only():
"""The agent dialect is single-sourced; a spec may only document how the
agent dialect differs from the wire (spelling, vocabulary, resolution)."""
for spec in ACTION_SPECS.values(): for spec in ACTION_SPECS.values():
renames = {k: v for k, v in spec.param_bridge.items() if k != v} assert "declaration dialect" not in spec.differences, spec.name
if renames:
assert spec.differences, (
f"'{spec.name}' renames {renames} across dialects but documents no differences."
)
# --- Manifest coverage --------------------------------------------------------------- # --- Manifest coverage ---------------------------------------------------------------
Loaded 100 of 127 files, more files were not shown because too many files have changed in this diff. Show more