refactor: unify agent history and harden runtime diagnostics
Preserve user guidance through history compression, stabilize Flash index targets, and retire withdrawn checks without applying stale verdicts. Improve runtime diagnostics, tool responses, and history rendering. Simplify documentation and comments. Validation: 2076 tests passed in the full suite; 381 affected tests and 26 frontend tests passed after cleanup. Ruff and the quality ratchet passed.
This commit is contained in:
1 parent
e983180b5b
commit
5d92939d91
127 files changed
+13674
-1924
No files matched your search
@@ -33,17 +33,17 @@
|
|||||||
|
|
||||||
## Key Highlights
|
## Key Highlights
|
||||||
|
|
||||||
* **Cross-App Automation & Autonomous AI Assistant**: Operates not just as a robust testing framework, but as an autonomous agent capable of handling complex cross-app workflows and daily tasks via natural language;
|
* **Cross-App Automation**: Executes testing workflows and everyday tasks on Android from natural language instructions.
|
||||||
* **Zero-Maintenance Test Automation**: Built upon a "Dynamic-First, Coordinate-Fallback" multimodal locating engine, eliminating fragile XPath/ID selector maintenance and remaining resilient to UI redesigns, system updates, and resolution drift;
|
* **Multimodal Targeting**: Uses element indices when available, with coordinate and visual locating fallbacks for custom interfaces.
|
||||||
* **One-Click Bug Repro & Logcat Diagnostics in IDE**: Native **Model Context Protocol (MCP)** integration allows **Antigravity, Claude Code, and Windsurf** to drive physical test devices via natural language, automatically capturing crash stacks from **Logcat** and keyframe screenshots;
|
* **IDE Diagnostics**: **Model Context Protocol (MCP)** integration lets **Antigravity, Claude Code, and Windsurf** drive test devices and collect **Logcat** output and screenshots.
|
||||||
* **Ultra-Fast Execution (3–5s per Step)**: Pioneered an **Optimistic Asynchronous Pipeline** that completely decouples UI interaction from heavy LLM reasoning, achieving rapid regression throughput in Flash mode;
|
* **Flash Execution**: A reactive observe-and-act loop with asynchronous history summaries, typically **3–5s per step**.
|
||||||
* **Popup Self-Healing & 10+ Hour Exploration**: Proprietary **Safety Net** double-checks targets before action execution to intercept and clear interfering system popups; Pro mode supports **10+ hours** of continuous exploratory & monkey-plus stability testing;
|
* **Pro Exploration**: Checks targets before individual actions and returns blocked actions to the Operator for recovery. Supports long-running exploratory and stability tests.
|
||||||
* **Industry-Leading SOTA**: Achieved **99%+ task completion** on Google Research's **AndroidWorld** benchmark (100+ complex multi-step tasks).
|
* **AndroidWorld Results**: **99%+ task completion** on Google Research's **AndroidWorld** benchmark (100+ multi-step tasks).
|
||||||
|
|
||||||
<a id="workflow-showcase"></a>
|
<a id="workflow-showcase"></a>
|
||||||
## Antigravity × ARTEMIS: Autonomous Testing Workflow
|
## Antigravity × ARTEMIS: Autonomous Testing Workflow
|
||||||
|
|
||||||
Experience seamless collaboration between **Antigravity** and **ARTEMIS** via native MCP integration — taking you from a natural language requirement to a production-grade diagnostic report in four automated steps:
|
**Antigravity** uses **ARTEMIS** through MCP to turn a test request into a plan, device execution, and a diagnostic report:
|
||||||
|
|
||||||
<table width="100%">
|
<table width="100%">
|
||||||
<tr>
|
<tr>
|
||||||
@@ -65,7 +65,7 @@ Experience seamless collaboration between **Antigravity** and **ARTEMIS** via na
|
|||||||
<img src="./docs/assets/workflow-3-exec.png" width="100%" alt="Step 3: Autonomous Test Execution" />
|
<img src="./docs/assets/workflow-3-exec.png" width="100%" alt="Step 3: Autonomous Test Execution" />
|
||||||
</td>
|
</td>
|
||||||
<td width="50%" align="center">
|
<td width="50%" align="center">
|
||||||
<b>4. Comprehensive Final Report</b><br>
|
<b>4. Final Report</b><br>
|
||||||
<sub>Delivers structured audit findings, metric tables, and raw datasets</sub><br><br>
|
<sub>Delivers structured audit findings, metric tables, and raw datasets</sub><br><br>
|
||||||
<img src="./docs/assets/workflow-4-report.png" width="100%" alt="Step 4: Final Report" />
|
<img src="./docs/assets/workflow-4-report.png" width="100%" alt="Step 4: Final Report" />
|
||||||
</td>
|
</td>
|
||||||
@@ -159,7 +159,8 @@ PYTHONPATH = "/path/to/artemis"
|
|||||||
"mobile_run_task": { "eager": true },
|
"mobile_run_task": { "eager": true },
|
||||||
"mobile_manage_task": { "eager": true },
|
"mobile_manage_task": { "eager": true },
|
||||||
"mobile_get_device_state": { "eager": true },
|
"mobile_get_device_state": { "eager": true },
|
||||||
"mobile_inspect_trace": { "eager": true }
|
"mobile_inspect_trace": { "eager": true },
|
||||||
|
"mobile_diagnose": { "eager": true }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -246,14 +247,14 @@ if __name__ == "__main__":
|
|||||||
</p>
|
</p>
|
||||||
|
|
||||||
* **Web Visual Test Console (`uv run artemis ui`)**: Real-time screen projection and interactive panel, supporting natural language test dispatch, live reasoning telemetry, action trajectories, and execution replay; manage server lifecycle anytime from any terminal using `uv run artemis restart`, `uv run artemis stop`, and `uv run artemis status`;
|
* **Web Visual Test Console (`uv run artemis ui`)**: Real-time screen projection and interactive panel, supporting natural language test dispatch, live reasoning telemetry, action trajectories, and execution replay; manage server lifecycle anytime from any terminal using `uv run artemis restart`, `uv run artemis stop`, and `uv run artemis status`;
|
||||||
* **Native MCP Protocol (IDE Collaboration)**: Operates as a standard MCP server seamlessly integrating with **Antigravity, Claude Code, Windsurf**, etc., directly driving real devices inside the IDE to verify bugs and run test cases;
|
* **MCP Server**: Connects **Antigravity, Claude Code, Windsurf**, and other MCP clients to real devices for bug reproduction and test execution;
|
||||||
* **Developer CLI (`uv run artemis run`)**: Direct terminal execution for automated test cases, exploratory stability inspection, or AndroidWorld benchmarks with high-fidelity structured terminal output;
|
* **Developer CLI (`uv run artemis run`)**: Direct terminal execution for automated test cases, exploratory stability inspection, or AndroidWorld benchmarks with high-fidelity structured terminal output;
|
||||||
* **Python SDK**: Integrates as a standard Python library into existing automated testing frameworks (e.g., pytest) or CI/CD pipelines with strongly typed Pydantic structured outputs and assertion support.
|
* **Python SDK**: Integrates as a standard Python library into existing automated testing frameworks (e.g., pytest) or CI/CD pipelines with strongly typed Pydantic structured outputs and assertion support.
|
||||||
|
|
||||||
<a id="benchmarks"></a>
|
<a id="benchmarks"></a>
|
||||||
## Benchmarks: AndroidWorld (SOTA 99%+)
|
## Benchmarks: AndroidWorld (SOTA 99%+)
|
||||||
|
|
||||||
Evaluated on [AndroidWorld](https://github.com/google-research/android_world) — Google Research's gold-standard benchmark spanning 20+ real apps and 100+ complex multi-step tasks: **Artemis demonstrated exceptional robustness across the entire benchmark suite, achieving a 99%+ completion rate.**
|
Artemis achieved a **99%+ completion rate** on [AndroidWorld](https://github.com/google-research/android_world), Google Research's benchmark spanning 20+ apps and 100+ multi-step tasks.
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="./docs/assets/androidworld_leaderboard.png?v=2" alt="AndroidWorld Benchmark Comparison" width="100%" />
|
<img src="./docs/assets/androidworld_leaderboard.png?v=2" alt="AndroidWorld Benchmark Comparison" width="100%" />
|
||||||
@@ -261,9 +262,9 @@ Evaluated on [AndroidWorld](https://github.com/google-research/android_world)
|
|||||||
|
|
||||||
## How ARTEMIS is Architected
|
## How ARTEMIS is Architected
|
||||||
|
|
||||||
* **Pre-Touch Pixel Gate & Speculative Chaining**: Eliminates "silent misclicks" from inference latency race conditions. Milliseconds before dispatch, a local UI guard intercepts unexpected dialogs (0 tokens, 0 cloud wait), a Micro-ROI gate verifies target stability, and speculative chained taps hit transient UI (e.g. auto-fading video controls) before they expire;
|
* **Pre-Execution Checks and Action Bursts**: Pro checks the target against the live UI tree and pixels before dispatching an individual action. Action bursts handle transient controls without waiting for another model turn.
|
||||||
* **Three-Layer Progressive Grounding Engine**: Fuses local OCR with accessibility hierarchies (~150ms, 0 tokens) to drive 85%+ of standard actions via drift-free numeric indices, gracefully falling back to spatial vision models for custom Canvas/Compose/Flutter UI and sandboxed CV probing for subtle pixel states;
|
* **Element Locating**: Combines accessibility hierarchies and OCR with visual models for custom Canvas, Compose, and Flutter interfaces.
|
||||||
* **Elastic Dual Engine with In-Flight Context Compactor**: Seamlessly toggles between high-throughput reactive CI loops (Flash Mode, 3–5s/step) and multi-step cognitive state graphs (Pro Mode), using background visual deltas and DOM pruning to slash token consumption by >70% for 10+ hours of continuous, unattended soak testing.
|
* **Shared History Compression**: Flash and Pro replace older screenshots with visual summaries and compress completed steps into searchable history chunks. Context thresholds control when raw turns are replaced.
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="./docs/assets/artemis_architecture_diagram.png" alt="ARTEMIS System Architecture Diagram" width="100%" />
|
<img src="./docs/assets/artemis_architecture_diagram.png" alt="ARTEMIS System Architecture Diagram" width="100%" />
|
||||||
@@ -274,7 +275,7 @@ Evaluated on [AndroidWorld](https://github.com/google-research/android_world)
|
|||||||
ARTEMIS supports two execution profiles tailored for different automation requirements:
|
ARTEMIS supports two execution profiles tailored for different automation requirements:
|
||||||
|
|
||||||
* **Flash Profile (`--profile flash`)**: Fast and token-efficient reactive loop (~3–5s per step): one model observes the live screen, thinks, and acts, with no graph orchestration. Ideal for routine, deterministic UI tasks. The loop is unbounded by default (`agent.flash.max_turns`, 0 = unlimited) because history is compressed rather than capped: Flash shares the Pro session transcript ledger (session-relative `T+mm:ss` clock, screenshots folded into visual summaries, older steps chunked into eras and recallable on demand via `search_history` / `replay_steps`) and can query the session recording through `video_analyzer`. Transient UI (auto-fading control bars, toasts) is handled by chaining taps into one `click_sequence`. *Limitations*: No task plan or notes, no pre-execution safety net, no checkpoint verification or final report, and no ADB shell.
|
* **Flash Profile (`--profile flash`)**: Fast and token-efficient reactive loop (~3–5s per step): one model observes the live screen, thinks, and acts, with no graph orchestration. Ideal for routine, deterministic UI tasks. The loop is unbounded by default (`agent.flash.max_turns`, 0 = unlimited) because history is compressed rather than capped: Flash shares the Pro session transcript ledger (session-relative `T+mm:ss` clock, screenshots folded into visual summaries, older steps chunked into eras and recallable on demand via `search_history` / `replay_steps`) and can query the session recording through `video_analyzer`. Transient UI (auto-fading control bars, toasts) is handled by chaining taps into one `click_sequence`. *Limitations*: No task plan or notes, no pre-execution safety net, no checkpoint verification or final report, and no ADB shell.
|
||||||
* **Pro Profile (`--profile pro`)**: Our most capable architecture (~15–40s per step), built as a multi-agent graph. A **Planner** maintains a living Markdown task plan with milestones and `verify` / `assert` check items; the **Operator** executes it with the full toolset (Explorer grounding whose `flash` / `pro` / `ultra` tier is a user setting per profile — `pro.explorer.mode` / `flash.explorer_mode` in `config/artemis.jsonc` or `--explorer-pro-mode` — never chosen by the agent; notes, history recall, video analysis, ADB diagnostics). Every single action passes a pre-execution **Safety Net** (XML-first, pixel fallback), while multi-action **fast-action bursts** fire back to back to beat turn latency on transient UI. A blocked or failed action opens an **execution incident** that stays in the Operator's context until a later action succeeds, so recovery is handled by the Operator itself with no separate repair agent. A read-only **Checker** verifies plan checkpoints and runs an exit final review against the original goal (`--verification-level`: `off` / `final` (default) / `checkpoints` / `strict`), and plan milestone edits get an advisory review. Handles 100+ step long-horizon workflows, `[Loop:continuous]` monitoring, and an optional written report.
|
* **Pro Profile (`--profile pro`)**: A planning and verification workflow (~15–40s per step), built as a multi-agent graph. A **Planner** maintains a living Markdown task plan with milestones and `verify` / `assert` check items; the **Operator** executes it with the full toolset (Explorer grounding whose `flash` / `pro` / `ultra` tier is a user setting per profile — `pro.explorer.mode` / `flash.explorer_mode` in `config/artemis.jsonc` or `--explorer-pro-mode` — never chosen by the agent; notes, history recall, video analysis, ADB diagnostics). Every single action passes a pre-execution **Safety Net** (XML-first, pixel fallback), while multi-action **fast-action bursts** fire back to back to beat turn latency on transient UI. A blocked or failed action opens an **execution incident** that stays in the Operator's context until a later action succeeds, so recovery is handled by the Operator itself with no separate repair agent. A read-only **Checker** verifies plan checkpoints and runs an exit final review against the original goal (`--verification-level`: `off` / `final` (default) / `checkpoints` / `strict`), and plan milestone edits get an advisory review. Handles 100+ step long-horizon workflows, `[Loop:continuous]` monitoring, and an optional written report.
|
||||||
|
|
||||||
## Roadmap
|
## Roadmap
|
||||||
|
|
||||||
|
|||||||
+17
-16
@@ -33,18 +33,18 @@
|
|||||||
|
|
||||||
## 核心亮点
|
## 核心亮点
|
||||||
|
|
||||||
* **跨 App 复杂自动化与 AI 个人助理**:不仅能执行严格的自动化测试用例,更具备接管真机的自主决策与长流程操作能力,一句话处理跨应用复杂业务与任务;
|
* **跨 App 自动化**:根据自然语言指令,在 Android 上执行测试流程和日常任务。
|
||||||
* **零脚本维护自动化测试**:基于“动态优先、坐标兜底”的多模态语义定位,彻底告别传统 XPath / 控件 ID 频繁失效的痛点,无惧 App 改版、分辨率差异与系统升级;
|
* **多模态定位**:优先使用元素索引,对自绘界面提供坐标和视觉定位兜底。
|
||||||
* **IDE 内一键 Bug 复现与 Logcat 诊断**:原生支持 **MCP 协议**,可在 **Antigravity、Claude Code、Windsurf** 中直接用自然语言驱动真机复现缺陷,自动抓取关键帧截图与 **Logcat 崩溃堆栈**,完成研发测试闭环;
|
* **IDE 内诊断**:通过 **MCP 协议**,让 **Antigravity、Claude Code、Windsurf** 操作测试设备,收集 **Logcat** 输出和截图。
|
||||||
* **极速执行吞吐(单步 3–5 秒)**:首创**全链路乐观异步流水线 (Optimistic Async Pipeline)**,解耦重度推理与页面操作,在 Flash 模式下实现丝滑的高频交互与快速回归验证;
|
* **Flash 执行**:使用观察、执行循环和异步历史摘要,单步通常约 **3–5 秒**。
|
||||||
* **干扰弹窗自愈与超长程巡检**:独创 **Safety Net** 执行前校验机制,自动识别并清除系统权限弹窗、通知遮挡等异常;Pro 模式支持连续 **10+ 小时** 无人值守稳定性巡检与探索性测试 (Exploratory / Monkey-plus Testing);
|
* **Pro 探索**:单个动作执行前校验目标,被拦截的动作交回 Operator 处理。支持长时间探索和稳定性测试。
|
||||||
* **业界顶尖 SOTA**:在 Google Research **AndroidWorld** 基准评测(100+ 复杂长程系统与应用交互任务)中取得 **99%+ 任务完成率**。
|
* **AndroidWorld 结果**:在 Google Research **AndroidWorld** 基准评测(100+ 多步任务)中取得 **99%+ 任务完成率**。
|
||||||
|
|
||||||
<a id="workflow-showcase"></a>
|
<a id="workflow-showcase"></a>
|
||||||
<a id="全流程演示"></a>
|
<a id="全流程演示"></a>
|
||||||
## Antigravity × ARTEMIS:全流程自主测试演示
|
## Antigravity × ARTEMIS:全流程自主测试演示
|
||||||
|
|
||||||
通过原生 MCP 协议,**Antigravity** 与 **ARTEMIS** 深度协同——只需一句自然语言指令,即可自动完成从需求理解、测试规划、真机执行到深度报告输出的完整闭环:
|
**Antigravity** 通过 MCP 调用 **ARTEMIS**,将测试需求转为计划、真机操作和诊断报告:
|
||||||
|
|
||||||
<table width="100%">
|
<table width="100%">
|
||||||
<tr>
|
<tr>
|
||||||
@@ -161,7 +161,8 @@ PYTHONPATH = "/path/to/artemis"
|
|||||||
"mobile_run_task": { "eager": true },
|
"mobile_run_task": { "eager": true },
|
||||||
"mobile_manage_task": { "eager": true },
|
"mobile_manage_task": { "eager": true },
|
||||||
"mobile_get_device_state": { "eager": true },
|
"mobile_get_device_state": { "eager": true },
|
||||||
"mobile_inspect_trace": { "eager": true }
|
"mobile_inspect_trace": { "eager": true },
|
||||||
|
"mobile_diagnose": { "eager": true }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -247,15 +248,15 @@ if __name__ == "__main__":
|
|||||||
</p>
|
</p>
|
||||||
|
|
||||||
* **Web 可视化测试控制台 (`uv run artemis ui`)**:集成设备实时投屏与交互面板,支持通过自然语言下发测试用例,实时观测推理步骤、操作轨迹、截图留存与异常状态回放;支持在任意终端使用 `uv run artemis restart`、`uv run artemis stop`、`uv run artemis status` 一键重启、关停或查看服务状态;
|
* **Web 可视化测试控制台 (`uv run artemis ui`)**:集成设备实时投屏与交互面板,支持通过自然语言下发测试用例,实时观测推理步骤、操作轨迹、截图留存与异常状态回放;支持在任意终端使用 `uv run artemis restart`、`uv run artemis stop`、`uv run artemis status` 一键重启、关停或查看服务状态;
|
||||||
* **原生 MCP 协议 (IDE 协同)**:作为标准 MCP 服务器无缝接入 **Antigravity、Claude Code、Windsurf** 等开发环境,在 IDE 中直接驱动真机完成自动化测试与 Bug 复现验证;
|
* **原生 MCP 协议 (IDE 协同)**:作为标准 MCP 服务器接入 **Antigravity、Claude Code、Windsurf** 等开发环境,在 IDE 中直接驱动真机完成自动化测试与 Bug 复现验证;
|
||||||
* **开发者命令行 CLI (`uv run artemis run`)**:支持通过终端直接执行自动化测试用例、探索性稳定性巡检或 AndroidWorld 基准评测,提供高保真结构化终端输出;
|
* **开发者命令行 CLI (`uv run artemis run`)**:支持通过终端直接执行自动化测试用例、探索性稳定性巡检或 AndroidWorld 基准评测,提供高保真结构化终端输出;
|
||||||
* **Python SDK**:作为标准 Python 库无缝集成至现有自动化测试框架(如 pytest)或 CI/CD 流水线,提供基于 Pydantic 的强类型结构化结果与断言支持。
|
* **Python SDK**:作为标准 Python 库集成至现有自动化测试框架(如 pytest)或 CI/CD 流水线,提供基于 Pydantic 的强类型结构化结果与断言支持。
|
||||||
|
|
||||||
<a id="benchmarks"></a>
|
<a id="benchmarks"></a>
|
||||||
<a id="基准评测"></a>
|
<a id="基准评测"></a>
|
||||||
## 基准评测:AndroidWorld (SOTA 99%+)
|
## 基准评测:AndroidWorld (SOTA 99%+)
|
||||||
|
|
||||||
在 Google Research 发布的业界基准评测 [AndroidWorld](https://github.com/google-research/android_world)(涵盖 20+ 款常用应用与 100+ 项复杂多步长程任务)中:**Artemis 在全套长程任务评测中展现了高鲁棒性,取得了超过 99% 的任务完成率。**
|
Artemis 在 Google Research 发布的 [AndroidWorld](https://github.com/google-research/android_world) 基准评测中取得 **99%+ 任务完成率**。该评测涵盖 20+ 款应用与 100+ 项多步任务。
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="./docs/assets/androidworld_leaderboard.png?v=2" alt="AndroidWorld 评测基准对比" width="100%" />
|
<img src="./docs/assets/androidworld_leaderboard.png?v=2" alt="AndroidWorld 评测基准对比" width="100%" />
|
||||||
@@ -263,9 +264,9 @@ if __name__ == "__main__":
|
|||||||
|
|
||||||
## ARTEMIS 是如何建构的
|
## ARTEMIS 是如何建构的
|
||||||
|
|
||||||
* **Pre-Touch 触控前像素守门与预测性连点**:彻底杜绝大模型推理延迟造成的「静默误触」。动作下发前毫秒级拦截意外弹窗(0 Token、0 云端等待),Micro-ROI 局部校验目标稳定性;针对视频全屏等瞬态控件,结合历史先验毫秒级连击唤醒,在淡出窗口期内瞬时闭环;
|
* **执行前校验与动作序列**:Pro 在下发单个动作前,通过实时 UI 树和像素校验目标。快速动作序列用于处理瞬态控件,无需等待下一轮模型响应。
|
||||||
* **三层递进式目标定位引擎 (Progressive Grounding)**:首层通过本地 OCR 与无障碍树几何融合(~150ms、0 Token),以无偏移数字索引承载 85%+ 常见交互;二层对 Flutter/Compose/Canvas 等自绘组件自动回退至空间视觉模型,三层辅以沙箱 CV 探针核验细微像素状态;
|
* **元素定位**:结合无障碍树、OCR 和视觉模型,支持 Canvas、Compose、Flutter 等自绘界面。
|
||||||
* **弹性双执行引擎与运行时上下文动态压缩**:兼顾 3–5 秒秒级响应的 CI 高吞吐回归(Flash 模式)与深度长程认知状态图探索(Pro 模式),后台异步生成视觉增量并剪除冗余 DOM,长程任务 Token 消耗锐降 70%+,支持 10+ 小时无人值守稳定性压测。
|
* **共享历史压缩**:Flash 和 Pro 将较早的截图替换为视觉摘要,并把已完成步骤压缩为可检索的历史分块。上下文阈值决定何时替换原始记录。
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="./docs/assets/artemis_architecture_diagram.png" alt="ARTEMIS 架构系统示意图" width="100%" />
|
<img src="./docs/assets/artemis_architecture_diagram.png" alt="ARTEMIS 架构系统示意图" width="100%" />
|
||||||
@@ -275,8 +276,8 @@ if __name__ == "__main__":
|
|||||||
|
|
||||||
ARTEMIS 提供两种运行模式以适应不同的自动化需求:
|
ARTEMIS 提供两种运行模式以适应不同的自动化需求:
|
||||||
|
|
||||||
* **Flash 模式 (`--profile flash`)**:轻量极速且省 Token 的响应式循环(单步约 3–5 秒):单个模型观察实时屏幕、思考、执行,不经过图编排,适合常规确定性 UI 操作。默认不限步数(`agent.flash.max_turns`,0 表示不限),因为历史是被压缩而不是被截断:Flash 与 Pro 共用同一套会话记录账本(相对测试时间 `T+mm:ss`、截图折叠为视觉摘要、更早的步骤分块归档并可通过 `search_history` / `replay_steps` 按需召回),并可调用 `video_analyzer` 分析整段会话录屏;对自动消失的控制栏、toast 等瞬态控件,用 `click_sequence` 把多次点击串成一个原子序列。*局限性*:没有任务计划与笔记、没有执行前安全网、没有检查点校验与最终报告、不能执行 ADB 命令。
|
* **Flash 模式 (`--profile flash`)**:使用历史压缩的响应式循环(单步约 3–5 秒):单个模型观察实时屏幕、思考、执行,不经过图编排,适合常规确定性 UI 操作。默认不限步数(`agent.flash.max_turns`,0 表示不限),因为历史是被压缩而不是被截断:Flash 与 Pro 共用同一套会话记录账本(相对测试时间 `T+mm:ss`、截图折叠为视觉摘要、更早的步骤分块归档并可通过 `search_history` / `replay_steps` 按需召回),并可调用 `video_analyzer` 分析整段会话录屏;对自动消失的控制栏、toast 等瞬态控件,用 `click_sequence` 把多次点击串成一个原子序列。*局限性*:没有任务计划与笔记、没有执行前安全网、没有检查点校验与最终报告、不能执行 ADB 命令。
|
||||||
* **Pro 模式 (`--profile pro`)**:最强大的架构(单步约 15–40 秒),由多智能体图编排:**Planner** 维护一份带里程碑和 `verify` / `assert` 检查项的 Markdown 任务计划;**Operator** 按计划执行,拥有全套工具(Explorer 元素定位,其 `flash` / `pro` / `ultra` 三档感知深度是按运行模式设置的用户配置——`config/artemis.jsonc` 里的 `pro.explorer.mode` / `flash.explorer_mode` 或 `--explorer-pro-mode`——智能体自身不会选择档位;笔记、历史召回、视频分析、ADB 诊断)。单个动作执行前都经过 **Safety Net** 校验(XML 优先、像素兜底),而一轮里的多个动作则作为**快速动作序列 (fast-action burst)** 连发,赶在瞬态控件消失前完成操作。动作被拦截或失败时会打开一条**执行事故 (execution incident)**,持续留在 Operator 的上下文里直到后续动作执行成功,恢复由 Operator 自己完成,不再有独立的修复智能体。只读的 **Checker** 校验计划中的检查点,并在退出前对照原始目标做终审(`--verification-level`:`off` / `final`(默认)/ `checkpoints` / `strict`),计划里程碑的改动会得到建议式复核。支持 100+ 步的长程复杂业务流、`[Loop:continuous]` 持续监控以及可选的书面报告。
|
* **Pro 模式 (`--profile pro`)**:包含计划和校验的执行流程(单步约 15–40 秒),由多智能体图编排:**Planner** 维护一份带里程碑和 `verify` / `assert` 检查项的 Markdown 任务计划;**Operator** 按计划执行,拥有全套工具(Explorer 元素定位,其 `flash` / `pro` / `ultra` 三档感知深度是按运行模式设置的用户配置——`config/artemis.jsonc` 里的 `pro.explorer.mode` / `flash.explorer_mode` 或 `--explorer-pro-mode`——智能体自身不会选择档位;笔记、历史召回、视频分析、ADB 诊断)。单个动作执行前都经过 **Safety Net** 校验(XML 优先、像素兜底),而一轮里的多个动作则作为**快速动作序列 (fast-action burst)** 连发,赶在瞬态控件消失前完成操作。动作被拦截或失败时会打开一条**执行事故 (execution incident)**,持续留在 Operator 的上下文里直到后续动作执行成功,恢复由 Operator 自己完成,不再有独立的修复智能体。只读的 **Checker** 校验计划中的检查点,并在退出前对照原始目标做终审(`--verification-level`:`off` / `final`(默认)/ `checkpoints` / `strict`),计划里程碑的改动会得到建议式复核。支持 100+ 步的长程复杂业务流、`[Loop:continuous]` 持续监控以及可选的书面报告。
|
||||||
|
|
||||||
## 路线图
|
## 路线图
|
||||||
|
|
||||||
|
|||||||
@@ -1070,13 +1070,14 @@ limitations under the License.
|
|||||||
<!-- Generic Tool Calls (Text format for note tools, video analyzer, and other sub-agents) -->
|
<!-- Generic Tool Calls (Text format for note tools, video analyzer, and other sub-agents) -->
|
||||||
@if (shouldShowTool(item.data, block.data) && !isDeviceActionTool(item.data)) {
|
@if (shouldShowTool(item.data, block.data) && !isDeviceActionTool(item.data)) {
|
||||||
<div class="step-generic-tool"
|
<div class="step-generic-tool"
|
||||||
[class.is-running]="isItemRunning(item, block)"
|
[class.is-running]="isToolLineRunning(item, block)"
|
||||||
[class.is-note-tool]="isNoteTool(item.data)"
|
[class.is-note-tool]="isNoteTool(item.data)"
|
||||||
[class.is-video-tool]="isVideoTool(item.data)"
|
[class.is-video-tool]="isVideoTool(item.data)"
|
||||||
[class.is-video-attention]="isVideoTool(item.data) && isVideoAnalysisAttention(item.data)"
|
[class.is-video-attention]="isVideoTool(item.data) && isVideoAnalysisAttention(item.data)"
|
||||||
|
[attr.title]="getCompressionPhaseTitle(item.data)"
|
||||||
>
|
>
|
||||||
<div class="tool-call-label">
|
<div class="tool-call-label">
|
||||||
<span class="stream-status-dot" [class.streaming]="isItemRunning(item, block)"></span>
|
<span class="stream-status-dot" [class.streaming]="isToolLineRunning(item, block)"></span>
|
||||||
<div class="tool-call-text">
|
<div class="tool-call-text">
|
||||||
<span>{{ isVideoTool(item.data) ? getVideoAnalysisLabel(item.data) : getToolDisplayLabel(item.data) }}{{ ((isNoteTool(item.data) && getToolKey(item.data)) || isVideoTool(item.data)) ? ':' : '' }}</span>
|
<span>{{ isVideoTool(item.data) ? getVideoAnalysisLabel(item.data) : getToolDisplayLabel(item.data) }}{{ ((isNoteTool(item.data) && getToolKey(item.data)) || isVideoTool(item.data)) ? ':' : '' }}</span>
|
||||||
<!-- Note Tool Pill -->
|
<!-- Note Tool Pill -->
|
||||||
|
|||||||
@@ -172,6 +172,9 @@ import {
|
|||||||
getUniqueGenericTools,
|
getUniqueGenericTools,
|
||||||
getToolKey,
|
getToolKey,
|
||||||
getToolDisplayLabel,
|
getToolDisplayLabel,
|
||||||
|
isCompressionTool,
|
||||||
|
isCompressionWaiting,
|
||||||
|
getCompressionPhaseLabel,
|
||||||
getToolIcon,
|
getToolIcon,
|
||||||
getToolTitle,
|
getToolTitle,
|
||||||
getToolTargetText,
|
getToolTargetText,
|
||||||
@@ -1465,6 +1468,20 @@ export class AgentStreamComponent implements AfterViewInit {
|
|||||||
return getToolDisplayLabel(tool, this.isFirstSaveNoteForKey(tool));
|
return getToolDisplayLabel(tool, this.isFirstSaveNoteForKey(tool));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Short plain-language phase for a compress_history line (hover title). */
|
||||||
|
public getCompressionPhaseTitle(tool: any): string | null {
|
||||||
|
return isCompressionTool(tool) ? getCompressionPhaseLabel(tool) : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A generic tool line pulses only while work is in flight. A compression
|
||||||
|
* whose summary is ready but held back by the context gate is waiting, not
|
||||||
|
* running, even though its trace status is still `running`.
|
||||||
|
*/
|
||||||
|
public isToolLineRunning(item: any, block: any): boolean {
|
||||||
|
return this.isItemRunning(item, block) && !isCompressionWaiting(item?.data);
|
||||||
|
}
|
||||||
|
|
||||||
public isNoteTool(tool: any): boolean {
|
public isNoteTool(tool: any): boolean {
|
||||||
return isNoteTool(tool);
|
return isNoteTool(tool);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -29,7 +29,7 @@
|
|||||||
<!-- Main Content Layout -->
|
<!-- Main Content Layout -->
|
||||||
<main class="home-content">
|
<main class="home-content">
|
||||||
|
|
||||||
<!-- Hero Intent Section (Modern Linear / Claude Style) -->
|
<!-- Task entry section -->
|
||||||
<section class="hero-section">
|
<section class="hero-section">
|
||||||
<h1 class="hero-title">
|
<h1 class="hero-title">
|
||||||
What would you like to do on your phone?
|
What would you like to do on your phone?
|
||||||
@@ -1718,7 +1718,7 @@
|
|||||||
</div>
|
</div>
|
||||||
}
|
}
|
||||||
|
|
||||||
<!-- Unified Intelligent Composer Surface (Apple / Linear / Claude Grade) -->
|
<!-- Task composer -->
|
||||||
<div class="composer-card" [class.is-pro-active]="selectedProfile() === 'pro'">
|
<div class="composer-card" [class.is-pro-active]="selectedProfile() === 'pro'">
|
||||||
|
|
||||||
<!-- Smart Intent Recommendation Strip (if detected) -->
|
<!-- Smart Intent Recommendation Strip (if detected) -->
|
||||||
@@ -1737,7 +1737,7 @@
|
|||||||
</div>
|
</div>
|
||||||
}
|
}
|
||||||
|
|
||||||
<!-- Borderless Seamless Textarea Input Area -->
|
<!-- Borderless task input -->
|
||||||
<div class="composer-input-area">
|
<div class="composer-input-area">
|
||||||
<textarea
|
<textarea
|
||||||
id="task-input-field"
|
id="task-input-field"
|
||||||
@@ -1768,7 +1768,7 @@
|
|||||||
<p class="desc-line">
|
<p class="desc-line">
|
||||||
<span class="mode-name">Pro</span>
|
<span class="mode-name">Pro</span>
|
||||||
<span class="bullet">·</span>
|
<span class="bullet">·</span>
|
||||||
<span>Our most powerful mode. Plans first, checks its own results and recovers when a tap is blocked. Handles <strong>100+ step</strong> workflows and live phone monitoring (<strong>15–40s</strong> / step).</span>
|
<span>Plans first, checks its results and recovers when a tap is blocked. Handles <strong>100+ step</strong> workflows and live phone monitoring (<strong>15–40s</strong> / step).</span>
|
||||||
</p>
|
</p>
|
||||||
}
|
}
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -353,7 +353,7 @@ $text-light: #94a3b8;
|
|||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
gap: 18px;
|
gap: 18px;
|
||||||
|
|
||||||
/* Hero Section (Modern Linear / Claude Style) */
|
/* Hero section */
|
||||||
.hero-section {
|
.hero-section {
|
||||||
text-align: center;
|
text-align: center;
|
||||||
padding: 16px 0 6px;
|
padding: 16px 0 6px;
|
||||||
@@ -3819,7 +3819,7 @@ $text-light: #94a3b8;
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Unified Sculptured Composer (Linear / Claude / ChatGPT Grade) */
|
/* Task composer */
|
||||||
.composer-card {
|
.composer-card {
|
||||||
background: rgba(255, 255, 255, 0.88);
|
background: rgba(255, 255, 255, 0.88);
|
||||||
backdrop-filter: blur(16px) saturate(190%);
|
backdrop-filter: blur(16px) saturate(190%);
|
||||||
|
|||||||
@@ -2,6 +2,9 @@ import {
|
|||||||
cleanErrorMessage,
|
cleanErrorMessage,
|
||||||
extractToolExtraParams,
|
extractToolExtraParams,
|
||||||
getCompressionLabel,
|
getCompressionLabel,
|
||||||
|
getCompressionPhase,
|
||||||
|
getCompressionPhaseLabel,
|
||||||
|
isCompressionWaiting,
|
||||||
getToolDisplayLabel,
|
getToolDisplayLabel,
|
||||||
getToolIcon,
|
getToolIcon,
|
||||||
getToolTargetText,
|
getToolTargetText,
|
||||||
@@ -35,6 +38,22 @@ describe('compress_history timeline line', () => {
|
|||||||
.toBe('Condensing step 5 into a short memory to free up room…');
|
.toBe('Condensing step 5 into a short memory to free up room…');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('says the summary is ready but held while working memory is still small', () => {
|
||||||
|
const held = trace('running', {
|
||||||
|
start_step: 1,
|
||||||
|
end_step: 4,
|
||||||
|
note: 'held',
|
||||||
|
context_tokens: 20000,
|
||||||
|
context_budget: 80000,
|
||||||
|
swap_at_tokens: 28000
|
||||||
|
});
|
||||||
|
expect(getCompressionLabel(held)).toBe(
|
||||||
|
'Short memory for steps 1–4 is ready; keeping the full record until working memory fills up · ≈ 20k of 28k tokens'
|
||||||
|
);
|
||||||
|
expect(getCompressionLabel(trace('running', { start_step: 1, end_step: 4, note: 'held' })))
|
||||||
|
.toBe('Short memory for steps 1–4 is ready; keeping the full record until working memory fills up');
|
||||||
|
});
|
||||||
|
|
||||||
it('reports the size reduction and the working memory once done', () => {
|
it('reports the size reduction and the working memory once done', () => {
|
||||||
const done = trace('success', {
|
const done = trace('success', {
|
||||||
start_step: 12,
|
start_step: 12,
|
||||||
@@ -54,13 +73,53 @@ describe('compress_history timeline line', () => {
|
|||||||
expect(getCompressionLabel(done)).toBe('Steps 1–4 condensed into a short memory · 3k → 1.5k tokens (2× smaller)');
|
expect(getCompressionLabel(done)).toBe('Steps 1–4 condensed into a short memory · 3k → 1.5k tokens (2× smaller)');
|
||||||
});
|
});
|
||||||
|
|
||||||
it('explains a forced snapshot and a failed attempt in plain words', () => {
|
it('explains a forced recap and a failed attempt in plain words', () => {
|
||||||
const forced = trace('success', { start_step: 1, end_step: 4, forced: true, context_tokens: 70000, context_budget: 80000 });
|
const forced = trace('success', { start_step: 1, end_step: 4, forced: true, context_tokens: 70000, context_budget: 80000 });
|
||||||
expect(getCompressionLabel(forced))
|
expect(getCompressionLabel(forced))
|
||||||
.toBe('Steps 1–4 replaced by a brief snapshot (memory was nearly full) · working memory ≈ 70k of 80k tokens');
|
.toBe('Steps 1–4 condensed into a recap to free up memory · working memory ≈ 70k of 80k tokens');
|
||||||
expect(getCompressionLabel(trace('failed', { start_step: 1, end_step: 4 })))
|
expect(getCompressionLabel(trace('failed', { start_step: 1, end_step: 4 })))
|
||||||
.toBe("Couldn't condense steps 1–4 yet; keeping the full record and retrying later");
|
.toBe("Couldn't condense steps 1–4 yet; keeping the full record and retrying later");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe('phase (args.phase from the backend)', () => {
|
||||||
|
it('reads the declared phase and maps it to one short plain-language label', () => {
|
||||||
|
const cases: Array<[string, string, string]> = [
|
||||||
|
['running', 'summarizing', 'Summarizing this stretch'],
|
||||||
|
['running', 'ready', 'Summary ready; kept in reserve until the context fills up'],
|
||||||
|
['success', 'applied', 'Replaced this stretch with its summary'],
|
||||||
|
['failed', 'failed', 'Summary failed; full record kept']
|
||||||
|
];
|
||||||
|
for (const [status, phase, label] of cases) {
|
||||||
|
const tool = trace(status, { start_step: 1, end_step: 4, phase });
|
||||||
|
expect(getCompressionPhase(tool)).toBe(phase as any);
|
||||||
|
expect(getCompressionPhaseLabel(tool)).toBe(label);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it('falls back to status and note on traces that carry no phase', () => {
|
||||||
|
expect(getCompressionPhase(trace('running', { start_step: 1, end_step: 4 }))).toBe('summarizing');
|
||||||
|
expect(getCompressionPhase(trace('running', { start_step: 1, end_step: 4, note: 'retrying' }))).toBe('summarizing');
|
||||||
|
expect(getCompressionPhase(trace('running', { start_step: 1, end_step: 4, note: 'held' }))).toBe('ready');
|
||||||
|
expect(getCompressionPhase(trace('success', { start_step: 1, end_step: 4 }))).toBe('applied');
|
||||||
|
expect(getCompressionPhase(trace('failed', { start_step: 1, end_step: 4 }))).toBe('failed');
|
||||||
|
expect(getCompressionPhase(trace('running', { start_step: 1, end_step: 4, phase: 'bogus' }))).toBe('summarizing');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('drives the timeline line from the phase even without a note', () => {
|
||||||
|
expect(getCompressionLabel(trace('running', { start_step: 1, end_step: 4, phase: 'ready' })))
|
||||||
|
.toBe('Short memory for steps 1–4 is ready; keeping the full record until working memory fills up');
|
||||||
|
expect(getCompressionLabel(trace('running', { start_step: 1, end_step: 4, phase: 'summarizing' })))
|
||||||
|
.toBe('Condensing steps 1–4 into a short memory to free up room…');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('treats a ready-but-held summary as waiting, not running', () => {
|
||||||
|
expect(isCompressionWaiting(trace('running', { start_step: 1, end_step: 4, phase: 'ready' }))).toBeTrue();
|
||||||
|
expect(isCompressionWaiting(trace('running', { start_step: 1, end_step: 4, note: 'held' }))).toBeTrue();
|
||||||
|
expect(isCompressionWaiting(trace('running', { start_step: 1, end_step: 4, phase: 'summarizing' }))).toBeFalse();
|
||||||
|
expect(isCompressionWaiting(trace('success', { start_step: 1, end_step: 4, phase: 'applied' }))).toBeFalse();
|
||||||
|
expect(isCompressionWaiting({ type: 'tool', name: 'save_note', status: 'running', payload: { args: { phase: 'ready' } } })).toBeFalse();
|
||||||
|
});
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe('cleanErrorMessage', () => {
|
describe('cleanErrorMessage', () => {
|
||||||
|
|||||||
@@ -725,21 +725,31 @@ export function getCompressionLabel(tool: any): string {
|
|||||||
: 'earlier steps';
|
: 'earlier steps';
|
||||||
const rangeCapitalized = range.charAt(0).toUpperCase() + range.slice(1);
|
const rangeCapitalized = range.charAt(0).toUpperCase() + range.slice(1);
|
||||||
const status = String(tool?.status || '').toLowerCase();
|
const status = String(tool?.status || '').toLowerCase();
|
||||||
|
const phase = getCompressionPhase(tool);
|
||||||
|
|
||||||
if (status === 'failed') {
|
if (status === 'failed' || phase === 'failed') {
|
||||||
return `Couldn't condense ${range} yet; keeping the full record and retrying later`;
|
return `Couldn't condense ${range} yet; keeping the full record and retrying later`;
|
||||||
}
|
}
|
||||||
if (status !== 'success') {
|
if (status !== 'success') {
|
||||||
return String(args.note || '').toLowerCase() === 'retrying'
|
const note = String(args.note || '').toLowerCase();
|
||||||
? `Retrying the memory summary for ${range}…`
|
if (note === 'retrying') return `Retrying the memory summary for ${range}…`;
|
||||||
: `Condensing ${range} into a short memory to free up room…`;
|
if (note === 'held' || phase === 'ready') {
|
||||||
|
const swapAt = Number(args.swap_at_tokens);
|
||||||
|
const heldContext = Number(args.context_tokens);
|
||||||
|
const heldParts = [`Short memory for ${range} is ready; keeping the full record until working memory fills up`];
|
||||||
|
if (heldContext > 0 && swapAt > 0) {
|
||||||
|
heldParts.push(`≈ ${formatTokenFigure(heldContext)} of ${formatTokenFigure(swapAt)} tokens`);
|
||||||
|
}
|
||||||
|
return heldParts.join(' · ');
|
||||||
|
}
|
||||||
|
return `Condensing ${range} into a short memory to free up room…`;
|
||||||
}
|
}
|
||||||
|
|
||||||
const parts: string[] = [];
|
const parts: string[] = [];
|
||||||
const source = Number(args.source_tokens);
|
const source = Number(args.source_tokens);
|
||||||
const summary = Number(args.summary_tokens);
|
const summary = Number(args.summary_tokens);
|
||||||
if (args.forced) {
|
if (args.forced) {
|
||||||
parts.push(`${rangeCapitalized} replaced by a brief snapshot (memory was nearly full)`);
|
parts.push(`${rangeCapitalized} condensed into a recap to free up memory`);
|
||||||
} else {
|
} else {
|
||||||
parts.push(`${rangeCapitalized} condensed into a short memory`);
|
parts.push(`${rangeCapitalized} condensed into a short memory`);
|
||||||
if (source > 0 && summary > 0) {
|
if (source > 0 && summary > 0) {
|
||||||
@@ -758,6 +768,57 @@ export function getCompressionLabel(tool: any): string {
|
|||||||
return parts.join(' · ');
|
return parts.join(' · ');
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Plain-language phases of one compress_history line. Mirrors
|
||||||
|
* COMPRESSION_PHASES in artemis/memory/chunking.py: the backend writes
|
||||||
|
* `args.phase`; the trace `status` (running/success/failed) is left alone.
|
||||||
|
*/
|
||||||
|
export type CompressionPhase = 'summarizing' | 'ready' | 'applied' | 'failed';
|
||||||
|
|
||||||
|
const COMPRESSION_PHASES: readonly CompressionPhase[] = ['summarizing', 'ready', 'applied', 'failed'];
|
||||||
|
|
||||||
|
export function isCompressionTool(tool: any): boolean {
|
||||||
|
const name = String(tool?.name || '').toLowerCase().replace(/^(_)?exec_/, '');
|
||||||
|
return name === 'compress_history';
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Phase of a compress_history line. Prefers the backend's `args.phase`;
|
||||||
|
* older traces without it fall back to status + note (`held` ⇒ ready).
|
||||||
|
*/
|
||||||
|
export function getCompressionPhase(tool: any): CompressionPhase {
|
||||||
|
const args = getToolArgs(tool) || {};
|
||||||
|
const declared = String(args.phase || '').toLowerCase() as CompressionPhase;
|
||||||
|
if (COMPRESSION_PHASES.includes(declared)) return declared;
|
||||||
|
const status = String(tool?.status || '').toLowerCase();
|
||||||
|
if (status === 'failed') return 'failed';
|
||||||
|
if (status === 'success') return 'applied';
|
||||||
|
if (String(args.note || '').toLowerCase() === 'held') return 'ready';
|
||||||
|
return 'summarizing';
|
||||||
|
}
|
||||||
|
|
||||||
|
/** One short, non-technical label per compression phase. */
|
||||||
|
export function getCompressionPhaseLabel(tool: any): string {
|
||||||
|
switch (getCompressionPhase(tool)) {
|
||||||
|
case 'ready':
|
||||||
|
return 'Summary ready; kept in reserve until the context fills up';
|
||||||
|
case 'applied':
|
||||||
|
return 'Replaced this stretch with its summary';
|
||||||
|
case 'failed':
|
||||||
|
return 'Summary failed; full record kept';
|
||||||
|
default:
|
||||||
|
return 'Summarizing this stretch';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A ready-but-held summary is waiting, not working: the card must not keep
|
||||||
|
* its running pulse even though the trace status is still `running`.
|
||||||
|
*/
|
||||||
|
export function isCompressionWaiting(tool: any): boolean {
|
||||||
|
return isCompressionTool(tool) && getCompressionPhase(tool) === 'ready';
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Get Material Symbol icon for tools
|
* Get Material Symbol icon for tools
|
||||||
*/
|
*/
|
||||||
|
|||||||
@@ -1,9 +1,9 @@
|
|||||||
{
|
{
|
||||||
"system": "You are a precise, evidence-driven verification expert (Checker). You audit declared check items against recorded execution evidence and issue verdicts. You never operate the device, never modify any state, and never negotiate with the executing agent — you only observe, probe read-only state where allowed, and judge.",
|
"system": "You are a precise, evidence-driven verification expert (Checker). You audit declared check items against recorded execution evidence and issue verdicts. You never operate the device, never modify any state, and never negotiate with the executing agent — you only observe, probe read-only state where allowed, and judge.",
|
||||||
"base_rules": "# Verdict Rules (always apply)\n- Quote every check item verbatim in `item_text` — never paraphrase.\n- Every verdict MUST carry concrete evidence: a specific probe output, a specific note excerpt, or a specific on-screen/screenshot observation. A `failed` verdict without concrete evidence is invalid and will be downgraded.\n- `inconclusive` means the evidence needed for a judgment was never captured or is genuinely ambiguous. When evidence was never collected, output `inconclusive` and state explicitly that the evidence is missing — do NOT output `failed` for lack of evidence.\n- Judge only what the check item declares. Do not invent additional criteria and do not relax the declared ones.",
|
"base_rules": "# Verdict Rules (always apply)\n- Quote every check item verbatim in `item_text` — never paraphrase.\n- Every verdict MUST carry concrete evidence: a specific probe output, a specific note excerpt, or a specific on-screen/screenshot observation. A `failed` verdict without concrete evidence is invalid and will be downgraded.\n- `inconclusive` means the evidence needed for a judgment was never captured or is genuinely ambiguous. When evidence was never collected, output `inconclusive` and state explicitly that the evidence is missing — do NOT output `failed` for lack of evidence.\n- Judge only what the check item declares. Do not invent additional criteria and do not relax the declared ones.",
|
||||||
"verify_semantics": "# `verify` items (completion acceptance criteria)\n- A `verify` item asks: was the anchored subgoal actually accomplished to this standard?\n- `failed` verify verdicts trigger a repair loop, so a failed verdict MUST include a `suggestion`: a concrete, actionable hint telling the executor what is missing or wrong.\n- The release decision (whether execution proceeds) is computed outside of you and is fail-open on errors; your job is only the truthful verdict value.",
|
"verify_semantics": "# `verify` items (completion acceptance criteria)\n- A `verify` item asks: was the anchored subgoal actually accomplished to this standard?\n- A `failed` verify verdict can send its milestone back for repair, so it must include a `suggestion`: a concrete, actionable hint telling the executor what is missing or wrong.\n- The release decision (whether execution proceeds) is computed outside of you and is fail-open on errors; your job is only the truthful verdict value.",
|
||||||
"assert_semantics": "# `assert` items (test assertions)\n- An `assert` item is a test expectation, not an instruction. A failed assertion is a legitimate, valuable test result: record it truthfully with evidence. Nothing will (or should) try to repair it.\n- Never mark an assert `failed` merely because evidence is absent — absence of captured evidence is `inconclusive` with an explicit note that the evidence is missing.\n- `suggestion` is not needed for assert verdicts.",
|
"assert_semantics": "# `assert` items (test assertions)\n- An `assert` item is a test expectation, not an instruction. A failed assertion is a legitimate, valuable test result: record it truthfully with evidence. Nothing will (or should) try to repair it.\n- Never mark an assert `failed` merely because evidence is absent — absence of captured evidence is `inconclusive` with an explicit note that the evidence is missing.\n- Precondition first: before judging an assert, confirm from the execution history (step replay, notes, anchor evidence) that the action(s) meant to trigger the asserted behavior were actually and correctly performed by the executor (the toggle was really tapped, the intended screen was really reached).\n- If that precondition is not established — the executor never performed the trigger, performed the wrong action, or the evidence cannot show it — output `inconclusive` and state explicitly that the precondition failed / the triggering action is not confirmed as executed. Never output `failed` here: a failed assert is attributed to the device under test, and an execution problem must not be reported as a device defect.\n- Output `failed` only when the precondition is established AND the device state, probe output, or recorded evidence still contradicts the expectation. The evidence must cite the concrete step(s) that performed the trigger plus the probe output / observation that contradicts the expectation, so the verdict reads as a device-defect diagnosis.\n- `suggestion` is not needed for assert verdicts.",
|
||||||
"anchor_guide": "# Evidence anchoring (checkpoint audit)\n- You are auditing a subgoal at its completion moment. The authoritative evidence is the execution history around the anchor step (the anchor step's pre-action screenshot and the preceding step's post-action state) plus persistently probeable device state.\n- For transient expectations (toasts, dialogs, momentary states) and ordering expectations, you MUST inspect the recorded history near the anchor step — the live device has already moved on and is NOT valid evidence for them.\n- Prefer persistent state probes and recorded notes over inference from summaries.",
|
"anchor_guide": "# Evidence anchoring (checkpoint audit)\n- You are auditing a subgoal at its completion moment. The authoritative evidence is the execution history around the anchor step (the anchor step's pre-action screenshot and the preceding step's post-action state) plus persistently probeable device state.\n- For transient expectations (toasts, dialogs, momentary states) and ordering expectations, you MUST inspect the recorded history near the anchor step — the live device has already moved on and is NOT valid evidence for them.\n- Prefer persistent state probes and recorded notes over inference from summaries.",
|
||||||
"final_guide": "# Final review (task exit audit)\n- Audit against the USER'S ORIGINAL GOAL, not merely the plan's checkboxes.\n- `@end` items: judge them using the FINAL device state provided to you.\n- `on_complete` items: cite the verdict ledger — an already-recorded PASS/FAIL from the item's checkpoint attempt is authoritative (an assert that ever failed stays failed). For items never checked, judge from the historical evidence near their subgoal's completion; if that evidence was never captured, output `inconclusive` stating the evidence is missing. NEVER re-judge an `on_complete` item from the current screen: the current screen only proves the present, not the past moment the item was declared for.\n- `unmet_subgoals`: list, verbatim, any plan subgoal marked complete whose verify criteria are demonstrably not met. Leave it empty otherwise; failed asserts alone do NOT make a subgoal unmet.",
|
"final_guide": "# Final review (task exit audit)\n- Audit against the user's original goal as amended by the user guidance received during the run (listed in the prompt), not merely the plan's checkboxes. Guidance outranks the plan and its check lines: judge a check line the guidance retired or changed by the guidance, not by its original wording.\n- `@end` items: judge them using the FINAL device state provided to you.\n- `on_complete` items: cite the verdict ledger — an already-recorded PASS/FAIL from the item's checkpoint attempt is authoritative (an assert that ever failed stays failed). For items never checked, judge from the historical evidence near their subgoal's completion; if that evidence was never captured, output `inconclusive` stating the evidence is missing. NEVER re-judge an `on_complete` item from the current screen: the current screen only proves the present, not the past moment the item was declared for.\n- `unmet_subgoals`: list, verbatim, any plan subgoal marked complete whose verify criteria are demonstrably not met. Leave it empty otherwise; failed asserts alone do NOT make a subgoal unmet.",
|
||||||
"probe_guide": "# Device probes (read-only)\n- `probe_device` runs a fixed, enumerated set of read-only state probes. Prefer probing persistent state over inferring it: alarms (`alarms`), battery status (`battery`), foreground activities (`foreground`), notifications (`notifications`), a settings value (`setting` with `namespace` + `key`), a content provider query (`content` with `uri`), installed packages (`packages`), a system property (`prop` with `key`).\n- Probes outside this table do not exist; you cannot run arbitrary commands."
|
"probe_guide": "# Device probes (read-only)\n- `probe_device` runs a fixed, enumerated set of read-only state probes. Prefer probing persistent state over inferring it: alarms (`alarms`), battery status (`battery`), foreground activities (`foreground`), notifications (`notifications`), a settings value (`setting` with `namespace` + `key`), a content provider query (`content` with `uri`), installed packages (`packages`), a system property (`prop` with `key`).\n- Probes outside this table do not exist; you cannot run arbitrary commands."
|
||||||
}
|
}
|
||||||
@@ -319,21 +319,54 @@ def build_checker_tools(
|
|||||||
# --- History formatting --------------------------------------------------------------
|
# --- History formatting --------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
def _format_history(ctx: ArtemisContext, limit: int = 60) -> str:
|
def _load_steps(ctx: ArtemisContext) -> list[dict] | str:
|
||||||
|
"""The agent-friendly step records, or the reason they are unavailable."""
|
||||||
if not ctx.data_engine:
|
if not ctx.data_engine:
|
||||||
return "No execution history available."
|
return "No execution history available."
|
||||||
try:
|
try:
|
||||||
steps = ctx.data_engine.get_agent_friendly_steps() or []
|
return list(ctx.data_engine.get_agent_friendly_steps() or [])
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
return f"Failed to load execution history: {e}"
|
return f"Failed to load execution history: {e}"
|
||||||
|
|
||||||
|
|
||||||
|
def _step_offset_label(step: dict, session_start: float | None) -> str:
|
||||||
|
"""Session-relative ``T+mm:ss`` label of a step: the clock every agent shares.
|
||||||
|
|
||||||
|
The recorded timestamp against the session start is authoritative; a
|
||||||
|
record that only carries the engine's ``"97.7s"`` label is converted.
|
||||||
|
"""
|
||||||
|
from artemis.memory.transcript import format_session_offset
|
||||||
|
|
||||||
|
ts = step.get("timestamp")
|
||||||
|
if isinstance(session_start, (int, float)) and isinstance(ts, (int, float)):
|
||||||
|
return format_session_offset(float(ts) - float(session_start))
|
||||||
|
rel = str(step.get("relative_time") or "").strip()
|
||||||
|
if rel.startswith("T+"):
|
||||||
|
return rel
|
||||||
|
if rel.endswith("s"):
|
||||||
|
try:
|
||||||
|
return format_session_offset(float(rel[:-1]))
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
return "T+??:??"
|
||||||
|
|
||||||
|
|
||||||
|
def _format_history(ctx: ArtemisContext, limit: int = 60, steps: list[dict] | None = None) -> str:
|
||||||
|
if steps is None:
|
||||||
|
steps = _load_steps(ctx)
|
||||||
|
if isinstance(steps, str):
|
||||||
|
return steps
|
||||||
if not steps:
|
if not steps:
|
||||||
return "No execution history available."
|
return "No execution history available."
|
||||||
|
session_start = getattr(ctx.data_engine, "session_start_time", None)
|
||||||
lines = []
|
lines = []
|
||||||
for s in steps[-limit:]:
|
for s in steps[-limit:]:
|
||||||
num = s.get("step_number")
|
num = s.get("step_number")
|
||||||
rel = s.get("relative_time", "")
|
summary = (s.get("summary") or "").strip()
|
||||||
summary = s.get("summary") or ""
|
|
||||||
action = s.get("action_taken")
|
action = s.get("action_taken")
|
||||||
|
if not summary and not action:
|
||||||
|
# Nothing to audit on this step: an empty row is only noise.
|
||||||
|
continue
|
||||||
action_str = ""
|
action_str = ""
|
||||||
if action and not summary:
|
if action and not summary:
|
||||||
# The shared renderer keeps the target's provenance visible: a
|
# The shared renderer keeps the target's provenance visible: a
|
||||||
@@ -343,8 +376,25 @@ def _format_history(ctx: ArtemisContext, limit: int = 60) -> str:
|
|||||||
action_str = format_actions_clean(action)[:160]
|
action_str = format_actions_clean(action)[:160]
|
||||||
except Exception:
|
except Exception:
|
||||||
action_str = str(action)[:160]
|
action_str = str(action)[:160]
|
||||||
lines.append(f"- Step {num} ({rel}): {summary or action_str}".rstrip())
|
label = _step_offset_label(s, session_start)
|
||||||
return "\n".join(lines)
|
lines.append(f"- Step {num} ({label}): {summary or action_str}".rstrip())
|
||||||
|
return "\n".join(lines) or "No execution history available."
|
||||||
|
|
||||||
|
|
||||||
|
def _format_user_guidance(ctx: ArtemisContext, steps: list[dict] | None = None) -> str:
|
||||||
|
"""Every mid-run user instruction, read from the step it was stamped on."""
|
||||||
|
if steps is None:
|
||||||
|
steps = _load_steps(ctx)
|
||||||
|
if isinstance(steps, str) or not steps:
|
||||||
|
return "(none)"
|
||||||
|
session_start = getattr(ctx.data_engine, "session_start_time", None)
|
||||||
|
lines = []
|
||||||
|
for s in steps:
|
||||||
|
instruction = (s.get("extra_metadata") or {}).get("injected_instruction")
|
||||||
|
if isinstance(instruction, str) and instruction.strip():
|
||||||
|
label = _step_offset_label(s, session_start)
|
||||||
|
lines.append(f'- Step {s.get("step_number")} ({label}): "{instruction.strip()}"')
|
||||||
|
return "\n".join(lines) or "(none)"
|
||||||
|
|
||||||
|
|
||||||
def _format_check_items(check_items: list) -> str:
|
def _format_check_items(check_items: list) -> str:
|
||||||
@@ -773,14 +823,16 @@ async def run_final_check(
|
|||||||
)
|
)
|
||||||
|
|
||||||
screenshot_b64, minimal_list = await _capture_final_screen(ctx)
|
screenshot_b64, minimal_list = await _capture_final_screen(ctx)
|
||||||
|
steps = _load_steps(ctx)
|
||||||
|
|
||||||
human_text = (
|
human_text = (
|
||||||
f"# User's Original Goal\n{goal}\n\n"
|
f"# User's Original Goal\n{goal}\n\n"
|
||||||
|
f"# User guidance received during the run\n{_format_user_guidance(ctx, steps)}\n\n"
|
||||||
f"# Final Task Plan\n{plan_text or '(no plan)'}\n\n"
|
f"# Final Task Plan\n{plan_text or '(no plan)'}\n\n"
|
||||||
f"# Declared Check Items\n{_format_check_items(items)}\n\n"
|
f"# Declared Check Items\n{_format_check_items(items)}\n\n"
|
||||||
f"# Verdict Ledger (checkpoint results, authoritative for on_complete"
|
f"# Verdict Ledger (checkpoint results, authoritative for on_complete"
|
||||||
f" items)\n{_format_ledger(ledger)}\n\n"
|
f" items)\n{_format_ledger(ledger)}\n\n"
|
||||||
f"# Execution History (summaries)\n{_format_history(ctx)}\n\n"
|
f"# Execution History (summaries)\n{_format_history(ctx, steps=steps)}\n\n"
|
||||||
f"# Final Screen Elements\n{minimal_list}"
|
f"# Final Screen Elements\n{minimal_list}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -356,9 +356,10 @@ class Diagnoser:
|
|||||||
if submit_tool:
|
if submit_tool:
|
||||||
submit_tool.invoke(args)
|
submit_tool.invoke(args)
|
||||||
|
|
||||||
outcome = f"Analysis: {analysis} | Actionable Steps: "
|
outcome = f"Analysis: {analysis}"
|
||||||
for step in steps:
|
if steps:
|
||||||
outcome += f"{step} "
|
numbered = "\n".join(f"{i}. {step}" for i, step in enumerate(steps, start=1))
|
||||||
|
outcome += f"\nActionable steps:\n{numbered}"
|
||||||
return outcome
|
return outcome
|
||||||
|
|
||||||
def _max_iteration_outcome(self, response, outcome: str) -> str:
|
def _max_iteration_outcome(self, response, outcome: str) -> str:
|
||||||
|
|||||||
@@ -61,7 +61,7 @@ UNIVERSAL_EXPLORER_TOOLS = [
|
|||||||
"function": {
|
"function": {
|
||||||
"name": "detect_objects",
|
"name": "detect_objects",
|
||||||
"description": (
|
"description": (
|
||||||
"[Perception] Locates elements, shapes and more using powerful VLM."
|
"[Perception] Locates elements, shapes and more using a vision-language model."
|
||||||
" Returns their normalized coordinates (0-1000 scale) with label dots."
|
" Returns their normalized coordinates (0-1000 scale) with label dots."
|
||||||
),
|
),
|
||||||
"parameters": {
|
"parameters": {
|
||||||
@@ -235,7 +235,7 @@ NATIVE_EXPLORER_TOOL_DECLARATIONS = [
|
|||||||
types.FunctionDeclaration(
|
types.FunctionDeclaration(
|
||||||
name="detect_objects",
|
name="detect_objects",
|
||||||
description=(
|
description=(
|
||||||
"[Perception] Locates elements, shapes and more using powerful"
|
"[Perception] Locates elements, shapes and more using a"
|
||||||
" VLM. Returns their normalized coordinates (in a [0, 1000]"
|
" VLM. Returns their normalized coordinates (in a [0, 1000]"
|
||||||
" scale) labeled with [D1], [D2] etc. in the text output, along"
|
" scale) labeled with [D1], [D2] etc. in the text output, along"
|
||||||
" with an annotated image containing corresponding visual label"
|
" with an annotated image containing corresponding visual label"
|
||||||
|
|||||||
@@ -14,10 +14,14 @@
|
|||||||
|
|
||||||
"""Context compression shared by Flash and Pro.
|
"""Context compression shared by Flash and Pro.
|
||||||
|
|
||||||
:class:`ScrubEdgeCompressor` strips XML at depth 1 and resolves screenshots
|
:class:`ScrubEdgeCompressor` maintains two edges over the observation history:
|
||||||
at depth K, then freezes the message. A late summary never backfills a frozen
|
a shallow text edge (``xml_scrub_depth``, default 1: as soon as a newer
|
||||||
message. FlashRunner uses this implementation; the Pro transcript ledger
|
observation exists) where the heavy text blocks and the per-turn ephemeral
|
||||||
reuses it over its active region (see the class docstring).
|
blocks leave, and the screenshot edge at depth K where the image is resolved
|
||||||
|
in place into its visual summary. A message past both edges is frozen; a late
|
||||||
|
summary never backfills a frozen message. FlashRunner uses this implementation;
|
||||||
|
the Pro transcript ledger reuses it over its active region (see the class
|
||||||
|
docstring).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Any, Callable
|
from typing import Any, Callable
|
||||||
@@ -25,6 +29,7 @@ from typing import Any, Callable
|
|||||||
from langchain_core.messages import BaseMessage
|
from langchain_core.messages import BaseMessage
|
||||||
|
|
||||||
from artemis.memory.step_memory import StepMemoryService
|
from artemis.memory.step_memory import StepMemoryService
|
||||||
|
from artemis.memory.transcript import EPHEMERAL_BLOCKS_KEY
|
||||||
from artemis.utils.logger import get_logger
|
from artemis.utils.logger import get_logger
|
||||||
|
|
||||||
logger = get_logger(__name__)
|
logger = get_logger(__name__)
|
||||||
@@ -35,6 +40,15 @@ HISTORY_SUMMARY_PREFIX = "--- Historical Visual Transition ---\n"
|
|||||||
_HISTORY_SUMMARY_PREFIX = HISTORY_SUMMARY_PREFIX
|
_HISTORY_SUMMARY_PREFIX = HISTORY_SUMMARY_PREFIX
|
||||||
_UI_LIST_MARKER = "--- UI Element List ---"
|
_UI_LIST_MARKER = "--- UI Element List ---"
|
||||||
|
|
||||||
|
#: Text of the label block both observation shapes place directly above the
|
||||||
|
#: screenshot. It only makes sense next to an image, so it leaves together
|
||||||
|
#: with the image it labels (the visual summary carries its own header).
|
||||||
|
SCREENSHOT_LABEL = "--- Current Screenshot ---"
|
||||||
|
|
||||||
|
#: Sentinel: the screenshot stays for now (summary pending inside the grace
|
||||||
|
#: window); the message is revisited for the image swap only.
|
||||||
|
_KEEP_IMAGE = object()
|
||||||
|
|
||||||
|
|
||||||
def _without_marked_suffix(text: str, markers: tuple[str, ...] = (_UI_LIST_MARKER,)) -> str:
|
def _without_marked_suffix(text: str, markers: tuple[str, ...] = (_UI_LIST_MARKER,)) -> str:
|
||||||
"""Remove a marked heavy block without discarding text before its marker."""
|
"""Remove a marked heavy block without discarding text before its marker."""
|
||||||
@@ -49,22 +63,32 @@ def _without_marked_suffix(text: str, markers: tuple[str, ...] = (_UI_LIST_MARKE
|
|||||||
|
|
||||||
|
|
||||||
class ScrubEdgeCompressor:
|
class ScrubEdgeCompressor:
|
||||||
"""Replace older observation details with summaries near the message tail.
|
"""Replace older observation details with summaries at two scrub edges.
|
||||||
|
|
||||||
Mutation discipline — every message is touched at most twice, always near
|
Depth 1 == the most recent image-bearing message. Messages newer than an
|
||||||
the tail, and is then frozen forever:
|
edge are byte-identical to what the model was first shown. Each message
|
||||||
|
is edited at most twice and is then frozen:
|
||||||
|
|
||||||
1. Depth 1 (XML): once a newer observation exists, historical UI Element
|
1. Text edge (depth > ``xml_scrub_depth``, default 1, i.e. as soon as a
|
||||||
lists are stripped (same semantics as the legacy ``prune_history_xml``).
|
newer observation exists), one pass, exactly once per message: the
|
||||||
2. Depth K (image, ``image_scrub_depth``): the K-th most recent
|
heavy blocks behind ``strip_markers`` (the UI element list; for Pro
|
||||||
image-bearing message has its screenshot resolved — replaced by the
|
also the plan recitation) are cut at their marker with the text before
|
||||||
ready visual summary; if the summary is still pending the image is
|
the marker kept, and every content block whose index is listed under
|
||||||
retained for up to ``pending_grace_steps`` further image-depths, after
|
:data:`~artemis.memory.transcript.EPHEMERAL_BLOCKS_KEY` in the
|
||||||
which it is replaced by a pending placeholder. A failed summary is
|
message's ``additional_kwargs`` (reminders, hints, per-turn notices)
|
||||||
replaced by an unavailable placeholder immediately. Both placeholders
|
is removed. Only the live observation therefore ever carries an
|
||||||
carry the DataEngine step reference.
|
indexed element list: a stale ``[n]`` index from an older screen can
|
||||||
3. Frozen: after resolution the message is never read or written again —
|
never be picked up as a target.
|
||||||
a late summary never backfills a frozen message.
|
2. Screenshot edge (depth >= ``image_scrub_depth``), IN PLACE: the ready
|
||||||
|
visual summary block takes the image block's position and the
|
||||||
|
:data:`SCREENSHOT_LABEL` block above it is removed. A failed summary
|
||||||
|
becomes an unavailable placeholder immediately; a summary still
|
||||||
|
pending keeps the image (and its label) for up to
|
||||||
|
``pending_grace_steps`` further depths, after which a pending
|
||||||
|
placeholder replaces it. Both placeholders carry the DataEngine step
|
||||||
|
reference. An image with no summary job at all is dropped silently.
|
||||||
|
3. Frozen: after the image is resolved the message is never read or
|
||||||
|
written again; a late summary never backfills a frozen message.
|
||||||
|
|
||||||
Contract: the message list is append-only (FlashRunner never removes or
|
Contract: the message list is append-only (FlashRunner never removes or
|
||||||
reorders entries), so bookkeeping is index-based and each pass only
|
reorders entries), so bookkeeping is index-based and each pass only
|
||||||
@@ -72,15 +96,14 @@ class ScrubEdgeCompressor:
|
|||||||
whole history.
|
whole history.
|
||||||
|
|
||||||
The Pro transcript ledger uses this discipline over its active region.
|
The Pro transcript ledger uses this discipline over its active region.
|
||||||
Its observation messages carry no
|
Its observation messages carry no ``tool_call_id``, so
|
||||||
``tool_call_id``, so ``summary_key_getter`` supplies the summary-job key
|
``summary_key_getter`` supplies the summary-job key (the DataEngine step
|
||||||
(the DataEngine step id) per message; ``strip_markers`` extends the
|
id) per message; ``strip_markers`` extends the strip to the Pro tail
|
||||||
depth-1 strip to the Pro tail blocks (UI element list + plan recitation);
|
blocks (UI element list + plan recitation); and ``tail_offset=1``
|
||||||
and ``tail_offset=1`` accounts for the live observation living outside
|
accounts for the live observation living outside the tracked list (the
|
||||||
the tracked list (the ledger renders it as a separate tail), so depth
|
ledger renders it as a separate tail), so depth arithmetic stays aligned
|
||||||
arithmetic and the keep-window stay aligned with the Flash semantics
|
with the Flash semantics where the live observation is *inside* the
|
||||||
where the live observation is *inside* the list. Defaults leave the
|
list.
|
||||||
Flash behavior byte-identical.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
@@ -90,7 +113,7 @@ class ScrubEdgeCompressor:
|
|||||||
prune_history_xml: bool = True,
|
prune_history_xml: bool = True,
|
||||||
image_scrub_depth: int = 3,
|
image_scrub_depth: int = 3,
|
||||||
pending_grace_steps: int = 3,
|
pending_grace_steps: int = 3,
|
||||||
xml_scrub_depth: int = 1,
|
xml_scrub_depth: int | None = 1,
|
||||||
summary_key_getter: Callable[[BaseMessage], str | None] | None = None,
|
summary_key_getter: Callable[[BaseMessage], str | None] | None = None,
|
||||||
strip_markers: tuple[str, ...] = (_UI_LIST_MARKER,),
|
strip_markers: tuple[str, ...] = (_UI_LIST_MARKER,),
|
||||||
tail_offset: int = 0,
|
tail_offset: int = 0,
|
||||||
@@ -99,7 +122,9 @@ class ScrubEdgeCompressor:
|
|||||||
self._prune_history_xml = prune_history_xml
|
self._prune_history_xml = prune_history_xml
|
||||||
self._image_scrub_depth = max(1, image_scrub_depth)
|
self._image_scrub_depth = max(1, image_scrub_depth)
|
||||||
self._pending_grace_steps = max(0, pending_grace_steps)
|
self._pending_grace_steps = max(0, pending_grace_steps)
|
||||||
self._xml_scrub_depth = max(1, xml_scrub_depth)
|
# Text edge: the heavy text blocks and ephemeral blocks leave once a
|
||||||
|
# message is deeper than this (None keeps the default of 1).
|
||||||
|
self._xml_scrub_depth = max(1, int(xml_scrub_depth)) if xml_scrub_depth is not None else 1
|
||||||
self._summary_key_getter = summary_key_getter
|
self._summary_key_getter = summary_key_getter
|
||||||
self._strip_markers = tuple(strip_markers) or (_UI_LIST_MARKER,)
|
self._strip_markers = tuple(strip_markers) or (_UI_LIST_MARKER,)
|
||||||
self._tail_offset = max(0, tail_offset)
|
self._tail_offset = max(0, tail_offset)
|
||||||
@@ -110,15 +135,28 @@ class ScrubEdgeCompressor:
|
|||||||
# {"idx", "key" (str tool_call_id | None), "legacy" (ordinal | None), "is_tool"}
|
# {"idx", "key" (str tool_call_id | None), "legacy" (ordinal | None), "is_tool"}
|
||||||
self._tracked: list[dict[str, Any]] = []
|
self._tracked: list[dict[str, Any]] = []
|
||||||
self._frozen: set[int] = set()
|
self._frozen: set[int] = set()
|
||||||
|
# Messages whose one-time text edits (strip + ephemeral deletion) are
|
||||||
|
# done; a message inside the pending-grace window sits here unfrozen.
|
||||||
|
self._text_scrubbed: set[int] = set()
|
||||||
|
# Strip-marker carriers without an image (e.g. a step whose screenshot
|
||||||
|
# failed); they are stripped when the edge passes their position.
|
||||||
self._xml_candidates: list[int] = []
|
self._xml_candidates: list[int] = []
|
||||||
|
|
||||||
def compress(self, messages: list[BaseMessage]) -> None:
|
def compress(
|
||||||
"""Advance the scrub edge over newly appended messages in-place."""
|
self, messages: list[BaseMessage], *, image_scrub_depth: int | None = None
|
||||||
|
) -> None:
|
||||||
|
"""Advance the scrub edge over newly appended messages in-place.
|
||||||
|
|
||||||
|
``image_scrub_depth`` overrides the constructor depth for this pass
|
||||||
|
(the transcript ledger passes its occupancy-driven depth). Because a
|
||||||
|
scrubbed message is frozen, a deeper pass after a shallower one never
|
||||||
|
backfills anything: relaxing the depth only lets *new* images live
|
||||||
|
longer, tightening it scrubs the extra depths once.
|
||||||
|
"""
|
||||||
self._discover(messages)
|
self._discover(messages)
|
||||||
if not self._tracked:
|
if not self._tracked and not self._xml_candidates:
|
||||||
return
|
return
|
||||||
self._scrub_xml(messages)
|
self._scrub_edge(messages, image_scrub_depth)
|
||||||
self._scrub_images(messages)
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Discovery (incremental; frozen prefix is never rescanned)
|
# Discovery (incremental; frozen prefix is never rescanned)
|
||||||
@@ -144,8 +182,6 @@ class ScrubEdgeCompressor:
|
|||||||
elif block.get("type") == "text" and self._has_strip_marker(block.get("text", "")):
|
elif block.get("type") == "text" and self._has_strip_marker(block.get("text", "")):
|
||||||
has_xml = True
|
has_xml = True
|
||||||
|
|
||||||
if has_xml:
|
|
||||||
self._xml_candidates.append(idx)
|
|
||||||
if has_image:
|
if has_image:
|
||||||
key = str(tool_call_id) if tool_call_id is not None else None
|
key = str(tool_call_id) if tool_call_id is not None else None
|
||||||
if key is None and self._summary_key_getter is not None:
|
if key is None and self._summary_key_getter is not None:
|
||||||
@@ -163,62 +199,24 @@ class ScrubEdgeCompressor:
|
|||||||
"is_tool": tool_call_id is not None,
|
"is_tool": tool_call_id is not None,
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
|
elif has_xml:
|
||||||
|
self._xml_candidates.append(idx)
|
||||||
self._scanned_until = len(messages)
|
self._scanned_until = len(messages)
|
||||||
|
|
||||||
def _has_strip_marker(self, text: str) -> bool:
|
def _has_strip_marker(self, text: str) -> bool:
|
||||||
return any(marker in text for marker in self._strip_markers)
|
return any(marker in text for marker in self._strip_markers)
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Depth-1 XML scrub
|
# The edges: text strip + ephemeral deletion (shallow), image swap (K)
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
def _scrub_xml(self, messages: list[BaseMessage]) -> None:
|
def _scrub_edge(
|
||||||
if not self._prune_history_xml:
|
self, messages: list[BaseMessage], image_scrub_depth: int | None = None
|
||||||
return
|
) -> None:
|
||||||
# Keep the UI list only in the newest xml_scrub_depth observation
|
|
||||||
# messages; everything older (or non-image XML carriers) is stripped.
|
|
||||||
# With a live tail outside the tracked list (tail_offset > 0) the tail
|
|
||||||
# itself occupies the newest slots of the keep window.
|
|
||||||
keep_count = max(0, self._xml_scrub_depth - self._tail_offset)
|
|
||||||
keep_indices = {rec["idx"] for rec in self._tracked[-keep_count:]} if keep_count else set()
|
|
||||||
still_pending: list[int] = []
|
|
||||||
for idx in self._xml_candidates:
|
|
||||||
if idx in keep_indices:
|
|
||||||
still_pending.append(idx)
|
|
||||||
continue
|
|
||||||
if idx in self._frozen:
|
|
||||||
continue
|
|
||||||
self._strip_xml_blocks(messages[idx])
|
|
||||||
self._xml_candidates = still_pending
|
|
||||||
|
|
||||||
def _strip_xml_blocks(self, msg: BaseMessage) -> None:
|
|
||||||
content = getattr(msg, "content", None)
|
|
||||||
if not isinstance(content, list):
|
|
||||||
return
|
|
||||||
new_blocks: list[Any] = []
|
|
||||||
for block in content:
|
|
||||||
if (
|
|
||||||
isinstance(block, dict)
|
|
||||||
and block.get("type") == "text"
|
|
||||||
and self._has_strip_marker(block.get("text", ""))
|
|
||||||
):
|
|
||||||
retained_text = _without_marked_suffix(block.get("text", ""), self._strip_markers)
|
|
||||||
if retained_text:
|
|
||||||
retained_block = dict(block)
|
|
||||||
retained_block["text"] = retained_text
|
|
||||||
new_blocks.append(retained_block)
|
|
||||||
continue
|
|
||||||
new_blocks.append(block)
|
|
||||||
msg.content = new_blocks
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# Depth-K image scrub
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
|
|
||||||
def _scrub_images(self, messages: list[BaseMessage]) -> None:
|
|
||||||
total = len(self._tracked)
|
total = len(self._tracked)
|
||||||
scrub_depth = self._image_scrub_depth
|
scrub_depth = max(1, int(image_scrub_depth or self._image_scrub_depth))
|
||||||
grace_limit = scrub_depth + self._pending_grace_steps
|
grace_limit = scrub_depth + self._pending_grace_steps
|
||||||
|
text_depth = self._xml_scrub_depth
|
||||||
|
|
||||||
for pos, rec in enumerate(self._tracked):
|
for pos, rec in enumerate(self._tracked):
|
||||||
if rec["idx"] in self._frozen:
|
if rec["idx"] in self._frozen:
|
||||||
@@ -226,12 +224,25 @@ class ScrubEdgeCompressor:
|
|||||||
# 1 == most recent image-bearing message; a live tail outside the
|
# 1 == most recent image-bearing message; a live tail outside the
|
||||||
# tracked list (tail_offset > 0) counts as the newest depths.
|
# tracked list (tail_offset > 0) counts as the newest depths.
|
||||||
depth = total - pos + self._tail_offset
|
depth = total - pos + self._tail_offset
|
||||||
if depth < scrub_depth:
|
|
||||||
continue # active window
|
|
||||||
if self._tail_offset == 0 and pos == total - 1:
|
if self._tail_offset == 0 and pos == total - 1:
|
||||||
continue # the live observation is never scrubbed
|
continue # the live observation is never scrubbed
|
||||||
|
past_text_edge = depth > text_depth
|
||||||
|
past_image_edge = depth >= scrub_depth
|
||||||
|
if not past_text_edge and not past_image_edge:
|
||||||
|
continue # newer than both edges: byte-identical
|
||||||
|
if not past_image_edge:
|
||||||
|
# Past the text edge only: strip once, keep the screenshot.
|
||||||
|
if rec["idx"] not in self._text_scrubbed:
|
||||||
|
self._rewrite(
|
||||||
|
messages[rec["idx"]],
|
||||||
|
rec["idx"],
|
||||||
|
replacement=_KEEP_IMAGE,
|
||||||
|
is_tool=rec["is_tool"],
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
summary, pending, failed, step_no = self._lookup(rec)
|
summary, pending, failed, step_no = self._lookup(rec)
|
||||||
|
replacement: Any
|
||||||
if summary is not None:
|
if summary is not None:
|
||||||
replacement = {
|
replacement = {
|
||||||
"type": "text",
|
"type": "text",
|
||||||
@@ -243,7 +254,7 @@ class ScrubEdgeCompressor:
|
|||||||
"text": f"[visual summary unavailable; evidence at DataEngine step {step_no}]",
|
"text": f"[visual summary unavailable; evidence at DataEngine step {step_no}]",
|
||||||
}
|
}
|
||||||
elif pending and depth <= grace_limit:
|
elif pending and depth <= grace_limit:
|
||||||
continue # grace period: retain the image, revisit next pass
|
replacement = _KEEP_IMAGE # grace: text edits now, image swap later
|
||||||
elif pending:
|
elif pending:
|
||||||
replacement = {
|
replacement = {
|
||||||
"type": "text",
|
"type": "text",
|
||||||
@@ -252,7 +263,26 @@ class ScrubEdgeCompressor:
|
|||||||
else:
|
else:
|
||||||
replacement = None # no summary job exists; drop the image silently
|
replacement = None # no summary job exists; drop the image silently
|
||||||
|
|
||||||
self._resolve(messages, rec, replacement)
|
self._resolve(messages, rec, replacement, text_edits=past_text_edge)
|
||||||
|
|
||||||
|
if self._xml_candidates:
|
||||||
|
# Imageless strip-marker carriers (a step whose screenshot failed)
|
||||||
|
# follow the same text edge, counted over every observation —
|
||||||
|
# image-bearing or not — plus the live tail: with the default
|
||||||
|
# depth 1 the list leaves as soon as any newer observation exists,
|
||||||
|
# so only the live observation ever carries an indexed element
|
||||||
|
# list. A carrier that is itself the newest observation stays.
|
||||||
|
observations = sorted({rec["idx"] for rec in self._tracked} | set(self._xml_candidates))
|
||||||
|
remaining: list[int] = []
|
||||||
|
for idx in self._xml_candidates:
|
||||||
|
newer = sum(1 for other in observations if other > idx) + self._tail_offset
|
||||||
|
if newer < text_depth:
|
||||||
|
remaining.append(idx)
|
||||||
|
continue
|
||||||
|
if idx not in self._frozen:
|
||||||
|
self._rewrite(messages[idx], idx, replacement=None, is_tool=False)
|
||||||
|
self._frozen.add(idx)
|
||||||
|
self._xml_candidates = remaining
|
||||||
|
|
||||||
def _lookup(self, rec: dict[str, Any]) -> tuple[str | None, bool, bool, int | None]:
|
def _lookup(self, rec: dict[str, Any]) -> tuple[str | None, bool, bool, int | None]:
|
||||||
"""Mirror the legacy keying: tool_call_id first, ordinal fallback."""
|
"""Mirror the legacy keying: tool_call_id first, ordinal fallback."""
|
||||||
@@ -280,42 +310,143 @@ class ScrubEdgeCompressor:
|
|||||||
self,
|
self,
|
||||||
messages: list[BaseMessage],
|
messages: list[BaseMessage],
|
||||||
rec: dict[str, Any],
|
rec: dict[str, Any],
|
||||||
replacement: dict[str, Any] | None,
|
replacement: Any,
|
||||||
|
*,
|
||||||
|
text_edits: bool = True,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Replace the message's image blocks and freeze it permanently."""
|
"""Rewrite one image-bearing message at the screenshot edge; freeze it
|
||||||
msg = messages[rec["idx"]]
|
unless the screenshot is kept for the pending-grace window. With a text
|
||||||
content = getattr(msg, "content", None)
|
edge deeper than the screenshot edge the text edits wait for their own
|
||||||
new_blocks: list[Any] = []
|
edge (``text_edits=False``); the message is then frozen only once both
|
||||||
if isinstance(content, list):
|
have happened."""
|
||||||
for block in content:
|
idx = rec["idx"]
|
||||||
if isinstance(block, dict) and block.get("type") in ("image_url", "image"):
|
self._rewrite(
|
||||||
continue
|
messages[idx],
|
||||||
if (
|
idx,
|
||||||
self._prune_history_xml
|
replacement=replacement,
|
||||||
and isinstance(block, dict)
|
is_tool=rec["is_tool"],
|
||||||
and block.get("type") == "text"
|
text_edits=text_edits,
|
||||||
and self._has_strip_marker(block.get("text", ""))
|
|
||||||
):
|
|
||||||
# Freezing implies fully scrubbed: no stale UI list survives
|
|
||||||
# even when xml_scrub_depth exceeds the image depth.
|
|
||||||
retained_text = _without_marked_suffix(
|
|
||||||
block.get("text", ""), self._strip_markers
|
|
||||||
)
|
)
|
||||||
|
if not text_edits:
|
||||||
|
return # text edge still ahead: revisit for the strip
|
||||||
|
if replacement is not _KEEP_IMAGE:
|
||||||
|
self._frozen.add(idx)
|
||||||
|
|
||||||
|
def _rewrite(
|
||||||
|
self,
|
||||||
|
msg: BaseMessage,
|
||||||
|
idx: int,
|
||||||
|
*,
|
||||||
|
replacement: Any,
|
||||||
|
is_tool: bool,
|
||||||
|
text_edits: bool = True,
|
||||||
|
) -> None:
|
||||||
|
"""The single-pass edit of one message (see the class docstring).
|
||||||
|
|
||||||
|
``replacement`` is the block that takes the first image's place (a
|
||||||
|
summary or placeholder), ``None`` to drop the image, or
|
||||||
|
:data:`_KEEP_IMAGE` to leave the image and its label in place. The
|
||||||
|
text edits (strip + ephemeral deletion) run only the first time the
|
||||||
|
message is rewritten and consume the ephemeral indices exactly once.
|
||||||
|
Any rewrite that happens *before* that first edit (the image swap when
|
||||||
|
the screenshot edge is shallower than the text edge, or a summary
|
||||||
|
arriving inside the grace window) can remove the label and the image
|
||||||
|
and so shift every later block; the surviving ephemeral indices are
|
||||||
|
then remapped to the new layout in ``additional_kwargs`` so the later
|
||||||
|
text edit still deletes the blocks that were marked, never a
|
||||||
|
neighbour that slid into their old position.
|
||||||
|
"""
|
||||||
|
content = getattr(msg, "content", None)
|
||||||
|
if not isinstance(content, list):
|
||||||
|
self._text_scrubbed.add(idx)
|
||||||
|
return
|
||||||
|
|
||||||
|
first_edit = idx not in self._text_scrubbed and text_edits
|
||||||
|
ephemeral: set[int] = set()
|
||||||
|
if first_edit:
|
||||||
|
ephemeral = self._ephemeral_indices(msg)
|
||||||
|
keep_image = replacement is _KEEP_IMAGE
|
||||||
|
|
||||||
|
new_blocks: list[Any] = []
|
||||||
|
# old position -> new position of every block that survives as itself
|
||||||
|
# (a replacement block takes the image's slot but is not the image).
|
||||||
|
position_map: dict[int, int] = {}
|
||||||
|
image_placed = False
|
||||||
|
for position, block in enumerate(content):
|
||||||
|
if position in ephemeral:
|
||||||
|
continue
|
||||||
|
if isinstance(block, dict):
|
||||||
|
kind = block.get("type")
|
||||||
|
if kind in ("image_url", "image"):
|
||||||
|
if keep_image:
|
||||||
|
position_map[position] = len(new_blocks)
|
||||||
|
new_blocks.append(block)
|
||||||
|
elif replacement is not None and not image_placed:
|
||||||
|
new_blocks.append(replacement) # in place of the image
|
||||||
|
image_placed = True
|
||||||
|
continue
|
||||||
|
if kind == "text":
|
||||||
|
text = block.get("text", "")
|
||||||
|
if not keep_image and text.strip() == SCREENSHOT_LABEL:
|
||||||
|
continue # the label leaves with the image it labelled
|
||||||
|
if first_edit and self._prune_history_xml and self._has_strip_marker(text):
|
||||||
|
retained_text = _without_marked_suffix(text, self._strip_markers)
|
||||||
if retained_text:
|
if retained_text:
|
||||||
retained_block = dict(block)
|
retained_block = dict(block)
|
||||||
retained_block["text"] = retained_text
|
retained_block["text"] = retained_text
|
||||||
|
position_map[position] = len(new_blocks)
|
||||||
new_blocks.append(retained_block)
|
new_blocks.append(retained_block)
|
||||||
continue
|
continue
|
||||||
|
position_map[position] = len(new_blocks)
|
||||||
new_blocks.append(block)
|
new_blocks.append(block)
|
||||||
|
|
||||||
if replacement is not None:
|
|
||||||
new_blocks.append(replacement)
|
|
||||||
|
|
||||||
# Defensive fallback: never leave ToolMessage or HumanMessage content empty
|
# Defensive fallback: never leave ToolMessage or HumanMessage content empty
|
||||||
if not new_blocks:
|
if not new_blocks:
|
||||||
new_blocks = [{"type": "text", "text": "Action dispatched." if rec["is_tool"] else ""}]
|
new_blocks = [{"type": "text", "text": "Action dispatched." if is_tool else ""}]
|
||||||
|
position_map = {}
|
||||||
|
|
||||||
msg.content = new_blocks
|
msg.content = new_blocks
|
||||||
self._frozen.add(rec["idx"])
|
if first_edit:
|
||||||
if rec["idx"] in self._xml_candidates:
|
self._text_scrubbed.add(idx)
|
||||||
self._xml_candidates.remove(rec["idx"])
|
if ephemeral:
|
||||||
|
# Consumed: the indices no longer describe the block layout.
|
||||||
|
msg.additional_kwargs.pop(EPHEMERAL_BLOCKS_KEY, None)
|
||||||
|
else:
|
||||||
|
self._remap_ephemeral(msg, position_map)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _remap_ephemeral(msg: BaseMessage, position_map: dict[int, int]) -> None:
|
||||||
|
"""Rewrite the not-yet-consumed ephemeral indices of ``msg`` through
|
||||||
|
``position_map`` (old block position -> new block position); indices of
|
||||||
|
blocks that did not survive are dropped and the key is removed when
|
||||||
|
nothing survives. Only ``additional_kwargs`` is touched, the content
|
||||||
|
blocks sent to the provider stay exactly as assembled."""
|
||||||
|
kwargs = getattr(msg, "additional_kwargs", None)
|
||||||
|
if not isinstance(kwargs, dict) or EPHEMERAL_BLOCKS_KEY not in kwargs:
|
||||||
|
return
|
||||||
|
remapped: list[int] = []
|
||||||
|
for value in kwargs.get(EPHEMERAL_BLOCKS_KEY) or []:
|
||||||
|
try:
|
||||||
|
old_position = int(value)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
continue
|
||||||
|
new_position = position_map.get(old_position)
|
||||||
|
if new_position is not None and new_position not in remapped:
|
||||||
|
remapped.append(new_position)
|
||||||
|
if remapped:
|
||||||
|
kwargs[EPHEMERAL_BLOCKS_KEY] = remapped
|
||||||
|
else:
|
||||||
|
kwargs.pop(EPHEMERAL_BLOCKS_KEY, None)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _ephemeral_indices(msg: BaseMessage) -> set[int]:
|
||||||
|
kwargs = getattr(msg, "additional_kwargs", None)
|
||||||
|
if not isinstance(kwargs, dict):
|
||||||
|
return set()
|
||||||
|
indices: set[int] = set()
|
||||||
|
for value in kwargs.get(EPHEMERAL_BLOCKS_KEY) or []:
|
||||||
|
try:
|
||||||
|
indices.add(int(value))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
continue
|
||||||
|
return indices
|
||||||
@@ -6,9 +6,9 @@ You are an autonomous and highly efficient Android Device Execution Agent. Your
|
|||||||
---
|
---
|
||||||
|
|
||||||
# 1. COGNITIVE PROTOCOL
|
# 1. COGNITIVE PROTOCOL
|
||||||
You operate in a dynamic observation-reasoning-execution loop. **CRITICAL RULE: YOU MUST NEVER INVOKE A TOOL SILENTLY!** In every single turn, you **MUST FIRST output a natural language reasoning paragraph BEFORE generating any tool call.**
|
In every turn, think through observation, planning and action in that order. Before calling an action tool or `report_task_status` (the Turn-Ending Action), your reply carries one short paragraph of connected prose reflecting those three phases; that written text is kept in the execution history and shown to the user.
|
||||||
|
|
||||||
- **Turn 1 Only (Initial Planning)**: Briefly decompose the overall `{{ goal }}` into high-level visual **Milestones** in your text output. Do not repeat this macro breakdown in subsequent turns.
|
- **Turn 1 Only (Initial Planning)**: Briefly decompose the objective into high-level visual **Milestones** in your text output. Do not repeat this macro breakdown in subsequent turns.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -23,7 +23,7 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
|
|||||||
---
|
---
|
||||||
|
|
||||||
# 3. READING YOUR HISTORY & THE SESSION CLOCK
|
# 3. READING YOUR HISTORY & THE SESSION CLOCK
|
||||||
- **Conversation = execution history**: The earlier turns of this conversation are your raw execution history. Each observation opens with `# CURRENT OBSERVATION [T+mm:ss]`, and each turn that executed a device action ends with an `--- Action Execution Result (T+mm:ss) ---` message (`Status: dispatched` means the device accepted the action; whether it had the intended effect is yours to judge from the observation that follows. `Status: failed` carries the `Error:` line). Every timestamp is a session-relative `T+mm:ss` offset from the start of this session; it is the only clock you have, for pacing waits as well as for reasoning about elapsed time.
|
- **Conversation = execution history**: The earlier turns of this conversation are your raw execution history. Each observation opens with `# CURRENT OBSERVATION [T+mm:ss]`. Every device action runs immediately and its tool result states what the device did (e.g. `Swiped up.`); whether it had the intended effect is yours to judge from the observation that follows. Only a turn whose action failed ends with an `--- Action Execution Result (T+mm:ss) ---` message (`Status: failed` with the `Error:` line). Every timestamp is a session-relative `T+mm:ss` offset from the start of this session; it is the only clock you have, for pacing waits as well as for reasoning about elapsed time.
|
||||||
- **Compression**: Screenshots older than a few turns are replaced by short `--- Historical Visual Transition ---` summaries and old `--- Visible UI Elements ---` lists are dropped. Long spans render as `[Era N | Steps a–b | T+.. → T+..]` blocks (a merged synopsis, segment titles, and a per-step action ledger). Because of this, if you spot critical text that you will need later (e.g., verification codes, specific titles, search lists), **you must explicitly verbalize and write those details into your text reasoning** so they persist across turns.
|
- **Compression**: Screenshots older than a few turns are replaced by short `--- Historical Visual Transition ---` summaries and old `--- Visible UI Elements ---` lists are dropped. Long spans render as `[Era N | Steps a–b | T+.. → T+..]` blocks (a merged synopsis, segment titles, and a per-step action ledger). Because of this, if you spot critical text that you will need later (e.g., verification codes, specific titles, search lists), **you must explicitly verbalize and write those details into your text reasoning** so they persist across turns.
|
||||||
{% if "search_history" in available_tools %}- **History Lookup (`search_history`)**: Compressed history keeps the step numbers and `T+` offsets but not every detail. When you need exactly what you did or saw at an earlier step (a value you read, the result of an action, which screen you were on), call `search_history` with keywords and/or the `[start, end]` step range shown in a chunk block; it searches the full stored history (screen descriptions, actions, results, reasoning, tool calls, notes, on-screen text) and every hit carries its step number. It is a pre-decision helper: read its answer, then act.{% endif %}
|
{% if "search_history" in available_tools %}- **History Lookup (`search_history`)**: Compressed history keeps the step numbers and `T+` offsets but not every detail. When you need exactly what you did or saw at an earlier step (a value you read, the result of an action, which screen you were on), call `search_history` with keywords and/or the `[start, end]` step range shown in a chunk block; it searches the full stored history (screen descriptions, actions, results, reasoning, tool calls, notes, on-screen text) and every hit carries its step number. It is a pre-decision helper: read its answer, then act.{% endif %}
|
||||||
{% if "replay_steps" in available_tools %}- **Step Replay (`replay_steps`)**: Replays one earlier step (or a short range) exactly as your context showed it — screen description, reasoning, tool calls with their results, the action and its execution result. Use the step numbers shown in the compressed history{% if "search_history" in available_tools %} or returned by `search_history`{% endif %}.{% endif %}
|
{% if "replay_steps" in available_tools %}- **Step Replay (`replay_steps`)**: Replays one earlier step (or a short range) exactly as your context showed it — screen description, reasoning, tool calls with their results, the action and its execution result. Use the step numbers shown in the compressed history{% if "search_history" in available_tools %} or returned by `search_history`{% endif %}.{% endif %}
|
||||||
@@ -39,12 +39,13 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
|
|||||||
---
|
---
|
||||||
|
|
||||||
# 5. ACTION RULES
|
# 5. ACTION RULES
|
||||||
1. **Normalized Coordinates (`0-1000` Scale)**: Always use normalized `[x, y]` coordinates in `0-1000` scale for any location-based target (e.g., `target=[500, 600]` or `click_sequence=[[300, 400], [500, 600]]`). Every coordinate target must carry `target_description` (one entry per point for `click_sequence`): a short, meaningful name for what you are aiming at, as a user would call it (e.g. 'play button', 'video body'), never a generic 'element' or 'button'. Nothing is inferred on your behalf: it is recorded as your own statement, and your history shows it marked as self-described.
|
1. **Prefer the element index.** The `--- Visible UI Elements ---` list numbers every element it found (`[3] Text: 'Wi-Fi' | Bounds: ...`). When your target is listed, address it by that index (`target=3`, a bare integer) with `click`, `long_press` or `input_text`: the tap lands on the element's center and the element's own text and bounds are recorded with the action. Heed the overlap warnings in the list; an overlapped element may not receive the tap. Indices belong to the screen they were listed on: swiping, scrolling or a page update renumbers them, so only ever use an index from the list in the current observation, never one remembered from an earlier turn. Within one turn, every index you call (even after an earlier action in the same turn) refers to the list in the current observation, not to the screen your earlier action produced.
|
||||||
2. **Do not determine element coordinates based on guesswork.** If the element is visible in the screenshot but its coordinates are missing from the provided `--- Visible UI Elements ---` list, please use the `ask_explorer` tool to locate the element first.
|
2. **Normalized Coordinates (`0-1000` Scale) otherwise.** When the target is visible but not listed, use normalized `[x, y]` coordinates in `0-1000` scale (e.g., `target=[500, 600]`; `click_sequence` always takes coordinate pairs, e.g. `sequence=[[300, 400], [500, 600]]`). Every coordinate target must carry `target_description` (one entry per point for `click_sequence`): a short, meaningful name for what you are aiming at, as a user would call it (e.g. 'play button', 'video body'), never a generic 'element' or 'button'. Nothing is inferred on your behalf: it is recorded as your own statement, and your history shows it marked as self-described.
|
||||||
{% if "manage_app" in available_tools %}3. **App Launching**: To start or reset an application, always use `manage_app(action="launch", app_name="...")` or `action="stop"`. Do not manually swipe across home screens.{% endif %}
|
3. **Do not determine element coordinates based on guesswork.** If the element is visible in the screenshot but neither listed nor safely locatable from the screenshot, use the `ask_explorer` tool to locate it first; its candidates join the element list and can be addressed by index or by coordinate.
|
||||||
|
{% if "manage_app" in available_tools %}4. **App Launching**: To start or reset an application, always use `manage_app(action="launch", app_name="...")` or `action="stop"`. Do not manually swipe across home screens.{% endif %}
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# 6. TERMINATION PROTOCOL (`report_task_status`)
|
# 6. TERMINATION PROTOCOL (`report_task_status`)
|
||||||
- **Completion (`status="completed"`)**: ONLY call `report_task_status(status="completed", ...)` after you have **visually verified** from the latest screen that the user's objective (`{{ goal }}`) is fully achieved.
|
- **Completion (`status="completed"`)**: ONLY call `report_task_status(status="completed", ...)` after you have **visually verified** from the latest screen that the objective is fully achieved.
|
||||||
- **Failure (`status="failed"`)**: If the task is genuinely impossible due to paywalls, unrecoverable crashes, or hitting an absolute dead end after sensible fallbacks, call `report_task_status(status="failed", ...)` with the exact blocking reason.
|
- **Failure (`status="failed"`)**: If the task is genuinely impossible due to paywalls, unrecoverable crashes, or hitting an absolute dead end after sensible fallbacks, call `report_task_status(status="failed", ...)` with the exact blocking reason.
|
||||||
+299
-96
@@ -22,10 +22,19 @@ the prompt is built every turn from a session :class:`TranscriptLedger` —
|
|||||||
a byte-stable system prefix, the committed earlier turns (append-only; old UI
|
a byte-stable system prefix, the committed earlier turns (append-only; old UI
|
||||||
lists stripped at depth 1, screenshots resolved to visual summaries at depth
|
lists stripped at depth 1, screenshots resolved to visual summaries at depth
|
||||||
K, long spans chunk-compressed), and a fresh tail carrying the current
|
K, long spans chunk-compressed), and a fresh tail carrying the current
|
||||||
observation under a ``# CURRENT OBSERVATION [T+mm:ss]`` header. Each
|
observation under a ``# CURRENT OBSERVATION [T+mm:ss]`` header. Actions run
|
||||||
committed turn ends with an ``--- Action Execution Result (T+mm:ss) ---``
|
synchronously and their tool messages already carry the outcome; only a turn
|
||||||
message, so every timestamp the model sees is a session-relative offset — the
|
whose action failed ends with an ``--- Action Execution Result (T+mm:ss) ---``
|
||||||
same clock the video analyzer uses for the session recording.
|
message (it carries the error). Every timestamp the model sees is a
|
||||||
|
session-relative offset — the same clock the video analyzer uses for the
|
||||||
|
session recording.
|
||||||
|
|
||||||
|
Per-turn notices on the tail (the reasoning reminder after a silent turn, the
|
||||||
|
no-tool-call nudge, the final-turn warning, the user-guidance wrapper) are
|
||||||
|
marked ephemeral: the ledger's scrub edge deletes them once the message leaves
|
||||||
|
the recent window, so they never reach the frozen history or a chunk capsule.
|
||||||
|
The verbatim body of a user instruction is the exception: it stays in the
|
||||||
|
active window until its turn is chunked, whose user lines then carry it on.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
@@ -45,9 +54,12 @@ from langchain_core.messages import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
from artemis.agents.flash.summarizer import VisualStepSummarizer, build_focus_context
|
from artemis.agents.flash.summarizer import VisualStepSummarizer, build_focus_context
|
||||||
|
from artemis.agents.operator.prompts import REASONING_REMINDER, UserGuidance, render_user_guidance
|
||||||
from artemis.agents.validator.tool_declarations import (
|
from artemis.agents.validator.tool_declarations import (
|
||||||
ASK_EXPLORER_TOOL,
|
ASK_EXPLORER_TOOL,
|
||||||
CLICK_SEQUENCE_TOOL,
|
CLICK_SEQUENCE_TOOL,
|
||||||
|
LIST_NOTES_TOOL,
|
||||||
|
READ_NOTE_TOOL,
|
||||||
REPORT_TASK_STATUS_TOOL,
|
REPORT_TASK_STATUS_TOOL,
|
||||||
VALIDATOR_TOOLS_DECLARATION,
|
VALIDATOR_TOOLS_DECLARATION,
|
||||||
capture_screenshot_and_parse_ui,
|
capture_screenshot_and_parse_ui,
|
||||||
@@ -66,7 +78,8 @@ from artemis.graph.perception import _check_injected_instruction_file
|
|||||||
from artemis.graph.state import State
|
from artemis.graph.state import State
|
||||||
from artemis.llm.structured import ParseFailure, parse_structured
|
from artemis.llm.structured import ParseFailure, parse_structured
|
||||||
from artemis.mcp.action_executor import McpActionExecutor
|
from artemis.mcp.action_executor import McpActionExecutor
|
||||||
from artemis.memory.transcript import PRO_UI_LIST_MARKER, TranscriptLedger
|
from artemis.mcp.observation import observe
|
||||||
|
from artemis.memory.transcript import PRO_UI_LIST_MARKER, TranscriptLedger, mark_ephemeral
|
||||||
from artemis.services.llm import (
|
from artemis.services.llm import (
|
||||||
RobustChatModelWrapper,
|
RobustChatModelWrapper,
|
||||||
acomplete,
|
acomplete,
|
||||||
@@ -85,12 +98,6 @@ from artemis.utils.logger import get_logger
|
|||||||
|
|
||||||
logger = get_logger(__name__)
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
#: Reminder carried by the first observation only (the system prompt states the rule).
|
|
||||||
_REASONING_FIRST_REMINDER = (
|
|
||||||
"CRITICAL RULE: In every single turn, you MUST FIRST output a natural"
|
|
||||||
" language reasoning/explanation paragraph BEFORE invoking any tool call."
|
|
||||||
)
|
|
||||||
|
|
||||||
_NO_TOOL_CALL_NOTICE = (
|
_NO_TOOL_CALL_NOTICE = (
|
||||||
"You did not call any tools last turn. Please make progress by calling an"
|
"You did not call any tools last turn. Please make progress by calling an"
|
||||||
" action tool or 'report_task_status'."
|
" action tool or 'report_task_status'."
|
||||||
@@ -116,17 +123,17 @@ class _TurnRecord:
|
|||||||
def result(self) -> dict | None:
|
def result(self) -> dict | None:
|
||||||
"""The turn's execution result in the validator-report shape.
|
"""The turn's execution result in the validator-report shape.
|
||||||
|
|
||||||
``None`` when no device action ran (helper-only turns have no result
|
``None`` unless an action failed: Flash executes synchronously and each
|
||||||
message, as in Pro).
|
action's tool message already carries its outcome, so a result message
|
||||||
|
would only repeat it. A failure is worth its own message because it
|
||||||
|
carries the error the turn ended on.
|
||||||
"""
|
"""
|
||||||
if not self.actions:
|
|
||||||
return None
|
|
||||||
for name, status, text in self.actions:
|
for name, status, text in self.actions:
|
||||||
if status != "success":
|
if status != "success":
|
||||||
# Executor messages name the action themselves ("Error
|
# Executor messages name the action themselves ("Error
|
||||||
# executing click: ..."); prefix only the ones that don't.
|
# executing click: ..."); prefix only the ones that don't.
|
||||||
return {"status": "failed", "error": text if name in text else f"{name}: {text}"}
|
return {"status": "failed", "error": text if name in text else f"{name}: {text}"}
|
||||||
return {"status": "dispatched"}
|
return None
|
||||||
|
|
||||||
|
|
||||||
class FlashRunner:
|
class FlashRunner:
|
||||||
@@ -164,7 +171,7 @@ class FlashRunner:
|
|||||||
if self.step_summarizer_cfg.enabled
|
if self.step_summarizer_cfg.enabled
|
||||||
else None
|
else None
|
||||||
)
|
)
|
||||||
# Publish the service on the composition root (ctx.step_memory, M2) so
|
# Publish the service on the composition root (ctx.step_memory) so
|
||||||
# any co-resident consumer shares this run's summary runtime.
|
# any co-resident consumer shares this run's summary runtime.
|
||||||
if self.summarizer is not None and getattr(ctx, "step_memory", None) is None:
|
if self.summarizer is not None and getattr(ctx, "step_memory", None) is None:
|
||||||
try:
|
try:
|
||||||
@@ -187,7 +194,13 @@ class FlashRunner:
|
|||||||
return getattr(setup, "video_recording_tools_enabled", False) is True
|
return getattr(setup, "video_recording_tools_enabled", False) is True
|
||||||
|
|
||||||
def _get_tools(self) -> list:
|
def _get_tools(self) -> list:
|
||||||
tools = list(VALIDATOR_TOOLS_DECLARATION)
|
# Flash has no tool that writes notes and its prompt never teaches
|
||||||
|
# them, so the Validator set's note readers are left out here.
|
||||||
|
tools = [
|
||||||
|
t
|
||||||
|
for t in VALIDATOR_TOOLS_DECLARATION
|
||||||
|
if t is not READ_NOTE_TOOL and t is not LIST_NOTES_TOOL
|
||||||
|
]
|
||||||
tools.insert(1, CLICK_SEQUENCE_TOOL)
|
tools.insert(1, CLICK_SEQUENCE_TOOL)
|
||||||
tools.append(ASK_EXPLORER_TOOL)
|
tools.append(ASK_EXPLORER_TOOL)
|
||||||
# Helper tools shared with the Pro operator: the history tools are
|
# Helper tools shared with the Pro operator: the history tools are
|
||||||
@@ -235,6 +248,9 @@ class FlashRunner:
|
|||||||
image_scrub_depth=getattr(cfg, "image_scrub_depth", 3),
|
image_scrub_depth=getattr(cfg, "image_scrub_depth", 3),
|
||||||
pending_grace_steps=getattr(cfg, "pending_grace_steps", 3),
|
pending_grace_steps=getattr(cfg, "pending_grace_steps", 3),
|
||||||
xml_scrub_depth=getattr(cfg, "xml_scrub_depth", 1),
|
xml_scrub_depth=getattr(cfg, "xml_scrub_depth", 1),
|
||||||
|
image_scrub_depth_relaxed=getattr(cfg, "image_scrub_depth_relaxed", None),
|
||||||
|
context_budget_tokens=getattr(cfg, "context_budget_tokens", None),
|
||||||
|
start_ratio=getattr(cfg, "start_ratio", None),
|
||||||
session_start=session_start if isinstance(session_start, (int, float)) else None,
|
session_start=session_start if isinstance(session_start, (int, float)) else None,
|
||||||
)
|
)
|
||||||
if engine is not None:
|
if engine is not None:
|
||||||
@@ -286,14 +302,19 @@ class FlashRunner:
|
|||||||
# Per-turn helpers (observe / think)
|
# Per-turn helpers (observe / think)
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
async def _read_injected_instruction(self) -> tuple[str | None, str | None]:
|
async def _read_injected_instruction(
|
||||||
"""Consumes this turn's injected instruction: ``(instruction, notice)``.
|
self, offset_label: str | None = None
|
||||||
|
) -> tuple[str | None, UserGuidance | None]:
|
||||||
|
"""Consumes this turn's injected instruction: ``(instruction, guidance)``.
|
||||||
|
|
||||||
``instruction`` is the user's text verbatim — what the step record,
|
``instruction`` is the user's text verbatim — what the step record,
|
||||||
the chunk ledger and the visual-transition lens receive (the same
|
the chunk ledger and the visual-transition lens receive (the same
|
||||||
shape Pro stamps). ``notice`` wraps it with the operator-facing
|
shape Pro stamps). ``guidance`` is the shared user-guidance pair
|
||||||
directive and rides only on the observation tail. Both are None
|
(:func:`render_user_guidance`, the Pro rendering without a plan to
|
||||||
when nothing was injected.
|
edit): the ephemeral ``--- User Guidance ---`` wrapper and the
|
||||||
|
persistent verbatim body, both riding on the observation tail.
|
||||||
|
``offset_label`` stamps the body with the session offset. Both are
|
||||||
|
None when nothing was injected.
|
||||||
"""
|
"""
|
||||||
if not (self.ctx.data_engine and self.ctx.data_engine.base_dir):
|
if not (self.ctx.data_engine and self.ctx.data_engine.base_dir):
|
||||||
return None, None
|
return None, None
|
||||||
@@ -308,18 +329,13 @@ class FlashRunner:
|
|||||||
if not (injected_payload and injected_payload.get("instruction")):
|
if not (injected_payload and injected_payload.get("instruction")):
|
||||||
return None, None
|
return None, None
|
||||||
instruction = str(injected_payload["instruction"])
|
instruction = str(injected_payload["instruction"])
|
||||||
notice = (
|
guidance = render_user_guidance(
|
||||||
"[REAL-TIME INJECTED INSTRUCTION from user]:"
|
instruction,
|
||||||
f" {instruction}\nYou MUST immediately"
|
release_loop=bool(injected_payload.get("release_loop")),
|
||||||
" follow this instruction and adjust your plan/actions."
|
has_plan=False,
|
||||||
|
offset_label=offset_label,
|
||||||
)
|
)
|
||||||
if injected_payload.get("release_loop"):
|
return instruction, guidance
|
||||||
notice += (
|
|
||||||
"\nThe user has explicitly authorized stopping any"
|
|
||||||
" ongoing monitoring loop; you may now wrap up and"
|
|
||||||
" complete the task."
|
|
||||||
)
|
|
||||||
return instruction, notice
|
|
||||||
|
|
||||||
def _build_tail(
|
def _build_tail(
|
||||||
self,
|
self,
|
||||||
@@ -328,19 +344,28 @@ class FlashRunner:
|
|||||||
img_bytes,
|
img_bytes,
|
||||||
xml_list,
|
xml_list,
|
||||||
*,
|
*,
|
||||||
injected: str | None = None,
|
injected: UserGuidance | None = None,
|
||||||
notices: list[str] | None = None,
|
notices: list[str] | None = None,
|
||||||
is_final: bool = False,
|
is_final: bool = False,
|
||||||
|
previous_turn_silent: bool = False,
|
||||||
) -> HumanMessage:
|
) -> HumanMessage:
|
||||||
"""Builds this turn's observation tail (Pro observation shape).
|
"""Builds this turn's observation tail (Pro observation shape).
|
||||||
|
|
||||||
Header, screenshot and UI list carry the same markers as the Pro
|
Header, screenshot and UI list carry the same markers as the Pro
|
||||||
operator's tail so the ledger's scrub edge treats both alike.
|
operator's tail so the ledger's scrub edge treats both alike. The
|
||||||
|
objective is stated once, in the system prompt; the tail never repeats
|
||||||
|
it. Every per-turn notice (nudges, the user-guidance wrapper, the
|
||||||
|
final-turn warning, the reasoning reminder after a silent turn) is
|
||||||
|
marked ephemeral so it leaves the transcript with the recent window.
|
||||||
|
The verbatim body of an injected instruction is the one exception: it
|
||||||
|
is a regular block, so a standing instruction stays in the active
|
||||||
|
window until the turn is chunked (the chunk's user lines carry it on).
|
||||||
"""
|
"""
|
||||||
blocks: list[dict] = []
|
# ``turns`` is kept for call-site symmetry: the tail has the same shape
|
||||||
if turns == 1:
|
# on every turn.
|
||||||
blocks.append({"type": "text", "text": f"Your objective is: {self.goal}"})
|
blocks: list[dict] = [
|
||||||
blocks.append({"type": "text", "text": f"# CURRENT OBSERVATION [{ledger.elapsed_label()}]"})
|
{"type": "text", "text": f"# CURRENT OBSERVATION [{ledger.elapsed_label()}]"}
|
||||||
|
]
|
||||||
if img_bytes:
|
if img_bytes:
|
||||||
img_b64 = base64.b64encode(img_bytes).decode("utf-8")
|
img_b64 = base64.b64encode(img_bytes).decode("utf-8")
|
||||||
blocks.append({"type": "text", "text": "--- Current Screenshot ---"})
|
blocks.append({"type": "text", "text": "--- Current Screenshot ---"})
|
||||||
@@ -349,15 +374,26 @@ class FlashRunner:
|
|||||||
)
|
)
|
||||||
if xml_list:
|
if xml_list:
|
||||||
blocks.append({"type": "text", "text": f"{PRO_UI_LIST_MARKER}\n{xml_list}"})
|
blocks.append({"type": "text", "text": f"{PRO_UI_LIST_MARKER}\n{xml_list}"})
|
||||||
for notice in notices or []:
|
ephemeral: list[int] = []
|
||||||
blocks.append({"type": "text", "text": notice})
|
# (text, ephemeral) in tail order.
|
||||||
|
per_turn: list[tuple[str, bool]] = [(text, True) for text in (notices or [])]
|
||||||
if injected:
|
if injected:
|
||||||
blocks.append({"type": "text", "text": injected})
|
# The wrapper is per-turn; the verbatim body right after it is a
|
||||||
|
# regular block that stays until the turn is chunked.
|
||||||
|
per_turn.append((injected.wrapper, True))
|
||||||
|
per_turn.append((injected.body, False))
|
||||||
if is_final:
|
if is_final:
|
||||||
blocks.append({"type": "text", "text": _FINAL_TURN_WARNING})
|
per_turn.append((_FINAL_TURN_WARNING, True))
|
||||||
if turns == 1:
|
if previous_turn_silent:
|
||||||
blocks.append({"type": "text", "text": _REASONING_FIRST_REMINDER})
|
per_turn.append((REASONING_REMINDER, True))
|
||||||
return HumanMessage(content=blocks)
|
for text, is_ephemeral in per_turn:
|
||||||
|
if is_ephemeral:
|
||||||
|
ephemeral.append(len(blocks))
|
||||||
|
blocks.append({"type": "text", "text": text})
|
||||||
|
tail = HumanMessage(content=blocks)
|
||||||
|
if ephemeral:
|
||||||
|
mark_ephemeral(tail, ephemeral)
|
||||||
|
return tail
|
||||||
|
|
||||||
def _extract_response_text(self, response) -> str:
|
def _extract_response_text(self, response) -> str:
|
||||||
"""Extracts the natural-language thought text from the model response."""
|
"""Extracts the natural-language thought text from the model response."""
|
||||||
@@ -368,6 +404,22 @@ class FlashRunner:
|
|||||||
)
|
)
|
||||||
return raw_text
|
return raw_text
|
||||||
|
|
||||||
|
def _extract_response_thinking(self, response) -> str:
|
||||||
|
"""Extracts the model's native thinking blocks (thought summaries), if any.
|
||||||
|
|
||||||
|
Providers running with ``include_thoughts`` return them as ``thinking``
|
||||||
|
content blocks next to the ``text`` blocks; they are recorded on the step
|
||||||
|
as ``operator_native_thinking`` and never fed back as reply text.
|
||||||
|
"""
|
||||||
|
if not isinstance(response.content, list):
|
||||||
|
return ""
|
||||||
|
parts = [
|
||||||
|
str(b.get("thinking") or "")
|
||||||
|
for b in response.content
|
||||||
|
if isinstance(b, dict) and b.get("type") == "thinking"
|
||||||
|
]
|
||||||
|
return "\n".join(p for p in parts if p.strip())
|
||||||
|
|
||||||
def _token_usage_from_response(self, response) -> dict | None:
|
def _token_usage_from_response(self, response) -> dict | None:
|
||||||
"""Extracts token usage metadata from the model response, if present."""
|
"""Extracts token usage metadata from the model response, if present."""
|
||||||
step_token_usage = None
|
step_token_usage = None
|
||||||
@@ -435,12 +487,20 @@ class FlashRunner:
|
|||||||
"total_tokens": max(1, prompt_tokens + completion_tokens),
|
"total_tokens": max(1, prompt_tokens + completion_tokens),
|
||||||
}
|
}
|
||||||
|
|
||||||
def _resolve_token_usage(self, response, messages: list[BaseMessage], raw_text: str) -> dict:
|
def _resolve_token_usage(
|
||||||
"""Extracts token usage from response metadata, estimating as fallback."""
|
self, response, messages: list[BaseMessage], raw_text: str
|
||||||
|
) -> tuple[dict, bool]:
|
||||||
|
"""Token usage for the call and whether the provider measured it.
|
||||||
|
|
||||||
|
Falls back to an estimate when the response carries no usage
|
||||||
|
metadata; the flag lets the caller keep an estimate out of the
|
||||||
|
ledger's chars-per-token calibration (an estimate is itself derived
|
||||||
|
from character counts, so calibrating on it would be circular).
|
||||||
|
"""
|
||||||
step_token_usage = self._token_usage_from_response(response)
|
step_token_usage = self._token_usage_from_response(response)
|
||||||
if not step_token_usage or step_token_usage.get("total_tokens", 0) <= 0:
|
if step_token_usage and step_token_usage.get("total_tokens", 0) > 0:
|
||||||
step_token_usage = self._estimate_token_usage(messages, raw_text)
|
return step_token_usage, True
|
||||||
return step_token_usage
|
return self._estimate_token_usage(messages, raw_text), False
|
||||||
|
|
||||||
def _record_llm_trace(self, step_token_usage: dict, raw_text: str) -> None:
|
def _record_llm_trace(self, step_token_usage: dict, raw_text: str) -> None:
|
||||||
"""Records the llm_call trace for this turn in the DataEngine."""
|
"""Records the llm_call trace for this turn in the DataEngine."""
|
||||||
@@ -490,6 +550,7 @@ class FlashRunner:
|
|||||||
pre_screenshot_bytes,
|
pre_screenshot_bytes,
|
||||||
xml_list,
|
xml_list,
|
||||||
messages: list[BaseMessage],
|
messages: list[BaseMessage],
|
||||||
|
native_text: str | None = None,
|
||||||
) -> dict:
|
) -> dict:
|
||||||
"""Records the final step, acknowledges the tool call, and flushes."""
|
"""Records the final step, acknowledges the tool call, and flushes."""
|
||||||
final_report = args
|
final_report = args
|
||||||
@@ -502,6 +563,7 @@ class FlashRunner:
|
|||||||
ui_tree=xml_list,
|
ui_tree=xml_list,
|
||||||
action_taken={"action": "report_task_status", "args": args},
|
action_taken={"action": "report_task_status", "args": args},
|
||||||
operator_raw_thinking=raw_text,
|
operator_raw_thinking=raw_text,
|
||||||
|
operator_native_thinking=native_text or None,
|
||||||
last_execution_result={"result": "Task completed with final report."},
|
last_execution_result={"result": "Task completed with final report."},
|
||||||
extra_metadata={"token_usage": step_token_usage},
|
extra_metadata={"token_usage": step_token_usage},
|
||||||
)
|
)
|
||||||
@@ -519,25 +581,56 @@ class FlashRunner:
|
|||||||
await self.summarizer.flush()
|
await self.summarizer.flush()
|
||||||
return final_report
|
return final_report
|
||||||
|
|
||||||
async def _capture_post_screenshot(self, exec_result, name: str, action_names):
|
async def _capture_post_screenshot(
|
||||||
"""Returns the post-action screenshot, capturing a fallback if needed."""
|
self, exec_result, name: str, action_names, state: State | None = None
|
||||||
|
):
|
||||||
|
"""Returns the post-action screenshot, capturing a fallback if needed.
|
||||||
|
|
||||||
|
When a device action produced no observation (a failed action, an
|
||||||
|
unknown tool, ...) the screen is observed through the very same
|
||||||
|
pipeline the executor's success path uses (:func:`observe`), so the
|
||||||
|
next observation tail carries the indexed minimal UI list and
|
||||||
|
``state`` gets the same refreshed screenshot path and indexed
|
||||||
|
elements. Nothing is captured for non-action tools.
|
||||||
|
"""
|
||||||
post_img_bytes = exec_result.screenshot_bytes
|
post_img_bytes = exec_result.screenshot_bytes
|
||||||
if not post_img_bytes and name in action_names:
|
if post_img_bytes or name not in action_names:
|
||||||
|
return post_img_bytes
|
||||||
try:
|
try:
|
||||||
controller = UnifiedMobileController(self.ctx)
|
obs, screenshot_bytes = await observe(self.ctx, self.controller, settle_ms=0)
|
||||||
screen_data = await controller.get_screen_data()
|
|
||||||
post_img_bytes = base64.b64decode(screen_data.base64)
|
|
||||||
if not exec_result.ui_elements_text:
|
|
||||||
exec_result.ui_elements_text = screen_data.elements
|
|
||||||
except Exception as shot_err:
|
except Exception as shot_err:
|
||||||
logger.warning(f"Failed to capture fallback screenshot in FlashRunner: {shot_err}")
|
logger.warning(f"Failed to capture fallback screenshot in FlashRunner: {shot_err}")
|
||||||
return post_img_bytes
|
return post_img_bytes
|
||||||
|
if not obs.ok:
|
||||||
|
logger.warning(f"Failed to capture fallback screenshot in FlashRunner: {obs.message}")
|
||||||
|
return post_img_bytes
|
||||||
|
post_img_bytes = screenshot_bytes
|
||||||
|
if not exec_result.ui_elements_text:
|
||||||
|
exec_result.ui_elements_text = obs.elements_text
|
||||||
|
if not exec_result.screenshot_path and obs.screenshot_path:
|
||||||
|
exec_result.screenshot_path = obs.screenshot_path
|
||||||
|
if state is not None:
|
||||||
|
if obs.hierarchy_ok:
|
||||||
|
state.indexed_points = [el["center"] for el in obs.elements]
|
||||||
|
state.indexed_elements = obs.elements
|
||||||
|
if obs.screenshot_path:
|
||||||
|
state.latest_screenshot = obs.screenshot_path
|
||||||
|
return post_img_bytes
|
||||||
|
|
||||||
def _extract_normalized_coordinates(self, name: str, args: dict):
|
def _extract_normalized_coordinates(self, name: str, args: dict, resolved=None):
|
||||||
"""Extracts and enriches coordinate metadata for the recorded action."""
|
"""Extracts and enriches coordinate metadata for the recorded action.
|
||||||
|
|
||||||
|
``resolved`` is the normalized point the executor sent a point action
|
||||||
|
to; it is the coordinate of record when the model addressed an element
|
||||||
|
by index (the arguments then carry no point at all).
|
||||||
|
"""
|
||||||
norm_coords = None
|
norm_coords = None
|
||||||
norm_start = None
|
norm_start = None
|
||||||
norm_end = None
|
norm_end = None
|
||||||
|
if name in ("click", "tap", "long_press", "input_text") and (
|
||||||
|
isinstance(resolved, (list, tuple)) and len(resolved) == 2
|
||||||
|
):
|
||||||
|
return [int(resolved[0]), int(resolved[1])], None, None
|
||||||
if name == "swipe":
|
if name == "swipe":
|
||||||
kind, target_val, _ = parse_swipe_parameters(args)
|
kind, target_val, _ = parse_swipe_parameters(args)
|
||||||
if kind == "coords" and isinstance(target_val, list) and len(target_val) == 4:
|
if kind == "coords" and isinstance(target_val, list) and len(target_val) == 4:
|
||||||
@@ -567,6 +660,40 @@ class FlashRunner:
|
|||||||
norm_coords = [int(float(nums[0])), int(float(nums[1]))]
|
norm_coords = [int(float(nums[0])), int(float(nums[1]))]
|
||||||
return norm_coords, norm_start, norm_end
|
return norm_coords, norm_start, norm_end
|
||||||
|
|
||||||
|
def _build_action_record(self, name: str, args: dict, exec_result) -> dict:
|
||||||
|
"""The action as the step record and the visual lens see it.
|
||||||
|
|
||||||
|
Flash records the model's own 0–1000 coordinates verbatim (an index
|
||||||
|
target records the point the executor resolved it to); the explicit
|
||||||
|
space marker keeps every later normalization pass (agent-friendly
|
||||||
|
steps, MCP inspector) a no-op on this record. The tool arguments ride
|
||||||
|
along under ``args``, and the target semantics the executor validated
|
||||||
|
(observed element fields for an index, the model's own
|
||||||
|
``target_description`` for a point) land top level in the shape Pro
|
||||||
|
records.
|
||||||
|
"""
|
||||||
|
metadata = getattr(exec_result, "metadata", None) or {}
|
||||||
|
resolved = metadata.get("target_coordinates")
|
||||||
|
norm_coords, norm_start, norm_end = self._extract_normalized_coordinates(
|
||||||
|
name, args, resolved=resolved
|
||||||
|
)
|
||||||
|
given = args.get("target") or args.get("coordinates") or args.get("sequence")
|
||||||
|
if isinstance(given, (int, float)) and not isinstance(given, bool):
|
||||||
|
given = None # An element index is not a coordinate.
|
||||||
|
action_dict = {
|
||||||
|
"action": name,
|
||||||
|
"coordinates": given or norm_coords,
|
||||||
|
COORDINATE_SPACE_KEY: COORDINATE_SPACE_NORMALIZED,
|
||||||
|
"args": args,
|
||||||
|
}
|
||||||
|
if norm_coords:
|
||||||
|
action_dict["normalized_coordinates"] = norm_coords
|
||||||
|
if norm_start and norm_end:
|
||||||
|
action_dict["normalized_start_coordinates"] = norm_start
|
||||||
|
action_dict["normalized_end_coordinates"] = norm_end
|
||||||
|
action_dict.update(metadata.get("target_semantics") or {})
|
||||||
|
return action_dict
|
||||||
|
|
||||||
def _record_action_step(
|
def _record_action_step(
|
||||||
self,
|
self,
|
||||||
name: str,
|
name: str,
|
||||||
@@ -578,6 +705,8 @@ class FlashRunner:
|
|||||||
xml_list,
|
xml_list,
|
||||||
post_img_bytes,
|
post_img_bytes,
|
||||||
injected: str | None = None,
|
injected: str | None = None,
|
||||||
|
native_text: str | None = None,
|
||||||
|
action_dict: dict | None = None,
|
||||||
):
|
):
|
||||||
"""Records telemetry / step in DataEngine; returns the step id or None."""
|
"""Records telemetry / step in DataEngine; returns the step id or None."""
|
||||||
recorded_step_id = None
|
recorded_step_id = None
|
||||||
@@ -585,31 +714,8 @@ class FlashRunner:
|
|||||||
if self.ctx.data_engine.current_step_id is None:
|
if self.ctx.data_engine.current_step_id is None:
|
||||||
self.ctx.data_engine.allocate_step_id()
|
self.ctx.data_engine.allocate_step_id()
|
||||||
|
|
||||||
norm_coords, norm_start, norm_end = self._extract_normalized_coordinates(name, args)
|
if action_dict is None:
|
||||||
|
action_dict = self._build_action_record(name, args, exec_result)
|
||||||
# Flash records the model's own 0–1000 coordinates verbatim; the
|
|
||||||
# explicit space marker keeps every later normalization pass
|
|
||||||
# (agent-friendly steps, MCP inspector) a no-op on this record.
|
|
||||||
action_dict = {
|
|
||||||
"action": name,
|
|
||||||
"coordinates": (
|
|
||||||
args.get("target")
|
|
||||||
or args.get("coordinates")
|
|
||||||
or args.get("sequence")
|
|
||||||
or norm_coords
|
|
||||||
),
|
|
||||||
COORDINATE_SPACE_KEY: COORDINATE_SPACE_NORMALIZED,
|
|
||||||
"args": args,
|
|
||||||
}
|
|
||||||
if norm_coords:
|
|
||||||
action_dict["normalized_coordinates"] = norm_coords
|
|
||||||
if norm_start and norm_end:
|
|
||||||
action_dict["normalized_start_coordinates"] = norm_start
|
|
||||||
action_dict["normalized_end_coordinates"] = norm_end
|
|
||||||
|
|
||||||
# The model's own statement of what it aimed at (target_description),
|
|
||||||
# validated and shaped by the executor into the fields Pro records.
|
|
||||||
action_dict.update((exec_result.metadata or {}).get("target_semantics") or {})
|
|
||||||
|
|
||||||
succeeded = exec_result.status == "success"
|
succeeded = exec_result.status == "success"
|
||||||
last_execution_result = {
|
last_execution_result = {
|
||||||
@@ -631,6 +737,7 @@ class FlashRunner:
|
|||||||
ui_tree=(exec_result.ui_elements_text or xml_list),
|
ui_tree=(exec_result.ui_elements_text or xml_list),
|
||||||
action_taken=action_dict,
|
action_taken=action_dict,
|
||||||
operator_raw_thinking=raw_text,
|
operator_raw_thinking=raw_text,
|
||||||
|
operator_native_thinking=native_text or None,
|
||||||
last_execution_result=last_execution_result,
|
last_execution_result=last_execution_result,
|
||||||
extra_metadata=extra_metadata,
|
extra_metadata=extra_metadata,
|
||||||
)
|
)
|
||||||
@@ -664,21 +771,35 @@ class FlashRunner:
|
|||||||
action_sequence: int,
|
action_sequence: int,
|
||||||
turn: _TurnRecord,
|
turn: _TurnRecord,
|
||||||
injected: str | None = None,
|
injected: str | None = None,
|
||||||
|
native_text: str | None = None,
|
||||||
|
index_elements: list[dict] | None = None,
|
||||||
):
|
):
|
||||||
"""Executes one tool call, records it, and updates loop state.
|
"""Executes one tool call, records it, and updates loop state.
|
||||||
|
|
||||||
The tool message carries the outcome text only; the post-action
|
The tool message carries the outcome text only; the post-action
|
||||||
screenshot and UI list become the next turn's observation tail.
|
screenshot and UI list become the next turn's observation tail.
|
||||||
Returns the (possibly updated) pre_screenshot_bytes, xml_list, and
|
``index_elements`` is the turn's element snapshot index targets
|
||||||
action_sequence for the next iteration.
|
resolve against (see :meth:`_process_tool_calls`); ``None`` resolves
|
||||||
|
against the state's current list. Returns the (possibly updated)
|
||||||
|
pre_screenshot_bytes, xml_list, and action_sequence for the next
|
||||||
|
iteration.
|
||||||
"""
|
"""
|
||||||
# Dynamic dispatch set: manifest device actions plus any backend
|
# Dynamic dispatch set: manifest device actions plus any backend
|
||||||
# extension tools, so extension steps are recorded like actions.
|
# extension tools, so extension steps are recorded like actions.
|
||||||
action_names = self.executor.action_tool_names
|
action_names = self.executor.action_tool_names
|
||||||
try:
|
try:
|
||||||
exec_result = await self.executor.execute(name, args, tc_id, state)
|
exec_result = await self.executor.execute(
|
||||||
|
name, args, tc_id, state, index_elements=index_elements
|
||||||
|
)
|
||||||
|
|
||||||
post_img_bytes = await self._capture_post_screenshot(exec_result, name, action_names)
|
post_img_bytes = await self._capture_post_screenshot(
|
||||||
|
exec_result, name, action_names, state
|
||||||
|
)
|
||||||
|
|
||||||
|
# One record shape for the DataEngine step and the visual lens.
|
||||||
|
action_dict = (
|
||||||
|
self._build_action_record(name, args, exec_result) if name in action_names else None
|
||||||
|
)
|
||||||
|
|
||||||
# Record telemetry / step in DataEngine
|
# Record telemetry / step in DataEngine
|
||||||
recorded_step_id = None
|
recorded_step_id = None
|
||||||
@@ -693,6 +814,8 @@ class FlashRunner:
|
|||||||
xml_list,
|
xml_list,
|
||||||
post_img_bytes,
|
post_img_bytes,
|
||||||
injected=injected,
|
injected=injected,
|
||||||
|
native_text=native_text,
|
||||||
|
action_dict=action_dict,
|
||||||
)
|
)
|
||||||
|
|
||||||
if name in action_names:
|
if name in action_names:
|
||||||
@@ -705,7 +828,9 @@ class FlashRunner:
|
|||||||
self.summarizer.dispatch(
|
self.summarizer.dispatch(
|
||||||
step_number=action_sequence,
|
step_number=action_sequence,
|
||||||
action_name=name,
|
action_name=name,
|
||||||
action_args=args,
|
# The recorded action minus its verb: the same shape the Pro
|
||||||
|
# summarizer hands the lens, so the phrase renders alike.
|
||||||
|
action_args={k: v for k, v in action_dict.items() if k != "action"},
|
||||||
pre_img_bytes=pre_screenshot_bytes,
|
pre_img_bytes=pre_screenshot_bytes,
|
||||||
post_img_bytes=post_img_bytes,
|
post_img_bytes=post_img_bytes,
|
||||||
exec_outcome=exec_result.text_summary,
|
exec_outcome=exec_result.text_summary,
|
||||||
@@ -780,12 +905,33 @@ class FlashRunner:
|
|||||||
action_sequence: int,
|
action_sequence: int,
|
||||||
turn: _TurnRecord,
|
turn: _TurnRecord,
|
||||||
injected: str | None = None,
|
injected: str | None = None,
|
||||||
|
native_text: str | None = None,
|
||||||
):
|
):
|
||||||
"""Dispatches the turn's tool calls.
|
"""Dispatches the turn's tool calls.
|
||||||
|
|
||||||
|
Every index target of the turn resolves against the element list the
|
||||||
|
model decided on: the list of the observation this turn was built
|
||||||
|
from, snapshotted here before the first action. Each action still
|
||||||
|
refreshes ``state.indexed_elements`` (that refreshed list is what the
|
||||||
|
next observation shows), so without the snapshot ``click(5)`` after
|
||||||
|
``click(3)`` would be resolved against the screen after the first
|
||||||
|
click.
|
||||||
|
|
||||||
|
The snapshot is the very list object the observation left on the
|
||||||
|
state (not a copy), and observations replace the state's list rather
|
||||||
|
than mutating it, so ``ask_explorer`` candidates registered before
|
||||||
|
any action land in the snapshot by construction. When a helper tool
|
||||||
|
appends candidates after an action has already replaced the state's
|
||||||
|
list, the explorer numbered them against that replaced list, so the
|
||||||
|
snapshot is re-taken from it: indices the explorer just handed out
|
||||||
|
resolve correctly, at the cost of the pre-action numbering (mixing
|
||||||
|
the two in one turn is the model's error, not a supported case).
|
||||||
|
|
||||||
Returns (final_report, pre_screenshot_bytes, xml_list, action_sequence);
|
Returns (final_report, pre_screenshot_bytes, xml_list, action_sequence);
|
||||||
final_report is non-None only when 'report_task_status' was called.
|
final_report is non-None only when 'report_task_status' was called.
|
||||||
"""
|
"""
|
||||||
|
action_names = self.executor.action_tool_names
|
||||||
|
index_elements = self._turn_index_snapshot(state)
|
||||||
for tc in tool_calls:
|
for tc in tool_calls:
|
||||||
name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
|
name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
|
||||||
args = tc.get("args") or {}
|
args = tc.get("args") or {}
|
||||||
@@ -802,9 +948,12 @@ class FlashRunner:
|
|||||||
pre_screenshot_bytes,
|
pre_screenshot_bytes,
|
||||||
xml_list,
|
xml_list,
|
||||||
messages,
|
messages,
|
||||||
|
native_text=native_text,
|
||||||
)
|
)
|
||||||
return final_report, pre_screenshot_bytes, xml_list, action_sequence
|
return final_report, pre_screenshot_bytes, xml_list, action_sequence
|
||||||
|
|
||||||
|
elements_before = getattr(state, "indexed_elements", None)
|
||||||
|
before_len = len(elements_before) if isinstance(elements_before, list) else 0
|
||||||
pre_screenshot_bytes, xml_list, action_sequence = await self._execute_and_record_action(
|
pre_screenshot_bytes, xml_list, action_sequence = await self._execute_and_record_action(
|
||||||
name,
|
name,
|
||||||
args,
|
args,
|
||||||
@@ -818,9 +967,46 @@ class FlashRunner:
|
|||||||
action_sequence,
|
action_sequence,
|
||||||
turn,
|
turn,
|
||||||
injected=injected,
|
injected=injected,
|
||||||
|
native_text=native_text,
|
||||||
|
index_elements=index_elements,
|
||||||
)
|
)
|
||||||
|
if name not in action_names:
|
||||||
|
index_elements = self._extend_index_snapshot(state, index_elements, before_len)
|
||||||
return None, pre_screenshot_bytes, xml_list, action_sequence
|
return None, pre_screenshot_bytes, xml_list, action_sequence
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _turn_index_snapshot(state: State) -> list[dict]:
|
||||||
|
"""The element list this turn's index targets resolve against.
|
||||||
|
|
||||||
|
The state's list object itself, so in-place appends (explorer
|
||||||
|
candidates) before any action join the snapshot; an empty list is
|
||||||
|
installed on the state when there is none, for the same reason.
|
||||||
|
"""
|
||||||
|
elements = getattr(state, "indexed_elements", None)
|
||||||
|
if not isinstance(elements, list):
|
||||||
|
elements = []
|
||||||
|
try:
|
||||||
|
state.indexed_elements = elements
|
||||||
|
except (AttributeError, TypeError, ValueError):
|
||||||
|
pass
|
||||||
|
return elements
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _extend_index_snapshot(state: State, snapshot: list[dict], before_len: int) -> list[dict]:
|
||||||
|
"""Re-takes the snapshot after a helper tool appended candidates to a
|
||||||
|
list that is no longer the snapshot (an action replaced it first).
|
||||||
|
|
||||||
|
Appends into the snapshot itself need nothing: the explorer numbered
|
||||||
|
them against the snapshot. Only growth of a different list object
|
||||||
|
moves the snapshot, since that is the list the new numbers refer to.
|
||||||
|
"""
|
||||||
|
current = getattr(state, "indexed_elements", None)
|
||||||
|
if not isinstance(current, list) or current is snapshot:
|
||||||
|
return snapshot
|
||||||
|
if len(current) > before_len:
|
||||||
|
return current
|
||||||
|
return snapshot
|
||||||
|
|
||||||
async def _prepare_conversation(self, state: State, tools_declaration: list):
|
async def _prepare_conversation(self, state: State, tools_declaration: list):
|
||||||
"""Installs the static prefix and captures the initial device state."""
|
"""Installs the static prefix and captures the initial device state."""
|
||||||
ledger = self._build_ledger()
|
ledger = self._build_ledger()
|
||||||
@@ -883,8 +1069,10 @@ class FlashRunner:
|
|||||||
previous_turn = None
|
previous_turn = None
|
||||||
|
|
||||||
# Check for real-time injected instructions: the verbatim text
|
# Check for real-time injected instructions: the verbatim text
|
||||||
# is stamped on the step, the wrapped notice goes to the tail.
|
# is stamped on the step, the guidance pair goes to the tail.
|
||||||
injected, injected_notice = await self._read_injected_instruction()
|
injected, injected_guidance = await self._read_injected_instruction(
|
||||||
|
ledger.elapsed_label()
|
||||||
|
)
|
||||||
|
|
||||||
# Tool restriction on the final turn (bounded loops only)
|
# Tool restriction on the final turn (bounded loops only)
|
||||||
is_final = limit is not None and turns == limit
|
is_final = limit is not None and turns == limit
|
||||||
@@ -893,9 +1081,12 @@ class FlashRunner:
|
|||||||
turns,
|
turns,
|
||||||
current_pre_screenshot_bytes,
|
current_pre_screenshot_bytes,
|
||||||
current_xml_list,
|
current_xml_list,
|
||||||
injected=injected_notice,
|
injected=injected_guidance,
|
||||||
notices=pending_notices,
|
notices=pending_notices,
|
||||||
is_final=is_final,
|
is_final=is_final,
|
||||||
|
# The ledger judged the turn it just committed: a bare tool
|
||||||
|
# call (or thinking only) earns a one-line reminder, no bounce.
|
||||||
|
previous_turn_silent=bool(getattr(ledger, "last_turn_silent", False)),
|
||||||
)
|
)
|
||||||
pending_notices = []
|
pending_notices = []
|
||||||
|
|
||||||
@@ -913,9 +1104,20 @@ class FlashRunner:
|
|||||||
|
|
||||||
# Extract thought and tool calls
|
# Extract thought and tool calls
|
||||||
raw_text = self._extract_response_text(response)
|
raw_text = self._extract_response_text(response)
|
||||||
|
native_text = self._extract_response_thinking(response)
|
||||||
|
|
||||||
# Extract token usage metadata from response
|
# Extract token usage metadata from response
|
||||||
step_token_usage = self._resolve_token_usage(response, messages, raw_text)
|
step_token_usage, measured = self._resolve_token_usage(response, messages, raw_text)
|
||||||
|
# The ledger's compaction thresholds read the runner's own prompt
|
||||||
|
# size, never the session-wide meter (which any agent overwrites).
|
||||||
|
# The prompt as sent (``messages`` minus the just-appended reply)
|
||||||
|
# rides along so the ledger can calibrate its chars-per-token
|
||||||
|
# ratio — only from a provider measurement, never from the
|
||||||
|
# estimate (same as the Pro operator).
|
||||||
|
ledger.record_prompt_tokens(
|
||||||
|
step_token_usage.get("prompt_tokens"),
|
||||||
|
messages=messages[:-1] if measured else None,
|
||||||
|
)
|
||||||
|
|
||||||
self._record_llm_trace(step_token_usage, raw_text)
|
self._record_llm_trace(step_token_usage, raw_text)
|
||||||
|
|
||||||
@@ -952,6 +1154,7 @@ class FlashRunner:
|
|||||||
action_sequence,
|
action_sequence,
|
||||||
turn,
|
turn,
|
||||||
injected=injected,
|
injected=injected,
|
||||||
|
native_text=native_text,
|
||||||
)
|
)
|
||||||
ledger.stage_turn(messages[turn_base:])
|
ledger.stage_turn(messages[turn_base:])
|
||||||
previous_turn = turn
|
previous_turn = turn
|
||||||
|
|||||||
@@ -260,24 +260,83 @@ class VisualStepSummarizer(StepMemoryService):
|
|||||||
def _action_phrase(action_name: str, action_args: dict[str, Any] | None) -> str:
|
def _action_phrase(action_name: str, action_args: dict[str, Any] | None) -> str:
|
||||||
"""The action as every other history reader sees it (``format_actions_clean``).
|
"""The action as every other history reader sees it (``format_actions_clean``).
|
||||||
|
|
||||||
A described coordinate target renders as ``'play button' (self-described)``,
|
``action_args`` is the recorded action minus its verb (what the Pro
|
||||||
an observed element as ``'Play'``; a Pro burst lists every member.
|
summarizer extracts from ``action_taken`` and what the Flash runner
|
||||||
|
builds from the same record shape). The action name always wins over
|
||||||
|
an argument of the same key: ``manage_app``'s own ``action="launch"``
|
||||||
|
argument becomes the ``intent`` the renderer reads, never the verb. A
|
||||||
|
described coordinate target renders as ``'play button'
|
||||||
|
(self-described)``, an observed element as ``'Play'``; a Pro burst
|
||||||
|
lists every member.
|
||||||
"""
|
"""
|
||||||
args = dict(action_args or {})
|
args = dict(action_args or {})
|
||||||
extra = args.pop("additional_actions", None)
|
extra = args.pop("additional_actions", None)
|
||||||
|
own_verb = args.pop("action", None)
|
||||||
|
if own_verb is not None and own_verb != action_name:
|
||||||
|
args.setdefault("intent", own_verb)
|
||||||
# Flash's coordinate swipe names its points ``start``/``end``; the
|
# Flash's coordinate swipe names its points ``start``/``end``; the
|
||||||
# history renderer reads ``start_coordinates``/``end_coordinates``.
|
# history renderer reads ``start_coordinates``/``end_coordinates``.
|
||||||
if action_name == "swipe" and "start" in args and "end" in args:
|
if action_name == "swipe":
|
||||||
args.setdefault("start_coordinates", args["start"])
|
raw = args.get("args") if isinstance(args.get("args"), dict) else {}
|
||||||
args.setdefault("end_coordinates", args["end"])
|
start = args.get("start", raw.get("start"))
|
||||||
|
end = args.get("end", raw.get("end"))
|
||||||
|
if start is not None and end is not None:
|
||||||
|
args.setdefault("start_coordinates", start)
|
||||||
|
args.setdefault("end_coordinates", end)
|
||||||
try:
|
try:
|
||||||
first = {"action": action_name, **args}
|
first = {**args, "action": action_name}
|
||||||
actions = [first, *extra] if isinstance(extra, list) and extra else first
|
actions = [first, *extra] if isinstance(extra, list) and extra else first
|
||||||
return format_actions_clean(actions)
|
return format_actions_clean(actions)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
logger.debug(f"Action phrase rendering fell back to raw args: {exc}", exc_info=True)
|
logger.debug(f"Action phrase rendering fell back to raw args: {exc}", exc_info=True)
|
||||||
return f"{action_name}({action_args})"
|
return f"{action_name}({action_args})"
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _verbatim_args(action_args: dict[str, Any] | None) -> dict[str, Any]:
|
||||||
|
"""The agent-facing arguments of the action.
|
||||||
|
|
||||||
|
A Flash record keeps the tool call's arguments under ``args``; a Pro
|
||||||
|
record is the arguments themselves. Bookkeeping keys never count.
|
||||||
|
"""
|
||||||
|
args = action_args or {}
|
||||||
|
raw = args.get("args")
|
||||||
|
if isinstance(raw, dict):
|
||||||
|
return dict(raw)
|
||||||
|
internal = {
|
||||||
|
"additional_actions",
|
||||||
|
"coordinate_space",
|
||||||
|
"normalized_coordinates",
|
||||||
|
"normalized_start_coordinates",
|
||||||
|
"normalized_end_coordinates",
|
||||||
|
}
|
||||||
|
return {k: v for k, v in args.items() if k not in internal}
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _phrase_carries(phrase: str, args: dict[str, Any]) -> bool:
|
||||||
|
"""Whether every argument value is visible in the rendered phrase.
|
||||||
|
|
||||||
|
Booleans and ``None`` never count (the phrase spells their effect, not
|
||||||
|
their value); everything else must appear verbatim, so an index
|
||||||
|
target, a repeat count or a duration the phrase folded away keeps the
|
||||||
|
verbatim argument line.
|
||||||
|
"""
|
||||||
|
haystack = phrase.lower()
|
||||||
|
|
||||||
|
def _shown(value: Any) -> bool:
|
||||||
|
if isinstance(value, (list, tuple)):
|
||||||
|
# A point renders whole (``at [880, 410]``); a sequence or a
|
||||||
|
# description list renders member by member.
|
||||||
|
return str(list(value)).lower() in haystack or (
|
||||||
|
bool(value) and all(_shown(v) for v in value)
|
||||||
|
)
|
||||||
|
return str(value).lower() in haystack
|
||||||
|
|
||||||
|
return all(
|
||||||
|
_shown(value)
|
||||||
|
for value in args.values()
|
||||||
|
if value is not None and not isinstance(value, bool)
|
||||||
|
)
|
||||||
|
|
||||||
def _meter_lens_call(self, response: Any) -> None:
|
def _meter_lens_call(self, response: Any) -> None:
|
||||||
"""Meter one raw-model lens call as an ``llm_usage`` trace, best-effort.
|
"""Meter one raw-model lens call as an ``llm_usage`` trace, best-effort.
|
||||||
|
|
||||||
@@ -338,11 +397,14 @@ class VisualStepSummarizer(StepMemoryService):
|
|||||||
dual = bool(pre_bytes) and bool(post_bytes)
|
dual = bool(pre_bytes) and bool(post_bytes)
|
||||||
template = self._prompt_template if dual else self._single_prompt_template
|
template = self._prompt_template if dual else self._single_prompt_template
|
||||||
rendered_prompt = Template(template).render(step_number=step_number)
|
rendered_prompt = Template(template).render(step_number=step_number)
|
||||||
lead_lines = [
|
phrase = self._action_phrase(action_name, action_args)
|
||||||
f"Step {step_number} Physical Action: {self._action_phrase(action_name, action_args)}",
|
lead_lines = [f"Step {step_number} Physical Action: {phrase}"]
|
||||||
f"Action arguments (verbatim): {action_name}({action_args})",
|
# The verbatim arguments are worth a line only when the phrase
|
||||||
f"Controller Outcome: {exec_outcome}",
|
# folded some of them away (an index target, a repeat count, ...).
|
||||||
]
|
verbatim = self._verbatim_args(action_args)
|
||||||
|
if verbatim and not self._phrase_carries(phrase, verbatim):
|
||||||
|
lead_lines.append(f"Action arguments (verbatim): {action_name}({verbatim})")
|
||||||
|
lead_lines.append(f"Controller Outcome: {exec_outcome}")
|
||||||
focus = input_data.get("focus")
|
focus = input_data.get("focus")
|
||||||
if isinstance(focus, dict) and focus:
|
if isinstance(focus, dict) and focus:
|
||||||
lead_lines.extend(["", render_focus_block(focus, dual=dual)])
|
lead_lines.extend(["", render_focus_block(focus, dual=dual)])
|
||||||
|
|||||||
@@ -1,3 +1,3 @@
|
|||||||
{
|
{
|
||||||
"main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n[% if \"wait_for_delay\" in available_tools %]- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n[% endif %]- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.\n 2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading state. This boundary probe must remain unidirectional; never reverse direction merely to prove completion. If the task declares an earlier exit condition (for example, find one verified match or process N items), that explicit condition may end the traversal without reaching the list boundary.\n 3. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n 4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content.\n- **Dynamic Monitoring & Reactive Loop Protocol (Monitor-Trigger-Action)**: When executing an active milestone matching `[Loop] <Goal> into note <TargetNote> (Exit: <ExitCondition>; Interval: <Interval>)` or `[Loop:continuous] <Goal> into note <TargetNote> (Interval: <Interval>)` (the milestone grammar itself is defined under Task Plan Grammar below), run this 5-step cyclic state machine:\n - **Step 1 (Baseline Sync)**: On the initial cycle, observe all visible entries/metrics in the primary monitoring view and save the full snaLine truncated
|
"main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n[% if \"wait_for_delay\" in available_tools %]- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n[% endif %]- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.\n 2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading state. This boundary probe must remain unidirectional; never reverse direction merely to prove completion. If the task declares an earlier exit condition (for example, find one verified match or process N items), that explicit condition may end the traversal without reaching the list boundary.\n 3. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n 4. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed with confirmed anchor continuity and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\", even if older execution history has been pruned. Resume from the last recorded anchor and direction. Only the single bounded recovery above, or a refreshed/mutated list, permits limited re-verification; otherwise continue the single pass over new content.\n- **Dynamic Monitoring & Reactive Loop Protocol (Monitor-Trigger-Action)**: When executing an active milestone matching `[Loop] <Goal> into note <TargetNote> (Exit: <ExitCondition>; Interval: <Interval>)` or `[Loop:continuous] <Goal> into note <TargetNote> (Interval: <Interval>)` (the milestone grammar itself is defined under Task Plan Grammar below), run this 5-step cyclic state machine:\n - **Step 1 (Baseline Sync)**: On the initial cycle, observe all visible entries/metrics in the primary monitoring view and save the full snaLine truncated
|
||||||
}
|
}
|
||||||
@@ -71,6 +71,9 @@ DEFERRING_TOOLS = {
|
|||||||
"ask_explorer",
|
"ask_explorer",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Bare key names press_key accepts (case-insensitive, optional KEYCODE_ prefix).
|
||||||
|
SUPPORTED_PRESS_KEYS = ("ENTER", "BACK", "HOME", "APP_SWITCH")
|
||||||
|
|
||||||
from artemis.agents.operator.prompts import (
|
from artemis.agents.operator.prompts import (
|
||||||
OPERATOR_MAX_TOOL_ITERATIONS,
|
OPERATOR_MAX_TOOL_ITERATIONS,
|
||||||
PromptBuilder,
|
PromptBuilder,
|
||||||
@@ -82,9 +85,9 @@ from artemis.agents.operator.prompts import (
|
|||||||
HistoricalStateHintPromptComponent,
|
HistoricalStateHintPromptComponent,
|
||||||
FeedbackPromptComponent,
|
FeedbackPromptComponent,
|
||||||
ExecutionIncidentPromptComponent,
|
ExecutionIncidentPromptComponent,
|
||||||
CheckItemsExplainerPromptComponent,
|
|
||||||
BackgroundTasksPromptComponent,
|
BackgroundTasksPromptComponent,
|
||||||
render_plan_ledger_bounce,
|
render_plan_ledger_bounce,
|
||||||
|
ReasoningReminderPromptComponent,
|
||||||
unwritten_action_streak,
|
unwritten_action_streak,
|
||||||
ToolLimitWarningPromptComponent,
|
ToolLimitWarningPromptComponent,
|
||||||
InjectedInstructionPromptComponent,
|
InjectedInstructionPromptComponent,
|
||||||
@@ -137,6 +140,24 @@ class OperatorNode:
|
|||||||
self._transcript_turn_base: int | None = None
|
self._transcript_turn_base: int | None = None
|
||||||
self._turn_plan_baseline: str | None = None
|
self._turn_plan_baseline: str | None = None
|
||||||
self._turn_ledger_streak: int = 0
|
self._turn_ledger_streak: int = 0
|
||||||
|
# Whether the previous turn's replies carried no visible reasoning text;
|
||||||
|
# the next observation tail then carries a one-line reminder (no extra
|
||||||
|
# model call). Legacy-path fallback only: with a transcript ledger the
|
||||||
|
# ledger's own ``last_turn_silent`` is authoritative.
|
||||||
|
self._previous_turn_silent: bool = False
|
||||||
|
|
||||||
|
def _previous_turn_was_silent(self) -> bool:
|
||||||
|
"""Whether the previous turn carried no visible reasoning text.
|
||||||
|
|
||||||
|
The transcript ledger judges its committed turns itself; the local flag
|
||||||
|
only serves the legacy path, which has no ledger.
|
||||||
|
"""
|
||||||
|
from artemis.memory import TranscriptLedger
|
||||||
|
|
||||||
|
ledger = getattr(self.ctx, "transcript_ledger", None)
|
||||||
|
if isinstance(ledger, TranscriptLedger):
|
||||||
|
return bool(ledger.last_turn_silent)
|
||||||
|
return self._previous_turn_silent
|
||||||
|
|
||||||
def _available_device_actions(self) -> frozenset[str]:
|
def _available_device_actions(self) -> frozenset[str]:
|
||||||
"""Device actions the installed actuator backend provides.
|
"""Device actions the installed actuator backend provides.
|
||||||
@@ -225,7 +246,6 @@ class OperatorNode:
|
|||||||
TemplatePromptComponent(),
|
TemplatePromptComponent(),
|
||||||
{"template_name": "main_template"},
|
{"template_name": "main_template"},
|
||||||
),
|
),
|
||||||
(CheckItemsExplainerPromptComponent(), {}),
|
|
||||||
(ExecutionIncidentPromptComponent(), {}),
|
(ExecutionIncidentPromptComponent(), {}),
|
||||||
(ObservationPromptComponent(), {}),
|
(ObservationPromptComponent(), {}),
|
||||||
(ScreenshotSimilarityPromptComponent(), {}),
|
(ScreenshotSimilarityPromptComponent(), {}),
|
||||||
@@ -233,10 +253,14 @@ class OperatorNode:
|
|||||||
(InjectedInstructionPromptComponent(), {}),
|
(InjectedInstructionPromptComponent(), {}),
|
||||||
(BackgroundTasksPromptComponent(), {}),
|
(BackgroundTasksPromptComponent(), {}),
|
||||||
(FeedbackPromptComponent(), {}),
|
(FeedbackPromptComponent(), {}),
|
||||||
|
(ReasoningReminderPromptComponent(), {}),
|
||||||
(ToolLimitWarningPromptComponent(), {}),
|
(ToolLimitWarningPromptComponent(), {}),
|
||||||
]
|
]
|
||||||
else:
|
else:
|
||||||
components = [(c, {}) for c in components] + [(ToolLimitWarningPromptComponent(), {})]
|
components = [(c, {}) for c in components] + [
|
||||||
|
(ReasoningReminderPromptComponent(), {}),
|
||||||
|
(ToolLimitWarningPromptComponent(), {}),
|
||||||
|
]
|
||||||
|
|
||||||
# Active and finished tasks are now passed as arguments
|
# Active and finished tasks are now passed as arguments
|
||||||
active_tasks = active_background_tasks or []
|
active_tasks = active_background_tasks or []
|
||||||
@@ -252,6 +276,7 @@ class OperatorNode:
|
|||||||
"current_step_num": current_step_num,
|
"current_step_num": current_step_num,
|
||||||
"active_background_tasks": active_tasks,
|
"active_background_tasks": active_tasks,
|
||||||
"newly_finished_tasks": newly_finished,
|
"newly_finished_tasks": newly_finished,
|
||||||
|
"previous_turn_silent": self._previous_turn_was_silent(),
|
||||||
"steps": steps,
|
"steps": steps,
|
||||||
}
|
}
|
||||||
kwargs_to_pass.update(extra_kwargs)
|
kwargs_to_pass.update(extra_kwargs)
|
||||||
@@ -280,6 +305,9 @@ class OperatorNode:
|
|||||||
image_scrub_depth=getattr(cfg, "image_scrub_depth", 3),
|
image_scrub_depth=getattr(cfg, "image_scrub_depth", 3),
|
||||||
pending_grace_steps=getattr(cfg, "pending_grace_steps", 3),
|
pending_grace_steps=getattr(cfg, "pending_grace_steps", 3),
|
||||||
xml_scrub_depth=getattr(cfg, "xml_scrub_depth", 1),
|
xml_scrub_depth=getattr(cfg, "xml_scrub_depth", 1),
|
||||||
|
image_scrub_depth_relaxed=getattr(cfg, "image_scrub_depth_relaxed", None),
|
||||||
|
context_budget_tokens=getattr(cfg, "context_budget_tokens", None),
|
||||||
|
start_ratio=getattr(cfg, "start_ratio", None),
|
||||||
session_start=session_start if isinstance(session_start, (int, float)) else None,
|
session_start=session_start if isinstance(session_start, (int, float)) else None,
|
||||||
)
|
)
|
||||||
# L2/L3 chunk compression (M3) rides the same flag; without a
|
# L2/L3 chunk compression (M3) rides the same flag; without a
|
||||||
@@ -337,6 +365,8 @@ class OperatorNode:
|
|||||||
"""
|
"""
|
||||||
from langchain_core.messages import HumanMessage
|
from langchain_core.messages import HumanMessage
|
||||||
|
|
||||||
|
from artemis.memory.transcript import mark_ephemeral
|
||||||
|
|
||||||
ledger = self._ensure_transcript_ledger(state)
|
ledger = self._ensure_transcript_ledger(state)
|
||||||
|
|
||||||
# 1. S region: rendered exactly once per session.
|
# 1. S region: rendered exactly once per session.
|
||||||
@@ -387,7 +417,6 @@ class OperatorNode:
|
|||||||
builder.add_human_content(f"# CURRENT OBSERVATION [{ledger.elapsed_label()}]")
|
builder.add_human_content(f"# CURRENT OBSERVATION [{ledger.elapsed_label()}]")
|
||||||
components = [
|
components = [
|
||||||
(PlanRecitationPromptComponent(), {}),
|
(PlanRecitationPromptComponent(), {}),
|
||||||
(CheckItemsExplainerPromptComponent(), {}),
|
|
||||||
(ExecutionIncidentPromptComponent(), {}),
|
(ExecutionIncidentPromptComponent(), {}),
|
||||||
(ObservationPromptComponent(), {}),
|
(ObservationPromptComponent(), {}),
|
||||||
(ScreenshotSimilarityPromptComponent(), {}),
|
(ScreenshotSimilarityPromptComponent(), {}),
|
||||||
@@ -395,6 +424,7 @@ class OperatorNode:
|
|||||||
(InjectedInstructionPromptComponent(), {}),
|
(InjectedInstructionPromptComponent(), {}),
|
||||||
(BackgroundTasksPromptComponent(), {}),
|
(BackgroundTasksPromptComponent(), {}),
|
||||||
(FeedbackPromptComponent(), {}),
|
(FeedbackPromptComponent(), {}),
|
||||||
|
(ReasoningReminderPromptComponent(), {}),
|
||||||
(ToolLimitWarningPromptComponent(), {}),
|
(ToolLimitWarningPromptComponent(), {}),
|
||||||
]
|
]
|
||||||
for component, extra_kwargs in components:
|
for component, extra_kwargs in components:
|
||||||
@@ -407,6 +437,7 @@ class OperatorNode:
|
|||||||
"active_background_tasks": active_background_tasks or [],
|
"active_background_tasks": active_background_tasks or [],
|
||||||
"newly_finished_tasks": newly_finished_tasks or [],
|
"newly_finished_tasks": newly_finished_tasks or [],
|
||||||
"steps": steps,
|
"steps": steps,
|
||||||
|
"previous_turn_silent": self._previous_turn_was_silent(),
|
||||||
}
|
}
|
||||||
kwargs_to_pass.update(extra_kwargs)
|
kwargs_to_pass.update(extra_kwargs)
|
||||||
await component(builder, state, self.ctx, **kwargs_to_pass)
|
await component(builder, state, self.ctx, **kwargs_to_pass)
|
||||||
@@ -418,6 +449,10 @@ class OperatorNode:
|
|||||||
else:
|
else:
|
||||||
tail_content.append(part)
|
tail_content.append(part)
|
||||||
tail = HumanMessage(content=tail_content)
|
tail = HumanMessage(content=tail_content)
|
||||||
|
# Per-turn notices (reminders, hints, guidance, incident blocks) are
|
||||||
|
# deleted by the scrub edge once the turn leaves the recent window.
|
||||||
|
if builder.ephemeral_indices:
|
||||||
|
mark_ephemeral(tail, builder.ephemeral_indices)
|
||||||
|
|
||||||
messages = ledger.render([tail])
|
messages = ledger.render([tail])
|
||||||
self._transcript_turn_base = len(messages) - 1
|
self._transcript_turn_base = len(messages) - 1
|
||||||
@@ -471,12 +506,27 @@ class OperatorNode:
|
|||||||
response = await invoke_llm_with_timeout_message(acomplete(bound_llm, current_messages))
|
response = await invoke_llm_with_timeout_message(acomplete(bound_llm, current_messages))
|
||||||
|
|
||||||
usage = usage_from_message(response)
|
usage = usage_from_message(response)
|
||||||
|
# The ledger's compaction thresholds read the operator's own
|
||||||
|
# prompt size, never the session-wide meter (which the
|
||||||
|
# Planner/Checker/sub-agent calls overwrite). The messages ride
|
||||||
|
# along so the ledger can calibrate its chars-per-token ratio.
|
||||||
|
transcript_ledger = getattr(self.ctx, "transcript_ledger", None)
|
||||||
if usage is not None:
|
if usage is not None:
|
||||||
logger.info(
|
logger.info(
|
||||||
f"LLM usage: prompt_tokens={usage['prompt_tokens']}"
|
f"LLM usage: prompt_tokens={usage['prompt_tokens']}"
|
||||||
f" (cached={usage['cached_tokens']}),"
|
f" (cached={usage['cached_tokens']}),"
|
||||||
f" completion_tokens={usage['completion_tokens']}"
|
f" completion_tokens={usage['completion_tokens']}"
|
||||||
)
|
)
|
||||||
|
if hasattr(transcript_ledger, "record_prompt_tokens"):
|
||||||
|
transcript_ledger.record_prompt_tokens(
|
||||||
|
usage.get("prompt_tokens"), messages=current_messages
|
||||||
|
)
|
||||||
|
elif hasattr(transcript_ledger, "record_prompt_tokens"):
|
||||||
|
# No usage metadata from the provider: record an estimate so
|
||||||
|
# the start gate and thresholds still see a context base.
|
||||||
|
from artemis.memory.transcript import estimate_prompt_tokens
|
||||||
|
|
||||||
|
transcript_ledger.record_prompt_tokens(estimate_prompt_tokens(current_messages))
|
||||||
|
|
||||||
if response.content:
|
if response.content:
|
||||||
if isinstance(response.content, str):
|
if isinstance(response.content, str):
|
||||||
@@ -638,18 +688,22 @@ class OperatorNode:
|
|||||||
"Scratchpad/helper tool call failed. Deferring"
|
"Scratchpad/helper tool call failed. Deferring"
|
||||||
f" screen actions: {other_tool_failure_msg}"
|
f" screen actions: {other_tool_failure_msg}"
|
||||||
)
|
)
|
||||||
for tc in action_calls:
|
# Every rejected call needs its own ToolMessage, but the
|
||||||
|
# full explanation travels once; the rest point at it.
|
||||||
|
for idx, tc in enumerate(action_calls):
|
||||||
tool_outputs.append(
|
tool_outputs.append(
|
||||||
ToolMessage(
|
ToolMessage(
|
||||||
tool_call_id=tc["id"],
|
tool_call_id=tc["id"],
|
||||||
|
name=normalize_name(tc["name"]),
|
||||||
content=(
|
content=(
|
||||||
"This screen action was rejected"
|
"Not executed: this screen action was"
|
||||||
" because an accompanying tool call"
|
" rejected because an accompanying tool"
|
||||||
" failed. Error:"
|
f" call failed. Error: {other_tool_failure_msg}."
|
||||||
f" {other_tool_failure_msg}. Currently,"
|
" No screen action ran this turn. Correct"
|
||||||
" no screen actions have been"
|
" the error, then re-issue the action."
|
||||||
" executed. Please correct the error"
|
if idx == 0
|
||||||
" and try again."
|
else "Not executed: see the accompanying"
|
||||||
|
" tool-call error above."
|
||||||
),
|
),
|
||||||
status="error",
|
status="error",
|
||||||
)
|
)
|
||||||
@@ -667,25 +721,23 @@ class OperatorNode:
|
|||||||
" Deferring terminal actions."
|
" Deferring terminal actions."
|
||||||
)
|
)
|
||||||
|
|
||||||
for tc in action_calls:
|
for idx, tc in enumerate(action_calls):
|
||||||
tool_outputs.append(
|
tool_outputs.append(
|
||||||
ToolMessage(
|
ToolMessage(
|
||||||
tool_call_id=tc["id"],
|
tool_call_id=tc["id"],
|
||||||
|
name=normalize_name(tc["name"]),
|
||||||
content=(
|
content=(
|
||||||
"Your pre-decision tools have been"
|
"Not executed: this screen action was"
|
||||||
" successfully processed. However, the"
|
" deferred because the same turn also"
|
||||||
" screen action you attempted to"
|
" called a result-dependent pre-decision"
|
||||||
" execute was rejected because you are"
|
" tool. No screen action ran this turn."
|
||||||
" not allowed to simultaneously"
|
" Read the tool results above, then"
|
||||||
" use result-dependent pre-decision"
|
" re-issue the action."
|
||||||
" tools and execute physical actions in"
|
if idx == 0
|
||||||
" the same turn."
|
else "Not executed: deferred for the same"
|
||||||
" Currently, no screen actions have"
|
" reason as the screen action above."
|
||||||
" been executed. Please review your"
|
|
||||||
" updated context and re-output your"
|
|
||||||
" intended screen action."
|
|
||||||
),
|
),
|
||||||
status="success",
|
status="error",
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -713,24 +765,36 @@ class OperatorNode:
|
|||||||
logger.info(f"Plan ledger gate: bouncing action. {bounce}")
|
logger.info(f"Plan ledger gate: bouncing action. {bounce}")
|
||||||
for tc in action_calls:
|
for tc in action_calls:
|
||||||
tool_outputs.append(
|
tool_outputs.append(
|
||||||
ToolMessage(tool_call_id=tc["id"], content=bounce, status="error")
|
ToolMessage(
|
||||||
|
tool_call_id=tc["id"],
|
||||||
|
name=normalize_name(tc["name"]),
|
||||||
|
content=bounce,
|
||||||
|
status="error",
|
||||||
|
)
|
||||||
)
|
)
|
||||||
action_calls_to_translate = []
|
action_calls_to_translate = []
|
||||||
elif len(action_calls) > burst_cap:
|
elif len(action_calls) > burst_cap:
|
||||||
# A multi-action turn is a fast-action burst that the Validator
|
# A multi-action turn is a fast-action burst that the Validator
|
||||||
# fires without the safety net; cap its length before it runs.
|
# fires without the safety net; cap its length before it runs.
|
||||||
validation_errors = True
|
validation_errors = True
|
||||||
for tc in action_calls:
|
burst_error = (
|
||||||
tool_outputs.append(
|
|
||||||
ToolMessage(
|
|
||||||
tool_call_id=tc["id"],
|
|
||||||
content=(
|
|
||||||
f"Error: {len(action_calls)} Turn-Ending Actions in one"
|
f"Error: {len(action_calls)} Turn-Ending Actions in one"
|
||||||
" turn exceed the fast-action burst limit of"
|
" turn exceed the fast-action burst limit of"
|
||||||
f" {burst_cap}. Nothing was executed. Re-issue at most"
|
f" {burst_cap}. Nothing was executed. Re-issue at most"
|
||||||
f" {burst_cap} actions (a burst is for sub-second"
|
f" {burst_cap} actions (a burst is for sub-second"
|
||||||
" sequences on transient UI, not for batching normal"
|
" sequences on transient UI, not for batching normal"
|
||||||
" steps), or a single vetted action."
|
" steps), or a single vetted action."
|
||||||
|
)
|
||||||
|
# One ToolMessage per rejected call, full text only on the first.
|
||||||
|
for idx, tc in enumerate(action_calls):
|
||||||
|
tool_outputs.append(
|
||||||
|
ToolMessage(
|
||||||
|
tool_call_id=tc["id"],
|
||||||
|
name=normalize_name(tc["name"]),
|
||||||
|
content=(
|
||||||
|
burst_error
|
||||||
|
if idx == 0
|
||||||
|
else "Not executed: see the burst-limit error above."
|
||||||
),
|
),
|
||||||
status="error",
|
status="error",
|
||||||
)
|
)
|
||||||
@@ -750,6 +814,7 @@ class OperatorNode:
|
|||||||
tool_outputs.append(
|
tool_outputs.append(
|
||||||
ToolMessage(
|
ToolMessage(
|
||||||
tool_call_id=tc["id"],
|
tool_call_id=tc["id"],
|
||||||
|
name=normalize_name(tc["name"]),
|
||||||
content=error,
|
content=error,
|
||||||
status="error",
|
status="error",
|
||||||
)
|
)
|
||||||
@@ -759,6 +824,7 @@ class OperatorNode:
|
|||||||
tool_outputs.append(
|
tool_outputs.append(
|
||||||
ToolMessage(
|
ToolMessage(
|
||||||
tool_call_id=tc["id"],
|
tool_call_id=tc["id"],
|
||||||
|
name=normalize_name(tc["name"]),
|
||||||
content="Action Recorded",
|
content="Action Recorded",
|
||||||
status="success",
|
status="success",
|
||||||
)
|
)
|
||||||
@@ -896,6 +962,7 @@ class OperatorNode:
|
|||||||
new_subagent_calls=new_subagent_calls,
|
new_subagent_calls=new_subagent_calls,
|
||||||
state=state,
|
state=state,
|
||||||
)
|
)
|
||||||
|
self._previous_turn_silent = not (raw_thinking and raw_thinking.strip())
|
||||||
|
|
||||||
# 8b. Transcript path: hold this turn's messages (observation tail +
|
# 8b. Transcript path: hold this turn's messages (observation tail +
|
||||||
# tool-loop products) for commit at the next build, when the turn's
|
# tool-loop products) for commit at the next build, when the turn's
|
||||||
@@ -1276,6 +1343,10 @@ class OperatorNode:
|
|||||||
"normalized_start_coordinates": [nx1, ny1],
|
"normalized_start_coordinates": [nx1, ny1],
|
||||||
"normalized_end_coordinates": [nx2, ny2],
|
"normalized_end_coordinates": [nx2, ny2],
|
||||||
"duration": duration,
|
"duration": duration,
|
||||||
|
# A directional scroll keeps the direction the agent issued so
|
||||||
|
# history and replays render "Swiped up (...)" rather than a
|
||||||
|
# bare coordinate path.
|
||||||
|
**({"direction": str(target)} if kind == "direction" else {}),
|
||||||
**({"target_description": described} if described else {}),
|
**({"target_description": described} if described else {}),
|
||||||
}
|
}
|
||||||
], None
|
], None
|
||||||
@@ -1290,9 +1361,15 @@ class OperatorNode:
|
|||||||
f"Error: 'key' must be a string, got {type(key).__name__}.",
|
f"Error: 'key' must be a string, got {type(key).__name__}.",
|
||||||
)
|
)
|
||||||
|
|
||||||
normalized_key = key[8:] if key.startswith("KEYCODE_") else key
|
normalized_key = key.strip().upper()
|
||||||
if normalized_key not in ["ENTER", "BACK", "HOME", "APP_SWITCH"]:
|
if normalized_key.startswith("KEYCODE_"):
|
||||||
return [], f"Error: Unsupported key '{key}'."
|
normalized_key = normalized_key[len("KEYCODE_") :]
|
||||||
|
if normalized_key not in SUPPORTED_PRESS_KEYS:
|
||||||
|
return (
|
||||||
|
[],
|
||||||
|
f"Error: Unsupported key '{key}'. Supported keys:"
|
||||||
|
f" {', '.join(SUPPORTED_PRESS_KEYS)}.",
|
||||||
|
)
|
||||||
return [{"action": "press_key", "keycode": f"KEYCODE_{normalized_key}"}], None
|
return [{"action": "press_key", "keycode": f"KEYCODE_{normalized_key}"}], None
|
||||||
|
|
||||||
elif tool_name == "manage_app":
|
elif tool_name == "manage_app":
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ from functools import lru_cache
|
|||||||
import io
|
import io
|
||||||
import json
|
import json
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from typing import NamedTuple
|
||||||
|
|
||||||
from jinja2 import Environment, StrictUndefined, Template
|
from jinja2 import Environment, StrictUndefined, Template
|
||||||
from langchain_core.messages import HumanMessage, SystemMessage
|
from langchain_core.messages import HumanMessage, SystemMessage
|
||||||
@@ -31,9 +32,8 @@ from artemis.tools.command_tool import (
|
|||||||
_is_output_long,
|
_is_output_long,
|
||||||
)
|
)
|
||||||
from artemis.utils.logger import get_logger
|
from artemis.utils.logger import get_logger
|
||||||
from artemis.utils.notes import get_note_file_path
|
|
||||||
from artemis.utils.plan_grammar import parse_plan, render_plan_grammar_spec
|
from artemis.utils.plan_grammar import parse_plan, render_plan_grammar_spec
|
||||||
from artemis.utils.task_tree import SELF_DESCRIBED_MARKER
|
from artemis.utils.task_tree import SELF_DESCRIBED_MARKER, action_intent_phrase
|
||||||
|
|
||||||
logger = get_logger(__name__)
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
@@ -149,11 +149,19 @@ class PromptBuilder:
|
|||||||
self.system_parts = []
|
self.system_parts = []
|
||||||
self.human_parts = []
|
self.human_parts = []
|
||||||
self.human_footer = None
|
self.human_footer = None
|
||||||
|
#: Indices into ``human_parts`` of blocks that only matter this turn (see
|
||||||
|
#: ``artemis.memory.transcript.EPHEMERAL_BLOCKS_KEY``).
|
||||||
|
self.ephemeral_indices: list[int] = []
|
||||||
|
|
||||||
def add_system_text(self, text: str):
|
def add_system_text(self, text: str):
|
||||||
self.system_parts.append(text)
|
self.system_parts.append(text)
|
||||||
|
|
||||||
def add_human_content(self, content: str | dict):
|
def add_human_content(self, content: str | dict, *, ephemeral: bool = False):
|
||||||
|
"""Append a block to the observation. ``ephemeral=True`` marks it as valid
|
||||||
|
for this turn only: the transcript scrub edge deletes it at depth K, so it
|
||||||
|
never reaches the frozen history or a chunk capsule."""
|
||||||
|
if ephemeral:
|
||||||
|
self.ephemeral_indices.append(len(self.human_parts))
|
||||||
self.human_parts.append(content)
|
self.human_parts.append(content)
|
||||||
|
|
||||||
def set_human_footer(self, content: str):
|
def set_human_footer(self, content: str):
|
||||||
@@ -171,10 +179,12 @@ class PromptBuilder:
|
|||||||
if self.human_footer:
|
if self.human_footer:
|
||||||
human_content.append({"type": "text", "text": self.human_footer})
|
human_content.append({"type": "text", "text": self.human_footer})
|
||||||
|
|
||||||
return [
|
human = HumanMessage(content=human_content)
|
||||||
SystemMessage(content=system_content),
|
if self.ephemeral_indices:
|
||||||
HumanMessage(content=human_content),
|
from artemis.memory.transcript import mark_ephemeral
|
||||||
]
|
|
||||||
|
mark_ephemeral(human, self.ephemeral_indices)
|
||||||
|
return [SystemMessage(content=system_content), human]
|
||||||
|
|
||||||
|
|
||||||
class PromptComponent:
|
class PromptComponent:
|
||||||
@@ -221,18 +231,42 @@ def resolve_operator_prompt_tools(ctx: ArtemisContext) -> frozenset[str]:
|
|||||||
return frozenset(available)
|
return frozenset(available)
|
||||||
|
|
||||||
|
|
||||||
def _operator_grammar_flags(ctx: ArtemisContext) -> tuple[bool, bool]:
|
def _operator_grammar_flags(ctx: ArtemisContext) -> tuple[bool, bool, bool]:
|
||||||
"""(include_checks, verification_active) for the template render."""
|
"""(midway_checks, final_check, verification_active) for the template render.
|
||||||
|
|
||||||
|
The two check gates are read separately so the grammar and the check-line
|
||||||
|
guidance are worded for what actually runs: with midway checks off there is
|
||||||
|
no repair loop to teach, and with both off the check-line grammar never
|
||||||
|
enters any prompt.
|
||||||
|
"""
|
||||||
setup = getattr(ctx, "execution_setup", None)
|
setup = getattr(ctx, "execution_setup", None)
|
||||||
# Grammar spec assembly is a function of configuration: with both check
|
midway = bool(setup and getattr(setup, "midway_checks_enabled", False))
|
||||||
# gates disabled, the check-line grammar never enters any prompt.
|
final = bool(setup and getattr(setup, "final_check_enabled", False))
|
||||||
include_checks = bool(setup and getattr(setup, "checks_enabled", False))
|
|
||||||
# The rejection/finding diagnosis trigger only exists while a mechanism
|
# The rejection/finding diagnosis trigger only exists while a mechanism
|
||||||
# that can produce rejections or findings is active.
|
# that can produce rejections or findings is active.
|
||||||
verification_active = include_checks or bool(
|
verification_active = (
|
||||||
setup and not getattr(setup, "disable_planner_validation", True)
|
midway or final or bool(setup and not getattr(setup, "disable_planner_validation", True))
|
||||||
)
|
)
|
||||||
return include_checks, verification_active
|
return midway, final, verification_active
|
||||||
|
|
||||||
|
|
||||||
|
def _checkpoint_max_repairs(ctx: ArtemisContext) -> int:
|
||||||
|
"""The midway repair budget recited in the static prompt."""
|
||||||
|
setup = getattr(ctx, "execution_setup", None)
|
||||||
|
max_repairs = getattr(setup, "checkpoint_max_repairs", None)
|
||||||
|
return max_repairs if isinstance(max_repairs, int) else 2
|
||||||
|
|
||||||
|
|
||||||
|
def _grammar_render_context(ctx: ArtemisContext) -> dict:
|
||||||
|
"""Template variables shared by the static and legacy system renders."""
|
||||||
|
midway, final, verification_active = _operator_grammar_flags(ctx)
|
||||||
|
return {
|
||||||
|
"plan_grammar": render_plan_grammar_spec(midway=midway, final=final),
|
||||||
|
"verification_active": verification_active,
|
||||||
|
"checks_active": midway or final,
|
||||||
|
"midway_checks_active": midway,
|
||||||
|
"checkpoint_max_repairs": _checkpoint_max_repairs(ctx),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
# --- M2 template split -----------------------------------------------------------
|
# --- M2 template split -----------------------------------------------------------
|
||||||
@@ -282,18 +316,15 @@ def render_transcript_static_system(
|
|||||||
|
|
||||||
available = resolve_operator_prompt_tools(ctx)
|
available = resolve_operator_prompt_tools(ctx)
|
||||||
static_template = apply_operator_prompt_contract(static_template, available_tools=available)
|
static_template = apply_operator_prompt_contract(static_template, available_tools=available)
|
||||||
include_checks, verification_active = _operator_grammar_flags(ctx)
|
|
||||||
return Template(static_template).render(
|
return Template(static_template).render(
|
||||||
initial_goal=state.initial_goal,
|
initial_goal=state.initial_goal,
|
||||||
subgoals_status="",
|
subgoals_status="",
|
||||||
plan_and_history="",
|
plan_and_history="",
|
||||||
unified_history="",
|
unified_history="",
|
||||||
plan_grammar=render_plan_grammar_spec(include_checks),
|
|
||||||
verification_active=verification_active,
|
|
||||||
checks_active=include_checks,
|
|
||||||
transcript_history=True,
|
transcript_history=True,
|
||||||
max_burst_actions=_max_burst_actions_for_prompt(),
|
max_burst_actions=_max_burst_actions_for_prompt(),
|
||||||
max_tool_calls=OPERATOR_MAX_TOOL_ITERATIONS,
|
max_tool_calls=OPERATOR_MAX_TOOL_ITERATIONS,
|
||||||
|
**_grammar_render_context(ctx),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -326,19 +357,15 @@ class TemplatePromptComponent(PromptComponent):
|
|||||||
|
|
||||||
plan_and_history = kwargs.get("plan_and_history", "No plan or history yet.")
|
plan_and_history = kwargs.get("plan_and_history", "No plan or history yet.")
|
||||||
|
|
||||||
include_checks, verification_active = _operator_grammar_flags(ctx)
|
|
||||||
|
|
||||||
full_prompt = Template(prompt_template).render(
|
full_prompt = Template(prompt_template).render(
|
||||||
initial_goal=state.initial_goal,
|
initial_goal=state.initial_goal,
|
||||||
subgoals_status="",
|
subgoals_status="",
|
||||||
plan_and_history=plan_and_history,
|
plan_and_history=plan_and_history,
|
||||||
unified_history="",
|
unified_history="",
|
||||||
plan_grammar=render_plan_grammar_spec(include_checks),
|
|
||||||
verification_active=verification_active,
|
|
||||||
checks_active=include_checks,
|
|
||||||
transcript_history=False,
|
transcript_history=False,
|
||||||
max_burst_actions=_max_burst_actions_for_prompt(),
|
max_burst_actions=_max_burst_actions_for_prompt(),
|
||||||
max_tool_calls=OPERATOR_MAX_TOOL_ITERATIONS,
|
max_tool_calls=OPERATOR_MAX_TOOL_ITERATIONS,
|
||||||
|
**_grammar_render_context(ctx),
|
||||||
)
|
)
|
||||||
|
|
||||||
parts = full_prompt.split("# CURRENT OBSERVATION")
|
parts = full_prompt.split("# CURRENT OBSERVATION")
|
||||||
@@ -400,7 +427,8 @@ class FeedbackPromptComponent(PromptComponent):
|
|||||||
" address any reverted subgoal accordingly; do not re-litigate"
|
" address any reverted subgoal accordingly; do not re-litigate"
|
||||||
" them. [planner] findings are advisory: a lightweight reviewer"
|
" them. [planner] findings are advisory: a lightweight reviewer"
|
||||||
" had a concern about a plan change that stayed applied — weigh"
|
" had a concern about a plan change that stayed applied — weigh"
|
||||||
" the reason against your own observations."
|
" the reason against your own observations.",
|
||||||
|
ephemeral=True,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -472,7 +500,9 @@ def render_execution_incident(incident: dict, steps: list) -> str:
|
|||||||
kind = incident.get("kind") or "exec_error"
|
kind = incident.get("kind") or "exec_error"
|
||||||
category = str(incident.get("category") or "general")
|
category = str(incident.get("category") or "general")
|
||||||
consecutive = int(incident.get("consecutive_failures") or 1)
|
consecutive = int(incident.get("consecutive_failures") or 1)
|
||||||
description = incident.get("action_description") or "the planned action"
|
# Intent form ("launch app 'X'"): the action did not happen, so the
|
||||||
|
# past-tense outcome phrase would misreport it as done.
|
||||||
|
description = action_intent_phrase(incident.get("action_description") or "the planned action")
|
||||||
reason = str(incident.get("reason") or "").strip()
|
reason = str(incident.get("reason") or "").strip()
|
||||||
burst_size = int(incident.get("burst_size") or 1)
|
burst_size = int(incident.get("burst_size") or 1)
|
||||||
index = int(incident.get("action_index") or 0)
|
index = int(incident.get("action_index") or 0)
|
||||||
@@ -505,8 +535,8 @@ def render_execution_incident(incident: dict, steps: list) -> str:
|
|||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
lines.append(
|
lines.append(
|
||||||
f"What happened: your planned action `{description}` was dispatched, but the"
|
f"What happened: your planned action `{description}` could"
|
||||||
f" device/executor reported: {reason}"
|
f" not be executed; the device/executor reported: {reason}"
|
||||||
)
|
)
|
||||||
|
|
||||||
# --- Category-specific evidence -------------------------------------------------
|
# --- Category-specific evidence -------------------------------------------------
|
||||||
@@ -547,7 +577,7 @@ def render_closed_incident(closed: dict) -> str:
|
|||||||
"""One-turn notice after an incident closes: settle the original intent."""
|
"""One-turn notice after an incident closes: settle the original intent."""
|
||||||
opened = closed.get("step_number")
|
opened = closed.get("step_number")
|
||||||
closed_at = closed.get("closed_at_step")
|
closed_at = closed.get("closed_at_step")
|
||||||
description = closed.get("action_description") or "the blocked action"
|
description = action_intent_phrase(closed.get("action_description") or "the blocked action")
|
||||||
header = (
|
header = (
|
||||||
f"--- Execution Incident (CLOSED at Step {closed_at}) ---"
|
f"--- Execution Incident (CLOSED at Step {closed_at}) ---"
|
||||||
if closed_at
|
if closed_at
|
||||||
@@ -567,61 +597,22 @@ class ExecutionIncidentPromptComponent(PromptComponent):
|
|||||||
then a one-turn CLOSED notice asking the Operator to settle the intent.
|
then a one-turn CLOSED notice asking the Operator to settle the intent.
|
||||||
|
|
||||||
Reads ``state.open_incident`` / ``state.last_closed_incident`` (both written
|
Reads ``state.open_incident`` / ``state.last_closed_incident`` (both written
|
||||||
by the Validator). The block is its own text part of the observation tail,
|
by the Validator). Both blocks are ephemeral: the open block is re-rendered
|
||||||
so the transcript ledger keeps it verbatim: the whole resolution effort
|
every turn the incident stays open, and the step record already carries the
|
||||||
stays legible across turns.
|
failed execution result, so older copies add nothing to the history.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
||||||
incident = getattr(state, "open_incident", None)
|
incident = getattr(state, "open_incident", None)
|
||||||
if isinstance(incident, dict) and incident.get("reason"):
|
if isinstance(incident, dict) and incident.get("reason"):
|
||||||
builder.add_human_content(
|
builder.add_human_content(
|
||||||
render_execution_incident(incident, kwargs.get("steps") or [])
|
render_execution_incident(incident, kwargs.get("steps") or []),
|
||||||
|
ephemeral=True,
|
||||||
)
|
)
|
||||||
return
|
return
|
||||||
closed = getattr(state, "last_closed_incident", None)
|
closed = getattr(state, "last_closed_incident", None)
|
||||||
if isinstance(closed, dict) and closed.get("kind"):
|
if isinstance(closed, dict) and closed.get("kind"):
|
||||||
builder.add_human_content(render_closed_incident(closed))
|
builder.add_human_content(render_closed_incident(closed), ephemeral=True)
|
||||||
|
|
||||||
|
|
||||||
class CheckItemsExplainerPromptComponent(PromptComponent):
|
|
||||||
"""Behavioral guidance for check lines, rendered iff the CURRENT plan
|
|
||||||
actually contains check lines (content-driven, not switch-driven: a resumed
|
|
||||||
plan carrying check lines still gets the explanation)."""
|
|
||||||
|
|
||||||
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
|
||||||
if not ctx or not getattr(ctx, "data_engine", None):
|
|
||||||
return
|
|
||||||
try:
|
|
||||||
task_plan_path = get_note_file_path(ctx.data_engine.base_dir, "task_plan")
|
|
||||||
if not task_plan_path.exists():
|
|
||||||
return
|
|
||||||
snapshot = parse_plan(task_plan_path.read_text(encoding="utf-8"))
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Failed to parse plan for check-items explainer: {e}")
|
|
||||||
return
|
|
||||||
if not snapshot.all_check_items:
|
|
||||||
return
|
|
||||||
setup = getattr(ctx, "execution_setup", None)
|
|
||||||
max_repairs = getattr(setup, "checkpoint_max_repairs", None)
|
|
||||||
if not isinstance(max_repairs, int):
|
|
||||||
max_repairs = 2
|
|
||||||
builder.add_human_content(
|
|
||||||
"--- About the plan's check lines ---\n"
|
|
||||||
"The task plan declares `- verify:` / `- assert:` check lines (see Task"
|
|
||||||
" Plan Grammar). Use `verify:` lines to confirm your work is complete,"
|
|
||||||
" but never declare a check passed yourself or record conclusions on"
|
|
||||||
" its behalf. Take NO extra actions for `assert:` lines and never"
|
|
||||||
" construct or fake state to satisfy one. Check lines must not be"
|
|
||||||
" deleted or reworded (deletions are automatically restored by the"
|
|
||||||
" system); keep them verbatim when rewriting the plan — adding new ones"
|
|
||||||
" is allowed. Simply mark completions per the plan grammar as usual;"
|
|
||||||
" checking runs asynchronously in the background and does not block you."
|
|
||||||
f" A failed `verify:` reopens its milestone at most {max_repairs} time(s);"
|
|
||||||
" once that repair budget is exhausted, its standing `finding:` line"
|
|
||||||
" disappears and the failure stands as the recorded result — do not"
|
|
||||||
" keep repairing it."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class BackgroundTasksPromptComponent(PromptComponent):
|
class BackgroundTasksPromptComponent(PromptComponent):
|
||||||
@@ -700,6 +691,26 @@ _LEDGER_BOUNCE_TAIL = (
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
#: One-line tail notice for the turn after a turn whose replies carried no visible
|
||||||
|
#: reasoning text. Thinking-mode models satisfy "reason first" inside their thought
|
||||||
|
#: channel and emit bare tool calls; the reminder costs no extra model call.
|
||||||
|
REASONING_REMINDER = (
|
||||||
|
"--- Reminder ---\n"
|
||||||
|
"Your previous turn carried no visible reasoning text (internal thinking is not"
|
||||||
|
" shown). This turn, write one short paragraph of plain prose before calling a"
|
||||||
|
" Turn-Ending Action: what the screen shows, whether the last action worked, and"
|
||||||
|
" what you do next and why."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ReasoningReminderPromptComponent(PromptComponent):
|
||||||
|
"""Renders :data:`REASONING_REMINDER` when the previous turn was silent."""
|
||||||
|
|
||||||
|
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
||||||
|
if kwargs.get("previous_turn_silent"):
|
||||||
|
builder.add_human_content(REASONING_REMINDER, ephemeral=True)
|
||||||
|
|
||||||
|
|
||||||
def render_plan_ledger_bounce(task_plan: str, streak: int, stale_turns: int) -> str | None:
|
def render_plan_ledger_bounce(task_plan: str, streak: int, stale_turns: int) -> str | None:
|
||||||
"""Return a plan-ledger validation error, if any."""
|
"""Return a plan-ledger validation error, if any."""
|
||||||
snapshot = parse_plan(task_plan or "")
|
snapshot = parse_plan(task_plan or "")
|
||||||
@@ -735,27 +746,128 @@ class ToolLimitWarningPromptComponent(PromptComponent):
|
|||||||
"\nWarning: your last turn used up its tool-call budget without a"
|
"\nWarning: your last turn used up its tool-call budget without a"
|
||||||
" Turn-Ending Action. Re-examine the task goal and your plan;"
|
" Turn-Ending Action. Re-examine the task goal and your plan;"
|
||||||
" the current approach may not be the right path. If the cause"
|
" the current approach may not be the right path. If the cause"
|
||||||
" is unclear, ask_diagnoser can help."
|
" is unclear, ask_diagnoser can help.",
|
||||||
|
ephemeral=True,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
#: Header of the user-guidance block (shared by the Flash runner and the Pro
|
||||||
|
#: operator; the operator's static prompt explains the channel once).
|
||||||
|
USER_GUIDANCE_MARKER = "--- User Guidance ---"
|
||||||
|
|
||||||
|
|
||||||
|
#: Prefix of the persistent user-instruction line (see :func:`render_user_guidance`).
|
||||||
|
USER_INSTRUCTION_PREFIX = "User instruction"
|
||||||
|
|
||||||
|
|
||||||
|
class UserGuidance(NamedTuple):
|
||||||
|
"""The two observation blocks delivering a mid-run user instruction.
|
||||||
|
|
||||||
|
``wrapper`` is the per-turn ``--- User Guidance ---`` framing (relay
|
||||||
|
notice, rank, release-loop wording): it is marked ephemeral and leaves
|
||||||
|
with the recent window. ``body`` is the verbatim instruction as one
|
||||||
|
self-contained line; it is added as a regular block so it stays in the
|
||||||
|
active window until the turn is chunk-compressed, at which point the
|
||||||
|
chunk's ``user_lines`` carry it verbatim. Standing instructions remain
|
||||||
|
visible after the turn in which they arrived.
|
||||||
|
"""
|
||||||
|
|
||||||
|
body: str
|
||||||
|
wrapper: str
|
||||||
|
|
||||||
|
|
||||||
|
def render_user_instruction(instruction: str, *, offset_label: str | None = None) -> str:
|
||||||
|
"""The persistent one-line body: ``User instruction (T+mm:ss): "..."``.
|
||||||
|
|
||||||
|
``offset_label`` is the session-relative time the instruction was
|
||||||
|
delivered when the call site has it; omitted otherwise.
|
||||||
|
"""
|
||||||
|
when = f" ({offset_label})" if offset_label else ""
|
||||||
|
return f'{USER_INSTRUCTION_PREFIX}{when}: "{instruction}"'
|
||||||
|
|
||||||
|
|
||||||
|
def render_user_guidance(
|
||||||
|
instruction: str,
|
||||||
|
*,
|
||||||
|
release_loop: bool = False,
|
||||||
|
has_plan: bool = True,
|
||||||
|
offset_label: str | None = None,
|
||||||
|
) -> UserGuidance:
|
||||||
|
"""The observation blocks delivering a mid-run user instruction.
|
||||||
|
|
||||||
|
One rendering for both profiles: the instruction is stated as what it is (an
|
||||||
|
external interruption from the user, relayed by the system) rather than as
|
||||||
|
advice to be weighed, and it explicitly outranks the plan and its check lines.
|
||||||
|
Returns ``(body, wrapper)``; callers add ``wrapper`` with ``ephemeral=True``
|
||||||
|
and ``body`` without (see :class:`UserGuidance`).
|
||||||
|
"""
|
||||||
|
rank = (
|
||||||
|
" It outranks the task plan and its check lines: edit the plan to match,"
|
||||||
|
" check lines included."
|
||||||
|
if has_plan
|
||||||
|
else " It outranks your current milestones: adjust them to match."
|
||||||
|
)
|
||||||
|
lines = [
|
||||||
|
USER_GUIDANCE_MARKER,
|
||||||
|
"The user watching this run sent the instruction quoted below; the system"
|
||||||
|
" relays it as an external interruption, like a stop signal." + rank,
|
||||||
|
"Act on it from this turn: reconcile it with the current screen and recent"
|
||||||
|
" history, then choose the next action. The quoted line stays in your"
|
||||||
|
" history as a standing instruction until the user sends another.",
|
||||||
|
]
|
||||||
|
if release_loop:
|
||||||
|
lines.append(
|
||||||
|
"The user has explicitly authorized stopping any ongoing monitoring loop;"
|
||||||
|
" you may now wrap up and complete the task."
|
||||||
|
)
|
||||||
|
return UserGuidance(
|
||||||
|
body=render_user_instruction(instruction, offset_label=offset_label),
|
||||||
|
wrapper=chr(10).join(lines),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def add_user_guidance(builder: PromptBuilder, guidance: UserGuidance) -> None:
|
||||||
|
"""Adds the two guidance blocks: the wrapper ephemeral, the body persistent."""
|
||||||
|
builder.add_human_content(guidance.wrapper, ephemeral=True)
|
||||||
|
builder.add_human_content(guidance.body)
|
||||||
|
|
||||||
|
|
||||||
class InjectedInstructionPromptComponent(PromptComponent):
|
class InjectedInstructionPromptComponent(PromptComponent):
|
||||||
|
"""The mid-run user instruction: an ephemeral wrapper plus the persistent
|
||||||
|
verbatim body (:class:`UserGuidance`).
|
||||||
|
|
||||||
|
The step record also keeps the instruction verbatim (``extra_metadata``),
|
||||||
|
which is what the chunk ledger renders once the turn is compressed.
|
||||||
|
"""
|
||||||
|
|
||||||
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
||||||
injected = getattr(state, "injected_instruction", None)
|
injected = getattr(state, "injected_instruction", None)
|
||||||
if injected:
|
if injected:
|
||||||
builder.add_human_content(
|
release_loop = bool(getattr(state, "user_stop_requested", False))
|
||||||
f"\n--- User Guidance ---\n"
|
add_user_guidance(
|
||||||
f"The user observing your progress has provided the following"
|
builder,
|
||||||
f" feedback or correction:\n"
|
render_user_guidance(
|
||||||
f'"{injected}"\n\n'
|
injected,
|
||||||
f"Please review this guidance, evaluate it against your current"
|
release_loop=release_loop,
|
||||||
f" screen state and recent history, "
|
has_plan=True,
|
||||||
f"and integrate it into your reasoning. Use it to refine your"
|
offset_label=_session_offset_label(ctx),
|
||||||
f" task plan and determine the "
|
),
|
||||||
f"most appropriate next action."
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _session_offset_label(ctx: ArtemisContext | None) -> str | None:
|
||||||
|
"""The transcript ledger's ``T+mm:ss`` label when a ledger is attached."""
|
||||||
|
ledger = getattr(ctx, "transcript_ledger", None) if ctx is not None else None
|
||||||
|
elapsed_label = getattr(ledger, "elapsed_label", None)
|
||||||
|
if ledger is None or not callable(elapsed_label):
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
label = elapsed_label()
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
return label if isinstance(label, str) and label else None
|
||||||
|
|
||||||
|
|
||||||
class ScreenshotSimilarityPromptComponent(PromptComponent):
|
class ScreenshotSimilarityPromptComponent(PromptComponent):
|
||||||
"""Background check: compare current screen against post-action screenshots of the last few steps.
|
"""Background check: compare current screen against post-action screenshots of the last few steps.
|
||||||
|
|
||||||
@@ -824,7 +936,7 @@ class ScreenshotSimilarityPromptComponent(PromptComponent):
|
|||||||
if matched_step_nums:
|
if matched_step_nums:
|
||||||
steps_str = ", ".join(matched_step_nums)
|
steps_str = ", ".join(matched_step_nums)
|
||||||
note_text = f"Note: the screen is unchanged since step {steps_str} (pixel-identical)."
|
note_text = f"Note: the screen is unchanged since step {steps_str} (pixel-identical)."
|
||||||
builder.add_human_content(note_text)
|
builder.add_human_content(note_text, ephemeral=True)
|
||||||
|
|
||||||
def _count_differing_pixels(
|
def _count_differing_pixels(
|
||||||
self,
|
self,
|
||||||
@@ -947,5 +1059,6 @@ class HistoricalStateHintPromptComponent(PromptComponent):
|
|||||||
builder.add_human_content(
|
builder.add_human_content(
|
||||||
f"Historical state hint: current screen closely resembles the"
|
f"Historical state hint: current screen closely resembles the"
|
||||||
f" post-action screen from Step {best_step_number}. Use"
|
f" post-action screen from Step {best_step_number}. Use"
|
||||||
" search_history / replay_steps only if its details are needed."
|
" search_history / replay_steps only if its details are needed.",
|
||||||
|
ephemeral=True,
|
||||||
)
|
)
|
||||||
@@ -21,7 +21,7 @@ The user explicitly requested to extract the following information:
|
|||||||
# Instruction
|
# Instruction
|
||||||
|
|
||||||
1. **Assess Confidence & Verify**: Based on the execution history, final state, and the target information to extract, determine if you can confidently formulate the final answer. If the current evidence is insufficient or you require more granular proof, you **MUST** actively gather information and details using your tools first. If your tools fail (e.g., return errors or time out), or if the steps were skipped/not observed, you **MUST** report this lack of evidence honestly. Do not assume, guess, or fabricate any unobserved steps.
|
1. **Assess Confidence & Verify**: Based on the execution history, final state, and the target information to extract, determine if you can confidently formulate the final answer. If the current evidence is insufficient or you require more granular proof, you **MUST** actively gather information and details using your tools first. If your tools fail (e.g., return errors or time out), or if the steps were skipped/not observed, you **MUST** report this lack of evidence honestly. Do not assume, guess, or fabricate any unobserved steps.
|
||||||
2. **Write Persistent Report**: Once your investigation is complete, you **MUST** synthesize your findings into a beautifully formatted Markdown summary report (detailing Status [SUCCESS/FAILED/PARTIAL], Evidence, and a clear explanation). You must write this report to persistent memory under the key "output" by calling:
|
2. **Write Persistent Report**: Once your investigation is complete, you **MUST** synthesize your findings into a Markdown summary report (detailing Status [SUCCESS/FAILED/PARTIAL], Evidence, and a clear explanation). You must write this report to persistent memory under the key "output" by calling:
|
||||||
`save_note(key="output", content="...your markdown report...")`
|
`save_note(key="output", content="...your markdown report...")`
|
||||||
3. **Final Response**: Only after saving the report, write a direct, helpful, and jargon-free final summary response to the user. Do not use any XML tags, code block wrappers, or special delimiters—just state the conclusion clearly and naturally, ensuring any extracted target information is clearly presented.
|
3. **Final Response**: Only after saving the report, write a direct, helpful, and jargon-free final summary response to the user. Do not use any XML tags, code block wrappers, or special delimiters—just state the conclusion clearly and naturally, ensuring any extracted target information is clearly presented.
|
||||||
|
|
||||||
|
|||||||
@@ -5,7 +5,7 @@
|
|||||||
"capabilities_operator": "## Execution Capabilities\n\nThe execution agent primarily interacts with the mobile device and analyzes screen recordings and logs. It can perform the following types of actions:\n- **UI Interaction**: Tapping, swiping, and entering text on the screen.\n- **App Management**: Launching and stopping specific applications.\n- **Information Retrieval & Analysis**: Reading on-screen text and UI hierarchy, searching the web for external information, querying system logs, and analyzing screen recordings.\n- **Terminal & Background Operations**: Executing arbitrary ADB shell commands on the device for exploration. It is capable of running long-running ADB commands asynchronously in the background as tasks.\n\n### Capability Boundaries & Planning Rules\n- **Cross-App Operations**: When a task requires interacting with multiple applications, split the operations into distinct milestones for each app. Avoid mixing actions for different apps in a single milestone.",
|
"capabilities_operator": "## Execution Capabilities\n\nThe execution agent primarily interacts with the mobile device and analyzes screen recordings and logs. It can perform the following types of actions:\n- **UI Interaction**: Tapping, swiping, and entering text on the screen.\n- **App Management**: Launching and stopping specific applications.\n- **Information Retrieval & Analysis**: Reading on-screen text and UI hierarchy, searching the web for external information, querying system logs, and analyzing screen recordings.\n- **Terminal & Background Operations**: Executing arbitrary ADB shell commands on the device for exploration. It is capable of running long-running ADB commands asynchronously in the background as tasks.\n\n### Capability Boundaries & Planning Rules\n- **Cross-App Operations**: When a task requires interacting with multiple applications, split the operations into distinct milestones for each app. Avoid mixing actions for different apps in a single milestone.",
|
||||||
"output_protocol_initial": "## Task Plan & Scratchpad Protocol\n\n1. **Articulate Thinking**: Begin by reasoning through the 5-step **Systematic Thinking Process** to establish your strategy, funnel approach, bounds, and pre-mortem checks.\n2. **Designate Notes in Milestones & Pre-generate Answer Cards**: If the task involves collecting findings, logging periodic monitoring events, recording candidates, or analytical summaries, designate clear, descriptive note names directly in the milestone descriptions (e.g., `candidates_list`, `monitoring_log`, `schedule_summary`). You can and should invoke the `save_note` tool to pre-generate these structured template files (answer card / scratchpad skeletons with Markdown headers, sections, or tables) before submitting `task_plan`. This provides downstream executors with a clear, ready-to-fill schema.\n3. **Submit Plan**: Submit your initial plan by invoking the `save_note` tool with `key='task_plan'`. Avoid writing the plan only in your text response, as it will only be recognized by the system if submitted via the tool.\n\n### Plan Formatting Rules\n- Format the plan strictly as a Markdown task list starting with `- [ ]` for uncompleted items.\n{{ plan_grammar }}\n- For linear macro-milestones: `- [ ] <Phase Description>`\n- For bounded repetitive, iterative, or periodic polling milestones, tag the item with `[Loop]` and declare its Exit Boundary and Interval (e.g., `- [ ] [Loop] <Overall Loop / Monitoring Goal> (Exit: <Condition>; Interval: <e.g., every 5 minutes>)`).\n- For unbounded continuous monitoring that only the user can stop, tag the item with `[Loop:continuous]` instead (e.g., `- [ ] [Loop:continuous] <Monitoring Goal> (Interval: <e.g., every 5 minutes>)`); never declare a self-serve exit condition for it.\n- **Strict Two-Tier Hierarchy & Loop Subtask Contract**:\n 1. **Subtask Nesting Rule**: ALL individual iteration instances, polling checks, candidate inspections, or baseline setups MUST be formatted as 2-space indented sub-tasks (` - [ ]`) directly nested under their parent `[Loop]` milestone.\n 2. **Mandatory First-Instance Initialization**: Every `[Loop]` milestone MUST be initialized with its first concrete child sub-task (e.g., ` - [ ] Inspect Candidate #1: ...` or ` - [ ] Polling Check #1 (Baseline & Initial Sync): ...`).\n 3. **No Flattened Cycles**: NEVER elevate individual checks, recurring polling cycles, or candidate inspections to top-level milestones outside the `[Loop]`. Top-level items are reserved exclusively for strategic macro-phases.\n- Ensure the content of the `task_plan` note contains only the task list, excluding introductions, explanations, conversational text, or headers.",
|
"output_protocol_initial": "## Task Plan & Scratchpad Protocol\n\n1. **Articulate Thinking**: Begin by reasoning through the 5-step **Systematic Thinking Process** to establish your strategy, funnel approach, bounds, and pre-mortem checks.\n2. **Designate Notes in Milestones & Pre-generate Answer Cards**: If the task involves collecting findings, logging periodic monitoring events, recording candidates, or analytical summaries, designate clear, descriptive note names directly in the milestone descriptions (e.g., `candidates_list`, `monitoring_log`, `schedule_summary`). You can and should invoke the `save_note` tool to pre-generate these structured template files (answer card / scratchpad skeletons with Markdown headers, sections, or tables) before submitting `task_plan`. This provides downstream executors with a clear, ready-to-fill schema.\n3. **Submit Plan**: Submit your initial plan by invoking the `save_note` tool with `key='task_plan'`. Avoid writing the plan only in your text response, as it will only be recognized by the system if submitted via the tool.\n\n### Plan Formatting Rules\n- Format the plan strictly as a Markdown task list starting with `- [ ]` for uncompleted items.\n{{ plan_grammar }}\n- For linear macro-milestones: `- [ ] <Phase Description>`\n- For bounded repetitive, iterative, or periodic polling milestones, tag the item with `[Loop]` and declare its Exit Boundary and Interval (e.g., `- [ ] [Loop] <Overall Loop / Monitoring Goal> (Exit: <Condition>; Interval: <e.g., every 5 minutes>)`).\n- For unbounded continuous monitoring that only the user can stop, tag the item with `[Loop:continuous]` instead (e.g., `- [ ] [Loop:continuous] <Monitoring Goal> (Interval: <e.g., every 5 minutes>)`); never declare a self-serve exit condition for it.\n- **Strict Two-Tier Hierarchy & Loop Subtask Contract**:\n 1. **Subtask Nesting Rule**: ALL individual iteration instances, polling checks, candidate inspections, or baseline setups MUST be formatted as 2-space indented sub-tasks (` - [ ]`) directly nested under their parent `[Loop]` milestone.\n 2. **Mandatory First-Instance Initialization**: Every `[Loop]` milestone MUST be initialized with its first concrete child sub-task (e.g., ` - [ ] Inspect Candidate #1: ...` or ` - [ ] Polling Check #1 (Baseline & Initial Sync): ...`).\n 3. **No Flattened Cycles**: NEVER elevate individual checks, recurring polling cycles, or candidate inspections to top-level milestones outside the `[Loop]`. Top-level items are reserved exclusively for strategic macro-phases.\n- Ensure the content of the `task_plan` note contains only the task list, excluding introductions, explanations, conversational text, or headers.",
|
||||||
"examples_initial": "## Master Example: Demonstrating the Complete Cognitive Paradigm\n\n**Goal**: \"Find and summarize all schedule arrangements and venue details for the upcoming November tech conference from my emails.\"\n\n### Naive / Incorrect Approaches:\n1. *Flawed Semantics & Too Granular*:\n- [ ] Tap search icon in Gmail\n- [ ] Type \"November\"\n- [ ] Open first email and tap back\n*(Fails: Micromanages atomic UI taps; assumes emails for November arrangements must have \"November\" in the title or be received in November)*\n2. *Vague & Unbounded Trap*:\n- [ ] Open email and check all emails for conference details\n*(Fails: Unbounded scan trap; causes infinite scrolling and step budget exhaustion on mobile)*\n\n---\n\n### Correct Approach (Full Cognitive Thinking & Robust Plan):\n\n#### Step-by-Step Thinking:\n1. **Intent & Semantic Expansion**:\n - *Core Objective*: Extract confirmed schedule dates, sessions, and venue info for the November tech conference.\n - *Semantic Insight*: Conference communications are often sent months in advance (e.g., ticket confirmations or registration emails in August/September). The subject line may say \"Global Developer Summit\", \"Tech Expo Agenda\", or \"Registration Confirmed\" rather than literally \"November conference\".\n - *Task Composition*: Linear setup (open email app and search) -> Iterative candidate inspection loop (examine each promising email) -> Final consolidation.\n2. **Visual Grounding**:\n - The device screenshot shows the Home screen with the Gmail app icon visible. The plan must start by opening the email client.\n3. **Tiered Strategy, Funnel Approach & Safeguards**:\n - *Tier 1 (Optimal)*: Use in-app search with rich semantic keywords (\"conference\", \"summit\", \"schedule\", \"agenda\", \"venue\", \"November\").\n - *Tier 2 (Fallback)*: If search is unavailable or yields zero results, fallback to chronological scanning of recent 3 months.\n - *Funnel (Broad Gathering -> Deep Inspection)*: Gather candidate emails from search results; do not discard candidates based solely on list previews. Open each candidate thread to inspect full body details.\n - *Feasibility Boundary*: Cap the candidate email inspection pool at top 15 emails to prevent infinite scrolling.\n4. **Risk Pre-mortem**:\n - *Risks*: Promotional marketing spam; conflicting drafts with preliminary dates; cancelled sessions.\n - *Defensive Check*: Instruct downstream executor to filter out promotional newsletters, verify session timestamps, and confirm final venue address.\n5. **Milestone Granularity & Scratchpad Blueprint**:\n - Focus milestones on business outcomes (search, inspect candidates, consolidate), avoiding micromanaged click-by-click steps.\n - Use indented sub-tasks to guide the loop of inspecting each candidate email, initializing with Item #1 and adding subsequent items as they are identified. This prevents the agent from losing state or re-inspecting the same items.\n - Designate note `november_conference_summary` as the destination for recording findings and pre-generate the structured template note.\n\n#### Tool Invocation:\n1. Pre-generate designated answer card note template:\n`save_note(key='november_conference_summary', content='# November Tech Conference Summary\\n\\n## Key Details\\n- Conference Name: \\n- Dates: \\n- Venue / Address: \\n\\n## Daily Schedule & Sessions\\n(To be populated during email inspection)\\n')`\n\n2. Submit milestone plan:\n`save_note(key='task_plan', content='- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\\n - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\\n- [ ] Review and consolidate all extracted information in `november_conference_summary` into a clear, finalized conference itinerary')`\n\n#### Content of `task_plan` Note:\n- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\n - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\n- [ ] ReviLine truncated
|
"examples_initial": "## Master Example: Demonstrating the Complete Cognitive Paradigm\n\n**Goal**: \"Find and summarize all schedule arrangements and venue details for the upcoming November tech conference from my emails.\"\n\n### Naive / Incorrect Approaches:\n1. *Flawed Semantics & Too Granular*:\n- [ ] Tap search icon in Gmail\n- [ ] Type \"November\"\n- [ ] Open first email and tap back\n*(Fails: Micromanages atomic UI taps; assumes emails for November arrangements must have \"November\" in the title or be received in November)*\n2. *Vague & Unbounded Trap*:\n- [ ] Open email and check all emails for conference details\n*(Fails: Unbounded scan trap; causes infinite scrolling and step budget exhaustion on mobile)*\n\n---\n\n### Correct Approach (Full Cognitive Thinking & Robust Plan):\n\n#### Step-by-Step Thinking:\n1. **Intent & Semantic Expansion**:\n - *Core Objective*: Extract confirmed schedule dates, sessions, and venue info for the November tech conference.\n - *Semantic Insight*: Conference communications are often sent months in advance (e.g., ticket confirmations or registration emails in August/September). The subject line may say \"Global Developer Summit\", \"Tech Expo Agenda\", or \"Registration Confirmed\" rather than literally \"November conference\".\n - *Task Composition*: Linear setup (open email app and search) -> Iterative candidate inspection loop (examine each promising email) -> Final consolidation.\n2. **Visual Grounding**:\n - The device screenshot shows the Home screen with the Gmail app icon visible. The plan must start by opening the email client.\n3. **Tiered Strategy, Funnel Approach & Safeguards**:\n - *Tier 1 (Optimal)*: Use in-app search with rich semantic keywords (\"conference\", \"summit\", \"schedule\", \"agenda\", \"venue\", \"November\").\n - *Tier 2 (Fallback)*: If search is unavailable or yields zero results, fallback to chronological scanning of recent 3 months.\n - *Funnel (Broad Gathering -> Deep Inspection)*: Gather candidate emails from search results; do not discard candidates based solely on list previews. Open each candidate thread to inspect full body details.\n - *Feasibility Boundary*: Cap the candidate email inspection pool at top 15 emails to prevent infinite scrolling.\n4. **Risk Pre-mortem**:\n - *Risks*: Promotional marketing spam; conflicting drafts with preliminary dates; cancelled sessions.\n - *Defensive Check*: Instruct downstream executor to filter out promotional newsletters, verify session timestamps, and confirm final venue address.\n5. **Milestone Granularity & Scratchpad Blueprint**:\n - Focus milestones on business outcomes (search, inspect candidates, consolidate), avoiding micromanaged click-by-click steps.\n - Use indented sub-tasks to guide the loop of inspecting each candidate email, initializing with Item #1 and adding subsequent items as they are identified. This prevents the agent from losing state or re-inspecting the same items.\n - Designate note `november_conference_summary` as the destination for recording findings and pre-generate the structured template note.\n\n#### Tool Invocation:\n1. Pre-generate designated answer card note template:\n`save_note(key='november_conference_summary', content='# November Tech Conference Summary\\n\\n## Key Details\\n- Conference Name: \\n- Dates: \\n- Venue / Address: \\n\\n## Daily Schedule & Sessions\\n(To be populated during email inspection)\\n')`\n\n2. Submit milestone plan:\n`save_note(key='task_plan', content='- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\\n - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\\n- [ ] Review and consolidate all extracted information in `november_conference_summary` into a clear, finalized conference itinerary')`\n\n#### Content of `task_plan` Note:\n- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\n - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\n- [ ] ReviLine truncated
|
||||||
"check_generation": "## Declaring Verification Standards (check lines)\nYou may attach check lines to the plan; an independent Checker judges them asynchronously.\n\n- **User-declared expectations become `- assert:` lines.** Whenever the user's goal states an explicit expectation (\"should show X\", \"must not crash\", \"the toast Y should appear\"), declare it as an assert line preserving the user's wording verbatim. Expectations about the whole run or the final state use the `@end` suffix (task-level lines at the end of the plan).\n- **`- verify:` lines only for milestones that genuinely need acceptance checks** — completion points of irreversible operations, or milestones that directly contribute to the final goal. Do NOT attach a verify line to every subgoal: restraint is an explicit requirement (checking cost scales with the number of check lines).\n- **Do not fabricate assertions** for subgoals that have nothing verifiable. Rewrite transient expectations as persistently probeable state wherever possible (e.g., \"alarm exists in the alarm list\" instead of \"a toast flashed\") — evidence for transient moments only exists if the execution history happened to capture it.\n- **Ordering requirements are post-hoc.** If the user demands \"A must be confirmed before B runs\", still generate the assertion preserving their semantics, but annotate in the plan that ordering is verified retroactively from history — the system performs post-hoc audits and CANNOT block B from executing. Never silently promise enforcement that does not exist.",
|
"check_generation": "## Declaring Verification Standards (check lines)\nYou may attach check lines to the plan. {% if midway_checks %}An independent Checker judges each line in the background when its milestone completes, and a failed verify reopens that milestone for repair{% if final_check %}; a final review at task exit then audits the whole run against the user's goal{% endif %}.{% else %}A final review at task exit judges every line once, from the recorded evidence around each milestone's completion; there is no midway repair loop.{% endif %}\n\n- **User-declared expectations become `- assert:` lines.** Whenever the user's goal states an explicit expectation (\"should show X\", \"must not crash\", \"the toast Y should appear\"), declare it as an assert line preserving the user's wording verbatim. Expectations about the whole run or the final state use the `@end` suffix (task-level lines at the end of the plan).\n- **`- verify:` lines only for milestones that genuinely need acceptance checks** — completion points of irreversible operations, or milestones that directly contribute to the final goal. Do NOT attach a verify line to every subgoal: restraint is an explicit requirement (checking cost scales with the number of check lines).\n- **Do not fabricate assertions** for subgoals that have nothing verifiable. Rewrite transient expectations as persistently probeable state wherever possible (e.g., \"alarm exists in the alarm list\" instead of \"a toast flashed\") — evidence for transient moments only exists if the execution history happened to capture it.\n- **Ordering requirements are post-hoc.** If the user demands \"A must be confirmed before B runs\", still generate the assertion preserving their semantics, but annotate in the plan that ordering is verified retroactively from history — the system performs post-hoc audits and CANNOT block B from executing. Never silently promise enforcement that does not exist.",
|
||||||
"check_audit": "- **Check Standard Protection**: Check lines (`- verify:` / `- assert:`) declared in the plan are the task's verification standards. FLAG any proposed plan that deletes, weakens, or rewords existing check lines — softening a declared standard is never an acceptable adaptation. Adding new check lines is allowed.",
|
"check_audit": "- **Check Standard Protection**: Check lines (`- verify:` / `- assert:`) declared in the plan are the task's verification standards. FLAG any proposed plan that deletes, weakens, or rewords existing check lines — softening a declared standard is never an acceptable adaptation. Adding new check lines is allowed.",
|
||||||
"role_validator": "## Role\nYou are the Task Plan Reviewer, a fast advisory second opinion. The Operator has modified the top-level task plan and the change is already applied. Your job is to spot a concrete concern — goal drift, a broken loop contract, a weakened check standard, a logical dead end — and, if there is one, state it with its reason so the Operator can decide. You never block or roll back anything.",
|
"role_validator": "## Role\nYou are the Task Plan Reviewer, a fast advisory second opinion. The Operator has modified the top-level task plan and the change is already applied. Your job is to spot a concrete concern — goal drift, a broken loop contract, a weakened check standard, a logical dead end — and, if there is one, state it with its reason so the Operator can decide. You never block or roll back anything.",
|
||||||
"validator_guidelines": "## Validation Principles\n- **Goal Alignment**: Ensure the new plan ultimately accomplishes the `Initial Goal` requested by the user. If the Operator has abandoned the core objective or fundamentally altered it, flag the plan.\n- **Logical Coherence**: Verify that the proposed milestones form a logical, achievable sequence to reach the goal. Support valid ongoing iteration/polling subgoals and trigger branches without misconstruing periodic checks as infinite loops. Flag the plan if it introduces contradictory steps, unhandled deadlocks, or obvious logical fallacies.\n{{ plan_grammar }}\n- **Loop & Monitoring Protection**: If the existing plan contains an active `[Loop]` or `[Loop:continuous]` milestone, you MUST FLAG any proposed plan that deletes the milestone or removes its loop tag, as well as any plan that marks a `[Loop:continuous]` continuous monitoring milestone as completed (`[x]`) without an explicit external user stop signal.\n- **Adaptive Acceptance**: The Operator operates in a dynamic environment and may encounter unexpected UI states or dynamic event triggers. You should approve alternative strategies, extra intermediate steps, loop status tracking, or necessary workarounds as long as they serve the primary goal effectively.\n- **Advisory Verdict & Constructive Feedback**: Your verdict is advisory — the change stays applied whether or not you approve it. Approve unless you have a concrete, evidence-backed concern; a vague unease is not a concern. If you flag the plan, give one or two sentences naming the flaw or goal deviation and the correction you suggest, concrete enough for the Operator to weigh it against its own observations."
|
"validator_guidelines": "## Validation Principles\n- **Goal Alignment**: Ensure the new plan ultimately accomplishes the `Initial Goal` requested by the user. If the Operator has abandoned the core objective or fundamentally altered it, flag the plan.\n- **Logical Coherence**: Verify that the proposed milestones form a logical, achievable sequence to reach the goal. Support valid ongoing iteration/polling subgoals and trigger branches without misconstruing periodic checks as infinite loops. Flag the plan if it introduces contradictory steps, unhandled deadlocks, or obvious logical fallacies.\n{{ plan_grammar }}\n- **Loop & Monitoring Protection**: If the existing plan contains an active `[Loop]` or `[Loop:continuous]` milestone, you MUST FLAG any proposed plan that deletes the milestone or removes its loop tag, as well as any plan that marks a `[Loop:continuous]` continuous monitoring milestone as completed (`[x]`) without an explicit external user stop signal.\n- **Adaptive Acceptance**: The Operator operates in a dynamic environment and may encounter unexpected UI states or dynamic event triggers. You should approve alternative strategies, extra intermediate steps, loop status tracking, or necessary workarounds as long as they serve the primary goal effectively.\n- **Advisory Verdict & Constructive Feedback**: Your verdict is advisory — the change stays applied whether or not you approve it. Approve unless you have a concrete, evidence-backed concern; a vague unease is not a concern. If you flag the plan, give one or two sentences naming the flaw or goal deviation and the correction you suggest, concrete enough for the Operator to weigh it against its own observations."
|
||||||
|
|||||||
@@ -82,11 +82,19 @@ def validate_plan_format(content: str) -> tuple[bool, str]:
|
|||||||
return True, ""
|
return True, ""
|
||||||
|
|
||||||
|
|
||||||
def _checks_enabled(ctx: ArtemisContext) -> bool:
|
def _check_gates(ctx: ArtemisContext) -> tuple[bool, bool]:
|
||||||
"""Config-driven prompt assembly gate: with both check gates disabled, no
|
"""(midway_checks, final_check): the two check gates, read separately so
|
||||||
check-related instruction reaches any planner prompt."""
|
the planner is taught the judgment moment that actually applies. With both
|
||||||
|
disabled, no check-related instruction reaches any planner prompt."""
|
||||||
setup = getattr(ctx, "execution_setup", None)
|
setup = getattr(ctx, "execution_setup", None)
|
||||||
return bool(setup and getattr(setup, "checks_enabled", False))
|
midway = bool(setup and getattr(setup, "midway_checks_enabled", False))
|
||||||
|
final = bool(setup and getattr(setup, "final_check_enabled", False))
|
||||||
|
return midway, final
|
||||||
|
|
||||||
|
|
||||||
|
def _checks_enabled(ctx: ArtemisContext) -> bool:
|
||||||
|
"""Config-driven prompt assembly gate (either check gate active)."""
|
||||||
|
return any(_check_gates(ctx))
|
||||||
|
|
||||||
|
|
||||||
def build_planner_system_blocks(prompts_data: dict, mode: str, include_checks: bool) -> list[str]:
|
def build_planner_system_blocks(prompts_data: dict, mode: str, include_checks: bool) -> list[str]:
|
||||||
@@ -152,12 +160,15 @@ async def run_async_planner_validation(
|
|||||||
with open(prompts_path, encoding="utf-8") as f:
|
with open(prompts_path, encoding="utf-8") as f:
|
||||||
prompts_data = json.load(f)
|
prompts_data = json.load(f)
|
||||||
|
|
||||||
include_checks = _checks_enabled(ctx)
|
midway, final = _check_gates(ctx)
|
||||||
|
include_checks = midway or final
|
||||||
mode_config = prompts_data["modes"]["validator"]
|
mode_config = prompts_data["modes"]["validator"]
|
||||||
system_blocks = build_planner_system_blocks(prompts_data, "validator", include_checks)
|
system_blocks = build_planner_system_blocks(prompts_data, "validator", include_checks)
|
||||||
system_content = "\n\n".join(prompts_data["blocks"][block] for block in system_blocks)
|
system_content = "\n\n".join(prompts_data["blocks"][block] for block in system_blocks)
|
||||||
system_message = Template(system_content).render(
|
system_message = Template(system_content).render(
|
||||||
plan_grammar=render_plan_grammar_spec(include_checks)
|
plan_grammar=render_plan_grammar_spec(midway=midway, final=final),
|
||||||
|
midway_checks=midway,
|
||||||
|
final_check=final,
|
||||||
)
|
)
|
||||||
|
|
||||||
history_str = "No history available."
|
history_str = "No history available."
|
||||||
@@ -228,15 +239,19 @@ class PlannerNode:
|
|||||||
mode = "initial_plan"
|
mode = "initial_plan"
|
||||||
mode_config = prompts_data["modes"][mode]
|
mode_config = prompts_data["modes"][mode]
|
||||||
|
|
||||||
include_checks = _checks_enabled(self.ctx)
|
midway, final = _check_gates(self.ctx)
|
||||||
|
include_checks = midway or final
|
||||||
# Assembly point (not in-template conditionals): check-generation
|
# Assembly point (not in-template conditionals): check-generation
|
||||||
# instructions only mount while the checking feature is active, so a
|
# instructions only mount while the checking feature is active, so a
|
||||||
# fully-disabled configuration yields a byte-identical planner prompt.
|
# fully-disabled configuration yields a byte-identical planner prompt.
|
||||||
|
# Inside the mounted block, the judgment moment is worded per gate.
|
||||||
system_blocks = build_planner_system_blocks(prompts_data, mode, include_checks)
|
system_blocks = build_planner_system_blocks(prompts_data, mode, include_checks)
|
||||||
system_content = "\n\n".join(prompts_data["blocks"][block] for block in system_blocks)
|
system_content = "\n\n".join(prompts_data["blocks"][block] for block in system_blocks)
|
||||||
|
|
||||||
system_message = Template(system_content).render(
|
system_message = Template(system_content).render(
|
||||||
plan_grammar=render_plan_grammar_spec(include_checks)
|
plan_grammar=render_plan_grammar_spec(midway=midway, final=final),
|
||||||
|
midway_checks=midway,
|
||||||
|
final_check=final,
|
||||||
)
|
)
|
||||||
|
|
||||||
# Get screenshot
|
# Get screenshot
|
||||||
|
|||||||
@@ -1,35 +0,0 @@
|
|||||||
# ROLE
|
|
||||||
You are the Summarizer Agent for a mobile automation system. Your task is to summarize the latest execution step, providing a high-information-density and succinct history summary of the current step without losing any information.
|
|
||||||
|
|
||||||
# OUTPUT FORMAT
|
|
||||||
- **Format Constraint**: You MUST output the summary as a **single, continuous paragraph**.
|
|
||||||
- **Prohibited Formats**: Do NOT use bullet points, numbered lists, markdown formatting, or multiple paragraphs.
|
|
||||||
- **Goal**: Maintain a consistent, easily parsable narrative for the Operator's historical record.
|
|
||||||
|
|
||||||
# CORE PHILOSOPHY & RULES
|
|
||||||
|
|
||||||
1. **Perspective**: You MUST write the summary from the Operator's first-person perspective using "I" (e.g., "I intended to...", "I clicked..."). NEVER use third-person terms like "The operator" or "The agent", nor second-person terms like "You" when referring to the Operator. For other agents (e.g., Explorer, Diagnoser), you can use their names.
|
|
||||||
2. **Zero Subjective Validation**: Absolutely avoid declaring semantic success, completion, or failure. In historical compression, any subjective validation will cause future planners to falsely believe a goal was definitively met or permanently broken.
|
|
||||||
- **BANNED WORDS**: "successfully", "completed", "entered", "navigated to", "failed", "unsuccessful", "could not", "achieved".
|
|
||||||
3. **Focus on Intent & Strategy**: The specific physical action (exact coordinates, swipes) is now recorded automatically. Focus primarily on the high-level Intent and reasoning. If you realize the Operator is on the wrong path and needs to backtrack or pivot, describe this as a shift in strategy.
|
|
||||||
4. **Intended Context Transitions**: Describe the intended navigation path ("From Where -> Intended Where"). Do not declare that the transition actually happened. (e.g., "From the Home screen, I intended to open the Network Settings view").
|
|
||||||
5. **Preserve Verifications & Iteration Progress**: If the Operator verified any state in this step, processed a specific candidate item/polling round, or explicitly mentioned what prerequisites or items are still missing, you MUST succinctly preserve these concrete progress details and verified items in the summary.
|
|
||||||
6. **Objective Incident Logs**: If the safety net intercepted the action, the device rejected it, or a fast-action burst aborted midway (an `[Result]` line starting with "Error:" / an execution incident), faithfully record what was blocked and the stated reason, without judging whether the Operator's next move will resolve it.
|
|
||||||
|
|
||||||
# LOOP & STAGNATION DETECTION
|
|
||||||
Use the `RECENT 10 STEPS HISTORY` to intelligently detect loops or stagnation:
|
|
||||||
1. **Physical Action Loops vs. Planned Iterations**:
|
|
||||||
- *Planned Iterations / Monitoring (Valid Progress)*: If the Operator is systematically processing different candidate items in a loop, or executing planned periodic polling cycles with wait intervals under a `[Loop]` milestone, record this objectively as planned progress (e.g., "Inspected Candidate #2", "Completed Polling Round #2 for new messages"). Do NOT flag planned iterations across different items or spaced polling as abnormal loops.
|
|
||||||
- *Unintended Action Loops (Anomalies)*: If the Operator is repeatedly clicking the exact same target, failing to progress on the same item, or oscillating blindly between identical states without updating progress notes, explicitly record the anomalous loop pattern and iteration count (e.g., "This is the 3rd cycle of failing to open the same folder").
|
|
||||||
2. **Cognitive Stagnation**: While you generally ignore internal background tool calls (e.g., reading logs, checking states) when counting physical action loops, if you detect continuous multiple steps of purely internal tool calls without any physical actions or note updates, objectively record this "stagnation" or "analysis loop".
|
|
||||||
|
|
||||||
# INPUT STRUCTURE
|
|
||||||
You will receive:
|
|
||||||
- **CURRENT PLAN & TASK PLAN**: The current task plan including milestones, active subgoals, and live iteration checklists to help you understand the strategic context and loop phase.
|
|
||||||
- **RECENT 10 STEPS HISTORY**: A brief history of the last 10 steps to help you detect loops and stagnation patterns.
|
|
||||||
- **CHRONOLOGICAL STEP TRACE (CURRENT STEP)**: The step-by-step chronological history of the current execution turn, including:
|
|
||||||
- `[Operator Monologue] / [Operator Native Thought]`: The operator's planning, intentions, progress counting, reasoning, verified states, and explicit mentions of missing prerequisites.
|
|
||||||
- `[Operator Tool Call] / [Operator Final Action]`: The actual action(s) planned and executed.
|
|
||||||
- `[Planned Fast-Action Burst]`: A multi-action turn executed back to back without the safety net, with each member's outcome.
|
|
||||||
- `[Pre-Execution Safety Net]`: Validation checks prior to action execution.
|
|
||||||
- `[Result]`: The Validator's execution outcome; an "Error:" line here is an execution incident the Operator must resolve.
|
|
||||||
@@ -12,7 +12,7 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
"""Pro-graph step summarizer node (M2: visual-transition lens dispatch).
|
"""Pro step summarizer dispatching visual-transition summaries.
|
||||||
|
|
||||||
Per the history-module redesign §5 (single step-level lens, confirmed
|
Per the history-module redesign §5 (single step-level lens, confirmed
|
||||||
2026-08-31) this node no longer generates a per-step text capsule with its own
|
2026-08-31) this node no longer generates a per-step text capsule with its own
|
||||||
|
|||||||
+87
-14
@@ -460,8 +460,20 @@ class MemoryTranscriptConfig(BaseModel):
|
|||||||
default=3,
|
default=3,
|
||||||
ge=1,
|
ge=1,
|
||||||
description=(
|
description=(
|
||||||
"Depth K of the screenshot scrub edge: the K-th most recent"
|
"Depth K of the screenshot scrub edge from the start gate up: the"
|
||||||
" historical screenshot is replaced by its visual summary."
|
" K-th most recent screenshot (the live one counts as depth 1) is"
|
||||||
|
" replaced by its visual summary."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
image_scrub_depth_relaxed: int = Field(
|
||||||
|
default=6,
|
||||||
|
ge=1,
|
||||||
|
description=(
|
||||||
|
"Screenshot scrub depth while the operator's measured context is"
|
||||||
|
" below budget*start_ratio (nothing is being replaced yet, so"
|
||||||
|
" more raw screenshots stay in view). Must be >= image_scrub_depth;"
|
||||||
|
" set equal to it to disable the occupancy-driven depth. A change"
|
||||||
|
" of depth never rewrites an already-scrubbed message."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
pending_grace_steps: int = Field(
|
pending_grace_steps: int = Field(
|
||||||
@@ -473,13 +485,17 @@ class MemoryTranscriptConfig(BaseModel):
|
|||||||
" image is replaced by a placeholder referencing the DataEngine step."
|
" image is replaced by a placeholder referencing the DataEngine step."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
xml_scrub_depth: int = Field(
|
xml_scrub_depth: int | None = Field(
|
||||||
default=1,
|
default=1,
|
||||||
ge=1,
|
ge=1,
|
||||||
description=(
|
description=(
|
||||||
"How many of the most recent observations keep their UI Element"
|
"Text edge of the scrub: once a message is deeper than this many"
|
||||||
" list; older lists are stripped (depth 1 matches the legacy"
|
" image-bearing observations (1 = as soon as a newer observation"
|
||||||
" prune_history_xml semantics)."
|
" exists) its UI element list, plan recitation and per-turn"
|
||||||
|
" ephemeral blocks are removed. Kept shallow on purpose: only the"
|
||||||
|
" live observation carries an indexed element list, so a stale"
|
||||||
|
" [n] index can never be picked up as a target. Screenshots follow"
|
||||||
|
" image_scrub_depth / image_scrub_depth_relaxed."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
context_budget_tokens: int = Field(
|
context_budget_tokens: int = Field(
|
||||||
@@ -491,14 +507,27 @@ class MemoryTranscriptConfig(BaseModel):
|
|||||||
" ratios below are applied against this budget."
|
" ratios below are applied against this budget."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
start_ratio: float = Field(
|
||||||
|
default=0.35,
|
||||||
|
ge=0.0,
|
||||||
|
lt=1.0,
|
||||||
|
description=(
|
||||||
|
"Start gate: below budget*start_ratio of the operator's measured"
|
||||||
|
" prompt size, closed segments only prepare their capsule in the"
|
||||||
|
" background — the original turns stay in the transcript. Ready"
|
||||||
|
" chunks replace their turns once the measured context reaches"
|
||||||
|
" this ratio (the hard threshold overrides it). 0 restores"
|
||||||
|
" swap-as-soon-as-ready."
|
||||||
|
),
|
||||||
|
)
|
||||||
soft_ratio: float = Field(
|
soft_ratio: float = Field(
|
||||||
default=0.7,
|
default=0.7,
|
||||||
gt=0.0,
|
gt=0.0,
|
||||||
lt=1.0,
|
lt=1.0,
|
||||||
description=(
|
description=(
|
||||||
"Soft threshold: when the last measured prompt size reaches"
|
"Soft threshold: when the operator's last measured prompt size"
|
||||||
" budget*soft_ratio, the oldest open segment is chunk-compressed"
|
" reaches budget*soft_ratio, the oldest open segment is"
|
||||||
" (L2)."
|
" chunk-compressed (L2)."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
hard_ratio: float = Field(
|
hard_ratio: float = Field(
|
||||||
@@ -506,9 +535,13 @@ class MemoryTranscriptConfig(BaseModel):
|
|||||||
gt=0.0,
|
gt=0.0,
|
||||||
le=1.0,
|
le=1.0,
|
||||||
description=(
|
description=(
|
||||||
"Hard threshold: when the last measured prompt size reaches"
|
"Hard threshold: when the operator's last measured prompt size"
|
||||||
" budget*hard_ratio, the frozen region collapses to the L3 session"
|
" reaches budget*hard_ratio, ready chunks swap as L2 blocks first;"
|
||||||
" snapshot (chunk headers merged; per-step index retained)."
|
" only if that would not bring the estimated context back under the"
|
||||||
|
" line does everything closed force-swap (pending chunks included)"
|
||||||
|
" and the whole frozen region fold into a recall-only era — L3: the"
|
||||||
|
" period paragraph with the per-step index. The fold is permanent;"
|
||||||
|
" later L2 chunks are appended after it."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
min_active_steps: int = Field(
|
min_active_steps: int = Field(
|
||||||
@@ -545,6 +578,23 @@ class MemoryTranscriptConfig(BaseModel):
|
|||||||
)
|
)
|
||||||
model_config = {"extra": "allow"}
|
model_config = {"extra": "allow"}
|
||||||
|
|
||||||
|
@model_validator(mode="after")
|
||||||
|
def _ratios_are_ordered(self) -> "MemoryTranscriptConfig":
|
||||||
|
"""The three thresholds form a ladder against the same budget."""
|
||||||
|
if not (0.0 <= self.start_ratio <= self.soft_ratio <= self.hard_ratio <= 1.0):
|
||||||
|
raise ValueError(
|
||||||
|
"memory.transcript ratios must satisfy"
|
||||||
|
" 0 <= start_ratio <= soft_ratio <= hard_ratio <= 1"
|
||||||
|
f" (got start_ratio={self.start_ratio}, soft_ratio={self.soft_ratio},"
|
||||||
|
f" hard_ratio={self.hard_ratio})."
|
||||||
|
)
|
||||||
|
if self.image_scrub_depth_relaxed < self.image_scrub_depth:
|
||||||
|
raise ValueError(
|
||||||
|
"memory.transcript.image_scrub_depth_relaxed must be >= image_scrub_depth"
|
||||||
|
f" (got {self.image_scrub_depth_relaxed} < {self.image_scrub_depth})."
|
||||||
|
)
|
||||||
|
return self
|
||||||
|
|
||||||
|
|
||||||
class MemoryRecallConfig(BaseModel):
|
class MemoryRecallConfig(BaseModel):
|
||||||
"""search_history tool options (agent.memory.recall)."""
|
"""search_history tool options (agent.memory.recall)."""
|
||||||
@@ -620,8 +670,21 @@ class MemoryChunkingConfig(BaseModel):
|
|||||||
default=2000,
|
default=2000,
|
||||||
ge=128,
|
ge=128,
|
||||||
description=(
|
description=(
|
||||||
"Size-threshold trigger: estimated source tokens (char/4 over the"
|
"Size-threshold trigger: estimated source tokens (transcript"
|
||||||
" open segment's transcript text) that force a chunk compression."
|
" characters through the session-calibrated ratio, 4 chars/token"
|
||||||
|
" by default) over the open segment's eligible portion that force"
|
||||||
|
" a chunk compression."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
min_steps: int = Field(
|
||||||
|
default=3,
|
||||||
|
ge=1,
|
||||||
|
description=(
|
||||||
|
"Minimum turns in a chunk closed by the size or pressure trigger:"
|
||||||
|
" below this the eligible turns keep accumulating instead of"
|
||||||
|
" becoming one-step chunks (each chunk costs a capsule call)."
|
||||||
|
" Milestone closes are exempt (a complete segment is a unit) and"
|
||||||
|
" the hard threshold waives it (emergency)."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
model: str = Field(
|
model: str = Field(
|
||||||
@@ -646,6 +709,16 @@ class MemoryChunkingConfig(BaseModel):
|
|||||||
" behavior."
|
" behavior."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@model_validator(mode="after")
|
||||||
|
def _min_steps_within_cap(self) -> "MemoryChunkingConfig":
|
||||||
|
if self.min_steps > self.max_steps:
|
||||||
|
raise ValueError(
|
||||||
|
"memory.chunking.min_steps must not exceed max_steps"
|
||||||
|
f" (got min_steps={self.min_steps}, max_steps={self.max_steps})."
|
||||||
|
)
|
||||||
|
return self
|
||||||
|
|
||||||
model_config = {"extra": "allow"}
|
model_config = {"extra": "allow"}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -221,6 +221,18 @@ class ArtemisContext(BaseModel):
|
|||||||
"""Plan content currently under async planner validation; becomes the new
|
"""Plan content currently under async planner validation; becomes the new
|
||||||
baseline when the validator approves it."""
|
baseline when the validator approves it."""
|
||||||
|
|
||||||
|
guidance_unprotected_checks: set[tuple[str, str]] = Field(default_factory=set)
|
||||||
|
"""``(kind, text)`` of every check line that existed in the task plan when a
|
||||||
|
mid-run user instruction arrived. User guidance outranks the plan's declared
|
||||||
|
standards, so these lines lose their machine restoration for the rest of the
|
||||||
|
run (the Operator may drop or reword them at any later write); check lines
|
||||||
|
added afterwards stay protected until the next instruction."""
|
||||||
|
guidance_retired_checks: set[tuple[str, str]] = Field(default_factory=set)
|
||||||
|
"""``(kind, text)`` of the check lines the Operator actually dropped under
|
||||||
|
that waiver. A verdict recorded for such a line before it was retired is a
|
||||||
|
result for a requirement the user withdrew: the run outcome reports it as
|
||||||
|
retired instead of counting it as a pass or failure."""
|
||||||
|
|
||||||
_genai_client: Any | None = PrivateAttr(default=None)
|
_genai_client: Any | None = PrivateAttr(default=None)
|
||||||
_video_blackboard: Any | None = PrivateAttr(default=None)
|
_video_blackboard: Any | None = PrivateAttr(default=None)
|
||||||
_video_circuit_breaker: Any | None = PrivateAttr(default=None)
|
_video_circuit_breaker: Any | None = PrivateAttr(default=None)
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ from artemis.core.diagnostics.adb_keys import (
|
|||||||
heal_adb_keys,
|
heal_adb_keys,
|
||||||
inspect_adb_keys,
|
inspect_adb_keys,
|
||||||
)
|
)
|
||||||
|
from artemis.core.diagnostics.device_smoke import smoke_test_device
|
||||||
from artemis.core.diagnostics.emulator_manager import (
|
from artemis.core.diagnostics.emulator_manager import (
|
||||||
EmulatorLaunchStage,
|
EmulatorLaunchStage,
|
||||||
EmulatorLaunchState,
|
EmulatorLaunchState,
|
||||||
@@ -32,11 +33,20 @@ from artemis.core.diagnostics.probes.credentials_probe import (
|
|||||||
LLMCredentialsProbe,
|
LLMCredentialsProbe,
|
||||||
VisionOCRProbe,
|
VisionOCRProbe,
|
||||||
)
|
)
|
||||||
|
from artemis.core.diagnostics.probes.host_probe import IntegrationHostProbe
|
||||||
from artemis.core.diagnostics.probes.runtime_probe import (
|
from artemis.core.diagnostics.probes.runtime_probe import (
|
||||||
PythonRuntimeProbe,
|
PythonRuntimeProbe,
|
||||||
SystemConfigProbe,
|
SystemConfigProbe,
|
||||||
)
|
)
|
||||||
from artemis.core.diagnostics.probes.toolchain_probe import ToolchainProbe
|
from artemis.core.diagnostics.probes.toolchain_probe import ToolchainProbe
|
||||||
|
from artemis.core.diagnostics.readiness import (
|
||||||
|
CHECK_ORDER,
|
||||||
|
Verdict,
|
||||||
|
adb_keys_corrupted,
|
||||||
|
base_verdict,
|
||||||
|
collect_readiness,
|
||||||
|
sort_by_fix_order,
|
||||||
|
)
|
||||||
from artemis.core.diagnostics.schema import (
|
from artemis.core.diagnostics.schema import (
|
||||||
DeviceInfo,
|
DeviceInfo,
|
||||||
ProbeAction,
|
ProbeAction,
|
||||||
@@ -53,6 +63,7 @@ __all__ = [
|
|||||||
"EmulatorLaunchStage",
|
"EmulatorLaunchStage",
|
||||||
"EmulatorLaunchState",
|
"EmulatorLaunchState",
|
||||||
"BaseProbe",
|
"BaseProbe",
|
||||||
|
"IntegrationHostProbe",
|
||||||
"PythonRuntimeProbe",
|
"PythonRuntimeProbe",
|
||||||
"SystemConfigProbe",
|
"SystemConfigProbe",
|
||||||
"AdbDeviceProbe",
|
"AdbDeviceProbe",
|
||||||
@@ -69,4 +80,11 @@ __all__ = [
|
|||||||
"get_adb_key_paths",
|
"get_adb_key_paths",
|
||||||
"inspect_adb_keys",
|
"inspect_adb_keys",
|
||||||
"heal_adb_keys",
|
"heal_adb_keys",
|
||||||
|
"smoke_test_device",
|
||||||
|
"CHECK_ORDER",
|
||||||
|
"Verdict",
|
||||||
|
"collect_readiness",
|
||||||
|
"base_verdict",
|
||||||
|
"adb_keys_corrupted",
|
||||||
|
"sort_by_fix_order",
|
||||||
]
|
]
|
||||||
@@ -0,0 +1,359 @@
|
|||||||
|
# Copyright 2026 Google LLC
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
"""End-to-end device observation smoke test.
|
||||||
|
|
||||||
|
The readiness ADB probe only proves that ``adb devices`` and ``dumpsys`` answer.
|
||||||
|
The most common "device shows Connected but ARTEMIS cannot observe it" failure
|
||||||
|
is UIAutomator not answering (screenshot / hierarchy dump hangs), which today
|
||||||
|
only surfaces as a failed task. :func:`smoke_test_device` performs exactly the
|
||||||
|
observation the ``mobile_get_device_state`` MCP tool performs -- build (or reuse)
|
||||||
|
the cached device controller through ``artemis.mcp.adb_server._get_controller``
|
||||||
|
and call ``controller.get_screen_data()`` -- and reports a structured verdict
|
||||||
|
with concrete repair guidance instead of raising.
|
||||||
|
|
||||||
|
Design notes
|
||||||
|
------------
|
||||||
|
* ``_get_controller`` is synchronous, caches one controller per serial for the
|
||||||
|
life of the process and never touches ``DeviceExecutionLock``. Its first call
|
||||||
|
for a serial blocks on ``UIAutomatorClient.get_screen_data()`` (three
|
||||||
|
``u2.connect`` attempts plus a screenshot and hierarchy dump), so it runs on a
|
||||||
|
worker thread here.
|
||||||
|
* ``AdbDriver.get_screen_data`` invokes the synchronous UIAutomator client on
|
||||||
|
the event-loop thread. A plain ``asyncio.wait_for`` around the coroutine can
|
||||||
|
therefore never fire while UIAutomator hangs. The coroutine is executed on a
|
||||||
|
dedicated daemon thread with its own event loop, and the deadline is enforced
|
||||||
|
on the join. A thread abandoned by a timeout finishes on its own once the
|
||||||
|
underlying adb / HTTP timeouts elapse; being a daemon it never delays
|
||||||
|
interpreter shutdown.
|
||||||
|
* ``u2.connect`` restarts the on-device UIAutomator server when it believes the
|
||||||
|
server is unhealthy. Running that against a device another ARTEMIS process is
|
||||||
|
driving would break the running task's session, so a device with a live
|
||||||
|
``DeviceExecutionLock`` owner is reported as busy instead of probed.
|
||||||
|
* Nothing is left behind: no screenshot is written to disk, no lock is taken,
|
||||||
|
and -- mirroring ``mobile_get_device_state`` -- the cached controller is kept
|
||||||
|
for reuse rather than closed.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import base64
|
||||||
|
from collections.abc import Callable
|
||||||
|
import os
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from typing import Any, TypeVar
|
||||||
|
|
||||||
|
from artemis.runtime.device_lock import DeviceExecutionLock
|
||||||
|
from artemis.utils.logger import get_logger
|
||||||
|
|
||||||
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
|
T = TypeVar("T")
|
||||||
|
|
||||||
|
#: On-device packages installed by uiautomator2 (``python -m uiautomator2 init``).
|
||||||
|
UIAUTOMATOR_PACKAGES: tuple[str, ...] = (
|
||||||
|
"com.github.uiautomator",
|
||||||
|
"com.github.uiautomator.test",
|
||||||
|
)
|
||||||
|
|
||||||
|
#: A real JPEG screenshot of any phone screen is tens of kilobytes; the Android
|
||||||
|
#: driver substitutes a 1x1 PNG (~70 bytes) when every capture path failed.
|
||||||
|
_MIN_PLAUSIBLE_SCREENSHOT_BYTES = 512
|
||||||
|
|
||||||
|
_SERIAL_PLACEHOLDER = "<serial>"
|
||||||
|
|
||||||
|
|
||||||
|
def _serial_or_placeholder(serial: str | None) -> str:
|
||||||
|
return serial or _SERIAL_PLACEHOLDER
|
||||||
|
|
||||||
|
|
||||||
|
def _uiautomator_fix(serial: str | None) -> list[str]:
|
||||||
|
s = _serial_or_placeholder(serial)
|
||||||
|
force_stops = ", then ".join(
|
||||||
|
f"adb -s {s} shell am force-stop {pkg}" for pkg in UIAUTOMATOR_PACKAGES
|
||||||
|
)
|
||||||
|
return [
|
||||||
|
"Unlock the phone and keep the screen on (UIAutomator cannot dump a locked or sleeping screen).",
|
||||||
|
f"Restart the on-device UIAutomator server, then retry: {force_stops}",
|
||||||
|
(
|
||||||
|
"If it keeps failing, reinstall the UIAutomator server APKs: "
|
||||||
|
f"python -m uiautomator2 purge --serial {s}, then python -m uiautomator2 init --serial {s} "
|
||||||
|
"(run with the ARTEMIS virtualenv Python), or reboot the device."
|
||||||
|
),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def fix_for_error(error: str | None, serial: str | None) -> list[str]:
|
||||||
|
"""Map an error string to concrete repair steps (substring match, case-insensitive)."""
|
||||||
|
if not error:
|
||||||
|
return []
|
||||||
|
text = error.lower()
|
||||||
|
s = _serial_or_placeholder(serial)
|
||||||
|
|
||||||
|
if "busy" in text or "another task" in text:
|
||||||
|
return [
|
||||||
|
"Wait for the running task to finish, or stop it with mobile_manage_task(action='stop', ...).",
|
||||||
|
"Then rerun the smoke test.",
|
||||||
|
]
|
||||||
|
if "unauthorized" in text:
|
||||||
|
return [
|
||||||
|
"Accept the 'Allow USB debugging' prompt on the phone (tick 'Always allow from this computer').",
|
||||||
|
f"If no prompt appears, re-plug the USB cable and run: adb -s {s} devices",
|
||||||
|
]
|
||||||
|
if "offline" in text:
|
||||||
|
return [
|
||||||
|
"Re-plug the USB cable (try another port or cable).",
|
||||||
|
"Run: adb reconnect (or adb kill-server, then adb start-server) and check adb devices -l.",
|
||||||
|
]
|
||||||
|
if "no android devices" in text or "no device" in text or "not found" in text:
|
||||||
|
return [
|
||||||
|
"Connect an Android device with USB debugging enabled, or start an emulator.",
|
||||||
|
"mobile_diagnose(launch_avd='<avd name>') boots a named AVD; adb devices -l lists what is attached.",
|
||||||
|
]
|
||||||
|
if (
|
||||||
|
"timeout" in text
|
||||||
|
or "timed out" in text
|
||||||
|
or "uiautomator" in text
|
||||||
|
or "not responding" in text
|
||||||
|
or "did not respond" in text
|
||||||
|
or "hierarchy" in text
|
||||||
|
or "screenshot" in text
|
||||||
|
or "http" in text
|
||||||
|
or "deadobject" in text
|
||||||
|
):
|
||||||
|
return _uiautomator_fix(serial)
|
||||||
|
return [
|
||||||
|
f"Check the raw hierarchy dump manually: adb -s {s} shell uiautomator dump, then adb -s {s} shell cat /sdcard/window_dump.xml",
|
||||||
|
f"Check the raw screenshot path: adb -s {s} exec-out screencap -p > screen.png",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
#: Exceptions the smoke test must never swallow.
|
||||||
|
_PASSTHROUGH_EXCEPTIONS: tuple[type[BaseException], ...] = (
|
||||||
|
KeyboardInterrupt,
|
||||||
|
SystemExit,
|
||||||
|
asyncio.CancelledError,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _describe_failure(exc: BaseException) -> str:
|
||||||
|
"""uiautomator2 errors derive from ``BaseException`` and often carry an empty message."""
|
||||||
|
text = str(exc).strip()
|
||||||
|
name = type(exc).__name__
|
||||||
|
if not text:
|
||||||
|
return name
|
||||||
|
return text if name in ("Exception", "RuntimeError") else f"{name}: {text}"
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_requested_serial(device_serial: str | None) -> str | None:
|
||||||
|
"""Mirror ``_get_controller``'s serial resolution (explicit arg, then env)."""
|
||||||
|
return (
|
||||||
|
device_serial or os.environ.get("ARTEMIS_DEVICE_ID") or os.environ.get("ADB_DEVICE_SERIAL")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _find_busy_owner(target_serial: str | None) -> tuple[str, Any] | None:
|
||||||
|
"""Return ``(device_key, owner)`` when a live ARTEMIS process owns the target device.
|
||||||
|
|
||||||
|
With no target serial ``_get_controller`` would pick the first attached device,
|
||||||
|
which cannot be matched against lock records cheaply; any live owner is then
|
||||||
|
treated as busy to stay on the safe side.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
owners = DeviceExecutionLock.get_active_owners()
|
||||||
|
except Exception as exc: # pylint: disable=broad-exception-caught
|
||||||
|
logger.debug(f"Device lock inspection failed, skipping busy guard: {exc}")
|
||||||
|
return None
|
||||||
|
if not owners:
|
||||||
|
return None
|
||||||
|
if target_serial is None:
|
||||||
|
key, owner = next(iter(owners.items()))
|
||||||
|
return key, owner
|
||||||
|
clean = DeviceExecutionLock._normalize_device_id(target_serial)
|
||||||
|
for key, owner in owners.items():
|
||||||
|
if key == clean or key.endswith(f"__{clean}"):
|
||||||
|
return key, owner
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def _run_in_daemon_thread(fn: Callable[[], T], timeout: float, what: str) -> T:
|
||||||
|
"""Run ``fn`` on a daemon thread and wait at most ``timeout`` seconds.
|
||||||
|
|
||||||
|
Raises :class:`TimeoutError` on deadline. The thread keeps running until
|
||||||
|
``fn`` returns on its own, but the caller is released immediately.
|
||||||
|
"""
|
||||||
|
loop = asyncio.get_running_loop()
|
||||||
|
future: asyncio.Future[T] = loop.create_future()
|
||||||
|
|
||||||
|
def _deliver(setter: Callable[[], None]) -> None:
|
||||||
|
try:
|
||||||
|
loop.call_soon_threadsafe(setter)
|
||||||
|
except RuntimeError:
|
||||||
|
# Event loop already closed: the caller gave up on us.
|
||||||
|
pass
|
||||||
|
|
||||||
|
def _worker() -> None:
|
||||||
|
try:
|
||||||
|
value = fn()
|
||||||
|
except BaseException as exc: # pylint: disable=broad-exception-caught
|
||||||
|
error: BaseException = exc
|
||||||
|
|
||||||
|
def _fail() -> None:
|
||||||
|
if not future.done():
|
||||||
|
future.set_exception(error)
|
||||||
|
|
||||||
|
_deliver(_fail)
|
||||||
|
return
|
||||||
|
|
||||||
|
def _succeed() -> None:
|
||||||
|
if not future.done():
|
||||||
|
future.set_result(value)
|
||||||
|
|
||||||
|
_deliver(_succeed)
|
||||||
|
|
||||||
|
threading.Thread(target=_worker, name=f"artemis-smoke-{what}", daemon=True).start()
|
||||||
|
try:
|
||||||
|
return await asyncio.wait_for(future, timeout=timeout)
|
||||||
|
except TimeoutError:
|
||||||
|
logger.warning(f"Device smoke test: {what} did not finish within {timeout:.1f}s")
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
def _run_coroutine_blocking(coro_factory: Callable[[], Any]) -> Any:
|
||||||
|
"""Execute a coroutine on a fresh event loop owned by the current thread."""
|
||||||
|
return asyncio.run(coro_factory())
|
||||||
|
|
||||||
|
|
||||||
|
def _count_elements(elements: Any) -> int | None:
|
||||||
|
if elements is None:
|
||||||
|
return None
|
||||||
|
if isinstance(elements, str):
|
||||||
|
return elements.count("<node")
|
||||||
|
if isinstance(elements, dict):
|
||||||
|
return 1
|
||||||
|
try:
|
||||||
|
return len(elements)
|
||||||
|
except TypeError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _decoded_length(b64: Any) -> int | None:
|
||||||
|
if not isinstance(b64, str) or not b64:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return len(base64.b64decode(b64))
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def smoke_test_device(
|
||||||
|
device_serial: str | None = None, timeout_seconds: float = 20.0
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Observe the device exactly like ``mobile_get_device_state`` and report a verdict.
|
||||||
|
|
||||||
|
Never raises. Returns::
|
||||||
|
|
||||||
|
{"ok": bool, "serial": str | None, "elapsed_seconds": float,
|
||||||
|
"screenshot_bytes": int | None, "element_count": int | None,
|
||||||
|
"error": str | None, "fix": list[str]}
|
||||||
|
"""
|
||||||
|
started = time.monotonic()
|
||||||
|
requested_serial = _resolve_requested_serial(device_serial)
|
||||||
|
result: dict[str, Any] = {
|
||||||
|
"ok": False,
|
||||||
|
"serial": requested_serial,
|
||||||
|
"elapsed_seconds": 0.0,
|
||||||
|
"screenshot_bytes": None,
|
||||||
|
"element_count": None,
|
||||||
|
"error": None,
|
||||||
|
"fix": [],
|
||||||
|
}
|
||||||
|
|
||||||
|
def _finish(error: str | None = None, cause: str | None = None) -> dict[str, Any]:
|
||||||
|
"""Record the verdict; ``cause`` is the raw failure text used to pick the fix."""
|
||||||
|
result["elapsed_seconds"] = round(time.monotonic() - started, 3)
|
||||||
|
result["error"] = error
|
||||||
|
result["ok"] = error is None
|
||||||
|
result["fix"] = fix_for_error(cause if cause is not None else error, result["serial"])
|
||||||
|
return result
|
||||||
|
|
||||||
|
busy = _find_busy_owner(requested_serial)
|
||||||
|
if busy is not None:
|
||||||
|
key, owner = busy
|
||||||
|
description = getattr(owner, "description", "") or "unknown task"
|
||||||
|
pid = getattr(owner, "pid", "?")
|
||||||
|
result["serial"] = requested_serial or getattr(owner, "device_id", None)
|
||||||
|
return _finish(
|
||||||
|
f"Device {key} is busy: another task is running on it ({description}, pid {pid});"
|
||||||
|
" skipping the screen capture so the running task is not disturbed."
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
from artemis.mcp import adb_server
|
||||||
|
|
||||||
|
controller = await _run_in_daemon_thread(
|
||||||
|
lambda: adb_server._get_controller(device_serial=device_serial),
|
||||||
|
timeout_seconds,
|
||||||
|
"controller-init",
|
||||||
|
)
|
||||||
|
except TimeoutError:
|
||||||
|
return _finish(
|
||||||
|
f"UIAutomator/controller initialization did not respond within {timeout_seconds:g}s"
|
||||||
|
)
|
||||||
|
except _PASSTHROUGH_EXCEPTIONS:
|
||||||
|
raise
|
||||||
|
except BaseException as exc: # pylint: disable=broad-exception-caught
|
||||||
|
cause = _describe_failure(exc)
|
||||||
|
return _finish(f"Failed to initialize Android device controller: {cause}", cause)
|
||||||
|
|
||||||
|
controller_ctx = getattr(controller, "ctx", None)
|
||||||
|
device = getattr(controller_ctx, "device", None)
|
||||||
|
device_id = getattr(device, "device_id", None)
|
||||||
|
if isinstance(device_id, str) and device_id:
|
||||||
|
result["serial"] = device_id
|
||||||
|
|
||||||
|
remaining = max(0.5, timeout_seconds - (time.monotonic() - started))
|
||||||
|
try:
|
||||||
|
device_data = await _run_in_daemon_thread(
|
||||||
|
lambda: _run_coroutine_blocking(controller.get_screen_data),
|
||||||
|
remaining,
|
||||||
|
"screen-data",
|
||||||
|
)
|
||||||
|
except TimeoutError:
|
||||||
|
return _finish(f"UIAutomator/screen capture did not respond within {timeout_seconds:g}s")
|
||||||
|
except _PASSTHROUGH_EXCEPTIONS:
|
||||||
|
raise
|
||||||
|
except BaseException as exc: # pylint: disable=broad-exception-caught
|
||||||
|
cause = _describe_failure(exc)
|
||||||
|
return _finish(f"Screen capture failed: {cause}", cause)
|
||||||
|
|
||||||
|
result["screenshot_bytes"] = _decoded_length(getattr(device_data, "base64", None))
|
||||||
|
result["element_count"] = _count_elements(getattr(device_data, "elements", None))
|
||||||
|
|
||||||
|
if result["screenshot_bytes"] is None:
|
||||||
|
return _finish("Screen capture returned no screenshot data")
|
||||||
|
if result["screenshot_bytes"] < _MIN_PLAUSIBLE_SCREENSHOT_BYTES:
|
||||||
|
return _finish(
|
||||||
|
f"Screenshot capture failed (driver returned a {result['screenshot_bytes']}-byte placeholder image)"
|
||||||
|
)
|
||||||
|
if not result["element_count"]:
|
||||||
|
return _finish(
|
||||||
|
"UIAutomator hierarchy dump returned no UI elements (screenshot worked, hierarchy did not)"
|
||||||
|
)
|
||||||
|
return _finish(None)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["UIAUTOMATOR_PACKAGES", "fix_for_error", "smoke_test_device"]
|
||||||
@@ -15,6 +15,7 @@
|
|||||||
"""System Readiness & Diagnostic Orchestration Engine."""
|
"""System Readiness & Diagnostic Orchestration Engine."""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
|
import os
|
||||||
import subprocess
|
import subprocess
|
||||||
import time
|
import time
|
||||||
from typing import Any
|
from typing import Any
|
||||||
@@ -33,6 +34,7 @@ from artemis.core.diagnostics.probes.runtime_probe import (
|
|||||||
from artemis.core.diagnostics.probes.toolchain_probe import ToolchainProbe
|
from artemis.core.diagnostics.probes.toolchain_probe import ToolchainProbe
|
||||||
from artemis.core.diagnostics.schema import (
|
from artemis.core.diagnostics.schema import (
|
||||||
DeviceInfo,
|
DeviceInfo,
|
||||||
|
ProbeAction,
|
||||||
ProbeCategory,
|
ProbeCategory,
|
||||||
ProbeResult,
|
ProbeResult,
|
||||||
ProbeStatus,
|
ProbeStatus,
|
||||||
@@ -50,6 +52,13 @@ class ReadinessEngine:
|
|||||||
|
|
||||||
_REPORT_CACHE_TTL_SECONDS = 2.0
|
_REPORT_CACHE_TTL_SECONDS = 2.0
|
||||||
|
|
||||||
|
#: Deadline for one probe inside a report. The ADB probe shells out to
|
||||||
|
#: ``adb`` without a deadline of its own, and a wedged ADB server would
|
||||||
|
#: otherwise hang every surface that runs the report (``artemis doctor``,
|
||||||
|
#: the console wizard, ``mobile_diagnose``). A probe that overruns is
|
||||||
|
#: reported as a FAIL with the recovery steps instead.
|
||||||
|
PROBE_TIMEOUT_SECONDS = 30.0
|
||||||
|
|
||||||
def __init__(self):
|
def __init__(self):
|
||||||
self._probes: dict[str, BaseProbe] = {}
|
self._probes: dict[str, BaseProbe] = {}
|
||||||
self._python_probe = PythonRuntimeProbe()
|
self._python_probe = PythonRuntimeProbe()
|
||||||
@@ -178,11 +187,22 @@ class ReadinessEngine:
|
|||||||
if categories is None or probe.category in categories
|
if categories is None or probe.category in categories
|
||||||
]
|
]
|
||||||
|
|
||||||
# Concurrently execute probes
|
# Concurrently execute probes. A probe that raises must not take the
|
||||||
results: list[ProbeResult] = await asyncio.gather(
|
# whole report down with it: it becomes a structured FAIL so the
|
||||||
*[probe.probe() for probe in target_probes],
|
# remaining probes still reach the user and the verdict stays honest.
|
||||||
return_exceptions=False,
|
outcomes = await asyncio.gather(
|
||||||
|
*[
|
||||||
|
asyncio.wait_for(probe.probe(), timeout=self.PROBE_TIMEOUT_SECONDS)
|
||||||
|
for probe in target_probes
|
||||||
|
],
|
||||||
|
return_exceptions=True,
|
||||||
)
|
)
|
||||||
|
results: list[ProbeResult] = [
|
||||||
|
outcome
|
||||||
|
if isinstance(outcome, ProbeResult)
|
||||||
|
else self._crashed_probe_result(probe, outcome)
|
||||||
|
for probe, outcome in zip(target_probes, outcomes, strict=True)
|
||||||
|
]
|
||||||
|
|
||||||
blockers = [r for r in results if r.is_blocker]
|
blockers = [r for r in results if r.is_blocker]
|
||||||
passed_blockers = [r for r in blockers if r.status == ProbeStatus.PASS]
|
passed_blockers = [r for r in blockers if r.status == ProbeStatus.PASS]
|
||||||
@@ -211,6 +231,62 @@ class ReadinessEngine:
|
|||||||
timestamp=time.time(),
|
timestamp=time.time(),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _crashed_probe_result(probe: BaseProbe, exc: object) -> ProbeResult:
|
||||||
|
"""Turn an exception escaping ``probe.probe()`` into a FAIL result.
|
||||||
|
|
||||||
|
Cancellation and other non-``Exception`` errors are re-raised: they are
|
||||||
|
not a diagnosis of the host, they are the event loop shutting us down.
|
||||||
|
"""
|
||||||
|
if isinstance(exc, BaseException) and not isinstance(exc, Exception):
|
||||||
|
raise exc
|
||||||
|
if isinstance(exc, TimeoutError):
|
||||||
|
logger.warning(
|
||||||
|
f"[ReadinessEngine] Probe '{probe.probe_id}' did not finish within"
|
||||||
|
f" {ReadinessEngine.PROBE_TIMEOUT_SECONDS:.0f}s; reporting it as FAIL."
|
||||||
|
)
|
||||||
|
return ProbeResult(
|
||||||
|
id=probe.probe_id,
|
||||||
|
category=probe.category,
|
||||||
|
title=probe.probe_id.replace("_", " ").title(),
|
||||||
|
status=ProbeStatus.FAIL,
|
||||||
|
is_blocker=probe.is_blocker,
|
||||||
|
summary="Probe timed out",
|
||||||
|
description=(
|
||||||
|
f"The '{probe.probe_id}' check did not finish within"
|
||||||
|
f" {ReadinessEngine.PROBE_TIMEOUT_SECONDS:.0f}s. A hung ADB server is the"
|
||||||
|
" usual cause: restart it and run the diagnosis again."
|
||||||
|
),
|
||||||
|
actions=[
|
||||||
|
ProbeAction(
|
||||||
|
action_type="command",
|
||||||
|
label="Restart ADB",
|
||||||
|
payload="adb kill-server && adb start-server",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
metadata={
|
||||||
|
"exception_type": "TimeoutError",
|
||||||
|
"timeout_seconds": ReadinessEngine.PROBE_TIMEOUT_SECONDS,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
logger.warning(
|
||||||
|
f"[ReadinessEngine] Probe '{probe.probe_id}' crashed; reporting it as FAIL: "
|
||||||
|
f"{type(exc).__name__}: {exc}"
|
||||||
|
)
|
||||||
|
return ProbeResult(
|
||||||
|
id=probe.probe_id,
|
||||||
|
category=probe.category,
|
||||||
|
title=probe.probe_id.replace("_", " ").title(),
|
||||||
|
status=ProbeStatus.FAIL,
|
||||||
|
is_blocker=probe.is_blocker,
|
||||||
|
summary="Probe crashed",
|
||||||
|
description=(
|
||||||
|
f"The '{probe.probe_id}' check raised {type(exc).__name__}: {exc}. "
|
||||||
|
"This is a diagnostics bug or a host permission problem, not a device fault."
|
||||||
|
),
|
||||||
|
metadata={"exception_type": type(exc).__name__, "exception": str(exc)},
|
||||||
|
)
|
||||||
|
|
||||||
async def heal_adb_keys(self, force: bool = False) -> dict[str, Any]:
|
async def heal_adb_keys(self, force: bool = False) -> dict[str, Any]:
|
||||||
"""Auto-heal corrupted or 0-byte ADB authentication keys."""
|
"""Auto-heal corrupted or 0-byte ADB authentication keys."""
|
||||||
from artemis.core.diagnostics.adb_keys import heal_adb_keys
|
from artemis.core.diagnostics.adb_keys import heal_adb_keys
|
||||||
|
|||||||
@@ -284,7 +284,7 @@ class VisionOCRProbe(BaseProbe):
|
|||||||
status=ProbeStatus.PASS,
|
status=ProbeStatus.PASS,
|
||||||
is_blocker=False,
|
is_blocker=False,
|
||||||
summary="Not Configured (Optional)",
|
summary="Not Configured (Optional)",
|
||||||
description="OCR_API_KEY is not set. Perception operates seamlessly in pure UI XML hierarchy mode.",
|
description="OCR_API_KEY is not set. Perception uses the UI XML hierarchy without OCR.",
|
||||||
metadata={"configured": False},
|
metadata={"configured": False},
|
||||||
actions=[
|
actions=[
|
||||||
ProbeAction(
|
ProbeAction(
|
||||||
|
|||||||
@@ -0,0 +1,427 @@
|
|||||||
|
# Copyright 2026 Google LLC
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
"""Integration-host probe: the process environment an IDE / MCP client launched us in.
|
||||||
|
|
||||||
|
The standard readiness probes answer "can Artemis drive a phone on this
|
||||||
|
machine?". This probe answers the question that precedes it when Artemis is
|
||||||
|
embedded in an AI coding assistant (Claude Code, Cursor, Codex, Antigravity,
|
||||||
|
...): "was the server launched the way the installer expects, and can it
|
||||||
|
spawn a background runner?". The failure modes it covers are the ones users
|
||||||
|
hit mostly through MCP, rarely from the CLI or the web console:
|
||||||
|
|
||||||
|
* the MCP client started the server with a different interpreter than the
|
||||||
|
project virtualenv, so the spawned runner imports a different package set;
|
||||||
|
* the ``.env`` file the settings loader reads is not where the user put the key;
|
||||||
|
* the traces directory is not writable from the client's sandbox;
|
||||||
|
* the daemon port is held by another process (a very common collision with
|
||||||
|
dev servers that also default to port 8000).
|
||||||
|
|
||||||
|
The probe is not registered in the default :class:`ReadinessEngine` probe set:
|
||||||
|
the web console has its own device wizard and would only render noise. The
|
||||||
|
``mobile_diagnose`` MCP tool and ``artemis doctor`` run it explicitly.
|
||||||
|
|
||||||
|
Only the unwritable-traces case blocks task execution; the result carries
|
||||||
|
``is_blocker`` accordingly, so the WARN findings degrade the verdict without
|
||||||
|
blocking it.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from artemis.core.diagnostics.probes.base import BaseProbe
|
||||||
|
from artemis.core.diagnostics.schema import (
|
||||||
|
ProbeAction,
|
||||||
|
ProbeCategory,
|
||||||
|
ProbeResult,
|
||||||
|
ProbeStatus,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _same_file(a: str | os.PathLike[str], b: str | os.PathLike[str]) -> bool:
|
||||||
|
"""Compare two interpreter paths tolerantly (symlinks, case on Windows)."""
|
||||||
|
try:
|
||||||
|
return os.path.samefile(a, b)
|
||||||
|
except OSError:
|
||||||
|
return os.path.normcase(os.path.abspath(a)) == os.path.normcase(os.path.abspath(b))
|
||||||
|
|
||||||
|
|
||||||
|
#: Client names accepted by ``artemis mcp --install`` (see interfaces/cli/commands/mcp.py).
|
||||||
|
MCP_CLIENT_NAMES: tuple[str, ...] = (
|
||||||
|
"antigravity",
|
||||||
|
"claude",
|
||||||
|
"cursor",
|
||||||
|
"windsurf",
|
||||||
|
"vscode",
|
||||||
|
"cline",
|
||||||
|
"roo",
|
||||||
|
"openclaw",
|
||||||
|
"codex",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _env_has_prefix(prefix: str, *, exclude: tuple[str, ...] = ()) -> bool:
|
||||||
|
return any(key.startswith(prefix) and key not in exclude for key in os.environ)
|
||||||
|
|
||||||
|
|
||||||
|
def detect_mcp_client() -> str | None:
|
||||||
|
"""Best-effort guess of which MCP client launched this process, from ``os.environ``.
|
||||||
|
|
||||||
|
Returns one of :data:`MCP_CLIENT_NAMES` or ``None`` when unsure. MCP hosts
|
||||||
|
spawn the server with their own process environment, so the variables the
|
||||||
|
IDE exports to its child processes leak through. Specific hosts are checked
|
||||||
|
before the generic VS Code markers because Cursor, Windsurf, Cline and Roo
|
||||||
|
are (or run inside) VS Code and inherit ``VSCODE_*`` / ``TERM_PROGRAM``.
|
||||||
|
"""
|
||||||
|
# Antigravity injects its language-server endpoint and CSRF token
|
||||||
|
# (ANTIGRAVITY_LS_ADDRESS / ANTIGRAVITY_CSRF_TOKEN) so child processes can
|
||||||
|
# call `agentapi`; mcp_server/notifiers/agentapi.py relies on the same pair.
|
||||||
|
if _env_has_prefix("ANTIGRAVITY_"):
|
||||||
|
return "antigravity"
|
||||||
|
# Claude Code marks every child with CLAUDECODE=1 plus CLAUDE_CODE_* details
|
||||||
|
# (entrypoint, session id, ...). Claude Desktop shares the same installer
|
||||||
|
# target, so both map to "claude".
|
||||||
|
if os.environ.get("CLAUDECODE") == "1" or _env_has_prefix("CLAUDE_CODE_"):
|
||||||
|
return "claude"
|
||||||
|
# Cursor tags its integrated terminal and spawned processes with
|
||||||
|
# CURSOR_TRACE_ID (and CURSOR_AGENT in agent mode).
|
||||||
|
if "CURSOR_TRACE_ID" in os.environ or _env_has_prefix("CURSOR_"):
|
||||||
|
return "cursor"
|
||||||
|
# Windsurf (Codeium) exports WINDSURF_* to its child processes.
|
||||||
|
if _env_has_prefix("WINDSURF_"):
|
||||||
|
return "windsurf"
|
||||||
|
# Codex CLI sets CODEX_SANDBOX* / CODEX_* for the processes it runs.
|
||||||
|
# CODEX_HOME alone is excluded: users set it by hand to relocate the Codex
|
||||||
|
# config directory, so on its own it does not prove Codex launched us.
|
||||||
|
if _env_has_prefix("CODEX_", exclude=("CODEX_HOME",)):
|
||||||
|
return "codex"
|
||||||
|
# OPENCLAW_WEBHOOK_URL is a notifier setting users put in any host's env
|
||||||
|
# block (see mcp_server/notifiers/webhook.py); only other OPENCLAW_* vars
|
||||||
|
# indicate the OpenClaw runtime itself.
|
||||||
|
if _env_has_prefix("OPENCLAW_", exclude=("OPENCLAW_WEBHOOK_URL",)):
|
||||||
|
return "openclaw"
|
||||||
|
# Cline / Roo Code run as VS Code extensions; their own prefixes are rare
|
||||||
|
# but unambiguous when present, and must win over the generic VS Code check.
|
||||||
|
if _env_has_prefix("CLINE_"):
|
||||||
|
return "cline"
|
||||||
|
if _env_has_prefix("ROO_"):
|
||||||
|
return "roo"
|
||||||
|
# Plain VS Code (Copilot agent mode) exports VSCODE_PID / VSCODE_IPC_HOOK
|
||||||
|
# and sets TERM_PROGRAM=vscode. Weakest signal, so it is checked last.
|
||||||
|
if (
|
||||||
|
"VSCODE_PID" in os.environ
|
||||||
|
or "VSCODE_IPC_HOOK" in os.environ
|
||||||
|
or os.environ.get("TERM_PROGRAM") == "vscode"
|
||||||
|
):
|
||||||
|
return "vscode"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def mcp_install_actions(client: str | None) -> list[ProbeAction]:
|
||||||
|
"""Actions that regenerate the MCP client config for ``client`` (or ask which one)."""
|
||||||
|
if client:
|
||||||
|
return [
|
||||||
|
ProbeAction(
|
||||||
|
action_type="command",
|
||||||
|
label="Regenerate MCP Config",
|
||||||
|
payload=f"uv run artemis mcp --install {client}",
|
||||||
|
)
|
||||||
|
]
|
||||||
|
return [
|
||||||
|
ProbeAction(
|
||||||
|
action_type="hint",
|
||||||
|
label="Regenerate MCP Config",
|
||||||
|
payload=(
|
||||||
|
"Could not tell which MCP client launched this server. Ask the user which IDE"
|
||||||
|
" they are using, then run `uv run artemis mcp --install <client>` with"
|
||||||
|
" <client> being one of: " + ", ".join(MCP_CLIENT_NAMES) + "."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def project_venv_python(project_root: Path) -> Path:
|
||||||
|
"""Return the interpreter path the installer writes into MCP client configs."""
|
||||||
|
if sys.platform == "win32":
|
||||||
|
return project_root / ".venv" / "Scripts" / "python.exe"
|
||||||
|
return project_root / ".venv" / "bin" / "python"
|
||||||
|
|
||||||
|
|
||||||
|
def directory_is_writable(path: Path) -> tuple[bool, str | None]:
|
||||||
|
"""Try to create and delete a scratch file; return (ok, error)."""
|
||||||
|
try:
|
||||||
|
path.mkdir(parents=True, exist_ok=True)
|
||||||
|
with tempfile.NamedTemporaryFile(dir=path, prefix=".artemis-diag-", delete=True):
|
||||||
|
pass
|
||||||
|
return True, None
|
||||||
|
except OSError as exc:
|
||||||
|
return False, str(exc)
|
||||||
|
|
||||||
|
|
||||||
|
def intended_traces_path() -> Path:
|
||||||
|
"""Return the traces directory the settings loader *would* pick, without creating it.
|
||||||
|
|
||||||
|
Mirrors the selection order of :func:`artemis.config.paths.get_default_traces_path`
|
||||||
|
(``ARTEMIS_TRACES_DIR`` env, else the user app dir when the install mode
|
||||||
|
keeps mutable state there, else ``ROOT_DIR/traces``) minus its ``mkdir``.
|
||||||
|
Used only when that function raised, so the probe can still name the
|
||||||
|
offending directory in its FAIL result instead of crashing.
|
||||||
|
"""
|
||||||
|
from artemis.config import paths as artemis_paths
|
||||||
|
|
||||||
|
env_traces = os.getenv(artemis_paths.ENV_ARTEMIS_TRACES_DIR)
|
||||||
|
if env_traces:
|
||||||
|
return Path(env_traces)
|
||||||
|
if artemis_paths._use_user_app_dir():
|
||||||
|
return artemis_paths.get_app_dir() / "traces"
|
||||||
|
return Path(artemis_paths.ROOT_DIR) / "traces"
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_traces_dir() -> tuple[Path, str | None]:
|
||||||
|
"""Return ``(traces_dir, error)``; ``error`` is set when the directory could not be created.
|
||||||
|
|
||||||
|
``get_default_traces_path`` creates the directory as a side effect, so on a
|
||||||
|
read-only parent it raises before :func:`directory_is_writable` ever runs.
|
||||||
|
Surface that as data rather than letting the probe itself crash.
|
||||||
|
"""
|
||||||
|
from artemis.config.paths import get_default_traces_path
|
||||||
|
|
||||||
|
try:
|
||||||
|
return Path(get_default_traces_path()), None
|
||||||
|
except OSError as exc:
|
||||||
|
return intended_traces_path(), str(exc)
|
||||||
|
|
||||||
|
|
||||||
|
class IntegrationHostProbe(BaseProbe):
|
||||||
|
"""Probe verifying the MCP / IDE host process can actually run Artemis tasks."""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def probe_id(self) -> str:
|
||||||
|
return "integration_host"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def category(self) -> ProbeCategory:
|
||||||
|
return ProbeCategory.RUNTIME
|
||||||
|
|
||||||
|
@property
|
||||||
|
def is_blocker(self) -> bool:
|
||||||
|
return True
|
||||||
|
|
||||||
|
async def probe(self) -> ProbeResult:
|
||||||
|
# Lazy imports keep this module importable in stripped test environments
|
||||||
|
# and avoid a core -> runtime import at package import time.
|
||||||
|
from artemis.config.paths import ROOT_DIR, get_env_file, is_source_checkout
|
||||||
|
from artemis.runtime.daemon_client import (
|
||||||
|
DEFAULT_DAEMON_HOST,
|
||||||
|
DEFAULT_DAEMON_PORT,
|
||||||
|
daemon_log_path,
|
||||||
|
is_artemis_daemon,
|
||||||
|
is_standalone_forced,
|
||||||
|
)
|
||||||
|
from artemis.runtime.server_lifecycle import is_port_in_use
|
||||||
|
|
||||||
|
project_root = Path(ROOT_DIR)
|
||||||
|
source_checkout = is_source_checkout()
|
||||||
|
server_python = Path(sys.executable)
|
||||||
|
# A project virtualenv is only meaningful for a source checkout. In a
|
||||||
|
# wheel install ROOT_DIR is site-packages, which never carries a .venv;
|
||||||
|
# the current interpreter also runs background tasks.
|
||||||
|
venv_python: Path | None = project_venv_python(project_root) if source_checkout else None
|
||||||
|
venv_exists = venv_python is not None and venv_python.is_file()
|
||||||
|
runner_python = venv_python if venv_exists else server_python
|
||||||
|
interpreter_matches_venv = venv_exists and _same_file(server_python, venv_python)
|
||||||
|
mcp_client = detect_mcp_client()
|
||||||
|
|
||||||
|
env_file = Path(get_env_file())
|
||||||
|
traces_dir, traces_error = resolve_traces_dir()
|
||||||
|
if traces_error is None:
|
||||||
|
traces_writable, traces_error = directory_is_writable(traces_dir)
|
||||||
|
else:
|
||||||
|
traces_writable = False
|
||||||
|
|
||||||
|
standalone = is_standalone_forced()
|
||||||
|
# Identity, not liveness: a dev server squatting on the port answers
|
||||||
|
# HTTP too, and must show up as a hijacked port, not as the daemon.
|
||||||
|
daemon_reachable = False if standalone else is_artemis_daemon()
|
||||||
|
port_in_use = is_port_in_use(DEFAULT_DAEMON_PORT, DEFAULT_DAEMON_HOST)
|
||||||
|
port_hijacked = port_in_use and not daemon_reachable and not standalone
|
||||||
|
|
||||||
|
metadata: dict[str, Any] = {
|
||||||
|
"os": sys.platform,
|
||||||
|
"mcp_client": mcp_client,
|
||||||
|
"project_root": str(project_root),
|
||||||
|
"is_source_checkout": source_checkout,
|
||||||
|
"server_python": str(server_python),
|
||||||
|
"venv_python": str(venv_python) if venv_python is not None else None,
|
||||||
|
"venv_exists": venv_exists,
|
||||||
|
"interpreter_matches_venv": interpreter_matches_venv,
|
||||||
|
"runner_python": str(runner_python),
|
||||||
|
"env_file": str(env_file),
|
||||||
|
"env_file_exists": env_file.is_file(),
|
||||||
|
"traces_dir": str(traces_dir),
|
||||||
|
"traces_dir_writable": traces_writable,
|
||||||
|
"traces_dir_error": traces_error,
|
||||||
|
"daemon": {
|
||||||
|
"host": DEFAULT_DAEMON_HOST,
|
||||||
|
"port": DEFAULT_DAEMON_PORT,
|
||||||
|
"standalone_forced": standalone,
|
||||||
|
"reachable": daemon_reachable,
|
||||||
|
"port_in_use": port_in_use,
|
||||||
|
"port_held_by_other_process": port_hijacked,
|
||||||
|
"log_path": str(daemon_log_path()),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
problems: list[str] = []
|
||||||
|
actions: list[ProbeAction] = []
|
||||||
|
status = ProbeStatus.PASS
|
||||||
|
|
||||||
|
if not traces_writable:
|
||||||
|
status = ProbeStatus.FAIL
|
||||||
|
problems.append(f"traces directory {traces_dir} is not writable ({traces_error})")
|
||||||
|
actions.append(
|
||||||
|
ProbeAction(
|
||||||
|
action_type="hint",
|
||||||
|
label="Fix Traces Directory",
|
||||||
|
payload=(
|
||||||
|
f"Make {traces_dir} writable for the user running the MCP server, or point "
|
||||||
|
"ARTEMIS_TRACES_DIR at a writable directory in the MCP server's env block, "
|
||||||
|
"then restart the MCP server."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if port_hijacked:
|
||||||
|
status = ProbeStatus.WARN if status is ProbeStatus.PASS else status
|
||||||
|
problems.append(
|
||||||
|
f"port {DEFAULT_DAEMON_PORT} is held by a process that is not the Artemis daemon"
|
||||||
|
)
|
||||||
|
actions.append(
|
||||||
|
ProbeAction(
|
||||||
|
action_type="hint",
|
||||||
|
label="Free the Daemon Port",
|
||||||
|
payload=(
|
||||||
|
f"Another program is listening on {DEFAULT_DAEMON_HOST}:{DEFAULT_DAEMON_PORT}"
|
||||||
|
" (often a dev server). Stop it, or set ARTEMIS_DAEMON_PORT to a free port in"
|
||||||
|
" the MCP server's env block and restart the MCP server. Tasks still run in"
|
||||||
|
" standalone mode meanwhile, without the shared scheduler."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if not source_checkout:
|
||||||
|
# Wheel / frozen install: no virtualenv expectation, nothing to warn about.
|
||||||
|
pass
|
||||||
|
elif venv_exists and not interpreter_matches_venv:
|
||||||
|
status = ProbeStatus.WARN if status is ProbeStatus.PASS else status
|
||||||
|
problems.append(
|
||||||
|
f"MCP server runs on {server_python}, not the project virtualenv {venv_python}"
|
||||||
|
)
|
||||||
|
actions.extend(mcp_install_actions(mcp_client))
|
||||||
|
actions.append(
|
||||||
|
ProbeAction(
|
||||||
|
action_type="hint",
|
||||||
|
label="Interpreter Mismatch",
|
||||||
|
payload=(
|
||||||
|
"The MCP client config should point `command` at the .venv interpreter so the"
|
||||||
|
" server and the background runner share one package set. Restart the MCP"
|
||||||
|
" server after changing the config."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
elif not venv_exists:
|
||||||
|
status = ProbeStatus.WARN if status is ProbeStatus.PASS else status
|
||||||
|
problems.append(
|
||||||
|
"no project virtualenv (.venv) found; runner falls back to the server interpreter"
|
||||||
|
)
|
||||||
|
actions.append(
|
||||||
|
ProbeAction(
|
||||||
|
action_type="command",
|
||||||
|
label="Create Virtualenv",
|
||||||
|
payload="uv sync",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if not env_file.is_file():
|
||||||
|
actions.append(
|
||||||
|
ProbeAction(
|
||||||
|
action_type="hint",
|
||||||
|
label="Env File Location",
|
||||||
|
payload=(
|
||||||
|
f"No .env file at {env_file}. API keys are read from that file or from the MCP"
|
||||||
|
" server's env block; the file is created by `artemis init`."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if not daemon_reachable and not port_hijacked:
|
||||||
|
actions.append(
|
||||||
|
ProbeAction(
|
||||||
|
action_type="hint",
|
||||||
|
label="Daemon Auto-Start",
|
||||||
|
payload=(
|
||||||
|
"Standalone mode is forced (ARTEMIS_STANDALONE=1); tasks run as detached"
|
||||||
|
" runner processes."
|
||||||
|
if standalone
|
||||||
|
else "The Artemis daemon is not running; mobile_run_task starts it on the"
|
||||||
|
f" first task. Its log is {daemon_log_path()}."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if status is ProbeStatus.PASS:
|
||||||
|
summary = "Host Ready"
|
||||||
|
description = (
|
||||||
|
f"MCP server runs on {server_python}; runner uses {runner_python}. "
|
||||||
|
f"Traces at {traces_dir}. "
|
||||||
|
+ (
|
||||||
|
"Daemon reachable."
|
||||||
|
if daemon_reachable
|
||||||
|
else "Daemon will auto-start on the first task."
|
||||||
|
)
|
||||||
|
)
|
||||||
|
actions.insert(
|
||||||
|
0,
|
||||||
|
ProbeAction(
|
||||||
|
action_type="hint",
|
||||||
|
label="Host OK",
|
||||||
|
payload="Interpreter, traces directory and daemon port look consistent.",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
summary = "Host Misconfigured" if status is ProbeStatus.FAIL else "Host Warning"
|
||||||
|
joined = "; ".join(problems)
|
||||||
|
description = joined[:1].upper() + joined[1:] + "."
|
||||||
|
|
||||||
|
# Only an unwritable traces directory stops tasks from running. The
|
||||||
|
# WARN findings (interpreter mismatch, missing venv, squatted port) are
|
||||||
|
# degradations the runner works around, so they must not turn the
|
||||||
|
# verdict into "blocked" — a pip/conda checkout without .venv would
|
||||||
|
# otherwise never leave that state.
|
||||||
|
return ProbeResult(
|
||||||
|
id=self.probe_id,
|
||||||
|
category=self.category,
|
||||||
|
title="MCP / IDE Integration Host",
|
||||||
|
status=status,
|
||||||
|
is_blocker=status is not ProbeStatus.WARN,
|
||||||
|
summary=summary,
|
||||||
|
description=description,
|
||||||
|
metadata=metadata,
|
||||||
|
actions=actions,
|
||||||
|
)
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
# Copyright 2026 Google LLC
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
"""Readiness collection and verdict shared by every diagnosis entry point.
|
||||||
|
|
||||||
|
``artemis doctor`` and the ``mobile_diagnose`` MCP tool must agree on which
|
||||||
|
probes run, in which order they are fixed, and what "ready" means. This module
|
||||||
|
holds exactly that common core; each entry point layers its own extra
|
||||||
|
conditions, fix policy and rendering on top.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
from collections.abc import Sequence
|
||||||
|
from typing import Literal
|
||||||
|
|
||||||
|
from artemis.core.diagnostics.engine import readiness_engine
|
||||||
|
from artemis.core.diagnostics.probes.host_probe import IntegrationHostProbe
|
||||||
|
from artemis.core.diagnostics.schema import ProbeResult, ProbeStatus, SystemReadinessReport
|
||||||
|
|
||||||
|
Verdict = Literal["ready", "degraded", "blocked"]
|
||||||
|
|
||||||
|
#: Fix order. Runtime problems mask everything after them, credentials mask
|
||||||
|
#: device problems (a task cannot start without a model), and the toolchain is
|
||||||
|
#: optional.
|
||||||
|
CHECK_ORDER: dict[str, int] = {
|
||||||
|
"python_runtime": 0,
|
||||||
|
"system_config": 1,
|
||||||
|
"integration_host": 2,
|
||||||
|
"gemini_api_key": 3,
|
||||||
|
"android_adb": 4,
|
||||||
|
"toolchain": 5,
|
||||||
|
"vision_ocr_key": 6,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def sort_by_fix_order(results: Sequence[ProbeResult]) -> list[ProbeResult]:
|
||||||
|
"""Order probe results so the first problem listed is the one to fix first."""
|
||||||
|
return sorted(results, key=lambda r: CHECK_ORDER.get(r.id, len(CHECK_ORDER)))
|
||||||
|
|
||||||
|
|
||||||
|
async def collect_readiness() -> tuple[SystemReadinessReport, ProbeResult]:
|
||||||
|
"""Run the engine probes and the integration-host probe concurrently."""
|
||||||
|
report, host = await asyncio.gather(
|
||||||
|
readiness_engine.run_all(force_refresh=True),
|
||||||
|
IntegrationHostProbe().probe(),
|
||||||
|
)
|
||||||
|
return report, host
|
||||||
|
|
||||||
|
|
||||||
|
def base_verdict(results: Sequence[ProbeResult]) -> Verdict:
|
||||||
|
"""Verdict from probe results alone.
|
||||||
|
|
||||||
|
Blocked when there is no blocker at all (nothing was checked) or any
|
||||||
|
blocker is not PASS; degraded when only optional checks fail; ready
|
||||||
|
otherwise.
|
||||||
|
"""
|
||||||
|
blockers = [r for r in results if r.is_blocker]
|
||||||
|
if not blockers or any(r.status is not ProbeStatus.PASS for r in blockers):
|
||||||
|
return "blocked"
|
||||||
|
if any(r.status is not ProbeStatus.PASS for r in results):
|
||||||
|
return "degraded"
|
||||||
|
return "ready"
|
||||||
|
|
||||||
|
|
||||||
|
def adb_keys_corrupted(results: Sequence[ProbeResult]) -> bool:
|
||||||
|
"""Whether the ADB probe reported corrupted RSA keys (the one self-heal both entry points share)."""
|
||||||
|
adb = next((r for r in results if r.id == "android_adb"), None)
|
||||||
|
if adb is None:
|
||||||
|
return False
|
||||||
|
return bool((adb.metadata.get("adb_keys") or {}).get("is_corrupted"))
|
||||||
@@ -666,6 +666,13 @@ class DataEngine:
|
|||||||
if session and session.end_time is not None and session.status not in ("running", "paused"):
|
if session and session.end_time is not None and session.status not in ("running", "paused"):
|
||||||
logger.debug(f"Session end already published for {session_id}; skipping duplicate")
|
logger.debug(f"Session end already published for {session_id}; skipping duplicate")
|
||||||
return
|
return
|
||||||
|
# Session-level LLM usage line (cache-hit ratios per source), best-effort.
|
||||||
|
try:
|
||||||
|
from artemis.services.token_meter import log_session_summary
|
||||||
|
|
||||||
|
log_session_summary(session_id)
|
||||||
|
except Exception as e:
|
||||||
|
logger.debug(f"Session usage summary skipped: {e}")
|
||||||
if session:
|
if session:
|
||||||
session.end_time = end_time
|
session.end_time = end_time
|
||||||
session.status = status
|
session.status = status
|
||||||
@@ -877,7 +884,7 @@ class DataEngine:
|
|||||||
if pre_image_name and post_image_name and pre_image_name == post_image_name:
|
if pre_image_name and post_image_name and pre_image_name == post_image_name:
|
||||||
post_image_name = None
|
post_image_name = None
|
||||||
|
|
||||||
# M5: persist the foreground app for the recall search surface (the
|
# Persist the foreground app for the recall search surface (the
|
||||||
# parameter was historically accepted but never stored). The explicit
|
# parameter was historically accepted but never stored). The explicit
|
||||||
# parameter wins; otherwise it is derived best-effort from the UI
|
# parameter wins; otherwise it is derived best-effort from the UI
|
||||||
# tree's package attributes — a pure string scan, no device call.
|
# tree's package attributes — a pure string scan, no device call.
|
||||||
@@ -889,7 +896,7 @@ class DataEngine:
|
|||||||
except (AttributeError, TypeError, ValueError):
|
except (AttributeError, TypeError, ValueError):
|
||||||
pass
|
pass
|
||||||
|
|
||||||
# M4: stamp perceptual hashes for the local screen-similarity hint
|
# Stamp perceptual hashes for the local screen-similarity hint
|
||||||
# (pure PIL dHash; best-effort, computed synchronously to keep the
|
# (pure PIL dHash; best-effort, computed synchronously to keep the
|
||||||
# record immutable once handed to the background writer).
|
# record immutable once handed to the background writer).
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -14,6 +14,7 @@
|
|||||||
|
|
||||||
import base64
|
import base64
|
||||||
from collections.abc import Callable
|
from collections.abc import Callable
|
||||||
|
import contextlib
|
||||||
import functools
|
import functools
|
||||||
import hashlib
|
import hashlib
|
||||||
import inspect
|
import inspect
|
||||||
@@ -175,6 +176,20 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
|
|||||||
def on_tool_error(self, error: Any, *, run_id: uuid.UUID, **kwargs: Any) -> Any:
|
def on_tool_error(self, error: Any, *, run_id: uuid.UUID, **kwargs: Any) -> Any:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _detached_lens() -> bool:
|
||||||
|
"""True inside ``detached_trace('lens:...')``: a background lens call
|
||||||
|
(visual summary, segment capsule) that belongs to no agent step."""
|
||||||
|
node_name = CURRENT_NODE_NAME.get() or ""
|
||||||
|
return CURRENT_TRACE_ID.get() is None and node_name.startswith("lens:")
|
||||||
|
|
||||||
|
def _current_step_id(self):
|
||||||
|
"""The step a trace attaches to: the engine's current step, or None for
|
||||||
|
a detached lens call (its reply must never read as that step's work)."""
|
||||||
|
if self._detached_lens():
|
||||||
|
return None
|
||||||
|
return self.ctx.data_engine.current_step_id
|
||||||
|
|
||||||
def on_llm_start(
|
def on_llm_start(
|
||||||
self,
|
self,
|
||||||
serialized: dict[str, Any],
|
serialized: dict[str, Any],
|
||||||
@@ -200,7 +215,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
|
|||||||
trace_id=run_id,
|
trace_id=run_id,
|
||||||
parent_trace_id=self._resolve_parent_id(parent_run_id, kwargs),
|
parent_trace_id=self._resolve_parent_id(parent_run_id, kwargs),
|
||||||
status="running",
|
status="running",
|
||||||
step_id=self.ctx.data_engine.current_step_id,
|
step_id=self._current_step_id(),
|
||||||
)
|
)
|
||||||
|
|
||||||
def on_chat_model_start(
|
def on_chat_model_start(
|
||||||
@@ -233,7 +248,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
|
|||||||
trace_id=run_id,
|
trace_id=run_id,
|
||||||
parent_trace_id=self._resolve_parent_id(parent_run_id, kwargs),
|
parent_trace_id=self._resolve_parent_id(parent_run_id, kwargs),
|
||||||
status="running",
|
status="running",
|
||||||
step_id=self.ctx.data_engine.current_step_id,
|
step_id=self._current_step_id(),
|
||||||
)
|
)
|
||||||
|
|
||||||
def _safe_uuid5(self, parent_id, name: str) -> uuid.UUID:
|
def _safe_uuid5(self, parent_id, name: str) -> uuid.UUID:
|
||||||
@@ -253,6 +268,11 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
|
|||||||
) -> Any:
|
) -> Any:
|
||||||
"""Run when LLM ends running."""
|
"""Run when LLM ends running."""
|
||||||
if self.ctx.data_engine:
|
if self.ctx.data_engine:
|
||||||
|
# A background lens call (see ``detached_trace``) belongs to no step:
|
||||||
|
# its reply is a compression product, never an agent's reasoning, so
|
||||||
|
# it is neither attached to the current step nor recorded as thinking.
|
||||||
|
detached = self._detached_lens()
|
||||||
|
step_id = self._current_step_id()
|
||||||
generations = getattr(response, "generations", [])
|
generations = getattr(response, "generations", [])
|
||||||
flat_generations = []
|
flat_generations = []
|
||||||
if generations:
|
if generations:
|
||||||
@@ -308,7 +328,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
|
|||||||
parent_trace_id=self._resolve_parent_id(parent_run_id),
|
parent_trace_id=self._resolve_parent_id(parent_run_id),
|
||||||
status="success",
|
status="success",
|
||||||
duration=duration,
|
duration=duration,
|
||||||
step_id=self.ctx.data_engine.current_step_id,
|
step_id=step_id,
|
||||||
)
|
)
|
||||||
|
|
||||||
native_thoughts = []
|
native_thoughts = []
|
||||||
@@ -327,6 +347,8 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
|
|||||||
elif gen_item.get("text") and str(gen_item.get("text")).strip():
|
elif gen_item.get("text") and str(gen_item.get("text")).strip():
|
||||||
raw_thoughts.append(str(gen_item.get("text")).strip())
|
raw_thoughts.append(str(gen_item.get("text")).strip())
|
||||||
|
|
||||||
|
if detached:
|
||||||
|
native_thoughts, raw_thoughts = [], []
|
||||||
if native_thoughts:
|
if native_thoughts:
|
||||||
self.ctx.data_engine.record_trace(
|
self.ctx.data_engine.record_trace(
|
||||||
type="thinking",
|
type="thinking",
|
||||||
@@ -335,7 +357,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
|
|||||||
trace_id=self._safe_uuid5(run_id, "thinking"),
|
trace_id=self._safe_uuid5(run_id, "thinking"),
|
||||||
parent_trace_id=run_id,
|
parent_trace_id=run_id,
|
||||||
status="success",
|
status="success",
|
||||||
step_id=self.ctx.data_engine.current_step_id,
|
step_id=step_id,
|
||||||
)
|
)
|
||||||
if raw_thoughts:
|
if raw_thoughts:
|
||||||
self.ctx.data_engine.record_trace(
|
self.ctx.data_engine.record_trace(
|
||||||
@@ -345,7 +367,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
|
|||||||
trace_id=self._safe_uuid5(run_id, "raw_thinking"),
|
trace_id=self._safe_uuid5(run_id, "raw_thinking"),
|
||||||
parent_trace_id=run_id,
|
parent_trace_id=run_id,
|
||||||
status="success",
|
status="success",
|
||||||
step_id=self.ctx.data_engine.current_step_id,
|
step_id=step_id,
|
||||||
)
|
)
|
||||||
|
|
||||||
def on_llm_error(
|
def on_llm_error(
|
||||||
@@ -379,7 +401,7 @@ class DataEngineCallbackHandler(BaseCallbackHandler):
|
|||||||
parent_trace_id=self._resolve_parent_id(parent_run_id),
|
parent_trace_id=self._resolve_parent_id(parent_run_id),
|
||||||
status="failed",
|
status="failed",
|
||||||
duration=duration,
|
duration=duration,
|
||||||
step_id=self.ctx.data_engine.current_step_id,
|
step_id=self._current_step_id(),
|
||||||
)
|
)
|
||||||
|
|
||||||
def _serialize_message(self, msg: BaseMessage) -> dict[str, Any]:
|
def _serialize_message(self, msg: BaseMessage) -> dict[str, Any]:
|
||||||
@@ -666,6 +688,36 @@ class TraceSpan:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@contextlib.contextmanager
|
||||||
|
def detached_trace(node_name: str | None = None):
|
||||||
|
"""Run a block outside the caller's trace span.
|
||||||
|
|
||||||
|
Background work (the step-memory lenses: visual-transition summaries,
|
||||||
|
segment capsules) is dispatched while an agent's step span is current, and
|
||||||
|
an ``asyncio`` task inherits that context. Every model call made inside
|
||||||
|
the block would then hang off the agent's span: the callback handler
|
||||||
|
records its reply as a ``thinking``/``raw_thinking`` child of that span, so
|
||||||
|
``replay_steps``/``search_history`` later show a capsule JSON as the
|
||||||
|
operator's own reasoning, and the stream deltas surface under the
|
||||||
|
operator's timeline entry.
|
||||||
|
|
||||||
|
Inside the block :data:`CURRENT_TRACE_ID` is cleared (the block's traces
|
||||||
|
have no parent span and can never be attributed to the caller's step) and
|
||||||
|
:data:`CURRENT_NODE_NAME` is replaced by ``node_name`` (when given), so
|
||||||
|
usage metering labels the calls after the background job rather than the
|
||||||
|
agent that happened to dispatch it. Both are restored on exit, exceptions
|
||||||
|
included. Only the current task's context is touched.
|
||||||
|
"""
|
||||||
|
token = CURRENT_TRACE_ID.set(None)
|
||||||
|
token_name = CURRENT_NODE_NAME.set(node_name) if node_name is not None else None
|
||||||
|
try:
|
||||||
|
yield
|
||||||
|
finally:
|
||||||
|
CURRENT_TRACE_ID.reset(token)
|
||||||
|
if token_name is not None:
|
||||||
|
CURRENT_NODE_NAME.reset(token_name)
|
||||||
|
|
||||||
|
|
||||||
def trace_langchain_tool(tool: BaseTool, ctx: ArtemisContext) -> BaseTool:
|
def trace_langchain_tool(tool: BaseTool, ctx: ArtemisContext) -> BaseTool:
|
||||||
"""Wraps a LangChain tool to trace its execution.
|
"""Wraps a LangChain tool to trace its execution.
|
||||||
|
|
||||||
|
|||||||
@@ -70,7 +70,7 @@ def _escape_for_adb_text(s: str) -> str:
|
|||||||
|
|
||||||
|
|
||||||
class AndroidAdbDriver(BaseDeviceDriver):
|
class AndroidAdbDriver(BaseDeviceDriver):
|
||||||
"""Production-grade Android Device Driver powered by ADB and UIAutomator2."""
|
"""Android device driver using ADB and UIAutomator2."""
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
|
|||||||
+162
-14
@@ -135,6 +135,10 @@ class TestSummary(BaseModel):
|
|||||||
inconclusive: int = 0
|
inconclusive: int = 0
|
||||||
unchecked: int = 0
|
unchecked: int = 0
|
||||||
failed_items: list[dict] = Field(default_factory=list)
|
failed_items: list[dict] = Field(default_factory=list)
|
||||||
|
#: Check lines the user retired mid-run (dropped from the plan under user
|
||||||
|
#: guidance): their earlier verdicts are reported here, never as failures.
|
||||||
|
retired: int = 0
|
||||||
|
retired_items: list[dict] = Field(default_factory=list)
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def total(self) -> int:
|
def total(self) -> int:
|
||||||
@@ -506,6 +510,112 @@ def _finding_registry(ctx) -> dict:
|
|||||||
return registry
|
return registry
|
||||||
|
|
||||||
|
|
||||||
|
def _finding_items(ctx) -> dict:
|
||||||
|
"""checkpoint_id -> ``{(kind, item_text), ...}`` of the failed verify items
|
||||||
|
that produced the headline currently held in :func:`_finding_registry`.
|
||||||
|
|
||||||
|
Sibling map of the registry (same lifetime, same keys) so a headline can
|
||||||
|
be retired when user guidance withdraws every item behind it.
|
||||||
|
"""
|
||||||
|
items = getattr(ctx, "checker_finding_items", None)
|
||||||
|
if not isinstance(items, dict):
|
||||||
|
items = {}
|
||||||
|
try:
|
||||||
|
ctx.checker_finding_items = items
|
||||||
|
except (AttributeError, TypeError, ValueError):
|
||||||
|
pass
|
||||||
|
return items
|
||||||
|
|
||||||
|
|
||||||
|
def _register_finding(ctx, checkpoint_id: str, verify_failures: list) -> None:
|
||||||
|
_finding_registry(ctx)[checkpoint_id] = _render_finding_headline(checkpoint_id, verify_failures)
|
||||||
|
_finding_items(ctx)[checkpoint_id] = {
|
||||||
|
(str(getattr(v, "kind", "") or ""), str(getattr(v, "item_text", "") or ""))
|
||||||
|
for v in verify_failures
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _clear_finding(ctx, checkpoint_id: str) -> bool:
|
||||||
|
"""Drops a checkpoint's standing headline; True when one was registered."""
|
||||||
|
_finding_items(ctx).pop(checkpoint_id, None)
|
||||||
|
return _finding_registry(ctx).pop(checkpoint_id, None) is not None
|
||||||
|
|
||||||
|
|
||||||
|
def _retired_checks(ctx) -> set[tuple[str, str]]:
|
||||||
|
retired = getattr(ctx, "guidance_retired_checks", None)
|
||||||
|
return retired if isinstance(retired, set) else set()
|
||||||
|
|
||||||
|
|
||||||
|
def retire_check_items(ctx, items) -> None:
|
||||||
|
"""Records ``(kind, text)`` check items the Operator dropped under user
|
||||||
|
guidance and withdraws their already-materialized findings.
|
||||||
|
|
||||||
|
Recording feeds the harvest gate (:func:`_active_verdicts`) and the run
|
||||||
|
outcome. Cleanup: every standing headline produced *only* by now-retired
|
||||||
|
items is popped and the plan's finding lines are re-projected; a headline
|
||||||
|
that still names a live item stays untouched.
|
||||||
|
"""
|
||||||
|
incoming = {(str(kind), str(text)) for kind, text in items}
|
||||||
|
if not incoming:
|
||||||
|
return
|
||||||
|
retired = _retired_checks(ctx) | incoming
|
||||||
|
ctx.guidance_retired_checks = retired
|
||||||
|
registry = _finding_registry(ctx)
|
||||||
|
produced_by = _finding_items(ctx)
|
||||||
|
stale = [cid for cid in list(registry) if produced_by.get(cid) and produced_by[cid] <= retired]
|
||||||
|
for cid in stale:
|
||||||
|
_clear_finding(ctx, cid)
|
||||||
|
logger.info(
|
||||||
|
f"Checkpoint {cid}: standing finding headline withdrawn — every item"
|
||||||
|
" behind it was retired by user guidance."
|
||||||
|
)
|
||||||
|
if stale:
|
||||||
|
sync_finding_lines(ctx)
|
||||||
|
|
||||||
|
|
||||||
|
def reinstate_check_items(ctx, items) -> None:
|
||||||
|
"""Removes ``(kind, text)`` items from the retired set: the Operator has
|
||||||
|
declared them in the plan again, so their verdicts drive side effects
|
||||||
|
once more and the run outcome counts them normally. A no-op when none of
|
||||||
|
the given items is retired."""
|
||||||
|
retired = _retired_checks(ctx)
|
||||||
|
if not retired:
|
||||||
|
return
|
||||||
|
back = {(str(kind), str(text)) for kind, text in items} & retired
|
||||||
|
if not back:
|
||||||
|
return
|
||||||
|
ctx.guidance_retired_checks = retired - back
|
||||||
|
logger.info(
|
||||||
|
"Check line(s) retired under user guidance are declared again and"
|
||||||
|
f" reinstated: {sorted(back)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _active_verdicts(ctx, checkpoint, verdicts: list) -> list:
|
||||||
|
"""The verdicts still allowed to drive side effects: those whose
|
||||||
|
``(kind, item_text)`` was not retired by user guidance. Ledger recording is
|
||||||
|
never filtered — this gate only protects execution state."""
|
||||||
|
retired = _retired_checks(ctx)
|
||||||
|
if not retired:
|
||||||
|
return list(verdicts)
|
||||||
|
active: list = []
|
||||||
|
dropped: list = []
|
||||||
|
for v in verdicts:
|
||||||
|
key = (str(getattr(v, "kind", "") or ""), str(getattr(v, "item_text", "") or ""))
|
||||||
|
(dropped if key in retired else active).append(v)
|
||||||
|
if dropped:
|
||||||
|
logger.info(
|
||||||
|
f"Checkpoint {checkpoint.checkpoint_id}: {len(dropped)} verdict(s)"
|
||||||
|
" recorded but dropped from side effects — their check items were"
|
||||||
|
" retired by user guidance: "
|
||||||
|
+ ", ".join(
|
||||||
|
f"[{k}] '{t}'"
|
||||||
|
for k, t in ((getattr(v, "kind", ""), getattr(v, "item_text", "")) for v in dropped)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return active
|
||||||
|
|
||||||
|
|
||||||
def _render_finding_headline(checkpoint_id: str, verify_failures: list) -> str:
|
def _render_finding_headline(checkpoint_id: str, verify_failures: list) -> str:
|
||||||
first = verify_failures[0]
|
first = verify_failures[0]
|
||||||
evidence = str(getattr(first, "evidence", "") or "").strip()
|
evidence = str(getattr(first, "evidence", "") or "").strip()
|
||||||
@@ -778,16 +888,26 @@ def harvest_run(
|
|||||||
# Layer 3: the per-checkpoint repair log accumulates every booked attempt.
|
# Layer 3: the per-checkpoint repair log accumulates every booked attempt.
|
||||||
_log_checker_note(ctx, run, verdicts)
|
_log_checker_note(ctx, run, verdicts)
|
||||||
|
|
||||||
|
# Guidance gate: a verdict for a check item the user retired mid-run (the
|
||||||
|
# Operator dropped its line under a guidance waiver) is a result for a
|
||||||
|
# withdrawn requirement — booked above, never a repair/halt trigger.
|
||||||
|
active = _active_verdicts(ctx, checkpoint, verdicts)
|
||||||
|
all_retired = bool(verdicts) and not active
|
||||||
|
|
||||||
# Applicability gate for side effects: the verdict must belong to the
|
# Applicability gate for side effects: the verdict must belong to the
|
||||||
# current attempt of its checkpoint AND the anchored subgoal's text must be
|
# current attempt of its checkpoint AND the anchored subgoal's text must be
|
||||||
# unchanged in the current plan. Stale/mismatched verdicts are ledger-only.
|
# unchanged in the current plan AND at least one verdict must still be
|
||||||
applicable = allow_side_effects and _subgoal_still_current(ctx, checkpoint.checkpoint_id)
|
# live. Stale/mismatched/fully retired verdicts are ledger-only.
|
||||||
|
applicable = (
|
||||||
|
allow_side_effects
|
||||||
|
and not all_retired
|
||||||
|
and _subgoal_still_current(ctx, checkpoint.checkpoint_id)
|
||||||
|
)
|
||||||
user_stopped = bool(state is not None and getattr(state, "user_stop_requested", False))
|
user_stopped = bool(state is not None and getattr(state, "user_stop_requested", False))
|
||||||
|
|
||||||
verify_failures = [v for v in verdicts if v.kind == "verify" and v.status == "failed"]
|
verify_failures = [v for v in active if v.kind == "verify" and v.status == "failed"]
|
||||||
assert_failures = [v for v in verdicts if v.kind == "assert" and v.status == "failed"]
|
assert_failures = [v for v in active if v.kind == "assert" and v.status == "failed"]
|
||||||
|
|
||||||
registry = _finding_registry(ctx)
|
|
||||||
registry_changed = False
|
registry_changed = False
|
||||||
reverted = False
|
reverted = False
|
||||||
|
|
||||||
@@ -799,9 +919,7 @@ def harvest_run(
|
|||||||
reverted = revert_subgoal_status(ctx, checkpoint.checkpoint_id)
|
reverted = revert_subgoal_status(ctx, checkpoint.checkpoint_id)
|
||||||
# Layer 2: register the standing plan headline for this unresolved
|
# Layer 2: register the standing plan headline for this unresolved
|
||||||
# finding; the projection below renders it under the subgoal.
|
# finding; the projection below renders it under the subgoal.
|
||||||
registry[checkpoint.checkpoint_id] = _render_finding_headline(
|
_register_finding(ctx, checkpoint.checkpoint_id, verify_failures)
|
||||||
checkpoint.checkpoint_id, verify_failures
|
|
||||||
)
|
|
||||||
registry_changed = True
|
registry_changed = True
|
||||||
for v in verify_failures:
|
for v in verify_failures:
|
||||||
suggestion = getattr(v, "suggestion", "") or ""
|
suggestion = getattr(v, "suggestion", "") or ""
|
||||||
@@ -820,16 +938,18 @@ def harvest_run(
|
|||||||
# Quota exhausted: the run is settled for this checkpoint — the
|
# Quota exhausted: the run is settled for this checkpoint — the
|
||||||
# standing headline is retired (the ledger and the checker note
|
# standing headline is retired (the ledger and the checker note
|
||||||
# keep the full record).
|
# keep the full record).
|
||||||
registry_changed = registry.pop(checkpoint.checkpoint_id, None) is not None
|
registry_changed = _clear_finding(ctx, checkpoint.checkpoint_id)
|
||||||
logger.warning(
|
logger.warning(
|
||||||
f"Checkpoint {checkpoint.checkpoint_id} exhausted its repair"
|
f"Checkpoint {checkpoint.checkpoint_id} exhausted its repair"
|
||||||
f" quota ({max_repairs}); verdict stays failed, no further"
|
f" quota ({max_repairs}); verdict stays failed, no further"
|
||||||
" repair loop."
|
" repair loop."
|
||||||
)
|
)
|
||||||
elif not verify_failures:
|
elif not verify_failures and not all_retired:
|
||||||
# A booked attempt without failed verify criteria resolves (or
|
# A booked attempt without failed verify criteria resolves (or
|
||||||
# releases) the checkpoint: its standing headline is retired.
|
# releases) the checkpoint: its standing headline is retired. (A fully
|
||||||
registry_changed = registry.pop(checkpoint.checkpoint_id, None) is not None
|
# retired report says nothing about the checkpoint — its headline, if
|
||||||
|
# any, was already handled by ``retire_check_items``.)
|
||||||
|
registry_changed = _clear_finding(ctx, checkpoint.checkpoint_id)
|
||||||
|
|
||||||
if registry_changed:
|
if registry_changed:
|
||||||
sync_finding_lines(ctx)
|
sync_finding_lines(ctx)
|
||||||
@@ -842,11 +962,21 @@ def harvest_run(
|
|||||||
"Assert failure under 'halt' policy: latching halt flag for exit settlement."
|
"Assert failure under 'halt' policy: latching halt flag for exit settlement."
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Event payload: ``verdicts`` carries every booked verdict (ledger view);
|
||||||
|
# ``applicable`` is False when nothing could take effect — including a
|
||||||
|
# report whose check items were all retired by user guidance; ``retired``
|
||||||
|
# lists the ``(kind, item_text)`` pairs of the verdicts dropped by that
|
||||||
|
# guidance gate (empty when none were).
|
||||||
_event(
|
_event(
|
||||||
"done",
|
"done",
|
||||||
verdict_payloads(verdicts),
|
verdict_payloads(verdicts),
|
||||||
applicable=bool(applicable),
|
applicable=bool(applicable),
|
||||||
reverted=bool(reverted),
|
reverted=bool(reverted),
|
||||||
|
retired=[
|
||||||
|
{"kind": p["kind"], "item_text": p["item_text"]}
|
||||||
|
for p in verdict_payloads(verdicts)
|
||||||
|
if (p["kind"], p["item_text"]) in _retired_checks(ctx)
|
||||||
|
],
|
||||||
repairs_used=int(ctx.checkpoint_repairs.get(checkpoint.checkpoint_id, 0)),
|
repairs_used=int(ctx.checkpoint_repairs.get(checkpoint.checkpoint_id, 0)),
|
||||||
)
|
)
|
||||||
return findings
|
return findings
|
||||||
@@ -1006,8 +1136,20 @@ def resolve_item_status(kind: str, records: list[dict]) -> str:
|
|||||||
return "unchecked"
|
return "unchecked"
|
||||||
|
|
||||||
|
|
||||||
def compute_test_summary(check_items: list[CheckItem], records: list[dict]) -> TestSummary:
|
def compute_test_summary(
|
||||||
|
check_items: list[CheckItem],
|
||||||
|
records: list[dict],
|
||||||
|
retired: set[tuple[str, str]] | None = None,
|
||||||
|
) -> TestSummary:
|
||||||
|
"""Folds the verdict ledger into pass/fail counts over the declared items.
|
||||||
|
|
||||||
|
``retired`` names check lines the Operator dropped under user guidance; a
|
||||||
|
line that is retired and no longer declared in the plan is reported under
|
||||||
|
``retired`` with its last recorded verdict, not as a pass or a failure —
|
||||||
|
the user withdrew that requirement.
|
||||||
|
"""
|
||||||
summary = TestSummary()
|
summary = TestSummary()
|
||||||
|
retired = retired or set()
|
||||||
by_item: dict[tuple[str, str], list[dict]] = {}
|
by_item: dict[tuple[str, str], list[dict]] = {}
|
||||||
for r in records:
|
for r in records:
|
||||||
by_item.setdefault((str(r.get("kind")), str(r.get("item_text"))), []).append(r)
|
by_item.setdefault((str(r.get("kind")), str(r.get("item_text"))), []).append(r)
|
||||||
@@ -1026,8 +1168,13 @@ def compute_test_summary(check_items: list[CheckItem], records: list[dict]) -> T
|
|||||||
seen.add(sig)
|
seen.add(sig)
|
||||||
all_items.append(sig)
|
all_items.append(sig)
|
||||||
|
|
||||||
|
declared = {(ci.kind, ci.text) for ci in check_items}
|
||||||
for kind, text in all_items:
|
for kind, text in all_items:
|
||||||
status = resolve_item_status(kind, by_item.get((kind, text), []))
|
status = resolve_item_status(kind, by_item.get((kind, text), []))
|
||||||
|
if (kind, text) in retired and (kind, text) not in declared:
|
||||||
|
summary.retired += 1
|
||||||
|
summary.retired_items.append({"item_text": text, "kind": kind, "last_status": status})
|
||||||
|
continue
|
||||||
if status == "passed":
|
if status == "passed":
|
||||||
summary.passed += 1
|
summary.passed += 1
|
||||||
elif status == "failed":
|
elif status == "failed":
|
||||||
@@ -1052,6 +1199,7 @@ def compute_run_outcome(
|
|||||||
records: list[dict],
|
records: list[dict],
|
||||||
*,
|
*,
|
||||||
verify_blocked: bool,
|
verify_blocked: bool,
|
||||||
|
retired: set[tuple[str, str]] | None = None,
|
||||||
) -> RunOutcome:
|
) -> RunOutcome:
|
||||||
"""Assembles the machine-readable run outcome.
|
"""Assembles the machine-readable run outcome.
|
||||||
|
|
||||||
@@ -1060,7 +1208,7 @@ def compute_run_outcome(
|
|||||||
completion. Assert failures never change ``task_status``: a finished task
|
completion. Assert failures never change ``task_status``: a finished task
|
||||||
with failed assertions is ``completed`` with ``tests.failed > 0``.
|
with failed assertions is ``completed`` with ``tests.failed > 0``.
|
||||||
"""
|
"""
|
||||||
tests = compute_test_summary(list(snapshot.all_check_items), records)
|
tests = compute_test_summary(list(snapshot.all_check_items), records, retired=retired)
|
||||||
if verify_blocked:
|
if verify_blocked:
|
||||||
task_status: Literal["completed", "partial", "blocked"] = "blocked"
|
task_status: Literal["completed", "partial", "blocked"] = "blocked"
|
||||||
elif snapshot.all_top_level_done:
|
elif snapshot.all_top_level_done:
|
||||||
|
|||||||
+59
-5
@@ -16,6 +16,7 @@ import asyncio
|
|||||||
import base64
|
import base64
|
||||||
import functools
|
import functools
|
||||||
import json
|
import json
|
||||||
|
from collections import Counter
|
||||||
from typing import Annotated, Literal
|
from typing import Annotated, Literal
|
||||||
|
|
||||||
from langchain_core.messages import ToolMessage
|
from langchain_core.messages import ToolMessage
|
||||||
@@ -49,6 +50,8 @@ from artemis.graph.checkpoints import (
|
|||||||
publish_checker_event,
|
publish_checker_event,
|
||||||
queue_checkpoints,
|
queue_checkpoints,
|
||||||
read_ledger,
|
read_ledger,
|
||||||
|
reinstate_check_items,
|
||||||
|
retire_check_items,
|
||||||
revert_subgoal_status,
|
revert_subgoal_status,
|
||||||
settle_all_checkpoints,
|
settle_all_checkpoints,
|
||||||
spawn_pending_checkpoints,
|
spawn_pending_checkpoints,
|
||||||
@@ -87,8 +90,10 @@ from artemis.utils.notes import (
|
|||||||
)
|
)
|
||||||
from artemis.utils.plan_grammar import (
|
from artemis.utils.plan_grammar import (
|
||||||
milestones_changed,
|
milestones_changed,
|
||||||
|
missing_check_items,
|
||||||
new_top_level_completions,
|
new_top_level_completions,
|
||||||
parse_plan,
|
parse_plan,
|
||||||
|
render_check_line,
|
||||||
restore_missing_check_items,
|
restore_missing_check_items,
|
||||||
subgoal_hash,
|
subgoal_hash,
|
||||||
unintended_milestone_edits,
|
unintended_milestone_edits,
|
||||||
@@ -432,7 +437,13 @@ async def exit_settlement_node(state: State, ctx: ArtemisContext):
|
|||||||
|
|
||||||
# END path: assemble the machine-readable run outcome.
|
# END path: assemble the machine-readable run outcome.
|
||||||
records = read_ledger(base_dir)
|
records = read_ledger(base_dir)
|
||||||
outcome = compute_run_outcome(snapshot, records, verify_blocked=verify_blocked)
|
retired = getattr(ctx, "guidance_retired_checks", None)
|
||||||
|
outcome = compute_run_outcome(
|
||||||
|
snapshot,
|
||||||
|
records,
|
||||||
|
verify_blocked=verify_blocked,
|
||||||
|
retired=retired if isinstance(retired, set) else None,
|
||||||
|
)
|
||||||
if snapshot.all_check_items or records or verify_blocked:
|
if snapshot.all_check_items or records or verify_blocked:
|
||||||
update["run_outcome"] = outcome.model_dump()
|
update["run_outcome"] = outcome.model_dump()
|
||||||
# BLOCKED/partial wrap-ups carry the last findings in the metadata file.
|
# BLOCKED/partial wrap-ups carry the last findings in the metadata file.
|
||||||
@@ -451,7 +462,8 @@ async def exit_settlement_node(state: State, ctx: ArtemisContext):
|
|||||||
f"Run outcome: task_status={outcome.task_status},"
|
f"Run outcome: task_status={outcome.task_status},"
|
||||||
f" tests(passed={outcome.tests.passed}, failed={outcome.tests.failed},"
|
f" tests(passed={outcome.tests.passed}, failed={outcome.tests.failed},"
|
||||||
f" inconclusive={outcome.tests.inconclusive},"
|
f" inconclusive={outcome.tests.inconclusive},"
|
||||||
f" unchecked={outcome.tests.unchecked})"
|
f" unchecked={outcome.tests.unchecked},"
|
||||||
|
f" retired={outcome.tests.retired})"
|
||||||
)
|
)
|
||||||
return update
|
return update
|
||||||
|
|
||||||
@@ -634,11 +646,53 @@ async def _process_plan_write(
|
|||||||
# deleted/rewritten check lines are merged back in a pure text operation.
|
# deleted/rewritten check lines are merged back in a pure text operation.
|
||||||
# This is content-driven and independent of any switch — a resumed plan
|
# This is content-driven and independent of any switch — a resumed plan
|
||||||
# with check lines stays protected even when checking is disabled.
|
# with check lines stays protected even when checking is disabled.
|
||||||
merged = restore_missing_check_items(content_before, content_after)
|
# The one exception is user guidance: it outranks the declared standards,
|
||||||
|
# so every check line that existed when an instruction arrived (recorded
|
||||||
|
# by perception in ``ctx.guidance_unprotected_checks``) may be dropped or
|
||||||
|
# reworded at any later write — the Operator usually re-plans over several
|
||||||
|
# surgical edits, turns after the guidance landed. Lines added after the
|
||||||
|
# instruction are protected again until the next one.
|
||||||
|
#
|
||||||
|
# ``missing_check_items`` diffs on the full signature (kind, when, text,
|
||||||
|
# parent hash), so rewording a milestone re-signs every check line under
|
||||||
|
# it. Such a line has merely moved: it is still declared, so it is neither
|
||||||
|
# merged back (that would duplicate it as a task-level ``@end`` orphan)
|
||||||
|
# nor retired (the ledger, the harvest gate and the run outcome all key a
|
||||||
|
# check item by ``(kind, text)``). Only a line whose ``(kind, text)`` is
|
||||||
|
# gone from the new plan counts as dropped.
|
||||||
|
before_snapshot = parse_plan(content_before)
|
||||||
|
after_snapshot = parse_plan(content_after)
|
||||||
|
missing = missing_check_items(before_snapshot, after_snapshot)
|
||||||
|
unprotected = getattr(ctx, "guidance_unprotected_checks", None)
|
||||||
|
if not isinstance(unprotected, set):
|
||||||
|
unprotected = set()
|
||||||
|
still_declared = Counter((ci.kind, ci.text) for ci in after_snapshot.check_items)
|
||||||
|
dropped: list = []
|
||||||
|
for ci in missing:
|
||||||
|
if still_declared.get((ci.kind, ci.text), 0) > 0:
|
||||||
|
still_declared[(ci.kind, ci.text)] -= 1
|
||||||
|
continue # moved under a reworded milestone / re-timed: still declared
|
||||||
|
dropped.append(ci)
|
||||||
|
waived = [ci for ci in dropped if (ci.kind, ci.text) in unprotected]
|
||||||
|
protected = [ci for ci in dropped if (ci.kind, ci.text) not in unprotected]
|
||||||
|
if waived:
|
||||||
|
logger.info(
|
||||||
|
"Plan write dropped declared check lines because of user guidance"
|
||||||
|
f" (the change stands): {[render_check_line(ci).strip() for ci in waived]}"
|
||||||
|
)
|
||||||
|
# Records the retirement (harvest gate + run outcome) and withdraws any
|
||||||
|
# standing finding headline those lines had already produced.
|
||||||
|
retire_check_items(ctx, [(ci.kind, ci.text) for ci in waived])
|
||||||
|
# A retired line the Operator declares again is live again: the harvest
|
||||||
|
# gate and the run outcome must agree with the plan on disk.
|
||||||
|
reinstate_check_items(ctx, [(ci.kind, ci.text) for ci in after_snapshot.check_items])
|
||||||
|
if protected:
|
||||||
|
merged = restore_missing_check_items(content_before, content_after, items=protected)
|
||||||
if merged is not None:
|
if merged is not None:
|
||||||
logger.info(
|
logger.info(
|
||||||
"Plan write removed or rewrote declared check lines; merging them"
|
"Plan write removed or rewrote declared check lines; merging them"
|
||||||
" back (check standards are protected deterministically)."
|
" back (check standards are protected deterministically):"
|
||||||
|
f" {[render_check_line(ci).strip() for ci in protected]}"
|
||||||
)
|
)
|
||||||
content_after = merged
|
content_after = merged
|
||||||
try:
|
try:
|
||||||
@@ -653,7 +707,7 @@ async def _process_plan_write(
|
|||||||
# never influences the parses below.
|
# never influences the parses below.
|
||||||
sync_finding_lines(ctx)
|
sync_finding_lines(ctx)
|
||||||
|
|
||||||
before = parse_plan(content_before)
|
before = before_snapshot
|
||||||
after = parse_plan(content_after)
|
after = parse_plan(content_after)
|
||||||
|
|
||||||
# Ratchet baseline: judge the current plan against the last *validated*
|
# Ratchet baseline: judge the current plan against the last *validated*
|
||||||
|
|||||||
@@ -92,6 +92,29 @@ def _check_injected_instruction_file(base_dir: str) -> dict | None:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _unprotect_current_check_lines(ctx: ArtemisContext) -> None:
|
||||||
|
"""Record the plan's current check lines as editable under user guidance."""
|
||||||
|
from artemis.utils.notes import get_note_file_path
|
||||||
|
from artemis.utils.plan_grammar import parse_plan
|
||||||
|
|
||||||
|
unprotected = getattr(ctx, "guidance_unprotected_checks", None)
|
||||||
|
if not isinstance(unprotected, set):
|
||||||
|
unprotected = set()
|
||||||
|
try:
|
||||||
|
plan_path = get_note_file_path(ctx.data_engine.base_dir, "task_plan")
|
||||||
|
text = plan_path.read_text(encoding="utf-8") if plan_path.exists() else ""
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Could not read task_plan for the guidance check-line waiver: {e}")
|
||||||
|
text = ""
|
||||||
|
added = {(ci.kind, ci.text) for ci in parse_plan(text).check_items}
|
||||||
|
unprotected |= added
|
||||||
|
ctx.guidance_unprotected_checks = unprotected
|
||||||
|
logger.info(
|
||||||
|
f"User guidance received: {len(added)} declared check line(s) may now be edited"
|
||||||
|
" by the Operator."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
async def perception_node(state: State, ctx: ArtemisContext) -> dict:
|
async def perception_node(state: State, ctx: ArtemisContext) -> dict:
|
||||||
"""Perception node (fast path): captures screenshot, extracts XML hierarchy, and performs OCR.
|
"""Perception node (fast path): captures screenshot, extracts XML hierarchy, and performs OCR.
|
||||||
|
|
||||||
@@ -114,6 +137,14 @@ async def perception_node(state: State, ctx: ArtemisContext) -> dict:
|
|||||||
if injected_payload:
|
if injected_payload:
|
||||||
injected_instruction = injected_payload.get("instruction")
|
injected_instruction = injected_payload.get("instruction")
|
||||||
user_stop_requested = bool(injected_payload.get("release_loop"))
|
user_stop_requested = bool(injected_payload.get("release_loop"))
|
||||||
|
if isinstance(injected_instruction, str) and injected_instruction.strip():
|
||||||
|
# User guidance outranks the plan's declared standards: every
|
||||||
|
# check line that exists right now loses its machine
|
||||||
|
# restoration for the rest of the run (kept on the run context,
|
||||||
|
# not the per-turn State), so the Operator can drop or reword
|
||||||
|
# it in whichever later write it gets to it. Lines added after
|
||||||
|
# this moment stay protected until the next instruction.
|
||||||
|
_unprotect_current_check_lines(ctx)
|
||||||
|
|
||||||
controller = UnifiedMobileController(ctx)
|
controller = UnifiedMobileController(ctx)
|
||||||
|
|
||||||
|
|||||||
@@ -116,7 +116,7 @@ NODE_VISIBILITY: dict[str, NodeVisibility] = {
|
|||||||
"operator_native_thinking",
|
"operator_native_thinking",
|
||||||
"user_stop_requested",
|
"user_stop_requested",
|
||||||
"initial_goal",
|
"initial_goal",
|
||||||
# M3: the turn's injected instruction is stamped verbatim onto the
|
# The turn's injected instruction is stamped verbatim onto the
|
||||||
# step record so the chunk ledger can preserve it (never-evict).
|
# step record so the chunk ledger can preserve it (never-evict).
|
||||||
"injected_instruction",
|
"injected_instruction",
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -12,307 +12,137 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
"""System, environment, and device diagnostics (artemis doctor)."""
|
"""System, environment, and device diagnostics (artemis doctor).
|
||||||
|
|
||||||
|
The command renders the same readiness probes the web console's device wizard
|
||||||
|
and the ``mobile_diagnose`` MCP tool run (:mod:`artemis.core.diagnostics`), so
|
||||||
|
the terminal, the browser and the IDE agree on one verdict. Only two rows are
|
||||||
|
CLI-specific and not engine probes: Node.js/npm and the Showcase UI build.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
from dataclasses import dataclass
|
||||||
|
import json
|
||||||
import platform as sys_platform
|
import platform as sys_platform
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
from typing import Any
|
||||||
import sys
|
|
||||||
|
|
||||||
from rich.console import Console
|
from rich.console import Console
|
||||||
|
from rich.markup import escape
|
||||||
from rich.panel import Panel
|
from rich.panel import Panel
|
||||||
from rich.table import Table
|
from rich.table import Table
|
||||||
import typer
|
import typer
|
||||||
|
|
||||||
from artemis.config import (
|
|
||||||
parse_llm_config,
|
|
||||||
settings,
|
|
||||||
)
|
|
||||||
from artemis.config.paths import ROOT_DIR
|
from artemis.config.paths import ROOT_DIR
|
||||||
from artemis.core.diagnostics.adb_keys import heal_adb_keys, inspect_adb_keys
|
from artemis.core.diagnostics.engine import readiness_engine
|
||||||
|
from artemis.core.diagnostics.readiness import (
|
||||||
|
Verdict,
|
||||||
|
adb_keys_corrupted,
|
||||||
|
base_verdict,
|
||||||
|
collect_readiness,
|
||||||
|
sort_by_fix_order,
|
||||||
|
)
|
||||||
|
from artemis.core.diagnostics.schema import ProbeResult, ProbeStatus
|
||||||
from artemis.platform import OSType, platform
|
from artemis.platform import OSType, platform
|
||||||
from artemis.toolchain import toolchain
|
from artemis.runtime.device_lock import DeviceExecutionLock
|
||||||
from artemis.utils.ocr_api import is_ocr_configured
|
|
||||||
|
|
||||||
|
|
||||||
def doctor_command(
|
@dataclass(frozen=True)
|
||||||
fix: bool = typer.Option(
|
class ExtraRow:
|
||||||
False,
|
"""A CLI-only diagnostic row that is not backed by an engine probe."""
|
||||||
"--fix",
|
|
||||||
"-f",
|
key: str
|
||||||
help="Automatically repair fixable environment issues (such as corrupted ADB authentication keys).",
|
title: str
|
||||||
|
status: str # "pass" | "missing"
|
||||||
|
status_markup: str
|
||||||
|
summary: str
|
||||||
|
detail: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class FixOutcome:
|
||||||
|
"""Result of one ``--fix`` repair step."""
|
||||||
|
|
||||||
|
fix: str
|
||||||
|
success: bool
|
||||||
|
message: str
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Collection
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
async def _collect() -> list[ProbeResult]:
|
||||||
|
report, host = await collect_readiness()
|
||||||
|
return sort_by_fix_order([*report.probes, host])
|
||||||
|
|
||||||
|
|
||||||
|
async def _apply_fixes(results: list[ProbeResult]) -> list[FixOutcome]:
|
||||||
|
"""Repair corrupted ADB keys and sweep stale device locks left by dead runners."""
|
||||||
|
outcomes: list[FixOutcome] = []
|
||||||
|
|
||||||
|
if adb_keys_corrupted(results):
|
||||||
|
result = await readiness_engine.heal_adb_keys()
|
||||||
|
outcomes.append(
|
||||||
|
FixOutcome(
|
||||||
|
fix="heal_adb_keys",
|
||||||
|
success=bool(result.get("success")),
|
||||||
|
message=str(result.get("message") or "")
|
||||||
|
or ("Corrupted RSA keys regenerated." if result.get("success") else "Heal failed."),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
outcomes.append(
|
||||||
|
FixOutcome(
|
||||||
|
fix="heal_adb_keys",
|
||||||
|
success=True,
|
||||||
|
message="ADB authentication keys are healthy; nothing to repair.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
removed = await asyncio.to_thread(DeviceExecutionLock.cleanup_stale_locks)
|
||||||
|
outcomes.append(
|
||||||
|
FixOutcome(
|
||||||
|
fix="cleanup_stale_locks",
|
||||||
|
success=True,
|
||||||
|
message=(
|
||||||
|
f"Removed {removed} stale device lock(s)/queue ticket(s)."
|
||||||
|
if removed
|
||||||
|
else "No stale device locks or queue tickets found."
|
||||||
),
|
),
|
||||||
) -> None:
|
|
||||||
"""Run diagnostics to inspect system dependencies, device connectivity, and configuration."""
|
|
||||||
console = Console()
|
|
||||||
console.print()
|
|
||||||
os_name = sys_platform.system()
|
|
||||||
arch_name = sys_platform.machine()
|
|
||||||
console.print(
|
|
||||||
Panel(
|
|
||||||
f"[bold cyan]☕ Artemis System & Environment Doctor[/bold cyan]\n"
|
|
||||||
f"[dim]Platform: {os_name} ({arch_name}) | PAL: {platform.os_type.value}[/dim]",
|
|
||||||
expand=False,
|
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
return outcomes
|
||||||
|
|
||||||
table = Table(show_header=True, header_style="bold magenta")
|
|
||||||
table.add_column("Component", style="dim", width=22)
|
|
||||||
table.add_column("Status", width=12)
|
|
||||||
table.add_column("Details & Recommendations")
|
|
||||||
|
|
||||||
all_passed = True
|
async def _diagnose(fix: bool) -> tuple[list[ProbeResult], list[FixOutcome]]:
|
||||||
|
results = await _collect()
|
||||||
# 1. Python Version
|
fixes: list[FixOutcome] = []
|
||||||
py_ver = f"{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}"
|
|
||||||
if sys.version_info >= (3, 12):
|
|
||||||
table.add_row(
|
|
||||||
"Python Runtime",
|
|
||||||
"[bold green]✔ OK[/bold green]",
|
|
||||||
f"Python {py_ver} ({arch_name}, 64-bit)",
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
all_passed = False
|
|
||||||
table.add_row(
|
|
||||||
"Python Runtime",
|
|
||||||
"[bold red]✖ Warning[/bold red]",
|
|
||||||
f"Python {py_ver} detected. Python >= 3.12 is strongly recommended.",
|
|
||||||
)
|
|
||||||
|
|
||||||
# 2. ADB Installation & Server
|
|
||||||
adb_path = toolchain.resolve("adb")
|
|
||||||
if adb_path:
|
|
||||||
try:
|
|
||||||
res = subprocess.run(
|
|
||||||
[adb_path, "version"],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
timeout=5,
|
|
||||||
encoding="utf-8",
|
|
||||||
errors="replace",
|
|
||||||
)
|
|
||||||
adb_info = res.stdout.splitlines()[0] if res.stdout else "Installed"
|
|
||||||
table.add_row(
|
|
||||||
"Android ADB", "[bold green]✔ OK[/bold green]", f"{adb_path} ({adb_info})"
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
table.add_row("Android ADB", "[bold green]✔ OK[/bold green]", f"{adb_path}")
|
|
||||||
else:
|
|
||||||
all_passed = False
|
|
||||||
hint = (
|
|
||||||
"Run: winget install Google.PlatformTools"
|
|
||||||
if platform.os_type == OSType.WINDOWS
|
|
||||||
else "Run: brew install android-platform-tools"
|
|
||||||
if platform.os_type == OSType.MACOS
|
|
||||||
else "Run: sudo apt-get install -y adb"
|
|
||||||
)
|
|
||||||
table.add_row(
|
|
||||||
"Android ADB",
|
|
||||||
"[bold red]✖ Missing[/bold red]",
|
|
||||||
f"adb not found in PATH or Android SDK. {hint}",
|
|
||||||
)
|
|
||||||
|
|
||||||
# 2b. ADB Authentication RSA Keys Check
|
|
||||||
key_status = inspect_adb_keys()
|
|
||||||
if key_status.is_corrupted:
|
|
||||||
if fix:
|
if fix:
|
||||||
heal_result = heal_adb_keys(adb_path=adb_path)
|
fixes = await _apply_fixes(results)
|
||||||
if heal_result.get("success"):
|
results = await _collect()
|
||||||
table.add_row(
|
return results, fixes
|
||||||
"ADB Auth Keys",
|
|
||||||
"[bold green]✔ Repaired[/bold green]",
|
|
||||||
"Corrupted RSA keys detected and automatically regenerated.",
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
all_passed = False
|
|
||||||
table.add_row(
|
|
||||||
"ADB Auth Keys",
|
|
||||||
"[bold red]✖ Error[/bold red]",
|
|
||||||
f"Failed to auto-heal keys: {heal_result.get('message')}",
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
all_passed = False
|
|
||||||
table.add_row(
|
|
||||||
"ADB Auth Keys",
|
|
||||||
"[bold red]✖ Corrupted[/bold red]",
|
|
||||||
f"{key_status.error_reason} Run [bold cyan]artemis doctor --fix[/bold cyan] to auto-repair.",
|
|
||||||
)
|
|
||||||
elif key_status.exists:
|
|
||||||
table.add_row(
|
|
||||||
"ADB Auth Keys",
|
|
||||||
"[bold green]✔ OK[/bold green]",
|
|
||||||
f"{key_status.key_path} ({key_status.key_size_bytes} bytes RSA)",
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
table.add_row(
|
|
||||||
"ADB Auth Keys",
|
|
||||||
"[bold cyan]○ Ready[/bold cyan]",
|
|
||||||
"Keys will be generated automatically on first device connection.",
|
|
||||||
)
|
|
||||||
|
|
||||||
# 3. Connected Android Devices
|
|
||||||
devices_info: list[tuple[str, str, str]] = []
|
|
||||||
if adb_path:
|
|
||||||
try:
|
|
||||||
res = subprocess.run(
|
|
||||||
[adb_path, "devices", "-l"],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
timeout=5,
|
|
||||||
encoding="utf-8",
|
|
||||||
errors="replace",
|
|
||||||
)
|
|
||||||
lines = [line_str.strip() for line_str in res.stdout.splitlines() if line_str.strip()]
|
|
||||||
for line in lines[1:]: # skip 'List of devices attached'
|
|
||||||
parts = line.split()
|
|
||||||
if len(parts) >= 2:
|
|
||||||
serial = parts[0]
|
|
||||||
state = parts[1]
|
|
||||||
extra = " ".join(parts[2:])
|
|
||||||
devices_info.append((serial, state, extra))
|
|
||||||
except (OSError, subprocess.SubprocessError):
|
|
||||||
# adb probe failed or timed out; report as "no devices detected".
|
|
||||||
pass
|
|
||||||
|
|
||||||
if devices_info:
|
# ---------------------------------------------------------------------------
|
||||||
active_devices = [d for d in devices_info if d[1] == "device"]
|
# CLI-only extra rows (not engine probes)
|
||||||
if active_devices:
|
# ---------------------------------------------------------------------------
|
||||||
summary = "\n".join(
|
|
||||||
[
|
|
||||||
f"• [bold cyan]{d[0]}[/bold cyan] ({d[2] or 'Authorized'})"
|
|
||||||
for d in active_devices
|
|
||||||
]
|
|
||||||
)
|
|
||||||
table.add_row("Android Devices", "[bold green]✔ Connected[/bold green]", summary)
|
|
||||||
else:
|
|
||||||
all_passed = False
|
|
||||||
table.add_row(
|
|
||||||
"Android Devices",
|
|
||||||
"[bold yellow]⚠ Unauthorized[/bold yellow]",
|
|
||||||
"Device detected but unauthorized. Check phone screen for USB Debugging prompt.",
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
table.add_row(
|
|
||||||
"Android Devices",
|
|
||||||
"[bold yellow]⚠ None Found[/bold yellow]",
|
|
||||||
"No Android device or emulator detected. Connect phone via USB or start emulator.",
|
|
||||||
)
|
|
||||||
|
|
||||||
# 4. LLM & Vision OCR Key Configuration
|
|
||||||
gemini_key = settings.get_api_key("google")
|
|
||||||
ocr_key = settings.get_api_key("ocr")
|
|
||||||
openai_key = settings.get_api_key("openai")
|
|
||||||
claude_key = settings.get_api_key("anthropic")
|
|
||||||
openrouter_key = settings.get_api_key("openrouter")
|
|
||||||
|
|
||||||
configured_keys = []
|
def _npm_row() -> ExtraRow:
|
||||||
if gemini_key:
|
|
||||||
val = gemini_key.get_secret_value()
|
|
||||||
masked = f"{val[:6]}...{val[-4:]}" if len(val) > 10 else "***"
|
|
||||||
configured_keys.append(f"Gemini LLM ({masked})")
|
|
||||||
if openai_key:
|
|
||||||
configured_keys.append("OpenAI")
|
|
||||||
if claude_key:
|
|
||||||
configured_keys.append("Anthropic")
|
|
||||||
if openrouter_key:
|
|
||||||
configured_keys.append("OpenRouter")
|
|
||||||
|
|
||||||
if configured_keys:
|
|
||||||
table.add_row(
|
|
||||||
"LLM Providers", "[bold green]✔ Configured[/bold green]", ", ".join(configured_keys)
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
all_passed = False
|
|
||||||
table.add_row(
|
|
||||||
"LLM Providers",
|
|
||||||
"[bold red]✖ Missing[/bold red]",
|
|
||||||
"No LLM key found in .env. Run [bold cyan]artemis init[/bold cyan] to configure.",
|
|
||||||
)
|
|
||||||
|
|
||||||
# 5. Vision OCR Key Check (Optional)
|
|
||||||
if is_ocr_configured() and ocr_key:
|
|
||||||
val = ocr_key.get_secret_value()
|
|
||||||
masked = f"{val[:6]}...{val[-4:]}" if len(val) > 10 else "***"
|
|
||||||
table.add_row(
|
|
||||||
"Vision OCR",
|
|
||||||
"[bold green]✔ Configured[/bold green]",
|
|
||||||
f"Google Cloud Vision API ({masked})",
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
table.add_row(
|
|
||||||
"Vision OCR",
|
|
||||||
"[bold cyan]○ Optional[/bold cyan]",
|
|
||||||
"OCR_API_KEY is not set. Perception operates in pure XML layout mode.",
|
|
||||||
)
|
|
||||||
|
|
||||||
# 6. Configuration File Health
|
|
||||||
try:
|
|
||||||
cfg = parse_llm_config()
|
|
||||||
planner_model = cfg.planner.model
|
|
||||||
table.add_row(
|
|
||||||
"Configuration",
|
|
||||||
"[bold green]✔ Valid[/bold green]",
|
|
||||||
f"config/artemis.jsonc parsed (Planner: {planner_model})",
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
all_passed = False
|
|
||||||
table.add_row(
|
|
||||||
"Configuration",
|
|
||||||
"[bold red]✖ Error[/bold red]",
|
|
||||||
f"Failed to parse config/artemis.jsonc: {e}",
|
|
||||||
)
|
|
||||||
|
|
||||||
# 7. Video Recording Tools (FFmpeg & scrcpy)
|
|
||||||
ffmpeg_path = toolchain.resolve("ffmpeg")
|
|
||||||
scrcpy_path = toolchain.resolve("scrcpy")
|
|
||||||
|
|
||||||
if ffmpeg_path:
|
|
||||||
table.add_row(
|
|
||||||
"FFmpeg Tools", "[bold green]✔ Installed[/bold green]", f"{ffmpeg_path} (Video ready)"
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
hint = (
|
|
||||||
"winget install Gyan.FFmpeg"
|
|
||||||
if platform.os_type == OSType.WINDOWS
|
|
||||||
else "brew install ffmpeg"
|
|
||||||
if platform.os_type == OSType.MACOS
|
|
||||||
else "sudo apt-get install -y ffmpeg"
|
|
||||||
)
|
|
||||||
table.add_row(
|
|
||||||
"FFmpeg Tools",
|
|
||||||
"[dim]⚪ Optional[/dim]",
|
|
||||||
f"Not found. ({hint})",
|
|
||||||
)
|
|
||||||
|
|
||||||
if scrcpy_path:
|
|
||||||
table.add_row(
|
|
||||||
"Screen Recorder",
|
|
||||||
"[bold green]✔ Installed[/bold green]",
|
|
||||||
f"{scrcpy_path} (Live stream ready)",
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
hint = (
|
|
||||||
"winget install Genymobile.scrcpy"
|
|
||||||
if platform.os_type == OSType.WINDOWS
|
|
||||||
else "brew install scrcpy"
|
|
||||||
if platform.os_type == OSType.MACOS
|
|
||||||
else "sudo apt-get install -y scrcpy"
|
|
||||||
)
|
|
||||||
table.add_row(
|
|
||||||
"Screen Recorder",
|
|
||||||
"[dim]⚪ Optional[/dim]",
|
|
||||||
f"scrcpy not found. ({hint})",
|
|
||||||
)
|
|
||||||
|
|
||||||
# 8. Node.js & npm (for UI compilation)
|
|
||||||
npm_path = shutil.which("npm")
|
npm_path = shutil.which("npm")
|
||||||
if npm_path:
|
if npm_path:
|
||||||
table.add_row(
|
return ExtraRow(
|
||||||
"Node.js / npm",
|
key="nodejs_npm",
|
||||||
"[bold green]✔ Installed[/bold green]",
|
title="Node.js / npm",
|
||||||
f"{npm_path} (Ready to build Showcase UI)",
|
status="pass",
|
||||||
|
status_markup="[bold green]✔ Installed[/bold green]",
|
||||||
|
summary="Installed",
|
||||||
|
detail=f"{npm_path} (Ready to build Showcase UI)",
|
||||||
)
|
)
|
||||||
else:
|
|
||||||
hint = (
|
hint = (
|
||||||
"winget install OpenJS.NodeJS.LTS"
|
"winget install OpenJS.NodeJS.LTS"
|
||||||
if platform.os_type == OSType.WINDOWS
|
if platform.os_type == OSType.WINDOWS
|
||||||
@@ -320,13 +150,17 @@ def doctor_command(
|
|||||||
if platform.os_type == OSType.MACOS
|
if platform.os_type == OSType.MACOS
|
||||||
else "sudo apt-get install -y nodejs npm"
|
else "sudo apt-get install -y nodejs npm"
|
||||||
)
|
)
|
||||||
table.add_row(
|
return ExtraRow(
|
||||||
"Node.js / npm",
|
key="nodejs_npm",
|
||||||
"[dim]⚪ Optional[/dim]",
|
title="Node.js / npm",
|
||||||
f"Not found in PATH. ({hint})",
|
status="missing",
|
||||||
|
status_markup="[dim]⚪ Optional[/dim]",
|
||||||
|
summary="Not Found",
|
||||||
|
detail=f"Not found in PATH. ({hint})",
|
||||||
)
|
)
|
||||||
|
|
||||||
# 9. Showcase UI Build Status
|
|
||||||
|
def _showcase_row() -> ExtraRow:
|
||||||
from artemis.resources import get_bundled_showcase_dist
|
from artemis.resources import get_bundled_showcase_dist
|
||||||
|
|
||||||
base_dist = ROOT_DIR / "apps" / "showcase_ui" / "dist"
|
base_dist = ROOT_DIR / "apps" / "showcase_ui" / "dist"
|
||||||
@@ -342,22 +176,98 @@ def doctor_command(
|
|||||||
if bundled_showcase is not None:
|
if bundled_showcase is not None:
|
||||||
found_showcase = bundled_showcase / "index.html"
|
found_showcase = bundled_showcase / "index.html"
|
||||||
if found_showcase:
|
if found_showcase:
|
||||||
table.add_row(
|
return ExtraRow(
|
||||||
"Showcase UI",
|
key="showcase_ui",
|
||||||
"[bold green]✔ Compiled[/bold green]",
|
title="Showcase UI",
|
||||||
f"{found_showcase.parent} (Angular ready)",
|
status="pass",
|
||||||
|
status_markup="[bold green]✔ Compiled[/bold green]",
|
||||||
|
summary="Compiled",
|
||||||
|
detail=f"{found_showcase.parent} (Angular ready)",
|
||||||
)
|
)
|
||||||
else:
|
return ExtraRow(
|
||||||
table.add_row(
|
key="showcase_ui",
|
||||||
"Showcase UI",
|
title="Showcase UI",
|
||||||
"[bold yellow]○ Not Compiled[/bold yellow]",
|
status="missing",
|
||||||
"Run ./start.sh or artemis ui to auto-compile.",
|
status_markup="[bold yellow]○ Not Compiled[/bold yellow]",
|
||||||
|
summary="Not Compiled",
|
||||||
|
detail="Run ./start.sh or artemis ui to auto-compile.",
|
||||||
)
|
)
|
||||||
|
|
||||||
console.print(table)
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Rendering
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _status_markup(result: ProbeResult) -> str:
|
||||||
|
status = result.status
|
||||||
|
if status is ProbeStatus.PASS:
|
||||||
|
return "[bold green]✔ OK[/bold green]"
|
||||||
|
if status is ProbeStatus.SKIPPED:
|
||||||
|
return "[dim]○ Skipped[/dim]"
|
||||||
|
summary = escape(result.summary)
|
||||||
|
if status is ProbeStatus.WARN:
|
||||||
|
# A warning keeps its summary whether or not it blocks (the host
|
||||||
|
# probe's warnings degrade the verdict without blocking it).
|
||||||
|
return f"[bold yellow]⚠ {summary}[/bold yellow]"
|
||||||
|
if not result.is_blocker:
|
||||||
|
return "[dim]⚪ Optional[/dim]"
|
||||||
|
return f"[bold red]✖ {summary}[/bold red]"
|
||||||
|
|
||||||
|
|
||||||
|
def _action_lines(result: ProbeResult) -> list[str]:
|
||||||
|
"""One plain-text line per remediation; ``&&`` chains split for PowerShell 5.1."""
|
||||||
|
lines: list[str] = []
|
||||||
|
for action in result.actions:
|
||||||
|
payload = action.payload.strip()
|
||||||
|
if not payload:
|
||||||
|
continue
|
||||||
|
if action.action_type == "command":
|
||||||
|
for part in payload.split("&&"):
|
||||||
|
if part.strip():
|
||||||
|
lines.append(f"Run: {part.strip()}")
|
||||||
|
else:
|
||||||
|
lines.append(payload)
|
||||||
|
return lines
|
||||||
|
|
||||||
|
|
||||||
|
def _details_markup(result: ProbeResult) -> str:
|
||||||
|
# Only summary/description/actions are ever rendered. Probe metadata can
|
||||||
|
# carry raw credentials (the console prefills its settings form from it)
|
||||||
|
# and must never reach a terminal or a log.
|
||||||
|
parts = [escape(result.description)]
|
||||||
|
if result.status is not ProbeStatus.PASS:
|
||||||
|
for line in _action_lines(result):
|
||||||
|
if line.startswith("Run: "):
|
||||||
|
parts.append(f"[bold cyan]Run:[/bold cyan] {escape(line[len('Run: ') :])}")
|
||||||
|
else:
|
||||||
|
parts.append(escape(line))
|
||||||
|
return "\n".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def _render_table(results: list[ProbeResult], extras: list[ExtraRow]) -> Table:
|
||||||
|
table = Table(show_header=True, header_style="bold magenta")
|
||||||
|
table.add_column("Component", style="dim", min_width=22, max_width=30)
|
||||||
|
table.add_column("Status", min_width=12, max_width=28)
|
||||||
|
table.add_column("Details & Recommendations", overflow="fold")
|
||||||
|
for result in results:
|
||||||
|
table.add_row(escape(result.title), _status_markup(result), _details_markup(result))
|
||||||
|
for extra in extras:
|
||||||
|
table.add_row(extra.title, extra.status_markup, escape(extra.detail))
|
||||||
|
return table
|
||||||
|
|
||||||
|
|
||||||
|
def _render_fixes(console: Console, fixes: list[FixOutcome]) -> None:
|
||||||
|
lines = []
|
||||||
|
for outcome in fixes:
|
||||||
|
mark = "[bold green]✔[/bold green]" if outcome.success else "[bold red]✖[/bold red]"
|
||||||
|
lines.append(f"{mark} {escape(outcome.fix)}: {escape(outcome.message)}")
|
||||||
|
console.print(Panel("\n".join(lines), title="Repairs (--fix)", expand=False))
|
||||||
console.print()
|
console.print()
|
||||||
|
|
||||||
if all_passed:
|
|
||||||
|
def _render_footer(console: Console, verdict: Verdict, results: list[ProbeResult]) -> None:
|
||||||
|
if verdict == "ready":
|
||||||
console.print(
|
console.print(
|
||||||
Panel(
|
Panel(
|
||||||
"[bold green]🎉 All system checks passed![/bold green]\n\n"
|
"[bold green]🎉 All system checks passed![/bold green]\n\n"
|
||||||
@@ -367,12 +277,128 @@ def doctor_command(
|
|||||||
expand=False,
|
expand=False,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
else:
|
return
|
||||||
|
|
||||||
|
if verdict == "degraded":
|
||||||
console.print(
|
console.print(
|
||||||
Panel(
|
Panel(
|
||||||
"[bold yellow]💡 Tip:[/bold yellow] Run [bold cyan]artemis init[/bold cyan] to quickly set up your API key and device.",
|
"[bold green]Required checks passed.[/bold green] Optional components are "
|
||||||
title="Action Required",
|
"missing; tasks can run, but some features (video replay, OCR) are off.\n\n"
|
||||||
|
"Run your first automation task now:\n"
|
||||||
|
' [bold cyan]artemis run "Open Settings and check Battery level"[/bold cyan]',
|
||||||
|
title="Status: Ready (optional gaps)",
|
||||||
expand=False,
|
expand=False,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
failing = [r.title for r in results if r.is_blocker and r.status is not ProbeStatus.PASS]
|
||||||
|
tips = [
|
||||||
|
"[bold yellow]💡 Blocking issues:[/bold yellow] " + escape(", ".join(failing)),
|
||||||
|
"",
|
||||||
|
"• API keys & device setup: run [bold cyan]artemis init[/bold cyan].",
|
||||||
|
"• Follow the [bold cyan]Run:[/bold cyan] lines above, then re-run "
|
||||||
|
"[bold cyan]artemis doctor[/bold cyan].",
|
||||||
|
"• Using an IDE (Claude Code, Cursor, ...)? Ask it to call "
|
||||||
|
"[bold cyan]mobile_diagnose[/bold cyan] - it runs these same checks and can "
|
||||||
|
"apply safe fixes from the assistant.",
|
||||||
|
]
|
||||||
|
if adb_keys_corrupted(results):
|
||||||
|
tips.append(
|
||||||
|
"• Corrupted ADB keys detected: run [bold cyan]artemis doctor --fix[/bold cyan]."
|
||||||
|
)
|
||||||
|
console.print(Panel("\n".join(tips), title="Action Required", expand=False))
|
||||||
|
|
||||||
|
|
||||||
|
def _json_document(
|
||||||
|
results: list[ProbeResult],
|
||||||
|
extras: list[ExtraRow],
|
||||||
|
fixes: list[FixOutcome],
|
||||||
|
verdict: Verdict,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
# Metadata is intentionally omitted: it may hold raw API keys.
|
||||||
|
checks = [
|
||||||
|
{
|
||||||
|
"id": r.id,
|
||||||
|
"title": r.title,
|
||||||
|
"status": r.status.value,
|
||||||
|
"required": r.is_blocker,
|
||||||
|
"summary": r.summary,
|
||||||
|
"detail": r.description,
|
||||||
|
"fix": []
|
||||||
|
if r.status is ProbeStatus.PASS
|
||||||
|
else [
|
||||||
|
{"type": a.action_type, "label": a.label, "payload": a.payload} for a in r.actions
|
||||||
|
],
|
||||||
|
}
|
||||||
|
for r in results
|
||||||
|
]
|
||||||
|
return {
|
||||||
|
"verdict": verdict,
|
||||||
|
"checks": checks,
|
||||||
|
"extras": {
|
||||||
|
e.key: {
|
||||||
|
"title": e.title,
|
||||||
|
"status": e.status,
|
||||||
|
"summary": e.summary,
|
||||||
|
"detail": e.detail,
|
||||||
|
}
|
||||||
|
for e in extras
|
||||||
|
},
|
||||||
|
"fixes": [{"fix": f.fix, "success": f.success, "message": f.message} for f in fixes],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Command
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def doctor_command(
|
||||||
|
fix: bool = typer.Option(
|
||||||
|
False,
|
||||||
|
"--fix",
|
||||||
|
"-f",
|
||||||
|
help=(
|
||||||
|
"Automatically repair fixable environment issues (corrupted ADB authentication "
|
||||||
|
"keys, stale device locks left by crashed runners)."
|
||||||
|
),
|
||||||
|
),
|
||||||
|
json_output: bool = typer.Option(
|
||||||
|
False,
|
||||||
|
"--json",
|
||||||
|
help="Print the diagnosis as a JSON document instead of a table.",
|
||||||
|
),
|
||||||
|
) -> None:
|
||||||
|
"""Run diagnostics to inspect system dependencies, device connectivity, and configuration."""
|
||||||
|
results, fixes = asyncio.run(_diagnose(fix))
|
||||||
|
extras = [_npm_row(), _showcase_row()]
|
||||||
|
verdict = base_verdict(results)
|
||||||
|
|
||||||
|
if json_output:
|
||||||
|
typer.echo(
|
||||||
|
json.dumps(
|
||||||
|
_json_document(results, extras, fixes, verdict), indent=2, ensure_ascii=False
|
||||||
|
)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
console = Console()
|
||||||
console.print()
|
console.print()
|
||||||
|
os_name = sys_platform.system()
|
||||||
|
arch_name = sys_platform.machine()
|
||||||
|
console.print(
|
||||||
|
Panel(
|
||||||
|
f"[bold cyan]☕ Artemis System & Environment Doctor[/bold cyan]\n"
|
||||||
|
f"[dim]Platform: {os_name} ({arch_name}) | PAL: {platform.os_type.value}[/dim]",
|
||||||
|
expand=False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if fixes:
|
||||||
|
_render_fixes(console, fixes)
|
||||||
|
console.print(_render_table(results, extras))
|
||||||
|
console.print()
|
||||||
|
_render_footer(console, verdict, results)
|
||||||
|
console.print()
|
||||||
|
|
||||||
|
if verdict == "blocked":
|
||||||
|
raise typer.Exit(code=1)
|
||||||
@@ -41,6 +41,7 @@ ARTEMIS_MCP_TOOLS = (
|
|||||||
"mobile_manage_task",
|
"mobile_manage_task",
|
||||||
"mobile_get_device_state",
|
"mobile_get_device_state",
|
||||||
"mobile_inspect_trace",
|
"mobile_inspect_trace",
|
||||||
|
"mobile_diagnose",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+182
-35
@@ -15,8 +15,16 @@
|
|||||||
"""MCP-backed action executor for the FlashRunner.
|
"""MCP-backed action executor for the FlashRunner.
|
||||||
|
|
||||||
Device actions use the in-process MCP session. Argument normalization,
|
Device actions use the in-process MCP session. Argument normalization,
|
||||||
coordinate descriptions, smart swipes, post-action observations, state updates,
|
element-index resolution, coordinate descriptions, smart swipes, post-action
|
||||||
tracing, and helper tools are handled here on the agent side.
|
observations, state updates, tracing, and helper tools are handled here on the
|
||||||
|
agent side.
|
||||||
|
|
||||||
|
Flash binds the same agent dialect as the Pro Operator: a ``click``,
|
||||||
|
``long_press`` or ``input_text`` target is either an element index into the
|
||||||
|
indexed ``--- Visible UI Elements ---`` list (resolved here to the element's
|
||||||
|
center, its observed text/bounds/id recorded) or a normalized ``[x, y]`` pair
|
||||||
|
carrying the model's own ``target_description``. The wire only ever sees
|
||||||
|
coordinates.
|
||||||
|
|
||||||
Device action status comes from ``ActionResult.ok``; helper tools report failures
|
Device action status comes from ``ActionResult.ok``; helper tools report failures
|
||||||
through ``ToolFailure`` or an explicit status.
|
through ``ToolFailure`` or an explicit status.
|
||||||
@@ -66,6 +74,9 @@ AGENT_TOOL_NAMES: frozenset[str] = (
|
|||||||
frozenset({"read_note", "list_notes", "ask_explorer", "video_analyzer"}) | HISTORY_TOOL_NAMES
|
frozenset({"read_note", "list_notes", "ask_explorer", "video_analyzer"}) | HISTORY_TOOL_NAMES
|
||||||
)
|
)
|
||||||
|
|
||||||
|
#: Actions whose ``target`` is a single point: an element index or an [x, y] pair.
|
||||||
|
_POINT_TARGET_ACTIONS: frozenset[str] = frozenset({"click", "long_press", "input_text"})
|
||||||
|
|
||||||
|
|
||||||
class _ArgError(ValueError):
|
class _ArgError(ValueError):
|
||||||
"""Argument-translation failure whose message is already fully formatted."""
|
"""Argument-translation failure whose message is already fully formatted."""
|
||||||
@@ -108,15 +119,28 @@ class McpActionExecutor:
|
|||||||
args: dict[str, Any],
|
args: dict[str, Any],
|
||||||
tool_call_id: str,
|
tool_call_id: str,
|
||||||
state: Any,
|
state: Any,
|
||||||
|
*,
|
||||||
|
index_elements: list[dict[str, Any]] | None = None,
|
||||||
) -> ToolExecutionResult:
|
) -> ToolExecutionResult:
|
||||||
"""Executes the named tool and wraps the outcome for the calling agent."""
|
"""Executes the named tool and wraps the outcome for the calling agent.
|
||||||
|
|
||||||
|
``index_elements`` is the element list an index target (``click(3)``)
|
||||||
|
resolves against; ``None`` means ``state.indexed_elements`` at call
|
||||||
|
time. A multi-action turn passes the list from the observation the
|
||||||
|
model decided on, so its later indices are not resolved against the
|
||||||
|
list refreshed by its earlier actions (``state.indexed_elements`` is
|
||||||
|
still refreshed after every action: it is what the next observation
|
||||||
|
shows).
|
||||||
|
"""
|
||||||
raw_name = name.split(":")[-1] if ":" in name else name
|
raw_name = name.split(":")[-1] if ":" in name else name
|
||||||
|
|
||||||
if raw_name in AGENT_TOOL_NAMES:
|
if raw_name in AGENT_TOOL_NAMES:
|
||||||
return await self._execute_agent_tool(raw_name, name, args, tool_call_id, state)
|
return await self._execute_agent_tool(raw_name, name, args, tool_call_id, state)
|
||||||
|
|
||||||
if raw_name in self.action_tool_names:
|
if raw_name in self.action_tool_names:
|
||||||
return await self._execute_device_action(raw_name, name, args, tool_call_id, state)
|
return await self._execute_device_action(
|
||||||
|
raw_name, name, args, tool_call_id, state, index_elements=index_elements
|
||||||
|
)
|
||||||
|
|
||||||
return ToolExecutionResult(
|
return ToolExecutionResult(
|
||||||
tool_call_id=tool_call_id,
|
tool_call_id=tool_call_id,
|
||||||
@@ -134,6 +158,8 @@ class McpActionExecutor:
|
|||||||
args: dict[str, Any],
|
args: dict[str, Any],
|
||||||
tool_call_id: str,
|
tool_call_id: str,
|
||||||
state: Any,
|
state: Any,
|
||||||
|
*,
|
||||||
|
index_elements: list[dict[str, Any]] | None = None,
|
||||||
) -> ToolExecutionResult:
|
) -> ToolExecutionResult:
|
||||||
span = TraceSpan(name=raw_name, trace_type="action", ctx=self.ctx)
|
span = TraceSpan(name=raw_name, trace_type="action", ctx=self.ctx)
|
||||||
span.payload = {"args": args}
|
span.payload = {"args": args}
|
||||||
@@ -144,12 +170,18 @@ class McpActionExecutor:
|
|||||||
shot_path: str | None = None
|
shot_path: str | None = None
|
||||||
xml_list: str | None = None
|
xml_list: str | None = None
|
||||||
target_semantics: dict[str, Any] = {}
|
target_semantics: dict[str, Any] = {}
|
||||||
|
target_coordinates: list[int] | None = None
|
||||||
with span:
|
with span:
|
||||||
try:
|
try:
|
||||||
session = await self._session_or_start()
|
session = await self._session_or_start()
|
||||||
wire_name, wire_args, finalize, target_semantics = self._translate(
|
wire_name, wire_args, finalize, target_semantics = self._translate(
|
||||||
raw_name, args, state
|
raw_name, args, state, index_elements=index_elements
|
||||||
)
|
)
|
||||||
|
if raw_name in _POINT_TARGET_ACTIONS:
|
||||||
|
# The normalized point the action was sent to: the recorded
|
||||||
|
# coordinates of an index target (the model named an index,
|
||||||
|
# not a point).
|
||||||
|
target_coordinates = wire_args.get("target")
|
||||||
extension = raw_name not in (REQUIRED_ACTIONS | OPTIONAL_ACTIONS)
|
extension = raw_name not in (REQUIRED_ACTIONS | OPTIONAL_ACTIONS)
|
||||||
|
|
||||||
if extension:
|
if extension:
|
||||||
@@ -214,6 +246,7 @@ class McpActionExecutor:
|
|||||||
"code": res.code.value,
|
"code": res.code.value,
|
||||||
"normalized_coordinates": res.normalized_coordinates,
|
"normalized_coordinates": res.normalized_coordinates,
|
||||||
"target_semantics": target_semantics,
|
"target_semantics": target_semantics,
|
||||||
|
"target_coordinates": target_coordinates,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -232,24 +265,35 @@ class McpActionExecutor:
|
|||||||
# --- Argument translation --------------------------------------------------------
|
# --- Argument translation --------------------------------------------------------
|
||||||
|
|
||||||
def _translate(
|
def _translate(
|
||||||
self, raw_name: str, args: dict[str, Any], state: Any
|
self,
|
||||||
|
raw_name: str,
|
||||||
|
args: dict[str, Any],
|
||||||
|
state: Any,
|
||||||
|
*,
|
||||||
|
index_elements: list[dict[str, Any]] | None = None,
|
||||||
) -> tuple[str, dict[str, Any], Any, dict[str, Any]]:
|
) -> tuple[str, dict[str, Any], Any, dict[str, Any]]:
|
||||||
"""Maps agent-facing args to wire args; returns (name, args, finalize, recorded).
|
"""Maps agent-facing args to wire args; returns (name, args, finalize, recorded).
|
||||||
|
|
||||||
``finalize(res)`` adds client-side context to swipe result messages.
|
``finalize(res)`` adds client-side context to swipe result messages.
|
||||||
``recorded`` contains validated target descriptions in the Pro Operator's
|
``recorded`` holds the target semantics in the Pro Operator's record
|
||||||
record format. Focused input and directional swipes have no coordinate
|
format: the observed element fields of an index target, or the validated
|
||||||
target, so they do not record a description.
|
``target_description`` of a coordinate target. Focused input and
|
||||||
|
directional swipes have no target, so they record nothing.
|
||||||
|
``index_elements`` overrides the list index targets resolve against
|
||||||
|
(see :meth:`execute`).
|
||||||
"""
|
"""
|
||||||
if raw_name == "click":
|
if raw_name == "click":
|
||||||
target = self._require_pair(
|
target, recorded = self._resolve_target(
|
||||||
args.get("target") or args.get("coordinates") or [], "click"
|
args.get("target") or args.get("coordinates"),
|
||||||
|
"click",
|
||||||
|
args,
|
||||||
|
state,
|
||||||
|
index_elements=index_elements,
|
||||||
)
|
)
|
||||||
recorded = self._require_description(args, "click")
|
|
||||||
return (
|
return (
|
||||||
"click",
|
"click",
|
||||||
{
|
{
|
||||||
"target": list(target),
|
"target": target,
|
||||||
"times": args.get("times", 1),
|
"times": args.get("times", 1),
|
||||||
"delay_ms": args.get("delay_ms", 100),
|
"delay_ms": args.get("delay_ms", 100),
|
||||||
},
|
},
|
||||||
@@ -258,14 +302,17 @@ class McpActionExecutor:
|
|||||||
)
|
)
|
||||||
|
|
||||||
if raw_name == "long_press":
|
if raw_name == "long_press":
|
||||||
target = self._require_pair(
|
target, recorded = self._resolve_target(
|
||||||
args.get("target") or args.get("coordinates") or [], "long press"
|
args.get("target") or args.get("coordinates"),
|
||||||
|
"long press",
|
||||||
|
args,
|
||||||
|
state,
|
||||||
|
index_elements=index_elements,
|
||||||
)
|
)
|
||||||
recorded = self._require_description(args, "long press")
|
|
||||||
return (
|
return (
|
||||||
"long_press",
|
"long_press",
|
||||||
{
|
{
|
||||||
"target": list(target),
|
"target": target,
|
||||||
"duration_ms": args.get("duration_ms", args.get("duration", 1000)),
|
"duration_ms": args.get("duration_ms", args.get("duration", 1000)),
|
||||||
},
|
},
|
||||||
None,
|
None,
|
||||||
@@ -277,8 +324,9 @@ class McpActionExecutor:
|
|||||||
target = None
|
target = None
|
||||||
recorded = {}
|
recorded = {}
|
||||||
if raw_target:
|
if raw_target:
|
||||||
target = list(self._require_pair(raw_target, "input text"))
|
target, recorded = self._resolve_target(
|
||||||
recorded = self._require_description(args, "input text")
|
raw_target, "input text", args, state, index_elements=index_elements
|
||||||
|
)
|
||||||
return (
|
return (
|
||||||
"input_text",
|
"input_text",
|
||||||
{
|
{
|
||||||
@@ -304,7 +352,7 @@ class McpActionExecutor:
|
|||||||
)
|
)
|
||||||
|
|
||||||
if raw_name == "swipe":
|
if raw_name == "swipe":
|
||||||
return self._translate_swipe(args, state)
|
return self._translate_swipe(args, state, index_elements=index_elements)
|
||||||
|
|
||||||
if raw_name == "press_key":
|
if raw_name == "press_key":
|
||||||
return "press_key", {"key": args.get("key", "BACK")}, None, {}
|
return "press_key", {"key": args.get("key", "BACK")}, None, {}
|
||||||
@@ -338,17 +386,101 @@ class McpActionExecutor:
|
|||||||
# Backend extension: pass the arguments straight through.
|
# Backend extension: pass the arguments straight through.
|
||||||
return raw_name, dict(args), None, {}
|
return raw_name, dict(args), None, {}
|
||||||
|
|
||||||
@staticmethod
|
def _screen_size(self) -> tuple[int, int]:
|
||||||
def _require_pair(raw: Any, label: str) -> tuple[int, int]:
|
"""The device resolution the indexed element list was built against."""
|
||||||
|
device = getattr(self.ctx, "device", None)
|
||||||
|
width = getattr(device, "device_width", None) if device else None
|
||||||
|
height = getattr(device, "device_height", None) if device else None
|
||||||
|
return (
|
||||||
|
width if isinstance(width, int) and width > 0 else 1080,
|
||||||
|
height if isinstance(height, int) and height > 0 else 2400,
|
||||||
|
)
|
||||||
|
|
||||||
|
def _resolve_target(
|
||||||
|
self,
|
||||||
|
raw: Any,
|
||||||
|
label: str,
|
||||||
|
args: dict[str, Any],
|
||||||
|
state: Any,
|
||||||
|
*,
|
||||||
|
index_elements: list[dict[str, Any]] | None = None,
|
||||||
|
) -> tuple[list[int], dict[str, Any]]:
|
||||||
|
"""Resolves a point target to ``([x, y] normalized, recorded semantics)``.
|
||||||
|
|
||||||
|
An element index carries observed semantics (text, bounds, id, class
|
||||||
|
straight from the indexed list); a coordinate pair carries only the
|
||||||
|
model's own ``target_description``. The two never mix (Pro parity).
|
||||||
|
"""
|
||||||
target = normalize_coordinate_target(raw)
|
target = normalize_coordinate_target(raw)
|
||||||
if not isinstance(target, (list, tuple)) or len(target) != 2:
|
if isinstance(target, bool):
|
||||||
prefix = {
|
target = None
|
||||||
"click": "Error during click",
|
if isinstance(target, (int, float)):
|
||||||
"long press": "Error during long press",
|
return self._resolve_index(int(target), label, state, index_elements=index_elements)
|
||||||
"input text": "Error during input text",
|
if isinstance(target, (list, tuple)) and len(target) == 2:
|
||||||
}[label]
|
recorded = self._require_description(args, label)
|
||||||
raise _ArgError(f"{prefix}: Invalid target format: {raw}")
|
return [int(target[0]), int(target[1])], recorded
|
||||||
return int(target[0]), int(target[1])
|
raise _ArgError(
|
||||||
|
f"Error during {label}: Invalid target format: {raw!r}. Use an element"
|
||||||
|
" index from the Visible UI Elements list (e.g. 3) or normalized [x, y]"
|
||||||
|
" coordinates with a target_description."
|
||||||
|
)
|
||||||
|
|
||||||
|
def _resolve_index(
|
||||||
|
self,
|
||||||
|
index: int,
|
||||||
|
label: str,
|
||||||
|
state: Any,
|
||||||
|
*,
|
||||||
|
index_elements: list[dict[str, Any]] | None = None,
|
||||||
|
) -> tuple[list[int], dict[str, Any]]:
|
||||||
|
"""Maps an element index onto the element's center (normalized 0-1000).
|
||||||
|
|
||||||
|
The recorded fields are what the Operator records for an index target,
|
||||||
|
with bounds normalized like every other Flash coordinate. The index is
|
||||||
|
resolved against ``index_elements`` when given, else against the
|
||||||
|
state's current list.
|
||||||
|
"""
|
||||||
|
elements = self._index_elements(state, index_elements)
|
||||||
|
if not 1 <= index <= len(elements):
|
||||||
|
hint = (
|
||||||
|
" The list is empty on this screen."
|
||||||
|
if not elements
|
||||||
|
else f" Active index range is 1 to {len(elements)}."
|
||||||
|
)
|
||||||
|
raise _ArgError(
|
||||||
|
f"Error during {label}: Invalid target index {index}.{hint} Use an index"
|
||||||
|
" shown in the Visible UI Elements list, or normalized [x, y]"
|
||||||
|
" coordinates with a target_description (ask_explorer can locate an"
|
||||||
|
" element that is visible but not listed)."
|
||||||
|
)
|
||||||
|
element = elements[index - 1]
|
||||||
|
width, height = self._screen_size()
|
||||||
|
|
||||||
|
def _norm_x(v: Any) -> int:
|
||||||
|
return int(max(0, min(1000, round(float(v) * 1000 / max(1, width)))))
|
||||||
|
|
||||||
|
def _norm_y(v: Any) -> int:
|
||||||
|
return int(max(0, min(1000, round(float(v) * 1000 / max(1, height)))))
|
||||||
|
|
||||||
|
center = element.get("center")
|
||||||
|
if not (isinstance(center, (list, tuple)) and len(center) == 2):
|
||||||
|
raise _ArgError(f"Error during {label}: element [{index}] has no usable center.")
|
||||||
|
bounds = element.get("bounds")
|
||||||
|
norm_bounds = None
|
||||||
|
if isinstance(bounds, (list, tuple)) and len(bounds) == 4:
|
||||||
|
norm_bounds = [
|
||||||
|
_norm_x(bounds[0]),
|
||||||
|
_norm_y(bounds[1]),
|
||||||
|
_norm_x(bounds[2]),
|
||||||
|
_norm_y(bounds[3]),
|
||||||
|
]
|
||||||
|
recorded = {
|
||||||
|
"target_text": element.get("text"),
|
||||||
|
"target_bounds": norm_bounds,
|
||||||
|
"target_resource_id": element.get("resource_id"),
|
||||||
|
"target_class": element.get("class"),
|
||||||
|
}
|
||||||
|
return [_norm_x(center[0]), _norm_y(center[1])], recorded
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _require_description(args: dict[str, Any], label: str) -> dict[str, Any]:
|
def _require_description(args: dict[str, Any], label: str) -> dict[str, Any]:
|
||||||
@@ -414,11 +546,23 @@ class McpActionExecutor:
|
|||||||
resolved.append([nx, ny])
|
resolved.append([nx, ny])
|
||||||
return resolved
|
return resolved
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _index_elements(
|
||||||
|
state: Any, index_elements: list[dict[str, Any]] | None
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
"""The element list index targets resolve against (``None`` -> state)."""
|
||||||
|
if index_elements is not None:
|
||||||
|
return index_elements
|
||||||
|
return getattr(state, "indexed_elements", None) or []
|
||||||
|
|
||||||
def _translate_swipe(
|
def _translate_swipe(
|
||||||
self, args: dict[str, Any], state: Any
|
self,
|
||||||
|
args: dict[str, Any],
|
||||||
|
state: Any,
|
||||||
|
*,
|
||||||
|
index_elements: list[dict[str, Any]] | None = None,
|
||||||
) -> tuple[str, dict[str, Any], Any, dict[str, Any]]:
|
) -> tuple[str, dict[str, Any], Any, dict[str, Any]]:
|
||||||
width = getattr(self.ctx.device, "device_width", 1080) if self.ctx.device else 1080
|
width, height = self._screen_size()
|
||||||
height = getattr(self.ctx.device, "device_height", 2400) if self.ctx.device else 2400
|
|
||||||
default_duration = args.get("duration", 400)
|
default_duration = args.get("duration", 400)
|
||||||
kind, target, final_duration = parse_swipe_parameters(
|
kind, target, final_duration = parse_swipe_parameters(
|
||||||
dict(args), default_duration=default_duration
|
dict(args), default_duration=default_duration
|
||||||
@@ -429,7 +573,7 @@ class McpActionExecutor:
|
|||||||
x1, y1, x2, y2, smart_dur = compute_smart_swipe_coordinates(
|
x1, y1, x2, y2, smart_dur = compute_smart_swipe_coordinates(
|
||||||
direction=target,
|
direction=target,
|
||||||
target=args.get("target"),
|
target=args.get("target"),
|
||||||
indexed_elements=getattr(state, "indexed_elements", None) if state else None,
|
indexed_elements=self._index_elements(state, index_elements) if state else None,
|
||||||
ui_hierarchy=getattr(state, "latest_ui_hierarchy", None) if state else None,
|
ui_hierarchy=getattr(state, "latest_ui_hierarchy", None) if state else None,
|
||||||
width=width,
|
width=width,
|
||||||
height=height,
|
height=height,
|
||||||
@@ -608,6 +752,9 @@ class McpActionExecutor:
|
|||||||
self.ctx, state, query, context_feedback or "", agent_name=self.agent_name
|
self.ctx, state, query, context_feedback or "", agent_name=self.agent_name
|
||||||
)
|
)
|
||||||
registered = register_candidates(self.ctx, state, outcome)
|
registered = register_candidates(self.ctx, state, outcome)
|
||||||
return render_text(query, outcome, registered), not outcome.error
|
# Candidates join ``state.indexed_elements``, and this executor's
|
||||||
|
# ``click`` resolves an index against that list (Pro parity), so the
|
||||||
|
# answer teaches the index syntax next to the coordinates.
|
||||||
|
return render_text(query, outcome, registered, index_targets=True), not outcome.error
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
return f"Error executing ask_explorer: {e}", False
|
return f"Error executing ask_explorer: {e}", False
|
||||||
+139
-339
@@ -21,49 +21,46 @@ the declarations drifted apart (``click.target`` took an element index in one pl
|
|||||||
and only coordinates in another). This module is now the single place all three are
|
and only coordinates in another). This module is now the single place all three are
|
||||||
*defined*; the historical sites import their surface from here, generated.
|
*defined*; the historical sites import their surface from here, generated.
|
||||||
|
|
||||||
An action has up to three dialects, and the differences between them are **deliberate
|
An action has one agent-facing dialect and one wire dialect:
|
||||||
and declared**, not accidental:
|
|
||||||
|
|
||||||
* ``operator`` -- the LangChain shell tools the Operator binds. The Operator receives
|
* ``operator`` -- the agent dialect. Both profiles receive the same indexed
|
||||||
an indexed "Visible UI Elements" list and its prompt teaches index-first targeting,
|
"Visible UI Elements" list, so target parameters accept ``int`` element indices
|
||||||
so target parameters accept ``int`` element indices as well as normalized
|
as well as normalized coordinates. The Pro Operator binds it as a LangChain shell
|
||||||
coordinates. The shells are declaration-only ("Action Recorded"): the Operator's
|
(declaration-only, "Action Recorded": the Operator's translate step lowers the
|
||||||
translate step lowers them into structured decisions.
|
call into structured decisions); the FlashRunner binds the very same description
|
||||||
* ``declaration`` -- the JSON ``ToolDeclaration`` bound by the Validator's
|
and parameters projected onto a JSON ``ToolDeclaration`` (:func:`tool_declaration`),
|
||||||
FlashRunner. These agents act on coordinates directly (the
|
and ``McpActionExecutor`` resolves an index against the current element list.
|
||||||
element list they see carries coordinates), so target parameters are coordinate
|
One definition, two bindings: the two profiles can no longer drift apart.
|
||||||
pairs and the wording teaches the act-then-observe loop ("The screen after X will
|
* ``declaration`` -- a JSON-only agent dialect, used solely for actions that have
|
||||||
be returned automatically").
|
no operator shell (``click_sequence`` is declared to Flash alone).
|
||||||
* ``wire`` -- what the action MCP server serves, mirroring the actuator protocol
|
* ``wire`` -- what the action MCP server serves, mirroring the actuator protocol
|
||||||
one-to-one: normalized coordinates, millisecond durations, no addressing sugar.
|
one-to-one: normalized coordinates, millisecond durations, no addressing sugar.
|
||||||
The client-side executor (``McpActionExecutor``) lowers the
|
The client-side executor (``McpActionExecutor``) lowers the agent dialect onto it.
|
||||||
agent dialects onto it.
|
|
||||||
|
|
||||||
A coordinate target names nothing by itself, so both agent dialects pair it with
|
A coordinate target names nothing by itself, so the agent dialect pairs it with
|
||||||
``target_description`` (``target_descriptions`` for a ``click_sequence``): the
|
``target_description`` (``target_descriptions`` for a ``click_sequence``): the
|
||||||
model's own statement of what it is aiming at. It is required whenever the target
|
model's own statement of what it is aiming at. It is required whenever the target
|
||||||
is a coordinate (always, in the declaration dialect), recorded verbatim on the
|
is a coordinate, recorded verbatim on the action, and never sent to the wire. An
|
||||||
action, and never sent to the wire. An element index needs none -- its text,
|
element index needs none -- its text, bounds and id are read from the indexed
|
||||||
bounds and id are read from the indexed list -- and the recorded fields stay
|
list -- and the recorded fields stay separate (``target_text``/``target_bounds``/...
|
||||||
separate (``target_text``/``target_bounds``/... for observed elements,
|
for observed elements, ``target_description`` for described coordinates) so no
|
||||||
``target_description`` for described coordinates) so no reader can mistake the
|
reader can mistake the model's belief for an observation.
|
||||||
model's belief for an observation.
|
|
||||||
|
|
||||||
Each ``ActionSpec.param_bridge`` maps operator parameter names onto declaration
|
``tests/unit/mcp/test_action_specs.py`` pins the generated schema of every model
|
||||||
parameter names, so every rename (``duration`` -> ``duration_ms``, ``gesture`` ->
|
surface and checks that the Flash declaration of each shared action is exactly the
|
||||||
``action``) is written down exactly once. ``tests/unit/mcp/test_action_specs.py``
|
operator shell's schema, so a divergence between what one profile sees and what the
|
||||||
re-derives the bridge from the generated schemas, so an undeclared divergence --
|
other sees -- the historical definition of drift -- fails CI instead of shipping.
|
||||||
today's definition of drift -- fails CI instead of shipping.
|
|
||||||
|
|
||||||
The classification of actions (required/optional/internal) stays in
|
The classification of actions (required/optional/internal) stays in
|
||||||
``action_manifest``; this module holds their schemas and teaching text.
|
``action_manifest``; this module holds their schemas and teaching text.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from collections.abc import Awaitable, Callable
|
from collections.abc import Awaitable, Callable
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass
|
||||||
from functools import cache
|
from functools import cache
|
||||||
import inspect
|
import inspect
|
||||||
from typing import Annotated, Any, Literal
|
import types
|
||||||
|
from typing import Annotated, Any, Literal, Union, get_args, get_origin
|
||||||
|
|
||||||
from langchain_core.tools import StructuredTool
|
from langchain_core.tools import StructuredTool
|
||||||
from mcp.types import CallToolResult
|
from mcp.types import CallToolResult
|
||||||
@@ -105,7 +102,11 @@ class ParamSpec:
|
|||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class OperatorDialect:
|
class OperatorDialect:
|
||||||
"""The Operator's LangChain shell: index-capable targets, declaration-only body."""
|
"""The agent dialect: index-capable targets, one wording for both profiles.
|
||||||
|
|
||||||
|
Bound by the Pro Operator as a declaration-only LangChain shell and by the
|
||||||
|
FlashRunner as the JSON projection of the same parameters.
|
||||||
|
"""
|
||||||
|
|
||||||
description: str
|
description: str
|
||||||
params: tuple[ParamSpec, ...]
|
params: tuple[ParamSpec, ...]
|
||||||
@@ -113,7 +114,7 @@ class OperatorDialect:
|
|||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class DeclarationDialect:
|
class DeclarationDialect:
|
||||||
"""The Validator/Flash JSON declaration, wording preserved."""
|
"""A JSON-only agent dialect, for actions without an operator shell."""
|
||||||
|
|
||||||
description: str
|
description: str
|
||||||
parameters: dict[str, Any]
|
parameters: dict[str, Any]
|
||||||
@@ -139,57 +140,43 @@ class ActionSpec:
|
|||||||
|
|
||||||
Attributes:
|
Attributes:
|
||||||
name: The canonical tool name every dialect shares.
|
name: The canonical tool name every dialect shares.
|
||||||
operator: Operator shell dialect, or ``None`` when the Operator does not bind
|
operator: The agent dialect (Operator shell and, projected, the Flash
|
||||||
this action.
|
declaration), or ``None`` when neither profile binds this action as a
|
||||||
declaration: Validator/Flash dialect, or ``None`` when no JSON declaration
|
shell.
|
||||||
exists (an action can be wire-only, reachable through executors but never
|
declaration: A JSON-only agent dialect for actions without an operator
|
||||||
declared to a model directly).
|
shell, or ``None`` (an action can be wire-only, reachable through
|
||||||
|
executors but never declared to a model directly).
|
||||||
wire: Action-server dialect; ``None`` only for purely virtual actions.
|
wire: Action-server dialect; ``None`` only for purely virtual actions.
|
||||||
param_bridge: operator param name -> declaration param name, for every
|
differences: Human-readable record of every deliberate divergence between
|
||||||
operator param, when both dialects exist. Identical names map to
|
the agent dialect and the wire (semantics, not just spelling). Empty
|
||||||
themselves; renames are the deliberate dialect differences.
|
means the dialects align modulo the documented dialect philosophy above.
|
||||||
differences: Human-readable record of every deliberate cross-dialect
|
|
||||||
divergence (semantics, not just spelling). Empty means the dialects
|
|
||||||
align modulo the documented dialect philosophy above.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
name: str
|
name: str
|
||||||
operator: OperatorDialect | None = None
|
operator: OperatorDialect | None = None
|
||||||
declaration: DeclarationDialect | None = None
|
declaration: DeclarationDialect | None = None
|
||||||
wire: WireDialect | None = None
|
wire: WireDialect | None = None
|
||||||
param_bridge: dict[str, str] = field(default_factory=dict)
|
|
||||||
differences: str = ""
|
differences: str = ""
|
||||||
|
|
||||||
|
|
||||||
# --- Shared type aliases -------------------------------------------------------------
|
# --- Shared type aliases -------------------------------------------------------------
|
||||||
|
|
||||||
#: Operator-dialect target: element index into the indexed UI list, or a normalized
|
#: Agent-dialect target: element index into the indexed UI list, or a normalized
|
||||||
#: [x, y] pair. Only the Operator sees this union; see the module docstring.
|
#: [x, y] pair.
|
||||||
IndexOrCoords = int | list[int]
|
IndexOrCoords = int | list[int]
|
||||||
|
|
||||||
Direction = Literal["up", "down", "left", "right"]
|
Direction = Literal["up", "down", "left", "right"]
|
||||||
|
|
||||||
_COORD_PAIR_JSON = {
|
#: Wording for the description that a coordinate target must carry. An element
|
||||||
"type": "array",
|
#: index already names its element (its text, id and bounds are recorded from the
|
||||||
"items": {"type": "integer"},
|
#: indexed list); a bare coordinate names nothing, so the model states what it is
|
||||||
}
|
#: aiming at and that statement is recorded as ``target_description``.
|
||||||
|
_TARGET_DESCRIPTION = (
|
||||||
#: Operator-dialect wording for the description that a coordinate target must carry.
|
|
||||||
#: An element index already names its element (its text, id and bounds are recorded
|
|
||||||
#: from the indexed list); a bare coordinate names nothing, so the model states what
|
|
||||||
#: it is aiming at and that statement is recorded as ``target_description``.
|
|
||||||
_OPERATOR_TARGET_DESCRIPTION = (
|
|
||||||
"What the target is, in a few words (e.g. 'play button', 'search input',"
|
"What the target is, in a few words (e.g. 'play button', 'search input',"
|
||||||
" 'video body'). REQUIRED when target is a coordinate pair; ignored for an"
|
" 'video body'). REQUIRED when target is a coordinate pair; ignored for an"
|
||||||
" element index."
|
" element index."
|
||||||
)
|
)
|
||||||
|
|
||||||
#: Declaration-dialect wording (coordinates-only dialect: always required).
|
|
||||||
_DECLARATION_TARGET_DESCRIPTION = (
|
|
||||||
"What the target is, in a few words (e.g. 'play button', 'search input',"
|
|
||||||
" 'video body'). Required: a coordinate names nothing by itself."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# --- Wire bindings -------------------------------------------------------------------
|
# --- Wire bindings -------------------------------------------------------------------
|
||||||
# Each binding reproduces the exact conversion of the historical hand-written FastMCP
|
# Each binding reproduces the exact conversion of the historical hand-written FastMCP
|
||||||
@@ -274,7 +261,7 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
ParamSpec(
|
ParamSpec(
|
||||||
"target_description",
|
"target_description",
|
||||||
str | None,
|
str | None,
|
||||||
_OPERATOR_TARGET_DESCRIPTION,
|
_TARGET_DESCRIPTION,
|
||||||
required=False,
|
required=False,
|
||||||
default=None,
|
default=None,
|
||||||
),
|
),
|
||||||
@@ -296,39 +283,6 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
),
|
),
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
declaration=DeclarationDialect(
|
|
||||||
description=(
|
|
||||||
"[ACTION] Tap/click on the target coordinate on the screen. The screen"
|
|
||||||
" after click will be returned automatically. For buttons, checkboxes,"
|
|
||||||
" tabs, icons, items in a list, tap ON the element. For text fields /"
|
|
||||||
" search bars / input boxes, tap INSIDE the input box to focus it."
|
|
||||||
),
|
|
||||||
parameters={
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"target": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {"type": "integer"},
|
|
||||||
"description": "Normalized coordinates [x, y] in 0-1000 scale.",
|
|
||||||
},
|
|
||||||
"target_description": {
|
|
||||||
"type": "string",
|
|
||||||
"description": _DECLARATION_TARGET_DESCRIPTION,
|
|
||||||
},
|
|
||||||
"times": {
|
|
||||||
"type": "integer",
|
|
||||||
"description": (
|
|
||||||
"Number of taps to perform (default 1). Set to 2 for double click."
|
|
||||||
),
|
|
||||||
},
|
|
||||||
"delay_ms": {
|
|
||||||
"type": "integer",
|
|
||||||
"description": "Delay between taps in milliseconds (default 100).",
|
|
||||||
},
|
|
||||||
},
|
|
||||||
"required": ["target", "target_description"],
|
|
||||||
},
|
|
||||||
),
|
|
||||||
wire=WireDialect(
|
wire=WireDialect(
|
||||||
description="Tap at a 0-1000 normalized [x, y] coordinate.",
|
description="Tap at a 0-1000 normalized [x, y] coordinate.",
|
||||||
params=(
|
params=(
|
||||||
@@ -338,17 +292,11 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
),
|
),
|
||||||
bind=_wire_click,
|
bind=_wire_click,
|
||||||
),
|
),
|
||||||
param_bridge={
|
|
||||||
"target": "target",
|
|
||||||
"target_description": "target_description",
|
|
||||||
"times": "times",
|
|
||||||
"delay_ms": "delay_ms",
|
|
||||||
},
|
|
||||||
differences=(
|
differences=(
|
||||||
"target: operator accepts an element index or a coordinate pair; the"
|
"target: the agent dialect accepts an element index or a coordinate pair"
|
||||||
" declaration and wire dialects take coordinates only."
|
" (an index is resolved client-side against the indexed element list); the"
|
||||||
" target_description: recorded, never sent to the wire; the operator"
|
" wire takes coordinates only. target_description: recorded, never sent"
|
||||||
" requires it only for coordinate targets, the declaration always."
|
" to the wire; required only for coordinate targets."
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
ActionSpec(
|
ActionSpec(
|
||||||
@@ -407,6 +355,9 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
" burst instead); sequence entries are normalized coordinate pairs in both"
|
" burst instead); sequence entries are normalized coordinate pairs in both"
|
||||||
" dialects (an element index is refused, never resolved client-side)."
|
" dialects (an element index is refused, never resolved client-side)."
|
||||||
" target_descriptions is recorded per entry and never sent to the wire."
|
" target_descriptions is recorded per entry and never sent to the wire."
|
||||||
|
" Unlike click, a burst is not resolved against the element list: its"
|
||||||
|
" whole point is to outrun a transient UI, and the list was captured"
|
||||||
|
" before the trigger tap."
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
ActionSpec(
|
ActionSpec(
|
||||||
@@ -427,7 +378,7 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
ParamSpec(
|
ParamSpec(
|
||||||
"target_description",
|
"target_description",
|
||||||
str | None,
|
str | None,
|
||||||
_OPERATOR_TARGET_DESCRIPTION,
|
_TARGET_DESCRIPTION,
|
||||||
required=False,
|
required=False,
|
||||||
default=None,
|
default=None,
|
||||||
),
|
),
|
||||||
@@ -440,31 +391,6 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
),
|
),
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
declaration=DeclarationDialect(
|
|
||||||
description=(
|
|
||||||
"[ACTION] Long press on a target coordinate on the screen. The screen"
|
|
||||||
" after long pressing will be returned automatically."
|
|
||||||
),
|
|
||||||
parameters={
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"target": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {"type": "integer"},
|
|
||||||
"description": "Normalized coordinates [x, y] in 0-1000 scale.",
|
|
||||||
},
|
|
||||||
"target_description": {
|
|
||||||
"type": "string",
|
|
||||||
"description": _DECLARATION_TARGET_DESCRIPTION,
|
|
||||||
},
|
|
||||||
"duration_ms": {
|
|
||||||
"type": "integer",
|
|
||||||
"description": "Duration of press in milliseconds (default 1000ms).",
|
|
||||||
},
|
|
||||||
},
|
|
||||||
"required": ["target", "target_description"],
|
|
||||||
},
|
|
||||||
),
|
|
||||||
wire=WireDialect(
|
wire=WireDialect(
|
||||||
description="Long-press at a 0-1000 normalized [x, y] coordinate.",
|
description="Long-press at a 0-1000 normalized [x, y] coordinate.",
|
||||||
params=(
|
params=(
|
||||||
@@ -473,19 +399,13 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
),
|
),
|
||||||
bind=_wire_long_press,
|
bind=_wire_long_press,
|
||||||
),
|
),
|
||||||
param_bridge={
|
|
||||||
"target": "target",
|
|
||||||
"target_description": "target_description",
|
|
||||||
"duration": "duration_ms",
|
|
||||||
},
|
|
||||||
differences=(
|
differences=(
|
||||||
"target: operator accepts an element index or a coordinate pair."
|
"target: the agent dialect accepts an element index or a coordinate pair;"
|
||||||
" target_description: recorded, never sent to the wire; the operator"
|
" the wire takes coordinates only. target_description: recorded, never"
|
||||||
" requires it only for coordinate targets, the declaration always."
|
" sent to the wire; required only for coordinate targets."
|
||||||
" duration: the operator spells the duration parameter `duration` (its"
|
" duration: the agent dialect spells the duration parameter `duration`"
|
||||||
" structured decisions and recorded traces carry that key); the"
|
" (structured decisions and recorded traces carry that key); the wire"
|
||||||
" declaration and wire dialects spell it `duration_ms`. Executors accept"
|
" spells it `duration_ms`. Executors accept both spellings."
|
||||||
" both spellings."
|
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
ActionSpec(
|
ActionSpec(
|
||||||
@@ -512,7 +432,7 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
ParamSpec(
|
ParamSpec(
|
||||||
"target_description",
|
"target_description",
|
||||||
str | None,
|
str | None,
|
||||||
_OPERATOR_TARGET_DESCRIPTION,
|
_TARGET_DESCRIPTION,
|
||||||
required=False,
|
required=False,
|
||||||
default=None,
|
default=None,
|
||||||
),
|
),
|
||||||
@@ -527,39 +447,6 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
),
|
),
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
declaration=DeclarationDialect(
|
|
||||||
description=(
|
|
||||||
"[ACTION] Type text into an input field on the screen. The screen after"
|
|
||||||
" typing will be returned automatically. Automatically taps inside the"
|
|
||||||
" input box at target [x, y] to focus before typing."
|
|
||||||
),
|
|
||||||
parameters={
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"text": {
|
|
||||||
"type": "string",
|
|
||||||
"description": "Text to type into the focused input field.",
|
|
||||||
},
|
|
||||||
"target": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {"type": "integer"},
|
|
||||||
"description": "Coordinates [x, y] of the input box in 0-1000 scale.",
|
|
||||||
},
|
|
||||||
"target_description": {
|
|
||||||
"type": "string",
|
|
||||||
"description": _DECLARATION_TARGET_DESCRIPTION,
|
|
||||||
},
|
|
||||||
"clear_exist": {
|
|
||||||
"type": "boolean",
|
|
||||||
"description": (
|
|
||||||
"Whether to clear existing text in the input box before typing"
|
|
||||||
" (default True)."
|
|
||||||
),
|
|
||||||
},
|
|
||||||
},
|
|
||||||
"required": ["text", "target", "target_description"],
|
|
||||||
},
|
|
||||||
),
|
|
||||||
wire=WireDialect(
|
wire=WireDialect(
|
||||||
description=("Type text, optionally focusing a 0-1000 normalized [x, y] target first."),
|
description=("Type text, optionally focusing a 0-1000 normalized [x, y] target first."),
|
||||||
params=(
|
params=(
|
||||||
@@ -569,18 +456,11 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
),
|
),
|
||||||
bind=_wire_input_text,
|
bind=_wire_input_text,
|
||||||
),
|
),
|
||||||
param_bridge={
|
|
||||||
"target": "target",
|
|
||||||
"target_description": "target_description",
|
|
||||||
"text": "text",
|
|
||||||
"clear_exist": "clear_exist",
|
|
||||||
},
|
|
||||||
differences=(
|
differences=(
|
||||||
"target: operator accepts an element index or a coordinate pair, and"
|
"target: the agent dialect accepts an element index or a coordinate pair,"
|
||||||
" requires a target; the wire dialect allows omitting the target to type"
|
" and requires a target; the wire allows omitting the target to type into"
|
||||||
" into the already-focused field. target_description: recorded, never"
|
" the already-focused field. target_description: recorded, never sent to"
|
||||||
" sent to the wire; the operator requires it only for coordinate targets,"
|
" the wire; required only for coordinate targets."
|
||||||
" the declaration always."
|
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
ActionSpec(
|
ActionSpec(
|
||||||
@@ -662,70 +542,6 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
),
|
),
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
declaration=DeclarationDialect(
|
|
||||||
description=(
|
|
||||||
"[ACTION] Perform a swipe, drag, or slider-adjustment gesture on the screen. The screen and UI hierarchy after swipe will be returned automatically.\n\n"
|
|
||||||
"• Directional Scrolling ('direction' or 'action'): Recommended for general browsing and standard page scrolling in most scenarios. Automatically computes safe swipe vectors and adaptive duration, retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings. Supports scoping to a sub-container via 'target'. If it fails on certain custom layouts, fall back to specifying exact coordinates ('start' and 'end') directly.\n"
|
|
||||||
"• Precise Coordinate Gestures ('start', 'end' or coordinates list): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars), drag-and-drop / list reordering, or as a reliable fallback when directional scrolling fails on specific containers. Always drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), swipe fully to the extreme boundary."
|
|
||||||
),
|
|
||||||
parameters={
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"direction": {
|
|
||||||
"type": "string",
|
|
||||||
"enum": ["up", "down", "left", "right"],
|
|
||||||
"description": (
|
|
||||||
"Direction for scrolling and swiping: 'up' (drags bottom-to-top, scrolling down to reveal content below),"
|
|
||||||
" 'down' (drags top-to-bottom, scrolling up to reveal content above),"
|
|
||||||
" 'left' (drags right-to-left, scrolling right),"
|
|
||||||
" 'right' (drags left-to-right, scrolling left)."
|
|
||||||
),
|
|
||||||
},
|
|
||||||
"start": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {"type": "integer"},
|
|
||||||
"description": (
|
|
||||||
"Start normalized coordinates [start_x, start_y] in 0-1000 scale."
|
|
||||||
),
|
|
||||||
},
|
|
||||||
"end": {
|
|
||||||
"type": "array",
|
|
||||||
"items": {"type": "integer"},
|
|
||||||
"description": (
|
|
||||||
"End normalized coordinates [end_x, end_y] in 0-1000 scale."
|
|
||||||
),
|
|
||||||
},
|
|
||||||
"target": {
|
|
||||||
"description": (
|
|
||||||
"Optional target element index (e.g. 2) or container bounds"
|
|
||||||
" [left, top, right, bottom] to scope the directional swipe within."
|
|
||||||
),
|
|
||||||
},
|
|
||||||
"action": {
|
|
||||||
"description": (
|
|
||||||
"Backward-compatible swipe gesture: smart direction string ('up', 'down', 'left', 'right')"
|
|
||||||
" OR precise custom coordinates [start_x, start_y, end_x, end_y] in 0-1000 scale."
|
|
||||||
),
|
|
||||||
},
|
|
||||||
"target_description": {
|
|
||||||
"type": "string",
|
|
||||||
"description": (
|
|
||||||
"What is being dragged, in a few words (e.g. 'brightness slider"
|
|
||||||
" knob'). Required for coordinate gestures ('start'/'end' or a"
|
|
||||||
" coordinates list); ignored for directional scrolling."
|
|
||||||
),
|
|
||||||
},
|
|
||||||
"duration": {
|
|
||||||
"type": "integer",
|
|
||||||
"description": (
|
|
||||||
"Optional swipe/drag duration in milliseconds (default 800). For drag-and-drop,"
|
|
||||||
" list reordering, or sliding/adjusting sliders (e.g., volume, brightness, SeekBars),"
|
|
||||||
" set duration >= 1000 (e.g. 1500). If omitted for directional swipe, duration is computed automatically."
|
|
||||||
),
|
|
||||||
},
|
|
||||||
},
|
|
||||||
},
|
|
||||||
),
|
|
||||||
wire=WireDialect(
|
wire=WireDialect(
|
||||||
description="Swipe between two 0-1000 normalized [x, y] points.",
|
description="Swipe between two 0-1000 normalized [x, y] points.",
|
||||||
params=(
|
params=(
|
||||||
@@ -735,23 +551,12 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
),
|
),
|
||||||
bind=_wire_swipe,
|
bind=_wire_swipe,
|
||||||
),
|
),
|
||||||
param_bridge={
|
|
||||||
"direction": "direction",
|
|
||||||
"start": "start",
|
|
||||||
"end": "end",
|
|
||||||
"target": "target",
|
|
||||||
"gesture": "action",
|
|
||||||
"target_description": "target_description",
|
|
||||||
"duration": "duration",
|
|
||||||
},
|
|
||||||
differences=(
|
differences=(
|
||||||
"target_description: recorded, never sent to the wire; required by both"
|
"target_description: recorded, never sent to the wire; required for"
|
||||||
" agent dialects for coordinate gestures only."
|
" coordinate gestures only. gesture: the legacy combined"
|
||||||
" gesture/action: the legacy combined direction-or-coordinates parameter is"
|
" direction-or-coordinates parameter (executors also accept the older"
|
||||||
" spelled `gesture` in the operator dialect and `action` in the"
|
" `action` spelling). Smart directional swipes exist only in the agent"
|
||||||
" declaration dialect (`action` collides with the operator's structured"
|
" dialect; the wire takes a resolved start/end pair, and direction"
|
||||||
" decision verb field). Smart directional swipes exist only in the agent"
|
|
||||||
" dialects; the wire takes a resolved start/end pair, and direction"
|
|
||||||
" resolution happens client-side against the live UI tree."
|
" resolution happens client-side against the live UI tree."
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
@@ -770,24 +575,6 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
),
|
),
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
declaration=DeclarationDialect(
|
|
||||||
description=(
|
|
||||||
"[ACTION] Press a physical or virtual system button. The screen after"
|
|
||||||
" pressing will be returned automatically."
|
|
||||||
),
|
|
||||||
parameters={
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"key": {
|
|
||||||
"type": "string",
|
|
||||||
"description": (
|
|
||||||
"Standard Android system button name (ENTER, BACK, HOME, APP_SWITCH)."
|
|
||||||
),
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": ["key"],
|
|
||||||
},
|
|
||||||
),
|
|
||||||
wire=WireDialect(
|
wire=WireDialect(
|
||||||
description=(
|
description=(
|
||||||
"Press a device key (home, back, enter, delete, tab, search, menu, app_switch)."
|
"Press a device key (home, back, enter, delete, tab, search, menu, app_switch)."
|
||||||
@@ -795,13 +582,11 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
params=(ParamSpec("key", str),),
|
params=(ParamSpec("key", str),),
|
||||||
bind=_wire_press_key,
|
bind=_wire_press_key,
|
||||||
),
|
),
|
||||||
param_bridge={"key": "key"},
|
|
||||||
differences=(
|
differences=(
|
||||||
"key vocabulary: the operator dialect is a closed uppercase enum (its"
|
"key vocabulary: the agent dialect is a closed uppercase enum (the"
|
||||||
" translate step emits Android KEYCODE_* names); the declaration dialect"
|
" Operator's translate step emits Android KEYCODE_* names); the wire"
|
||||||
" is an open string taught the same uppercase names; the wire takes the"
|
" takes the actuator's bare lowercase key words. `to_canonical_call` and"
|
||||||
" actuator's bare lowercase key words. `to_canonical_call` and the"
|
" the executors normalize between them."
|
||||||
" executors normalize between them."
|
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
ActionSpec(
|
ActionSpec(
|
||||||
@@ -817,26 +602,6 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
),
|
),
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
declaration=DeclarationDialect(
|
|
||||||
description=(
|
|
||||||
"[ACTION] Launch or force stop a specified application. The screen"
|
|
||||||
" after launch/stop will be returned automatically."
|
|
||||||
),
|
|
||||||
parameters={
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"action": {
|
|
||||||
"type": "string",
|
|
||||||
"description": "The action type ('launch' or 'stop').",
|
|
||||||
},
|
|
||||||
"app_name": {
|
|
||||||
"type": "string",
|
|
||||||
"description": "Display name or package name of the application.",
|
|
||||||
},
|
|
||||||
},
|
|
||||||
"required": ["action", "app_name"],
|
|
||||||
},
|
|
||||||
),
|
|
||||||
wire=WireDialect(
|
wire=WireDialect(
|
||||||
description="Launch or stop an app by human-readable name or package.",
|
description="Launch or stop an app by human-readable name or package.",
|
||||||
params=(
|
params=(
|
||||||
@@ -845,7 +610,6 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
),
|
),
|
||||||
bind=_wire_manage_app,
|
bind=_wire_manage_app,
|
||||||
),
|
),
|
||||||
param_bridge={"action": "action", "app_name": "app_name"},
|
|
||||||
),
|
),
|
||||||
ActionSpec(
|
ActionSpec(
|
||||||
name="wait_for_delay",
|
name="wait_for_delay",
|
||||||
@@ -867,32 +631,11 @@ _SPECS: tuple[ActionSpec, ...] = (
|
|||||||
),
|
),
|
||||||
),
|
),
|
||||||
),
|
),
|
||||||
declaration=DeclarationDialect(
|
|
||||||
description=(
|
|
||||||
"[ACTION] Pause execution and wait for a specified duration in milliseconds."
|
|
||||||
" The screen after pause will be returned automatically."
|
|
||||||
),
|
|
||||||
parameters={
|
|
||||||
"type": "object",
|
|
||||||
"properties": {
|
|
||||||
"time_in_ms": {
|
|
||||||
"type": "integer",
|
|
||||||
"description": (
|
|
||||||
"The exact duration to pause in milliseconds (e.g., 2000 for 2s,"
|
|
||||||
" 60000 for 1 min, 180000 for 3 mins, 300000 for 5 mins)."
|
|
||||||
" Convert any required waiting duration into milliseconds."
|
|
||||||
),
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"required": ["time_in_ms"],
|
|
||||||
},
|
|
||||||
),
|
|
||||||
wire=WireDialect(
|
wire=WireDialect(
|
||||||
description="Wait for a fixed number of milliseconds.",
|
description="Wait for a fixed number of milliseconds.",
|
||||||
params=(ParamSpec("time_in_ms", int),),
|
params=(ParamSpec("time_in_ms", int),),
|
||||||
bind=_wire_wait_for_delay,
|
bind=_wire_wait_for_delay,
|
||||||
),
|
),
|
||||||
param_bridge={"time_in_ms": "time_in_ms"},
|
|
||||||
),
|
),
|
||||||
ActionSpec(
|
ActionSpec(
|
||||||
name="wait_for_text",
|
name="wait_for_text",
|
||||||
@@ -1007,16 +750,73 @@ def operator_shell_tool(name: str) -> StructuredTool:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
_SCALAR_JSON: dict[Any, dict[str, str]] = {
|
||||||
|
bool: {"type": "boolean"},
|
||||||
|
int: {"type": "integer"},
|
||||||
|
float: {"type": "number"},
|
||||||
|
str: {"type": "string"},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _json_schema(annotation: Any) -> dict[str, Any]:
|
||||||
|
"""Projects one Python-typed parameter annotation onto JSON schema.
|
||||||
|
|
||||||
|
``None`` members of a union only make the parameter optional (its
|
||||||
|
``required`` flag says so); they never appear in the projected schema, so
|
||||||
|
the declaration carries no ``null`` alternatives or ``default`` noise.
|
||||||
|
"""
|
||||||
|
origin = get_origin(annotation)
|
||||||
|
if origin in (Union, types.UnionType):
|
||||||
|
members = [a for a in get_args(annotation) if a is not type(None)]
|
||||||
|
schemas = [_json_schema(m) for m in members]
|
||||||
|
return schemas[0] if len(schemas) == 1 else {"anyOf": schemas}
|
||||||
|
if origin is Literal:
|
||||||
|
return {"type": "string", "enum": list(get_args(annotation))}
|
||||||
|
if origin is list:
|
||||||
|
(item,) = get_args(annotation) or (str,)
|
||||||
|
return {"type": "array", "items": _json_schema(item)}
|
||||||
|
try:
|
||||||
|
return dict(_SCALAR_JSON[annotation])
|
||||||
|
except KeyError:
|
||||||
|
raise TypeError(f"No JSON projection for parameter annotation {annotation!r}.") from None
|
||||||
|
|
||||||
|
|
||||||
|
def _projected_parameters(dialect: OperatorDialect) -> dict[str, Any]:
|
||||||
|
"""The operator shell's parameters as a JSON-schema ``parameters`` object."""
|
||||||
|
properties: dict[str, Any] = {}
|
||||||
|
required: list[str] = []
|
||||||
|
for p in dialect.params:
|
||||||
|
schema = _json_schema(p.annotation)
|
||||||
|
if p.description:
|
||||||
|
schema["description"] = p.description
|
||||||
|
properties[p.name] = schema
|
||||||
|
if p.required:
|
||||||
|
required.append(p.name)
|
||||||
|
return {"type": "object", "properties": properties, "required": required}
|
||||||
|
|
||||||
|
|
||||||
|
@cache
|
||||||
def tool_declaration(name: str) -> ToolDeclaration:
|
def tool_declaration(name: str) -> ToolDeclaration:
|
||||||
"""Builds the Validator/Flash ``ToolDeclaration`` for one action."""
|
"""Builds the Flash ``ToolDeclaration`` for one action.
|
||||||
dialect = ACTION_SPECS[name].declaration
|
|
||||||
if dialect is None:
|
A shared action is the operator shell projected onto JSON (same description,
|
||||||
raise ValueError(f"Action '{name}' has no declaration dialect.")
|
same parameters, same targeting semantics); a JSON-only action uses its own
|
||||||
|
``DeclarationDialect``.
|
||||||
|
"""
|
||||||
|
spec = ACTION_SPECS[name]
|
||||||
|
if spec.declaration is not None:
|
||||||
return ToolDeclaration(
|
return ToolDeclaration(
|
||||||
name=name,
|
name=name,
|
||||||
description=dialect.description,
|
description=spec.declaration.description,
|
||||||
parameters=dialect.parameters,
|
parameters=spec.declaration.parameters,
|
||||||
)
|
)
|
||||||
|
if spec.operator is not None:
|
||||||
|
return ToolDeclaration(
|
||||||
|
name=name,
|
||||||
|
description=spec.operator.description,
|
||||||
|
parameters=_projected_parameters(spec.operator),
|
||||||
|
)
|
||||||
|
raise ValueError(f"Action '{name}' has no agent dialect to declare.")
|
||||||
|
|
||||||
|
|
||||||
def wire_dialects() -> tuple[ActionSpec, ...]:
|
def wire_dialects() -> tuple[ActionSpec, ...]:
|
||||||
|
|||||||
+315
-133
@@ -12,7 +12,7 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
"""History compression into segment chunks, eras, and a session snapshot.
|
"""History compression into segment chunks, eras, and recall-only periods.
|
||||||
|
|
||||||
Each chunk keeps three bands: synopsis and effects, interval summaries, and a
|
Each chunk keeps three bands: synopsis and effects, interval summaries, and a
|
||||||
per-step action ledger. Capsule generation reads the operator's turn transcripts
|
per-step action ledger. Capsule generation reads the operator's turn transcripts
|
||||||
@@ -20,12 +20,18 @@ alongside the recorded step facts. Milestone changes, segment size, and token
|
|||||||
limits determine chunk boundaries.
|
limits determine chunk boundaries.
|
||||||
|
|
||||||
Original messages remain in context until their capsule is ready. Failed capsule
|
Original messages remain in context until their capsule is ready. Failed capsule
|
||||||
jobs retain their source text for retry. Only the hard token limit forces pending
|
jobs retain their source text for retry. At the hard token limit, ready chunks
|
||||||
chunks into the session snapshot. Older chunks fold into eras and then period
|
swap as ordinary L2 blocks first; only when that cannot bring the context back
|
||||||
summaries; full step records remain available through the history tools.
|
under the line does everything closed force-swap (pending chunks included) and
|
||||||
|
the whole frozen region fold into a recall-only era — L3: the period paragraph
|
||||||
|
plus the per-step minimal index (step number, session offset, action phrase).
|
||||||
|
The fold is monotonic: later L2 swaps append new chunk blocks after the folded
|
||||||
|
eras and never restore the folded chunks. Older chunks also fold into eras and
|
||||||
|
then recall-only periods by count overflow; full step records remain available
|
||||||
|
through the history tools.
|
||||||
|
|
||||||
Checkpoint annotations persist independently of capsule generation. Frozen
|
Checkpoint annotations persist independently of capsule generation. Frozen
|
||||||
context is rebuilt only when a capsule or snapshot replaces the source turns.
|
context is rebuilt only when a compression event replaces or folds turns.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
@@ -61,6 +67,29 @@ _NOTE_TOOLS = ("save_note", "update_note", "append_note")
|
|||||||
# Reuse one tool trace per chunk across running, success, and failed states.
|
# Reuse one tool trace per chunk across running, success, and failed states.
|
||||||
COMPRESSION_TRACE_NAME = "compress_history"
|
COMPRESSION_TRACE_NAME = "compress_history"
|
||||||
|
|
||||||
|
#: Plain-language phases carried in the trace payload (``args.phase``) so the
|
||||||
|
#: timeline can say what the compression is *doing* independently of the
|
||||||
|
#: trace ``status`` (running/success/failed), which other code keys on.
|
||||||
|
#: summarizing — the segment closed and its capsule is generating (running)
|
||||||
|
#: ready — the capsule is on hand but the start gate holds the swap (running)
|
||||||
|
#: applied — the compressed block replaced the raw turns (success)
|
||||||
|
#: failed — capsule generation gave up; the full record is kept (failed)
|
||||||
|
COMPRESSION_PHASE_SUMMARIZING = "summarizing"
|
||||||
|
COMPRESSION_PHASE_READY = "ready"
|
||||||
|
COMPRESSION_PHASE_APPLIED = "applied"
|
||||||
|
COMPRESSION_PHASE_FAILED = "failed"
|
||||||
|
COMPRESSION_PHASES = (
|
||||||
|
COMPRESSION_PHASE_SUMMARIZING,
|
||||||
|
COMPRESSION_PHASE_READY,
|
||||||
|
COMPRESSION_PHASE_APPLIED,
|
||||||
|
COMPRESSION_PHASE_FAILED,
|
||||||
|
)
|
||||||
|
_STATUS_TO_PHASE = {
|
||||||
|
"running": COMPRESSION_PHASE_SUMMARIZING,
|
||||||
|
"success": COMPRESSION_PHASE_APPLIED,
|
||||||
|
"failed": COMPRESSION_PHASE_FAILED,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Band ③ — mechanical per-step action ledger
|
# Band ③ — mechanical per-step action ledger
|
||||||
@@ -161,7 +190,7 @@ def extract_note_writes(step: dict) -> list[dict[str, str]]:
|
|||||||
|
|
||||||
|
|
||||||
def validate_interval_coverage(intervals: Any, start_step: int, end_step: int) -> bool:
|
def validate_interval_coverage(intervals: Any, start_step: int, end_step: int) -> bool:
|
||||||
"""Band ② hard constraint: the interval union covers [start, end] seamlessly."""
|
"""Band ② hard constraint: the interval union covers [start, end] without gaps or overlaps."""
|
||||||
if not isinstance(intervals, list) or not intervals:
|
if not isinstance(intervals, list) or not intervals:
|
||||||
return False
|
return False
|
||||||
expected = start_step
|
expected = start_step
|
||||||
@@ -289,7 +318,7 @@ class StepCapsuleLens(StepLens):
|
|||||||
" keys doing/did/effect/entry_state/exit_state/verified_facts/"
|
" keys doing/did/effect/entry_state/exit_state/verified_facts/"
|
||||||
"unresolved/failed_paths/important_entities/intervals. Never use"
|
"unresolved/failed_paths/important_entities/intervals. Never use"
|
||||||
" verdict words (successfully, completed, failed, ...); intervals"
|
" verdict words (successfully, completed, failed, ...); intervals"
|
||||||
" must cover the step range seamlessly. Return only JSON."
|
" must cover the full step range without gaps or overlaps. Return only JSON."
|
||||||
)
|
)
|
||||||
|
|
||||||
def _get_llm(self):
|
def _get_llm(self):
|
||||||
@@ -396,8 +425,9 @@ class StepCapsuleLens(StepLens):
|
|||||||
def _cap_transcript(cls, transcript: str) -> str:
|
def _cap_transcript(cls, transcript: str) -> str:
|
||||||
"""Cap one turn transcript in place, keeping its head and its tail.
|
"""Cap one turn transcript in place, keeping its head and its tail.
|
||||||
|
|
||||||
The tail is kept intact because the validator result message closes
|
The tail is kept intact because the turn's last messages (the final
|
||||||
every turn and outranks everything above it (fact priority); the cut
|
action's tool result, or the failure result message that closes a
|
||||||
|
failed turn) outrank everything above them (fact priority); the cut
|
||||||
lands in the middle and is announced with its size.
|
lands in the middle and is announced with its size.
|
||||||
"""
|
"""
|
||||||
cap = cls.MAX_TURN_TRANSCRIPT_CHARS
|
cap = cls.MAX_TURN_TRANSCRIPT_CHARS
|
||||||
@@ -475,8 +505,9 @@ class StepCapsuleLens(StepLens):
|
|||||||
async def _invoke(self, llm: Any, messages: list[BaseMessage], model_label: str):
|
async def _invoke(self, llm: Any, messages: list[BaseMessage], model_label: str):
|
||||||
response = await asyncio.wait_for(llm.ainvoke(messages), timeout=90.0)
|
response = await asyncio.wait_for(llm.ainvoke(messages), timeout=90.0)
|
||||||
# Raw-model bypass metering (gateway-wrapped models meter themselves);
|
# Raw-model bypass metering (gateway-wrapped models meter themselves);
|
||||||
# capsule prompts are small and must not clobber the session's
|
# capsule prompts are small and are kept out of the session meter's
|
||||||
# last_prompt_tokens (the compaction thresholds' live context base).
|
# last_prompt_tokens (the compaction base itself is operator-owned on
|
||||||
|
# the transcript ledger and never reads the session meter).
|
||||||
try:
|
try:
|
||||||
from artemis.services.llm import RobustChatModelWrapper
|
from artemis.services.llm import RobustChatModelWrapper
|
||||||
from artemis.services.token_meter import record_llm_usage
|
from artemis.services.token_meter import record_llm_usage
|
||||||
@@ -588,7 +619,8 @@ class ChunkState:
|
|||||||
trace_id: Any = None
|
trace_id: Any = None
|
||||||
trace_step_id: Any = None
|
trace_step_id: Any = None
|
||||||
announced_at: float | None = None
|
announced_at: float | None = None
|
||||||
# Measure source at close and summary at swap; estimate tokens as chars // 4.
|
# Measure source at close and summary at swap; tokens are derived through
|
||||||
|
# the ledger's calibrated chars-per-token ratio (chars // 4 by default).
|
||||||
# Forced swaps without a capsule leave summary_chars unset.
|
# Forced swaps without a capsule leave summary_chars unset.
|
||||||
source_chars: int = 0
|
source_chars: int = 0
|
||||||
summary_chars: int | None = None
|
summary_chars: int | None = None
|
||||||
@@ -762,8 +794,13 @@ def render_era_period_paragraph(era: EraState) -> str:
|
|||||||
def render_era_block(era: EraState) -> str:
|
def render_era_block(era: EraState) -> str:
|
||||||
"""Era block: ① merged headers, ② degraded to titles, ③ per-chunk ledgers.
|
"""Era block: ① merged headers, ② degraded to titles, ③ per-chunk ledgers.
|
||||||
|
|
||||||
A recall-only era includes the step range and session offsets for video
|
A recall-only era (L3) includes the step range and session offsets for
|
||||||
alignment, a synopsis, recall guidance, and preserved user instructions.
|
video alignment, a synopsis, recall guidance, and the per-step minimal
|
||||||
|
index of every member chunk (step number, session offset, action phrase
|
||||||
|
— results only via search_history). Pending chunks contribute their
|
||||||
|
index too. User-injected instruction lines are never evicted: they sit
|
||||||
|
inside the index at their step, or are appended when a chunk carries no
|
||||||
|
index text.
|
||||||
"""
|
"""
|
||||||
if era.recall_only:
|
if era.recall_only:
|
||||||
lines = [
|
lines = [
|
||||||
@@ -774,9 +811,12 @@ def render_era_block(era: EraState) -> str:
|
|||||||
),
|
),
|
||||||
RECALL_GUIDANCE_TEMPLATE.format(start=era.start_step_number, end=era.end_step_number),
|
RECALL_GUIDANCE_TEMPLATE.format(start=era.start_step_number, end=era.end_step_number),
|
||||||
]
|
]
|
||||||
# Never-evict: user-injected instruction lines survive even recall-only.
|
|
||||||
for chunk in era.chunks:
|
for chunk in era.chunks:
|
||||||
lines.extend(chunk.user_lines)
|
if chunk.minimal_index:
|
||||||
|
lines.append(chunk.minimal_index)
|
||||||
|
rendered = "\n".join(lines)
|
||||||
|
for chunk in era.chunks:
|
||||||
|
lines.extend(line for line in chunk.user_lines if line not in rendered)
|
||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
|
|
||||||
merged = merge_structured_fields([c.band1 for c in era.chunks if c.band1])
|
merged = merge_structured_fields([c.band1 for c in era.chunks if c.band1])
|
||||||
@@ -816,48 +856,6 @@ def render_era_block(era: EraState) -> str:
|
|||||||
return "\n".join(parts)
|
return "\n".join(parts)
|
||||||
|
|
||||||
|
|
||||||
def render_l3_snapshot(
|
|
||||||
eras: list[EraState],
|
|
||||||
chunks: list[ChunkState],
|
|
||||||
*,
|
|
||||||
goal: str | None,
|
|
||||||
plan_text: str | None,
|
|
||||||
) -> str:
|
|
||||||
"""L3 emergency snapshot: merged knowledge plane + minimal per-step index.
|
|
||||||
|
|
||||||
The chunk headers are mechanically set-merged (no summary-of-summary); the
|
|
||||||
per-step skeleton survives at minimal width so step-level time perception
|
|
||||||
never breaks.
|
|
||||||
"""
|
|
||||||
all_chunks = [c for era in eras for c in era.chunks] + list(chunks)
|
|
||||||
merged = merge_structured_fields([c.band1 for c in all_chunks if c.band1])
|
|
||||||
exit_state = ""
|
|
||||||
for chunk in reversed(all_chunks):
|
|
||||||
if chunk.band1:
|
|
||||||
exit_state = chunk.band1.get("exit_state") or ""
|
|
||||||
break
|
|
||||||
|
|
||||||
start = all_chunks[0].start_step_number if all_chunks else 0
|
|
||||||
end = all_chunks[-1].end_step_number if all_chunks else 0
|
|
||||||
parts = [
|
|
||||||
f"[Session snapshot | Steps {start}–{end} | hard-threshold fallback]",
|
|
||||||
f"Overall goal: {goal or '-'}",
|
|
||||||
"Plan state:",
|
|
||||||
(plan_text or "-").rstrip(),
|
|
||||||
f"Verified facts: {'; '.join(merged['verified_facts']) or '-'}",
|
|
||||||
f"Unresolved: {'; '.join(merged['unresolved']) or '-'}",
|
|
||||||
f"Failed paths: {'; '.join(merged['failed_paths']) or '-'}",
|
|
||||||
f"Important entities: {'; '.join(merged['important_entities']) or '-'}",
|
|
||||||
f"Device/app state: {exit_state or '-'}",
|
|
||||||
"",
|
|
||||||
"--- Step index (minimal width, per chunk) ---",
|
|
||||||
]
|
|
||||||
for chunk in all_chunks:
|
|
||||||
parts.append(f"[{chunk.step_range_label}]")
|
|
||||||
parts.append(chunk.minimal_index)
|
|
||||||
return "\n".join(parts)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# HistoryChunkManager — triggers, compression events, F-region rendering
|
# HistoryChunkManager — triggers, compression events, F-region rendering
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -885,10 +883,13 @@ class HistoryChunkManager:
|
|||||||
):
|
):
|
||||||
self._engine = engine
|
self._engine = engine
|
||||||
self._ctx = ctx
|
self._ctx = ctx
|
||||||
|
# Accepted for the callers' wiring; the frozen region no longer
|
||||||
|
# recites the goal (the prompt carries it elsewhere).
|
||||||
self._goal = goal
|
self._goal = goal
|
||||||
|
|
||||||
cc = chunking_config
|
cc = chunking_config
|
||||||
self._max_steps = int(getattr(cc, "max_steps", 12) or 12)
|
self._max_steps = int(getattr(cc, "max_steps", 12) or 12)
|
||||||
|
self._min_steps = max(1, min(self._max_steps, int(getattr(cc, "min_steps", 3) or 3)))
|
||||||
self._target_source_tokens = int(getattr(cc, "target_source_tokens", 2000) or 2000)
|
self._target_source_tokens = int(getattr(cc, "target_source_tokens", 2000) or 2000)
|
||||||
self._model_name = getattr(cc, "model", None) or "gemini-3.8-flash"
|
self._model_name = getattr(cc, "model", None) or "gemini-3.8-flash"
|
||||||
self._max_chunks = int(getattr(cc, "max_chunks", 8) or 8)
|
self._max_chunks = int(getattr(cc, "max_chunks", 8) or 8)
|
||||||
@@ -897,6 +898,8 @@ class HistoryChunkManager:
|
|||||||
|
|
||||||
tc = transcript_config
|
tc = transcript_config
|
||||||
self._budget = int(getattr(tc, "context_budget_tokens", 80000) or 80000)
|
self._budget = int(getattr(tc, "context_budget_tokens", 80000) or 80000)
|
||||||
|
start_ratio = getattr(tc, "start_ratio", None)
|
||||||
|
self._start_ratio = float(0.35 if start_ratio is None else start_ratio)
|
||||||
self._soft_ratio = float(getattr(tc, "soft_ratio", 0.7) or 0.7)
|
self._soft_ratio = float(getattr(tc, "soft_ratio", 0.7) or 0.7)
|
||||||
self._hard_ratio = float(getattr(tc, "hard_ratio", 0.9) or 0.9)
|
self._hard_ratio = float(getattr(tc, "hard_ratio", 0.9) or 0.9)
|
||||||
self._min_active_steps = int(getattr(tc, "min_active_steps", 5) or 5)
|
self._min_active_steps = int(getattr(tc, "min_active_steps", 5) or 5)
|
||||||
@@ -916,7 +919,7 @@ class HistoryChunkManager:
|
|||||||
# whose original turns still live in the transcript until their
|
# whose original turns still live in the transcript until their
|
||||||
# capsule header is ready (ready-gated swap). An entry's ``chunk`` is
|
# capsule header is ready (ready-gated swap). An entry's ``chunk`` is
|
||||||
# None when the segment had no step records — it can then only leave
|
# None when the segment had no step records — it can then only leave
|
||||||
# the queue through the hard-threshold emergency swap.
|
# the queue through the hard-threshold force-swap.
|
||||||
self._chunks: list[ChunkState] = []
|
self._chunks: list[ChunkState] = []
|
||||||
self._eras: list[EraState] = []
|
self._eras: list[EraState] = []
|
||||||
self._awaiting: list[dict[str, Any]] = []
|
self._awaiting: list[dict[str, Any]] = []
|
||||||
@@ -1065,11 +1068,11 @@ class HistoryChunkManager:
|
|||||||
logger.error(f"History chunk compression event failed: {e}")
|
logger.error(f"History chunk compression event failed: {e}")
|
||||||
|
|
||||||
def _on_render_inner(self, ledger) -> None:
|
def _on_render_inner(self, ledger) -> None:
|
||||||
base_tokens = self._context_base_tokens()
|
base_tokens = self._context_base_tokens(ledger)
|
||||||
soft = base_tokens is not None and base_tokens >= self._budget * self._soft_ratio
|
soft = base_tokens is not None and base_tokens >= self._budget * self._soft_ratio
|
||||||
hard = base_tokens is not None and base_tokens >= self._budget * self._hard_ratio
|
hard = base_tokens is not None and base_tokens >= self._budget * self._hard_ratio
|
||||||
|
|
||||||
closed_any = self._close_new_segments(ledger, soft)
|
closed_any = self._close_new_segments(ledger, soft, hard)
|
||||||
if closed_any or soft or hard:
|
if closed_any or soft or hard:
|
||||||
# Failure rung of the degradation ladder: exhausted capsules are
|
# Failure rung of the degradation ladder: exhausted capsules are
|
||||||
# re-dispatched (the lens retries a fallback model per attempt);
|
# re-dispatched (the lens retries a fallback model per attempt);
|
||||||
@@ -1079,12 +1082,20 @@ class HistoryChunkManager:
|
|||||||
self._harvest_capsules()
|
self._harvest_capsules()
|
||||||
self._swap_ready_segments(ledger, hard=hard, base_tokens=base_tokens)
|
self._swap_ready_segments(ledger, hard=hard, base_tokens=base_tokens)
|
||||||
|
|
||||||
def _close_new_segments(self, ledger, soft: bool) -> bool:
|
def _close_new_segments(self, ledger, soft: bool, hard: bool = False) -> bool:
|
||||||
"""Trigger evaluation: close due segments and dispatch their capsules.
|
"""Trigger evaluation: close due segments and dispatch their capsules.
|
||||||
|
|
||||||
Closing NEVER freezes turns (ready-gated swap): the segment's original
|
Closing NEVER freezes turns (ready-gated swap): the segment's original
|
||||||
messages stay live in the transcript; a closed segment joins the
|
messages stay live in the transcript; a closed segment joins the
|
||||||
``_awaiting`` queue until its capsule header is ready.
|
``_awaiting`` queue until its capsule header is ready.
|
||||||
|
|
||||||
|
Minimum chunk length: the size and pressure triggers only close once
|
||||||
|
at least ``min_steps`` eligible turns have accumulated in the open
|
||||||
|
segment — turns age past the sliding-window floor one per render, so
|
||||||
|
without it a heavy session closes a one-turn chunk (one capsule call)
|
||||||
|
every render. A milestone close is exempt (a complete segment is a
|
||||||
|
unit, however short) and the hard threshold waives it (emergency:
|
||||||
|
whatever is eligible closes so the swap has material).
|
||||||
"""
|
"""
|
||||||
turns = ledger.unchunked_turns()
|
turns = ledger.unchunked_turns()
|
||||||
if len(turns) <= self._min_active_steps:
|
if len(turns) <= self._min_active_steps:
|
||||||
@@ -1128,17 +1139,19 @@ class HistoryChunkManager:
|
|||||||
if tail_segment is not None:
|
if tail_segment is not None:
|
||||||
tail_portion = tail_segment[1][: max(0, remaining_eligible - consumed_prefix)]
|
tail_portion = tail_segment[1][: max(0, remaining_eligible - consumed_prefix)]
|
||||||
|
|
||||||
|
long_enough = hard or len(tail_portion) >= self._min_steps
|
||||||
|
|
||||||
size_event = False
|
size_event = False
|
||||||
if tail_segment is not None and tail_portion:
|
if tail_segment is not None and tail_portion and long_enough:
|
||||||
tail_chars = ledger.turn_text_chars(tail_portion)
|
tail_chars = ledger.turn_text_chars(tail_portion)
|
||||||
size_event = (
|
size_event = (
|
||||||
len(tail_segment[1]) >= self._max_steps
|
len(tail_segment[1]) >= self._max_steps
|
||||||
or (tail_chars // 4) >= self._target_source_tokens
|
or self._chars_to_tokens(tail_chars, ledger) >= self._target_source_tokens
|
||||||
)
|
)
|
||||||
|
|
||||||
if size_event:
|
if size_event:
|
||||||
selected.append((tail_segment[0], tail_portion, "size"))
|
selected.append((tail_segment[0], tail_portion, "size"))
|
||||||
elif not milestone_event and soft and tail_portion:
|
elif not milestone_event and soft and tail_portion and long_enough:
|
||||||
# Soft threshold with nothing else due: close the oldest open
|
# Soft threshold with nothing else due: close the oldest open
|
||||||
# segment's eligible portion (bounded by the chunk size cap).
|
# segment's eligible portion (bounded by the chunk size cap).
|
||||||
selected.append((tail_segment[0], tail_portion[: self._max_steps], "pressure"))
|
selected.append((tail_segment[0], tail_portion[: self._max_steps], "pressure"))
|
||||||
@@ -1176,6 +1189,10 @@ class HistoryChunkManager:
|
|||||||
" retained until a capsule lands."
|
" retained until a capsule lands."
|
||||||
)
|
)
|
||||||
self._capsule_service.submit(key, payload)
|
self._capsule_service.submit(key, payload)
|
||||||
|
# A fresh attempt: if the chunk readies again while the start
|
||||||
|
# gate still holds it, the timeline gets a fresh ``held`` note
|
||||||
|
# instead of staying on ``retrying``.
|
||||||
|
entry["held_announced"] = False
|
||||||
self._announce(chunk, "running", note="retrying")
|
self._announce(chunk, "running", note="retrying")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Capsule re-dispatch for {key} failed: {e}")
|
logger.error(f"Capsule re-dispatch for {key} failed: {e}")
|
||||||
@@ -1185,41 +1202,91 @@ class HistoryChunkManager:
|
|||||||
|
|
||||||
Swaps consume the transcript's oldest unchunked turns, so only a
|
Swaps consume the transcript's oldest unchunked turns, so only a
|
||||||
contiguous READY prefix may swap — an older pending segment keeps its
|
contiguous READY prefix may swap — an older pending segment keeps its
|
||||||
(and every younger segment's) original text live. The hard threshold
|
(and every younger segment's) original text live.
|
||||||
is the sole exception: everything closed force-swaps, pending chunks
|
|
||||||
included, and the frozen region renders as the L3 snapshot.
|
Start gate: closing a segment (milestone/size triggers) only *prepares*
|
||||||
|
its capsule in the background; the original text is replaced only once
|
||||||
|
the operator's measured context has reached ``budget * start_ratio``.
|
||||||
|
Below that the ready chunks are held in the awaiting queue, so at low
|
||||||
|
occupancy the model keeps reading the full transcript while capsules
|
||||||
|
are already on hand for when pressure arrives. An unknown base (no
|
||||||
|
measured operator call yet, or a provider without usage metadata)
|
||||||
|
cannot gate and falls through to swap-on-ready.
|
||||||
|
|
||||||
|
Hard threshold ladder (L2 first, L3 last): at ``budget * hard_ratio``
|
||||||
|
the ready prefix swaps as ordinary L2 chunk blocks when the estimated
|
||||||
|
context after that swap drops back under the hard line (rung 1). Only
|
||||||
|
when no ready prefix exists, or the L2 swap would still leave the
|
||||||
|
context at or above the line, does everything closed force-swap —
|
||||||
|
pending chunks included — and the whole frozen region fold into a
|
||||||
|
recall-only era (rung 2: L3 = period paragraph + per-step minimal
|
||||||
|
index). With nothing awaiting at all, the last resort is folding the
|
||||||
|
existing frozen region the same way without consuming turns (rung 3).
|
||||||
|
The fold is a monotonic state change: later L2 swaps append new chunk
|
||||||
|
blocks after the folded eras and never restore the folded chunks;
|
||||||
|
repeated hard renders with nothing left to fold leave the frozen
|
||||||
|
blocks byte-identical.
|
||||||
|
|
||||||
``base_tokens`` is the prompt size before the swap, used to estimate
|
``base_tokens`` is the prompt size before the swap, used to estimate
|
||||||
the remaining context on the last swapped chunk's trace.
|
the remaining context on the last swapped chunk's trace.
|
||||||
"""
|
"""
|
||||||
|
hard_line = self._budget * self._hard_ratio
|
||||||
if not self._awaiting:
|
if not self._awaiting:
|
||||||
|
if hard and self._collapse_frozen_region():
|
||||||
|
ledger.freeze_turns(0, self._render_frozen_blocks())
|
||||||
|
logger.info(
|
||||||
|
"History compression: hard threshold with nothing awaiting;"
|
||||||
|
" frozen region folded into the recall-only period."
|
||||||
|
)
|
||||||
return
|
return
|
||||||
swap: list[dict[str, Any]] = []
|
|
||||||
if hard:
|
# An entry without a chunk (turns with no step records: nothing to
|
||||||
swap, self._awaiting = self._awaiting, []
|
# summarize, nothing to render) is consumable at once — it must not
|
||||||
else:
|
# pin the queue, or no L2 swap ever happens and the hard line jumps
|
||||||
while self._awaiting:
|
# straight to the fold.
|
||||||
chunk = self._awaiting[0]["chunk"]
|
ready_prefix: list[dict[str, Any]] = []
|
||||||
if chunk is None or chunk.status != "ready":
|
for entry in self._awaiting:
|
||||||
|
chunk = entry["chunk"]
|
||||||
|
if chunk is not None and chunk.status != "ready":
|
||||||
break
|
break
|
||||||
swap.append(self._awaiting.pop(0))
|
ready_prefix.append(entry)
|
||||||
|
|
||||||
|
swap: list[dict[str, Any]] = []
|
||||||
|
fold = False
|
||||||
|
if hard:
|
||||||
|
estimate = self._estimate_context_after(base_tokens, ready_prefix, ledger)
|
||||||
|
if ready_prefix and estimate is not None and estimate < hard_line:
|
||||||
|
swap = ready_prefix # rung 1: L2 is enough
|
||||||
|
else:
|
||||||
|
swap = list(self._awaiting) # rung 2: force everything, fold
|
||||||
|
fold = True
|
||||||
|
elif base_tokens is not None and base_tokens < self._budget * self._start_ratio:
|
||||||
|
self._announce_held(base_tokens, ledger)
|
||||||
|
return
|
||||||
|
else:
|
||||||
|
swap = ready_prefix
|
||||||
if not swap:
|
if not swap:
|
||||||
return
|
return
|
||||||
|
self._awaiting = self._awaiting[len(swap) :]
|
||||||
|
|
||||||
for entry in swap:
|
for entry in swap:
|
||||||
if entry["chunk"] is not None:
|
if entry["chunk"] is not None:
|
||||||
self._chunks.append(entry["chunk"])
|
self._chunks.append(entry["chunk"])
|
||||||
|
if fold:
|
||||||
|
self._collapse_frozen_region()
|
||||||
|
else:
|
||||||
self._fold_eras()
|
self._fold_eras()
|
||||||
blocks = self._render_frozen_blocks(hard=hard)
|
blocks = self._render_frozen_blocks()
|
||||||
consumed = sum(len(e["turns"]) for e in swap)
|
consumed = sum(len(e["turns"]) for e in swap)
|
||||||
ledger.freeze_turns(consumed, blocks)
|
ledger.freeze_turns(consumed, blocks)
|
||||||
logger.info(
|
logger.info(
|
||||||
f"History compression swap: froze {consumed} turns"
|
f"History compression swap: froze {consumed} turns"
|
||||||
f" ({len(self._chunks)} chunks, {len(self._eras)} eras,"
|
f" ({len(self._chunks)} chunks, {len(self._eras)} eras,"
|
||||||
f" {len(self._awaiting)} still awaiting, hard={hard})."
|
f" {len(self._awaiting)} still awaiting, hard={hard}, folded={fold})."
|
||||||
)
|
)
|
||||||
# Pending chunks contribute only their minimal index to the L3 snapshot.
|
# A folded chunk contributes only its minimal index (its ① fields are
|
||||||
# Include that size in the context estimate, without a compression ratio.
|
# merged into the period paragraph); a pending chunk has no summary
|
||||||
|
# size to report, so its trace carries no compression ratio.
|
||||||
swapped = [e for e in swap if e["chunk"] is not None]
|
swapped = [e for e in swap if e["chunk"] is not None]
|
||||||
source_tokens_total = 0
|
source_tokens_total = 0
|
||||||
replacement_tokens_total = 0
|
replacement_tokens_total = 0
|
||||||
@@ -1227,12 +1294,12 @@ class HistoryChunkManager:
|
|||||||
chunk = entry["chunk"]
|
chunk = entry["chunk"]
|
||||||
if chunk.status == "ready":
|
if chunk.status == "ready":
|
||||||
chunk.summary_chars = len(render_chunk_block(chunk))
|
chunk.summary_chars = len(render_chunk_block(chunk))
|
||||||
replacement_chars = chunk.summary_chars
|
|
||||||
else:
|
else:
|
||||||
chunk.summary_chars = None
|
chunk.summary_chars = None
|
||||||
replacement_chars = len(chunk.minimal_index or "")
|
source_tokens_total += self._chars_to_tokens(chunk.source_chars, ledger)
|
||||||
source_tokens_total += chunk.source_chars // 4
|
replacement_tokens_total += self._chars_to_tokens(
|
||||||
replacement_tokens_total += replacement_chars // 4
|
self._replacement_chars(chunk, folded=fold), ledger
|
||||||
|
)
|
||||||
context_tokens: int | None = None
|
context_tokens: int | None = None
|
||||||
if base_tokens is not None:
|
if base_tokens is not None:
|
||||||
context_tokens = max(
|
context_tokens = max(
|
||||||
@@ -1242,19 +1309,26 @@ class HistoryChunkManager:
|
|||||||
for index, entry in enumerate(swapped):
|
for index, entry in enumerate(swapped):
|
||||||
chunk = entry["chunk"]
|
chunk = entry["chunk"]
|
||||||
turns = len(entry["turns"])
|
turns = len(entry["turns"])
|
||||||
forced = hard and chunk.status != "ready"
|
plural = "s" if turns != 1 else ""
|
||||||
if not forced:
|
forced = fold and chunk.status != "ready"
|
||||||
result = f"Replaced {turns} turn{'s' if turns != 1 else ''} with the summary."
|
if forced:
|
||||||
else:
|
|
||||||
result = (
|
result = (
|
||||||
"Context budget reached before the summary was ready;"
|
"Context budget reached before the summary was ready;"
|
||||||
f" replaced {turns} turn{'s' if turns != 1 else ''} with a brief"
|
f" {turns} turn{plural} folded into the period summary."
|
||||||
" snapshot."
|
|
||||||
)
|
)
|
||||||
|
elif fold:
|
||||||
|
result = (
|
||||||
|
f"Replaced {turns} turn{plural} with the summary, folded into"
|
||||||
|
" the period summary (context budget reached)."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
result = f"Replaced {turns} turn{plural} with the summary."
|
||||||
extra: dict[str, Any] = {
|
extra: dict[str, Any] = {
|
||||||
"source_tokens": chunk.source_chars // 4,
|
"source_tokens": self._chars_to_tokens(chunk.source_chars, ledger),
|
||||||
"summary_tokens": (
|
"summary_tokens": (
|
||||||
chunk.summary_chars // 4 if chunk.summary_chars is not None else None
|
self._chars_to_tokens(chunk.summary_chars, ledger)
|
||||||
|
if chunk.summary_chars is not None
|
||||||
|
else None
|
||||||
),
|
),
|
||||||
}
|
}
|
||||||
# Show the context estimate once per swap, on its last trace.
|
# Show the context estimate once per swap, on its last trace.
|
||||||
@@ -1264,20 +1338,118 @@ class HistoryChunkManager:
|
|||||||
extra["context_estimated"] = True
|
extra["context_estimated"] = True
|
||||||
self._announce(chunk, "success", result=result, forced=forced, extra=extra)
|
self._announce(chunk, "success", result=result, forced=forced, extra=extra)
|
||||||
|
|
||||||
|
def _collapse_frozen_region(self) -> bool:
|
||||||
|
"""Fold the whole frozen region into recall-only eras (hard-threshold
|
||||||
|
L3). Every existing era becomes recall-only and the loose chunks fold
|
||||||
|
into one new recall-only era (a later fold appends another era, so
|
||||||
|
ordinals and step order stay monotonic). Returns whether anything
|
||||||
|
changed — False means the region is already fully folded, so the
|
||||||
|
caller can leave the frozen blocks untouched (byte-stable)."""
|
||||||
|
changed = False
|
||||||
|
for era in self._eras:
|
||||||
|
if not era.recall_only:
|
||||||
|
era.recall_only = True
|
||||||
|
changed = True
|
||||||
|
if self._chunks:
|
||||||
|
self._era_counter += 1
|
||||||
|
self._eras.append(
|
||||||
|
EraState(ordinal=self._era_counter, chunks=self._chunks, recall_only=True)
|
||||||
|
)
|
||||||
|
self._chunks = []
|
||||||
|
changed = True
|
||||||
|
return changed
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _chars_to_tokens(chars: int, ledger: Any = None) -> int:
|
||||||
|
"""Characters → tokens through the ledger's session-calibrated ratio
|
||||||
|
(:meth:`TranscriptLedger.chars_to_tokens`); ``chars // 4`` without a
|
||||||
|
ledger. Identical to ``// 4`` at the default ratio."""
|
||||||
|
convert = getattr(ledger, "chars_to_tokens", None)
|
||||||
|
if callable(convert):
|
||||||
|
try:
|
||||||
|
return int(convert(chars))
|
||||||
|
except (TypeError, ValueError, ZeroDivisionError):
|
||||||
|
pass
|
||||||
|
return max(0, int(chars)) // 4
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _replacement_chars(chunk: ChunkState, *, folded: bool = False) -> int:
|
||||||
|
"""Characters a swapped chunk contributes in place of its turns: the
|
||||||
|
full block when ready and rendered as a chunk block, only the minimal
|
||||||
|
index when still pending or folded into a recall-only era."""
|
||||||
|
if chunk.status == "ready" and not folded:
|
||||||
|
return len(render_chunk_block(chunk))
|
||||||
|
return len(chunk.minimal_index or "")
|
||||||
|
|
||||||
|
def _estimate_context_after(
|
||||||
|
self, base_tokens: int | None, entries: list[dict[str, Any]], ledger: Any = None
|
||||||
|
) -> int | None:
|
||||||
|
"""Estimated prompt size after swapping ``entries`` as L2 blocks."""
|
||||||
|
if base_tokens is None:
|
||||||
|
return None
|
||||||
|
delta = 0
|
||||||
|
for entry in entries:
|
||||||
|
chunk = entry["chunk"]
|
||||||
|
if chunk is None:
|
||||||
|
continue
|
||||||
|
delta += self._chars_to_tokens(
|
||||||
|
self._replacement_chars(chunk), ledger
|
||||||
|
) - self._chars_to_tokens(chunk.source_chars, ledger)
|
||||||
|
return max(0, int(base_tokens) + delta)
|
||||||
|
|
||||||
|
def _announce_held(self, base_tokens: int, ledger: Any = None) -> None:
|
||||||
|
"""Once per chunk: its capsule is ready but the start gate holds the
|
||||||
|
original text live. The timeline line stays ``running`` with a
|
||||||
|
``held`` note so the UI can say the summary is ready and waiting."""
|
||||||
|
threshold = int(self._budget * self._start_ratio)
|
||||||
|
for entry in self._awaiting:
|
||||||
|
chunk = entry["chunk"]
|
||||||
|
if chunk is None or chunk.status != "ready" or entry.get("held_announced"):
|
||||||
|
continue
|
||||||
|
entry["held_announced"] = True
|
||||||
|
self._announce(
|
||||||
|
chunk,
|
||||||
|
"running",
|
||||||
|
phase=COMPRESSION_PHASE_READY,
|
||||||
|
note="held",
|
||||||
|
extra={
|
||||||
|
"summary_tokens": self._chars_to_tokens(len(render_chunk_block(chunk)), ledger),
|
||||||
|
"context_tokens": int(base_tokens),
|
||||||
|
"context_budget": self._budget,
|
||||||
|
"swap_at_tokens": threshold,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
def _partition(self, eligible: list[dict]) -> list[tuple[str | None, list[dict]]]:
|
def _partition(self, eligible: list[dict]) -> list[tuple[str | None, list[dict]]]:
|
||||||
"""Split eligible turns into consecutive same-hash segments."""
|
"""Split eligible turns into consecutive same-hash segments.
|
||||||
|
|
||||||
|
A turn without a stamp (no recorded step — a reply without a tool
|
||||||
|
call, a helper-only turn — or a step not stamped yet) joins the
|
||||||
|
running segment. Leading unstamped turns have no running segment to
|
||||||
|
join: they are held and become the head of the first stamped segment
|
||||||
|
instead of forming a stampless segment of their own, which would
|
||||||
|
close as a chunk with no step records (``_create_chunk`` returns
|
||||||
|
``None`` for it) and sit at the head of the awaiting queue.
|
||||||
|
"""
|
||||||
segments: list[tuple[str | None, list[dict]]] = []
|
segments: list[tuple[str | None, list[dict]]] = []
|
||||||
current_hash: str | None = None
|
current_hash: str | None = None
|
||||||
current: list[dict] = []
|
current: list[dict] = []
|
||||||
|
started = False
|
||||||
for turn in eligible:
|
for turn in eligible:
|
||||||
key = turn.get("step_key")
|
key = turn.get("step_key")
|
||||||
stamped = self._step_hashes.get(str(key)) if key is not None else None
|
stamped = self._step_hashes.get(str(key)) if key is not None else None
|
||||||
|
if stamped is None and not started:
|
||||||
|
current.append(turn) # leading unstamped turns wait for a segment
|
||||||
|
continue
|
||||||
if stamped is None:
|
if stamped is None:
|
||||||
stamped = current_hash # unknown stamps join the running segment
|
stamped = current_hash # unknown stamps join the running segment
|
||||||
if current and stamped == current_hash:
|
if not started:
|
||||||
|
started = True
|
||||||
|
current_hash = stamped
|
||||||
|
current.append(turn)
|
||||||
|
elif stamped == current_hash:
|
||||||
current.append(turn)
|
current.append(turn)
|
||||||
else:
|
else:
|
||||||
if current:
|
|
||||||
segments.append((current_hash, current))
|
segments.append((current_hash, current))
|
||||||
current_hash = stamped
|
current_hash = stamped
|
||||||
current = [turn]
|
current = [turn]
|
||||||
@@ -1290,21 +1462,26 @@ class HistoryChunkManager:
|
|||||||
seg_turns[i : i + self._max_steps] for i in range(0, len(seg_turns), self._max_steps)
|
seg_turns[i : i + self._max_steps] for i in range(0, len(seg_turns), self._max_steps)
|
||||||
]
|
]
|
||||||
|
|
||||||
def _context_base_tokens(self) -> int | None:
|
def _context_base_tokens(self, ledger=None) -> int | None:
|
||||||
|
"""The live context base for the start/soft/hard thresholds.
|
||||||
|
|
||||||
|
An injected ``meter_getter`` wins (tests / custom wiring); otherwise
|
||||||
|
the base is the owning operator's last measured prompt size recorded
|
||||||
|
on the ledger (:meth:`TranscriptLedger.record_prompt_tokens`). The
|
||||||
|
session-wide token meter is deliberately NOT consulted: its
|
||||||
|
``last_prompt_tokens`` is overwritten by whichever agent called last
|
||||||
|
(Planner, Validator, Checker, sub-agents, lenses), so it does not
|
||||||
|
describe the operator's context at all.
|
||||||
|
"""
|
||||||
if self._meter_getter is not None:
|
if self._meter_getter is not None:
|
||||||
try:
|
try:
|
||||||
return self._meter_getter()
|
return self._meter_getter()
|
||||||
except Exception:
|
except Exception:
|
||||||
return None
|
return None
|
||||||
try:
|
try:
|
||||||
from artemis.services.token_meter import get_meter
|
last = getattr(ledger, "last_prompt_tokens", None)
|
||||||
|
return int(last) if last else None
|
||||||
session_id = getattr(self._engine, "current_session_id", None)
|
except (TypeError, ValueError):
|
||||||
if not session_id:
|
|
||||||
return None
|
|
||||||
last = get_meter(session_id).last_prompt_tokens
|
|
||||||
return last or None
|
|
||||||
except Exception:
|
|
||||||
return None
|
return None
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
@@ -1395,7 +1572,11 @@ class HistoryChunkManager:
|
|||||||
chunk.source_chars = int(ledger.turn_text_chars(slice_turns))
|
chunk.source_chars = int(ledger.turn_text_chars(slice_turns))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.debug(f"Chunk source size unavailable for {chunk.capsule_key}: {e}")
|
logger.debug(f"Chunk source size unavailable for {chunk.capsule_key}: {e}")
|
||||||
self._announce(chunk, "running", extra={"source_tokens": chunk.source_chars // 4})
|
self._announce(
|
||||||
|
chunk,
|
||||||
|
"running",
|
||||||
|
extra={"source_tokens": self._chars_to_tokens(chunk.source_chars, ledger)},
|
||||||
|
)
|
||||||
return chunk
|
return chunk
|
||||||
|
|
||||||
def _announce(
|
def _announce(
|
||||||
@@ -1408,19 +1589,29 @@ class HistoryChunkManager:
|
|||||||
note: str | None = None,
|
note: str | None = None,
|
||||||
forced: bool = False,
|
forced: bool = False,
|
||||||
extra: dict[str, Any] | None = None,
|
extra: dict[str, Any] | None = None,
|
||||||
|
phase: str | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Update the chunk's timeline trace without interrupting compression."""
|
"""Update the chunk's timeline trace without interrupting compression.
|
||||||
|
|
||||||
|
``phase`` is the machine-readable plain-language phase written to
|
||||||
|
``args.phase`` (one of ``COMPRESSION_PHASES``). It defaults from the
|
||||||
|
trace ``status`` — running → summarizing, success → applied,
|
||||||
|
failed → failed — and the held path passes ``ready`` explicitly.
|
||||||
|
"""
|
||||||
if chunk.trace_id is None:
|
if chunk.trace_id is None:
|
||||||
return
|
return
|
||||||
engine = self._engine
|
engine = self._engine
|
||||||
if engine is None or not hasattr(engine, "record_trace"):
|
if engine is None or not hasattr(engine, "record_trace"):
|
||||||
return
|
return
|
||||||
|
if phase is None:
|
||||||
|
phase = _STATUS_TO_PHASE.get(status, COMPRESSION_PHASE_SUMMARIZING)
|
||||||
args: dict[str, Any] = {
|
args: dict[str, Any] = {
|
||||||
"start_step": chunk.start_step_number,
|
"start_step": chunk.start_step_number,
|
||||||
"end_step": chunk.end_step_number,
|
"end_step": chunk.end_step_number,
|
||||||
"steps": len(chunk.source_step_ids),
|
"steps": len(chunk.source_step_ids),
|
||||||
"milestone": chunk.milestone_label,
|
"milestone": chunk.milestone_label,
|
||||||
"trigger": chunk.trigger,
|
"trigger": chunk.trigger,
|
||||||
|
"phase": phase,
|
||||||
}
|
}
|
||||||
if note:
|
if note:
|
||||||
args["note"] = note
|
args["note"] = note
|
||||||
@@ -1450,9 +1641,15 @@ class HistoryChunkManager:
|
|||||||
logger.debug(f"Compression announcement skipped for {chunk.capsule_key}: {e}")
|
logger.debug(f"Compression announcement skipped for {chunk.capsule_key}: {e}")
|
||||||
|
|
||||||
def _on_capsule_failed(self, key: JobKey) -> None:
|
def _on_capsule_failed(self, key: JobKey) -> None:
|
||||||
"""Mark the trace failed; keep the chunk pending for a later retry."""
|
"""Mark the trace failed; keep the chunk pending for a later retry.
|
||||||
for chunk in self._all_chunks():
|
|
||||||
if chunk.status != "pending" or chunk.capsule_key != key:
|
Only a chunk still awaiting can be retried (:meth:`_redispatch_failed_capsules`
|
||||||
|
walks the awaiting queue); a pending chunk that was already force-swapped
|
||||||
|
keeps its swap trace, since no retry follows for it.
|
||||||
|
"""
|
||||||
|
for entry in self._awaiting:
|
||||||
|
chunk = entry["chunk"]
|
||||||
|
if chunk is None or chunk.status != "pending" or chunk.capsule_key != key:
|
||||||
continue
|
continue
|
||||||
self._announce(
|
self._announce(
|
||||||
chunk,
|
chunk,
|
||||||
@@ -1584,15 +1781,18 @@ class HistoryChunkManager:
|
|||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
def _fold_eras(self) -> None:
|
def _fold_eras(self) -> None:
|
||||||
|
"""Count overflow: the oldest loose chunks merge into an era; the
|
||||||
|
oldest ledger-bearing eras beyond ``max_eras`` become recall-only
|
||||||
|
(eras already folded by a hard event do not count against the cap)."""
|
||||||
overflow = len(self._chunks) - self._max_chunks
|
overflow = len(self._chunks) - self._max_chunks
|
||||||
if overflow > 0:
|
if overflow > 0:
|
||||||
folded, self._chunks = self._chunks[:overflow], self._chunks[overflow:]
|
folded, self._chunks = self._chunks[:overflow], self._chunks[overflow:]
|
||||||
self._era_counter += 1
|
self._era_counter += 1
|
||||||
self._eras.append(EraState(ordinal=self._era_counter, chunks=folded))
|
self._eras.append(EraState(ordinal=self._era_counter, chunks=folded))
|
||||||
era_overflow = len(self._eras) - self._max_eras
|
with_ledger = [era for era in self._eras if not era.recall_only]
|
||||||
|
era_overflow = len(with_ledger) - self._max_eras
|
||||||
if era_overflow > 0:
|
if era_overflow > 0:
|
||||||
for era in self._eras[:era_overflow]:
|
for era in with_ledger[:era_overflow]:
|
||||||
if not era.recall_only:
|
|
||||||
era.recall_only = True
|
era.recall_only = True
|
||||||
logger.info(
|
logger.info(
|
||||||
f"Era {era.ordinal} overflowed to the recall-only period"
|
f"Era {era.ordinal} overflowed to the recall-only period"
|
||||||
@@ -1600,15 +1800,9 @@ class HistoryChunkManager:
|
|||||||
f"{era.end_step_number})."
|
f"{era.end_step_number})."
|
||||||
)
|
)
|
||||||
|
|
||||||
def _render_frozen_blocks(self, *, hard: bool) -> list[BaseMessage]:
|
def _render_frozen_blocks(self) -> list[BaseMessage]:
|
||||||
if not self._chunks and not self._eras:
|
"""Frozen region, oldest first: era blocks (recall-only or merged),
|
||||||
return []
|
then the loose chunk blocks."""
|
||||||
if hard:
|
|
||||||
plan_text = self._read_plan_text()
|
|
||||||
text = render_l3_snapshot(
|
|
||||||
self._eras, self._chunks, goal=self._goal, plan_text=plan_text
|
|
||||||
)
|
|
||||||
return [HumanMessage(content=[{"type": "text", "text": text}])]
|
|
||||||
blocks: list[BaseMessage] = []
|
blocks: list[BaseMessage] = []
|
||||||
for era in self._eras:
|
for era in self._eras:
|
||||||
blocks.append(HumanMessage(content=[{"type": "text", "text": render_era_block(era)}]))
|
blocks.append(HumanMessage(content=[{"type": "text", "text": render_era_block(era)}]))
|
||||||
@@ -1618,18 +1812,6 @@ class HistoryChunkManager:
|
|||||||
)
|
)
|
||||||
return blocks
|
return blocks
|
||||||
|
|
||||||
def _read_plan_text(self) -> str | None:
|
|
||||||
try:
|
|
||||||
from artemis.utils.notes import get_note_file_path
|
|
||||||
|
|
||||||
base_dir = getattr(self._engine, "base_dir", None)
|
|
||||||
if not base_dir:
|
|
||||||
return None
|
|
||||||
path = get_note_file_path(base_dir, "task_plan")
|
|
||||||
return path.read_text(encoding="utf-8") if path.exists() else None
|
|
||||||
except Exception:
|
|
||||||
return None
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Draining
|
# Draining
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|||||||
@@ -2,14 +2,14 @@
|
|||||||
You are the Segment Capsule writer for a mobile automation agent's history compression. You receive one contiguous segment of executed steps and produce the two LLM-authored bands of the segment's history chunk:
|
You are the Segment Capsule writer for a mobile automation agent's history compression. You receive one contiguous segment of executed steps and produce the two LLM-authored bands of the segment's history chunk:
|
||||||
|
|
||||||
- Band ① — Synopsis & effects: what this segment was doing, what was actually done, and what effects/artifacts it left behind, plus structured fields.
|
- Band ① — Synopsis & effects: what this segment was doing, what was actually done, and what effects/artifacts it left behind, plus structured fields.
|
||||||
- Band ② — Compressed step summary: an interval narrative ("Steps xx–xx did ..., Step xx did ...") that seamlessly covers every step of the segment.
|
- Band ② — Compressed step summary: an interval narrative ("Steps xx–xx did ..., Step xx did ...") covering every step of the segment.
|
||||||
|
|
||||||
The mechanical per-step action ledger (band ③) is assembled outside of you; do NOT reproduce it.
|
The mechanical per-step action ledger (band ③) is assembled outside of you; do NOT reproduce it.
|
||||||
|
|
||||||
# INPUT SHAPE
|
# INPUT SHAPE
|
||||||
The segment arrives turn by turn, in order. Each turn block has:
|
The segment arrives turn by turn, in order. Each turn block has:
|
||||||
- `Recorded steps:` — the mechanical facts of the step(s) executed in that turn: step number, session offset, the exact action, the controller/validator result, the visual transition summary, the notes written (tool → note key: gist), and any user-injected instruction.
|
- `Recorded steps:` — the mechanical facts of the step(s) executed in that turn: step number, session offset, the exact action, the controller/validator result, the visual transition summary, the notes written (tool → note key: gist), and any user-injected instruction.
|
||||||
- `Transcript as seen by the operator during this turn:` — the turn exactly as it stood in the operator's own context: the observation text, the operator's reasoning (including `(thinking)` blocks), every `[tool call]` with its arguments, every `[tool result]` returned to the operator (explorer answers, recalled history, note reads, ADB output, screenshot placeholders), and the `--- Action Execution Result ---` message. Your capsule REPLACES this transcript in the operator's context — everything the operator learned from it that still matters must survive in your output.
|
- `Transcript as seen by the operator during this turn:` — the turn exactly as it stood in the operator's own context: the observation text, the operator's reasoning (including `(thinking)` blocks), every `[tool call]` with its arguments, every `[tool result]` returned to the operator (explorer answers, recalled history, note reads, ADB output, screenshot placeholders), and the `--- Action Execution Result ---` message that closes the turn (every turn on the Pro path; only when an action failed on the Flash path). Your capsule REPLACES this transcript in the operator's context — everything the operator learned from it that still matters must survive in your output.
|
||||||
|
|
||||||
A turn block without a transcript falls back to a reasoning excerpt; treat it the same way with less evidence.
|
A turn block without a transcript falls back to a reasoning excerpt; treat it the same way with less evidence.
|
||||||
|
|
||||||
@@ -45,7 +45,7 @@ Field `intervals`: an ordered array of `{"start_step": N, "end_step": M, "text":
|
|||||||
- **NEVER narrate step by step** ("Step 5: I tapped X. Step 6: I tapped Y."). One interval is a behavior plus the observed outcome, e.g. "Opened the 'Network & internet' section and returned to the main list; the section showed Wi‑Fi, mobile network and VPN rows."
|
- **NEVER narrate step by step** ("Step 5: I tapped X. Step 6: I tapped Y."). One interval is a behavior plus the observed outcome, e.g. "Opened the 'Network & internet' section and returned to the main list; the section showed Wi‑Fi, mobile network and VPN rows."
|
||||||
- **Merge homogeneous consecutive actions into one interval** (repeated scrolls, a sweep through several sections, a polling loop, retries of the same target) — describe the run and its count/result once. A step that does something of a different nature gets its own single-step interval (`start_step == end_step`).
|
- **Merge homogeneous consecutive actions into one interval** (repeated scrolls, a sweep through several sections, a polling loop, retries of the same target) — describe the run and its count/result once. A step that does something of a different nature gets its own single-step interval (`start_step == end_step`).
|
||||||
- Every line's text must reference concrete behavior for those steps (the step numbers come from the `Recorded steps:` lines): the target's visible text, the screen or state observed afterwards, values read, errors shown.
|
- Every line's text must reference concrete behavior for those steps (the step numbers come from the `Recorded steps:` lines): the target's visible text, the screen or state observed afterwards, values read, errors shown.
|
||||||
- **The union of intervals MUST cover the segment's step range seamlessly** — no gaps, no overlaps, in ascending order. This is machine-checked; a gap forces regeneration.
|
- **The union of intervals MUST cover the segment's full step range** — no gaps, no overlaps, in ascending order. This is machine-checked; a gap forces regeneration.
|
||||||
- Keep the neutral-wording contract in every line.
|
- Keep the neutral-wording contract in every line.
|
||||||
|
|
||||||
# LENGTH TARGET (SOFT)
|
# LENGTH TARGET (SOFT)
|
||||||
|
|||||||
@@ -20,12 +20,13 @@ with alias resolution (tool_call_id / legacy ordinals), and status queries.
|
|||||||
|
|
||||||
What a summary *contains* (the lens semantics) lives in subclasses — the
|
What a summary *contains* (the lens semantics) lives in subclasses — the
|
||||||
service only guarantees how jobs are scheduled, retried, keyed, and drained.
|
service only guarantees how jobs are scheduled, retried, keyed, and drained.
|
||||||
Both Flash and (from M2 on) Pro profiles share this runtime.
|
Flash and Pro profiles share this runtime.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
from artemis.data_engine.trace import detached_trace
|
||||||
from artemis.utils.logger import get_logger
|
from artemis.utils.logger import get_logger
|
||||||
|
|
||||||
logger = get_logger(__name__)
|
logger = get_logger(__name__)
|
||||||
@@ -67,12 +68,20 @@ class StepMemoryService:
|
|||||||
where available; tool_call_ids (and legacy ordinals) remain queryable
|
where available; tool_call_ids (and legacy ordinals) remain queryable
|
||||||
through an alias map so message-level consumers need no migration.
|
through an alias map so message-level consumers need no migration.
|
||||||
5. Bounded concurrency: at most ``max_concurrency`` attempts run at once.
|
5. Bounded concurrency: at most ``max_concurrency`` attempts run at once.
|
||||||
|
6. Detached tracing: every job runs inside
|
||||||
|
:func:`~artemis.data_engine.trace.detached_trace`, so a lens's model
|
||||||
|
calls never attach to the trace span (the operator's step) that was
|
||||||
|
current when the job was submitted — no lens needs to know.
|
||||||
|
|
||||||
Subclasses implement :meth:`_attempt` (one summarization attempt for a
|
Subclasses implement :meth:`_attempt` (one summarization attempt for a
|
||||||
payload) and may override :meth:`_on_status` to persist pending/failed
|
payload) and may override :meth:`_on_status` to persist pending/failed
|
||||||
status transitions.
|
status transitions.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
#: Trace node label of a lens-less subclass's jobs (``lens:<name>``); a
|
||||||
|
#: configured :class:`StepLens` contributes its own ``name`` instead.
|
||||||
|
lens_name: str | None = None
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
ctx: Any,
|
ctx: Any,
|
||||||
@@ -131,7 +140,25 @@ class StepMemoryService:
|
|||||||
# Scheduling core
|
# Scheduling core
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
@property
|
||||||
|
def trace_node_name(self) -> str:
|
||||||
|
"""Node label the job's traces and usage records carry (``lens:<name>``)."""
|
||||||
|
name = getattr(self._lens, "name", None) or self.lens_name or type(self).__name__.lower()
|
||||||
|
return f"lens:{name}"
|
||||||
|
|
||||||
async def _run_until_ready(self, key: JobKey) -> None:
|
async def _run_until_ready(self, key: JobKey) -> None:
|
||||||
|
"""Run one job to completion, detached from the submitter's trace span.
|
||||||
|
|
||||||
|
The task was created while the submitting agent's step span was
|
||||||
|
current and inherits that context; the whole job — every attempt,
|
||||||
|
retry sleeps included — runs under :func:`detached_trace` so nothing
|
||||||
|
a lens records (model calls, their thinking/raw_thinking children,
|
||||||
|
stream deltas, usage) can be attributed to the submitter's step.
|
||||||
|
"""
|
||||||
|
with detached_trace(self.trace_node_name):
|
||||||
|
await self._run_attempts(key)
|
||||||
|
|
||||||
|
async def _run_attempts(self, key: JobKey) -> None:
|
||||||
"""Retry one job independently, bounded by the configured retry limit.
|
"""Retry one job independently, bounded by the configured retry limit.
|
||||||
|
|
||||||
Attempts at most ``1 + retry_limit`` times; on exhaustion the job
|
Attempts at most ``1 + retry_limit`` times; on exhaustion the job
|
||||||
|
|||||||
+341
-13
@@ -22,11 +22,16 @@ Messages are organized into four regions:
|
|||||||
- **A (active window)**: raw per-turn messages, append-only. Each committed
|
- **A (active window)**: raw per-turn messages, append-only. Each committed
|
||||||
turn contributes its tail observation HumanMessage, the operator AIMessages
|
turn contributes its tail observation HumanMessage, the operator AIMessages
|
||||||
(tool_calls and native thinking preserved by reference), the in-turn
|
(tool_calls and native thinking preserved by reference), the in-turn
|
||||||
ToolMessages, and the turn's validator result message. The scrub edge
|
ToolMessages, and the turn's result message (every turn on the Pro path;
|
||||||
(:class:`~artemis.agents.flash.context_compressor.ScrubEdgeCompressor`) strips
|
only when an action failed on the Flash path). The
|
||||||
old UI lists and plan recitations at depth 1 and resolves screenshots to
|
scrub edge (:class:`~artemis.agents.flash.context_compressor.ScrubEdgeCompressor`)
|
||||||
visual summaries at depth K — messages are never
|
has two depths: a shallow text edge (UI list, plan recitation and ephemeral
|
||||||
removed or reordered, so tool-call/response pairs are never split.
|
per-turn blocks leave as soon as a newer observation exists) and a
|
||||||
|
screenshot edge that follows the occupancy tier (relaxed below the start
|
||||||
|
gate, tightened past it), where the screenshot is replaced in place by its
|
||||||
|
visual summary. Every rewrite happens once and is never undone; everything
|
||||||
|
newer is byte-identical to what the model was first shown. Messages are
|
||||||
|
never removed or reordered, so tool-call/response pairs are never split.
|
||||||
- **T (current tail)**: built fresh every turn by the operator; passed to
|
- **T (current tail)**: built fresh every turn by the operator; passed to
|
||||||
:meth:`render` and only enters A when the turn is committed.
|
:meth:`render` and only enters A when the turn is committed.
|
||||||
|
|
||||||
@@ -47,7 +52,7 @@ import json
|
|||||||
import time
|
import time
|
||||||
from typing import Any, Callable
|
from typing import Any, Callable
|
||||||
|
|
||||||
from langchain_core.messages import BaseMessage, HumanMessage
|
from langchain_core.messages import AIMessage, BaseMessage, HumanMessage
|
||||||
|
|
||||||
from artemis.memory.step_memory import StepMemoryService
|
from artemis.memory.step_memory import StepMemoryService
|
||||||
from artemis.utils.logger import get_logger
|
from artemis.utils.logger import get_logger
|
||||||
@@ -67,6 +72,28 @@ EXECUTION_RESULT_MARKER = "--- Action Execution Result"
|
|||||||
#: Header prefix of the cold-start restored-history block.
|
#: Header prefix of the cold-start restored-history block.
|
||||||
RESTORED_HISTORY_HEADER = "[Restored history]"
|
RESTORED_HISTORY_HEADER = "[Restored history]"
|
||||||
|
|
||||||
|
#: ``HumanMessage.additional_kwargs`` key listing the indices of content blocks that
|
||||||
|
#: are only meaningful for the turn they were built for (reminders, hints, user
|
||||||
|
#: guidance, per-turn notices). The scrub edge deletes them at its text edge, so
|
||||||
|
#: they never reach the frozen region or a chunk capsule. The indices always
|
||||||
|
#: describe the message's *current* block layout: a rewrite that removes blocks
|
||||||
|
#: before the text edge (the screenshot swap when the image edge is shallower)
|
||||||
|
#: remaps the surviving indices in place, and the key is dropped once the blocks
|
||||||
|
#: are deleted. ``additional_kwargs`` is never sent to the provider, so the
|
||||||
|
#: marking costs the model nothing.
|
||||||
|
EPHEMERAL_BLOCKS_KEY = "ephemeral_blocks"
|
||||||
|
|
||||||
|
|
||||||
|
def mark_ephemeral(message: BaseMessage, indices) -> None:
|
||||||
|
"""Flag content blocks of ``message`` (by index) as ephemeral, see
|
||||||
|
:data:`EPHEMERAL_BLOCKS_KEY`."""
|
||||||
|
existing = list(message.additional_kwargs.get(EPHEMERAL_BLOCKS_KEY, []))
|
||||||
|
for index in indices:
|
||||||
|
index = int(index)
|
||||||
|
if index not in existing:
|
||||||
|
existing.append(index)
|
||||||
|
message.additional_kwargs[EPHEMERAL_BLOCKS_KEY] = existing
|
||||||
|
|
||||||
|
|
||||||
def format_session_offset(seconds: float) -> str:
|
def format_session_offset(seconds: float) -> str:
|
||||||
"""Render a session-start offset as ``T+mm:ss`` (minutes never wrap)."""
|
"""Render a session-start offset as ``T+mm:ss`` (minutes never wrap)."""
|
||||||
@@ -135,6 +162,8 @@ def render_turn_transcript(messages: list[BaseMessage]) -> str:
|
|||||||
names_by_id: dict[str, str] = {}
|
names_by_id: dict[str, str] = {}
|
||||||
for msg in messages:
|
for msg in messages:
|
||||||
kind = str(getattr(msg, "type", "") or "message")
|
kind = str(getattr(msg, "type", "") or "message")
|
||||||
|
if isinstance(msg, AIMessage):
|
||||||
|
kind = "ai" # streamed replies report type "AIMessageChunk"
|
||||||
role = _ROLE_LABELS.get(kind, kind)
|
role = _ROLE_LABELS.get(kind, kind)
|
||||||
if kind == "tool":
|
if kind == "tool":
|
||||||
call_id = getattr(msg, "tool_call_id", None)
|
call_id = getattr(msg, "tool_call_id", None)
|
||||||
@@ -157,6 +186,71 @@ def render_turn_transcript(messages: list[BaseMessage]) -> str:
|
|||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
#: Tokens charged per image block in the provider-less prompt estimate
|
||||||
|
#: (Gemini's flat per-image cost; the Flash runner uses the same figure).
|
||||||
|
IMAGE_BLOCK_TOKENS = 258
|
||||||
|
|
||||||
|
#: Characters per token assumed before any calibrated measurement.
|
||||||
|
DEFAULT_CHARS_PER_TOKEN = 4.0
|
||||||
|
|
||||||
|
#: Bounds of the calibrated chars-per-token ratio: ~1 for CJK-heavy prompts,
|
||||||
|
#: ~4 for English prose; anything outside is a measurement artefact.
|
||||||
|
_CHARS_PER_TOKEN_BOUNDS = (1.0, 6.0)
|
||||||
|
|
||||||
|
#: Smallest text growth (characters) between two consecutive measured prompts
|
||||||
|
#: for their difference to count as a calibration sample (see
|
||||||
|
#: :meth:`TranscriptLedger.record_prompt_tokens`). Below this the provider's
|
||||||
|
#: rounding and a few hundred tokens of turn-to-turn jitter dominate the
|
||||||
|
#: quotient.
|
||||||
|
MIN_CALIBRATION_DELTA_CHARS = 2000
|
||||||
|
|
||||||
|
|
||||||
|
def measure_prompt_content(messages: list[Any]) -> tuple[int, int]:
|
||||||
|
"""Count ``(text_chars, image_count)`` over a prompt's messages.
|
||||||
|
|
||||||
|
The walk covers what the provider tokenizes as text: string content in
|
||||||
|
full; in list content the ``text`` blocks by their text, ``image_url`` /
|
||||||
|
``image`` blocks as one image each, any other block by its ``str()``; and
|
||||||
|
each message's ``tool_calls`` by the JSON of their arguments (an
|
||||||
|
AIMessage's calls are sent back to the model verbatim, and the ledger's
|
||||||
|
turn transcripts size them the same way).
|
||||||
|
"""
|
||||||
|
text_chars = 0
|
||||||
|
image_count = 0
|
||||||
|
for msg in messages:
|
||||||
|
content = getattr(msg, "content", None)
|
||||||
|
if isinstance(content, str):
|
||||||
|
text_chars += len(content)
|
||||||
|
elif isinstance(content, list):
|
||||||
|
for block in content:
|
||||||
|
if isinstance(block, dict):
|
||||||
|
if block.get("type") in ("image_url", "image") or "image_url" in block:
|
||||||
|
image_count += 1
|
||||||
|
elif block.get("type") == "text":
|
||||||
|
text_chars += len(str(block.get("text", "")))
|
||||||
|
else:
|
||||||
|
text_chars += len(str(block))
|
||||||
|
else:
|
||||||
|
text_chars += len(str(block))
|
||||||
|
elif content is not None:
|
||||||
|
text_chars += len(str(content))
|
||||||
|
for call in getattr(msg, "tool_calls", None) or []:
|
||||||
|
if isinstance(call, dict):
|
||||||
|
text_chars += len(
|
||||||
|
json.dumps(call.get("args") or {}, ensure_ascii=False, default=str)
|
||||||
|
)
|
||||||
|
return text_chars, image_count
|
||||||
|
|
||||||
|
|
||||||
|
def estimate_prompt_tokens(messages: list[Any]) -> int:
|
||||||
|
"""Provider-less prompt size estimate: ``text_chars // 4`` plus
|
||||||
|
:data:`IMAGE_BLOCK_TOKENS` per image block (see
|
||||||
|
:func:`measure_prompt_content` for what counts). Used as the context base
|
||||||
|
when a provider reports no usage metadata; never below 1."""
|
||||||
|
text_chars, image_count = measure_prompt_content(messages)
|
||||||
|
return max(1, text_chars // 4 + image_count * IMAGE_BLOCK_TOKENS)
|
||||||
|
|
||||||
|
|
||||||
class TranscriptLedger:
|
class TranscriptLedger:
|
||||||
"""Append-only four-region message ledger for one Pro session.
|
"""Append-only four-region message ledger for one Pro session.
|
||||||
|
|
||||||
@@ -172,7 +266,10 @@ class TranscriptLedger:
|
|||||||
prune_history_xml: bool = True,
|
prune_history_xml: bool = True,
|
||||||
image_scrub_depth: int = 3,
|
image_scrub_depth: int = 3,
|
||||||
pending_grace_steps: int = 3,
|
pending_grace_steps: int = 3,
|
||||||
xml_scrub_depth: int = 1,
|
xml_scrub_depth: int | None = None,
|
||||||
|
image_scrub_depth_relaxed: int | None = None,
|
||||||
|
context_budget_tokens: int | None = None,
|
||||||
|
start_ratio: float | None = None,
|
||||||
clock: Callable[[], float] | None = None,
|
clock: Callable[[], float] | None = None,
|
||||||
session_start: float | None = None,
|
session_start: float | None = None,
|
||||||
):
|
):
|
||||||
@@ -204,6 +301,51 @@ class TranscriptLedger:
|
|||||||
self._frozen_blocks: list[BaseMessage] = []
|
self._frozen_blocks: list[BaseMessage] = []
|
||||||
self._chunker: Any | None = None
|
self._chunker: Any | None = None
|
||||||
|
|
||||||
|
# The owning operator's last measured prompt size (tokens). This is
|
||||||
|
# the live context base the chunker's start/soft/hard thresholds read;
|
||||||
|
# only the operator's own calls write it, so Planner/Checker/lens
|
||||||
|
# calls sharing the session can never masquerade as the operator's
|
||||||
|
# context (see HistoryChunkManager._context_base_tokens).
|
||||||
|
self._last_prompt_tokens: int | None = None
|
||||||
|
# Session-calibrated characters-per-token ratio (see
|
||||||
|
# record_prompt_tokens): the chunker converts transcript characters
|
||||||
|
# to tokens through it, so CJK-heavy sessions (~1 char/token on
|
||||||
|
# Gemini) are not under-estimated 4x by the fixed //4 default.
|
||||||
|
# Calibrated differentially: ``_calibration_sample`` is the previous
|
||||||
|
# measured ``(text_chars, image_count, prompt_tokens)`` and the
|
||||||
|
# pooled deltas between consecutive same-image-count prompts give
|
||||||
|
# the session's single best ratio.
|
||||||
|
self._chars_per_token: float = DEFAULT_CHARS_PER_TOKEN
|
||||||
|
self._calibration_sample: tuple[int, int, int] | None = None
|
||||||
|
self._pooled_delta_chars = 0
|
||||||
|
self._pooled_delta_tokens = 0
|
||||||
|
|
||||||
|
# Whether the last committed turn carried no visible reasoning text (bare
|
||||||
|
# tool calls / thinking only). Set by ``commit_staged``; both runners read
|
||||||
|
# it to decide whether to attach the shared reasoning reminder.
|
||||||
|
self._last_turn_silent: bool = False
|
||||||
|
|
||||||
|
# Text edge (default 1): UI list, plan recitation and ephemeral blocks
|
||||||
|
# leave as soon as a newer observation exists; screenshots follow K.
|
||||||
|
self._xml_scrub_depth = xml_scrub_depth
|
||||||
|
|
||||||
|
# Occupancy-driven screenshot depth. Below the start gate
|
||||||
|
# (``last_prompt_tokens < budget * start_ratio``, or no measurement
|
||||||
|
# yet) the scrub edge sits at the relaxed depth so the model keeps
|
||||||
|
# more raw screenshots while nothing is being replaced; from the
|
||||||
|
# start gate up it tightens to ``image_scrub_depth``. Scrubbing is
|
||||||
|
# irreversible, so a change of depth never rewrites a message:
|
||||||
|
# tightening scrubs the extra depths once, relaxing lets new images
|
||||||
|
# live longer. See :attr:`effective_image_scrub_depth`.
|
||||||
|
self._image_scrub_depth = max(1, int(image_scrub_depth))
|
||||||
|
self._image_scrub_depth_relaxed = (
|
||||||
|
max(self._image_scrub_depth, int(image_scrub_depth_relaxed))
|
||||||
|
if image_scrub_depth_relaxed is not None
|
||||||
|
else self._image_scrub_depth
|
||||||
|
)
|
||||||
|
self._context_budget_tokens = int(context_budget_tokens) if context_budget_tokens else None
|
||||||
|
self._start_ratio = float(start_ratio) if start_ratio is not None else None
|
||||||
|
|
||||||
# id(message) -> summary-job key (DataEngine step id). A side map keeps
|
# id(message) -> summary-job key (DataEngine step id). A side map keeps
|
||||||
# the key out of the serialized message payload entirely.
|
# the key out of the serialized message payload entirely.
|
||||||
self._step_keys: dict[int, str] = {}
|
self._step_keys: dict[int, str] = {}
|
||||||
@@ -235,6 +377,186 @@ class TranscriptLedger:
|
|||||||
"""The current session offset as ``T+mm:ss``."""
|
"""The current session offset as ``T+mm:ss``."""
|
||||||
return format_session_offset(self.elapsed_seconds())
|
return format_session_offset(self.elapsed_seconds())
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Context base (operator-measured prompt size)
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
@property
|
||||||
|
def last_prompt_tokens(self) -> int | None:
|
||||||
|
"""The operator's last measured prompt size, or None before the
|
||||||
|
first measured call (or when the provider reports no usage)."""
|
||||||
|
return self._last_prompt_tokens
|
||||||
|
|
||||||
|
@property
|
||||||
|
def last_turn_silent(self) -> bool:
|
||||||
|
"""True when the last committed turn had no visible reasoning text.
|
||||||
|
|
||||||
|
A turn is silent when it carries at least one AI message and none of
|
||||||
|
them shows text: string content that is empty or whitespace, or list
|
||||||
|
content without a non-empty ``text`` block (native ``thinking`` blocks
|
||||||
|
are not visible text). A turn with no AI message at all is not silent.
|
||||||
|
"""
|
||||||
|
return self._last_turn_silent
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _ai_message_has_visible_text(message: BaseMessage) -> bool:
|
||||||
|
content = getattr(message, "content", None)
|
||||||
|
if isinstance(content, str):
|
||||||
|
return bool(content.strip())
|
||||||
|
if isinstance(content, list):
|
||||||
|
for block in content:
|
||||||
|
if isinstance(block, str):
|
||||||
|
if block.strip():
|
||||||
|
return True
|
||||||
|
elif isinstance(block, dict) and block.get("type") == "text":
|
||||||
|
if str(block.get("text") or "").strip():
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
return False
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def _turn_is_silent(cls, messages: list[BaseMessage]) -> bool:
|
||||||
|
# Streamed replies are ``AIMessageChunk`` (type "AIMessageChunk"), so
|
||||||
|
# the class, not the type string, identifies the model's messages.
|
||||||
|
ai_messages = [m for m in messages if isinstance(m, AIMessage)]
|
||||||
|
if not ai_messages:
|
||||||
|
return False
|
||||||
|
return not any(cls._ai_message_has_visible_text(m) for m in ai_messages)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def chars_per_token(self) -> float:
|
||||||
|
"""Calibrated characters-per-token ratio (4.0 until a measured call
|
||||||
|
with its messages is recorded)."""
|
||||||
|
return self._chars_per_token
|
||||||
|
|
||||||
|
def chars_to_tokens(self, chars: int) -> int:
|
||||||
|
"""Convert transcript characters to tokens through the calibrated
|
||||||
|
ratio (identical to ``chars // 4`` at the default ratio)."""
|
||||||
|
return int(max(0, int(chars)) / self._chars_per_token)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def occupancy(self) -> float | None:
|
||||||
|
"""``last_prompt_tokens / context_budget_tokens``, or None when either
|
||||||
|
is unknown."""
|
||||||
|
if self._last_prompt_tokens is None or not self._context_budget_tokens:
|
||||||
|
return None
|
||||||
|
return self._last_prompt_tokens / self._context_budget_tokens
|
||||||
|
|
||||||
|
@property
|
||||||
|
def below_start_gate(self) -> bool:
|
||||||
|
"""Whether the measured occupancy is under the start gate. An unknown
|
||||||
|
occupancy (no measured call yet) counts as below: a session starts
|
||||||
|
empty."""
|
||||||
|
if self._start_ratio is None:
|
||||||
|
return False
|
||||||
|
ratio = self.occupancy
|
||||||
|
return ratio is None or ratio < self._start_ratio
|
||||||
|
|
||||||
|
@property
|
||||||
|
def effective_image_scrub_depth(self) -> int:
|
||||||
|
"""The screenshot depth the scrub edge uses at the next render: the
|
||||||
|
relaxed depth below the start gate, the tight depth from there up."""
|
||||||
|
if self.below_start_gate:
|
||||||
|
return self._image_scrub_depth_relaxed
|
||||||
|
return self._image_scrub_depth
|
||||||
|
|
||||||
|
def record_prompt_tokens(
|
||||||
|
self, prompt_tokens: int | None, messages: list[Any] | None = None
|
||||||
|
) -> None:
|
||||||
|
"""Record the operator's measured prompt size for one call.
|
||||||
|
|
||||||
|
Called by the ledger's owning operator (Flash runner / Pro operator)
|
||||||
|
after each of its own LLM calls; within a multi-call turn the last
|
||||||
|
(largest) call wins. Non-positive or missing values are ignored so a
|
||||||
|
provider without usage metadata leaves the base unknown rather than
|
||||||
|
zero (and the calibration state below is left untouched).
|
||||||
|
|
||||||
|
When ``messages`` (the exact list sent to the model) accompanies a
|
||||||
|
measured size, the call also calibrates :attr:`chars_per_token`,
|
||||||
|
**differentially**: the ledger remembers the previous measured
|
||||||
|
sample ``(text_chars, image_count, prompt_tokens)`` and, when the new
|
||||||
|
sample carries the same number of images, grows the text by at least
|
||||||
|
:data:`MIN_CALIBRATION_DELTA_CHARS` and costs more tokens, it pools
|
||||||
|
the pair's differences (``pooled_chars += Δchars``,
|
||||||
|
``pooled_tokens += Δtokens``) and sets the ratio to
|
||||||
|
``pooled_chars / pooled_tokens``, clamped to [1.0, 6.0]. The pooled
|
||||||
|
quotient is the maximum-likelihood single ratio for the session (no
|
||||||
|
smoothing constant to tune, no last-pair noise). The reference is
|
||||||
|
replaced only when it was consumed, when the image count changed, or
|
||||||
|
when the prompt shrank in tokens (a compaction or scrub reset the
|
||||||
|
baseline); a sample that moved too little keeps the reference, so
|
||||||
|
several small turns accumulate into one usable difference instead of
|
||||||
|
being discarded one by one (real sessions often grow < 2000 chars
|
||||||
|
per turn and would otherwise never calibrate).
|
||||||
|
|
||||||
|
Why differences: the provider's ``prompt_tokens`` also counts terms
|
||||||
|
that are not in ``messages`` or not proportional to their text — the
|
||||||
|
bound tool/function schemas, and full-size screenshots that tile to
|
||||||
|
~1100–1800 tokens rather than the flat :data:`IMAGE_BLOCK_TOKENS`.
|
||||||
|
A whole-prompt quotient charges all of that to the text, and it
|
||||||
|
also averages over the static prefix, whereas the chunker applies
|
||||||
|
the ratio to the transcript *tail*, whose density differs. On real
|
||||||
|
Gemini Operator traces (10 sessions, 2026-09) the two disagree by
|
||||||
|
5-30% with a session-dependent sign, so the size-based chunk
|
||||||
|
trigger fired early in some sessions and late in others. Between
|
||||||
|
two prompts with the same image count the constant terms cancel and
|
||||||
|
the difference measures the tail's own rate. Until the first
|
||||||
|
usable difference exists the whole-prompt quotient
|
||||||
|
``text_chars / (prompt_tokens - images * IMAGE_BLOCK_TOKENS)`` is
|
||||||
|
used as a fallback; once a pooled difference exists the pooled ratio
|
||||||
|
is authoritative and the fallback is no longer applied.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
value = int(prompt_tokens) if prompt_tokens is not None else 0
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return
|
||||||
|
if value <= 0:
|
||||||
|
return
|
||||||
|
self._last_prompt_tokens = value
|
||||||
|
if messages is None:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
text_chars, image_count = measure_prompt_content(messages)
|
||||||
|
except Exception as exc:
|
||||||
|
logger.debug(f"Prompt calibration skipped: {exc}", exc_info=True)
|
||||||
|
return
|
||||||
|
if text_chars <= 0:
|
||||||
|
return
|
||||||
|
|
||||||
|
low, high = _CHARS_PER_TOKEN_BOUNDS
|
||||||
|
previous = self._calibration_sample
|
||||||
|
sample = (text_chars, image_count, value)
|
||||||
|
|
||||||
|
if previous is None:
|
||||||
|
self._calibration_sample = sample
|
||||||
|
else:
|
||||||
|
prev_chars, prev_images, prev_tokens = previous
|
||||||
|
delta_chars = text_chars - prev_chars
|
||||||
|
delta_tokens = value - prev_tokens
|
||||||
|
if image_count != prev_images or delta_tokens < 0:
|
||||||
|
# Not comparable (image set changed) or the prompt actually
|
||||||
|
# shrank (compaction / scrub): start a fresh baseline here.
|
||||||
|
self._calibration_sample = sample
|
||||||
|
elif delta_chars >= MIN_CALIBRATION_DELTA_CHARS:
|
||||||
|
if delta_tokens > 0:
|
||||||
|
self._pooled_delta_chars += delta_chars
|
||||||
|
self._pooled_delta_tokens += delta_tokens
|
||||||
|
pooled = self._pooled_delta_chars / self._pooled_delta_tokens
|
||||||
|
self._chars_per_token = min(high, max(low, pooled))
|
||||||
|
self._calibration_sample = sample
|
||||||
|
return
|
||||||
|
self._calibration_sample = sample # text grew at no cost: unusable
|
||||||
|
# else: moved too little to measure yet (in either direction) —
|
||||||
|
# keep the reference so the movement accumulates against it.
|
||||||
|
|
||||||
|
if self._pooled_delta_tokens > 0:
|
||||||
|
# A pooled difference already exists: it is authoritative, and
|
||||||
|
# the whole-prompt quotient (which mis-charges schemas and image
|
||||||
|
# tiling to the text) is never applied again.
|
||||||
|
return
|
||||||
|
text_tokens = max(1, value - image_count * IMAGE_BLOCK_TOKENS)
|
||||||
|
self._chars_per_token = min(high, max(low, text_chars / text_tokens))
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# S region
|
# S region
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
@@ -320,6 +642,7 @@ class TranscriptLedger:
|
|||||||
return
|
return
|
||||||
staged = self._staged
|
staged = self._staged
|
||||||
self._staged = None
|
self._staged = None
|
||||||
|
self._last_turn_silent = self._turn_is_silent(staged)
|
||||||
|
|
||||||
if step_key is not None:
|
if step_key is not None:
|
||||||
observation = self._first_image_message(staged)
|
observation = self._first_image_message(staged)
|
||||||
@@ -449,18 +772,23 @@ class TranscriptLedger:
|
|||||||
def render(self, tail: list[BaseMessage]) -> list[BaseMessage]:
|
def render(self, tail: list[BaseMessage]) -> list[BaseMessage]:
|
||||||
"""Advance compression and return ``S + F + A + tail``.
|
"""Advance compression and return ``S + F + A + tail``.
|
||||||
|
|
||||||
Order per turn: the chunker runs first (a triggered compression event
|
Order per turn: the occupancy band is read once (from the operator's
|
||||||
deep-mutates the frozen region and advances the F/A boundary), then
|
last measured prompt), the scrub edge advances over the active window
|
||||||
the scrub edge advances over the active window. The returned list is a
|
at that band's screenshot depth, then the chunker runs (a triggered
|
||||||
fresh container: appending to it (the operator's in-turn tool loop)
|
compression event deep-mutates the frozen region and advances the F/A
|
||||||
never mutates the ledger regions.
|
boundary). Scrub-before-chunk guarantees that the turns a chunk
|
||||||
|
closes over — all older than the sliding-window floor — have already
|
||||||
|
had their screenshots resolved when the relaxed depth is no deeper
|
||||||
|
than the floor plus one. The returned list is a fresh container:
|
||||||
|
appending to it (the operator's in-turn tool loop) never mutates the
|
||||||
|
ledger regions.
|
||||||
"""
|
"""
|
||||||
|
self._compressor.compress(self._active, image_scrub_depth=self.effective_image_scrub_depth)
|
||||||
if self._chunker is not None:
|
if self._chunker is not None:
|
||||||
try:
|
try:
|
||||||
self._chunker.on_render(self)
|
self._chunker.on_render(self)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"History chunker render hook failed: {e}")
|
logger.error(f"History chunker render hook failed: {e}")
|
||||||
self._compressor.compress(self._active)
|
|
||||||
return [
|
return [
|
||||||
*self._static,
|
*self._static,
|
||||||
*self._restored,
|
*self._restored,
|
||||||
|
|||||||
@@ -192,7 +192,7 @@
|
|||||||
// ==========================================================================
|
// ==========================================================================
|
||||||
// 🌐 4.3 Shared Capabilities & Global Overrides
|
// 🌐 4.3 Shared Capabilities & Global Overrides
|
||||||
// ==========================================================================
|
// ==========================================================================
|
||||||
// 🧠 Unified Step-Memory Runtime (history-module redesign v3, M0–M5)
|
// Shared step memory and history compression
|
||||||
// Transcript ledger + layered compression for the Pro operator.
|
// Transcript ledger + layered compression for the Pro operator.
|
||||||
"memory": {
|
"memory": {
|
||||||
// Background summarization runtime (visual/capsule lenses)
|
// Background summarization runtime (visual/capsule lenses)
|
||||||
@@ -204,15 +204,33 @@
|
|||||||
"transcript": {
|
"transcript": {
|
||||||
// ROLLBACK SWITCH: set false to restore the legacy per-turn 2-message
|
// ROLLBACK SWITCH: set false to restore the legacy per-turn 2-message
|
||||||
// prompt path byte-for-byte (L2/L3 chunk compression rides this same
|
// prompt path byte-for-byte (L2/L3 chunk compression rides this same
|
||||||
// flag). Default on since M5 (2026-09-01) per the on-device A/B:
|
// flag). See services/token_meter.py for measured session token usage
|
||||||
// turn latency −77% P50, cache-hit calls 43%→88%, output tokens −74%.
|
// and cache-hit ratios.
|
||||||
"enabled": true,
|
"enabled": true,
|
||||||
"context_budget_tokens": 80000,
|
"context_budget_tokens": 80000,
|
||||||
|
// Below start_ratio of the operator's own measured prompt size, closed
|
||||||
|
// segments only prepare their summary; the original turns are kept in
|
||||||
|
// context and replaced once occupancy reaches this ratio. Measurements
|
||||||
|
// come from the operator; other agents' calls do not affect this gate.
|
||||||
|
"start_ratio": 0.35,
|
||||||
"soft_ratio": 0.7,
|
"soft_ratio": 0.7,
|
||||||
"hard_ratio": 0.9,
|
"hard_ratio": 0.9,
|
||||||
"min_active_steps": 5,
|
"min_active_steps": 5,
|
||||||
|
// Screenshot edge: 6 while the operator's measured context is below
|
||||||
|
// start_ratio (nothing is being replaced yet), 3 from there up. A
|
||||||
|
// screenshot reaching the edge is replaced in place by its visual
|
||||||
|
// summary (the 'Current Screenshot' label leaves with it) and never
|
||||||
|
// restored; the depth only decides how long new ones stay.
|
||||||
"image_scrub_depth": 3,
|
"image_scrub_depth": 3,
|
||||||
|
"image_scrub_depth_relaxed": 6,
|
||||||
"pending_grace_steps": 3,
|
"pending_grace_steps": 3,
|
||||||
|
// Text edge: once a newer observation exists (depth 1) the UI element
|
||||||
|
// list, the plan recitation and the per-turn ephemeral blocks
|
||||||
|
// (reminders, hints, guidance wrappers) leave the older observation.
|
||||||
|
// User instruction bodies remain in history. The text edge stays
|
||||||
|
// shallow on purpose: only the live observation carries an indexed
|
||||||
|
// element list, so a stale [n] index can never be picked up as a
|
||||||
|
// target.
|
||||||
"xml_scrub_depth": 1,
|
"xml_scrub_depth": 1,
|
||||||
// Local historical-state hint (no model call); distance calibrated
|
// Local historical-state hint (no model call); distance calibrated
|
||||||
// 2026-09-01 from 460 dHash-stamped on-device steps (same screen <=4,
|
// 2026-09-01 from 460 dHash-stamped on-device steps (same screen <=4,
|
||||||
@@ -223,6 +241,10 @@
|
|||||||
// Segment (L2) chunk compression: three-band chunks, era folding
|
// Segment (L2) chunk compression: three-band chunks, era folding
|
||||||
"chunking": {
|
"chunking": {
|
||||||
"max_steps": 12,
|
"max_steps": 12,
|
||||||
|
// Size/pressure closes wait until at least this many turns are
|
||||||
|
// eligible, so heavy turns do not become one-step chunks (one capsule
|
||||||
|
// call each); milestone closes are exempt, the hard line waives it.
|
||||||
|
"min_steps": 3,
|
||||||
"target_source_tokens": 2000,
|
"target_source_tokens": 2000,
|
||||||
"model": "gemini-3.8-flash",
|
"model": "gemini-3.8-flash",
|
||||||
"max_chunks": 8
|
"max_chunks": 8
|
||||||
|
|||||||
@@ -33,6 +33,7 @@ from artemis.runtime.daemon_client import (
|
|||||||
ensure_daemon_running,
|
ensure_daemon_running,
|
||||||
get_daemon_session,
|
get_daemon_session,
|
||||||
get_daemon_status,
|
get_daemon_status,
|
||||||
|
is_artemis_daemon,
|
||||||
is_daemon_running,
|
is_daemon_running,
|
||||||
stop_task_on_daemon,
|
stop_task_on_daemon,
|
||||||
submit_batch_to_daemon,
|
submit_batch_to_daemon,
|
||||||
@@ -84,6 +85,7 @@ __all__ = [
|
|||||||
"get_daemon_session",
|
"get_daemon_session",
|
||||||
"get_daemon_status",
|
"get_daemon_status",
|
||||||
"get_server_status",
|
"get_server_status",
|
||||||
|
"is_artemis_daemon",
|
||||||
"is_daemon_running",
|
"is_daemon_running",
|
||||||
"is_cancel_requested",
|
"is_cancel_requested",
|
||||||
"is_port_in_use",
|
"is_port_in_use",
|
||||||
|
|||||||
@@ -17,6 +17,7 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
|
from http.client import HTTPException
|
||||||
import os
|
import os
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
import subprocess
|
import subprocess
|
||||||
@@ -62,6 +63,31 @@ def is_daemon_running(
|
|||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def is_artemis_daemon(
|
||||||
|
host: str = DEFAULT_DAEMON_HOST,
|
||||||
|
port: int = DEFAULT_DAEMON_PORT,
|
||||||
|
timeout: float = 0.5,
|
||||||
|
) -> bool:
|
||||||
|
"""Identity probe: is the process on ``host:port`` the Artemis daemon?
|
||||||
|
|
||||||
|
:func:`is_daemon_running` is a liveness probe that accepts any HTTP
|
||||||
|
listener (a dev server on port 8000 answers ``200`` on ``/``). Diagnostics
|
||||||
|
need to tell the daemon apart from a port squatter, so this asks the
|
||||||
|
daemon's lightweight ``/api/system/server-status`` route and requires the
|
||||||
|
JSON shape only Artemis serves.
|
||||||
|
"""
|
||||||
|
url = f"http://{host}:{port}/api/system/server-status"
|
||||||
|
try:
|
||||||
|
req = urllib.request.Request(url, headers={"User-Agent": "Artemis-Daemon-Probe"})
|
||||||
|
with urllib.request.urlopen(req, timeout=timeout) as response:
|
||||||
|
if response.status != 200:
|
||||||
|
return False
|
||||||
|
payload = json.loads(response.read().decode("utf-8", errors="replace") or "null")
|
||||||
|
except (OSError, ValueError, HTTPException):
|
||||||
|
return False
|
||||||
|
return isinstance(payload, dict) and "current_pid" in payload and "port" in payload
|
||||||
|
|
||||||
|
|
||||||
def daemon_log_path(port: int = DEFAULT_DAEMON_PORT) -> Path:
|
def daemon_log_path(port: int = DEFAULT_DAEMON_PORT) -> Path:
|
||||||
"""Return the log file path used by a background-spawned Daemon on the given port."""
|
"""Return the log file path used by a background-spawned Daemon on the given port."""
|
||||||
from artemis.config.paths import get_app_dir
|
from artemis.config.paths import get_app_dir
|
||||||
|
|||||||
+111
-16
@@ -12,20 +12,26 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
"""Session-level LLM token metering (M0: pure recording, no decisions).
|
"""Session-level LLM token usage and cache-hit metering.
|
||||||
|
|
||||||
Every gateway-managed LLM call reports its real ``usage_metadata`` here. The
|
Every gateway-managed LLM call reports its real ``usage_metadata`` here. The
|
||||||
meter accumulates per-session totals and records one ``llm_usage`` trace per
|
meter accumulates per-session totals and records one ``llm_usage`` trace per
|
||||||
call into the DataEngine, carrying:
|
call into the DataEngine, carrying:
|
||||||
|
|
||||||
- the call's measured ``prompt_tokens`` — the last call's measured input is the
|
- the call's measured ``prompt_tokens`` (``context_base_tokens`` in the trace
|
||||||
best available estimate of the *current context base* for the next call
|
payload, kept for the timeline). Note: the session meter's
|
||||||
(the compaction thresholds of M2/M3 will consume this, never char-count
|
``last_prompt_tokens`` is whichever call happened last in the session —
|
||||||
heuristics);
|
Planner, Validator, Checker, sub-agents included — so the compaction
|
||||||
|
thresholds do NOT read it; the operator records its own prompt size on the
|
||||||
|
transcript ledger (``TranscriptLedger.record_prompt_tokens``);
|
||||||
- provider-reported cache hits (Gemini ``cached_content_token_count`` /
|
- provider-reported cache hits (Gemini ``cached_content_token_count`` /
|
||||||
LangChain ``input_token_details.cache_read``) to establish the cache-hit-rate
|
LangChain ``input_token_details.cache_read``) to establish the cache-hit-rate
|
||||||
baseline;
|
baseline;
|
||||||
- running per-session totals.
|
- running per-session totals, including ``session_cached_ratio``
|
||||||
|
(cached / prompt tokens over the session) and the same ratio for the call's
|
||||||
|
own ``source`` (``source_cached_ratio``), so a session whose operator calls
|
||||||
|
never hit the prompt cache is visible on its ``llm_usage`` traces and in the
|
||||||
|
session summary line (:func:`log_session_summary`).
|
||||||
|
|
||||||
This module never raises into the LLM call path and makes no decisions.
|
This module never raises into the LLM call path and makes no decisions.
|
||||||
"""
|
"""
|
||||||
@@ -71,8 +77,15 @@ def extract_usage(response: Any) -> dict[str, int] | None:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def cached_ratio(cached_tokens: int, prompt_tokens: int) -> float:
|
||||||
|
"""``cached / prompt`` rounded to three decimals; 0.0 with no prompt tokens."""
|
||||||
|
if prompt_tokens <= 0:
|
||||||
|
return 0.0
|
||||||
|
return round(cached_tokens / prompt_tokens, 3)
|
||||||
|
|
||||||
|
|
||||||
class SessionTokenMeter:
|
class SessionTokenMeter:
|
||||||
"""Accumulates measured LLM usage for one session."""
|
"""Accumulates measured LLM usage for one session, in total and per source."""
|
||||||
|
|
||||||
def __init__(self, session_id: str):
|
def __init__(self, session_id: str):
|
||||||
self.session_id = session_id
|
self.session_id = session_id
|
||||||
@@ -83,36 +96,116 @@ class SessionTokenMeter:
|
|||||||
self.cached_tokens = 0
|
self.cached_tokens = 0
|
||||||
self.cache_hit_calls = 0
|
self.cache_hit_calls = 0
|
||||||
self.last_prompt_tokens = 0
|
self.last_prompt_tokens = 0
|
||||||
|
# Per-source totals (source = gateway endpoint key or lens label):
|
||||||
|
# {"llm_calls", "prompt_tokens", "cached_tokens", "cache_hit_calls"}.
|
||||||
|
self.sources: dict[str, dict[str, int]] = {}
|
||||||
|
|
||||||
def record(self, usage: dict[str, int], *, update_last_prompt: bool = True) -> dict[str, int]:
|
def record(
|
||||||
|
self,
|
||||||
|
usage: dict[str, int],
|
||||||
|
*,
|
||||||
|
update_last_prompt: bool = True,
|
||||||
|
source: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
"""Accumulates one call's usage; returns a session snapshot.
|
"""Accumulates one call's usage; returns a session snapshot.
|
||||||
|
|
||||||
``update_last_prompt=False`` accumulates the totals without touching
|
``update_last_prompt=False`` accumulates the totals without touching
|
||||||
``last_prompt_tokens`` — background lens calls carry tiny prompts that
|
``last_prompt_tokens`` — background lens calls carry tiny prompts that
|
||||||
must not masquerade as the live context base consumed by the L2/L3
|
must not masquerade as the live context base consumed by the L2/L3
|
||||||
compaction thresholds.
|
compaction thresholds. With a ``source`` the call also joins that
|
||||||
|
source's totals and the snapshot carries ``source_cached_ratio``.
|
||||||
"""
|
"""
|
||||||
with self._lock:
|
with self._lock:
|
||||||
self.llm_calls += 1
|
prompt = usage.get("prompt_tokens", 0)
|
||||||
self.prompt_tokens += usage.get("prompt_tokens", 0)
|
|
||||||
self.completion_tokens += usage.get("completion_tokens", 0)
|
|
||||||
cached = usage.get("cached_tokens", 0)
|
cached = usage.get("cached_tokens", 0)
|
||||||
|
self.llm_calls += 1
|
||||||
|
self.prompt_tokens += prompt
|
||||||
|
self.completion_tokens += usage.get("completion_tokens", 0)
|
||||||
self.cached_tokens += cached
|
self.cached_tokens += cached
|
||||||
if cached > 0:
|
if cached > 0:
|
||||||
self.cache_hit_calls += 1
|
self.cache_hit_calls += 1
|
||||||
if update_last_prompt:
|
if update_last_prompt:
|
||||||
self.last_prompt_tokens = usage.get("prompt_tokens", 0)
|
self.last_prompt_tokens = prompt
|
||||||
return self.snapshot()
|
snapshot = self._snapshot_locked()
|
||||||
|
if source:
|
||||||
|
totals = self.sources.setdefault(
|
||||||
|
source,
|
||||||
|
{"llm_calls": 0, "prompt_tokens": 0, "cached_tokens": 0, "cache_hit_calls": 0},
|
||||||
|
)
|
||||||
|
totals["llm_calls"] += 1
|
||||||
|
totals["prompt_tokens"] += prompt
|
||||||
|
totals["cached_tokens"] += cached
|
||||||
|
if cached > 0:
|
||||||
|
totals["cache_hit_calls"] += 1
|
||||||
|
snapshot["source_cached_ratio"] = cached_ratio(
|
||||||
|
totals["cached_tokens"], totals["prompt_tokens"]
|
||||||
|
)
|
||||||
|
return snapshot
|
||||||
|
|
||||||
def snapshot(self) -> dict[str, int]:
|
def _snapshot_locked(self) -> dict[str, Any]:
|
||||||
return {
|
return {
|
||||||
"session_llm_calls": self.llm_calls,
|
"session_llm_calls": self.llm_calls,
|
||||||
"session_prompt_tokens": self.prompt_tokens,
|
"session_prompt_tokens": self.prompt_tokens,
|
||||||
"session_completion_tokens": self.completion_tokens,
|
"session_completion_tokens": self.completion_tokens,
|
||||||
"session_cached_tokens": self.cached_tokens,
|
"session_cached_tokens": self.cached_tokens,
|
||||||
"session_cache_hit_calls": self.cache_hit_calls,
|
"session_cache_hit_calls": self.cache_hit_calls,
|
||||||
|
"session_cached_ratio": cached_ratio(self.cached_tokens, self.prompt_tokens),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
def snapshot(self) -> dict[str, Any]:
|
||||||
|
with self._lock:
|
||||||
|
return self._snapshot_locked()
|
||||||
|
|
||||||
|
def summary(self) -> dict[str, Any]:
|
||||||
|
"""Session totals plus every source's totals with its ``cached_ratio``."""
|
||||||
|
with self._lock:
|
||||||
|
summary: dict[str, Any] = {"session_id": self.session_id, **self._snapshot_locked()}
|
||||||
|
summary["sources"] = {
|
||||||
|
source: {
|
||||||
|
**totals,
|
||||||
|
"cached_ratio": cached_ratio(totals["cached_tokens"], totals["prompt_tokens"]),
|
||||||
|
}
|
||||||
|
for source, totals in self.sources.items()
|
||||||
|
}
|
||||||
|
return summary
|
||||||
|
|
||||||
|
|
||||||
|
def format_session_summary(session_id: Any) -> str:
|
||||||
|
"""One log line: session cache ratio, then each source's calls and ratio."""
|
||||||
|
summary = get_meter(session_id).summary()
|
||||||
|
parts = [
|
||||||
|
f"LLM usage for session {summary['session_id']}:"
|
||||||
|
f" {summary['session_llm_calls']} calls,"
|
||||||
|
f" {summary['session_prompt_tokens']} prompt tokens,"
|
||||||
|
f" {summary['session_cached_tokens']} cached"
|
||||||
|
f" (cached_ratio={summary['session_cached_ratio']:.3f},"
|
||||||
|
f" cache-hit calls {summary['session_cache_hit_calls']})"
|
||||||
|
]
|
||||||
|
for source, totals in summary["sources"].items():
|
||||||
|
parts.append(
|
||||||
|
f"{source}: {totals['llm_calls']} calls, cached_ratio={totals['cached_ratio']:.3f}"
|
||||||
|
)
|
||||||
|
return "; ".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def log_session_summary(session_id: Any) -> dict[str, Any] | None:
|
||||||
|
"""Log the session's usage summary (INFO) at session end, best-effort.
|
||||||
|
|
||||||
|
Returns the summary dict, or None when the session never metered a call
|
||||||
|
(nothing is logged then). Never raises.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
key = str(session_id)
|
||||||
|
with _LOCK:
|
||||||
|
meter = _METERS.get(key)
|
||||||
|
if meter is None or meter.llm_calls == 0:
|
||||||
|
return None
|
||||||
|
logger.info(format_session_summary(key))
|
||||||
|
return meter.summary()
|
||||||
|
except Exception as exc:
|
||||||
|
logger.debug(f"Token meter session summary skipped: {exc}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def get_meter(session_id: Any) -> SessionTokenMeter:
|
def get_meter(session_id: Any) -> SessionTokenMeter:
|
||||||
"""Returns (creating on first use) the meter for a session id."""
|
"""Returns (creating on first use) the meter for a session id."""
|
||||||
@@ -147,7 +240,9 @@ def record_llm_usage(
|
|||||||
if not session_id or usage is None:
|
if not session_id or usage is None:
|
||||||
return None
|
return None
|
||||||
|
|
||||||
snapshot = get_meter(session_id).record(usage, update_last_prompt=update_last_prompt)
|
snapshot = get_meter(session_id).record(
|
||||||
|
usage, update_last_prompt=update_last_prompt, source=source
|
||||||
|
)
|
||||||
payload: dict[str, Any] = {
|
payload: dict[str, Any] = {
|
||||||
**usage,
|
**usage,
|
||||||
"context_base_tokens": usage["prompt_tokens"],
|
"context_base_tokens": usage["prompt_tokens"],
|
||||||
|
|||||||
@@ -125,23 +125,52 @@ def _adb_binary() -> str:
|
|||||||
return "adb"
|
return "adb"
|
||||||
|
|
||||||
|
|
||||||
|
#: Inline display cap for command output, in characters. Beyond it the full
|
||||||
|
#: text is cached under a TaskId and only the tail is shown to the model.
|
||||||
|
_LONG_OUTPUT_CHARS = 50000
|
||||||
|
|
||||||
|
#: How many trailing lines of an over-long output stay visible inline.
|
||||||
|
_LONG_OUTPUT_TAIL_LINES = 200
|
||||||
|
|
||||||
|
|
||||||
def _is_output_long(output: str) -> bool:
|
def _is_output_long(output: str) -> bool:
|
||||||
"""Checks if command output exceeds the max threshold for inline display."""
|
"""Whether command output is too long to show inline.
|
||||||
return len(output.splitlines()) > 500 or len(output) > 50000
|
|
||||||
|
The trigger is size alone: more than ``_LONG_OUTPUT_CHARS`` characters.
|
||||||
|
A line count by itself never trips it, so a 1000-line / 4 KB listing
|
||||||
|
(``seq 1 1000``) is shown in full instead of being cut to its tail.
|
||||||
|
"""
|
||||||
|
return len(output) > _LONG_OUTPUT_CHARS
|
||||||
|
|
||||||
|
|
||||||
def _format_long_output_response(_task_id: str, output: str, intro: str) -> str:
|
def _format_long_output_response(task_id: str, output: str, intro: str) -> str:
|
||||||
"""Formats truncated output with guidance to use the analyzer tool."""
|
"""Formats truncated output with guidance to use the analyzer tool."""
|
||||||
lines = output.splitlines()
|
lines = output.splitlines()
|
||||||
last_200 = "\n".join(lines[-200:])
|
tail = "\n".join(lines[-_LONG_OUTPUT_TAIL_LINES:])
|
||||||
return (
|
return (
|
||||||
f"{intro}\nWarning: Output is too long ({len(lines)} lines,"
|
f"{intro}\nWarning: Output is too long ({len(lines)} lines,"
|
||||||
f" {len(output)} chars) and has been truncated.\n--- Last 200 lines of"
|
f" {len(output)} chars) and has been truncated.\n--- Last"
|
||||||
f" output ---\n{last_200}\n------------------------\nYou can use the"
|
f" {_LONG_OUTPUT_TAIL_LINES} lines of output ---\n{tail}\n"
|
||||||
" 'analyze_task_output' tool to get more information."
|
f"------------------------\nThe full output is kept under TaskId"
|
||||||
|
f" {task_id}: use analyze_task_output with TaskId {task_id} to query it."
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _render_output_body(output: str) -> str:
|
||||||
|
"""Renders command output for the model; whitespace-only output reads as empty."""
|
||||||
|
return output if output.strip() else "(empty output)"
|
||||||
|
|
||||||
|
|
||||||
|
def _abbreviate(text: str, limit: int = 40) -> str:
|
||||||
|
"""Cuts ``text`` to ``limit`` characters, marking the cut only when one happened."""
|
||||||
|
return text if len(text) <= limit else f"{text[:limit]}..."
|
||||||
|
|
||||||
|
|
||||||
|
def _terminal_label(terminal_id: str | None) -> str:
|
||||||
|
"""Renders a terminal id for the model; a task without one shows as ``-``."""
|
||||||
|
return terminal_id or "-"
|
||||||
|
|
||||||
|
|
||||||
def _filter_persistent_env(env: dict[str, str]) -> dict[str, str]:
|
def _filter_persistent_env(env: dict[str, str]) -> dict[str, str]:
|
||||||
"""Drops the platform-provided variables from a captured shell environment."""
|
"""Drops the platform-provided variables from a captured shell environment."""
|
||||||
kept: dict[str, str] = {}
|
kept: dict[str, str] = {}
|
||||||
@@ -700,8 +729,7 @@ class RunAdbCommandTool(ArtemisTool):
|
|||||||
intro += "."
|
intro += "."
|
||||||
if run_persistent:
|
if run_persistent:
|
||||||
intro += f" TerminalID: {terminal_id}."
|
intro += f" TerminalID: {terminal_id}."
|
||||||
body = clean_output if clean_output.strip() else "(empty output)"
|
return f"{intro}\nOutput:\n{_render_output_body(clean_output)}"
|
||||||
return f"{intro}\nOutput:\n{body}"
|
|
||||||
|
|
||||||
# We spawn the host subprocess: adb -s <serial> shell <script>.
|
# We spawn the host subprocess: adb -s <serial> shell <script>.
|
||||||
# The script travels as an argument so the remote shell exits with it;
|
# The script travels as an argument so the remote shell exits with it;
|
||||||
@@ -779,7 +807,7 @@ class RunAdbCommandTool(ArtemisTool):
|
|||||||
registry.finished[task_id]["notified"] = True
|
registry.finished[task_id]["notified"] = True
|
||||||
return _format_long_output_response(task_id, clean_output, intro)
|
return _format_long_output_response(task_id, clean_output, intro)
|
||||||
|
|
||||||
return f"{intro}\nOutput:\n{clean_output}"
|
return f"{intro}\nOutput:\n{_render_output_body(clean_output)}"
|
||||||
|
|
||||||
# pylint: disable=too-many-arguments,too-many-positional-arguments
|
# pylint: disable=too-many-arguments,too-many-positional-arguments
|
||||||
async def _hand_off_to_background(
|
async def _hand_off_to_background(
|
||||||
@@ -896,8 +924,8 @@ class ManageTaskTool(ArtemisTool):
|
|||||||
active_tasks = []
|
active_tasks = []
|
||||||
for tid, t in registry.background.items():
|
for tid, t in registry.background.items():
|
||||||
active_tasks.append(
|
active_tasks.append(
|
||||||
f"- {tid}: Command='{t.command[:40]}...', Phone"
|
f"- {tid}: Command='{_abbreviate(t.command)}', Phone"
|
||||||
f" Cwd='{t.cwd}', TerminalID='{t.terminal_id}',"
|
f" Cwd='{t.cwd}', TerminalID={_terminal_label(t.terminal_id)},"
|
||||||
f" Interactive={t.interactive}"
|
f" Interactive={t.interactive}"
|
||||||
)
|
)
|
||||||
if not active_tasks:
|
if not active_tasks:
|
||||||
@@ -908,7 +936,7 @@ class ManageTaskTool(ArtemisTool):
|
|||||||
finished_tasks = []
|
finished_tasks = []
|
||||||
for tid, t in registry.finished.items():
|
for tid, t in registry.finished.items():
|
||||||
finished_tasks.append(
|
finished_tasks.append(
|
||||||
f"- {tid}: Command='{t['command'][:40]}...', Status='{t['status']}'"
|
f"- {tid}: Command='{_abbreviate(t['command'])}', Status='{t['status']}'"
|
||||||
)
|
)
|
||||||
if not finished_tasks:
|
if not finished_tasks:
|
||||||
return "No recently finished tasks."
|
return "No recently finished tasks."
|
||||||
@@ -924,7 +952,7 @@ class ManageTaskTool(ArtemisTool):
|
|||||||
if action == "status":
|
if action == "status":
|
||||||
task_info = _get_task_info(task_id, ctx)
|
task_info = _get_task_info(task_id, ctx)
|
||||||
if not task_info:
|
if not task_info:
|
||||||
return f"Task {task_id} not found."
|
return ToolFailure(f"Error: Task {task_id} not found.")
|
||||||
|
|
||||||
output_text = task_info.get("output", "")
|
output_text = task_info.get("output", "")
|
||||||
if _ENV_START_MARKER in output_text:
|
if _ENV_START_MARKER in output_text:
|
||||||
@@ -935,7 +963,7 @@ class ManageTaskTool(ArtemisTool):
|
|||||||
f"Status: {task_info.get('status')}\n"
|
f"Status: {task_info.get('status')}\n"
|
||||||
f"Command: {task_info.get('command')}\n"
|
f"Command: {task_info.get('command')}\n"
|
||||||
f"Cwd: {task_info.get('cwd')}\n"
|
f"Cwd: {task_info.get('cwd')}\n"
|
||||||
f"TerminalID: {task_info.get('terminal_id')}"
|
f"TerminalID: {_terminal_label(task_info.get('terminal_id'))}"
|
||||||
)
|
)
|
||||||
exit_code = task_info.get("exit_code")
|
exit_code = task_info.get("exit_code")
|
||||||
if exit_code is not None:
|
if exit_code is not None:
|
||||||
@@ -948,16 +976,18 @@ class ManageTaskTool(ArtemisTool):
|
|||||||
|
|
||||||
if action == "kill":
|
if action == "kill":
|
||||||
if not task:
|
if not task:
|
||||||
return f"Task {task_id} is not active or already finished."
|
return ToolFailure(f"Error: Task {task_id} is not active or already finished.")
|
||||||
try:
|
try:
|
||||||
await task.stop(ctx, status="killed")
|
await task.stop(ctx, status="killed")
|
||||||
return f"Task {task_id} successfully terminated."
|
|
||||||
except Exception as e: # pylint: disable=broad-exception-caught
|
except Exception as e: # pylint: disable=broad-exception-caught
|
||||||
return ToolFailure(f"Failed to terminate task {task_id}: {e}")
|
return ToolFailure(f"Failed to terminate task {task_id}: {e}")
|
||||||
|
if task.exit_code is not None:
|
||||||
|
return f"Terminated task {task_id} (exit code {task.exit_code})."
|
||||||
|
return f"Terminated task {task_id}."
|
||||||
|
|
||||||
if action == "send_input":
|
if action == "send_input":
|
||||||
if not task:
|
if not task:
|
||||||
return f"Task {task_id} is not active."
|
return ToolFailure(f"Error: Task {task_id} is not active.")
|
||||||
if not input_str:
|
if not input_str:
|
||||||
return ToolFailure("Error: Input is required for send_input action.")
|
return ToolFailure("Error: Input is required for send_input action.")
|
||||||
if not task.interactive or task.process.stdin is None:
|
if not task.interactive or task.process.stdin is None:
|
||||||
@@ -968,7 +998,7 @@ class ManageTaskTool(ArtemisTool):
|
|||||||
try:
|
try:
|
||||||
task.process.stdin.write(input_str.encode())
|
task.process.stdin.write(input_str.encode())
|
||||||
await task.process.stdin.drain()
|
await task.process.stdin.drain()
|
||||||
return f"Input successfully sent to task {task_id}."
|
return f"Sent input to task {task_id}."
|
||||||
except Exception as e: # pylint: disable=broad-exception-caught
|
except Exception as e: # pylint: disable=broad-exception-caught
|
||||||
return ToolFailure(f"Failed to send input: {e}")
|
return ToolFailure(f"Failed to send input: {e}")
|
||||||
|
|
||||||
|
|||||||
@@ -209,6 +209,9 @@ class RegisteredCandidate:
|
|||||||
bounds: tuple[int, int, int, int] | None = None
|
bounds: tuple[int, int, int, int] | None = None
|
||||||
#: True when the candidate mapped onto an element that was already indexed.
|
#: True when the candidate mapped onto an element that was already indexed.
|
||||||
reused: bool = False
|
reused: bool = False
|
||||||
|
#: The text the agent's own indexed list shows for the reused element, so
|
||||||
|
#: the answer can name it the way the agent already knows it.
|
||||||
|
existing_text: str = ""
|
||||||
|
|
||||||
|
|
||||||
#: A bare-point candidate this close to an existing indexed center (in
|
#: A bare-point candidate this close to an existing indexed center (in
|
||||||
@@ -331,6 +334,7 @@ def register_candidates(
|
|||||||
description=cand.description,
|
description=cand.description,
|
||||||
bounds=_element_pixel_bounds(existing),
|
bounds=_element_pixel_bounds(existing),
|
||||||
reused=True,
|
reused=True,
|
||||||
|
existing_text=str(existing.get("text") or ""),
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
continue
|
continue
|
||||||
@@ -371,8 +375,20 @@ def register_candidates(
|
|||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
def render_text(query: str, outcome: ExplorerOutcome, registered: list[RegisteredCandidate]) -> str:
|
def render_text(
|
||||||
"""Plain-text answer for the calling agent."""
|
query: str,
|
||||||
|
outcome: ExplorerOutcome,
|
||||||
|
registered: list[RegisteredCandidate],
|
||||||
|
*,
|
||||||
|
index_targets: bool = False,
|
||||||
|
) -> str:
|
||||||
|
"""Plain-text answer for the calling agent.
|
||||||
|
|
||||||
|
``index_targets`` selects the targeting syntax the caller's ``click``
|
||||||
|
accepts: the Pro Operator and the Flash / MCP action executor take a bare
|
||||||
|
integer index next to coordinates; the Validator's text entry takes
|
||||||
|
coordinates only.
|
||||||
|
"""
|
||||||
if registered:
|
if registered:
|
||||||
lines = [
|
lines = [
|
||||||
f"Explorer located {len(registered)} candidate(s) for '{query}' in the"
|
f"Explorer located {len(registered)} candidate(s) for '{query}' in the"
|
||||||
@@ -381,12 +397,18 @@ def render_text(query: str, outcome: ExplorerOutcome, registered: list[Registere
|
|||||||
for c in registered:
|
for c in registered:
|
||||||
line = f"- [{c.index}] '{c.description}' at normalized [{c.coords[0]}, {c.coords[1]}]"
|
line = f"- [{c.index}] '{c.description}' at normalized [{c.coords[0]}, {c.coords[1]}]"
|
||||||
if c.reused:
|
if c.reused:
|
||||||
line += " (already in your indexed list)"
|
line += f" (matches your indexed element [{c.index}] '{c.existing_text}')"
|
||||||
lines.append(line)
|
lines.append(line)
|
||||||
|
if index_targets:
|
||||||
lines.append(
|
lines.append(
|
||||||
"Act on a candidate by its index (e.g. target=[index]) or by the"
|
"Act on a candidate by its index (target=3, a bare integer) or by its"
|
||||||
" normalized coordinate. Candidates are ranked by confidence."
|
" normalized coordinate. Candidates are ranked by confidence."
|
||||||
)
|
)
|
||||||
|
else:
|
||||||
|
lines.append(
|
||||||
|
"Act on a candidate by its normalized coordinate (target=[x, y]) with a"
|
||||||
|
" target_description. Candidates are ranked by confidence."
|
||||||
|
)
|
||||||
if outcome.message:
|
if outcome.message:
|
||||||
lines.append(f"Explorer notes: {outcome.message}")
|
lines.append(f"Explorer notes: {outcome.message}")
|
||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
@@ -444,7 +466,7 @@ def render_operator_blocks(
|
|||||||
registered: list[RegisteredCandidate],
|
registered: list[RegisteredCandidate],
|
||||||
) -> str | list[dict[str, Any]]:
|
) -> str | list[dict[str, Any]]:
|
||||||
"""Multimodal answer for the Operator: text plus the annotated screenshot."""
|
"""Multimodal answer for the Operator: text plus the annotated screenshot."""
|
||||||
text = render_text(query, outcome, registered)
|
text = render_text(query, outcome, registered, index_targets=True)
|
||||||
screenshot_path = getattr(state, "latest_screenshot", None)
|
screenshot_path = getattr(state, "latest_screenshot", None)
|
||||||
image_path = (
|
image_path = (
|
||||||
render_annotated_image(ctx, screenshot_path, registered) if screenshot_path else None
|
render_annotated_image(ctx, screenshot_path, registered) if screenshot_path else None
|
||||||
@@ -475,10 +497,11 @@ async def ask_explorer_text(
|
|||||||
*,
|
*,
|
||||||
agent_name: str = "validator",
|
agent_name: str = "validator",
|
||||||
) -> str:
|
) -> str:
|
||||||
"""Text-only entry used by the Validator / Flash runner and MCP executors."""
|
"""Text-only, coordinate-only entry used by the Validator (Flash and the MCP
|
||||||
|
executor reach the explorer through ``McpActionExecutor`` with index targets)."""
|
||||||
outcome = await locate(ctx, state, query, context_feedback, agent_name=agent_name)
|
outcome = await locate(ctx, state, query, context_feedback, agent_name=agent_name)
|
||||||
registered = register_candidates(ctx, state, outcome)
|
registered = register_candidates(ctx, state, outcome)
|
||||||
return render_text(query, outcome, registered)
|
return render_text(query, outcome, registered, index_targets=False)
|
||||||
|
|
||||||
|
|
||||||
class AskExplorerTool(ArtemisTool):
|
class AskExplorerTool(ArtemisTool):
|
||||||
|
|||||||
@@ -15,9 +15,9 @@
|
|||||||
"""replay_steps: full replay of a step range, exactly as the executing agent saw it.
|
"""replay_steps: full replay of a step range, exactly as the executing agent saw it.
|
||||||
|
|
||||||
Each step is rendered with :func:`~artemis.utils.task_tree.render_step_replay`
|
Each step is rendered with :func:`~artemis.utils.task_tree.render_step_replay`
|
||||||
(``[Screen]`` description, reasoning, every non-internal tool call with name /
|
(``T+mm:ss`` step header, ``[Screen]`` description, reasoning, every
|
||||||
arguments / result, the planned action, safety-net interception and the
|
non-internal tool call with name / arguments / result, the executed action,
|
||||||
execution result). Two loose bounds keep one tool result finite (config
|
safety-net interception and the execution result). Two loose bounds keep one tool result finite (config
|
||||||
``agent.memory.replay``): at most ``max_steps`` steps per call, and a
|
``agent.memory.replay``): at most ``max_steps`` steps per call, and a
|
||||||
``max_tokens`` budget applied by dropping *whole trailing steps* — a replayed
|
``max_tokens`` budget applied by dropping *whole trailing steps* — a replayed
|
||||||
step is never cut mid-sentence. Screenshots are deliberately not part of the
|
step is never cut mid-sentence. Screenshots are deliberately not part of the
|
||||||
@@ -79,6 +79,10 @@ def replay_steps_text(
|
|||||||
return ToolFailure(f"Error: no recorded steps in range {start}–{end}.")
|
return ToolFailure(f"Error: no recorded steps in range {start}–{end}.")
|
||||||
|
|
||||||
budget_tokens = max(1, int(max_tokens or DEFAULT_MAX_TOKENS))
|
budget_tokens = max(1, int(max_tokens or DEFAULT_MAX_TOKENS))
|
||||||
|
# Step headers carry the session clock (``T+mm:ss``) every agent prompt
|
||||||
|
# and search_history use; without a session start they fall back to the
|
||||||
|
# stored relative time.
|
||||||
|
session_start = getattr(reader, "session_start_time", None)
|
||||||
rendered: list[str] = []
|
rendered: list[str] = []
|
||||||
shown: list[dict] = []
|
shown: list[dict] = []
|
||||||
used = 0
|
used = 0
|
||||||
@@ -88,7 +92,7 @@ def replay_steps_text(
|
|||||||
dropped += 1
|
dropped += 1
|
||||||
continue
|
continue
|
||||||
try:
|
try:
|
||||||
text = render_step_replay(step)
|
text = render_step_replay(step, session_start=session_start)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
return ToolFailure(f"Error rendering steps {start}–{end}: {e}")
|
return ToolFailure(f"Error rendering steps {start}–{end}: {e}")
|
||||||
cost = _estimate_tokens(text) + 1
|
cost = _estimate_tokens(text) + 1
|
||||||
|
|||||||
@@ -29,6 +29,7 @@ from pathlib import Path
|
|||||||
import sqlite3
|
import sqlite3
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
from artemis.core.tool_failure import ToolFailure
|
||||||
from artemis.utils.logger import get_logger
|
from artemis.utils.logger import get_logger
|
||||||
from artemis.utils.visualization import overlay_action_on_screenshot
|
from artemis.utils.visualization import overlay_action_on_screenshot
|
||||||
|
|
||||||
@@ -47,10 +48,18 @@ class ScreenshotResult:
|
|||||||
image_bytes: bytes | None = None
|
image_bytes: bytes | None = None
|
||||||
#: True when ``which == "overlay"`` and the action was actually drawn.
|
#: True when ``which == "overlay"`` and the action was actually drawn.
|
||||||
overlay_drawn: bool = False
|
overlay_drawn: bool = False
|
||||||
|
#: True when the lookup could not serve the request (bad arguments, no
|
||||||
|
#: history, unknown step, unreadable file). A step that simply has no
|
||||||
|
#: screenshot of the requested kind is an answer, not an error.
|
||||||
|
is_error: bool = False
|
||||||
|
|
||||||
def to_content_blocks(self) -> list[dict[str, Any]] | str:
|
def to_content_blocks(self) -> list[dict[str, Any]] | str:
|
||||||
"""The tool's return value: content blocks with the image, or the
|
"""The tool's return value: content blocks with the image, the
|
||||||
plain-text explanation when there is no image to attach."""
|
plain-text answer when there is no image to attach, or a
|
||||||
|
:class:`ToolFailure` (status ``error`` downstream) when the lookup
|
||||||
|
failed."""
|
||||||
|
if self.is_error:
|
||||||
|
return ToolFailure(self.description)
|
||||||
if not self.image_bytes:
|
if not self.image_bytes:
|
||||||
return self.description
|
return self.description
|
||||||
encoded = base64.b64encode(self.image_bytes).decode("utf-8")
|
encoded = base64.b64encode(self.image_bytes).decode("utf-8")
|
||||||
@@ -60,41 +69,43 @@ class ScreenshotResult:
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _failure(step_number: int, which: str, description: str) -> ScreenshotResult:
|
||||||
|
return ScreenshotResult(step_number, which, description, is_error=True)
|
||||||
|
|
||||||
|
|
||||||
def load_step_screenshot(reader: Any, step_number: Any, which: str = "pre") -> ScreenshotResult:
|
def load_step_screenshot(reader: Any, step_number: Any, which: str = "pre") -> ScreenshotResult:
|
||||||
"""Loads (and for ``overlay`` annotates) one step screenshot from ``reader``."""
|
"""Loads (and for ``overlay`` annotates) one step screenshot from ``reader``."""
|
||||||
which = str(which or "pre").lower()
|
which = str(which or "pre").lower()
|
||||||
try:
|
try:
|
||||||
number = int(step_number)
|
number = int(step_number)
|
||||||
except (TypeError, ValueError):
|
except (TypeError, ValueError):
|
||||||
return ScreenshotResult(
|
return _failure(-1, which, f"Error: step_number must be an integer, got {step_number!r}.")
|
||||||
-1, which, f"Error: step_number must be an integer, got {step_number!r}."
|
|
||||||
)
|
|
||||||
if which not in WHICH_CHOICES:
|
if which not in WHICH_CHOICES:
|
||||||
return ScreenshotResult(
|
return _failure(
|
||||||
number, which, f"Error: 'which' must be 'pre', 'post' or 'overlay', got '{which}'."
|
number, which, f"Error: 'which' must be 'pre', 'post' or 'overlay', got '{which}'."
|
||||||
)
|
)
|
||||||
if reader is None:
|
if reader is None:
|
||||||
return ScreenshotResult(number, which, "Error: no execution history available.")
|
return _failure(number, which, "Error: no execution history available.")
|
||||||
|
|
||||||
source = "pre" if which == "overlay" else which
|
source = "pre" if which == "overlay" else which
|
||||||
try:
|
try:
|
||||||
path = reader.get_step_image_path(number, source)
|
path = reader.get_step_image_path(number, source)
|
||||||
except (sqlite3.Error, OSError, ValueError) as e:
|
except (sqlite3.Error, OSError, ValueError) as e:
|
||||||
return ScreenshotResult(number, which, f"Error looking up step {number}: {e}")
|
return _failure(number, which, f"Error looking up step {number}: {e}")
|
||||||
if path is None:
|
if path is None:
|
||||||
try:
|
try:
|
||||||
record = reader.get_step_record(number)
|
record = reader.get_step_record(number)
|
||||||
except (sqlite3.Error, ValueError):
|
except (sqlite3.Error, ValueError):
|
||||||
record = None
|
record = None
|
||||||
if record is None:
|
if record is None:
|
||||||
return ScreenshotResult(number, which, f"Error: step {number} not found.")
|
return _failure(number, which, f"Error: step {number} not found.")
|
||||||
return ScreenshotResult(
|
return ScreenshotResult(
|
||||||
number, which, f"No {source}-action screenshot recorded for step {number}."
|
number, which, f"No {source}-action screenshot recorded for step {number}."
|
||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
image_bytes = Path(path).read_bytes()
|
image_bytes = Path(path).read_bytes()
|
||||||
except OSError as e:
|
except OSError as e:
|
||||||
return ScreenshotResult(number, which, f"Error reading screenshot: {e}")
|
return _failure(number, which, f"Error reading screenshot: {e}")
|
||||||
|
|
||||||
if which != "overlay":
|
if which != "overlay":
|
||||||
return ScreenshotResult(
|
return ScreenshotResult(
|
||||||
|
|||||||
+165
-14
@@ -30,6 +30,12 @@ Hard boundaries (config ``agent.memory.recall``):
|
|||||||
- every result carries a step number / step id;
|
- every result carries a step number / step id;
|
||||||
- large raw sources return only excerpts around the match plus a reference.
|
- large raw sources return only excerpts around the match plus a reference.
|
||||||
|
|
||||||
|
Two surfaces per step: the *scoring* haystack keeps the raw record (action
|
||||||
|
JSON, execution result JSON, UI-tree XML) so resource ids, bounds and package
|
||||||
|
names stay searchable; the *excerpt* shown as ``Match:`` is built from the
|
||||||
|
readable twin (the ledger action phrase, result status/error words, element
|
||||||
|
text / content descriptions / OCR text) so the model never reads JSON.
|
||||||
|
|
||||||
A ``step_range`` additionally returns the full-width ``build_action_ledger``
|
A ``step_range`` additionally returns the full-width ``build_action_ledger``
|
||||||
rows of that range — the re-entry point for compressed-history marker lines.
|
rows of that range — the re-entry point for compressed-history marker lines.
|
||||||
"""
|
"""
|
||||||
@@ -104,6 +110,15 @@ def result_search_text(result: Any) -> str:
|
|||||||
the haystack via :func:`_screen_text`), so base64 never enters the search."""
|
the haystack via :func:`_screen_text`), so base64 never enters the search."""
|
||||||
if result is None:
|
if result is None:
|
||||||
return ""
|
return ""
|
||||||
|
# A content-block list stored as its JSON string (trace serialization)
|
||||||
|
# is unwrapped first so its text blocks render, not the JSON.
|
||||||
|
if isinstance(result, str) and result.lstrip().startswith("[{"):
|
||||||
|
try:
|
||||||
|
parsed = json.loads(result)
|
||||||
|
except ValueError:
|
||||||
|
parsed = None
|
||||||
|
if isinstance(parsed, list) and any(isinstance(b, dict) and "type" in b for b in parsed):
|
||||||
|
return result_search_text(parsed)
|
||||||
if isinstance(result, list):
|
if isinstance(result, list):
|
||||||
texts = [
|
texts = [
|
||||||
str(block.get("text") or "")
|
str(block.get("text") or "")
|
||||||
@@ -135,23 +150,32 @@ def _events_text(step: dict) -> str:
|
|||||||
return "\n".join(parts)
|
return "\n".join(parts)
|
||||||
|
|
||||||
|
|
||||||
def _action_text(action_taken: Any) -> str:
|
def _rendered_action(action_taken: Any) -> str:
|
||||||
"""Combine the rendered action with raw fields for search.
|
"""The shared ledger phrase of a step's action (``format_actions_clean``)."""
|
||||||
|
|
||||||
Rendered excerpts identify self-described targets. Raw JSON also makes
|
|
||||||
resource IDs, bounds, and other fields searchable.
|
|
||||||
"""
|
|
||||||
if not action_taken:
|
if not action_taken:
|
||||||
return ""
|
return ""
|
||||||
try:
|
try:
|
||||||
rendered = format_actions_clean(action_taken)
|
return format_actions_clean(action_taken)
|
||||||
except Exception:
|
except Exception:
|
||||||
rendered = ""
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def _action_text(action_taken: Any) -> str:
|
||||||
|
"""Combine the rendered action with raw fields for *scoring*.
|
||||||
|
|
||||||
|
Rendered text identifies self-described targets. Raw JSON also makes
|
||||||
|
resource IDs, bounds, and other fields searchable. Never shown as-is:
|
||||||
|
the excerpt comes from :func:`_step_display_text`.
|
||||||
|
"""
|
||||||
|
if not action_taken:
|
||||||
|
return ""
|
||||||
|
rendered = _rendered_action(action_taken)
|
||||||
raw = json.dumps(action_taken, ensure_ascii=False, default=str)
|
raw = json.dumps(action_taken, ensure_ascii=False, default=str)
|
||||||
return f"{rendered}\n{raw}" if rendered else raw
|
return f"{rendered}\n{raw}" if rendered else raw
|
||||||
|
|
||||||
|
|
||||||
def _step_haystack(step: dict) -> str:
|
def _step_haystack(step: dict) -> str:
|
||||||
|
"""The scoring surface of a step: every recorded field, raw JSON included."""
|
||||||
extra = step.get("extra_metadata") or {}
|
extra = step.get("extra_metadata") or {}
|
||||||
fields = [
|
fields = [
|
||||||
str(step.get("summary") or ""),
|
str(step.get("summary") or ""),
|
||||||
@@ -170,6 +194,70 @@ def _step_haystack(step: dict) -> str:
|
|||||||
return "\n".join(f for f in fields if f)
|
return "\n".join(f for f in fields if f)
|
||||||
|
|
||||||
|
|
||||||
|
def _result_display_text(result: Any) -> str:
|
||||||
|
"""The execution result as words: its status and result/error strings
|
||||||
|
(an execution incident renders as its one-line phrase), never JSON."""
|
||||||
|
if not result:
|
||||||
|
return ""
|
||||||
|
if not isinstance(result, dict):
|
||||||
|
return str(result)
|
||||||
|
parts: list[str] = []
|
||||||
|
status = result.get("status")
|
||||||
|
if status:
|
||||||
|
parts.append(f"status {status}")
|
||||||
|
try:
|
||||||
|
from artemis.utils.task_tree import format_result_clean
|
||||||
|
|
||||||
|
incident_or_error = format_result_clean(result)
|
||||||
|
except Exception:
|
||||||
|
incident_or_error = None
|
||||||
|
if incident_or_error:
|
||||||
|
parts.append(str(incident_or_error))
|
||||||
|
else:
|
||||||
|
for key in ("result", "error", "error_msg", "message"):
|
||||||
|
value = result.get(key)
|
||||||
|
if isinstance(value, str) and value.strip():
|
||||||
|
parts.append(value.strip())
|
||||||
|
return " | ".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def _events_display_text(step: dict) -> str:
|
||||||
|
"""Tool calls and reasoning of a step as the replay shows them (name,
|
||||||
|
arguments, result text), without the raw event JSON."""
|
||||||
|
parts: list[str] = []
|
||||||
|
for event in step.get("interleaved_events") or []:
|
||||||
|
etype = event.get("type")
|
||||||
|
if etype == "tool_call":
|
||||||
|
name = str(event.get("name") or "")
|
||||||
|
args = event.get("args") or {}
|
||||||
|
result_text = result_search_text(event.get("result"))
|
||||||
|
try:
|
||||||
|
call = f"{name}({json.dumps(args, ensure_ascii=False, default=str)})"
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
call = name
|
||||||
|
parts.append(f"{call} -> {result_text}" if result_text else call)
|
||||||
|
elif etype and "thought" in etype:
|
||||||
|
parts.append(str(event.get("content") or ""))
|
||||||
|
return "\n".join(p for p in parts if p)
|
||||||
|
|
||||||
|
|
||||||
|
def _step_display_text(step: dict) -> str:
|
||||||
|
"""The *excerpt* surface of a step: the same fields as the haystack, as the
|
||||||
|
agent would read them. The summary is left out — it already heads the
|
||||||
|
result as the ``Screen:`` line."""
|
||||||
|
extra = step.get("extra_metadata") or {}
|
||||||
|
fields = [
|
||||||
|
_rendered_action(step.get("action_taken")),
|
||||||
|
_result_display_text(step.get("last_execution_result")),
|
||||||
|
str(step.get("operator_raw_thinking") or ""),
|
||||||
|
_events_display_text(step),
|
||||||
|
str(extra.get("injected_instruction") or ""),
|
||||||
|
str(extra.get("foreground_app") or ""),
|
||||||
|
]
|
||||||
|
# Excerpts flatten whitespace, so the field boundary must survive as text.
|
||||||
|
return " | ".join(f.strip() for f in fields if f and f.strip())
|
||||||
|
|
||||||
|
|
||||||
def _image_text(reader: Any, image_name: str | None) -> str:
|
def _image_text(reader: Any, image_name: str | None) -> str:
|
||||||
"""Raw XML/OCR text stored for one screenshot — the textual description
|
"""Raw XML/OCR text stored for one screenshot — the textual description
|
||||||
of that image (package/activity strings live inside the XML blob)."""
|
of that image (package/activity strings live inside the XML blob)."""
|
||||||
@@ -212,9 +300,37 @@ def referenced_image_names(step: dict) -> list[str]:
|
|||||||
return names
|
return names
|
||||||
|
|
||||||
|
|
||||||
def _screen_text(reader: Any, step: dict) -> str:
|
#: UI-tree node attributes whose values are human-readable screen content.
|
||||||
"""Description text of the screenshots behind a step: its own pre- and
|
_UI_NODE_TEXT_KEYS = ("text", "content-desc", "content_desc", "resource-id", "resource_id")
|
||||||
post-action screenshots plus any screenshot a tool result embedded."""
|
|
||||||
|
|
||||||
|
def _collect_screen_strings(value: Any, out: list[str], depth: int = 0) -> None:
|
||||||
|
"""Walks a stored UI tree / OCR blob and appends its readable values
|
||||||
|
(element text, content description, resource id, OCR text) in order."""
|
||||||
|
if depth > 50:
|
||||||
|
return
|
||||||
|
if isinstance(value, str):
|
||||||
|
text = " ".join(value.split())
|
||||||
|
if text and text not in out:
|
||||||
|
out.append(text)
|
||||||
|
elif isinstance(value, dict):
|
||||||
|
for key in _UI_NODE_TEXT_KEYS:
|
||||||
|
text = value.get(key)
|
||||||
|
if isinstance(text, str):
|
||||||
|
text = " ".join(text.split())
|
||||||
|
if text and text not in out:
|
||||||
|
out.append(text)
|
||||||
|
for child_key in ("children", "nodes", "elements"):
|
||||||
|
children = value.get(child_key)
|
||||||
|
if isinstance(children, (list, tuple)):
|
||||||
|
for child in children:
|
||||||
|
_collect_screen_strings(child, out, depth + 1)
|
||||||
|
elif isinstance(value, (list, tuple)):
|
||||||
|
for child in value:
|
||||||
|
_collect_screen_strings(child, out, depth + 1)
|
||||||
|
|
||||||
|
|
||||||
|
def _screen_names(step: dict) -> list[str]:
|
||||||
names: list[str] = []
|
names: list[str] = []
|
||||||
for name in (
|
for name in (
|
||||||
step.get("pre_image_name"),
|
step.get("pre_image_name"),
|
||||||
@@ -223,7 +339,32 @@ def _screen_text(reader: Any, step: dict) -> str:
|
|||||||
):
|
):
|
||||||
if name and name not in names:
|
if name and name not in names:
|
||||||
names.append(name)
|
names.append(name)
|
||||||
return "\n".join(t for t in (_image_text(reader, n) for n in names) if t)
|
return names
|
||||||
|
|
||||||
|
|
||||||
|
def _screen_text(reader: Any, step: dict) -> str:
|
||||||
|
"""Description text of the screenshots behind a step (its own pre- and
|
||||||
|
post-action screenshots plus any screenshot a tool result embedded) —
|
||||||
|
the raw XML/OCR blobs, for *scoring* only."""
|
||||||
|
return "\n".join(t for t in (_image_text(reader, n) for n in _screen_names(step)) if t)
|
||||||
|
|
||||||
|
|
||||||
|
def _screen_display_text(reader: Any, step: dict) -> str:
|
||||||
|
"""The readable content of the same screenshots for the *excerpt*: element
|
||||||
|
text / content descriptions / resource ids and OCR text joined by
|
||||||
|
``" | "``, never the UI-tree JSON."""
|
||||||
|
strings: list[str] = []
|
||||||
|
for name in _screen_names(step):
|
||||||
|
try:
|
||||||
|
record = reader.storage.get_image(name)
|
||||||
|
except (sqlite3.Error, ValueError, AttributeError):
|
||||||
|
continue
|
||||||
|
if record is None:
|
||||||
|
continue
|
||||||
|
for blob in (record.ui_tree, record.ocr_result):
|
||||||
|
if blob:
|
||||||
|
_collect_screen_strings(blob, strings)
|
||||||
|
return " | ".join(strings)
|
||||||
|
|
||||||
|
|
||||||
def _step_label(step: dict, session_start: float | None) -> str:
|
def _step_label(step: dict, session_start: float | None) -> str:
|
||||||
@@ -321,12 +462,22 @@ def search_history_text(
|
|||||||
total = score + screen_score
|
total = score + screen_score
|
||||||
if total <= 0:
|
if total <= 0:
|
||||||
continue
|
continue
|
||||||
source = haystack if score >= screen_score else screen
|
|
||||||
lines = [f"[{_step_label(step, session_start)} | id {step.get('step_id')}]"]
|
lines = [f"[{_step_label(step, session_start)} | id {step.get('step_id')}]"]
|
||||||
summary = step.get("summary")
|
summary = step.get("summary")
|
||||||
if summary:
|
if summary:
|
||||||
lines.append(f" Screen: {_clamp(str(summary), EXCERPT_CHARS)}")
|
lines.append(f" Screen: {_clamp(str(summary), EXCERPT_CHARS)}")
|
||||||
lines.append(f" Match: {_excerpt(source, terms, EXCERPT_CHARS)}")
|
# The excerpt comes from the readable twin of whichever surface
|
||||||
|
# scored higher; the raw JSON only ever scores. When the hit sits
|
||||||
|
# in the summary alone, the Screen line already shows it.
|
||||||
|
display = (
|
||||||
|
_step_display_text(step)
|
||||||
|
if score >= screen_score
|
||||||
|
else _screen_display_text(reader, step)
|
||||||
|
)
|
||||||
|
display_hit = _score(display, terms) > 0
|
||||||
|
summary_hit = bool(summary) and _score(str(summary), terms) > 0
|
||||||
|
if display and (display_hit or not summary_hit):
|
||||||
|
lines.append(f" Match: {_excerpt(display, terms, EXCERPT_CHARS)}")
|
||||||
number = step.get("step_number") or 0
|
number = step.get("step_number") or 0
|
||||||
results.append((total, number, "\n".join(lines)))
|
results.append((total, number, "\n".join(lines)))
|
||||||
|
|
||||||
|
|||||||
@@ -29,6 +29,29 @@ from artemis.utils.logger import get_logger
|
|||||||
|
|
||||||
logger = get_logger(__name__)
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
|
#: How many line ranges a merged block's header lists before it is cut.
|
||||||
|
_MAX_HEADER_RANGES = 8
|
||||||
|
|
||||||
|
|
||||||
|
def _compress_line_numbers(indices: list[int]) -> str:
|
||||||
|
"""Renders sorted line numbers as ranges: ``6``, ``69-79``, ``699-798``.
|
||||||
|
|
||||||
|
Consecutive runs collapse into ``start-end``; at most
|
||||||
|
``_MAX_HEADER_RANGES`` ranges are listed, the rest replaced by ``...``.
|
||||||
|
"""
|
||||||
|
runs: list[str] = []
|
||||||
|
start = prev = indices[0]
|
||||||
|
for idx in indices[1:]:
|
||||||
|
if idx == prev + 1:
|
||||||
|
prev = idx
|
||||||
|
continue
|
||||||
|
runs.append(str(start) if start == prev else f"{start}-{prev}")
|
||||||
|
start = prev = idx
|
||||||
|
runs.append(str(start) if start == prev else f"{start}-{prev}")
|
||||||
|
if len(runs) > _MAX_HEADER_RANGES:
|
||||||
|
return ", ".join(runs[:_MAX_HEADER_RANGES]) + ", ..."
|
||||||
|
return ", ".join(runs)
|
||||||
|
|
||||||
|
|
||||||
# pylint: disable=too-many-branches,too-many-locals,too-many-statements
|
# pylint: disable=too-many-branches,too-many-locals,too-many-statements
|
||||||
def search_and_merge_logs(
|
def search_and_merge_logs(
|
||||||
@@ -99,7 +122,7 @@ def search_and_merge_logs(
|
|||||||
truncated = True
|
truncated = True
|
||||||
break
|
break
|
||||||
|
|
||||||
match_str = ", ".join(map(str, match_indices))
|
match_str = _compress_line_numbers(match_indices)
|
||||||
matches.append(f"--- Match at line(s) {match_str} ---")
|
matches.append(f"--- Match at line(s) {match_str} ---")
|
||||||
for idx in range(start, end):
|
for idx in range(start, end):
|
||||||
line_content = log_lines[idx]
|
line_content = log_lines[idx]
|
||||||
|
|||||||
+11
-11
@@ -148,7 +148,7 @@ class SaveNoteTool(ArtemisTool):
|
|||||||
else "."
|
else "."
|
||||||
)
|
)
|
||||||
save_note_content(base_dir, k, c)
|
save_note_content(base_dir, k, c)
|
||||||
agent_outcome = f"Successfully saved note to {k}.md."
|
agent_outcome = f"Saved note '{k}'."
|
||||||
status = "success"
|
status = "success"
|
||||||
except Exception as e: # pylint: disable=broad-exception-caught
|
except Exception as e: # pylint: disable=broad-exception-caught
|
||||||
agent_outcome = f"Failed to save note {k}.md: {e}"
|
agent_outcome = f"Failed to save note {k}.md: {e}"
|
||||||
@@ -227,7 +227,7 @@ class AppendNoteTool(ArtemisTool):
|
|||||||
else "."
|
else "."
|
||||||
)
|
)
|
||||||
append_note_content(base_dir, k, c)
|
append_note_content(base_dir, k, c)
|
||||||
agent_outcome = f"Successfully appended note to {k}.md."
|
agent_outcome = f"Appended to note '{k}'."
|
||||||
status = "success"
|
status = "success"
|
||||||
except Exception as e: # pylint: disable=broad-exception-caught
|
except Exception as e: # pylint: disable=broad-exception-caught
|
||||||
agent_outcome = f"Failed to append note {k}.md: {e}"
|
agent_outcome = f"Failed to append note {k}.md: {e}"
|
||||||
@@ -309,9 +309,9 @@ class UpdateNoteTool(ArtemisTool):
|
|||||||
)
|
)
|
||||||
warning = update_note_content(base_dir, k, t, r)
|
warning = update_note_content(base_dir, k, t, r)
|
||||||
if warning:
|
if warning:
|
||||||
agent_outcome = f"Successfully updated note '{k}'.\nWARNING: {warning}"
|
agent_outcome = f"Updated note '{k}'.\nWARNING: {warning}"
|
||||||
else:
|
else:
|
||||||
agent_outcome = f"Successfully updated note '{k}'."
|
agent_outcome = f"Updated note '{k}'."
|
||||||
status = "success"
|
status = "success"
|
||||||
except Exception as e: # pylint: disable=broad-exception-caught
|
except Exception as e: # pylint: disable=broad-exception-caught
|
||||||
agent_outcome = f"Failed to update note '{k}': {e}"
|
agent_outcome = f"Failed to update note '{k}': {e}"
|
||||||
@@ -642,7 +642,7 @@ class SaveNotePureTool(ArtemisTool):
|
|||||||
else "."
|
else "."
|
||||||
)
|
)
|
||||||
save_note_content(base_dir, k, c)
|
save_note_content(base_dir, k, c)
|
||||||
return f"Successfully saved note to {k}.md."
|
return f"Saved note '{k}'."
|
||||||
except Exception as e: # pylint: disable=broad-exception-caught
|
except Exception as e: # pylint: disable=broad-exception-caught
|
||||||
return ToolFailure(f"Failed to save note {k}.md: {e}")
|
return ToolFailure(f"Failed to save note {k}.md: {e}")
|
||||||
|
|
||||||
@@ -709,8 +709,8 @@ class UpdateNotePureTool(ArtemisTool):
|
|||||||
)
|
)
|
||||||
warning = update_note_content(base_dir, k, t, r)
|
warning = update_note_content(base_dir, k, t, r)
|
||||||
if warning:
|
if warning:
|
||||||
return f"Successfully updated note '{k}'.\nWARNING: {warning}"
|
return f"Updated note '{k}'.\nWARNING: {warning}"
|
||||||
return f"Successfully updated note '{k}'."
|
return f"Updated note '{k}'."
|
||||||
except Exception as e: # pylint: disable=broad-exception-caught
|
except Exception as e: # pylint: disable=broad-exception-caught
|
||||||
return ToolFailure(f"Failed to update note '{k}': {e}")
|
return ToolFailure(f"Failed to update note '{k}': {e}")
|
||||||
|
|
||||||
@@ -774,7 +774,7 @@ class AppendNotePureTool(ArtemisTool):
|
|||||||
else "."
|
else "."
|
||||||
)
|
)
|
||||||
append_note_content(base_dir, k, c)
|
append_note_content(base_dir, k, c)
|
||||||
return f"Successfully appended note to {k}.md."
|
return f"Appended to note '{k}'."
|
||||||
except Exception as e: # pylint: disable=broad-exception-caught
|
except Exception as e: # pylint: disable=broad-exception-caught
|
||||||
return ToolFailure(f"Failed to append note {k}.md: {e}")
|
return ToolFailure(f"Failed to append note {k}.md: {e}")
|
||||||
|
|
||||||
@@ -791,7 +791,7 @@ def get_append_note_tool_pure(ctx: ArtemisContext) -> BaseTool:
|
|||||||
|
|
||||||
save_note_wrapper = ToolWrapper(
|
save_note_wrapper = ToolWrapper(
|
||||||
tool_fn_getter=get_save_note_tool,
|
tool_fn_getter=get_save_note_tool,
|
||||||
on_success_fn=lambda key: f"Successfully saved note '{key}'.",
|
on_success_fn=lambda key: f"Saved note '{key}'.",
|
||||||
on_failure_fn=lambda key: f"Failed to save note '{key}'.",
|
on_failure_fn=lambda key: f"Failed to save note '{key}'.",
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -810,13 +810,13 @@ list_notes_wrapper = ToolWrapper(
|
|||||||
|
|
||||||
update_note_wrapper = ToolWrapper(
|
update_note_wrapper = ToolWrapper(
|
||||||
tool_fn_getter=get_update_note_tool,
|
tool_fn_getter=get_update_note_tool,
|
||||||
on_success_fn=lambda key: f"Successfully updated note '{key}'.",
|
on_success_fn=lambda key: f"Updated note '{key}'.",
|
||||||
on_failure_fn=lambda key: f"Failed to update note '{key}'.",
|
on_failure_fn=lambda key: f"Failed to update note '{key}'.",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
append_note_wrapper = ToolWrapper(
|
append_note_wrapper = ToolWrapper(
|
||||||
tool_fn_getter=get_append_note_tool,
|
tool_fn_getter=get_append_note_tool,
|
||||||
on_success_fn=lambda key: f"Successfully appended note '{key}'.",
|
on_success_fn=lambda key: f"Appended to note '{key}'.",
|
||||||
on_failure_fn=lambda key: f"Failed to append note '{key}'.",
|
on_failure_fn=lambda key: f"Failed to append note '{key}'.",
|
||||||
)
|
)
|
||||||
@@ -63,7 +63,7 @@ class VideoAnalyzerArgs(BaseModel):
|
|||||||
|
|
||||||
OPERATOR_VIDEO_ANALYZER_DOCSTRING = (
|
OPERATOR_VIDEO_ANALYZER_DOCSTRING = (
|
||||||
"[VIDEO] Use this tool to analyze any specific time range of the video"
|
"[VIDEO] Use this tool to analyze any specific time range of the video"
|
||||||
" recording. This tool delegates to a powerful video-analyzing subagent"
|
" recording. This tool delegates to a video analysis subagent"
|
||||||
" with full access to the continuous screen recording of your entire"
|
" with full access to the continuous screen recording of your entire"
|
||||||
" execution session.\n\n"
|
" execution session.\n\n"
|
||||||
"- This tool overcomes your limitation of only receiving discrete"
|
"- This tool overcomes your limitation of only receiving discrete"
|
||||||
|
|||||||
@@ -15,6 +15,8 @@
|
|||||||
"""Utilities for handling app locking and initial app launch logic."""
|
"""Utilities for handling app locking and initial app launch logic."""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
import re
|
||||||
|
|
||||||
from artemis.context import AppLaunchResult, ArtemisContext
|
from artemis.context import AppLaunchResult, ArtemisContext
|
||||||
from artemis.controllers.platform_specific_commands_controller import (
|
from artemis.controllers.platform_specific_commands_controller import (
|
||||||
@@ -28,43 +30,180 @@ from artemis.utils.logger import get_logger
|
|||||||
logger = get_logger(__name__)
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
SYSTEM_OVERLAYS_VALID_FOR_LAUNCH = {
|
# ``ActivityRecord{<hash> u<user> <package>/<activity> t<task id>}`` as printed by
|
||||||
"com.google.android.permissioncontroller",
|
# ``dumpsys activity activities`` on every line that names an activity.
|
||||||
"com.android.permissioncontroller",
|
_ACTIVITY_RECORD_RE = re.compile(
|
||||||
"com.google.android.packageinstaller",
|
r"ActivityRecord\{[0-9a-f]+ u\d+ (?P<package>[\w.]+)/(?P<activity>[\w.$]+) t(?P<task>\d+)\}"
|
||||||
"com.android.packageinstaller",
|
)
|
||||||
}
|
# ``* Task{<hash> #<id> type=... A=<uid>:<affinity> ... visible=true ...}`` task headers
|
||||||
|
# (``* TaskRecord{<hash> #<id> A=<affinity> ...}`` on Android 11 and older).
|
||||||
|
_TASK_HEADER_RE = re.compile(r"^\s*\* Task(?:Record)?\{[0-9a-f]+ #(?P<task>\d+)(?P<attrs>[^}]*)\}")
|
||||||
|
# Device-side pre-filter: the full dump runs to hundreds of KB on a busy device;
|
||||||
|
# only the task headers, history entries and resumed-activity lines are parsed.
|
||||||
|
_FOREGROUND_DUMP_CMD = (
|
||||||
|
"dumpsys activity activities | grep -E"
|
||||||
|
r" '^ *\* (Task|TaskRecord)\{|^ *\* Hist |ResumedActivity'"
|
||||||
|
)
|
||||||
|
# ``* Hist #<index>: ActivityRecord{...}`` entries; index 0 is the task's base activity.
|
||||||
|
_HIST_RE = re.compile(r"^\s*\* Hist\s+#(?P<index>\d+): (?P<record>ActivityRecord\{[^}]*\})")
|
||||||
|
# ``topResumedActivity=ActivityRecord{...}`` (per task, top task first) and the global
|
||||||
|
# ``ResumedActivity: ActivityRecord{...}`` line; the first one seen is the foreground one.
|
||||||
|
_RESUMED_RE = re.compile(r"ResumedActivity[:=]\s*(?P<record>ActivityRecord\{[^}]*\})")
|
||||||
|
|
||||||
|
|
||||||
def get_focused_task_package(ctx: ArtemisContext) -> str | None:
|
@dataclass(frozen=True)
|
||||||
"""Inspect the active/focused task stack to find the package affinity of the top task.
|
class ForegroundTask:
|
||||||
|
"""What ``dumpsys activity activities`` says is on top of the display.
|
||||||
|
|
||||||
Uses adb shell dumpsys activity activities to extract this info.
|
``resumed_package`` is the package of the resumed activity (what the user sees).
|
||||||
|
``base_package`` is the package of the task's root activity (``Hist #0``), and
|
||||||
|
``packages`` holds every package that has an activity in that task, so an app whose
|
||||||
|
task currently shows a helper activity from another package (Settings search,
|
||||||
|
permission dialogs, account pickers, system update pages, ...) is still
|
||||||
|
recognised as the foreground app.
|
||||||
|
"""
|
||||||
|
|
||||||
|
task_id: int | None = None
|
||||||
|
affinity: str | None = None
|
||||||
|
resumed_component: str | None = None
|
||||||
|
resumed_package: str | None = None
|
||||||
|
base_package: str | None = None
|
||||||
|
packages: frozenset[str] = field(default_factory=frozenset)
|
||||||
|
|
||||||
|
def owns(self, app_package: str) -> bool:
|
||||||
|
"""Whether ``app_package`` is the app the foreground task belongs to.
|
||||||
|
|
||||||
|
The task affinity is the app's own claim on the task (by default the
|
||||||
|
package name, or a package-prefixed name such as
|
||||||
|
``com.android.settings.root``); it stays with the task even when the
|
||||||
|
app's root activity has been destroyed and only a companion activity
|
||||||
|
(e.g. the Settings search UI) remains in the history."""
|
||||||
|
affinity = self.affinity or ""
|
||||||
|
return (
|
||||||
|
app_package == self.resumed_package
|
||||||
|
or app_package == self.base_package
|
||||||
|
or app_package in self.packages
|
||||||
|
or affinity == app_package
|
||||||
|
or affinity.startswith(app_package + ".")
|
||||||
|
)
|
||||||
|
|
||||||
|
def describe(self) -> str:
|
||||||
|
task = f"task #{self.task_id}" if self.task_id is not None else "task ?"
|
||||||
|
details = [f"base={self.base_package}", f"resumed={self.resumed_component}"]
|
||||||
|
if self.affinity:
|
||||||
|
details.append(f"affinity={self.affinity}")
|
||||||
|
others = sorted(self.packages - {self.base_package, self.resumed_package, None})
|
||||||
|
if others:
|
||||||
|
details.append(f"also={','.join(others)}")
|
||||||
|
return f"{task} ({', '.join(details)})"
|
||||||
|
|
||||||
|
|
||||||
|
def parse_foreground_task(dump: str) -> ForegroundTask | None:
|
||||||
|
"""Parse ``dumpsys activity activities`` output into the foreground task summary.
|
||||||
|
|
||||||
|
The foreground task is the one holding the resumed activity; when no activity is
|
||||||
|
resumed (mid-transition) the first visible task with history entries is used.
|
||||||
|
Returns None when the dump contains no task information at all.
|
||||||
|
"""
|
||||||
|
task_attrs: dict[int, str] = {}
|
||||||
|
task_order: list[int] = []
|
||||||
|
task_activities: dict[int, list[tuple[int, str, str]]] = {}
|
||||||
|
resumed: re.Match[str] | None = None
|
||||||
|
|
||||||
|
for line in dump.splitlines():
|
||||||
|
header = _TASK_HEADER_RE.match(line)
|
||||||
|
if header:
|
||||||
|
task_id = int(header.group("task"))
|
||||||
|
if task_id not in task_attrs:
|
||||||
|
task_attrs[task_id] = header.group("attrs")
|
||||||
|
task_order.append(task_id)
|
||||||
|
continue
|
||||||
|
hist = _HIST_RE.match(line)
|
||||||
|
if hist:
|
||||||
|
record = _ACTIVITY_RECORD_RE.search(hist.group("record"))
|
||||||
|
if record:
|
||||||
|
task_activities.setdefault(int(record.group("task")), []).append(
|
||||||
|
(int(hist.group("index")), record.group("package"), record.group("activity"))
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
if resumed is None:
|
||||||
|
match = _RESUMED_RE.search(line)
|
||||||
|
if match:
|
||||||
|
resumed = _ACTIVITY_RECORD_RE.search(match.group("record"))
|
||||||
|
|
||||||
|
if not task_attrs and resumed is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
task_id: int | None = None
|
||||||
|
if resumed is not None:
|
||||||
|
task_id = int(resumed.group("task"))
|
||||||
|
else:
|
||||||
|
for candidate in task_order:
|
||||||
|
if "visible=true" in task_attrs[candidate] and candidate in task_activities:
|
||||||
|
task_id = candidate
|
||||||
|
break
|
||||||
|
|
||||||
|
activities = sorted(task_activities.get(task_id, []))
|
||||||
|
affinity_match = re.search(r"\bA=(?:\d+:)?(\S+)", task_attrs.get(task_id, ""))
|
||||||
|
return ForegroundTask(
|
||||||
|
task_id=task_id,
|
||||||
|
affinity=affinity_match.group(1) if affinity_match else None,
|
||||||
|
resumed_component=(
|
||||||
|
f"{resumed.group('package')}/{resumed.group('activity')}" if resumed else None
|
||||||
|
),
|
||||||
|
resumed_package=resumed.group("package") if resumed else None,
|
||||||
|
base_package=activities[0][1] if activities else None,
|
||||||
|
packages=frozenset(package for _, package, _ in activities),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def get_foreground_task(ctx: ArtemisContext) -> ForegroundTask | None:
|
||||||
|
"""Read the foreground task from ``dumpsys activity activities`` on the device.
|
||||||
|
|
||||||
|
Blocking (a synchronous ADB shell round-trip); async callers go through
|
||||||
|
:func:`get_foreground_task_async` so the event loop keeps serving.
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
device = get_adb_device(ctx)
|
device = get_adb_device(ctx)
|
||||||
output = str(device.shell("dumpsys activity activities | grep topDisplayFocusedRootTask"))
|
if device is None:
|
||||||
if "topDisplayFocusedRootTask=" in output:
|
|
||||||
segment = output.split("topDisplayFocusedRootTask=")[-1]
|
|
||||||
if "A=" in segment:
|
|
||||||
affinity = segment.split("A=")[-1].split("}")[0].split()[0].strip()
|
|
||||||
if ":" in affinity:
|
|
||||||
return affinity.split(":")[-1]
|
|
||||||
return affinity
|
|
||||||
if "{" in segment and "}" in segment:
|
|
||||||
braces_content = segment.split("{")[-1].split("}")[0]
|
|
||||||
tokens = braces_content.split()
|
|
||||||
for token in tokens:
|
|
||||||
if "." in token and not token.startswith("#") and not token.isdigit():
|
|
||||||
if ":" in token:
|
|
||||||
return token.split(":")[-1]
|
|
||||||
return token
|
|
||||||
return None
|
return None
|
||||||
|
dump = str(device.shell(_FOREGROUND_DUMP_CMD))
|
||||||
|
if "Task" not in dump:
|
||||||
|
# No grep on the device (or nothing matched): parse the full dump.
|
||||||
|
dump = str(device.shell("dumpsys activity activities"))
|
||||||
|
return parse_foreground_task(dump)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.debug(f"Failed to retrieve focused task package via dumpsys: {e}")
|
logger.debug(f"Failed to retrieve foreground task via dumpsys: {e}")
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def get_foreground_task_async(ctx: ArtemisContext) -> ForegroundTask | None:
|
||||||
|
""":func:`get_foreground_task` off the event loop."""
|
||||||
|
return await asyncio.to_thread(get_foreground_task, ctx)
|
||||||
|
|
||||||
|
|
||||||
|
async def _observe_foreground(
|
||||||
|
ctx: ArtemisContext, app_package: str
|
||||||
|
) -> tuple[bool, str | None, ForegroundTask | None]:
|
||||||
|
"""Report whether ``app_package`` is in the foreground.
|
||||||
|
|
||||||
|
Fast path: the focused window's package equals the target. Otherwise the app also
|
||||||
|
counts as foreground when the top task belongs to it (see ``ForegroundTask``).
|
||||||
|
A ``None`` focused package means the window manager is mid-transition and is
|
||||||
|
reported as "not yet" without consulting the task stack.
|
||||||
|
|
||||||
|
Returns (is_foreground, focused_package, foreground_task); the task is only read
|
||||||
|
when the fast path fails on a non-null focused package.
|
||||||
|
"""
|
||||||
|
current_package = await get_current_foreground_package_async(ctx)
|
||||||
|
if current_package == app_package:
|
||||||
|
return True, current_package, None
|
||||||
|
if current_package is None:
|
||||||
|
return False, None, None
|
||||||
|
task = await get_foreground_task_async(ctx)
|
||||||
|
return task is not None and task.owns(app_package), current_package, task
|
||||||
|
|
||||||
|
|
||||||
async def _poll_for_app_ready(
|
async def _poll_for_app_ready(
|
||||||
ctx: ArtemisContext,
|
ctx: ArtemisContext,
|
||||||
app_package: str,
|
app_package: str,
|
||||||
@@ -88,20 +227,16 @@ async def _poll_for_app_ready(
|
|||||||
polls = int(max_poll_seconds / poll_interval)
|
polls = int(max_poll_seconds / poll_interval)
|
||||||
|
|
||||||
for i in range(polls):
|
for i in range(polls):
|
||||||
current_package = await get_current_foreground_package_async(ctx)
|
ready, current_package, task = await _observe_foreground(ctx, app_package)
|
||||||
|
|
||||||
|
if ready:
|
||||||
if current_package == app_package:
|
if current_package == app_package:
|
||||||
logger.success(f"App {app_package} is ready (took ~{i * poll_interval:.1f}s)")
|
logger.success(f"App {app_package} is ready (took ~{i * poll_interval:.1f}s)")
|
||||||
return True, None
|
else:
|
||||||
|
|
||||||
if current_package in SYSTEM_OVERLAYS_VALID_FOR_LAUNCH:
|
|
||||||
task_package = get_focused_task_package(ctx)
|
|
||||||
if task_package == app_package:
|
|
||||||
logger.success(
|
logger.success(
|
||||||
f"App {app_package} launch succeeded (system overlay"
|
f"App {app_package} is ready (focused window belongs to"
|
||||||
f" '{current_package}' focused, but top task stack affinity"
|
f" '{current_package}', but the foreground {task.describe()} is owned"
|
||||||
f" matches expected app package '{task_package}', took"
|
f" by {app_package}, took ~{i * poll_interval:.1f}s)"
|
||||||
f" ~{i * poll_interval:.1f}s)"
|
|
||||||
)
|
)
|
||||||
return True, None
|
return True, None
|
||||||
|
|
||||||
@@ -110,16 +245,19 @@ async def _poll_for_app_ready(
|
|||||||
else:
|
else:
|
||||||
logger.debug(
|
logger.debug(
|
||||||
f"Poll {i + 1}/{polls}: Wrong app in foreground (expected"
|
f"Poll {i + 1}/{polls}: Wrong app in foreground (expected"
|
||||||
f" '{app_package}', got '{current_package}'). Still waiting..."
|
f" '{app_package}', got '{current_package}', foreground"
|
||||||
|
f" {task.describe() if task else 'task unknown'}). Still waiting..."
|
||||||
)
|
)
|
||||||
|
|
||||||
if i < polls - 1:
|
if i < polls - 1:
|
||||||
await asyncio.sleep(poll_interval)
|
await asyncio.sleep(poll_interval)
|
||||||
|
|
||||||
current_package = await get_current_foreground_package_async(ctx)
|
current_package = await get_current_foreground_package_async(ctx)
|
||||||
|
task = await get_foreground_task_async(ctx)
|
||||||
error_msg = (
|
error_msg = (
|
||||||
f"Timeout waiting for {app_package} to load after {max_poll_seconds}s. "
|
f"Timeout waiting for {app_package} to load after {max_poll_seconds}s. "
|
||||||
f"Current foreground: {current_package}"
|
f"Current foreground: {current_package}; foreground"
|
||||||
|
f" {task.describe() if task else 'task unknown'}"
|
||||||
)
|
)
|
||||||
logger.error(error_msg)
|
logger.error(error_msg)
|
||||||
return False, error_msg
|
return False, error_msg
|
||||||
@@ -225,10 +363,10 @@ async def _handle_initial_app_launch(
|
|||||||
logger.info(f"Starting initial app launch for package: {locked_app_package}")
|
logger.info(f"Starting initial app launch for package: {locked_app_package}")
|
||||||
|
|
||||||
try:
|
try:
|
||||||
current_package = await get_current_foreground_package_async(ctx)
|
already_foreground, current_package, _ = await _observe_foreground(ctx, locked_app_package)
|
||||||
logger.info(f"Current foreground app: {current_package}")
|
logger.info(f"Current foreground app: {current_package}")
|
||||||
|
|
||||||
if current_package == locked_app_package:
|
if already_foreground:
|
||||||
logger.info(f"App {locked_app_package} is already in foreground")
|
logger.info(f"App {locked_app_package} is already in foreground")
|
||||||
return AppLaunchResult(
|
return AppLaunchResult(
|
||||||
locked_app_package=locked_app_package,
|
locked_app_package=locked_app_package,
|
||||||
|
|||||||
@@ -93,8 +93,10 @@ async def validate_api_key(
|
|||||||
async with httpx.AsyncClient(timeout=timeout) as client:
|
async with httpx.AsyncClient(timeout=timeout) as client:
|
||||||
if is_google_family_provider(clean_provider):
|
if is_google_family_provider(clean_provider):
|
||||||
# Test Google Gemini via generativelanguage API
|
# Test Google Gemini via generativelanguage API
|
||||||
url = f"https://generativelanguage.googleapis.com/v1beta/models?key={clean_key}&pageSize=1"
|
# The key travels in a header, never in the URL: httpx logs request
|
||||||
resp = await client.get(url)
|
# URLs at INFO level and those logs end up in files.
|
||||||
|
url = "https://generativelanguage.googleapis.com/v1beta/models?pageSize=1"
|
||||||
|
resp = await client.get(url, headers={"x-goog-api-key": clean_key})
|
||||||
if resp.status_code == 200:
|
if resp.status_code == 200:
|
||||||
return True, "Google Gemini API key verified successfully."
|
return True, "Google Gemini API key verified successfully."
|
||||||
err_msg = _extract_error_message(resp)
|
err_msg = _extract_error_message(resp)
|
||||||
@@ -102,7 +104,7 @@ async def validate_api_key(
|
|||||||
|
|
||||||
elif clean_provider in ("ocr", "vision", "google_vision"):
|
elif clean_provider in ("ocr", "vision", "google_vision"):
|
||||||
# Test Google Cloud Vision OCR API
|
# Test Google Cloud Vision OCR API
|
||||||
url = f"https://vision.googleapis.com/v1/images:annotate?key={clean_key}"
|
url = "https://vision.googleapis.com/v1/images:annotate"
|
||||||
payload = {
|
payload = {
|
||||||
"requests": [
|
"requests": [
|
||||||
{
|
{
|
||||||
@@ -112,7 +114,9 @@ async def validate_api_key(
|
|||||||
]
|
]
|
||||||
}
|
}
|
||||||
resp = await client.post(
|
resp = await client.post(
|
||||||
url, json=payload, headers={"Content-Type": "application/json"}
|
url,
|
||||||
|
json=payload,
|
||||||
|
headers={"Content-Type": "application/json", "x-goog-api-key": clean_key},
|
||||||
)
|
)
|
||||||
if resp.status_code == 200:
|
if resp.status_code == 200:
|
||||||
return True, "Google Cloud Vision OCR API key verified successfully."
|
return True, "Google Cloud Vision OCR API key verified successfully."
|
||||||
|
|||||||
@@ -376,17 +376,18 @@ def format_read_note_success(
|
|||||||
start_line: int | None = None,
|
start_line: int | None = None,
|
||||||
end_line: int | None = None,
|
end_line: int | None = None,
|
||||||
) -> str:
|
) -> str:
|
||||||
"""Formats the success message for reading a note."""
|
"""The read result: a one-line header naming the note and its extent, then the content."""
|
||||||
range_str = ""
|
|
||||||
if start_line is not None or end_line is not None:
|
if start_line is not None or end_line is not None:
|
||||||
range_str = f" (lines {start_line or 1} to {end_line or 'end'})"
|
extent = f"lines {start_line or 1} to {end_line or 'end'}"
|
||||||
return f"Successfully read note '{key}'{range_str}. '{key}' note content:\n{content}"
|
else:
|
||||||
|
extent = f"{len(content.splitlines())} lines"
|
||||||
|
return f"Note '{key}' ({extent}):\n{content}"
|
||||||
|
|
||||||
|
|
||||||
def format_read_note_failure(key: str, error: str) -> str:
|
def format_read_note_failure(key: str, error: str) -> str:
|
||||||
"""Formats the failure message for reading a note."""
|
"""Formats the failure message for reading a note."""
|
||||||
if "not found" in error.lower():
|
if "not found" in error.lower():
|
||||||
return ToolFailure(f"Note '{key}' not found in scratchpad.")
|
return ToolFailure(f"Note '{key}' not found. Use list_notes to see the saved keys.")
|
||||||
return ToolFailure(f"Failed to read note '{key}': {error}")
|
return ToolFailure(f"Failed to read note '{key}': {error}")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -314,18 +314,23 @@ def render_check_line(item: CheckItem, indent: int = 2) -> str:
|
|||||||
return f"{prefix}- {item.kind}{suffix}: {item.text}"
|
return f"{prefix}- {item.kind}{suffix}: {item.text}"
|
||||||
|
|
||||||
|
|
||||||
def restore_missing_check_items(content_before: str, content_after: str) -> str | None:
|
def restore_missing_check_items(
|
||||||
|
content_before: str,
|
||||||
|
content_after: str,
|
||||||
|
items: list[CheckItem] | None = None,
|
||||||
|
) -> str | None:
|
||||||
"""Deterministic guard: merge check lines deleted/rewritten by a plan write
|
"""Deterministic guard: merge check lines deleted/rewritten by a plan write
|
||||||
back into the new content.
|
back into the new content.
|
||||||
|
|
||||||
Returns the merged plan text, or ``None`` when nothing is missing. Lines
|
Returns the merged plan text, or ``None`` when nothing is missing. Lines
|
||||||
whose parent subgoal still exists are re-anchored directly under it; lines
|
whose parent subgoal still exists are re-anchored directly under it; lines
|
||||||
whose parent subgoal was removed are converted to task-level ``@end`` items
|
whose parent subgoal was removed are converted to task-level ``@end`` items
|
||||||
appended at the end of the plan.
|
appended at the end of the plan. ``items`` restricts the merge to the given
|
||||||
|
missing items (the caller may waive some of them).
|
||||||
"""
|
"""
|
||||||
before = parse_plan(content_before)
|
before = parse_plan(content_before)
|
||||||
after = parse_plan(content_after)
|
after = parse_plan(content_after)
|
||||||
missing = missing_check_items(before, after)
|
missing = missing_check_items(before, after) if items is None else list(items)
|
||||||
if not missing:
|
if not missing:
|
||||||
return None
|
return None
|
||||||
|
|
||||||
@@ -438,23 +443,68 @@ Every checklist line MUST match `<indent>- [<status>] <text>`:
|
|||||||
- `{CONTINUOUS_LOOP_TAG}` tags an UNBOUNDED continuous-monitoring milestone: it must stay `[{STATUS_ACTIVE}]` and can never be marked `[{STATUS_DONE}]`, deleted, or untagged by you — the system mechanically rejects such edits. Only an explicit external stop signal injected by the user unlocks its completion; that signal is an external interruption delivered by the system, never something inferred from the screen or the plan.
|
- `{CONTINUOUS_LOOP_TAG}` tags an UNBOUNDED continuous-monitoring milestone: it must stay `[{STATUS_ACTIVE}]` and can never be marked `[{STATUS_DONE}]`, deleted, or untagged by you — the system mechanically rejects such edits. Only an explicit external stop signal injected by the user unlocks its completion; that signal is an external interruption delivered by the system, never something inferred from the screen or the plan.
|
||||||
- The task terminates only when every top-level milestone is `[{STATUS_DONE}]`."""
|
- The task terminates only when every top-level milestone is `[{STATUS_DONE}]`."""
|
||||||
|
|
||||||
_CHECK_GRAMMAR_EXTENSION = """### Check Line Grammar (declared verification standards)
|
_CHECK_GRAMMAR_HEADER = """### Check Line Grammar (declared verification standards)
|
||||||
A top-level milestone may carry indented check lines, and the plan may end with task-level check lines at zero indentation:
|
A top-level milestone may carry indented check lines, and the plan may end with task-level check lines at zero indentation:"""
|
||||||
- ` - verify: <expected state>` — an acceptance criterion for its parent milestone. It is judged by an independent Checker at the moment the milestone is marked completed; a failed verify triggers a repair loop.
|
|
||||||
- ` - assert: <expected observation>` — a test assertion. It is judged by the independent Checker and a failure is recorded verbatim as a legitimate test result; assertions are NEVER repaired, worked around, or satisfied by constructing state.
|
#: Judgment-moment wording of the check lines, keyed by the midway gate.
|
||||||
- `- verify@end: ...` / `- assert@end: ...` — the `@end` suffix defers judgment to task exit, using the final device state. Items without `@end` are judged with the evidence available when their anchored milestone completes.
|
_VERIFY_JUDGED_MIDWAY = (
|
||||||
- Capability boundary (declare honestly, never promise more): all checks are POST-HOC audits over recorded evidence. They detect and record violations after the fact; they CANNOT block an action from happening, so "B must not run before A is confirmed" ordering rules are verified retroactively, not enforced. Transient prompts that were never captured in the execution history cannot be recovered — prefer expressing expectations as persistently probeable state.
|
"It is judged by an independent Checker at the moment the milestone is marked"
|
||||||
- ` - finding: ...` — a SYSTEM-authored standing headline pinned under a milestone whose verify criterion failed. You never author these lines: the system re-renders them on every plan write (deleting one only makes it reappear) and removes them automatically once the verify passes again or its repair budget is exhausted. Each finding names a `checker-...` note holding the full details and repair log — read it with `read_note`; note keys with the `checker-` prefix are reserved for the system and are read-only for you."""
|
" completed; a failed verify reopens the milestone for repair."
|
||||||
|
)
|
||||||
|
_VERIFY_JUDGED_AT_EXIT = (
|
||||||
|
"It is judged once at task exit from the recorded evidence around the"
|
||||||
|
" milestone's completion; there is no midway repair loop."
|
||||||
|
)
|
||||||
|
_ASSERT_JUDGED_MIDWAY = (
|
||||||
|
"It is judged by the independent Checker when its milestone completes (at task exit for `@end`)"
|
||||||
|
)
|
||||||
|
_ASSERT_JUDGED_AT_EXIT = "It is judged once at task exit from the recorded evidence"
|
||||||
|
|
||||||
|
_CHECK_GRAMMAR_COMMON = """- `- verify@end: ...` / `- assert@end: ...` — the `@end` suffix defers judgment to task exit, using the final device state. Items without `@end` are judged from the evidence recorded around their anchored milestone's completion.
|
||||||
|
- Capability boundary (declare honestly, never promise more): all checks are POST-HOC audits over recorded evidence. They detect and record violations after the fact; they CANNOT block an action from happening, so "B must not run before A is confirmed" ordering rules are verified retroactively, not enforced. Transient prompts that were never captured in the execution history cannot be recovered — prefer expressing expectations as persistently probeable state."""
|
||||||
|
|
||||||
|
_FINDING_GRAMMAR = """- ` - finding: ...` — a SYSTEM-authored standing headline pinned under a milestone whose verify criterion failed. You never author these lines: the system re-renders them on every plan write (deleting one only makes it reappear) and removes them automatically once the verify passes again or its repair budget is exhausted. Each finding names a `checker-...` note holding the full details and repair log — read it with `read_note`; note keys with the `checker-` prefix are reserved for the system and are read-only for you."""
|
||||||
|
|
||||||
|
|
||||||
def render_plan_grammar_spec(include_checks: bool) -> str:
|
def _render_check_grammar(*, midway: bool, final: bool) -> str:
|
||||||
|
"""The check-line grammar worded for the active gates.
|
||||||
|
|
||||||
|
Midway on: verify lines are judged at milestone completion and can reopen
|
||||||
|
the milestone; finding lines exist. Midway off (final only): every line is
|
||||||
|
judged once at task exit, and no repair loop or finding line exists, so
|
||||||
|
neither is taught.
|
||||||
|
"""
|
||||||
|
verify_when = _VERIFY_JUDGED_MIDWAY if midway else _VERIFY_JUDGED_AT_EXIT
|
||||||
|
assert_when = _ASSERT_JUDGED_MIDWAY if midway else _ASSERT_JUDGED_AT_EXIT
|
||||||
|
lines = [
|
||||||
|
_CHECK_GRAMMAR_HEADER,
|
||||||
|
"- ` - verify: <expected state>` — an acceptance criterion for its parent"
|
||||||
|
f" milestone. {verify_when}",
|
||||||
|
f"- ` - assert: <expected observation>` — a test assertion. {assert_when} and a"
|
||||||
|
" failure is recorded verbatim as a legitimate test result; assertions are"
|
||||||
|
" NEVER repaired, worked around, or satisfied by constructing state.",
|
||||||
|
_CHECK_GRAMMAR_COMMON,
|
||||||
|
]
|
||||||
|
if midway:
|
||||||
|
lines.append(_FINDING_GRAMMAR)
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def render_plan_grammar_spec(
|
||||||
|
include_checks: bool | None = None, *, midway: bool = False, final: bool = False
|
||||||
|
) -> str:
|
||||||
"""Renders the grammar spec taught to agents.
|
"""Renders the grammar spec taught to agents.
|
||||||
|
|
||||||
``include_checks=False`` (both check gates disabled) returns exactly the
|
``midway`` / ``final`` are the two check gates
|
||||||
base grammar — zero context pollution from the checking feature.
|
(``ExecutionSetup.midway_checks_enabled`` / ``final_check_enabled``); the
|
||||||
|
check-line grammar is worded for whichever is active. Both off returns
|
||||||
|
exactly the base grammar — zero context pollution from the checking
|
||||||
|
feature. ``include_checks`` is a compatibility shim meaning "both gates".
|
||||||
"""
|
"""
|
||||||
if include_checks:
|
if include_checks is not None:
|
||||||
return _PLAN_GRAMMAR_BASE + "\n\n" + _CHECK_GRAMMAR_EXTENSION
|
midway = final = bool(include_checks)
|
||||||
|
if midway or final:
|
||||||
|
return _PLAN_GRAMMAR_BASE + "\n\n" + _render_check_grammar(midway=midway, final=final)
|
||||||
return _PLAN_GRAMMAR_BASE
|
return _PLAN_GRAMMAR_BASE
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+134
-24
@@ -119,8 +119,6 @@ def format_action_clean(action_obj) -> str:
|
|||||||
direction = action_obj.get("intent")
|
direction = action_obj.get("intent")
|
||||||
duration = action_obj.get("duration") or action_obj.get("duration_ms")
|
duration = action_obj.get("duration") or action_obj.get("duration_ms")
|
||||||
duration_str = f" over {duration}ms" if duration else ""
|
duration_str = f" over {duration}ms" if duration else ""
|
||||||
if direction:
|
|
||||||
return f"Swiped {direction}{duration_str}"
|
|
||||||
|
|
||||||
start_coords = action_obj.get("start_coordinates")
|
start_coords = action_obj.get("start_coordinates")
|
||||||
end_coords = action_obj.get("end_coordinates")
|
end_coords = action_obj.get("end_coordinates")
|
||||||
@@ -129,6 +127,14 @@ def format_action_clean(action_obj) -> str:
|
|||||||
if isinstance(coords_list, list) and len(coords_list) == 4:
|
if isinstance(coords_list, list) and len(coords_list) == 4:
|
||||||
start_coords = coords_list[:2]
|
start_coords = coords_list[:2]
|
||||||
end_coords = coords_list[2:]
|
end_coords = coords_list[2:]
|
||||||
|
if not start_coords or not end_coords:
|
||||||
|
start_coords = action_obj.get("normalized_start_coordinates")
|
||||||
|
end_coords = action_obj.get("normalized_end_coordinates")
|
||||||
|
if direction:
|
||||||
|
# A directional swipe is what the agent issued; the recorded path
|
||||||
|
# (when any) is the detail, not the headline.
|
||||||
|
path = f" (from {start_coords} to {end_coords})" if start_coords and end_coords else ""
|
||||||
|
return f"Swiped {direction}{path}{duration_str}"
|
||||||
dragged = _target_label(action_obj, fallback="")
|
dragged = _target_label(action_obj, fallback="")
|
||||||
dragged_str = f" {dragged}" if dragged else ""
|
dragged_str = f" {dragged}" if dragged else ""
|
||||||
return f"Swiped{dragged_str} from {start_coords} to {end_coords}{duration_str}"
|
return f"Swiped{dragged_str} from {start_coords} to {end_coords}{duration_str}"
|
||||||
@@ -176,8 +182,32 @@ def format_action_clean(action_obj) -> str:
|
|||||||
)
|
)
|
||||||
return f"Tapped sequence of targets: {labelled}"
|
return f"Tapped sequence of targets: {labelled}"
|
||||||
return f"Tapped sequence of targets: {seq}"
|
return f"Tapped sequence of targets: {seq}"
|
||||||
|
elif act_type == "wait_for_text":
|
||||||
|
state = str(action_obj.get("wait_state") or "appear")
|
||||||
|
timeout = action_obj.get("timeout_ms")
|
||||||
|
timeout_str = f" (timeout {timeout}ms)" if timeout else ""
|
||||||
|
return f"Waited for text '{action_obj.get('text', '')}' to {state}{timeout_str}"
|
||||||
|
elif act_type == "open_link":
|
||||||
|
return f"Opened link '{action_obj.get('url', '')}'"
|
||||||
|
elif act_type == "erase_one_char":
|
||||||
|
return "Erased one character"
|
||||||
|
elif act_type == "focus_and_clear_text":
|
||||||
|
return f"Cleared text at {coords}"
|
||||||
else:
|
else:
|
||||||
return f"Action: {act_type} with args: {json.dumps(action_obj, ensure_ascii=False)}"
|
# Unknown action: show only the agent-facing arguments, never the
|
||||||
|
# bookkeeping keys the record carries for other consumers.
|
||||||
|
internal = {
|
||||||
|
"action",
|
||||||
|
"name",
|
||||||
|
"intent",
|
||||||
|
"coordinate_space",
|
||||||
|
"normalized_coordinates",
|
||||||
|
"normalized_start_coordinates",
|
||||||
|
"normalized_end_coordinates",
|
||||||
|
}
|
||||||
|
shown = {k: v for k, v in action_obj.items() if k not in internal and v is not None}
|
||||||
|
args_str = f" with args: {json.dumps(shown, ensure_ascii=False)}" if shown else ""
|
||||||
|
return f"Action: {act_type}{args_str}"
|
||||||
|
|
||||||
|
|
||||||
def format_actions_clean(actions) -> str:
|
def format_actions_clean(actions) -> str:
|
||||||
@@ -192,6 +222,48 @@ def format_actions_clean(actions) -> str:
|
|||||||
return format_action_clean(actions)
|
return format_action_clean(actions)
|
||||||
|
|
||||||
|
|
||||||
|
#: Past-tense outcome prefix (``format_action_clean``) -> intent-form prefix.
|
||||||
|
#: Longest prefixes first so "Double tapped" / "Tapped sequence" win over "Tapped".
|
||||||
|
_INTENT_PREFIXES: tuple[tuple[str, str], ...] = (
|
||||||
|
("Tapped sequence of targets: ", "tap sequence of targets: "),
|
||||||
|
("Double tapped ", "double tap "),
|
||||||
|
("Long pressed ", "long press "),
|
||||||
|
("Tapped ", "tap "),
|
||||||
|
("Inputted ", "type "),
|
||||||
|
("Swiped ", "swipe "),
|
||||||
|
("Pressed key ", "press key "),
|
||||||
|
("Launched app ", "launch app "),
|
||||||
|
("Stopped app ", "stop app "),
|
||||||
|
("Managed app ", "manage app "),
|
||||||
|
("Waited for text ", "wait for text "),
|
||||||
|
("Waited for ", "wait "),
|
||||||
|
("Opened link ", "open link "),
|
||||||
|
("Erased one character", "erase one character"),
|
||||||
|
("Cleared text ", "clear text "),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def action_intent_phrase(description: str) -> str:
|
||||||
|
"""The intent form of a rendered action phrase: ``Tapped 'Wi-Fi row' at
|
||||||
|
[500, 520]`` -> ``tap 'Wi-Fi row' at [500, 520]``.
|
||||||
|
|
||||||
|
``format_action_clean`` renders actions as past-tense outcomes, which
|
||||||
|
reads as success; a failure or incident context must describe what the
|
||||||
|
agent *tried* to do instead. Phrases without a known prefix (bursts,
|
||||||
|
unknown action types) are returned unchanged.
|
||||||
|
"""
|
||||||
|
text = str(description or "")
|
||||||
|
for outcome, intent in _INTENT_PREFIXES:
|
||||||
|
if text.startswith(outcome):
|
||||||
|
return intent + text[len(outcome) :]
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def format_action_intent(action_obj) -> str:
|
||||||
|
"""Intent-form rendering of one action (``launch app 'X'``, ``swipe up``)."""
|
||||||
|
return action_intent_phrase(format_action_clean(action_obj))
|
||||||
|
|
||||||
|
|
||||||
def _is_terminal_attempt_failure(attempts) -> bool:
|
def _is_terminal_attempt_failure(attempts) -> bool:
|
||||||
if not attempts:
|
if not attempts:
|
||||||
return False
|
return False
|
||||||
@@ -216,11 +288,17 @@ def failed_execution_error(result_obj) -> str | None:
|
|||||||
|
|
||||||
|
|
||||||
def format_incident_clean(incident: dict) -> str:
|
def format_incident_clean(incident: dict) -> str:
|
||||||
"""One-line rendering of an execution incident for history and result lines."""
|
"""One-line rendering of an execution incident for history and result lines.
|
||||||
|
|
||||||
|
The action is quoted in intent form (``launch app 'X'``): it did not
|
||||||
|
happen, so the past-tense outcome phrase would misreport it as done.
|
||||||
|
"""
|
||||||
kind = incident.get("kind") or "exec_error"
|
kind = incident.get("kind") or "exec_error"
|
||||||
category = incident.get("category") or "general"
|
category = incident.get("category") or "general"
|
||||||
consecutive = incident.get("consecutive_failures") or 1
|
consecutive = incident.get("consecutive_failures") or 1
|
||||||
description = incident.get("action_description") or format_action_clean(incident.get("action"))
|
description = action_intent_phrase(
|
||||||
|
incident.get("action_description") or format_action_clean(incident.get("action"))
|
||||||
|
)
|
||||||
reason = str(incident.get("reason") or "").strip()
|
reason = str(incident.get("reason") or "").strip()
|
||||||
burst_size = int(incident.get("burst_size") or 1)
|
burst_size = int(incident.get("burst_size") or 1)
|
||||||
index = int(incident.get("action_index") or 0)
|
index = int(incident.get("action_index") or 0)
|
||||||
@@ -296,6 +374,21 @@ def safe_parse_validation_result(result: Any) -> list:
|
|||||||
return []
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
#: Block labels of a detailed / replayed step. Role-neutral on purpose: the
|
||||||
|
#: same renderer replays a Flash turn (no Operator) and a Pro Operator turn,
|
||||||
|
#: and every Pro agent (Checker, Diagnoser, Outputter) reads them.
|
||||||
|
REASONING_BLOCK_LABEL = "[Reasoning & tool calls]"
|
||||||
|
ACTION_BLOCK_LABEL = "[Action]"
|
||||||
|
BURST_BLOCK_LABEL = "[Fast-Action Burst]"
|
||||||
|
|
||||||
|
#: Continuation lines of a " - " bullet stay inside the bullet.
|
||||||
|
_BULLET_CONTINUATION = "\n "
|
||||||
|
|
||||||
|
|
||||||
|
def _indent_continuation(text: str) -> str:
|
||||||
|
return text.replace("\n", _BULLET_CONTINUATION)
|
||||||
|
|
||||||
|
|
||||||
#: Limit tool-result text in the live context, which is rebuilt every turn.
|
#: Limit tool-result text in the live context, which is rebuilt every turn.
|
||||||
LIVE_RESULT_CHARS = 120
|
LIVE_RESULT_CHARS = 120
|
||||||
|
|
||||||
@@ -544,9 +637,27 @@ def _burst_member_status(result: Any, index: int) -> str:
|
|||||||
return f" (FAILED: {last.strip()})"
|
return f" (FAILED: {last.strip()})"
|
||||||
|
|
||||||
|
|
||||||
|
def replay_time_label(step: dict, session_start: Any) -> str:
|
||||||
|
"""The step-header clock of a replay: the session-relative ``T+mm:ss``
|
||||||
|
offset every agent prompt and ``search_history`` use, or the legacy
|
||||||
|
``Start: 12.3s`` text when no session start / step timestamp is known."""
|
||||||
|
ts = step.get("timestamp")
|
||||||
|
if (
|
||||||
|
isinstance(session_start, (int, float))
|
||||||
|
and not isinstance(session_start, bool)
|
||||||
|
and isinstance(ts, (int, float))
|
||||||
|
and not isinstance(ts, bool)
|
||||||
|
):
|
||||||
|
from artemis.memory.chunking import step_offset_label
|
||||||
|
|
||||||
|
return step_offset_label(step, float(session_start))
|
||||||
|
return f"Start: {step.get('relative_time') or 'N/A'}"
|
||||||
|
|
||||||
|
|
||||||
def render_step_replay(
|
def render_step_replay(
|
||||||
step: dict,
|
step: dict,
|
||||||
*,
|
*,
|
||||||
|
session_start: Any = None,
|
||||||
result_chars: int = REPLAY_RESULT_CHARS,
|
result_chars: int = REPLAY_RESULT_CHARS,
|
||||||
include_summary: bool = True,
|
include_summary: bool = True,
|
||||||
) -> str:
|
) -> str:
|
||||||
@@ -556,6 +667,9 @@ def render_step_replay(
|
|||||||
/ ``get_agent_friendly_steps``): coordinates already normalized, traces
|
/ ``get_agent_friendly_steps``): coordinates already normalized, traces
|
||||||
already expanded into ``interleaved_events``. Used by ``replay_steps`` and
|
already expanded into ``interleaved_events``. Used by ``replay_steps`` and
|
||||||
the MCP trace inspector. Screenshots are fetched separately.
|
the MCP trace inspector. Screenshots are fetched separately.
|
||||||
|
|
||||||
|
``session_start`` (the reader's ``session_start_time``) makes the header
|
||||||
|
carry the ``T+mm:ss`` session clock; see :func:`replay_time_label`.
|
||||||
"""
|
"""
|
||||||
return _render_step_detailed(
|
return _render_step_detailed(
|
||||||
step=step,
|
step=step,
|
||||||
@@ -566,6 +680,7 @@ def render_step_replay(
|
|||||||
is_most_recent=False,
|
is_most_recent=False,
|
||||||
result_chars=result_chars,
|
result_chars=result_chars,
|
||||||
include_summary=include_summary,
|
include_summary=include_summary,
|
||||||
|
time_label=replay_time_label(step, session_start),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -597,15 +712,17 @@ def _render_step_detailed(
|
|||||||
*,
|
*,
|
||||||
result_chars: int = LIVE_RESULT_CHARS,
|
result_chars: int = LIVE_RESULT_CHARS,
|
||||||
include_summary: bool = False,
|
include_summary: bool = False,
|
||||||
|
time_label: str | None = None,
|
||||||
) -> str:
|
) -> str:
|
||||||
# 1. Step Header
|
# 1. Step Header (``time_label`` overrides the live window's "Start: 12.3s")
|
||||||
status_str = "Most Recent Step, " if is_most_recent else ""
|
status_str = "Most Recent Step, " if is_most_recent else ""
|
||||||
|
clock = time_label or f"Start: {relative_time}"
|
||||||
|
|
||||||
# In the live window the summary is omitted from the detailed view header
|
# In the live window the summary is omitted from the detailed view header
|
||||||
# as it is already fully detailed below; replay callers opt in because the
|
# as it is already fully detailed below; replay callers opt in because the
|
||||||
# summary is the Operator's own claim about the step, which an auditor
|
# summary is the Operator's own claim about the step, which an auditor
|
||||||
# compares against the evidence that follows.
|
# compares against the evidence that follows.
|
||||||
step_line = f"- **Step {step['step_number']} ({status_str}Start: {relative_time})**"
|
step_line = f"- **Step {step['step_number']} ({status_str}{clock})**"
|
||||||
if include_summary:
|
if include_summary:
|
||||||
screen_line = _screen_description_line(step, summary)
|
screen_line = _screen_description_line(step, summary)
|
||||||
if screen_line:
|
if screen_line:
|
||||||
@@ -695,9 +812,10 @@ def _render_step_detailed(
|
|||||||
if raw_thinking and raw_thinking.strip() and not thought_exists(raw_thinking):
|
if raw_thinking and raw_thinking.strip() and not thought_exists(raw_thinking):
|
||||||
operator_events.append({"type": "thought", "content": raw_thinking.strip()})
|
operator_events.append({"type": "thought", "content": raw_thinking.strip()})
|
||||||
|
|
||||||
# 4. Render Operator Loop
|
# 4. Render the acting model's reasoning and tool calls (role-neutral: the
|
||||||
|
# same block replays a Flash turn and a Pro Operator turn)
|
||||||
if operator_events:
|
if operator_events:
|
||||||
step_line += "\n * [Operator Decision Loop]:"
|
step_line += f"\n * {REASONING_BLOCK_LABEL}:"
|
||||||
ACTION_TOOLS = {
|
ACTION_TOOLS = {
|
||||||
"click",
|
"click",
|
||||||
"swipe",
|
"swipe",
|
||||||
@@ -722,7 +840,7 @@ def _render_step_detailed(
|
|||||||
)
|
)
|
||||||
cleaned = re.sub(r"</?thought>", "", cleaned, flags=re.DOTALL).strip()
|
cleaned = re.sub(r"</?thought>", "", cleaned, flags=re.DOTALL).strip()
|
||||||
if cleaned:
|
if cleaned:
|
||||||
step_line += f"\n - {cleaned}"
|
step_line += f"\n - {_indent_continuation(cleaned)}"
|
||||||
elif e_type == "tool_call":
|
elif e_type == "tool_call":
|
||||||
if name in ACTION_TOOLS:
|
if name in ACTION_TOOLS:
|
||||||
continue
|
continue
|
||||||
@@ -735,17 +853,13 @@ def _render_step_detailed(
|
|||||||
if formatted:
|
if formatted:
|
||||||
step_line += f"\n - [Tool Call]: {formatted}"
|
step_line += f"\n - [Tool Call]: {formatted}"
|
||||||
|
|
||||||
# Check if the execution failed
|
# 5. Render the action the step executed (single vetted action or a burst).
|
||||||
exec_error = None
|
# A failed action's error is printed once, on the [Result] line below.
|
||||||
if isinstance(result, dict) and result.get("status") in ("failed", "error"):
|
|
||||||
exec_error = failed_execution_error(result)
|
|
||||||
|
|
||||||
# 5. Render Planned Action (single vetted action or fast-action burst)
|
|
||||||
if action:
|
if action:
|
||||||
actions = action if isinstance(action, list) else [action]
|
actions = action if isinstance(action, list) else [action]
|
||||||
if len(actions) > 1:
|
if len(actions) > 1:
|
||||||
step_line += (
|
step_line += (
|
||||||
f"\n * [Planned Fast-Action Burst]: {len(actions)} actions fired back to"
|
f"\n * {BURST_BLOCK_LABEL}: {len(actions)} actions fired back to"
|
||||||
" back without the safety net"
|
" back without the safety net"
|
||||||
)
|
)
|
||||||
for i, member in enumerate(actions):
|
for i, member in enumerate(actions):
|
||||||
@@ -756,15 +870,11 @@ def _render_step_detailed(
|
|||||||
action_clean = format_action_clean(actions[0])
|
action_clean = format_action_clean(actions[0])
|
||||||
if intercepted:
|
if intercepted:
|
||||||
step_line += (
|
step_line += (
|
||||||
f"\n * [Planned Action]: {action_clean}"
|
f"\n * {ACTION_BLOCK_LABEL}: {action_clean}"
|
||||||
" (Intercepted by Pre-Execution Safety Net)"
|
" (Intercepted by Pre-Execution Safety Net)"
|
||||||
)
|
)
|
||||||
elif exec_error:
|
|
||||||
step_line += (
|
|
||||||
f"\n * [Planned Action]: {action_clean} -> (Execution failed: {exec_error})"
|
|
||||||
)
|
|
||||||
else:
|
else:
|
||||||
step_line += f"\n * [Planned Action]: {action_clean}"
|
step_line += f"\n * {ACTION_BLOCK_LABEL}: {action_clean}"
|
||||||
|
|
||||||
# 6. Render Pre-Execution Safety Net
|
# 6. Render Pre-Execution Safety Net
|
||||||
if intercepted:
|
if intercepted:
|
||||||
@@ -892,7 +1002,7 @@ def build_plan_and_history(
|
|||||||
# Ensure the last step is always in detailed_step_ids
|
# Ensure the last step is always in detailed_step_ids
|
||||||
detailed_step_ids.add(full_info_step_id)
|
detailed_step_ids.add(full_info_step_id)
|
||||||
|
|
||||||
# M4: chunk blocks replace the per-step lines of already-chunked
|
# Chunk blocks replace the per-step lines of already-chunked
|
||||||
# ranges. Chunks only ever cover frozen (old) turns, so they are
|
# ranges. Chunks only ever cover frozen (old) turns, so they are
|
||||||
# emitted first, in chronological order; steps that must render
|
# emitted first, in chronological order; steps that must render
|
||||||
# detailed (recent window / most-recent step) are never suppressed.
|
# detailed (recent window / most-recent step) are never suppressed.
|
||||||
|
|||||||
+25
-3
@@ -192,7 +192,7 @@
|
|||||||
// ==========================================================================
|
// ==========================================================================
|
||||||
// 🌐 4.3 Shared Capabilities & Global Overrides
|
// 🌐 4.3 Shared Capabilities & Global Overrides
|
||||||
// ==========================================================================
|
// ==========================================================================
|
||||||
// 🧠 Unified Step-Memory Runtime (history-module redesign v3, M0–M5)
|
// Shared step memory and history compression
|
||||||
// Transcript ledger + layered compression for the Pro operator.
|
// Transcript ledger + layered compression for the Pro operator.
|
||||||
"memory": {
|
"memory": {
|
||||||
// Background summarization runtime (visual/capsule lenses)
|
// Background summarization runtime (visual/capsule lenses)
|
||||||
@@ -204,15 +204,33 @@
|
|||||||
"transcript": {
|
"transcript": {
|
||||||
// ROLLBACK SWITCH: set false to restore the legacy per-turn 2-message
|
// ROLLBACK SWITCH: set false to restore the legacy per-turn 2-message
|
||||||
// prompt path byte-for-byte (L2/L3 chunk compression rides this same
|
// prompt path byte-for-byte (L2/L3 chunk compression rides this same
|
||||||
// flag). Default on since M5 (2026-09-01) per the on-device A/B:
|
// flag). See services/token_meter.py for measured session token usage
|
||||||
// turn latency −77% P50, cache-hit calls 43%→88%, output tokens −74%.
|
// and cache-hit ratios.
|
||||||
"enabled": true,
|
"enabled": true,
|
||||||
"context_budget_tokens": 80000,
|
"context_budget_tokens": 80000,
|
||||||
|
// Below start_ratio of the operator's own measured prompt size, closed
|
||||||
|
// segments only prepare their summary; the original turns are kept in
|
||||||
|
// context and replaced once occupancy reaches this ratio. Measurements
|
||||||
|
// come from the operator; other agents' calls do not affect this gate.
|
||||||
|
"start_ratio": 0.35,
|
||||||
"soft_ratio": 0.7,
|
"soft_ratio": 0.7,
|
||||||
"hard_ratio": 0.9,
|
"hard_ratio": 0.9,
|
||||||
"min_active_steps": 5,
|
"min_active_steps": 5,
|
||||||
|
// Screenshot edge: 6 while the operator's measured context is below
|
||||||
|
// start_ratio (nothing is being replaced yet), 3 from there up. A
|
||||||
|
// screenshot reaching the edge is replaced in place by its visual
|
||||||
|
// summary (the 'Current Screenshot' label leaves with it) and never
|
||||||
|
// restored; the depth only decides how long new ones stay.
|
||||||
"image_scrub_depth": 3,
|
"image_scrub_depth": 3,
|
||||||
|
"image_scrub_depth_relaxed": 6,
|
||||||
"pending_grace_steps": 3,
|
"pending_grace_steps": 3,
|
||||||
|
// Text edge: once a newer observation exists (depth 1) the UI element
|
||||||
|
// list, the plan recitation and the per-turn ephemeral blocks
|
||||||
|
// (reminders, hints, guidance wrappers) leave the older observation.
|
||||||
|
// User instruction bodies remain in history. The text edge stays
|
||||||
|
// shallow on purpose: only the live observation carries an indexed
|
||||||
|
// element list, so a stale [n] index can never be picked up as a
|
||||||
|
// target.
|
||||||
"xml_scrub_depth": 1,
|
"xml_scrub_depth": 1,
|
||||||
// Local historical-state hint (no model call); distance calibrated
|
// Local historical-state hint (no model call); distance calibrated
|
||||||
// 2026-09-01 from 460 dHash-stamped on-device steps (same screen <=4,
|
// 2026-09-01 from 460 dHash-stamped on-device steps (same screen <=4,
|
||||||
@@ -223,6 +241,10 @@
|
|||||||
// Segment (L2) chunk compression: three-band chunks, era folding
|
// Segment (L2) chunk compression: three-band chunks, era folding
|
||||||
"chunking": {
|
"chunking": {
|
||||||
"max_steps": 12,
|
"max_steps": 12,
|
||||||
|
// Size/pressure closes wait until at least this many turns are
|
||||||
|
// eligible, so heavy turns do not become one-step chunks (one capsule
|
||||||
|
// call each); milestone closes are exempt, the hard line waives it.
|
||||||
|
"min_steps": 3,
|
||||||
"target_source_tokens": 2000,
|
"target_source_tokens": 2000,
|
||||||
"model": "gemini-3.8-flash",
|
"model": "gemini-3.8-flash",
|
||||||
"max_chunks": 8
|
"max_chunks": 8
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# 🔌 Universal MCP Server for ARTEMIS
|
# 🔌 Universal MCP Server for ARTEMIS
|
||||||
|
|
||||||
This directory contains the universal **Model Context Protocol (MCP)** server for **ARTEMIS**, enabling seamless mobile device automation inside AI IDEs and agents like **Antigravity**, **Cursor**, **Claude Code**, **OpenClaw**, and **Windsurf**.
|
This directory contains the **Model Context Protocol (MCP)** server for **ARTEMIS**. It exposes mobile device automation to clients such as **Antigravity**, **Cursor**, **Claude Code**, **OpenClaw**, and **Windsurf**.
|
||||||
|
|
||||||
## 🏗️ Architecture
|
## 🏗️ Architecture
|
||||||
|
|
||||||
@@ -24,7 +24,8 @@ mcp_server/
|
|||||||
│ ├── task_runner.py # mobile_run_task
|
│ ├── task_runner.py # mobile_run_task
|
||||||
│ ├── task_manager.py # mobile_manage_task
|
│ ├── task_manager.py # mobile_manage_task
|
||||||
│ ├── device_state.py # mobile_get_device_state
|
│ ├── device_state.py # mobile_get_device_state
|
||||||
│ └── inspect_trace.py # mobile_inspect_trace
|
│ ├── inspect_trace.py # mobile_inspect_trace
|
||||||
|
│ └── diagnose.py # mobile_diagnose
|
||||||
└── utils/ # Environment and device utilities
|
└── utils/ # Environment and device utilities
|
||||||
├── device_utils.py # Cross-platform ADB and emulator resolver
|
├── device_utils.py # Cross-platform ADB and emulator resolver
|
||||||
└── env_utils.py # Python interpreter and process manager
|
└── env_utils.py # Python interpreter and process manager
|
||||||
@@ -56,6 +57,7 @@ The included [`rules.md`](./rules.md) file contains the **Mobile Testing Mindset
|
|||||||
2. **Flash vs. Pro Routing Strategy**: Guides the AI to choose **Flash** for rapid, straightforward UI actions (no step cap by default) and **Pro** for tasks that need a persistent plan, verified checkpoints, polling loops, or ADB / video / log diagnosis.
|
2. **Flash vs. Pro Routing Strategy**: Guides the AI to choose **Flash** for rapid, straightforward UI actions (no step cap by default) and **Pro** for tasks that need a persistent plan, verified checkpoints, polling loops, or ADB / video / log diagnosis.
|
||||||
3. **Latency & Timing Compensation**: Clarifies the difference between AI exploratory latency (e.g., model decision intervals) and the deterministic timing requirements of final test code.
|
3. **Latency & Timing Compensation**: Clarifies the difference between AI exploratory latency (e.g., model decision intervals) and the deterministic timing requirements of final test code.
|
||||||
4. **"Dynamic-First, Coordinate-Fallback" Locator Pattern**: Teaches the AI to prioritize dynamic UI locators (Resource IDs, OCR text, semantics) for layout resilience, while implementing absolute coordinate fallbacks for maximum execution reliability.
|
4. **"Dynamic-First, Coordinate-Fallback" Locator Pattern**: Teaches the AI to prioritize dynamic UI locators (Resource IDs, OCR text, semantics) for layout resilience, while implementing absolute coordinate fallbacks for maximum execution reliability.
|
||||||
|
5. **Environment Self-Diagnosis**: Tells the AI to call `mobile_diagnose` first whenever a tool errors or the user says ARTEMIS is not working, and how to act on the returned fix list (run local commands itself, relay device prompts to the user, never move API keys through the chat, reload the MCP server after config changes).
|
||||||
|
|
||||||
### How Global Rules & MCP are Installed
|
### How Global Rules & MCP are Installed
|
||||||
When you run `uv run artemis mcp --install all` (or target a specific IDE like `--install cursor`, `--install claude`, etc.), the installer **automatically synchronizes both global MCP server configuration and global testing rules** into the corresponding user home directories (without creating project-level rule files in your workspace):
|
When you run `uv run artemis mcp --install all` (or target a specific IDE like `--install cursor`, `--install claude`, etc.), the installer **automatically synchronizes both global MCP server configuration and global testing rules** into the corresponding user home directories (without creating project-level rule files in your workspace):
|
||||||
@@ -75,6 +77,7 @@ When you run `uv run artemis mcp --install all` (or target a specific IDE like `
|
|||||||
* **`mobile_manage_task`**: Manages task lifecycle (`status`, `stop`, `inject_instruction`), returning task state and assigned `device_serial`. Pass `release_loop=True` with `inject_instruction` to gracefully end a `[Loop:continuous]` monitoring task — this explicit signal (not "please stop" wording) is what unlocks the loop milestone's completion.
|
* **`mobile_manage_task`**: Manages task lifecycle (`status`, `stop`, `inject_instruction`), returning task state and assigned `device_serial`. Pass `release_loop=True` with `inject_instruction` to gracefully end a `[Loop:continuous]` monitoring task — this explicit signal (not "please stop" wording) is what unlocks the loop milestone's completion.
|
||||||
* **`mobile_get_device_state`**: Real-time observer (`screenshot` or OCR+XML `hierarchy`) with optional `device_serial`.
|
* **`mobile_get_device_state`**: Real-time observer (`screenshot` or OCR+XML `hierarchy`) with optional `device_serial`.
|
||||||
* **`mobile_inspect_trace`**: Granular trace inspection, visual action overlays, agent reasoning, and `device_serial` tracking.
|
* **`mobile_inspect_trace`**: Granular trace inspection, visual action overlays, agent reasoning, and `device_serial` tracking.
|
||||||
|
* **`mobile_diagnose`**: Environment doctor for the IDE. Reuses the readiness probes behind `artemis doctor` and the web console's device wizard (Python runtime, config, LLM credentials, ADB / devices / RSA keys / emulators, video toolchain) plus an MCP-host probe (server interpreter vs project `.venv`, detected MCP client, `.env` location, traces directory, daemon port collisions). Returns a `verdict` (`ready` | `degraded` | `blocked`), an ordered `next_steps` fix list the AI agent can act on (`Run:` one-command lines vs `Guidance:` for the user), scrubbed per-check details, `tasks` holding or waiting for devices, log paths, and the most recent failed task with its `recent_errors`. Optional deep checks: `verify_credentials=true` (live API-key validation, ~12s, result in `credentials`), `probe_device=true` (end-to-end screenshot + UIAutomator hierarchy, ~20s, result in `device_probe`), `launch_avd="<name>"` (boots an installed AVD in the background; re-run after ~60s). `attempt_fix=true` applies the safe self-heals (regenerate corrupted ADB keys, restart the ADB server, clear stale device locks / queue tickets left by crashed runners).
|
||||||
|
|
||||||
### 📱 Device Selection & Multi-Device Execution
|
### 📱 Device Selection & Multi-Device Execution
|
||||||
ARTEMIS supports parallel execution across multiple connected Android devices and emulators:
|
ARTEMIS supports parallel execution across multiple connected Android devices and emulators:
|
||||||
|
|||||||
+3
-1
@@ -37,7 +37,9 @@ mcp = FastMCP(
|
|||||||
"Use mobile_run_task to launch autonomous UI workflows on connected Android devices or emulators, "
|
"Use mobile_run_task to launch autonomous UI workflows on connected Android devices or emulators, "
|
||||||
"mobile_manage_task to check status or steer execution, "
|
"mobile_manage_task to check status or steer execution, "
|
||||||
"mobile_get_device_state to inspect real-time device screen/hierarchy, "
|
"mobile_get_device_state to inspect real-time device screen/hierarchy, "
|
||||||
"and mobile_inspect_trace to inspect detailed execution steps and visual action overlays."
|
"mobile_inspect_trace to inspect detailed execution steps and visual action overlays, "
|
||||||
|
"and mobile_diagnose whenever a tool errors, no device is found, or the user says "
|
||||||
|
"ARTEMIS is not working: it checks the environment and returns ordered fix steps."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
+14
-2
@@ -1,9 +1,9 @@
|
|||||||
# Mobile Testing Mindset (ARTEMIS Integration)
|
# Mobile Testing Mindset (ARTEMIS Integration)
|
||||||
|
|
||||||
As a senior test engineer, you must ensure that all test code you write is **fully executable, stable, and exceptionally robust**. You must never guess or assume how an interaction behaves on a mobile device, no matter how simple the task may seem. Instead, you must leverage **ARTEMIS**, a powerful mobile UI automation framework, to discover and verify the exact execution flow, and author your test scripts based strictly on these concrete, verified steps. Furthermore, if you need to investigate or research specific software or hardware behaviors, you should utilize ARTEMIS to perform active explorations on the device, whether by mimicking human interactions or executing precise ADB commands.
|
Write executable tests based on device behavior verified with **ARTEMIS**. Before authoring a test, explore the target application and confirm its screens, transitions, and interactions. Use ARTEMIS device actions or ADB commands to investigate software and hardware behavior instead of assuming how an interaction works.
|
||||||
|
|
||||||
### 1. The Runnable Code Principle & ARTEMIS Exploration
|
### 1. The Runnable Code Principle & ARTEMIS Exploration
|
||||||
- **When tasked with authoring tests**, your primary goal is to deliver **runnable, production-grade test code**.
|
- **When tasked with authoring tests**, deliver runnable test code with verified interactions and explicit wait conditions.
|
||||||
- Before writing any test code, you must use the ARTEMIS MCP tools to interactively run and explore the target application. This allows you to discover the exact sequence of UI states, transitions, and required interactions.
|
- Before writing any test code, you must use the ARTEMIS MCP tools to interactively run and explore the target application. This allows you to discover the exact sequence of UI states, transitions, and required interactions.
|
||||||
- Analyze the user's target testing framework to exploit its native capabilities and maximize test stability.
|
- Analyze the user's target testing framework to exploit its native capabilities and maximize test stability.
|
||||||
- **Timing & Latency Management (Exploration vs. Execution)**:
|
- **Timing & Latency Management (Exploration vs. Execution)**:
|
||||||
@@ -54,3 +54,15 @@ Deeply understand and select between ARTEMIS's dual execution models (**ARTEMIS
|
|||||||
2. **Catch**: If the dynamic attempt fails, fall back to the precise absolute coordinates verified during your ARTEMIS exploration.
|
2. **Catch**: If the dynamic attempt fails, fall back to the precise absolute coordinates verified during your ARTEMIS exploration.
|
||||||
- **Coordinate-Only Frameworks**: If the framework only supports coordinates, ensure the coordinates are well-documented, and where possible, parameterized or made relative to screen boundaries to mitigate resolution differences.
|
- **Coordinate-Only Frameworks**: If the framework only supports coordinates, ensure the coordinates are well-documented, and where possible, parameterized or made relative to screen boundaries to mitigate resolution differences.
|
||||||
- **Dynamic-Only Frameworks**: If the framework does not support coordinate-based clicks, focus entirely on generating highly robust dynamic locators, leveraging ARTEMIS's element descriptions and XML tree analysis.
|
- **Dynamic-Only Frameworks**: If the framework does not support coordinate-based clicks, focus entirely on generating highly robust dynamic locators, leveraging ARTEMIS's element descriptions and XML tree analysis.
|
||||||
|
|
||||||
|
### 5. Environment Self-Diagnosis (`mobile_diagnose`)
|
||||||
|
- **Diagnose before guessing**: Whenever an ARTEMIS tool returns an error, a task fails to start or is rejected, no device is found, or the user reports that ARTEMIS "doesn't work" in the IDE, call `mobile_diagnose` first. Do not troubleshoot by re-running `mobile_run_task` on trial and error, and do not ask the user to run `adb` commands by hand before you have the report.
|
||||||
|
- **If `mobile_diagnose` is not in your tool list at all**, the MCP server itself failed to start: check the IDE's MCP server log and run `uv run artemis doctor` in the project directory (the CLI shares the same checks).
|
||||||
|
- **Follow `next_steps` in order**: The report lists fixes in dependency order (Python runtime → config → MCP host → LLM credentials → device → optional video toolchain). `Run:` lines are single, local, non-destructive shell commands (one command per line, no `&&`) you may execute yourself (ask before installing software with winget / brew / apt). `Guidance:` lines describe something only the user can do (unlock the phone, tap "Allow" on the USB-debugging prompt, plug in a cable, free a port, tell you which IDE they use) — relay them verbatim. `Docs:` lines are references.
|
||||||
|
- **Let ARTEMIS self-heal first**: Call `mobile_diagnose(attempt_fix=true)` before asking for manual ADB intervention; it regenerates corrupted ADB RSA keys, restarts the ADB server when that is safe (no device ready, no task holding a device), and removes stale device locks / queue tickets left behind by crashed runners.
|
||||||
|
- **Deeper checks on demand**: `verify_credentials=true` validates the configured API keys live (~12s; use when a key exists but tasks fail with auth / quota errors; result in `credentials`). `probe_device=true` runs a true end-to-end check — screenshot plus UI hierarchy through UIAutomator (~20s; use when the device shows connected but observation or tasks fail; result in `device_probe`).
|
||||||
|
- **Emulators**: `launch_avd="<name>"` starts an installed Android Virtual Device in the background via ARTEMIS's emulator manager and returns immediately; boot takes 1-3 minutes, so re-run `mobile_diagnose` after ~60s. Never run `emulator -avd` in the shell yourself — it is a foreground process and hangs the shell.
|
||||||
|
- **Busy devices**: `tasks: {active, queued}` lists tasks currently holding or waiting for devices. A busy device means wait, or stop the task with `mobile_manage_task(action="stop", trace_id=...)`.
|
||||||
|
- **Credentials never go through the chat**: If the LLM key is missing, ask the user to add it to the `.env` file at the path the report shows (`host.env_file`) or to the MCP server's `env` block, then to restart the MCP server. `artemis init` is interactive and cannot be driven from a tool call.
|
||||||
|
- **Restart after configuration changes**: Settings are read when the server starts. After editing `.env`, `config/artemis.jsonc`, or the MCP client configuration, reload the MCP server in the IDE before calling `mobile_diagnose` again; otherwise it reports the old state.
|
||||||
|
- **Failed tasks**: `logs.last_failed_task` gives the trace id, its `stderr_log`, and `recent_errors` (the error lines from that stderr, so you need not open the file); read them before re-running the task. Repeat `mobile_diagnose` until `verdict` is `ready` or `degraded` before delegating tasks again (`degraded` means tasks can run but something optional is off: video tools, the OCR key, a secondary credential, or a host warning such as an interpreter mismatch or a squatted daemon port; relay its `[OPTIONAL]` steps to the user, do not loop on them).
|
||||||
@@ -15,6 +15,7 @@
|
|||||||
"""MCP Tools package for ARTEMIS."""
|
"""MCP Tools package for ARTEMIS."""
|
||||||
|
|
||||||
from mcp_server.tools.device_state import mobile_get_device_state
|
from mcp_server.tools.device_state import mobile_get_device_state
|
||||||
|
from mcp_server.tools.diagnose import mobile_diagnose
|
||||||
from mcp_server.tools.inspect_trace import mobile_inspect_trace
|
from mcp_server.tools.inspect_trace import mobile_inspect_trace
|
||||||
from mcp_server.tools.task_manager import mobile_manage_task
|
from mcp_server.tools.task_manager import mobile_manage_task
|
||||||
from mcp_server.tools.task_runner import mobile_run_task
|
from mcp_server.tools.task_runner import mobile_run_task
|
||||||
@@ -24,4 +25,5 @@ __all__ = [
|
|||||||
"mobile_manage_task",
|
"mobile_manage_task",
|
||||||
"mobile_get_device_state",
|
"mobile_get_device_state",
|
||||||
"mobile_inspect_trace",
|
"mobile_inspect_trace",
|
||||||
|
"mobile_diagnose",
|
||||||
]
|
]
|
||||||
File diff suppressed because it is too large.
Load diff
@@ -284,9 +284,7 @@ async def run_part_4():
|
|||||||
print_status_snapshot("Part 4 Running (Mixed Queueing)")
|
print_status_snapshot("Part 4 Running (Mixed Queueing)")
|
||||||
|
|
||||||
await asyncio.gather(t4a, t4b, t4c)
|
await asyncio.gather(t4a, t4b, t4c)
|
||||||
print(
|
print("Part 4 passed: cross-device concurrency and single-device queueing.\n")
|
||||||
"🎉 Part 4 Success: Cross-device concurrency and single-device queueing work together seamlessly!\n"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
async def main():
|
async def main():
|
||||||
|
|||||||
+1
-1
@@ -17,7 +17,7 @@
|
|||||||
|
|
||||||
Fastest and simplest way to launch the Artemis Web UI across all platforms:
|
Fastest and simplest way to launch the Artemis Web UI across all platforms:
|
||||||
- Automatically detects if the UI is already running and opens the browser directly.
|
- Automatically detects if the UI is already running and opens the browser directly.
|
||||||
- Uses uv or existing virtualenv seamlessly.
|
- Uses uv or an existing virtualenv.
|
||||||
- Zero-configuration one-click start.
|
- Zero-configuration one-click start.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|||||||
+96
-36
@@ -3,33 +3,39 @@
|
|||||||
"type": "function",
|
"type": "function",
|
||||||
"function": {
|
"function": {
|
||||||
"name": "click",
|
"name": "click",
|
||||||
"description": "[ACTION] Tap/click on the target coordinate on the screen. The screen after click will be returned automatically. For buttons, checkboxes, tabs, icons, items in a list, tap ON the element. For text fields / search bars / input boxes, tap INSIDE the input box to focus it.",
|
"description": "[ACTION] Click on the target location on the screen (supports element index or absolute normalized coordinates).",
|
||||||
"parameters": {
|
"parameters": {
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"properties": {
|
"properties": {
|
||||||
"target": {
|
"target": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"type": "integer"
|
||||||
|
},
|
||||||
|
{
|
||||||
"type": "array",
|
"type": "array",
|
||||||
"items": {
|
"items": {
|
||||||
"type": "integer"
|
"type": "integer"
|
||||||
},
|
}
|
||||||
"description": "Normalized coordinates [x, y] in 0-1000 scale."
|
}
|
||||||
|
],
|
||||||
|
"description": "Click target. Can be an element index number (int, e.g. 3) OR normalized coordinates (list of 2 integers, e.g. [500, 600])."
|
||||||
},
|
},
|
||||||
"target_description": {
|
"target_description": {
|
||||||
"type": "string",
|
"type": "string",
|
||||||
"description": "What the target is, in a few words (e.g. 'play button', 'search input', 'video body'). Required: a coordinate names nothing by itself."
|
"description": "What the target is, in a few words (e.g. 'play button', 'search input', 'video body'). REQUIRED when target is a coordinate pair; ignored for an element index."
|
||||||
},
|
},
|
||||||
"times": {
|
"times": {
|
||||||
"type": "integer",
|
"type": "integer",
|
||||||
"description": "Number of taps to perform (default 1). Set to 2 for double click."
|
"description": "Number of consecutive clicks on this target. Use this for double-clicks or multi-clicks (e.g. 7 to enter developer mode). Default is 1."
|
||||||
},
|
},
|
||||||
"delay_ms": {
|
"delay_ms": {
|
||||||
"type": "integer",
|
"type": "integer",
|
||||||
"description": "Delay between taps in milliseconds (default 100)."
|
"description": "Delay in milliseconds between consecutive clicks. Default is 100."
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"required": [
|
"required": [
|
||||||
"target",
|
"target"
|
||||||
"target_description"
|
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -76,29 +82,35 @@
|
|||||||
"type": "function",
|
"type": "function",
|
||||||
"function": {
|
"function": {
|
||||||
"name": "long_press",
|
"name": "long_press",
|
||||||
"description": "[ACTION] Long press on a target coordinate on the screen. The screen after long pressing will be returned automatically.",
|
"description": "[ACTION] Long press on the target location on the screen (supports element index or absolute normalized coordinates).",
|
||||||
"parameters": {
|
"parameters": {
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"properties": {
|
"properties": {
|
||||||
"target": {
|
"target": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"type": "integer"
|
||||||
|
},
|
||||||
|
{
|
||||||
"type": "array",
|
"type": "array",
|
||||||
"items": {
|
"items": {
|
||||||
"type": "integer"
|
"type": "integer"
|
||||||
},
|
}
|
||||||
"description": "Normalized coordinates [x, y] in 0-1000 scale."
|
}
|
||||||
|
],
|
||||||
|
"description": "Long press target. Can be an element index number (int, e.g. 3) OR normalized coordinates (list of 2 integers, e.g. [500, 600])."
|
||||||
},
|
},
|
||||||
"target_description": {
|
"target_description": {
|
||||||
"type": "string",
|
"type": "string",
|
||||||
"description": "What the target is, in a few words (e.g. 'play button', 'search input', 'video body'). Required: a coordinate names nothing by itself."
|
"description": "What the target is, in a few words (e.g. 'play button', 'search input', 'video body'). REQUIRED when target is a coordinate pair; ignored for an element index."
|
||||||
},
|
},
|
||||||
"duration_ms": {
|
"duration": {
|
||||||
"type": "integer",
|
"type": "integer",
|
||||||
"description": "Duration of press in milliseconds (default 1000ms)."
|
"description": "Long press duration in milliseconds (default 1000)."
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"required": [
|
"required": [
|
||||||
"target",
|
"target"
|
||||||
"target_description"
|
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -107,34 +119,40 @@
|
|||||||
"type": "function",
|
"type": "function",
|
||||||
"function": {
|
"function": {
|
||||||
"name": "input_text",
|
"name": "input_text",
|
||||||
"description": "[ACTION] Type text into an input field on the screen. The screen after typing will be returned automatically. Automatically taps inside the input box at target [x, y] to focus before typing.",
|
"description": "[ACTION] Type text into the target input field (supports replacing whole text or appending to the end, and multi-line strings with '\\n').",
|
||||||
"parameters": {
|
"parameters": {
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"properties": {
|
"properties": {
|
||||||
"text": {
|
"text": {
|
||||||
"type": "string",
|
"type": "string",
|
||||||
"description": "Text to type into the focused input field."
|
"description": "The text content to input. Supports multi-line content with '\\n'."
|
||||||
},
|
},
|
||||||
"target": {
|
"target": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"type": "integer"
|
||||||
|
},
|
||||||
|
{
|
||||||
"type": "array",
|
"type": "array",
|
||||||
"items": {
|
"items": {
|
||||||
"type": "integer"
|
"type": "integer"
|
||||||
},
|
}
|
||||||
"description": "Coordinates [x, y] of the input box in 0-1000 scale."
|
}
|
||||||
|
],
|
||||||
|
"description": "Input target field. Can be an input box element index number (int, e.g. 3) OR normalized coordinates (list of 2 integers, e.g. [500, 600])."
|
||||||
},
|
},
|
||||||
"target_description": {
|
"target_description": {
|
||||||
"type": "string",
|
"type": "string",
|
||||||
"description": "What the target is, in a few words (e.g. 'play button', 'search input', 'video body'). Required: a coordinate names nothing by itself."
|
"description": "What the target is, in a few words (e.g. 'play button', 'search input', 'video body'). REQUIRED when target is a coordinate pair; ignored for an element index."
|
||||||
},
|
},
|
||||||
"clear_exist": {
|
"clear_exist": {
|
||||||
"type": "boolean",
|
"type": "boolean",
|
||||||
"description": "Whether to clear existing text in the input box before typing (default True)."
|
"description": "Whether to clear existing text before typing. True (default): clear/replace entire text. False: append at the end of existing content."
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"required": [
|
"required": [
|
||||||
"text",
|
"text",
|
||||||
"target",
|
"target"
|
||||||
"target_description"
|
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -143,7 +161,7 @@
|
|||||||
"type": "function",
|
"type": "function",
|
||||||
"function": {
|
"function": {
|
||||||
"name": "swipe",
|
"name": "swipe",
|
||||||
"description": "[ACTION] Perform a swipe, drag, or slider-adjustment gesture on the screen. The screen and UI hierarchy after swipe will be returned automatically.\n\n• Directional Scrolling ('direction' or 'action'): Recommended for general browsing and standard page scrolling in most scenarios. Automatically computes safe swipe vectors and adaptive duration, retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings. Supports scoping to a sub-container via 'target'. If it fails on certain custom layouts, fall back to specifying exact coordinates ('start' and 'end') directly.\n• Precise Coordinate Gestures ('start', 'end' or coordinates list): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars), drag-and-drop / list reordering, or as a reliable fallback when directional scrolling fails on specific containers. Always drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), swipe fully to the extreme boundary.",
|
"description": "[ACTION] Perform a swipe, drag, or slider-adjustment gesture on the screen.\n\n• Directional Scrolling ('direction'): Recommended for general browsing and standard page scrolling in most scenarios. Automatically computes safe swipe vectors and adaptive duration, retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings. Supports scoping to a sub-container via 'target'. If it fails on certain custom layouts, fall back to specifying exact coordinates ('start' and 'end') directly.\n• Precise Coordinate Gestures ('start', 'end'): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars), drag-and-drop / list reordering, or as a reliable fallback when directional scrolling fails on specific containers. Always drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), swipe fully to the extreme boundary.\n\nArgs:\n direction: Smart directional scrolling ('up', 'down', 'left', 'right'). Automatically computes safe swipe vectors, retaining 40% visual overlap: 'up' (reveal content below), 'down' (reveal content above), 'left', 'right'.\n start: Start normalized coordinates [start_x, start_y] in 0-1000 scale.\n end: End normalized coordinates [end_x, end_y] in 0-1000 scale.\n target: Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within.\n gesture: Backward-compatible parameter: direction string OR custom coordinates list [start_x, start_y, end_x, end_y] in 0-1000 scale.\n target_description: What is being dragged, in a few words (e.g. 'brightness slider knob'). REQUIRED for coordinate gestures ('start'/'end' or a coordinates list); ignored for directional scrolling.\n duration: Optional gesture duration in milliseconds (default 800).",
|
||||||
"parameters": {
|
"parameters": {
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"properties": {
|
"properties": {
|
||||||
@@ -162,30 +180,62 @@
|
|||||||
"items": {
|
"items": {
|
||||||
"type": "integer"
|
"type": "integer"
|
||||||
},
|
},
|
||||||
"description": "Start normalized coordinates [start_x, start_y] in 0-1000 scale."
|
"description": "Start normalized coordinates [start_x, start_y] in 0-1000 scale for precise, local interactions (e.g. adjusting sliders, SeekBars, fine range selection, or drag-and-drop)."
|
||||||
},
|
},
|
||||||
"end": {
|
"end": {
|
||||||
"type": "array",
|
"type": "array",
|
||||||
"items": {
|
"items": {
|
||||||
"type": "integer"
|
"type": "integer"
|
||||||
},
|
},
|
||||||
"description": "End normalized coordinates [end_x, end_y] in 0-1000 scale."
|
"description": "End normalized coordinates [end_x, end_y] in 0-1000 scale for precise, local interactions (e.g. adjusting sliders, SeekBars, fine range selection, or drag-and-drop)."
|
||||||
},
|
},
|
||||||
"target": {
|
"target": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"type": "integer"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "integer"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "string"
|
||||||
|
}
|
||||||
|
],
|
||||||
"description": "Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within."
|
"description": "Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within."
|
||||||
},
|
},
|
||||||
"action": {
|
"gesture": {
|
||||||
|
"anyOf": [
|
||||||
|
{
|
||||||
|
"type": "string",
|
||||||
|
"enum": [
|
||||||
|
"up",
|
||||||
|
"down",
|
||||||
|
"left",
|
||||||
|
"right"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "integer"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
"description": "Backward-compatible swipe gesture: smart direction string ('up', 'down', 'left', 'right') OR precise custom coordinates [start_x, start_y, end_x, end_y] in 0-1000 scale."
|
"description": "Backward-compatible swipe gesture: smart direction string ('up', 'down', 'left', 'right') OR precise custom coordinates [start_x, start_y, end_x, end_y] in 0-1000 scale."
|
||||||
},
|
},
|
||||||
"target_description": {
|
"target_description": {
|
||||||
"type": "string",
|
"type": "string",
|
||||||
"description": "What is being dragged, in a few words (e.g. 'brightness slider knob'). Required for coordinate gestures ('start'/'end' or a coordinates list); ignored for directional scrolling."
|
"description": "What is being dragged, in a few words (e.g. 'brightness slider knob'). REQUIRED for coordinate gestures ('start'/'end' or a coordinates list); ignored for directional scrolling."
|
||||||
},
|
},
|
||||||
"duration": {
|
"duration": {
|
||||||
"type": "integer",
|
"type": "integer",
|
||||||
"description": "Optional swipe/drag duration in milliseconds (default 800). For drag-and-drop, list reordering, or sliding/adjusting sliders (e.g., volume, brightness, SeekBars), set duration >= 1000 (e.g. 1500). If omitted for directional swipe, duration is computed automatically."
|
"description": "Optional swipe/drag duration in milliseconds (default 800; computed automatically for directional swipes)."
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
},
|
||||||
|
"required": []
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
@@ -193,12 +243,18 @@
|
|||||||
"type": "function",
|
"type": "function",
|
||||||
"function": {
|
"function": {
|
||||||
"name": "press_key",
|
"name": "press_key",
|
||||||
"description": "[ACTION] Press a physical or virtual system button. The screen after pressing will be returned automatically.",
|
"description": "[ACTION] Press a physical or virtual system button (e.g. ENTER, BACK, HOME, APP_SWITCH).",
|
||||||
"parameters": {
|
"parameters": {
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"properties": {
|
"properties": {
|
||||||
"key": {
|
"key": {
|
||||||
"type": "string",
|
"type": "string",
|
||||||
|
"enum": [
|
||||||
|
"ENTER",
|
||||||
|
"BACK",
|
||||||
|
"HOME",
|
||||||
|
"APP_SWITCH"
|
||||||
|
],
|
||||||
"description": "Standard Android system button name (ENTER, BACK, HOME, APP_SWITCH)."
|
"description": "Standard Android system button name (ENTER, BACK, HOME, APP_SWITCH)."
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
@@ -212,13 +268,17 @@
|
|||||||
"type": "function",
|
"type": "function",
|
||||||
"function": {
|
"function": {
|
||||||
"name": "manage_app",
|
"name": "manage_app",
|
||||||
"description": "[ACTION] Launch or force stop a specified application. The screen after launch/stop will be returned automatically.",
|
"description": "[ACTION] Launch or force stop a specified application.",
|
||||||
"parameters": {
|
"parameters": {
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"properties": {
|
"properties": {
|
||||||
"action": {
|
"action": {
|
||||||
"type": "string",
|
"type": "string",
|
||||||
"description": "The action type ('launch' or 'stop')."
|
"enum": [
|
||||||
|
"launch",
|
||||||
|
"stop"
|
||||||
|
],
|
||||||
|
"description": "The action type."
|
||||||
},
|
},
|
||||||
"app_name": {
|
"app_name": {
|
||||||
"type": "string",
|
"type": "string",
|
||||||
@@ -236,13 +296,13 @@
|
|||||||
"type": "function",
|
"type": "function",
|
||||||
"function": {
|
"function": {
|
||||||
"name": "wait_for_delay",
|
"name": "wait_for_delay",
|
||||||
"description": "[ACTION] Pause execution and wait for a specified duration in milliseconds. The screen after pause will be returned automatically.",
|
"description": "[ACTION] Pause execution and wait for a specified duration in milliseconds.\n\nUse this whenever you need time to elapse—whether for UI loading, animations,\nscreen transitions, or longer scheduled delays and intervals specified in the task.",
|
||||||
"parameters": {
|
"parameters": {
|
||||||
"type": "object",
|
"type": "object",
|
||||||
"properties": {
|
"properties": {
|
||||||
"time_in_ms": {
|
"time_in_ms": {
|
||||||
"type": "integer",
|
"type": "integer",
|
||||||
"description": "The exact duration to pause in milliseconds (e.g., 2000 for 2s, 60000 for 1 min, 180000 for 3 mins, 300000 for 5 mins). Convert any required waiting duration into milliseconds."
|
"description": "The exact duration to wait in milliseconds. Accurately convert the required time duration into milliseconds based on your objective or plan (e.g., 2000 for 2s, 5000 for 5s, 60000 for 1 minute, 180000 for 3 minutes, 300000 for 5 minutes)."
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"required": [
|
"required": [
|
||||||
|
|||||||
@@ -0,0 +1,302 @@
|
|||||||
|
# Copyright 2026 Google LLC
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
"""A Flash multi-action turn resolves every index against one element list.
|
||||||
|
|
||||||
|
Defect: each action's post-observation replaced ``state.indexed_elements``
|
||||||
|
and the executor resolved index targets against the state at call time, so
|
||||||
|
``click(3)`` then ``click(5)`` decided on one screen sent the second tap to
|
||||||
|
whatever the screen after the first click listed as ``[5]``. The runner now
|
||||||
|
snapshots the list once per turn and threads it through
|
||||||
|
``McpActionExecutor.execute(index_elements=...)``; the state's list is still
|
||||||
|
refreshed after every action (that is what the next observation shows).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from unittest.mock import AsyncMock, Mock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from langchain_core.messages import ToolMessage
|
||||||
|
|
||||||
|
from artemis.agents.flash.runner import FlashRunner, _TurnRecord
|
||||||
|
from artemis.context import ArtemisContext
|
||||||
|
from artemis.graph.state import State
|
||||||
|
from artemis.mcp.action_executor import McpActionExecutor, _ArgError
|
||||||
|
from artemis.mcp.action_types import ActionResult, ObserveResult
|
||||||
|
|
||||||
|
|
||||||
|
def _el(index: int, text: str, cx: int, cy: int) -> dict:
|
||||||
|
return {
|
||||||
|
"index": index,
|
||||||
|
"center": [cx, cy],
|
||||||
|
"text": text,
|
||||||
|
"bounds": [cx - 100, cy - 50, cx + 100, cy + 50],
|
||||||
|
"class": "android.widget.TextView",
|
||||||
|
"resource_id": f"id/{text.lower().replace(' ', '_')}",
|
||||||
|
"is_ocr": False,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# The list of the observation the model decided on (1080x2400 device).
|
||||||
|
ORIGINAL = [
|
||||||
|
_el(1, "Home", 540, 200),
|
||||||
|
_el(2, "Search", 540, 600),
|
||||||
|
_el(3, "Wi-Fi", 540, 1000),
|
||||||
|
_el(4, "Bluetooth", 540, 1400),
|
||||||
|
_el(5, "Display", 540, 1800),
|
||||||
|
]
|
||||||
|
# The screen after the first click: fewer elements, different numbering.
|
||||||
|
REFRESHED = [_el(1, "Back", 100, 100), _el(2, "Wi-Fi toggle", 900, 300)]
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeSession:
|
||||||
|
"""An action session whose observations replay a scripted list sequence."""
|
||||||
|
|
||||||
|
started = True
|
||||||
|
|
||||||
|
def __init__(self, observations: list[list[dict]]):
|
||||||
|
self.calls: list[tuple[str, dict]] = []
|
||||||
|
self._observations = list(observations)
|
||||||
|
|
||||||
|
async def call(self, name, args):
|
||||||
|
self.calls.append((name, dict(args)))
|
||||||
|
return ActionResult.success(name, f"{name} dispatched")
|
||||||
|
|
||||||
|
async def observe(self, settle_ms=0):
|
||||||
|
elements = self._observations.pop(0) if self._observations else []
|
||||||
|
return ObserveResult(
|
||||||
|
ok=True,
|
||||||
|
elements_text="\n".join(f"[{e['index']}] {e['text']}" for e in elements),
|
||||||
|
elements=elements,
|
||||||
|
hierarchy_ok=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def mock_context():
|
||||||
|
ctx = Mock(spec=ArtemisContext)
|
||||||
|
ctx.llm_config = Mock()
|
||||||
|
llm_cfg = Mock()
|
||||||
|
llm_cfg.model = "gemini-2.5-flash"
|
||||||
|
llm_cfg.temperature = 0.1
|
||||||
|
ctx.llm_config.get_agent.return_value = llm_cfg
|
||||||
|
ctx.device = Mock()
|
||||||
|
ctx.device.device_width = 1080
|
||||||
|
ctx.device.device_height = 2400
|
||||||
|
ctx.data_engine = None
|
||||||
|
ctx.adb_client = None
|
||||||
|
ctx.driver = Mock()
|
||||||
|
return ctx
|
||||||
|
|
||||||
|
|
||||||
|
def _runner(mock_context, observations):
|
||||||
|
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||||
|
runner = FlashRunner(mock_context, goal="Open Wi-Fi and Display")
|
||||||
|
runner.summarizer = None
|
||||||
|
runner.executor._session = _FakeSession(observations)
|
||||||
|
# No device: the executor's observation already refreshed the state.
|
||||||
|
runner._capture_post_screenshot = AsyncMock(return_value=None)
|
||||||
|
return runner
|
||||||
|
|
||||||
|
|
||||||
|
def _state(elements):
|
||||||
|
state = State(initial_goal="g")
|
||||||
|
state.indexed_elements = list(elements)
|
||||||
|
state.indexed_points = [e["center"] for e in elements]
|
||||||
|
return state
|
||||||
|
|
||||||
|
|
||||||
|
def _click(index, tc_id):
|
||||||
|
return {"name": "click", "args": {"target": index}, "id": tc_id}
|
||||||
|
|
||||||
|
|
||||||
|
# --- Executor: the explicit list wins over the state ---------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _executor():
|
||||||
|
ctx = Mock()
|
||||||
|
ctx.device.device_width = 1080
|
||||||
|
ctx.device.device_height = 2400
|
||||||
|
actuator = Mock()
|
||||||
|
actuator.controller = Mock()
|
||||||
|
return McpActionExecutor(ctx, actuator=actuator)
|
||||||
|
|
||||||
|
|
||||||
|
def test_resolve_index_uses_the_given_list_and_falls_back_to_the_state():
|
||||||
|
executor = _executor()
|
||||||
|
state = _state(REFRESHED)
|
||||||
|
|
||||||
|
# None -> the state's current list (Pro operator / validator / explorer callers).
|
||||||
|
point, recorded = executor._resolve_index(2, "click", state)
|
||||||
|
assert point == [833, 125] and recorded["target_text"] == "Wi-Fi toggle"
|
||||||
|
|
||||||
|
# An explicit list: index 5 exists only there.
|
||||||
|
point, recorded = executor._resolve_index(5, "click", state, index_elements=ORIGINAL)
|
||||||
|
assert point == [500, 750] and recorded["target_text"] == "Display"
|
||||||
|
assert recorded["target_resource_id"] == "id/display"
|
||||||
|
|
||||||
|
# The snapshot bounds the valid range, not the state.
|
||||||
|
with pytest.raises(_ArgError, match="Active index range is 1 to 2"):
|
||||||
|
executor._resolve_index(5, "click", state, index_elements=REFRESHED)
|
||||||
|
_, _, _, recorded = executor._translate(
|
||||||
|
"long_press", {"target": 4}, state, index_elements=ORIGINAL
|
||||||
|
)
|
||||||
|
assert recorded["target_text"] == "Bluetooth"
|
||||||
|
_, wire, _, recorded = executor._translate(
|
||||||
|
"input_text", {"target": 2, "text": "hi"}, state, index_elements=ORIGINAL
|
||||||
|
)
|
||||||
|
assert wire["target"] == [500, 250] and recorded["target_text"] == "Search"
|
||||||
|
|
||||||
|
|
||||||
|
# --- Runner: one snapshot per turn ------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_two_index_clicks_in_one_turn_resolve_against_the_original_list(mock_context):
|
||||||
|
"""``click(3)`` then ``click(5)``: the first click's observation lists only
|
||||||
|
two elements, yet the second index still lands on the original ``[5]``
|
||||||
|
(Display), with its semantics recorded from the original element."""
|
||||||
|
runner = _runner(mock_context, observations=[REFRESHED, [_el(1, "Display page", 1, 1)]])
|
||||||
|
state = _state(ORIGINAL)
|
||||||
|
messages: list = []
|
||||||
|
turn = _TurnRecord()
|
||||||
|
|
||||||
|
final, _, xml_list, _ = await runner._process_tool_calls(
|
||||||
|
[_click(3, "tc-1"), _click(5, "tc-2")],
|
||||||
|
state,
|
||||||
|
messages,
|
||||||
|
"thought",
|
||||||
|
{},
|
||||||
|
b"pre",
|
||||||
|
"xml",
|
||||||
|
0,
|
||||||
|
turn,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert final is None
|
||||||
|
session = runner.executor._session
|
||||||
|
assert [args["target"] for _, args in session.calls] == [[500, 417], [500, 750]]
|
||||||
|
assert [status for _, status, _ in turn.actions] == ["success", "success"]
|
||||||
|
tool_msgs = [m for m in messages if isinstance(m, ToolMessage)]
|
||||||
|
assert [m.status for m in tool_msgs] == ["success", "success"]
|
||||||
|
# The state kept being refreshed: the NEXT observation shows the newest list.
|
||||||
|
assert [e["text"] for e in state.indexed_elements] == ["Display page"]
|
||||||
|
assert xml_list == "[1] Display page"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_second_index_beyond_the_refreshed_list_is_still_valid(mock_context):
|
||||||
|
"""Without the snapshot ``click(5)`` after the refresh was an
|
||||||
|
'Active index range is 1 to 2' error; with it the turn's own list is
|
||||||
|
authoritative and the recorded metadata comes from that list."""
|
||||||
|
runner = _runner(mock_context, observations=[REFRESHED, REFRESHED])
|
||||||
|
state = _state(ORIGINAL)
|
||||||
|
captured = []
|
||||||
|
real_execute = runner.executor.execute
|
||||||
|
|
||||||
|
async def spy(name, args, tc_id, st, **kwargs):
|
||||||
|
result = await real_execute(name, args, tc_id, st, **kwargs)
|
||||||
|
captured.append(result)
|
||||||
|
return result
|
||||||
|
|
||||||
|
runner.executor.execute = spy
|
||||||
|
|
||||||
|
await runner._process_tool_calls(
|
||||||
|
[_click(4, "tc-1"), _click(5, "tc-2")],
|
||||||
|
state,
|
||||||
|
[],
|
||||||
|
"thought",
|
||||||
|
{},
|
||||||
|
b"pre",
|
||||||
|
"xml",
|
||||||
|
0,
|
||||||
|
_TurnRecord(),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert [r.status for r in captured] == ["success", "success"]
|
||||||
|
assert [r.metadata["target_coordinates"] for r in captured] == [[500, 583], [500, 750]]
|
||||||
|
assert [r.metadata["target_semantics"]["target_text"] for r in captured] == [
|
||||||
|
"Bluetooth",
|
||||||
|
"Display",
|
||||||
|
]
|
||||||
|
assert captured[1].metadata["target_semantics"]["target_bounds"] == [407, 729, 593, 771]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_explorer_candidates_registered_before_an_action_join_the_snapshot(mock_context):
|
||||||
|
"""``ask_explorer`` appends to the state's list in place; before any
|
||||||
|
action that list is the snapshot, so the new index resolves in the same
|
||||||
|
turn even though the action's observation then replaces the state list."""
|
||||||
|
runner = _runner(mock_context, observations=[REFRESHED])
|
||||||
|
state = _state(ORIGINAL)
|
||||||
|
|
||||||
|
async def fake_explorer(query, context_feedback, st):
|
||||||
|
st.indexed_points.append([540, 2200])
|
||||||
|
st.indexed_elements.append(_el(6, "Hidden gear", 540, 2200))
|
||||||
|
return "Found 'gear' at index 6.", True
|
||||||
|
|
||||||
|
runner.executor._ask_explorer = fake_explorer
|
||||||
|
|
||||||
|
await runner._process_tool_calls(
|
||||||
|
[
|
||||||
|
{"name": "ask_explorer", "args": {"query": "gear"}, "id": "tc-0"},
|
||||||
|
_click(6, "tc-1"),
|
||||||
|
],
|
||||||
|
state,
|
||||||
|
[],
|
||||||
|
"thought",
|
||||||
|
{},
|
||||||
|
b"pre",
|
||||||
|
"xml",
|
||||||
|
0,
|
||||||
|
_TurnRecord(),
|
||||||
|
)
|
||||||
|
|
||||||
|
session = runner.executor._session
|
||||||
|
assert [args["target"] for _, args in session.calls] == [[500, 917]]
|
||||||
|
assert [e["text"] for e in state.indexed_elements] == ["Back", "Wi-Fi toggle"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_explorer_after_an_action_moves_the_snapshot_to_the_list_it_numbered(mock_context):
|
||||||
|
"""After an action replaced the state list, the explorer numbers its
|
||||||
|
candidates against that replaced list; the snapshot follows so the index
|
||||||
|
the explorer just handed out resolves correctly."""
|
||||||
|
runner = _runner(mock_context, observations=[REFRESHED, REFRESHED])
|
||||||
|
state = _state(ORIGINAL)
|
||||||
|
|
||||||
|
async def fake_explorer(query, context_feedback, st):
|
||||||
|
st.indexed_points.append([200, 2000])
|
||||||
|
st.indexed_elements.append(_el(len(st.indexed_elements) + 1, "Gear", 200, 2000))
|
||||||
|
return "Found.", True
|
||||||
|
|
||||||
|
runner.executor._ask_explorer = fake_explorer
|
||||||
|
|
||||||
|
await runner._process_tool_calls(
|
||||||
|
[
|
||||||
|
_click(3, "tc-1"),
|
||||||
|
{"name": "ask_explorer", "args": {"query": "gear"}, "id": "tc-2"},
|
||||||
|
_click(3, "tc-3"), # the explorer's index on the refreshed list
|
||||||
|
],
|
||||||
|
state,
|
||||||
|
[],
|
||||||
|
"thought",
|
||||||
|
{},
|
||||||
|
b"pre",
|
||||||
|
"xml",
|
||||||
|
0,
|
||||||
|
_TurnRecord(),
|
||||||
|
)
|
||||||
|
|
||||||
|
session = runner.executor._session
|
||||||
|
assert [args["target"] for _, args in session.calls] == [[500, 417], [185, 833]]
|
||||||
@@ -13,8 +13,8 @@
|
|||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
"""§1.3 assembly discipline: prompts are functions of configuration x scenario.
|
"""§1.3 assembly discipline: prompts are functions of configuration x scenario.
|
||||||
Both check gates off => zero context pollution; a plan carrying check lines
|
Both check gates off => zero context pollution; with a gate on, the check-line
|
||||||
gets the behavior explainer regardless of switches."""
|
guidance lives once in the static system prompt, worded for that gate."""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -23,10 +23,10 @@ from unittest.mock import MagicMock
|
|||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from artemis.agents.operator.prompts import (
|
from artemis.agents.operator.prompts import (
|
||||||
CheckItemsExplainerPromptComponent,
|
|
||||||
FeedbackPromptComponent,
|
FeedbackPromptComponent,
|
||||||
PromptBuilder,
|
PromptBuilder,
|
||||||
TemplatePromptComponent,
|
TemplatePromptComponent,
|
||||||
|
render_transcript_static_system,
|
||||||
)
|
)
|
||||||
from artemis.agents.planner.planner import build_planner_system_blocks
|
from artemis.agents.planner.planner import build_planner_system_blocks
|
||||||
from artemis.context import ArtemisContext, ExecutionSetup
|
from artemis.context import ArtemisContext, ExecutionSetup
|
||||||
@@ -95,35 +95,47 @@ def _state(**overrides):
|
|||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_explainer_renders_iff_plan_has_check_lines(tmp_path):
|
async def test_check_line_guidance_lives_in_the_static_prompt_once():
|
||||||
component = CheckItemsExplainerPromptComponent()
|
"""The check-line guidance is a static-prompt section gated on the check
|
||||||
|
gates (never a per-turn observation block), worded for the active gate."""
|
||||||
|
off = await _render_main_template(disable_checker=True)
|
||||||
|
assert "*Check Lines*" not in off
|
||||||
|
assert "never declare a check passed" not in off
|
||||||
|
|
||||||
# Plan without check lines -> nothing rendered
|
final_only = await _render_main_template(disable_midway_checks=True, disable_final_check=False)
|
||||||
builder = PromptBuilder()
|
assert final_only.count("*Check Lines*") == 1
|
||||||
ctx = _mock_ctx(tmp_path, plan="- [ ] G\n")
|
assert "never declare a check passed" in final_only
|
||||||
await component(builder, _state(), ctx)
|
assert "restores deleted ones unless user guidance called for the change" in final_only
|
||||||
assert builder.human_parts == []
|
# No midway repair loop -> no repair budget to recite, and the grammar says so.
|
||||||
|
assert "reopens its milestone at most" not in final_only
|
||||||
|
assert "there is no midway repair loop" in final_only
|
||||||
|
|
||||||
# Plan with check lines -> explainer rendered
|
midway = await _render_main_template(disable_midway_checks=False, checkpoint_max_repairs=3)
|
||||||
builder2 = PromptBuilder()
|
assert midway.count("*Check Lines*") == 1
|
||||||
ctx2 = _mock_ctx(tmp_path, plan="- [ ] G\n - verify: V\n")
|
assert "reopens its milestone at most 3 times" in midway
|
||||||
await component(builder2, _state(), ctx2)
|
assert "- finding:" in midway
|
||||||
joined = "\n".join(p for p in builder2.human_parts if isinstance(p, str))
|
|
||||||
assert "check lines" in joined
|
|
||||||
assert "never declare a check passed yourself" in joined
|
def test_transcript_static_system_recites_the_repair_budget_from_setup():
|
||||||
assert "automatically restored" in joined
|
ctx = _mock_ctx(disable_midway_checks=False, checkpoint_max_repairs=5)
|
||||||
|
ctx.actuator = None
|
||||||
|
prompts = json.loads(OPERATOR_JSON.read_text(encoding="utf-8"))
|
||||||
|
text = render_transcript_static_system(prompts, ctx, _state(initial_goal="G"))
|
||||||
|
assert text.count("*Check Lines*") == 1
|
||||||
|
assert "reopens its milestone at most 5 times" in text
|
||||||
|
|
||||||
|
quiet = _mock_ctx(disable_checker=True)
|
||||||
|
quiet.actuator = None
|
||||||
|
assert "*Check Lines*" not in render_transcript_static_system(
|
||||||
|
prompts, quiet, _state(initial_goal="G")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_explainer_is_content_driven_not_switch_driven(tmp_path):
|
async def test_static_prompt_declares_the_user_guidance_channel_once():
|
||||||
"""A resumed old plan with check lines still gets the explanation even when
|
text = await _render_main_template(disable_checker=True)
|
||||||
every check switch is off."""
|
assert text.count("--- User Guidance ---") == 1
|
||||||
component = CheckItemsExplainerPromptComponent()
|
assert "outranks the task plan and its check lines" in text
|
||||||
builder = PromptBuilder()
|
|
||||||
ctx = _mock_ctx(tmp_path, plan="- [ ] G\n - assert: A\n", disable_checker=True)
|
|
||||||
await component(builder, _state(), ctx)
|
|
||||||
joined = "\n".join(p for p in builder.human_parts if isinstance(p, str))
|
|
||||||
assert "check lines" in joined
|
|
||||||
|
|
||||||
|
|
||||||
async def _render_main_template(**setup_kwargs) -> str:
|
async def _render_main_template(**setup_kwargs) -> str:
|
||||||
|
|||||||
@@ -400,7 +400,7 @@ async def test_replay_steps_result_is_a_plain_tool_message(tmp_path):
|
|||||||
assert tool_msg.name == "replay_steps"
|
assert tool_msg.name == "replay_steps"
|
||||||
assert "- **Step 3 (Start: 9.0s)**" in tool_msg.content
|
assert "- **Step 3 (Start: 9.0s)**" in tool_msg.content
|
||||||
assert "[Screen]: Saved the alarm." in tool_msg.content
|
assert "[Screen]: Saved the alarm." in tool_msg.content
|
||||||
assert "[Planned Action]: Tapped 'Save' at [0.5, 0.5]" in tool_msg.content
|
assert "[Action]: Tapped 'Save' at [0.5, 0.5]" in tool_msg.content
|
||||||
|
|
||||||
|
|
||||||
# --- Streamed-reasoning transcript (replay of the live Thought/Work text) -------------
|
# --- Streamed-reasoning transcript (replay of the live Thought/Work text) -------------
|
||||||
@@ -647,6 +647,89 @@ def test_format_history_marks_self_described_targets_only():
|
|||||||
assert "target_resource_id" not in text
|
assert "target_resource_id" not in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_format_history_uses_session_offsets_and_skips_empty_steps():
|
||||||
|
"""Rows carry the session-relative ``T+mm:ss`` clock every agent shares
|
||||||
|
(from the step timestamp, else from the engine's ``97.7s`` label), and a
|
||||||
|
step with neither summary nor action renders no row."""
|
||||||
|
from artemis.agents.checker.checker import _format_history
|
||||||
|
|
||||||
|
ctx = _mock_ctx()
|
||||||
|
ctx.data_engine.session_start_time = 1000.0
|
||||||
|
ctx.data_engine.get_agent_friendly_steps.return_value = [
|
||||||
|
{"step_number": 1, "timestamp": 1097.7, "summary": "Opened the app"},
|
||||||
|
{"step_number": 2, "relative_time": "12.4s", "summary": "", "action_taken": None},
|
||||||
|
{
|
||||||
|
"step_number": 3,
|
||||||
|
"relative_time": "130.0s",
|
||||||
|
"summary": None,
|
||||||
|
"action_taken": {"action": "click", "coordinates": [1, 2], "target_text": "Login"},
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
lines = _format_history(ctx).splitlines()
|
||||||
|
assert len(lines) == 2
|
||||||
|
assert lines[0] == "- Step 1 (T+01:37): Opened the app"
|
||||||
|
assert lines[1].startswith("- Step 3 (T+02:10): ")
|
||||||
|
assert "'Login'" in lines[1]
|
||||||
|
|
||||||
|
|
||||||
|
async def _final_check_human_text(ctx) -> str:
|
||||||
|
report = CheckReport(verdicts=[])
|
||||||
|
structured = MagicMock()
|
||||||
|
structured.ainvoke = AsyncMock(return_value=report)
|
||||||
|
llm = MagicMock()
|
||||||
|
llm.with_structured_output.return_value = structured
|
||||||
|
response = MagicMock()
|
||||||
|
response.tool_calls = []
|
||||||
|
acomplete_mock = AsyncMock(return_value=response)
|
||||||
|
|
||||||
|
with (
|
||||||
|
patch("artemis.agents.checker.checker.get_llm", return_value=llm),
|
||||||
|
patch("artemis.agents.checker.checker.acomplete", new=acomplete_mock),
|
||||||
|
patch(
|
||||||
|
"artemis.agents.checker.checker._capture_final_screen",
|
||||||
|
new=AsyncMock(return_value=(None, "elements")),
|
||||||
|
),
|
||||||
|
):
|
||||||
|
await run_final_check(ctx, goal="the goal", plan_text="- [x] G", ledger=[], check_items=[])
|
||||||
|
|
||||||
|
messages = acomplete_mock.await_args.args[1]
|
||||||
|
human = messages[-1]
|
||||||
|
return "\n".join(
|
||||||
|
block["text"] for block in human.content if isinstance(block, dict) and "text" in block
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_final_review_lists_user_guidance_from_the_step_records():
|
||||||
|
"""Every mid-run user instruction reaches the final review, read from the
|
||||||
|
step it was stamped on, so the goal is audited as amended by it."""
|
||||||
|
ctx = _mock_ctx()
|
||||||
|
ctx.data_engine.get_agent_friendly_steps.return_value = [
|
||||||
|
{"step_number": 1, "relative_time": "3.0s", "summary": "s1", "extra_metadata": {}},
|
||||||
|
{
|
||||||
|
"step_number": 2,
|
||||||
|
"relative_time": "65.0s",
|
||||||
|
"summary": "s2",
|
||||||
|
"extra_metadata": {"injected_instruction": "Skip the login"},
|
||||||
|
},
|
||||||
|
]
|
||||||
|
text = await _final_check_human_text(ctx)
|
||||||
|
section = text.split("# User guidance received during the run\n", 1)[1].split("\n\n", 1)[0]
|
||||||
|
assert section == '- Step 2 (T+01:05): "Skip the login"'
|
||||||
|
# The system prompt audits the goal as amended by that guidance.
|
||||||
|
from artemis.agents.checker.checker import _load_prompts
|
||||||
|
|
||||||
|
assert "as amended by the user guidance" in _load_prompts()["final_guide"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_final_review_reports_no_guidance_explicitly():
|
||||||
|
ctx = _mock_ctx()
|
||||||
|
text = await _final_check_human_text(ctx)
|
||||||
|
assert "# User guidance received during the run\n(none)\n" in text
|
||||||
|
|
||||||
|
|
||||||
# --- tool results: status is structural, never sniffed from the words -----------------
|
# --- tool results: status is structural, never sniffed from the words -----------------
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -332,3 +332,39 @@ async def test_run_tool_message_status_is_structural(
|
|||||||
assert messages[0].tool_call_id == "c1"
|
assert messages[0].tool_call_id == "c1"
|
||||||
assert messages[0].status == expected_status
|
assert messages[0].status == expected_status
|
||||||
assert messages[0].content == str(result)
|
assert messages[0].content == str(result)
|
||||||
|
|
||||||
|
|
||||||
|
# --- submit_answer rendering ------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_submit_answer_renders_numbered_steps(mock_context):
|
||||||
|
agent = Diagnoser(mock_context)
|
||||||
|
call = {
|
||||||
|
"name": "submit_answer",
|
||||||
|
"args": {
|
||||||
|
"analysis": "Airplane mode blocks the toggle.",
|
||||||
|
"actionable_steps": ["Turn off airplane mode", "Retry the toggle"],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
outcome = agent._handle_submit_call(call, [])
|
||||||
|
|
||||||
|
assert outcome == (
|
||||||
|
"Analysis: Airplane mode blocks the toggle.\n"
|
||||||
|
"Actionable steps:\n"
|
||||||
|
"1. Turn off airplane mode\n"
|
||||||
|
"2. Retry the toggle"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_submit_answer_without_steps_omits_section(mock_context):
|
||||||
|
agent = Diagnoser(mock_context)
|
||||||
|
call = {
|
||||||
|
"name": "submit_answer",
|
||||||
|
"args": {"analysis": "Nothing to fix.", "actionable_steps": []},
|
||||||
|
}
|
||||||
|
|
||||||
|
outcome = agent._handle_submit_call(call, [])
|
||||||
|
|
||||||
|
assert outcome == "Analysis: Nothing to fix."
|
||||||
|
assert "Actionable" not in outcome
|
||||||
@@ -154,7 +154,14 @@ def test_render_burst_abort_and_exec_error():
|
|||||||
|
|
||||||
plain = _disappeared(kind=KIND_EXEC_ERROR, category="general")
|
plain = _disappeared(kind=KIND_EXEC_ERROR, category="general")
|
||||||
text = render_execution_incident(plain, STEPS)
|
text = render_execution_incident(plain, STEPS)
|
||||||
assert "was dispatched, but the device/executor reported" in text
|
# A failed action is quoted as an intent, never as a done outcome, and
|
||||||
|
# the sentence does not claim it was dispatched.
|
||||||
|
assert (
|
||||||
|
"your planned action `tap 'Skip' at [950, 288]` could not be executed; the"
|
||||||
|
" device/executor reported:" in text
|
||||||
|
)
|
||||||
|
assert "was dispatched" not in text
|
||||||
|
assert "`Tapped 'Skip'" not in text
|
||||||
|
|
||||||
|
|
||||||
def test_render_without_history_omits_the_trigger_hint():
|
def test_render_without_history_omits_the_trigger_hint():
|
||||||
@@ -179,15 +186,17 @@ async def test_component_renders_only_while_an_incident_is_open():
|
|||||||
builder, SimpleNamespace(open_incident=_disappeared()), ctx, steps=STEPS
|
builder, SimpleNamespace(open_incident=_disappeared()), ctx, steps=STEPS
|
||||||
)
|
)
|
||||||
assert len(builder.human_parts) == 1
|
assert len(builder.human_parts) == 1
|
||||||
# Its own text part: the transcript scrub keeps it verbatim across turns.
|
|
||||||
assert builder.human_parts[0].startswith(EXECUTION_INCIDENT_MARKER)
|
assert builder.human_parts[0].startswith(EXECUTION_INCIDENT_MARKER)
|
||||||
|
# Re-rendered every turn while open, so older copies leave the history.
|
||||||
|
assert builder.ephemeral_indices == [0]
|
||||||
|
|
||||||
|
|
||||||
def test_render_closed_notice_settles_the_intent():
|
def test_render_closed_notice_settles_the_intent():
|
||||||
closed = dict(_disappeared(), closed_at_step=15)
|
closed = dict(_disappeared(), closed_at_step=15)
|
||||||
text = render_closed_incident(closed)
|
text = render_closed_incident(closed)
|
||||||
assert text.startswith("--- Execution Incident (CLOSED at Step 15) ---")
|
assert text.startswith("--- Execution Incident (CLOSED at Step 15) ---")
|
||||||
assert "opened at Step 12 on `Tapped 'Skip' at [950, 288]`" in text
|
# The blocked action is quoted in intent form: it never happened.
|
||||||
|
assert "opened at Step 12 on `tap 'Skip' at [950, 288]`" in text
|
||||||
assert "already served, still pending, or no longer needed" in text
|
assert "already served, still pending, or no longer needed" in text
|
||||||
|
|
||||||
|
|
||||||
@@ -201,6 +210,7 @@ async def test_component_renders_closed_notice_once_no_incident_is_open():
|
|||||||
await ExecutionIncidentPromptComponent()(builder, state, ctx, steps=STEPS)
|
await ExecutionIncidentPromptComponent()(builder, state, ctx, steps=STEPS)
|
||||||
assert len(builder.human_parts) == 1
|
assert len(builder.human_parts) == 1
|
||||||
assert builder.human_parts[0].startswith("--- Execution Incident (CLOSED at Step 15) ---")
|
assert builder.human_parts[0].startswith("--- Execution Incident (CLOSED at Step 15) ---")
|
||||||
|
assert builder.ephemeral_indices == [0]
|
||||||
|
|
||||||
# An open incident takes precedence over a stale closed notice.
|
# An open incident takes precedence over a stale closed notice.
|
||||||
builder = PromptBuilder()
|
builder = PromptBuilder()
|
||||||
|
|||||||
@@ -14,13 +14,18 @@
|
|||||||
|
|
||||||
"""Unit tests for Universal FlashRunner."""
|
"""Unit tests for Universal FlashRunner."""
|
||||||
|
|
||||||
from unittest.mock import Mock, patch
|
import base64
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from unittest.mock import AsyncMock, Mock, patch
|
||||||
import pytest
|
import pytest
|
||||||
from langchain_core.messages import HumanMessage, ToolMessage
|
from langchain_core.messages import HumanMessage, ToolMessage
|
||||||
|
|
||||||
from artemis.agents.explorer.constants import ASK_EXPLORER_DESCRIPTION
|
from artemis.agents.explorer.constants import ASK_EXPLORER_DESCRIPTION
|
||||||
from artemis.agents.flash.runner import FlashRunner
|
from artemis.agents.flash.runner import FlashRunner
|
||||||
from artemis.agents.validator.tool_declarations import ASK_EXPLORER_TOOL
|
from artemis.agents.validator.tool_declarations import (
|
||||||
|
ASK_EXPLORER_TOOL,
|
||||||
|
ToolExecutionResult,
|
||||||
|
)
|
||||||
from artemis.context import ArtemisContext
|
from artemis.context import ArtemisContext
|
||||||
|
|
||||||
|
|
||||||
@@ -57,6 +62,138 @@ def test_flash_runner_tools_initialization(mock_context):
|
|||||||
assert "report_task_status" in tool_names
|
assert "report_task_status" in tool_names
|
||||||
# Validator's report_failure_analysis should NOT be in FlashRunner
|
# Validator's report_failure_analysis should NOT be in FlashRunner
|
||||||
assert "report_failure_analysis" not in tool_names
|
assert "report_failure_analysis" not in tool_names
|
||||||
|
# Flash has no note-writing tool and its prompt never teaches notes:
|
||||||
|
# the Validator set's note readers are not bound.
|
||||||
|
assert "read_note" not in tool_names
|
||||||
|
assert "list_notes" not in tool_names
|
||||||
|
assert tool_names[:2] == ["click", "click_sequence"]
|
||||||
|
|
||||||
|
|
||||||
|
# --- Post-action screenshot fallback (failed action / unknown tool) ----------------
|
||||||
|
|
||||||
|
|
||||||
|
def _screen(elements):
|
||||||
|
return SimpleNamespace(
|
||||||
|
base64=base64.b64encode(b"POST-IMG").decode("ascii"),
|
||||||
|
elements=elements,
|
||||||
|
width=1080,
|
||||||
|
height=2400,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _failed_click(tc_id="tc1"):
|
||||||
|
return ToolExecutionResult(
|
||||||
|
tool_call_id=tc_id,
|
||||||
|
tool_name="click",
|
||||||
|
status="error",
|
||||||
|
text_summary="Error during click: Invalid target format: 3",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fallback_observation_renders_the_indexed_ui_list(mock_context, tmp_path):
|
||||||
|
"""After an action that produced no observation, the fallback capture must
|
||||||
|
hand the next tail the same indexed minimal list the success path builds
|
||||||
|
(never the raw element dict repr) and refresh the state like it."""
|
||||||
|
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||||
|
runner = FlashRunner(mock_context, goal="g")
|
||||||
|
runner.controller = Mock()
|
||||||
|
runner.controller.get_screen_data = AsyncMock(
|
||||||
|
return_value=_screen(
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"text": "Settings",
|
||||||
|
"content-desc": "",
|
||||||
|
"bounds": "[40,120][1040,270]",
|
||||||
|
"class": "android.widget.TextView",
|
||||||
|
"resource-id": "android:id/title",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
)
|
||||||
|
)
|
||||||
|
state = Mock()
|
||||||
|
state.latest_screenshot = "old.jpg"
|
||||||
|
exec_result = _failed_click()
|
||||||
|
|
||||||
|
with patch("artemis.mcp.observation.get_temp_dir", return_value=tmp_path):
|
||||||
|
post = await runner._capture_post_screenshot(
|
||||||
|
exec_result, "click", frozenset({"click"}), state
|
||||||
|
)
|
||||||
|
|
||||||
|
assert post == b"POST-IMG"
|
||||||
|
assert isinstance(exec_result.ui_elements_text, str)
|
||||||
|
assert exec_result.ui_elements_text == "[1] Text: 'Settings' | Bounds: [37,50][962,112]"
|
||||||
|
assert "{'text'" not in exec_result.ui_elements_text
|
||||||
|
# State refreshed exactly as the executor's success path does.
|
||||||
|
assert state.indexed_elements[0]["index"] == 1
|
||||||
|
assert state.indexed_elements[0]["text"] == "Settings"
|
||||||
|
assert state.indexed_points == [state.indexed_elements[0]["center"]]
|
||||||
|
assert state.latest_screenshot != "old.jpg"
|
||||||
|
assert state.latest_screenshot.startswith(str(tmp_path))
|
||||||
|
assert exec_result.screenshot_path == state.latest_screenshot
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fallback_observation_is_skipped_for_non_action_tools(mock_context):
|
||||||
|
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||||
|
runner = FlashRunner(mock_context, goal="g")
|
||||||
|
runner.controller = Mock()
|
||||||
|
runner.controller.get_screen_data = AsyncMock(return_value=_screen([]))
|
||||||
|
state = Mock()
|
||||||
|
exec_result = ToolExecutionResult(
|
||||||
|
tool_call_id="tc", tool_name="ask_explorer", status="success", text_summary="found"
|
||||||
|
)
|
||||||
|
|
||||||
|
post = await runner._capture_post_screenshot(
|
||||||
|
exec_result, "ask_explorer", frozenset({"click"}), state
|
||||||
|
)
|
||||||
|
|
||||||
|
assert post is None
|
||||||
|
assert exec_result.ui_elements_text is None
|
||||||
|
runner.controller.get_screen_data.assert_not_awaited()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fallback_observation_keeps_the_executors_ui_list(mock_context, tmp_path):
|
||||||
|
"""An action that reported a UI list but no screenshot keeps its own list."""
|
||||||
|
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||||
|
runner = FlashRunner(mock_context, goal="g")
|
||||||
|
runner.controller = Mock()
|
||||||
|
runner.controller.get_screen_data = AsyncMock(
|
||||||
|
return_value=_screen([{"text": "Other", "bounds": "[0,0][100,100]"}])
|
||||||
|
)
|
||||||
|
exec_result = ToolExecutionResult(
|
||||||
|
tool_call_id="tc",
|
||||||
|
tool_name="click",
|
||||||
|
status="success",
|
||||||
|
text_summary="click executed",
|
||||||
|
ui_elements_text="[1] Text: 'Mine' | Bounds: [0,0][10,10]",
|
||||||
|
)
|
||||||
|
|
||||||
|
with patch("artemis.mcp.observation.get_temp_dir", return_value=tmp_path):
|
||||||
|
post = await runner._capture_post_screenshot(
|
||||||
|
exec_result, "click", frozenset({"click"}), None
|
||||||
|
)
|
||||||
|
|
||||||
|
assert post == b"POST-IMG"
|
||||||
|
assert exec_result.ui_elements_text == "[1] Text: 'Mine' | Bounds: [0,0][10,10]"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fallback_observation_failure_is_contained(mock_context):
|
||||||
|
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||||
|
runner = FlashRunner(mock_context, goal="g")
|
||||||
|
runner.controller = Mock()
|
||||||
|
runner.controller.get_screen_data = AsyncMock(side_effect=RuntimeError("adb gone"))
|
||||||
|
state = Mock()
|
||||||
|
state.latest_screenshot = "old.jpg"
|
||||||
|
exec_result = _failed_click()
|
||||||
|
|
||||||
|
post = await runner._capture_post_screenshot(exec_result, "click", frozenset({"click"}), state)
|
||||||
|
|
||||||
|
assert post is None
|
||||||
|
assert exec_result.ui_elements_text is None
|
||||||
|
assert state.latest_screenshot == "old.jpg"
|
||||||
|
|
||||||
|
|
||||||
def test_flash_runner_executor_follows_the_flash_explorer_tier(mock_context):
|
def test_flash_runner_executor_follows_the_flash_explorer_tier(mock_context):
|
||||||
@@ -202,10 +339,12 @@ def test_flash_records_render_in_the_ledger_with_their_arguments(mock_context):
|
|||||||
typed = _record(runner, "input_text", {"target": [500, 600], "text": "hello"})
|
typed = _record(runner, "input_text", {"target": [500, 600], "text": "hello"})
|
||||||
assert format_actions_clean(typed) == "Inputted 'hello' into field at [500, 600]"
|
assert format_actions_clean(typed) == "Inputted 'hello' into field at [500, 600]"
|
||||||
|
|
||||||
|
# A directional swipe names the direction first; the recorded canonical
|
||||||
|
# path is the parenthesised detail.
|
||||||
swipe_dir = _record(runner, "swipe", {"direction": "up"})
|
swipe_dir = _record(runner, "swipe", {"direction": "up"})
|
||||||
assert format_actions_clean(swipe_dir) == "Swiped up"
|
assert format_actions_clean(swipe_dir) == "Swiped up (from [600, 700] to [600, 300])"
|
||||||
swipe_fa = _record(runner, "swipe", {"action": "down"})
|
swipe_fa = _record(runner, "swipe", {"action": "down"})
|
||||||
assert format_actions_clean(swipe_fa) == "Swiped down"
|
assert format_actions_clean(swipe_fa) == "Swiped down (from [600, 300] to [600, 700])"
|
||||||
swipe_coords = _record(runner, "swipe", {"coordinates": [556, 289, 556, 124]})
|
swipe_coords = _record(runner, "swipe", {"coordinates": [556, 289, 556, 124]})
|
||||||
assert format_actions_clean(swipe_coords) == "Swiped from [556, 289] to [556, 124]"
|
assert format_actions_clean(swipe_coords) == "Swiped from [556, 289] to [556, 124]"
|
||||||
|
|
||||||
@@ -246,3 +385,174 @@ def test_flash_records_the_executors_target_semantics(mock_context):
|
|||||||
bare = _record(runner, "click", {"target": [320, 399]}, metadata={"target_semantics": {}})
|
bare = _record(runner, "click", {"target": [320, 399]}, metadata={"target_semantics": {}})
|
||||||
assert "target_description" not in bare
|
assert "target_description" not in bare
|
||||||
assert "target_text" not in bare
|
assert "target_text" not in bare
|
||||||
|
|
||||||
|
|
||||||
|
def test_flash_records_an_index_target_as_the_resolved_point_with_observed_semantics(
|
||||||
|
mock_context,
|
||||||
|
):
|
||||||
|
"""An element index is not a coordinate: the record carries the normalized
|
||||||
|
point the executor resolved it to, plus the element's observed fields in
|
||||||
|
the shape Pro records (never a self-described label)."""
|
||||||
|
from artemis.utils.task_tree import format_actions_clean
|
||||||
|
|
||||||
|
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||||
|
runner = FlashRunner(mock_context, goal="g")
|
||||||
|
|
||||||
|
click = _record(
|
||||||
|
runner,
|
||||||
|
"click",
|
||||||
|
{"target": 3},
|
||||||
|
metadata={
|
||||||
|
"target_coordinates": [500, 520],
|
||||||
|
"target_semantics": {
|
||||||
|
"target_text": "Wi-Fi",
|
||||||
|
"target_bounds": [37, 480, 962, 560],
|
||||||
|
"target_resource_id": "android:id/title",
|
||||||
|
"target_class": "android.widget.TextView",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert click["coordinates"] == [500, 520]
|
||||||
|
assert click["normalized_coordinates"] == [500, 520]
|
||||||
|
assert click["target_text"] == "Wi-Fi"
|
||||||
|
assert click["target_bounds"] == [37, 480, 962, 560]
|
||||||
|
assert "target_description" not in click
|
||||||
|
assert click["args"] == {"target": 3} # the tool call itself, verbatim
|
||||||
|
assert format_actions_clean(click) == "Tapped 'Wi-Fi' at [500, 520]"
|
||||||
|
|
||||||
|
typed = _record(
|
||||||
|
runner,
|
||||||
|
"input_text",
|
||||||
|
{"target": 2, "text": "hello"},
|
||||||
|
metadata={"target_coordinates": [500, 300], "target_semantics": {"target_text": "Search"}},
|
||||||
|
)
|
||||||
|
assert typed["coordinates"] == [500, 300]
|
||||||
|
assert format_actions_clean(typed) == "Inputted 'hello' into 'Search' at [500, 300]"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_visual_lens_receives_the_recorded_action_shape(mock_context):
|
||||||
|
"""The lens gets the recorded action minus its verb (what the Pro
|
||||||
|
summarizer hands it), so ``manage_app``'s own ``action`` argument can no
|
||||||
|
longer overwrite the action name in the rendered phrase."""
|
||||||
|
from artemis.agents.flash.runner import _TurnRecord
|
||||||
|
from artemis.agents.flash.summarizer import VisualStepSummarizer
|
||||||
|
|
||||||
|
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||||
|
runner = FlashRunner(mock_context, goal="g")
|
||||||
|
runner.summarizer = Mock()
|
||||||
|
runner.executor.execute = AsyncMock(
|
||||||
|
return_value=ToolExecutionResult(
|
||||||
|
tool_call_id="tc",
|
||||||
|
tool_name="manage_app",
|
||||||
|
status="success",
|
||||||
|
text_summary="Launched app 'Settings' (com.android.settings); foreground confirmed.",
|
||||||
|
screenshot_bytes=b"post",
|
||||||
|
metadata={"target_semantics": {}, "target_coordinates": None},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
args = {"action": "launch", "app_name": "Settings"}
|
||||||
|
|
||||||
|
await runner._execute_and_record_action(
|
||||||
|
"manage_app", args, "tc", Mock(), [], "thought", {}, b"pre", "xml", 0, _TurnRecord()
|
||||||
|
)
|
||||||
|
|
||||||
|
runner.summarizer.dispatch.assert_called_once()
|
||||||
|
kwargs = runner.summarizer.dispatch.call_args.kwargs
|
||||||
|
assert kwargs["action_name"] == "manage_app"
|
||||||
|
assert "action" not in kwargs["action_args"]
|
||||||
|
assert kwargs["action_args"]["args"] == args
|
||||||
|
assert VisualStepSummarizer._action_phrase("manage_app", kwargs["action_args"]) == (
|
||||||
|
"Launched app 'Settings'"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- Native thinking (thought summaries) reach the step record ----------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _thinking_response(content):
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
return SimpleNamespace(content=content, tool_calls=[])
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_response_thinking_reads_only_thinking_blocks(mock_context):
|
||||||
|
"""``include_thoughts`` replies carry ``thinking`` blocks next to ``text``
|
||||||
|
blocks; only the former are native thinking, and plain-string replies
|
||||||
|
carry none."""
|
||||||
|
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||||
|
runner = FlashRunner(mock_context, goal="g")
|
||||||
|
|
||||||
|
blocks = [
|
||||||
|
{"type": "thinking", "thinking": "**Verifying**\n\nMaps is open."},
|
||||||
|
{"type": "text", "text": "Tapping the search bar."},
|
||||||
|
{"type": "thinking", "thinking": " "},
|
||||||
|
]
|
||||||
|
assert runner._extract_response_thinking(_thinking_response(blocks)) == (
|
||||||
|
"**Verifying**\n\nMaps is open."
|
||||||
|
)
|
||||||
|
assert runner._extract_response_text(_thinking_response(blocks)) == "Tapping the search bar."
|
||||||
|
assert runner._extract_response_thinking(_thinking_response("plain reply")) == ""
|
||||||
|
assert (
|
||||||
|
runner._extract_response_thinking(_thinking_response([{"type": "text", "text": "t"}])) == ""
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_record_action_step_persists_native_thinking(mock_context):
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||||
|
runner = FlashRunner(mock_context, goal="g")
|
||||||
|
runner.ctx.data_engine = Mock()
|
||||||
|
runner.ctx.data_engine.current_step_id = None
|
||||||
|
runner.ctx.data_engine.record_step.return_value = "step-1"
|
||||||
|
exec_result = SimpleNamespace(
|
||||||
|
status="success", text_summary="click executed", metadata={}, ui_elements_text="ui"
|
||||||
|
)
|
||||||
|
|
||||||
|
runner._record_action_step(
|
||||||
|
"click",
|
||||||
|
{"target": [1, 2]},
|
||||||
|
exec_result,
|
||||||
|
"thought",
|
||||||
|
{},
|
||||||
|
b"pre",
|
||||||
|
["xml"],
|
||||||
|
b"post",
|
||||||
|
native_text="native summary",
|
||||||
|
)
|
||||||
|
kwargs = runner.ctx.data_engine.record_step.call_args.kwargs
|
||||||
|
assert kwargs["operator_raw_thinking"] == "thought"
|
||||||
|
assert kwargs["operator_native_thinking"] == "native summary"
|
||||||
|
|
||||||
|
runner._record_action_step(
|
||||||
|
"click", {"target": [1, 2]}, exec_result, "thought", {}, b"pre", ["xml"], b"post"
|
||||||
|
)
|
||||||
|
assert runner.ctx.data_engine.record_step.call_args.kwargs["operator_native_thinking"] is None
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_final_report_persists_native_thinking(mock_context):
|
||||||
|
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||||
|
runner = FlashRunner(mock_context, goal="g")
|
||||||
|
runner.ctx.data_engine = Mock()
|
||||||
|
runner.ctx.data_engine.current_step_id = None
|
||||||
|
runner.summarizer = None
|
||||||
|
messages = []
|
||||||
|
|
||||||
|
report = await runner._finalize_task_report(
|
||||||
|
"report_task_status",
|
||||||
|
{"status": "completed"},
|
||||||
|
"tc",
|
||||||
|
"final text",
|
||||||
|
{},
|
||||||
|
b"pre",
|
||||||
|
["xml"],
|
||||||
|
messages,
|
||||||
|
native_text="native summary",
|
||||||
|
)
|
||||||
|
assert report == {"status": "completed"}
|
||||||
|
kwargs = runner.ctx.data_engine.record_step.call_args.kwargs
|
||||||
|
assert kwargs["operator_raw_thinking"] == "final text"
|
||||||
|
assert kwargs["operator_native_thinking"] == "native summary"
|
||||||
|
assert isinstance(messages[-1], ToolMessage)
|
||||||
@@ -17,33 +17,42 @@
|
|||||||
Covers: the ``video_analyzer`` availability gate (same as the Pro graph) and
|
Covers: the ``video_analyzer`` availability gate (same as the Pro graph) and
|
||||||
its prompt segment, the unbounded-by-default turn limit, the Pro-shaped
|
its prompt segment, the unbounded-by-default turn limit, the Pro-shaped
|
||||||
observation tail (``# CURRENT OBSERVATION [T+mm:ss]`` header, screenshot,
|
observation tail (``# CURRENT OBSERVATION [T+mm:ss]`` header, screenshot,
|
||||||
``--- Visible UI Elements ---`` list), and the end-to-end loop: committed
|
``--- Visible UI Elements ---`` list, ephemeral per-turn notices, the
|
||||||
turns carry text-only tool messages plus an ``--- Action Execution Result
|
reasoning reminder after a silent turn), and the end-to-end loop: committed
|
||||||
(T+mm:ss) ---`` message, every recorded step of a multi-action turn is
|
turns carry text-only tool messages and no execution-result message unless
|
||||||
registered on the committed turn, the final-turn tool restriction only
|
an action failed, every recorded step of a multi-action turn is registered
|
||||||
applies to bounded loops, and a no-tool-call turn is nudged, not terminated.
|
on the committed turn, the final-turn tool restriction only applies to
|
||||||
|
bounded loops, and a no-tool-call turn is nudged, not terminated.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import base64
|
import base64
|
||||||
import re
|
import re
|
||||||
from unittest.mock import AsyncMock, Mock, patch
|
from unittest.mock import AsyncMock, Mock, PropertyMock, patch
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage, ToolMessage
|
from langchain_core.messages import AIMessage, HumanMessage, SystemMessage, ToolMessage
|
||||||
|
|
||||||
from artemis.agents.flash.runner import FlashRunner, _TurnRecord
|
from artemis.agents.flash.runner import _FINAL_TURN_WARNING, FlashRunner, _TurnRecord
|
||||||
|
from artemis.agents.operator.prompts import (
|
||||||
|
REASONING_REMINDER,
|
||||||
|
USER_GUIDANCE_MARKER,
|
||||||
|
UserGuidance,
|
||||||
|
render_user_guidance,
|
||||||
|
render_user_instruction,
|
||||||
|
)
|
||||||
from artemis.agents.validator.tool_declarations import ToolExecutionResult
|
from artemis.agents.validator.tool_declarations import ToolExecutionResult
|
||||||
from artemis.context import ArtemisContext
|
from artemis.context import ArtemisContext
|
||||||
from artemis.graph.state import State
|
from artemis.graph.state import State
|
||||||
from artemis.memory.transcript import (
|
from artemis.memory.transcript import (
|
||||||
|
EPHEMERAL_BLOCKS_KEY,
|
||||||
EXECUTION_RESULT_MARKER,
|
EXECUTION_RESULT_MARKER,
|
||||||
PRO_UI_LIST_MARKER,
|
PRO_UI_LIST_MARKER,
|
||||||
TranscriptLedger,
|
TranscriptLedger,
|
||||||
)
|
)
|
||||||
|
|
||||||
OBSERVATION_HEADER_RE = re.compile(r"^# CURRENT OBSERVATION \[T\+\d{2,}:\d{2}\]$")
|
OBSERVATION_HEADER_RE = re.compile(r"^# CURRENT OBSERVATION \[T\+\d{2,}:\d{2}\]$")
|
||||||
RESULT_RE = re.compile(
|
FAILED_RESULT_RE = re.compile(
|
||||||
rf"^{re.escape(EXECUTION_RESULT_MARKER)} \(T\+\d{{2,}}:\d{{2}}\) ---\nStatus: dispatched$"
|
rf"^{re.escape(EXECUTION_RESULT_MARKER)} \(T\+\d{{2,}}:\d{{2}}\) ---\nStatus: failed\n"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -173,31 +182,139 @@ def test_turn_limit_semantics(mock_context):
|
|||||||
|
|
||||||
|
|
||||||
def test_observation_tail_has_pro_shape(mock_context):
|
def test_observation_tail_has_pro_shape(mock_context):
|
||||||
|
"""The objective lives in the system prompt only; the tail opens with the
|
||||||
|
observation header on turn 1 exactly as on every later turn, and carries
|
||||||
|
no reminder unless the previous turn was silent."""
|
||||||
with patch("artemis.controllers.unified_controller.get_driver"):
|
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||||
runner = FlashRunner(mock_context, goal="Open Settings")
|
runner = FlashRunner(mock_context, goal="Open Settings")
|
||||||
ledger = TranscriptLedger()
|
ledger = TranscriptLedger()
|
||||||
|
|
||||||
tail = runner._build_tail(ledger, 1, b"IMG", "[1] Settings")
|
tail = runner._build_tail(ledger, 1, b"IMG", "[1] Settings")
|
||||||
texts = [b["text"] for b in tail.content if b["type"] == "text"]
|
texts = [b["text"] for b in tail.content if b["type"] == "text"]
|
||||||
assert texts[0] == "Your objective is: Open Settings"
|
assert OBSERVATION_HEADER_RE.match(texts[0])
|
||||||
assert OBSERVATION_HEADER_RE.match(texts[1])
|
assert texts[1] == "--- Current Screenshot ---"
|
||||||
assert texts[2] == "--- Current Screenshot ---"
|
assert texts[2] == f"{PRO_UI_LIST_MARKER}\n[1] Settings"
|
||||||
assert texts[3] == f"{PRO_UI_LIST_MARKER}\n[1] Settings"
|
assert len(texts) == 3
|
||||||
assert sum(1 for b in tail.content if b["type"] == "image_url") == 1
|
assert sum(1 for b in tail.content if b["type"] == "image_url") == 1
|
||||||
|
joined = "".join(texts)
|
||||||
|
assert "Open Settings" not in joined and "objective" not in joined.lower()
|
||||||
|
assert "CRITICAL RULE" not in joined and REASONING_REMINDER not in joined
|
||||||
|
assert EPHEMERAL_BLOCKS_KEY not in tail.additional_kwargs
|
||||||
|
|
||||||
later = runner._build_tail(
|
later = runner._build_tail(
|
||||||
ledger, 5, None, None, injected="[INJECTED] stop", notices=["nudge"], is_final=True
|
ledger,
|
||||||
|
5,
|
||||||
|
None,
|
||||||
|
None,
|
||||||
|
injected=UserGuidance(body='User instruction: "stop"', wrapper="[INJECTED]"),
|
||||||
|
notices=["nudge"],
|
||||||
|
is_final=True,
|
||||||
)
|
)
|
||||||
texts = [b["text"] for b in later.content]
|
texts = [b["text"] for b in later.content]
|
||||||
assert OBSERVATION_HEADER_RE.match(texts[0])
|
assert OBSERVATION_HEADER_RE.match(texts[0])
|
||||||
assert "Your objective is" not in "".join(texts)
|
assert "objective" not in "".join(texts).lower()
|
||||||
assert texts[1:] == [
|
assert texts[1:] == [
|
||||||
"nudge",
|
"nudge",
|
||||||
"[INJECTED] stop",
|
"[INJECTED]",
|
||||||
"[WARNING] This is your final turn; only 'report_task_status' is available.",
|
'User instruction: "stop"',
|
||||||
|
_FINAL_TURN_WARNING,
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_per_turn_notices_are_marked_ephemeral(mock_context):
|
||||||
|
"""Nudges, the user-guidance wrapper, the final-turn warning and the
|
||||||
|
reasoning reminder are only meaningful for the turn they were built for:
|
||||||
|
every one of them is flagged ephemeral (by block index) so the scrub edge
|
||||||
|
drops them before the message freezes; the observation blocks never are."""
|
||||||
|
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||||
|
runner = FlashRunner(mock_context, goal="Open Settings")
|
||||||
|
ledger = TranscriptLedger()
|
||||||
|
guidance = render_user_guidance("stop", release_loop=False, has_plan=False)
|
||||||
|
|
||||||
|
tail = runner._build_tail(
|
||||||
|
ledger,
|
||||||
|
3,
|
||||||
|
b"IMG",
|
||||||
|
"[1] Settings",
|
||||||
|
notices=["nudge"],
|
||||||
|
injected=guidance,
|
||||||
|
is_final=True,
|
||||||
|
previous_turn_silent=True,
|
||||||
|
)
|
||||||
|
texts = [b.get("text") for b in tail.content]
|
||||||
|
# header, screenshot label, image, UI list, then the per-turn notices with
|
||||||
|
# the persistent instruction body right after its wrapper
|
||||||
|
assert texts[4:] == [
|
||||||
|
"nudge",
|
||||||
|
guidance.wrapper,
|
||||||
|
guidance.body,
|
||||||
|
_FINAL_TURN_WARNING,
|
||||||
|
REASONING_REMINDER,
|
||||||
|
]
|
||||||
|
assert tail.additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [4, 5, 7, 8]
|
||||||
|
|
||||||
|
|
||||||
|
def test_injected_instruction_body_outlives_its_wrapper(mock_context):
|
||||||
|
"""Defect: a standing instruction ("don't send messages from now on")
|
||||||
|
vanished after one turn because the whole guidance block was ephemeral.
|
||||||
|
Now the wrapper is ephemeral and the verbatim body is a regular block, so
|
||||||
|
it stays in the active window until the turn is chunked."""
|
||||||
|
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||||
|
runner = FlashRunner(mock_context, goal="g")
|
||||||
|
ledger = TranscriptLedger()
|
||||||
|
guidance = render_user_guidance(
|
||||||
|
"don't send messages from now on", has_plan=False, offset_label="T+01:05"
|
||||||
|
)
|
||||||
|
|
||||||
|
tail = runner._build_tail(ledger, 2, b"IMG", "[1] Send", injected=guidance)
|
||||||
|
texts = [b.get("text") for b in tail.content]
|
||||||
|
wrapper_index = texts.index(guidance.wrapper)
|
||||||
|
body_index = texts.index(guidance.body)
|
||||||
|
ephemeral = tail.additional_kwargs[EPHEMERAL_BLOCKS_KEY]
|
||||||
|
assert wrapper_index in ephemeral
|
||||||
|
assert body_index not in ephemeral
|
||||||
|
assert body_index == wrapper_index + 1
|
||||||
|
assert guidance.body == 'User instruction (T+01:05): "don\'t send messages from now on"'
|
||||||
|
assert guidance.body == render_user_instruction(
|
||||||
|
"don't send messages from now on", offset_label="T+01:05"
|
||||||
|
)
|
||||||
|
# The wrapper frames the quoted line but no longer carries the words.
|
||||||
|
assert guidance.wrapper.startswith(USER_GUIDANCE_MARKER)
|
||||||
|
assert "don't send messages" not in guidance.wrapper
|
||||||
|
|
||||||
|
# Scrub edge parity: once the turn has left the live position (a newer
|
||||||
|
# committed observation exists) the wrapper is gone and the body is
|
||||||
|
# still in the committed turn.
|
||||||
|
ledger.stage_turn([tail, AIMessage(content="ok")])
|
||||||
|
ledger.commit_staged(step_key="s1")
|
||||||
|
ledger.stage_turn([runner._build_tail(ledger, 3, b"IMG", "[1] Send"), AIMessage(content="ok")])
|
||||||
|
ledger.commit_staged(step_key="s2")
|
||||||
|
rendered = ledger.render([runner._build_tail(ledger, 4, b"IMG", "[1] Send")])
|
||||||
|
committed = next(
|
||||||
|
m
|
||||||
|
for m in rendered
|
||||||
|
if isinstance(m, HumanMessage) and guidance.body in [b.get("text") for b in m.content]
|
||||||
|
)
|
||||||
|
committed_texts = [b.get("text", "") for b in committed.content if isinstance(b, dict)]
|
||||||
|
assert guidance.wrapper not in committed_texts
|
||||||
|
assert not any(t.startswith(USER_GUIDANCE_MARKER) for t in committed_texts)
|
||||||
|
|
||||||
|
|
||||||
|
def test_reasoning_reminder_follows_a_silent_turn_only(mock_context):
|
||||||
|
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||||
|
runner = FlashRunner(mock_context, goal="g")
|
||||||
|
ledger = TranscriptLedger()
|
||||||
|
|
||||||
|
quiet = runner._build_tail(ledger, 2, None, "[1] x", previous_turn_silent=False)
|
||||||
|
assert REASONING_REMINDER not in [b["text"] for b in quiet.content]
|
||||||
|
|
||||||
|
reminded = runner._build_tail(ledger, 2, None, "[1] x", previous_turn_silent=True)
|
||||||
|
texts = [b["text"] for b in reminded.content]
|
||||||
|
assert texts[-1] == REASONING_REMINDER
|
||||||
|
assert texts.count(REASONING_REMINDER) == 1
|
||||||
|
assert reminded.additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [len(texts) - 1]
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Injected instruction: verbatim text for the record and the lens, wrapped
|
# Injected instruction: verbatim text for the record and the lens, wrapped
|
||||||
# notice for the observation tail only
|
# notice for the observation tail only
|
||||||
@@ -209,8 +326,9 @@ async def test_injected_instruction_splits_verbatim_text_from_operator_notice(
|
|||||||
mock_context, tmp_path
|
mock_context, tmp_path
|
||||||
):
|
):
|
||||||
"""The user's words are what the step record, the chunk ledger and the
|
"""The user's words are what the step record, the chunk ledger and the
|
||||||
visual lens receive (Pro parity); the 'You MUST immediately ...' directive
|
visual lens receive (Pro parity); the shared ``--- User Guidance ---``
|
||||||
is operator-facing and rides on the observation tail alone."""
|
block (the Pro rendering, without a plan to edit) rides on the
|
||||||
|
observation tail alone."""
|
||||||
import json
|
import json
|
||||||
|
|
||||||
(tmp_path / "injected_instruction.json").write_text(
|
(tmp_path / "injected_instruction.json").write_text(
|
||||||
@@ -225,11 +343,15 @@ async def test_injected_instruction_splits_verbatim_text_from_operator_notice(
|
|||||||
instruction, notice = await runner._read_injected_instruction()
|
instruction, notice = await runner._read_injected_instruction()
|
||||||
|
|
||||||
assert instruction == "Skip the popup and log in"
|
assert instruction == "Skip the popup and log in"
|
||||||
assert notice.startswith(
|
assert notice == render_user_guidance(
|
||||||
"[REAL-TIME INJECTED INSTRUCTION from user]: Skip the popup and log in"
|
"Skip the popup and log in", release_loop=True, has_plan=False
|
||||||
)
|
)
|
||||||
assert "You MUST immediately follow this instruction" in notice
|
assert notice.wrapper.startswith(USER_GUIDANCE_MARKER)
|
||||||
assert "authorized stopping any ongoing monitoring loop" in notice
|
assert notice.body == 'User instruction: "Skip the popup and log in"'
|
||||||
|
assert "outranks your current milestones" in notice.wrapper
|
||||||
|
assert "task plan" not in notice.wrapper
|
||||||
|
assert "authorized stopping any ongoing monitoring loop" in notice.wrapper
|
||||||
|
assert "REAL-TIME INJECTED" not in notice.wrapper and "You MUST" not in notice.wrapper
|
||||||
assert not (tmp_path / "injected_instruction.json").exists()
|
assert not (tmp_path / "injected_instruction.json").exists()
|
||||||
|
|
||||||
with patch("artemis.controllers.unified_controller.get_driver"):
|
with patch("artemis.controllers.unified_controller.get_driver"):
|
||||||
@@ -267,7 +389,7 @@ def _make_runner(mock_context, responses, max_turns=None):
|
|||||||
|
|
||||||
runner._invoke_model = AsyncMock(side_effect=_invoke)
|
runner._invoke_model = AsyncMock(side_effect=_invoke)
|
||||||
runner.executor.execute = AsyncMock(
|
runner.executor.execute = AsyncMock(
|
||||||
side_effect=lambda name, args, tc_id, state: _exec_result(
|
side_effect=lambda name, args, tc_id, state, **_: _exec_result(
|
||||||
tc_id, name, f"IMG-{tc_id}".encode()
|
tc_id, name, f"IMG-{tc_id}".encode()
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
@@ -289,9 +411,10 @@ _INITIAL_OBSERVATION = ("shot0.png", b"IMG0", "[1] Settings")
|
|||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_run_builds_prompt_from_ledger_with_session_offsets(mock_context):
|
async def test_run_builds_prompt_from_ledger_with_session_offsets(mock_context):
|
||||||
"""Turn 2 must see: the static system prefix, the committed turn 1 (its
|
"""Turn 2 must see: the static system prefix, the committed turn 1 (its
|
||||||
tail, the AI message, text-only tool messages and an execution-result
|
tail, the AI message, text-only tool messages that already carry each
|
||||||
message with a ``T+`` offset) and a fresh observation tail carrying the
|
action's outcome — no execution-result message repeats them) and a fresh
|
||||||
last action's post screenshot — the Pro transcript shape, with no cap."""
|
observation tail carrying the last action's post screenshot — the Pro
|
||||||
|
transcript shape, with no cap."""
|
||||||
responses = [
|
responses = [
|
||||||
AIMessage(
|
AIMessage(
|
||||||
content="I will tap then wait.",
|
content="I will tap then wait.",
|
||||||
@@ -327,33 +450,146 @@ async def test_run_builds_prompt_from_ledger_with_session_offsets(mock_context):
|
|||||||
tail1 = turn2_messages[1]
|
tail1 = turn2_messages[1]
|
||||||
assert isinstance(tail1, HumanMessage)
|
assert isinstance(tail1, HumanMessage)
|
||||||
tail1_texts = [b["text"] for b in tail1.content if b["type"] == "text"]
|
tail1_texts = [b["text"] for b in tail1.content if b["type"] == "text"]
|
||||||
assert tail1_texts[0] == "Your objective is: Open Settings"
|
assert OBSERVATION_HEADER_RE.match(tail1_texts[0])
|
||||||
assert OBSERVATION_HEADER_RE.match(tail1_texts[1])
|
assert "objective" not in "".join(tail1_texts).lower()
|
||||||
|
assert "Open Settings" in turn2_messages[0].content # stated once, in the prefix
|
||||||
|
|
||||||
assert isinstance(turn2_messages[2], AIMessage)
|
assert isinstance(turn2_messages[2], AIMessage)
|
||||||
tool_msgs = [m for m in turn2_messages if isinstance(m, ToolMessage)]
|
tool_msgs = [m for m in turn2_messages if isinstance(m, ToolMessage)]
|
||||||
assert [m.content for m in tool_msgs] == ["click executed", "wait_for_delay executed"]
|
assert [m.content for m in tool_msgs] == ["click executed", "wait_for_delay executed"]
|
||||||
|
|
||||||
result_msg = turn2_messages[5]
|
# Every action succeeded: the tool messages are the outcome, nothing repeats them.
|
||||||
assert isinstance(result_msg, HumanMessage)
|
assert not any(
|
||||||
assert RESULT_RE.match(result_msg.content[0]["text"])
|
isinstance(m, HumanMessage)
|
||||||
|
and any(EXECUTION_RESULT_MARKER in b.get("text", "") for b in m.content)
|
||||||
|
for m in turn2_messages
|
||||||
|
)
|
||||||
|
|
||||||
tail2 = turn2_messages[6]
|
tail2 = turn2_messages[5]
|
||||||
tail2_texts = [b["text"] for b in tail2.content if b["type"] == "text"]
|
tail2_texts = [b["text"] for b in tail2.content if b["type"] == "text"]
|
||||||
assert OBSERVATION_HEADER_RE.match(tail2_texts[0])
|
assert OBSERVATION_HEADER_RE.match(tail2_texts[0])
|
||||||
assert "Your objective is" not in "".join(tail2_texts)
|
assert "objective" not in "".join(tail2_texts).lower()
|
||||||
images = [b for b in tail2.content if b["type"] == "image_url"]
|
images = [b for b in tail2.content if b["type"] == "image_url"]
|
||||||
assert len(images) == 1
|
assert len(images) == 1
|
||||||
assert images[0]["image_url"]["url"].endswith(base64.b64encode(b"IMG-tc2").decode())
|
assert images[0]["image_url"]["url"].endswith(base64.b64encode(b"IMG-tc2").decode())
|
||||||
assert tail2_texts[1] == "--- Current Screenshot ---"
|
assert tail2_texts[1] == "--- Current Screenshot ---"
|
||||||
assert tail2_texts[2] == f"{PRO_UI_LIST_MARKER}\n[1] next"
|
assert tail2_texts[2] == f"{PRO_UI_LIST_MARKER}\n[1] next"
|
||||||
assert len(turn2_messages) == 7
|
assert len(turn2_messages) == 6
|
||||||
|
|
||||||
# Both actions of the multi-action turn were registered on the committed turn.
|
# Both actions of the multi-action turn were registered on the committed turn.
|
||||||
ledger = mock_context.transcript_ledger
|
ledger = mock_context.transcript_ledger
|
||||||
assert ledger.unchunked_turns()[0]["step_keys"] == ["tc1", "tc2"]
|
assert ledger.unchunked_turns()[0]["step_keys"] == ["tc1", "tc2"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_run_failed_action_turn_ends_with_the_error_result(mock_context):
|
||||||
|
"""A failed action is the one case worth a result message: it carries the
|
||||||
|
error the turn ended on, after the tool message."""
|
||||||
|
responses = [
|
||||||
|
AIMessage(
|
||||||
|
content="Tapping.",
|
||||||
|
tool_calls=[{"name": "click", "args": {"target": 99}, "id": "tc1"}],
|
||||||
|
),
|
||||||
|
_report(),
|
||||||
|
]
|
||||||
|
with (
|
||||||
|
patch("artemis.controllers.unified_controller.get_driver"),
|
||||||
|
patch(
|
||||||
|
"artemis.agents.flash.runner.capture_screenshot_and_parse_ui",
|
||||||
|
AsyncMock(return_value=_INITIAL_OBSERVATION),
|
||||||
|
),
|
||||||
|
):
|
||||||
|
runner = _make_runner(mock_context, responses, max_turns=0)
|
||||||
|
runner.executor.execute = AsyncMock(
|
||||||
|
return_value=ToolExecutionResult(
|
||||||
|
tool_call_id="tc1",
|
||||||
|
tool_name="click",
|
||||||
|
status="error",
|
||||||
|
text_summary="Error during click: Invalid target index 99.",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
runner._capture_post_screenshot = AsyncMock(return_value=None)
|
||||||
|
await runner.run(State(initial_goal="Open Settings"))
|
||||||
|
|
||||||
|
turn2_messages = runner.calls[1]["messages"]
|
||||||
|
result_msgs = [
|
||||||
|
m
|
||||||
|
for m in turn2_messages
|
||||||
|
if isinstance(m, HumanMessage)
|
||||||
|
and any(EXECUTION_RESULT_MARKER in b.get("text", "") for b in m.content)
|
||||||
|
]
|
||||||
|
assert len(result_msgs) == 1
|
||||||
|
text = result_msgs[0].content[0]["text"]
|
||||||
|
assert FAILED_RESULT_RE.match(text)
|
||||||
|
assert "Invalid target index 99" in text
|
||||||
|
assert isinstance(turn2_messages[turn2_messages.index(result_msgs[0]) - 1], ToolMessage)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_run_reminds_after_a_silent_turn_without_a_bounce(mock_context):
|
||||||
|
"""The ledger judges the committed turn (sibling logic, authoritative
|
||||||
|
here); the runner only reads ``last_turn_silent`` when it builds the
|
||||||
|
next tail: one reminder block, marked ephemeral, one model call per turn."""
|
||||||
|
responses = [
|
||||||
|
AIMessage(
|
||||||
|
content="", # bare tool call, no visible reasoning
|
||||||
|
tool_calls=[
|
||||||
|
{"name": "click", "args": {"target": 1}, "id": "tc1"},
|
||||||
|
],
|
||||||
|
),
|
||||||
|
_report(),
|
||||||
|
]
|
||||||
|
with (
|
||||||
|
patch("artemis.controllers.unified_controller.get_driver"),
|
||||||
|
patch(
|
||||||
|
"artemis.agents.flash.runner.capture_screenshot_and_parse_ui",
|
||||||
|
AsyncMock(return_value=_INITIAL_OBSERVATION),
|
||||||
|
),
|
||||||
|
patch.object(TranscriptLedger, "last_turn_silent", new_callable=PropertyMock) as silent,
|
||||||
|
):
|
||||||
|
silent.side_effect = lambda: silent.call_count > 1 # turn 1: False, later: True
|
||||||
|
runner = _make_runner(mock_context, responses, max_turns=0)
|
||||||
|
await runner.run(State(initial_goal="Open Settings"))
|
||||||
|
|
||||||
|
assert runner._invoke_model.await_count == 2
|
||||||
|
tail1 = runner.calls[0]["messages"][-1]
|
||||||
|
assert REASONING_REMINDER not in [b.get("text") for b in tail1.content]
|
||||||
|
tail2 = runner.calls[1]["messages"][-1]
|
||||||
|
texts = [b.get("text") for b in tail2.content]
|
||||||
|
assert texts.count(REASONING_REMINDER) == 1
|
||||||
|
assert tail2.additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [texts.index(REASONING_REMINDER)]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_run_calibrates_the_ledger_only_from_measured_usage(mock_context):
|
||||||
|
"""A provider-reported prompt size calibrates the chars-per-token ratio
|
||||||
|
with the sent messages; an estimated size (no usage metadata) records the
|
||||||
|
base but never calibrates — the estimate is itself derived from character
|
||||||
|
counts, so calibrating on it would be circular."""
|
||||||
|
measured = AIMessage(
|
||||||
|
content="Tapping.",
|
||||||
|
tool_calls=[{"name": "click", "args": {"target": 1}, "id": "tc1"}],
|
||||||
|
usage_metadata={"input_tokens": 1200, "output_tokens": 5, "total_tokens": 1205},
|
||||||
|
)
|
||||||
|
responses = [measured, _report()]
|
||||||
|
with (
|
||||||
|
patch("artemis.controllers.unified_controller.get_driver"),
|
||||||
|
patch(
|
||||||
|
"artemis.agents.flash.runner.capture_screenshot_and_parse_ui",
|
||||||
|
AsyncMock(return_value=_INITIAL_OBSERVATION),
|
||||||
|
),
|
||||||
|
patch.object(TranscriptLedger, "record_prompt_tokens") as record,
|
||||||
|
):
|
||||||
|
runner = _make_runner(mock_context, responses, max_turns=0)
|
||||||
|
await runner.run(State(initial_goal="Open Settings"))
|
||||||
|
|
||||||
|
assert record.call_count == 2
|
||||||
|
first_call, second_call = record.call_args_list
|
||||||
|
assert first_call.args[0] == 1200 and first_call.kwargs["messages"] is not None
|
||||||
|
# The report turn carries no usage metadata: estimated, recorded, not calibrated.
|
||||||
|
assert second_call.args[0] >= 1 and second_call.kwargs["messages"] is None
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_run_final_turn_restricts_tools_and_fails_without_report(mock_context):
|
async def test_run_final_turn_restricts_tools_and_fails_without_report(mock_context):
|
||||||
responses = [AIMessage(content="I give up.", tool_calls=[])]
|
responses = [AIMessage(content="I give up.", tool_calls=[])]
|
||||||
@@ -411,11 +647,14 @@ def test_turn_record_result_is_none_for_helper_only_turns():
|
|||||||
assert _TurnRecord().result() is None
|
assert _TurnRecord().result() is None
|
||||||
|
|
||||||
|
|
||||||
def test_turn_record_result_is_dispatched_when_every_action_was_accepted():
|
def test_turn_record_result_is_none_when_every_action_succeeded():
|
||||||
|
"""The tool messages already say ``Swiped up.``; a ``Status: dispatched``
|
||||||
|
line after them would only duplicate the outcome (and put two human
|
||||||
|
messages back to back)."""
|
||||||
record = _TurnRecord(
|
record = _TurnRecord(
|
||||||
actions=[("click", "success", "Tapped at [500, 500]"), ("swipe", "success", "Swiped up.")]
|
actions=[("click", "success", "Tapped at [500, 500]"), ("swipe", "success", "Swiped up.")]
|
||||||
)
|
)
|
||||||
assert record.result() == {"status": "dispatched"}
|
assert record.result() is None
|
||||||
|
|
||||||
|
|
||||||
def test_turn_record_failed_result_keeps_an_executor_message_that_names_the_action():
|
def test_turn_record_failed_result_keeps_an_executor_message_that_names_the_action():
|
||||||
|
|||||||
@@ -12,13 +12,11 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
"""Unit tests for the Flash scrub-edge compressor (history redesign §3.2, M1).
|
"""Tests for screenshot summaries, text cleanup, and frozen history.
|
||||||
|
|
||||||
Covers the M1 acceptance items: the legacy-equivalent success-path product
|
Covers pending-summary recovery and expiry, failed-summary placeholders,
|
||||||
(asserted directly since the legacy ``compress_flash_messages`` reference was
|
independent text and image depths, ephemeral-block remapping, and stable
|
||||||
removed in M5), the three race regressions (grace recovery, grace exhaustion
|
content after both cleanup passes.
|
||||||
without backfill, failed-summary placeholder), the freeze invariant (bytes
|
|
||||||
behind the scrub edge never change), and depth-1 XML stripping equivalence.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
@@ -83,9 +81,10 @@ def test_success_path_product_behind_the_scrub_edge(mock_context):
|
|||||||
by the ``--- Historical Visual Transition ---`` summary block.
|
by the ``--- Historical Visual Transition ---`` summary block.
|
||||||
|
|
||||||
(Asserted directly; the legacy ``compress_flash_messages`` reference this
|
(Asserted directly; the legacy ``compress_flash_messages`` reference this
|
||||||
was originally diffed against was removed in M5.) Intended M1 timing
|
was originally diffed against was removed in M5.) The swap happens at
|
||||||
change: the swap happens at depth K, so the newest K-1 historical steps
|
depth K, so the newest K-1 historical steps still carry their screenshots;
|
||||||
still carry their screenshots.
|
their UI lists leave earlier, at the shallow text edge (depth 1), so only
|
||||||
|
the live observation ever carries an indexed element list.
|
||||||
"""
|
"""
|
||||||
summarizer = VisualStepSummarizer(mock_context)
|
summarizer = VisualStepSummarizer(mock_context)
|
||||||
for i in range(1, 7):
|
for i in range(1, 7):
|
||||||
@@ -120,8 +119,8 @@ def test_success_path_product_behind_the_scrub_edge(mock_context):
|
|||||||
assert _has_image(new_messages[6])
|
assert _has_image(new_messages[6])
|
||||||
assert "Tree 6" in str(new_messages[6].content)
|
assert "Tree 6" in str(new_messages[6].content)
|
||||||
|
|
||||||
# Intended timing change: step 5 (depth 2 < K) keeps its screenshot under
|
# Step 5 (depth 2 < K) is past the text edge only: screenshot intact,
|
||||||
# the scrub edge; its XML was still stripped at depth 1.
|
# UI list already gone, summary not yet swapped in.
|
||||||
assert _has_image(new_messages[5])
|
assert _has_image(new_messages[5])
|
||||||
assert "Visual summary 5." not in str(new_messages[5].content)
|
assert "Visual summary 5." not in str(new_messages[5].content)
|
||||||
assert "Tree 5" not in str(new_messages[5].content)
|
assert "Tree 5" not in str(new_messages[5].content)
|
||||||
@@ -249,10 +248,10 @@ def test_freeze_invariant_bytes_never_change(mock_context):
|
|||||||
assert "Late summary 2." not in " ".join(str(m.content) for m in messages)
|
assert "Late summary 2." not in " ".join(str(m.content) for m in messages)
|
||||||
|
|
||||||
|
|
||||||
def test_xml_depth1_strip_matches_legacy_semantics(mock_context):
|
def test_strip_at_the_edge_keeps_combined_block_prefix_and_live_list(mock_context):
|
||||||
"""Depth-1 XML stripping keeps combined-block prefixes and the live list."""
|
"""The XML strip at the edge keeps combined-block prefixes and the live list."""
|
||||||
messages: list = []
|
messages: list = []
|
||||||
compressor = ScrubEdgeCompressor(summarizer=None, prune_history_xml=True)
|
compressor = ScrubEdgeCompressor(summarizer=None, prune_history_xml=True, image_scrub_depth=2)
|
||||||
|
|
||||||
combined = ToolMessage(
|
combined = ToolMessage(
|
||||||
tool_call_id="tc1",
|
tool_call_id="tc1",
|
||||||
@@ -267,13 +266,44 @@ def test_xml_depth1_strip_matches_legacy_semantics(mock_context):
|
|||||||
)
|
)
|
||||||
_simulate_turns(compressor, messages, [combined, _step(2)])
|
_simulate_turns(compressor, messages, [combined, _step(2)])
|
||||||
|
|
||||||
# Historical combined block: prefix retained, tree stripped (the exact
|
# Combined block at the edge: prefix retained, tree stripped (the exact
|
||||||
# legacy ``compress_flash_messages`` product for this input).
|
# legacy ``compress_flash_messages`` product for this input).
|
||||||
assert messages[0].content[0] == {"type": "text", "text": "Tapped Settings."}
|
assert messages[0].content[0] == {"type": "text", "text": "Tapped Settings."}
|
||||||
# Live observation keeps its UI list untouched.
|
# Live observation keeps its UI list untouched.
|
||||||
assert "Tree 2" in str(messages[1].content)
|
assert "Tree 2" in str(messages[1].content)
|
||||||
|
|
||||||
|
|
||||||
|
def test_imageless_ui_list_carrier_is_stripped_as_soon_as_a_newer_observation_exists(
|
||||||
|
mock_context,
|
||||||
|
):
|
||||||
|
"""A step whose screenshot failed still carries a UI list; it follows the
|
||||||
|
text edge like any observation: while it is the newest observation it keeps
|
||||||
|
the list, and it is stripped as soon as a newer observation exists — even
|
||||||
|
when that newer one has no image either — so only the live observation ever
|
||||||
|
carries an indexed element list."""
|
||||||
|
messages: list = []
|
||||||
|
compressor = ScrubEdgeCompressor(summarizer=None, image_scrub_depth=2)
|
||||||
|
|
||||||
|
def imageless(n: int) -> ToolMessage:
|
||||||
|
return ToolMessage(
|
||||||
|
tool_call_id=f"tc{n}",
|
||||||
|
name="click",
|
||||||
|
content=[
|
||||||
|
{"type": "text", "text": f"Tapped {n}, screenshot failed."},
|
||||||
|
{"type": "text", "text": f"--- UI Element List ---\nTree {n}"},
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
_simulate_turns(compressor, messages, [imageless(1)])
|
||||||
|
assert "Tree 1" in str(messages[0].content) # newest observation: kept
|
||||||
|
_simulate_turns(compressor, messages, [imageless(2)]) # screenshots keep failing
|
||||||
|
assert messages[0].content == [{"type": "text", "text": "Tapped 1, screenshot failed."}]
|
||||||
|
assert "Tree 2" in str(messages[1].content)
|
||||||
|
_simulate_turns(compressor, messages, [_step(3)])
|
||||||
|
assert messages[1].content == [{"type": "text", "text": "Tapped 2, screenshot failed."}]
|
||||||
|
assert "Tree 3" in str(messages[2].content) # live observation untouched
|
||||||
|
|
||||||
|
|
||||||
def test_prune_history_xml_disabled_keeps_ui_lists(mock_context):
|
def test_prune_history_xml_disabled_keeps_ui_lists(mock_context):
|
||||||
"""With the switch off, UI lists survive both depth-1 and freeze scrubs."""
|
"""With the switch off, UI lists survive both depth-1 and freeze scrubs."""
|
||||||
summarizer = VisualStepSummarizer(mock_context)
|
summarizer = VisualStepSummarizer(mock_context)
|
||||||
@@ -317,3 +347,348 @@ def test_untracked_messages_are_never_touched(mock_context):
|
|||||||
assert messages[2].content == "thinking"
|
assert messages[2].content == "thinking"
|
||||||
# Image-only tool message froze with the defensive fallback text.
|
# Image-only tool message froze with the defensive fallback text.
|
||||||
assert messages[1].content == [{"type": "text", "text": "Action dispatched."}]
|
assert messages[1].content == [{"type": "text", "text": "Action dispatched."}]
|
||||||
|
|
||||||
|
|
||||||
|
def test_depth_override_tightens_once_and_deeper_pass_never_backfills(mock_context):
|
||||||
|
"""A per-call depth override: a shallower pass scrubs the extra depths in
|
||||||
|
one go; a deeper pass afterwards touches nothing already frozen."""
|
||||||
|
summarizer = VisualStepSummarizer(mock_context)
|
||||||
|
for i in range(1, 9):
|
||||||
|
summarizer._step_inputs[f"tc{i}"] = {"step_number": i}
|
||||||
|
summarizer._summaries[f"tc{i}"] = f"Summary {i}."
|
||||||
|
messages: list = [_initial_observation()]
|
||||||
|
compressor = ScrubEdgeCompressor(summarizer=summarizer, image_scrub_depth=3)
|
||||||
|
|
||||||
|
for i in range(1, 9):
|
||||||
|
compressor.compress(messages, image_scrub_depth=6)
|
||||||
|
messages.append(_step(i))
|
||||||
|
compressor.compress(messages, image_scrub_depth=6)
|
||||||
|
with_images = [m for m in messages if _has_image(m)]
|
||||||
|
# Depth 6 with the live step at depth 1: tc4..tc7 (depths 5..2) keep
|
||||||
|
# their screenshots, tc3 (depth 6) is scrubbed.
|
||||||
|
assert [m.tool_call_id for m in with_images] == ["tc4", "tc5", "tc6", "tc7", "tc8"]
|
||||||
|
|
||||||
|
compressor.compress(messages, image_scrub_depth=3) # tighten: three more scrubbed at once
|
||||||
|
with_images = [m for m in messages if _has_image(m)]
|
||||||
|
assert [m.tool_call_id for m in with_images] == ["tc7", "tc8"]
|
||||||
|
frozen = {idx: _content_fingerprint(messages[idx]) for idx in compressor._frozen}
|
||||||
|
|
||||||
|
compressor.compress(messages, image_scrub_depth=6) # relax: nothing restored
|
||||||
|
assert [m.tool_call_id for m in messages if _has_image(m)] == ["tc7", "tc8"]
|
||||||
|
assert {idx: _content_fingerprint(messages[idx]) for idx in frozen} == frozen
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Screenshot replacement, ephemeral blocks, and frozen content
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _labelled_step(i: int, *, footer: str | None = None) -> ToolMessage:
|
||||||
|
"""A step in the Flash observation shape with the screenshot label block."""
|
||||||
|
blocks = [
|
||||||
|
{"type": "text", "text": f"Action {i} completed."},
|
||||||
|
{"type": "text", "text": "--- Current Screenshot ---"},
|
||||||
|
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,IMG_{i}"}},
|
||||||
|
{"type": "text", "text": f"--- UI Element List ---\nTree {i}"},
|
||||||
|
]
|
||||||
|
if footer is not None:
|
||||||
|
blocks.append({"type": "text", "text": footer})
|
||||||
|
return ToolMessage(tool_call_id=f"tc{i}", name="click", content=blocks)
|
||||||
|
|
||||||
|
|
||||||
|
def test_summary_takes_the_images_place_and_the_label_leaves_with_it(mock_context):
|
||||||
|
"""(i) The visual summary block sits exactly where the image block was —
|
||||||
|
not appended after whatever followed — and no ``--- Current Screenshot ---``
|
||||||
|
label is left dangling above it."""
|
||||||
|
summarizer = VisualStepSummarizer(mock_context)
|
||||||
|
summarizer._summaries["tc1"] = "Summary 1."
|
||||||
|
messages: list = []
|
||||||
|
compressor = ScrubEdgeCompressor(summarizer=summarizer, image_scrub_depth=2)
|
||||||
|
_simulate_turns(
|
||||||
|
compressor, messages, [_labelled_step(1, footer="Footer note 1"), _labelled_step(2)]
|
||||||
|
)
|
||||||
|
|
||||||
|
assert messages[0].content == [
|
||||||
|
{"type": "text", "text": "Action 1 completed."},
|
||||||
|
{"type": "text", "text": "--- Historical Visual Transition ---\nSummary 1."},
|
||||||
|
{"type": "text", "text": "Footer note 1"},
|
||||||
|
]
|
||||||
|
assert "--- Current Screenshot ---" not in str(messages[0].content)
|
||||||
|
# The live step keeps its label above its screenshot.
|
||||||
|
assert messages[1].content[1] == {"type": "text", "text": "--- Current Screenshot ---"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_ephemeral_blocks_vanish_at_the_text_edge_and_not_before(mock_context):
|
||||||
|
"""(ii) Blocks listed under ``EPHEMERAL_BLOCKS_KEY`` survive only while
|
||||||
|
the message is the live observation; at the text edge (a newer observation
|
||||||
|
exists) they are deleted together with the UI list, the screenshot stays
|
||||||
|
until depth K; the consumed indices are cleared at the text edge."""
|
||||||
|
from artemis.memory.transcript import EPHEMERAL_BLOCKS_KEY, mark_ephemeral
|
||||||
|
|
||||||
|
summarizer = VisualStepSummarizer(mock_context)
|
||||||
|
summarizer._summaries["tc1"] = "Summary 1."
|
||||||
|
step1 = ToolMessage(
|
||||||
|
tool_call_id="tc1",
|
||||||
|
name="click",
|
||||||
|
content=[
|
||||||
|
{"type": "text", "text": "Action 1 completed."},
|
||||||
|
{"type": "text", "text": "[reminder: state your reasoning]"},
|
||||||
|
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,IMG_1"}},
|
||||||
|
{"type": "text", "text": "--- UI Element List ---\nTree 1"},
|
||||||
|
{"type": "text", "text": "[hint: valid this turn only]"},
|
||||||
|
],
|
||||||
|
)
|
||||||
|
mark_ephemeral(step1, [1, 4])
|
||||||
|
|
||||||
|
messages: list = []
|
||||||
|
compressor = ScrubEdgeCompressor(summarizer=summarizer, image_scrub_depth=3)
|
||||||
|
_simulate_turns(compressor, messages, [step1])
|
||||||
|
# Live observation: everything, ephemeral blocks included, still there.
|
||||||
|
assert "[reminder: state your reasoning]" in str(messages[0].content)
|
||||||
|
assert "[hint: valid this turn only]" in str(messages[0].content)
|
||||||
|
assert messages[0].additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [1, 4]
|
||||||
|
|
||||||
|
_simulate_turns(compressor, messages, [_step(2)]) # step 1 reaches depth 2
|
||||||
|
# Text edge: notices and UI list gone, the screenshot stays (depth 2 < K).
|
||||||
|
assert messages[0].content == [
|
||||||
|
{"type": "text", "text": "Action 1 completed."},
|
||||||
|
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,IMG_1"}},
|
||||||
|
]
|
||||||
|
assert EPHEMERAL_BLOCKS_KEY not in messages[0].additional_kwargs
|
||||||
|
|
||||||
|
_simulate_turns(compressor, messages, [_step(3)]) # step 1 reaches depth 3
|
||||||
|
assert messages[0].content == [
|
||||||
|
{"type": "text", "text": "Action 1 completed."},
|
||||||
|
{"type": "text", "text": "--- Historical Visual Transition ---\nSummary 1."},
|
||||||
|
]
|
||||||
|
assert EPHEMERAL_BLOCKS_KEY not in messages[0].additional_kwargs
|
||||||
|
|
||||||
|
|
||||||
|
def test_grace_does_the_text_edits_at_k_once_and_only_the_image_swap_waits(mock_context):
|
||||||
|
"""A pending summary keeps the screenshot (and its label) inside the grace
|
||||||
|
window, but the strip and the ephemeral deletion happen at K exactly once;
|
||||||
|
the later swap puts the summary where the image was and freezes."""
|
||||||
|
from artemis.memory.transcript import mark_ephemeral
|
||||||
|
|
||||||
|
summarizer = VisualStepSummarizer(mock_context)
|
||||||
|
summarizer._step_inputs["tc1"] = {"step_number": 1} # pending
|
||||||
|
step1 = _labelled_step(1, footer="[per-turn notice]")
|
||||||
|
mark_ephemeral(step1, [4])
|
||||||
|
|
||||||
|
messages: list = []
|
||||||
|
compressor = ScrubEdgeCompressor(
|
||||||
|
summarizer=summarizer, image_scrub_depth=2, pending_grace_steps=3
|
||||||
|
)
|
||||||
|
_simulate_turns(compressor, messages, [step1, _labelled_step(2)])
|
||||||
|
assert messages[0].content == [
|
||||||
|
{"type": "text", "text": "Action 1 completed."},
|
||||||
|
{"type": "text", "text": "--- Current Screenshot ---"},
|
||||||
|
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,IMG_1"}},
|
||||||
|
]
|
||||||
|
after_text_edits = _content_fingerprint(messages[0])
|
||||||
|
_simulate_turns(compressor, messages, [_labelled_step(3)]) # still pending, in grace
|
||||||
|
assert _content_fingerprint(messages[0]) == after_text_edits
|
||||||
|
|
||||||
|
summarizer._summaries["tc1"] = "Late but in time."
|
||||||
|
_simulate_turns(compressor, messages, [_labelled_step(4)])
|
||||||
|
assert messages[0].content == [
|
||||||
|
{"type": "text", "text": "Action 1 completed."},
|
||||||
|
{"type": "text", "text": "--- Historical Visual Transition ---\nLate but in time."},
|
||||||
|
]
|
||||||
|
assert 0 in compressor._frozen
|
||||||
|
|
||||||
|
|
||||||
|
def test_messages_past_the_edge_are_byte_identical_on_every_later_pass(mock_context):
|
||||||
|
"""(iii) Once a message is past the edge, compressing again — with new
|
||||||
|
turns appended, depths relaxed or tightened, summaries arriving late —
|
||||||
|
leaves it byte-identical. Each pass rewrites exactly the two messages
|
||||||
|
that reached an edge (the text edge at depth 2, the screenshot edge at
|
||||||
|
K) and nothing older."""
|
||||||
|
summarizer = VisualStepSummarizer(mock_context)
|
||||||
|
for i in range(1, 13):
|
||||||
|
summarizer._step_inputs[f"tc{i}"] = {"step_number": i}
|
||||||
|
summarizer._summaries[f"tc{i}"] = f"Summary {i}."
|
||||||
|
messages: list = []
|
||||||
|
compressor = ScrubEdgeCompressor(summarizer=summarizer, image_scrub_depth=3)
|
||||||
|
|
||||||
|
previous: list[str] = []
|
||||||
|
for i in range(1, 13):
|
||||||
|
messages.append(_labelled_step(i))
|
||||||
|
compressor.compress(messages)
|
||||||
|
current = [_content_fingerprint(m) for m in messages]
|
||||||
|
# Everything before the messages that reached an edge is unchanged.
|
||||||
|
changed = [idx for idx, fp in enumerate(previous) if current[idx] != fp]
|
||||||
|
# The message reaching depth 2 (text edge) is index len-1 of the previous
|
||||||
|
# pass's list, the one reaching depth 3 (screenshot edge) is len-2; the
|
||||||
|
# new live message is not in that list yet.
|
||||||
|
expected = [idx for idx in (len(previous) - 2, len(previous) - 1) if idx >= 0]
|
||||||
|
assert changed == expected, changed
|
||||||
|
previous = current
|
||||||
|
|
||||||
|
# No new messages: a repeated pass, relaxed or not, is a no-op byte for byte.
|
||||||
|
compressor.compress(messages)
|
||||||
|
compressor.compress(messages, image_scrub_depth=6)
|
||||||
|
assert [_content_fingerprint(m) for m in messages] == previous
|
||||||
|
# Tightening rewrites exactly the message at the new depth, nothing older.
|
||||||
|
compressor.compress(messages, image_scrub_depth=2)
|
||||||
|
changed = [i for i, fp in enumerate(previous) if _content_fingerprint(messages[i]) != fp]
|
||||||
|
assert changed == [len(messages) - 2]
|
||||||
|
|
||||||
|
|
||||||
|
def test_text_edge_deeper_than_screenshot_edge_keeps_the_edges_independent(mock_context):
|
||||||
|
"""xml_scrub_depth=4 with K=3: the screenshot is resolved at depth 3 while
|
||||||
|
the UI list stays until the message is deeper than 4; the message is only
|
||||||
|
frozen once both edits have happened."""
|
||||||
|
summarizer = VisualStepSummarizer(mock_context)
|
||||||
|
for i in range(1, 8):
|
||||||
|
summarizer._summaries[f"tc{i}"] = f"Summary {i}."
|
||||||
|
messages: list = []
|
||||||
|
compressor = ScrubEdgeCompressor(summarizer=summarizer, image_scrub_depth=3, xml_scrub_depth=4)
|
||||||
|
_simulate_turns(compressor, messages, [_labelled_step(i) for i in range(1, 4)])
|
||||||
|
# Step 1 at depth 3: image swapped, label gone, UI list still there.
|
||||||
|
assert "Summary 1." in str(messages[0].content)
|
||||||
|
assert not _has_image(messages[0])
|
||||||
|
assert "--- Current Screenshot ---" not in str(messages[0].content)
|
||||||
|
assert "Tree 1" in str(messages[0].content)
|
||||||
|
|
||||||
|
_simulate_turns(compressor, messages, [_labelled_step(4), _labelled_step(5)])
|
||||||
|
# Step 1 at depth 5 (> 4): UI list gone now; the summary is untouched.
|
||||||
|
assert "Tree 1" not in str(messages[0].content)
|
||||||
|
assert messages[0].content == [
|
||||||
|
{"type": "text", "text": "Action 1 completed."},
|
||||||
|
{"type": "text", "text": "--- Historical Visual Transition ---\nSummary 1."},
|
||||||
|
]
|
||||||
|
frozen = _content_fingerprint(messages[0])
|
||||||
|
_simulate_turns(compressor, messages, [_labelled_step(6), _labelled_step(7)])
|
||||||
|
assert _content_fingerprint(messages[0]) == frozen
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Ephemeral indices stay aligned when the image edge is shallower than the
|
||||||
|
# text edge (a rewrite before the text edit shifts the later blocks)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _step_with_notices(i: int) -> ToolMessage:
|
||||||
|
"""Flash observation shape with one ephemeral notice on each side of the
|
||||||
|
screenshot and a body block *after* the notice that follows the image, so
|
||||||
|
a stale index would delete the body instead of the notice."""
|
||||||
|
from artemis.memory.transcript import mark_ephemeral
|
||||||
|
|
||||||
|
msg = ToolMessage(
|
||||||
|
tool_call_id=f"tc{i}",
|
||||||
|
name="click",
|
||||||
|
content=[
|
||||||
|
{"type": "text", "text": f"Action {i} completed."},
|
||||||
|
{"type": "text", "text": f"[hint {i}: before the image]"},
|
||||||
|
{"type": "text", "text": "--- Current Screenshot ---"},
|
||||||
|
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,IMG_{i}"}},
|
||||||
|
{"type": "text", "text": f"--- UI Element List ---\nTree {i}"},
|
||||||
|
{"type": "text", "text": f"[reminder {i}: after the image]"},
|
||||||
|
{"type": "text", "text": f"Body note {i}"},
|
||||||
|
],
|
||||||
|
)
|
||||||
|
mark_ephemeral(msg, [1, 5])
|
||||||
|
return msg
|
||||||
|
|
||||||
|
|
||||||
|
def test_image_edge_before_text_edge_remaps_ephemeral_indices_to_the_summary_layout(
|
||||||
|
mock_context,
|
||||||
|
):
|
||||||
|
"""K=3 with xml_scrub_depth=4: the summary swap at depth 3 removes the
|
||||||
|
label block, so the notice after the image moves up by one. The stored
|
||||||
|
indices must follow it; the later text pass then deletes both notices and
|
||||||
|
leaves the body block intact."""
|
||||||
|
from artemis.memory.transcript import EPHEMERAL_BLOCKS_KEY
|
||||||
|
|
||||||
|
summarizer = VisualStepSummarizer(mock_context)
|
||||||
|
for i in range(1, 6):
|
||||||
|
summarizer._summaries[f"tc{i}"] = f"Summary {i}."
|
||||||
|
messages: list = []
|
||||||
|
compressor = ScrubEdgeCompressor(summarizer=summarizer, image_scrub_depth=3, xml_scrub_depth=4)
|
||||||
|
_simulate_turns(compressor, messages, [_step_with_notices(i) for i in range(1, 4)])
|
||||||
|
|
||||||
|
# Image pass (depth 3): label + image -> summary; notices still present.
|
||||||
|
assert messages[0].content == [
|
||||||
|
{"type": "text", "text": "Action 1 completed."},
|
||||||
|
{"type": "text", "text": "[hint 1: before the image]"},
|
||||||
|
{"type": "text", "text": "--- Historical Visual Transition ---\nSummary 1."},
|
||||||
|
{"type": "text", "text": "--- UI Element List ---\nTree 1"},
|
||||||
|
{"type": "text", "text": "[reminder 1: after the image]"},
|
||||||
|
{"type": "text", "text": "Body note 1"},
|
||||||
|
]
|
||||||
|
assert messages[0].additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [1, 4]
|
||||||
|
|
||||||
|
_simulate_turns(compressor, messages, [_step_with_notices(4), _step_with_notices(5)])
|
||||||
|
# Text pass (depth 5 > 4): notices and UI list gone, body block intact.
|
||||||
|
assert messages[0].content == [
|
||||||
|
{"type": "text", "text": "Action 1 completed."},
|
||||||
|
{"type": "text", "text": "--- Historical Visual Transition ---\nSummary 1."},
|
||||||
|
{"type": "text", "text": "Body note 1"},
|
||||||
|
]
|
||||||
|
assert EPHEMERAL_BLOCKS_KEY not in messages[0].additional_kwargs
|
||||||
|
assert 0 in compressor._frozen
|
||||||
|
|
||||||
|
|
||||||
|
def test_image_edge_before_text_edge_remaps_when_the_image_is_dropped(mock_context):
|
||||||
|
"""Same ordering with no summary job at all: label and image both leave
|
||||||
|
(two blocks removed), the notice after the image moves up by two."""
|
||||||
|
from artemis.memory.transcript import EPHEMERAL_BLOCKS_KEY
|
||||||
|
|
||||||
|
messages: list = []
|
||||||
|
compressor = ScrubEdgeCompressor(summarizer=None, image_scrub_depth=3, xml_scrub_depth=4)
|
||||||
|
_simulate_turns(compressor, messages, [_step_with_notices(i) for i in range(1, 4)])
|
||||||
|
|
||||||
|
assert messages[0].content == [
|
||||||
|
{"type": "text", "text": "Action 1 completed."},
|
||||||
|
{"type": "text", "text": "[hint 1: before the image]"},
|
||||||
|
{"type": "text", "text": "--- UI Element List ---\nTree 1"},
|
||||||
|
{"type": "text", "text": "[reminder 1: after the image]"},
|
||||||
|
{"type": "text", "text": "Body note 1"},
|
||||||
|
]
|
||||||
|
assert messages[0].additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [1, 3]
|
||||||
|
|
||||||
|
_simulate_turns(compressor, messages, [_step_with_notices(4), _step_with_notices(5)])
|
||||||
|
assert messages[0].content == [
|
||||||
|
{"type": "text", "text": "Action 1 completed."},
|
||||||
|
{"type": "text", "text": "Body note 1"},
|
||||||
|
]
|
||||||
|
assert EPHEMERAL_BLOCKS_KEY not in messages[0].additional_kwargs
|
||||||
|
assert 0 in compressor._frozen
|
||||||
|
|
||||||
|
|
||||||
|
def test_grace_swap_before_the_text_edge_remaps_ephemeral_indices(mock_context):
|
||||||
|
"""Pending summary inside the grace window with the text edge still ahead:
|
||||||
|
the keep-image revisit changes nothing (indices untouched); the late swap
|
||||||
|
removes the label and remaps; the text pass then deletes the right blocks."""
|
||||||
|
from artemis.memory.transcript import EPHEMERAL_BLOCKS_KEY
|
||||||
|
|
||||||
|
summarizer = VisualStepSummarizer(mock_context)
|
||||||
|
summarizer._step_inputs["tc1"] = {"step_number": 1} # pending
|
||||||
|
messages: list = []
|
||||||
|
compressor = ScrubEdgeCompressor(
|
||||||
|
summarizer=summarizer, image_scrub_depth=3, xml_scrub_depth=5, pending_grace_steps=3
|
||||||
|
)
|
||||||
|
_simulate_turns(compressor, messages, [_step_with_notices(i) for i in range(1, 4)])
|
||||||
|
# Depth 3, pending, in grace: byte-identical, indices untouched.
|
||||||
|
assert _has_image(messages[0])
|
||||||
|
assert "--- Current Screenshot ---" in str(messages[0].content)
|
||||||
|
assert messages[0].additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [1, 5]
|
||||||
|
|
||||||
|
summarizer._summaries["tc1"] = "Late but in time."
|
||||||
|
_simulate_turns(compressor, messages, [_step_with_notices(4)]) # depth 4: swap
|
||||||
|
assert messages[0].content[2] == {
|
||||||
|
"type": "text",
|
||||||
|
"text": "--- Historical Visual Transition ---\nLate but in time.",
|
||||||
|
}
|
||||||
|
assert messages[0].additional_kwargs[EPHEMERAL_BLOCKS_KEY] == [1, 4]
|
||||||
|
|
||||||
|
_simulate_turns(compressor, messages, [_step_with_notices(5), _step_with_notices(6)])
|
||||||
|
# Depth 6 > 5: text pass on the remapped layout.
|
||||||
|
assert messages[0].content == [
|
||||||
|
{"type": "text", "text": "Action 1 completed."},
|
||||||
|
{"type": "text", "text": "--- Historical Visual Transition ---\nLate but in time."},
|
||||||
|
{"type": "text", "text": "Body note 1"},
|
||||||
|
]
|
||||||
|
assert EPHEMERAL_BLOCKS_KEY not in messages[0].additional_kwargs
|
||||||
|
assert 0 in compressor._frozen
|
||||||
@@ -569,8 +569,9 @@ async def test_raw_lens_call_meters_llm_usage_without_touching_context_base(mock
|
|||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_focus_block_and_provenance_marked_action_phrase_reach_the_lens(mock_context):
|
async def test_focus_block_and_provenance_marked_action_phrase_reach_the_lens(mock_context):
|
||||||
"""The lens input opens with the action as every history reader sees it
|
"""The lens input opens with the action as every history reader sees it
|
||||||
(self-described target marked), the verbatim arguments, the outcome, and
|
(self-described target marked), the outcome, and the operator focus
|
||||||
the operator focus block; the system prompt teaches how to use it."""
|
block; the phrase already carries every argument, so no verbatim
|
||||||
|
argument line repeats them. The system prompt teaches how to use it."""
|
||||||
summarizer = VisualStepSummarizer(mock_context)
|
summarizer = VisualStepSummarizer(mock_context)
|
||||||
captured: list = []
|
captured: list = []
|
||||||
|
|
||||||
@@ -609,9 +610,9 @@ async def test_focus_block_and_provenance_marked_action_phrase_reach_the_lens(mo
|
|||||||
assert "Absence is a fact" in system_text
|
assert "Absence is a fact" in system_text
|
||||||
assert lead_text.startswith(
|
assert lead_text.startswith(
|
||||||
"Step 3 Physical Action: Tapped 'Wi-Fi toggle' (self-described) at [880, 410]\n"
|
"Step 3 Physical Action: Tapped 'Wi-Fi toggle' (self-described) at [880, 410]\n"
|
||||||
|
"Controller Outcome: Tapped at [880, 410] (normalized)."
|
||||||
)
|
)
|
||||||
assert "Action arguments (verbatim): click({'target': [880, 410]" in lead_text
|
assert "Action arguments (verbatim)" not in lead_text
|
||||||
assert "Controller Outcome: Tapped at [880, 410] (normalized)." in lead_text
|
|
||||||
assert FOCUS_BLOCK_HEADER in lead_text
|
assert FOCUS_BLOCK_HEADER in lead_text
|
||||||
assert "Task goal: Turn on Wi-Fi" in lead_text
|
assert "Task goal: Turn on Wi-Fi" in lead_text
|
||||||
assert "Active sub-goal: Enable Wi-Fi > Tap the Wi-Fi switch" in lead_text
|
assert "Active sub-goal: Enable Wi-Fi > Tap the Wi-Fi switch" in lead_text
|
||||||
@@ -688,6 +689,102 @@ async def test_single_frame_focus_label_forbids_describing_the_outcome(mock_cont
|
|||||||
assert summarizer.get_job_payload(2)["focus"] is None
|
assert summarizer.get_job_payload(2)["focus"] is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_action_phrase_action_name_wins_over_an_action_argument():
|
||||||
|
"""``manage_app``'s own ``action="launch"`` argument is the intent the
|
||||||
|
renderer reads, never the verb: the phrase is ``Launched app 'X'`` for
|
||||||
|
the Flash record shape, the Pro record shape and a raw argument dict."""
|
||||||
|
flash_record = {
|
||||||
|
"coordinates": None,
|
||||||
|
"coordinate_space": "normalized",
|
||||||
|
"args": {"action": "launch", "app_name": "Settings"},
|
||||||
|
}
|
||||||
|
assert VisualStepSummarizer._action_phrase("manage_app", flash_record) == (
|
||||||
|
"Launched app 'Settings'"
|
||||||
|
)
|
||||||
|
pro_record = {"intent": "stop", "app_name": "com.android.settings"}
|
||||||
|
assert VisualStepSummarizer._action_phrase("manage_app", pro_record) == (
|
||||||
|
"Stopped app 'com.android.settings'"
|
||||||
|
)
|
||||||
|
raw = {"action": "launch", "app_name": "Settings"}
|
||||||
|
assert VisualStepSummarizer._action_phrase("manage_app", raw) == "Launched app 'Settings'"
|
||||||
|
assert "Action: launch" not in VisualStepSummarizer._action_phrase("manage_app", raw)
|
||||||
|
|
||||||
|
# The Flash record shape renders every other action through the same path.
|
||||||
|
key = {"coordinates": None, "args": {"key": "BACK"}}
|
||||||
|
assert VisualStepSummarizer._action_phrase("press_key", key) == "Pressed key 'BACK'"
|
||||||
|
swipe = {
|
||||||
|
"coordinates": [600, 700, 600, 300],
|
||||||
|
"normalized_start_coordinates": [600, 700],
|
||||||
|
"normalized_end_coordinates": [600, 300],
|
||||||
|
"args": {"direction": "up"},
|
||||||
|
}
|
||||||
|
assert VisualStepSummarizer._action_phrase("swipe", swipe) == (
|
||||||
|
"Swiped up (from [600, 700] to [600, 300])"
|
||||||
|
)
|
||||||
|
index_click = {
|
||||||
|
"coordinates": [500, 520],
|
||||||
|
"normalized_coordinates": [500, 520],
|
||||||
|
"target_text": "Wi-Fi",
|
||||||
|
"args": {"target": 3},
|
||||||
|
}
|
||||||
|
assert VisualStepSummarizer._action_phrase("click", index_click) == (
|
||||||
|
"Tapped 'Wi-Fi' at [500, 520]"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_verbatim_argument_line_appears_only_when_the_phrase_folds_arguments(
|
||||||
|
mock_context,
|
||||||
|
):
|
||||||
|
"""``Launched app 'Settings'`` carries its arguments: no verbatim line. An
|
||||||
|
index target (``target=3``) is folded into the resolved point, so the
|
||||||
|
verbatim line keeps the index the model actually named."""
|
||||||
|
summarizer = VisualStepSummarizer(mock_context)
|
||||||
|
captured: list = []
|
||||||
|
|
||||||
|
async def mock_ainvoke(messages):
|
||||||
|
captured.append(messages)
|
||||||
|
return AIMessage(content="In this step, I acted and the screen showed the result.")
|
||||||
|
|
||||||
|
summarizer._llm = Mock()
|
||||||
|
summarizer._llm.ainvoke = AsyncMock(side_effect=mock_ainvoke)
|
||||||
|
|
||||||
|
summarizer.dispatch(
|
||||||
|
step_number=1,
|
||||||
|
action_name="manage_app",
|
||||||
|
action_args={"coordinates": None, "args": {"action": "launch", "app_name": "Settings"}},
|
||||||
|
pre_img_bytes=b"pre",
|
||||||
|
post_img_bytes=b"post",
|
||||||
|
exec_outcome="Launched app 'Settings' (com.android.settings); foreground confirmed.",
|
||||||
|
)
|
||||||
|
summarizer.dispatch(
|
||||||
|
step_number=2,
|
||||||
|
action_name="click",
|
||||||
|
action_args={
|
||||||
|
"coordinates": [500, 520],
|
||||||
|
"normalized_coordinates": [500, 520],
|
||||||
|
"target_text": "Wi-Fi",
|
||||||
|
"args": {"target": 3},
|
||||||
|
},
|
||||||
|
pre_img_bytes=b"pre",
|
||||||
|
post_img_bytes=b"post",
|
||||||
|
exec_outcome="Tapped at [500, 520] (normalized).",
|
||||||
|
)
|
||||||
|
await summarizer.flush()
|
||||||
|
|
||||||
|
lead_by_step = {}
|
||||||
|
for messages in captured:
|
||||||
|
text = messages[1].content[0]["text"]
|
||||||
|
lead_by_step[text.split(" ")[1]] = text
|
||||||
|
launch = lead_by_step["1"]
|
||||||
|
assert launch.startswith("Step 1 Physical Action: Launched app 'Settings'\nController Outcome:")
|
||||||
|
assert "Action arguments (verbatim)" not in launch
|
||||||
|
assert "Action: launch" not in launch
|
||||||
|
click = lead_by_step["2"]
|
||||||
|
assert click.startswith("Step 2 Physical Action: Tapped 'Wi-Fi' at [500, 520]\n")
|
||||||
|
assert "Action arguments (verbatim): click({'target': 3})" in click
|
||||||
|
|
||||||
|
|
||||||
def test_action_phrase_renders_flash_swipe_points_and_bursts():
|
def test_action_phrase_renders_flash_swipe_points_and_bursts():
|
||||||
"""Flash's ``start``/``end`` swipe points and a Pro burst's additional
|
"""Flash's ``start``/``end`` swipe points and a Pro burst's additional
|
||||||
actions render through the shared history phrase, never as None."""
|
actions render through the shared history phrase, never as None."""
|
||||||
|
|||||||
@@ -263,9 +263,9 @@ async def test_history_analyzer_read_note_tool_call():
|
|||||||
] # Index 5 contains the ToolMessage for read_note
|
] # Index 5 contains the ToolMessage for read_note
|
||||||
assert isinstance(read_msg, ToolMessage)
|
assert isinstance(read_msg, ToolMessage)
|
||||||
assert read_msg.tool_call_id == "call_read"
|
assert read_msg.tool_call_id == "call_read"
|
||||||
assert (
|
assert read_msg.content == (
|
||||||
read_msg.content == "Successfully read note 'tactical_plan'. 'tactical_plan' note"
|
f"Note 'tactical_plan' ({len(fake_plan_content.splitlines())} lines):"
|
||||||
f" content:\n{fake_plan_content}"
|
f"\n{fake_plan_content}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -306,10 +306,7 @@ def test_history_analyzer_robust_tools_behavior():
|
|||||||
patch("pathlib.Path.read_text", return_value="Tactical plan content"),
|
patch("pathlib.Path.read_text", return_value="Tactical plan content"),
|
||||||
):
|
):
|
||||||
result_read = read_tool.invoke({"key": "tactical_plan.md"})
|
result_read = read_tool.invoke({"key": "tactical_plan.md"})
|
||||||
assert (
|
assert result_read == "Note 'tactical_plan.md' (1 lines):\nTactical plan content"
|
||||||
result_read == "Successfully read note 'tactical_plan.md'. 'tactical_plan.md'"
|
|
||||||
" note content:\nTactical plan content"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
|
|||||||
@@ -399,6 +399,14 @@ async def test_operator_node_multiple_actions():
|
|||||||
assert decisions[1]["action"] == "tap"
|
assert decisions[1]["action"] == "tap"
|
||||||
assert decisions[1]["coordinates"] == [54, 120]
|
assert decisions[1]["coordinates"] == [54, 120]
|
||||||
|
|
||||||
|
# Each action's ToolMessage names its tool, like helper-tool results do
|
||||||
|
# (the loop appends them to the same list it sent to the model).
|
||||||
|
from langchain_core.messages import ToolMessage
|
||||||
|
|
||||||
|
sent = mock_llm.ainvoke.call_args[0][0]
|
||||||
|
recorded = [(m.name, m.content) for m in sent if isinstance(m, ToolMessage)]
|
||||||
|
assert recorded == [("input_text", "Action Recorded"), ("click", "Action Recorded")]
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_operator_node_no_record_step():
|
async def test_operator_node_no_record_step():
|
||||||
@@ -1467,6 +1475,40 @@ async def test_operator_swipe_translation():
|
|||||||
assert actions[0]["coordinates"] == [648, 720, 648, 1680]
|
assert actions[0]["coordinates"] == [648, 720, 648, 1680]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_operator_press_key_translation_is_case_insensitive():
|
||||||
|
from artemis.agents.operator.operator import OperatorNode
|
||||||
|
from artemis.context import ArtemisContext
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
mock_ctx = MagicMock(spec=ArtemisContext)
|
||||||
|
mock_state = MagicMock()
|
||||||
|
mock_state.operator_raw_data = {"width": 1080, "height": 2400}
|
||||||
|
node = OperatorNode(mock_ctx, transcript_config=LEGACY_TRANSCRIPT)
|
||||||
|
|
||||||
|
for spelling in ("back", "Back", "BACK", "KEYCODE_BACK", "keycode_back", " back "):
|
||||||
|
actions, err = node._translate_and_validate_tool(
|
||||||
|
{"name": "press_key", "args": {"key": spelling}}, mock_state
|
||||||
|
)
|
||||||
|
assert err is None, spelling
|
||||||
|
assert actions == [{"action": "press_key", "keycode": "KEYCODE_BACK"}], spelling
|
||||||
|
|
||||||
|
actions, err = node._translate_and_validate_tool(
|
||||||
|
{"name": "press_key", "args": {"key": "app_switch"}}, mock_state
|
||||||
|
)
|
||||||
|
assert err is None
|
||||||
|
assert actions == [{"action": "press_key", "keycode": "KEYCODE_APP_SWITCH"}]
|
||||||
|
|
||||||
|
# Unsupported keys get a self-explanatory error that lists the accepted names.
|
||||||
|
actions, err = node._translate_and_validate_tool(
|
||||||
|
{"name": "press_key", "args": {"key": "volume_up"}}, mock_state
|
||||||
|
)
|
||||||
|
assert actions == []
|
||||||
|
assert err == (
|
||||||
|
"Error: Unsupported key 'volume_up'. Supported keys: ENTER, BACK, HOME, APP_SWITCH."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
# --- helper tool results: status is structural, never sniffed from the words ----------
|
# --- helper tool results: status is structural, never sniffed from the words ----------
|
||||||
|
|
||||||
|
|
||||||
@@ -1537,5 +1579,79 @@ async def test_operator_helper_tool_message_status_is_structural(result, expecte
|
|||||||
tool_msgs = {m.tool_call_id: m for m in second_turn_messages if isinstance(m, ToolMessage)}
|
tool_msgs = {m.tool_call_id: m for m in second_turn_messages if isinstance(m, ToolMessage)}
|
||||||
assert tool_msgs["call_read"].status == expected_status
|
assert tool_msgs["call_read"].status == expected_status
|
||||||
assert tool_msgs["call_read"].content == str(result)
|
assert tool_msgs["call_read"].content == str(result)
|
||||||
# The accompanying click is rejected only when the helper actually failed.
|
# The accompanying click never ran: rejected when the helper failed, deferred
|
||||||
assert tool_msgs["call_click"].status == ("error" if expected_status == "error" else "success")
|
# otherwise. Both are reported as errors and never claim success.
|
||||||
|
assert tool_msgs["call_click"].status == "error"
|
||||||
|
assert tool_msgs["call_click"].content.startswith("Not executed:")
|
||||||
|
assert "success" not in tool_msgs["call_click"].content.lower()
|
||||||
|
if expected_status == "error":
|
||||||
|
assert str(result) in tool_msgs["call_click"].content
|
||||||
|
else:
|
||||||
|
assert "deferred" in tool_msgs["call_click"].content
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_operator_burst_limit_error_text_is_not_repeated_per_call():
|
||||||
|
"""An over-long burst answers every tool_call id, but the full burst-limit
|
||||||
|
paragraph travels once; the remaining ToolMessages just point at it."""
|
||||||
|
from langchain_core.messages import ToolMessage
|
||||||
|
|
||||||
|
mock_ctx = MagicMock(spec=ArtemisContext)
|
||||||
|
mock_ctx.execution_setup = None
|
||||||
|
mock_ctx.data_engine = None
|
||||||
|
|
||||||
|
mock_state = MagicMock()
|
||||||
|
mock_state.subagent_calls = []
|
||||||
|
mock_state.initial_goal = "Test goal"
|
||||||
|
mock_state.open_incident = None
|
||||||
|
|
||||||
|
turn_1 = MagicMock()
|
||||||
|
turn_1.tool_calls = [
|
||||||
|
{
|
||||||
|
"name": "click",
|
||||||
|
"args": {"target": [50, 50], "target_description": f"button {i}"},
|
||||||
|
"id": f"call_click_{i}",
|
||||||
|
}
|
||||||
|
for i in range(6)
|
||||||
|
]
|
||||||
|
turn_2 = MagicMock()
|
||||||
|
turn_2.tool_calls = [
|
||||||
|
{
|
||||||
|
"name": "click",
|
||||||
|
"args": {"target": [50, 50], "target_description": "button"},
|
||||||
|
"id": "call_click_final",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
mock_llm = MagicMock()
|
||||||
|
mock_llm.ainvoke = AsyncMock(side_effect=[turn_1, turn_2])
|
||||||
|
mock_llm.bind_tools.return_value = mock_llm
|
||||||
|
|
||||||
|
with (
|
||||||
|
patch("artemis.agents.operator.operator.get_llm", return_value=mock_llm),
|
||||||
|
patch.object(OperatorNode, "_max_burst_actions", return_value=4),
|
||||||
|
):
|
||||||
|
node = OperatorNode(mock_ctx, transcript_config=LEGACY_TRANSCRIPT)
|
||||||
|
node_update = await node(mock_state)
|
||||||
|
|
||||||
|
assert mock_llm.ainvoke.await_count == 2
|
||||||
|
second_turn_messages = mock_llm.ainvoke.call_args_list[1].args[0]
|
||||||
|
burst_ids = {f"call_click_{i}" for i in range(6)}
|
||||||
|
tool_msgs = {
|
||||||
|
m.tool_call_id: m
|
||||||
|
for m in second_turn_messages
|
||||||
|
if isinstance(m, ToolMessage) and m.tool_call_id in burst_ids
|
||||||
|
}
|
||||||
|
assert set(tool_msgs) == burst_ids
|
||||||
|
assert all(m.status == "error" for m in tool_msgs.values())
|
||||||
|
|
||||||
|
first = tool_msgs["call_click_0"].content
|
||||||
|
assert first.startswith("Error: 6 Turn-Ending Actions in one turn exceed")
|
||||||
|
assert "burst limit of 4" in first
|
||||||
|
for i in range(1, 6):
|
||||||
|
assert tool_msgs[f"call_click_{i}"].content == (
|
||||||
|
"Not executed: see the burst-limit error above."
|
||||||
|
)
|
||||||
|
|
||||||
|
# Nothing from the rejected burst ran; turn 2's single click did.
|
||||||
|
decisions = json.loads(node_update["structured_decisions"])
|
||||||
|
assert len(decisions) == 1 and decisions[0]["action"] == "tap"
|
||||||
@@ -0,0 +1,122 @@
|
|||||||
|
# Copyright 2026 Google LLC
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
"""The Pro operator's context base on a provider without usage metadata.
|
||||||
|
|
||||||
|
The transcript ledger's start gate and soft/hard thresholds read the
|
||||||
|
operator's own measured prompt size; when the provider reports no usage the
|
||||||
|
operator records a provider-less estimate instead (Flash already does), so
|
||||||
|
the thresholds still see a base. A measured call also hands the ledger the
|
||||||
|
messages it was measured on, for the chars-per-token calibration.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import base64
|
||||||
|
from unittest.mock import AsyncMock, MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from artemis.agents.operator.operator import OperatorNode
|
||||||
|
from artemis.config.agent import MemoryTranscriptConfig
|
||||||
|
from artemis.context import ArtemisContext
|
||||||
|
from artemis.memory.step_memory import StepMemoryService
|
||||||
|
from artemis.memory.transcript import TranscriptLedger, estimate_prompt_tokens
|
||||||
|
|
||||||
|
SCREENSHOT_B64 = base64.b64encode(b"fake-jpeg-bytes").decode("utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def _ctx():
|
||||||
|
ctx = MagicMock(spec=ArtemisContext)
|
||||||
|
ctx.execution_setup = None
|
||||||
|
ctx.actuator = None
|
||||||
|
ctx.data_engine = None
|
||||||
|
ctx.step_memory = StepMemoryService(ctx=None)
|
||||||
|
ctx.transcript_ledger = None
|
||||||
|
return ctx
|
||||||
|
|
||||||
|
|
||||||
|
def _state():
|
||||||
|
state = MagicMock()
|
||||||
|
state.subagent_calls = []
|
||||||
|
state.initial_goal = "Estimate goal"
|
||||||
|
state.operator_feedback = None
|
||||||
|
state.injected_instruction = None
|
||||||
|
state.operator_tool_limit_exceeded = False
|
||||||
|
state.structured_decisions = None
|
||||||
|
state.last_execution_result = None
|
||||||
|
state.current_step_id = None
|
||||||
|
state.latest_ui_hierarchy = []
|
||||||
|
state.operator_raw_data = {
|
||||||
|
"screenshot_b64": SCREENSHOT_B64,
|
||||||
|
"xml_hierarchy": [],
|
||||||
|
"ocr_results": None,
|
||||||
|
"width": 1080,
|
||||||
|
"height": 2400,
|
||||||
|
}
|
||||||
|
return state
|
||||||
|
|
||||||
|
|
||||||
|
def _llm(captured: list, usage_metadata=None):
|
||||||
|
mock_llm = MagicMock()
|
||||||
|
response = MagicMock()
|
||||||
|
response.tool_calls = []
|
||||||
|
response.content = "no action this turn"
|
||||||
|
# usage_from_message only trusts a non-empty dict; a MagicMock attribute
|
||||||
|
# (the default) reads as "no usage reported".
|
||||||
|
if usage_metadata is not None:
|
||||||
|
response.usage_metadata = usage_metadata
|
||||||
|
|
||||||
|
async def ainvoke(*args, **kwargs):
|
||||||
|
captured.append(list(args[0]))
|
||||||
|
return response
|
||||||
|
|
||||||
|
mock_llm.ainvoke = AsyncMock(side_effect=ainvoke)
|
||||||
|
mock_llm.bind_tools.return_value = mock_llm
|
||||||
|
return mock_llm
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_operator_records_an_estimate_when_the_provider_reports_no_usage():
|
||||||
|
ctx = _ctx()
|
||||||
|
captured: list = []
|
||||||
|
with patch("artemis.agents.operator.operator.get_llm", return_value=_llm(captured)):
|
||||||
|
node = OperatorNode(ctx, transcript_config=MemoryTranscriptConfig(enabled=True))
|
||||||
|
await node(_state())
|
||||||
|
|
||||||
|
ledger = ctx.transcript_ledger
|
||||||
|
assert isinstance(ledger, TranscriptLedger)
|
||||||
|
assert captured, "the operator made no model call"
|
||||||
|
assert ledger.last_prompt_tokens == estimate_prompt_tokens(captured[-1])
|
||||||
|
assert ledger.last_prompt_tokens > 0
|
||||||
|
# No measurement, no calibration.
|
||||||
|
assert ledger.chars_per_token == 4.0
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_operator_measured_usage_calibrates_the_ledger_ratio():
|
||||||
|
ctx = _ctx()
|
||||||
|
captured: list = []
|
||||||
|
llm = _llm(
|
||||||
|
captured,
|
||||||
|
usage_metadata={"input_tokens": 50_000, "output_tokens": 10, "total_tokens": 50_010},
|
||||||
|
)
|
||||||
|
with patch("artemis.agents.operator.operator.get_llm", return_value=llm):
|
||||||
|
node = OperatorNode(ctx, transcript_config=MemoryTranscriptConfig(enabled=True))
|
||||||
|
await node(_state())
|
||||||
|
|
||||||
|
ledger = ctx.transcript_ledger
|
||||||
|
assert ledger.last_prompt_tokens == 50_000
|
||||||
|
# The prompt is a few thousand chars of English plus one image; measured
|
||||||
|
# at 50k tokens the ratio clamps to the lower bound — proof the messages
|
||||||
|
# reached the calibration, not just the token figure.
|
||||||
|
assert ledger.chars_per_token == 1.0
|
||||||
@@ -0,0 +1,147 @@
|
|||||||
|
# Copyright 2026 Google LLC
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
"""Tests for the Operator reasoning reminder.
|
||||||
|
|
||||||
|
Thinking-mode models satisfy "reason before acting" inside their thought channel
|
||||||
|
and emit bare tool calls. After such a silent turn the next observation tail
|
||||||
|
carries a one-line reminder; nothing is bounced and no extra model call is made.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from unittest.mock import AsyncMock, MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from artemis.agents.operator.operator import OperatorNode
|
||||||
|
from artemis.agents.operator.prompts import (
|
||||||
|
REASONING_REMINDER,
|
||||||
|
apply_operator_prompt_contract,
|
||||||
|
load_operator_prompts,
|
||||||
|
)
|
||||||
|
from artemis.config.agent import MemoryTranscriptConfig
|
||||||
|
from artemis.context import ArtemisContext
|
||||||
|
|
||||||
|
LEGACY_TRANSCRIPT = MemoryTranscriptConfig(enabled=False)
|
||||||
|
|
||||||
|
PLAN_WITH_LEAF = """- [x] Open Google Maps and search for SFO
|
||||||
|
- [/] Read the commute duration and record it into note `commute_eta_info`
|
||||||
|
- [/] Read the driving duration
|
||||||
|
- [/] Tap the "Driving" tab so the fastest route is highlighted
|
||||||
|
- [ ] Draft the ETA message
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _make_node(tmp_path):
|
||||||
|
notes = tmp_path / "notes"
|
||||||
|
notes.mkdir(parents=True)
|
||||||
|
(notes / "task_plan.md").write_text(PLAN_WITH_LEAF, encoding="utf-8")
|
||||||
|
|
||||||
|
ctx = MagicMock(spec=ArtemisContext)
|
||||||
|
ctx.execution_setup = None
|
||||||
|
ctx.data_engine = MagicMock()
|
||||||
|
ctx.data_engine.base_dir = str(tmp_path)
|
||||||
|
ctx.data_engine.current_session_id = "s"
|
||||||
|
ctx.data_engine.get_agent_friendly_steps.return_value = []
|
||||||
|
|
||||||
|
state = MagicMock()
|
||||||
|
state.subagent_calls = []
|
||||||
|
state.initial_goal = "goal"
|
||||||
|
state.open_incident = None
|
||||||
|
return OperatorNode(ctx, transcript_config=LEGACY_TRANSCRIPT), state
|
||||||
|
|
||||||
|
|
||||||
|
def _click_response(content):
|
||||||
|
response = MagicMock()
|
||||||
|
response.content = content
|
||||||
|
response.tool_calls = [
|
||||||
|
{
|
||||||
|
"name": "click",
|
||||||
|
"args": {"target": [50, 50], "target_description": "button"},
|
||||||
|
"id": "call_0",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
return response
|
||||||
|
|
||||||
|
|
||||||
|
def _texts(messages):
|
||||||
|
out = []
|
||||||
|
for m in messages:
|
||||||
|
content = getattr(m, "content", None)
|
||||||
|
if isinstance(content, str):
|
||||||
|
out.append(content)
|
||||||
|
elif isinstance(content, list):
|
||||||
|
out.extend(b.get("text", "") for b in content if isinstance(b, dict))
|
||||||
|
return "\n".join(out)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_reminder_follows_a_silent_turn_and_costs_no_extra_call(tmp_path):
|
||||||
|
node, state = _make_node(tmp_path)
|
||||||
|
seen = []
|
||||||
|
replies = iter(
|
||||||
|
[
|
||||||
|
_click_response(""), # turn 1: silent
|
||||||
|
_click_response("Tapping the Driving tab."), # turn 2: reminded, speaks
|
||||||
|
_click_response([{"type": "thinking", "thinking": "internal"}]), # turn 3: silent
|
||||||
|
_click_response("Done."), # turn 4
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
async def ainvoke(messages, *args, **kwargs):
|
||||||
|
seen.append(list(messages))
|
||||||
|
return next(replies)
|
||||||
|
|
||||||
|
llm = MagicMock()
|
||||||
|
llm.ainvoke = AsyncMock(side_effect=ainvoke)
|
||||||
|
llm.bind_tools.return_value = llm
|
||||||
|
with patch("artemis.agents.operator.operator.get_llm", return_value=llm):
|
||||||
|
for _ in range(4):
|
||||||
|
await node(state)
|
||||||
|
|
||||||
|
assert llm.ainvoke.await_count == 4, "one model call per turn, never a bounce"
|
||||||
|
assert REASONING_REMINDER not in _texts(seen[0]) # nothing to remind on turn 1
|
||||||
|
assert REASONING_REMINDER in _texts(seen[1]) # turn 1 was silent
|
||||||
|
assert REASONING_REMINDER not in _texts(seen[2]) # turn 2 spoke
|
||||||
|
assert REASONING_REMINDER in _texts(seen[3]) # thinking-only turn 3 counts as silent
|
||||||
|
assert _texts(seen[3]).count(REASONING_REMINDER) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_ledger_silence_verdict_outranks_the_local_flag(tmp_path):
|
||||||
|
"""With a transcript ledger, the ledger's own judgment of the previous
|
||||||
|
turn is authoritative; the local flag only serves the legacy path."""
|
||||||
|
from artemis.memory import TranscriptLedger
|
||||||
|
|
||||||
|
node, _ = _make_node(tmp_path)
|
||||||
|
node._previous_turn_silent = True
|
||||||
|
assert node._previous_turn_was_silent() is True # no ledger: local flag
|
||||||
|
|
||||||
|
ledger = MagicMock(spec=TranscriptLedger)
|
||||||
|
ledger.last_turn_silent = False
|
||||||
|
node.ctx.transcript_ledger = ledger
|
||||||
|
assert node._previous_turn_was_silent() is False
|
||||||
|
|
||||||
|
ledger.last_turn_silent = True
|
||||||
|
node._previous_turn_silent = False
|
||||||
|
assert node._previous_turn_was_silent() is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_prompt_asks_for_one_paragraph_not_phase_sections():
|
||||||
|
prompt = apply_operator_prompt_contract(load_operator_prompts()["main_template"])
|
||||||
|
assert prompt.count("one short paragraph of connected prose reflecting these three phases") == 1
|
||||||
|
assert "in your raw thinking" not in prompt
|
||||||
|
assert "in your written reply" not in prompt
|
||||||
|
assert (
|
||||||
|
"bounced"
|
||||||
|
not in prompt.split("# THE THREE-PHASE COGNITIVE PROTOCOL")[1].split("## Phase 1")[0]
|
||||||
|
)
|
||||||
@@ -45,6 +45,7 @@ from artemis.config.agent import MemoryTranscriptConfig
|
|||||||
from artemis.context import ArtemisContext
|
from artemis.context import ArtemisContext
|
||||||
from artemis.memory.step_memory import StepMemoryService
|
from artemis.memory.step_memory import StepMemoryService
|
||||||
from artemis.memory.transcript import (
|
from artemis.memory.transcript import (
|
||||||
|
EPHEMERAL_BLOCKS_KEY,
|
||||||
EXECUTION_RESULT_MARKER,
|
EXECUTION_RESULT_MARKER,
|
||||||
PLAN_RECITATION_MARKER,
|
PLAN_RECITATION_MARKER,
|
||||||
RESTORED_HISTORY_HEADER,
|
RESTORED_HISTORY_HEADER,
|
||||||
@@ -53,8 +54,8 @@ from artemis.memory.transcript import (
|
|||||||
|
|
||||||
# SHA-256 snapshots of the legacy system message with the fixed inputs below.
|
# SHA-256 snapshots of the legacy system message with the fixed inputs below.
|
||||||
# Update these when an intentional template change alters the rendered prompt.
|
# Update these when an intentional template change alters the rendered prompt.
|
||||||
GOLDEN_EMPTY_PLAN = "d8cf83565c766d32893b124c974060bbd98de2d862c3b9925e546ea0c946c324"
|
GOLDEN_EMPTY_PLAN = "513e0341014b9a2342f608ad4dd173b7dcf5704ce53ee028ba302a9a1b696f96"
|
||||||
GOLDEN_SENTINEL_PLAN = "8f871b5e5700ba5ee07ed03b80a8f82664abe68b0c170d3185bf1ee35f7821ed"
|
GOLDEN_SENTINEL_PLAN = "012fe1b08645266d5a7e518023e20fc6fbfa65d4f2d985847ebb06e0184c67ee"
|
||||||
|
|
||||||
SCREENSHOT_B64 = base64.b64encode(b"fake-jpeg-bytes").decode("utf-8")
|
SCREENSHOT_B64 = base64.b64encode(b"fake-jpeg-bytes").decode("utf-8")
|
||||||
|
|
||||||
@@ -103,6 +104,8 @@ def test_transcript_static_system_is_stable_and_carries_no_history():
|
|||||||
# ...and no unrendered slot or trailing observation header survives.
|
# ...and no unrendered slot or trailing observation header survives.
|
||||||
assert "{{" not in first
|
assert "{{" not in first
|
||||||
assert not first.endswith("# CURRENT OBSERVATION\n")
|
assert not first.endswith("# CURRENT OBSERVATION\n")
|
||||||
|
# The history guide names the marker a resolved screenshot renders as.
|
||||||
|
assert "--- Historical Visual Transition ---" in first
|
||||||
|
|
||||||
|
|
||||||
def _transcript_ctx():
|
def _transcript_ctx():
|
||||||
@@ -214,7 +217,9 @@ async def test_transcript_mode_two_turns_build_four_regions(tmp_path):
|
|||||||
# Committed turn-1 observation followed by its validator result.
|
# Committed turn-1 observation followed by its validator result.
|
||||||
assert any(EXECUTION_RESULT_MARKER in str(m.content) for m in active)
|
assert any(EXECUTION_RESULT_MARKER in str(m.content) for m in active)
|
||||||
committed_obs = active[0]
|
committed_obs = active[0]
|
||||||
# Depth-1 scrub removed the old plan recitation copy.
|
# Text edge at depth 1: the committed observation has lost its plan
|
||||||
|
# recitation as soon as the next tail exists (its screenshot stays
|
||||||
|
# until the screenshot edge).
|
||||||
assert PLAN_RECITATION_MARKER not in str(committed_obs.content)
|
assert PLAN_RECITATION_MARKER not in str(committed_obs.content)
|
||||||
# The live tail still recites the plan.
|
# The live tail still recites the plan.
|
||||||
assert PLAN_RECITATION_MARKER in str(turn2_messages[-1].content)
|
assert PLAN_RECITATION_MARKER in str(turn2_messages[-1].content)
|
||||||
@@ -223,6 +228,52 @@ async def test_transcript_mode_two_turns_build_four_regions(tmp_path):
|
|||||||
assert re.search(r"T\+\d{2,}:\d{2}", str(turn2_messages[-1].content))
|
assert re.search(r"T\+\d{2,}:\d{2}", str(turn2_messages[-1].content))
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_transcript_tail_marks_per_turn_notices_ephemeral():
|
||||||
|
"""The hand-built tail carries the builder's ephemeral indices: per-turn
|
||||||
|
notices are deleted by the scrub edge; header, plan recitation and the
|
||||||
|
observation itself are not."""
|
||||||
|
from artemis.agents.operator.prompts import USER_GUIDANCE_MARKER
|
||||||
|
|
||||||
|
ctx = _transcript_ctx()
|
||||||
|
captured: list = []
|
||||||
|
mock_llm = _no_action_llm(captured)
|
||||||
|
with patch("artemis.agents.operator.operator.get_llm", return_value=mock_llm):
|
||||||
|
node = OperatorNode(ctx, transcript_config=MemoryTranscriptConfig(enabled=True))
|
||||||
|
await node(
|
||||||
|
_transcript_state(
|
||||||
|
injected_instruction="Skip the login step",
|
||||||
|
user_stop_requested=False,
|
||||||
|
operator_tool_limit_exceeded=True,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
tail = captured[0][-1]
|
||||||
|
assert isinstance(tail, HumanMessage)
|
||||||
|
texts = {
|
||||||
|
i: block.get("text", "")
|
||||||
|
for i, block in enumerate(tail.content)
|
||||||
|
if isinstance(block, dict) and block.get("type") == "text"
|
||||||
|
}
|
||||||
|
guidance = next(i for i, t in texts.items() if t.startswith(USER_GUIDANCE_MARKER))
|
||||||
|
warning = next(i for i, t in texts.items() if "tool-call budget" in t)
|
||||||
|
assert set(tail.additional_kwargs[EPHEMERAL_BLOCKS_KEY]) == {guidance, warning}
|
||||||
|
# The user's stop bit is relayed on the guidance block itself.
|
||||||
|
assert "authorized stopping" not in texts[guidance]
|
||||||
|
# The verbatim instruction is its own persistent block right after the
|
||||||
|
# wrapper (stamped with the ledger's session offset), never ephemeral:
|
||||||
|
# a standing instruction must outlive the turn it arrived on.
|
||||||
|
body = guidance + 1
|
||||||
|
assert re.fullmatch(r'User instruction \(T\+\d{2,}:\d{2}\): "Skip the login step"', texts[body])
|
||||||
|
assert body not in tail.additional_kwargs[EPHEMERAL_BLOCKS_KEY]
|
||||||
|
assert "Skip the login step" not in texts[guidance]
|
||||||
|
# Header, plan recitation and the observation stay in the history.
|
||||||
|
assert texts[0].startswith("# CURRENT OBSERVATION")
|
||||||
|
plan = next(i for i, t in texts.items() if t.startswith(PLAN_RECITATION_MARKER))
|
||||||
|
screenshot = next(i for i, t in texts.items() if t == "--- Current Screenshot ---")
|
||||||
|
assert not {0, plan, screenshot} & set(tail.additional_kwargs[EPHEMERAL_BLOCKS_KEY])
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_transcript_actionless_turn_commits_without_validator_message():
|
async def test_transcript_actionless_turn_commits_without_validator_message():
|
||||||
ctx = _transcript_ctx()
|
ctx = _transcript_ctx()
|
||||||
|
|||||||
@@ -287,7 +287,7 @@ async def test_outputter_builds_concise_history(mock_get_llm):
|
|||||||
}
|
}
|
||||||
]
|
]
|
||||||
mock_data_engine.get_agent_friendly_steps.return_value = steps
|
mock_data_engine.get_agent_friendly_steps.return_value = steps
|
||||||
# M5: the transcript flag defaults on, so the outputter's compiled view
|
# The transcript flag defaults on, so the outputter's compiled view
|
||||||
# consults the chunk store; no persisted chunks keeps the classic path.
|
# consults the chunk store; no persisted chunks keeps the classic path.
|
||||||
mock_data_engine.get_history_chunks.return_value = []
|
mock_data_engine.get_history_chunks.return_value = []
|
||||||
mock_data_engine.base_dir = "/tmp/mock_session"
|
mock_data_engine.base_dir = "/tmp/mock_session"
|
||||||
|
|||||||
@@ -12,7 +12,7 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
"""SummarizerNode tests (M2: visual-transition lens dispatch).
|
"""Tests for SummarizerNode visual-transition dispatch.
|
||||||
|
|
||||||
Per the history redesign §5 the Pro SummarizerNode no longer runs its own LLM
|
Per the history redesign §5 the Pro SummarizerNode no longer runs its own LLM
|
||||||
call to produce a text capsule: it dispatches the step to the shared
|
call to produce a text capsule: it dispatches the step to the shared
|
||||||
|
|||||||
@@ -0,0 +1,364 @@
|
|||||||
|
# Copyright 2026 Google LLC
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
"""Unit tests for the end-to-end device observation smoke test (no real device)."""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import base64
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from unittest.mock import AsyncMock, MagicMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from artemis.core.diagnostics import smoke_test_device
|
||||||
|
from artemis.core.diagnostics import device_smoke
|
||||||
|
from artemis.runtime.device_lock import DeviceExecutionLock, DeviceLockOwner
|
||||||
|
|
||||||
|
SERIAL = "59100DLCR0033X"
|
||||||
|
FAKE_JPEG = b"\xff\xd8\xff" + bytes(range(256)) * 8 + b"\xff\xd9"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def idle_devices(monkeypatch):
|
||||||
|
"""No live lock owners unless a test installs some."""
|
||||||
|
monkeypatch.setattr(DeviceExecutionLock, "get_active_owners", classmethod(lambda cls: {}))
|
||||||
|
monkeypatch.delenv("ARTEMIS_DEVICE_ID", raising=False)
|
||||||
|
monkeypatch.delenv("ADB_DEVICE_SERIAL", raising=False)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def patch_controller(monkeypatch):
|
||||||
|
"""Install a fake ``_get_controller`` and return the controller it hands out."""
|
||||||
|
|
||||||
|
def _install(get_screen_data=None, factory=None):
|
||||||
|
controller = MagicMock()
|
||||||
|
controller.ctx = SimpleNamespace(device=SimpleNamespace(device_id=SERIAL))
|
||||||
|
if get_screen_data is not None:
|
||||||
|
controller.get_screen_data = get_screen_data
|
||||||
|
calls: list[str | None] = []
|
||||||
|
|
||||||
|
def _fake_get_controller(device_serial=None):
|
||||||
|
calls.append(device_serial)
|
||||||
|
if factory is not None:
|
||||||
|
return factory(device_serial)
|
||||||
|
return controller
|
||||||
|
|
||||||
|
monkeypatch.setattr("artemis.mcp.adb_server._get_controller", _fake_get_controller)
|
||||||
|
controller.calls = calls
|
||||||
|
return controller
|
||||||
|
|
||||||
|
return _install
|
||||||
|
|
||||||
|
|
||||||
|
def _screen_data(elements, jpeg=FAKE_JPEG):
|
||||||
|
return SimpleNamespace(
|
||||||
|
base64=base64.b64encode(jpeg).decode("ascii"),
|
||||||
|
elements=elements,
|
||||||
|
width=1080,
|
||||||
|
height=2400,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_success_reports_bytes_elements_serial_and_elapsed(patch_controller):
|
||||||
|
elements = [{"class": "android.widget.FrameLayout"}, {"text": "Hello"}, {"text": "World"}]
|
||||||
|
controller = patch_controller(AsyncMock(return_value=_screen_data(elements)))
|
||||||
|
|
||||||
|
result = await smoke_test_device(device_serial=SERIAL, timeout_seconds=5)
|
||||||
|
|
||||||
|
assert result["ok"] is True
|
||||||
|
assert result["error"] is None
|
||||||
|
assert result["fix"] == []
|
||||||
|
assert result["serial"] == SERIAL
|
||||||
|
assert result["screenshot_bytes"] == len(FAKE_JPEG)
|
||||||
|
assert result["element_count"] == 3
|
||||||
|
assert isinstance(result["elapsed_seconds"], float)
|
||||||
|
assert result["elapsed_seconds"] >= 0.0
|
||||||
|
assert controller.calls == [SERIAL]
|
||||||
|
controller.get_screen_data.assert_awaited_once()
|
||||||
|
assert set(result) == {
|
||||||
|
"ok",
|
||||||
|
"serial",
|
||||||
|
"elapsed_seconds",
|
||||||
|
"screenshot_bytes",
|
||||||
|
"element_count",
|
||||||
|
"error",
|
||||||
|
"fix",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_serial_comes_from_controller_when_not_requested(patch_controller):
|
||||||
|
patch_controller(AsyncMock(return_value=_screen_data([{"text": "x"}])))
|
||||||
|
|
||||||
|
result = await smoke_test_device()
|
||||||
|
|
||||||
|
assert result["ok"] is True
|
||||||
|
assert result["serial"] == SERIAL
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_xml_string_elements_are_counted_by_node(patch_controller):
|
||||||
|
xml = '<hierarchy><node class="a"><node class="b"/><node class="c"/></node></hierarchy>'
|
||||||
|
patch_controller(AsyncMock(return_value=_screen_data(xml)))
|
||||||
|
|
||||||
|
result = await smoke_test_device(device_serial=SERIAL)
|
||||||
|
|
||||||
|
assert result["ok"] is True
|
||||||
|
assert result["element_count"] == 3
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_init_failure_no_devices_points_to_launch_avd(patch_controller):
|
||||||
|
def _boom(device_serial):
|
||||||
|
raise Exception("No Android devices found at localhost:5037")
|
||||||
|
|
||||||
|
patch_controller(factory=_boom)
|
||||||
|
|
||||||
|
result = await smoke_test_device(device_serial=SERIAL, timeout_seconds=5)
|
||||||
|
|
||||||
|
assert result["ok"] is False
|
||||||
|
assert result["serial"] == SERIAL
|
||||||
|
assert result["screenshot_bytes"] is None
|
||||||
|
assert result["element_count"] is None
|
||||||
|
assert "No Android devices found" in result["error"]
|
||||||
|
assert any("launch_avd" in step for step in result["fix"])
|
||||||
|
assert any("mobile_diagnose" in step for step in result["fix"])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_init_failure_target_not_found_points_to_launch_avd(patch_controller):
|
||||||
|
def _boom(device_serial):
|
||||||
|
raise Exception(
|
||||||
|
f"Target Android device '{device_serial}' not found at localhost:5037 among available devices."
|
||||||
|
)
|
||||||
|
|
||||||
|
patch_controller(factory=_boom)
|
||||||
|
|
||||||
|
result = await smoke_test_device(device_serial="emulator-5554")
|
||||||
|
|
||||||
|
assert result["ok"] is False
|
||||||
|
assert result["serial"] == "emulator-5554"
|
||||||
|
assert any("launch_avd" in step for step in result["fix"])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_screen_data_timeout_reports_timeout_and_force_stop(patch_controller):
|
||||||
|
async def _hang():
|
||||||
|
await asyncio.sleep(3)
|
||||||
|
return _screen_data([{"text": "late"}])
|
||||||
|
|
||||||
|
patch_controller(_hang)
|
||||||
|
|
||||||
|
result = await smoke_test_device(device_serial=SERIAL, timeout_seconds=0.3)
|
||||||
|
|
||||||
|
assert result["ok"] is False
|
||||||
|
assert result["serial"] == SERIAL
|
||||||
|
assert result["error"] == "UIAutomator/screen capture did not respond within 0.3s"
|
||||||
|
assert result["elapsed_seconds"] < 2.5
|
||||||
|
joined = "\n".join(result["fix"])
|
||||||
|
assert f"adb -s {SERIAL} shell am force-stop com.github.uiautomator" in joined
|
||||||
|
assert "uiautomator2 init" in joined
|
||||||
|
assert "screen on" in joined
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_init_timeout_is_reported_without_blocking(patch_controller):
|
||||||
|
import time
|
||||||
|
|
||||||
|
def _slow(device_serial):
|
||||||
|
time.sleep(2)
|
||||||
|
return MagicMock()
|
||||||
|
|
||||||
|
patch_controller(factory=_slow)
|
||||||
|
|
||||||
|
result = await smoke_test_device(device_serial=SERIAL, timeout_seconds=0.3)
|
||||||
|
|
||||||
|
assert result["ok"] is False
|
||||||
|
assert "did not respond within 0.3s" in result["error"]
|
||||||
|
assert result["elapsed_seconds"] < 1.5
|
||||||
|
assert any("force-stop" in step for step in result["fix"])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_generic_screen_data_exception_falls_back_to_manual_dump(patch_controller):
|
||||||
|
patch_controller(AsyncMock(side_effect=RuntimeError("weird failure 42")))
|
||||||
|
|
||||||
|
result = await smoke_test_device(device_serial=SERIAL)
|
||||||
|
|
||||||
|
assert result["ok"] is False
|
||||||
|
assert result["error"] == "Screen capture failed: weird failure 42"
|
||||||
|
assert any("uiautomator dump" in step for step in result["fix"])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_base_exception_from_uiautomator2_is_captured_not_raised(patch_controller):
|
||||||
|
class HTTPTimeoutError(BaseException):
|
||||||
|
"""uiautomator2 errors derive from BaseException."""
|
||||||
|
|
||||||
|
patch_controller(
|
||||||
|
AsyncMock(side_effect=HTTPTimeoutError("HTTP request timeout: read timed out"))
|
||||||
|
)
|
||||||
|
|
||||||
|
result = await smoke_test_device(device_serial=SERIAL)
|
||||||
|
|
||||||
|
assert result["ok"] is False
|
||||||
|
assert "HTTPTimeoutError" in result["error"]
|
||||||
|
assert any("force-stop" in step for step in result["fix"])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_unauthorized_and_offline_guidance(patch_controller):
|
||||||
|
def _unauth(device_serial):
|
||||||
|
raise Exception("device unauthorized. This adb server's $ADB_VENDOR_KEYS is not set")
|
||||||
|
|
||||||
|
patch_controller(factory=_unauth)
|
||||||
|
result = await smoke_test_device(device_serial=SERIAL)
|
||||||
|
assert result["ok"] is False
|
||||||
|
assert any("USB debugging" in step for step in result["fix"])
|
||||||
|
|
||||||
|
def _offline(device_serial):
|
||||||
|
raise Exception("device offline")
|
||||||
|
|
||||||
|
patch_controller(factory=_offline)
|
||||||
|
result = await smoke_test_device(device_serial=SERIAL)
|
||||||
|
assert result["ok"] is False
|
||||||
|
assert any("adb reconnect" in step for step in result["fix"])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_empty_hierarchy_is_not_ok(patch_controller):
|
||||||
|
patch_controller(AsyncMock(return_value=_screen_data([])))
|
||||||
|
|
||||||
|
result = await smoke_test_device(device_serial=SERIAL)
|
||||||
|
|
||||||
|
assert result["ok"] is False
|
||||||
|
assert result["screenshot_bytes"] == len(FAKE_JPEG)
|
||||||
|
assert result["element_count"] == 0
|
||||||
|
assert "no UI elements" in result["error"]
|
||||||
|
assert any("force-stop" in step for step in result["fix"])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_placeholder_screenshot_is_not_ok(patch_controller):
|
||||||
|
one_pixel_png = base64.b64decode(
|
||||||
|
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
|
||||||
|
)
|
||||||
|
patch_controller(AsyncMock(return_value=_screen_data([{"text": "x"}], jpeg=one_pixel_png)))
|
||||||
|
|
||||||
|
result = await smoke_test_device(device_serial=SERIAL)
|
||||||
|
|
||||||
|
assert result["ok"] is False
|
||||||
|
assert result["screenshot_bytes"] == len(one_pixel_png)
|
||||||
|
assert "placeholder" in result["error"]
|
||||||
|
|
||||||
|
|
||||||
|
def _owner(device_id: str, description: str = "task-1") -> DeviceLockOwner:
|
||||||
|
return DeviceLockOwner(
|
||||||
|
pid=4242,
|
||||||
|
process_created_at=1.0,
|
||||||
|
token="tok",
|
||||||
|
device_id=device_id,
|
||||||
|
description=description,
|
||||||
|
acquired_at="2026-09-09T00:00:00Z",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_busy_device_is_reported_without_touching_the_controller(
|
||||||
|
patch_controller, monkeypatch
|
||||||
|
):
|
||||||
|
controller = patch_controller(AsyncMock(return_value=_screen_data([{"text": "x"}])))
|
||||||
|
monkeypatch.setattr(
|
||||||
|
DeviceExecutionLock,
|
||||||
|
"get_active_owners",
|
||||||
|
classmethod(lambda cls: {SERIAL: _owner(SERIAL, "youtube smoke")}),
|
||||||
|
)
|
||||||
|
|
||||||
|
result = await smoke_test_device(device_serial=SERIAL)
|
||||||
|
|
||||||
|
assert result["ok"] is False
|
||||||
|
assert result["serial"] == SERIAL
|
||||||
|
assert "busy" in result["error"]
|
||||||
|
assert "youtube smoke" in result["error"]
|
||||||
|
assert "4242" in result["error"]
|
||||||
|
assert any("mobile_manage_task" in step for step in result["fix"])
|
||||||
|
assert controller.calls == []
|
||||||
|
controller.get_screen_data.assert_not_awaited()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_busy_guard_matches_scoped_lock_keys(patch_controller, monkeypatch):
|
||||||
|
controller = patch_controller(AsyncMock(return_value=_screen_data([{"text": "x"}])))
|
||||||
|
monkeypatch.setattr(
|
||||||
|
DeviceExecutionLock,
|
||||||
|
"get_active_owners",
|
||||||
|
classmethod(lambda cls: {f"endpoint1__{SERIAL}": _owner(SERIAL)}),
|
||||||
|
)
|
||||||
|
|
||||||
|
result = await smoke_test_device(device_serial=SERIAL)
|
||||||
|
|
||||||
|
assert result["ok"] is False
|
||||||
|
assert "busy" in result["error"]
|
||||||
|
assert controller.calls == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_busy_guard_ignores_other_devices(patch_controller, monkeypatch):
|
||||||
|
controller = patch_controller(AsyncMock(return_value=_screen_data([{"text": "x"}])))
|
||||||
|
monkeypatch.setattr(
|
||||||
|
DeviceExecutionLock,
|
||||||
|
"get_active_owners",
|
||||||
|
classmethod(lambda cls: {"emulator-5554": _owner("emulator-5554")}),
|
||||||
|
)
|
||||||
|
|
||||||
|
result = await smoke_test_device(device_serial=SERIAL)
|
||||||
|
|
||||||
|
assert result["ok"] is True
|
||||||
|
assert controller.calls == [SERIAL]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_busy_guard_without_target_serial_treats_any_owner_as_busy(
|
||||||
|
patch_controller, monkeypatch
|
||||||
|
):
|
||||||
|
controller = patch_controller(AsyncMock(return_value=_screen_data([{"text": "x"}])))
|
||||||
|
monkeypatch.setattr(
|
||||||
|
DeviceExecutionLock,
|
||||||
|
"get_active_owners",
|
||||||
|
classmethod(lambda cls: {"emulator-5554": _owner("emulator-5554")}),
|
||||||
|
)
|
||||||
|
|
||||||
|
result = await smoke_test_device()
|
||||||
|
|
||||||
|
assert result["ok"] is False
|
||||||
|
assert result["serial"] == "emulator-5554"
|
||||||
|
assert "busy" in result["error"]
|
||||||
|
assert controller.calls == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_fix_for_error_generic_fallback_uses_serial():
|
||||||
|
fix = device_smoke.fix_for_error("something odd", SERIAL)
|
||||||
|
assert fix
|
||||||
|
assert all(SERIAL in step for step in fix)
|
||||||
|
assert any("uiautomator dump" in step for step in fix)
|
||||||
|
assert device_smoke.fix_for_error(None, SERIAL) == []
|
||||||
|
assert "<serial>" in device_smoke.fix_for_error("timeout", None)[1]
|
||||||
|
|
||||||
|
|
||||||
|
def test_smoke_test_is_exported_from_diagnostics_package():
|
||||||
|
import artemis.core.diagnostics as diagnostics
|
||||||
|
|
||||||
|
assert "smoke_test_device" in diagnostics.__all__
|
||||||
|
assert diagnostics.smoke_test_device is device_smoke.smoke_test_device
|
||||||
@@ -429,3 +429,87 @@ async def test_emulator_manager_lifecycle():
|
|||||||
dismiss_res = manager.dismiss()
|
dismiss_res = manager.dismiss()
|
||||||
assert dismiss_res["success"] is True
|
assert dismiss_res["success"] is True
|
||||||
assert manager.get_status().status == EmulatorLaunchStage.IDLE
|
assert manager.get_status().status == EmulatorLaunchStage.IDLE
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_build_report_turns_crashing_probe_into_fail_result():
|
||||||
|
engine = ReadinessEngine()
|
||||||
|
healthy = ProbeResult(
|
||||||
|
id="healthy",
|
||||||
|
category=ProbeCategory.RUNTIME,
|
||||||
|
title="Healthy",
|
||||||
|
status=ProbeStatus.PASS,
|
||||||
|
is_blocker=True,
|
||||||
|
summary="Ready",
|
||||||
|
description="Ready",
|
||||||
|
)
|
||||||
|
good = Mock()
|
||||||
|
good.probe_id = "healthy"
|
||||||
|
good.category = ProbeCategory.RUNTIME
|
||||||
|
good.is_blocker = True
|
||||||
|
good.probe = AsyncMock(return_value=healthy)
|
||||||
|
|
||||||
|
bad = Mock()
|
||||||
|
bad.probe_id = "integration_host"
|
||||||
|
bad.category = ProbeCategory.RUNTIME
|
||||||
|
bad.is_blocker = True
|
||||||
|
bad.probe = AsyncMock(side_effect=PermissionError(13, "Permission denied", "/ro/traces"))
|
||||||
|
engine._probes = {"healthy": good, "integration_host": bad}
|
||||||
|
|
||||||
|
report = await engine._build_report()
|
||||||
|
|
||||||
|
assert report.overall_ready is False
|
||||||
|
assert report.blocker_count == 2
|
||||||
|
assert report.passed_blocker_count == 1
|
||||||
|
by_id = {r.id: r for r in report.probes}
|
||||||
|
assert by_id["healthy"].status is ProbeStatus.PASS
|
||||||
|
crashed = by_id["integration_host"]
|
||||||
|
assert crashed.status is ProbeStatus.FAIL
|
||||||
|
assert crashed.is_blocker is True
|
||||||
|
assert crashed.category is ProbeCategory.RUNTIME
|
||||||
|
assert crashed.summary == "Probe crashed"
|
||||||
|
assert "PermissionError" in crashed.description
|
||||||
|
assert "Permission denied" in crashed.description
|
||||||
|
assert crashed.metadata["exception_type"] == "PermissionError"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_build_report_turns_hung_probe_into_fail_result(monkeypatch):
|
||||||
|
"""A probe that never returns (wedged ADB server) is cut off at the
|
||||||
|
engine's per-probe deadline and reported as a FAIL with the restart step,
|
||||||
|
so no surface running the report can hang."""
|
||||||
|
monkeypatch.setattr(ReadinessEngine, "PROBE_TIMEOUT_SECONDS", 0.05)
|
||||||
|
engine = ReadinessEngine()
|
||||||
|
|
||||||
|
async def never_returns():
|
||||||
|
await asyncio.sleep(10)
|
||||||
|
|
||||||
|
hung = Mock()
|
||||||
|
hung.probe_id = "android_adb"
|
||||||
|
hung.category = ProbeCategory.DEVICE
|
||||||
|
hung.is_blocker = True
|
||||||
|
hung.probe = never_returns
|
||||||
|
engine._probes = {"android_adb": hung}
|
||||||
|
|
||||||
|
report = await engine._build_report()
|
||||||
|
|
||||||
|
result = report.probes[0]
|
||||||
|
assert result.status is ProbeStatus.FAIL
|
||||||
|
assert result.summary == "Probe timed out"
|
||||||
|
assert result.metadata["exception_type"] == "TimeoutError"
|
||||||
|
assert any("adb kill-server" in a.payload for a in result.actions)
|
||||||
|
assert report.overall_ready is False
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_build_report_does_not_swallow_cancellation():
|
||||||
|
engine = ReadinessEngine()
|
||||||
|
probe = Mock()
|
||||||
|
probe.probe_id = "cancelled"
|
||||||
|
probe.category = ProbeCategory.RUNTIME
|
||||||
|
probe.is_blocker = True
|
||||||
|
probe.probe = AsyncMock(side_effect=asyncio.CancelledError())
|
||||||
|
engine._probes = {"cancelled": probe}
|
||||||
|
|
||||||
|
with pytest.raises(asyncio.CancelledError):
|
||||||
|
await engine._build_report()
|
||||||
@@ -0,0 +1,471 @@
|
|||||||
|
# Copyright 2026 Google LLC
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
"""Unit tests for the MCP / IDE integration-host readiness probe."""
|
||||||
|
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from artemis.core.diagnostics import IntegrationHostProbe, ReadinessEngine
|
||||||
|
from artemis.core.diagnostics.probes import host_probe
|
||||||
|
from artemis.core.diagnostics.schema import ProbeCategory, ProbeStatus
|
||||||
|
|
||||||
|
|
||||||
|
#: Environment keys / prefixes any IDE might export; scrubbed so the developer's
|
||||||
|
#: own host (this test suite is often run from inside Claude Code or Cursor) does
|
||||||
|
#: not leak into the detection tests.
|
||||||
|
_SIGNAL_PREFIXES = (
|
||||||
|
"ANTIGRAVITY_",
|
||||||
|
"CLAUDE_CODE_",
|
||||||
|
"CURSOR_",
|
||||||
|
"WINDSURF_",
|
||||||
|
"CODEX_",
|
||||||
|
"OPENCLAW_",
|
||||||
|
"CLINE_",
|
||||||
|
"ROO_",
|
||||||
|
"VSCODE_",
|
||||||
|
)
|
||||||
|
_SIGNAL_KEYS = ("CLAUDECODE", "TERM_PROGRAM")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def clean_ide_env(monkeypatch):
|
||||||
|
for key in list(os.environ):
|
||||||
|
if key in _SIGNAL_KEYS or key.startswith(_SIGNAL_PREFIXES):
|
||||||
|
monkeypatch.delenv(key, raising=False)
|
||||||
|
return monkeypatch
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def host_env(tmp_path, monkeypatch, clean_ide_env):
|
||||||
|
"""A consistent, healthy host layout the individual tests then break."""
|
||||||
|
venv_python = host_probe.project_venv_python(tmp_path)
|
||||||
|
venv_python.parent.mkdir(parents=True)
|
||||||
|
venv_python.write_bytes(b"")
|
||||||
|
traces = tmp_path / "traces"
|
||||||
|
|
||||||
|
monkeypatch.setattr("artemis.config.paths.ROOT_DIR", tmp_path)
|
||||||
|
monkeypatch.setattr("artemis.config.paths.get_env_file", lambda: tmp_path / ".env")
|
||||||
|
monkeypatch.setattr("artemis.config.paths.get_default_traces_path", lambda: traces)
|
||||||
|
monkeypatch.setattr("artemis.config.paths.is_source_checkout", lambda: True)
|
||||||
|
monkeypatch.setattr(sys, "executable", str(venv_python))
|
||||||
|
|
||||||
|
monkeypatch.setattr("artemis.runtime.daemon_client.is_standalone_forced", lambda: False)
|
||||||
|
monkeypatch.setattr("artemis.runtime.daemon_client.is_artemis_daemon", lambda: True)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"artemis.runtime.daemon_client.daemon_log_path", lambda: tmp_path / "daemon.log"
|
||||||
|
)
|
||||||
|
monkeypatch.setattr("artemis.runtime.server_lifecycle.is_port_in_use", lambda *a, **k: True)
|
||||||
|
return tmp_path
|
||||||
|
|
||||||
|
|
||||||
|
def test_probe_is_not_part_of_the_default_console_report():
|
||||||
|
assert "integration_host" not in ReadinessEngine()._probes
|
||||||
|
probe = IntegrationHostProbe()
|
||||||
|
assert probe.probe_id == "integration_host"
|
||||||
|
assert probe.category is ProbeCategory.RUNTIME
|
||||||
|
assert probe.is_blocker is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_consistent_host_passes(host_env):
|
||||||
|
result = await IntegrationHostProbe().probe()
|
||||||
|
|
||||||
|
assert result.status is ProbeStatus.PASS
|
||||||
|
assert result.is_blocker is True
|
||||||
|
assert result.summary == "Host Ready"
|
||||||
|
meta = result.metadata
|
||||||
|
assert meta["venv_exists"] is True
|
||||||
|
assert meta["interpreter_matches_venv"] is True
|
||||||
|
assert Path(meta["runner_python"]) == host_probe.project_venv_python(host_env)
|
||||||
|
assert meta["env_file_exists"] is False
|
||||||
|
assert meta["traces_dir_writable"] is True
|
||||||
|
assert (host_env / "traces").is_dir()
|
||||||
|
assert meta["daemon"] == {
|
||||||
|
"host": "127.0.0.1",
|
||||||
|
"port": 8000,
|
||||||
|
"standalone_forced": False,
|
||||||
|
"reachable": True,
|
||||||
|
"port_in_use": True,
|
||||||
|
"port_held_by_other_process": False,
|
||||||
|
"log_path": str(host_env / "daemon.log"),
|
||||||
|
}
|
||||||
|
labels = [a.label for a in result.actions]
|
||||||
|
assert labels[0] == "Host OK"
|
||||||
|
assert "Env File Location" in labels # informational even when passing
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_daemon_port_held_by_other_process_warns(host_env, monkeypatch):
|
||||||
|
monkeypatch.setattr("artemis.runtime.daemon_client.is_artemis_daemon", lambda: False)
|
||||||
|
|
||||||
|
result = await IntegrationHostProbe().probe()
|
||||||
|
|
||||||
|
assert result.status is ProbeStatus.WARN
|
||||||
|
assert result.is_blocker is False # tasks still run standalone
|
||||||
|
assert result.metadata["daemon"]["port_held_by_other_process"] is True
|
||||||
|
assert "port 8000 is held" in result.description.lower()
|
||||||
|
port_fix = next(a for a in result.actions if a.label == "Free the Daemon Port")
|
||||||
|
assert "ARTEMIS_DAEMON_PORT" in port_fix.payload
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_daemon_not_running_and_port_free_is_informational(host_env, monkeypatch):
|
||||||
|
monkeypatch.setattr("artemis.runtime.daemon_client.is_artemis_daemon", lambda: False)
|
||||||
|
monkeypatch.setattr("artemis.runtime.server_lifecycle.is_port_in_use", lambda *a, **k: False)
|
||||||
|
|
||||||
|
result = await IntegrationHostProbe().probe()
|
||||||
|
|
||||||
|
assert result.status is ProbeStatus.PASS
|
||||||
|
assert "auto-start" in result.description
|
||||||
|
auto = next(a for a in result.actions if a.label == "Daemon Auto-Start")
|
||||||
|
assert str(host_env / "daemon.log") in auto.payload
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_standalone_mode_skips_daemon_probe(host_env, monkeypatch):
|
||||||
|
monkeypatch.setattr("artemis.runtime.daemon_client.is_standalone_forced", lambda: True)
|
||||||
|
monkeypatch.setattr("artemis.runtime.daemon_client.is_artemis_daemon", lambda: False)
|
||||||
|
|
||||||
|
result = await IntegrationHostProbe().probe()
|
||||||
|
|
||||||
|
assert result.status is ProbeStatus.PASS
|
||||||
|
assert result.metadata["daemon"]["standalone_forced"] is True
|
||||||
|
assert result.metadata["daemon"]["port_held_by_other_process"] is False
|
||||||
|
auto = next(a for a in result.actions if a.label == "Daemon Auto-Start")
|
||||||
|
assert "ARTEMIS_STANDALONE" in auto.payload
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_interpreter_mismatch_warns_and_points_at_installer(host_env, monkeypatch):
|
||||||
|
other = host_env / "other-python.exe"
|
||||||
|
other.write_bytes(b"")
|
||||||
|
monkeypatch.setattr(sys, "executable", str(other))
|
||||||
|
|
||||||
|
result = await IntegrationHostProbe().probe()
|
||||||
|
|
||||||
|
assert result.status is ProbeStatus.WARN
|
||||||
|
assert result.metadata["interpreter_matches_venv"] is False
|
||||||
|
assert Path(result.metadata["runner_python"]) == host_probe.project_venv_python(host_env)
|
||||||
|
assert result.metadata["mcp_client"] is None
|
||||||
|
assert not [a for a in result.actions if a.action_type == "command"]
|
||||||
|
regen = next(a for a in result.actions if a.label == "Regenerate MCP Config")
|
||||||
|
assert regen.action_type == "hint"
|
||||||
|
assert "ask the user which ide" in regen.payload.lower()
|
||||||
|
assert "uv run artemis mcp --install <client>" in regen.payload
|
||||||
|
for name in host_probe.MCP_CLIENT_NAMES:
|
||||||
|
assert name in regen.payload
|
||||||
|
assert next(a for a in result.actions if a.label == "Interpreter Mismatch")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_interpreter_mismatch_emits_runnable_install_for_detected_client(
|
||||||
|
host_env, monkeypatch
|
||||||
|
):
|
||||||
|
monkeypatch.setenv("CLAUDECODE", "1")
|
||||||
|
other = host_env / "other-python.exe"
|
||||||
|
other.write_bytes(b"")
|
||||||
|
monkeypatch.setattr(sys, "executable", str(other))
|
||||||
|
|
||||||
|
result = await IntegrationHostProbe().probe()
|
||||||
|
|
||||||
|
assert result.status is ProbeStatus.WARN
|
||||||
|
assert result.metadata["mcp_client"] == "claude"
|
||||||
|
regen = next(a for a in result.actions if a.action_type == "command")
|
||||||
|
assert regen.label == "Regenerate MCP Config"
|
||||||
|
assert regen.payload == "uv run artemis mcp --install claude"
|
||||||
|
assert "<client>" not in regen.payload
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_metadata_reports_client_even_when_host_is_healthy(host_env, monkeypatch):
|
||||||
|
monkeypatch.setenv("CURSOR_TRACE_ID", "abc")
|
||||||
|
|
||||||
|
result = await IntegrationHostProbe().probe()
|
||||||
|
|
||||||
|
assert result.status is ProbeStatus.PASS
|
||||||
|
assert result.metadata["mcp_client"] == "cursor"
|
||||||
|
|
||||||
|
|
||||||
|
def test_detect_mcp_client_unknown_in_clean_environment(clean_ide_env):
|
||||||
|
assert host_probe.detect_mcp_client() is None
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("variables", "expected"),
|
||||||
|
[
|
||||||
|
({"ANTIGRAVITY_LS_ADDRESS": "127.0.0.1:1"}, "antigravity"),
|
||||||
|
({"ANTIGRAVITY_CSRF_TOKEN": "t"}, "antigravity"),
|
||||||
|
({"CLAUDECODE": "1"}, "claude"),
|
||||||
|
({"CLAUDE_CODE_ENTRYPOINT": "cli"}, "claude"),
|
||||||
|
({"CURSOR_TRACE_ID": "abc"}, "cursor"),
|
||||||
|
({"CURSOR_AGENT": "1"}, "cursor"),
|
||||||
|
({"WINDSURF_SESSION": "x"}, "windsurf"),
|
||||||
|
({"CODEX_SANDBOX": "seatbelt"}, "codex"),
|
||||||
|
({"CODEX_HOME": "/tmp/codex", "CODEX_SANDBOX_NETWORK_DISABLED": "1"}, "codex"),
|
||||||
|
({"OPENCLAW_SESSION_ID": "s"}, "openclaw"),
|
||||||
|
({"CLINE_TASK_ID": "1"}, "cline"),
|
||||||
|
({"ROO_CODE_TASK": "1"}, "roo"),
|
||||||
|
({"VSCODE_PID": "42"}, "vscode"),
|
||||||
|
({"TERM_PROGRAM": "vscode"}, "vscode"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_detect_mcp_client_signals(clean_ide_env, variables, expected):
|
||||||
|
for key, value in variables.items():
|
||||||
|
clean_ide_env.setenv(key, value)
|
||||||
|
assert host_probe.detect_mcp_client() == expected
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"variables",
|
||||||
|
[
|
||||||
|
{"CODEX_HOME": "/home/me/.codex"}, # user relocated the Codex config dir
|
||||||
|
{"OPENCLAW_WEBHOOK_URL": "https://hooks.example/x"}, # notifier setting, any host
|
||||||
|
{"TERM_PROGRAM": "iTerm.app"},
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_detect_mcp_client_ignores_weak_or_user_set_variables(clean_ide_env, variables):
|
||||||
|
for key, value in variables.items():
|
||||||
|
clean_ide_env.setenv(key, value)
|
||||||
|
assert host_probe.detect_mcp_client() is None
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("variables", "expected"),
|
||||||
|
[
|
||||||
|
# Claude Code or Codex running inside a VS Code / Cursor terminal: the
|
||||||
|
# agent that spawned the server wins over the editor around it.
|
||||||
|
({"CLAUDECODE": "1", "VSCODE_PID": "1", "TERM_PROGRAM": "vscode"}, "claude"),
|
||||||
|
({"CLAUDECODE": "1", "CURSOR_TRACE_ID": "x"}, "claude"),
|
||||||
|
({"CODEX_SANDBOX": "seatbelt", "VSCODE_PID": "1"}, "codex"),
|
||||||
|
# Cursor / Windsurf / Cline are VS Code forks or extensions and inherit
|
||||||
|
# the generic VS Code markers.
|
||||||
|
({"CURSOR_TRACE_ID": "x", "VSCODE_PID": "1", "TERM_PROGRAM": "vscode"}, "cursor"),
|
||||||
|
({"WINDSURF_SESSION": "x", "TERM_PROGRAM": "vscode"}, "windsurf"),
|
||||||
|
({"CLINE_TASK_ID": "1", "VSCODE_PID": "1"}, "cline"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_detect_mcp_client_specific_host_beats_generic_vscode(clean_ide_env, variables, expected):
|
||||||
|
for key, value in variables.items():
|
||||||
|
clean_ide_env.setenv(key, value)
|
||||||
|
assert host_probe.detect_mcp_client() == expected
|
||||||
|
|
||||||
|
|
||||||
|
def test_mcp_install_actions_shapes():
|
||||||
|
(cmd,) = host_probe.mcp_install_actions("antigravity")
|
||||||
|
assert cmd.action_type == "command"
|
||||||
|
assert cmd.payload == "uv run artemis mcp --install antigravity"
|
||||||
|
|
||||||
|
(hint,) = host_probe.mcp_install_actions(None)
|
||||||
|
assert hint.action_type == "hint"
|
||||||
|
assert "antigravity, claude, cursor, windsurf, vscode, cline, roo, openclaw, codex" in (
|
||||||
|
hint.payload
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_missing_venv_warns_with_uv_sync(host_env, monkeypatch):
|
||||||
|
host_probe.project_venv_python(host_env).unlink()
|
||||||
|
monkeypatch.setattr(sys, "executable", str(host_env / "system-python.exe"))
|
||||||
|
|
||||||
|
result = await IntegrationHostProbe().probe()
|
||||||
|
|
||||||
|
assert result.status is ProbeStatus.WARN
|
||||||
|
assert result.metadata["venv_exists"] is False
|
||||||
|
assert result.metadata["runner_python"] == str(host_env / "system-python.exe")
|
||||||
|
assert any(a.payload == "uv sync" for a in result.actions)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_unwritable_traces_dir_fails_even_with_other_warnings(host_env, monkeypatch):
|
||||||
|
monkeypatch.setattr(host_probe, "directory_is_writable", lambda _p: (False, "denied"))
|
||||||
|
monkeypatch.setattr("artemis.runtime.daemon_client.is_artemis_daemon", lambda: False)
|
||||||
|
|
||||||
|
result = await IntegrationHostProbe().probe()
|
||||||
|
|
||||||
|
assert result.status is ProbeStatus.FAIL
|
||||||
|
assert result.is_blocker is True
|
||||||
|
assert result.summary == "Host Misconfigured"
|
||||||
|
assert result.metadata["traces_dir_writable"] is False
|
||||||
|
assert result.metadata["traces_dir_error"] == "denied"
|
||||||
|
assert result.actions[0].label == "Fix Traces Directory"
|
||||||
|
assert "ARTEMIS_TRACES_DIR" in result.actions[0].payload
|
||||||
|
# The port problem is still reported alongside the blocking failure.
|
||||||
|
assert "port 8000 is held" in result.description.lower()
|
||||||
|
|
||||||
|
|
||||||
|
def test_directory_is_writable_reports_real_errors(tmp_path):
|
||||||
|
ok, error = host_probe.directory_is_writable(tmp_path / "new" / "nested")
|
||||||
|
assert ok is True and error is None
|
||||||
|
|
||||||
|
blocker = tmp_path / "file"
|
||||||
|
blocker.write_text("x")
|
||||||
|
ok, error = host_probe.directory_is_writable(blocker / "child")
|
||||||
|
assert ok is False and error
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_wheel_install_without_venv_is_not_a_venv_warning(host_env, monkeypatch):
|
||||||
|
"""A wheel install has no .venv under site-packages; that is normal, not a warning."""
|
||||||
|
host_probe.project_venv_python(host_env).unlink()
|
||||||
|
system_python = host_env / "system-python.exe"
|
||||||
|
monkeypatch.setattr(sys, "executable", str(system_python))
|
||||||
|
monkeypatch.setattr("artemis.config.paths.is_source_checkout", lambda: False)
|
||||||
|
|
||||||
|
result = await IntegrationHostProbe().probe()
|
||||||
|
|
||||||
|
assert result.status is ProbeStatus.PASS
|
||||||
|
assert result.summary == "Host Ready"
|
||||||
|
meta = result.metadata
|
||||||
|
assert meta["is_source_checkout"] is False
|
||||||
|
assert meta["venv_python"] is None
|
||||||
|
assert meta["venv_exists"] is False
|
||||||
|
assert meta["interpreter_matches_venv"] is False
|
||||||
|
assert meta["runner_python"] == str(system_python)
|
||||||
|
assert "venv" not in result.description.lower()
|
||||||
|
assert not [a for a in result.actions if a.payload == "uv sync"]
|
||||||
|
assert not [a for a in result.actions if a.label == "Interpreter Mismatch"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_wheel_install_ignores_stray_venv_next_to_site_packages(host_env, monkeypatch):
|
||||||
|
"""Even if a .venv happens to exist under ROOT_DIR, a non-source install never compares against it."""
|
||||||
|
other = host_env / "other-python.exe"
|
||||||
|
other.write_bytes(b"")
|
||||||
|
monkeypatch.setattr(sys, "executable", str(other))
|
||||||
|
monkeypatch.setattr("artemis.config.paths.is_source_checkout", lambda: False)
|
||||||
|
|
||||||
|
result = await IntegrationHostProbe().probe()
|
||||||
|
|
||||||
|
assert result.status is ProbeStatus.PASS
|
||||||
|
assert result.metadata["venv_python"] is None
|
||||||
|
assert result.metadata["runner_python"] == str(other)
|
||||||
|
assert not [a for a in result.actions if a.label == "Interpreter Mismatch"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_source_checkout_without_venv_still_warns_with_uv_sync(host_env, monkeypatch):
|
||||||
|
host_probe.project_venv_python(host_env).unlink()
|
||||||
|
monkeypatch.setattr(sys, "executable", str(host_env / "system-python.exe"))
|
||||||
|
monkeypatch.setattr("artemis.config.paths.is_source_checkout", lambda: True)
|
||||||
|
|
||||||
|
result = await IntegrationHostProbe().probe()
|
||||||
|
|
||||||
|
assert result.status is ProbeStatus.WARN
|
||||||
|
assert result.metadata["is_source_checkout"] is True
|
||||||
|
assert result.metadata["venv_python"] == str(host_probe.project_venv_python(host_env))
|
||||||
|
assert result.metadata["venv_exists"] is False
|
||||||
|
assert "no project virtualenv" in result.description.lower()
|
||||||
|
assert any(a.payload == "uv sync" for a in result.actions)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_traces_path_creation_error_becomes_fail_not_crash(host_env, monkeypatch):
|
||||||
|
"""get_default_traces_path mkdirs as a side effect; a read-only parent must not crash the probe."""
|
||||||
|
|
||||||
|
def denied():
|
||||||
|
raise PermissionError(13, "Permission denied", str(host_env / "traces"))
|
||||||
|
|
||||||
|
monkeypatch.setattr("artemis.config.paths.get_default_traces_path", denied)
|
||||||
|
monkeypatch.setenv("ARTEMIS_TRACES_DIR", str(host_env / "ro" / "traces"))
|
||||||
|
|
||||||
|
result = await IntegrationHostProbe().probe()
|
||||||
|
|
||||||
|
assert result.status is ProbeStatus.FAIL
|
||||||
|
assert result.is_blocker is True
|
||||||
|
assert result.summary == "Host Misconfigured"
|
||||||
|
meta = result.metadata
|
||||||
|
assert meta["traces_dir_writable"] is False
|
||||||
|
assert "Permission denied" in meta["traces_dir_error"]
|
||||||
|
# The intended path is still named, resolved without touching the disk.
|
||||||
|
assert Path(meta["traces_dir"]) == host_env / "ro" / "traces"
|
||||||
|
assert not (host_env / "ro").exists()
|
||||||
|
assert result.actions[0].label == "Fix Traces Directory"
|
||||||
|
assert "ARTEMIS_TRACES_DIR" in result.actions[0].payload
|
||||||
|
assert "not writable" in result.description
|
||||||
|
|
||||||
|
|
||||||
|
def test_intended_traces_path_mirrors_selection_without_mkdir(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setattr("artemis.config.paths.ROOT_DIR", tmp_path)
|
||||||
|
monkeypatch.delenv("ARTEMIS_TRACES_DIR", raising=False)
|
||||||
|
|
||||||
|
monkeypatch.setattr("artemis.config.paths._use_user_app_dir", lambda: False)
|
||||||
|
assert host_probe.intended_traces_path() == tmp_path / "traces"
|
||||||
|
|
||||||
|
monkeypatch.setattr("artemis.config.paths._use_user_app_dir", lambda: True)
|
||||||
|
monkeypatch.setattr("artemis.config.paths.get_app_dir", lambda: tmp_path / "app")
|
||||||
|
assert host_probe.intended_traces_path() == tmp_path / "app" / "traces"
|
||||||
|
|
||||||
|
monkeypatch.setenv("ARTEMIS_TRACES_DIR", str(tmp_path / "env-traces"))
|
||||||
|
assert host_probe.intended_traces_path() == tmp_path / "env-traces"
|
||||||
|
|
||||||
|
assert not (tmp_path / "traces").exists()
|
||||||
|
assert not (tmp_path / "app").exists()
|
||||||
|
assert not (tmp_path / "env-traces").exists()
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Daemon identity (a dev server on port 8000 is not the daemon)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeResponse:
|
||||||
|
def __init__(self, status: int, body: bytes):
|
||||||
|
self.status = status
|
||||||
|
self._body = body
|
||||||
|
|
||||||
|
def read(self) -> bytes:
|
||||||
|
return self._body
|
||||||
|
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *exc):
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"status, body, expected",
|
||||||
|
[
|
||||||
|
(200, b'{"running": true, "port": 8000, "current_pid": 42}', True),
|
||||||
|
(200, b"<!doctype html><title>Vite App</title>", False), # dev server on /
|
||||||
|
(200, b'{"hello": "world"}', False),
|
||||||
|
(404, b"", False),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_is_artemis_daemon_requires_the_server_status_shape(monkeypatch, status, body, expected):
|
||||||
|
from artemis.runtime import daemon_client
|
||||||
|
|
||||||
|
seen: dict[str, str] = {}
|
||||||
|
|
||||||
|
def fake_urlopen(req, timeout):
|
||||||
|
seen["url"] = req.full_url
|
||||||
|
return _FakeResponse(status, body)
|
||||||
|
|
||||||
|
monkeypatch.setattr(daemon_client.urllib.request, "urlopen", fake_urlopen)
|
||||||
|
assert daemon_client.is_artemis_daemon("127.0.0.1", 8000) is expected
|
||||||
|
assert seen["url"] == "http://127.0.0.1:8000/api/system/server-status"
|
||||||
|
|
||||||
|
|
||||||
|
def test_is_artemis_daemon_is_false_when_nothing_listens(monkeypatch):
|
||||||
|
from artemis.runtime import daemon_client
|
||||||
|
|
||||||
|
def refuse(req, timeout):
|
||||||
|
raise ConnectionRefusedError()
|
||||||
|
|
||||||
|
monkeypatch.setattr(daemon_client.urllib.request, "urlopen", refuse)
|
||||||
|
assert daemon_client.is_artemis_daemon("127.0.0.1", 8000) is False
|
||||||
@@ -0,0 +1,157 @@
|
|||||||
|
# Copyright 2026 Google LLC
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
"""Unit tests for the readiness core shared by ``artemis doctor`` and ``mobile_diagnose``."""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import time
|
||||||
|
from unittest.mock import AsyncMock
|
||||||
|
|
||||||
|
from artemis.core.diagnostics import readiness
|
||||||
|
from artemis.core.diagnostics.probes.host_probe import IntegrationHostProbe
|
||||||
|
from artemis.core.diagnostics.readiness import (
|
||||||
|
CHECK_ORDER,
|
||||||
|
adb_keys_corrupted,
|
||||||
|
base_verdict,
|
||||||
|
collect_readiness,
|
||||||
|
sort_by_fix_order,
|
||||||
|
)
|
||||||
|
from artemis.core.diagnostics.schema import (
|
||||||
|
ProbeCategory,
|
||||||
|
ProbeResult,
|
||||||
|
ProbeStatus,
|
||||||
|
SystemReadinessReport,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _probe(
|
||||||
|
probe_id: str,
|
||||||
|
status: ProbeStatus = ProbeStatus.PASS,
|
||||||
|
*,
|
||||||
|
blocker: bool = True,
|
||||||
|
metadata: dict | None = None,
|
||||||
|
) -> ProbeResult:
|
||||||
|
return ProbeResult(
|
||||||
|
id=probe_id,
|
||||||
|
title=probe_id,
|
||||||
|
category=ProbeCategory.RUNTIME,
|
||||||
|
status=status,
|
||||||
|
summary=status.value,
|
||||||
|
description=f"{probe_id} is {status.value}",
|
||||||
|
is_blocker=blocker,
|
||||||
|
metadata=metadata or {},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# base_verdict
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_base_verdict_without_any_blocker_is_blocked():
|
||||||
|
assert base_verdict([]) == "blocked"
|
||||||
|
assert base_verdict([_probe("toolchain", blocker=False)]) == "blocked"
|
||||||
|
|
||||||
|
|
||||||
|
def test_base_verdict_failing_blocker_is_blocked():
|
||||||
|
results = [
|
||||||
|
_probe("python_runtime"),
|
||||||
|
_probe("android_adb", ProbeStatus.FAIL),
|
||||||
|
_probe("toolchain", blocker=False),
|
||||||
|
]
|
||||||
|
assert base_verdict(results) == "blocked"
|
||||||
|
|
||||||
|
|
||||||
|
def test_base_verdict_passing_blockers_with_warning_optional_is_degraded():
|
||||||
|
results = [
|
||||||
|
_probe("python_runtime"),
|
||||||
|
_probe("android_adb"),
|
||||||
|
_probe("toolchain", ProbeStatus.WARN, blocker=False),
|
||||||
|
]
|
||||||
|
assert base_verdict(results) == "degraded"
|
||||||
|
|
||||||
|
|
||||||
|
def test_base_verdict_all_pass_is_ready():
|
||||||
|
results = [_probe("python_runtime"), _probe("android_adb"), _probe("toolchain", blocker=False)]
|
||||||
|
assert base_verdict(results) == "ready"
|
||||||
|
|
||||||
|
|
||||||
|
def test_base_verdict_warning_blocker_is_blocked_not_degraded():
|
||||||
|
results = [_probe("python_runtime"), _probe("integration_host", ProbeStatus.WARN)]
|
||||||
|
assert base_verdict(results) == "blocked"
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# adb_keys_corrupted
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_adb_keys_corrupted_reads_the_adb_probe_flag():
|
||||||
|
corrupted = _probe("android_adb", metadata={"adb_keys": {"is_corrupted": True}})
|
||||||
|
healthy = _probe("android_adb", metadata={"adb_keys": {"is_corrupted": False}})
|
||||||
|
assert adb_keys_corrupted([_probe("python_runtime"), corrupted]) is True
|
||||||
|
assert adb_keys_corrupted([_probe("python_runtime"), healthy]) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_adb_keys_corrupted_is_false_without_probe_or_metadata():
|
||||||
|
assert adb_keys_corrupted([]) is False
|
||||||
|
assert adb_keys_corrupted([_probe("python_runtime")]) is False
|
||||||
|
assert adb_keys_corrupted([_probe("android_adb")]) is False
|
||||||
|
assert adb_keys_corrupted([_probe("android_adb", metadata={"adb_keys": None})]) is False
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# ordering / collection
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_sort_by_fix_order_follows_check_order_and_appends_unknown_ids():
|
||||||
|
results = [
|
||||||
|
_probe("custom_probe"),
|
||||||
|
_probe("toolchain", blocker=False),
|
||||||
|
_probe("android_adb"),
|
||||||
|
_probe("python_runtime"),
|
||||||
|
]
|
||||||
|
ordered = [r.id for r in sort_by_fix_order(results)]
|
||||||
|
assert ordered == ["python_runtime", "android_adb", "toolchain", "custom_probe"]
|
||||||
|
assert list(CHECK_ORDER) == [
|
||||||
|
"python_runtime",
|
||||||
|
"system_config",
|
||||||
|
"integration_host",
|
||||||
|
"gemini_api_key",
|
||||||
|
"android_adb",
|
||||||
|
"toolchain",
|
||||||
|
"vision_ocr_key",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_collect_readiness_gathers_engine_report_and_host_probe(monkeypatch):
|
||||||
|
report = SystemReadinessReport(
|
||||||
|
overall_ready=True,
|
||||||
|
blocker_count=1,
|
||||||
|
passed_blocker_count=1,
|
||||||
|
probes=[_probe("python_runtime")],
|
||||||
|
os_type="windows",
|
||||||
|
timestamp=time.time(),
|
||||||
|
)
|
||||||
|
host = _probe("integration_host")
|
||||||
|
run_all = AsyncMock(return_value=report)
|
||||||
|
monkeypatch.setattr(readiness.readiness_engine, "run_all", run_all)
|
||||||
|
monkeypatch.setattr(IntegrationHostProbe, "probe", AsyncMock(return_value=host))
|
||||||
|
|
||||||
|
got_report, got_host = asyncio.run(collect_readiness())
|
||||||
|
|
||||||
|
assert got_report is report
|
||||||
|
assert got_host is host
|
||||||
|
run_all.assert_awaited_once_with(force_refresh=True)
|
||||||
@@ -392,7 +392,7 @@ def _foreground_engine(tmp_path):
|
|||||||
|
|
||||||
|
|
||||||
def test_record_step_persists_explicit_foreground_app(tmp_path):
|
def test_record_step_persists_explicit_foreground_app(tmp_path):
|
||||||
"""M5: the historically dropped foreground_app parameter is persisted."""
|
"""The foreground_app parameter is persisted."""
|
||||||
engine = _foreground_engine(tmp_path)
|
engine = _foreground_engine(tmp_path)
|
||||||
engine.record_step(
|
engine.record_step(
|
||||||
foreground_app="com.android.settings",
|
foreground_app="com.android.settings",
|
||||||
|
|||||||
@@ -0,0 +1,79 @@
|
|||||||
|
# Copyright 2026 Google LLC
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
"""A detached lens call (segment capsule, visual summary) must never be
|
||||||
|
recorded as the current step's reasoning: the callback handler attaches
|
||||||
|
``thinking``/``raw_thinking`` traces by the DataEngine's global
|
||||||
|
``current_step_id``, which is independent of the trace context, so the
|
||||||
|
handler itself has to recognise the detached lens context."""
|
||||||
|
|
||||||
|
import uuid
|
||||||
|
from unittest.mock import Mock
|
||||||
|
|
||||||
|
from langchain_core.messages import AIMessage
|
||||||
|
from langchain_core.outputs import ChatGeneration, LLMResult
|
||||||
|
|
||||||
|
from artemis.data_engine.context_vars import CURRENT_NODE_NAME, CURRENT_TRACE_ID
|
||||||
|
from artemis.data_engine.trace import DataEngineCallbackHandler, detached_trace
|
||||||
|
|
||||||
|
CAPSULE = '{"doing": "scrolling", "did": "swiped", "entry_state": "home"}'
|
||||||
|
|
||||||
|
|
||||||
|
def _handler():
|
||||||
|
ctx = Mock()
|
||||||
|
ctx.data_engine = Mock()
|
||||||
|
ctx.data_engine.current_step_id = uuid.uuid4()
|
||||||
|
ctx.data_engine.record_trace = Mock()
|
||||||
|
return DataEngineCallbackHandler(ctx), ctx
|
||||||
|
|
||||||
|
|
||||||
|
def _result(text: str) -> LLMResult:
|
||||||
|
return LLMResult(generations=[[ChatGeneration(message=AIMessage(content=text))]])
|
||||||
|
|
||||||
|
|
||||||
|
def _recorded(ctx) -> dict[str, dict]:
|
||||||
|
return {c.kwargs["type"]: c.kwargs for c in ctx.data_engine.record_trace.call_args_list}
|
||||||
|
|
||||||
|
|
||||||
|
def test_agent_call_reply_is_recorded_as_raw_thinking_under_the_step():
|
||||||
|
handler, ctx = _handler()
|
||||||
|
span = uuid.uuid4()
|
||||||
|
token = CURRENT_TRACE_ID.set(span)
|
||||||
|
token_name = CURRENT_NODE_NAME.set("operator")
|
||||||
|
try:
|
||||||
|
handler.on_llm_end(_result("I see the settings list."), run_id=uuid.uuid4())
|
||||||
|
finally:
|
||||||
|
CURRENT_TRACE_ID.reset(token)
|
||||||
|
CURRENT_NODE_NAME.reset(token_name)
|
||||||
|
recorded = _recorded(ctx)
|
||||||
|
assert recorded["llm_call"]["step_id"] == ctx.data_engine.current_step_id
|
||||||
|
assert recorded["raw_thinking"]["step_id"] == ctx.data_engine.current_step_id
|
||||||
|
assert recorded["raw_thinking"]["payload"]["thought"] == "I see the settings list."
|
||||||
|
|
||||||
|
|
||||||
|
def test_detached_lens_reply_is_neither_thinking_nor_attached_to_the_step():
|
||||||
|
handler, ctx = _handler()
|
||||||
|
span = uuid.uuid4()
|
||||||
|
token = CURRENT_TRACE_ID.set(span)
|
||||||
|
token_name = CURRENT_NODE_NAME.set("operator")
|
||||||
|
try:
|
||||||
|
with detached_trace("lens:step_capsule"):
|
||||||
|
handler.on_llm_end(_result(CAPSULE), run_id=uuid.uuid4())
|
||||||
|
finally:
|
||||||
|
CURRENT_TRACE_ID.reset(token)
|
||||||
|
CURRENT_NODE_NAME.reset(token_name)
|
||||||
|
recorded = _recorded(ctx)
|
||||||
|
assert "raw_thinking" not in recorded
|
||||||
|
assert "thinking" not in recorded
|
||||||
|
assert recorded["llm_call"]["step_id"] is None
|
||||||
@@ -208,6 +208,219 @@ async def test_process_plan_write_only_queues(tmp_path):
|
|||||||
assert ctx.checkpoint_tasks == {}
|
assert ctx.checkpoint_tasks == {}
|
||||||
|
|
||||||
|
|
||||||
|
VERIFY_LINE = " - verify: the alarm list shows 7:30 AM\n"
|
||||||
|
ASSERT_LINE = " - assert: a toast appeared\n"
|
||||||
|
|
||||||
|
|
||||||
|
def _guidance_arrives(ctx, plan_text):
|
||||||
|
"""What perception_node does when a non-empty user instruction lands: the
|
||||||
|
check lines that exist at that moment lose their machine restoration."""
|
||||||
|
from artemis.utils.plan_grammar import parse_plan
|
||||||
|
|
||||||
|
current = getattr(ctx, "guidance_unprotected_checks", None)
|
||||||
|
if not isinstance(current, set):
|
||||||
|
current = set()
|
||||||
|
ctx.guidance_unprotected_checks = current | {
|
||||||
|
(ci.kind, ci.text) for ci in parse_plan(plan_text).check_items
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
async def _plan_write(tmp_path, ctx, before, after):
|
||||||
|
"""One accepted task_plan write through the shared post-write pipeline;
|
||||||
|
returns the plan text as it stands on disk afterwards."""
|
||||||
|
from artemis.graph.graph import _process_plan_write
|
||||||
|
|
||||||
|
task_plan_path = _write_plan(tmp_path, after)
|
||||||
|
await _process_plan_write(ctx, _make_state(), task_plan_path, before, after, "ok", True)
|
||||||
|
return task_plan_path.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
NEW_VERIFY_LINE = " - verify: the build number is recorded\n"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_pre_guidance_check_lines_may_be_dropped_at_any_later_write(tmp_path):
|
||||||
|
"""Without guidance a deleted check line grows back. Once user guidance has
|
||||||
|
arrived, the lines that existed at that moment stay editable for the rest
|
||||||
|
of the run: the Operator re-plans over several writes, turns later, and
|
||||||
|
none of them is undone."""
|
||||||
|
without_verify = PLAN_WITH_CHECKS.replace(VERIFY_LINE, "")
|
||||||
|
without_both = without_verify.replace(ASSERT_LINE, "")
|
||||||
|
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
|
||||||
|
ctx.guidance_unprotected_checks = set()
|
||||||
|
|
||||||
|
# Ordinary write (no guidance): the deleted line grows back.
|
||||||
|
assert VERIFY_LINE in await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, without_verify)
|
||||||
|
|
||||||
|
# Turn N: guidance arrives; the Operator only advances a milestone.
|
||||||
|
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
|
||||||
|
started_next = PLAN_WITH_CHECKS.replace("- [ ] Next milestone", "- [/] Next milestone")
|
||||||
|
assert await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, started_next) == started_next
|
||||||
|
|
||||||
|
# Turn N+2: the first check line goes, and stands.
|
||||||
|
dropped_verify = started_next.replace(VERIFY_LINE, "")
|
||||||
|
assert await _plan_write(tmp_path, ctx, started_next, dropped_verify) == dropped_verify
|
||||||
|
# Turn N+3: the second one goes too, and also stands (no single-write window).
|
||||||
|
dropped_both = dropped_verify.replace(ASSERT_LINE, "")
|
||||||
|
assert await _plan_write(tmp_path, ctx, dropped_verify, dropped_both) == dropped_both
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_check_lines_added_after_guidance_stay_protected(tmp_path):
|
||||||
|
"""A check line the Operator adds after the instruction is a new standard:
|
||||||
|
deleting it later is restored, while the pre-guidance line still may go."""
|
||||||
|
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
|
||||||
|
ctx.guidance_unprotected_checks = set()
|
||||||
|
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
|
||||||
|
|
||||||
|
with_new = PLAN_WITH_CHECKS.replace(ASSERT_LINE, ASSERT_LINE + NEW_VERIFY_LINE)
|
||||||
|
assert await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, with_new) == with_new
|
||||||
|
|
||||||
|
dropped_new = with_new.replace(NEW_VERIFY_LINE, "")
|
||||||
|
on_disk = await _plan_write(tmp_path, ctx, with_new, dropped_new)
|
||||||
|
assert NEW_VERIFY_LINE in on_disk # restored: it postdates the guidance
|
||||||
|
|
||||||
|
dropped_old = with_new.replace(VERIFY_LINE, "")
|
||||||
|
on_disk = await _plan_write(tmp_path, ctx, with_new, dropped_old)
|
||||||
|
assert VERIFY_LINE not in on_disk # stands: it predates the guidance
|
||||||
|
assert NEW_VERIFY_LINE in on_disk
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_second_guidance_waives_the_lines_that_exist_by_then(tmp_path):
|
||||||
|
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
|
||||||
|
ctx.guidance_unprotected_checks = set()
|
||||||
|
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
|
||||||
|
with_new = PLAN_WITH_CHECKS.replace(ASSERT_LINE, ASSERT_LINE + NEW_VERIFY_LINE)
|
||||||
|
assert await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, with_new) == with_new
|
||||||
|
dropped_new = with_new.replace(NEW_VERIFY_LINE, "")
|
||||||
|
assert NEW_VERIFY_LINE in await _plan_write(tmp_path, ctx, with_new, dropped_new)
|
||||||
|
|
||||||
|
_guidance_arrives(ctx, with_new) # the new line exists now: it is waived too
|
||||||
|
assert NEW_VERIFY_LINE not in await _plan_write(tmp_path, ctx, with_new, dropped_new)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_mixed_write_restores_only_the_protected_lines(tmp_path):
|
||||||
|
"""One write drops a pre-guidance line and a post-guidance line: only the
|
||||||
|
protected one is merged back."""
|
||||||
|
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
|
||||||
|
ctx.guidance_unprotected_checks = set()
|
||||||
|
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
|
||||||
|
with_new = PLAN_WITH_CHECKS.replace(ASSERT_LINE, ASSERT_LINE + NEW_VERIFY_LINE)
|
||||||
|
assert await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, with_new) == with_new
|
||||||
|
|
||||||
|
dropped_both = with_new.replace(VERIFY_LINE, "").replace(NEW_VERIFY_LINE, "")
|
||||||
|
on_disk = await _plan_write(tmp_path, ctx, with_new, dropped_both)
|
||||||
|
assert VERIFY_LINE not in on_disk
|
||||||
|
assert NEW_VERIFY_LINE in on_disk
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_guided_drop_records_the_retired_lines_and_the_outcome_excludes_them(tmp_path):
|
||||||
|
"""A line dropped under guidance is remembered as retired; a verdict the
|
||||||
|
ledger recorded for it earlier is reported as retired, not as a failure,
|
||||||
|
while a line still declared keeps its verdict."""
|
||||||
|
from artemis.graph.checkpoints import compute_test_summary
|
||||||
|
from artemis.utils.plan_grammar import parse_plan
|
||||||
|
|
||||||
|
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
|
||||||
|
ctx.guidance_unprotected_checks = set()
|
||||||
|
ctx.guidance_retired_checks = set()
|
||||||
|
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
|
||||||
|
without_verify = PLAN_WITH_CHECKS.replace(VERIFY_LINE, "")
|
||||||
|
assert await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, without_verify) == without_verify
|
||||||
|
assert ctx.guidance_retired_checks == {("verify", "the alarm list shows 7:30 AM")}
|
||||||
|
|
||||||
|
records = [
|
||||||
|
{"kind": "verify", "item_text": "the alarm list shows 7:30 AM", "status": "failed"},
|
||||||
|
{"kind": "assert", "item_text": "a toast appeared", "status": "passed"},
|
||||||
|
]
|
||||||
|
summary = compute_test_summary(
|
||||||
|
list(parse_plan(without_verify).all_check_items),
|
||||||
|
records,
|
||||||
|
retired=ctx.guidance_retired_checks,
|
||||||
|
)
|
||||||
|
assert (summary.passed, summary.failed, summary.retired) == (1, 0, 1)
|
||||||
|
assert summary.retired_items == [
|
||||||
|
{"item_text": "the alarm list shows 7:30 AM", "kind": "verify", "last_status": "failed"}
|
||||||
|
]
|
||||||
|
# A retired line that is declared again (the Operator re-added it) counts normally.
|
||||||
|
summary = compute_test_summary(
|
||||||
|
list(parse_plan(PLAN_WITH_CHECKS).all_check_items),
|
||||||
|
records,
|
||||||
|
retired=ctx.guidance_retired_checks,
|
||||||
|
)
|
||||||
|
assert (summary.passed, summary.failed, summary.retired) == (1, 1, 0)
|
||||||
|
|
||||||
|
|
||||||
|
REWORDED_MILESTONE = PLAN_WITH_CHECKS.replace(f"- [x] {GOAL_TEXT}", "- [x] Create the 7:30 alarm")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_rewording_a_milestone_keeps_its_check_lines_in_place(tmp_path):
|
||||||
|
"""The check lines under a reworded milestone change signature (parent
|
||||||
|
hash) but stay declared: they are neither duplicated as task-level @end
|
||||||
|
orphans nor, under guidance, retired."""
|
||||||
|
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
|
||||||
|
ctx.guidance_unprotected_checks = set()
|
||||||
|
ctx.guidance_retired_checks = set()
|
||||||
|
|
||||||
|
# No guidance: nothing to restore, the plan stands as written.
|
||||||
|
on_disk = await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, REWORDED_MILESTONE)
|
||||||
|
assert on_disk == REWORDED_MILESTONE
|
||||||
|
assert "@end" not in on_disk
|
||||||
|
|
||||||
|
# Guidance armed: the same edit retires nothing.
|
||||||
|
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
|
||||||
|
on_disk = await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, REWORDED_MILESTONE)
|
||||||
|
assert on_disk == REWORDED_MILESTONE
|
||||||
|
assert ctx.guidance_retired_checks == set()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_redeclaring_a_retired_line_reinstates_it(tmp_path):
|
||||||
|
"""Dropped under guidance, then added back: the harvest gate must see the
|
||||||
|
line as live again, matching what compute_test_summary reports."""
|
||||||
|
from artemis.graph.checkpoints import reinstate_check_items
|
||||||
|
|
||||||
|
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
|
||||||
|
ctx.guidance_unprotected_checks = set()
|
||||||
|
ctx.guidance_retired_checks = set()
|
||||||
|
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
|
||||||
|
without_verify = PLAN_WITH_CHECKS.replace(VERIFY_LINE, "")
|
||||||
|
await _plan_write(tmp_path, ctx, PLAN_WITH_CHECKS, without_verify)
|
||||||
|
assert ctx.guidance_retired_checks == {RETIRED_VERIFY}
|
||||||
|
|
||||||
|
await _plan_write(tmp_path, ctx, without_verify, PLAN_WITH_CHECKS)
|
||||||
|
assert ctx.guidance_retired_checks == set()
|
||||||
|
|
||||||
|
# Direct helper: unrelated items leave the set untouched.
|
||||||
|
ctx.guidance_retired_checks = {RETIRED_VERIFY}
|
||||||
|
reinstate_check_items(ctx, [RETIRED_ASSERT])
|
||||||
|
assert ctx.guidance_retired_checks == {RETIRED_VERIFY}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_rejected_plan_write_leaves_the_waiver_untouched(tmp_path):
|
||||||
|
from langchain_core.messages import ToolMessage
|
||||||
|
|
||||||
|
from artemis.graph.graph import _process_plan_write
|
||||||
|
|
||||||
|
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
|
||||||
|
ctx.guidance_unprotected_checks = set()
|
||||||
|
_guidance_arrives(ctx, PLAN_WITH_CHECKS)
|
||||||
|
waived = set(ctx.guidance_unprotected_checks)
|
||||||
|
reworded = PLAN_WITH_CHECKS.replace("- [ ] Next milestone", "- [ ] The next milestone")
|
||||||
|
task_plan_path = _write_plan(tmp_path, reworded)
|
||||||
|
result = await _process_plan_write(
|
||||||
|
ctx, _make_state(), task_plan_path, PLAN_WITH_CHECKS, reworded, "ok", False
|
||||||
|
)
|
||||||
|
assert isinstance(result, ToolMessage)
|
||||||
|
assert task_plan_path.read_text(encoding="utf-8") == PLAN_WITH_CHECKS
|
||||||
|
assert ctx.guidance_unprotected_checks == waived
|
||||||
|
|
||||||
|
|
||||||
# --- §8.1 / §8.10: spawn after record_step; record_step unconditional ----------------
|
# --- §8.1 / §8.10: spawn after record_step; record_step unconditional ----------------
|
||||||
|
|
||||||
|
|
||||||
@@ -515,6 +728,146 @@ async def test_assert_fail_halt_policy_latches_halt(tmp_path):
|
|||||||
assert ctx.assert_halt is True
|
assert ctx.assert_halt is True
|
||||||
|
|
||||||
|
|
||||||
|
# --- Retired check items: late verdicts are ledger-only ------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
RETIRED_VERIFY = ("verify", "the alarm list shows 7:30 AM")
|
||||||
|
RETIRED_ASSERT = ("assert", "a toast appeared")
|
||||||
|
|
||||||
|
|
||||||
|
def _last_done_event(ctx):
|
||||||
|
calls = [
|
||||||
|
c for c in ctx.data_engine._publish.call_args_list if c.args[1].get("status") == "done"
|
||||||
|
]
|
||||||
|
return calls[-1].args[1]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_late_failing_verify_for_retired_line_is_ledger_only(tmp_path):
|
||||||
|
"""The user retired the verify line but kept the milestone: a late failing
|
||||||
|
verdict for it neither reopens the subgoal nor produces findings/headline,
|
||||||
|
yet the ledger still books it and the done event reports it as retired."""
|
||||||
|
plan_path = _write_plan(tmp_path, PLAN_WITH_CHECKS.replace(VERIFY_LINE, ""))
|
||||||
|
ctx = _make_ctx(tmp_path)
|
||||||
|
ctx.guidance_retired_checks = {RETIRED_VERIFY}
|
||||||
|
run = _done_run(ctx, CheckReport(verdicts=[_verdict("verify", "failed")]))
|
||||||
|
|
||||||
|
findings = harvest_run(ctx, _make_state(), run, allow_side_effects=True)
|
||||||
|
|
||||||
|
assert findings == []
|
||||||
|
assert f"- [x] {GOAL_TEXT}" in plan_path.read_text(encoding="utf-8")
|
||||||
|
assert "- finding:" not in plan_path.read_text(encoding="utf-8")
|
||||||
|
assert ctx.checkpoint_repairs == {}
|
||||||
|
assert getattr(ctx, "checker_findings", {}) == {}
|
||||||
|
records = read_ledger(tmp_path)
|
||||||
|
assert [(r["kind"], r["status"]) for r in records] == [("verify", "failed")]
|
||||||
|
event = _last_done_event(ctx)
|
||||||
|
assert event["applicable"] is False
|
||||||
|
assert event["reverted"] is False
|
||||||
|
assert event["retired"] == [{"kind": "verify", "item_text": RETIRED_VERIFY[1]}]
|
||||||
|
assert [v["status"] for v in event["verdicts"]] == ["failed"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_failed_assert_for_retired_line_never_latches_halt(tmp_path):
|
||||||
|
_write_plan(tmp_path, PLAN_WITH_CHECKS.replace(ASSERT_LINE, ""))
|
||||||
|
ctx = _make_ctx(tmp_path, assert_failure_policy="halt")
|
||||||
|
ctx.guidance_retired_checks = {RETIRED_ASSERT}
|
||||||
|
run = _done_run(ctx, CheckReport(verdicts=[_verdict("assert", "failed")]))
|
||||||
|
|
||||||
|
harvest_run(ctx, _make_state(), run, allow_side_effects=True)
|
||||||
|
|
||||||
|
assert ctx.assert_halt is False
|
||||||
|
assert read_ledger(tmp_path)[0]["status"] == "failed" # still booked
|
||||||
|
assert _last_done_event(ctx)["applicable"] is False
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_non_retired_sibling_verdict_still_takes_effect(tmp_path):
|
||||||
|
"""Same report: the retired verify is dropped, the live assert still halts
|
||||||
|
and the live verify sibling still repairs."""
|
||||||
|
plan_path = _write_plan(tmp_path, PLAN_WITH_CHECKS.replace(VERIFY_LINE, ""))
|
||||||
|
ctx = _make_ctx(tmp_path, assert_failure_policy="halt")
|
||||||
|
ctx.guidance_retired_checks = {RETIRED_VERIFY}
|
||||||
|
run = _done_run(
|
||||||
|
ctx,
|
||||||
|
CheckReport(verdicts=[_verdict("verify", "failed"), _verdict("assert", "failed")]),
|
||||||
|
)
|
||||||
|
|
||||||
|
findings = harvest_run(ctx, _make_state(), run, allow_side_effects=True)
|
||||||
|
|
||||||
|
assert findings == [] # the retired verify produces no repair finding
|
||||||
|
assert f"- [x] {GOAL_TEXT}" in plan_path.read_text(encoding="utf-8")
|
||||||
|
assert ctx.assert_halt is True # the live assert sibling still counts
|
||||||
|
event = _last_done_event(ctx)
|
||||||
|
assert event["applicable"] is True
|
||||||
|
assert event["retired"] == [{"kind": "verify", "item_text": RETIRED_VERIFY[1]}]
|
||||||
|
|
||||||
|
# And a live verify sibling next to a retired one still drives the repair,
|
||||||
|
# with the headline naming only the live criterion.
|
||||||
|
live_line = " - verify: the alarm is enabled\n"
|
||||||
|
plan_path = _write_plan(tmp_path, PLAN_WITH_CHECKS.replace(VERIFY_LINE, live_line))
|
||||||
|
ctx = _make_ctx(tmp_path)
|
||||||
|
ctx.guidance_retired_checks = {RETIRED_VERIFY}
|
||||||
|
run = _done_run(
|
||||||
|
ctx,
|
||||||
|
CheckReport(
|
||||||
|
verdicts=[
|
||||||
|
_verdict("verify", "failed"),
|
||||||
|
_verdict("verify", "failed", text="the alarm is enabled", evidence="toggle off"),
|
||||||
|
]
|
||||||
|
),
|
||||||
|
)
|
||||||
|
findings = harvest_run(ctx, _make_state(), run, allow_side_effects=True)
|
||||||
|
content = plan_path.read_text(encoding="utf-8")
|
||||||
|
assert f"- [/] {GOAL_TEXT}" in content
|
||||||
|
assert "- finding: verify failed — 'the alarm is enabled'" in content
|
||||||
|
assert RETIRED_VERIFY[1] not in " ".join(findings)
|
||||||
|
assert any("the alarm is enabled" in f for f in findings)
|
||||||
|
assert ctx.checker_finding_items[GOAL_KEY] == {("verify", "the alarm is enabled")}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_retiring_an_item_withdraws_its_registered_headline(tmp_path):
|
||||||
|
"""A headline registered by an earlier failing attempt disappears from the
|
||||||
|
plan when the user guidance retires the item that produced it; a headline
|
||||||
|
that still names a live item stays."""
|
||||||
|
from artemis.graph.checkpoints import retire_check_items
|
||||||
|
|
||||||
|
plan_path = _write_plan(tmp_path)
|
||||||
|
ctx = _make_ctx(tmp_path)
|
||||||
|
run = _done_run(ctx, CheckReport(verdicts=[_verdict("verify", "failed")]))
|
||||||
|
harvest_run(ctx, _make_state(), run, allow_side_effects=True)
|
||||||
|
assert "- finding: verify failed" in plan_path.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
retire_check_items(ctx, [RETIRED_ASSERT]) # unrelated item: headline stays
|
||||||
|
assert "- finding: verify failed" in plan_path.read_text(encoding="utf-8")
|
||||||
|
assert ctx.guidance_retired_checks == {RETIRED_ASSERT}
|
||||||
|
|
||||||
|
retire_check_items(ctx, [RETIRED_VERIFY])
|
||||||
|
assert "- finding:" not in plan_path.read_text(encoding="utf-8")
|
||||||
|
assert ctx.checker_findings == {}
|
||||||
|
assert ctx.checker_finding_items == {}
|
||||||
|
assert ctx.guidance_retired_checks == {RETIRED_ASSERT, RETIRED_VERIFY}
|
||||||
|
|
||||||
|
# The plan-write path reaches the same helper: dropping a waived line
|
||||||
|
# after a headline was registered removes the headline on that write.
|
||||||
|
plan_path = _write_plan(tmp_path)
|
||||||
|
ctx = _make_ctx(tmp_path, disable_planner_validation=True)
|
||||||
|
ctx.guidance_unprotected_checks = set()
|
||||||
|
ctx.guidance_retired_checks = set()
|
||||||
|
run = _done_run(ctx, CheckReport(verdicts=[_verdict("verify", "failed")]))
|
||||||
|
harvest_run(ctx, _make_state(), run, allow_side_effects=True)
|
||||||
|
with_finding = plan_path.read_text(encoding="utf-8")
|
||||||
|
assert "- finding: verify failed" in with_finding
|
||||||
|
_guidance_arrives(ctx, with_finding)
|
||||||
|
without_verify = with_finding.replace(VERIFY_LINE, "")
|
||||||
|
on_disk = await _plan_write(tmp_path, ctx, with_finding, without_verify)
|
||||||
|
assert VERIFY_LINE not in on_disk
|
||||||
|
assert "- finding:" not in on_disk
|
||||||
|
assert ctx.guidance_retired_checks == {RETIRED_VERIFY}
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_ledger_is_append_only_fail_then_pass_both_present(tmp_path):
|
async def test_ledger_is_append_only_fail_then_pass_both_present(tmp_path):
|
||||||
_write_plan(tmp_path)
|
_write_plan(tmp_path)
|
||||||
|
|||||||
@@ -0,0 +1,95 @@
|
|||||||
|
# Copyright 2026 Google LLC
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
"""Perception side of the guidance waiver: a non-empty user instruction records
|
||||||
|
the plan's current check lines in ``ctx.guidance_unprotected_checks`` on the run
|
||||||
|
context (which outlives the per-turn State rebuild); stop-only or empty
|
||||||
|
payloads do not, and an ordinary turn never touches the set."""
|
||||||
|
|
||||||
|
from unittest.mock import AsyncMock, MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from artemis.graph.perception import perception_node
|
||||||
|
from artemis.graph.state import State
|
||||||
|
|
||||||
|
|
||||||
|
async def _run_perception(ctx, payload):
|
||||||
|
with (
|
||||||
|
patch("artemis.graph.perception.is_ocr_configured", return_value=False),
|
||||||
|
patch("artemis.graph.perception.UnifiedMobileController") as controller_cls,
|
||||||
|
patch("artemis.graph.perception._check_injected_instruction_file", return_value=payload),
|
||||||
|
):
|
||||||
|
device_data = MagicMock()
|
||||||
|
device_data.width = 1080
|
||||||
|
device_data.height = 2400
|
||||||
|
device_data.base64 = "ZHVtbXk="
|
||||||
|
device_data.elements = []
|
||||||
|
controller = MagicMock()
|
||||||
|
controller.get_screen_data = AsyncMock(return_value=device_data)
|
||||||
|
controller_cls.return_value = controller
|
||||||
|
|
||||||
|
state = MagicMock(spec=State)
|
||||||
|
state.structured_decisions = []
|
||||||
|
return await perception_node(state, ctx)
|
||||||
|
|
||||||
|
|
||||||
|
PLAN = "- [/] Set the alarm\n - verify: the alarm list shows 7:30 AM\n- [ ] Next milestone\n"
|
||||||
|
|
||||||
|
|
||||||
|
def _ctx(tmp_path):
|
||||||
|
from artemis.utils.notes import get_note_file_path
|
||||||
|
|
||||||
|
ctx = MagicMock()
|
||||||
|
ctx.data_engine = MagicMock()
|
||||||
|
ctx.data_engine.base_dir = tmp_path
|
||||||
|
ctx.data_engine.current_step_id = "step_1"
|
||||||
|
ctx.device = MagicMock()
|
||||||
|
ctx.guidance_unprotected_checks = set()
|
||||||
|
plan_path = get_note_file_path(tmp_path, "task_plan")
|
||||||
|
plan_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
plan_path.write_text(PLAN, encoding="utf-8")
|
||||||
|
return ctx
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_instruction_waives_the_current_check_lines_and_still_reaches_state(tmp_path):
|
||||||
|
ctx = _ctx(tmp_path)
|
||||||
|
update = await _run_perception(ctx, {"instruction": "Skip the toast check"})
|
||||||
|
assert ctx.guidance_unprotected_checks == {("verify", "the alarm list shows 7:30 AM")}
|
||||||
|
assert update["injected_instruction"] == "Skip the toast check"
|
||||||
|
assert update["user_stop_requested"] is False
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_stop_only_or_blank_payload_does_not_arm(tmp_path):
|
||||||
|
ctx = _ctx(tmp_path)
|
||||||
|
update = await _run_perception(ctx, {"release_loop": True})
|
||||||
|
assert update["user_stop_requested"] is True
|
||||||
|
assert ctx.guidance_unprotected_checks == set()
|
||||||
|
|
||||||
|
update = await _run_perception(ctx, {"instruction": " ", "release_loop": True})
|
||||||
|
assert update["user_stop_requested"] is True
|
||||||
|
assert ctx.guidance_unprotected_checks == set()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_ordinary_turn_leaves_the_waiver_as_it_was(tmp_path):
|
||||||
|
"""The per-turn reset applies to State (``injected_instruction`` goes back
|
||||||
|
to None); the waiver on the context lives for the rest of the run."""
|
||||||
|
ctx = _ctx(tmp_path)
|
||||||
|
ctx.guidance_unprotected_checks = {("verify", "earlier line")}
|
||||||
|
update = await _run_perception(ctx, None)
|
||||||
|
assert update["injected_instruction"] is None
|
||||||
|
assert ctx.guidance_unprotected_checks == {("verify", "earlier line")}
|
||||||
@@ -77,6 +77,11 @@ async def test_ask_explorer_runs_the_pipeline_for_the_executor_agent():
|
|||||||
assert result.status == "success"
|
assert result.status == "success"
|
||||||
assert "Explorer located 1 candidate(s) for 'blue Send button'" in result.text_summary
|
assert "Explorer located 1 candidate(s) for 'blue Send button'" in result.text_summary
|
||||||
assert "[1] 'Send' at normalized [500, 500]" in result.text_summary
|
assert "[1] 'Send' at normalized [500, 500]" in result.text_summary
|
||||||
|
# This executor's click resolves an index against the element list the
|
||||||
|
# candidates just joined, so the guidance teaches the index syntax (Pro
|
||||||
|
# parity) next to the coordinates.
|
||||||
|
assert "target=3, a bare integer" in result.text_summary
|
||||||
|
assert "normalized coordinate" in result.text_summary
|
||||||
# Candidates were registered on the state for index-based actions.
|
# Candidates were registered on the state for index-based actions.
|
||||||
assert state.indexed_points == [[540, 1200]]
|
assert state.indexed_points == [[540, 1200]]
|
||||||
assert state.indexed_elements[0]["index"] == 1
|
assert state.indexed_elements[0]["index"] == 1
|
||||||
|
|||||||
@@ -12,12 +12,14 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
"""Coordinate targets must be described by the model, never inferred.
|
"""Targets are either observed (an element index) or described (coordinates).
|
||||||
|
|
||||||
The Flash/Validator dialect addresses every target by coordinates, so the
|
Flash binds the Pro Operator's agent dialect: a click, long press or focused
|
||||||
executor refuses a click, long press, focused input, coordinate swipe or click
|
input names an element index from the indexed UI list -- resolved here to the
|
||||||
sequence that does not say what it is aiming at. The description is recorded
|
element's center with its observed text/bounds/id recorded -- or a coordinate
|
||||||
by the runner and never reaches the wire.
|
pair, which the model must describe itself. A coordinate swipe or click
|
||||||
|
sequence must be described too. Nothing is inferred, the description never
|
||||||
|
reaches the wire, and the wire only ever sees coordinates.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from unittest.mock import Mock
|
from unittest.mock import Mock
|
||||||
@@ -44,6 +46,123 @@ def _state():
|
|||||||
return state
|
return state
|
||||||
|
|
||||||
|
|
||||||
|
def _indexed_state():
|
||||||
|
"""Two observed elements (pixel centers, the shape ``observe()`` writes)."""
|
||||||
|
state = _state()
|
||||||
|
state.indexed_elements = [
|
||||||
|
{
|
||||||
|
"index": 1,
|
||||||
|
"center": [540, 1248],
|
||||||
|
"text": "Wi-Fi",
|
||||||
|
"bounds": [40, 1152, 1040, 1344],
|
||||||
|
"class": "android.widget.TextView",
|
||||||
|
"resource_id": "android:id/title",
|
||||||
|
"is_ocr": False,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"index": 2,
|
||||||
|
"center": [540, 720],
|
||||||
|
"text": "Search settings",
|
||||||
|
"bounds": [80, 660, 1000, 780],
|
||||||
|
"class": "android.widget.EditText",
|
||||||
|
"resource_id": "com.android.settings:id/search",
|
||||||
|
"is_ocr": False,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
state.indexed_points = [el["center"] for el in state.indexed_elements]
|
||||||
|
return state
|
||||||
|
|
||||||
|
|
||||||
|
# --- Index targets: resolved against the element list, observed semantics recorded ---
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("index", [1, "1", 1.0, [1]])
|
||||||
|
def test_click_index_resolves_to_the_elements_center_and_records_observed_fields(index):
|
||||||
|
executor = _make_executor()
|
||||||
|
wire_name, wire_args, _, recorded = executor._translate(
|
||||||
|
"click", {"target": index}, _indexed_state()
|
||||||
|
)
|
||||||
|
assert wire_name == "click"
|
||||||
|
# 540/1080 -> 500, 1248/2400 -> 520 (normalized 0-1000)
|
||||||
|
assert wire_args == {"target": [500, 520], "times": 1, "delay_ms": 100}
|
||||||
|
assert recorded == {
|
||||||
|
"target_text": "Wi-Fi",
|
||||||
|
"target_bounds": [37, 480, 963, 560],
|
||||||
|
"target_resource_id": "android:id/title",
|
||||||
|
"target_class": "android.widget.TextView",
|
||||||
|
}
|
||||||
|
assert "target_description" not in recorded
|
||||||
|
|
||||||
|
|
||||||
|
def test_index_target_needs_no_description_and_ignores_a_stray_one():
|
||||||
|
"""An index already names its element; the model's belief never mixes with
|
||||||
|
the observation on the record."""
|
||||||
|
executor = _make_executor()
|
||||||
|
_, wire_args, _, recorded = executor._translate(
|
||||||
|
"click", {"target": 2, "target_description": "the search box"}, _indexed_state()
|
||||||
|
)
|
||||||
|
assert wire_args["target"] == [500, 300]
|
||||||
|
assert recorded["target_text"] == "Search settings"
|
||||||
|
assert "target_description" not in recorded
|
||||||
|
|
||||||
|
|
||||||
|
def test_long_press_and_input_text_resolve_indices_too():
|
||||||
|
executor = _make_executor()
|
||||||
|
_, press_args, _, recorded = executor._translate(
|
||||||
|
"long_press", {"target": 1, "duration": 1500}, _indexed_state()
|
||||||
|
)
|
||||||
|
assert press_args == {"target": [500, 520], "duration_ms": 1500}
|
||||||
|
assert recorded["target_text"] == "Wi-Fi"
|
||||||
|
|
||||||
|
_, type_args, _, recorded = executor._translate(
|
||||||
|
"input_text", {"text": "wifi", "target": 2}, _indexed_state()
|
||||||
|
)
|
||||||
|
assert type_args == {"text": "wifi", "target": [500, 300], "clear_exist": True}
|
||||||
|
assert recorded["target_resource_id"] == "com.android.settings:id/search"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("name, label", [("click", "click"), ("long_press", "long press")])
|
||||||
|
def test_out_of_range_index_is_refused_with_the_active_range(name, label):
|
||||||
|
executor = _make_executor()
|
||||||
|
with pytest.raises(_ArgError) as excinfo:
|
||||||
|
executor._translate(name, {"target": 7}, _indexed_state())
|
||||||
|
message = str(excinfo.value)
|
||||||
|
assert message.startswith(f"Error during {label}: Invalid target index 7.")
|
||||||
|
assert "1 to 2" in message
|
||||||
|
assert "ask_explorer" in message
|
||||||
|
|
||||||
|
|
||||||
|
def test_index_against_an_empty_list_is_refused():
|
||||||
|
executor = _make_executor()
|
||||||
|
with pytest.raises(_ArgError) as excinfo:
|
||||||
|
executor._translate("click", {"target": 1}, _state())
|
||||||
|
message = str(excinfo.value)
|
||||||
|
assert "Invalid target index 1" in message
|
||||||
|
assert "empty" in message
|
||||||
|
|
||||||
|
|
||||||
|
def test_unusable_target_is_refused_with_both_syntaxes_named():
|
||||||
|
executor = _make_executor()
|
||||||
|
with pytest.raises(_ArgError) as excinfo:
|
||||||
|
executor._translate("click", {"target": "the button"}, _indexed_state())
|
||||||
|
message = str(excinfo.value)
|
||||||
|
assert message.startswith("Error during click: Invalid target format")
|
||||||
|
assert "element index" in message and "[x, y]" in message
|
||||||
|
|
||||||
|
|
||||||
|
def test_index_resolution_uses_the_live_device_resolution():
|
||||||
|
"""The element list is built against the device size ``observe()`` wrote
|
||||||
|
on the context; a different resolution maps the same pixels differently."""
|
||||||
|
executor = _make_executor(width=720, height=1600)
|
||||||
|
state = _indexed_state()
|
||||||
|
state.indexed_elements[0]["center"] = [360, 800]
|
||||||
|
_, wire_args, _, _ = executor._translate("click", {"target": 1}, state)
|
||||||
|
assert wire_args["target"] == [500, 500]
|
||||||
|
|
||||||
|
|
||||||
|
# --- Coordinate targets: the model describes what it aims at -------------------------
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
@pytest.mark.parametrize(
|
||||||
"name, args, label",
|
"name, args, label",
|
||||||
[
|
[
|
||||||
|
|||||||
@@ -17,12 +17,11 @@
|
|||||||
Two kinds of protection:
|
Two kinds of protection:
|
||||||
|
|
||||||
* **Fixture pins** freeze the exact schema each model surface receives (operator
|
* **Fixture pins** freeze the exact schema each model surface receives (operator
|
||||||
shells, Validator/Flash declarations, the action server manifest). A schema change
|
shells, Flash declarations, the action server manifest). A schema change must be a
|
||||||
must be a conscious act: regenerate the fixture in the same commit and say why.
|
conscious act: regenerate the fixture in the same commit and say why.
|
||||||
* **Bridge derivation** re-derives every cross-dialect parameter difference from the
|
* **Projection identity** checks that the Flash declaration of every shared action
|
||||||
generated schemas and requires it to be declared in ``ActionSpec.param_bridge``.
|
is the operator shell's own description and parameters, so the two profiles can
|
||||||
Undeclared divergence between what one agent sees and what another sees -- the
|
never see different tools under one name -- the historical definition of drift.
|
||||||
historical definition of drift -- fails here instead of shipping.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
@@ -130,47 +129,71 @@ def test_validator_declaration_order_is_stable():
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# --- Bridge derivation: dialect differences must be declared -------------------------
|
# --- Projection identity: Flash binds the operator shell, projected ------------------
|
||||||
|
|
||||||
|
|
||||||
def _operator_param_names(name: str) -> list[str]:
|
def _without_optional_noise(schema: dict) -> dict:
|
||||||
schema = convert_to_openai_tool(operator_shell_tool(name))
|
"""The pydantic-generated property schema minus what only marks optionality.
|
||||||
return list(schema["function"]["parameters"].get("properties", {}))
|
|
||||||
|
``default`` and the ``null`` alternative say "optional"; the declaration says
|
||||||
|
that through ``required`` alone.
|
||||||
|
"""
|
||||||
|
out = {k: v for k, v in schema.items() if k not in ("anyOf", "default")}
|
||||||
|
if "anyOf" in schema:
|
||||||
|
members = [m for m in schema["anyOf"] if m.get("type") != "null"]
|
||||||
|
out.update(members[0] if len(members) == 1 else {"anyOf": members})
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
def _declaration_param_names(name: str) -> list[str]:
|
def test_flash_declarations_are_the_operator_shells_projected():
|
||||||
return list(tool_declaration(name).parameters.get("properties", {}))
|
"""One definition, two bindings: for every action with an operator shell, the
|
||||||
|
Flash ``ToolDeclaration`` carries the shell's description, parameter names,
|
||||||
|
descriptions, types (index-or-coordinates unions included) and required set."""
|
||||||
def test_param_bridge_is_exact():
|
|
||||||
"""Every operator<->declaration parameter difference is declared, none invented."""
|
|
||||||
for spec in ACTION_SPECS.values():
|
for spec in ACTION_SPECS.values():
|
||||||
if spec.operator is None or spec.declaration is None:
|
if spec.operator is None:
|
||||||
assert spec.param_bridge == {}, (
|
|
||||||
f"'{spec.name}' declares a param_bridge without both dialects."
|
|
||||||
)
|
|
||||||
continue
|
continue
|
||||||
operator_names = _operator_param_names(spec.name)
|
shell = convert_to_openai_tool(operator_shell_tool(spec.name))["function"]
|
||||||
declaration_names = _declaration_param_names(spec.name)
|
declaration = tool_declaration(spec.name)
|
||||||
assert set(spec.param_bridge) == set(operator_names), (
|
assert declaration.description == shell["description"], spec.name
|
||||||
f"'{spec.name}': param_bridge keys must cover exactly the operator"
|
shell_props = shell["parameters"]["properties"]
|
||||||
f" params. bridge={sorted(spec.param_bridge)} operator={operator_names}"
|
decl_props = declaration.parameters["properties"]
|
||||||
|
assert list(decl_props) == list(shell_props), spec.name
|
||||||
|
for param, schema in shell_props.items():
|
||||||
|
assert decl_props[param] == _without_optional_noise(schema), (
|
||||||
|
f"'{spec.name}.{param}' diverges between the operator shell and the"
|
||||||
|
" Flash declaration."
|
||||||
)
|
)
|
||||||
assert set(spec.param_bridge.values()) == set(declaration_names), (
|
assert declaration.parameters["required"] == shell["parameters"].get("required", []), (
|
||||||
f"'{spec.name}': param_bridge values must cover exactly the declaration"
|
spec.name
|
||||||
f" params. bridge={sorted(spec.param_bridge.values())}"
|
|
||||||
f" declaration={declaration_names}"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_renamed_params_are_documented():
|
def test_point_targets_accept_an_index_or_coordinates_in_both_profiles():
|
||||||
"""A cross-dialect rename is a deliberate difference; it must carry prose."""
|
"""click / long_press / input_text address an element by index or by
|
||||||
|
normalized coordinates; the description is required only for coordinates."""
|
||||||
|
for name in ("click", "long_press", "input_text"):
|
||||||
|
target = tool_declaration(name).parameters["properties"]["target"]
|
||||||
|
assert target["anyOf"] == [
|
||||||
|
{"type": "integer"},
|
||||||
|
{"type": "array", "items": {"type": "integer"}},
|
||||||
|
], name
|
||||||
|
assert "target_description" not in tool_declaration(name).parameters["required"], name
|
||||||
|
assert tool_declaration("click_sequence").parameters["required"] == [
|
||||||
|
"sequence",
|
||||||
|
"target_descriptions",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_shared_descriptions_are_identical_between_profiles():
|
||||||
|
for name in ("swipe", "press_key", "manage_app", "wait_for_delay"):
|
||||||
|
assert tool_declaration(name).description == operator_shell_tool(name).description
|
||||||
|
|
||||||
|
|
||||||
|
def test_documented_differences_concern_the_wire_only():
|
||||||
|
"""The agent dialect is single-sourced; a spec may only document how the
|
||||||
|
agent dialect differs from the wire (spelling, vocabulary, resolution)."""
|
||||||
for spec in ACTION_SPECS.values():
|
for spec in ACTION_SPECS.values():
|
||||||
renames = {k: v for k, v in spec.param_bridge.items() if k != v}
|
assert "declaration dialect" not in spec.differences, spec.name
|
||||||
if renames:
|
|
||||||
assert spec.differences, (
|
|
||||||
f"'{spec.name}' renames {renames} across dialects but documents no differences."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# --- Manifest coverage ---------------------------------------------------------------
|
# --- Manifest coverage ---------------------------------------------------------------
|
||||||
|
|||||||
Loaded 100 of 127 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user