Compare commits
13
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4c717afd15 | ||
|
|
ea93354a15 | ||
|
|
386f10817d | ||
|
|
7bcdb2acca | ||
|
|
b780a60e8d | ||
|
|
dccddaa7e8 | ||
|
|
cf110b74cd | ||
|
|
6de0b1b276 | ||
|
|
889a951c9b | ||
|
|
9c98b6bf8a | ||
|
|
1c1d700e9b | ||
|
|
c89588a80d | ||
|
|
fd92bafcb5 |
No files matched your search
@@ -0,0 +1,52 @@
|
||||
# Artemis Plugins 协作指南
|
||||
|
||||
## Scope
|
||||
|
||||
- 本文件适用于本仓库,当前用户指令优先。主程序 `artemis` 是独立仓库,两者分别提交、验证和推送。
|
||||
- 本仓库负责插件适配、工具契约、构建和市场索引;主程序负责模型、调度、资源锁、录屏、证据、任务与最终验收。
|
||||
- 修改适配器、工具、包或市场前,读 [DESIGN.md](DESIGN.md);具体调用规则见插件包内的 SKILL.md,验证结论见对应 VERIFICATION.md。
|
||||
- 先区分本地源码、远端发布引用和机器已安装版本。遇到旧 OCU/Cua 候选现场与市场文档不一致时,先查分叉和实际版本,不覆盖本地改动,不把旧候选文档直接当作当前市场状态。
|
||||
- 已有未发布提交或脏文件不属于本次任务时,隔离检出最新目标分支,只移植本次提交;不使用重置、清理、强推或整仓暂存来跳过比较。
|
||||
|
||||
## Commands and Evidence
|
||||
|
||||
以下是原生命令;宿主要求 RTK 时按宿主规则包装,不把本机工具路径写成项目依赖。
|
||||
|
||||
| 用途 | 工作目录 | 命令或入口 |
|
||||
| --- | --- | --- |
|
||||
| 比较当前状态 | 仓库根目录 | `git status --short --branch`、`git log --left-right --oneline HEAD...origin/main` |
|
||||
| Cua 确定性回归 | `plugins/cua-driver` | `go test ./...`、`go vet ./...`;Go 1.22+,Windows 适配 |
|
||||
| 构建参数 | 仓库根目录 | `python scripts/build_cua_driver.py --help`;固定上游 ZIP 和 Go 路径的用法见 [README](README.md) |
|
||||
| Windows 实测 | 仓库根目录 | 按对应验证记录运行 `tests/cua_smoke.py`、`tests/cua_fixture_validation.py`;使用已有 ARTEMIS 依赖和正确的 `PYTHONPATH` |
|
||||
| 提交前 | 仓库根目录 | `git diff --check`、`git diff --cached --check`,核对精确暂存路径 |
|
||||
|
||||
浏览器及前台实测先读 [Cua 验证记录](plugins/cua-driver/VERIFICATION.md),确认脚本会操作的测试窗口、应用及桌面资源。执行涉及用户应用的交互任务时,沿用主程序的 ARTEMIS 探索和资源锁,不为文档验证启动业务任务。
|
||||
|
||||
## Adapter Contracts
|
||||
|
||||
- 当前市场使用 Cua,旧 OCU 的源码、附件、用户安装和历史证据继续保留。版本以目标提交的 `catalog.json`、清单与发行说明核对,不能由本地未发布文件推定。
|
||||
- 复用宿主 `desktop:local`、任务固定版本、步骤与录像。插件不新增模型循环、业务队列、独立录屏器、自动回放或默认业务回滚流程。
|
||||
- 工具的 schema、`plugins/cua-driver/tools.json`、清单能力声明、实现和包内 Skill 同步更新。参数互斥、类型、窗口范围和错误语义在发送动作前校验。
|
||||
- `pid`、`window_id` 标识实际目标;应用名与 PID 筛选不能悄悄扩大范围。发现空结果和驱动错误分别处理,未知 PID 不使用 0。
|
||||
- 控件使用最新 `snapshot_id` 派生的 token;像素动作使用最新 `capture_id` 及同图坐标。窗口移动、缩放、尺寸或编号失效时重新观察,坐标映射不能重复缩放。
|
||||
- 浏览器前台输入先激活并核对准确 HWND,随后保持目标前台;只读观察不抢焦点。激活未确认不发送输入,前台操作会占用用户桌面,不承诺无干扰后台执行。
|
||||
- 每次调用只执行一次动作。明确未执行、部分执行、结果不明和已验证分别返回;结果未知的写操作不自动重发,投递成功不等于业务成功。
|
||||
- 操作后返回真实观察与来源;截图保存到任务 trace,拒绝覆盖和路径逃逸,附件有真实文件与 SHA256。灰屏或只有浏览器外壳不能当作有效业务观察。
|
||||
- 会话关闭、取消和宿主退出要回收适配器、Driver/UIA 子进程;原有 Windows Job Object 生命周期保持有效。
|
||||
|
||||
## Packaging and Release
|
||||
|
||||
- 固定上游版本、提交与 ZIP SHA256,保留许可证和包内来源记录;仅分发所需运行依赖,不执行全局安装器,不包含云环境或模型服务。
|
||||
- 包版本、MCP 握手版本、构建来源信息、发行标签和市场索引一致。只改适配层时按既有 +artemis.N 修订后缀约定发布新的不可变包,不覆盖旧版本附件。
|
||||
- 发布包、校验值、发行说明和实际验证结果齐全后再更新市场。主程序升级不自动升级或授权用户插件;任务结束后由用户操作更新。
|
||||
- 文档变更检查链接、命令与实际源码;代码变更运行相应 Go 回归和静态检查,必要时完成隔离窗口实测。契约检查、窗口烟测和完整业务验收分别报告。
|
||||
- Windows x64 预览及未测范围如实保留。候选阶段小样本、约 26.1% token 差异或 PNG 保存成功不能外推为稳定成功率、费用或跨端验证通过。
|
||||
- 只暂存本次具体路径;不提交 `artifacts/`、可执行文件、上游下载包、执行记录、凭据或个人目录。推送后读取远端分支并核对发布提交。
|
||||
|
||||
## References
|
||||
|
||||
- [适配与市场设计](DESIGN.md)
|
||||
- [安装与构建](README.md)
|
||||
- [Cua 调用规则](plugins/cua-driver/SKILL.md)
|
||||
- [Cua 发行说明](plugins/cua-driver/RELEASE.md)
|
||||
- [Cua 实测范围](plugins/cua-driver/VERIFICATION.md)
|
||||
@@ -0,0 +1,90 @@
|
||||
# Artemis Plugins 设计约定
|
||||
|
||||
## Overview
|
||||
|
||||
本仓库维护 ARTEMIS 插件适配与市场分发。当前发布基线是 Cua Driver 0.38.0 的 Windows x64 适配,市场版本为 `0.38.0+artemis.1`,保留预览标识。后续版本以目标分支的清单、市场索引、来源记录和发行说明为准。
|
||||
|
||||
这是适配器架构与行为设计,插件没有独立产品网页或通用配色。主程序网页的视觉规则由其根目录 DESIGN.md 管理;包内 [SKILL.md](plugins/cua-driver/SKILL.md) 指导调用方如何使用实际工具,不代替宿主权限或证据校验。
|
||||
|
||||
## System architecture
|
||||
|
||||
| 层 | 职责 |
|
||||
| --- | --- |
|
||||
| ARTEMIS 宿主 | 模型、任务准入、队列与资源互斥、插件安装授权和版本固定、录屏、步骤、附件、报告及最终验收 |
|
||||
| Go 适配器 | 通过 stdio MCP 暴露所需工具;校验工具参数、目标身份、截图几何与错误;转换真实观察、动作回执和附件 |
|
||||
| 固定 Cua Driver | 执行窗口发现、UIA、截图和输入;会话自有 Driver/UIA 子进程随宿主生命周期回收 |
|
||||
| 市场与安装包 | `catalog.json` 描述兼容平台、包地址、SHA256 与验证边界;构建产出自包含 ZIP、适配源码、许可证和 `provenance.json` |
|
||||
|
||||
插件沿用宿主的 `desktop:local` 资源和任务版本固定,不增设业务调度、AI、录屏、环境恢复或远程云服务。主程序仍可在同一任务中按需使用安卓和电脑能力,最终裁定由对应来源的真实证据支持。
|
||||
|
||||
旧 OCU 从市场入口下架,但其源码、旧发行附件、已安装版本和历史证据保留。插件安装、授权、更新及切换由用户发起,不能在任务运行中静默更换实现。
|
||||
|
||||
## Observation and action flow
|
||||
|
||||
1. 按应用名或已知 PID 发现目标窗口;同时给出筛选时取交集。未知 PID 省略或设 null,无效 PID 在驱动调用前拒绝,驱动错误不伪装成空列表。
|
||||
2. 读取窗口截图与 UIA 树,保存实际 `pid`、`window_id`、`snapshot_id`、`capture_id`、尺寸、位置与焦点信息。
|
||||
3. UIA 控件动作使用当前 token;像素动作只使用同一张截图上的窗口局部坐标,交给适配器完成所需映射。
|
||||
4. 在输入前校验参数互斥、窗口身份、截图有效性、坐标范围和必要的前台激活。旧图、窗口移动或缩放后重新观察。
|
||||
5. 一次调用只执行一次请求动作,随后返回真实观察与明确回执;新的 snapshot/token 和 capture 替代旧标识。
|
||||
6. 需要留档时保存当前窗口 PNG,读回真实文件并返回哈希与附件;宿主负责关联任务步骤、来源和报告。
|
||||
|
||||
应用名、窗口标题、地址栏及页面内容共同帮助核对身份;不能把无关前台应用的 PID 当成目标,也不能只因同一插件返回过一张图就复用到另一窗口。
|
||||
|
||||
## Input and coordinate contracts
|
||||
|
||||
| 工具 | 约束 |
|
||||
| --- | --- |
|
||||
| `click` | 使用最新控件 token;不与像素坐标混用 |
|
||||
| `click_at`、`drag` | 使用最新 `capture_id` 和同图坐标;拖拽两端均需校验 |
|
||||
| `scroll` | 最新截图中的滚动区域坐标与 `capture_id`,不用控件 token;刻度不是精确像素或固定页数,需确认实际位移 |
|
||||
| `press_key`、`type_text` | 操作目标窗口当前焦点;不传控件 token 或 x/y,先用独立动作聚焦 |
|
||||
| `set_value` | 必须给出当前可赋值控件 token,不能把同名文本标签当作输入框 |
|
||||
| `save_screenshot` | filename 只是文件名;保存到任务 trace,拒绝覆盖、路径逃逸和缺失真实文件的附件 |
|
||||
|
||||
UIA 的屏幕坐标不是截图局部像素。滚动映射、点击及拖拽的既有映射分别核对,避免重复缩放;旧 `capture_id`、越界、窗口几何变化或截图读取失败均在派发前拒绝。定位失败后刷新观察,不能原样重复同一错误动作。
|
||||
|
||||
## Foreground and evidence
|
||||
|
||||
Chromium 浏览器点击、按键、输入和滚动使用明确的 foreground 路径。适配器先调用既有激活能力并核实准确 HWND,再发送一次输入,随后保持目标前台供渲染、后续观察和录像。激活未确认时不发送业务输入,输入结果不明时不重发。
|
||||
|
||||
只读观察及截图保存不抢焦点,`capture_focus` 记录采集前后的前台状态。正文灰屏或只剩外壳时,先核对同一窗口的前台状态,再显式恢复目标并观察;不通过重复刷新、提交表单或业务点击来尝试恢复。
|
||||
|
||||
这条路径会占用用户前台。当前没有接入独立浏览器 DOM、已有登录会话的调试通道、隔离桌面或 Cua 云环境,不能承诺用户与模型互不干扰地操作同一桌面。
|
||||
|
||||
回执区分未执行、部分执行、无法确认及已验证。投递成功、PNG 保存或截图哈希一致只证明各自操作,不证明业务检查通过。安卓截图不能充当电脑证据,空白截图也不能充当网页缺陷证据。
|
||||
|
||||
## Lifecycle and distribution
|
||||
|
||||
每个插件会话启动自己的适配器、Driver 和 UIA helper;Windows Job Object 负责取消、关闭和宿主退出后的进程回收。遥测设置只作用于该子进程环境,不修改用户全局设置。
|
||||
|
||||
构建固定上游 Cua Driver 的版本、提交和 ZIP SHA256,只打包所需可执行运行依赖及适配源码、工具描述、许可证、使用规则和验证记录。用户安装自包含 ZIP,不需要为此另装 Go、Python、Node 或另一套 Codex MCP。
|
||||
|
||||
插件清单、握手版本和包名保持一致;`provenance.json` 记录实际上游、校验值和适配修订。只修改适配时使用新的 `+artemis.N` 包,既有同版本包不可覆盖。市场变更引用已验证的 HTTPS 安装包和哈希。
|
||||
|
||||
## Verification and change entrypoints
|
||||
|
||||
| 变更 | 主要入口与验证 |
|
||||
| --- | --- |
|
||||
| 窗口发现 | 适配器的窗口查找实现及对应 Go 回归;独立应用名发现脚本 |
|
||||
| 输入与状态回执 | `main.go`、`tools.json`、`plugin.json`、包内 Skill 共同维护;明确未执行与结果未知场景 |
|
||||
| 坐标与前台 | 发布分支的 `coordinates.go`、`foreground.go` 及对应测试;覆盖缩放、旧图、越界、移动、激活失败和只读不抢焦点 |
|
||||
| 包与市场 | `scripts/build_cua_driver.py`、来源记录、`catalog.json`、发行说明及公开下载校验 |
|
||||
| 真机联调 | 主程序调度与桌面资源锁内运行隔离窗口测试;真实业务、录像和最终裁定另外验收 |
|
||||
|
||||
具体 Go 与构建命令见 [AGENTS.md](AGENTS.md)。修改行为时同步工具声明、实现、文档及相应回归;只更新文档时检查命令、相对链接和所指版本,无需操作用户桌面或调用付费模型。
|
||||
|
||||
[VERIFICATION.md](plugins/cua-driver/VERIFICATION.md) 区分候选对照、应用名发现、输入、坐标和前台修订。候选的独立验收 3/3 与 AI 裁定 2 次通过、1 次未判定不能合并成“全部通过”;约 26.1% token 差异包含对照缺陷和重试成本。复杂浏览器控制台、多 DPI、多屏和长程混合任务继续按实际覆盖范围声明。
|
||||
|
||||
## Working across divergent checkouts
|
||||
|
||||
本地目录可能保留 OCU 与早期 Cua 候选改动,而发布引用已经包含后续修订。先检查 Git 状态、分叉提交和具体文件,分别记录源码状态、发布状态及实际安装状态;旧候选文档的结论不能覆盖发布记录。
|
||||
|
||||
发布范围较窄时可在最新目标分支的隔离工作树中移植本次提交,推送后核对远端 SHA。原目录的未归并分支和用户改动继续保留,不能为了发布文档而顺带推送全部历史、覆盖文件或删除本地开发现场。
|
||||
|
||||
## 声明式工作区扩展
|
||||
|
||||
`plugins/task-review` 使用清单版本 3 的 workspace 接口。插件声明面板位置和宽度、字段标签/选项/报告属性;宿主负责安全渲染、响应式布局、任务绑定存储、并发保存与报告缓存。纯界面扩展不声明 MCP、Skill、设备资源或调度器,不参与模型执行插件选择。
|
||||
|
||||
评审由人工填写,使用「通过 / 失败」按钮,备注独立选填。未装或停用时不贡献界面与新报告字段;卸载保留任务内记录,重装后恢复;任务删除时清理。插件不可改写原任务执行状态或既有复验结论。
|
||||
|
||||
工作区扩展从 ARTEMIS 2.3.0 起提供。catalog-v4.json 服务 2.4.0+,catalog-v3.json 保留给 2.3.x,旧 catalog.json 保留给严格校验清单版本的 2.2.x。2.3.x 不能解析 original_case 面板,人工评审 1.1.0 仅加入 v4 索引;旧索引保留兼容版本。执行插件更新应同步兼容的索引。先发布并回读核验不可变附件,再提交包含新插件的市场索引。
|
||||
@@ -1,38 +1,44 @@
|
||||
# Artemis Plugins
|
||||
|
||||
ARTEMIS 2.0.0 的插件市场仓库,用于分发经过验证的插件,供用户在 ARTEMIS 插件页面安装。
|
||||
ARTEMIS 的插件市场。配套 ARTEMIS 2.2.0 发布的电脑控制插件为 **Cua 0.38.0+artemis.1**,支持 Windows x64,通过窗口截图与可访问性树操作电脑应用和浏览器窗口,复用 ARTEMIS 的调度、资源互斥、步骤、截图附件和录像。
|
||||
|
||||
电脑控制 1.0.0 以开发预览提供市场安装,适用于 Windows x64 和 ARTEMIS 2.0.0 开发版。已验证基础窗口、浏览器表单和 Flash 手机联动;复杂控制台、多 DPI、多显示器和长程场景仍需验证。具体结果见 [验证记录](plugins/open-computer-use/VERIFICATION.md)。
|
||||
在 ARTEMIS 的「插件」页面安装「电脑控制 · Cua」,授权并启用即可。安卓任务可根据提示词按需调用电脑控制,不需要单独配置 Codex MCP 或安装 Go、Node、Python。
|
||||
|
||||
## 仓库内容
|
||||
## 从 OCU 切换
|
||||
|
||||
- `catalog.json`:市场索引,当前契约版本为 1,空列表表示没有已发布插件。
|
||||
- `plugins/open-computer-use`:首发插件清单、适配 Skill 与验证记录。
|
||||
- `scripts/build_computer_use.py`:校验固定的 npm 1.1.0 安装包与上游源码,在构建阶段应用 Windows 截图和输入适配,运行 Go 测试并编译,生成包含源码、许可证的 ZIP 与 SHA-256;不执行 npm 安装脚本。
|
||||
- 安装包通过本仓库的发行附件分发;`artifacts/` 下的本地测试包不进入 Git。
|
||||
Open Computer Use(OCU)已从市场索引下架,当前市场只提供 Cua。已安装 OCU 的用户请先安装并启用 Cua,再停用 OCU,避免同一任务加载两套电脑控制工具。排队、暂停或运行中的任务结束后再切换。
|
||||
|
||||
首发计划仅适配 [open-codex-computer-use](https://github.com/iFurySt/open-codex-computer-use),通过同一插件操作 Windows 应用及浏览器窗口。用户在插件页面安装、授权并启用后,安卓任务可由 AI 按需调用,无需选择组合执行目标。
|
||||
旧插件的源码、发行附件、历史任务与证据继续保留。下架不会删除用户的已安装版本,也不会替其他用户自动授权新插件。需要复核旧任务时仍可按其固定版本处理。
|
||||
|
||||
## 本地构建和验证
|
||||
## 验证范围
|
||||
|
||||
构建机需要 Python 和 Go 1.22+;最终用户安装包不需要 Go。准备 npm `open-computer-use@1.1.0` 原始 tarball,以及上游提交 `7d8cb3fa5618d1ed8ec71773434ac3677f1922c9` 的干净 Git 检出:
|
||||
本版本固定上游 [Cua Driver 0.38.0](https://github.com/trycua/cua/releases/tag/cua-driver-rs-v0.38.0),增加 ARTEMIS 窗口归属与焦点适配、已知无标题 Shell/菜单兼容补丁,修正滚动重复缩放。插件继续标注开发预览,升级内容见[发行说明](plugins/cua-driver/RELEASE.md)。
|
||||
|
||||
此版本通过 Go 检查与构建、菜单/任务栏 9 次调用、隔离 Brave 表单 38 次调用验证,并参与 2.2.0 仅 PC 与跨端集中回归。旧版窗口录像、对照与性能记录保留各自版本范围,不计为新版重测。复杂控制台、多 DPI、多显示器与混合长程任务仍需进一步验证,详见[验证记录](plugins/cua-driver/VERIFICATION.md)。
|
||||
|
||||
适配修订另覆盖应用名发现、浏览器键盘与中文输入、截图缩放与坐标校验,以及前台激活后保持目标可见。Cua 需要前台输入时会占用用户桌面,不提供与用户操作互不干扰的执行环境;复杂业务是否通过仍以任务实际证据为准。
|
||||
|
||||
## 构建与验证
|
||||
|
||||
当前发布源码为 `0.38.0+artemis.1`。构建机需要 Python、Go、Rust 与 Windows C++/Spectre 依赖;最终用户只安装 ZIP。准备固定上游源码并应用包内兼容补丁后,从官方发行下载 `cua-driver-rs-0.38.0-windows-x86_64-binary.zip`;脚本校验来源后构建,不执行全局安装器,详见[构建说明](plugins/cua-driver/BUILD.md)。
|
||||
|
||||
```sh
|
||||
python scripts/build_computer_use.py open-computer-use-1.1.0.tgz --upstream-source <上游源码目录> --go <go可执行文件>
|
||||
python tests/windows_capture_smoke.py artifacts/open-computer-use.exe
|
||||
python tests/browser_smoke.py artifacts/open-computer-use.exe --chrome <Chrome可执行文件>
|
||||
python scripts/build_cua_driver.py <上游ZIP> --upstream-source <已应用补丁的上游源码目录> --go <Go可执行文件>
|
||||
python tests/cua_smoke.py <安装后的artemis-cua-adapter.exe> artifacts/cua-smoke
|
||||
python tests/cua_fixture_validation.py <安装后的artemis-cua-adapter.exe>
|
||||
python tests/cua_discovery_smoke.py <安装后的artemis-cua-adapter.exe> --app Brave --output artifacts/cua-discovery.json
|
||||
```
|
||||
|
||||
第二条命令使用安装了 ARTEMIS 依赖的 Python,创建并关闭两个临时窗口,验证遮挡截图、坐标对应和按钮实际收到点击。不会操作用户已有应用。
|
||||
验证脚本使用已安装 ARTEMIS 依赖的 Python,并将 ARTEMIS 源码目录放入 `PYTHONPATH`。前两项实测只操作自行创建的测试窗口;窗口发现实测只读取指定的已打开应用,不改变其页面。历史 A/B 脚本仍固定 OCU 1.1.0 与 Cua 0.1.2,便于复现已有测量;它不是安装后必须执行的步骤。
|
||||
|
||||
第三条命令创建独立 Chrome 配置和本地测试表单,通过 MCP 填写中文、提交一次并读回结果,退出后关闭测试浏览器。
|
||||
`catalog.json` 是市场索引,包含清单、HTTPS 安装包地址、SHA256 和验证说明。`artifacts/` 保存本机安装包与测试资料,不进入 Git。版本文件不可原地覆盖,任务固定使用其启动时的版本。
|
||||
|
||||
ARTEMIS 适配将屏幕区域截取改为目标窗口 `PrintWindow`;失败或空白画面不回退为截屏。鼠标消息根据目标窗口的子控件路由,工具声明不再向模型推荐 Windows 未实现的点击方式。适配源文件与构建后的完整源码都随包分发,未修改上游仓库。
|
||||
插件版本跟随所用 Cua Driver 上游版本。市场已发布适配版为 `0.36.0+artemis.4`,发行标签 `cua-driver-v0.36.0+artemis.4`;当前源码候选为 `0.38.0+artemis.1`,对应上游标签 `cua-driver-rs-v0.38.0`。只修改 ARTEMIS 适配而未升级上游时使用 `+artemis.N` 后缀区分不可变安装包;包内 `provenance.json` 记录上游提交、原包校验值和适配修订,构建会检查版本对应关系。
|
||||
|
||||
## 发布约定
|
||||
[发行说明](plugins/cua-driver/RELEASE.md) · [ARTEMIS 主项目](https://git.playones.com/chenmingxuan/Artemis-Eapil)
|
||||
|
||||
每个可安装版本需要声明支持的 ARTEMIS 版本和平台,固定运行依赖,提供安装包校验值、更新说明及实际验证记录。插件配置不包含账号密钥、个人目录或设备信息。
|
||||
## 工作区扩展:任务人工评审 1.1.0
|
||||
|
||||
安装使用最新已验证且兼容的版本;后续升级由用户在 ARTEMIS 中操作,正在执行的任务固定插件版本。
|
||||
[task-review](plugins/task-review/README.md) 使用 ARTEMIS 清单版本 3,为任务回放增加原始用例与人工评审侧栏,支持自动保存和上一条/下一条,并将选填结果写入报告。此插件不运行 MCP 或设备操作,需要 ARTEMIS 2.4.0+,在主程序插件市场安装并启用;2.2.x 尚不支持。
|
||||
|
||||
[ARTEMIS 主项目](https://git.playones.com/chenmingxuan/Artemis-Eapil)
|
||||
构建:`python scripts/build_task_review.py`。输出 ZIP 与校验文件位于 `artifacts/task-review/`。验证范围见 [VERIFICATION](plugins/task-review/VERIFICATION.md)。发行标签 `task-review-v1.1.0`,市场安装包与 SHA256 以 `catalog-v4.json` 为准;`catalog.json` 保留给 2.2.x。
|
||||
+282
@@ -0,0 +1,282 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"plugins": [
|
||||
{
|
||||
"manifest": {
|
||||
"schema_version": 2,
|
||||
"id": "cua-driver",
|
||||
"name": "电脑控制 · Cua",
|
||||
"version": "0.38.0+artemis.1",
|
||||
"description": "通过 Cua Driver 的窗口截图与可访问性树操作 Windows 应用及浏览器窗口;复用 ARTEMIS 的任务调度、步骤证据与录像。",
|
||||
"artemis_version": ">=2.0.0,<3.0.0",
|
||||
"platforms": [
|
||||
"win32"
|
||||
],
|
||||
"architectures": [
|
||||
"amd64"
|
||||
],
|
||||
"source": "https://github.com/trycua/cua",
|
||||
"license": "MIT",
|
||||
"server": {
|
||||
"transport": "stdio",
|
||||
"command": "${plugin_dir}/bin/artemis-cua-adapter.exe"
|
||||
},
|
||||
"skill": "SKILL.md",
|
||||
"targets": [
|
||||
"computer",
|
||||
"mixed"
|
||||
],
|
||||
"resources": [
|
||||
"desktop:local"
|
||||
],
|
||||
"observation": {
|
||||
"tool": "list_windows",
|
||||
"arguments": {
|
||||
"on_screen_only": true
|
||||
},
|
||||
"refresh_tools": [
|
||||
"get_window_state"
|
||||
]
|
||||
},
|
||||
"capabilities": {
|
||||
"kind": "visual",
|
||||
"recording": "windows",
|
||||
"tools": {
|
||||
"list_windows": {
|
||||
"effect": "read",
|
||||
"observation": true,
|
||||
"idempotent": true,
|
||||
"target_argument": "",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
"get_window_state": {
|
||||
"effect": "read",
|
||||
"observation": true,
|
||||
"idempotent": true,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"verify_state": {
|
||||
"effect": "read",
|
||||
"observation": false,
|
||||
"idempotent": true,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"click": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"drag": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"type_text": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"press_key": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"set_value": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"scroll": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"save_screenshot": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image",
|
||||
"file"
|
||||
]
|
||||
},
|
||||
"click_at": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"bring_to_front": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": true,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image",
|
||||
"state"
|
||||
]
|
||||
}
|
||||
},
|
||||
"required_tools": [
|
||||
"list_windows",
|
||||
"get_window_state",
|
||||
"save_screenshot"
|
||||
]
|
||||
}
|
||||
},
|
||||
"release_stage": "preview",
|
||||
"archive_url": "https://git.playones.com/chenmingxuan/Artemis-Plugins/releases/download/cua-driver-v0.38.0+artemis.1/cua-driver-0.38.0+artemis.1-windows-amd64.zip",
|
||||
"sha256": "c007da16d58c4165a1a344113e650e68bd92a69ad3ce4ca8285818c5426dc0f4",
|
||||
"verification": "Windows x64 预览,配套 ARTEMIS 2.2.0。固定 Driver 0.38.0、源码提交和官方 ZIP 校验值;包含已知无标题 Shell/菜单兼容补丁及重新构建运行库。Go 测试、vet、构建及包内源码/二进制校验通过;菜单/任务栏 9 次调用、隔离 Brave 表单 38 次调用通过,并参与仅 PC 与跨端集中回归。前台输入占用桌面;显示光标与系统鼠标偶发分离待复现,未全面覆盖多屏/DPI、长时间压力、其他托盘菜单及原失败业务全流程。详情见 plugins/cua-driver/VERIFICATION.md。"
|
||||
},
|
||||
{
|
||||
"manifest": {
|
||||
"schema_version": 3,
|
||||
"id": "task-review",
|
||||
"name": "任务人工评审",
|
||||
"version": "1.0.0",
|
||||
"description": "在步骤回放中查看初始需求提示词,由人工标记 AI 执行准确性与测试结果正确性,按需填写备注并随报告导出。",
|
||||
"artemis_version": ">=2.3.0,<3.0.0",
|
||||
"platforms": [
|
||||
"win32",
|
||||
"linux",
|
||||
"darwin"
|
||||
],
|
||||
"architectures": [
|
||||
"amd64",
|
||||
"arm64"
|
||||
],
|
||||
"license": "Apache-2.0",
|
||||
"targets": [],
|
||||
"capabilities": {
|
||||
"kind": "other",
|
||||
"recording": "none",
|
||||
"tools": {}
|
||||
},
|
||||
"workspace": {
|
||||
"panels": [
|
||||
{
|
||||
"id": "requirements",
|
||||
"title": "任务需求",
|
||||
"side": "left",
|
||||
"kind": "initial_prompt",
|
||||
"width": {
|
||||
"preferred": 340,
|
||||
"minimum": 240,
|
||||
"maximum": 480
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "review",
|
||||
"title": "人工评审",
|
||||
"side": "right",
|
||||
"kind": "form",
|
||||
"width": {
|
||||
"preferred": 360,
|
||||
"minimum": 260,
|
||||
"maximum": 520
|
||||
}
|
||||
}
|
||||
],
|
||||
"fields": [
|
||||
{
|
||||
"id": "execution_accuracy",
|
||||
"label": "AI执行准确性",
|
||||
"description": "Artemis自动化是否按照内部实际测试用例步骤执行",
|
||||
"kind": "choice",
|
||||
"options": [
|
||||
"通过",
|
||||
"失败"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "execution_note",
|
||||
"label": "AI执行准确性备注",
|
||||
"kind": "text",
|
||||
"max_length": 2000
|
||||
},
|
||||
{
|
||||
"id": "result_correctness",
|
||||
"label": "测试结果正确性",
|
||||
"description": "Artemis 按照内部实际用例步骤执行后,人工再次确认结果正确性",
|
||||
"kind": "choice",
|
||||
"options": [
|
||||
"通过",
|
||||
"失败"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "result_note",
|
||||
"label": "测试结果正确性备注",
|
||||
"kind": "text",
|
||||
"max_length": 2000
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"release_stage": "stable",
|
||||
"archive_url": "https://git.playones.com/chenmingxuan/Artemis-Plugins/releases/download/task-review-v1.0.0/task-review-1.0.0.zip",
|
||||
"sha256": "5487fef365cfa9ec5d2eda7be60b5a5110bcf2edd24b2d10495ccda8ca02c454",
|
||||
"verification": "ARTEMIS 2.3.0+ 声明式工作区插件,无 MCP、模型或设备操作。Windows 隔离示例验证 Markdown 初始需求、左右侧栏、人工中文评分、选填备注与 Word/Excel 导出;宿主默认回归 3197 项、前端 336 项及构建通过。macOS/Linux 未实测;真实业务结果由人工评审,详见 plugins/task-review/VERIFICATION.md。"
|
||||
}
|
||||
]
|
||||
}
|
||||
+189
-1
@@ -1,4 +1,192 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"plugins": []
|
||||
"plugins": [
|
||||
{
|
||||
"manifest": {
|
||||
"schema_version": 2,
|
||||
"id": "cua-driver",
|
||||
"name": "电脑控制 · Cua",
|
||||
"version": "0.38.0+artemis.1",
|
||||
"description": "通过 Cua Driver 的窗口截图与可访问性树操作 Windows 应用及浏览器窗口;复用 ARTEMIS 的任务调度、步骤证据与录像。",
|
||||
"artemis_version": ">=2.0.0,<3.0.0",
|
||||
"platforms": [
|
||||
"win32"
|
||||
],
|
||||
"architectures": [
|
||||
"amd64"
|
||||
],
|
||||
"source": "https://github.com/trycua/cua",
|
||||
"license": "MIT",
|
||||
"server": {
|
||||
"transport": "stdio",
|
||||
"command": "${plugin_dir}/bin/artemis-cua-adapter.exe"
|
||||
},
|
||||
"skill": "SKILL.md",
|
||||
"targets": [
|
||||
"computer",
|
||||
"mixed"
|
||||
],
|
||||
"resources": [
|
||||
"desktop:local"
|
||||
],
|
||||
"observation": {
|
||||
"tool": "list_windows",
|
||||
"arguments": {
|
||||
"on_screen_only": true
|
||||
},
|
||||
"refresh_tools": [
|
||||
"get_window_state"
|
||||
]
|
||||
},
|
||||
"capabilities": {
|
||||
"kind": "visual",
|
||||
"recording": "windows",
|
||||
"tools": {
|
||||
"list_windows": {
|
||||
"effect": "read",
|
||||
"observation": true,
|
||||
"idempotent": true,
|
||||
"target_argument": "",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
"get_window_state": {
|
||||
"effect": "read",
|
||||
"observation": true,
|
||||
"idempotent": true,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"verify_state": {
|
||||
"effect": "read",
|
||||
"observation": false,
|
||||
"idempotent": true,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"click": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"drag": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"type_text": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"press_key": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"set_value": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"scroll": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"save_screenshot": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image",
|
||||
"file"
|
||||
]
|
||||
},
|
||||
"click_at": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"bring_to_front": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": true,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image",
|
||||
"state"
|
||||
]
|
||||
}
|
||||
},
|
||||
"required_tools": [
|
||||
"list_windows",
|
||||
"get_window_state",
|
||||
"save_screenshot"
|
||||
]
|
||||
}
|
||||
},
|
||||
"release_stage": "preview",
|
||||
"archive_url": "https://git.playones.com/chenmingxuan/Artemis-Plugins/releases/download/cua-driver-v0.38.0+artemis.1/cua-driver-0.38.0+artemis.1-windows-amd64.zip",
|
||||
"sha256": "c007da16d58c4165a1a344113e650e68bd92a69ad3ce4ca8285818c5426dc0f4",
|
||||
"verification": "Windows x64 预览,配套 ARTEMIS 2.2.0。固定 Driver 0.38.0、源码提交和官方 ZIP 校验值;包含已知无标题 Shell/菜单兼容补丁及重新构建运行库。Go 测试、vet、构建及包内源码/二进制校验通过;菜单/任务栏 9 次调用、隔离 Brave 表单 38 次调用通过,并参与仅 PC 与跨端集中回归。前台输入占用桌面;显示光标与系统鼠标偶发分离待复现,未全面覆盖多屏/DPI、长时间压力、其他托盘菜单及原失败业务全流程。详情见 plugins/cua-driver/VERIFICATION.md。"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
# 构建 0.38.0+artemis.1
|
||||
|
||||
开发机需要 Python、Go、上游固定的 Rust 工具链、Windows C++ x64 工具链及对应 Spectre 运行库。用户安装插件 ZIP 不需要这些工具。不会运行 Cua 全局安装器。
|
||||
|
||||
1. 获取官方 `cua-driver-rs-v0.38.0` 源码,核对 HEAD 为 `85e2b1318bd77ccf3cfa88ccdf875a4bb7eaed19`。保留完整源码;稀疏检出至少包括 `libs/cua-driver/rust`、`libs/cua`、`libs/images/sandbox-images.json`。
|
||||
2. 在上游源码根目录应用本插件的 `patches/cua-driver-0.38.0-untitled-shell.patch`。补丁只修改 Windows 精确窗口解析,不放宽普通窗口发现。
|
||||
3. 下载官方 Windows x64 ZIP。在具备上述编译依赖的终端,从 Artemis-Plugins 根目录执行:
|
||||
|
||||
```sh
|
||||
python scripts/build_cua_driver.py <官方ZIP> --upstream-source <上游源码根目录> --go <Go可执行文件>
|
||||
```
|
||||
|
||||
构建脚本校验官方 ZIP SHA256、上游 HEAD、唯一修改文件与补丁内容,使用 `cargo build --release --locked` 重新构建 Driver、UIA helper 和 SDK DLL,再运行 Go 测试并构建适配器。仅游标主题工具沿用官方原包二进制。
|
||||
|
||||
`provenance.json` 区分官方 ZIP 来源、兼容补丁及重新构建文件,记录各自 SHA256。打包后先运行确定性测试及授权的隔离窗口回归,再安装到本机或发布。市场索引与发布附件需另行更新,不由构建脚本自动修改。
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2025 Cua AI, Inc.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Cua 电脑控制 0.38.0+artemis.1
|
||||
|
||||
2026-10-27,与 ARTEMIS 2.2.0 配套发布。同步官方 [Cua Driver 0.38.0](https://github.com/trycua/cua/releases/tag/cua-driver-rs-v0.38.0),替换原 0.36.0 驱动。插件仍为 Windows x64 开发预览。既有任务与旧安装包保留原版本,主程序升级不会自动更新或授权插件;当前任务结束后在插件市场单独更新。
|
||||
|
||||
补充可见的任务栏、托盘及原生菜单窗口发现。操作后读取实际键盘焦点,只在 PID、线程、拥有关系和窗口类型验证通过时,将观察目标跟随到相关菜单或 Shell 窗口;无关应用、同进程的其他普通窗口及归属不明的菜单不会自动接管。返回 keyboard_focus、observation_target 和实际截图来源,后续控件 token 和坐标绑定实际观察窗口。
|
||||
|
||||
官方驱动按精确 HWND 解析窗口时仍过滤空标题,导致已找到的原生菜单被报为不存在。此包对固定源码增加一处兼容补丁,仅放行已知 Shell/菜单类型的无标题窗口,保留可见性和 PID/HWND 校验;普通应用发现与叠加层过滤保持原逻辑。Driver、UIA helper 与 SDK DLL 从带补丁的源码重新构建,包内记录补丁和二进制 SHA256,详见 [构建说明](BUILD.md)。
|
||||
|
||||
0.38.0 已在驱动内映射截图坐标,因此移除旧滚动预缩放,避免重复转换。适配器继续校验最新 capture_id、窗口归属、物理边界、尺寸和坐标范围。未知写入结果不会重发;取证失败不转发未经窗口校验的图片。
|
||||
|
||||
前台动作沿用先激活并核实目标、再投递一次输入的路径;只读观察不抢焦点。截图成功、动作投递与业务验证分别记录。共享 ARTEMIS 的 desktop:local 锁、调度、证据和录像,不新增控制通道或模型循环。
|
||||
|
||||
上游提交:`85e2b1318bd77ccf3cfa88ccdf875a4bb7eaed19`。官方 Windows x64 ZIP SHA256:`5edaa74d0afa1b55a9668dbc4f32736507bd2bd4d75b18ca1dc8881bb12402b5`。MIT 许可证随包保留;不包含 Cua Spaces 或云环境。用户安装 ZIP 即可,构建依赖仅供开发者使用。实测结果与未测范围见 [验证记录](VERIFICATION.md)。
|
||||
|
||||
发布前 Go 测试、vet 与构建通过,核对包内源码、补丁和运行库校验值。此版本已有菜单/任务栏 9 次调用、隔离 Brave 表单 38 次调用验证,并参与 ARTEMIS 2.2.0 的仅 PC 与跨端集中回归。多屏/DPI、长时间压力和原失败业务全流程未全面覆盖;显示光标与系统鼠标偶发分离仍待复现,不宣称本次已修复。
|
||||
|
||||
发布附件包含 Windows amd64 ZIP 与 SHA256SUMS.txt;包内 provenance.json 记录固定上游、兼容补丁和重新构建运行库的校验值。
|
||||
@@ -0,0 +1,19 @@
|
||||
# Cua 电脑控制(Windows)
|
||||
|
||||
通过本地 Cua Driver 操作用户授权的 Windows 窗口。调度、资源互斥、模型、步骤、截图和录像由 ARTEMIS 提供;插件本身不运行 AI,不另起后台服务。
|
||||
|
||||
1. 先按应用名发现窗口,例如 `list_windows({"app":"Brave","on_screen_only":true})`;未知 `pid` 必须省略或填 `null`,不能填 `0`。`app` 与非空 `pid` 同时提供时会取交集,只有该应用实际返回的 PID 才能用于后续调用。空结果时先按 `lookup.suggested_arguments` 纠正筛选(去掉错误 PID 或检查最小化窗口),不能直接宣称浏览器不可用,也不能把无关前台进程号当成目标 PID。取得目标的 `pid`、`window_id` 后读取真实画面;只操作任务要求的窗口,不关闭其他应用。
|
||||
2. `get_window_state` 同时读取截图和 UIA 控件树。最新 `snapshot_id` 加 `:N` 即控件 `element_token`。优先用 `click` 点击最新控件;视觉目标用 `click_at`,坐标仅取本次截图上的窗口局部像素,不能用屏幕坐标,需携带本次 `capture_id`。两种点击参数互斥,不混用。
|
||||
3. 所有窗口操作明确传 `pid`、`window_id`。`set_value` 必须带最新 `element_token`,用于设置具有 set_value 动作的输入框;同名 Text 标签不能当成 Edit 输入框。`press_key` 和 `type_text` 只操作当前焦点,不能传 `element_token`、`x`、`y`(也不要补 0/null)。需要聚焦控件时先 `click` / `click_at`,再输入。`press_key` 传单键及 `modifiers` 数组。
|
||||
4. 每个动作后插件自动返回一次新截图和控件树,新 snapshot 替换旧 token。若操作打开原生菜单、托盘或把焦点切到任务栏,检查 `observation_target.observed` 和 `artemis.source.target_id`,后续使用实际观察目标的 PID/HWND 与新 token;旧窗口编号不代表当前焦点。`keyboard_focus` 与 `post_action_focus` 记录真实焦点。归属不明或取证失败时先只读核对,不重复业务动作。原生菜单可向其拥有者窗口发送 Escape 关闭。仅在画面变化或需要核验时再次观察。
|
||||
5. 默认后台操作;明确指定 foreground 时,适配器先使用 Cua 自带 bring_to_front 激活并核实目标,随后保持目标在前台,供页面渲染和连续观察;不会每次动作后切回原应用,也不能承诺不影响用户前台工作。Brave/Chrome/Edge 等 Chromium 浏览器的点击(尤其滚动后)、快捷键、文字输入和滚动直接使用 `delivery_mode=foreground`。例如新标签页:`press_key(key="t",modifiers=["ctrl"],delivery_mode="foreground")`;地址导航:Ctrl+L → `type_text(text=网址,delivery_mode="foreground")` → Enter,各调用均带同一目标 pid/window_id。检查新页面再继续。只读观察/保存截图不会抢焦点;若正文灰屏或控件树缺少网页正文,检查 capture_focus,调用同一 pid/window_id 的 bring_to_front,再观察页面;不要靠重复刷新或业务点击恢复。用户切换窗口后需要重新确认前台,不能拿灰屏判网站失败。`effect=unverifiable/suspected_noop` 仅代表投递/效果未确认,必须检查返回画面和读回值;`refused` 是未执行。其他应用后台无效时,可在任务授权范围内显式 foreground 后再次检查。不要仅凭工具成功声称测试通过。
|
||||
6. `scroll` 必须带最新截图中滚动区域内的 `x,y`,不能传 `element_token`。使用 up/down/left/right、amount、by=line/page;滚轮刻度不保证固定像素或整页距离,根据返回画面确认实际位移再继续。`get_window_state(since="latest")` 可在同一窗口读取差异,但新 snapshot_id 仍替换旧 token。
|
||||
7. `save_screenshot` 保存最新窗口 PNG 到当前任务 trace,filename 仅是文件名,不能传路径,不覆盖已有文件。实际成功返回 SHA256 和附件;不得把工具内存截图当成已保存文件。
|
||||
|
||||
不提供 Cua 自带调度、录屏、回放、配置修改、安装器、任意脚本或自动升级接口。浏览器本轮使用窗口视觉/UIA 通路;未接入 DOM 或已有登录浏览器调试权限。此包基于官方 0.38.0,另含有明确来源记录的无标题 Shell 窗口兼容补丁,构建方式见 BUILD.md,实测范围见 VERIFICATION.md。
|
||||
|
||||
## 截图坐标与证据
|
||||
|
||||
- `click_at`、`scroll`、`drag` 都携带最新返回截图的 `capture_id`,坐标取自同一张图的实际宽高。动作返回的新截图可能改变分辨率;旧编号、旧坐标不可沿用。窗口移动、缩放或坐标越界被拒绝后重新观察,不反复重试原参数。
|
||||
- 缩放转换由 Cua Driver 处理,适配器校验截图绑定与范围,不自行乘 DPI 或加窗口偏移。优先用最新 `element_token` 点击可访问控件;UIA 的屏幕 `frame` 不能作为截图坐标。
|
||||
- 操作及保存截图前,从已有观察核对应用、窗口标题和页面身份(浏览器结合地址栏、选中标签及页面内容)。身份不符时定位正确页面再操作。无位移的滚动不能证明到底;先检查遮挡、焦点和滚动容器,再决定补一次边界验证或换路径。
|
||||
@@ -0,0 +1,101 @@
|
||||
# Cua 验证记录
|
||||
|
||||
## 0.38.0+artemis.1 发布核对(2026-10-27)
|
||||
|
||||
重新执行 Go 测试、vet 和构建,核对最终包内源码、兼容补丁及运行库 SHA256。10 月 26 日主程序集中回归中,此插件参与仅 PC 输入、双击、清空和两轮 PC 到 Android 的校验码传递,均有两端独立证据;该组正式任务共 9 项通过,另有准备探索超出指定页面范围而失败,未计入通过数。模型统一 gpt-6-luna/high,未将历史 OCU 对照性能数据作为此次发行性能结论。
|
||||
|
||||
下述 10 月 25 日的菜单/任务栏及浏览器验证仍对应同一动作实现。此次发布主要补全发行文档、构建来源、校验值和市场索引,不操作用户桌面。显示光标与系统鼠标偶发分离尚待复现;长期压力、全部多屏/DPI、其他托盘应用和原失败业务全流程仍未全面覆盖。
|
||||
|
||||
## 0.38.0+artemis.1 本地升级验证(2026-10-25)
|
||||
|
||||
官方 Driver 0.38.0、固定提交及 Windows x64 ZIP SHA256 均已核对。Driver、UIA helper 和 SDK DLL 使用原锁文件从固定源码构建,唯一源码差异为精确 PID/HWND 解析允许已知 Shell/菜单类型空标题;补丁及重新构建文件 SHA256 随包保存。未运行全局安装器,未修改市场索引。
|
||||
|
||||
Go 测试、vet、适配器与运行库构建通过。新增确定性回归覆盖:补充 Shell 窗口时保留应用/PID 筛选与去重、真实键盘焦点选择、拒绝外部应用或无关同进程窗口、无 owner 的原生菜单按已验证菜单拥有者与线程归属定位,以及滚动/拖拽不重复缩放。Python 构建与测试脚本通过 Ruff 检查。
|
||||
|
||||
在 ARTEMIS `desktop:local` 锁内完成两组隔离桌面验证,未调用付费模型或重跑用户业务任务:
|
||||
|
||||
- `tests/cua_shell_smoke.py --taskbar`:9 次调用。自有 WinForms 窗口的菜单按钮只派发一次,发现并获取无标题 `#32768` 菜单的 PNG 与控件树;来源匹配实际菜单 HWND,窗口枚举可再次找到菜单。Escape 关闭菜单,独立进程计数确认按钮触发 1 次、菜单业务项触发 0 次。Win+B 后焦点及截图目标跟随 `Shell_TrayWnd`。
|
||||
- `tests/cua_browser_smoke.py --foreground-check`:38 次调用。临时 Brave profile 和本地表单验证 Ctrl+T、新标签导航、UIA 值设置、中文真实输入、表单恰好提交一次、实际滚动、页底按钮与 PNG 哈希。普通测试窗口遮挡后的显式前台切换、5 秒后焦点保持、只读观察不抢焦点和显式恢复均通过。
|
||||
|
||||
首次菜单测试出现前台 HWND 为空且按钮计数为 0,用户反馈可能手动点走窗口;该次保留失败证据,不能用作已确认的驱动故障。随后完整菜单与浏览器验证通过。记录保存在主程序的本地 `artifacts/validation/2026-10-25/cua-shell-038-repeat` 与 `cua-browser-038`。
|
||||
|
||||
本轮未验证真实 Codex/CodexM 退出、其他托盘应用菜单、全部多屏/DPI 组合或长程业务成功率。旧版本的模型、性能及录像数据仍属于下列历史记录,不计入本次验证。
|
||||
|
||||
## 候选阶段记录
|
||||
|
||||
以下记录对应市场替换前的候选测试。适配版本 0.1.2;上游 Cua Driver 0.36.0(预发布),仅 Windows x64。默认模型和默认插件保持原样。
|
||||
|
||||
端到端联调还发现 ARTEMIS SDK 配置构造遗漏 `recording`,导致任务指定窗口录制却使用默认屏幕录制。主项目已补传该配置,覆盖从任务默认值、SDK Builder 到 PluginRecordings 的回归;新工作进程实际录像索引已确认 `scope=window`。早期屏幕录像只保留在本机,测量报告披露各次实际录制范围,不混同为统一录制条件。
|
||||
|
||||
已验证:
|
||||
|
||||
- 11 个工具符合 ARTEMIS schema 2 清单,命名、只读声明与窗口参数匹配。
|
||||
- HWND/PID 与物理窗口边界核对、真实窗口截图、动作后新 snapshot/token。
|
||||
- 独立窗口确认点击实际到达;表单值与勾选状态正确,蓝色小目标命中,滚动到末尾后完成。
|
||||
- 后台滚动的明确预检拒绝与前台执行区分;未自动重试写操作。
|
||||
- 新 PNG 保存到 trace,读回 SHA256 校验,附件被证据索引接收;拒绝覆盖已有文件与路径逃逸。
|
||||
- 使用 ARTEMIS 原有录像管理器录制 PC 窗口,H.264 文件确实生成;关闭会话后候选适配器、Driver 和 UIA 子进程均退出。
|
||||
- Go 回归覆盖动作回执、未知结果、拒绝、参数边界与两种点击路径互斥;Python 工具经过 Ruff 检查。
|
||||
|
||||
联调发现并修正:0.1.0 同一点击工具允许控件与像素参数,模型混填导致上游拒绝;0.1.1 拆分 `click` 与 `click_at`。0.1.2 补充上游 Windows 类级 `background_unavailable` 的未执行回执;带 actuator cause 的失败仍保留不确定状态。
|
||||
|
||||
实际 A/B 数据由 `tests/benchmark_plugins.py` 和 `tests/summarize_benchmark.py` 生成,保存于本机 `artifacts/`。测试使用独立窗口、固定任务和统一调度,比较表单、34 像素视觉目标、滚动及截图留档;不能由这些小样本外推浏览器、所有 DPI、真实路由器或长程任务的成功率。模型无工具调用、兜底模型与网关缓存情况必须随数据披露。
|
||||
|
||||
2026-10-17 完成一个复合用例各 3 次:Flash,主模型 `mimo-v2.6-flash/high`,保留 `gpt-6-luna` 兜底,480 秒任务时限。对照为当前 OCU 1.1.0 适配包。独立窗口事件判定结果如下(token 包含执行、摘要、压缩与最终核验):
|
||||
|
||||
| 指标 | OCU 1.1.0 | Cua 0.1.2 |
|
||||
| --- | --- | --- |
|
||||
| 全部动作与截图独立验收 | 0/3 | 3/3 |
|
||||
| 蓝色目标恰好命中一次、没有误点 | 3/3 | 3/3 |
|
||||
| 平均总 token | 683,748 | 504,954 |
|
||||
| 缓存输入 / 总输入 | 51.0% | 49.7% |
|
||||
| 平均任务结束/停止耗时 | 478.8 秒(超时) | 303.2 秒 |
|
||||
| 平均插件调用耗时合计 | 32.2 秒 | 5.2 秒 |
|
||||
|
||||
OCU 在滚动回退中漏传 `WindowBounds`,导致反复报坐标不可用;此为对照版本的具体缺陷,本次未中途修改。Cua 第一轮有一次旧 token 拒绝,刷新后成功。两者的小目标点击均成功,不能据此宣称 Cua 普遍点击更准。Cua 的 token 少 26.1% 是本轮观察值,包含旧插件失败重试开销,不是所有成功任务的固定节省。
|
||||
|
||||
Cua 第三轮虽然独立验收通过,ARTEMIS 最终核验却没有生成结论,保留 `inconclusive`;AI 裁定实际为 2 次通过、1 次未判定。三次截图均可解码、SHA256 与保存回执一致;最后一轮窗口录像全片解码通过,约 173 秒、5.8 MiB。测试结束后队列空闲,候选适配器、Driver、UIA helper 和测试窗口进程均已退出。
|
||||
|
||||
主项目 SDK 配置修复通过默认回归 2761 项(2 项跳过、8 项未选中),Ruff、质量基线和核心类型检查通过。候选继续仅在非默认测试配置启用;没有修改市场索引。录像范围的中途修复、模型兜底以及最终核验延迟均在本地报告中保留,不将本次开发测试包装为严格受控的性能实验。
|
||||
|
||||
未验证:真实业务跨端长程任务、Chrome/Brave 特定页面、多个 DPI/显示器、非 Windows 平台、DOM 路由。这些未测范围仍不作稳定性保证;市场替换决定见下一节。
|
||||
|
||||
## 市场替换版本 0.36.0
|
||||
|
||||
0.36.0 沿用 0.1.2 已测驱动和动作实现,仅更新发行版本、名称及文档,作为市场默认电脑控制插件取代 OCU。上游仍固定 0.36.0 预发布版,市场继续标注开发预览;上文的未测范围和 AI 裁定缺失情况仍然有效。旧 OCU 从 catalog.json 下架,既有源码、发行附件和任务证据保留。已安装 OCU 的用户应安装并启用 Cua,再停用 OCU;不会静默为其他用户授权新代码。
|
||||
|
||||
## 窗口发现修订 0.36.0+artemis.1
|
||||
|
||||
修复复跑任务在按应用名找 Brave 时填入 PID 0,随后误用无关前台应用 PID 的失败链路。原生窗口基准没有覆盖这一入口,原有 A/B 数据不作为浏览器发现能力的验证。
|
||||
|
||||
新增 Go 回归覆盖无效 PID 在调用驱动前拒绝、null 省略、冲突筛选保持原范围并提示同应用重查、可见窗口为空时保留目标检查最小化窗口,以及驱动错误不得伪装成空列表。`tests/cua_discovery_smoke.py` 对已打开的指定应用只读验证主机 schema、适配器、应用名发现、null PID、冲突 PID 恢复,以及实际截图和 UIA 树;不点击、不导航、不修改用户页面。
|
||||
|
||||
本修订实测通过:真实 Brave 应用名枚举、null PID、省略 PID、冲突 PID 后按建议恢复,并取得真实截图及 UIA 树。原生窗口表单、34 像素目标、滚动、PNG 附件校验和窗口录像回归通过;Go 测试、vet 与 Python Ruff 检查通过。未重跑原业务的整段移动端/PC布局验收,不将窗口发现修复等同于原任务通过。
|
||||
|
||||
## 输入参数修订 0.36.0+artemis.2
|
||||
|
||||
任务 f1928dc9 的窗口发现、截图和 UIA 树已经成功,但后续三次 Ctrl+T 同时传 element_token 与 x/y,上游在执行前拒绝,未完成导航。旧适配层把此错误归为不确定并刷新截图,产生新 token 后仍重复相同冲突。
|
||||
|
||||
修订在工具 schema 中分开聚焦和键盘输入:按键/打字只接受窗口当前焦点,set_value 必须指定控件,scroll 仅接受成对坐标。Go 回归覆盖真实冲突参数(包括 x=0,y=0)、明确 not_started/none 回执,以及在接触 HWND/驱动前拒绝,不自动重试写动作。
|
||||
|
||||
`tests/cua_browser_smoke.py` 新增独立 Brave profile 和本地表单,全部操作经过打包后的 Cua MCP,未使用 CDP/DOM 控制。实测通过 Ctrl+T 新建标签、Ctrl+L 地址栏导航、UIA set_value、聚焦后的中文 type_text、表单恰好提交一次并读回、实际滚动位移、底部按钮及 PNG SHA256。测试窗口退出,不操作用户原有标签页。
|
||||
|
||||
实测还发现 Chromium 后台点击可能沿用滚动前的控件位置;前台路径会解析可见位置,已验证滚动后点击成功,schema/Skill 明确要求浏览器使用 foreground。滚轮 amount/by 不是精确滚动像素或页数,测试按页面滚动事件决定是否继续。不能据小样本宣称所有浏览器、DPI、跨端业务都通过;原业务完整任务未重跑,移动端问题不在本修订范围。
|
||||
|
||||
## 截图坐标修订 0.36.0+artemis.3
|
||||
|
||||
对照实际任务中 2558×1538 原图、1280×770 后续截图,以及超界 (1700, 1000) 滚动仍被派发的回执,核对固定上游提交的 Windows ScrollTool:其前台路径直接 bitmap_to_screen,而点击/拖拽使用 capture 映射或 snapshot 缩放。修复仅对滚动补映射,不重复缩放点击和拖拽。
|
||||
|
||||
Go 确定性回归覆盖上述尺寸、原始尺寸无二次缩放、不同裁边、负窗口原点、非整数点、缺失/过期编号、非数值/越界、拖拽两端、窗口移动/缩放及原始截图读取失败。Go test、vet 与构建通过。本修订没有重跑付费模型业务任务,也未声称真实多显示器、多 DPI 或跨端成功率已验证;待使用新插件进行业务复测。
|
||||
|
||||
补充用户业务复测:2026-10-17 使用 `gpt-6-luna / high` 与 `0.36.0+artemis.3` 完成手机 Chrome、电脑 Brave 的混合页面探索,33 步、约 410 秒,执行状态为 completed,最终核验为 passed。记录确认移动端 Desktop site 未勾选、两端页底观察及追加滚动;报告明确保留 PC 中段局部未逐屏覆盖和文字证据冲突。此通过仅针对本次探索与报告要求,不表示页面所有区域无缺陷,也不外推多显示器、多 DPI 或整体成功率。
|
||||
|
||||
## 前台取证修订 0.36.0+artemis.4
|
||||
|
||||
2026-10-17 排查发现,前台点击后能短暂捕获网页正文,随后同窗口仅剩浏览器外壳和灰色正文,窗口录像也出现相同表现。固定上游在 foreground 输入后异步恢复原前台,ARTEMIS 的动作后截图则是独立调用;浏览器后台渲染和截图可能受到该时序影响。原业务没有重跑,保留 blocked,不能归因为网站缺陷。
|
||||
|
||||
适配修复使用上游既有 bring_to_front,在显式 foreground 输入前完成持久激活并核实准确 HWND。拒绝、仅提升层级、激活后丢失前台及传输异常都不发送业务输入;不确定输入不重发。只读观察不改变前台,记录 capture_focus,模型可显式恢复同一目标后重新观察。
|
||||
|
||||
Go 回归与 vet 通过。`tests/cua_browser_smoke.py --foreground-check` 在 ARTEMIS 的 desktop:local 资源锁内运行,使用临时 Brave profile、本地表单及另一个普通测试窗口代表原前台应用,不操作用户原有标签。38 次工具调用验证:从被遮挡窗口执行前台输入后,等待 5 秒(超过上游 3 秒恢复轮询)仍保持目标前台;人为切走后只读观察不抢焦点,bring_to_front 恢复并保存真实截图;导航后等待仍能读取正文;新标签、中文输入、表单恰好提交一次、实际滚动、页底点击和 PNG 校验均通过。
|
||||
|
||||
本次未复测原业务全流程、Codex 窗口本身或多屏 DPI;以普通前台窗口验证同一前台恢复机制,不将局部回归称为原任务通过。此修订随 ARTEMIS 2.0.0 配套发布,市场继续保留预览标识;仅更新主程序不会自动更新已安装插件。
|
||||
@@ -0,0 +1,90 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"math"
|
||||
)
|
||||
|
||||
// A frame belongs to one adapter session and one concrete window. Encoded
|
||||
// pixels, DWM frame dimensions and Windows logical pixels are distinct spaces.
|
||||
type pointerFrame struct {
|
||||
id string
|
||||
width, height int
|
||||
window obj
|
||||
}
|
||||
|
||||
func targetKey(args obj) string {
|
||||
return fmt.Sprintf("%d:%d", num(args["pid"]), num(args["window_id"]))
|
||||
}
|
||||
func prependFrameHint(r obj, f pointerFrame) {
|
||||
// Keep coordinate metadata before long trees so bounded model context cannot
|
||||
// silently drop the only capture ID while retaining the screenshot.
|
||||
r["content"] = append([]any{textBlock(fmt.Sprintf("Screenshot capture_id=%s, size=%dx%d; pixel x=0..%d, y=0..%d. Use this new ID for click_at/scroll/drag.", f.id, f.width, f.height, f.width-1, f.height-1))}, items(r["content"])...)
|
||||
}
|
||||
func isPointerTool(name string) bool { return name == "click_at" || name == "drag" || name == "scroll" }
|
||||
func copyArgs(args obj) obj {
|
||||
out := obj{}
|
||||
for k, v := range args {
|
||||
out[k] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
func capturedFrame(r, src obj) (pointerFrame, bool) {
|
||||
w, h := imageSize(r)
|
||||
id := str(object(r["structuredContent"])["capture_id"])
|
||||
if src == nil || id == "" || w <= 0 || h <= 0 {
|
||||
return pointerFrame{}, false
|
||||
}
|
||||
return pointerFrame{id, w, h, copyArgs(object(src["window"]))}, true
|
||||
}
|
||||
func coordinate(v any) (float64, bool) {
|
||||
var n float64
|
||||
switch x := v.(type) {
|
||||
case float64:
|
||||
n = x
|
||||
case int:
|
||||
n = float64(x)
|
||||
case int64:
|
||||
n = float64(x)
|
||||
default:
|
||||
return 0, false
|
||||
}
|
||||
return n, !math.IsNaN(n) && !math.IsInf(n, 0)
|
||||
}
|
||||
func (f pointerFrame) validate(name string, args obj) error {
|
||||
if f.id == "" || str(args["capture_id"]) != f.id {
|
||||
return errors.New("capture_id is missing or stale; use the latest screenshot for this window")
|
||||
}
|
||||
pairs := [][2]string{{"x", "y"}}
|
||||
if name == "drag" {
|
||||
pairs = [][2]string{{"from_x", "from_y"}, {"to_x", "to_y"}}
|
||||
}
|
||||
for _, pair := range pairs {
|
||||
x, xok := coordinate(args[pair[0]])
|
||||
y, yok := coordinate(args[pair[1]])
|
||||
if !xok || !yok || x < 0 || y < 0 || x > float64(f.width-1) || y > float64(f.height-1) {
|
||||
return fmt.Errorf("%s/%s must be inside screenshot %dx%d (0..%d, 0..%d); no screen or Android normalized coordinates", pair[0], pair[1], f.width, f.height, f.width-1, f.height-1)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func sameWindow(a, b obj) bool {
|
||||
for _, k := range []string{"left", "top", "width", "height"} {
|
||||
if num(a[k]) != num(b[k]) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return num(a["width"]) > 0 && num(a["height"]) > 0
|
||||
}
|
||||
|
||||
// Only scroll needs pre-scaling in the pinned driver. click_at is capture-bound
|
||||
// upstream and drag uses the driver's snapshot ratio; never scale either twice.
|
||||
// v0.38.0 scroll and drag use their latest window screenshot context.
|
||||
// Admission stays host-bound to the exact capture; the upstream schema does
|
||||
// not accept capture_id for these two tools. Do not rescale x/y here.
|
||||
func withoutCaptureBinding(args obj) obj {
|
||||
out := copyArgs(args)
|
||||
delete(out, "capture_id")
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"math"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func fixtureFrame(w, h int) pointerFrame {
|
||||
return pointerFrame{"current", w, h, obj{"left": -2560, "top": 120, "width": 2560, "height": 1540}}
|
||||
}
|
||||
|
||||
func TestScrollPixelsReachUpstreamWithoutDoubleScaling(t *testing.T) {
|
||||
for _, size := range [][2]int{{1280, 770}, {2558, 1538}, {2560, 1540}} {
|
||||
frame := fixtureFrame(size[0], size[1])
|
||||
args := obj{"pid": 1, "window_id": 2, "capture_id": "current", "x": 900.5, "y": 600.25, "direction": "down"}
|
||||
if err := frame.validate("scroll", args); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := withoutCaptureBinding(args)
|
||||
if got["x"] != 900.5 || got["y"] != 600.25 || got["pid"] != 1 || got["window_id"] != 2 {
|
||||
t.Fatal(got)
|
||||
}
|
||||
if _, ok := got["capture_id"]; ok {
|
||||
t.Fatal("upstream scroll rejects capture_id")
|
||||
}
|
||||
if args["capture_id"] != "current" {
|
||||
t.Fatal("input evidence mutated")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestInvalidScrollCoordinatesAreRejectedBeforeDispatch(t *testing.T) {
|
||||
frame := fixtureFrame(1280, 770)
|
||||
for _, bad := range []obj{
|
||||
{"capture_id": "current", "x": 1700, "y": 1000},
|
||||
{"capture_id": "old", "x": 900, "y": 600},
|
||||
{"x": 900, "y": 600},
|
||||
{"capture_id": "current", "x": -1, "y": 0},
|
||||
{"capture_id": "current", "x": 1280, "y": 0},
|
||||
{"capture_id": "current", "x": 0, "y": 770},
|
||||
{"capture_id": "current", "x": "900", "y": 600},
|
||||
{"capture_id": "current", "x": math.NaN(), "y": 600},
|
||||
{"capture_id": "current", "x": 900, "y": math.Inf(1)},
|
||||
} {
|
||||
if frame.validate("scroll", bad) == nil {
|
||||
t.Fatalf("accepted %v", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMovedOrResizedWindowInvalidatesFrame(t *testing.T) {
|
||||
frame := fixtureFrame(1280, 770)
|
||||
for _, key := range []string{"left", "top", "width", "height"} {
|
||||
changed := copyArgs(frame.window)
|
||||
changed[key] = num(changed[key]) + 1
|
||||
if sameWindow(frame.window, changed) {
|
||||
t.Fatal("accepted changed " + key)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestClickAndBothDragEndpointsUseLatestFrame(t *testing.T) {
|
||||
f := fixtureFrame(1280, 770)
|
||||
click := obj{"capture_id": "current", "x": 1279, "y": 769}
|
||||
if err := f.validate("click_at", click); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if click["x"] != 1279 || click["y"] != 769 {
|
||||
t.Fatal("click scaled twice")
|
||||
}
|
||||
drag := obj{"capture_id": "current", "from_x": 0, "from_y": 0, "to_x": 1279, "to_y": 769}
|
||||
if err := f.validate("drag", drag); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, key := range []string{"from_x", "from_y", "to_x", "to_y"} {
|
||||
bad := copyArgs(drag)
|
||||
bad[key] = 3000
|
||||
if f.validate("drag", bad) == nil {
|
||||
t.Fatal("accepted out of bounds " + key)
|
||||
}
|
||||
}
|
||||
drag["capture_id"] = "old"
|
||||
if f.validate("drag", drag) == nil {
|
||||
t.Fatal("accepted stale drag")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAllPixelSchemasRequireCapture(t *testing.T) {
|
||||
a := newAdapter(nil)
|
||||
for _, name := range []string{"click_at", "scroll", "drag"} {
|
||||
found := false
|
||||
for _, key := range items(a.schemas[name]["required"]) {
|
||||
if key == "capture_id" {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatal(name + " lacks capture binding")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
package main
|
||||
|
||||
// Use Cua's existing persistent activation before explicit foreground input.
|
||||
// Cua then snapshots the target itself as its "previous foreground", so its
|
||||
// asynchronous restore cannot race ARTEMIS's separately requested observation.
|
||||
// Read-only calls and background delivery never activate or restore a window.
|
||||
func callWithForeground(call func(string, obj) (obj, error), name string, args obj, current func() int64) (obj, error) {
|
||||
if name != "bring_to_front" && args["delivery_mode"] != "foreground" {
|
||||
return call(name, args)
|
||||
}
|
||||
target := obj{"pid": args["pid"], "window_id": args["window_id"]}
|
||||
focus, err := call("bring_to_front", target)
|
||||
if err != nil {
|
||||
// Only activation was attempted. Do not send (or replay) the business
|
||||
// action when the activation result is unknown.
|
||||
return failure("Foreground activation could not be confirmed; requested input was not sent: "+err.Error(), "not_started", "possible", "foreground_unavailable"), nil
|
||||
}
|
||||
activation := object(focus["structuredContent"])
|
||||
actual := current()
|
||||
activation["verified_foreground_window_id"] = actual
|
||||
activation["policy"] = "keep_target_foreground"
|
||||
if focus["isError"] == true || activation["landed_on_target"] != true || actual != num(args["window_id"]) {
|
||||
r := failure("Target window did not retain foreground; requested input was not sent. Inspect the window before retrying.", "not_started", "possible", "foreground_unavailable")
|
||||
object(r["structuredContent"])["foreground_activation"] = activation
|
||||
return r, nil
|
||||
}
|
||||
if name == "bring_to_front" {
|
||||
activation["artemis"] = receipt("completed", "confirmed", "")
|
||||
focus["structuredContent"] = activation
|
||||
return focus, nil
|
||||
}
|
||||
r, err := call(name, args)
|
||||
if err == nil {
|
||||
s := object(r["structuredContent"])
|
||||
s["foreground_activation"] = activation
|
||||
r["structuredContent"] = s
|
||||
}
|
||||
return r, err
|
||||
}
|
||||
|
||||
func annotateCaptureFocus(r obj, target, before, after int64) {
|
||||
s := object(r["structuredContent"])
|
||||
s["capture_focus"] = obj{
|
||||
"target_window_id": target,
|
||||
"foreground_window_id_before": before,
|
||||
"foreground_window_id_after": after,
|
||||
"target_foreground_before": before == target,
|
||||
"target_foreground_after": after == target,
|
||||
"changed_during_capture": before != after,
|
||||
}
|
||||
r["structuredContent"] = s
|
||||
if before != target || after != target {
|
||||
r["content"] = append(items(r["content"]), textBlock("Capture was not continuously foreground. If the page body is blank or missing from the tree, use bring_to_front for this same pid/window_id, then observe again; do not infer a website failure or repeat a business action."))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestForegroundActivatesBeforeInputAndDoesNotRestore(t *testing.T) {
|
||||
var calls []string
|
||||
foreground := int64(10) // The user's foreground app, e.g. Codex.
|
||||
call := func(name string, args obj) (obj, error) {
|
||||
calls = append(calls, name)
|
||||
if name == "bring_to_front" {
|
||||
if len(args) != 2 || num(args["window_id"]) != 20 {
|
||||
t.Fatal("activation changed the target or forwarded input arguments")
|
||||
}
|
||||
foreground = 20
|
||||
return obj{"structuredContent": obj{"landed_on_target": true}}, nil
|
||||
}
|
||||
// Simulate upstream's activate/act/restore: it now saves the target,
|
||||
// not the user's previous app, as its restore destination.
|
||||
previous := foreground
|
||||
foreground = 20
|
||||
foreground = previous
|
||||
return obj{"structuredContent": obj{"effect": "unverifiable"}}, nil
|
||||
}
|
||||
r, err := callWithForeground(call, "press_key", obj{"pid": 2, "window_id": 20, "key": "t", "delivery_mode": "foreground"}, func() int64 { return foreground })
|
||||
if err != nil || r["isError"] == true || foreground != 20 || !reflect.DeepEqual(calls, []string{"bring_to_front", "press_key"}) {
|
||||
t.Fatalf("foreground handoff failed: %v %v %v %v", calls, foreground, r, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnconfirmedActivationNeverSendsInput(t *testing.T) {
|
||||
for _, mode := range []string{"refused", "raised_only", "lost_focus", "transport"} {
|
||||
t.Run(mode, func(t *testing.T) {
|
||||
calls := 0
|
||||
r, err := callWithForeground(func(name string, _ obj) (obj, error) {
|
||||
calls++
|
||||
if name != "bring_to_front" {
|
||||
t.Fatal("input sent without focus")
|
||||
}
|
||||
if mode == "transport" {
|
||||
return nil, errors.New("disconnected")
|
||||
}
|
||||
return obj{"isError": mode == "refused", "structuredContent": obj{"raised": true, "landed_on_target": mode != "raised_only"}}, nil
|
||||
}, "click", obj{"pid": 2, "window_id": 20, "delivery_mode": "foreground"}, func() int64 {
|
||||
if mode == "lost_focus" {
|
||||
return 10
|
||||
}
|
||||
return 20
|
||||
})
|
||||
rec := object(object(r["structuredContent"])["artemis"])
|
||||
if err != nil || calls != 1 || r["isError"] != true || rec["execution"] != "not_started" || rec["retryable"] != false {
|
||||
t.Fatalf("unsafe activation handling: %v %v", r, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackgroundAndObservationsNeverActivate(t *testing.T) {
|
||||
for _, name := range []string{"get_window_state", "verify_state", "save_screenshot", "click"} {
|
||||
var calls []string
|
||||
_, err := callWithForeground(func(tool string, _ obj) (obj, error) {
|
||||
calls = append(calls, tool)
|
||||
return obj{}, nil
|
||||
}, name, obj{"pid": 2, "window_id": 20}, func() int64 { t.Fatal("unexpected focus probe"); return 0 })
|
||||
if err != nil || !reflect.DeepEqual(calls, []string{name}) {
|
||||
t.Fatal(calls, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestUncertainInputIsNotReplayed(t *testing.T) {
|
||||
calls := 0
|
||||
_, err := callWithForeground(func(name string, _ obj) (obj, error) {
|
||||
calls++
|
||||
if name == "bring_to_front" {
|
||||
return obj{"structuredContent": obj{"landed_on_target": true}}, nil
|
||||
}
|
||||
return nil, errors.New("input reply lost")
|
||||
}, "type_text", obj{"pid": 2, "window_id": 20, "delivery_mode": "foreground"}, func() int64 { return 20 })
|
||||
if calls != 2 || err == nil {
|
||||
t.Fatal(calls, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCaptureRecordsFocusWithoutHidingBackgroundEvidence(t *testing.T) {
|
||||
r := obj{"content": []any{obj{"type": "image", "data": "original"}}}
|
||||
annotateCaptureFocus(r, 20, 20, 10)
|
||||
s := object(object(r["structuredContent"])["capture_focus"])
|
||||
if s["changed_during_capture"] != true || s["target_foreground_after"] != false || len(items(r["content"])) != 2 || object(items(r["content"])[0])["data"] != "original" {
|
||||
t.Fatal(r)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
module artemis-cua-adapter
|
||||
|
||||
go 1.22
|
||||
@@ -0,0 +1,539 @@
|
||||
// ARTEMIS adapter for the pinned MIT Cua Driver. No model, scheduler, or recorder.
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
_ "embed"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"image"
|
||||
_ "image/jpeg"
|
||||
_ "image/png"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
//go:embed tools.json
|
||||
var toolsJSON []byte
|
||||
|
||||
const adapterVersion = "0.38.0+artemis.1"
|
||||
|
||||
type obj = map[string]any
|
||||
type request struct {
|
||||
JSONRPC string `json:"jsonrpc"`
|
||||
ID json.RawMessage `json:"id"`
|
||||
Method string `json:"method"`
|
||||
Params obj `json:"params"`
|
||||
}
|
||||
type peer struct {
|
||||
cmd *exec.Cmd
|
||||
input io.WriteCloser
|
||||
lines chan []byte
|
||||
next int
|
||||
mu sync.Mutex
|
||||
dead bool
|
||||
job uintptr
|
||||
}
|
||||
|
||||
func object(v any) obj {
|
||||
m, _ := v.(map[string]any)
|
||||
if m == nil {
|
||||
return obj{}
|
||||
}
|
||||
return m
|
||||
}
|
||||
func str(v any) string { s, _ := v.(string); return s }
|
||||
func num(v any) int64 {
|
||||
switch n := v.(type) {
|
||||
case float64:
|
||||
return int64(n)
|
||||
case int64:
|
||||
return n
|
||||
case int:
|
||||
return int64(n)
|
||||
}
|
||||
return 0
|
||||
}
|
||||
func items(v any) []any { a, _ := v.([]any); return a }
|
||||
func textBlock(s string) obj { return obj{"type": "text", "text": s} }
|
||||
func receipt(execution, side, code string) obj {
|
||||
return obj{"version": 1, "execution": execution, "side_effect": side, "retryable": false, "error_code": code}
|
||||
}
|
||||
func failure(msg, execution, side, code string) obj {
|
||||
return obj{"isError": true, "content": []any{textBlock(msg)}, "structuredContent": obj{"artemis": receipt(execution, side, code)}}
|
||||
}
|
||||
func writeTool(name string) bool {
|
||||
return name != "list_windows" && name != "get_window_state" && name != "verify_state"
|
||||
}
|
||||
|
||||
func startPeer(exe string) (*peer, error) {
|
||||
c := exec.Command(exe, "mcp", "--direct")
|
||||
c.Env = append(os.Environ(), "DO_NOT_TRACK=1", "CUA_TELEMETRY=0", "CUA_DRIVER_RS_TELEMETRY_ENABLED=false")
|
||||
configureProcess(c)
|
||||
in, err := c.StdinPipe()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out, err := c.StdoutPipe()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
c.Stderr = os.Stderr
|
||||
if err = c.Start(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
job, err := ownProcess(c.Process.Pid)
|
||||
if err != nil {
|
||||
_ = c.Process.Kill()
|
||||
_ = c.Wait()
|
||||
return nil, err
|
||||
}
|
||||
p := &peer{cmd: c, input: in, lines: make(chan []byte, 32), job: job}
|
||||
go func() {
|
||||
defer close(p.lines)
|
||||
s := bufio.NewScanner(out)
|
||||
s.Buffer(make([]byte, 65536), 32<<20)
|
||||
for s.Scan() {
|
||||
p.lines <- append([]byte(nil), s.Bytes()...)
|
||||
}
|
||||
}()
|
||||
_, err = p.call("initialize", obj{"protocolVersion": "2024-11-05", "clientInfo": obj{"name": "artemis-cua-adapter", "version": adapterVersion}, "capabilities": obj{}})
|
||||
if err != nil {
|
||||
p.stop()
|
||||
return nil, err
|
||||
}
|
||||
err = json.NewEncoder(in).Encode(obj{"jsonrpc": "2.0", "method": "notifications/initialized"})
|
||||
if err != nil {
|
||||
p.stop()
|
||||
return nil, err
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
func (p *peer) stop() {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if p.dead {
|
||||
return
|
||||
}
|
||||
p.dead = true
|
||||
_ = p.input.Close()
|
||||
closeJob(p.job)
|
||||
_ = p.cmd.Process.Kill()
|
||||
_ = p.cmd.Wait()
|
||||
}
|
||||
func (p *peer) call(method string, params obj) (obj, error) {
|
||||
p.mu.Lock()
|
||||
dead := p.dead
|
||||
p.mu.Unlock()
|
||||
if dead {
|
||||
return nil, errors.New("driver closed")
|
||||
}
|
||||
p.next++
|
||||
id := p.next
|
||||
if err := json.NewEncoder(p.input).Encode(obj{"jsonrpc": "2.0", "id": id, "method": method, "params": params}); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
timer := time.NewTimer(45 * time.Second)
|
||||
defer timer.Stop()
|
||||
for {
|
||||
select {
|
||||
case raw, ok := <-p.lines:
|
||||
if !ok {
|
||||
return nil, errors.New("driver disconnected")
|
||||
}
|
||||
var r obj
|
||||
if json.Unmarshal(raw, &r) != nil {
|
||||
continue
|
||||
}
|
||||
if num(r["id"]) != int64(id) {
|
||||
continue
|
||||
}
|
||||
if r["error"] != nil {
|
||||
return nil, errors.New("driver protocol error")
|
||||
}
|
||||
return object(r["result"]), nil
|
||||
case <-timer.C:
|
||||
p.stop()
|
||||
return nil, errors.New("driver response timed out; action not replayed")
|
||||
}
|
||||
}
|
||||
}
|
||||
func (p *peer) tool(name string, args obj) (obj, error) {
|
||||
return p.call("tools/call", obj{"name": name, "arguments": args})
|
||||
}
|
||||
|
||||
type adapter struct {
|
||||
peer *peer
|
||||
tools []any
|
||||
schemas map[string]obj
|
||||
captures map[string]pointerFrame
|
||||
}
|
||||
|
||||
func newAdapter(p *peer) *adapter {
|
||||
a := &adapter{peer: p, schemas: map[string]obj{}, captures: map[string]pointerFrame{}}
|
||||
_ = json.Unmarshal(toolsJSON, &a.tools)
|
||||
for _, v := range a.tools {
|
||||
t := object(v)
|
||||
a.schemas[str(t["name"])] = object(t["inputSchema"])
|
||||
}
|
||||
return a
|
||||
}
|
||||
func (a *adapter) validate(name string, args obj) error {
|
||||
schema, ok := a.schemas[name]
|
||||
if !ok {
|
||||
return errors.New("tool is not exposed")
|
||||
}
|
||||
if name == "press_key" || name == "type_text" {
|
||||
for _, key := range []string{"element_token", "x", "y"} {
|
||||
if _, supplied := args[key]; supplied {
|
||||
return errors.New(name + " uses the window's current focus; omit element_token/x/y. Focus with click/click_at first, or use set_value for a UIA field")
|
||||
}
|
||||
}
|
||||
}
|
||||
props := object(schema["properties"])
|
||||
for key := range args {
|
||||
if _, ok := props[key]; !ok {
|
||||
return fmt.Errorf("unsupported argument: %s", key)
|
||||
}
|
||||
}
|
||||
for _, key := range items(schema["required"]) {
|
||||
if _, ok := args[str(key)]; !ok {
|
||||
return fmt.Errorf("missing argument: %s", str(key))
|
||||
}
|
||||
}
|
||||
if name == "list_windows" {
|
||||
_, err := windowQuery(args)
|
||||
return err
|
||||
}
|
||||
if num(args["pid"]) <= 0 || num(args["window_id"]) <= 0 {
|
||||
return errors.New("a concrete pid and window_id are required")
|
||||
}
|
||||
if name == "set_value" && strings.TrimSpace(str(args["element_token"])) == "" {
|
||||
return errors.New("set_value requires the latest element_token; use type_text for the current focus")
|
||||
}
|
||||
if name == "save_screenshot" && !validFilename(str(args["filename"])) {
|
||||
return errors.New("filename must be a new plain PNG filename")
|
||||
}
|
||||
if name == "get_window_state" && (args["include_screenshot"] == false || args["include_accessibility_tree"] == false) {
|
||||
return errors.New("candidate observations require both screenshot and accessibility tree")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func validFilename(s string) bool {
|
||||
return s != "" && len(s) <= 160 && strings.EqualFold(filepath.Ext(s), ".png") && !strings.ContainsAny(s, "/\\:<>\"|?*\x00") && s != "." && s != ".." && !strings.HasSuffix(s, ".") && !strings.HasSuffix(s, " ")
|
||||
}
|
||||
func actionReceipt(r obj, write bool) obj {
|
||||
if !write {
|
||||
if r["isError"] == true {
|
||||
return receipt("not_started", "none", "observation_failed")
|
||||
}
|
||||
return receipt("completed", "none", "")
|
||||
}
|
||||
s := object(r["structuredContent"])
|
||||
// v0.36.0 class-level preflight refusal has no effect field. It occurs
|
||||
// before input injection. Do not classify actuator failures with a cause
|
||||
// as safe-to-repeat; retain their uncertain execution below.
|
||||
if s["code"] == "background_unavailable" && s["cause"] == nil && s["event_kind"] != nil && r["isError"] == true {
|
||||
return receipt("not_started", "none", "background_unavailable")
|
||||
}
|
||||
switch str(s["effect"]) {
|
||||
case "confirmed":
|
||||
return receipt("completed", "confirmed", "")
|
||||
case "partial":
|
||||
return receipt("partial", "confirmed", "partial_delivery")
|
||||
case "refused":
|
||||
return receipt("not_started", "none", "driver_refused")
|
||||
case "unverifiable", "suspected_noop":
|
||||
return receipt("completed", "possible", str(s["effect"]))
|
||||
default:
|
||||
return receipt("unknown", "possible", "unclassified_result")
|
||||
}
|
||||
}
|
||||
func imageSize(r obj) (int, int) {
|
||||
for _, v := range items(r["content"]) {
|
||||
c := object(v)
|
||||
if c["type"] == "image" {
|
||||
raw, err := base64.StdEncoding.DecodeString(str(c["data"]))
|
||||
if err == nil {
|
||||
cfg, _, err := image.DecodeConfig(bytes.NewReader(raw))
|
||||
if err == nil {
|
||||
return cfg.Width, cfg.Height
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return 0, 0
|
||||
}
|
||||
func source(r obj, args obj, started float64) obj {
|
||||
s := object(r["structuredContent"])
|
||||
w, h := imageSize(r)
|
||||
if r["isError"] == true || w <= 0 || h <= 0 || num(s["pid"]) != num(args["pid"]) || num(s["window_id"]) != num(args["window_id"]) {
|
||||
return nil
|
||||
}
|
||||
bounds := object(s["window_bounds"])
|
||||
rect := obj{"left": num(bounds["x"]), "top": num(bounds["y"]), "width": num(bounds["width"]), "height": num(bounds["height"])}
|
||||
screen, ok := verifyWindow(num(args["pid"]), num(args["window_id"]), rect)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
return obj{"kind": "computer", "target_id": fmt.Sprint(num(args["window_id"])), "label": s["window_title"], "captured_at": started, "width": w, "height": h, "window": rect, "screen": screen}
|
||||
}
|
||||
func (a *adapter) observe(args obj) (obj, obj, error) {
|
||||
delete(a.captures, targetKey(args))
|
||||
started := float64(time.Now().UnixMilli()) / 1000
|
||||
before := foregroundWindow()
|
||||
focusBefore := keyboardFocus()
|
||||
r, err := a.peer.tool("get_window_state", args)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
annotateCaptureFocus(r, num(args["window_id"]), before, foregroundWindow())
|
||||
object(r["structuredContent"])["keyboard_focus"] = obj{"before": focusBefore, "after": keyboardFocus()}
|
||||
src := source(r, args, started)
|
||||
if frame, ok := capturedFrame(r, src); ok {
|
||||
a.captures[targetKey(args)] = frame
|
||||
prependFrameHint(r, frame)
|
||||
}
|
||||
return r, src, nil
|
||||
}
|
||||
func observationArgs(args obj) obj {
|
||||
return obj{"pid": args["pid"], "window_id": args["window_id"], "max_image_dimension": 1280, "max_elements": 250, "tree_format": "markdown"}
|
||||
}
|
||||
func (a *adapter) handle(name string, args obj) obj {
|
||||
if err := a.validate(name, args); err != nil {
|
||||
return failure(err.Error(), "not_started", "none", "invalid_input")
|
||||
}
|
||||
if name != "list_windows" {
|
||||
if !windowOwned(num(args["pid"]), num(args["window_id"])) {
|
||||
return failure("window is gone or belongs to a different process", "not_started", "none", "invalid_target")
|
||||
}
|
||||
}
|
||||
if name == "save_screenshot" {
|
||||
return a.save(args)
|
||||
}
|
||||
var r, src obj
|
||||
var err error
|
||||
if name == "get_window_state" {
|
||||
for k, v := range observationArgs(args) {
|
||||
if _, ok := args[k]; !ok {
|
||||
args[k] = v
|
||||
}
|
||||
}
|
||||
r, src, err = a.observe(args)
|
||||
} else if name == "list_windows" {
|
||||
query, _ := windowQuery(args) // Already validated above.
|
||||
r, err = a.peer.tool(name, query)
|
||||
if err == nil {
|
||||
r = supplementShellWindows(r, query, nativeShellWindows())
|
||||
r = windowListing(r, query)
|
||||
}
|
||||
} else {
|
||||
if isPointerTool(name) {
|
||||
frame, ok := a.captures[targetKey(args)]
|
||||
if !ok {
|
||||
return failure("No current screenshot; get_window_state before a pixel action", "not_started", "none", "invalid_coordinates")
|
||||
}
|
||||
if err := frame.validate(name, args); err != nil {
|
||||
return failure(err.Error(), "not_started", "none", "invalid_coordinates")
|
||||
}
|
||||
if _, ok := verifyWindow(num(args["pid"]), num(args["window_id"]), frame.window); !ok {
|
||||
delete(a.captures, targetKey(args))
|
||||
return failure("Window moved or resized; get_window_state again", "not_started", "none", "stale_capture")
|
||||
}
|
||||
if name == "scroll" || name == "drag" {
|
||||
// v0.38.0 maps screenshot pixels itself. Keep x/y unchanged.
|
||||
args = withoutCaptureBinding(args)
|
||||
}
|
||||
}
|
||||
upstream := name
|
||||
if name == "click_at" {
|
||||
upstream = "click"
|
||||
}
|
||||
if name == "verify_state" {
|
||||
delete(a.captures, targetKey(args))
|
||||
}
|
||||
if writeTool(name) {
|
||||
delete(a.captures, targetKey(args))
|
||||
}
|
||||
r, err = callWithForeground(a.peer.tool, upstream, args, foregroundWindow)
|
||||
}
|
||||
if err != nil {
|
||||
a.peer.stop()
|
||||
if writeTool(name) {
|
||||
return failure(err.Error(), "unknown", "possible", "transport_lost")
|
||||
}
|
||||
return failure(err.Error(), "not_started", "none", "transport_lost")
|
||||
}
|
||||
structured := object(r["structuredContent"])
|
||||
rec := object(structured["artemis"])
|
||||
if len(rec) == 0 {
|
||||
rec = actionReceipt(r, writeTool(name))
|
||||
}
|
||||
if writeTool(name) && rec["execution"] != "not_started" {
|
||||
// Observe once after delivery; NEVER repeat an uncertain write.
|
||||
focus := keyboardFocus()
|
||||
next, reason := postActionTarget(args, nativeWindowInfo(num(args["window_id"])), focus, nativeShellWindows())
|
||||
structured["post_action_focus"] = focus
|
||||
structured["observation_target"] = obj{"requested": obj{"pid": args["pid"], "window_id": args["window_id"]},
|
||||
"observed": obj{"pid": next["pid"], "window_id": next["window_id"]}, "reason": reason}
|
||||
observation, origin, e := a.observe(next)
|
||||
if e == nil && origin != nil {
|
||||
src = origin
|
||||
structured["observation"] = observation["structuredContent"]
|
||||
r["content"] = append(items(r["content"]), items(observation["content"])...)
|
||||
} else {
|
||||
structured["observation_error"] = "Post-action evidence unavailable; inspect before another action."
|
||||
if e != nil {
|
||||
structured["observation_error"] = e.Error()
|
||||
} else if observation != nil {
|
||||
var errors []string
|
||||
if observation["isError"] == true {
|
||||
for _, item := range items(observation["content"]) {
|
||||
block := object(item)
|
||||
if block["type"] == "text" {
|
||||
text := []rune(str(block["text"]))
|
||||
if len(text) > 2000 {
|
||||
text = text[:2000]
|
||||
}
|
||||
errors = append(errors, string(text))
|
||||
}
|
||||
}
|
||||
}
|
||||
// Failed identity/geometry validation must never forward its image
|
||||
// or full tree as task evidence, even inside diagnostics.
|
||||
structured["observation_diagnostics"] = obj{"driver_errors": errors,
|
||||
"native_target": nativeWindowInfo(num(next["window_id"])), "isError": observation["isError"],
|
||||
"image_size": func() []int { w, h := imageSize(observation); return []int{w, h} }()}
|
||||
}
|
||||
}
|
||||
}
|
||||
if name == "verify_state" { // Read-only verification may include a screenshot; it is not a task verdict.
|
||||
src = source(r, args, float64(time.Now().UnixMilli())/1000)
|
||||
if frame, ok := capturedFrame(r, src); ok {
|
||||
a.captures[targetKey(args)] = frame
|
||||
prependFrameHint(r, frame)
|
||||
}
|
||||
}
|
||||
if src != nil {
|
||||
rec["source"] = src
|
||||
} else if name == "get_window_state" && r["isError"] != true {
|
||||
return failure("capture target or geometry could not be validated", "not_started", "none", "invalid_capture")
|
||||
}
|
||||
if structured["effect"] == "refused" {
|
||||
r["isError"] = true
|
||||
}
|
||||
structured["artemis"] = rec
|
||||
r["structuredContent"] = structured
|
||||
return r
|
||||
}
|
||||
func (a *adapter) save(args obj) obj {
|
||||
root := os.Getenv("ARTEMIS_ARTIFACTS_DIR")
|
||||
if root == "" {
|
||||
return failure("task artifact directory unavailable", "not_started", "none", "missing_artifacts")
|
||||
}
|
||||
r, src, err := a.observe(observationArgs(args))
|
||||
if err != nil || src == nil {
|
||||
return failure("fresh window capture failed", "not_started", "none", "capture_failed")
|
||||
}
|
||||
var png []byte
|
||||
for _, v := range items(r["content"]) {
|
||||
b := object(v)
|
||||
if b["type"] == "image" && b["mimeType"] == "image/png" {
|
||||
png, _ = base64.StdEncoding.DecodeString(str(b["data"]))
|
||||
break
|
||||
}
|
||||
}
|
||||
if len(png) == 0 {
|
||||
return failure("expected PNG capture", "not_started", "none", "capture_format")
|
||||
}
|
||||
if err = os.MkdirAll(root, 0700); err != nil {
|
||||
return failure("artifact directory unavailable", "not_started", "none", "file_error")
|
||||
}
|
||||
path := filepath.Join(root, str(args["filename"]))
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600)
|
||||
if err != nil {
|
||||
return failure("file exists or cannot be created", "not_started", "none", "file_error")
|
||||
}
|
||||
n, writeErr := f.Write(png)
|
||||
closeErr := f.Close()
|
||||
actual, readErr := os.ReadFile(path)
|
||||
if writeErr != nil || closeErr != nil || readErr != nil || n != len(png) || !bytes.Equal(actual, png) {
|
||||
return failure("artifact write could not be verified", "partial", "possible", "file_error")
|
||||
}
|
||||
rec := receipt("completed", "confirmed", "")
|
||||
rec["source"] = src
|
||||
rec["attachments"] = []any{obj{"name": str(args["filename"]), "sha256": fmt.Sprintf("%x", sha256.Sum256(actual)), "media_type": "image/png"}}
|
||||
object(r["structuredContent"])["artemis"] = rec
|
||||
r["content"] = append(items(r["content"]), textBlock("Saved verified task attachment: "+str(args["filename"])))
|
||||
return r
|
||||
}
|
||||
func serve(a *adapter, in io.Reader, out io.Writer) {
|
||||
input := make(chan request)
|
||||
go func() {
|
||||
defer close(input)
|
||||
defer a.peer.stop()
|
||||
s := bufio.NewScanner(in)
|
||||
s.Buffer(make([]byte, 65536), 2<<20)
|
||||
for s.Scan() {
|
||||
var q request
|
||||
if json.Unmarshal(s.Bytes(), &q) == nil {
|
||||
if q.Method == "notifications/cancelled" {
|
||||
a.peer.stop()
|
||||
continue
|
||||
}
|
||||
input <- q
|
||||
}
|
||||
}
|
||||
}()
|
||||
enc := json.NewEncoder(out)
|
||||
for q := range input {
|
||||
if len(q.ID) == 0 {
|
||||
continue
|
||||
}
|
||||
var result any
|
||||
var rpcErr any
|
||||
switch q.Method {
|
||||
case "initialize":
|
||||
result = obj{"protocolVersion": "2024-11-05", "capabilities": obj{"tools": obj{}}, "serverInfo": obj{"name": "artemis-cua-driver", "version": adapterVersion}}
|
||||
case "ping":
|
||||
result = obj{}
|
||||
case "tools/list":
|
||||
result = obj{"tools": a.tools}
|
||||
case "tools/call":
|
||||
result = a.handle(str(q.Params["name"]), object(q.Params["arguments"]))
|
||||
default:
|
||||
rpcErr = obj{"code": -32601, "message": "method not found"}
|
||||
}
|
||||
response := obj{"jsonrpc": "2.0", "id": q.ID}
|
||||
if rpcErr != nil {
|
||||
response["error"] = rpcErr
|
||||
} else {
|
||||
response["result"] = result
|
||||
}
|
||||
if enc.Encode(response) != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
func main() {
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
os.Exit(1)
|
||||
}
|
||||
p, err := startPeer(filepath.Join(filepath.Dir(exe), "cua-driver.exe"))
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "Cua startup failed:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer p.stop()
|
||||
serve(newAdapter(p), os.Stdin, os.Stdout)
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
package main
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestUnknownAndRefusedAreNotConfirmed(t *testing.T) {
|
||||
cases := []struct{ effect, execution, side string }{{"confirmed", "completed", "confirmed"}, {"partial", "partial", "confirmed"}, {"unverifiable", "completed", "possible"}, {"suspected_noop", "completed", "possible"}, {"refused", "not_started", "none"}, {"", "unknown", "possible"}}
|
||||
for _, c := range cases {
|
||||
r := actionReceipt(obj{"structuredContent": obj{"effect": c.effect}}, true)
|
||||
if r["execution"] != c.execution || r["side_effect"] != c.side || r["retryable"] != false {
|
||||
t.Fatalf("%s: %v", c.effect, r)
|
||||
}
|
||||
}
|
||||
}
|
||||
func TestAttachmentCannotEscape(t *testing.T) {
|
||||
for _, s := range []string{"../out.png", "..\\out.png", "C:out.png", "a.png:stream", "", "a.png ", "a.jpg"} {
|
||||
if validFilename(s) {
|
||||
t.Fatal(s)
|
||||
}
|
||||
}
|
||||
if !validFilename("留档.png") {
|
||||
t.Fatal("plain Unicode PNG rejected")
|
||||
}
|
||||
}
|
||||
func TestUnexposedAndMissingTargetRejected(t *testing.T) {
|
||||
a := newAdapter(nil)
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
args obj
|
||||
}{{"run_script", obj{}}, {"click", obj{"pid": 1, "window_id": 2, "scope": "desktop"}}, {"get_window_state", obj{"pid": 1}}, {"click", obj{"pid": 0, "window_id": 2}}} {
|
||||
if a.validate(c.name, c.args) == nil {
|
||||
t.Fatal(c)
|
||||
}
|
||||
}
|
||||
}
|
||||
func TestSupportedArguments(t *testing.T) {
|
||||
a := newAdapter(nil)
|
||||
if err := a.validate("click", obj{"pid": 1, "window_id": 2, "element_token": "s00000001:0"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackgroundPreflightVsActuatorUncertainty(t *testing.T) {
|
||||
r := obj{"isError": true, "structuredContent": obj{"code": "background_unavailable", "event_kind": "mouse_scroll"}}
|
||||
if actionReceipt(r, true)["execution"] != "not_started" {
|
||||
t.Fatal("preflight refusal must be explicit")
|
||||
}
|
||||
object(r["structuredContent"])["cause"] = "unknown actuator failure"
|
||||
if actionReceipt(r, true)["execution"] != "unknown" {
|
||||
t.Fatal("actuator uncertainty must not be hidden")
|
||||
}
|
||||
}
|
||||
|
||||
func TestClickModesAreDisjoint(t *testing.T) {
|
||||
a := newAdapter(nil)
|
||||
if a.validate("click", obj{"pid": 1, "window_id": 2, "element_token": "s1:0", "capture_id": "capture"}) == nil {
|
||||
t.Fatal("mixed click modes")
|
||||
}
|
||||
if a.validate("click_at", obj{"pid": 1, "window_id": 2, "x": 5, "y": 5}) == nil {
|
||||
t.Fatal("pixel action lacks capture")
|
||||
}
|
||||
}
|
||||
|
||||
func TestKeyboardRejectsAmbiguousTargetsBeforeWindowOrDriverAccess(t *testing.T) {
|
||||
a := newAdapter(nil) // Any upstream access would panic.
|
||||
for _, name := range []string{"press_key", "type_text"} {
|
||||
for _, extra := range []obj{
|
||||
{"element_token": "s00000001:9", "x": 576, "y": 25},
|
||||
{"element_token": "s00000002:9", "x": 0, "y": 0},
|
||||
{"x": 0, "y": 0}, {"element_token": nil},
|
||||
} {
|
||||
args := obj{"pid": 1, "window_id": 2}
|
||||
if name == "press_key" {
|
||||
args["key"] = "t"
|
||||
args["modifiers"] = []any{"ctrl"}
|
||||
} else {
|
||||
args["text"] = "test"
|
||||
}
|
||||
for k, v := range extra {
|
||||
args[k] = v
|
||||
}
|
||||
r := a.handle(name, args)
|
||||
rec := object(object(r["structuredContent"])["artemis"])
|
||||
if r["isError"] != true || rec["execution"] != "not_started" || rec["side_effect"] != "none" || rec["error_code"] != "invalid_input" {
|
||||
t.Fatalf("%s ambiguous action was not rejected before delivery: %v", name, r)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestInputContractsHaveOneTargetingMode(t *testing.T) {
|
||||
a := newAdapter(nil)
|
||||
for _, name := range []string{"press_key", "type_text"} {
|
||||
props := object(a.schemas[name]["properties"])
|
||||
for _, key := range []string{"element_token", "x", "y"} {
|
||||
if _, exists := props[key]; exists {
|
||||
t.Fatalf("%s exposes ambiguous %s", name, key)
|
||||
}
|
||||
}
|
||||
}
|
||||
for name, args := range map[string]obj{
|
||||
"press_key": {"pid": 1, "window_id": 2, "key": "t", "modifiers": []any{"ctrl"}, "delivery_mode": "foreground"},
|
||||
"type_text": {"pid": 1, "window_id": 2, "text": "中文"},
|
||||
"set_value": {"pid": 1, "window_id": 2, "element_token": "s00000001:9", "value": "value"},
|
||||
"scroll": {"pid": 1, "window_id": 2, "direction": "down", "x": 100, "y": 100, "capture_id": "current"},
|
||||
} {
|
||||
if err := a.validate(name, args); err != nil {
|
||||
t.Fatalf("valid %s: %v", name, err)
|
||||
}
|
||||
}
|
||||
for _, token := range []any{nil, "", " "} {
|
||||
if a.validate("set_value", obj{"pid": 1, "window_id": 2, "value": "v", "element_token": token}) == nil {
|
||||
t.Fatal("set_value must target a concrete element")
|
||||
}
|
||||
}
|
||||
if a.validate("set_value", obj{"pid": 1, "window_id": 2, "value": "v"}) == nil {
|
||||
t.Fatal("set_value requires an element")
|
||||
}
|
||||
for _, extra := range []obj{{"x": 1}, {"x": 1, "y": 1, "element_token": "s00000001:9"}} {
|
||||
args := obj{"pid": 1, "window_id": 2, "direction": "down"}
|
||||
for k, v := range extra {
|
||||
args[k] = v
|
||||
}
|
||||
if a.validate("scroll", args) == nil {
|
||||
t.Fatal("ambiguous scroll accepted")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
diff --git a/libs/cua-driver/rust/crates/platform-windows/src/win32/windows.rs b/libs/cua-driver/rust/crates/platform-windows/src/win32/windows.rs
|
||||
index b13588d..b0bccf5 100644
|
||||
--- a/libs/cua-driver/rust/crates/platform-windows/src/win32/windows.rs
|
||||
+++ b/libs/cua-driver/rust/crates/platform-windows/src/win32/windows.rs
|
||||
@@ -281,14 +281,26 @@ fn window_info_by_handle(hwnd: u64) -> Option<WindowInfo> {
|
||||
return None;
|
||||
}
|
||||
let title_len = unsafe { GetWindowTextLengthW(native) };
|
||||
- if title_len == 0 {
|
||||
+ // An exact PID/HWND remains a valid identity for untitled native shell
|
||||
+ // surfaces. Keep ordinary application/overlay discovery filters intact.
|
||||
+ let untitled_shell = matches!(
|
||||
+ window_class_name(native).as_str(),
|
||||
+ "Shell_TrayWnd"
|
||||
+ | "Shell_SecondaryTrayWnd"
|
||||
+ | "NotifyIconOverflowWindow"
|
||||
+ | "TopLevelWindowForOverflowXamlIsland"
|
||||
+ | "#32768"
|
||||
+ | "Progman"
|
||||
+ | "WorkerW"
|
||||
+ );
|
||||
+ if title_len == 0 && !untitled_shell {
|
||||
return None;
|
||||
}
|
||||
let mut buf = vec![0u16; (title_len + 1) as usize];
|
||||
unsafe { GetWindowTextW(native, &mut buf) };
|
||||
let title_len = buf.iter().position(|&c| c == 0).unwrap_or(buf.len());
|
||||
let title = String::from_utf16_lossy(&buf[..title_len]);
|
||||
- if title.trim().is_empty() {
|
||||
+ if title.trim().is_empty() && !untitled_shell {
|
||||
return None;
|
||||
}
|
||||
let minimized = unsafe { IsIconic(native) }.as_bool();
|
||||
@@ -0,0 +1,181 @@
|
||||
{
|
||||
"schema_version": 2,
|
||||
"id": "cua-driver",
|
||||
"name": "电脑控制 · Cua",
|
||||
"version": "0.38.0+artemis.1",
|
||||
"description": "通过 Cua Driver 的窗口截图与可访问性树操作 Windows 应用及浏览器窗口;复用 ARTEMIS 的任务调度、步骤证据与录像。",
|
||||
"artemis_version": ">=2.0.0,<3.0.0",
|
||||
"platforms": [
|
||||
"win32"
|
||||
],
|
||||
"architectures": [
|
||||
"amd64"
|
||||
],
|
||||
"source": "https://github.com/trycua/cua",
|
||||
"license": "MIT",
|
||||
"server": {
|
||||
"transport": "stdio",
|
||||
"command": "${plugin_dir}/bin/artemis-cua-adapter.exe"
|
||||
},
|
||||
"skill": "SKILL.md",
|
||||
"targets": [
|
||||
"computer",
|
||||
"mixed"
|
||||
],
|
||||
"resources": [
|
||||
"desktop:local"
|
||||
],
|
||||
"observation": {
|
||||
"tool": "list_windows",
|
||||
"arguments": {
|
||||
"on_screen_only": true
|
||||
},
|
||||
"refresh_tools": [
|
||||
"get_window_state"
|
||||
]
|
||||
},
|
||||
"capabilities": {
|
||||
"kind": "visual",
|
||||
"recording": "windows",
|
||||
"tools": {
|
||||
"list_windows": {
|
||||
"effect": "read",
|
||||
"observation": true,
|
||||
"idempotent": true,
|
||||
"target_argument": "",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text"
|
||||
]
|
||||
},
|
||||
"get_window_state": {
|
||||
"effect": "read",
|
||||
"observation": true,
|
||||
"idempotent": true,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"verify_state": {
|
||||
"effect": "read",
|
||||
"observation": false,
|
||||
"idempotent": true,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"click": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"drag": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"type_text": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"press_key": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"set_value": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"scroll": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"save_screenshot": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image",
|
||||
"file"
|
||||
]
|
||||
},
|
||||
"click_at": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": false,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"bring_to_front": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": true,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image",
|
||||
"state"
|
||||
]
|
||||
}
|
||||
},
|
||||
"required_tools": [
|
||||
"list_windows",
|
||||
"get_window_state",
|
||||
"save_screenshot"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
package main
|
||||
|
||||
import "strings"
|
||||
|
||||
// Shell surfaces often have an empty title and are omitted by upstream's
|
||||
// ordinary application-window enumeration. They still require exact HWND/PID.
|
||||
func shellKind(class string) string {
|
||||
switch class {
|
||||
case "Shell_TrayWnd", "Shell_SecondaryTrayWnd":
|
||||
return "taskbar"
|
||||
case "NotifyIconOverflowWindow", "TopLevelWindowForOverflowXamlIsland":
|
||||
return "tray"
|
||||
case "#32768":
|
||||
return "menu"
|
||||
case "Progman", "WorkerW":
|
||||
return "desktop"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func shellMatches(window, query obj) bool {
|
||||
if query["pid"] != nil && num(query["pid"]) != num(window["pid"]) {
|
||||
return false
|
||||
}
|
||||
app := strings.TrimSuffix(strings.ToLower(str(query["app"])), ".exe")
|
||||
name := strings.TrimSuffix(strings.ToLower(str(window["app_name"])), ".exe")
|
||||
return (app == "" || strings.Contains(name, app)) && (query["on_screen_only"] != true || window["is_on_screen"] == true)
|
||||
}
|
||||
|
||||
func supplementShellWindows(result, query obj, native []obj) obj {
|
||||
if result["isError"] == true {
|
||||
return result
|
||||
}
|
||||
data := object(result["structuredContent"])
|
||||
windows, ok := data["windows"].([]any)
|
||||
if !ok { // Do not turn an upstream observation failure into a valid list.
|
||||
return result
|
||||
}
|
||||
known := map[int64]obj{}
|
||||
for _, value := range windows {
|
||||
w := object(value)
|
||||
known[num(w["window_id"])] = w
|
||||
}
|
||||
for _, w := range native {
|
||||
if shellKind(str(w["class_name"])) == "" || !shellMatches(w, query) {
|
||||
continue
|
||||
}
|
||||
if old, exists := known[num(w["window_id"])]; exists {
|
||||
old["class_name"] = w["class_name"]
|
||||
old["window_kind"] = shellKind(str(w["class_name"]))
|
||||
} else {
|
||||
w["window_kind"] = shellKind(str(w["class_name"]))
|
||||
windows = append(windows, w)
|
||||
known[num(w["window_id"])] = w
|
||||
}
|
||||
}
|
||||
data["windows"] = windows
|
||||
result["structuredContent"] = data
|
||||
return result
|
||||
}
|
||||
|
||||
// Follow only native keyboard focus or a uniquely associated active menu.
|
||||
// A different foreground application never silently becomes screenshot scope.
|
||||
func postActionTarget(args, requested, focus obj, candidates []obj) (obj, string) {
|
||||
selected := object(focus["focus_root"])
|
||||
if focus["probe_ok"] != true {
|
||||
return observationArgs(args), "keyboard_focus_unavailable"
|
||||
}
|
||||
if focus["menu_active"] == true {
|
||||
owner := object(focus["menu_owner"])
|
||||
var menus []obj
|
||||
for _, w := range candidates {
|
||||
if shellKind(str(w["class_name"])) == "menu" && w["is_on_screen"] == true &&
|
||||
num(w["pid"]) == num(args["pid"]) && num(owner["pid"]) == num(args["pid"]) &&
|
||||
(num(w["owner_id"]) == num(owner["window_id"]) ||
|
||||
(num(w["thread_id"]) != 0 && num(w["thread_id"]) == num(owner["thread_id"]))) {
|
||||
menus = append(menus, w)
|
||||
}
|
||||
}
|
||||
if len(menus) == 1 {
|
||||
selected = menus[0]
|
||||
} else if len(menus) > 1 {
|
||||
return observationArgs(args), "ambiguous_active_menu_observe_candidates_explicitly"
|
||||
}
|
||||
}
|
||||
if num(selected["window_id"]) == num(args["window_id"]) {
|
||||
return observationArgs(args), "keyboard_focus_remains_in_requested_window"
|
||||
}
|
||||
if num(selected["pid"]) != num(args["pid"]) || selected["is_on_screen"] != true {
|
||||
return observationArgs(args), "unrelated_or_unverified_focus_not_followed"
|
||||
}
|
||||
kind := shellKind(str(selected["class_name"]))
|
||||
relatedShell := shellKind(str(requested["class_name"])) != "" && kind != "" && kind != "desktop"
|
||||
menuOwner := object(focus["menu_owner"])
|
||||
// Native #32768 menus can report GW_OWNER=0. GetGUIThreadInfo still
|
||||
// identifies their owning window; the candidate was also matched by PID
|
||||
// and UI thread above. Do not broaden this to arbitrary same-PID windows.
|
||||
relatedPopup := kind == "menu" && (num(selected["owner_id"]) == num(args["window_id"]) ||
|
||||
(focus["menu_active"] == true && num(menuOwner["window_id"]) == num(args["window_id"])))
|
||||
if !relatedShell && !relatedPopup {
|
||||
return observationArgs(args), "unrelated_focus_not_followed"
|
||||
}
|
||||
next := observationArgs(args)
|
||||
next["pid"], next["window_id"] = selected["pid"], selected["window_id"]
|
||||
return next, "associated_native_keyboard_focus"
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
package main
|
||||
|
||||
import "testing"
|
||||
|
||||
func shellFixtureWindow(pid, hwnd int64, class string) obj {
|
||||
return obj{"pid": pid, "window_id": hwnd, "class_name": class,
|
||||
"is_on_screen": true, "app_name": "explorer.exe", "title": ""}
|
||||
}
|
||||
|
||||
func TestUntitledShellDiscoveryKeepsAppAndPIDScope(t *testing.T) {
|
||||
native := []obj{shellFixtureWindow(1, 10, "Shell_TrayWnd"), shellFixtureWindow(2, 20, "NotifyIconOverflowWindow")}
|
||||
for _, query := range []obj{{"pid": 1}, {"app": "Explorer", "pid": 1}, {"app": "explorer.exe", "pid": 1}} {
|
||||
r := supplementShellWindows(obj{"structuredContent": obj{"windows": []any{}}}, query, native)
|
||||
windows := items(object(r["structuredContent"])["windows"])
|
||||
if len(windows) != 1 || num(object(windows[0])["window_id"]) != 10 {
|
||||
t.Fatal(r)
|
||||
}
|
||||
}
|
||||
r := supplementShellWindows(obj{"structuredContent": obj{"windows": []any{}}}, obj{"app": "Brave"}, native)
|
||||
if len(items(object(r["structuredContent"])["windows"])) != 0 {
|
||||
t.Fatal("widened app scope")
|
||||
}
|
||||
broken := obj{"isError": true}
|
||||
if supplementShellWindows(broken, obj{}, native)["isError"] != true {
|
||||
t.Fatal("observation error hidden")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWinBObservationFollowsActualTaskbarFocus(t *testing.T) {
|
||||
args := obj{"pid": 1, "window_id": 10}
|
||||
requested := shellFixtureWindow(1, 10, "Progman")
|
||||
focus := obj{"probe_ok": true, "focus_root": shellFixtureWindow(1, 20, "Shell_TrayWnd"),
|
||||
"foreground_window": requested} // Foreground HWND alone would give the wrong answer.
|
||||
next, reason := postActionTarget(args, requested, focus, nil)
|
||||
if num(next["window_id"]) != 20 || reason != "associated_native_keyboard_focus" {
|
||||
t.Fatal(next, reason)
|
||||
}
|
||||
if num(args["window_id"]) != 10 {
|
||||
t.Fatal("requested target history mutated")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPostActionNeverCapturesUnrelatedApplicationOrAmbiguousMenu(t *testing.T) {
|
||||
args := obj{"pid": 1, "window_id": 10}
|
||||
requested := shellFixtureWindow(1, 10, "Progman")
|
||||
for _, other := range []obj{shellFixtureWindow(2, 20, "Shell_TrayWnd"), shellFixtureWindow(1, 20, "OtherApp")} {
|
||||
next, _ := postActionTarget(args, requested, obj{"probe_ok": true, "focus_root": other}, nil)
|
||||
if num(next["window_id"]) != 10 {
|
||||
t.Fatal("unrelated focus captured", next)
|
||||
}
|
||||
}
|
||||
menu := shellFixtureWindow(1, 30, "#32768")
|
||||
menu["owner_id"] = int64(10)
|
||||
menu2 := shellFixtureWindow(1, 40, "#32768")
|
||||
menu2["owner_id"] = int64(10)
|
||||
focus := obj{"probe_ok": true, "menu_active": true, "menu_owner": requested}
|
||||
next, reason := postActionTarget(args, requested, focus, []obj{menu, menu2})
|
||||
if num(next["window_id"]) != 10 || reason != "ambiguous_active_menu_observe_candidates_explicitly" {
|
||||
t.Fatal(next, reason)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOwnedNativeMenuAndOverflowHaveIndependentObservationTargets(t *testing.T) {
|
||||
args := obj{"pid": 1, "window_id": 10}
|
||||
owner := shellFixtureWindow(1, 10, "Shell_TrayWnd")
|
||||
menu := shellFixtureWindow(1, 30, "#32768")
|
||||
menu["owner_id"] = int64(10)
|
||||
focus := obj{"probe_ok": true, "menu_active": true, "menu_owner": owner}
|
||||
next, _ := postActionTarget(args, owner, focus, []obj{menu})
|
||||
if num(next["window_id"]) != 30 {
|
||||
t.Fatal(next)
|
||||
}
|
||||
next, _ = postActionTarget(args, owner, obj{"probe_ok": true,
|
||||
"focus_root": shellFixtureWindow(1, 40, "TopLevelWindowForOverflowXamlIsland")}, nil)
|
||||
if num(next["window_id"]) != 40 {
|
||||
t.Fatal(next)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNativeMenuWithoutGWOwnerRequiresVerifiedMenuOwnerAndThread(t *testing.T) {
|
||||
args := obj{"pid": 1, "window_id": 10}
|
||||
owner := shellFixtureWindow(1, 10, "ordinary-form")
|
||||
owner["thread_id"] = int64(7)
|
||||
menu := shellFixtureWindow(1, 30, "#32768")
|
||||
menu["owner_id"], menu["thread_id"] = int64(0), int64(7)
|
||||
focus := obj{"probe_ok": true, "menu_active": true, "menu_owner": owner}
|
||||
next, _ := postActionTarget(args, owner, focus, []obj{menu})
|
||||
if num(next["window_id"]) != 30 {
|
||||
t.Fatal(next)
|
||||
}
|
||||
menu["thread_id"] = int64(8)
|
||||
next, _ = postActionTarget(args, owner, focus, []obj{menu})
|
||||
if num(next["window_id"]) != 10 {
|
||||
t.Fatal("unrelated menu followed", next)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"sync"
|
||||
"syscall"
|
||||
"unsafe"
|
||||
)
|
||||
|
||||
type windowRect struct{ left, top, right, bottom int32 }
|
||||
|
||||
func windowClass(hwnd int64) string {
|
||||
var value [256]uint16
|
||||
user32.NewProc("GetClassNameW").Call(uintptr(hwnd), uintptr(unsafe.Pointer(&value[0])), uintptr(len(value)))
|
||||
return syscall.UTF16ToString(value[:])
|
||||
}
|
||||
|
||||
func nativeWindowInfo(hwnd int64) obj {
|
||||
if hwnd == 0 {
|
||||
return obj{}
|
||||
}
|
||||
var pid uint32
|
||||
thread, _, _ := user32.NewProc("GetWindowThreadProcessId").Call(uintptr(hwnd), uintptr(unsafe.Pointer(&pid)))
|
||||
if pid == 0 {
|
||||
return obj{}
|
||||
}
|
||||
visible, _, _ := user32.NewProc("IsWindowVisible").Call(uintptr(hwnd))
|
||||
owner, _, _ := user32.NewProc("GetWindow").Call(uintptr(hwnd), 4)
|
||||
root, _, _ := user32.NewProc("GetAncestor").Call(uintptr(hwnd), 2)
|
||||
var title [1024]uint16
|
||||
user32.NewProc("GetWindowTextW").Call(uintptr(hwnd), uintptr(unsafe.Pointer(&title[0])), uintptr(len(title)))
|
||||
app := ""
|
||||
process, _, _ := kernel32.NewProc("OpenProcess").Call(0x1000, 0, uintptr(pid))
|
||||
if process != 0 {
|
||||
var path [32768]uint16
|
||||
size := uint32(len(path))
|
||||
ok, _, _ := kernel32.NewProc("QueryFullProcessImageNameW").Call(process, 0, uintptr(unsafe.Pointer(&path[0])), uintptr(unsafe.Pointer(&size)))
|
||||
if ok != 0 {
|
||||
app = filepath.Base(syscall.UTF16ToString(path[:]))
|
||||
}
|
||||
kernel32.NewProc("CloseHandle").Call(process)
|
||||
}
|
||||
info := obj{"pid": int64(pid), "window_id": hwnd, "thread_id": int64(thread),
|
||||
"root_window_id": int64(root), "owner_id": int64(owner), "class_name": windowClass(hwnd),
|
||||
"title": syscall.UTF16ToString(title[:]), "app_name": app, "is_on_screen": visible != 0,
|
||||
"z_index": nil, "minimized": false}
|
||||
runtime.LockOSThread()
|
||||
defer runtime.UnlockOSThread()
|
||||
previous, _, _ := user32.NewProc("SetThreadDpiAwarenessContext").Call(^uintptr(3))
|
||||
if previous != 0 {
|
||||
defer user32.NewProc("SetThreadDpiAwarenessContext").Call(previous)
|
||||
var rect windowRect
|
||||
ok, _, _ := user32.NewProc("GetWindowRect").Call(uintptr(hwnd), uintptr(unsafe.Pointer(&rect)))
|
||||
if ok != 0 {
|
||||
info["bounds"] = obj{"x": rect.left, "y": rect.top, "width": rect.right - rect.left, "height": rect.bottom - rect.top}
|
||||
info["is_on_screen"] = visible != 0 && rect.right > rect.left && rect.bottom > rect.top
|
||||
}
|
||||
}
|
||||
return info
|
||||
}
|
||||
|
||||
// A single static callback avoids NewCallback's non-reclaimable slot leak.
|
||||
var shellEnumerationLock sync.Mutex
|
||||
var shellEnumeration *[]obj
|
||||
var shellEnumerator = syscall.NewCallback(func(hwnd, _ uintptr) uintptr {
|
||||
if shellKind(windowClass(int64(hwnd))) != "" {
|
||||
info := nativeWindowInfo(int64(hwnd))
|
||||
if info["is_on_screen"] == true {
|
||||
*shellEnumeration = append(*shellEnumeration, info)
|
||||
}
|
||||
}
|
||||
return 1
|
||||
})
|
||||
|
||||
func nativeShellWindows() []obj {
|
||||
shellEnumerationLock.Lock()
|
||||
defer shellEnumerationLock.Unlock()
|
||||
var windows []obj
|
||||
shellEnumeration = &windows
|
||||
user32.NewProc("EnumWindows").Call(shellEnumerator, 0)
|
||||
shellEnumeration = nil
|
||||
return windows
|
||||
}
|
||||
|
||||
func keyboardFocus() obj {
|
||||
foreground := nativeWindowInfo(foregroundWindow())
|
||||
var gui struct {
|
||||
size, flags uint32
|
||||
active, focus, capture, menuOwner, moveSize, caret uintptr
|
||||
rect windowRect
|
||||
}
|
||||
gui.size = uint32(unsafe.Sizeof(gui))
|
||||
ok, _, _ := user32.NewProc("GetGUIThreadInfo").Call(0, uintptr(unsafe.Pointer(&gui)))
|
||||
result := obj{"probe_ok": ok != 0, "foreground_window": foreground}
|
||||
if ok != 0 {
|
||||
focused := nativeWindowInfo(int64(gui.focus))
|
||||
result["focus_window"] = focused
|
||||
result["focus_root"] = nativeWindowInfo(num(focused["root_window_id"]))
|
||||
result["active_window"] = nativeWindowInfo(int64(gui.active))
|
||||
result["menu_owner"] = nativeWindowInfo(int64(gui.menuOwner))
|
||||
result["menu_active"] = gui.flags&0x1c != 0
|
||||
}
|
||||
return result
|
||||
}
|
||||
@@ -0,0 +1,668 @@
|
||||
[
|
||||
{
|
||||
"name": "list_windows",
|
||||
"description": "Find the authorized application window. If its PID is unknown, call with app only, e.g. {\"app\":\"Brave\",\"on_screen_only\":true}; omit pid or use null, never 0. Use the returned matching pid/window_id for later actions. Empty filtered results are not proof that the app is unavailable; follow lookup.suggested_arguments when present.",
|
||||
"inputSchema": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"app": {
|
||||
"description": "Application name substring (case-insensitive). Start with this when the target PID is unknown. app and a non-null pid are combined filters; do not copy the PID of an unrelated foreground app.",
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"on_screen_only": {
|
||||
"description": "When true, drop windows that aren't currently on-screen. Default false.",
|
||||
"type": "boolean"
|
||||
},
|
||||
"pid": {
|
||||
"description": "Known positive Windows process ID from a matching list_windows result. Unknown: omit or null; never 0. With app, both filters must match the same window.",
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
],
|
||||
"minimum": 1,
|
||||
"maximum": 4294967295
|
||||
}
|
||||
},
|
||||
"type": "object",
|
||||
"required": []
|
||||
},
|
||||
"annotations": {
|
||||
"readOnlyHint": true,
|
||||
"idempotentHint": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "get_window_state",
|
||||
"description": "Read fresh window screenshot and compact UIA tree. Every [N] uses element_token=<snapshot_id>:N; a new snapshot invalidates old tokens. Pixel x/y are screenshot-local; use capture_id. since=latest returns changes. Keep the new snapshot_id.",
|
||||
"inputSchema": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"max_elements": {
|
||||
"description": "Cap on the total number of UIA nodes walked. Truncates depth-first; markdown and structured elements truncate together. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a la",
|
||||
"minimum": 1,
|
||||
"type": "integer"
|
||||
},
|
||||
"max_image_dimension": {
|
||||
"description": "Per-call long-edge limit for the returned screenshot, in pixels (aspect ratio preserved). Explicit values override configured behavior; 0 returns native resolution. Omit to preserve the configured default.",
|
||||
"minimum": 0,
|
||||
"type": "integer"
|
||||
},
|
||||
"pid": {
|
||||
"description": "Positive process ID returned for the target app by list_windows; never guess or use another app foreground PID.",
|
||||
"type": "integer"
|
||||
},
|
||||
"query": {
|
||||
"description": "Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count.",
|
||||
"type": "string"
|
||||
},
|
||||
"since": {
|
||||
"description": "A `snapshot_id` from an earlier get_window_state of the same window, or \"latest\" for this window's most recent read with the same query/max_elements/max_depth. Use it on every re-read after acting. Returns only what changed against that sna",
|
||||
"type": "string"
|
||||
},
|
||||
"window_id": {
|
||||
"description": "Exact HWND returned for this application by list_windows. Required and must belong to pid.",
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"pid",
|
||||
"window_id"
|
||||
],
|
||||
"type": "object"
|
||||
},
|
||||
"annotations": {
|
||||
"readOnlyHint": true,
|
||||
"idempotentHint": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "verify_state",
|
||||
"description": "Read-only assertion for exact window. satisfied/unsatisfied/unknown describes the observation, not the whole task verdict.",
|
||||
"inputSchema": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"expect": {
|
||||
"description": "One to eight predicates, combined with logical AND.",
|
||||
"items": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"element": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"enabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"exists": {
|
||||
"description": "Assert that at least one trusted element matches the selector.\n\nElement walks are not yet exhaustive on every platform, so absence\ncannot be proven. `false` is rejected instead of returning an\nindefinitely-unknown predicate.",
|
||||
"type": "boolean"
|
||||
},
|
||||
"selected": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"selector": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"label_contains": {
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"role": {
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [],
|
||||
"type": "object"
|
||||
},
|
||||
"value_equals": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"selector"
|
||||
],
|
||||
"type": "object"
|
||||
},
|
||||
"window": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"bounds": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"height": {
|
||||
"type": "number"
|
||||
},
|
||||
"tolerance_px": {
|
||||
"maximum": 100,
|
||||
"minimum": 0,
|
||||
"type": "number"
|
||||
},
|
||||
"width": {
|
||||
"type": "number"
|
||||
},
|
||||
"x": {
|
||||
"type": "number"
|
||||
},
|
||||
"y": {
|
||||
"type": "number"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"x",
|
||||
"y",
|
||||
"width",
|
||||
"height"
|
||||
],
|
||||
"type": "object"
|
||||
},
|
||||
"exists": {
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"required": [],
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"required": [],
|
||||
"type": "object"
|
||||
},
|
||||
"maxItems": 8,
|
||||
"minItems": 1,
|
||||
"type": "array"
|
||||
},
|
||||
"include_screenshot": {
|
||||
"description": "Return the final window screenshot as image content for a multimodal\ncaller. The driver does not interpret that image.",
|
||||
"type": "boolean"
|
||||
},
|
||||
"pid": {
|
||||
"description": "Positive process ID returned for the target app by list_windows; never guess or use another app foreground PID.",
|
||||
"minimum": 1,
|
||||
"type": "integer"
|
||||
},
|
||||
"stable_samples": {
|
||||
"default": 2,
|
||||
"description": "Consecutive satisfied samples required before returning success.",
|
||||
"maximum": 5,
|
||||
"minimum": 1,
|
||||
"type": "integer"
|
||||
},
|
||||
"timeout_ms": {
|
||||
"default": 5000,
|
||||
"description": "Bounded wait. Zero performs one sample.",
|
||||
"maximum": 10000,
|
||||
"minimum": 0,
|
||||
"type": "integer"
|
||||
},
|
||||
"window_id": {
|
||||
"description": "Exact HWND returned for this application by list_windows. Required and must belong to pid.",
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"pid",
|
||||
"window_id",
|
||||
"expect"
|
||||
],
|
||||
"type": "object"
|
||||
},
|
||||
"annotations": {
|
||||
"readOnlyHint": true,
|
||||
"idempotentHint": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "click",
|
||||
"description": "Click the latest UIA element_token. This tool has no pixel coordinates. For visual targets use click_at. Returns effect and fresh observation; old tokens become stale. For Chromium/Brave use foreground, especially after scrolling: it resolves off-screen element bounds before input. Background posted clicks may use old/off-screen positions; inspect actual evidence before retrying.",
|
||||
"inputSchema": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"button": {
|
||||
"description": "Mouse button. Default \"left\".",
|
||||
"enum": [
|
||||
"left",
|
||||
"right",
|
||||
"middle"
|
||||
],
|
||||
"type": "string"
|
||||
},
|
||||
"count": {
|
||||
"description": "Click count — 1 (single), 2 (double), 3 (triple). Default 1.",
|
||||
"maximum": 3,
|
||||
"minimum": 1,
|
||||
"type": "integer"
|
||||
},
|
||||
"delivery_mode": {
|
||||
"description": "background does not activate the window. foreground uses Cua bring_to_front before input and leaves the target foreground for subsequent rendering and evidence; it does not switch back to the user's previous app after each action.",
|
||||
"enum": [
|
||||
"background",
|
||||
"foreground"
|
||||
],
|
||||
"type": "string"
|
||||
},
|
||||
"element_token": {
|
||||
"description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current sna",
|
||||
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
|
||||
"type": "string"
|
||||
},
|
||||
"pid": {
|
||||
"description": "Positive process ID returned for the target app by list_windows; never guess or use another app foreground PID.",
|
||||
"type": "integer"
|
||||
},
|
||||
"window_id": {
|
||||
"description": "Exact HWND returned for this application by list_windows. Required and must belong to pid.",
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"type": "object",
|
||||
"required": [
|
||||
"pid",
|
||||
"window_id",
|
||||
"element_token"
|
||||
]
|
||||
},
|
||||
"annotations": {
|
||||
"readOnlyHint": false,
|
||||
"idempotentHint": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "drag",
|
||||
"description": "Drag between screenshot-local coordinates. Returns actual delivery effect and fresh post-action observation. Requires the latest capture_id. Stale, moved/resized or out-of-image targets are rejected before input.",
|
||||
"inputSchema": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"delivery_mode": {
|
||||
"description": "background does not activate the window. foreground uses Cua bring_to_front before input and leaves the target foreground for subsequent rendering and evidence; it does not switch back to the user's previous app after each action.",
|
||||
"enum": [
|
||||
"background",
|
||||
"foreground"
|
||||
],
|
||||
"type": "string"
|
||||
},
|
||||
"duration_ms": {
|
||||
"description": "Wall-clock duration of drag path. Default: 500.",
|
||||
"maximum": 10000,
|
||||
"minimum": 0,
|
||||
"type": "integer"
|
||||
},
|
||||
"from_x": {
|
||||
"description": "Drag-start X in window-local screenshot pixels.",
|
||||
"type": "number"
|
||||
},
|
||||
"from_y": {
|
||||
"description": "Drag-start Y in window-local screenshot pixels.",
|
||||
"type": "number"
|
||||
},
|
||||
"pid": {
|
||||
"description": "Positive process ID returned for the target app by list_windows; never guess or use another app foreground PID.",
|
||||
"type": "integer"
|
||||
},
|
||||
"to_x": {
|
||||
"description": "Drag-end X in window-local screenshot pixels.",
|
||||
"type": "number"
|
||||
},
|
||||
"to_y": {
|
||||
"description": "Drag-end Y in window-local screenshot pixels.",
|
||||
"type": "number"
|
||||
},
|
||||
"window_id": {
|
||||
"description": "Exact HWND returned for this application by list_windows. Required and must belong to pid.",
|
||||
"type": "integer"
|
||||
},
|
||||
"capture_id": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "capture_id from the latest returned screenshot for this exact window. Each action returns a new screenshot; use its new ID and dimensions."
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"pid",
|
||||
"window_id",
|
||||
"from_x",
|
||||
"from_y",
|
||||
"to_x",
|
||||
"to_y",
|
||||
"capture_id"
|
||||
],
|
||||
"type": "object"
|
||||
},
|
||||
"annotations": {
|
||||
"readOnlyHint": false,
|
||||
"idempotentHint": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "type_text",
|
||||
"description": "Type text at the current focus in the exact window. No element_token or coordinates. Focus a field with click/click_at or the address bar with Ctrl+L first; use set_value for direct UIA replacement. Chromium/Brave input requires foreground. Fresh observation follows.",
|
||||
"inputSchema": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"delivery_mode": {
|
||||
"description": "background does not activate the window. foreground uses Cua bring_to_front before input and leaves the target foreground for subsequent rendering and evidence; it does not switch back to the user's previous app after each action.",
|
||||
"enum": [
|
||||
"background",
|
||||
"foreground"
|
||||
],
|
||||
"type": "string"
|
||||
},
|
||||
"pid": {
|
||||
"description": "Positive process ID returned for the target app by list_windows; never guess or use another app foreground PID.",
|
||||
"type": "integer"
|
||||
},
|
||||
"text": {
|
||||
"description": "Text to insert at the focused element's cursor.",
|
||||
"type": "string"
|
||||
},
|
||||
"window_id": {
|
||||
"description": "Exact HWND returned for this application by list_windows. Required and must belong to pid.",
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"pid",
|
||||
"window_id",
|
||||
"text"
|
||||
],
|
||||
"type": "object"
|
||||
},
|
||||
"annotations": {
|
||||
"readOnlyHint": false,
|
||||
"idempotentHint": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "press_key",
|
||||
"description": "Send a key and modifiers to the exact window current focus. No element_token or coordinates. For browser shortcuts (Ctrl+T, Ctrl+L, Enter), use foreground. To target a field, focus it with click/click_at first. Fresh observation follows; inspect before repeating.",
|
||||
"inputSchema": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"delivery_mode": {
|
||||
"description": "background does not activate the window. foreground uses Cua bring_to_front before input and leaves the target foreground for subsequent rendering and evidence; it does not switch back to the user's previous app after each action.",
|
||||
"enum": [
|
||||
"background",
|
||||
"foreground"
|
||||
],
|
||||
"type": "string"
|
||||
},
|
||||
"key": {
|
||||
"description": "Key name (return, tab, escape, up, down, left, right, space, delete, home, end, pageup, pagedown, f1-f12, letter, digit).",
|
||||
"type": "string"
|
||||
},
|
||||
"modifiers": {
|
||||
"description": "Optional modifier names held while the key is pressed (ctrl/shift/alt/win).",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
"pid": {
|
||||
"description": "Positive process ID returned for the target app by list_windows; never guess or use another app foreground PID.",
|
||||
"type": "integer"
|
||||
},
|
||||
"window_id": {
|
||||
"description": "Exact HWND returned for this application by list_windows. Required and must belong to pid.",
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"pid",
|
||||
"window_id",
|
||||
"key"
|
||||
],
|
||||
"type": "object"
|
||||
},
|
||||
"annotations": {
|
||||
"readOnlyHint": false,
|
||||
"idempotentHint": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "set_value",
|
||||
"description": "Set the latest UIA element_token value. Returns delivery effect/readback and fresh screenshot/tree.",
|
||||
"inputSchema": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"delivery_mode": {
|
||||
"description": "background does not activate the window. foreground uses Cua bring_to_front before input and leaves the target foreground for subsequent rendering and evidence; it does not switch back to the user's previous app after each action.",
|
||||
"enum": [
|
||||
"background",
|
||||
"foreground"
|
||||
],
|
||||
"type": "string"
|
||||
},
|
||||
"element_token": {
|
||||
"description": "Element handle `<snapshot_id>:<row>` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current sna",
|
||||
"pattern": "^s[0-9a-f]{8}:[0-9]+$",
|
||||
"type": "string"
|
||||
},
|
||||
"pid": {
|
||||
"description": "Positive process ID returned for the target app by list_windows; never guess or use another app foreground PID.",
|
||||
"type": "integer"
|
||||
},
|
||||
"value": {
|
||||
"description": "New value. UIA will coerce to the element's native type.",
|
||||
"type": "string"
|
||||
},
|
||||
"window_id": {
|
||||
"description": "Exact HWND returned for this application by list_windows. Required and must belong to pid.",
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"pid",
|
||||
"window_id",
|
||||
"value",
|
||||
"element_token"
|
||||
],
|
||||
"type": "object"
|
||||
},
|
||||
"annotations": {
|
||||
"readOnlyHint": false,
|
||||
"idempotentHint": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "scroll",
|
||||
"description": "Scroll at x,y in the latest window screenshot. No element_token. Use a point inside the intended scroll surface, amount=1..50, by=line/page. Chromium/Brave requires foreground. Verify actual content movement in the returned observation. Requires the latest capture_id. Stale, moved/resized or out-of-image targets are rejected before input.",
|
||||
"inputSchema": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"amount": {
|
||||
"description": "Number of scroll ticks. Default 3.",
|
||||
"maximum": 50,
|
||||
"minimum": 1,
|
||||
"type": "integer"
|
||||
},
|
||||
"by": {
|
||||
"description": "Scroll granularity. Default: line.",
|
||||
"enum": [
|
||||
"line",
|
||||
"page"
|
||||
],
|
||||
"type": "string"
|
||||
},
|
||||
"delivery_mode": {
|
||||
"description": "background does not activate the window. foreground uses Cua bring_to_front before input and leaves the target foreground for subsequent rendering and evidence; it does not switch back to the user's previous app after each action.",
|
||||
"enum": [
|
||||
"background",
|
||||
"foreground"
|
||||
],
|
||||
"type": "string"
|
||||
},
|
||||
"direction": {
|
||||
"description": "Scroll direction.",
|
||||
"enum": [
|
||||
"up",
|
||||
"down",
|
||||
"left",
|
||||
"right"
|
||||
],
|
||||
"type": "string"
|
||||
},
|
||||
"pid": {
|
||||
"description": "Positive process ID returned for the target app by list_windows; never guess or use another app foreground PID.",
|
||||
"type": "integer"
|
||||
},
|
||||
"window_id": {
|
||||
"description": "Exact HWND returned for this application by list_windows. Required and must belong to pid.",
|
||||
"type": "integer"
|
||||
},
|
||||
"x": {
|
||||
"description": "X inside the scroll area in latest window-local screenshot pixels; never screen coordinates. Must be paired with the other coordinate.",
|
||||
"type": "number"
|
||||
},
|
||||
"y": {
|
||||
"description": "Y inside the scroll area in latest window-local screenshot pixels; never screen coordinates. Must be paired with the other coordinate.",
|
||||
"type": "number"
|
||||
},
|
||||
"capture_id": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "capture_id from the latest returned screenshot for this exact window. Each action returns a new screenshot; use its new ID and dimensions."
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"pid",
|
||||
"window_id",
|
||||
"direction",
|
||||
"x",
|
||||
"y",
|
||||
"capture_id"
|
||||
],
|
||||
"type": "object"
|
||||
},
|
||||
"annotations": {
|
||||
"readOnlyHint": false,
|
||||
"idempotentHint": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "save_screenshot",
|
||||
"description": "Save fresh window screenshot to this task trace. Plain .png filename only; no overwrite. Returns verified attachment hash, screenshot and latest tree.",
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"pid": {
|
||||
"type": "integer",
|
||||
"minimum": 1,
|
||||
"description": "Positive process ID returned for the target app by list_windows; never guess or use another app foreground PID."
|
||||
},
|
||||
"window_id": {
|
||||
"type": "integer",
|
||||
"minimum": 1,
|
||||
"description": "Exact HWND returned for this application by list_windows. Required and must belong to pid."
|
||||
},
|
||||
"filename": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"pid",
|
||||
"window_id",
|
||||
"filename"
|
||||
]
|
||||
},
|
||||
"annotations": {
|
||||
"readOnlyHint": false,
|
||||
"idempotentHint": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "click_at",
|
||||
"description": "Click a visual target at x,y in the latest window screenshot, with its exact capture_id. Never pass an element token. Returns effect and fresh observation. Use the latest returned image dimensions; no screen coordinates or Android normalization.",
|
||||
"inputSchema": {
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"button": {
|
||||
"description": "Mouse button. Default \"left\".",
|
||||
"enum": [
|
||||
"left",
|
||||
"right",
|
||||
"middle"
|
||||
],
|
||||
"type": "string"
|
||||
},
|
||||
"capture_id": {
|
||||
"description": "Optional ID from get_window_state or get_desktop_state. When present, x and y are admitted against that exact capture and the ID is consumed before native dispatch.",
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
"count": {
|
||||
"description": "Click count — 1 (single), 2 (double), 3 (triple). Default 1.",
|
||||
"maximum": 3,
|
||||
"minimum": 1,
|
||||
"type": "integer"
|
||||
},
|
||||
"delivery_mode": {
|
||||
"description": "background does not activate the window. foreground uses Cua bring_to_front before input and leaves the target foreground for subsequent rendering and evidence; it does not switch back to the user's previous app after each action.",
|
||||
"enum": [
|
||||
"background",
|
||||
"foreground"
|
||||
],
|
||||
"type": "string"
|
||||
},
|
||||
"pid": {
|
||||
"description": "Positive process ID returned for the target app by list_windows; never guess or use another app foreground PID.",
|
||||
"type": "integer"
|
||||
},
|
||||
"window_id": {
|
||||
"description": "Exact HWND returned for this application by list_windows. Required and must belong to pid.",
|
||||
"type": "integer"
|
||||
},
|
||||
"x": {
|
||||
"description": "X in window-local screenshot pixels — same space as the PNG get_window_state returns. Must be provided together with y.",
|
||||
"type": "number"
|
||||
},
|
||||
"y": {
|
||||
"description": "Y in window-local screenshot pixels. Must be provided together with x.",
|
||||
"type": "number"
|
||||
}
|
||||
},
|
||||
"type": "object",
|
||||
"required": [
|
||||
"pid",
|
||||
"window_id",
|
||||
"x",
|
||||
"y",
|
||||
"capture_id"
|
||||
]
|
||||
},
|
||||
"annotations": {
|
||||
"readOnlyHint": false,
|
||||
"idempotentHint": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "bring_to_front",
|
||||
"description": "Activate the exact authorized window and leave it foreground across observations and actions. Use when a background browser body is blank or its page tree is missing. This changes the user foreground; it does not navigate, refresh, or repeat any business action. Read the returned fresh screenshot before continuing.",
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"pid",
|
||||
"window_id"
|
||||
],
|
||||
"properties": {
|
||||
"pid": {
|
||||
"type": "integer",
|
||||
"minimum": 1,
|
||||
"maximum": 4294967295
|
||||
},
|
||||
"window_id": {
|
||||
"type": "integer",
|
||||
"minimum": 1
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
},
|
||||
"annotations": {
|
||||
"readOnlyHint": false,
|
||||
"destructiveHint": false,
|
||||
"idempotentHint": true,
|
||||
"openWorldHint": false
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,95 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"math"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Optional filters must remain optional across JSON-schema clients. In
|
||||
// particular, null means unspecified; zero is not a wildcard process ID.
|
||||
func windowQuery(args obj) (obj, error) {
|
||||
query := obj{}
|
||||
for key, value := range args {
|
||||
if value != nil {
|
||||
query[key] = value
|
||||
}
|
||||
}
|
||||
if value, ok := query["app"]; ok {
|
||||
app, valid := value.(string)
|
||||
if !valid {
|
||||
return nil, errors.New("app must be an application name or null")
|
||||
}
|
||||
if app = strings.TrimSpace(app); app == "" {
|
||||
delete(query, "app")
|
||||
} else {
|
||||
query["app"] = app
|
||||
}
|
||||
}
|
||||
if value, ok := query["pid"]; ok {
|
||||
valid := num(value) > 0 && num(value) <= math.MaxUint32
|
||||
switch n := value.(type) {
|
||||
case float64:
|
||||
valid = valid && n == math.Trunc(n)
|
||||
case int, int64:
|
||||
default:
|
||||
valid = false
|
||||
}
|
||||
if !valid {
|
||||
return nil, errors.New("pid must be a positive Windows process ID from a matching list_windows result. If unknown, omit pid or use null and search by app, e.g. {\"app\":\"Brave\",\"on_screen_only\":true}. Never use 0 or a different app's foreground PID")
|
||||
}
|
||||
}
|
||||
return query, nil
|
||||
}
|
||||
|
||||
// Upstream's empty-result hint names the unrelated foreground process. Do not
|
||||
// present that PID as a recovery candidate. Keep the exact filters and provide
|
||||
// a separate suggested query; never silently widen the caller's PID scope.
|
||||
func windowListing(result obj, query obj) obj {
|
||||
if result["isError"] == true {
|
||||
return result
|
||||
}
|
||||
data := object(result["structuredContent"])
|
||||
windows, ok := data["windows"].([]any)
|
||||
if !ok {
|
||||
return failure("Driver returned no window list; this is an observation error, not evidence that the application is absent", "not_started", "none", "invalid_window_list")
|
||||
}
|
||||
lookup := obj{"status": "matched", "filters": query, "count": len(windows)}
|
||||
lines := []string{fmt.Sprintf("Found %d matching window(s).", len(windows))}
|
||||
for _, value := range windows {
|
||||
window := object(value)
|
||||
lines = append(lines, fmt.Sprintf("- %s (pid %d), window_id %d, title %q", str(window["app_name"]), num(window["pid"]), num(window["window_id"]), str(window["title"])))
|
||||
if kind := str(window["window_kind"]); kind != "" {
|
||||
lines[len(lines)-1] += fmt.Sprintf(" [%s, class %s]", kind, str(window["class_name"]))
|
||||
}
|
||||
}
|
||||
if len(windows) == 0 {
|
||||
lookup["status"] = "no_match"
|
||||
lines = append(lines, "No windows match these filters. This alone does not prove the application is unavailable or screenshot capture is broken. Do not substitute another application's foreground PID.")
|
||||
if str(query["app"]) != "" && query["pid"] != nil {
|
||||
next := obj{}
|
||||
for key, value := range query {
|
||||
if key != "pid" {
|
||||
next[key] = value
|
||||
}
|
||||
}
|
||||
lookup["suggested_arguments"] = next
|
||||
lines = append(lines, "app and pid are combined filters. Retry list_windows for the same app with pid omitted; use only a PID returned for that app.")
|
||||
} else if query["on_screen_only"] == true {
|
||||
next := obj{}
|
||||
for key, value := range query {
|
||||
next[key] = value
|
||||
}
|
||||
next["on_screen_only"] = false
|
||||
lookup["suggested_arguments"] = next
|
||||
lines = append(lines, "Retry the same filters with on_screen_only=false to check for minimized/off-screen windows. A listed window still needs a fresh get_window_state before interaction.")
|
||||
} else {
|
||||
lines = append(lines, "Verify the requested application name or PID. When the PID is unknown, search by the known app name with pid omitted. Do not guess a PID or operate an unrelated window.")
|
||||
}
|
||||
}
|
||||
data["lookup"] = lookup
|
||||
result["structuredContent"] = data
|
||||
result["content"] = []any{textBlock(strings.Join(lines, "\n"))}
|
||||
return result
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type lookupInput struct{ bytes.Buffer }
|
||||
|
||||
func (*lookupInput) Close() error { return nil }
|
||||
|
||||
func TestInvalidDiscoveryPIDStopsBeforeDriver(t *testing.T) {
|
||||
for _, pid := range []any{0, -1, 1.5, "123", true, float64(4294967296)} {
|
||||
result := newAdapter(nil).handle("list_windows", obj{"app": "Brave", "pid": pid})
|
||||
if result["isError"] != true || object(object(result["structuredContent"])["artemis"])["execution"] != "not_started" {
|
||||
t.Fatalf("invalid PID %v reached driver or was reported as no windows: %v", pid, result)
|
||||
}
|
||||
if !strings.Contains(str(object(items(result["content"])[0])["text"]), "omit pid") {
|
||||
t.Fatalf("missing recovery instruction for %v", pid)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAppDiscoveryOmitsNullPIDAtDriverBoundary(t *testing.T) {
|
||||
input := &lookupInput{}
|
||||
lines := make(chan []byte, 1)
|
||||
lines <- []byte(`{"id":1,"result":{"content":[],"structuredContent":{"windows":[{"app_name":"brave.exe","pid":72,"window_id":81,"title":"Browser"}]}}}`)
|
||||
a := newAdapter(&peer{input: input, lines: lines})
|
||||
args := obj{"app": " Brave ", "pid": nil, "on_screen_only": true}
|
||||
result := a.handle("list_windows", args)
|
||||
if result["isError"] == true || len(items(object(result["structuredContent"])["windows"])) != 1 {
|
||||
t.Fatalf("failed app-only discovery: %v", result)
|
||||
}
|
||||
var sent obj
|
||||
if err := json.Unmarshal(input.Bytes(), &sent); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
forwarded := object(object(sent["params"])["arguments"])
|
||||
if _, present := forwarded["pid"]; present || forwarded["app"] != "Brave" || forwarded["on_screen_only"] != true {
|
||||
t.Fatalf("incorrect upstream query: %v", forwarded)
|
||||
}
|
||||
if args["app"] != " Brave " || len(args) != 3 {
|
||||
t.Fatal("changed original arguments used in trace")
|
||||
}
|
||||
}
|
||||
|
||||
func TestConflictingFiltersPreserveScopeAndRemoveForegroundHint(t *testing.T) {
|
||||
query := obj{"app": "Brave", "pid": 14492, "on_screen_only": true}
|
||||
result := windowListing(obj{
|
||||
"content": []any{textBlock(`No windows. Current frontmost app is "studio64.exe" (pid 14492).`)},
|
||||
"structuredContent": obj{"windows": []any{}},
|
||||
}, query)
|
||||
data := object(result["structuredContent"])
|
||||
lookup := object(data["lookup"])
|
||||
next := object(lookup["suggested_arguments"])
|
||||
if len(items(data["windows"])) != 0 || lookup["status"] != "no_match" || object(lookup["filters"])["pid"] != 14492 {
|
||||
t.Fatal("original filter or no-match result was hidden")
|
||||
}
|
||||
if _, exists := next["pid"]; exists || next["app"] != "Brave" || next["on_screen_only"] != true {
|
||||
t.Fatalf("incorrect app-only recovery: %v", next)
|
||||
}
|
||||
text := str(object(items(result["content"])[0])["text"])
|
||||
if strings.Contains(text, "studio64.exe") || !strings.Contains(text, "pid omitted") {
|
||||
t.Fatalf("misleading recovery: %s", text)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmptyVisibleQuerySuggestsSameTargetOffScreen(t *testing.T) {
|
||||
query := obj{"pid": 12, "on_screen_only": true}
|
||||
result := windowListing(obj{"structuredContent": obj{"windows": []any{}}}, query)
|
||||
next := object(object(object(result["structuredContent"])["lookup"])["suggested_arguments"])
|
||||
if next["pid"] != 12 || next["on_screen_only"] != false || query["on_screen_only"] != true {
|
||||
t.Fatalf("changed target or original query: %v", next)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLookupErrorsNeverBecomeEmptySuccess(t *testing.T) {
|
||||
for _, result := range []obj{
|
||||
{"isError": true, "content": []any{textBlock("driver unavailable")}},
|
||||
{"structuredContent": obj{}},
|
||||
} {
|
||||
if windowListing(result, obj{"app": "Brave"})["isError"] != true {
|
||||
t.Fatal("observation failure became a successful empty list")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"runtime"
|
||||
"syscall"
|
||||
"unsafe"
|
||||
)
|
||||
|
||||
var user32 = syscall.NewLazyDLL("user32.dll")
|
||||
var kernel32 = syscall.NewLazyDLL("kernel32.dll")
|
||||
|
||||
func foregroundWindow() int64 {
|
||||
hwnd, _, _ := user32.NewProc("GetForegroundWindow").Call()
|
||||
return int64(hwnd)
|
||||
}
|
||||
|
||||
func configureProcess(c *exec.Cmd) {
|
||||
c.SysProcAttr = &syscall.SysProcAttr{HideWindow: true}
|
||||
}
|
||||
func windowOwned(pid, hwnd int64) bool {
|
||||
var actual uint32
|
||||
r, _, _ := user32.NewProc("IsWindow").Call(uintptr(hwnd))
|
||||
if r == 0 {
|
||||
return false
|
||||
}
|
||||
user32.NewProc("GetWindowThreadProcessId").Call(uintptr(hwnd), uintptr(unsafe.Pointer(&actual)))
|
||||
return int64(actual) == pid
|
||||
}
|
||||
func verifyWindow(pid, hwnd int64, b obj) (obj, bool) {
|
||||
// DPI context belongs to an OS thread, not a Go goroutine. Apply it around
|
||||
// the actual geometry reads; setting it once while starting a child leaks
|
||||
// context and later GetSystemMetrics can return virtualized dimensions.
|
||||
runtime.LockOSThread()
|
||||
defer runtime.UnlockOSThread()
|
||||
setDPI := user32.NewProc("SetThreadDpiAwarenessContext")
|
||||
previous, _, _ := setDPI.Call(^uintptr(3))
|
||||
if previous == 0 {
|
||||
return nil, false
|
||||
}
|
||||
defer setDPI.Call(previous)
|
||||
if !windowOwned(pid, hwnd) {
|
||||
return nil, false
|
||||
}
|
||||
var r struct{ L, T, R, B int32 }
|
||||
code, _, _ := syscall.NewLazyDLL("dwmapi.dll").NewProc("DwmGetWindowAttribute").Call(uintptr(hwnd), 9, uintptr(unsafe.Pointer(&r)), unsafe.Sizeof(r))
|
||||
if code != 0 || r.R <= r.L || r.B <= r.T {
|
||||
// Shell menus/taskbars can lack a DWM extended frame. Match their
|
||||
// actual HWND rectangle; never substitute desktop or another window.
|
||||
if shellKind(windowClass(hwnd)) == "" {
|
||||
return nil, false
|
||||
}
|
||||
ok, _, _ := user32.NewProc("GetWindowRect").Call(uintptr(hwnd), uintptr(unsafe.Pointer(&r)))
|
||||
if ok == 0 {
|
||||
return nil, false
|
||||
}
|
||||
}
|
||||
// Cua uses DWM physical bounds; refuse evidence after a window move/resize.
|
||||
if int64(r.L) != num(b["left"]) || int64(r.T) != num(b["top"]) || int64(r.R-r.L) != num(b["width"]) || int64(r.B-r.T) != num(b["height"]) {
|
||||
return nil, false
|
||||
}
|
||||
metric := func(n uintptr) int64 { x, _, _ := user32.NewProc("GetSystemMetrics").Call(n); return int64(int32(x)) }
|
||||
return obj{"left": metric(76), "top": metric(77), "width": metric(78), "height": metric(79)}, true
|
||||
}
|
||||
func ownProcess(pid int) (uintptr, error) {
|
||||
job, _, e := kernel32.NewProc("CreateJobObjectW").Call(0, 0)
|
||||
if job == 0 {
|
||||
return 0, e
|
||||
}
|
||||
// JOBOBJECT_EXTENDED_LIMIT_INFORMATION, Windows x64. Kill the entire owned
|
||||
// child tree when the adapter dies, including Cua's UIA helper.
|
||||
data := make([]byte, 144)
|
||||
*(*uint32)(unsafe.Pointer(&data[16])) = 0x2000
|
||||
ok, _, _ := kernel32.NewProc("SetInformationJobObject").Call(job, 9, uintptr(unsafe.Pointer(&data[0])), uintptr(len(data)))
|
||||
h, _, _ := kernel32.NewProc("OpenProcess").Call(0x0101, 0, uintptr(pid))
|
||||
if h == 0 || ok == 0 {
|
||||
closeJob(job)
|
||||
return 0, fmt.Errorf("cannot own driver process")
|
||||
}
|
||||
defer kernel32.NewProc("CloseHandle").Call(h)
|
||||
ok, _, e = kernel32.NewProc("AssignProcessToJobObject").Call(job, h)
|
||||
if ok == 0 {
|
||||
closeJob(job)
|
||||
return 0, e
|
||||
}
|
||||
return job, nil
|
||||
}
|
||||
func closeJob(h uintptr) {
|
||||
if h != 0 {
|
||||
kernel32.NewProc("CloseHandle").Call(h)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,202 @@
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
https://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2026 Google LLC.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
https://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -0,0 +1,17 @@
|
||||
# 任务人工评审 1.1.0
|
||||
|
||||
需要 ARTEMIS 2.4.0 或更高的 2.x 版本。升级主程序后,可在「插件 → 高级接入」导入本目录或发行 ZIP。2.2.x 不支持此接口。
|
||||
|
||||
安装并启用后,下载的批量模板在原七列后附加“预置条件、操作步骤、预期结果”;也兼容旧七列表格。未启用插件时模板和校验仍保持七列。新增三列不进入 AI 提示词。
|
||||
|
||||
打开「原始用例」,左侧标题为用例编号与用例标题,依次展示三项原始内容;旧任务没有记录时明确显示缺失,不拿 AI 提示词代替。打开「人工评审」,填写「用例执行准确性」「测试结果正确性」及各自备注。全部选填,空项不阻止保存或导出。只有任务结束后可以编辑;可修改、清空并查看历史。
|
||||
|
||||
此插件没有 MCP、Skill、AI 调用、设备操作或任务执行资源。评审不覆盖原有测试结论和人工复验。安装并启用时,单任务 Word/Excel、批次汇总及资料包报告显示评审字段,未填字段为空。禁用或卸载后隐藏面板及新导出中的字段,历史数据由宿主保留,重新安装启用后恢复。已下载报告不会改写;删除任务会清理该任务的评审。
|
||||
|
||||
`workspace.panels[].width` 声明首选、最小和最大宽度(CSS 像素);宿主根据播放器和屏幕空间调整布局,窄屏可低于声明的最小值以避免溢出。默认收起、左右独立展开;窄屏一次展开一侧;单侧展开始终保持在回放左边或右边。
|
||||
|
||||
在仓库根目录运行 `python scripts/build_task_review.py` 生成安装 ZIP 和 SHA256SUMS.txt。产物默认在 `artifacts/task-review/`,不提交构建产物。发行标签为 `task-review-v1.1.0`,安装包及 SHA256 以 catalog-v4.json 为准。
|
||||
|
||||
回放标题栏的侧栏按钮控制左右展开;两项评审使用「通过 / 失败」分段按钮,再次点击已选项可清空。导出保留相同中文值。
|
||||
|
||||
Flash/Pro/探索及安卓、电脑、跨端任务均可显示面板,步骤和视频模式均支持。右侧上一条/下一条按同批次已结束任务的执行顺序切换;独立任务使用当前历史列表。上一条/下一条位于保存评审下方。选择结果后立即自动保存,备注停止输入 600 毫秒后保存;切换前先保存,失败或冲突时保留当前任务和输入,手动保存可重试。打开或读取任务不触发保存,面板展开状态保持。备注框缩为一行起始高度,可手动拉高。原字段 ID 不变,1.0.0 评审继续读取。
|
||||
@@ -0,0 +1,11 @@
|
||||
# 任务人工评审 1.1.0
|
||||
|
||||
需要 ARTEMIS 2.4.0+(2.x)。先更新宿主并重启,再通过「插件 → 高级接入」导入 task-review-1.1.0.zip 并启用。
|
||||
|
||||
- 启用插件后,批量模板附加预置条件、操作步骤、预期结果。未启用时仍是七列;新增列不进入模型提示词。
|
||||
- 左侧展示原始用例,用例编号和标题置顶;历史缺失字段明确提示。
|
||||
- 「AI执行准确性」改为「用例执行准确性」,保留字段 ID 和旧评审值。
|
||||
- 备注框缩短;增加上一条、下一条,按钮位于保存评审下方;用户修改后自动保存,切换前保存,失败留在当前任务,连续修改不会被旧回执覆盖。
|
||||
- 面板适用于所有执行模式、设备目标及步骤/视频视图。
|
||||
|
||||
发行标签为 task-review-v1.1.0。验证范围与结果见 [验证记录](VERIFICATION.md)。
|
||||
@@ -0,0 +1,40 @@
|
||||
# 任务人工评审 1.1.0 验证
|
||||
|
||||
日期:2026-10-28;宿主 ARTEMIS 2.4.0。本轮在 Windows 隔离环境完成。
|
||||
|
||||
- 宿主发布前最终默认确定性测试:3267 passed、2 skipped、8 deselected。
|
||||
- 前端 ChromeHeadless:347 项通过;生产构建通过。
|
||||
- 宿主 Ruff 格式/静态检查、质量基线、核心 Pyright、Git diff 空白检查通过。
|
||||
- 新旧导入模板及插件启用门控、原始用例与 AI 提示词隔离、批次评审顺序通过。
|
||||
- 结果选择立即自动保存、备注 600ms 防抖、清空、连续修改串行修订、旧回执与旧读取不覆盖新内容、关闭前保存、切换前保存、失败留在当前项通过。
|
||||
- Windows Chrome 实际浏览器隔离夹具验证:视频视图中原始用例与评审面板;导航位于保存下方;自动保存、切换往返、宽屏/窄屏、深浅色;未出现页面脚本异常。
|
||||
- 清单通过宿主 check_plugin.py 检查,字段 ID 保持不变。
|
||||
- 2.4.0 通过独立 catalog-v4.json 获取 1.1.0;保留 catalog-v3.json 的 1.0.0,使用 2.3.0 的严格面板模型验证旧索引仍可解析。官方旧地址迁移与自定义地址保留的回归通过。
|
||||
|
||||
浏览器使用拦截 API 的示例数据,未填写真实任务的人工评审,未操作用户设备或调用模型。未做 macOS/Linux 实测,未运行原 192 项批次或高并发设备压力测试。
|
||||
|
||||
以下为旧版历史记录,不能作为本轮验证结果。
|
||||
|
||||
---
|
||||
|
||||
# 任务人工评审 1.0.0 验证记录
|
||||
|
||||
日期:2026-10-27。发布目标:task-review-v1.0.0。
|
||||
|
||||
宿主:ARTEMIS 2.3.0。插件基于独立插件仓库 b780a60,未夹带其他未发布插件变更;清单最低宿主版本为 2.3.0,2.2.x 不支持。
|
||||
|
||||
## 已验证
|
||||
|
||||
- 宿主默认确定性测试:3197 passed,2 skipped,8 deselected。
|
||||
- 前端 ChromeHeadless:336 项通过;生产构建通过。
|
||||
- 宿主 Ruff 格式/静态检查、质量基线、核心 Pyright 通过。
|
||||
- 清单和字段验证、仅启用插件贡献字段、纯界面插件不参与执行选择。
|
||||
- 原始任务提示词 Markdown、部分填写、清空、失败重试保留输入、修订冲突、历史记录、切换任务不串写。
|
||||
- 运行中只读,终态可评审;禁用/卸载隐藏贡献,重装恢复,删除任务/批次清理;复跑不继承。
|
||||
- Word/Excel 单任务和批次汇总、ZIP 报告包含已启用贡献,未填为空;评审和插件状态变更使旧缓存失效。
|
||||
- 使用生产前端与隔离示例任务进行浏览器验证:标题栏左右开关、普通浮窗/影院模式、浅色/深色、单侧在左右展开;600/740px 容器左右布局的浏览器回归通过。
|
||||
- 实际通过网页保存中文「通过」,生成 Word/Excel 并确认值为中文,另一未填结果为空。
|
||||
|
||||
## 范围
|
||||
|
||||
验证使用 Windows 本地环境与临时数据库,不操作用户设备、业务应用或现有任务,不调用模型。未在 macOS/Linux 实测,不将此记录视为真实业务用例通过或市场发布证明。此插件无运行进程和 MCP,因此不需要设备动作、取消进程或录像验收。
|
||||
@@ -0,0 +1,84 @@
|
||||
{
|
||||
"schema_version": 3,
|
||||
"id": "task-review",
|
||||
"name": "任务人工评审",
|
||||
"version": "1.1.0",
|
||||
"description": "在任务回放中查看导入的原始用例,逐条填写用例执行准确性与测试结果正确性,按需填写备注并随报告导出。",
|
||||
"artemis_version": ">=2.4.0,<3.0.0",
|
||||
"platforms": [
|
||||
"win32",
|
||||
"linux",
|
||||
"darwin"
|
||||
],
|
||||
"architectures": [
|
||||
"amd64",
|
||||
"arm64"
|
||||
],
|
||||
"license": "Apache-2.0",
|
||||
"targets": [],
|
||||
"capabilities": {
|
||||
"kind": "other",
|
||||
"recording": "none",
|
||||
"tools": {}
|
||||
},
|
||||
"workspace": {
|
||||
"panels": [
|
||||
{
|
||||
"id": "requirements",
|
||||
"title": "原始用例",
|
||||
"side": "left",
|
||||
"kind": "original_case",
|
||||
"width": {
|
||||
"preferred": 340,
|
||||
"minimum": 240,
|
||||
"maximum": 480
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "review",
|
||||
"title": "人工评审",
|
||||
"side": "right",
|
||||
"kind": "form",
|
||||
"width": {
|
||||
"preferred": 360,
|
||||
"minimum": 260,
|
||||
"maximum": 520
|
||||
}
|
||||
}
|
||||
],
|
||||
"fields": [
|
||||
{
|
||||
"id": "execution_accuracy",
|
||||
"label": "用例执行准确性",
|
||||
"description": "核对实际操作是否符合用例步骤。",
|
||||
"kind": "choice",
|
||||
"options": [
|
||||
"通过",
|
||||
"失败"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "execution_note",
|
||||
"label": "用例执行准确性备注",
|
||||
"kind": "text",
|
||||
"max_length": 2000
|
||||
},
|
||||
{
|
||||
"id": "result_correctness",
|
||||
"label": "测试结果正确性",
|
||||
"description": "核对测试结果是否符合用例的预期结果。",
|
||||
"kind": "choice",
|
||||
"options": [
|
||||
"通过",
|
||||
"失败"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "result_note",
|
||||
"label": "测试结果正确性备注",
|
||||
"kind": "text",
|
||||
"max_length": 2000
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,176 @@
|
||||
"""Build a self-contained Windows plugin from a hash-pinned Cua release."""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import urllib.request
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SOURCE = ROOT / "plugins/cua-driver"
|
||||
UPSTREAM_VERSION = "0.38.0"
|
||||
COMMIT = "85e2b1318bd77ccf3cfa88ccdf875a4bb7eaed19"
|
||||
DIGEST = "5edaa74d0afa1b55a9668dbc4f32736507bd2bd4d75b18ca1dc8881bb12402b5"
|
||||
URL = "https://github.com/trycua/cua/releases/download/cua-driver-rs-v0.38.0/cua-driver-rs-0.38.0-windows-x86_64-binary.zip"
|
||||
PATCH = SOURCE / "patches/cua-driver-0.38.0-untitled-shell.patch"
|
||||
PATCHED_PATH = "libs/cua-driver/rust/crates/platform-windows/src/win32/windows.rs"
|
||||
|
||||
|
||||
def build_patched_runtime(source: Path, cargo: str) -> dict:
|
||||
source = source.resolve()
|
||||
head = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=source, text=True).strip()
|
||||
if head != COMMIT:
|
||||
raise ValueError("Patched Cua source must use the pinned upstream commit")
|
||||
changes = subprocess.check_output(["git", "diff", "--name-only", "HEAD"], cwd=source, text=True)
|
||||
if changes.splitlines() != [PATCHED_PATH]:
|
||||
raise ValueError("Cua source may contain only the reviewed untitled-shell patch")
|
||||
diff = subprocess.check_output(
|
||||
["git", "diff", "--binary", "HEAD", "--", PATCHED_PATH], cwd=source
|
||||
)
|
||||
if diff.replace(b"\r\n", b"\n") != PATCH.read_bytes().replace(b"\r\n", b"\n"):
|
||||
raise ValueError("Cua source does not match the packaged compatibility patch")
|
||||
project = source / "libs/cua-driver/rust"
|
||||
command = [
|
||||
cargo,
|
||||
"build",
|
||||
"--release",
|
||||
"--locked",
|
||||
"-p",
|
||||
"cua-driver",
|
||||
"-p",
|
||||
"cua-driver-uia",
|
||||
"-p",
|
||||
"cua-driver-sdk",
|
||||
]
|
||||
subprocess.run(command, cwd=project, check=True, timeout=1800)
|
||||
binaries = {
|
||||
name: project / "target/release" / name
|
||||
for name in ("cua-driver.exe", "cua-driver-uia.exe", "cua_driver_sdk.dll")
|
||||
}
|
||||
if not all(path.is_file() for path in binaries.values()):
|
||||
raise ValueError("Patched Cua runtime build is incomplete")
|
||||
return binaries
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(__doc__)
|
||||
parser.add_argument("upstream_zip", type=Path)
|
||||
parser.add_argument("--go", default="go")
|
||||
parser.add_argument("--cargo", default="cargo")
|
||||
parser.add_argument("--upstream-source", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, default=ROOT / "artifacts")
|
||||
args = parser.parse_args()
|
||||
version = json.loads((SOURCE / "plugin.json").read_text("utf-8"))["version"]
|
||||
if version.split("+", 1)[0] != UPSTREAM_VERSION:
|
||||
raise ValueError("Plugin version must match the pinned Cua Driver version")
|
||||
match = re.search(r'const adapterVersion = "([^"]+)"', (SOURCE / "main.go").read_text("utf-8"))
|
||||
if not match or match[1] != version:
|
||||
raise ValueError("Adapter MCP version must match plugin.json")
|
||||
if hashlib.sha256(args.upstream_zip.read_bytes()).hexdigest() != DIGEST:
|
||||
raise ValueError("Cua release archive hash mismatch")
|
||||
patched_runtime = build_patched_runtime(args.upstream_source, args.cargo)
|
||||
output = args.output.resolve()
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
adapter = output / "artemis-cua-adapter.exe"
|
||||
env = {
|
||||
**os.environ,
|
||||
"CGO_ENABLED": "0",
|
||||
"GOOS": "windows",
|
||||
"GOARCH": "amd64",
|
||||
"GOTOOLCHAIN": "local",
|
||||
}
|
||||
for command in (
|
||||
[args.go, "test", "./..."],
|
||||
[
|
||||
args.go,
|
||||
"build",
|
||||
"-trimpath",
|
||||
"-buildvcs=false",
|
||||
"-ldflags=-s -w",
|
||||
"-o",
|
||||
str(adapter),
|
||||
".",
|
||||
],
|
||||
):
|
||||
subprocess.run(command, cwd=SOURCE, env=env, check=True, timeout=180)
|
||||
license_file = SOURCE / "LICENSE.upstream"
|
||||
if not license_file.exists():
|
||||
with urllib.request.urlopen(
|
||||
f"https://raw.githubusercontent.com/trycua/cua/{COMMIT}/LICENSE.md",
|
||||
timeout=30,
|
||||
) as response:
|
||||
license_file.write_bytes(response.read())
|
||||
target = output / f"cua-driver-{version}-windows-amd64.zip"
|
||||
with (
|
||||
zipfile.ZipFile(target, "w", zipfile.ZIP_DEFLATED) as dest,
|
||||
zipfile.ZipFile(args.upstream_zip) as upstream,
|
||||
):
|
||||
for name in (
|
||||
"plugin.json",
|
||||
"SKILL.md",
|
||||
"LICENSE.upstream",
|
||||
"RELEASE.md",
|
||||
"VERIFICATION.md",
|
||||
"BUILD.md",
|
||||
):
|
||||
dest.write(SOURCE / name, name)
|
||||
for source in sorted(SOURCE.glob("*.go")):
|
||||
dest.write(source, f"source/{source.name}")
|
||||
for name in ("go.mod", "tools.json"):
|
||||
dest.write(SOURCE / name, f"source/{name}")
|
||||
dest.write(PATCH, f"source/patches/{PATCH.name}")
|
||||
dest.write(adapter, "bin/artemis-cua-adapter.exe")
|
||||
for name in upstream.namelist():
|
||||
# Only the release's executable runtime; no global installer/hooks.
|
||||
if Path(name).name in {
|
||||
"cua-driver.exe",
|
||||
"cua-driver-uia.exe",
|
||||
"cua-cursor-theme.exe",
|
||||
"cua_driver_sdk.dll",
|
||||
}:
|
||||
basename = Path(name).name
|
||||
if basename not in patched_runtime:
|
||||
dest.writestr(f"bin/{basename}", upstream.read(name))
|
||||
for name, binary in patched_runtime.items():
|
||||
dest.write(binary, f"bin/{name}")
|
||||
dest.writestr(
|
||||
"provenance.json",
|
||||
json.dumps(
|
||||
{
|
||||
"source_commit": COMMIT,
|
||||
"upstream_version": UPSTREAM_VERSION,
|
||||
"plugin_version": version,
|
||||
"adapter_revision": int(version.rsplit(".", 1)[1]),
|
||||
"prerelease": True,
|
||||
"upstream_release_channel": "stable",
|
||||
"url": URL,
|
||||
"sha256": DIGEST,
|
||||
"compatibility_patch": {
|
||||
"file": f"source/patches/{PATCH.name}",
|
||||
"sha256": hashlib.sha256(PATCH.read_bytes()).hexdigest(),
|
||||
"rebuilt_binaries": {
|
||||
name: hashlib.sha256(binary.read_bytes()).hexdigest()
|
||||
for name, binary in patched_runtime.items()
|
||||
},
|
||||
"cargo_locked": True,
|
||||
},
|
||||
},
|
||||
indent=2,
|
||||
),
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"archive": str(target),
|
||||
"sha256": hashlib.sha256(target.read_bytes()).hexdigest(),
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,30 @@
|
||||
"""Build the declarative task review plugin without runtime dependencies."""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import zipfile
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--output", type=Path, default=Path("artifacts/task-review"))
|
||||
args = parser.parse_args()
|
||||
package = Path(__file__).resolve().parents[1] / "plugins" / "task-review"
|
||||
manifest = json.loads((package / "plugin.json").read_text(encoding="utf-8"))
|
||||
args.output.mkdir(parents=True, exist_ok=True)
|
||||
target = args.output / f"task-review-{manifest['version']}.zip"
|
||||
with zipfile.ZipFile(target, "w", compression=zipfile.ZIP_DEFLATED) as archive:
|
||||
for name in ("plugin.json", "README.md", "LICENSE", "VERIFICATION.md", "RELEASE.md"):
|
||||
info = zipfile.ZipInfo(name, date_time=(2026, 10, 27, 0, 0, 0))
|
||||
info.compress_type = zipfile.ZIP_DEFLATED
|
||||
info.external_attr = 0o100644 << 16
|
||||
archive.writestr(info, (package / name).read_bytes())
|
||||
digest = hashlib.sha256(target.read_bytes()).hexdigest()
|
||||
(args.output / "SHA256SUMS.txt").write_text(f"{digest} {target.name}\n", encoding="ascii")
|
||||
print(f"{target}\nSHA256 {digest}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,47 @@
|
||||
param([Parameter(Mandatory=$true)][string]$StateFile, [string]$Title='ARTEMIS Plugin Benchmark')
|
||||
Add-Type -TypeDefinition 'using System; using System.Runtime.InteropServices; public static class BenchmarkDpi { [DllImport("user32.dll")] public static extern IntPtr SetThreadDpiAwarenessContext(IntPtr c); }'
|
||||
[void][BenchmarkDpi]::SetThreadDpiAwarenessContext([IntPtr](-4))
|
||||
Add-Type -AssemblyName System.Windows.Forms
|
||||
Add-Type -AssemblyName System.Drawing
|
||||
$form = New-Object System.Windows.Forms.Form
|
||||
$form.Text=$Title
|
||||
$form.Size=New-Object System.Drawing.Size(880,660)
|
||||
$form.StartPosition='Manual'
|
||||
$form.Location=New-Object System.Drawing.Point(80,80)
|
||||
$form.TopMost=$true
|
||||
$form.Font=New-Object System.Drawing.Font('Segoe UI',11)
|
||||
$state=@{pid=$PID;title=$Title;applied=$false;ssid='';night=$false;apply_count=0;invalid_apply=0;target_clicks=0;wrong_clicks=0;finished=$false;events=@()}
|
||||
function Save-State {
|
||||
$state | ConvertTo-Json -Depth 6 | Set-Content -LiteralPath $StateFile -Encoding UTF8
|
||||
}
|
||||
function Log-Event([string]$name) {$state.events += @{event=$name;at=[DateTimeOffset]::UtcNow.ToUnixTimeMilliseconds()};Save-State}
|
||||
$heading=New-Object System.Windows.Forms.Label
|
||||
$heading.Text='Router test console - owned fixture';$heading.AutoSize=$true;$heading.Location=New-Object System.Drawing.Point(20,18)
|
||||
$form.Controls.Add($heading)
|
||||
$label=New-Object System.Windows.Forms.Label
|
||||
$label.Text='Network name';$label.Location=New-Object System.Drawing.Point(20,60);$label.AutoSize=$true;$form.Controls.Add($label)
|
||||
$networkInput=New-Object System.Windows.Forms.TextBox
|
||||
$networkInput.AccessibleName='Network name';$networkInput.Location=New-Object System.Drawing.Point(170,56);$networkInput.Size=New-Object System.Drawing.Size(280,32);$form.Controls.Add($networkInput)
|
||||
$check=New-Object System.Windows.Forms.CheckBox
|
||||
$check.Text='Night mode';$check.AccessibleName='Night mode';$check.Location=New-Object System.Drawing.Point(470,56);$check.Size=New-Object System.Drawing.Size(140,32);$form.Controls.Add($check)
|
||||
$apply=New-Object System.Windows.Forms.Button
|
||||
$apply.Text='Apply';$apply.AccessibleName='Apply';$apply.Location=New-Object System.Drawing.Point(630,54);$apply.Size=New-Object System.Drawing.Size(130,36);$form.Controls.Add($apply)
|
||||
$status=New-Object System.Windows.Forms.Label
|
||||
$status.Text='Not applied';$status.AutoSize=$true;$status.Location=New-Object System.Drawing.Point(20,104);$form.Controls.Add($status)
|
||||
$apply.Add_Click({$state.apply_count++;$state.ssid=$networkInput.Text;$state.night=$check.Checked;$state.applied=($networkInput.Text -eq 'ARTEMIS-LAB' -and $check.Checked);if(!$state.applied){$state.invalid_apply++};$status.Text="Saved: $($networkInput.Text); Night mode: $($check.Checked)";Log-Event 'apply'})
|
||||
$hint=New-Object System.Windows.Forms.Label
|
||||
$hint.Text='Visual task: click the BLUE circle only, then scroll the panel to Finish.';$hint.AutoSize=$true;$hint.Location=New-Object System.Drawing.Point(20,150);$form.Controls.Add($hint)
|
||||
$canvas=New-Object System.Windows.Forms.Panel
|
||||
$canvas.Location=New-Object System.Drawing.Point(20,185);$canvas.Size=New-Object System.Drawing.Size(760,100);$canvas.BackColor=[System.Drawing.Color]::White;$form.Controls.Add($canvas)
|
||||
$canvas.Add_Paint({param($sender,$e) for($i=0;$i -lt 8;$i++){$color=if($i -eq 5){[System.Drawing.Brushes]::RoyalBlue}else{[System.Drawing.Brushes]::IndianRed};$e.Graphics.FillEllipse($color,25+90*$i,24,34,34)}})
|
||||
$canvas.Add_MouseClick({param($sender,$e) $hit=$false;for($i=0;$i -lt 8;$i++){if([Math]::Pow($e.X-(42+90*$i),2)+[Math]::Pow($e.Y-41,2) -le 289){$hit=$true;if($i -eq 5){$state.target_clicks++;$hint.Text='Blue target selected. Scroll panel to Finish.';Log-Event 'target'}else{$state.wrong_clicks++;Log-Event 'wrong-target'}}};if(!$hit){$state.wrong_clicks++;Log-Event 'miss'}})
|
||||
$panel=New-Object System.Windows.Forms.Panel
|
||||
$panel.AccessibleName='Scrollable settings';$panel.Location=New-Object System.Drawing.Point(20,310);$panel.Size=New-Object System.Drawing.Size(760,260);$panel.AutoScroll=$true;$panel.BorderStyle='FixedSingle';$form.Controls.Add($panel)
|
||||
for($i=0;$i -lt 12;$i++){$row=New-Object System.Windows.Forms.Label;$row.Text="Diagnostic row $($i+1) - scroll down for Finish";$row.AutoSize=$true;$row.Location=New-Object System.Drawing.Point(15,(15+60*$i));$panel.Controls.Add($row)}
|
||||
$finish=New-Object System.Windows.Forms.Button
|
||||
$finish.Text='Finish';$finish.AccessibleName='Finish';$finish.Location=New-Object System.Drawing.Point(20,760);$finish.Size=New-Object System.Drawing.Size(160,40);$panel.Controls.Add($finish)
|
||||
$finish.Add_Click({$state.scroll_value=$panel.VerticalScroll.Value;$state.finished=($state.target_clicks -eq 1 -and $state.wrong_clicks -eq 0 -and $state.scroll_value -gt 0);$finish.Text=if($state.finished){'Finished successfully'}else{'Target not selected'};Log-Event 'finish'})
|
||||
$timer=New-Object System.Windows.Forms.Timer
|
||||
$timer.Interval=900000;$timer.Add_Tick({$form.Close()})
|
||||
$form.Add_Shown({Save-State;$timer.Start()})
|
||||
try{[System.Windows.Forms.Application]::Run($form)}finally{$timer.Dispose();$form.Dispose()}
|
||||
@@ -0,0 +1,279 @@
|
||||
"""Opt-in sequential A/B through ARTEMIS's existing queue, never a second agent.
|
||||
|
||||
Usage: python tests/benchmark_plugins.py --traces <ARTEMIS traces> --output <new folder>
|
||||
The user-started service must already run. Only this script's owned fixture is closed.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import time
|
||||
import urllib.request
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
PROMPT = """只操作测试窗口 ARTEMIS Plugin Benchmark(进程 PID {pid}),不要观察或操作其他应用。先按此 PID 找到窗口。
|
||||
1. 将 Network name 填为 ARTEMIS-LAB,勾选 Night mode,点击 Apply,确认显示 Saved: ARTEMIS-LAB; Night mode: True。
|
||||
2. 只点击唯一的蓝色圆形一次,不要点击红色圆形。在下方 Scrollable settings 面板向下滚动,找到并点击 Finish,确认 Finished successfully。
|
||||
3. 保存这个窗口的截图到任务 trace,文件名 proof.png,然后结束并如实说明实际结果。不要关闭窗口。
|
||||
"""
|
||||
|
||||
|
||||
def api(base, route, body=None):
|
||||
request = urllib.request.Request(
|
||||
base + route,
|
||||
data=json.dumps(body).encode() if body is not None else None,
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
with urllib.request.urlopen(request, timeout=30) as response:
|
||||
return json.load(response)
|
||||
|
||||
|
||||
def collect(traces, sid):
|
||||
with sqlite3.connect(
|
||||
f"file:{(traces / 'data_engine.db').as_posix()}?mode=ro", uri=True
|
||||
) as db:
|
||||
db.row_factory = sqlite3.Row
|
||||
session = dict(
|
||||
db.execute("SELECT * FROM sessions WHERE session_id=?", (sid,)).fetchone()
|
||||
)
|
||||
events = [
|
||||
dict(row)
|
||||
for row in db.execute(
|
||||
"SELECT name,type,status,duration,payload FROM traces WHERE session_id=? ORDER BY timestamp",
|
||||
(sid,),
|
||||
)
|
||||
]
|
||||
steps = db.execute(
|
||||
"SELECT count(*) FROM steps WHERE session_id=?", (sid,)
|
||||
).fetchone()[0]
|
||||
usage, calls, requests = [], [], []
|
||||
for event in events:
|
||||
payload = json.loads(event["payload"] or "{}")
|
||||
if event["name"] == "llm_usage":
|
||||
usage.append(payload)
|
||||
if "plugin_id" in payload and "tool_name" in payload:
|
||||
# Tool trace contains the real call receipt, not the separate log line.
|
||||
calls.append(
|
||||
{
|
||||
"name": payload["tool_name"],
|
||||
"status": event["status"],
|
||||
"duration": event["duration"],
|
||||
"args": payload.get("args"),
|
||||
"result": payload.get("result"),
|
||||
}
|
||||
)
|
||||
if event["name"] == "llm_request":
|
||||
requests.append(payload)
|
||||
return {
|
||||
"session": session,
|
||||
"steps": steps,
|
||||
"llm_usage": usage,
|
||||
"plugin_calls": calls,
|
||||
"requests": requests,
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(__doc__)
|
||||
parser.add_argument("--base", default="http://127.0.0.1:8000")
|
||||
parser.add_argument("--traces", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--model", default="mimo-v2.6-flash")
|
||||
parser.add_argument("--repetitions", type=int, default=3)
|
||||
parser.add_argument("--resume", action="store_true")
|
||||
args = parser.parse_args()
|
||||
args.output.mkdir(parents=True, exist_ok=args.resume)
|
||||
config = {
|
||||
"model": args.model,
|
||||
"reasoning_effort": "high",
|
||||
"execution_target": "computer",
|
||||
"record_video": True,
|
||||
"recording": {"codec": "h264", "computer_scope": "window"},
|
||||
"analyze_video": False,
|
||||
"timeout_seconds": 480,
|
||||
"request_timeout_seconds": 120,
|
||||
"max_retries": 1,
|
||||
}
|
||||
protocol = {
|
||||
"prompt": PROMPT,
|
||||
"config": config,
|
||||
"repetitions": args.repetitions,
|
||||
"acceptance": "applied && finished && exactly one target click && no wrong clicks && verified proof.png",
|
||||
"order": "OCU,Cua / Cua,OCU / OCU,Cua",
|
||||
"fixture": "benchmark-fixture.ps1",
|
||||
}
|
||||
protocol_path = args.output / "protocol.json"
|
||||
if args.resume and protocol_path.exists():
|
||||
if json.loads(protocol_path.read_text("utf-8")) != protocol:
|
||||
raise ValueError(
|
||||
"Resume requires the original prompt, model and task settings"
|
||||
)
|
||||
else:
|
||||
protocol_path.write_text(
|
||||
json.dumps(protocol, ensure_ascii=False, indent=2), "utf-8"
|
||||
)
|
||||
for repetition in range(args.repetitions):
|
||||
order = (
|
||||
["open-computer-use", "cua-driver"]
|
||||
if repetition % 2 == 0
|
||||
else ["cua-driver", "open-computer-use"]
|
||||
)
|
||||
for plugin in order:
|
||||
folder = args.output / f"{repetition + 1}-{plugin}"
|
||||
if args.resume and (folder / "result.json").is_file():
|
||||
continue
|
||||
queue = api(args.base, "/api/status")
|
||||
if queue.get("active_tasks") or queue.get("queue"):
|
||||
raise RuntimeError(
|
||||
"Other tasks are active/queued; benchmark will not interfere"
|
||||
)
|
||||
folder.mkdir()
|
||||
state_file = folder / "fixture.json"
|
||||
stderr = (folder / "fixture-stderr.log").open("w")
|
||||
fixture = subprocess.Popen(
|
||||
[
|
||||
"powershell.exe",
|
||||
"-NoProfile",
|
||||
"-File",
|
||||
str(Path(__file__).with_name("benchmark-fixture.ps1").resolve()),
|
||||
"-StateFile",
|
||||
str(state_file.resolve()),
|
||||
],
|
||||
creationflags=subprocess.CREATE_NO_WINDOW,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=stderr,
|
||||
)
|
||||
sid = str(uuid.uuid4())
|
||||
done = False
|
||||
try:
|
||||
for _ in range(100):
|
||||
if state_file.exists():
|
||||
break
|
||||
time.sleep(0.1)
|
||||
if not state_file.exists():
|
||||
raise RuntimeError("fixture did not start")
|
||||
body = {
|
||||
"goal": PROMPT.format(pid=fixture.pid),
|
||||
"session_id": sid,
|
||||
"profile": "flash",
|
||||
"ingress": "client",
|
||||
"runtime_overrides": {
|
||||
**config,
|
||||
"plugins": [
|
||||
{
|
||||
"id": plugin,
|
||||
"profile": "benchmark"
|
||||
if plugin == "cua-driver"
|
||||
else "default",
|
||||
"version": "0.1.2"
|
||||
if plugin == "cua-driver"
|
||||
else "1.1.0",
|
||||
}
|
||||
],
|
||||
},
|
||||
}
|
||||
(folder / "request.json").write_text(
|
||||
json.dumps(body, ensure_ascii=False, indent=2), "utf-8"
|
||||
)
|
||||
started = time.monotonic()
|
||||
response = api(args.base, "/api/run", body)
|
||||
(folder / "submission.json").write_text(json.dumps(response), "utf-8")
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "submitted",
|
||||
"plugin": plugin,
|
||||
"round": repetition + 1,
|
||||
"trace_id": sid,
|
||||
"response": response,
|
||||
}
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
if response.get("status") in {"rejected", "error"}:
|
||||
raise RuntimeError("task was rejected")
|
||||
while time.monotonic() - started < 600:
|
||||
time.sleep(15)
|
||||
status = api(args.base, f"/api/sessions/{sid}")
|
||||
state = json.loads(state_file.read_text("utf-8-sig"))
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "progress",
|
||||
"trace_id": sid,
|
||||
"status": status["status"],
|
||||
"elapsed": round(time.monotonic() - started),
|
||||
"applied": state["applied"],
|
||||
"finished": state["finished"],
|
||||
}
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
if status["status"] not in {
|
||||
"running",
|
||||
"pending",
|
||||
"queued",
|
||||
"stopping",
|
||||
"starting",
|
||||
}:
|
||||
done = True
|
||||
break
|
||||
if not done:
|
||||
api(args.base, "/api/stop", {"session_id": sid})
|
||||
time.sleep(3)
|
||||
elapsed = time.monotonic() - started
|
||||
result = collect(args.traces.resolve(), sid)
|
||||
state = json.loads(state_file.read_text("utf-8-sig"))
|
||||
attachment = (
|
||||
args.traces / sid / "artifacts/plugins" / plugin / "proof.png"
|
||||
)
|
||||
result.update(
|
||||
trace_id=sid,
|
||||
plugin=plugin,
|
||||
repetition=repetition + 1,
|
||||
fixture=state,
|
||||
observed_elapsed=elapsed,
|
||||
watchdog_timeout=not done,
|
||||
actual_pass=bool(
|
||||
state["applied"]
|
||||
and state["finished"]
|
||||
and state["target_clicks"] == 1
|
||||
and state["wrong_clicks"] == 0
|
||||
and attachment.exists()
|
||||
),
|
||||
attachment_exists=attachment.exists(),
|
||||
)
|
||||
(folder / "result.json").write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2), "utf-8"
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"event": "done",
|
||||
"trace_id": sid,
|
||||
"actual_pass": result["actual_pass"],
|
||||
}
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
# Release/recording cleanup belongs to ARTEMIS, wait before next fixture.
|
||||
for _ in range(15):
|
||||
if not api(args.base, "/api/status").get("active_tasks"):
|
||||
break
|
||||
time.sleep(2)
|
||||
finally:
|
||||
if not done:
|
||||
try:
|
||||
api(args.base, "/api/stop", {"session_id": sid})
|
||||
except (OSError, ValueError) as error:
|
||||
print(f"Cleanup request failed for {sid}: {error}", flush=True)
|
||||
if fixture.poll() is None:
|
||||
fixture.terminate()
|
||||
fixture.wait(10)
|
||||
stderr.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,4 +1,8 @@
|
||||
param([string]$Title, [string]$ReadyFile, [switch]$Cover, [int]$Width=500, [int]$Height=300)
|
||||
param([string]$Title, [string]$ReadyFile, [switch]$Cover, [int]$Width=500, [int]$Height=300, [switch]$PerMonitorDpi, [switch]$NormalWindow)
|
||||
if ($PerMonitorDpi) {
|
||||
Add-Type -TypeDefinition 'using System; using System.Runtime.InteropServices; public static class FixtureDpi { [DllImport("user32.dll")] public static extern IntPtr SetThreadDpiAwarenessContext(IntPtr c); }'
|
||||
[void][FixtureDpi]::SetThreadDpiAwarenessContext([IntPtr](-4))
|
||||
}
|
||||
Add-Type -AssemblyName System.Windows.Forms
|
||||
Add-Type -AssemblyName System.Drawing
|
||||
$form = New-Object System.Windows.Forms.Form
|
||||
@@ -6,7 +10,7 @@ $form.Text = $Title
|
||||
$form.Size = New-Object System.Drawing.Size($Width, $Height)
|
||||
$form.StartPosition = 'Manual'
|
||||
$form.Location = New-Object System.Drawing.Point(100, 100)
|
||||
$form.TopMost = $true
|
||||
$form.TopMost = -not $NormalWindow
|
||||
$form.BackColor = if ($Cover) { [System.Drawing.Color]::Magenta } else { [System.Drawing.Color]::Lime }
|
||||
$label = New-Object System.Windows.Forms.Label
|
||||
$label.Text = if ($Cover) { 'OCCLUDER - NOT TEST EVIDENCE' } else { 'ARTEMIS window capture fixture' }
|
||||
@@ -20,6 +24,8 @@ $button.Location = New-Object System.Drawing.Point(100, 100)
|
||||
$button.Size = New-Object System.Drawing.Size(180, 50)
|
||||
$button.BackColor = [System.Drawing.Color]::Red
|
||||
$button.Add_Click({ [System.IO.File]::WriteAllText(($ReadyFile + '.clicked'), 'clicked') })
|
||||
$button.Add_MouseDown({ param($sender, $event) [System.IO.File]::AppendAllText(($ReadyFile + '.mouse'), "down:$($event.X),$($event.Y)`n") })
|
||||
$button.Add_MouseUp({ param($sender, $event) [System.IO.File]::AppendAllText(($ReadyFile + '.mouse'), "up:$($event.X),$($event.Y)`n") })
|
||||
$form.Controls.Add($button)
|
||||
$timer = New-Object System.Windows.Forms.Timer
|
||||
$timer.Interval = 120000
|
||||
|
||||
@@ -0,0 +1,545 @@
|
||||
"""Opt-in Cua browser regression: real keyboard, form, scrolling and evidence.
|
||||
|
||||
Uses a disposable browser profile and a loopback event oracle, no CDP/DOM control.
|
||||
Only the launched browser process tree is cleaned up. No user tabs are changed.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import jsonschema
|
||||
import psutil
|
||||
from artemis.plugins.models import ServerConfig
|
||||
from artemis.plugins.session import PluginSession
|
||||
from artemis.runtime.device_lock import DeviceExecutionLock
|
||||
|
||||
|
||||
async def model_task(base, pid, url, output, events, traces, model):
|
||||
"""Opt-in paid model run through the user's existing idle ARTEMIS service."""
|
||||
sid = str(uuid.uuid4())
|
||||
submitted = done = False
|
||||
async with httpx.AsyncClient(base_url=base, trust_env=False, timeout=30) as client:
|
||||
|
||||
async def api(path, body=None):
|
||||
r = (
|
||||
await client.get(path)
|
||||
if body is None
|
||||
else await client.post(path, json=body)
|
||||
)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
status = await api("/api/status")
|
||||
assert not status.get("active_tasks") and not status.get("queue"), (
|
||||
"ARTEMIS is busy"
|
||||
)
|
||||
body = {
|
||||
"goal": f"仅操作 PID={pid} 的独立 Brave 测试窗口,不操作其他浏览器窗口或手机。新建一个标签页并通过地址栏打开 {url}/form。将 Test parameter 输入框填写为 ARTEMIS 中文参数,点击 Apply parameter 一次,核对页面显示 Saved: ARTEMIS 中文参数。然后向下滚动到页面底部,点击 Finish scrolling 一次,确认显示 Finished。把最终窗口截图保存到任务 trace,文件名 browser-model-proof.png。以实际页面证据判定结果。",
|
||||
"session_id": sid,
|
||||
"profile": "flash",
|
||||
"ingress": "client",
|
||||
"runtime_overrides": {
|
||||
"execution_target": "computer",
|
||||
"plugins": [
|
||||
{
|
||||
"id": "cua-driver",
|
||||
"profile": "benchmark",
|
||||
"version": "0.36.0+artemis.3",
|
||||
}
|
||||
],
|
||||
"model": model,
|
||||
"reasoning_effort": "high",
|
||||
"timeout_seconds": 300,
|
||||
"request_timeout_seconds": 90,
|
||||
"max_retries": 1,
|
||||
"record_video": False,
|
||||
"analyze_video": False,
|
||||
},
|
||||
}
|
||||
(output / "request.json").write_text(
|
||||
json.dumps(body, ensure_ascii=False, indent=2), "utf-8"
|
||||
)
|
||||
try:
|
||||
reply = await api("/api/run", body)
|
||||
assert reply.get("status") not in {"rejected", "error"}, reply
|
||||
submitted = True
|
||||
print(json.dumps({"trace_id": sid, "submission": reply}), flush=True)
|
||||
async with asyncio.timeout(420):
|
||||
while True:
|
||||
await asyncio.sleep(10)
|
||||
status = await api(f"/api/sessions/{sid}")
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"trace_id": sid,
|
||||
"status": status["status"],
|
||||
"saved": any(e["path"] == "/save" for e in events),
|
||||
"finished": any(e["path"] == "/finish" for e in events),
|
||||
}
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
if status["status"] not in {
|
||||
"running",
|
||||
"pending",
|
||||
"queued",
|
||||
"stopping",
|
||||
"starting",
|
||||
}:
|
||||
done = True
|
||||
break
|
||||
proof = (
|
||||
traces / sid / "artifacts/plugins/cua-driver/browser-model-proof.png"
|
||||
)
|
||||
report = {
|
||||
"trace_id": sid,
|
||||
"status": status["status"],
|
||||
"events": list(events),
|
||||
"actual_pass": (
|
||||
[e["body"] for e in events if e["path"] == "/save"]
|
||||
== ["ARTEMIS 中文参数"]
|
||||
and len([e for e in events if e["path"] == "/finish"]) == 1
|
||||
and any(
|
||||
e["path"] == "/scroll" and float(e["body"]) > 1000
|
||||
for e in events
|
||||
)
|
||||
and proof.is_file()
|
||||
),
|
||||
}
|
||||
(output / "model-report.json").write_text(
|
||||
json.dumps(report, ensure_ascii=False, indent=2), "utf-8"
|
||||
)
|
||||
assert report["actual_pass"], report
|
||||
print(json.dumps({"trace_id": sid, "actual_pass": True}), flush=True)
|
||||
finally:
|
||||
if submitted and not done:
|
||||
await api("/api/stop", {"session_id": sid})
|
||||
|
||||
|
||||
async def run(
|
||||
executable: Path,
|
||||
browser: Path,
|
||||
output: Path,
|
||||
base=None,
|
||||
traces=None,
|
||||
model="gpt-6.1-sol",
|
||||
foreground_check=False,
|
||||
):
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
events = []
|
||||
calls = []
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
def do_GET(self):
|
||||
events.append({"path": self.path})
|
||||
if self.path == "/form":
|
||||
body = """<!doctype html><meta charset="utf-8">
|
||||
<title>ARTEMIS Cua Browser Form</title>
|
||||
<style>body{font:24px sans-serif;margin:40px}input,button{font:inherit;padding:12px;display:block;margin:20px 0}</style>
|
||||
<h1>ARTEMIS Cua Browser Form</h1>
|
||||
<label>Test parameter<input aria-label="Test parameter" id="value"></label>
|
||||
<button aria-label="Apply parameter" onclick="fetch('/save',{method:'POST',body:document.getElementById('value').value}).then(()=>{document.getElementById('result').textContent='Saved: '+document.getElementById('value').value})">Apply parameter</button>
|
||||
<div id="result" aria-live="polite">Pending</div>
|
||||
<div style="height:1600px">Scroll down to finish</div>
|
||||
<button aria-label="Finish scrolling" onclick="fetch('/finish',{method:'POST',body:String(window.scrollY)}).then(()=>{this.textContent='Finished';this.setAttribute('aria-label','Finished');document.body.style.backgroundColor='#c4f9d0'})">Finish scrolling</button>
|
||||
<script>addEventListener('scroll',()=>fetch('/scroll',{method:'POST',body:String(window.scrollY)}))</script>
|
||||
"""
|
||||
else:
|
||||
body = "<title>ARTEMIS Cua Browser Start</title><h1>Isolated browser test</h1>"
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/html; charset=utf-8")
|
||||
self.end_headers()
|
||||
self.wfile.write(body.encode())
|
||||
|
||||
def do_POST(self):
|
||||
events.append(
|
||||
{
|
||||
"path": self.path,
|
||||
"body": self.rfile.read(
|
||||
int(self.headers["Content-Length"])
|
||||
).decode(),
|
||||
}
|
||||
)
|
||||
self.send_response(200)
|
||||
self.end_headers()
|
||||
|
||||
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
|
||||
threading.Thread(target=server.serve_forever, daemon=True).start()
|
||||
url = f"http://127.0.0.1:{server.server_port}"
|
||||
session = None
|
||||
process = cover_process = None
|
||||
try:
|
||||
with tempfile.TemporaryDirectory(prefix="artemis-cua-browser-") as profile:
|
||||
try:
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
str(browser),
|
||||
"--user-data-dir=" + profile,
|
||||
"--no-first-run",
|
||||
"--no-default-browser-check",
|
||||
"--disable-extensions",
|
||||
"--disable-background-mode",
|
||||
"--force-renderer-accessibility",
|
||||
"--window-size=1000,800",
|
||||
"--window-position=100,100",
|
||||
url + "/start",
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
creationflags=subprocess.CREATE_NO_WINDOW,
|
||||
)
|
||||
if base:
|
||||
await model_task(
|
||||
base, process.pid, url, output, events, traces, model
|
||||
)
|
||||
return
|
||||
session = await PluginSession(
|
||||
ServerConfig(
|
||||
command=str(executable),
|
||||
env={"ARTEMIS_ARTIFACTS_DIR": str(output / "attachments")},
|
||||
),
|
||||
executable.parent,
|
||||
).start()
|
||||
schemas = {t.name: t.inputSchema for t in session.tools}
|
||||
|
||||
async def call(name, args, *, invalid=False):
|
||||
if not invalid:
|
||||
jsonschema.validate(args, schemas[name])
|
||||
started = time.monotonic()
|
||||
result = await session.call(name, args)
|
||||
data = result.structuredContent or {}
|
||||
calls.append(
|
||||
{
|
||||
"tool": name,
|
||||
"arguments": args,
|
||||
"seconds": round(time.monotonic() - started, 3),
|
||||
"is_error": result.isError,
|
||||
"result": data,
|
||||
"text": [
|
||||
b.text for b in result.content if b.type == "text"
|
||||
],
|
||||
}
|
||||
)
|
||||
(output / "calls.json").write_text(
|
||||
json.dumps(calls, ensure_ascii=False, indent=2), "utf-8"
|
||||
)
|
||||
if not invalid:
|
||||
assert not result.isError, calls[-1]
|
||||
return data
|
||||
|
||||
async with asyncio.timeout(20):
|
||||
while True:
|
||||
listing = await call("list_windows", {"pid": process.pid})
|
||||
own = listing["windows"]
|
||||
if own:
|
||||
break
|
||||
await asyncio.sleep(0.2)
|
||||
target = {"pid": process.pid, "window_id": own[0]["window_id"]}
|
||||
|
||||
async def observe():
|
||||
return await call("get_window_state", target)
|
||||
|
||||
async def wait_tree(text):
|
||||
async with asyncio.timeout(10):
|
||||
while True:
|
||||
state = await observe()
|
||||
if text in state["tree_markdown"]:
|
||||
return state
|
||||
await asyncio.sleep(0.2)
|
||||
|
||||
async def key(key, modifiers=None):
|
||||
return await call(
|
||||
"press_key",
|
||||
{
|
||||
**target,
|
||||
"key": key,
|
||||
"modifiers": modifiers or [],
|
||||
"delivery_mode": "foreground",
|
||||
},
|
||||
)
|
||||
|
||||
def token(state, label, role="Button"):
|
||||
match = re.search(
|
||||
r"\[(\d+)\] " + re.escape(role) + ' "' + re.escape(label) + '"',
|
||||
state["tree_markdown"],
|
||||
)
|
||||
assert match, state["tree_markdown"]
|
||||
return state["snapshot_id"] + ":" + match[1]
|
||||
|
||||
if foreground_check:
|
||||
ready = Path(profile) / "cover-ready"
|
||||
cover_process = await asyncio.create_subprocess_exec(
|
||||
"powershell.exe",
|
||||
"-NoProfile",
|
||||
"-File",
|
||||
str(Path(__file__).with_name("capture-fixture.ps1")),
|
||||
"-Title",
|
||||
"ARTEMIS foreground handoff fixture",
|
||||
"-ReadyFile",
|
||||
str(ready),
|
||||
"-Cover",
|
||||
"-NormalWindow",
|
||||
"-Width",
|
||||
"1100",
|
||||
"-Height",
|
||||
"900",
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
creationflags=subprocess.CREATE_NO_WINDOW,
|
||||
)
|
||||
async with asyncio.timeout(20):
|
||||
while not ready.exists():
|
||||
await asyncio.sleep(0.1)
|
||||
cover_windows = await call(
|
||||
"list_windows", {"pid": cover_process.pid}
|
||||
)
|
||||
cover = {
|
||||
k: cover_windows["windows"][0][k] for k in ("pid", "window_id")
|
||||
}
|
||||
await call("bring_to_front", cover)
|
||||
hidden = await observe()
|
||||
assert not hidden["capture_focus"]["target_foreground_before"]
|
||||
assert not hidden["capture_focus"]["target_foreground_after"]
|
||||
# Explicit foreground input must not snap back to the cover.
|
||||
await key("l", ["ctrl"])
|
||||
await asyncio.sleep(5) # Past Cua's 3-second restore poller.
|
||||
focused = await observe()
|
||||
assert focused["capture_focus"]["target_foreground_after"]
|
||||
await key("escape")
|
||||
# Manual focus loss is reported, never silently corrected by
|
||||
# read-only capture. Existing bring_to_front recovers it.
|
||||
await call("bring_to_front", cover)
|
||||
hidden = await observe()
|
||||
assert not hidden["capture_focus"]["target_foreground_after"]
|
||||
await call("bring_to_front", target)
|
||||
await asyncio.sleep(5)
|
||||
focused = await observe()
|
||||
assert focused["capture_focus"]["target_foreground_before"]
|
||||
assert focused["capture_focus"]["target_foreground_after"]
|
||||
await call(
|
||||
"save_screenshot",
|
||||
{**target, "filename": "foreground-recovered.png"},
|
||||
)
|
||||
await wait_tree("ARTEMIS Cua Browser Start")
|
||||
# Reproduce task f1928... invalid form before any keyboard dispatch.
|
||||
bad = await call(
|
||||
"press_key",
|
||||
{
|
||||
**target,
|
||||
"key": "t",
|
||||
"modifiers": ["ctrl"],
|
||||
"element_token": "s00000001:9",
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
},
|
||||
invalid=True,
|
||||
)
|
||||
assert bad["artemis"]["execution"] == "not_started"
|
||||
assert bad["artemis"]["side_effect"] == "none"
|
||||
assert "observation" not in bad
|
||||
await key("t", ["ctrl"])
|
||||
state = await observe()
|
||||
assert "ARTEMIS Cua Browser Start" not in state.get("window_title", "")
|
||||
await key("l", ["ctrl"])
|
||||
await call(
|
||||
"type_text",
|
||||
{**target, "text": url + "/form", "delivery_mode": "foreground"},
|
||||
)
|
||||
await key("return")
|
||||
state = await wait_tree("Test parameter")
|
||||
if foreground_check:
|
||||
await asyncio.sleep(5)
|
||||
state = await observe()
|
||||
assert state["capture_focus"]["target_foreground_after"]
|
||||
assert "Test parameter" in state["tree_markdown"]
|
||||
# Direct UIA replacement is a separate, unambiguous tool.
|
||||
await call(
|
||||
"set_value",
|
||||
{
|
||||
**target,
|
||||
"element_token": token(state, "Test parameter", "Edit"),
|
||||
"value": "seed",
|
||||
},
|
||||
)
|
||||
state = await observe()
|
||||
await call(
|
||||
"click",
|
||||
{
|
||||
**target,
|
||||
"element_token": token(state, "Test parameter", "Edit"),
|
||||
"delivery_mode": "foreground",
|
||||
},
|
||||
)
|
||||
await key("a", ["ctrl"])
|
||||
await call(
|
||||
"type_text",
|
||||
{
|
||||
**target,
|
||||
"text": "ARTEMIS 中文参数",
|
||||
"delivery_mode": "foreground",
|
||||
},
|
||||
)
|
||||
state = await wait_tree("ARTEMIS 中文参数")
|
||||
await call(
|
||||
"click",
|
||||
{**target, "element_token": token(state, "Apply parameter")},
|
||||
)
|
||||
state = await wait_tree("Saved: ARTEMIS 中文参数")
|
||||
assert [e["body"] for e in events if e["path"] == "/save"] == [
|
||||
"ARTEMIS 中文参数"
|
||||
]
|
||||
source = state["artemis"]["source"]
|
||||
# Wheel ticks do not guarantee an exact pixel/page distance.
|
||||
# Observe each actual movement before another bounded scroll.
|
||||
for _ in range(3):
|
||||
before = sum(e["path"] == "/scroll" for e in events)
|
||||
await call(
|
||||
"scroll",
|
||||
{
|
||||
**target,
|
||||
"direction": "down",
|
||||
"amount": 8,
|
||||
"by": "page",
|
||||
"x": source["width"] // 2,
|
||||
"y": source["height"] // 2,
|
||||
"capture_id": state["capture_id"],
|
||||
"delivery_mode": "foreground",
|
||||
},
|
||||
)
|
||||
async with asyncio.timeout(5):
|
||||
while sum(e["path"] == "/scroll" for e in events) <= before:
|
||||
await asyncio.sleep(0.1)
|
||||
state = await observe()
|
||||
source = state["artemis"]["source"]
|
||||
if any(
|
||||
e["path"] == "/scroll" and float(e["body"]) > 1000
|
||||
for e in events
|
||||
):
|
||||
break
|
||||
assert any(
|
||||
e["path"] == "/scroll" and float(e["body"]) > 1000 for e in events
|
||||
)
|
||||
state = await observe()
|
||||
await call(
|
||||
"click",
|
||||
{
|
||||
**target,
|
||||
"element_token": token(state, "Finish scrolling"),
|
||||
"delivery_mode": "foreground",
|
||||
},
|
||||
)
|
||||
await wait_tree("Finished")
|
||||
assert len([e for e in events if e["path"] == "/finish"]) == 1
|
||||
saved = await call(
|
||||
"save_screenshot", {**target, "filename": "browser-proof.png"}
|
||||
)
|
||||
attachment = saved["artemis"]["attachments"][0]
|
||||
assert (
|
||||
hashlib.sha256(
|
||||
(output / "attachments" / attachment["name"]).read_bytes()
|
||||
).hexdigest()
|
||||
== attachment["sha256"]
|
||||
)
|
||||
report = {
|
||||
"passed": True,
|
||||
"foreground_handoff_and_readonly_recovery": foreground_check,
|
||||
"checks": [
|
||||
"mixed parameters refused before dispatch",
|
||||
"Ctrl+T new tab",
|
||||
"Ctrl+L URL navigation",
|
||||
"UIA set_value",
|
||||
"focused Unicode text input",
|
||||
"single form submission readback",
|
||||
"actual page scroll",
|
||||
"finish button",
|
||||
"PNG hash",
|
||||
],
|
||||
"events": events,
|
||||
"calls": len(calls),
|
||||
}
|
||||
(output / "report.json").write_text(
|
||||
json.dumps(report, ensure_ascii=False, indent=2), "utf-8"
|
||||
)
|
||||
print(json.dumps(report, ensure_ascii=False))
|
||||
finally:
|
||||
(output / "events.json").write_text(
|
||||
json.dumps(events, ensure_ascii=False, indent=2), "utf-8"
|
||||
)
|
||||
if session:
|
||||
await session.aclose()
|
||||
if cover_process and cover_process.returncode is None:
|
||||
cover_process.terminate()
|
||||
await cover_process.wait()
|
||||
if process:
|
||||
try:
|
||||
parent = psutil.Process(process.pid)
|
||||
owned = [*parent.children(recursive=True), parent]
|
||||
for proc in owned:
|
||||
try:
|
||||
proc.terminate()
|
||||
except psutil.NoSuchProcess:
|
||||
pass
|
||||
_, alive = psutil.wait_procs(owned, timeout=5)
|
||||
for proc in alive:
|
||||
proc.kill()
|
||||
psutil.wait_procs(alive, timeout=5)
|
||||
except psutil.NoSuchProcess:
|
||||
pass
|
||||
finally:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(__doc__)
|
||||
parser.add_argument("executable", type=Path)
|
||||
parser.add_argument("--browser", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument(
|
||||
"--model-base", help="Opt-in paid model test through an existing ARTEMIS server"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--traces", type=Path, help="That server's traces directory (model test only)"
|
||||
)
|
||||
parser.add_argument("--model", default="gpt-6.1-sol")
|
||||
parser.add_argument("--foreground-check", action="store_true")
|
||||
args = parser.parse_args()
|
||||
if args.model_base and not args.traces:
|
||||
parser.error("--model-base requires --traces")
|
||||
if args.foreground_check and args.model_base:
|
||||
parser.error("--foreground-check is deterministic, not a model task")
|
||||
lock = None
|
||||
if not args.model_base:
|
||||
lock = DeviceExecutionLock(
|
||||
"cua-browser-regression",
|
||||
description="Cua browser regression",
|
||||
resource_ids=["desktop:local"],
|
||||
)
|
||||
lock.acquire(timeout=0)
|
||||
try:
|
||||
asyncio.run(
|
||||
run(
|
||||
args.executable.resolve(),
|
||||
args.browser,
|
||||
args.output.resolve(),
|
||||
args.model_base,
|
||||
args.traces,
|
||||
args.model,
|
||||
args.foreground_check,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
if lock:
|
||||
lock.release()
|
||||
@@ -0,0 +1,88 @@
|
||||
"""Opt-in read-only app discovery against an already open, named application."""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import jsonschema
|
||||
from artemis.plugins.models import ServerConfig
|
||||
from artemis.plugins.session import PluginSession
|
||||
|
||||
|
||||
async def run(executable: Path, app: str, output: Path):
|
||||
session = await PluginSession(
|
||||
ServerConfig(command=str(executable)), executable.parent
|
||||
).start()
|
||||
try:
|
||||
schema = next(t.inputSchema for t in session.tools if t.name == "list_windows")
|
||||
query = {"app": app, "on_screen_only": True}
|
||||
bad_query = {**query, "pid": 0}
|
||||
try:
|
||||
jsonschema.validate(bad_query, schema)
|
||||
except jsonschema.ValidationError:
|
||||
pass
|
||||
else:
|
||||
raise AssertionError("Host schema still accepts PID 0")
|
||||
bad = await session.call("list_windows", bad_query)
|
||||
assert bad.isError
|
||||
assert bad.structuredContent["artemis"]["execution"] == "not_started"
|
||||
assert "omit pid" in "\n".join(b.text for b in bad.content if b.type == "text")
|
||||
|
||||
jsonschema.validate(query, schema)
|
||||
discovered = await session.call("list_windows", query)
|
||||
assert not discovered.isError
|
||||
windows = discovered.structuredContent["windows"]
|
||||
assert windows, "Open the intended app before running this opt-in probe"
|
||||
|
||||
nullable = {**query, "pid": None}
|
||||
jsonschema.validate(nullable, schema)
|
||||
found = await session.call("list_windows", nullable)
|
||||
assert not found.isError and found.structuredContent["windows"]
|
||||
assert "pid" not in found.structuredContent["lookup"]["filters"]
|
||||
|
||||
# This Python test process is not the requested application. A conflict
|
||||
# must stay empty and suggest a new app-only query, without a guessed PID.
|
||||
conflict = await session.call("list_windows", {**query, "pid": os.getpid()})
|
||||
assert not conflict.isError and conflict.structuredContent["windows"] == []
|
||||
next_query = conflict.structuredContent["lookup"]["suggested_arguments"]
|
||||
assert next_query == query
|
||||
assert "frontmost" not in "\n".join(
|
||||
b.text for b in conflict.content if b.type == "text"
|
||||
)
|
||||
recovered = await session.call("list_windows", next_query)
|
||||
assert not recovered.isError and recovered.structuredContent["windows"]
|
||||
|
||||
window = recovered.structuredContent["windows"][0]
|
||||
target = {k: window[k] for k in ("pid", "window_id")}
|
||||
state = await session.call("get_window_state", {**target, "max_elements": 100})
|
||||
assert not state.isError, state
|
||||
assert any(b.type == "image" for b in state.content)
|
||||
assert state.structuredContent["snapshot_id"]
|
||||
assert state.structuredContent["tree_markdown"]
|
||||
assert state.structuredContent["artemis"]["source"]["target_id"] == str(
|
||||
target["window_id"]
|
||||
)
|
||||
report = {
|
||||
"app": app,
|
||||
"zero_pid_rejected_by_host_and_adapter": True,
|
||||
"app_only_window_count": len(windows),
|
||||
"null_pid_omitted": True,
|
||||
"conflicting_pid_recovered_without_substitution": True,
|
||||
"screenshot_and_tree": True,
|
||||
}
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
output.write_text(json.dumps(report, ensure_ascii=False, indent=2), "utf-8")
|
||||
print(json.dumps(report, ensure_ascii=False))
|
||||
finally:
|
||||
await session.aclose()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(__doc__)
|
||||
parser.add_argument("executable", type=Path)
|
||||
parser.add_argument("--app", required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
asyncio.run(run(args.executable.resolve(), args.app, args.output))
|
||||
@@ -0,0 +1,118 @@
|
||||
"""Validate the benchmark's independent oracle and Cua foreground escalation."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from artemis.plugins.models import ServerConfig
|
||||
from artemis.plugins.session import PluginSession
|
||||
|
||||
|
||||
async def main(executable):
|
||||
with tempfile.TemporaryDirectory(
|
||||
prefix="artemis-benchmark-validation-"
|
||||
) as temporary:
|
||||
state_file = Path(temporary) / "state.json"
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"powershell.exe",
|
||||
"-NoProfile",
|
||||
"-File",
|
||||
str(Path(__file__).with_name("benchmark-fixture.ps1")),
|
||||
"-StateFile",
|
||||
str(state_file),
|
||||
creationflags=subprocess.CREATE_NO_WINDOW,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
session = None
|
||||
try:
|
||||
for _ in range(100):
|
||||
if state_file.exists():
|
||||
break
|
||||
await asyncio.sleep(0.1)
|
||||
assert state_file.exists()
|
||||
session = await PluginSession(
|
||||
ServerConfig(command=str(executable)), executable.parent
|
||||
).start()
|
||||
listing = await session.call("list_windows", {"pid": proc.pid})
|
||||
target = {
|
||||
"pid": proc.pid,
|
||||
"window_id": listing.structuredContent["windows"][0]["window_id"],
|
||||
}
|
||||
|
||||
async def element(label, tool, **extra):
|
||||
state = await session.call("get_window_state", target)
|
||||
text = "\n".join(getattr(c, "text", "") for c in state.content)
|
||||
match = re.search(r'\[(\d+)\][^\n]*"' + re.escape(label) + '"', text)
|
||||
assert match, text
|
||||
token = state.structuredContent["snapshot_id"] + ":" + match[1]
|
||||
result = await session.call(
|
||||
tool, {**target, "element_token": token, **extra}
|
||||
)
|
||||
assert not result.isError, result
|
||||
|
||||
await element("Network name", "set_value", value="ARTEMIS-LAB")
|
||||
await element("Night mode", "click")
|
||||
await element("Apply", "click")
|
||||
assert json.loads(state_file.read_text("utf-8-sig"))["applied"]
|
||||
state = await session.call("get_window_state", target)
|
||||
pixel = await session.call(
|
||||
"click_at",
|
||||
{
|
||||
**target,
|
||||
"x": 512,
|
||||
"y": 256,
|
||||
"capture_id": state.structuredContent["capture_id"],
|
||||
},
|
||||
)
|
||||
assert not pixel.isError, pixel
|
||||
scrolled = await session.call(
|
||||
"scroll",
|
||||
{
|
||||
**target,
|
||||
"direction": "down",
|
||||
"amount": 4,
|
||||
"by": "page",
|
||||
"x": 400,
|
||||
"y": 470,
|
||||
"capture_id": pixel.structuredContent["observation"]["capture_id"],
|
||||
},
|
||||
)
|
||||
if scrolled.isError:
|
||||
assert (
|
||||
scrolled.structuredContent["artemis"]["execution"] == "not_started"
|
||||
), scrolled
|
||||
state = await session.call("get_window_state", target)
|
||||
scrolled = await session.call(
|
||||
"scroll",
|
||||
{
|
||||
**target,
|
||||
"direction": "down",
|
||||
"amount": 4,
|
||||
"by": "page",
|
||||
"x": 400,
|
||||
"y": 470,
|
||||
"capture_id": state.structuredContent["capture_id"],
|
||||
"delivery_mode": "foreground",
|
||||
},
|
||||
)
|
||||
assert not scrolled.isError, scrolled
|
||||
await element("Finish", "click")
|
||||
result = json.loads(state_file.read_text("utf-8-sig"))
|
||||
print(json.dumps(result))
|
||||
assert (
|
||||
result["applied"] and result["finished"] and result["wrong_clicks"] == 0
|
||||
)
|
||||
finally:
|
||||
if session:
|
||||
await session.aclose()
|
||||
proc.terminate()
|
||||
await asyncio.wait_for(proc.wait(), 10)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main(Path(sys.argv[1]).resolve()))
|
||||
@@ -0,0 +1,163 @@
|
||||
"""Opt-in native-menu and Win+B regression, under the host desktop lease."""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import tempfile
|
||||
|
||||
from artemis.plugins.models import ServerConfig
|
||||
from artemis.plugins.session import PluginSession
|
||||
from artemis.runtime.device_lock import DeviceExecutionLock
|
||||
|
||||
|
||||
async def run(executable: Path, output: Path, taskbar: bool):
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
with tempfile.TemporaryDirectory(prefix="artemis-shell-fixture-") as folder:
|
||||
state_file = Path(folder) / "state.json"
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"powershell.exe",
|
||||
"-NoProfile",
|
||||
"-File",
|
||||
str(Path(__file__).with_name("shell-fixture.ps1")),
|
||||
"-StateFile",
|
||||
str(state_file),
|
||||
creationflags=subprocess.CREATE_NO_WINDOW,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
session = None
|
||||
calls = []
|
||||
try:
|
||||
async with asyncio.timeout(15):
|
||||
while not state_file.exists():
|
||||
if proc.returncode is not None:
|
||||
raise RuntimeError((await proc.stderr.read()).decode(errors="replace"))
|
||||
await asyncio.sleep(0.1)
|
||||
state = json.loads(state_file.read_text(encoding="utf-8-sig"))
|
||||
target = {"pid": state["pid"], "window_id": state["hwnd"]}
|
||||
session = await PluginSession(
|
||||
ServerConfig(command=str(executable)), executable.parent
|
||||
).start()
|
||||
|
||||
async def call(name, args):
|
||||
result = await session.call(name, args)
|
||||
calls.append(
|
||||
{
|
||||
"name": name,
|
||||
"arguments": args,
|
||||
"result": result.model_dump(mode="json", by_alias=True, exclude_none=True),
|
||||
}
|
||||
)
|
||||
assert not result.isError, "\n".join(
|
||||
block.text[:1500] for block in result.content if block.type == "text"
|
||||
)
|
||||
return result
|
||||
|
||||
def tree(result):
|
||||
return "\n".join(getattr(block, "text", "") for block in result.content)
|
||||
|
||||
result = await call("get_window_state", target)
|
||||
match = re.search(r'\[(\d+)\][^\n]*"Open native menu"', tree(result))
|
||||
assert match, tree(result)
|
||||
opened = await call(
|
||||
"click",
|
||||
{
|
||||
**target,
|
||||
"element_token": result.structuredContent["snapshot_id"] + ":" + match[1],
|
||||
"count": 1,
|
||||
"delivery_mode": "foreground",
|
||||
},
|
||||
)
|
||||
observation = opened.structuredContent["observation_target"]
|
||||
menu = observation["observed"]
|
||||
assert menu["window_id"] != target["window_id"], observation
|
||||
assert observation["reason"] == "associated_native_keyboard_focus", observation
|
||||
assert opened.structuredContent["artemis"]["source"]["target_id"] == str(
|
||||
menu["window_id"]
|
||||
)
|
||||
observed = await call("get_window_state", menu)
|
||||
assert "Increment once" in tree(observed), tree(observed)
|
||||
png = next(block for block in observed.content if block.type == "image")
|
||||
image = base64.b64decode(png.data, validate=True)
|
||||
(output / "native-menu.png").write_bytes(image)
|
||||
listed = await call("list_windows", {"pid": proc.pid})
|
||||
assert any(
|
||||
window["window_id"] == menu["window_id"] and window.get("window_kind") == "menu"
|
||||
for window in listed.structuredContent["windows"]
|
||||
)
|
||||
# Escape belongs to the owner queue; no fixture or user business action.
|
||||
await call("press_key", {**target, "key": "escape", "delivery_mode": "foreground"})
|
||||
final_state = json.loads(state_file.read_text(encoding="utf-8-sig"))
|
||||
assert final_state["selected_count"] == 0, final_state
|
||||
assert final_state["menu_attempts"] == 1, final_state
|
||||
|
||||
shell_verified = False
|
||||
if taskbar:
|
||||
windows = await call("list_windows", {"app": "explorer", "on_screen_only": True})
|
||||
desktop = next(
|
||||
window
|
||||
for window in windows.structuredContent["windows"]
|
||||
if window.get("class_name") == "Progman"
|
||||
)
|
||||
desktop = {key: desktop[key] for key in ("pid", "window_id")}
|
||||
await call("get_window_state", desktop)
|
||||
key_result = await call(
|
||||
"press_key",
|
||||
{**desktop, "key": "b", "modifiers": ["win"], "delivery_mode": "foreground"},
|
||||
)
|
||||
switched = key_result.structuredContent["observation_target"]
|
||||
assert switched["observed"]["window_id"] != desktop["window_id"], switched
|
||||
assert switched["reason"] == "associated_native_keyboard_focus", switched
|
||||
assert key_result.structuredContent["post_action_focus"]["focus_root"][
|
||||
"class_name"
|
||||
] in {"Shell_TrayWnd", "Shell_SecondaryTrayWnd"}
|
||||
assert key_result.structuredContent["artemis"]["source"]["target_id"] == str(
|
||||
switched["observed"]["window_id"]
|
||||
)
|
||||
shell_verified = True
|
||||
await call("bring_to_front", target)
|
||||
report = {
|
||||
"native_menu_discovered_and_captured": True,
|
||||
"menu_source_matches_actual_hwnd": True,
|
||||
"menu_png_sha256": hashlib.sha256(image).hexdigest(),
|
||||
"win_b_focus_and_capture": shell_verified,
|
||||
"fixture_business_submissions": 0,
|
||||
"calls": len(calls),
|
||||
}
|
||||
(output / "summary.json").write_text(json.dumps(report, indent=2), encoding="utf-8")
|
||||
print(json.dumps(report))
|
||||
finally:
|
||||
# Save only this fixture's protocol evidence; no pairing payloads.
|
||||
(output / "calls.json").write_text(
|
||||
json.dumps(calls, ensure_ascii=False, indent=2), encoding="utf-8"
|
||||
)
|
||||
if session:
|
||||
await session.aclose()
|
||||
if proc.returncode is None:
|
||||
proc.terminate()
|
||||
await proc.wait()
|
||||
if state_file.exists():
|
||||
(output / "fixture-state.json").write_text(
|
||||
state_file.read_text(encoding="utf-8-sig"), encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(__doc__)
|
||||
parser.add_argument("executable", type=Path)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--taskbar", action="store_true")
|
||||
args = parser.parse_args()
|
||||
lock = DeviceExecutionLock(
|
||||
"cua-shell-regression", description="Cua shell regression", resource_ids=["desktop:local"]
|
||||
)
|
||||
lock.acquire(timeout=0)
|
||||
try:
|
||||
asyncio.run(run(args.executable.resolve(), args.output.resolve(), args.taskbar))
|
||||
finally:
|
||||
lock.release()
|
||||
@@ -0,0 +1,141 @@
|
||||
"""Opt-in actual HWND capture, action, artifact, recording and shutdown proof."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
from artemis.config.agent import RecordingConfig
|
||||
from artemis.plugins.evidence import (
|
||||
decode_receipt,
|
||||
persist_attachments,
|
||||
source_identity,
|
||||
validate_tools,
|
||||
)
|
||||
from artemis.plugins.models import PluginCallResult, PluginManifest, ServerConfig
|
||||
from artemis.plugins.recording import PluginRecordings
|
||||
from artemis.plugins.session import PluginSession
|
||||
|
||||
|
||||
async def run(executable, output):
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
with tempfile.TemporaryDirectory(prefix="artemis-cua-smoke-") as temporary:
|
||||
ready = Path(temporary) / "ready"
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"powershell.exe",
|
||||
"-NoProfile",
|
||||
"-File",
|
||||
str(Path(__file__).with_name("capture-fixture.ps1")),
|
||||
"-Title",
|
||||
"ARTEMIS Cua Smoke",
|
||||
"-ReadyFile",
|
||||
str(ready),
|
||||
"-PerMonitorDpi",
|
||||
creationflags=subprocess.CREATE_NO_WINDOW,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
session = recorder = None
|
||||
try:
|
||||
for _ in range(100):
|
||||
if ready.exists():
|
||||
break
|
||||
await asyncio.sleep(0.1)
|
||||
assert ready.exists()
|
||||
artifacts = output / "artifacts/plugins/cua-driver"
|
||||
session = await PluginSession(
|
||||
ServerConfig(
|
||||
command=str(executable),
|
||||
env={"ARTEMIS_ARTIFACTS_DIR": str(artifacts)},
|
||||
),
|
||||
executable.parent,
|
||||
).start()
|
||||
manifest = PluginManifest.model_validate_json(
|
||||
(
|
||||
Path(__file__).parents[1] / "plugins/cua-driver/plugin.json"
|
||||
).read_text("utf-8")
|
||||
)
|
||||
validate_tools(manifest, session.tools)
|
||||
windows = await session.call("list_windows", {"pid": proc.pid})
|
||||
target = {
|
||||
"pid": proc.pid,
|
||||
"window_id": windows.structuredContent["windows"][0]["window_id"],
|
||||
}
|
||||
raw = await session.call("get_window_state", target)
|
||||
assert not raw.isError, raw
|
||||
outcome = PluginCallResult(
|
||||
status="success", structured_content=raw.structuredContent
|
||||
)
|
||||
decode_receipt(outcome)
|
||||
assert outcome.source and outcome.source.width > 0
|
||||
snapshot = raw.structuredContent["snapshot_id"]
|
||||
clicked = await session.call(
|
||||
"click", {**target, "element_token": f"{snapshot}:0"}
|
||||
)
|
||||
assert not clicked.isError, clicked
|
||||
assert Path(str(ready) + ".clicked").exists()
|
||||
assert clicked.structuredContent["artemis"]["source"]
|
||||
assert clicked.structuredContent["observation"]["snapshot_id"] != snapshot
|
||||
saved = await session.call(
|
||||
"save_screenshot", {**target, "filename": "cua-proof.png"}
|
||||
)
|
||||
assert not saved.isError, saved
|
||||
receipt = saved.structuredContent["artemis"]
|
||||
files = persist_attachments(output, "cua-driver", receipt["attachments"])
|
||||
duplicate = await session.call(
|
||||
"save_screenshot", {**target, "filename": "cua-proof.png"}
|
||||
)
|
||||
assert (
|
||||
duplicate.isError
|
||||
and duplicate.structuredContent["artemis"]["execution"] == "not_started"
|
||||
)
|
||||
context = SimpleNamespace(
|
||||
data_engine=SimpleNamespace(
|
||||
base_dir=output, session_start_time=time.time()
|
||||
),
|
||||
agent_config=SimpleNamespace(
|
||||
recording=RecordingConfig(codec="h264", computer_scope="window")
|
||||
),
|
||||
execution_setup=SimpleNamespace(
|
||||
record_video_enabled=True, video_recording_tools_enabled=False
|
||||
),
|
||||
)
|
||||
recorder = PluginRecordings(context)
|
||||
await recorder.observe(
|
||||
source_identity("cua-driver", outcome.source),
|
||||
"cua-driver",
|
||||
outcome.source,
|
||||
)
|
||||
assert recorder.active
|
||||
await asyncio.sleep(5)
|
||||
await recorder.stop()
|
||||
clips = [
|
||||
s for track in recorder.sources.values() for s in track["segments"]
|
||||
]
|
||||
assert clips and clips[0]["duration"] >= 3
|
||||
result = {
|
||||
"actual_click": True,
|
||||
"source": outcome.source.model_dump(),
|
||||
"attachments": files,
|
||||
"recordings": clips,
|
||||
}
|
||||
(output / "result.json").write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2), "utf-8"
|
||||
)
|
||||
print(json.dumps(result, ensure_ascii=False))
|
||||
finally:
|
||||
if recorder:
|
||||
await recorder.stop()
|
||||
if session:
|
||||
await session.aclose()
|
||||
proc.terminate()
|
||||
await asyncio.wait_for(proc.wait(), 10)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
asyncio.run(run(Path(sys.argv[1]).resolve(), Path(sys.argv[2]).resolve()))
|
||||
@@ -0,0 +1,51 @@
|
||||
param([Parameter(Mandatory = $true)][string]$StateFile)
|
||||
$ErrorActionPreference = 'Stop'
|
||||
Add-Type -AssemblyName System.Windows.Forms
|
||||
Add-Type -AssemblyName System.Drawing
|
||||
Add-Type -ReferencedAssemblies System.Windows.Forms,System.Drawing,System.Web.Extensions @'
|
||||
using System;
|
||||
using System.IO;
|
||||
using System.Collections.Generic;
|
||||
using System.Diagnostics;
|
||||
using System.Runtime.InteropServices;
|
||||
using System.Web.Script.Serialization;
|
||||
using System.Windows.Forms;
|
||||
using System.Drawing;
|
||||
public sealed class ArtemisMenuFixture : Form {
|
||||
[DllImport("user32.dll")] static extern IntPtr CreatePopupMenu();
|
||||
[DllImport("user32.dll", CharSet=CharSet.Unicode)] static extern bool AppendMenuW(IntPtr menu, uint flags, UIntPtr id, string text);
|
||||
[DllImport("user32.dll")] static extern uint TrackPopupMenu(IntPtr menu, uint flags, int x, int y, int reserved, IntPtr owner, IntPtr rect);
|
||||
[DllImport("user32.dll")] static extern bool DestroyMenu(IntPtr menu);
|
||||
readonly string stateFile;
|
||||
int count, attempts;
|
||||
long result = -1;
|
||||
string error = "";
|
||||
public ArtemisMenuFixture(string path) {
|
||||
stateFile = path;
|
||||
Text = "ARTEMIS native menu fixture";
|
||||
Size = new Size(560, 360);
|
||||
StartPosition = FormStartPosition.CenterScreen;
|
||||
var button = new Button { Text="Open native menu", Location=new Point(30,30), Size=new Size(240,55) };
|
||||
button.Click += delegate {
|
||||
var menu = CreatePopupMenu();
|
||||
try {
|
||||
attempts++;
|
||||
Save();
|
||||
if (!AppendMenuW(menu, 0, new UIntPtr(1), "Increment once")) throw new Exception("AppendMenuW failed");
|
||||
result = TrackPopupMenu(menu, 0x100, Left+45, Top+140, 0, Handle, IntPtr.Zero);
|
||||
if (result == 1) count++;
|
||||
} catch (Exception ex) { error = ex.Message; }
|
||||
finally { DestroyMenu(menu); Save(); }
|
||||
};
|
||||
Controls.Add(button);
|
||||
Shown += delegate { Save(); };
|
||||
}
|
||||
void Save() {
|
||||
File.WriteAllText(stateFile, new JavaScriptSerializer().Serialize(new Dictionary<string,object> {
|
||||
{"pid",Process.GetCurrentProcess().Id}, {"hwnd",Handle.ToInt64()}, {"selected_count",count},
|
||||
{"menu_attempts",attempts}, {"menu_result",result}, {"menu_error",error}
|
||||
}));
|
||||
}
|
||||
}
|
||||
'@
|
||||
[System.Windows.Forms.Application]::Run([ArtemisMenuFixture]::new($StateFile))
|
||||
@@ -0,0 +1,208 @@
|
||||
"""Summarize measured trace usage; never infer token counts from text length."""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import sqlite3
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
from statistics import mean
|
||||
|
||||
|
||||
def enrich(row, data, traces):
|
||||
"""Read actual evidence separately; the fixture never feeds the agent."""
|
||||
session_root = traces / row["trace_id"]
|
||||
outcome = session_root / "run_outcome.json"
|
||||
if outcome.exists():
|
||||
value = json.loads(outcome.read_text("utf-8"))
|
||||
row["ai_verdict"] = value.get("test_result", {}).get("status")
|
||||
row["ai_verdict_reason"] = value.get("test_result", {}).get("reason")
|
||||
row["outcome_code"] = value.get("code")
|
||||
recordings = session_root / "evidence/recordings/recordings.json"
|
||||
row["recordings"] = []
|
||||
if recordings.exists():
|
||||
for source in json.loads(recordings.read_text("utf-8")).get("sources", []):
|
||||
row["recordings"].append(
|
||||
{
|
||||
"scope": source.get("scope"),
|
||||
"status": source.get("status"),
|
||||
"area": source.get("area"),
|
||||
"segments": [
|
||||
{
|
||||
**segment,
|
||||
"exists": (session_root / segment["file"]).is_file(),
|
||||
}
|
||||
for segment in source.get("segments", [])
|
||||
],
|
||||
}
|
||||
)
|
||||
proof = session_root / "artifacts/plugins" / row["plugin"] / "proof.png"
|
||||
row["proof_sha256"] = (
|
||||
hashlib.sha256(proof.read_bytes()).hexdigest() if proof.is_file() else None
|
||||
)
|
||||
start = data["session"]["start_time"]
|
||||
finish = next(
|
||||
(e["at"] / 1000 for e in data["fixture"]["events"] if e["event"] == "finish"),
|
||||
None,
|
||||
)
|
||||
row["fixture_finish_seconds"] = (
|
||||
round(finish - start, 3) if finish is not None else None
|
||||
)
|
||||
row["evidence_ready_seconds"] = (
|
||||
round(max(finish, proof.stat().st_mtime) - start, 3)
|
||||
if finish is not None and proof.is_file() and row["actual_pass"]
|
||||
else None
|
||||
)
|
||||
row["post_evidence_seconds"] = (
|
||||
round(row["seconds"] - row["evidence_ready_seconds"], 3)
|
||||
if row["evidence_ready_seconds"] is not None
|
||||
else None
|
||||
)
|
||||
with sqlite3.connect(
|
||||
f"file:{(traces / 'data_engine.db').as_posix()}?mode=ro", uri=True
|
||||
) as db:
|
||||
errors = []
|
||||
observed_plugins = set()
|
||||
for status, payload in db.execute(
|
||||
"SELECT status,payload FROM traces WHERE session_id=? AND type='tool' ORDER BY timestamp",
|
||||
(row["trace_id"],),
|
||||
):
|
||||
value = json.loads(payload or "{}")
|
||||
if value.get("plugin_id"):
|
||||
observed_plugins.add(value["plugin_id"])
|
||||
if value.get("plugin_id") and status != "success":
|
||||
# Keep the diagnostic first line, not private window inventory.
|
||||
message = str(value.get("error", "")).splitlines()
|
||||
errors.append(
|
||||
{
|
||||
"plugin": value["plugin_id"],
|
||||
"tool": value.get("tool_name"),
|
||||
"message": message[0] if message else "",
|
||||
}
|
||||
)
|
||||
row["tool_errors"] = errors
|
||||
row["observed_plugins"] = sorted(observed_plugins)
|
||||
|
||||
|
||||
def summarize(root, traces=None):
|
||||
rows = []
|
||||
for path in sorted(root.glob("*-*/result.json")):
|
||||
data = json.loads(path.read_text("utf-8"))
|
||||
usage = data["llm_usage"]
|
||||
calls = data["plugin_calls"]
|
||||
session = data["session"]
|
||||
requests = data.get("requests", [])
|
||||
usage_by_node = defaultdict(
|
||||
lambda: {"calls": 0, "input": 0, "cached": 0, "output": 0}
|
||||
)
|
||||
models_from_usage = Counter()
|
||||
for item in usage:
|
||||
source_parts = item.get("source", "").split(":")
|
||||
model_index = 2 if source_parts[0] == "lens" else 1
|
||||
model = (
|
||||
source_parts[model_index]
|
||||
if len(source_parts) > model_index
|
||||
else "unknown"
|
||||
)
|
||||
models_from_usage[model] += 1
|
||||
node = usage_by_node[item.get("node", "unknown")]
|
||||
node["calls"] += 1
|
||||
node["input"] += item.get("prompt_tokens", 0)
|
||||
node["cached"] += item.get("cached_tokens", 0)
|
||||
node["output"] += item.get("completion_tokens", 0)
|
||||
models = Counter(
|
||||
p.get("model", "unknown") for p in requests if p.get("phase") == "completed"
|
||||
)
|
||||
input_tokens = sum(u.get("prompt_tokens", 0) for u in usage)
|
||||
cached = sum(u.get("cached_tokens", 0) for u in usage)
|
||||
output_tokens = sum(u.get("completion_tokens", 0) for u in usage)
|
||||
duration = (session.get("end_time") or session["start_time"]) - session[
|
||||
"start_time"
|
||||
]
|
||||
row = {
|
||||
"plugin": data["plugin"],
|
||||
"round": data["repetition"],
|
||||
"trace_id": data["trace_id"],
|
||||
"actual_pass": data["actual_pass"],
|
||||
"execution_status": session["status"],
|
||||
"input_tokens": input_tokens,
|
||||
"cached_tokens": cached,
|
||||
"uncached_input": input_tokens - cached,
|
||||
"output_tokens": output_tokens,
|
||||
"total_tokens": input_tokens + output_tokens,
|
||||
"cache_rate": cached / input_tokens if input_tokens else None,
|
||||
"llm_calls": len(usage),
|
||||
"models": dict(models),
|
||||
"models_from_usage": dict(models_from_usage),
|
||||
"usage_by_node": dict(usage_by_node),
|
||||
"request_phases": dict(Counter(p.get("phase") for p in requests)),
|
||||
"request_seconds_by_role": {
|
||||
role: round(
|
||||
sum(
|
||||
p.get("elapsed_seconds", 0)
|
||||
for p in requests
|
||||
if p.get("role") == role
|
||||
),
|
||||
3,
|
||||
)
|
||||
for role in sorted({p.get("role", "unknown") for p in requests})
|
||||
},
|
||||
"fallback_requests": sum(bool(p.get("is_fallback")) for p in requests),
|
||||
"retried_requests": sum((p.get("attempt") or 1) > 1 for p in requests),
|
||||
"executor_text_only_responses": sum(
|
||||
p.get("role") == "FlashRunner"
|
||||
and p.get("phase") == "completed"
|
||||
and not p.get("tools")
|
||||
for p in requests
|
||||
),
|
||||
"seconds": round(duration, 3),
|
||||
"plugin_calls": len(calls),
|
||||
"plugin_seconds": round(sum(c.get("duration") or 0 for c in calls), 3),
|
||||
"plugin_errors": sum(c["status"] != "success" for c in calls),
|
||||
"scroll_calls": sum(c["name"] == "scroll" for c in calls),
|
||||
"fixture": data["fixture"],
|
||||
}
|
||||
if traces is not None:
|
||||
enrich(row, data, traces)
|
||||
rows.append(row)
|
||||
aggregates = {}
|
||||
for plugin in sorted({r["plugin"] for r in rows}):
|
||||
group = [r for r in rows if r["plugin"] == plugin]
|
||||
aggregate = {"runs": len(group), "passed": sum(r["actual_pass"] for r in group)}
|
||||
for key in (
|
||||
"input_tokens",
|
||||
"cached_tokens",
|
||||
"uncached_input",
|
||||
"output_tokens",
|
||||
"total_tokens",
|
||||
"seconds",
|
||||
"llm_calls",
|
||||
"plugin_calls",
|
||||
"plugin_seconds",
|
||||
"plugin_errors",
|
||||
"scroll_calls",
|
||||
):
|
||||
aggregate[f"mean_{key}"] = round(mean(r[key] for r in group), 3)
|
||||
aggregate["cache_rate"] = sum(r["cached_tokens"] for r in group) / max(
|
||||
1, sum(r["input_tokens"] for r in group)
|
||||
)
|
||||
aggregates[plugin] = aggregate
|
||||
report = {"runs": rows, "aggregate": aggregates}
|
||||
(root / "summary.json").write_text(
|
||||
json.dumps(report, ensure_ascii=False, indent=2), "utf-8"
|
||||
)
|
||||
return report
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(__doc__)
|
||||
parser.add_argument("directory", type=Path)
|
||||
parser.add_argument("--traces", type=Path)
|
||||
options = parser.parse_args()
|
||||
print(
|
||||
json.dumps(
|
||||
summarize(options.directory, options.traces)["aggregate"],
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
Reference in new issue
Block a user