fix: keep Cua foreground actions on the target window
This commit is contained in:
chenmingxuan committed
1 parent
889a951c9b
commit
6de0b1b276
13 files changed
+318
-22
No files matched your search
@@ -1,6 +1,6 @@
|
||||
# Artemis Plugins
|
||||
|
||||
ARTEMIS 2.0.0 的插件市场。当前电脑控制插件为 **Cua 0.36.0+artemis.1**，支持 Windows x64，通过窗口截图与可访问性树操作电脑应用和浏览器窗口，复用 ARTEMIS 的调度、资源互斥、步骤、截图附件和录像。
|
||||
ARTEMIS 2.0.0 的插件市场。当前电脑控制插件为 **Cua 0.36.0+artemis.4**，支持 Windows x64，通过窗口截图与可访问性树操作电脑应用和浏览器窗口，复用 ARTEMIS 的调度、资源互斥、步骤、截图附件和录像。
|
||||
|
||||
在 ARTEMIS 的「插件」页面安装「电脑控制 · Cua」，授权并启用即可。安卓任务可根据提示词按需调用电脑控制，不需要单独配置 Codex MCP 或安装 Go、Node、Python。
|
||||
|
||||
@@ -16,6 +16,8 @@ Open Computer Use（OCU）已从市场索引下架，当前市场只提供 Cua
|
||||
|
||||
已测试原生窗口表单、34 像素视觉目标、滚动、截图保存、ARTEMIS 窗口录像和子进程退出。对照测试独立验收 3/3 完成；ARTEMIS AI 裁定为 2 次通过、1 次未判定。浏览器复杂控制台、多 DPI、多显示器与安卓混合长程任务仍需进一步验证，详见 [验证记录](plugins/cua-driver/VERIFICATION.md)。
|
||||
|
||||
适配修订另覆盖应用名发现、浏览器键盘与中文输入、截图缩放与坐标校验，以及前台激活后保持目标可见。Cua 需要前台输入时会占用用户桌面，不提供与用户操作互不干扰的执行环境；复杂业务是否通过仍以任务实际证据为准。
|
||||
|
||||
## 构建与验证
|
||||
|
||||
构建机需要 Python 和 Go；最终用户只安装 ZIP。从固定上游发行下载 `cua-driver-rs-0.36.0-windows-x86_64-binary.zip`，构建脚本校验 SHA256 后打包，不执行全局安装器。
|
||||
@@ -31,6 +33,6 @@ python tests/cua_discovery_smoke.py <安装后的artemis-cua-adapter.exe> --app
|
||||
|
||||
`catalog.json` 是市场索引，包含清单、HTTPS 安装包地址、SHA256 和验证说明。`artifacts/` 保存本机安装包与测试资料，不进入 Git。版本文件不可原地覆盖，任务固定使用其启动时的版本。
|
||||
|
||||
插件版本跟随所用 Cua Driver 上游版本：当前为 `0.36.0`，对应上游标签 `cua-driver-rs-v0.36.0`，当前适配修订为 `0.36.0+artemis.1`，本仓库发行标签为 `cua-driver-v0.36.0+artemis.1`。只修改 ARTEMIS 适配而未升级上游时使用 `+artemis.N` 后缀区分不可变安装包；包内 `provenance.json` 记录上游提交、原包校验值和适配修订，构建会检查版本对应关系。
|
||||
插件版本跟随所用 Cua Driver 上游版本：当前为 `0.36.0`，对应上游标签 `cua-driver-rs-v0.36.0`，当前适配修订为 `0.36.0+artemis.4`，本仓库发行标签为 `cua-driver-v0.36.0+artemis.4`。只修改 ARTEMIS 适配而未升级上游时使用 `+artemis.N` 后缀区分不可变安装包；包内 `provenance.json` 记录上游提交、原包校验值和适配修订，构建会检查版本对应关系。
|
||||
|
||||
[发行说明](plugins/cua-driver/RELEASE.md) · [ARTEMIS 主项目](https://git.playones.com/chenmingxuan/Artemis-Eapil)
|
||||
@@ -1,14 +1,16 @@
|
||||
# Cua 电脑控制 0.36.0+artemis.3
|
||||
# Cua 电脑控制 0.36.0+artemis.4
|
||||
|
||||
作为 ARTEMIS 插件市场的默认电脑控制实现，替代已下架的 Open Computer Use。此修订修复缩放截图后的滚动坐标转换，加入截图编号及范围校验，保留上一修订的键盘参数修复；上游仍是固定的 Cua Driver 0.36.0 预发布版，市场标注开发预览。
|
||||
此修订修复前台操作完成后立即恢复其他应用、后续浏览器观察退回后台的问题。上游仍固定 Cua Driver 0.36.0 预发布版；仅增加 ARTEMIS 适配修订，没有接入独立环境、DOM 控制或新的后台通道。
|
||||
|
||||
本修订：上游 Windows 0.36.0 的滚动路径没有应用截图缩放比，适配器现按实际原始截图尺寸转换；每次滚动前仅内部读取原始截图（不读取 UIA、不发送给模型、不增加模型调用），避免推算 DPI 或假定固定裁边。点击和拖拽继续由上游转换，避免重复缩放。所有像素操作要求最新 capture_id，过期、越界或窗口移动/缩放时拒绝派发。截图编号和尺寸置于文字结果开头，避免被长控件树挤出上下文。Windows 几何读取在锁定 OS 线程后设置并还原 DPI 上下文。
|
||||
明确使用 foreground 的操作先调用上游自带 bring_to_front，核实目标激活后再发送一次输入。目标保持前台供后续渲染、观察和录像，不逐次切回原应用。激活失败或结果未知时不发送业务输入；写操作结果未知时不重发。独立 bring_to_front 工具可恢复被遮挡的窗口，不必刷新页面或重复业务动作。只读观察不抢焦点，capture_focus 记录采集前后的前台 HWND 和目标状态，供识别灰屏与焦点切换。PNG 保存成功仍不代表网页验收通过。
|
||||
|
||||
保留前一修订的滚动缩放、capture_id、范围与窗口几何校验，以及键盘参数修复。原生后台操作继续遵循 Cua 支持范围；foreground 会占用用户前台，不承诺用户与模型可同时操作同一桌面。
|
||||
|
||||
上一修订内容：`press_key` / `type_text` 只对目标窗口的当前焦点操作，不再暴露互斥的 element_token/x/y 参数；聚焦使用独立点击工具，控件赋值由必填 element_token 的 `set_value` 完成。`scroll` 仅按截图坐标定位滚动区域。旧格式在适配层返回明确未执行回执，不发送输入、不重新截图。补充浏览器前台输入与滚动后点击规则，不自动重试结果不确定的写操作。保留上一修订的应用名发现和无效 PID 修复。
|
||||
|
||||
上游提交：`6d6c734c127025fb39ad5d585a10382808b39dc1`。Windows x64 原始 ZIP SHA256：`f8c47024b626f4e0e70f8428d2998df39e4879110c5722a29d5fd03e367353ad`。采用 MIT Driver，许可证随包提供；不包含 Cua Spaces、云环境或模型服务。
|
||||
|
||||
提供 11 个窗口观察、控件/像素点击、输入、按键、拖动、滚动及截图工具。操作后返回真实窗口观察，保留动作回执，截图校验 PID/HWND 与物理边界，生成文件保存到当前任务 trace。共享 ARTEMIS 的 `desktop:local` 资源、任务固定版本、步骤和录像，不另建调度器。
|
||||
提供 12 个窗口激活、观察、控件/像素点击、输入、按键、拖动、滚动及截图工具。操作后返回真实窗口观察，保留动作回执，截图校验 PID/HWND 与物理边界，生成文件保存到当前任务 trace。共享 ARTEMIS 的 `desktop:local` 资源、任务固定版本、步骤和录像，不另建调度器。
|
||||
|
||||
适配器与依赖随包分发，无需额外安装 Go、Python、Node 或另一份 Codex MCP。关闭、取消或宿主退出后通过 Windows Job Object 回收 Driver/UIA 子进程；仅在该子进程环境关闭遥测。
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
2. `get_window_state` 同时读取截图和 UIA 控件树。最新 `snapshot_id` 加 `:N` 即控件 `element_token`。优先用 `click` 点击最新控件；视觉目标用 `click_at`，坐标仅取本次截图上的窗口局部像素，不能用屏幕坐标，需携带本次 `capture_id`。两种点击参数互斥，不混用。
|
||||
3. 所有窗口操作明确传 `pid`、`window_id`。`set_value` 必须带最新 `element_token`，用于设置具有 set_value 动作的输入框；同名 Text 标签不能当成 Edit 输入框。`press_key` 和 `type_text` 只操作当前焦点，不能传 `element_token`、`x`、`y`（也不要补 0/null）。需要聚焦控件时先 `click` / `click_at`，再输入。`press_key` 传单键及 `modifiers` 数组。
|
||||
4. 每个动作后插件自动返回一次新截图和控件树，新 snapshot 替换旧 token。接着用新 token，无需重复获取相同状态。仅当画面确实还在变化或需要核验时再次观察。一次调用只执行一次动作，超时不自动重发。
|
||||
5. 默认后台操作；Brave/Chrome/Edge 等 Chromium 浏览器的点击（尤其滚动后）、快捷键、文字输入和滚动直接使用 `delivery_mode=foreground`。例如新标签页：`press_key(key="t",modifiers=["ctrl"],delivery_mode="foreground")`；地址导航：Ctrl+L → `type_text(text=网址,delivery_mode="foreground")` → Enter，各调用均带同一目标 pid/window_id。检查新页面再继续。`effect=unverifiable/suspected_noop` 仅代表投递/效果未确认，必须检查返回画面和读回值；`refused` 是未执行。其他应用后台无效时，可在任务授权范围内显式 foreground 后再次检查。不要仅凭工具成功声称测试通过。
|
||||
5. 默认后台操作；明确指定 foreground 时，适配器先使用 Cua 自带 bring_to_front 激活并核实目标，随后保持目标在前台，供页面渲染和连续观察；不会每次动作后切回原应用，也不能承诺不影响用户前台工作。Brave/Chrome/Edge 等 Chromium 浏览器的点击（尤其滚动后）、快捷键、文字输入和滚动直接使用 `delivery_mode=foreground`。例如新标签页：`press_key(key="t",modifiers=["ctrl"],delivery_mode="foreground")`；地址导航：Ctrl+L → `type_text(text=网址,delivery_mode="foreground")` → Enter，各调用均带同一目标 pid/window_id。检查新页面再继续。只读观察/保存截图不会抢焦点；若正文灰屏或控件树缺少网页正文，检查 capture_focus，调用同一 pid/window_id 的 bring_to_front，再观察页面；不要靠重复刷新或业务点击恢复。用户切换窗口后需要重新确认前台，不能拿灰屏判网站失败。`effect=unverifiable/suspected_noop` 仅代表投递/效果未确认，必须检查返回画面和读回值；`refused` 是未执行。其他应用后台无效时，可在任务授权范围内显式 foreground 后再次检查。不要仅凭工具成功声称测试通过。
|
||||
6. `scroll` 必须带最新截图中滚动区域内的 `x,y`，不能传 `element_token`。使用 up/down/left/right、amount、by=line/page；滚轮刻度不保证固定像素或整页距离，根据返回画面确认实际位移再继续。`get_window_state(since="latest")` 可在同一窗口读取差异，但新 snapshot_id 仍替换旧 token。
|
||||
7. `save_screenshot` 保存最新窗口 PNG 到当前任务 trace，filename 仅是文件名，不能传路径，不覆盖已有文件。实际成功返回 SHA256 和附件；不得把工具内存截图当成已保存文件。
|
||||
|
||||
|
||||
@@ -68,3 +68,13 @@ Cua 第三轮虽然独立验收通过，ARTEMIS 最终核验却没有生成结
|
||||
Go 确定性回归覆盖上述尺寸、原始尺寸无二次缩放、不同裁边、负窗口原点、非整数点、缺失/过期编号、非数值/越界、拖拽两端、窗口移动/缩放及原始截图读取失败。Go test、vet 与构建通过。本修订没有重跑付费模型业务任务，也未声称真实多显示器、多 DPI 或跨端成功率已验证；待使用新插件进行业务复测。
|
||||
|
||||
补充用户业务复测：2026-10-17 使用 `gpt-6-luna / high` 与 `0.36.0+artemis.3` 完成手机 Chrome、电脑 Brave 的混合页面探索，33 步、约 410 秒，执行状态为 completed，最终核验为 passed。记录确认移动端 Desktop site 未勾选、两端页底观察及追加滚动；报告明确保留 PC 中段局部未逐屏覆盖和文字证据冲突。此通过仅针对本次探索与报告要求，不表示页面所有区域无缺陷，也不外推多显示器、多 DPI 或整体成功率。
|
||||
|
||||
## 前台取证修订 0.36.0+artemis.4
|
||||
|
||||
2026-10-17 排查发现，前台点击后能短暂捕获网页正文，随后同窗口仅剩浏览器外壳和灰色正文，窗口录像也出现相同表现。固定上游在 foreground 输入后异步恢复原前台，ARTEMIS 的动作后截图则是独立调用；浏览器后台渲染和截图可能受到该时序影响。原业务没有重跑，保留 blocked，不能归因为网站缺陷。
|
||||
|
||||
适配修复使用上游既有 bring_to_front，在显式 foreground 输入前完成持久激活并核实准确 HWND。拒绝、仅提升层级、激活后丢失前台及传输异常都不发送业务输入；不确定输入不重发。只读观察不改变前台，记录 capture_focus，模型可显式恢复同一目标后重新观察。
|
||||
|
||||
Go 回归与 vet 通过。`tests/cua_browser_smoke.py --foreground-check` 在 ARTEMIS 的 desktop:local 资源锁内运行，使用临时 Brave profile、本地表单及另一个普通测试窗口代表原前台应用，不操作用户原有标签。38 次工具调用验证：从被遮挡窗口执行前台输入后，等待 5 秒（超过上游 3 秒恢复轮询）仍保持目标前台；人为切走后只读观察不抢焦点，bring_to_front 恢复并保存真实截图；导航后等待仍能读取正文；新标签、中文输入、表单恰好提交一次、实际滚动、页底点击和 PNG 校验均通过。
|
||||
|
||||
本次未复测原业务全流程、Codex 窗口本身或多屏 DPI；以普通前台窗口验证同一前台恢复机制，不将局部回归称为原任务通过。此修订随 ARTEMIS 2.0.0 配套发布，市场继续保留预览标识；仅更新主程序不会自动更新已安装插件。
|
||||
@@ -0,0 +1,55 @@
|
||||
package main
|
||||
|
||||
// Use Cua's existing persistent activation before explicit foreground input.
|
||||
// Cua then snapshots the target itself as its "previous foreground", so its
|
||||
// asynchronous restore cannot race ARTEMIS's separately requested observation.
|
||||
// Read-only calls and background delivery never activate or restore a window.
|
||||
func callWithForeground(call func(string, obj) (obj, error), name string, args obj, current func() int64) (obj, error) {
|
||||
if name != "bring_to_front" && args["delivery_mode"] != "foreground" {
|
||||
return call(name, args)
|
||||
}
|
||||
target := obj{"pid": args["pid"], "window_id": args["window_id"]}
|
||||
focus, err := call("bring_to_front", target)
|
||||
if err != nil {
|
||||
// Only activation was attempted. Do not send (or replay) the business
|
||||
// action when the activation result is unknown.
|
||||
return failure("Foreground activation could not be confirmed; requested input was not sent: "+err.Error(), "not_started", "possible", "foreground_unavailable"), nil
|
||||
}
|
||||
activation := object(focus["structuredContent"])
|
||||
actual := current()
|
||||
activation["verified_foreground_window_id"] = actual
|
||||
activation["policy"] = "keep_target_foreground"
|
||||
if focus["isError"] == true || activation["landed_on_target"] != true || actual != num(args["window_id"]) {
|
||||
r := failure("Target window did not retain foreground; requested input was not sent. Inspect the window before retrying.", "not_started", "possible", "foreground_unavailable")
|
||||
object(r["structuredContent"])["foreground_activation"] = activation
|
||||
return r, nil
|
||||
}
|
||||
if name == "bring_to_front" {
|
||||
activation["artemis"] = receipt("completed", "confirmed", "")
|
||||
focus["structuredContent"] = activation
|
||||
return focus, nil
|
||||
}
|
||||
r, err := call(name, args)
|
||||
if err == nil {
|
||||
s := object(r["structuredContent"])
|
||||
s["foreground_activation"] = activation
|
||||
r["structuredContent"] = s
|
||||
}
|
||||
return r, err
|
||||
}
|
||||
|
||||
func annotateCaptureFocus(r obj, target, before, after int64) {
|
||||
s := object(r["structuredContent"])
|
||||
s["capture_focus"] = obj{
|
||||
"target_window_id": target,
|
||||
"foreground_window_id_before": before,
|
||||
"foreground_window_id_after": after,
|
||||
"target_foreground_before": before == target,
|
||||
"target_foreground_after": after == target,
|
||||
"changed_during_capture": before != after,
|
||||
}
|
||||
r["structuredContent"] = s
|
||||
if before != target || after != target {
|
||||
r["content"] = append(items(r["content"]), textBlock("Capture was not continuously foreground. If the page body is blank or missing from the tree, use bring_to_front for this same pid/window_id, then observe again; do not infer a website failure or repeat a business action."))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestForegroundActivatesBeforeInputAndDoesNotRestore(t *testing.T) {
|
||||
var calls []string
|
||||
foreground := int64(10) // The user's foreground app, e.g. Codex.
|
||||
call := func(name string, args obj) (obj, error) {
|
||||
calls = append(calls, name)
|
||||
if name == "bring_to_front" {
|
||||
if len(args) != 2 || num(args["window_id"]) != 20 {
|
||||
t.Fatal("activation changed the target or forwarded input arguments")
|
||||
}
|
||||
foreground = 20
|
||||
return obj{"structuredContent": obj{"landed_on_target": true}}, nil
|
||||
}
|
||||
// Simulate upstream's activate/act/restore: it now saves the target,
|
||||
// not the user's previous app, as its restore destination.
|
||||
previous := foreground
|
||||
foreground = 20
|
||||
foreground = previous
|
||||
return obj{"structuredContent": obj{"effect": "unverifiable"}}, nil
|
||||
}
|
||||
r, err := callWithForeground(call, "press_key", obj{"pid": 2, "window_id": 20, "key": "t", "delivery_mode": "foreground"}, func() int64 { return foreground })
|
||||
if err != nil || r["isError"] == true || foreground != 20 || !reflect.DeepEqual(calls, []string{"bring_to_front", "press_key"}) {
|
||||
t.Fatalf("foreground handoff failed: %v %v %v %v", calls, foreground, r, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnconfirmedActivationNeverSendsInput(t *testing.T) {
|
||||
for _, mode := range []string{"refused", "raised_only", "lost_focus", "transport"} {
|
||||
t.Run(mode, func(t *testing.T) {
|
||||
calls := 0
|
||||
r, err := callWithForeground(func(name string, _ obj) (obj, error) {
|
||||
calls++
|
||||
if name != "bring_to_front" {
|
||||
t.Fatal("input sent without focus")
|
||||
}
|
||||
if mode == "transport" {
|
||||
return nil, errors.New("disconnected")
|
||||
}
|
||||
return obj{"isError": mode == "refused", "structuredContent": obj{"raised": true, "landed_on_target": mode != "raised_only"}}, nil
|
||||
}, "click", obj{"pid": 2, "window_id": 20, "delivery_mode": "foreground"}, func() int64 {
|
||||
if mode == "lost_focus" {
|
||||
return 10
|
||||
}
|
||||
return 20
|
||||
})
|
||||
rec := object(object(r["structuredContent"])["artemis"])
|
||||
if err != nil || calls != 1 || r["isError"] != true || rec["execution"] != "not_started" || rec["retryable"] != false {
|
||||
t.Fatalf("unsafe activation handling: %v %v", r, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackgroundAndObservationsNeverActivate(t *testing.T) {
|
||||
for _, name := range []string{"get_window_state", "verify_state", "save_screenshot", "click"} {
|
||||
var calls []string
|
||||
_, err := callWithForeground(func(tool string, _ obj) (obj, error) {
|
||||
calls = append(calls, tool)
|
||||
return obj{}, nil
|
||||
}, name, obj{"pid": 2, "window_id": 20}, func() int64 { t.Fatal("unexpected focus probe"); return 0 })
|
||||
if err != nil || !reflect.DeepEqual(calls, []string{name}) {
|
||||
t.Fatal(calls, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestUncertainInputIsNotReplayed(t *testing.T) {
|
||||
calls := 0
|
||||
_, err := callWithForeground(func(name string, _ obj) (obj, error) {
|
||||
calls++
|
||||
if name == "bring_to_front" {
|
||||
return obj{"structuredContent": obj{"landed_on_target": true}}, nil
|
||||
}
|
||||
return nil, errors.New("input reply lost")
|
||||
}, "type_text", obj{"pid": 2, "window_id": 20, "delivery_mode": "foreground"}, func() int64 { return 20 })
|
||||
if calls != 2 || err == nil {
|
||||
t.Fatal(calls, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCaptureRecordsFocusWithoutHidingBackgroundEvidence(t *testing.T) {
|
||||
r := obj{"content": []any{obj{"type": "image", "data": "original"}}}
|
||||
annotateCaptureFocus(r, 20, 20, 10)
|
||||
s := object(object(r["structuredContent"])["capture_focus"])
|
||||
if s["changed_during_capture"] != true || s["target_foreground_after"] != false || len(items(r["content"])) != 2 || object(items(r["content"])[0])["data"] != "original" {
|
||||
t.Fatal(r)
|
||||
}
|
||||
}
|
||||
@@ -25,7 +25,7 @@ import (
|
||||
//go:embed tools.json
|
||||
var toolsJSON []byte
|
||||
|
||||
const adapterVersion = "0.36.0+artemis.3"
|
||||
const adapterVersion = "0.36.0+artemis.4"
|
||||
|
||||
type obj = map[string]any
|
||||
type request struct {
|
||||
@@ -290,10 +290,12 @@ func source(r obj, args obj, started float64) obj {
|
||||
func (a *adapter) observe(args obj) (obj, obj, error) {
|
||||
delete(a.captures, targetKey(args))
|
||||
started := float64(time.Now().UnixMilli()) / 1000
|
||||
before := foregroundWindow()
|
||||
r, err := a.peer.tool("get_window_state", args)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
annotateCaptureFocus(r, num(args["window_id"]), before, foregroundWindow())
|
||||
src := source(r, args, started)
|
||||
if frame, ok := capturedFrame(r, src); ok {
|
||||
a.captures[targetKey(args)] = frame
|
||||
@@ -384,7 +386,7 @@ func (a *adapter) handle(name string, args obj) obj {
|
||||
if writeTool(name) {
|
||||
delete(a.captures, targetKey(args))
|
||||
}
|
||||
r, err = a.peer.tool(upstream, args)
|
||||
r, err = callWithForeground(a.peer.tool, upstream, args, foregroundWindow)
|
||||
}
|
||||
if err != nil {
|
||||
a.peer.stop()
|
||||
@@ -393,8 +395,11 @@ func (a *adapter) handle(name string, args obj) obj {
|
||||
}
|
||||
return failure(err.Error(), "not_started", "none", "transport_lost")
|
||||
}
|
||||
rec := actionReceipt(r, writeTool(name))
|
||||
structured := object(r["structuredContent"])
|
||||
rec := object(structured["artemis"])
|
||||
if len(rec) == 0 {
|
||||
rec = actionReceipt(r, writeTool(name))
|
||||
}
|
||||
if writeTool(name) && rec["execution"] != "not_started" {
|
||||
// Observe once after delivery; NEVER repeat an uncertain write.
|
||||
observation, origin, e := a.observe(observationArgs(args))
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"schema_version": 2,
|
||||
"id": "cua-driver",
|
||||
"name": "电脑控制 · Cua",
|
||||
"version": "0.36.0+artemis.3",
|
||||
"version": "0.36.0+artemis.4",
|
||||
"description": "通过 Cua Driver 的窗口截图与可访问性树操作 Windows 应用及浏览器窗口；复用 ARTEMIS 的任务调度、步骤证据与录像。",
|
||||
"artemis_version": ">=2.0.0,<3.0.0",
|
||||
"platforms": [
|
||||
@@ -158,6 +158,18 @@
|
||||
"text",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
"bring_to_front": {
|
||||
"effect": "write",
|
||||
"observation": false,
|
||||
"idempotent": true,
|
||||
"target_argument": "window_id",
|
||||
"cancellation": "unknown",
|
||||
"evidence": [
|
||||
"text",
|
||||
"image",
|
||||
"state"
|
||||
]
|
||||
}
|
||||
},
|
||||
"required_tools": [
|
||||
|
||||
@@ -235,7 +235,7 @@
|
||||
"type": "integer"
|
||||
},
|
||||
"delivery_mode": {
|
||||
"description": "background (default) avoids raising the window; unsupported input is refused. Use foreground for Chromium/Brave keyboard input and scrolling within task scope. Inspect the returned state before retrying an uncertain action.",
|
||||
"description": "background does not activate the window. foreground uses Cua bring_to_front before input and leaves the target foreground for subsequent rendering and evidence; it does not switch back to the user's previous app after each action.",
|
||||
"enum": [
|
||||
"background",
|
||||
"foreground"
|
||||
@@ -275,7 +275,7 @@
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"delivery_mode": {
|
||||
"description": "background (default) avoids raising the window; unsupported input is refused. Use foreground for Chromium/Brave keyboard input and scrolling within task scope. Inspect the returned state before retrying an uncertain action.",
|
||||
"description": "background does not activate the window. foreground uses Cua bring_to_front before input and leaves the target foreground for subsequent rendering and evidence; it does not switch back to the user's previous app after each action.",
|
||||
"enum": [
|
||||
"background",
|
||||
"foreground"
|
||||
@@ -341,7 +341,7 @@
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"delivery_mode": {
|
||||
"description": "background (default) avoids raising the window; unsupported input is refused. Use foreground for Chromium/Brave keyboard input and scrolling within task scope. Inspect the returned state before retrying an uncertain action.",
|
||||
"description": "background does not activate the window. foreground uses Cua bring_to_front before input and leaves the target foreground for subsequent rendering and evidence; it does not switch back to the user's previous app after each action.",
|
||||
"enum": [
|
||||
"background",
|
||||
"foreground"
|
||||
@@ -380,7 +380,7 @@
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"delivery_mode": {
|
||||
"description": "background (default) avoids raising the window; unsupported input is refused. Use foreground for Chromium/Brave keyboard input and scrolling within task scope. Inspect the returned state before retrying an uncertain action.",
|
||||
"description": "background does not activate the window. foreground uses Cua bring_to_front before input and leaves the target foreground for subsequent rendering and evidence; it does not switch back to the user's previous app after each action.",
|
||||
"enum": [
|
||||
"background",
|
||||
"foreground"
|
||||
@@ -426,7 +426,7 @@
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"delivery_mode": {
|
||||
"description": "background (default) avoids raising the window; unsupported input is refused. Use foreground for Chromium/Brave keyboard input and scrolling within task scope. Inspect the returned state before retrying an uncertain action.",
|
||||
"description": "background does not activate the window. foreground uses Cua bring_to_front before input and leaves the target foreground for subsequent rendering and evidence; it does not switch back to the user's previous app after each action.",
|
||||
"enum": [
|
||||
"background",
|
||||
"foreground"
|
||||
@@ -485,7 +485,7 @@
|
||||
"type": "string"
|
||||
},
|
||||
"delivery_mode": {
|
||||
"description": "background (default) avoids raising the window; unsupported input is refused. Use foreground for Chromium/Brave keyboard input and scrolling within task scope. Inspect the returned state before retrying an uncertain action.",
|
||||
"description": "background does not activate the window. foreground uses Cua bring_to_front before input and leaves the target foreground for subsequent rendering and evidence; it does not switch back to the user's previous app after each action.",
|
||||
"enum": [
|
||||
"background",
|
||||
"foreground"
|
||||
@@ -598,7 +598,7 @@
|
||||
"type": "integer"
|
||||
},
|
||||
"delivery_mode": {
|
||||
"description": "background (default) avoids raising the window; unsupported input is refused. Use foreground for Chromium/Brave keyboard input and scrolling within task scope. Inspect the returned state before retrying an uncertain action.",
|
||||
"description": "background does not activate the window. foreground uses Cua bring_to_front before input and leaves the target foreground for subsequent rendering and evidence; it does not switch back to the user's previous app after each action.",
|
||||
"enum": [
|
||||
"background",
|
||||
"foreground"
|
||||
@@ -635,5 +635,34 @@
|
||||
"readOnlyHint": false,
|
||||
"idempotentHint": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "bring_to_front",
|
||||
"description": "Activate the exact authorized window and leave it foreground across observations and actions. Use when a background browser body is blank or its page tree is missing. This changes the user foreground; it does not navigate, refresh, or repeat any business action. Read the returned fresh screenshot before continuing.",
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"pid",
|
||||
"window_id"
|
||||
],
|
||||
"properties": {
|
||||
"pid": {
|
||||
"type": "integer",
|
||||
"minimum": 1,
|
||||
"maximum": 4294967295
|
||||
},
|
||||
"window_id": {
|
||||
"type": "integer",
|
||||
"minimum": 1
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
},
|
||||
"annotations": {
|
||||
"readOnlyHint": false,
|
||||
"destructiveHint": false,
|
||||
"idempotentHint": true,
|
||||
"openWorldHint": false
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -11,6 +11,11 @@ import (
|
||||
var user32 = syscall.NewLazyDLL("user32.dll")
|
||||
var kernel32 = syscall.NewLazyDLL("kernel32.dll")
|
||||
|
||||
func foregroundWindow() int64 {
|
||||
hwnd, _, _ := user32.NewProc("GetForegroundWindow").Call()
|
||||
return int64(hwnd)
|
||||
}
|
||||
|
||||
func configureProcess(c *exec.Cmd) {
|
||||
c.SysProcAttr = &syscall.SysProcAttr{HideWindow: true}
|
||||
}
|
||||
|
||||
@@ -99,7 +99,7 @@ def main():
|
||||
"source_commit": COMMIT,
|
||||
"upstream_version": UPSTREAM_VERSION,
|
||||
"plugin_version": version,
|
||||
"adapter_revision": 4,
|
||||
"adapter_revision": 5,
|
||||
"prerelease": True,
|
||||
"url": URL,
|
||||
"sha256": DIGEST,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
param([string]$Title, [string]$ReadyFile, [switch]$Cover, [int]$Width=500, [int]$Height=300, [switch]$PerMonitorDpi)
|
||||
param([string]$Title, [string]$ReadyFile, [switch]$Cover, [int]$Width=500, [int]$Height=300, [switch]$PerMonitorDpi, [switch]$NormalWindow)
|
||||
if ($PerMonitorDpi) {
|
||||
Add-Type -TypeDefinition 'using System; using System.Runtime.InteropServices; public static class FixtureDpi { [DllImport("user32.dll")] public static extern IntPtr SetThreadDpiAwarenessContext(IntPtr c); }'
|
||||
[void][FixtureDpi]::SetThreadDpiAwarenessContext([IntPtr](-4))
|
||||
@@ -10,7 +10,7 @@ $form.Text = $Title
|
||||
$form.Size = New-Object System.Drawing.Size($Width, $Height)
|
||||
$form.StartPosition = 'Manual'
|
||||
$form.Location = New-Object System.Drawing.Point(100, 100)
|
||||
$form.TopMost = $true
|
||||
$form.TopMost = -not $NormalWindow
|
||||
$form.BackColor = if ($Cover) { [System.Drawing.Color]::Magenta } else { [System.Drawing.Color]::Lime }
|
||||
$label = New-Object System.Windows.Forms.Label
|
||||
$label.Text = if ($Cover) { 'OCCLUDER - NOT TEST EVIDENCE' } else { 'ARTEMIS window capture fixture' }
|
||||
|
||||
@@ -22,6 +22,7 @@ import jsonschema
|
||||
import psutil
|
||||
from artemis.plugins.models import ServerConfig
|
||||
from artemis.plugins.session import PluginSession
|
||||
from artemis.runtime.device_lock import DeviceExecutionLock
|
||||
|
||||
|
||||
async def model_task(base, pid, url, output, events, traces, model):
|
||||
@@ -133,6 +134,7 @@ async def run(
|
||||
base=None,
|
||||
traces=None,
|
||||
model="gpt-6.1-sol",
|
||||
foreground_check=False,
|
||||
):
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
events = []
|
||||
@@ -179,7 +181,7 @@ async def run(
|
||||
threading.Thread(target=server.serve_forever, daemon=True).start()
|
||||
url = f"http://127.0.0.1:{server.server_port}"
|
||||
session = None
|
||||
process = None
|
||||
process = cover_process = None
|
||||
try:
|
||||
with tempfile.TemporaryDirectory(prefix="artemis-cua-browser-") as profile:
|
||||
try:
|
||||
@@ -276,6 +278,60 @@ async def run(
|
||||
assert match, state["tree_markdown"]
|
||||
return state["snapshot_id"] + ":" + match[1]
|
||||
|
||||
if foreground_check:
|
||||
ready = Path(profile) / "cover-ready"
|
||||
cover_process = await asyncio.create_subprocess_exec(
|
||||
"powershell.exe",
|
||||
"-NoProfile",
|
||||
"-File",
|
||||
str(Path(__file__).with_name("capture-fixture.ps1")),
|
||||
"-Title",
|
||||
"ARTEMIS foreground handoff fixture",
|
||||
"-ReadyFile",
|
||||
str(ready),
|
||||
"-Cover",
|
||||
"-NormalWindow",
|
||||
"-Width",
|
||||
"1100",
|
||||
"-Height",
|
||||
"900",
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
creationflags=subprocess.CREATE_NO_WINDOW,
|
||||
)
|
||||
async with asyncio.timeout(20):
|
||||
while not ready.exists():
|
||||
await asyncio.sleep(0.1)
|
||||
cover_windows = await call(
|
||||
"list_windows", {"pid": cover_process.pid}
|
||||
)
|
||||
cover = {
|
||||
k: cover_windows["windows"][0][k] for k in ("pid", "window_id")
|
||||
}
|
||||
await call("bring_to_front", cover)
|
||||
hidden = await observe()
|
||||
assert not hidden["capture_focus"]["target_foreground_before"]
|
||||
assert not hidden["capture_focus"]["target_foreground_after"]
|
||||
# Explicit foreground input must not snap back to the cover.
|
||||
await key("l", ["ctrl"])
|
||||
await asyncio.sleep(5) # Past Cua's 3-second restore poller.
|
||||
focused = await observe()
|
||||
assert focused["capture_focus"]["target_foreground_after"]
|
||||
await key("escape")
|
||||
# Manual focus loss is reported, never silently corrected by
|
||||
# read-only capture. Existing bring_to_front recovers it.
|
||||
await call("bring_to_front", cover)
|
||||
hidden = await observe()
|
||||
assert not hidden["capture_focus"]["target_foreground_after"]
|
||||
await call("bring_to_front", target)
|
||||
await asyncio.sleep(5)
|
||||
focused = await observe()
|
||||
assert focused["capture_focus"]["target_foreground_before"]
|
||||
assert focused["capture_focus"]["target_foreground_after"]
|
||||
await call(
|
||||
"save_screenshot",
|
||||
{**target, "filename": "foreground-recovered.png"},
|
||||
)
|
||||
await wait_tree("ARTEMIS Cua Browser Start")
|
||||
# Reproduce task f1928... invalid form before any keyboard dispatch.
|
||||
bad = await call(
|
||||
@@ -303,6 +359,11 @@ async def run(
|
||||
)
|
||||
await key("return")
|
||||
state = await wait_tree("Test parameter")
|
||||
if foreground_check:
|
||||
await asyncio.sleep(5)
|
||||
state = await observe()
|
||||
assert state["capture_focus"]["target_foreground_after"]
|
||||
assert "Test parameter" in state["tree_markdown"]
|
||||
# Direct UIA replacement is a separate, unambiguous tool.
|
||||
await call(
|
||||
"set_value",
|
||||
@@ -393,6 +454,7 @@ async def run(
|
||||
)
|
||||
report = {
|
||||
"passed": True,
|
||||
"foreground_handoff_and_readonly_recovery": foreground_check,
|
||||
"checks": [
|
||||
"mixed parameters refused before dispatch",
|
||||
"Ctrl+T new tab",
|
||||
@@ -417,6 +479,9 @@ async def run(
|
||||
)
|
||||
if session:
|
||||
await session.aclose()
|
||||
if cover_process and cover_process.returncode is None:
|
||||
cover_process.terminate()
|
||||
await cover_process.wait()
|
||||
if process:
|
||||
try:
|
||||
parent = psutil.Process(process.pid)
|
||||
@@ -449,9 +514,21 @@ if __name__ == "__main__":
|
||||
"--traces", type=Path, help="That server's traces directory (model test only)"
|
||||
)
|
||||
parser.add_argument("--model", default="gpt-6.1-sol")
|
||||
parser.add_argument("--foreground-check", action="store_true")
|
||||
args = parser.parse_args()
|
||||
if args.model_base and not args.traces:
|
||||
parser.error("--model-base requires --traces")
|
||||
if args.foreground_check and args.model_base:
|
||||
parser.error("--foreground-check is deterministic, not a model task")
|
||||
lock = None
|
||||
if not args.model_base:
|
||||
lock = DeviceExecutionLock(
|
||||
"cua-browser-regression",
|
||||
description="Cua browser regression",
|
||||
resource_ids=["desktop:local"],
|
||||
)
|
||||
lock.acquire(timeout=0)
|
||||
try:
|
||||
asyncio.run(
|
||||
run(
|
||||
args.executable.resolve(),
|
||||
@@ -460,5 +537,9 @@ if __name__ == "__main__":
|
||||
args.model_base,
|
||||
args.traces,
|
||||
args.model,
|
||||
args.foreground_check,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
if lock:
|
||||
lock.release()
|
||||
Reference in new issue
Block a user