feat: add gesture sequences and calibrated pixel drags

This commit is contained in:
chenmingxuan committed 2026-11-02 11:53:27 +08:00
1 parent 2a3bbaa3fb
commit 479a185192
31 files changed
+2440 -44

No files matched your search

+2
View File
@@ -30,6 +30,8 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
- **Time-Sensitive Tasks & Transient UI**: Because of the turn latency above, a target element that you see on the current screenshot (e.g., auto-dismissing overlays, temporary message bars, menus or control bars with short timeouts) may have already vanished by the time your physical click command reaches the device! This is one of the most common reasons why the UI does not change as expected after a single step. We call these **Time-Sensitive Tasks**.
- *Mental Model*: When facing these tasks, any attempt to interact step-by-step (such as tapping once to wake up a menu/bar and waiting for the next turn to tap the target button) WILL fail because the element will auto-dismiss during your decision delay. You MUST combine the summon action on the trigger element (`Trigger`) and the target action (`Target`) into a single atomic sequence.
{% if "click_sequence" in available_tools %} - *Chained Execution*: Use `click_sequence(sequence=[[trigger_x, trigger_y], [target_x, target_y]])` in `0-1000` normalized scale to execute both taps consecutively with a 50ms delay, catching the transient element before the timeout closes it.{% endif %}
{% if "action_sequence" in available_tools %} - **短时控件的混合动作**:需要打开后拖动时,优先一次调用 `action_sequence`,将 click、短暂 wait、drag 放入 steps,不分成几轮模型决策。后续目标须来自同布局的真实观察,不能猜测。逐项读取回执;部分执行后不得整组重发,先观察当前状态。{% endif %}
{% if "drag" in available_tools %} - **刻度校准**:局部滑条、时间轴优先用 `drag`。放大后可用当前原图的 pixel 坐标,或两个真实刻度 anchors 计算目标;时间统一换算成秒等同一数值单位。pan 用固定指针 indicator_position,thumb 用真实滑块起点。不要自动越过终点,也不要一味拖慢;短距离可从 300ms 起步。动作后比较同名刻度或真实选中值,仅根据已观察残差修正。回执中的 calibration 是计算/派发信息,不能当作已选中目标值;精度不足先放大。{% endif %}
---
+22 -2
View File
@@ -253,7 +253,14 @@ class FlashRunner:
t
for t in tools
if t.name not in DEVICE_ACTIONS
and t.name not in {"ask_explorer", "video_analyzer", "click_sequence"}
and t.name
not in {
"ask_explorer",
"video_analyzer",
"click_sequence",
"action_sequence",
"drag",
}
]
# With an actuator installed, drop declarations for device actions the
# backend does not implement (and append its extension tools); without
@@ -781,7 +788,20 @@ class FlashRunner:
action_dict["normalized_start_coordinates"] = norm_start
action_dict["normalized_end_coordinates"] = norm_end
action_dict.update(metadata.get("target_semantics") or {})
for key in ("input_verification", "execution", "side_effect", "source"):
for key in (
"input_verification",
"execution",
"side_effect",
"source",
"sequence_results",
"completed_steps",
"calibration",
"coordinate_space",
"coordinates",
"duration_ms",
"screen_size",
"normalized_coordinates",
):
if key in metadata:
action_dict[key] = metadata[key]
if metadata.get("plugin_id"):