fix: guard transient controls and verify timeline endpoints
This commit is contained in:
chenmingxuan committed
1 parent
479a185192
commit
00e62a663a
29 files changed
+1150
-90
No files matched your search
+76
-10
@@ -3,7 +3,7 @@
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "click",
|
||||
"description": "[ACTION] Click on the target location on the screen (supports element index or absolute normalized coordinates).",
|
||||
"description": "[ACTION] Click on the target location on the screen (supports element index or absolute normalized coordinates). Indexed resource targets are checked against live bounds before input; missing/moved targets refuse the tap. For floating or auto-hiding panels, use a guarded action_sequence with opener and target in the same call; an open panel in a screenshot may already have expired.",
|
||||
"parameters": {
|
||||
"properties": {
|
||||
"target": {
|
||||
@@ -54,7 +54,7 @@
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "action_sequence",
|
||||
"description": "[ACTION] Execute a short sequence of click, wait and drag steps in ONE call, in order, without model turns or screenshots between members. Use for temporary overlays: open the control, wait briefly, then operate it before it disappears. Use observed coordinates, or positions learned from an earlier observation of the same restored layout; element indices are not accepted. Each click/drag needs target_description. All members are validated before any input. The configured fast-action limit applies. The first failure stops the remainder; receipts identify completed and unexecuted members. Never replay the whole sequence after a partial result. Verify the final screen before claiming the intended result.",
|
||||
"description": "[ACTION] Execute a short sequence of click, wait and drag steps in ONE call, in order, without model turns between members; explicit visibility guards may read the live UI locally. Use for temporary overlays: open the control, wait briefly, then operate it before it disappears. For clicks prefer target=the exact observed resource_id string: live bounds are resolved locally, refusing absent or ambiguous targets. Otherwise use observed [x,y] in the declared coordinate_space; do not confuse original screenshot pixels with 0..1000 normalized coordinates. Element indices are not accepted. Each click/drag needs target_description. All members are validated before any input. The configured fast-action limit applies. The first failure stops the remainder; receipts identify completed and unexecuted members. Never replay the whole sequence after a partial result. Every wait counts as a member. click.times (1..10) performs consecutive taps within ONE member, like the ordinary click tool; prefer this over expanding repeated +/- taps into many members/model turns. Each resource-targeted tap is revalidated locally; missing targets stop remaining taps and receipts preserve completed_taps. For maximum/minimum prefer observed native +/- controls, confirming that further same-direction adjustment stops changing handle/ticks. An estimated drag endpoint can be imprecise. For every auto-hiding adjustment, including the reverse direction, keep opener/wait/operation in one call. A prior screenshot does not prove the panel is still open. Use click.unless_visible and drag.require_visible with an observed panel resource_id to avoid toggling an open panel closed or dragging its underlying content. Never invent resource IDs. Verify actual ticks/handle endpoints before claiming maximum or minimum.",
|
||||
"parameters": {
|
||||
"properties": {
|
||||
"steps": {
|
||||
@@ -70,13 +70,22 @@
|
||||
"type": "string"
|
||||
},
|
||||
"target": {
|
||||
"items": {
|
||||
"type": "integer"
|
||||
},
|
||||
"maxItems": 2,
|
||||
"minItems": 2,
|
||||
"title": "Target",
|
||||
"type": "array"
|
||||
"anyOf": [
|
||||
{
|
||||
"items": {
|
||||
"type": "integer"
|
||||
},
|
||||
"maxItems": 2,
|
||||
"minItems": 2,
|
||||
"type": "array"
|
||||
},
|
||||
{
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
}
|
||||
],
|
||||
"description": "Prefer an exact observed resource_id: resolve its live bounds before tapping, refusing absent/ambiguous targets. Otherwise use [x,y] in coordinate_space; never mix screenshot pixels with normalized coordinates.",
|
||||
"title": "Target"
|
||||
},
|
||||
"target_description": {
|
||||
"minLength": 1,
|
||||
@@ -91,6 +100,35 @@
|
||||
],
|
||||
"title": "Coordinate Space",
|
||||
"type": "string"
|
||||
},
|
||||
"times": {
|
||||
"default": 1,
|
||||
"description": "Consecutive taps within this ONE member, e.g. 10 for a scale +/- button; no model turns between taps.",
|
||||
"maximum": 10,
|
||||
"minimum": 1,
|
||||
"title": "Times",
|
||||
"type": "integer"
|
||||
},
|
||||
"delay_ms": {
|
||||
"default": 100,
|
||||
"maximum": 1000,
|
||||
"minimum": 0,
|
||||
"title": "Delay Ms",
|
||||
"type": "integer"
|
||||
},
|
||||
"unless_visible": {
|
||||
"anyOf": [
|
||||
{
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"description": "Exact previously observed resource_id inside a toggled panel (not the opener). Skip this opener click when that control is already visible; otherwise click. Avoid closing a still-open panel.",
|
||||
"title": "Unless Visible"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
@@ -235,6 +273,20 @@
|
||||
"default": null,
|
||||
"description": "Computes the endpoint from observed anchors; then set end=null."
|
||||
},
|
||||
"require_visible": {
|
||||
"anyOf": [
|
||||
{
|
||||
"minLength": 1,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"description": "Exact previously observed resource_id of a panel control. Check live visibility before dragging; refuse input if absent. Use for auto-hiding controls.",
|
||||
"title": "Require Visible"
|
||||
},
|
||||
"action": {
|
||||
"const": "drag",
|
||||
"title": "Action",
|
||||
@@ -273,6 +325,8 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
"maxItems": 10,
|
||||
"minItems": 1,
|
||||
"type": "array"
|
||||
}
|
||||
},
|
||||
@@ -287,7 +341,7 @@
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "drag",
|
||||
"description": "[ACTION] Precisely drag a local slider or timeline. Use explicit start/end or calibration from TWO observed numeric scale anchors. coordinate_space is normalized (0..1000) or pixel (the current screenshot's actual pixels); pixel avoids quantization on a zoomed scale. Default duration_ms=300: slower is not always more precise and may trigger long-press recognition. Do not overshoot the endpoint automatically. For calibration, axis=x/y; mode=thumb moves the handle to target_value, mode=pan shifts the scale under indicator_position. Use one numeric unit, e.g. seconds since midnight, and two anchors from the CURRENT zoom/layout. Provide exactly one of end or calibration. Returned geometry is a plan/dispatch receipt, not proof of the selected value: compare the post-action scale/value and correct only the observed residual; zoom further if the pixel resolution is insufficient.",
|
||||
"description": "[ACTION] Precisely drag a local slider or timeline. Use explicit start/end or calibration from TWO observed numeric scale anchors. coordinate_space is normalized (0..1000) or pixel (the current screenshot's actual pixels); pixel avoids quantization on a zoomed scale. Default duration_ms=300. If a panned timeline coasts past the release position, try a slower pan (e.g. 1500ms), then wait for its ticks to stop before measuring the residual. A transient release-time bubble can disagree with the final fixed-pointer scale; do not treat it as final position proof. A permanently visible timeline can be panned independently of its zoom popup. Do not overshoot the endpoint automatically. For calibration, axis=x/y; mode=thumb moves the handle to target_value, mode=pan shifts the scale under indicator_position. Use one numeric unit, e.g. seconds since midnight, and two anchors from the CURRENT zoom/layout. Provide exactly one of end or calibration. Returned geometry is a plan/dispatch receipt, not proof of the selected value: compare the post-action scale/value and correct only the observed residual; zoom further if the pixel resolution is insufficient.",
|
||||
"parameters": {
|
||||
"properties": {
|
||||
"start": {
|
||||
@@ -410,6 +464,18 @@
|
||||
],
|
||||
"default": null,
|
||||
"description": "Observed numeric anchors that compute the endpoint. When supplied, end MUST be null; otherwise set calibration null."
|
||||
},
|
||||
"require_visible": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"description": "Previously observed panel resource_id; refuse the drag if it is no longer visible."
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
|
||||
Reference in new issue
Block a user