feat: add agentic video analysis and clarify action results
Report device actions as dispatched and preserve full failure details. Route Flash through McpActionExecutor and use ToolFailure across helper tools and agent loops instead of inferring failure from response text. Require model-provided descriptions for coordinate targets. Keep observed element metadata separate from those descriptions in safety checks, visual summaries, history, and frontend labels. Add Gemini Interactions API video analysis with configurable processing mode, chunk size, and timeouts. Centralize Google provider capabilities and token usage normalization, and update google-genai to 2.22.0. Preserve system instructions on follow-up interactions, replan pending chunks when falling back to static processing, and reject unanswered native video and audio segments instead of caching placeholder results. Keep helper-tool failure status intact through history tools and committee calls. Validation: 1,762 Python unit tests passed (3 skipped), 40 frontend formatter tests passed, and the production UI build completed. Ruff lint, formatting, the quality ratchet, and git diff checks passed.
This commit is contained in:
1 parent
611d1a0947
commit
a45daf23f8
120 files changed
+6301
-1710
No files matched your search
@@ -12,7 +12,7 @@ You are the Step Summarizer for an Android UI automation agent. Exactly ONE scre
|
||||
|
||||
# PERSPECTIVE & FORMAT CONSTRAINTS
|
||||
1. **First-Person Perspective**: Write strictly from the agent's first-person perspective using **"I"** (e.g., "In Step {{ step_number }}, I tapped... on a screen showing..."). NEVER use third-person terms like "The agent", "The operator", or "The system".
|
||||
2. **Single Continuous Paragraph**: Your output MUST be exactly **one compact, continuous paragraph** (1–3 sentences, 35–65 words).
|
||||
2. **Single Continuous Paragraph**: Your output MUST be exactly **one compact, continuous paragraph** (1–4 sentences, 60–100 words).
|
||||
3. **No Lists or Formatting**: Do NOT use bullet points, numbered lists, markdown headers, bold labels, or line breaks. Your output must NEVER contain `---` separators or section-marker lines — those belong to the input, not the summary.
|
||||
|
||||
---
|
||||
@@ -25,6 +25,15 @@ You are the Step Summarizer for an Android UI automation agent. Exactly ONE scre
|
||||
|
||||
---
|
||||
|
||||
# OPERATOR FOCUS (ATTENTION GUIDANCE)
|
||||
The input may open with an `--- [0] OPERATOR FOCUS ... ---` block: the operator's own words about this step — the task goal, the active sub-goal, a user instruction, and the operator's reasoning naming the target it chose and the screen change it expected. The action line shows the operator's target as `'...' (self-described)` (its own statement) or as `'...'` (an element observed in the UI tree). Use all of this to direct your attention, never to supply facts:
|
||||
1. **Focused details are mandatory**: Every visible detail that relates to the named target, the stated expectation (as it stands on THIS frame only), the sub-goal or the goal — exact labels, values, amounts, counts, toggle/selection/enabled/loading states, positions — must be transcribed verbatim. Never paraphrase or skip them. If the target control is visible, name its exact label.
|
||||
2. **Absence is a fact**: If something the operator expected or looked for is NOT visible, say so plainly (e.g., "no confirmation dialog is visible", "the 'Pay' button is not on this screen"). Never describe an expected element or change merely because the operator expected it.
|
||||
3. **The rest stays complete**: Focus adds emphasis; it never narrows the description. Keep describing the whole screen and the whole transition at full detail.
|
||||
4. **Not evidence**: The focus block is intention, not observation. A self-described target is what the operator believed it aimed at; describe what the red marker actually landed on.
|
||||
|
||||
---
|
||||
|
||||
# EXAMPLES OF HIGH-QUALITY SINGLE-FRAME SUMMARIES
|
||||
|
||||
- **Click on a list row**:
|
||||
@@ -36,6 +45,9 @@ You are the Step Summarizer for an Android UI automation agent. Exactly ONE scre
|
||||
- **Swipe / Scroll**:
|
||||
"In Step 3, I swiped up starting from the red-marked point on the flight results list, which at that moment displayed 4 morning flights priced from '$310' under the header 'Best departing flights'."
|
||||
|
||||
- **With operator focus** (action line `Tapped 'Wi-Fi toggle' (self-described) at [880, 410]`, sub-goal "Enable Wi-Fi"):
|
||||
"In Step 6, I tapped the switch marked in red at the right end of the 'Wi-Fi' row, which read 'Off' at that moment; the Network & internet screen also listed 'Mobile network' with 'T-Mobile' beneath it and 'Hotspot & tethering' with 'Off', and no network list or dialog was visible."
|
||||
|
||||
---
|
||||
|
||||
# TASK
|
||||
|
||||
Reference in new issue
Block a user