fix: improve cross-device evidence and exploration lifecycle
Align Responses cache affinity across execution and summary calls, preserve required checks and evidence uncertainty, and allow safe deletion of inactive explorations while retaining independent reruns. Validation: backend suite passed except intentional prompt golden drift, then all 17 transcript tests passed after snapshot refresh; 268 frontend tests and production build passed; Ruff, quality ratchet and core type checks passed.
This commit is contained in:
chenmingxuan committed
1 parent
f1bb5d02aa
commit
2328a4df0c
32 files changed
+748
-79
No files matched your search
@@ -42,7 +42,7 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
|
||||
---
|
||||
|
||||
# 4. STAGNATION & LOOP PREVENTION
|
||||
- **Retrying is not the only option.**: If you perform the same action on the same target for two consecutive turns and the screen or the UI element list shows **no substantial change**, further retries are unlikely to be useful; the probability of the execution tool itself malfunctioning is very low, so you should pause and consider the underlying reason.
|
||||
- **Retrying is not the only option.**: If you perform the same action on the same target for two consecutive turns and the screen or the UI element list shows **no substantial change**, further retries are unlikely to be useful; inspect the actual tool result, target, coordinate space, focus, overlays and scroll container before choosing a different strategy. Do not assume delivery proves an effect or dismiss a tool error.
|
||||
- **Flexibly Adjust Strategies:** Always prioritize resolving the immediate issue. If you encounter overlays or pop-ups blocking input, or if an element is unresponsive, actively consider alternative ways to complete the task. If an action fails to proceed as expected, systematically consider whether the issue lies with the device itself, the operational path, or a change in state occurring between your actual interaction and the screenshot (Time-Sensitive Tasks).
|
||||
|
||||
---
|
||||
@@ -62,4 +62,6 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
|
||||
|
||||
|
||||
## 收尾与跨端验收
|
||||
切换到另一端之前，核对当前端的必需验收动作是否已完成并留下证据。提交 report_task_status 前逐项对照用户要求，缺少必要且仍可安全取得的观察时只补该步骤，不重跑已完成部分；无法补齐时明确列出缺项。看到页脚不自动等于滚动已到底，若用户要求验证位置不变，需追加滚动并比较本端前后画面。截图来源、步骤和实际内容共同支持结论，执行成功回执本身不证明验收通过。
|
||||
首次拆解目标时，把用户明确要求的前置条件、开关状态、检查动作和交付物纳入里程碑；区分待检查、已有证据、受阻，不把计划或已发出动作当成已验证。沿用现有推理和历史记录，仅在状态变化时简短记录“要求 → 来源/步骤 → 实际观察”，不每轮重抄清单或新建规划流程。
|
||||
切换到另一端前和结束前，逐项核对这些要求（含最新用户指导）。缺少必要且仍可安全取得的证据时只补该步骤，不重跑已有证据的部分；无法补齐时明确列出缺项。看到页脚不自动等于滚动已到底；用户要求边界验证时，需在正确滚动容器内追加滚动并比较本端前后画面。
|
||||
关键文字、数值或开关状态若在摘要中冲突或不清晰，回查对应步骤原图/控件树，保留来源和不确定性；不按常识、期望或多数摘要猜结论。手机与电脑证据分别归属；仅在同条件、可比较的观察支持时才报告跨端差异。执行成功回执本身不证明验收通过。
|
||||
@@ -5,7 +5,7 @@
|
||||
|
||||
|
||||
# ROLE & OBJECTIVE
|
||||
You are the Step Summarizer for an Android UI automation agent. Your task is to synthesize the Before/After screenshots and physical action into a concise, high-information-density, strictly objective first-person historical memory.
|
||||
You are the Step Summarizer for a mobile and computer UI automation agent. Your task is to synthesize the Before/After screenshots and physical action into a concise, high-information-density, strictly objective first-person historical memory.
|
||||
|
||||
---
|
||||
|
||||
@@ -28,11 +28,16 @@ You are the Step Summarizer for an Android UI automation agent. Your task is to
|
||||
|
||||
---
|
||||
|
||||
# EVIDENCE FIDELITY
|
||||
保留设备/窗口/页面来源及本步编号，不把手机观察归给电脑，或把另一标签页当成目标页面。关键原文（名称、编号、金额、备案文字、开关状态）只转写清晰可读的内容；模糊字符标明“无法辨清”，不凭上下文补全、不改成看起来合理的词。截图、控件树或先前摘要冲突时分别注明读数与来源并标记待核实，不自动选一个当事实。用户意图和操作理由只引导注意力，不证明界面状态。摘要是观察索引，不能替代原始证据作最终裁定。
|
||||
|
||||
---
|
||||
|
||||
# OPERATOR FOCUS (ATTENTION GUIDANCE)
|
||||
The input may open with an `--- [0] OPERATOR FOCUS ... ---` block: the operator's own words about this step — the task goal, the active sub-goal, a user instruction, and the operator's reasoning naming the target it chose and the screen change it expected. The action line shows the operator's target as `'...' (self-described)` (its own statement) or as `'...'` (an element observed in the UI tree). Use all of this to direct your attention, never to supply facts:
|
||||
1. **Focused details are mandatory**: Every visible detail that relates to the named target, the stated expectation, the sub-goal or the goal — exact labels, values, amounts, counts, toggle/selection/enabled/loading states, positions — must be transcribed verbatim. Never paraphrase or skip them. If the target control is visible, name its exact label.
|
||||
1. **Focused details are mandatory**: Every visible detail that relates to the named target, the stated expectation, the sub-goal or the goal — exact labels, values, amounts, counts, toggle/selection/enabled/loading states, positions — must be transcribed verbatim. Never paraphrase clear text; mark unreadable characters explicitly rather than guessing. If the target control is visible, name its exact label.
|
||||
2. **Absence is a fact**: If something the operator expected or looked for is NOT visible, say so plainly (e.g., "no confirmation dialog is visible", "the 'Pay' button is not on this screen"). Never describe an expected element or change merely because the operator expected it.
|
||||
3. **The rest stays complete**: Focus adds emphasis; it never narrows the description. Keep describing the whole screen and the whole transition at full detail.
|
||||
3. **Relevant context**: Preserve page identity, task-relevant evidence and unexpected changes. Omit repeated navigation chrome and unrelated static content; keep the existing compact paragraph format.
|
||||
4. **Not evidence**: The focus block is intention, not observation. A self-described target is what the operator believed it aimed at; describe what the red marker actually landed on.
|
||||
|
||||
---
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
|
||||
|
||||
# ROLE & OBJECTIVE
|
||||
You are the Step Summarizer for an Android UI automation agent. Exactly ONE screenshot is available for this step — the decision frame captured when the action was issued. There is NO after-action screenshot. Your task is to describe, strictly objectively and in first person, what THIS screen showed and where the physical action landed.
|
||||
You are the Step Summarizer for a mobile and computer UI automation agent. Exactly ONE screenshot is available for this step — the decision frame captured when the action was issued. There is NO after-action screenshot. Your task is to describe, strictly objectively and in first person, what THIS screen showed and where the physical action landed.
|
||||
|
||||
---
|
||||
|
||||
@@ -31,11 +31,16 @@ You are the Step Summarizer for an Android UI automation agent. Exactly ONE scre
|
||||
|
||||
---
|
||||
|
||||
# EVIDENCE FIDELITY
|
||||
保留设备/窗口/页面来源及本步编号，不把手机观察归给电脑，或把另一标签页当成目标页面。关键原文（名称、编号、金额、备案文字、开关状态）只转写清晰可读的内容；模糊字符标明“无法辨清”，不凭上下文补全、不改成看起来合理的词。截图、控件树或先前摘要冲突时分别注明读数与来源并标记待核实，不自动选一个当事实。用户意图和操作理由只引导注意力，不证明界面状态。摘要是观察索引，不能替代原始证据作最终裁定。
|
||||
|
||||
---
|
||||
|
||||
# OPERATOR FOCUS (ATTENTION GUIDANCE)
|
||||
The input may open with an `--- [0] OPERATOR FOCUS ... ---` block: the operator's own words about this step — the task goal, the active sub-goal, a user instruction, and the operator's reasoning naming the target it chose and the screen change it expected. The action line shows the operator's target as `'...' (self-described)` (its own statement) or as `'...'` (an element observed in the UI tree). Use all of this to direct your attention, never to supply facts:
|
||||
1. **Focused details are mandatory**: Every visible detail that relates to the named target, the stated expectation (as it stands on THIS frame only), the sub-goal or the goal — exact labels, values, amounts, counts, toggle/selection/enabled/loading states, positions — must be transcribed verbatim. Never paraphrase or skip them. If the target control is visible, name its exact label.
|
||||
1. **Focused details are mandatory**: Every visible detail that relates to the named target, the stated expectation (as it stands on THIS frame only), the sub-goal or the goal — exact labels, values, amounts, counts, toggle/selection/enabled/loading states, positions — must be transcribed verbatim. Never paraphrase clear text; mark unreadable characters explicitly rather than guessing. If the target control is visible, name its exact label.
|
||||
2. **Absence is a fact**: If something the operator expected or looked for is NOT visible, say so plainly (e.g., "no confirmation dialog is visible", "the 'Pay' button is not on this screen"). Never describe an expected element or change merely because the operator expected it.
|
||||
3. **The rest stays complete**: Focus adds emphasis; it never narrows the description. Keep describing the whole screen and the whole transition at full detail.
|
||||
3. **Relevant context**: Preserve page identity, task-relevant evidence and unexpected changes. Omit repeated navigation chrome and unrelated static content; keep the existing compact paragraph format.
|
||||
4. **Not evidence**: The focus block is intention, not observation. A self-described target is what the operator believed it aimed at; describe what the red marker actually landed on.
|
||||
|
||||
---
|
||||
|
||||
@@ -254,6 +254,13 @@ class VisualStepSummarizer(StepMemoryService):
|
||||
else:
|
||||
key = action_key if action_key is not None else step_number
|
||||
|
||||
receipt_outcome = exec_outcome
|
||||
# Keep the tree in its labelled evidence block, and any distinct
|
||||
# controller status/errors around it. Deduplicate before clipping.
|
||||
if post_img_bytes and post_ui_elements_text and post_ui_elements_text.strip():
|
||||
exec_outcome = exec_outcome.replace(
|
||||
post_ui_elements_text.strip(), "[UI observation in AFTER ACTION UI ELEMENTS]", 1
|
||||
)
|
||||
if action_name.startswith("plug_"):
|
||||
from artemis.config.constants import PLUGIN_SUMMARY_TEXT_CHARS
|
||||
from artemis.plugins.prompt_budget import bounded_result_text
|
||||
@@ -287,7 +294,7 @@ class VisualStepSummarizer(StepMemoryService):
|
||||
if unchanged
|
||||
else f"步骤 {step_number}：只读观察 {action_name}；未执行写操作。"
|
||||
"以下为工具返回摘录，截图仍在本步骤证据中；不代表视觉验收通过。\n"
|
||||
) + bounded_result_text(exec_outcome, limit=PLUGIN_SUMMARY_TEXT_CHARS)
|
||||
) + bounded_result_text(receipt_outcome, limit=PLUGIN_SUMMARY_TEXT_CHARS)
|
||||
# Evidence has already been persisted by the caller. As with a
|
||||
# completed visual summary, do not retain every frame in memory.
|
||||
payload.update(pre_img_bytes=None, post_img_bytes=None, focus=None)
|
||||
@@ -528,8 +535,13 @@ class VisualStepSummarizer(StepMemoryService):
|
||||
HumanMessage(content=content_blocks),
|
||||
]
|
||||
|
||||
from artemis.llm.cache_affinity import cache_affinity_options
|
||||
|
||||
response = await asyncio.wait_for(
|
||||
self._llm.ainvoke(adapt_responses_messages(messages, self._model_name)),
|
||||
self._llm.ainvoke(
|
||||
adapt_responses_messages(messages, self._model_name),
|
||||
**cache_affinity_options(self.ctx, "lens:visual_transition", self._model_name),
|
||||
),
|
||||
timeout=self._timeout_seconds,
|
||||
)
|
||||
self._meter_lens_call(response)
|
||||
|
||||
Reference in new issue
Block a user