fix: improve cross-device evidence and exploration lifecycle
Align Responses cache affinity across execution and summary calls, preserve required checks and evidence uncertainty, and allow safe deletion of inactive explorations while retaining independent reruns. Validation: backend suite passed except intentional prompt golden drift, then all 17 transcript tests passed after snapshot refresh; 268 frontend tests and production build passed; Ruff, quality ratchet and core type checks passed.
This commit is contained in:
1 parent
f1bb5d02aa
commit
2328a4df0c
32 files changed
+748
-79
No files matched your search
@@ -42,7 +42,7 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
|
||||
---
|
||||
|
||||
# 4. STAGNATION & LOOP PREVENTION
|
||||
- **Retrying is not the only option.**: If you perform the same action on the same target for two consecutive turns and the screen or the UI element list shows **no substantial change**, further retries are unlikely to be useful; the probability of the execution tool itself malfunctioning is very low, so you should pause and consider the underlying reason.
|
||||
- **Retrying is not the only option.**: If you perform the same action on the same target for two consecutive turns and the screen or the UI element list shows **no substantial change**, further retries are unlikely to be useful; inspect the actual tool result, target, coordinate space, focus, overlays and scroll container before choosing a different strategy. Do not assume delivery proves an effect or dismiss a tool error.
|
||||
- **Flexibly Adjust Strategies:** Always prioritize resolving the immediate issue. If you encounter overlays or pop-ups blocking input, or if an element is unresponsive, actively consider alternative ways to complete the task. If an action fails to proceed as expected, systematically consider whether the issue lies with the device itself, the operational path, or a change in state occurring between your actual interaction and the screenshot (Time-Sensitive Tasks).
|
||||
|
||||
---
|
||||
@@ -62,4 +62,6 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
|
||||
|
||||
|
||||
## 收尾与跨端验收
|
||||
切换到另一端之前,核对当前端的必需验收动作是否已完成并留下证据。提交 report_task_status 前逐项对照用户要求,缺少必要且仍可安全取得的观察时只补该步骤,不重跑已完成部分;无法补齐时明确列出缺项。看到页脚不自动等于滚动已到底,若用户要求验证位置不变,需追加滚动并比较本端前后画面。截图来源、步骤和实际内容共同支持结论,执行成功回执本身不证明验收通过。
|
||||
首次拆解目标时,把用户明确要求的前置条件、开关状态、检查动作和交付物纳入里程碑;区分待检查、已有证据、受阻,不把计划或已发出动作当成已验证。沿用现有推理和历史记录,仅在状态变化时简短记录“要求 → 来源/步骤 → 实际观察”,不每轮重抄清单或新建规划流程。
|
||||
切换到另一端前和结束前,逐项核对这些要求(含最新用户指导)。缺少必要且仍可安全取得的证据时只补该步骤,不重跑已有证据的部分;无法补齐时明确列出缺项。看到页脚不自动等于滚动已到底;用户要求边界验证时,需在正确滚动容器内追加滚动并比较本端前后画面。
|
||||
关键文字、数值或开关状态若在摘要中冲突或不清晰,回查对应步骤原图/控件树,保留来源和不确定性;不按常识、期望或多数摘要猜结论。手机与电脑证据分别归属;仅在同条件、可比较的观察支持时才报告跨端差异。执行成功回执本身不证明验收通过。
|
||||
@@ -5,7 +5,7 @@
|
||||
|
||||
|
||||
# ROLE & OBJECTIVE
|
||||
You are the Step Summarizer for an Android UI automation agent. Your task is to synthesize the Before/After screenshots and physical action into a concise, high-information-density, strictly objective first-person historical memory.
|
||||
You are the Step Summarizer for a mobile and computer UI automation agent. Your task is to synthesize the Before/After screenshots and physical action into a concise, high-information-density, strictly objective first-person historical memory.
|
||||
|
||||
---
|
||||
|
||||
@@ -28,11 +28,16 @@ You are the Step Summarizer for an Android UI automation agent. Your task is to
|
||||
|
||||
---
|
||||
|
||||
# EVIDENCE FIDELITY
|
||||
保留设备/窗口/页面来源及本步编号,不把手机观察归给电脑,或把另一标签页当成目标页面。关键原文(名称、编号、金额、备案文字、开关状态)只转写清晰可读的内容;模糊字符标明“无法辨清”,不凭上下文补全、不改成看起来合理的词。截图、控件树或先前摘要冲突时分别注明读数与来源并标记待核实,不自动选一个当事实。用户意图和操作理由只引导注意力,不证明界面状态。摘要是观察索引,不能替代原始证据作最终裁定。
|
||||
|
||||
---
|
||||
|
||||
# OPERATOR FOCUS (ATTENTION GUIDANCE)
|
||||
The input may open with an `--- [0] OPERATOR FOCUS ... ---` block: the operator's own words about this step — the task goal, the active sub-goal, a user instruction, and the operator's reasoning naming the target it chose and the screen change it expected. The action line shows the operator's target as `'...' (self-described)` (its own statement) or as `'...'` (an element observed in the UI tree). Use all of this to direct your attention, never to supply facts:
|
||||
1. **Focused details are mandatory**: Every visible detail that relates to the named target, the stated expectation, the sub-goal or the goal — exact labels, values, amounts, counts, toggle/selection/enabled/loading states, positions — must be transcribed verbatim. Never paraphrase or skip them. If the target control is visible, name its exact label.
|
||||
1. **Focused details are mandatory**: Every visible detail that relates to the named target, the stated expectation, the sub-goal or the goal — exact labels, values, amounts, counts, toggle/selection/enabled/loading states, positions — must be transcribed verbatim. Never paraphrase clear text; mark unreadable characters explicitly rather than guessing. If the target control is visible, name its exact label.
|
||||
2. **Absence is a fact**: If something the operator expected or looked for is NOT visible, say so plainly (e.g., "no confirmation dialog is visible", "the 'Pay' button is not on this screen"). Never describe an expected element or change merely because the operator expected it.
|
||||
3. **The rest stays complete**: Focus adds emphasis; it never narrows the description. Keep describing the whole screen and the whole transition at full detail.
|
||||
3. **Relevant context**: Preserve page identity, task-relevant evidence and unexpected changes. Omit repeated navigation chrome and unrelated static content; keep the existing compact paragraph format.
|
||||
4. **Not evidence**: The focus block is intention, not observation. A self-described target is what the operator believed it aimed at; describe what the red marker actually landed on.
|
||||
|
||||
---
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
|
||||
|
||||
# ROLE & OBJECTIVE
|
||||
You are the Step Summarizer for an Android UI automation agent. Exactly ONE screenshot is available for this step — the decision frame captured when the action was issued. There is NO after-action screenshot. Your task is to describe, strictly objectively and in first person, what THIS screen showed and where the physical action landed.
|
||||
You are the Step Summarizer for a mobile and computer UI automation agent. Exactly ONE screenshot is available for this step — the decision frame captured when the action was issued. There is NO after-action screenshot. Your task is to describe, strictly objectively and in first person, what THIS screen showed and where the physical action landed.
|
||||
|
||||
---
|
||||
|
||||
@@ -31,11 +31,16 @@ You are the Step Summarizer for an Android UI automation agent. Exactly ONE scre
|
||||
|
||||
---
|
||||
|
||||
# EVIDENCE FIDELITY
|
||||
保留设备/窗口/页面来源及本步编号,不把手机观察归给电脑,或把另一标签页当成目标页面。关键原文(名称、编号、金额、备案文字、开关状态)只转写清晰可读的内容;模糊字符标明“无法辨清”,不凭上下文补全、不改成看起来合理的词。截图、控件树或先前摘要冲突时分别注明读数与来源并标记待核实,不自动选一个当事实。用户意图和操作理由只引导注意力,不证明界面状态。摘要是观察索引,不能替代原始证据作最终裁定。
|
||||
|
||||
---
|
||||
|
||||
# OPERATOR FOCUS (ATTENTION GUIDANCE)
|
||||
The input may open with an `--- [0] OPERATOR FOCUS ... ---` block: the operator's own words about this step — the task goal, the active sub-goal, a user instruction, and the operator's reasoning naming the target it chose and the screen change it expected. The action line shows the operator's target as `'...' (self-described)` (its own statement) or as `'...'` (an element observed in the UI tree). Use all of this to direct your attention, never to supply facts:
|
||||
1. **Focused details are mandatory**: Every visible detail that relates to the named target, the stated expectation (as it stands on THIS frame only), the sub-goal or the goal — exact labels, values, amounts, counts, toggle/selection/enabled/loading states, positions — must be transcribed verbatim. Never paraphrase or skip them. If the target control is visible, name its exact label.
|
||||
1. **Focused details are mandatory**: Every visible detail that relates to the named target, the stated expectation (as it stands on THIS frame only), the sub-goal or the goal — exact labels, values, amounts, counts, toggle/selection/enabled/loading states, positions — must be transcribed verbatim. Never paraphrase clear text; mark unreadable characters explicitly rather than guessing. If the target control is visible, name its exact label.
|
||||
2. **Absence is a fact**: If something the operator expected or looked for is NOT visible, say so plainly (e.g., "no confirmation dialog is visible", "the 'Pay' button is not on this screen"). Never describe an expected element or change merely because the operator expected it.
|
||||
3. **The rest stays complete**: Focus adds emphasis; it never narrows the description. Keep describing the whole screen and the whole transition at full detail.
|
||||
3. **Relevant context**: Preserve page identity, task-relevant evidence and unexpected changes. Omit repeated navigation chrome and unrelated static content; keep the existing compact paragraph format.
|
||||
4. **Not evidence**: The focus block is intention, not observation. A self-described target is what the operator believed it aimed at; describe what the red marker actually landed on.
|
||||
|
||||
---
|
||||
|
||||
@@ -254,6 +254,13 @@ class VisualStepSummarizer(StepMemoryService):
|
||||
else:
|
||||
key = action_key if action_key is not None else step_number
|
||||
|
||||
receipt_outcome = exec_outcome
|
||||
# Keep the tree in its labelled evidence block, and any distinct
|
||||
# controller status/errors around it. Deduplicate before clipping.
|
||||
if post_img_bytes and post_ui_elements_text and post_ui_elements_text.strip():
|
||||
exec_outcome = exec_outcome.replace(
|
||||
post_ui_elements_text.strip(), "[UI observation in AFTER ACTION UI ELEMENTS]", 1
|
||||
)
|
||||
if action_name.startswith("plug_"):
|
||||
from artemis.config.constants import PLUGIN_SUMMARY_TEXT_CHARS
|
||||
from artemis.plugins.prompt_budget import bounded_result_text
|
||||
@@ -287,7 +294,7 @@ class VisualStepSummarizer(StepMemoryService):
|
||||
if unchanged
|
||||
else f"步骤 {step_number}:只读观察 {action_name};未执行写操作。"
|
||||
"以下为工具返回摘录,截图仍在本步骤证据中;不代表视觉验收通过。\n"
|
||||
) + bounded_result_text(exec_outcome, limit=PLUGIN_SUMMARY_TEXT_CHARS)
|
||||
) + bounded_result_text(receipt_outcome, limit=PLUGIN_SUMMARY_TEXT_CHARS)
|
||||
# Evidence has already been persisted by the caller. As with a
|
||||
# completed visual summary, do not retain every frame in memory.
|
||||
payload.update(pre_img_bytes=None, post_img_bytes=None, focus=None)
|
||||
@@ -528,8 +535,13 @@ class VisualStepSummarizer(StepMemoryService):
|
||||
HumanMessage(content=content_blocks),
|
||||
]
|
||||
|
||||
from artemis.llm.cache_affinity import cache_affinity_options
|
||||
|
||||
response = await asyncio.wait_for(
|
||||
self._llm.ainvoke(adapt_responses_messages(messages, self._model_name)),
|
||||
self._llm.ainvoke(
|
||||
adapt_responses_messages(messages, self._model_name),
|
||||
**cache_affinity_options(self.ctx, "lens:visual_transition", self._model_name),
|
||||
),
|
||||
timeout=self._timeout_seconds,
|
||||
)
|
||||
self._meter_lens_call(response)
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
{
|
||||
"main_template": "# 输出语言要求\n所有自行撰写的自然语言说明必须使用简体中文,包括每一步的操作说明、计划与子任务、检查反馈、诊断结论、历史摘要和最终报告。英文示例仅示范格式,不决定输出语言。\n保留工具名、参数名、JSON 字段名、枚举值、状态码、计划语法标记、模板变量、命令、路径、URL、包名及标识符原文。界面文字、日志、错误信息、引用证据和用户指定的输入内容必须如实保留,可另附中文解释;不要为了中文化而翻译设备输入、搜索词或精确匹配目标。\n遵守各自原有输出结构;结构化结果只将允许自由填写的说明文本写成简体中文。只描述已有证据,不编造操作结果。\n\n每次调用设备操作工具前,用一小段简体中文说明当前观察、即将执行的动作和目的;操作后仅在获得结果或新观察时说明实际变化。不要只在最终答复中使用中文。工具参数中的自由描述文本也使用简体中文,精确引用的界面文字仍保留原文。\n\n# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n[% if \"wait_for_delay\" in available_tools %]- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n[% endif %]- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.\n 2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading state. This boundary probe must remain unidirectional; never reverse direction merely to prove completion. If the task declares an earlier exit condition (for example, find one verified match or process N items), that explicit condition may end the traversal without reaching the list boundary.\n 3. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtLine truncated
|
||||
"main_template": "# 输出语言要求\n所有自行撰写的自然语言说明必须使用简体中文,包括每一步的操作说明、计划与子任务、检查反馈、诊断结论、历史摘要和最终报告。英文示例仅示范格式,不决定输出语言。\n保留工具名、参数名、JSON 字段名、枚举值、状态码、计划语法标记、模板变量、命令、路径、URL、包名及标识符原文。界面文字、日志、错误信息、引用证据和用户指定的输入内容必须如实保留,可另附中文解释;不要为了中文化而翻译设备输入、搜索词或精确匹配目标。\n遵守各自原有输出结构;结构化结果只将允许自由填写的说明文本写成简体中文。只描述已有证据,不编造操作结果。\n\n每次调用设备操作工具前,用一小段简体中文说明当前观察、即将执行的动作和目的;操作后仅在获得结果或新观察时说明实际变化。不要只在最终答复中使用中文。工具参数中的自由描述文本也使用简体中文,精确引用的界面文字仍保留原文。\n\n# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **要求与证据**:沿用现有任务计划,将用户明确要求的前置条件、开关状态、检查动作和交付物列为必要项,区分待检查、已有证据、受阻。仅在状态变化时简短记下要求、来源/步骤和实际观察,不每轮重抄清单。跨端切换前和结束前核对必要项及最新用户指导,只补尚缺且可安全取得的证据;无法补齐时明确列出缺项。用户希望发生、动作已发出或计划已勾选都不证明验收通过。\n- **证据冲突**:关键文字、数值或开关状态有冲突/不清晰时,回查对应步骤原图或控件树,保留来源和不确定性,不按常识或多数摘要猜测。手机与电脑证据分别归属,仅在同条件观察支持时报告跨端差异。保持已有可靠证据,只回查缺失或矛盾部分。\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n[% if \"wait_for_delay\" in available_tools %]- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n[% endif %]- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor and carry that anchor plus the scan direction in short-term memory or the traversal ledger. After the scroll, evaluate this handoff before doing anything unrelated: if the anchor remains visible with new content beyond it, record the new content and continue in the same direction without scrolling back to verify; if the anchor is missing, continuity is uncertain, so perform at most one minimal reverse recovery to re-establish overlap, then resume the original direction. While the traversal's declared exit condition remains unresolved, keep this traversal active and do not advance to another milestone or mark it complete.\n 2. **Boundary-Proven Completion**: A single scroll that reveals no new content is not sufficient evidence that the list is complete. For an exhaustive traversal, stop only when an explicit end-of-list indicator is visible, or when one additional successful swipe in the same direction and within the same list container leaves the same final anchor visible with no new items and no loading statLine truncated
|
||||
}
|
||||
@@ -81,6 +81,7 @@ async def discuss(repository, session_id, message):
|
||||
"""Explicit message POST may start a reply; reads never start any work."""
|
||||
key = message["id"]
|
||||
with repository._transaction() as connection:
|
||||
repository._require(connection, session_id)
|
||||
old = repository._get(connection, "exploration_discussion", key)
|
||||
if old and old["status"] in {"running", "completed"}:
|
||||
return
|
||||
|
||||
@@ -62,6 +62,8 @@ class ExplorationStore:
|
||||
session = self._get(connection, "exploration", identifier(session_id))
|
||||
if session is None:
|
||||
raise KeyError("探索会话不存在")
|
||||
if session.get("status") == "deleting":
|
||||
raise ExplorationConflict("探索正在删除,不能继续操作")
|
||||
return session
|
||||
|
||||
def _operation(
|
||||
@@ -75,6 +77,8 @@ class ExplorationStore:
|
||||
if previous:
|
||||
if previous["fingerprint"] != fingerprint:
|
||||
raise ExplorationConflict("同一 request_id 的内容与已提交请求不同")
|
||||
if previous.get("deleted"):
|
||||
raise ExplorationConflict("此请求关联的探索已删除,请重新创建任务")
|
||||
return previous["result"]
|
||||
result = operation()
|
||||
self._put(
|
||||
@@ -299,6 +303,59 @@ class ExplorationStore:
|
||||
).fetchall()
|
||||
return [json.loads(row[0]) for row in rows]
|
||||
|
||||
def begin_delete(self, session_id: str, check_runs: Callable[[list[dict]], None]) -> list[dict]:
|
||||
"""Fence new work before deleting files; failed cleanup can be retried."""
|
||||
with self._transaction() as connection:
|
||||
session = self._get(connection, "exploration", identifier(session_id))
|
||||
if session is None:
|
||||
raise KeyError("探索会话不存在")
|
||||
if session.get("active_run_id") or session.get("active_demonstration_id"):
|
||||
raise ExplorationConflict("探索仍有执行或示范任务,请先停止并等待结束后删除")
|
||||
rows = connection.execute(
|
||||
"SELECT kind,payload FROM automation_records "
|
||||
"WHERE json_extract(payload,'$.exploration_id')=?",
|
||||
(session_id,),
|
||||
).fetchall()
|
||||
runs = []
|
||||
for row in rows:
|
||||
item = json.loads(row["payload"])
|
||||
if row["kind"] == "exploration_run":
|
||||
if item.get("outcome") is None:
|
||||
raise ExplorationConflict("探索执行尚未结束,请先停止并等待结束后删除")
|
||||
runs.append(item)
|
||||
elif row["kind"] in {"exploration_generation", "exploration_discussion"}:
|
||||
if item.get("status") in {"pending", "running"}:
|
||||
raise ExplorationConflict("探索正在分析或生成内容,请等待结束后删除")
|
||||
check_runs(runs)
|
||||
session["status"] = "deleting"
|
||||
self._save_session(connection, session)
|
||||
return runs
|
||||
|
||||
def finish_delete(self, session_id: str) -> None:
|
||||
with self._transaction() as connection:
|
||||
session = self._get(connection, "exploration", session_id)
|
||||
if not session or session.get("status") != "deleting":
|
||||
raise ExplorationConflict("探索尚未准备删除")
|
||||
connection.execute(
|
||||
"DELETE FROM automation_records WHERE kind IN "
|
||||
"('exploration_run','exploration_message','exploration_case',"
|
||||
"'exploration_demo','exploration_generation','exploration_discussion') "
|
||||
"AND json_extract(payload,'$.exploration_id')=?",
|
||||
(session_id,),
|
||||
)
|
||||
connection.execute(
|
||||
"DELETE FROM automation_records WHERE kind='exploration' AND id=?", (session_id,)
|
||||
)
|
||||
# Retain only idempotency tombstones, not snapshots of deleted messages/cases.
|
||||
connection.execute(
|
||||
"UPDATE automation_records SET payload="
|
||||
"json_set(json_remove(payload,'$.result'),'$.deleted',json('true')) "
|
||||
"WHERE kind='exploration_request' AND "
|
||||
"(json_extract(payload,'$.result.id')=? OR "
|
||||
"json_extract(payload,'$.result.exploration_id')=?)",
|
||||
(session_id, session_id),
|
||||
)
|
||||
|
||||
def set_run_state(
|
||||
self,
|
||||
run_id: str,
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
"""Per-conversation routing hints shared by agent and auxiliary Responses calls."""
|
||||
|
||||
from collections.abc import Mapping
|
||||
import hashlib
|
||||
import re
|
||||
from typing import Any
|
||||
from uuid import UUID
|
||||
|
||||
|
||||
def cache_affinity_options(
|
||||
ctx: Any, role: str | None, model_name: str | None, options: dict | None = None
|
||||
) -> dict:
|
||||
"""Keep body and header routing keys aligned without changing model defaults.
|
||||
|
||||
An exploration survives multiple execution sessions. Each model role keeps
|
||||
its own cache scope so background summaries do not compete with the agent.
|
||||
Set hints per call: raw models/HTTP clients may be shared by concurrent tasks.
|
||||
Explicit caller keys and headers take precedence (including underscore aliases).
|
||||
"""
|
||||
result = dict(options or {})
|
||||
engine = getattr(ctx, "data_engine", None)
|
||||
session = getattr(ctx, "exploration_id", None) or getattr(engine, "current_session_id", None)
|
||||
if isinstance(session, (str, UUID)) and str(session):
|
||||
identity = f"{session}:{role or 'model'}:{model_name}"
|
||||
result.setdefault(
|
||||
"prompt_cache_key", "artemis:" + hashlib.sha256(identity.encode()).hexdigest()[:40]
|
||||
)
|
||||
key = result.get("prompt_cache_key")
|
||||
if not isinstance(key, str) or not re.fullmatch(r"[-0-9A-Za-z._:]{1,128}", key):
|
||||
return result
|
||||
existing = result.get("extra_headers")
|
||||
if existing is not None and not isinstance(existing, Mapping):
|
||||
return result
|
||||
headers = dict(existing or {})
|
||||
names = {name.lower().replace("_", "-") for name in headers}
|
||||
for name in ("Thread-Id", "Session-Id"):
|
||||
if name.lower() not in names:
|
||||
headers[name] = key
|
||||
result["extra_headers"] = headers
|
||||
return result
|
||||
@@ -4,6 +4,7 @@ import base64
|
||||
from contextvars import ContextVar
|
||||
from copy import deepcopy
|
||||
import io
|
||||
import hashlib
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
@@ -208,6 +209,26 @@ class RequestAttempt:
|
||||
self.publish()
|
||||
|
||||
|
||||
async def observe_request_headers(request):
|
||||
"""Correlate a wire request and record only hashes of routing headers."""
|
||||
attempt = ACTIVE_ATTEMPT.get()
|
||||
if attempt is None:
|
||||
return
|
||||
request.headers.setdefault("X-Client-Request-Id", attempt.data["request_id"])
|
||||
routing = {}
|
||||
for name in ("thread-id", "thread_id", "session-id", "session_id"):
|
||||
value = request.headers.get(name)
|
||||
if value is not None:
|
||||
routing[name] = hashlib.sha256(value.encode()).hexdigest()[:16]
|
||||
attempt.data.update(
|
||||
request_sent_at=time.time(),
|
||||
routing_header_fingerprints=routing,
|
||||
client_request_id_matches=request.headers.get("X-Client-Request-Id")
|
||||
== attempt.data["request_id"],
|
||||
)
|
||||
attempt.publish()
|
||||
|
||||
|
||||
async def observe_response_headers(response):
|
||||
"""httpx hook: only allowlisted headers, never body/auth/request headers."""
|
||||
attempt = ACTIVE_ATTEMPT.get()
|
||||
|
||||
@@ -89,7 +89,7 @@ class ModelFactory:
|
||||
from langchain_openai import ChatOpenAI
|
||||
from openai import DefaultAsyncHttpxClient
|
||||
|
||||
from artemis.llm.diagnostics import observe_response_headers
|
||||
from artemis.llm.diagnostics import observe_request_headers, observe_response_headers
|
||||
|
||||
api_key = endpoint.api_key or (
|
||||
settings.OPENAI_API_KEY.get_secret_value() if settings.OPENAI_API_KEY else None
|
||||
@@ -114,7 +114,10 @@ class ModelFactory:
|
||||
"store": False,
|
||||
"http_async_client": DefaultAsyncHttpxClient(
|
||||
timeout=endpoint.timeout_seconds,
|
||||
event_hooks={"response": [observe_response_headers]},
|
||||
event_hooks={
|
||||
"request": [observe_request_headers],
|
||||
"response": [observe_response_headers],
|
||||
},
|
||||
),
|
||||
}
|
||||
return ChatOpenAI(**{k: v for k, v in kwargs.items() if v is not None})
|
||||
@@ -504,8 +504,16 @@ class StepCapsuleLens(StepLens):
|
||||
return parsed
|
||||
|
||||
async def _invoke(self, llm: Any, messages: list[BaseMessage], model_label: str):
|
||||
from artemis.llm.cache_affinity import cache_affinity_options
|
||||
|
||||
request_messages = adapt_responses_messages(messages, model_label)
|
||||
response = await asyncio.wait_for(llm.ainvoke(request_messages), timeout=90.0)
|
||||
response = await asyncio.wait_for(
|
||||
llm.ainvoke(
|
||||
request_messages,
|
||||
**cache_affinity_options(self._ctx, "lens:step_capsule", model_label),
|
||||
),
|
||||
timeout=90.0,
|
||||
)
|
||||
# Raw-model bypass metering (gateway-wrapped models meter themselves);
|
||||
# capsule prompts are small and are kept out of the session meter's
|
||||
# last_prompt_tokens (the compaction base itself is operator-owned on
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# ROLE
|
||||
You are the Segment Capsule writer for a mobile automation agent's history compression. You receive one contiguous segment of executed steps and produce the two LLM-authored bands of the segment's history chunk:
|
||||
You are the Segment Capsule writer for a mobile and computer automation agent's history compression. You receive one contiguous segment of executed steps and produce the two LLM-authored bands of the segment's history chunk:
|
||||
|
||||
- Band ① — Synopsis & effects: what this segment was doing, what was actually done, and what effects/artifacts it left behind, plus structured fields.
|
||||
- Band ② — Compressed step summary: an interval narrative ("Steps xx–xx did ..., Step xx did ...") covering every step of the segment.
|
||||
@@ -21,7 +21,8 @@ A turn block without a transcript falls back to a reasoning excerpt; treat it th
|
||||
2. **First-person perspective**: write from the agent's perspective using "I". Never "the agent" / "the operator".
|
||||
3. **Planned iterations are not anomalies**: systematic processing of different candidate items, or periodic polling cycles under a `[Loop]` / `[Loop:continuous]` milestone, is planned progress — record it objectively (e.g. "polling round #2"). Only flag a loop when the exact same target is repeated without progress or states oscillate blindly; then record the pattern and count factually.
|
||||
4. **Preserve, don't abstract**: be deliberately detailed and stay close to the source wording. Prefer longer output over abstracting facts beyond recognition. Faithfully carry visible text, codes, prices, account names, file names, error banners — and the concrete content of tool results the operator relied on (an explorer's answer, a recalled step, a note's contents, a command's output).
|
||||
5. **Fact priority**: user instructions > validator/controller recorded results > tool results returned to the operator (explorer / recall / notes / ADB) > UI/OCR deltas > visual transition summaries > the operator's reasoning.
|
||||
5. **Requirements are not observations**: user instructions govern what to do, not what happened. Controller receipts establish delivery, not goal achievement. Preserve recorded observations with device/window/page and step attribution; distinguish UI/OCR readings, visual summaries, tool reports and operator hypotheses. Conflicting or unreadable critical text stays unresolved with its competing sources; do not silently choose a spelling or merge different devices' observations into one fact.
|
||||
6. **Requirement continuity**: preserve outstanding required checks, blockers and evidence links in the existing fields. Carry observed requirement facts in `verified_facts` with source/step; keep missing checks and contradictions in `unresolved`. Planned actions, user expectations and checked plan boxes alone cannot promote an item to verified. Do not repeat the same checklist across all bands.
|
||||
|
||||
# BAND ① REQUIREMENTS
|
||||
Band ① is the segment-level account; band ② below it is the interval-level account. The two bands must not say the same thing twice: ① never re-lists what the intervals of ② already narrate, and it should come out shorter than ② (the `doing` + `did` + `effect` prose together — not counting the structured fields — shorter than the ② intervals together).
|
||||
|
||||
@@ -139,6 +139,13 @@ class PluginRuntime:
|
||||
lines.append(
|
||||
"只调用实际提供的工具。电脑插件的像素坐标以其返回的截图为准,不使用 Android 的 0–1000 坐标归一化。每个资源的截图和读数必须分开核对;只有文字观察时不能猜像素坐标。"
|
||||
)
|
||||
lines.append(
|
||||
"电脑操作及保存截图前,用已有观察核对应用、窗口和页面身份;浏览器结合地址栏、"
|
||||
"选中标签与页面内容,不能仅凭窗口还存在就认定页面正确。身份不符时先定位目标。"
|
||||
"动作返回的新截图可能改变尺寸与编号,坐标和元素标识只取自同一张最新图;"
|
||||
"不沿用旧图坐标、不自行叠加 DPI 缩放。无位移时先排查回执、焦点、遮挡和滚动容器,"
|
||||
"不连续重试同一点,也不把无效滚动当作已经到底。"
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
async def load(self, plugin_id: str) -> PluginCallResult:
|
||||
|
||||
+3
-12
@@ -23,14 +23,13 @@ from collections.abc import Awaitable, Callable, Coroutine
|
||||
from contextvars import ContextVar, Token
|
||||
from contextlib import aclosing
|
||||
import functools
|
||||
import hashlib
|
||||
import logging
|
||||
from pathlib import Path
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from typing import Any, Literal, TypeVar, overload
|
||||
from uuid import UUID, uuid4
|
||||
from uuid import uuid4
|
||||
|
||||
from langchain_core.language_models.chat_models import BaseChatModel
|
||||
from langchain_core.messages import AIMessage, BaseMessage, HumanMessage
|
||||
@@ -48,6 +47,7 @@ from artemis.config import (
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.data_engine.trace import CURRENT_TRACE_ID, DataEngineCallbackHandler
|
||||
from artemis.llm.diagnostics import ACTIVE_ATTEMPT, RequestAttempt, error_text, input_images
|
||||
from artemis.llm.cache_affinity import cache_affinity_options
|
||||
from artemis.llm.reliability import (
|
||||
CircuitBreaker,
|
||||
FailureCategory,
|
||||
@@ -637,16 +637,7 @@ class RobustChatModelWrapper:
|
||||
|
||||
def _prepare_model_call(self, args: tuple, kwargs: dict) -> tuple[tuple, dict]:
|
||||
model_name = self.endpoint.model_name if self.endpoint else None
|
||||
engine = getattr(self.ctx, "data_engine", None)
|
||||
session = getattr(self.ctx, "exploration_id", None) or getattr(
|
||||
engine, "current_session_id", None
|
||||
)
|
||||
if isinstance(session, (str, UUID)) and str(session):
|
||||
identity = f"{session}:{self.role or 'model'}:{model_name}"
|
||||
kwargs = {
|
||||
"prompt_cache_key": "artemis:" + hashlib.sha256(identity.encode()).hexdigest()[:40],
|
||||
**kwargs,
|
||||
}
|
||||
kwargs = cache_affinity_options(self.ctx, self.role, model_name, kwargs)
|
||||
if (
|
||||
args
|
||||
and isinstance(args[0], (list, tuple))
|
||||
|
||||
Reference in new issue
Block a user