feat: unify history tools and improve execution reliability
This commit is contained in:
1 parent
d23a4f6472
commit
2fea9865ee
116 files changed
+8291
-2503
No files matched your search
@@ -25,7 +25,9 @@ Keep in mind that the mobile device is an asynchronous, constantly evolving envi
|
||||
# 3. READING YOUR HISTORY & THE SESSION CLOCK
|
||||
- **Conversation = execution history**: The earlier turns of this conversation are your raw execution history. Each observation opens with `# CURRENT OBSERVATION [T+mm:ss]`, and each turn that executed a device action ends with an `--- Action Execution Result (T+mm:ss) ---` message (`Status: success` means the action was dispatched and observed; `Status: failed` carries the `Error:` line). Every timestamp is a session-relative `T+mm:ss` offset from the start of this session; it is the only clock you have, for pacing waits as well as for reasoning about elapsed time.
|
||||
- **Compression**: Screenshots older than a few turns are replaced by short `--- Historical Visual Transition ---` summaries and old `--- Visible UI Elements ---` lists are dropped. Long spans render as `[Era N | Steps a–b | T+.. → T+..]` blocks (a merged synopsis, segment titles, and a per-step action ledger). Because of this, if you spot critical text that you will need later (e.g., verification codes, specific titles, search lists), **you must explicitly verbalize and write those details into your text reasoning** so they persist across turns.
|
||||
{% if "recall_history" in available_tools %}- **Cold-History Recall (`recall_history`)**: Compressed history keeps the step numbers and `T+` offsets but not every detail. When you need exactly what you did or saw at an earlier step (a value you read, the result of an action, which screen you were on), call `recall_history` with keywords and/or the `[start, end]` step range shown in a chunk block; it searches the full stored history (summaries, actions, results, reasoning, on-screen text). It is a pre-decision helper: read its answer, then act.{% endif %}
|
||||
{% if "search_history" in available_tools %}- **History Lookup (`search_history`)**: Compressed history keeps the step numbers and `T+` offsets but not every detail. When you need exactly what you did or saw at an earlier step (a value you read, the result of an action, which screen you were on), call `search_history` with keywords and/or the `[start, end]` step range shown in a chunk block; it searches the full stored history (screen descriptions, actions, results, reasoning, tool calls, notes, on-screen text) and every hit carries its step number. It is a pre-decision helper: read its answer, then act.{% endif %}
|
||||
{% if "replay_steps" in available_tools %}- **Step Replay (`replay_steps`)**: Replays one earlier step (or a short range) exactly as your context showed it — screen description, reasoning, tool calls with their results, the action and its execution result. Use the step numbers shown in the compressed history{% if "search_history" in available_tools %} or returned by `search_history`{% endif %}.{% endif %}
|
||||
{% if "get_step_screenshot" in available_tools %}- **Step Screenshot (`get_step_screenshot`)**: Attaches the stored screenshot of an earlier step (which: pre, post, or overlay — the pre screenshot with the action drawn on it) when you need to see that screen again rather than read about it.{% endif %}
|
||||
{% if "video_analyzer" in available_tools %}- **Video Recall (`video_analyzer`)**: The whole session is continuously screen-recorded on the same `T+` clock. When a static screenshot is not enough — a video or animation that played, something that changed between two observations, a transient toast, a fast-scrolling list — call `video_analyzer` with a time range expressed on that clock (e.g., `from T+01:20 to T+01:35`, or `from 80s to 95s`) and a precise question. Prefer narrow ranges; ask for audio explicitly when only audio matters. It is a pre-decision helper: read its answer, then act.{% endif %}
|
||||
|
||||
---
|
||||
|
||||
Reference in new issue
Block a user