feat: enable remote SDK and resilient device orchestration
This commit is contained in:
somew1nd committed
1 parent
3cc711df51
commit
ddfa420945
110 files changed
+14450
-12954
No files matched your search
@@ -0,0 +1,897 @@
|
||||
# ARTEMIS Pro Context, History, and Summarization Redesign
|
||||
|
||||
Status: Proposal (2026-08-25)
|
||||
|
||||
Scope: ARTEMIS Pro / LangGraph execution mode
|
||||
|
||||
Audience: maintainers of Operator, Validator, Summarizer, DataEngine, and model routing
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
This proposal improves long-term history, visually grounded neutral summaries,
|
||||
context compression, and history recall without weakening Pro mode's current
|
||||
reasoning continuity. The goal is not to give Operator more raw history by
|
||||
default. It is to increase information density while preserving the current
|
||||
`10K-12K` input range and moving all additional image interpretation to a
|
||||
non-blocking background path.
|
||||
|
||||
The design preserves mechanisms that already work:
|
||||
|
||||
- the previous Operator step's complete raw or native thinking;
|
||||
- the exact previous action and Validator result;
|
||||
- the current screenshot and UI element list;
|
||||
- complete, auditable raw step records in DataEngine; and
|
||||
- asynchronous Summarizer scheduling that never blocks the graph's critical path.
|
||||
|
||||
It adds the following capabilities:
|
||||
|
||||
1. Preserve the previous model message, thought signature, action call, and Validator result as a continuous conversation tail.
|
||||
2. Keep the current neutral summary contract while allowing the Summarizer to use the decision screenshot and an optional post-action screenshot as context.
|
||||
3. Bind thinking, actions, available images, validation results, and summaries with a stable `step_id`.
|
||||
4. Treat the active milestone only as a temporary working-set label; use plan-independent history chunks for long-term compression.
|
||||
5. Add a bounded, on-demand `recall_history` tool for complete steps, older screenshots, and UI text.
|
||||
6. Introduce a fixed token budget, value-density history selection, and same-turn tool-output limits.
|
||||
7. Separate stable system rules from dynamic plans and history to improve prefix caching and multi-turn continuity.
|
||||
|
||||
The steady-state Operator context becomes:
|
||||
|
||||
```text
|
||||
Current screenshot/XML
|
||||
+ complete previous-turn reasoning and action result
|
||||
+ recent visually grounded step summaries
|
||||
+ relevant history-chunk summaries
|
||||
+ cold-history recall on demand
|
||||
```
|
||||
|
||||
The steady-state target remains `10K-12K tokens`, with zero additional model
|
||||
calls and zero summary waits on the critical path.
|
||||
|
||||
## 2. Goals and Constraints
|
||||
|
||||
### 2.1 First Priority: Preserve Speed
|
||||
|
||||
The engineering definition of preserving speed is:
|
||||
|
||||
- no new synchronous model calls in normal steps;
|
||||
- Operator never waits for summaries, indexes, or history chunks;
|
||||
- first-call input P50 and P95 remain close to the current baseline;
|
||||
- background summaries do not compete at the same request priority as Operator;
|
||||
- normal Operator requests still contain only one current screenshot;
|
||||
- history improvements replace low-density content instead of expanding normal context; and
|
||||
- any background failure immediately falls back to the current detailed history.
|
||||
|
||||
Absolute zero latency change cannot be guaranteed by adding synchronous input.
|
||||
This design therefore uses a token-neutral, zero-wait critical path and validates
|
||||
P50 and P90 through paired A/B tests.
|
||||
|
||||
### 2.2 History and Summary Goals
|
||||
|
||||
- Preserve the previous strategy, expected result, and unresolved questions.
|
||||
- Ground older step summaries in available visual evidence instead of guessing from action text.
|
||||
- Keep important history visible across plan rewrites, milestone renames, and goal rollback.
|
||||
- Make every compressed record traceable to original steps, screenshots, and tool traces.
|
||||
- Never discard failures, recovery work, side effects, user constraints, or critical values through ordinary compression.
|
||||
- Allow Operator to recall older information during long tasks.
|
||||
|
||||
### 2.3 Use Model Capacity Effectively
|
||||
|
||||
"Maximize" means maximizing useful reasoning information, not filling a one-million-token window:
|
||||
|
||||
- the strongest model directly handles current visual state, continuous reasoning, and high-value history;
|
||||
- available older screenshots help the Summarizer understand the UI, action target, and explicit records;
|
||||
- long context acts as an exception and deep-recall buffer, not the normal working set;
|
||||
- native multi-turn messages and thought signatures are preserved;
|
||||
- stable system prefixes are reused whenever possible; and
|
||||
- the model does not repeatedly reread duplicate plans, raw logs, or low-value click traces.
|
||||
|
||||
### 2.4 Non-Goals
|
||||
|
||||
- Do not delete or overwrite raw history in DataEngine.
|
||||
- Do not send multiple historical screenshots to Operator by default.
|
||||
- Do not treat plan milestones as permanent fact boundaries.
|
||||
- Do not replace Validator's objective judgment with a summary.
|
||||
- Do not require a new model API in the first phase.
|
||||
- Do not enlarge every step merely to use a larger context window.
|
||||
|
||||
## 3. Current Baseline
|
||||
|
||||
### 3.1 Pro Graph Path
|
||||
|
||||
```text
|
||||
Planner
|
||||
-> Convergence
|
||||
-> Perception
|
||||
-> Operator
|
||||
-> Execution Check
|
||||
-> Validator
|
||||
-> Summarizer
|
||||
-> Convergence
|
||||
```
|
||||
|
||||
The Summarizer node creates a background task and returns immediately. This
|
||||
behavior must remain unchanged.
|
||||
|
||||
### 3.2 Current Operator History
|
||||
|
||||
`build_plan_and_history` currently:
|
||||
|
||||
- renders the latest completed step in detail;
|
||||
- may include raw/native thinking, actions, tool calls, Validator results, and Failure Analyzer traces;
|
||||
- renders older summaries when available while retaining exact actions;
|
||||
- preserves steps under the active subgoal plus the latest three global steps;
|
||||
- puts the full current plan in the Operator prompt; and
|
||||
- rebuilds messages on every Operator turn.
|
||||
|
||||
The current system already has continuity. Removing complete reasoning from the
|
||||
latest step would prevent the model from understanding the previous strategy,
|
||||
expectation, and decision basis.
|
||||
|
||||
### 3.3 Remove Separate Short-Term Memory
|
||||
|
||||
Remove the existing `<short_term_memory>` extraction, state storage, and next-turn
|
||||
injection path without introducing a replacement relay object. Cross-turn
|
||||
continuity should come from the complete previous Operator message, exact action,
|
||||
Validator result, current plan, long-term Notes, and auditable execution history.
|
||||
|
||||
### 3.4 Current Pro Summaries
|
||||
|
||||
The current Pro Summarizer receives only text:
|
||||
|
||||
- current plan and history;
|
||||
- current-step thinking;
|
||||
- action and execution result; and
|
||||
- Failure Analyzer trace.
|
||||
|
||||
It does not load `pre_image_name` or `post_image_name`. Images should reduce
|
||||
context misunderstanding, not expand the Summarizer's authority to judge results.
|
||||
|
||||
### 3.5 Existing Cold History
|
||||
|
||||
Complete steps, screenshots, and UI data already exist in DataEngine.
|
||||
HistoryAnalyzer and Outputter offer partial retrieval, but Operator lacks one
|
||||
unified direct history-recall tool.
|
||||
|
||||
### 3.6 Measured Baseline
|
||||
|
||||
Current identifiable Pro Operator calls in `traces/data_engine.db` show:
|
||||
|
||||
- 74 direct Operator model calls across 43 Operator turns;
|
||||
- median input of about 10,221 tokens;
|
||||
- input P90 of about 11,648 tokens and P95 of about 12,019 tokens;
|
||||
- maximum input of about 12,747 tokens;
|
||||
- median model-call latency of about 5.24 seconds and P90 of about 16.42 seconds;
|
||||
- median full Operator-turn latency of about 20.02 seconds; and
|
||||
- median observed step interval of about 17.15 seconds.
|
||||
|
||||
Input length and model latency show no reliable correlation in the observed
|
||||
`8.8K-12.7K` range. That result must not be extrapolated to an `80K-100K`
|
||||
dynamic context.
|
||||
|
||||
## 4. Design Principles
|
||||
|
||||
### 4.1 Separate Raw Records from the Model View
|
||||
|
||||
DataEngine stores complete facts. The prompt contains only the most valuable view
|
||||
for the current turn. Compression changes what the model sees, never the source record.
|
||||
|
||||
### 4.2 Continuous Reasoning Before Compression
|
||||
|
||||
The complete previous reasoning, action expectation, action call, and Validator
|
||||
result are required on the hot path and cannot be replaced by an ordinary summary.
|
||||
|
||||
### 4.3 Images Provide Context, Not Judgment
|
||||
|
||||
The Summarizer may inspect screenshots that actually exist for a step, but it
|
||||
must preserve the current prompt's neutral responsibility. It may compress
|
||||
intent, strategy, explicit verification, iteration progress, missing conditions,
|
||||
and objective Failure Analyzer intervention. It must not independently decide
|
||||
whether an action succeeded, failed, completed, or navigated to a new page.
|
||||
|
||||
A missing post-action screenshot means only that no independent post-action
|
||||
visual evidence is available. It never means the screen was unchanged or the
|
||||
action had no effect.
|
||||
|
||||
### 4.4 Plans Are Dynamic Indexes, Not History Keys
|
||||
|
||||
The current subgoal may increase relevance but must not control permanent
|
||||
retention. History identity uses stable `step_id` values, time ranges, and source references.
|
||||
|
||||
### 4.5 Validator Facts Take Priority
|
||||
|
||||
```text
|
||||
Explicit user instructions
|
||||
> objective Validator or Controller results
|
||||
> activity, package, XML, or OCR differences
|
||||
> sourced content in visually grounded summaries
|
||||
> Operator or Summarizer inference
|
||||
```
|
||||
|
||||
### 4.6 Background Enhancements Cannot Become Dependencies
|
||||
|
||||
If a visual summary or history chunk is not ready, Operator immediately continues
|
||||
with the existing detailed record.
|
||||
|
||||
## 5. Target Architecture
|
||||
|
||||
```text
|
||||
+-----------------------------+
|
||||
| DataEngine raw facts |
|
||||
| steps / traces / images / UI|
|
||||
+--------------+--------------+
|
||||
|
|
||||
Validator completes step ---------------+
|
||||
|
|
||||
+--------------v--------------+
|
||||
| Step Memory Record |
|
||||
| exact action/result/sources |
|
||||
+-------+--------------+-------+
|
||||
| |
|
||||
immediate | asynchronous
|
||||
| v
|
||||
| +--------------------+
|
||||
| | Context Summarizer |
|
||||
| | decision image + |
|
||||
| | optional post/UI |
|
||||
| +----------+---------+
|
||||
| |
|
||||
+-------v----------------v-------+
|
||||
| Pro Memory Store |
|
||||
| hot history/summaries/chunks |
|
||||
+---------------+---------------+
|
||||
|
|
||||
Perception current screenshot/XML -------+
|
||||
v
|
||||
+-----------------------------+
|
||||
| Token-Budget Prompt Builder |
|
||||
+--------------+--------------+
|
||||
v
|
||||
Operator
|
||||
|
|
||||
recall_history on demand
|
||||
```
|
||||
|
||||
## 6. Layered History Model
|
||||
|
||||
### 6.1 L0: Current Observation
|
||||
|
||||
Every Operator turn receives the current screenshot, package/activity, compact
|
||||
XML/OCR element list, screen dimensions and coordinate mapping, user-injected
|
||||
instructions, active plan path, and background-task state. Historical images do
|
||||
not enter normal requests.
|
||||
|
||||
### 6.2 L1: Hot History and Conversation Tail
|
||||
|
||||
```text
|
||||
Previous AIMessage
|
||||
- model-visible thinking / thought signature
|
||||
- raw reasoning text
|
||||
- turn-ending action tool call
|
||||
|
||||
Validator Function/Tool Result
|
||||
- whether execution occurred
|
||||
- execution state
|
||||
- Failure Analyzer repair
|
||||
- observed post-action state
|
||||
|
||||
Current HumanMessage
|
||||
- current screenshot
|
||||
- current XML
|
||||
```
|
||||
|
||||
Keep one complete prior Operator turn normally and up to two when the last step
|
||||
failed, remains uncertain, is recovering, or has an unfinished strategy. Do not
|
||||
duplicate the same complete reasoning in `Execution History`. Until the provider
|
||||
can preserve real messages and thought signatures, continue rendering raw/native
|
||||
thinking verbatim.
|
||||
|
||||
### 6.3 L2: Visually Grounded, Neutral Step Summaries
|
||||
|
||||
Older steps use the current "summary plus exact action" structure, with a
|
||||
multimodal summary input:
|
||||
|
||||
```text
|
||||
Step N
|
||||
Neutral contextual summary
|
||||
Exact action
|
||||
Validator outcome
|
||||
Unresolved/recovery marker
|
||||
```
|
||||
|
||||
The summary never replaces the exact action or overrides Validator. Screenshots
|
||||
only clarify context; the output does not make a new result judgment.
|
||||
|
||||
### 6.4 Current Work Segment
|
||||
|
||||
The active milestone may organize recent summaries, loop and candidate progress,
|
||||
unresolved items, and the most recent entry state. It remains a temporary label
|
||||
that can change with the plan and never becomes a permanent history identity.
|
||||
|
||||
### 6.5 L3: Stable History Chunks
|
||||
|
||||
Long-term history is chunked by stable step ranges and token thresholds:
|
||||
|
||||
```text
|
||||
Chunk 1: Steps 1-10
|
||||
Chunk 2: Steps 11-19
|
||||
Chunk 3: Steps 20-31
|
||||
```
|
||||
|
||||
Close a chunk after roughly `1.5K-2K tokens` of step summaries or 8-12 steps.
|
||||
Close early on a major package/activity change, user goal change, Failure Analyzer
|
||||
boundary, complete strategy change, long work-segment pause, irreversible side
|
||||
effect, or important external result.
|
||||
|
||||
Each chunk preserves its source range and step IDs, attempted path, verified
|
||||
facts, failed paths, unresolved questions, important entities, entry and exit
|
||||
states, and raw source references. Later plan rewrites do not alter chunk identity.
|
||||
|
||||
### 6.6 L4: Cold History
|
||||
|
||||
Retain every screenshot actually saved for a step, raw XML/OCR, raw/native
|
||||
thinking, complete tool calls and results, ADB and log output, the full Failure
|
||||
Analyzer trace, all summary versions, Notes, and plan revisions. Cold history is
|
||||
excluded from normal prompts and loaded only through local matching or
|
||||
`recall_history`.
|
||||
|
||||
## 7. Stable Data Model
|
||||
|
||||
### 7.1 StepMemoryRecord
|
||||
|
||||
```python
|
||||
class StepMemoryRecord:
|
||||
step_id: UUID
|
||||
session_id: UUID
|
||||
step_number: int
|
||||
timestamp: datetime
|
||||
|
||||
plan_revision: str | None
|
||||
current_subgoal_hash: str | None
|
||||
|
||||
operator_message_ref: str | None
|
||||
thought_signature: str | None
|
||||
raw_thinking_ref: str | None
|
||||
native_thinking_ref: str | None
|
||||
action: dict
|
||||
validator_result: dict | None
|
||||
failure_recovery: dict | None
|
||||
|
||||
pre_image_name: str | None
|
||||
post_image_name: str | None
|
||||
pre_ui_ref: str | None
|
||||
post_ui_ref: str | None
|
||||
ui_diff: dict | None
|
||||
|
||||
summary: str | None
|
||||
summary_status: str
|
||||
summary_model: str | None
|
||||
summary_version: int
|
||||
|
||||
summary_decision_image_name: str | None
|
||||
summary_post_image_name: str | None
|
||||
summary_used_ui_diff: bool
|
||||
```
|
||||
|
||||
The `summary_status` state machine is:
|
||||
|
||||
```text
|
||||
pending -> visual_ready -> visual_failed -> stale
|
||||
```
|
||||
|
||||
Exact actions, Validator results, and raw sources remain immediately available
|
||||
regardless of summary state.
|
||||
|
||||
### 7.2 SummaryEvidenceMetadata
|
||||
|
||||
The summary remains one neutral natural-language paragraph. Deterministic source
|
||||
metadata is stored separately:
|
||||
|
||||
```json
|
||||
{
|
||||
"step_id": "...",
|
||||
"decision_image_name": "image-a",
|
||||
"post_image_name": null,
|
||||
"used_ui_diff": false,
|
||||
"model": "gemini-3.7-flash",
|
||||
"prompt_version": 2,
|
||||
"summary_version": 3
|
||||
}
|
||||
```
|
||||
|
||||
The system derives these fields from actual inputs. A null `post_image_name`
|
||||
expresses no action result or screen-change judgment.
|
||||
|
||||
### 7.3 HistoryChunk
|
||||
|
||||
```python
|
||||
class HistoryChunk:
|
||||
chunk_id: UUID
|
||||
session_id: UUID
|
||||
start_step_id: UUID
|
||||
end_step_id: UUID
|
||||
source_step_ids: list[UUID]
|
||||
digest: str
|
||||
verified_facts: list[str]
|
||||
unresolved: list[str]
|
||||
failed_paths: list[str]
|
||||
important_entities: list[str]
|
||||
entry_state: str | None
|
||||
exit_state: str | None
|
||||
model: str
|
||||
version: int
|
||||
```
|
||||
|
||||
Chunk summaries are versioned append-only records and never damage source summaries.
|
||||
|
||||
## 8. Visually Grounded Neutral Pro Summarizer
|
||||
|
||||
### 8.1 Inputs
|
||||
|
||||
Each background summary call uses only inputs that actually exist:
|
||||
|
||||
1. the decision screenshot seen by Operator, optionally annotated with action coordinates;
|
||||
2. an independent post-action screenshot only when Validator captured and saved it;
|
||||
3. the exact action name and parameters;
|
||||
4. the Controller result;
|
||||
5. the final Validator result;
|
||||
6. Failure Analyzer recovery records;
|
||||
7. available package/activity information;
|
||||
8. available UI/OCR differences;
|
||||
9. current-step thinking; and
|
||||
10. a small number of recent summaries for loop and strategy continuity.
|
||||
|
||||
Do not resend the full plan and ten detailed steps. Build the message dynamically,
|
||||
never substitute the decision screenshot for a missing post-action screenshot,
|
||||
and never infer "no screen change" from missing post-action evidence.
|
||||
|
||||
### 8.2 Output Rules
|
||||
|
||||
`artemis/agents/summarizer/summarizer.md` remains the sole authority for summary
|
||||
semantics. Preserve its philosophy and one-paragraph output. Use Operator's first
|
||||
person to compress intent, strategy, explicit progress, and verified facts. Do
|
||||
not independently claim success, failure, completion, non-completion, or page
|
||||
navigation. A successful Controller return does not prove task success. Do not
|
||||
guess unreadable values or judge outcomes by whether two images look similar.
|
||||
|
||||
Failure Analyzer intervention is summarized as an objective trigger and action,
|
||||
without claiming that the repair worked. Step ID, evidence names, model, prompt
|
||||
version, and summary version are system metadata rather than model-generated text.
|
||||
|
||||
The existing prohibition on outcome-judgment terms remains, including
|
||||
`successfully`, `completed`, `entered`, `navigated to`, `failed`, `unsuccessful`,
|
||||
`could not`, and `achieved`.
|
||||
|
||||
### 8.3 Model Selection
|
||||
|
||||
Quality-first default:
|
||||
|
||||
```text
|
||||
Per-step summary: Gemini 3.7 Flash + available screenshots
|
||||
History-chunk summary: Gemini 3.7 Flash
|
||||
```
|
||||
|
||||
Single-quota, strict-isolation option:
|
||||
|
||||
```text
|
||||
Real-time step summary: Gemini 3.5 Flash-Lite + available screenshots
|
||||
Idle reprocessing for violations or important omissions: Gemini 3.7 + available screenshots
|
||||
History-chunk summary: Gemini 3.7 Flash
|
||||
```
|
||||
|
||||
The recommended quality ceiling remains 3.7 with available images. Flash-Lite is
|
||||
a resource-isolation option, not a replacement for complex strategy, loop, and
|
||||
recovery reasoning.
|
||||
|
||||
### 8.4 Scheduling and Isolation
|
||||
|
||||
- Operator never awaits Summarizer.
|
||||
- Background summary concurrency is limited to one.
|
||||
- Operator requests have absolute priority.
|
||||
- Summary timeout or failure never affects graph execution.
|
||||
- Existing detailed history remains the fallback.
|
||||
- Prefer a separate endpoint, project, or quota.
|
||||
- Without resource isolation, do not start a new 3.7 summary while Operator is active.
|
||||
- Give non-preemptible background calls a short timeout.
|
||||
- An optional final flush must not delay the next normal step.
|
||||
|
||||
## 9. Stable History Chunks
|
||||
|
||||
Milestones reflect a temporary interpretation of the task and are not stable fact
|
||||
boundaries. A plan edit can otherwise hide relevant steps, split one path across
|
||||
milestones, make rollback lose a prior strategy, or desynchronize summaries from
|
||||
current labels. Subgoal identity therefore affects relevance only.
|
||||
|
||||
```python
|
||||
if current_open_chunk.summary_tokens >= 1800:
|
||||
close_chunk()
|
||||
elif current_open_chunk.step_count >= 12:
|
||||
close_chunk()
|
||||
elif important_boundary_event:
|
||||
close_chunk()
|
||||
```
|
||||
|
||||
Prevent recursive summary loss by generating each chunk from source step
|
||||
summaries, retaining `source_step_ids`, merging structured fact sets rather than
|
||||
guessing from prose, and retaining every summary version.
|
||||
|
||||
## 10. Operator Cold-History Recall
|
||||
|
||||
Expose one unified tool:
|
||||
|
||||
```python
|
||||
recall_history(
|
||||
query: str,
|
||||
step_range: list[int] | None = None,
|
||||
include_details: bool = False,
|
||||
include_images: bool = False,
|
||||
max_results: int = 5,
|
||||
) -> HistoryRecallResult
|
||||
```
|
||||
|
||||
Search visually grounded summaries, history chunks, raw actions, Validator
|
||||
results, Operator thinking, Notes, XML/OCR text, package/activity data, Failure
|
||||
Analyzer traces, and image references. Phase one can use SQLite FTS, keywords,
|
||||
step ranges, package/activity, and screen hashes without another model or vector database.
|
||||
|
||||
Return at most five results and 2K tokens by default. Every result includes a
|
||||
step ID or number. Detailed output remains token-bounded. Image recall returns no
|
||||
more than the images that actually exist for one step. Large raw results return
|
||||
only relevant excerpts and references.
|
||||
|
||||
Operator should recall history when the current screen resembles an older state,
|
||||
an old exact value is needed, two consecutive steps make no progress, the same
|
||||
action is about to be attempted a third time, a summary conflicts with current
|
||||
observation, the user asks to return to an earlier state, or Failure Analyzer
|
||||
needs review. Ordinary steps must not recall history speculatively.
|
||||
|
||||
Before prompt construction, a local package/activity, screenshot-hash, and UI-
|
||||
signature match may inject a short hint without a model call or historical image:
|
||||
|
||||
```text
|
||||
Historical state hint: The current screen closely resembles the post-action
|
||||
screen from Step 17. Use recall_history only if its details are needed.
|
||||
```
|
||||
|
||||
## 11. Token Budget and History Selection
|
||||
|
||||
Recommended steady-state budgets:
|
||||
|
||||
```text
|
||||
First Operator call target: 10K-12K
|
||||
First-call soft limit: 12K
|
||||
First-call emergency hard limit: 16K
|
||||
Same-turn soft limit: 20K
|
||||
Same-turn hard limit: 28K
|
||||
```
|
||||
|
||||
```python
|
||||
history_budget = (
|
||||
operator_total_budget
|
||||
- stable_system_tokens
|
||||
- tool_schema_tokens
|
||||
- current_observation_tokens
|
||||
- active_plan_tokens
|
||||
- same_turn_reserve
|
||||
)
|
||||
```
|
||||
|
||||
Always retain the previous complete Operator turn, current user constraints,
|
||||
active plan path, unresolved items, recent failures and recovery, irreversible
|
||||
actions or external side effects, the previous chunk's exit state, and current
|
||||
loop or candidate progress.
|
||||
|
||||
Score optional history using current-subgoal relevance, unresolved status,
|
||||
failure or recovery relevance, screen similarity, entity overlap, path boundary,
|
||||
explicit user references, recency, resolved status, and duplication. Fill the
|
||||
remaining budget by `value_density = priority / token_cost`.
|
||||
|
||||
When over budget:
|
||||
|
||||
1. Remove duplicate renderings.
|
||||
2. Keep the previous real message and remove its duplicate detailed history text.
|
||||
3. Replace older detailed steps with visual summaries.
|
||||
4. Replace groups of old summaries with stable chunks.
|
||||
5. Remove resolved, unrelated, low-value chunks.
|
||||
6. Keep only relevant excerpts and references from large tool output.
|
||||
|
||||
Never evict hard user constraints, unresolved items, irreversible side effects,
|
||||
critical values, recent recovery state, or the previous complete reasoning and expectation.
|
||||
|
||||
## 12. Prompt and Cache Layout
|
||||
|
||||
Keep the stable Operator role, safety rules, cognitive protocol, and output
|
||||
contract in `SystemMessage`. Put the previous real AI message, thought signature,
|
||||
action call, and Validator tool result in the conversation tail. Put the current
|
||||
goal, active plan path, budgeted history, current screenshot, and UI elements in
|
||||
the dynamic `HumanMessage`. Keep tool schemas and ordering stable when possible.
|
||||
|
||||
If the provider adapter supports stateful interactions, preserve one to three
|
||||
recent Operator interactions with `previous_interaction_id`. Start a new
|
||||
interaction when the token threshold is reached or the plan changes materially,
|
||||
seeding it with the stable prefix and chunk summaries. Until then, retain the
|
||||
current message-rebuild path.
|
||||
|
||||
## 13. Same-Turn Context Control
|
||||
|
||||
Large ADB, Notes, Explorer, log, or XML results can expand later model calls in
|
||||
the same Operator turn. Store complete results in a trace or artifact and expose
|
||||
a local envelope:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "success",
|
||||
"important_lines": ["..."],
|
||||
"errors": [],
|
||||
"tail": ["..."],
|
||||
"full_result_ref": "trace://...",
|
||||
"truncated": true,
|
||||
"original_chars": 87321
|
||||
}
|
||||
```
|
||||
|
||||
Apply character, line, and estimated-token limits; retain matched lines, errors,
|
||||
and head/tail context; never call another LLM synchronously to summarize tool
|
||||
output. Compress older results at the soft limit and require a decision instead
|
||||
of accumulating more exploration at the hard limit.
|
||||
|
||||
## 14. Configuration
|
||||
|
||||
Add a dedicated Pro memory section instead of reusing Flash `step_summarizer`:
|
||||
|
||||
```jsonc
|
||||
"agent": {
|
||||
"pro": {
|
||||
"memory": {
|
||||
"enabled": true,
|
||||
"preserve_previous_operator_turn": true,
|
||||
"max_hot_turns": 2,
|
||||
"operator_target_input_tokens": 12000,
|
||||
"operator_hard_input_tokens": 16000,
|
||||
"same_turn_soft_tokens": 20000,
|
||||
"same_turn_hard_tokens": 28000,
|
||||
"visual_step_summary": {
|
||||
"enabled": true,
|
||||
"model": "gemini-3.7-flash",
|
||||
"thinking_level": "low",
|
||||
"include_pre_image": true,
|
||||
"include_post_image_when_available": true,
|
||||
"include_ui_diff": true,
|
||||
"max_concurrency": 1,
|
||||
"operator_never_waits": true
|
||||
},
|
||||
"history_chunks": {
|
||||
"enabled": true,
|
||||
"max_steps": 12,
|
||||
"target_source_tokens": 1800,
|
||||
"model": "gemini-3.7-flash"
|
||||
},
|
||||
"recall": {
|
||||
"enabled": true,
|
||||
"max_results": 5,
|
||||
"max_text_tokens": 2000,
|
||||
"max_image_steps": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For a speed-first single-quota deployment, override only the visual step model
|
||||
with `gemini-3.5-flash-lite`.
|
||||
|
||||
## 15. Implementation Scope
|
||||
|
||||
### 15.1 Operator
|
||||
|
||||
Update Operator, prompt builders, and configuration to add a token-budget memory
|
||||
assembler, preserve recent real messages and Validator results, avoid duplicate
|
||||
history rendering, move dynamic state out of the stable system prompt, remove
|
||||
`<short_term_memory>`, add `recall_history`, inject local screen-similarity hints,
|
||||
and enforce same-turn output limits.
|
||||
|
||||
### 15.2 Summarizer
|
||||
|
||||
Load decision and optional post-action images by stable step ID, include available
|
||||
Validator and UI-diff data, send multimodal messages, preserve the neutral
|
||||
one-paragraph contract, atomically update versioned evidence metadata, retain old
|
||||
summaries on failure, enforce concurrency and timeout limits, and never block Operator.
|
||||
|
||||
### 15.3 Graph and State
|
||||
|
||||
Store the previous Operator `AIMessage`, thought signature, and Validator function
|
||||
response. Remove the `short_term_memory` field while safely ignoring it in older
|
||||
tasks. Read ready memory snapshots at the Perception/Operator boundary and retain
|
||||
the current asynchronous Summarizer semantics.
|
||||
|
||||
### 15.4 DataEngine
|
||||
|
||||
Add summary status, version, evidence, thought-signature, message-reference, and
|
||||
UI-diff fields; persist history chunks and local recall indexes; support atomic
|
||||
updates by step ID; and preserve every older summary version and source.
|
||||
|
||||
### 15.5 Task Tree
|
||||
|
||||
Separate history visibility from rendering detail, render visual summaries and
|
||||
stable chunks, use current subgoal only for relevance, avoid repeating the latest
|
||||
step when it is already present in the real message tail, and retain exact actions
|
||||
and validation results.
|
||||
|
||||
### 15.6 Memory Module
|
||||
|
||||
Suggested files:
|
||||
|
||||
```text
|
||||
artemis/memory/pro_memory.py
|
||||
artemis/memory/history_chunker.py
|
||||
artemis/memory/history_retriever.py
|
||||
artemis/memory/token_budget.py
|
||||
artemis/tools/history_recall.py
|
||||
```
|
||||
|
||||
Responsibilities include memory snapshots, value-based selection, chunk closure
|
||||
and background summarization, FTS and screen-signature retrieval, the recall tool,
|
||||
and provider-independent token estimation.
|
||||
|
||||
### 15.7 Configuration
|
||||
|
||||
Every capability must have an independent switch, default to current Pro fallback
|
||||
behavior, tolerate missing configuration and older tasks, and support both the
|
||||
3.7 quality mode and Flash-Lite isolation mode.
|
||||
|
||||
## 16. Delivery Phases
|
||||
|
||||
### Phase 0: Baseline and Observability
|
||||
|
||||
Freeze a representative long-task suite. Record input tokens, TTFT, model latency,
|
||||
full-turn latency, summary-ready latency, failures, retries, history size,
|
||||
duplicate-action rate, and Failure Analyzer activation without changing prompts.
|
||||
|
||||
### Phase 1: Protect and Improve Hot History
|
||||
|
||||
Preserve complete recent reasoning, remove the duplicate short-term-memory path,
|
||||
avoid duplicate rendering, add the total token threshold, and locally truncate
|
||||
large tool envelopes. Prove that speed does not regress first.
|
||||
|
||||
### Phase 2: Pro Visual Step Summaries
|
||||
|
||||
Load actual decision and optional post-action images by stable step ID while
|
||||
preserving the neutral output contract and text fallback. Compare 3.7 with images,
|
||||
Flash-Lite with images, and current 3.7 text-only behavior. Confirm no Operator wait.
|
||||
|
||||
### Phase 3: Stable History Chunks
|
||||
|
||||
Add milestone-independent rolling chunks, generate them from source summaries,
|
||||
treat milestones as temporary working-set labels, and add source references and versions.
|
||||
|
||||
### Phase 4: Operator Cold-History Recall
|
||||
|
||||
Reuse step, screenshot, and UI search capabilities behind one bounded
|
||||
`recall_history` tool, add local screen-similarity hints, and limit text and images.
|
||||
|
||||
### Phase 5: Real Multi-Turn Messages and Cache Optimization
|
||||
|
||||
Preserve real AI messages, tool calls, and thought signatures; separate stable
|
||||
system and dynamic human content; evaluate stateful provider APIs; and safely
|
||||
restart interactions at token thresholds. Begin only after the first four phases stabilize.
|
||||
|
||||
## 17. Test Plan
|
||||
|
||||
Unit tests must cover hot-history preservation, removal of short-term-memory
|
||||
parsing, action/Validator pairing, duplicate suppression, two-turn recovery,
|
||||
image lookup by step ID, optional post-image behavior, neutral-output validation,
|
||||
Validator precedence, concurrency, timeout, zero-wait behavior, chunk thresholds,
|
||||
source traceability, recall bounds, image limits, and hard token protections.
|
||||
|
||||
Integration tests must include tasks longer than 30 steps, repeated similar
|
||||
screens, plan rewrites, cross-app flows, dialogs, Toasts, WebViews, Canvas,
|
||||
Failure Analyzer recovery, repeated ADB exploration, long lists, monitoring loops,
|
||||
summary timeout or quota exhaustion, and exact-value recall from older screenshots.
|
||||
|
||||
Run blinded evaluations on real traces comparing 3.7 text-only, Flash-Lite with
|
||||
available images, and 3.7 with available images. Reviewers score fidelity to
|
||||
intent and explicit facts, unsupported visual claims, unauthorized outcome
|
||||
judgments, anomaly retention, next-step usefulness, generation time, and ready latency.
|
||||
|
||||
## 18. Acceptance Criteria
|
||||
|
||||
Under the same device, task, model, and comparable service load:
|
||||
|
||||
- first-call input P50 increases by no more than 3% and P95 by no more than 5%;
|
||||
- Operator model-call and full-turn P50/P90 latency regress by no more than 5%;
|
||||
- observed step-interval P50/P90 regress by no more than 5%;
|
||||
- Operator waits for summaries exactly zero times;
|
||||
- normal Operator requests contain zero historical images; and
|
||||
- images returned by `recall_history(include_images=True)` count only as explicit recall.
|
||||
|
||||
Run at least 30 paired tasks and report confidence intervals. Quality must not
|
||||
regress for previous-strategy recall, overall task success, or Failure Analyzer
|
||||
recovery. Visual-fact accuracy, important-fact retention across plan edits,
|
||||
repeat-action rate, history-induced rollback rate, and top-five recall hit rate
|
||||
must improve. Summary/Validator conflict should approach zero.
|
||||
|
||||
## 19. Failure Modes and Fallbacks
|
||||
|
||||
- If a visual summary is pending, use complete previous reasoning, exact action, and Validator result.
|
||||
- If visual summarization fails, set `visual_failed`, preserve old summaries, and do not block for retries.
|
||||
- If screenshot evidence is incomplete, include only actual evidence and never infer the missing post-action state.
|
||||
- If chunk generation fails, keep source summaries and retry the pending chunk during an idle window.
|
||||
- If recall output is too large, return high-scoring excerpts and step/artifact references.
|
||||
- If background work competes with Operator, switch to Lite, pause new summaries, or use a separate quota.
|
||||
- If stateful interactions are incompatible, return to prompt reconstruction while preserving raw/native thinking.
|
||||
|
||||
## 20. Rollout Flags
|
||||
|
||||
```text
|
||||
pro_memory_enabled
|
||||
preserve_operator_message_tail
|
||||
visual_step_summary
|
||||
history_chunks
|
||||
operator_history_recall
|
||||
stateful_operator_interactions
|
||||
```
|
||||
|
||||
Recommended order: enable metrics and token thresholds; run visual summaries in
|
||||
shadow mode; use them after blinded validation; enable stable chunks; enable
|
||||
`recall_history`; then evaluate real multi-turn interactions. Disabling any flag
|
||||
must restore current Pro behavior.
|
||||
|
||||
## 21. Improvements Over the Current System
|
||||
|
||||
| Capability | Current Pro | Redesigned Pro |
|
||||
|---|---|---|
|
||||
| Previous reasoning | Re-rendered as detailed history | Preserved and ready for a real message/thought-signature chain |
|
||||
| Short-term strategy | Optional `<short_term_memory>` | Carried by the complete previous message, Validator, plan, and Notes |
|
||||
| Step summary | 3.7 reads text only | Neutral 3.7 summary reads actual available images, UI, and Validator data |
|
||||
| Exact action | Retained beside summary | Still retained and never replaced by visual summary |
|
||||
| History pruning | Current subgoal plus latest three steps | Working set plus stable chunks and relevance budget |
|
||||
| Plan rewrite | May change old-step visibility | Stable step/chunk references survive label changes |
|
||||
| Old screenshots | No direct Operator recall | One actual step can be recalled on demand |
|
||||
| Cold history | Primarily for HistoryAnalyzer/Outputter | Locally searchable and bounded for Operator |
|
||||
| Context control | Mainly a step window | Total token budget, value density, and same-turn limits |
|
||||
| Cache use | Highly dynamic system prompt | Stable system prefix separated from runtime state |
|
||||
| Background failure | Summary may be delayed | Explicit state machine and zero-wait fallback |
|
||||
|
||||
The objective is not to display more history. It is to preserve recent reasoning,
|
||||
give older summaries real visual context, decouple long-term identity from plan
|
||||
milestones, make compressed records precisely recallable, and keep normal input
|
||||
size and critical-path model-call count unchanged.
|
||||
|
||||
## 22. Recommended Defaults
|
||||
|
||||
```text
|
||||
Normal Operator input: 10K-12K tokens
|
||||
First-call soft/hard limits: 12K / 16K
|
||||
Same-turn soft/hard limits: 20K / 28K
|
||||
|
||||
Complete hot turns: 1 normally, up to 2 for failure/uncertainty
|
||||
Current-work summaries: budget-controlled, not a fixed count
|
||||
History chunk: 8-12 steps or about 1.8K source-summary tokens
|
||||
|
||||
Historical images in normal input: 0
|
||||
Summary input images: 0-1 decision, 0-1 independent post-action
|
||||
Explicit recall images: actual images from at most one step
|
||||
|
||||
Preferred step-summary model: Gemini 3.7 Flash + images
|
||||
Single-quota speed mode: Gemini 3.5 Flash-Lite + images
|
||||
History-chunk model: Gemini 3.7 Flash
|
||||
Background summary concurrency: 1
|
||||
|
||||
Additional critical-path calls: 0
|
||||
Critical-path summary waits: 0
|
||||
```
|
||||
|
||||
## 23. Final Decision
|
||||
|
||||
Do not remove previous reasoning and replace detailed history with an image-free
|
||||
capsule. Preserve complete previous-turn reasoning, upgrade older steps to
|
||||
visually grounded neutral summaries, use milestone-independent rolling chunks,
|
||||
allow bounded cold-history recall, and enforce a fixed token budget.
|
||||
|
||||
The first, highest-value implementation slice is:
|
||||
|
||||
1. preserve complete previous reasoning and remove the duplicate short-term-memory path;
|
||||
2. add the decision screenshot and optional actual post-action screenshot to the current neutral Pro Summarizer;
|
||||
3. bind summaries with stable step IDs; and
|
||||
4. establish speed baselines and blinded visual-summary evaluations.
|
||||
|
||||
Do not expand the normal history window or remove the current detailed-history
|
||||
fallback until all four items have passed validation.
|
||||
Reference in new issue
Block a user