914 lines
36 KiB
Markdown
914 lines
36 KiB
Markdown
# ARTEMIS Pro Context, History, and Summarization Redesign
|
||
|
||
Status: Proposal (2026-08-25) — partially superseded
|
||
|
||
> **2026-08-31**: `docs/plans/history-module-redesign.md` (unified v3) supersedes
|
||
> §6/§11/§12 of this document: Pro moves from per-turn prompt recompilation to an
|
||
> incremental append-only transcript with layered compaction (this doc's Phase 5
|
||
> promoted to the core architecture). §7 (data model), §8 (visually grounded
|
||
> neutral summaries), §9 (stable chunks), §10 (recall), §13 (same-turn envelopes)
|
||
> are absorbed and remain authoritative where referenced.
|
||
>
|
||
> **Final status (2026-09-01)**: the unified v3 plan is fully implemented
|
||
> (M0–M5, work in worktree `zealous-gates-4626b1`, uncommitted). Everything
|
||
> this document delegated forward has landed, including §3.3 (the
|
||
> `<short_term_memory>` relay was removed end to end in M5 — state field,
|
||
> operator extraction, prompt component, visibility manifest entries, and the
|
||
> operator.json instruction block; render-time tag scrubbing is kept for older
|
||
> recorded responses, per this section's own "safely ignoring" clause). See
|
||
> history-module-redesign.md §7 for the per-milestone implementation notes.
|
||
|
||
Scope: ARTEMIS Pro / LangGraph execution mode
|
||
|
||
Audience: maintainers of Operator, Validator, Summarizer, DataEngine, and model routing
|
||
|
||
## 1. Executive Summary
|
||
|
||
This proposal improves long-term history, visually grounded neutral summaries,
|
||
context compression, and history recall without weakening Pro mode's current
|
||
reasoning continuity. The goal is not to give Operator more raw history by
|
||
default. It is to increase information density while preserving the current
|
||
`10K-12K` input range and moving all additional image interpretation to a
|
||
non-blocking background path.
|
||
|
||
The design preserves mechanisms that already work:
|
||
|
||
- the previous Operator step's complete raw or native thinking;
|
||
- the exact previous action and Validator result;
|
||
- the current screenshot and UI element list;
|
||
- complete, auditable raw step records in DataEngine; and
|
||
- asynchronous Summarizer scheduling that never blocks the graph's critical path.
|
||
|
||
It adds the following capabilities:
|
||
|
||
1. Preserve the previous model message, thought signature, action call, and Validator result as a continuous conversation tail.
|
||
2. Keep the current neutral summary contract while allowing the Summarizer to use the decision screenshot and an optional post-action screenshot as context.
|
||
3. Bind thinking, actions, available images, validation results, and summaries with a stable `step_id`.
|
||
4. Treat the active milestone only as a temporary working-set label; use plan-independent history chunks for long-term compression.
|
||
5. Add a bounded, on-demand `recall_history` tool for complete steps, older screenshots, and UI text.
|
||
6. Introduce a fixed token budget, value-density history selection, and same-turn tool-output limits.
|
||
7. Separate stable system rules from dynamic plans and history to improve prefix caching and multi-turn continuity.
|
||
|
||
The steady-state Operator context becomes:
|
||
|
||
```text
|
||
Current screenshot/XML
|
||
+ complete previous-turn reasoning and action result
|
||
+ recent visually grounded step summaries
|
||
+ relevant history-chunk summaries
|
||
+ cold-history recall on demand
|
||
```
|
||
|
||
The steady-state target remains `10K-12K tokens`, with zero additional model
|
||
calls and zero summary waits on the critical path.
|
||
|
||
## 2. Goals and Constraints
|
||
|
||
### 2.1 First Priority: Preserve Speed
|
||
|
||
The engineering definition of preserving speed is:
|
||
|
||
- no new synchronous model calls in normal steps;
|
||
- Operator never waits for summaries, indexes, or history chunks;
|
||
- first-call input P50 and P95 remain close to the current baseline;
|
||
- background summaries do not compete at the same request priority as Operator;
|
||
- normal Operator requests still contain only one current screenshot;
|
||
- history improvements replace low-density content instead of expanding normal context; and
|
||
- any background failure immediately falls back to the current detailed history.
|
||
|
||
Absolute zero latency change cannot be guaranteed by adding synchronous input.
|
||
This design therefore uses a token-neutral, zero-wait critical path and validates
|
||
P50 and P90 through paired A/B tests.
|
||
|
||
### 2.2 History and Summary Goals
|
||
|
||
- Preserve the previous strategy, expected result, and unresolved questions.
|
||
- Ground older step summaries in available visual evidence instead of guessing from action text.
|
||
- Keep important history visible across plan rewrites, milestone renames, and goal rollback.
|
||
- Make every compressed record traceable to original steps, screenshots, and tool traces.
|
||
- Never discard failures, recovery work, side effects, user constraints, or critical values through ordinary compression.
|
||
- Allow Operator to recall older information during long tasks.
|
||
|
||
### 2.3 Use Model Capacity Effectively
|
||
|
||
"Maximize" means maximizing useful reasoning information, not filling a one-million-token window:
|
||
|
||
- the strongest model directly handles current visual state, continuous reasoning, and high-value history;
|
||
- available older screenshots help the Summarizer understand the UI, action target, and explicit records;
|
||
- long context acts as an exception and deep-recall buffer, not the normal working set;
|
||
- native multi-turn messages and thought signatures are preserved;
|
||
- stable system prefixes are reused whenever possible; and
|
||
- the model does not repeatedly reread duplicate plans, raw logs, or low-value click traces.
|
||
|
||
### 2.4 Non-Goals
|
||
|
||
- Do not delete or overwrite raw history in DataEngine.
|
||
- Do not send multiple historical screenshots to Operator by default.
|
||
- Do not treat plan milestones as permanent fact boundaries.
|
||
- Do not replace Validator's objective judgment with a summary.
|
||
- Do not require a new model API in the first phase.
|
||
- Do not enlarge every step merely to use a larger context window.
|
||
|
||
## 3. Current Baseline
|
||
|
||
### 3.1 Pro Graph Path
|
||
|
||
```text
|
||
Planner
|
||
-> Convergence
|
||
-> Perception
|
||
-> Operator
|
||
-> Execution Check
|
||
-> Validator
|
||
-> Summarizer
|
||
-> Convergence
|
||
```
|
||
|
||
The Summarizer node creates a background task and returns immediately. This
|
||
behavior must remain unchanged.
|
||
|
||
### 3.2 Current Operator History
|
||
|
||
`build_plan_and_history` currently:
|
||
|
||
- renders the latest completed step in detail;
|
||
- may include raw/native thinking, actions, tool calls, Validator results, and Failure Analyzer traces;
|
||
- renders older summaries when available while retaining exact actions;
|
||
- preserves steps under the active subgoal plus the latest three global steps;
|
||
- puts the full current plan in the Operator prompt; and
|
||
- rebuilds messages on every Operator turn.
|
||
|
||
The current system already has continuity. Removing complete reasoning from the
|
||
latest step would prevent the model from understanding the previous strategy,
|
||
expectation, and decision basis.
|
||
|
||
### 3.3 Remove Separate Short-Term Memory
|
||
|
||
Remove the existing `<short_term_memory>` extraction, state storage, and next-turn
|
||
injection path without introducing a replacement relay object. Cross-turn
|
||
continuity should come from the complete previous Operator message, exact action,
|
||
Validator result, current plan, long-term Notes, and auditable execution history.
|
||
|
||
### 3.4 Current Pro Summaries
|
||
|
||
The current Pro Summarizer receives only text:
|
||
|
||
- current plan and history;
|
||
- current-step thinking;
|
||
- action and execution result; and
|
||
- Failure Analyzer trace.
|
||
|
||
It does not load `pre_image_name` or `post_image_name`. Images should reduce
|
||
context misunderstanding, not expand the Summarizer's authority to judge results.
|
||
|
||
### 3.5 Existing Cold History
|
||
|
||
Complete steps, screenshots, and UI data already exist in DataEngine.
|
||
HistoryAnalyzer and Outputter offer partial retrieval, but Operator lacks one
|
||
unified direct history-recall tool.
|
||
|
||
### 3.6 Measured Baseline
|
||
|
||
Current identifiable Pro Operator calls in `traces/data_engine.db` show:
|
||
|
||
- 74 direct Operator model calls across 43 Operator turns;
|
||
- median input of about 10,221 tokens;
|
||
- input P90 of about 11,648 tokens and P95 of about 12,019 tokens;
|
||
- maximum input of about 12,747 tokens;
|
||
- median model-call latency of about 5.24 seconds and P90 of about 16.42 seconds;
|
||
- median full Operator-turn latency of about 20.02 seconds; and
|
||
- median observed step interval of about 17.15 seconds.
|
||
|
||
Input length and model latency show no reliable correlation in the observed
|
||
`8.8K-12.7K` range. That result must not be extrapolated to an `80K-100K`
|
||
dynamic context.
|
||
|
||
## 4. Design Principles
|
||
|
||
### 4.1 Separate Raw Records from the Model View
|
||
|
||
DataEngine stores complete facts. The prompt contains only the most valuable view
|
||
for the current turn. Compression changes what the model sees, never the source record.
|
||
|
||
### 4.2 Continuous Reasoning Before Compression
|
||
|
||
The complete previous reasoning, action expectation, action call, and Validator
|
||
result are required on the hot path and cannot be replaced by an ordinary summary.
|
||
|
||
### 4.3 Images Provide Context, Not Judgment
|
||
|
||
The Summarizer may inspect screenshots that actually exist for a step, but it
|
||
must preserve the current prompt's neutral responsibility. It may compress
|
||
intent, strategy, explicit verification, iteration progress, missing conditions,
|
||
and objective Failure Analyzer intervention. It must not independently decide
|
||
whether an action succeeded, failed, completed, or navigated to a new page.
|
||
|
||
A missing post-action screenshot means only that no independent post-action
|
||
visual evidence is available. It never means the screen was unchanged or the
|
||
action had no effect.
|
||
|
||
### 4.4 Plans Are Dynamic Indexes, Not History Keys
|
||
|
||
The current subgoal may increase relevance but must not control permanent
|
||
retention. History identity uses stable `step_id` values, time ranges, and source references.
|
||
|
||
### 4.5 Validator Facts Take Priority
|
||
|
||
```text
|
||
Explicit user instructions
|
||
> objective Validator or Controller results
|
||
> activity, package, XML, or OCR differences
|
||
> sourced content in visually grounded summaries
|
||
> Operator or Summarizer inference
|
||
```
|
||
|
||
### 4.6 Background Enhancements Cannot Become Dependencies
|
||
|
||
If a visual summary or history chunk is not ready, Operator immediately continues
|
||
with the existing detailed record.
|
||
|
||
## 5. Target Architecture
|
||
|
||
```text
|
||
+-----------------------------+
|
||
| DataEngine raw facts |
|
||
| steps / traces / images / UI|
|
||
+--------------+--------------+
|
||
|
|
||
Validator completes step ---------------+
|
||
|
|
||
+--------------v--------------+
|
||
| Step Memory Record |
|
||
| exact action/result/sources |
|
||
+-------+--------------+-------+
|
||
| |
|
||
immediate | asynchronous
|
||
| v
|
||
| +--------------------+
|
||
| | Context Summarizer |
|
||
| | decision image + |
|
||
| | optional post/UI |
|
||
| +----------+---------+
|
||
| |
|
||
+-------v----------------v-------+
|
||
| Pro Memory Store |
|
||
| hot history/summaries/chunks |
|
||
+---------------+---------------+
|
||
|
|
||
Perception current screenshot/XML -------+
|
||
v
|
||
+-----------------------------+
|
||
| Token-Budget Prompt Builder |
|
||
+--------------+--------------+
|
||
v
|
||
Operator
|
||
|
|
||
recall_history on demand
|
||
```
|
||
|
||
## 6. Layered History Model
|
||
|
||
### 6.1 L0: Current Observation
|
||
|
||
Every Operator turn receives the current screenshot, package/activity, compact
|
||
XML/OCR element list, screen dimensions and coordinate mapping, user-injected
|
||
instructions, active plan path, and background-task state. Historical images do
|
||
not enter normal requests.
|
||
|
||
### 6.2 L1: Hot History and Conversation Tail
|
||
|
||
```text
|
||
Previous AIMessage
|
||
- model-visible thinking / thought signature
|
||
- raw reasoning text
|
||
- turn-ending action tool call
|
||
|
||
Validator Function/Tool Result
|
||
- whether execution occurred
|
||
- execution state
|
||
- Failure Analyzer repair
|
||
- observed post-action state
|
||
|
||
Current HumanMessage
|
||
- current screenshot
|
||
- current XML
|
||
```
|
||
|
||
Keep one complete prior Operator turn normally and up to two when the last step
|
||
failed, remains uncertain, is recovering, or has an unfinished strategy. Do not
|
||
duplicate the same complete reasoning in `Execution History`. Until the provider
|
||
can preserve real messages and thought signatures, continue rendering raw/native
|
||
thinking verbatim.
|
||
|
||
### 6.3 L2: Visually Grounded, Neutral Step Summaries
|
||
|
||
Older steps use the current "summary plus exact action" structure, with a
|
||
multimodal summary input:
|
||
|
||
```text
|
||
Step N
|
||
Neutral contextual summary
|
||
Exact action
|
||
Validator outcome
|
||
Unresolved/recovery marker
|
||
```
|
||
|
||
The summary never replaces the exact action or overrides Validator. Screenshots
|
||
only clarify context; the output does not make a new result judgment.
|
||
|
||
### 6.4 Current Work Segment
|
||
|
||
The active milestone may organize recent summaries, loop and candidate progress,
|
||
unresolved items, and the most recent entry state. It remains a temporary label
|
||
that can change with the plan and never becomes a permanent history identity.
|
||
|
||
### 6.5 L3: Stable History Chunks
|
||
|
||
Long-term history is chunked by stable step ranges and token thresholds:
|
||
|
||
```text
|
||
Chunk 1: Steps 1-10
|
||
Chunk 2: Steps 11-19
|
||
Chunk 3: Steps 20-31
|
||
```
|
||
|
||
Close a chunk after roughly `1.5K-2K tokens` of step summaries or 8-12 steps.
|
||
Close early on a major package/activity change, user goal change, Failure Analyzer
|
||
boundary, complete strategy change, long work-segment pause, irreversible side
|
||
effect, or important external result.
|
||
|
||
Each chunk preserves its source range and step IDs, attempted path, verified
|
||
facts, failed paths, unresolved questions, important entities, entry and exit
|
||
states, and raw source references. Later plan rewrites do not alter chunk identity.
|
||
|
||
### 6.6 L4: Cold History
|
||
|
||
Retain every screenshot actually saved for a step, raw XML/OCR, raw/native
|
||
thinking, complete tool calls and results, ADB and log output, the full Failure
|
||
Analyzer trace, all summary versions, Notes, and plan revisions. Cold history is
|
||
excluded from normal prompts and loaded only through local matching or
|
||
`recall_history`.
|
||
|
||
## 7. Stable Data Model
|
||
|
||
### 7.1 StepMemoryRecord
|
||
|
||
```python
|
||
class StepMemoryRecord:
|
||
step_id: UUID
|
||
session_id: UUID
|
||
step_number: int
|
||
timestamp: datetime
|
||
|
||
plan_revision: str | None
|
||
current_subgoal_hash: str | None
|
||
|
||
operator_message_ref: str | None
|
||
thought_signature: str | None
|
||
raw_thinking_ref: str | None
|
||
native_thinking_ref: str | None
|
||
action: dict
|
||
validator_result: dict | None
|
||
failure_recovery: dict | None
|
||
|
||
pre_image_name: str | None
|
||
post_image_name: str | None
|
||
pre_ui_ref: str | None
|
||
post_ui_ref: str | None
|
||
ui_diff: dict | None
|
||
|
||
summary: str | None
|
||
summary_status: str
|
||
summary_model: str | None
|
||
summary_version: int
|
||
|
||
summary_decision_image_name: str | None
|
||
summary_post_image_name: str | None
|
||
summary_used_ui_diff: bool
|
||
```
|
||
|
||
The `summary_status` state machine is:
|
||
|
||
```text
|
||
pending -> visual_ready -> visual_failed -> stale
|
||
```
|
||
|
||
Exact actions, Validator results, and raw sources remain immediately available
|
||
regardless of summary state.
|
||
|
||
### 7.2 SummaryEvidenceMetadata
|
||
|
||
The summary remains one neutral natural-language paragraph. Deterministic source
|
||
metadata is stored separately:
|
||
|
||
```json
|
||
{
|
||
"step_id": "...",
|
||
"decision_image_name": "image-a",
|
||
"post_image_name": null,
|
||
"used_ui_diff": false,
|
||
"model": "gemini-3.7-flash",
|
||
"prompt_version": 2,
|
||
"summary_version": 3
|
||
}
|
||
```
|
||
|
||
The system derives these fields from actual inputs. A null `post_image_name`
|
||
expresses no action result or screen-change judgment.
|
||
|
||
### 7.3 HistoryChunk
|
||
|
||
```python
|
||
class HistoryChunk:
|
||
chunk_id: UUID
|
||
session_id: UUID
|
||
start_step_id: UUID
|
||
end_step_id: UUID
|
||
source_step_ids: list[UUID]
|
||
digest: str
|
||
verified_facts: list[str]
|
||
unresolved: list[str]
|
||
failed_paths: list[str]
|
||
important_entities: list[str]
|
||
entry_state: str | None
|
||
exit_state: str | None
|
||
model: str
|
||
version: int
|
||
```
|
||
|
||
Chunk summaries are versioned append-only records and never damage source summaries.
|
||
|
||
## 8. Visually Grounded Neutral Pro Summarizer
|
||
|
||
### 8.1 Inputs
|
||
|
||
Each background summary call uses only inputs that actually exist:
|
||
|
||
1. the decision screenshot seen by Operator, optionally annotated with action coordinates;
|
||
2. an independent post-action screenshot only when Validator captured and saved it;
|
||
3. the exact action name and parameters;
|
||
4. the Controller result;
|
||
5. the final Validator result;
|
||
6. Failure Analyzer recovery records;
|
||
7. available package/activity information;
|
||
8. available UI/OCR differences;
|
||
9. current-step thinking; and
|
||
10. a small number of recent summaries for loop and strategy continuity.
|
||
|
||
Do not resend the full plan and ten detailed steps. Build the message dynamically,
|
||
never substitute the decision screenshot for a missing post-action screenshot,
|
||
and never infer "no screen change" from missing post-action evidence.
|
||
|
||
### 8.2 Output Rules
|
||
|
||
`artemis/agents/summarizer/summarizer.md` remains the sole authority for summary
|
||
semantics. Preserve its philosophy and one-paragraph output. Use Operator's first
|
||
person to compress intent, strategy, explicit progress, and verified facts. Do
|
||
not independently claim success, failure, completion, non-completion, or page
|
||
navigation. A successful Controller return does not prove task success. Do not
|
||
guess unreadable values or judge outcomes by whether two images look similar.
|
||
|
||
Failure Analyzer intervention is summarized as an objective trigger and action,
|
||
without claiming that the repair worked. Step ID, evidence names, model, prompt
|
||
version, and summary version are system metadata rather than model-generated text.
|
||
|
||
The existing prohibition on outcome-judgment terms remains, including
|
||
`successfully`, `completed`, `entered`, `navigated to`, `failed`, `unsuccessful`,
|
||
`could not`, and `achieved`.
|
||
|
||
### 8.3 Model Selection
|
||
|
||
Quality-first default:
|
||
|
||
```text
|
||
Per-step summary: Gemini 3.7 Flash + available screenshots
|
||
History-chunk summary: Gemini 3.7 Flash
|
||
```
|
||
|
||
Single-quota, strict-isolation option:
|
||
|
||
```text
|
||
Real-time step summary: Gemini 3.5 Flash-Lite + available screenshots
|
||
Idle reprocessing for violations or important omissions: Gemini 3.7 + available screenshots
|
||
History-chunk summary: Gemini 3.7 Flash
|
||
```
|
||
|
||
The recommended quality ceiling remains 3.7 with available images. Flash-Lite is
|
||
a resource-isolation option, not a replacement for complex strategy, loop, and
|
||
recovery reasoning.
|
||
|
||
### 8.4 Scheduling and Isolation
|
||
|
||
- Operator never awaits Summarizer.
|
||
- Background summary concurrency is limited to one.
|
||
- Operator requests have absolute priority.
|
||
- Summary timeout or failure never affects graph execution.
|
||
- Existing detailed history remains the fallback.
|
||
- Prefer a separate endpoint, project, or quota.
|
||
- Without resource isolation, do not start a new 3.7 summary while Operator is active.
|
||
- Give non-preemptible background calls a short timeout.
|
||
- An optional final flush must not delay the next normal step.
|
||
|
||
## 9. Stable History Chunks
|
||
|
||
Milestones reflect a temporary interpretation of the task and are not stable fact
|
||
boundaries. A plan edit can otherwise hide relevant steps, split one path across
|
||
milestones, make rollback lose a prior strategy, or desynchronize summaries from
|
||
current labels. Subgoal identity therefore affects relevance only.
|
||
|
||
```python
|
||
if current_open_chunk.summary_tokens >= 1800:
|
||
close_chunk()
|
||
elif current_open_chunk.step_count >= 12:
|
||
close_chunk()
|
||
elif important_boundary_event:
|
||
close_chunk()
|
||
```
|
||
|
||
Prevent recursive summary loss by generating each chunk from source step
|
||
summaries, retaining `source_step_ids`, merging structured fact sets rather than
|
||
guessing from prose, and retaining every summary version.
|
||
|
||
## 10. Operator Cold-History Recall
|
||
|
||
Expose one unified tool:
|
||
|
||
```python
|
||
recall_history(
|
||
query: str,
|
||
step_range: list[int] | None = None,
|
||
include_details: bool = False,
|
||
include_images: bool = False,
|
||
max_results: int = 5,
|
||
) -> HistoryRecallResult
|
||
```
|
||
|
||
Search visually grounded summaries, history chunks, raw actions, Validator
|
||
results, Operator thinking, Notes, XML/OCR text, package/activity data, Failure
|
||
Analyzer traces, and image references. Phase one can use SQLite FTS, keywords,
|
||
step ranges, package/activity, and screen hashes without another model or vector database.
|
||
|
||
Return at most five results and 2K tokens by default. Every result includes a
|
||
step ID or number. Detailed output remains token-bounded. Image recall returns no
|
||
more than the images that actually exist for one step. Large raw results return
|
||
only relevant excerpts and references.
|
||
|
||
Operator should recall history when the current screen resembles an older state,
|
||
an old exact value is needed, two consecutive steps make no progress, the same
|
||
action is about to be attempted a third time, a summary conflicts with current
|
||
observation, the user asks to return to an earlier state, or Failure Analyzer
|
||
needs review. Ordinary steps must not recall history speculatively.
|
||
|
||
Before prompt construction, a local package/activity, screenshot-hash, and UI-
|
||
signature match may inject a short hint without a model call or historical image:
|
||
|
||
```text
|
||
Historical state hint: The current screen closely resembles the post-action
|
||
screen from Step 17. Use recall_history only if its details are needed.
|
||
```
|
||
|
||
## 11. Token Budget and History Selection
|
||
|
||
Recommended steady-state budgets:
|
||
|
||
```text
|
||
First Operator call target: 10K-12K
|
||
First-call soft limit: 12K
|
||
First-call emergency hard limit: 16K
|
||
Same-turn soft limit: 20K
|
||
Same-turn hard limit: 28K
|
||
```
|
||
|
||
```python
|
||
history_budget = (
|
||
operator_total_budget
|
||
- stable_system_tokens
|
||
- tool_schema_tokens
|
||
- current_observation_tokens
|
||
- active_plan_tokens
|
||
- same_turn_reserve
|
||
)
|
||
```
|
||
|
||
Always retain the previous complete Operator turn, current user constraints,
|
||
active plan path, unresolved items, recent failures and recovery, irreversible
|
||
actions or external side effects, the previous chunk's exit state, and current
|
||
loop or candidate progress.
|
||
|
||
Score optional history using current-subgoal relevance, unresolved status,
|
||
failure or recovery relevance, screen similarity, entity overlap, path boundary,
|
||
explicit user references, recency, resolved status, and duplication. Fill the
|
||
remaining budget by `value_density = priority / token_cost`.
|
||
|
||
When over budget:
|
||
|
||
1. Remove duplicate renderings.
|
||
2. Keep the previous real message and remove its duplicate detailed history text.
|
||
3. Replace older detailed steps with visual summaries.
|
||
4. Replace groups of old summaries with stable chunks.
|
||
5. Remove resolved, unrelated, low-value chunks.
|
||
6. Keep only relevant excerpts and references from large tool output.
|
||
|
||
Never evict hard user constraints, unresolved items, irreversible side effects,
|
||
critical values, recent recovery state, or the previous complete reasoning and expectation.
|
||
|
||
## 12. Prompt and Cache Layout
|
||
|
||
Keep the stable Operator role, safety rules, cognitive protocol, and output
|
||
contract in `SystemMessage`. Put the previous real AI message, thought signature,
|
||
action call, and Validator tool result in the conversation tail. Put the current
|
||
goal, active plan path, budgeted history, current screenshot, and UI elements in
|
||
the dynamic `HumanMessage`. Keep tool schemas and ordering stable when possible.
|
||
|
||
If the provider adapter supports stateful interactions, preserve one to three
|
||
recent Operator interactions with `previous_interaction_id`. Start a new
|
||
interaction when the token threshold is reached or the plan changes materially,
|
||
seeding it with the stable prefix and chunk summaries. Until then, retain the
|
||
current message-rebuild path.
|
||
|
||
## 13. Same-Turn Context Control
|
||
|
||
Large ADB, Notes, Explorer, log, or XML results can expand later model calls in
|
||
the same Operator turn. Store complete results in a trace or artifact and expose
|
||
a local envelope:
|
||
|
||
```json
|
||
{
|
||
"status": "success",
|
||
"important_lines": ["..."],
|
||
"errors": [],
|
||
"tail": ["..."],
|
||
"full_result_ref": "trace://...",
|
||
"truncated": true,
|
||
"original_chars": 87321
|
||
}
|
||
```
|
||
|
||
Apply character, line, and estimated-token limits; retain matched lines, errors,
|
||
and head/tail context; never call another LLM synchronously to summarize tool
|
||
output. Compress older results at the soft limit and require a decision instead
|
||
of accumulating more exploration at the hard limit.
|
||
|
||
## 14. Configuration
|
||
|
||
Add a dedicated Pro memory section instead of reusing Flash `step_summarizer`:
|
||
|
||
```jsonc
|
||
"agent": {
|
||
"pro": {
|
||
"memory": {
|
||
"enabled": true,
|
||
"preserve_previous_operator_turn": true,
|
||
"max_hot_turns": 2,
|
||
"operator_target_input_tokens": 12000,
|
||
"operator_hard_input_tokens": 16000,
|
||
"same_turn_soft_tokens": 20000,
|
||
"same_turn_hard_tokens": 28000,
|
||
"visual_step_summary": {
|
||
"enabled": true,
|
||
"model": "gemini-3.7-flash",
|
||
"thinking_level": "low",
|
||
"include_pre_image": true,
|
||
"include_post_image_when_available": true,
|
||
"include_ui_diff": true,
|
||
"max_concurrency": 1,
|
||
"operator_never_waits": true
|
||
},
|
||
"history_chunks": {
|
||
"enabled": true,
|
||
"max_steps": 12,
|
||
"target_source_tokens": 1800,
|
||
"model": "gemini-3.7-flash"
|
||
},
|
||
"recall": {
|
||
"enabled": true,
|
||
"max_results": 5,
|
||
"max_text_tokens": 2000,
|
||
"max_image_steps": 1
|
||
}
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
For a speed-first single-quota deployment, override only the visual step model
|
||
with `gemini-3.5-flash-lite`.
|
||
|
||
## 15. Implementation Scope
|
||
|
||
### 15.1 Operator
|
||
|
||
Update Operator, prompt builders, and configuration to add a token-budget memory
|
||
assembler, preserve recent real messages and Validator results, avoid duplicate
|
||
history rendering, move dynamic state out of the stable system prompt, remove
|
||
`<short_term_memory>`, add `recall_history`, inject local screen-similarity hints,
|
||
and enforce same-turn output limits.
|
||
|
||
### 15.2 Summarizer
|
||
|
||
Load decision and optional post-action images by stable step ID, include available
|
||
Validator and UI-diff data, send multimodal messages, preserve the neutral
|
||
one-paragraph contract, atomically update versioned evidence metadata, retain old
|
||
summaries on failure, enforce concurrency and timeout limits, and never block Operator.
|
||
|
||
### 15.3 Graph and State
|
||
|
||
Store the previous Operator `AIMessage`, thought signature, and Validator function
|
||
response. Remove the `short_term_memory` field while safely ignoring it in older
|
||
tasks. Read ready memory snapshots at the Perception/Operator boundary and retain
|
||
the current asynchronous Summarizer semantics.
|
||
|
||
### 15.4 DataEngine
|
||
|
||
Add summary status, version, evidence, thought-signature, message-reference, and
|
||
UI-diff fields; persist history chunks and local recall indexes; support atomic
|
||
updates by step ID; and preserve every older summary version and source.
|
||
|
||
### 15.5 Task Tree
|
||
|
||
Separate history visibility from rendering detail, render visual summaries and
|
||
stable chunks, use current subgoal only for relevance, avoid repeating the latest
|
||
step when it is already present in the real message tail, and retain exact actions
|
||
and validation results.
|
||
|
||
### 15.6 Memory Module
|
||
|
||
Suggested files:
|
||
|
||
```text
|
||
artemis/memory/pro_memory.py
|
||
artemis/memory/history_chunker.py
|
||
artemis/memory/history_retriever.py
|
||
artemis/memory/token_budget.py
|
||
artemis/tools/history_recall.py
|
||
```
|
||
|
||
Responsibilities include memory snapshots, value-based selection, chunk closure
|
||
and background summarization, FTS and screen-signature retrieval, the recall tool,
|
||
and provider-independent token estimation.
|
||
|
||
### 15.7 Configuration
|
||
|
||
Every capability must have an independent switch, default to current Pro fallback
|
||
behavior, tolerate missing configuration and older tasks, and support both the
|
||
3.7 quality mode and Flash-Lite isolation mode.
|
||
|
||
## 16. Delivery Phases
|
||
|
||
### Phase 0: Baseline and Observability
|
||
|
||
Freeze a representative long-task suite. Record input tokens, TTFT, model latency,
|
||
full-turn latency, summary-ready latency, failures, retries, history size,
|
||
duplicate-action rate, and Failure Analyzer activation without changing prompts.
|
||
|
||
### Phase 1: Protect and Improve Hot History
|
||
|
||
Preserve complete recent reasoning, remove the duplicate short-term-memory path,
|
||
avoid duplicate rendering, add the total token threshold, and locally truncate
|
||
large tool envelopes. Prove that speed does not regress first.
|
||
|
||
### Phase 2: Pro Visual Step Summaries
|
||
|
||
Load actual decision and optional post-action images by stable step ID while
|
||
preserving the neutral output contract and text fallback. Compare 3.7 with images,
|
||
Flash-Lite with images, and current 3.7 text-only behavior. Confirm no Operator wait.
|
||
|
||
### Phase 3: Stable History Chunks
|
||
|
||
Add milestone-independent rolling chunks, generate them from source summaries,
|
||
treat milestones as temporary working-set labels, and add source references and versions.
|
||
|
||
### Phase 4: Operator Cold-History Recall
|
||
|
||
Reuse step, screenshot, and UI search capabilities behind one bounded
|
||
`recall_history` tool, add local screen-similarity hints, and limit text and images.
|
||
|
||
### Phase 5: Real Multi-Turn Messages and Cache Optimization
|
||
|
||
Preserve real AI messages, tool calls, and thought signatures; separate stable
|
||
system and dynamic human content; evaluate stateful provider APIs; and safely
|
||
restart interactions at token thresholds. Begin only after the first four phases stabilize.
|
||
|
||
## 17. Test Plan
|
||
|
||
Unit tests must cover hot-history preservation, removal of short-term-memory
|
||
parsing, action/Validator pairing, duplicate suppression, two-turn recovery,
|
||
image lookup by step ID, optional post-image behavior, neutral-output validation,
|
||
Validator precedence, concurrency, timeout, zero-wait behavior, chunk thresholds,
|
||
source traceability, recall bounds, image limits, and hard token protections.
|
||
|
||
Integration tests must include tasks longer than 30 steps, repeated similar
|
||
screens, plan rewrites, cross-app flows, dialogs, Toasts, WebViews, Canvas,
|
||
Failure Analyzer recovery, repeated ADB exploration, long lists, monitoring loops,
|
||
summary timeout or quota exhaustion, and exact-value recall from older screenshots.
|
||
|
||
Run blinded evaluations on real traces comparing 3.7 text-only, Flash-Lite with
|
||
available images, and 3.7 with available images. Reviewers score fidelity to
|
||
intent and explicit facts, unsupported visual claims, unauthorized outcome
|
||
judgments, anomaly retention, next-step usefulness, generation time, and ready latency.
|
||
|
||
## 18. Acceptance Criteria
|
||
|
||
Under the same device, task, model, and comparable service load:
|
||
|
||
- first-call input P50 increases by no more than 3% and P95 by no more than 5%;
|
||
- Operator model-call and full-turn P50/P90 latency regress by no more than 5%;
|
||
- observed step-interval P50/P90 regress by no more than 5%;
|
||
- Operator waits for summaries exactly zero times;
|
||
- normal Operator requests contain zero historical images; and
|
||
- images returned by `recall_history(include_images=True)` count only as explicit recall.
|
||
|
||
Run at least 30 paired tasks and report confidence intervals. Quality must not
|
||
regress for previous-strategy recall, overall task success, or Failure Analyzer
|
||
recovery. Visual-fact accuracy, important-fact retention across plan edits,
|
||
repeat-action rate, history-induced rollback rate, and top-five recall hit rate
|
||
must improve. Summary/Validator conflict should approach zero.
|
||
|
||
## 19. Failure Modes and Fallbacks
|
||
|
||
- If a visual summary is pending, use complete previous reasoning, exact action, and Validator result.
|
||
- If visual summarization fails, set `visual_failed`, preserve old summaries, and do not block for retries.
|
||
- If screenshot evidence is incomplete, include only actual evidence and never infer the missing post-action state.
|
||
- If chunk generation fails, keep source summaries and retry the pending chunk during an idle window.
|
||
- If recall output is too large, return high-scoring excerpts and step/artifact references.
|
||
- If background work competes with Operator, switch to Lite, pause new summaries, or use a separate quota.
|
||
- If stateful interactions are incompatible, return to prompt reconstruction while preserving raw/native thinking.
|
||
|
||
## 20. Rollout Flags
|
||
|
||
```text
|
||
pro_memory_enabled
|
||
preserve_operator_message_tail
|
||
visual_step_summary
|
||
history_chunks
|
||
operator_history_recall
|
||
stateful_operator_interactions
|
||
```
|
||
|
||
Recommended order: enable metrics and token thresholds; run visual summaries in
|
||
shadow mode; use them after blinded validation; enable stable chunks; enable
|
||
`recall_history`; then evaluate real multi-turn interactions. Disabling any flag
|
||
must restore current Pro behavior.
|
||
|
||
## 21. Improvements Over the Current System
|
||
|
||
| Capability | Current Pro | Redesigned Pro |
|
||
|---|---|---|
|
||
| Previous reasoning | Re-rendered as detailed history | Preserved and ready for a real message/thought-signature chain |
|
||
| Short-term strategy | Optional `<short_term_memory>` | Carried by the complete previous message, Validator, plan, and Notes |
|
||
| Step summary | 3.7 reads text only | Neutral 3.7 summary reads actual available images, UI, and Validator data |
|
||
| Exact action | Retained beside summary | Still retained and never replaced by visual summary |
|
||
| History pruning | Current subgoal plus latest three steps | Working set plus stable chunks and relevance budget |
|
||
| Plan rewrite | May change old-step visibility | Stable step/chunk references survive label changes |
|
||
| Old screenshots | No direct Operator recall | One actual step can be recalled on demand |
|
||
| Cold history | Primarily for HistoryAnalyzer/Outputter | Locally searchable and bounded for Operator |
|
||
| Context control | Mainly a step window | Total token budget, value density, and same-turn limits |
|
||
| Cache use | Highly dynamic system prompt | Stable system prefix separated from runtime state |
|
||
| Background failure | Summary may be delayed | Explicit state machine and zero-wait fallback |
|
||
|
||
The objective is not to display more history. It is to preserve recent reasoning,
|
||
give older summaries real visual context, decouple long-term identity from plan
|
||
milestones, make compressed records precisely recallable, and keep normal input
|
||
size and critical-path model-call count unchanged.
|
||
|
||
## 22. Recommended Defaults
|
||
|
||
```text
|
||
Normal Operator input: 10K-12K tokens
|
||
First-call soft/hard limits: 12K / 16K
|
||
Same-turn soft/hard limits: 20K / 28K
|
||
|
||
Complete hot turns: 1 normally, up to 2 for failure/uncertainty
|
||
Current-work summaries: budget-controlled, not a fixed count
|
||
History chunk: 8-12 steps or about 1.8K source-summary tokens
|
||
|
||
Historical images in normal input: 0
|
||
Summary input images: 0-1 decision, 0-1 independent post-action
|
||
Explicit recall images: actual images from at most one step
|
||
|
||
Preferred step-summary model: Gemini 3.7 Flash + images
|
||
Single-quota speed mode: Gemini 3.5 Flash-Lite + images
|
||
History-chunk model: Gemini 3.7 Flash
|
||
Background summary concurrency: 1
|
||
|
||
Additional critical-path calls: 0
|
||
Critical-path summary waits: 0
|
||
```
|
||
|
||
## 23. Final Decision
|
||
|
||
Do not remove previous reasoning and replace detailed history with an image-free
|
||
capsule. Preserve complete previous-turn reasoning, upgrade older steps to
|
||
visually grounded neutral summaries, use milestone-independent rolling chunks,
|
||
allow bounded cold-history recall, and enforce a fixed token budget.
|
||
|
||
The first, highest-value implementation slice is:
|
||
|
||
1. preserve complete previous reasoning and remove the duplicate short-term-memory path;
|
||
2. add the decision screenshot and optional actual post-action screenshot to the current neutral Pro Summarizer;
|
||
3. bind summaries with stable step IDs; and
|
||
4. establish speed baselines and blinded visual-summary evaluations.
|
||
|
||
Do not expand the normal history window or remove the current detailed-history
|
||
fallback until all four items have passed validation.
|