Artemis-Eapil/docs/design/pro-context-memory-redesign.md

ARTEMIS Pro Context, History, and Summarization Redesign

Status: Proposal (2026-08-25) — partially superseded

2026-08-31: docs/plans/history-module-redesign.md (unified v3) supersedes §6/§11/§12 of this document: Pro moves from per-turn prompt recompilation to an incremental append-only transcript with layered compaction (this doc's Phase 5 promoted to the core architecture). §7 (data model), §8 (visually grounded neutral summaries), §9 (stable chunks), §10 (recall), §13 (same-turn envelopes) are absorbed and remain authoritative where referenced.

Final status (2026-09-01): the unified v3 plan is fully implemented (M0–M5, work in worktree zealous-gates-4626b1, uncommitted). Everything this document delegated forward has landed, including §3.3 (the <short_term_memory> relay was removed end to end in M5 — state field, operator extraction, prompt component, visibility manifest entries, and the operator.json instruction block; render-time tag scrubbing is kept for older recorded responses, per this section's own "safely ignoring" clause). See history-module-redesign.md §7 for the per-milestone implementation notes.

Scope: ARTEMIS Pro / LangGraph execution mode

Audience: maintainers of Operator, Validator, Summarizer, DataEngine, and model routing

1. Executive Summary

This proposal improves long-term history, visually grounded neutral summaries, context compression, and history recall without weakening Pro mode's current reasoning continuity. The goal is not to give Operator more raw history by default. It is to increase information density while preserving the current 10K-12K input range and moving all additional image interpretation to a non-blocking background path.

The design preserves mechanisms that already work:

• the previous Operator step's complete raw or native thinking;
• the exact previous action and Validator result;
• the current screenshot and UI element list;
• complete, auditable raw step records in DataEngine; and
• asynchronous Summarizer scheduling that never blocks the graph's critical path.

It adds the following capabilities:

1. Preserve the previous model message, thought signature, action call, and Validator result as a continuous conversation tail.
2. Keep the current neutral summary contract while allowing the Summarizer to use the decision screenshot and an optional post-action screenshot as context.
3. Bind thinking, actions, available images, validation results, and summaries with a stable step_id.
4. Treat the active milestone only as a temporary working-set label; use plan-independent history chunks for long-term compression.
5. Add a bounded, on-demand recall_history tool for complete steps, older screenshots, and UI text.
6. Introduce a fixed token budget, value-density history selection, and same-turn tool-output limits.
7. Separate stable system rules from dynamic plans and history to improve prefix caching and multi-turn continuity.

The steady-state Operator context becomes:

Current screenshot/XML
  + complete previous-turn reasoning and action result
  + recent visually grounded step summaries
  + relevant history-chunk summaries
  + cold-history recall on demand

The steady-state target remains 10K-12K tokens, with zero additional model calls and zero summary waits on the critical path.

2. Goals and Constraints

2.1 First Priority: Preserve Speed

The engineering definition of preserving speed is:

• no new synchronous model calls in normal steps;
• Operator never waits for summaries, indexes, or history chunks;
• first-call input P50 and P95 remain close to the current baseline;
• background summaries do not compete at the same request priority as Operator;
• normal Operator requests still contain only one current screenshot;
• history improvements replace low-density content instead of expanding normal context; and
• any background failure immediately falls back to the current detailed history.

Absolute zero latency change cannot be guaranteed by adding synchronous input. This design therefore uses a token-neutral, zero-wait critical path and validates P50 and P90 through paired A/B tests.

2.2 History and Summary Goals

• Preserve the previous strategy, expected result, and unresolved questions.
• Ground older step summaries in available visual evidence instead of guessing from action text.
• Keep important history visible across plan rewrites, milestone renames, and goal rollback.
• Make every compressed record traceable to original steps, screenshots, and tool traces.
• Never discard failures, recovery work, side effects, user constraints, or critical values through ordinary compression.
• Allow Operator to recall older information during long tasks.

2.3 Use Model Capacity Effectively

"Maximize" means maximizing useful reasoning information, not filling a one-million-token window:

• the strongest model directly handles current visual state, continuous reasoning, and high-value history;
• available older screenshots help the Summarizer understand the UI, action target, and explicit records;
• long context acts as an exception and deep-recall buffer, not the normal working set;
• native multi-turn messages and thought signatures are preserved;
• stable system prefixes are reused whenever possible; and
• the model does not repeatedly reread duplicate plans, raw logs, or low-value click traces.

2.4 Non-Goals

• Do not delete or overwrite raw history in DataEngine.
• Do not send multiple historical screenshots to Operator by default.
• Do not treat plan milestones as permanent fact boundaries.
• Do not replace Validator's objective judgment with a summary.
• Do not require a new model API in the first phase.
• Do not enlarge every step merely to use a larger context window.

3. Current Baseline

3.1 Pro Graph Path

Planner
  -> Convergence
  -> Perception
  -> Operator
  -> Execution Check
  -> Validator
  -> Summarizer
  -> Convergence

The Summarizer node creates a background task and returns immediately. This behavior must remain unchanged.

3.2 Current Operator History

build_plan_and_history currently:

• renders the latest completed step in detail;
• may include raw/native thinking, actions, tool calls, Validator results, and Failure Analyzer traces;
• renders older summaries when available while retaining exact actions;
• preserves steps under the active subgoal plus the latest three global steps;
• puts the full current plan in the Operator prompt; and
• rebuilds messages on every Operator turn.

The current system already has continuity. Removing complete reasoning from the latest step would prevent the model from understanding the previous strategy, expectation, and decision basis.

3.3 Remove Separate Short-Term Memory

Remove the existing <short_term_memory> extraction, state storage, and next-turn injection path without introducing a replacement relay object. Cross-turn continuity should come from the complete previous Operator message, exact action, Validator result, current plan, long-term Notes, and auditable execution history.

3.4 Current Pro Summaries

The current Pro Summarizer receives only text:

• current plan and history;
• current-step thinking;
• action and execution result; and
• Failure Analyzer trace.

It does not load pre_image_name or post_image_name. Images should reduce context misunderstanding, not expand the Summarizer's authority to judge results.

3.5 Existing Cold History

Complete steps, screenshots, and UI data already exist in DataEngine. HistoryAnalyzer and Outputter offer partial retrieval, but Operator lacks one unified direct history-recall tool.

3.6 Measured Baseline

Current identifiable Pro Operator calls in traces/data_engine.db show:

• 74 direct Operator model calls across 43 Operator turns;
• median input of about 10,221 tokens;
• input P90 of about 11,648 tokens and P95 of about 12,019 tokens;
• maximum input of about 12,747 tokens;
• median model-call latency of about 5.24 seconds and P90 of about 16.42 seconds;
• median full Operator-turn latency of about 20.02 seconds; and
• median observed step interval of about 17.15 seconds.

Input length and model latency show no reliable correlation in the observed 8.8K-12.7K range. That result must not be extrapolated to an 80K-100K dynamic context.

4. Design Principles

4.1 Separate Raw Records from the Model View

DataEngine stores complete facts. The prompt contains only the most valuable view for the current turn. Compression changes what the model sees, never the source record.

4.2 Continuous Reasoning Before Compression

The complete previous reasoning, action expectation, action call, and Validator result are required on the hot path and cannot be replaced by an ordinary summary.

4.3 Images Provide Context, Not Judgment

The Summarizer may inspect screenshots that actually exist for a step, but it must preserve the current prompt's neutral responsibility. It may compress intent, strategy, explicit verification, iteration progress, missing conditions, and objective Failure Analyzer intervention. It must not independently decide whether an action succeeded, failed, completed, or navigated to a new page.

A missing post-action screenshot means only that no independent post-action visual evidence is available. It never means the screen was unchanged or the action had no effect.

4.4 Plans Are Dynamic Indexes, Not History Keys

The current subgoal may increase relevance but must not control permanent retention. History identity uses stable step_id values, time ranges, and source references.

4.5 Validator Facts Take Priority

Explicit user instructions
  > objective Validator or Controller results
  > activity, package, XML, or OCR differences
  > sourced content in visually grounded summaries
  > Operator or Summarizer inference

4.6 Background Enhancements Cannot Become Dependencies

If a visual summary or history chunk is not ready, Operator immediately continues with the existing detailed record.

5. Target Architecture

                         +-----------------------------+
                         | DataEngine raw facts        |
                         | steps / traces / images / UI|
                         +--------------+--------------+
                                        |
Validator completes step ---------------+
                                        |
                         +--------------v--------------+
                         | Step Memory Record          |
                         | exact action/result/sources |
                         +-------+--------------+-------+
                                 |              |
                         immediate              | asynchronous
                                 |              v
                                 |     +--------------------+
                                 |     | Context Summarizer |
                                 |     | decision image +   |
                                 |     | optional post/UI   |
                                 |     +----------+---------+
                                 |                |
                         +-------v----------------v-------+
                         | Pro Memory Store              |
                         | hot history/summaries/chunks  |
                         +---------------+---------------+
                                         |
Perception current screenshot/XML -------+
                                         v
                         +-----------------------------+
                         | Token-Budget Prompt Builder |
                         +--------------+--------------+
                                        v
                                    Operator
                                        |
                              recall_history on demand

6. Layered History Model

6.1 L0: Current Observation

Every Operator turn receives the current screenshot, package/activity, compact XML/OCR element list, screen dimensions and coordinate mapping, user-injected instructions, active plan path, and background-task state. Historical images do not enter normal requests.

6.2 L1: Hot History and Conversation Tail

Previous AIMessage
  - model-visible thinking / thought signature
  - raw reasoning text
  - turn-ending action tool call

Validator Function/Tool Result
  - whether execution occurred
  - execution state
  - Failure Analyzer repair
  - observed post-action state

Current HumanMessage
  - current screenshot
  - current XML

Keep one complete prior Operator turn normally and up to two when the last step failed, remains uncertain, is recovering, or has an unfinished strategy. Do not duplicate the same complete reasoning in Execution History. Until the provider can preserve real messages and thought signatures, continue rendering raw/native thinking verbatim.

6.3 L2: Visually Grounded, Neutral Step Summaries

Older steps use the current "summary plus exact action" structure, with a multimodal summary input:

Step N
  Neutral contextual summary
  Exact action
  Validator outcome
  Unresolved/recovery marker

The summary never replaces the exact action or overrides Validator. Screenshots only clarify context; the output does not make a new result judgment.

6.4 Current Work Segment

The active milestone may organize recent summaries, loop and candidate progress, unresolved items, and the most recent entry state. It remains a temporary label that can change with the plan and never becomes a permanent history identity.

6.5 L3: Stable History Chunks

Long-term history is chunked by stable step ranges and token thresholds:

Chunk 1: Steps 1-10
Chunk 2: Steps 11-19
Chunk 3: Steps 20-31

Close a chunk after roughly 1.5K-2K tokens of step summaries or 8-12 steps. Close early on a major package/activity change, user goal change, Failure Analyzer boundary, complete strategy change, long work-segment pause, irreversible side effect, or important external result.

Each chunk preserves its source range and step IDs, attempted path, verified facts, failed paths, unresolved questions, important entities, entry and exit states, and raw source references. Later plan rewrites do not alter chunk identity.

6.6 L4: Cold History

Retain every screenshot actually saved for a step, raw XML/OCR, raw/native thinking, complete tool calls and results, ADB and log output, the full Failure Analyzer trace, all summary versions, Notes, and plan revisions. Cold history is excluded from normal prompts and loaded only through local matching or recall_history.

7. Stable Data Model

7.1 StepMemoryRecord

class StepMemoryRecord:
    step_id: UUID
    session_id: UUID
    step_number: int
    timestamp: datetime

    plan_revision: str | None
    current_subgoal_hash: str | None

    operator_message_ref: str | None
    thought_signature: str | None
    raw_thinking_ref: str | None
    native_thinking_ref: str | None
    action: dict
    validator_result: dict | None
    failure_recovery: dict | None

    pre_image_name: str | None
    post_image_name: str | None
    pre_ui_ref: str | None
    post_ui_ref: str | None
    ui_diff: dict | None

    summary: str | None
    summary_status: str
    summary_model: str | None
    summary_version: int

    summary_decision_image_name: str | None
    summary_post_image_name: str | None
    summary_used_ui_diff: bool

The summary_status state machine is:

pending -> visual_ready -> visual_failed -> stale

Exact actions, Validator results, and raw sources remain immediately available regardless of summary state.

7.2 SummaryEvidenceMetadata

The summary remains one neutral natural-language paragraph. Deterministic source metadata is stored separately:

{
  "step_id": "...",
  "decision_image_name": "image-a",
  "post_image_name": null,
  "used_ui_diff": false,
  "model": "gemini-3.7-flash",
  "prompt_version": 2,
  "summary_version": 3
}

The system derives these fields from actual inputs. A null post_image_name expresses no action result or screen-change judgment.

7.3 HistoryChunk

class HistoryChunk:
    chunk_id: UUID
    session_id: UUID
    start_step_id: UUID
    end_step_id: UUID
    source_step_ids: list[UUID]
    digest: str
    verified_facts: list[str]
    unresolved: list[str]
    failed_paths: list[str]
    important_entities: list[str]
    entry_state: str | None
    exit_state: str | None
    model: str
    version: int

Chunk summaries are versioned append-only records and never damage source summaries.

8. Visually Grounded Neutral Pro Summarizer

8.1 Inputs

Each background summary call uses only inputs that actually exist:

1. the decision screenshot seen by Operator, optionally annotated with action coordinates;
2. an independent post-action screenshot only when Validator captured and saved it;
3. the exact action name and parameters;
4. the Controller result;
5. the final Validator result;
6. Failure Analyzer recovery records;
7. available package/activity information;
8. available UI/OCR differences;
9. current-step thinking; and
10. a small number of recent summaries for loop and strategy continuity.

Do not resend the full plan and ten detailed steps. Build the message dynamically, never substitute the decision screenshot for a missing post-action screenshot, and never infer "no screen change" from missing post-action evidence.

8.2 Output Rules

artemis/agents/summarizer/summarizer.md remains the sole authority for summary semantics. Preserve its philosophy and one-paragraph output. Use Operator's first person to compress intent, strategy, explicit progress, and verified facts. Do not independently claim success, failure, completion, non-completion, or page navigation. A successful Controller return does not prove task success. Do not guess unreadable values or judge outcomes by whether two images look similar.

Failure Analyzer intervention is summarized as an objective trigger and action, without claiming that the repair worked. Step ID, evidence names, model, prompt version, and summary version are system metadata rather than model-generated text.

The existing prohibition on outcome-judgment terms remains, including successfully, completed, entered, navigated to, failed, unsuccessful, could not, and achieved.

8.3 Model Selection

Quality-first default:

Per-step summary: Gemini 3.7 Flash + available screenshots
History-chunk summary: Gemini 3.7 Flash

Single-quota, strict-isolation option:

Real-time step summary: Gemini 3.5 Flash-Lite + available screenshots
Idle reprocessing for violations or important omissions: Gemini 3.7 + available screenshots
History-chunk summary: Gemini 3.7 Flash

The recommended quality ceiling remains 3.7 with available images. Flash-Lite is a resource-isolation option, not a replacement for complex strategy, loop, and recovery reasoning.

8.4 Scheduling and Isolation

• Operator never awaits Summarizer.
• Background summary concurrency is limited to one.
• Operator requests have absolute priority.
• Summary timeout or failure never affects graph execution.
• Existing detailed history remains the fallback.
• Prefer a separate endpoint, project, or quota.
• Without resource isolation, do not start a new 3.7 summary while Operator is active.
• Give non-preemptible background calls a short timeout.
• An optional final flush must not delay the next normal step.

9. Stable History Chunks

Milestones reflect a temporary interpretation of the task and are not stable fact boundaries. A plan edit can otherwise hide relevant steps, split one path across milestones, make rollback lose a prior strategy, or desynchronize summaries from current labels. Subgoal identity therefore affects relevance only.

if current_open_chunk.summary_tokens >= 1800:
    close_chunk()
elif current_open_chunk.step_count >= 12:
    close_chunk()
elif important_boundary_event:
    close_chunk()

Prevent recursive summary loss by generating each chunk from source step summaries, retaining source_step_ids, merging structured fact sets rather than guessing from prose, and retaining every summary version.

10. Operator Cold-History Recall

Expose one unified tool:

recall_history(
    query: str,
    step_range: list[int] | None = None,
    include_details: bool = False,
    include_images: bool = False,
    max_results: int = 5,
) -> HistoryRecallResult

Search visually grounded summaries, history chunks, raw actions, Validator results, Operator thinking, Notes, XML/OCR text, package/activity data, Failure Analyzer traces, and image references. Phase one can use SQLite FTS, keywords, step ranges, package/activity, and screen hashes without another model or vector database.

Return at most five results and 2K tokens by default. Every result includes a step ID or number. Detailed output remains token-bounded. Image recall returns no more than the images that actually exist for one step. Large raw results return only relevant excerpts and references.

Operator should recall history when the current screen resembles an older state, an old exact value is needed, two consecutive steps make no progress, the same action is about to be attempted a third time, a summary conflicts with current observation, the user asks to return to an earlier state, or Failure Analyzer needs review. Ordinary steps must not recall history speculatively.

Before prompt construction, a local package/activity, screenshot-hash, and UI- signature match may inject a short hint without a model call or historical image:

Historical state hint: The current screen closely resembles the post-action
screen from Step 17. Use recall_history only if its details are needed.

11. Token Budget and History Selection

Recommended steady-state budgets:

First Operator call target:      10K-12K
First-call soft limit:           12K
First-call emergency hard limit: 16K
Same-turn soft limit:            20K
Same-turn hard limit:            28K

history_budget = (
    operator_total_budget
    - stable_system_tokens
    - tool_schema_tokens
    - current_observation_tokens
    - active_plan_tokens
    - same_turn_reserve
)

Always retain the previous complete Operator turn, current user constraints, active plan path, unresolved items, recent failures and recovery, irreversible actions or external side effects, the previous chunk's exit state, and current loop or candidate progress.

Score optional history using current-subgoal relevance, unresolved status, failure or recovery relevance, screen similarity, entity overlap, path boundary, explicit user references, recency, resolved status, and duplication. Fill the remaining budget by value_density = priority / token_cost.

When over budget:

1. Remove duplicate renderings.
2. Keep the previous real message and remove its duplicate detailed history text.
3. Replace older detailed steps with visual summaries.
4. Replace groups of old summaries with stable chunks.
5. Remove resolved, unrelated, low-value chunks.
6. Keep only relevant excerpts and references from large tool output.

Never evict hard user constraints, unresolved items, irreversible side effects, critical values, recent recovery state, or the previous complete reasoning and expectation.

12. Prompt and Cache Layout

Keep the stable Operator role, safety rules, cognitive protocol, and output contract in SystemMessage. Put the previous real AI message, thought signature, action call, and Validator tool result in the conversation tail. Put the current goal, active plan path, budgeted history, current screenshot, and UI elements in the dynamic HumanMessage. Keep tool schemas and ordering stable when possible.

If the provider adapter supports stateful interactions, preserve one to three recent Operator interactions with previous_interaction_id. Start a new interaction when the token threshold is reached or the plan changes materially, seeding it with the stable prefix and chunk summaries. Until then, retain the current message-rebuild path.

13. Same-Turn Context Control

Large ADB, Notes, Explorer, log, or XML results can expand later model calls in the same Operator turn. Store complete results in a trace or artifact and expose a local envelope:

{
  "status": "success",
  "important_lines": ["..."],
  "errors": [],
  "tail": ["..."],
  "full_result_ref": "trace://...",
  "truncated": true,
  "original_chars": 87321
}

Apply character, line, and estimated-token limits; retain matched lines, errors, and head/tail context; never call another LLM synchronously to summarize tool output. Compress older results at the soft limit and require a decision instead of accumulating more exploration at the hard limit.

14. Configuration

Add a dedicated Pro memory section instead of reusing Flash step_summarizer:

"agent": {
  "pro": {
    "memory": {
      "enabled": true,
      "preserve_previous_operator_turn": true,
      "max_hot_turns": 2,
      "operator_target_input_tokens": 12000,
      "operator_hard_input_tokens": 16000,
      "same_turn_soft_tokens": 20000,
      "same_turn_hard_tokens": 28000,
      "visual_step_summary": {
        "enabled": true,
        "model": "gemini-3.7-flash",
        "thinking_level": "low",
        "include_pre_image": true,
        "include_post_image_when_available": true,
        "include_ui_diff": true,
        "max_concurrency": 1,
        "operator_never_waits": true
      },
      "history_chunks": {
        "enabled": true,
        "max_steps": 12,
        "target_source_tokens": 1800,
        "model": "gemini-3.7-flash"
      },
      "recall": {
        "enabled": true,
        "max_results": 5,
        "max_text_tokens": 2000,
        "max_image_steps": 1
      }
    }
  }
}

For a speed-first single-quota deployment, override only the visual step model with gemini-3.5-flash-lite.

15. Implementation Scope

15.1 Operator

Update Operator, prompt builders, and configuration to add a token-budget memory assembler, preserve recent real messages and Validator results, avoid duplicate history rendering, move dynamic state out of the stable system prompt, remove <short_term_memory>, add recall_history, inject local screen-similarity hints, and enforce same-turn output limits.

15.2 Summarizer

Load decision and optional post-action images by stable step ID, include available Validator and UI-diff data, send multimodal messages, preserve the neutral one-paragraph contract, atomically update versioned evidence metadata, retain old summaries on failure, enforce concurrency and timeout limits, and never block Operator.

15.3 Graph and State

Store the previous Operator AIMessage, thought signature, and Validator function response. Remove the short_term_memory field while safely ignoring it in older tasks. Read ready memory snapshots at the Perception/Operator boundary and retain the current asynchronous Summarizer semantics.

15.4 DataEngine

Add summary status, version, evidence, thought-signature, message-reference, and UI-diff fields; persist history chunks and local recall indexes; support atomic updates by step ID; and preserve every older summary version and source.

15.5 Task Tree

Separate history visibility from rendering detail, render visual summaries and stable chunks, use current subgoal only for relevance, avoid repeating the latest step when it is already present in the real message tail, and retain exact actions and validation results.

15.6 Memory Module

Suggested files:

artemis/memory/pro_memory.py
artemis/memory/history_chunker.py
artemis/memory/history_retriever.py
artemis/memory/token_budget.py
artemis/tools/history_recall.py

Responsibilities include memory snapshots, value-based selection, chunk closure and background summarization, FTS and screen-signature retrieval, the recall tool, and provider-independent token estimation.

15.7 Configuration

Every capability must have an independent switch, default to current Pro fallback behavior, tolerate missing configuration and older tasks, and support both the 3.7 quality mode and Flash-Lite isolation mode.

16. Delivery Phases

Phase 0: Baseline and Observability

Freeze a representative long-task suite. Record input tokens, TTFT, model latency, full-turn latency, summary-ready latency, failures, retries, history size, duplicate-action rate, and Failure Analyzer activation without changing prompts.

Phase 1: Protect and Improve Hot History

Preserve complete recent reasoning, remove the duplicate short-term-memory path, avoid duplicate rendering, add the total token threshold, and locally truncate large tool envelopes. Prove that speed does not regress first.

Phase 2: Pro Visual Step Summaries

Load actual decision and optional post-action images by stable step ID while preserving the neutral output contract and text fallback. Compare 3.7 with images, Flash-Lite with images, and current 3.7 text-only behavior. Confirm no Operator wait.

Phase 3: Stable History Chunks

Add milestone-independent rolling chunks, generate them from source summaries, treat milestones as temporary working-set labels, and add source references and versions.

Phase 4: Operator Cold-History Recall

Reuse step, screenshot, and UI search capabilities behind one bounded recall_history tool, add local screen-similarity hints, and limit text and images.

Phase 5: Real Multi-Turn Messages and Cache Optimization

Preserve real AI messages, tool calls, and thought signatures; separate stable system and dynamic human content; evaluate stateful provider APIs; and safely restart interactions at token thresholds. Begin only after the first four phases stabilize.

17. Test Plan

Unit tests must cover hot-history preservation, removal of short-term-memory parsing, action/Validator pairing, duplicate suppression, two-turn recovery, image lookup by step ID, optional post-image behavior, neutral-output validation, Validator precedence, concurrency, timeout, zero-wait behavior, chunk thresholds, source traceability, recall bounds, image limits, and hard token protections.

Integration tests must include tasks longer than 30 steps, repeated similar screens, plan rewrites, cross-app flows, dialogs, Toasts, WebViews, Canvas, Failure Analyzer recovery, repeated ADB exploration, long lists, monitoring loops, summary timeout or quota exhaustion, and exact-value recall from older screenshots.

Run blinded evaluations on real traces comparing 3.7 text-only, Flash-Lite with available images, and 3.7 with available images. Reviewers score fidelity to intent and explicit facts, unsupported visual claims, unauthorized outcome judgments, anomaly retention, next-step usefulness, generation time, and ready latency.

18. Acceptance Criteria

Under the same device, task, model, and comparable service load:

• first-call input P50 increases by no more than 3% and P95 by no more than 5%;
• Operator model-call and full-turn P50/P90 latency regress by no more than 5%;
• observed step-interval P50/P90 regress by no more than 5%;
• Operator waits for summaries exactly zero times;
• normal Operator requests contain zero historical images; and
• images returned by recall_history(include_images=True) count only as explicit recall.

Run at least 30 paired tasks and report confidence intervals. Quality must not regress for previous-strategy recall, overall task success, or Failure Analyzer recovery. Visual-fact accuracy, important-fact retention across plan edits, repeat-action rate, history-induced rollback rate, and top-five recall hit rate must improve. Summary/Validator conflict should approach zero.

19. Failure Modes and Fallbacks

• If a visual summary is pending, use complete previous reasoning, exact action, and Validator result.
• If visual summarization fails, set visual_failed, preserve old summaries, and do not block for retries.
• If screenshot evidence is incomplete, include only actual evidence and never infer the missing post-action state.
• If chunk generation fails, keep source summaries and retry the pending chunk during an idle window.
• If recall output is too large, return high-scoring excerpts and step/artifact references.
• If background work competes with Operator, switch to Lite, pause new summaries, or use a separate quota.
• If stateful interactions are incompatible, return to prompt reconstruction while preserving raw/native thinking.

20. Rollout Flags

pro_memory_enabled
preserve_operator_message_tail
visual_step_summary
history_chunks
operator_history_recall
stateful_operator_interactions

Recommended order: enable metrics and token thresholds; run visual summaries in shadow mode; use them after blinded validation; enable stable chunks; enable recall_history; then evaluate real multi-turn interactions. Disabling any flag must restore current Pro behavior.

21. Improvements Over the Current System

Capability	Current Pro	Redesigned Pro
Previous reasoning	Re-rendered as detailed history	Preserved and ready for a real message/thought-signature chain
Short-term strategy	Optional <short_term_memory>	Carried by the complete previous message, Validator, plan, and Notes
Step summary	3.7 reads text only	Neutral 3.7 summary reads actual available images, UI, and Validator data
Exact action	Retained beside summary	Still retained and never replaced by visual summary
History pruning	Current subgoal plus latest three steps	Working set plus stable chunks and relevance budget
Plan rewrite	May change old-step visibility	Stable step/chunk references survive label changes
Old screenshots	No direct Operator recall	One actual step can be recalled on demand
Cold history	Primarily for HistoryAnalyzer/Outputter	Locally searchable and bounded for Operator
Context control	Mainly a step window	Total token budget, value density, and same-turn limits
Cache use	Highly dynamic system prompt	Stable system prefix separated from runtime state
Background failure	Summary may be delayed	Explicit state machine and zero-wait fallback

The objective is not to display more history. It is to preserve recent reasoning, give older summaries real visual context, decouple long-term identity from plan milestones, make compressed records precisely recallable, and keep normal input size and critical-path model-call count unchanged.

22. Recommended Defaults

Normal Operator input:             10K-12K tokens
First-call soft/hard limits:       12K / 16K
Same-turn soft/hard limits:        20K / 28K

Complete hot turns:                1 normally, up to 2 for failure/uncertainty
Current-work summaries:            budget-controlled, not a fixed count
History chunk:                     8-12 steps or about 1.8K source-summary tokens

Historical images in normal input: 0
Summary input images:              0-1 decision, 0-1 independent post-action
Explicit recall images:            actual images from at most one step

Preferred step-summary model:      Gemini 3.7 Flash + images
Single-quota speed mode:           Gemini 3.5 Flash-Lite + images
History-chunk model:               Gemini 3.7 Flash
Background summary concurrency:    1

Additional critical-path calls:    0
Critical-path summary waits:       0

23. Final Decision

Do not remove previous reasoning and replace detailed history with an image-free capsule. Preserve complete previous-turn reasoning, upgrade older steps to visually grounded neutral summaries, use milestone-independent rolling chunks, allow bounded cold-history recall, and enforce a fixed token budget.

The first, highest-value implementation slice is:

1. preserve complete previous reasoning and remove the duplicate short-term-memory path;
2. add the decision screenshot and optional actual post-action screenshot to the current neutral Pro Summarizer;
3. bind summaries with stable step IDs; and
4. establish speed baselines and blinded visual-summary evaluations.

Do not expand the normal history window or remove the current detailed-history fallback until all four items have passed validation.