feat: enable remote SDK and resilient device orchestration
This commit is contained in:
somew1nd committed
1 parent
3cc711df51
commit
ddfa420945
110 files changed
+14450
-12954
No files matched your search
@@ -0,0 +1,897 @@
|
||||
# ARTEMIS Pro Context, History, and Summarization Redesign
|
||||
|
||||
Status: Proposal (2026-08-25)
|
||||
|
||||
Scope: ARTEMIS Pro / LangGraph execution mode
|
||||
|
||||
Audience: maintainers of Operator, Validator, Summarizer, DataEngine, and model routing
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
This proposal improves long-term history, visually grounded neutral summaries,
|
||||
context compression, and history recall without weakening Pro mode's current
|
||||
reasoning continuity. The goal is not to give Operator more raw history by
|
||||
default. It is to increase information density while preserving the current
|
||||
`10K-12K` input range and moving all additional image interpretation to a
|
||||
non-blocking background path.
|
||||
|
||||
The design preserves mechanisms that already work:
|
||||
|
||||
- the previous Operator step's complete raw or native thinking;
|
||||
- the exact previous action and Validator result;
|
||||
- the current screenshot and UI element list;
|
||||
- complete, auditable raw step records in DataEngine; and
|
||||
- asynchronous Summarizer scheduling that never blocks the graph's critical path.
|
||||
|
||||
It adds the following capabilities:
|
||||
|
||||
1. Preserve the previous model message, thought signature, action call, and Validator result as a continuous conversation tail.
|
||||
2. Keep the current neutral summary contract while allowing the Summarizer to use the decision screenshot and an optional post-action screenshot as context.
|
||||
3. Bind thinking, actions, available images, validation results, and summaries with a stable `step_id`.
|
||||
4. Treat the active milestone only as a temporary working-set label; use plan-independent history chunks for long-term compression.
|
||||
5. Add a bounded, on-demand `recall_history` tool for complete steps, older screenshots, and UI text.
|
||||
6. Introduce a fixed token budget, value-density history selection, and same-turn tool-output limits.
|
||||
7. Separate stable system rules from dynamic plans and history to improve prefix caching and multi-turn continuity.
|
||||
|
||||
The steady-state Operator context becomes:
|
||||
|
||||
```text
|
||||
Current screenshot/XML
|
||||
+ complete previous-turn reasoning and action result
|
||||
+ recent visually grounded step summaries
|
||||
+ relevant history-chunk summaries
|
||||
+ cold-history recall on demand
|
||||
```
|
||||
|
||||
The steady-state target remains `10K-12K tokens`, with zero additional model
|
||||
calls and zero summary waits on the critical path.
|
||||
|
||||
## 2. Goals and Constraints
|
||||
|
||||
### 2.1 First Priority: Preserve Speed
|
||||
|
||||
The engineering definition of preserving speed is:
|
||||
|
||||
- no new synchronous model calls in normal steps;
|
||||
- Operator never waits for summaries, indexes, or history chunks;
|
||||
- first-call input P50 and P95 remain close to the current baseline;
|
||||
- background summaries do not compete at the same request priority as Operator;
|
||||
- normal Operator requests still contain only one current screenshot;
|
||||
- history improvements replace low-density content instead of expanding normal context; and
|
||||
- any background failure immediately falls back to the current detailed history.
|
||||
|
||||
Absolute zero latency change cannot be guaranteed by adding synchronous input.
|
||||
This design therefore uses a token-neutral, zero-wait critical path and validates
|
||||
P50 and P90 through paired A/B tests.
|
||||
|
||||
### 2.2 History and Summary Goals
|
||||
|
||||
- Preserve the previous strategy, expected result, and unresolved questions.
|
||||
- Ground older step summaries in available visual evidence instead of guessing from action text.
|
||||
- Keep important history visible across plan rewrites, milestone renames, and goal rollback.
|
||||
- Make every compressed record traceable to original steps, screenshots, and tool traces.
|
||||
- Never discard failures, recovery work, side effects, user constraints, or critical values through ordinary compression.
|
||||
- Allow Operator to recall older information during long tasks.
|
||||
|
||||
### 2.3 Use Model Capacity Effectively
|
||||
|
||||
"Maximize" means maximizing useful reasoning information, not filling a one-million-token window:
|
||||
|
||||
- the strongest model directly handles current visual state, continuous reasoning, and high-value history;
|
||||
- available older screenshots help the Summarizer understand the UI, action target, and explicit records;
|
||||
- long context acts as an exception and deep-recall buffer, not the normal working set;
|
||||
- native multi-turn messages and thought signatures are preserved;
|
||||
- stable system prefixes are reused whenever possible; and
|
||||
- the model does not repeatedly reread duplicate plans, raw logs, or low-value click traces.
|
||||
|
||||
### 2.4 Non-Goals
|
||||
|
||||
- Do not delete or overwrite raw history in DataEngine.
|
||||
- Do not send multiple historical screenshots to Operator by default.
|
||||
- Do not treat plan milestones as permanent fact boundaries.
|
||||
- Do not replace Validator's objective judgment with a summary.
|
||||
- Do not require a new model API in the first phase.
|
||||
- Do not enlarge every step merely to use a larger context window.
|
||||
|
||||
## 3. Current Baseline
|
||||
|
||||
### 3.1 Pro Graph Path
|
||||
|
||||
```text
|
||||
Planner
|
||||
-> Convergence
|
||||
-> Perception
|
||||
-> Operator
|
||||
-> Execution Check
|
||||
-> Validator
|
||||
-> Summarizer
|
||||
-> Convergence
|
||||
```
|
||||
|
||||
The Summarizer node creates a background task and returns immediately. This
|
||||
behavior must remain unchanged.
|
||||
|
||||
### 3.2 Current Operator History
|
||||
|
||||
`build_plan_and_history` currently:
|
||||
|
||||
- renders the latest completed step in detail;
|
||||
- may include raw/native thinking, actions, tool calls, Validator results, and Failure Analyzer traces;
|
||||
- renders older summaries when available while retaining exact actions;
|
||||
- preserves steps under the active subgoal plus the latest three global steps;
|
||||
- puts the full current plan in the Operator prompt; and
|
||||
- rebuilds messages on every Operator turn.
|
||||
|
||||
The current system already has continuity. Removing complete reasoning from the
|
||||
latest step would prevent the model from understanding the previous strategy,
|
||||
expectation, and decision basis.
|
||||
|
||||
### 3.3 Remove Separate Short-Term Memory
|
||||
|
||||
Remove the existing `<short_term_memory>` extraction, state storage, and next-turn
|
||||
injection path without introducing a replacement relay object. Cross-turn
|
||||
continuity should come from the complete previous Operator message, exact action,
|
||||
Validator result, current plan, long-term Notes, and auditable execution history.
|
||||
|
||||
### 3.4 Current Pro Summaries
|
||||
|
||||
The current Pro Summarizer receives only text:
|
||||
|
||||
- current plan and history;
|
||||
- current-step thinking;
|
||||
- action and execution result; and
|
||||
- Failure Analyzer trace.
|
||||
|
||||
It does not load `pre_image_name` or `post_image_name`. Images should reduce
|
||||
context misunderstanding, not expand the Summarizer's authority to judge results.
|
||||
|
||||
### 3.5 Existing Cold History
|
||||
|
||||
Complete steps, screenshots, and UI data already exist in DataEngine.
|
||||
HistoryAnalyzer and Outputter offer partial retrieval, but Operator lacks one
|
||||
unified direct history-recall tool.
|
||||
|
||||
### 3.6 Measured Baseline
|
||||
|
||||
Current identifiable Pro Operator calls in `traces/data_engine.db` show:
|
||||
|
||||
- 74 direct Operator model calls across 43 Operator turns;
|
||||
- median input of about 10,221 tokens;
|
||||
- input P90 of about 11,648 tokens and P95 of about 12,019 tokens;
|
||||
- maximum input of about 12,747 tokens;
|
||||
- median model-call latency of about 5.24 seconds and P90 of about 16.42 seconds;
|
||||
- median full Operator-turn latency of about 20.02 seconds; and
|
||||
- median observed step interval of about 17.15 seconds.
|
||||
|
||||
Input length and model latency show no reliable correlation in the observed
|
||||
`8.8K-12.7K` range. That result must not be extrapolated to an `80K-100K`
|
||||
dynamic context.
|
||||
|
||||
## 4. Design Principles
|
||||
|
||||
### 4.1 Separate Raw Records from the Model View
|
||||
|
||||
DataEngine stores complete facts. The prompt contains only the most valuable view
|
||||
for the current turn. Compression changes what the model sees, never the source record.
|
||||
|
||||
### 4.2 Continuous Reasoning Before Compression
|
||||
|
||||
The complete previous reasoning, action expectation, action call, and Validator
|
||||
result are required on the hot path and cannot be replaced by an ordinary summary.
|
||||
|
||||
### 4.3 Images Provide Context, Not Judgment
|
||||
|
||||
The Summarizer may inspect screenshots that actually exist for a step, but it
|
||||
must preserve the current prompt's neutral responsibility. It may compress
|
||||
intent, strategy, explicit verification, iteration progress, missing conditions,
|
||||
and objective Failure Analyzer intervention. It must not independently decide
|
||||
whether an action succeeded, failed, completed, or navigated to a new page.
|
||||
|
||||
A missing post-action screenshot means only that no independent post-action
|
||||
visual evidence is available. It never means the screen was unchanged or the
|
||||
action had no effect.
|
||||
|
||||
### 4.4 Plans Are Dynamic Indexes, Not History Keys
|
||||
|
||||
The current subgoal may increase relevance but must not control permanent
|
||||
retention. History identity uses stable `step_id` values, time ranges, and source references.
|
||||
|
||||
### 4.5 Validator Facts Take Priority
|
||||
|
||||
```text
|
||||
Explicit user instructions
|
||||
> objective Validator or Controller results
|
||||
> activity, package, XML, or OCR differences
|
||||
> sourced content in visually grounded summaries
|
||||
> Operator or Summarizer inference
|
||||
```
|
||||
|
||||
### 4.6 Background Enhancements Cannot Become Dependencies
|
||||
|
||||
If a visual summary or history chunk is not ready, Operator immediately continues
|
||||
with the existing detailed record.
|
||||
|
||||
## 5. Target Architecture
|
||||
|
||||
```text
|
||||
+-----------------------------+
|
||||
| DataEngine raw facts |
|
||||
| steps / traces / images / UI|
|
||||
+--------------+--------------+
|
||||
|
|
||||
Validator completes step ---------------+
|
||||
|
|
||||
+--------------v--------------+
|
||||
| Step Memory Record |
|
||||
| exact action/result/sources |
|
||||
+-------+--------------+-------+
|
||||
| |
|
||||
immediate | asynchronous
|
||||
| v
|
||||
| +--------------------+
|
||||
| | Context Summarizer |
|
||||
| | decision image + |
|
||||
| | optional post/UI |
|
||||
| +----------+---------+
|
||||
| |
|
||||
+-------v----------------v-------+
|
||||
| Pro Memory Store |
|
||||
| hot history/summaries/chunks |
|
||||
+---------------+---------------+
|
||||
|
|
||||
Perception current screenshot/XML -------+
|
||||
v
|
||||
+-----------------------------+
|
||||
| Token-Budget Prompt Builder |
|
||||
+--------------+--------------+
|
||||
v
|
||||
Operator
|
||||
|
|
||||
recall_history on demand
|
||||
```
|
||||
|
||||
## 6. Layered History Model
|
||||
|
||||
### 6.1 L0: Current Observation
|
||||
|
||||
Every Operator turn receives the current screenshot, package/activity, compact
|
||||
XML/OCR element list, screen dimensions and coordinate mapping, user-injected
|
||||
instructions, active plan path, and background-task state. Historical images do
|
||||
not enter normal requests.
|
||||
|
||||
### 6.2 L1: Hot History and Conversation Tail
|
||||
|
||||
```text
|
||||
Previous AIMessage
|
||||
- model-visible thinking / thought signature
|
||||
- raw reasoning text
|
||||
- turn-ending action tool call
|
||||
|
||||
Validator Function/Tool Result
|
||||
- whether execution occurred
|
||||
- execution state
|
||||
- Failure Analyzer repair
|
||||
- observed post-action state
|
||||
|
||||
Current HumanMessage
|
||||
- current screenshot
|
||||
- current XML
|
||||
```
|
||||
|
||||
Keep one complete prior Operator turn normally and up to two when the last step
|
||||
failed, remains uncertain, is recovering, or has an unfinished strategy. Do not
|
||||
duplicate the same complete reasoning in `Execution History`. Until the provider
|
||||
can preserve real messages and thought signatures, continue rendering raw/native
|
||||
thinking verbatim.
|
||||
|
||||
### 6.3 L2: Visually Grounded, Neutral Step Summaries
|
||||
|
||||
Older steps use the current "summary plus exact action" structure, with a
|
||||
multimodal summary input:
|
||||
|
||||
```text
|
||||
Step N
|
||||
Neutral contextual summary
|
||||
Exact action
|
||||
Validator outcome
|
||||
Unresolved/recovery marker
|
||||
```
|
||||
|
||||
The summary never replaces the exact action or overrides Validator. Screenshots
|
||||
only clarify context; the output does not make a new result judgment.
|
||||
|
||||
### 6.4 Current Work Segment
|
||||
|
||||
The active milestone may organize recent summaries, loop and candidate progress,
|
||||
unresolved items, and the most recent entry state. It remains a temporary label
|
||||
that can change with the plan and never becomes a permanent history identity.
|
||||
|
||||
### 6.5 L3: Stable History Chunks
|
||||
|
||||
Long-term history is chunked by stable step ranges and token thresholds:
|
||||
|
||||
```text
|
||||
Chunk 1: Steps 1-10
|
||||
Chunk 2: Steps 11-19
|
||||
Chunk 3: Steps 20-31
|
||||
```
|
||||
|
||||
Close a chunk after roughly `1.5K-2K tokens` of step summaries or 8-12 steps.
|
||||
Close early on a major package/activity change, user goal change, Failure Analyzer
|
||||
boundary, complete strategy change, long work-segment pause, irreversible side
|
||||
effect, or important external result.
|
||||
|
||||
Each chunk preserves its source range and step IDs, attempted path, verified
|
||||
facts, failed paths, unresolved questions, important entities, entry and exit
|
||||
states, and raw source references. Later plan rewrites do not alter chunk identity.
|
||||
|
||||
### 6.6 L4: Cold History
|
||||
|
||||
Retain every screenshot actually saved for a step, raw XML/OCR, raw/native
|
||||
thinking, complete tool calls and results, ADB and log output, the full Failure
|
||||
Analyzer trace, all summary versions, Notes, and plan revisions. Cold history is
|
||||
excluded from normal prompts and loaded only through local matching or
|
||||
`recall_history`.
|
||||
|
||||
## 7. Stable Data Model
|
||||
|
||||
### 7.1 StepMemoryRecord
|
||||
|
||||
```python
|
||||
class StepMemoryRecord:
|
||||
step_id: UUID
|
||||
session_id: UUID
|
||||
step_number: int
|
||||
timestamp: datetime
|
||||
|
||||
plan_revision: str | None
|
||||
current_subgoal_hash: str | None
|
||||
|
||||
operator_message_ref: str | None
|
||||
thought_signature: str | None
|
||||
raw_thinking_ref: str | None
|
||||
native_thinking_ref: str | None
|
||||
action: dict
|
||||
validator_result: dict | None
|
||||
failure_recovery: dict | None
|
||||
|
||||
pre_image_name: str | None
|
||||
post_image_name: str | None
|
||||
pre_ui_ref: str | None
|
||||
post_ui_ref: str | None
|
||||
ui_diff: dict | None
|
||||
|
||||
summary: str | None
|
||||
summary_status: str
|
||||
summary_model: str | None
|
||||
summary_version: int
|
||||
|
||||
summary_decision_image_name: str | None
|
||||
summary_post_image_name: str | None
|
||||
summary_used_ui_diff: bool
|
||||
```
|
||||
|
||||
The `summary_status` state machine is:
|
||||
|
||||
```text
|
||||
pending -> visual_ready -> visual_failed -> stale
|
||||
```
|
||||
|
||||
Exact actions, Validator results, and raw sources remain immediately available
|
||||
regardless of summary state.
|
||||
|
||||
### 7.2 SummaryEvidenceMetadata
|
||||
|
||||
The summary remains one neutral natural-language paragraph. Deterministic source
|
||||
metadata is stored separately:
|
||||
|
||||
```json
|
||||
{
|
||||
"step_id": "...",
|
||||
"decision_image_name": "image-a",
|
||||
"post_image_name": null,
|
||||
"used_ui_diff": false,
|
||||
"model": "gemini-3.7-flash",
|
||||
"prompt_version": 2,
|
||||
"summary_version": 3
|
||||
}
|
||||
```
|
||||
|
||||
The system derives these fields from actual inputs. A null `post_image_name`
|
||||
expresses no action result or screen-change judgment.
|
||||
|
||||
### 7.3 HistoryChunk
|
||||
|
||||
```python
|
||||
class HistoryChunk:
|
||||
chunk_id: UUID
|
||||
session_id: UUID
|
||||
start_step_id: UUID
|
||||
end_step_id: UUID
|
||||
source_step_ids: list[UUID]
|
||||
digest: str
|
||||
verified_facts: list[str]
|
||||
unresolved: list[str]
|
||||
failed_paths: list[str]
|
||||
important_entities: list[str]
|
||||
entry_state: str | None
|
||||
exit_state: str | None
|
||||
model: str
|
||||
version: int
|
||||
```
|
||||
|
||||
Chunk summaries are versioned append-only records and never damage source summaries.
|
||||
|
||||
## 8. Visually Grounded Neutral Pro Summarizer
|
||||
|
||||
### 8.1 Inputs
|
||||
|
||||
Each background summary call uses only inputs that actually exist:
|
||||
|
||||
1. the decision screenshot seen by Operator, optionally annotated with action coordinates;
|
||||
2. an independent post-action screenshot only when Validator captured and saved it;
|
||||
3. the exact action name and parameters;
|
||||
4. the Controller result;
|
||||
5. the final Validator result;
|
||||
6. Failure Analyzer recovery records;
|
||||
7. available package/activity information;
|
||||
8. available UI/OCR differences;
|
||||
9. current-step thinking; and
|
||||
10. a small number of recent summaries for loop and strategy continuity.
|
||||
|
||||
Do not resend the full plan and ten detailed steps. Build the message dynamically,
|
||||
never substitute the decision screenshot for a missing post-action screenshot,
|
||||
and never infer "no screen change" from missing post-action evidence.
|
||||
|
||||
### 8.2 Output Rules
|
||||
|
||||
`artemis/agents/summarizer/summarizer.md` remains the sole authority for summary
|
||||
semantics. Preserve its philosophy and one-paragraph output. Use Operator's first
|
||||
person to compress intent, strategy, explicit progress, and verified facts. Do
|
||||
not independently claim success, failure, completion, non-completion, or page
|
||||
navigation. A successful Controller return does not prove task success. Do not
|
||||
guess unreadable values or judge outcomes by whether two images look similar.
|
||||
|
||||
Failure Analyzer intervention is summarized as an objective trigger and action,
|
||||
without claiming that the repair worked. Step ID, evidence names, model, prompt
|
||||
version, and summary version are system metadata rather than model-generated text.
|
||||
|
||||
The existing prohibition on outcome-judgment terms remains, including
|
||||
`successfully`, `completed`, `entered`, `navigated to`, `failed`, `unsuccessful`,
|
||||
`could not`, and `achieved`.
|
||||
|
||||
### 8.3 Model Selection
|
||||
|
||||
Quality-first default:
|
||||
|
||||
```text
|
||||
Per-step summary: Gemini 3.7 Flash + available screenshots
|
||||
History-chunk summary: Gemini 3.7 Flash
|
||||
```
|
||||
|
||||
Single-quota, strict-isolation option:
|
||||
|
||||
```text
|
||||
Real-time step summary: Gemini 3.5 Flash-Lite + available screenshots
|
||||
Idle reprocessing for violations or important omissions: Gemini 3.7 + available screenshots
|
||||
History-chunk summary: Gemini 3.7 Flash
|
||||
```
|
||||
|
||||
The recommended quality ceiling remains 3.7 with available images. Flash-Lite is
|
||||
a resource-isolation option, not a replacement for complex strategy, loop, and
|
||||
recovery reasoning.
|
||||
|
||||
### 8.4 Scheduling and Isolation
|
||||
|
||||
- Operator never awaits Summarizer.
|
||||
- Background summary concurrency is limited to one.
|
||||
- Operator requests have absolute priority.
|
||||
- Summary timeout or failure never affects graph execution.
|
||||
- Existing detailed history remains the fallback.
|
||||
- Prefer a separate endpoint, project, or quota.
|
||||
- Without resource isolation, do not start a new 3.7 summary while Operator is active.
|
||||
- Give non-preemptible background calls a short timeout.
|
||||
- An optional final flush must not delay the next normal step.
|
||||
|
||||
## 9. Stable History Chunks
|
||||
|
||||
Milestones reflect a temporary interpretation of the task and are not stable fact
|
||||
boundaries. A plan edit can otherwise hide relevant steps, split one path across
|
||||
milestones, make rollback lose a prior strategy, or desynchronize summaries from
|
||||
current labels. Subgoal identity therefore affects relevance only.
|
||||
|
||||
```python
|
||||
if current_open_chunk.summary_tokens >= 1800:
|
||||
close_chunk()
|
||||
elif current_open_chunk.step_count >= 12:
|
||||
close_chunk()
|
||||
elif important_boundary_event:
|
||||
close_chunk()
|
||||
```
|
||||
|
||||
Prevent recursive summary loss by generating each chunk from source step
|
||||
summaries, retaining `source_step_ids`, merging structured fact sets rather than
|
||||
guessing from prose, and retaining every summary version.
|
||||
|
||||
## 10. Operator Cold-History Recall
|
||||
|
||||
Expose one unified tool:
|
||||
|
||||
```python
|
||||
recall_history(
|
||||
query: str,
|
||||
step_range: list[int] | None = None,
|
||||
include_details: bool = False,
|
||||
include_images: bool = False,
|
||||
max_results: int = 5,
|
||||
) -> HistoryRecallResult
|
||||
```
|
||||
|
||||
Search visually grounded summaries, history chunks, raw actions, Validator
|
||||
results, Operator thinking, Notes, XML/OCR text, package/activity data, Failure
|
||||
Analyzer traces, and image references. Phase one can use SQLite FTS, keywords,
|
||||
step ranges, package/activity, and screen hashes without another model or vector database.
|
||||
|
||||
Return at most five results and 2K tokens by default. Every result includes a
|
||||
step ID or number. Detailed output remains token-bounded. Image recall returns no
|
||||
more than the images that actually exist for one step. Large raw results return
|
||||
only relevant excerpts and references.
|
||||
|
||||
Operator should recall history when the current screen resembles an older state,
|
||||
an old exact value is needed, two consecutive steps make no progress, the same
|
||||
action is about to be attempted a third time, a summary conflicts with current
|
||||
observation, the user asks to return to an earlier state, or Failure Analyzer
|
||||
needs review. Ordinary steps must not recall history speculatively.
|
||||
|
||||
Before prompt construction, a local package/activity, screenshot-hash, and UI-
|
||||
signature match may inject a short hint without a model call or historical image:
|
||||
|
||||
```text
|
||||
Historical state hint: The current screen closely resembles the post-action
|
||||
screen from Step 17. Use recall_history only if its details are needed.
|
||||
```
|
||||
|
||||
## 11. Token Budget and History Selection
|
||||
|
||||
Recommended steady-state budgets:
|
||||
|
||||
```text
|
||||
First Operator call target: 10K-12K
|
||||
First-call soft limit: 12K
|
||||
First-call emergency hard limit: 16K
|
||||
Same-turn soft limit: 20K
|
||||
Same-turn hard limit: 28K
|
||||
```
|
||||
|
||||
```python
|
||||
history_budget = (
|
||||
operator_total_budget
|
||||
- stable_system_tokens
|
||||
- tool_schema_tokens
|
||||
- current_observation_tokens
|
||||
- active_plan_tokens
|
||||
- same_turn_reserve
|
||||
)
|
||||
```
|
||||
|
||||
Always retain the previous complete Operator turn, current user constraints,
|
||||
active plan path, unresolved items, recent failures and recovery, irreversible
|
||||
actions or external side effects, the previous chunk's exit state, and current
|
||||
loop or candidate progress.
|
||||
|
||||
Score optional history using current-subgoal relevance, unresolved status,
|
||||
failure or recovery relevance, screen similarity, entity overlap, path boundary,
|
||||
explicit user references, recency, resolved status, and duplication. Fill the
|
||||
remaining budget by `value_density = priority / token_cost`.
|
||||
|
||||
When over budget:
|
||||
|
||||
1. Remove duplicate renderings.
|
||||
2. Keep the previous real message and remove its duplicate detailed history text.
|
||||
3. Replace older detailed steps with visual summaries.
|
||||
4. Replace groups of old summaries with stable chunks.
|
||||
5. Remove resolved, unrelated, low-value chunks.
|
||||
6. Keep only relevant excerpts and references from large tool output.
|
||||
|
||||
Never evict hard user constraints, unresolved items, irreversible side effects,
|
||||
critical values, recent recovery state, or the previous complete reasoning and expectation.
|
||||
|
||||
## 12. Prompt and Cache Layout
|
||||
|
||||
Keep the stable Operator role, safety rules, cognitive protocol, and output
|
||||
contract in `SystemMessage`. Put the previous real AI message, thought signature,
|
||||
action call, and Validator tool result in the conversation tail. Put the current
|
||||
goal, active plan path, budgeted history, current screenshot, and UI elements in
|
||||
the dynamic `HumanMessage`. Keep tool schemas and ordering stable when possible.
|
||||
|
||||
If the provider adapter supports stateful interactions, preserve one to three
|
||||
recent Operator interactions with `previous_interaction_id`. Start a new
|
||||
interaction when the token threshold is reached or the plan changes materially,
|
||||
seeding it with the stable prefix and chunk summaries. Until then, retain the
|
||||
current message-rebuild path.
|
||||
|
||||
## 13. Same-Turn Context Control
|
||||
|
||||
Large ADB, Notes, Explorer, log, or XML results can expand later model calls in
|
||||
the same Operator turn. Store complete results in a trace or artifact and expose
|
||||
a local envelope:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "success",
|
||||
"important_lines": ["..."],
|
||||
"errors": [],
|
||||
"tail": ["..."],
|
||||
"full_result_ref": "trace://...",
|
||||
"truncated": true,
|
||||
"original_chars": 87321
|
||||
}
|
||||
```
|
||||
|
||||
Apply character, line, and estimated-token limits; retain matched lines, errors,
|
||||
and head/tail context; never call another LLM synchronously to summarize tool
|
||||
output. Compress older results at the soft limit and require a decision instead
|
||||
of accumulating more exploration at the hard limit.
|
||||
|
||||
## 14. Configuration
|
||||
|
||||
Add a dedicated Pro memory section instead of reusing Flash `step_summarizer`:
|
||||
|
||||
```jsonc
|
||||
"agent": {
|
||||
"pro": {
|
||||
"memory": {
|
||||
"enabled": true,
|
||||
"preserve_previous_operator_turn": true,
|
||||
"max_hot_turns": 2,
|
||||
"operator_target_input_tokens": 12000,
|
||||
"operator_hard_input_tokens": 16000,
|
||||
"same_turn_soft_tokens": 20000,
|
||||
"same_turn_hard_tokens": 28000,
|
||||
"visual_step_summary": {
|
||||
"enabled": true,
|
||||
"model": "gemini-3.7-flash",
|
||||
"thinking_level": "low",
|
||||
"include_pre_image": true,
|
||||
"include_post_image_when_available": true,
|
||||
"include_ui_diff": true,
|
||||
"max_concurrency": 1,
|
||||
"operator_never_waits": true
|
||||
},
|
||||
"history_chunks": {
|
||||
"enabled": true,
|
||||
"max_steps": 12,
|
||||
"target_source_tokens": 1800,
|
||||
"model": "gemini-3.7-flash"
|
||||
},
|
||||
"recall": {
|
||||
"enabled": true,
|
||||
"max_results": 5,
|
||||
"max_text_tokens": 2000,
|
||||
"max_image_steps": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For a speed-first single-quota deployment, override only the visual step model
|
||||
with `gemini-3.5-flash-lite`.
|
||||
|
||||
## 15. Implementation Scope
|
||||
|
||||
### 15.1 Operator
|
||||
|
||||
Update Operator, prompt builders, and configuration to add a token-budget memory
|
||||
assembler, preserve recent real messages and Validator results, avoid duplicate
|
||||
history rendering, move dynamic state out of the stable system prompt, remove
|
||||
`<short_term_memory>`, add `recall_history`, inject local screen-similarity hints,
|
||||
and enforce same-turn output limits.
|
||||
|
||||
### 15.2 Summarizer
|
||||
|
||||
Load decision and optional post-action images by stable step ID, include available
|
||||
Validator and UI-diff data, send multimodal messages, preserve the neutral
|
||||
one-paragraph contract, atomically update versioned evidence metadata, retain old
|
||||
summaries on failure, enforce concurrency and timeout limits, and never block Operator.
|
||||
|
||||
### 15.3 Graph and State
|
||||
|
||||
Store the previous Operator `AIMessage`, thought signature, and Validator function
|
||||
response. Remove the `short_term_memory` field while safely ignoring it in older
|
||||
tasks. Read ready memory snapshots at the Perception/Operator boundary and retain
|
||||
the current asynchronous Summarizer semantics.
|
||||
|
||||
### 15.4 DataEngine
|
||||
|
||||
Add summary status, version, evidence, thought-signature, message-reference, and
|
||||
UI-diff fields; persist history chunks and local recall indexes; support atomic
|
||||
updates by step ID; and preserve every older summary version and source.
|
||||
|
||||
### 15.5 Task Tree
|
||||
|
||||
Separate history visibility from rendering detail, render visual summaries and
|
||||
stable chunks, use current subgoal only for relevance, avoid repeating the latest
|
||||
step when it is already present in the real message tail, and retain exact actions
|
||||
and validation results.
|
||||
|
||||
### 15.6 Memory Module
|
||||
|
||||
Suggested files:
|
||||
|
||||
```text
|
||||
artemis/memory/pro_memory.py
|
||||
artemis/memory/history_chunker.py
|
||||
artemis/memory/history_retriever.py
|
||||
artemis/memory/token_budget.py
|
||||
artemis/tools/history_recall.py
|
||||
```
|
||||
|
||||
Responsibilities include memory snapshots, value-based selection, chunk closure
|
||||
and background summarization, FTS and screen-signature retrieval, the recall tool,
|
||||
and provider-independent token estimation.
|
||||
|
||||
### 15.7 Configuration
|
||||
|
||||
Every capability must have an independent switch, default to current Pro fallback
|
||||
behavior, tolerate missing configuration and older tasks, and support both the
|
||||
3.7 quality mode and Flash-Lite isolation mode.
|
||||
|
||||
## 16. Delivery Phases
|
||||
|
||||
### Phase 0: Baseline and Observability
|
||||
|
||||
Freeze a representative long-task suite. Record input tokens, TTFT, model latency,
|
||||
full-turn latency, summary-ready latency, failures, retries, history size,
|
||||
duplicate-action rate, and Failure Analyzer activation without changing prompts.
|
||||
|
||||
### Phase 1: Protect and Improve Hot History
|
||||
|
||||
Preserve complete recent reasoning, remove the duplicate short-term-memory path,
|
||||
avoid duplicate rendering, add the total token threshold, and locally truncate
|
||||
large tool envelopes. Prove that speed does not regress first.
|
||||
|
||||
### Phase 2: Pro Visual Step Summaries
|
||||
|
||||
Load actual decision and optional post-action images by stable step ID while
|
||||
preserving the neutral output contract and text fallback. Compare 3.7 with images,
|
||||
Flash-Lite with images, and current 3.7 text-only behavior. Confirm no Operator wait.
|
||||
|
||||
### Phase 3: Stable History Chunks
|
||||
|
||||
Add milestone-independent rolling chunks, generate them from source summaries,
|
||||
treat milestones as temporary working-set labels, and add source references and versions.
|
||||
|
||||
### Phase 4: Operator Cold-History Recall
|
||||
|
||||
Reuse step, screenshot, and UI search capabilities behind one bounded
|
||||
`recall_history` tool, add local screen-similarity hints, and limit text and images.
|
||||
|
||||
### Phase 5: Real Multi-Turn Messages and Cache Optimization
|
||||
|
||||
Preserve real AI messages, tool calls, and thought signatures; separate stable
|
||||
system and dynamic human content; evaluate stateful provider APIs; and safely
|
||||
restart interactions at token thresholds. Begin only after the first four phases stabilize.
|
||||
|
||||
## 17. Test Plan
|
||||
|
||||
Unit tests must cover hot-history preservation, removal of short-term-memory
|
||||
parsing, action/Validator pairing, duplicate suppression, two-turn recovery,
|
||||
image lookup by step ID, optional post-image behavior, neutral-output validation,
|
||||
Validator precedence, concurrency, timeout, zero-wait behavior, chunk thresholds,
|
||||
source traceability, recall bounds, image limits, and hard token protections.
|
||||
|
||||
Integration tests must include tasks longer than 30 steps, repeated similar
|
||||
screens, plan rewrites, cross-app flows, dialogs, Toasts, WebViews, Canvas,
|
||||
Failure Analyzer recovery, repeated ADB exploration, long lists, monitoring loops,
|
||||
summary timeout or quota exhaustion, and exact-value recall from older screenshots.
|
||||
|
||||
Run blinded evaluations on real traces comparing 3.7 text-only, Flash-Lite with
|
||||
available images, and 3.7 with available images. Reviewers score fidelity to
|
||||
intent and explicit facts, unsupported visual claims, unauthorized outcome
|
||||
judgments, anomaly retention, next-step usefulness, generation time, and ready latency.
|
||||
|
||||
## 18. Acceptance Criteria
|
||||
|
||||
Under the same device, task, model, and comparable service load:
|
||||
|
||||
- first-call input P50 increases by no more than 3% and P95 by no more than 5%;
|
||||
- Operator model-call and full-turn P50/P90 latency regress by no more than 5%;
|
||||
- observed step-interval P50/P90 regress by no more than 5%;
|
||||
- Operator waits for summaries exactly zero times;
|
||||
- normal Operator requests contain zero historical images; and
|
||||
- images returned by `recall_history(include_images=True)` count only as explicit recall.
|
||||
|
||||
Run at least 30 paired tasks and report confidence intervals. Quality must not
|
||||
regress for previous-strategy recall, overall task success, or Failure Analyzer
|
||||
recovery. Visual-fact accuracy, important-fact retention across plan edits,
|
||||
repeat-action rate, history-induced rollback rate, and top-five recall hit rate
|
||||
must improve. Summary/Validator conflict should approach zero.
|
||||
|
||||
## 19. Failure Modes and Fallbacks
|
||||
|
||||
- If a visual summary is pending, use complete previous reasoning, exact action, and Validator result.
|
||||
- If visual summarization fails, set `visual_failed`, preserve old summaries, and do not block for retries.
|
||||
- If screenshot evidence is incomplete, include only actual evidence and never infer the missing post-action state.
|
||||
- If chunk generation fails, keep source summaries and retry the pending chunk during an idle window.
|
||||
- If recall output is too large, return high-scoring excerpts and step/artifact references.
|
||||
- If background work competes with Operator, switch to Lite, pause new summaries, or use a separate quota.
|
||||
- If stateful interactions are incompatible, return to prompt reconstruction while preserving raw/native thinking.
|
||||
|
||||
## 20. Rollout Flags
|
||||
|
||||
```text
|
||||
pro_memory_enabled
|
||||
preserve_operator_message_tail
|
||||
visual_step_summary
|
||||
history_chunks
|
||||
operator_history_recall
|
||||
stateful_operator_interactions
|
||||
```
|
||||
|
||||
Recommended order: enable metrics and token thresholds; run visual summaries in
|
||||
shadow mode; use them after blinded validation; enable stable chunks; enable
|
||||
`recall_history`; then evaluate real multi-turn interactions. Disabling any flag
|
||||
must restore current Pro behavior.
|
||||
|
||||
## 21. Improvements Over the Current System
|
||||
|
||||
| Capability | Current Pro | Redesigned Pro |
|
||||
|---|---|---|
|
||||
| Previous reasoning | Re-rendered as detailed history | Preserved and ready for a real message/thought-signature chain |
|
||||
| Short-term strategy | Optional `<short_term_memory>` | Carried by the complete previous message, Validator, plan, and Notes |
|
||||
| Step summary | 3.7 reads text only | Neutral 3.7 summary reads actual available images, UI, and Validator data |
|
||||
| Exact action | Retained beside summary | Still retained and never replaced by visual summary |
|
||||
| History pruning | Current subgoal plus latest three steps | Working set plus stable chunks and relevance budget |
|
||||
| Plan rewrite | May change old-step visibility | Stable step/chunk references survive label changes |
|
||||
| Old screenshots | No direct Operator recall | One actual step can be recalled on demand |
|
||||
| Cold history | Primarily for HistoryAnalyzer/Outputter | Locally searchable and bounded for Operator |
|
||||
| Context control | Mainly a step window | Total token budget, value density, and same-turn limits |
|
||||
| Cache use | Highly dynamic system prompt | Stable system prefix separated from runtime state |
|
||||
| Background failure | Summary may be delayed | Explicit state machine and zero-wait fallback |
|
||||
|
||||
The objective is not to display more history. It is to preserve recent reasoning,
|
||||
give older summaries real visual context, decouple long-term identity from plan
|
||||
milestones, make compressed records precisely recallable, and keep normal input
|
||||
size and critical-path model-call count unchanged.
|
||||
|
||||
## 22. Recommended Defaults
|
||||
|
||||
```text
|
||||
Normal Operator input: 10K-12K tokens
|
||||
First-call soft/hard limits: 12K / 16K
|
||||
Same-turn soft/hard limits: 20K / 28K
|
||||
|
||||
Complete hot turns: 1 normally, up to 2 for failure/uncertainty
|
||||
Current-work summaries: budget-controlled, not a fixed count
|
||||
History chunk: 8-12 steps or about 1.8K source-summary tokens
|
||||
|
||||
Historical images in normal input: 0
|
||||
Summary input images: 0-1 decision, 0-1 independent post-action
|
||||
Explicit recall images: actual images from at most one step
|
||||
|
||||
Preferred step-summary model: Gemini 3.7 Flash + images
|
||||
Single-quota speed mode: Gemini 3.5 Flash-Lite + images
|
||||
History-chunk model: Gemini 3.7 Flash
|
||||
Background summary concurrency: 1
|
||||
|
||||
Additional critical-path calls: 0
|
||||
Critical-path summary waits: 0
|
||||
```
|
||||
|
||||
## 23. Final Decision
|
||||
|
||||
Do not remove previous reasoning and replace detailed history with an image-free
|
||||
capsule. Preserve complete previous-turn reasoning, upgrade older steps to
|
||||
visually grounded neutral summaries, use milestone-independent rolling chunks,
|
||||
allow bounded cold-history recall, and enforce a fixed token budget.
|
||||
|
||||
The first, highest-value implementation slice is:
|
||||
|
||||
1. preserve complete previous reasoning and remove the duplicate short-term-memory path;
|
||||
2. add the decision screenshot and optional actual post-action screenshot to the current neutral Pro Summarizer;
|
||||
3. bind summaries with stable step IDs; and
|
||||
4. establish speed baselines and blinded visual-summary evaluations.
|
||||
|
||||
Do not expand the normal history window or remove the current detailed-history
|
||||
fallback until all four items have passed validation.
|
||||
@@ -0,0 +1,606 @@
|
||||
# ARTEMIS Video Analysis Memory and Reliability Design
|
||||
|
||||
Status: implemented (2026-08-25)
|
||||
Audience: ARTEMIS runtime, perception, tracing, and mobile automation maintainers
|
||||
|
||||
## 1. Executive summary
|
||||
|
||||
The current video analyzer treats its blackboard as state owned by one
|
||||
`VideoAnalyzer` object. Tool invocations create new analyzer objects, so the
|
||||
state is lost between calls. A failed parent analysis also discards useful
|
||||
work completed by child analyzers. Overlap detection is advisory and does not
|
||||
prevent the same interval from being sent to a video model again.
|
||||
|
||||
This design moves video memory out of the agent and into a video-scoped,
|
||||
persistent blackboard. The blackboard is the source of truth for:
|
||||
|
||||
- which intervals have been analyzed for a normalized query and modality;
|
||||
- observations and evidence extracted from those intervals;
|
||||
- successful, running, and failed segment attempts;
|
||||
- resumable partial results across tool calls;
|
||||
- concurrency leases that prevent duplicate model requests.
|
||||
|
||||
The analyzer becomes a coordinator over this state. It calculates the set
|
||||
difference between a requested interval and completed coverage, analyzes only
|
||||
the gaps, commits every successful chunk immediately, and returns partial
|
||||
results without erasing prior work.
|
||||
|
||||
The first implementation deliberately uses SQLite and the existing trace
|
||||
directory. Vector search, semantic graphs, provider-managed context caches,
|
||||
and entity re-identification are future layers, not prerequisites for fixing
|
||||
correctness and reliability.
|
||||
|
||||
## 2. Goals
|
||||
|
||||
### 2.1 Functional goals
|
||||
|
||||
1. Share analyzed intervals and results across every `VideoAnalyzer` created
|
||||
for the same recording.
|
||||
2. Avoid repeating an identical successful `(video, interval, query,
|
||||
modality)` analysis.
|
||||
3. Persist each successful chunk before sibling chunks or the parent agent
|
||||
finish.
|
||||
4. Preserve successful chunks when some chunks fail.
|
||||
5. Expose previous observations to both native Gemini and universal-model
|
||||
coordinators.
|
||||
6. Keep evidence tied to recording-relative timestamps and durable files.
|
||||
7. Prevent concurrent duplicate requests with an expiring lease.
|
||||
8. Maintain existing public tool signatures.
|
||||
|
||||
### 2.2 Reliability goals
|
||||
|
||||
- A process or model failure may lose only the currently running chunk.
|
||||
- An expired running lease is reclaimable.
|
||||
- Retrying a request analyzes only uncovered or failed chunks.
|
||||
- A partial result explicitly identifies failed intervals.
|
||||
- SQLite writes are idempotent and safe under multiple analyzer instances.
|
||||
- Evidence used by persistent memory is not deleted by temporary-file cleanup.
|
||||
|
||||
### 2.3 Non-goals for phase one
|
||||
|
||||
- Semantic similarity reuse between materially different queries.
|
||||
- Cross-recording memory or user identity memory.
|
||||
- Replacing Gemini or the configured universal VLM.
|
||||
- Hosting an open-weight video model or accessing model KV caches.
|
||||
- Full WorldMM semantic graphs, HippoRAG, VLM2Vec, or ReflectWorld ReID.
|
||||
- Permanent retention of uploaded provider files.
|
||||
|
||||
## 3. Current failure modes
|
||||
|
||||
### 3.1 Agent-owned state
|
||||
|
||||
`VideoAnalyzer.__init__` initializes an empty list. `video_tool.py` creates a
|
||||
new analyzer for every tool call. Consequently, a later invocation cannot see
|
||||
the earlier invocation's ledger.
|
||||
|
||||
### 3.2 Advisory overlap handling
|
||||
|
||||
The analyzer calculates high-overlap warnings but continues with extraction,
|
||||
compression, upload, and inference. A warning in a prompt is not an
|
||||
idempotency mechanism.
|
||||
|
||||
### 3.3 Commit-at-the-end behavior
|
||||
|
||||
Some event observations are appended during child analysis, but they are only
|
||||
in memory. If the parent fails or cleanup runs, the trace can contain expensive
|
||||
model activity with no reusable result.
|
||||
|
||||
### 3.4 Mutable recording ambiguity
|
||||
|
||||
An open-ended interval is currently cached as `(start, None)`. The meaning of
|
||||
`None` changes while recording continues, so this cannot be a stable cache
|
||||
identity.
|
||||
|
||||
### 3.5 Disposable evidence
|
||||
|
||||
Extracted screenshots live in temporary directories registered for cleanup.
|
||||
The ledger may therefore reference files that no longer exist.
|
||||
|
||||
## 4. Design principles
|
||||
|
||||
### 4.1 Memory before reasoning
|
||||
|
||||
Perception results are written before the coordinator generates its final
|
||||
answer. Answer generation reads stored evidence; it is not the owner of that
|
||||
evidence.
|
||||
|
||||
### 4.2 Evidence-backed records
|
||||
|
||||
Every observation carries an interval, query, confidence, modality, model
|
||||
metadata where available, and an optional durable screenshot. Summaries never
|
||||
stand alone without temporal provenance.
|
||||
|
||||
### 4.3 Objective observation versus query result
|
||||
|
||||
The design has two logical layers:
|
||||
|
||||
- **Observations** describe visible or audible events and can help later
|
||||
queries.
|
||||
- **Segment results** answer one normalized query over one complete interval
|
||||
and define exact reusable coverage.
|
||||
|
||||
An observation about a button is useful context for many questions, but a
|
||||
segment result for "did login succeed" must not automatically satisfy a
|
||||
different query such as "how long did the spinner remain visible".
|
||||
|
||||
### 4.4 Immutable interval identity
|
||||
|
||||
An analysis segment always has numeric start and end timestamps. Open-ended
|
||||
requests are resolved to the safe available end before they are claimed or
|
||||
committed.
|
||||
|
||||
### 4.5 At-least-once execution, idempotent commit
|
||||
|
||||
Provider requests can be retried. Database constraints and deterministic keys
|
||||
make commits idempotent. Exact-once provider execution is not promised, but
|
||||
duplicate execution is strongly suppressed by leases.
|
||||
|
||||
## 5. Architecture
|
||||
|
||||
```text
|
||||
video_analyzer tool
|
||||
|
|
||||
v
|
||||
VideoAnalyzer coordinator ---------> persistent blackboard ledger
|
||||
| | |
|
||||
| | +-- observations/evidence
|
||||
| +-- segment coverage and attempts
|
||||
|
|
||||
+-- resolve stable numeric interval
|
||||
+-- subtract completed coverage
|
||||
+-- claim uncovered chunks with leases
|
||||
+-- run child analysis
|
||||
+-- commit each successful chunk immediately
|
||||
+-- mark failed chunks retryable
|
||||
+-- synthesize cached + new + partial results
|
||||
```
|
||||
|
||||
The blackboard is acquired from `ArtemisContext`, not constructed as private
|
||||
agent state. Multiple analyzers in the same context share the object, while
|
||||
SQLite makes the data available to later instances and process restarts.
|
||||
|
||||
## 6. Recording identity
|
||||
|
||||
The preferred blackboard key is the active `RecordingSession.video_id`:
|
||||
|
||||
```text
|
||||
video:<uuid>
|
||||
```
|
||||
|
||||
If no active recording is available, the fallback is the DataEngine session:
|
||||
|
||||
```text
|
||||
session:<uuid>
|
||||
```
|
||||
|
||||
The context retains the resolved blackboard instance so analysis after a
|
||||
recording stops continues to use the same key. Extracted recording artifacts
|
||||
also carry an explicit source revision:
|
||||
|
||||
```text
|
||||
<video-uuid>:<generation>:<sealed-end>
|
||||
```
|
||||
|
||||
A generation advances whenever the recorder rolls a physical segment. The
|
||||
recorder tracks the sealed numeric prefix, and every `end=None` analysis is
|
||||
resolved to the current safe numeric end before it is claimed.
|
||||
|
||||
## 7. Persistence model
|
||||
|
||||
### 7.1 `video_analysis_segments`
|
||||
|
||||
One row represents a complete query-specific analysis of a numeric interval.
|
||||
|
||||
| Column | Meaning |
|
||||
| --- | --- |
|
||||
| `board_key` | Recording/session namespace |
|
||||
| `query_key` | SHA-256 of the normalized query |
|
||||
| `query_text` | Original query for debugging and retrieval |
|
||||
| `modality` | `video` or `audio` |
|
||||
| `start_ms`, `end_ms` | Stable integer interval |
|
||||
| `status` | `running`, `succeeded`, `retryable_failed`, `permanent_failed` |
|
||||
| `lease_owner`, `lease_expires_at` | Duplicate-work suppression |
|
||||
| `attempt_count` | Number of acquired attempts |
|
||||
| `summary`, `analysis` | Reusable completed result |
|
||||
| `error` | Last structured failure text |
|
||||
| `created_at`, `updated_at` | Audit timestamps |
|
||||
|
||||
The unique key is:
|
||||
|
||||
```text
|
||||
(board_key, query_key, modality, start_ms, end_ms)
|
||||
```
|
||||
|
||||
### 7.2 `video_analysis_observations`
|
||||
|
||||
One row represents a timestamped event or fact.
|
||||
|
||||
| Column | Meaning |
|
||||
| --- | --- |
|
||||
| `observation_id` | Deterministic content fingerprint |
|
||||
| `board_key` | Recording/session namespace |
|
||||
| `start_ms`, `end_ms` | Event interval |
|
||||
| `query_text` | Query that caused extraction |
|
||||
| `summary` | Concise observation |
|
||||
| `confidence` | Model-reported confidence |
|
||||
| `modality` | `video` or `audio` |
|
||||
| `screenshot_path` | Durable evidence path |
|
||||
| `extra_json` | Forward-compatible metadata |
|
||||
| `created_at` | Audit timestamp |
|
||||
|
||||
The deterministic observation ID makes repeated commits safe.
|
||||
|
||||
### 7.3 Evidence layout
|
||||
|
||||
```text
|
||||
<trace-session>/video_blackboard/evidence/<content-id>.jpg
|
||||
```
|
||||
|
||||
Temporary screenshots are copied into this directory before their source is
|
||||
registered for deletion. Stored records reference the durable copy.
|
||||
|
||||
## 8. Segment state machine
|
||||
|
||||
```text
|
||||
lease expires
|
||||
running ------------------------------+
|
||||
| |
|
||||
| success v
|
||||
+----------> succeeded retryable_failed
|
||||
| |
|
||||
| retryable error | claim again
|
||||
+----------> retryable_failed ------+
|
||||
|
|
||||
| non-retryable/budget exhausted
|
||||
+----------> permanent_failed
|
||||
```
|
||||
|
||||
Claim behavior is transactional:
|
||||
|
||||
1. A missing row is inserted as `running`.
|
||||
2. `succeeded` returns its cached result without acquiring work.
|
||||
3. `running` with a live lease returns `in_progress`.
|
||||
4. An expired lease or retryable failure is atomically reclaimed and increments
|
||||
`attempt_count`.
|
||||
|
||||
Exhausted failures store a normalized category and category-derived
|
||||
retryability. Permanent failure is reserved for validated bad input,
|
||||
authentication/policy errors, cancellation, or explicit decisions.
|
||||
|
||||
## 9. Coverage and planning
|
||||
|
||||
Coverage is query- and modality-specific. Given requested interval `R` and
|
||||
the union of successful intervals `S`, the work planner calculates:
|
||||
|
||||
```text
|
||||
missing = R - union(S)
|
||||
```
|
||||
|
||||
Each missing interval is split into bounded chunks. Completed chunks are
|
||||
included in the returned result; only gaps are sent to the provider.
|
||||
|
||||
Interval math uses integer milliseconds internally to prevent floating-point
|
||||
key drift. API-facing timestamps remain seconds.
|
||||
|
||||
Observations from other queries are still placed in the ledger as context, but
|
||||
they do not count as completed coverage for the new query.
|
||||
|
||||
## 10. Execution flow
|
||||
|
||||
### 10.1 Tool invocation
|
||||
|
||||
1. Acquire the context-scoped blackboard.
|
||||
2. Refresh the local ledger from persistent observations.
|
||||
3. Give the coordinator the full ledger snapshot.
|
||||
4. Coordinator requests segment analysis.
|
||||
|
||||
### 10.2 Child analysis
|
||||
|
||||
1. Resolve a numeric interval from recording metadata.
|
||||
2. Calculate missing intervals.
|
||||
3. For every chunk, acquire a lease.
|
||||
4. Reuse `succeeded`; skip live `running`; execute only `claimed`.
|
||||
5. Extract/compress/upload and invoke the configured VLM.
|
||||
6. Persist timeline observations as soon as they are parsed.
|
||||
7. Persist the complete segment result before returning to the parent.
|
||||
8. On final failure, mark that exact chunk failed and keep all sibling results.
|
||||
|
||||
### 10.3 Parent synthesis
|
||||
|
||||
The parent receives cached and newly completed results in the same format. If
|
||||
some chunks failed, it receives an explicit partial-result marker and failed
|
||||
intervals. It may answer with qualified confidence or request a targeted retry.
|
||||
|
||||
## 11. Retry and degradation policy
|
||||
|
||||
The desired production order is:
|
||||
|
||||
1. retry transient upload/poll/generation failures with exponential backoff
|
||||
and jitter;
|
||||
2. retry the exact chunk without expanding its identity;
|
||||
3. split a repeatedly failing chunk into smaller children;
|
||||
4. use the configured fallback video model;
|
||||
5. fall back to high-frequency keyframes, OCR, and UI hierarchy around known
|
||||
action timestamps;
|
||||
6. return partial evidence with machine-readable failed intervals.
|
||||
|
||||
The runtime implements this order with structured failure classes, bounded
|
||||
bisection, configured fallback models, and a primary-model circuit breaker.
|
||||
|
||||
## 12. Provider and sampling strategy
|
||||
|
||||
Provider context caching is an optimization, not the system of record. ARTEMIS
|
||||
must remain correct if a provider cache misses or expires.
|
||||
|
||||
Default 1 FPS video sampling is supplemented by exact, dense seeks around
|
||||
DataEngine action timestamps. Quiet intervals remain sparse while taps,
|
||||
transitions, toasts, and spinners receive a higher sampling budget.
|
||||
|
||||
## 13. Concurrency
|
||||
|
||||
- SQLite uses WAL mode and short `BEGIN IMMEDIATE` claim transactions.
|
||||
- The unique segment key prevents duplicate rows.
|
||||
- A UUID lease owner identifies one execution attempt.
|
||||
- Leases expire so crashes do not permanently block work.
|
||||
- Existing API and transcode semaphores still bound local concurrency.
|
||||
- Conflict cleaning affects an analyzer's presentation ledger; it never
|
||||
deletes persistent observations written by another analyzer.
|
||||
|
||||
## 14. Cleanup and retention
|
||||
|
||||
Three artifact classes have different policies:
|
||||
|
||||
1. Trimmed/compressed chunks: temporary, deleted after the call.
|
||||
2. Provider-uploaded files: temporary, deleted best-effort after inference.
|
||||
3. Blackboard evidence: durable for the trace/session retention period.
|
||||
|
||||
Deleting a trace/session removes blackboard rows through DataEngine and
|
||||
evidence through the session directory lifecycle.
|
||||
|
||||
## 15. Observability
|
||||
|
||||
Required counters and trace fields:
|
||||
|
||||
- requested duration;
|
||||
- cached duration;
|
||||
- newly analyzed duration;
|
||||
- number of claimed, cached, in-progress, succeeded, and failed chunks;
|
||||
- provider attempts and error class;
|
||||
- blackboard key and query key;
|
||||
- partial versus complete outcome;
|
||||
- evidence paths attached to observations.
|
||||
|
||||
No prompt or trace status should report success merely because an error string
|
||||
was returned as normal text.
|
||||
|
||||
## 16. Compatibility and migration
|
||||
|
||||
- Public `video_analyzer` and `video_analyzer_pure` arguments do not change.
|
||||
- Existing in-memory ledger dictionaries remain the presentation format.
|
||||
- When DataEngine persistence is unavailable, the blackboard falls back to a
|
||||
context-scoped in-memory store. This preserves tests and embedded usages but
|
||||
does not promise process-restart recovery.
|
||||
- Databases are migrated with `CREATE TABLE IF NOT EXISTS`; no destructive
|
||||
migration is required.
|
||||
- Existing traces without blackboard rows continue to work and start empty.
|
||||
|
||||
## 17. Security and privacy
|
||||
|
||||
- Evidence remains local under the trace directory.
|
||||
- Provider files are deleted best-effort after requests.
|
||||
- Blackboard rows must not store credentials or entire model prompts.
|
||||
- Retention follows trace retention.
|
||||
- Future cross-session or identity memory requires a separate consent and
|
||||
privacy design; it is outside this scope.
|
||||
|
||||
## 18. Testing strategy
|
||||
|
||||
### 18.1 Unit tests
|
||||
|
||||
- persistence across two blackboard instances using the same SQLite file;
|
||||
- deterministic observation deduplication;
|
||||
- interval union and missing-range calculation;
|
||||
- exact successful segment reuse;
|
||||
- live lease suppression and expired lease reclamation;
|
||||
- failed chunk remains retryable;
|
||||
- evidence is copied to durable storage;
|
||||
- two `VideoAnalyzer` instances share prior ledger state;
|
||||
- repeated `exec_spawn_sub_agent` does not invoke child inference again;
|
||||
- mixed success/failure returns partial output and preserves success.
|
||||
|
||||
### 18.2 Integration tests
|
||||
|
||||
- analyze overlapping intervals against a fixture recording;
|
||||
- interrupt after one successful chunk and resume;
|
||||
- concurrent identical requests result in one provider call;
|
||||
- active recording resolves open-ended requests to numeric endpoints;
|
||||
- evidence remains readable after analyzer cleanup.
|
||||
|
||||
### 18.3 Device tests
|
||||
|
||||
Per ARTEMIS mobile testing rules, end-to-end mobile flows must be explored and
|
||||
verified on a connected device before production test scripts are finalized.
|
||||
The refactor's deterministic unit tests do not require a device; live timing,
|
||||
fast UI transitions, and recording alignment do.
|
||||
|
||||
## 19. Rollout
|
||||
|
||||
1. **Complete:** SQLite blackboard, sharing, observation persistence, exact
|
||||
coverage reuse, leases, and tests.
|
||||
2. **Complete:** failure classification, jitter, bounded bisection, fallback,
|
||||
and circuit breaking.
|
||||
3. **Complete:** recording generations, sealed-prefix metadata, and
|
||||
generation-scoped caches.
|
||||
4. **Complete:** action-aware sampling and real audio delivery.
|
||||
5. **Deferred optimization:** provider file/context caching, gated by an
|
||||
explicit retention policy.
|
||||
6. **Future accuracy layer:** semantic/event hierarchical retrieval after
|
||||
production traces provide a representative evaluation corpus.
|
||||
|
||||
## 20. Acceptance criteria
|
||||
|
||||
- Two analyzer instances in one task see the same ledger.
|
||||
- Repeating an identical successful interval/query causes zero new child VLM
|
||||
calls.
|
||||
- If one of several chunks fails, successful chunks are available to the next
|
||||
invocation and are not rerun.
|
||||
- A crashed `running` chunk becomes reclaimable after lease expiry.
|
||||
- Persistent evidence survives normal analyzer temporary-file cleanup.
|
||||
- Existing video-analyzer unit tests and new blackboard tests pass.
|
||||
|
||||
Additional reliability criteria:
|
||||
|
||||
- timeout/media failures recover through bounded recursive subdivision;
|
||||
- provider-wide failures do not trigger request fan-out;
|
||||
- primary-model failures select the configured `utils.video_analyzer` fallback;
|
||||
- the primary circuit opens after a configured consecutive-failure threshold;
|
||||
- audio analysis is independently resumable and reusable;
|
||||
- recording generation changes invalidate stale trim-cache identities;
|
||||
- frames are sampled densely around recorded mobile actions;
|
||||
- traces include blackboard status, attempts, failure categories, successful
|
||||
duration, observation count, and circuit state.
|
||||
|
||||
## 21. Implemented production architecture
|
||||
|
||||
The completed refactor extends the phase-one blackboard with the following
|
||||
production controls:
|
||||
|
||||
1. Failures are normalized as rate limit, provider outage, timeout,
|
||||
connection, authentication, bad request, media processing, cancellation,
|
||||
or unknown. The class determines retryability, bisection, and fallback.
|
||||
2. Timeout, media, oversized, and unknown failures can recursively bisect a
|
||||
chunk down to configured size/depth bounds. Provider outages, connections,
|
||||
and rate limits do not bisect because fan-out would amplify load.
|
||||
3. Coordinator, video child, and audio child use the configured utility-model
|
||||
fallback. A context-scoped circuit breaker bypasses an unhealthy primary.
|
||||
4. Recording results expose `video_id`, `generation`, `sealed_until`, and
|
||||
`source_revision`. Trim caches are scoped by video UUID and generation;
|
||||
open-ended requests are resolved to numeric safe ends.
|
||||
5. Universal video analysis receives sparse baseline frames, dense frames
|
||||
around DataEngine mobile-action timestamps, and the real audio track.
|
||||
Audio-only analysis sends a standard base64 audio content block. Models
|
||||
that reject audio degrade only that video chunk to visual-only analysis.
|
||||
6. Audio uses the same lease, exact reuse, completion, and failure state
|
||||
machine as video.
|
||||
7. DataEngine session deletion removes blackboard rows by session/video key;
|
||||
evidence is deleted with the trace directory. Universal-path temporary
|
||||
artifacts are cleaned on early exit.
|
||||
|
||||
The persistence schema adds `error_category`, `model_name`, and
|
||||
`source_generation` to every segment attempt. Migrations are additive and
|
||||
safe for existing SQLite traces.
|
||||
|
||||
## 22. State-of-the-art review and adoption decision
|
||||
|
||||
The relevant frontier is not “send the full video to a bigger model.” Recent
|
||||
systems converge on four architectural ideas:
|
||||
|
||||
1. **Agentic coarse-to-fine localization.** [VideoAgent](https://arxiv.org/abs/2403.10517)
|
||||
iteratively retrieves a small number of frames; [VideoTree](https://arxiv.org/abs/2405.19209)
|
||||
builds a query-adaptive temporal hierarchy.
|
||||
2. **Persistent hierarchical multimodal memory.** The memory-augmented
|
||||
[VideoAgent](https://arxiv.org/abs/2403.11481) separates memory construction
|
||||
and query-time tool use. [VideoARM](https://arxiv.org/abs/2512.12360) uses an
|
||||
observe-think-act-memorize loop over hierarchical memory.
|
||||
3. **Adaptive redundancy reduction.** [LongVU](https://arxiv.org/abs/2410.17434)
|
||||
removes redundant frames and preserves query-relevant temporal/spatial
|
||||
detail instead of sampling uniformly.
|
||||
4. **Retrieval over very long media.** [VideoRAG](https://arxiv.org/abs/2502.01549)
|
||||
combines graph/text grounding with multimodal retrieval. This is useful for
|
||||
many-hour corpora, but heavier than one mobile-test recording.
|
||||
|
||||
Provider guidance supports the same separation: Gemini's
|
||||
[video-understanding documentation](https://ai.google.dev/gemini-api/docs/video-understanding)
|
||||
describes direct audio/video processing and reusable Files API inputs, while
|
||||
[context caching](https://ai.google.dev/gemini-api/docs/caching) is an
|
||||
optimization for repeated common input. Neither is a durable work ledger.
|
||||
|
||||
### 22.1 Adopted directly
|
||||
|
||||
- VideoARM/VideoAgent's observe → reason → act → memorize loop;
|
||||
- query-scoped coarse-to-fine temporal analysis;
|
||||
- separation of objective observations and answered query spans;
|
||||
- adaptive sampling, with mobile action timestamps as the domain-specific
|
||||
relevance signal;
|
||||
- independent visual/audio tools and evidence-grounded synthesis.
|
||||
|
||||
### 22.2 Not copied wholesale
|
||||
|
||||
The research implementations assume offline benchmark videos, often require
|
||||
GPU feature extractors, and optimize multiple-choice VideoQA. ARTEMIS operates
|
||||
on a growing recorder stream, must survive provider/process failures, prevent
|
||||
duplicate paid calls, and tie evidence to test-relative action time. Importing
|
||||
VideoARM or VideoRAG would add a second orchestration runtime without solving
|
||||
leases, idempotent commit, active-recording identity, retention, or mobile
|
||||
alignment.
|
||||
|
||||
ARTEMIS therefore ports their architectural invariants, not their
|
||||
benchmark-specific code, and supplies the production controls the prototypes
|
||||
do not provide.
|
||||
|
||||
## 23. Runtime configuration
|
||||
|
||||
All controls live under `video_analyzer` and flow through the SDK builder:
|
||||
|
||||
| Setting | Default | Purpose |
|
||||
| --- | ---: | --- |
|
||||
| `chunk_size_seconds` | 60 | Initial coarse chunk size |
|
||||
| `min_chunk_seconds` | 4 | Smallest bisection leaf |
|
||||
| `max_split_depth` | 4 | Bound failure-driven fan-out |
|
||||
| `circuit_breaker_threshold` | 3 | Failures before primary circuit opens |
|
||||
| `circuit_breaker_cooldown_seconds` | 60 | Half-open delay |
|
||||
| `action_window_seconds` | 2 | Dense window on either side of an action |
|
||||
| `dense_action_fps` | 4 | Sampling rate in action windows |
|
||||
| `max_dense_action_frames` | 24 | Dense-frame budget per chunk |
|
||||
| `native_max_retries` | 1 | Native-provider retries before universal fallback |
|
||||
| `model_call_timeout_seconds` | 120 | Hard bound for one model response |
|
||||
|
||||
The configured `llm_config.utils.video_analyzer.fallback` supplies the model
|
||||
fallback. Public video tool signatures remain unchanged.
|
||||
|
||||
## 24. Verification status
|
||||
|
||||
Local verification covers persistence, exact reuse, leases, partial commit,
|
||||
structured failures, circuit breaking, fallback selection, timeout bisection,
|
||||
audio reuse, real audio content blocks, action-aware sampling, exact frame
|
||||
seeking on an MP4 fixture, generation-scoped caches, configuration
|
||||
propagation, and existing analyzer/controller behavior.
|
||||
|
||||
### 24.1 Real-device acceptance (2026-08-25)
|
||||
|
||||
A connected Pixel 10 (`59260DLCR00360`) completed a deterministic Google
|
||||
Calculator recording with independently observed UI ground truth:
|
||||
|
||||
- `12+34=46` at 21.15s;
|
||||
- `7×8=56` at 32.44s;
|
||||
- history opened at 39.01s and closed at 42.12s;
|
||||
- the final screen returned to result `56`.
|
||||
|
||||
Three query-independent video analyses all matched that ground truth. Each
|
||||
query was then replayed twice with the identical interval and query. All six
|
||||
replays returned `CACHED VIDEO ANALYSIS` in roughly 3ms, with no new child
|
||||
analysis. Final blackboard state was three succeeded segments, 127.8 covered
|
||||
query-seconds, and eight timestamped observations. The machine-readable
|
||||
report and retained MP4 are produced by
|
||||
`scripts/real_device_video_acceptance.py`.
|
||||
|
||||
The run also exercised real provider degradation: native Gemini returned a
|
||||
503 and later stalled a stream. The affected query preserved prior successes,
|
||||
retried only its own uncovered work, then completed through the configured
|
||||
universal fallback. Based on that trace, native attempts now default to two
|
||||
total calls with a 120-second response bound before fallback.
|
||||
|
||||
Device exploration also exposed two non-video runtime faults that affect
|
||||
perceived stability and were fixed as part of the gate:
|
||||
|
||||
1. UIAutomator may return current string/dictionary bounds with a stale cached
|
||||
center. Dynamic taps now always derive their center from fresh bounds and
|
||||
use the cached center only when bounds are unavailable.
|
||||
2. A stale desktop IPC port previously blocked every DataEngine event for one
|
||||
second. IPC connects now use a short localhost timeout plus exponential
|
||||
backoff, and detached agent initialization has a 30-second hard bound.
|
||||
|
||||
The acceptance result demonstrates 100% accuracy and reuse for this defined
|
||||
fixture and run. It is not a mathematical guarantee for every future app,
|
||||
video, device, or provider outage; production SLOs require a larger repeated
|
||||
corpus and ongoing monitoring.
|
||||
Reference in new issue
Block a user