# ARTEMIS Video Analysis Memory and Reliability Design Status: implemented (2026-08-25) Audience: ARTEMIS runtime, perception, tracing, and mobile automation maintainers ## 1. Executive summary The current video analyzer treats its blackboard as state owned by one `VideoAnalyzer` object. Tool invocations create new analyzer objects, so the state is lost between calls. A failed parent analysis also discards useful work completed by child analyzers. Overlap detection is advisory and does not prevent the same interval from being sent to a video model again. This design moves video memory out of the agent and into a video-scoped, persistent blackboard. The blackboard is the source of truth for: - which intervals have been analyzed for a normalized query and modality; - observations and evidence extracted from those intervals; - successful, running, and failed segment attempts; - resumable partial results across tool calls; - concurrency leases that prevent duplicate model requests. The analyzer becomes a coordinator over this state. It calculates the set difference between a requested interval and completed coverage, analyzes only the gaps, commits every successful chunk immediately, and returns partial results without erasing prior work. The first implementation deliberately uses SQLite and the existing trace directory. Vector search, semantic graphs, provider-managed context caches, and entity re-identification are future layers, not prerequisites for fixing correctness and reliability. ## 2. Goals ### 2.1 Functional goals 1. Share analyzed intervals and results across every `VideoAnalyzer` created for the same recording. 2. Avoid repeating an identical successful `(video, interval, query, modality)` analysis. 3. Persist each successful chunk before sibling chunks or the parent agent finish. 4. Preserve successful chunks when some chunks fail. 5. Expose previous observations to both native Gemini and universal-model coordinators. 6. Keep evidence tied to recording-relative timestamps and durable files. 7. Prevent concurrent duplicate requests with an expiring lease. 8. Maintain existing public tool signatures. ### 2.2 Reliability goals - A process or model failure may lose only the currently running chunk. - An expired running lease is reclaimable. - Retrying a request analyzes only uncovered or failed chunks. - A partial result explicitly identifies failed intervals. - SQLite writes are idempotent and safe under multiple analyzer instances. - Evidence used by persistent memory is not deleted by temporary-file cleanup. ### 2.3 Non-goals for phase one - Semantic similarity reuse between materially different queries. - Cross-recording memory or user identity memory. - Replacing Gemini or the configured universal VLM. - Hosting an open-weight video model or accessing model KV caches. - Full WorldMM semantic graphs, HippoRAG, VLM2Vec, or ReflectWorld ReID. - Permanent retention of uploaded provider files. ## 3. Current failure modes ### 3.1 Agent-owned state `VideoAnalyzer.__init__` initializes an empty list. `video_tool.py` creates a new analyzer for every tool call. Consequently, a later invocation cannot see the earlier invocation's ledger. ### 3.2 Advisory overlap handling The analyzer calculates high-overlap warnings but continues with extraction, compression, upload, and inference. A warning in a prompt is not an idempotency mechanism. ### 3.3 Commit-at-the-end behavior Some event observations are appended during child analysis, but they are only in memory. If the parent fails or cleanup runs, the trace can contain expensive model activity with no reusable result. ### 3.4 Mutable recording ambiguity An open-ended interval is currently cached as `(start, None)`. The meaning of `None` changes while recording continues, so this cannot be a stable cache identity. ### 3.5 Disposable evidence Extracted screenshots live in temporary directories registered for cleanup. The ledger may therefore reference files that no longer exist. ## 4. Design principles ### 4.1 Memory before reasoning Perception results are written before the coordinator generates its final answer. Answer generation reads stored evidence; it is not the owner of that evidence. ### 4.2 Evidence-backed records Every observation carries an interval, query, confidence, modality, model metadata where available, and an optional durable screenshot. Summaries never stand alone without temporal provenance. ### 4.3 Objective observation versus query result The design has two logical layers: - **Observations** describe visible or audible events and can help later queries. - **Segment results** answer one normalized query over one complete interval and define exact reusable coverage. An observation about a button is useful context for many questions, but a segment result for "did login succeed" must not automatically satisfy a different query such as "how long did the spinner remain visible". ### 4.4 Immutable interval identity An analysis segment always has numeric start and end timestamps. Open-ended requests are resolved to the safe available end before they are claimed or committed. ### 4.5 At-least-once execution, idempotent commit Provider requests can be retried. Database constraints and deterministic keys make commits idempotent. Exact-once provider execution is not promised, but duplicate execution is strongly suppressed by leases. ## 5. Architecture ```text video_analyzer tool | v VideoAnalyzer coordinator ---------> persistent blackboard ledger | | | | | +-- observations/evidence | +-- segment coverage and attempts | +-- resolve stable numeric interval +-- subtract completed coverage +-- claim uncovered chunks with leases +-- run child analysis +-- commit each successful chunk immediately +-- mark failed chunks retryable +-- synthesize cached + new + partial results ``` The blackboard is acquired from `ArtemisContext`, not constructed as private agent state. Multiple analyzers in the same context share the object, while SQLite makes the data available to later instances and process restarts. ## 6. Recording identity The preferred blackboard key is the active `RecordingSession.video_id`: ```text video: ``` If no active recording is available, the fallback is the DataEngine session: ```text session: ``` The context retains the resolved blackboard instance so analysis after a recording stops continues to use the same key. Extracted recording artifacts also carry an explicit source revision: ```text :: ``` A generation advances whenever the recorder rolls a physical segment. The recorder tracks the sealed numeric prefix, and every `end=None` analysis is resolved to the current safe numeric end before it is claimed. ## 7. Persistence model ### 7.1 `video_analysis_segments` One row represents a complete query-specific analysis of a numeric interval. | Column | Meaning | | --- | --- | | `board_key` | Recording/session namespace | | `query_key` | SHA-256 of the normalized query | | `query_text` | Original query for debugging and retrieval | | `modality` | `video` or `audio` | | `start_ms`, `end_ms` | Stable integer interval | | `status` | `running`, `succeeded`, `retryable_failed`, `permanent_failed` | | `lease_owner`, `lease_expires_at` | Duplicate-work suppression | | `attempt_count` | Number of acquired attempts | | `summary`, `analysis` | Reusable completed result | | `error` | Last structured failure text | | `created_at`, `updated_at` | Audit timestamps | The unique key is: ```text (board_key, query_key, modality, start_ms, end_ms) ``` ### 7.2 `video_analysis_observations` One row represents a timestamped event or fact. | Column | Meaning | | --- | --- | | `observation_id` | Deterministic content fingerprint | | `board_key` | Recording/session namespace | | `start_ms`, `end_ms` | Event interval | | `query_text` | Query that caused extraction | | `summary` | Concise observation | | `confidence` | Model-reported confidence | | `modality` | `video` or `audio` | | `screenshot_path` | Durable evidence path | | `extra_json` | Forward-compatible metadata | | `created_at` | Audit timestamp | The deterministic observation ID makes repeated commits safe. ### 7.3 Evidence layout ```text /video_blackboard/evidence/.jpg ``` Temporary screenshots are copied into this directory before their source is registered for deletion. Stored records reference the durable copy. ## 8. Segment state machine ```text lease expires running ------------------------------+ | | | success v +----------> succeeded retryable_failed | | | retryable error | claim again +----------> retryable_failed ------+ | | non-retryable/budget exhausted +----------> permanent_failed ``` Claim behavior is transactional: 1. A missing row is inserted as `running`. 2. `succeeded` returns its cached result without acquiring work. 3. `running` with a live lease returns `in_progress`. 4. An expired lease or retryable failure is atomically reclaimed and increments `attempt_count`. Exhausted failures store a normalized category and category-derived retryability. Permanent failure is reserved for validated bad input, authentication/policy errors, cancellation, or explicit decisions. ## 9. Coverage and planning Coverage is query- and modality-specific. Given requested interval `R` and the union of successful intervals `S`, the work planner calculates: ```text missing = R - union(S) ``` Each missing interval is split into bounded chunks. Completed chunks are included in the returned result; only gaps are sent to the provider. Interval math uses integer milliseconds internally to prevent floating-point key drift. API-facing timestamps remain seconds. Observations from other queries are still placed in the ledger as context, but they do not count as completed coverage for the new query. ## 10. Execution flow ### 10.1 Tool invocation 1. Acquire the context-scoped blackboard. 2. Refresh the local ledger from persistent observations. 3. Give the coordinator the full ledger snapshot. 4. Coordinator requests segment analysis. ### 10.2 Child analysis 1. Resolve a numeric interval from recording metadata. 2. Calculate missing intervals. 3. For every chunk, acquire a lease. 4. Reuse `succeeded`; skip live `running`; execute only `claimed`. 5. Extract/compress/upload and invoke the configured VLM. 6. Persist timeline observations as soon as they are parsed. 7. Persist the complete segment result before returning to the parent. 8. On final failure, mark that exact chunk failed and keep all sibling results. ### 10.3 Parent synthesis The parent receives cached and newly completed results in the same format. If some chunks failed, it receives an explicit partial-result marker and failed intervals. It may answer with qualified confidence or request a targeted retry. ## 11. Retry and degradation policy The desired production order is: 1. retry transient upload/poll/generation failures with exponential backoff and jitter; 2. retry the exact chunk without expanding its identity; 3. split a repeatedly failing chunk into smaller children; 4. use the configured fallback video model; 5. fall back to high-frequency keyframes, OCR, and UI hierarchy around known action timestamps; 6. return partial evidence with machine-readable failed intervals. The runtime implements this order with structured failure classes, bounded bisection, configured fallback models, and a primary-model circuit breaker. ## 12. Provider and sampling strategy Provider context caching is an optimization, not the system of record. ARTEMIS must remain correct if a provider cache misses or expires. Default 1 FPS video sampling is supplemented by exact, dense seeks around DataEngine action timestamps. Quiet intervals remain sparse while taps, transitions, toasts, and spinners receive a higher sampling budget. ## 13. Concurrency - SQLite uses WAL mode and short `BEGIN IMMEDIATE` claim transactions. - The unique segment key prevents duplicate rows. - A UUID lease owner identifies one execution attempt. - Leases expire so crashes do not permanently block work. - Existing API and transcode semaphores still bound local concurrency. - Conflict cleaning affects an analyzer's presentation ledger; it never deletes persistent observations written by another analyzer. ## 14. Cleanup and retention Three artifact classes have different policies: 1. Trimmed/compressed chunks: temporary, deleted after the call. 2. Provider-uploaded files: temporary, deleted best-effort after inference. 3. Blackboard evidence: durable for the trace/session retention period. Deleting a trace/session removes blackboard rows through DataEngine and evidence through the session directory lifecycle. ## 15. Observability Required counters and trace fields: - requested duration; - cached duration; - newly analyzed duration; - number of claimed, cached, in-progress, succeeded, and failed chunks; - provider attempts and error class; - blackboard key and query key; - partial versus complete outcome; - evidence paths attached to observations. No prompt or trace status should report success merely because an error string was returned as normal text. ## 16. Compatibility and migration - Public `video_analyzer` and `video_analyzer_pure` arguments do not change. - Existing in-memory ledger dictionaries remain the presentation format. - When DataEngine persistence is unavailable, the blackboard falls back to a context-scoped in-memory store. This preserves tests and embedded usages but does not promise process-restart recovery. - Databases are migrated with `CREATE TABLE IF NOT EXISTS`; no destructive migration is required. - Existing traces without blackboard rows continue to work and start empty. ## 17. Security and privacy - Evidence remains local under the trace directory. - Provider files are deleted best-effort after requests. - Blackboard rows must not store credentials or entire model prompts. - Retention follows trace retention. - Future cross-session or identity memory requires a separate consent and privacy design; it is outside this scope. ## 18. Testing strategy ### 18.1 Unit tests - persistence across two blackboard instances using the same SQLite file; - deterministic observation deduplication; - interval union and missing-range calculation; - exact successful segment reuse; - live lease suppression and expired lease reclamation; - failed chunk remains retryable; - evidence is copied to durable storage; - two `VideoAnalyzer` instances share prior ledger state; - repeated `exec_spawn_sub_agent` does not invoke child inference again; - mixed success/failure returns partial output and preserves success. ### 18.2 Integration tests - analyze overlapping intervals against a fixture recording; - interrupt after one successful chunk and resume; - concurrent identical requests result in one provider call; - active recording resolves open-ended requests to numeric endpoints; - evidence remains readable after analyzer cleanup. ### 18.3 Device tests Per ARTEMIS mobile testing rules, end-to-end mobile flows must be explored and verified on a connected device before production test scripts are finalized. The refactor's deterministic unit tests do not require a device; live timing, fast UI transitions, and recording alignment do. ## 19. Rollout 1. **Complete:** SQLite blackboard, sharing, observation persistence, exact coverage reuse, leases, and tests. 2. **Complete:** failure classification, jitter, bounded bisection, fallback, and circuit breaking. 3. **Complete:** recording generations, sealed-prefix metadata, and generation-scoped caches. 4. **Complete:** action-aware sampling and real audio delivery. 5. **Deferred optimization:** provider file/context caching, gated by an explicit retention policy. 6. **Future accuracy layer:** semantic/event hierarchical retrieval after production traces provide a representative evaluation corpus. ## 20. Acceptance criteria - Two analyzer instances in one task see the same ledger. - Repeating an identical successful interval/query causes zero new child VLM calls. - If one of several chunks fails, successful chunks are available to the next invocation and are not rerun. - A crashed `running` chunk becomes reclaimable after lease expiry. - Persistent evidence survives normal analyzer temporary-file cleanup. - Existing video-analyzer unit tests and new blackboard tests pass. Additional reliability criteria: - timeout/media failures recover through bounded recursive subdivision; - provider-wide failures do not trigger request fan-out; - primary-model failures select the configured `utils.video_analyzer` fallback; - the primary circuit opens after a configured consecutive-failure threshold; - audio analysis is independently resumable and reusable; - recording generation changes invalidate stale trim-cache identities; - frames are sampled densely around recorded mobile actions; - traces include blackboard status, attempts, failure categories, successful duration, observation count, and circuit state. ## 21. Implemented production architecture The completed refactor extends the phase-one blackboard with the following production controls: 1. Failures are normalized as rate limit, provider outage, timeout, connection, authentication, bad request, media processing, cancellation, or unknown. The class determines retryability, bisection, and fallback. 2. Timeout, media, oversized, and unknown failures can recursively bisect a chunk down to configured size/depth bounds. Provider outages, connections, and rate limits do not bisect because fan-out would amplify load. 3. Coordinator, video child, and audio child use the configured utility-model fallback. A context-scoped circuit breaker bypasses an unhealthy primary. 4. Recording results expose `video_id`, `generation`, `sealed_until`, and `source_revision`. Trim caches are scoped by video UUID and generation; open-ended requests are resolved to numeric safe ends. 5. Universal video analysis receives sparse baseline frames, dense frames around DataEngine mobile-action timestamps, and the real audio track. Audio-only analysis sends a standard base64 audio content block. Models that reject audio degrade only that video chunk to visual-only analysis. 6. Audio uses the same lease, exact reuse, completion, and failure state machine as video. 7. DataEngine session deletion removes blackboard rows by session/video key; evidence is deleted with the trace directory. Universal-path temporary artifacts are cleaned on early exit. The persistence schema adds `error_category`, `model_name`, and `source_generation` to every segment attempt. Migrations are additive and safe for existing SQLite traces. ## 22. State-of-the-art review and adoption decision The relevant frontier is not “send the full video to a bigger model.” Recent systems converge on four architectural ideas: 1. **Agentic coarse-to-fine localization.** [VideoAgent](https://arxiv.org/abs/2403.10517) iteratively retrieves a small number of frames; [VideoTree](https://arxiv.org/abs/2405.19209) builds a query-adaptive temporal hierarchy. 2. **Persistent hierarchical multimodal memory.** The memory-augmented [VideoAgent](https://arxiv.org/abs/2403.11481) separates memory construction and query-time tool use. [VideoARM](https://arxiv.org/abs/2512.12360) uses an observe-think-act-memorize loop over hierarchical memory. 3. **Adaptive redundancy reduction.** [LongVU](https://arxiv.org/abs/2410.17434) removes redundant frames and preserves query-relevant temporal/spatial detail instead of sampling uniformly. 4. **Retrieval over very long media.** [VideoRAG](https://arxiv.org/abs/2502.01549) combines graph/text grounding with multimodal retrieval. This is useful for many-hour corpora, but heavier than one mobile-test recording. Provider guidance supports the same separation: Gemini's [video-understanding documentation](https://ai.google.dev/gemini-api/docs/video-understanding) describes direct audio/video processing and reusable Files API inputs, while [context caching](https://ai.google.dev/gemini-api/docs/caching) is an optimization for repeated common input. Neither is a durable work ledger. ### 22.1 Adopted directly - VideoARM/VideoAgent's observe → reason → act → memorize loop; - query-scoped coarse-to-fine temporal analysis; - separation of objective observations and answered query spans; - adaptive sampling, with mobile action timestamps as the domain-specific relevance signal; - independent visual/audio tools and evidence-grounded synthesis. ### 22.2 Not copied wholesale The research implementations assume offline benchmark videos, often require GPU feature extractors, and optimize multiple-choice VideoQA. ARTEMIS operates on a growing recorder stream, must survive provider/process failures, prevent duplicate paid calls, and tie evidence to test-relative action time. Importing VideoARM or VideoRAG would add a second orchestration runtime without solving leases, idempotent commit, active-recording identity, retention, or mobile alignment. ARTEMIS therefore ports their architectural invariants, not their benchmark-specific code, and supplies the production controls the prototypes do not provide. ## 23. Runtime configuration All controls live under `video_analyzer` and flow through the SDK builder: | Setting | Default | Purpose | | --- | ---: | --- | | `chunk_size_seconds` | 60 | Initial coarse chunk size | | `min_chunk_seconds` | 4 | Smallest bisection leaf | | `max_split_depth` | 4 | Bound failure-driven fan-out | | `circuit_breaker_threshold` | 3 | Failures before primary circuit opens | | `circuit_breaker_cooldown_seconds` | 60 | Half-open delay | | `action_window_seconds` | 2 | Dense window on either side of an action | | `dense_action_fps` | 4 | Sampling rate in action windows | | `max_dense_action_frames` | 24 | Dense-frame budget per chunk | | `native_max_retries` | 1 | Native-provider retries before universal fallback | | `model_call_timeout_seconds` | 120 | Hard bound for one model response | The configured `llm_config.utils.video_analyzer.fallback` supplies the model fallback. Public video tool signatures remain unchanged. ## 24. Verification status Local verification covers persistence, exact reuse, leases, partial commit, structured failures, circuit breaking, fallback selection, timeout bisection, audio reuse, real audio content blocks, action-aware sampling, exact frame seeking on an MP4 fixture, generation-scoped caches, configuration propagation, and existing analyzer/controller behavior. ### 24.1 Real-device acceptance (2026-08-25) A connected Pixel 10 (`59260DLCR00360`) completed a deterministic Google Calculator recording with independently observed UI ground truth: - `12+34=46` at 21.15s; - `7×8=56` at 32.44s; - history opened at 39.01s and closed at 42.12s; - the final screen returned to result `56`. Three query-independent video analyses all matched that ground truth. Each query was then replayed twice with the identical interval and query. All six replays returned `CACHED VIDEO ANALYSIS` in roughly 3ms, with no new child analysis. Final blackboard state was three succeeded segments, 127.8 covered query-seconds, and eight timestamped observations. The machine-readable report and retained MP4 are produced by `scripts/real_device_video_acceptance.py`. The run also exercised real provider degradation: native Gemini returned a 503 and later stalled a stream. The affected query preserved prior successes, retried only its own uncovered work, then completed through the configured universal fallback. Based on that trace, native attempts now default to two total calls with a 120-second response bound before fallback. Device exploration also exposed two non-video runtime faults that affect perceived stability and were fixed as part of the gate: 1. UIAutomator may return current string/dictionary bounds with a stale cached center. Dynamic taps now always derive their center from fresh bounds and use the cached center only when bounds are unavailable. 2. A stale desktop IPC port previously blocked every DataEngine event for one second. IPC connects now use a short localhost timeout plus exponential backoff, and detached agent initialization has a 30-second hard bound. The acceptance result demonstrates 100% accuracy and reuse for this defined fixture and run. It is not a mathematical guarantee for every future app, video, device, or provider outage; production SLOs require a larger repeated corpus and ongoing monitoring.