feat: initial release of ARTEMIS Android Agent & Automation Platform
ARTEMIS is an autonomous multimodal Android agent and mobile UI automation platform engineered for physical devices and emulators. It combines state-of-the-art multimodal Vision-Language Models (VLMs) with hierarchical closed-loop planning, resilient self-healing, and deep Android subsystem integration. Key Features & Architectural Highlights: - Dual-Engine Orchestration: * Pro Engine: Multi-agent closed-loop (Planner -> Operator -> Validator/Failure Analyzer -> Summarizer) powered by LangGraph. * Flash Engine: Ultra-fast reactive loop for deterministic, low-latency UI execution (<35 steps). - Multimodal Perception & Grounding: * Hybrid visual screenshot grounding, real-time OCR extraction, and XML accessibility hierarchy parsing. * Dynamic-First, Coordinate-Fallback locating strategy for maximum resilience against UI drift. - Autonomous Self-Healing & Diagnostics: * Failure analyzer and safety net for popup dismissal, crash recovery, and auto-retry. * Diagnostic sub-agents for log analysis, video recording inspection, and object detection. - Unified CLI & Python SDK: * Rich CLI with commands: init, doctor, run, trace, batch, bench, server, and mcp. * Fluent Python SDK with Pydantic structured output models and package locking. - Multi-Provider LLM Architecture: * Support for Google Gemini, OpenAI, Anthropic Claude, Vertex AI, OpenRouter, xAI Grok, and local endpoints (Ollama/vLLM). * Centralized configuration in config/artemis.jsonc. - Native MCP & Observability: * Model Context Protocol (MCP) server endpoints for AI IDEs. * Full trace recording with before/after screenshots, action overlays, and execution replays. Change-Id: I430b93ee2b76b54cc5a3c6eeccd15d2cf29830a1
This commit is contained in:
somew1nd committed
commit
206a9ce921
229 files changed
+49653
No files matched your search
@@ -0,0 +1,16 @@
|
||||
.git
|
||||
.venv
|
||||
__pycache__
|
||||
*.pyc
|
||||
traces/
|
||||
latte-traces/
|
||||
debug/replay/
|
||||
debug_pub/replay/
|
||||
*.mp4
|
||||
*.jpg
|
||||
*.png
|
||||
pytest_temp/
|
||||
/scratch/
|
||||
android_world*
|
||||
/dist/
|
||||
node_modules/
|
||||
@@ -0,0 +1,19 @@
|
||||
# ==============================================================================
|
||||
# ☕ Latte Environment Configuration
|
||||
# ==============================================================================
|
||||
|
||||
# 1. Primary LLM Provider API Key (Required for Agent Planning & Reasoning)
|
||||
GEMINI_API_KEY=your_gemini_api_key_here
|
||||
|
||||
# 2. Google Cloud Vision OCR API Key (Required for fast text perception & XML fusion)
|
||||
OCR_API_KEY=your_google_cloud_vision_api_key_here
|
||||
|
||||
# Optional: Alternative LLM Providers
|
||||
# OPENAI_API_KEY=your_openai_api_key_here
|
||||
# ANTHROPIC_API_KEY=your_anthropic_api_key_here
|
||||
# OPEN_ROUTER_API_KEY=your_openrouter_api_key_here
|
||||
# XAI_API_KEY=your_xai_api_key_here
|
||||
|
||||
# Optional: Target Android Device Serial (Leave commented for auto-detection)
|
||||
# ADB_DEVICE_SERIAL=emulator-5554
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
.gcloudignore
|
||||
.git
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
traces/
|
||||
latte-traces/
|
||||
debug/replay/
|
||||
debug_pub/replay/
|
||||
*.mp4
|
||||
*.jpg
|
||||
*.png
|
||||
pytest_temp/
|
||||
/scratch/
|
||||
android_world*
|
||||
/dist/
|
||||
!frontend/dist/**
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
# ==============================================================================
|
||||
# ☕ Latte Framework - Master Git Ignore Configuration
|
||||
# ==============================================================================
|
||||
|
||||
# --- 1. Environment Variables & Virtual Environments (Recursive) ---
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
uv.lock
|
||||
**/.venv/
|
||||
**/venv/
|
||||
**/ENV/
|
||||
**/env/
|
||||
**/.pyenv/
|
||||
.python-version
|
||||
|
||||
# --- 2. Node & Web Frontend Dependencies / Artifacts (Recursive) ---
|
||||
**/node_modules/
|
||||
**/node_modules
|
||||
**/.angular/
|
||||
**/.angular/cache/
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
|
||||
# --- 3. Python Caches, Bytecode & Build Artifacts (Recursive) ---
|
||||
**/__pycache__/
|
||||
**/*.py[cod]
|
||||
**/*$py.class
|
||||
**/*.so
|
||||
**/*.pyd
|
||||
**/*.c
|
||||
**/build/
|
||||
**/dist/
|
||||
**/*.egg-info/
|
||||
**/.eggs/
|
||||
develop-eggs/
|
||||
downloads/
|
||||
eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
MANIFEST
|
||||
|
||||
# --- 4. Testing, Coverage & Linters (Recursive) ---
|
||||
**/.pytest_cache/
|
||||
**/.ruff_cache/
|
||||
**/.mypy_cache/
|
||||
**/.pytype/
|
||||
.coverage
|
||||
.coverage.*
|
||||
.cache
|
||||
coverage.xml
|
||||
nosetests.xml
|
||||
htmlcov/
|
||||
pytest_temp/
|
||||
tests_tmp/
|
||||
tests/**/outputs/
|
||||
|
||||
# --- 5. Runtime Logs, Spans, Replay & Traces (Recursive) ---
|
||||
traces/
|
||||
traces_test/
|
||||
latte-traces/
|
||||
data_engine.db*
|
||||
**/replay/outputs/
|
||||
**/replay/data/
|
||||
**/*.log
|
||||
**/*.log.*
|
||||
**/*.out
|
||||
**/*.tmp
|
||||
.latte_ipc_port
|
||||
.latte_paused
|
||||
.latte/
|
||||
.artemis_ipc_port
|
||||
.artemis_paused
|
||||
.artemis/
|
||||
*.bak
|
||||
live_screenshot*
|
||||
recording.mp4
|
||||
recording.mkv
|
||||
screenshot.jpg
|
||||
screen.png
|
||||
|
||||
# --- 6. Workspace Scratch, Dumps & Temporary Experimentation ---
|
||||
scratch/
|
||||
artifacts/
|
||||
ignores/
|
||||
*.docx
|
||||
*gdoc.html
|
||||
*.bug.log
|
||||
|
||||
# --- 7. AI Agent Workspace & IDE Metadata ---
|
||||
.agents/
|
||||
_agents/
|
||||
jetski/
|
||||
.jetski*
|
||||
.benchmarks/
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
+103
@@ -0,0 +1,103 @@
|
||||
# How to Contribute
|
||||
|
||||
We'd love to accept your patches and contributions to ARTEMIS! There are just a few small guidelines you need to follow.
|
||||
|
||||
---
|
||||
|
||||
## Contributor License Agreement (CLA)
|
||||
|
||||
Contributions to this project must be accompanied by a Contributor License Agreement (CLA). You (or your employer) retain the copyright to your contribution; this simply gives us permission to use and redistribute your contributions as part of the project.
|
||||
|
||||
- Head over to <https://cla.developers.google.com/> to check your current agreements on file or to sign a new one.
|
||||
- If you are an individual contributing original source code, you must sign the **Individual CLA**.
|
||||
- If you are contributing on behalf of a company or organization, a **Corporate CLA** must be executed by your employer.
|
||||
- You generally only need to submit a CLA once, so if you have already submitted one for another Google project, you do not need to sign it again.
|
||||
|
||||
---
|
||||
|
||||
## Community Guidelines
|
||||
|
||||
This project adheres to [Google's Open Source Community Guidelines](https://opensource.google/conduct/). By participating, all contributors and maintainers are expected to uphold these principles and maintain a welcoming, inclusive, and professional environment.
|
||||
|
||||
---
|
||||
|
||||
## Code Reviews
|
||||
|
||||
All submissions, including submissions by project maintainers, require review. We use standard GitHub pull requests for this purpose:
|
||||
|
||||
1. Consult [GitHub Help](https://docs.github.com/articles/about-pull-requests/) for information on using pull requests.
|
||||
2. Maintainers will review your changes, offer feedback, and approve when ready.
|
||||
3. Once approved and all CI checks pass, a project maintainer will merge your PR.
|
||||
|
||||
---
|
||||
|
||||
## Development & Contribution Workflow
|
||||
|
||||
### 1. Setting Up Your Development Environment
|
||||
|
||||
- **Prerequisites**:
|
||||
- Python **3.12+**
|
||||
- Android SDK Platform Tools (`adb` available in `PATH`)
|
||||
- [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip`
|
||||
|
||||
- **Clone & Install**:
|
||||
```bash
|
||||
git clone https://github.com/<your-username>/artemis.git
|
||||
cd artemis
|
||||
|
||||
# Install all dependencies and setup pre-commit hooks
|
||||
make setup
|
||||
```
|
||||
*(Alternatively: `uv sync --dev` and `uv run pre-commit install`)*
|
||||
|
||||
### 2. Making Changes
|
||||
|
||||
- Create a descriptive topic branch off `main`:
|
||||
```bash
|
||||
git checkout -b feat/my-new-feature
|
||||
# or
|
||||
git checkout -b fix/issue-description
|
||||
```
|
||||
|
||||
- Follow the project's coding and style conventions:
|
||||
- **Absolute Imports**: Always use absolute imports (`from artemis.sdk import Agent`). Relative imports are disallowed.
|
||||
- **Typing**: Use comprehensive type annotations on all public functions, classes, and methods.
|
||||
- **Testing**: Add or update test cases under `tests/` for any modified or new behavior.
|
||||
|
||||
### 3. Local Verification
|
||||
|
||||
Before creating a pull request, ensure all linters, type checks, and tests pass:
|
||||
|
||||
```bash
|
||||
# Format code (Ruff)
|
||||
make format
|
||||
|
||||
# Lint code (Ruff)
|
||||
make lint
|
||||
|
||||
# Run static type checker (Pyright)
|
||||
make typecheck
|
||||
|
||||
# Run test suite (Pytest)
|
||||
make test
|
||||
```
|
||||
|
||||
### 4. Submitting a Pull Request
|
||||
|
||||
1. Push your branch to your fork:
|
||||
```bash
|
||||
git push origin feat/my-new-feature
|
||||
```
|
||||
2. Open a Pull Request against the `main` branch of the ARTEMIS repository.
|
||||
3. Complete the PR template, describing:
|
||||
- The motivation behind the change.
|
||||
- Any related issue numbers (e.g., `Fixes #42`).
|
||||
- How the change was tested (device, emulator, unit tests).
|
||||
4. Ensure the CLA check automatically passes on your PR.
|
||||
|
||||
---
|
||||
|
||||
## Reporting Bugs and Feature Requests
|
||||
|
||||
- **Bug Reports**: Open an issue describing the bug, including steps to reproduce, device/emulator specifications, Android version, logs, and screenshots if applicable.
|
||||
- **Feature Requests**: Open an issue detailing the use case, proposed API or workflow, and alternatives considered.
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install necessary build tools
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
curl \
|
||||
git \
|
||||
adb \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy application source code and apps
|
||||
COPY artemis/ /app/artemis/
|
||||
COPY apps/ /app/apps/
|
||||
COPY config/ /app/config/
|
||||
COPY pyproject.toml setup.py README.md LICENSE /app/
|
||||
|
||||
# Install server and core dependencies
|
||||
RUN pip install --no-cache-dir \
|
||||
uvicorn \
|
||||
fastapi \
|
||||
pydantic \
|
||||
pydantic-settings \
|
||||
python-dotenv \
|
||||
sseclient-py \
|
||||
websockets \
|
||||
adbutils \
|
||||
jinja2 \
|
||||
colorama \
|
||||
psutil \
|
||||
httpx \
|
||||
retry \
|
||||
google-auth \
|
||||
langchain \
|
||||
langchain-core \
|
||||
langchain-community \
|
||||
langchain-google-genai \
|
||||
langchain-google-vertexai \
|
||||
langchain-mcp-adapters \
|
||||
google-genai \
|
||||
langgraph \
|
||||
typer \
|
||||
pillow \
|
||||
packaging \
|
||||
uiautomator2 \
|
||||
opencv-python-headless \
|
||||
uuid-utils \
|
||||
posthog \
|
||||
mcp \
|
||||
inquirer \
|
||||
rich \
|
||||
requests \
|
||||
imageio-ffmpeg \
|
||||
langchain-openai
|
||||
|
||||
# Install artemis package locally (no-deps to prevent pulling unnecessary heavy optional libraries)
|
||||
RUN pip install --no-cache-dir --no-deps -e /app
|
||||
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV PYTHONPATH=/app
|
||||
ENV ARTEMIS_APP_DIR=/app
|
||||
ENV PORT=8080
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
CMD ["uvicorn", "apps.admin_console.server:app", "--host", "0.0.0.0", "--port", "8080"]
|
||||
@@ -0,0 +1,202 @@
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
https://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2026 Google LLC.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
https://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -0,0 +1,56 @@
|
||||
.PHONY: help test install setup clean precommit-install precommit lint format typecheck
|
||||
|
||||
help: ## Show this help message
|
||||
@echo 'Usage: make [target]'
|
||||
@echo ''
|
||||
@echo 'Available targets:'
|
||||
@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf " %-20s %s\n", $$1, $$2}' $(MAKEFILE_LIST)
|
||||
|
||||
test: ## Run all tests
|
||||
@echo "🧪 Running all tests..."
|
||||
@uv run pytest -v
|
||||
|
||||
install: ## Install dependencies
|
||||
@echo "📦 Installing dependencies..."
|
||||
@uv sync --dev
|
||||
|
||||
setup: ## Setup project (install dependencies + pre-commit hooks)
|
||||
@echo "🚀 Setting up project..."
|
||||
@$(MAKE) install
|
||||
@$(MAKE) precommit-install
|
||||
@echo ""
|
||||
@echo "✅ Setup complete! You're ready to start developing."
|
||||
|
||||
lint: ## Run linting checks
|
||||
@echo "🔍 Running linting checks..."
|
||||
@uv run ruff format --check
|
||||
@uv run ruff check
|
||||
|
||||
format: ## Format code
|
||||
@echo "✨ Formatting code..."
|
||||
@uv run ruff format
|
||||
@uv run ruff check --fix
|
||||
|
||||
typecheck: ## Run type checking
|
||||
@echo "🔍 Running type checks..."
|
||||
@uv run pyright
|
||||
|
||||
precommit-install: ## Install pre-commit hooks
|
||||
@echo "🔧 Installing pre-commit hooks..."
|
||||
@uv run pre-commit install
|
||||
@echo "✅ Pre-commit hooks installed successfully!"
|
||||
@echo ""
|
||||
@echo "Pre-commit will now run automatically on every commit."
|
||||
@echo "To run manually: make precommit"
|
||||
|
||||
precommit: ## Run pre-commit hooks manually on all files
|
||||
@echo "🔍 Running pre-commit checks..."
|
||||
@uv run pre-commit run --all-files
|
||||
|
||||
clean: ## Clean up generated files and caches
|
||||
@echo "🧹 Cleaning up..."
|
||||
@rm -rf .pytest_cache
|
||||
@rm -rf .ruff_cache
|
||||
@rm -rf **/__pycache__
|
||||
@find . -type d -name "*.egg-info" -exec rm -rf {} + 2>/dev/null || true
|
||||
@echo "✅ Cleanup complete"
|
||||
@@ -0,0 +1,399 @@
|
||||
# ARTEMIS ☕📱
|
||||
|
||||
**Autonomous Multimodal Android Agent & Mobile UI Automation Platform**
|
||||
|
||||
[](https://www.python.org/downloads/)
|
||||
[](LICENSE)
|
||||
[](https://github.com/langchain-ai/langgraph)
|
||||
[](https://ai.google.dev/)
|
||||
|
||||
ARTEMIS is a next-generation, production-grade autonomous agent and testing framework engineered for Android devices and emulators. Combining state-of-the-art multimodal Large Vision-Language Models (VLMs) with hierarchical closed-loop planning, resilient self-healing, and deep Android subsystem integration (ADB, UIAutomator2, accessibility trees), ARTEMIS enables autonomous UI navigation, robust automated testing, and intelligent device diagnostics.
|
||||
|
||||
---
|
||||
|
||||
## 📑 Table of Contents
|
||||
|
||||
- [Key Features](#-key-features)
|
||||
- [Architecture Overview](#-architecture-overview)
|
||||
- [Prerequisites](#-prerequisites)
|
||||
- [Installation & Setup](#-installation--setup)
|
||||
- [Quick Start](#-quick-start)
|
||||
- [Command Line Interface (CLI)](#command-line-interface-cli)
|
||||
- [Python SDK](#python-sdk)
|
||||
- [Execution Profiles: Flash vs. Pro](#-execution-profiles-flash-vs-pro)
|
||||
- [Model Context Protocol (MCP) Integration](#-model-context-protocol-mcp-integration)
|
||||
- [Configuration Guide](#-configuration-guide)
|
||||
- [Using Non-Google LLMs](#using-non-google-llms)
|
||||
- [Trace Inspection & Observability](#-trace-inspection--observability)
|
||||
- [Development & Testing](#-development--testing)
|
||||
- [License](#-license)
|
||||
|
||||
---
|
||||
|
||||
## ✨ Key Features
|
||||
|
||||
- **Dual-Engine Execution**:
|
||||
- **ARTEMIS Flash**: High-speed, reactive single-agent loop for deterministic, low-latency tasks (< 35 steps).
|
||||
- **ARTEMIS Pro**: Multi-agent closed-loop graph orchestration (Planner $\rightarrow$ Operator $\rightarrow$ Failure Analyzer $\rightarrow$ Summarizer) powered by LangGraph.
|
||||
- **Multimodal Perception**:
|
||||
- Combined visual screenshot grounding, real-time OCR, accessibility hierarchy (XML) parsing, and dynamic video analysis.
|
||||
- **Resilient Locating Strategy ("Dynamic-First, Coordinate-Fallback")**:
|
||||
- Prioritizes semantic IDs, XPath, and accessibility text matching to withstand UI layout drift; seamlessly falls back to calibrated absolute/relative coordinates.
|
||||
- **Autonomous Self-Healing**:
|
||||
- Built-in failure analyzer and safety net detect blocked flows, system dialogs, app crashes, or unexpected states and auto-recover in real time.
|
||||
- **App Locking & Session Boundaries**:
|
||||
- Restricts execution context to target application packages with automatic foreground recovery.
|
||||
- **Model Context Protocol (MCP) Native**:
|
||||
- Exposes Android automation, state inspection, and trace querying as MCP tools for IDEs and AI orchestrators.
|
||||
- **Structured Pydantic Outputs**:
|
||||
- Automatically parses agent observations and findings into validated Pydantic schemas.
|
||||
|
||||
---
|
||||
|
||||
## 🏛 Architecture Overview
|
||||
|
||||
ARTEMIS operates on a closed-loop multi-agent architecture:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────┐
|
||||
│ User Goal / Task │
|
||||
└────────────────┬────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────┐
|
||||
│ Planner Agent │
|
||||
│ (Hierarchical Task Breakdown) │
|
||||
└───────────────┬───────────────┘
|
||||
│
|
||||
┌──────────────┴──────────────┐
|
||||
▼ ▼
|
||||
┌─────────────────────────────────┐ ┌───────────────────────────────┐
|
||||
│ Operator Agent │ │ Diagnostic Agents │
|
||||
│ (Screen VLM + XML Grounding) │ │ (Log Reader, Video Analyzer, │
|
||||
└────────────────┬────────────────┘ │ Object Detector, Explorer) │
|
||||
│ └───────────────────────────────┘
|
||||
▼
|
||||
┌─────────────────────────────────┐
|
||||
│ Execution & Device Control │
|
||||
│ (ADB / UIAutomator2 / Taps) │
|
||||
└────────────────┬────────────────┘
|
||||
│
|
||||
[Action Success / State Change?]
|
||||
│ │
|
||||
(Yes) (No / Blocked)
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────────┐ ┌─────────────────────────────────────────┐
|
||||
│ Next Step / Complete │ │ Failure Analyzer & Safety Net │
|
||||
└──────────┬───────────┘ │ (Popup dismissal, Retry, Self-healing) │
|
||||
│ └────────────────────┬────────────────────┘
|
||||
▼ │
|
||||
┌──────────────────────┐ │
|
||||
│ Summarizer / Output │◄──────────────────────┘
|
||||
│ (Reports, Traces) │
|
||||
└──────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Prerequisites
|
||||
|
||||
1. **Python**: Version `3.12` or higher.
|
||||
2. **Android SDK / ADB**:
|
||||
- `adb` must be installed and accessible in your system `PATH`.
|
||||
- Verify with `adb devices`.
|
||||
3. **Android Device or Emulator**:
|
||||
- A physically connected Android device with **USB Debugging** enabled, or
|
||||
- An active Android Emulator (AVD) running on your local machine.
|
||||
4. **Screen Recording & Video Tools** *(optional for `--with-video-recording-tools`)*:
|
||||
- **`scrcpy`**: Required for live Android device screen video recording (`apt install scrcpy` / `brew install scrcpy`).
|
||||
- **`ffmpeg`**: Required for video transcoding, trimming, and audio extraction (`apt install ffmpeg` / `brew install ffmpeg`).
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Installation & Setup
|
||||
|
||||
### 1. Clone the Repository
|
||||
|
||||
```bash
|
||||
git clone https://github.com/google/artemis.git
|
||||
cd artemis
|
||||
```
|
||||
|
||||
### 2. Install Dependencies
|
||||
|
||||
Using [`uv`](https://docs.astral.sh/uv/) (recommended):
|
||||
|
||||
```bash
|
||||
# Install dependencies into a virtual environment
|
||||
uv sync --dev
|
||||
```
|
||||
|
||||
Or using standard `pip`:
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -e ".[dev]"
|
||||
```
|
||||
|
||||
### 3. Quick Setup (10 Seconds)
|
||||
|
||||
Run the interactive setup wizard to configure your API key and connect your device:
|
||||
|
||||
```bash
|
||||
artemis init
|
||||
```
|
||||
|
||||
Verify your environment and connected devices anytime with the doctor diagnostic tool:
|
||||
|
||||
```bash
|
||||
artemis doctor
|
||||
```
|
||||
|
||||
*(Alternatively, copy `.env.example` to `.env` and set `GEMINI_API_KEY` for LLM reasoning and `OCR_API_KEY` for Vision perception.)*
|
||||
|
||||
---
|
||||
|
||||
## 💡 Quick Start
|
||||
|
||||
### Command Line Interface (CLI)
|
||||
|
||||
ARTEMIS installs a unified command-line tool `artemis`:
|
||||
|
||||
```bash
|
||||
# 1. Environment & Device Diagnostics
|
||||
artemis doctor
|
||||
|
||||
# 2. Interactive Setup Wizard
|
||||
artemis init
|
||||
|
||||
# 3. Basic task execution (ARTEMIS Pro by default)
|
||||
artemis run "Open YouTube and search for Lo-Fi Hip Hop"
|
||||
|
||||
# 4. Fast reactive profile (ARTEMIS Flash)
|
||||
artemis run "Open Settings and verify Airplane Mode is turned off" --profile flash
|
||||
|
||||
# 5. Execute with trace recording and structured output
|
||||
artemis run "Open Maps, search for coffee shops, and extract the top 3 results" \
|
||||
--test-name "coffee_search_test" \
|
||||
--traces-path "./traces" \
|
||||
--output-description '{"places": "list of shop names", "ratings": "list of rating scores"}'
|
||||
|
||||
# 6. Enable dynamic video recording analysis
|
||||
artemis run "Play the latest video in the feed and verify playback" --with-video-recording-tools
|
||||
```
|
||||
|
||||
### Python SDK
|
||||
|
||||
Use the fluent SDK to integrate ARTEMIS into your test suites or automation workflows:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from pydantic import BaseModel, Field
|
||||
from artemis.sdk import Agent
|
||||
|
||||
|
||||
class VideoSearchResult(BaseModel):
|
||||
query: str = Field(..., description="The search term used")
|
||||
top_video_title: str = Field(..., description="Title of the first video result")
|
||||
channel_name: str = Field(..., description="Channel name of the first video")
|
||||
|
||||
|
||||
async def main():
|
||||
# 1. Initialize the agent
|
||||
agent = Agent()
|
||||
await agent.init()
|
||||
|
||||
try:
|
||||
# 2. Build task specification
|
||||
task = (
|
||||
agent.new_task("Search for 'Pixel 9 Pro' and get the first video details")
|
||||
.with_name("youtube_search_verification")
|
||||
.with_locked_app_package("com.google.android.youtube")
|
||||
.with_output_format(VideoSearchResult)
|
||||
.with_max_steps(50)
|
||||
.build()
|
||||
)
|
||||
|
||||
# 3. Execute task
|
||||
result: VideoSearchResult = await agent.run_task(request=task)
|
||||
|
||||
if result:
|
||||
print("Search Results:")
|
||||
print(f"- Query: {result.query}")
|
||||
print(f"- First Video: {result.top_video_title}")
|
||||
print(f"- Channel: {result.channel_name}")
|
||||
else:
|
||||
print("Task did not complete successfully.")
|
||||
|
||||
finally:
|
||||
# 4. Clean up agent session & device connections
|
||||
await agent.clean()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Execution Profiles: Flash vs. Pro
|
||||
|
||||
| Feature | ARTEMIS Flash (`--profile flash`) | ARTEMIS Pro (`--profile default`) |
|
||||
| :--- | :--- | :--- |
|
||||
| **Model Loop** | Single reactive agent (Observe-Think-Act) | Multi-node LangGraph orchestration |
|
||||
| **Best For** | Short, deterministic tasks (< 35 steps) | Deep, complex, multi-stage workflows |
|
||||
| **Step Latency** | ~3–5 seconds / step | ~15–30 seconds / turn (with planning & validation) |
|
||||
| **Planning** | Direct goal execution | Hierarchical multi-step plan decomposition |
|
||||
| **Diagnostics** | Basic error reporting | Deep diagnostics, log analysis & self-healing |
|
||||
|
||||
---
|
||||
|
||||
## 🔌 Model Context Protocol (MCP) Integration
|
||||
|
||||
ARTEMIS includes built-in MCP servers and tools to allow AI IDEs (such as Claude Desktop, Cursor, and Gemini Jetski) to control Android devices directly:
|
||||
|
||||
| Tool Name | Description |
|
||||
| :--- | :--- |
|
||||
| `mobile_run_task` | Launches an autonomous background automation task on the connected device. |
|
||||
| `mobile_manage_task` | Inspects status, terminates tasks, or injects real-time steering instructions. |
|
||||
| `mobile_get_device_state` | Retrieves real-time screenshots or simplified accessibility UI trees. |
|
||||
| `mobile_inspect_trace` | Inspects step-by-step screenshots, visual action overlays, and execution logs. |
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ Configuration Guide
|
||||
|
||||
All model configurations, thinking budgets, agent capabilities, and tool filtering are consolidated into a single self-explanatory configuration file: [`config/artemis.jsonc`](config/artemis.jsonc).
|
||||
|
||||
```jsonc
|
||||
{
|
||||
// 1. Global Default Model (Applied across all sub-agents unless overridden)
|
||||
"default": {
|
||||
"provider": "google",
|
||||
"model": "gemini-3.6-flash",
|
||||
"thinking_level": "medium",
|
||||
"fallback": {
|
||||
"provider": "google",
|
||||
"model": "gemini-3.5-flash"
|
||||
}
|
||||
},
|
||||
|
||||
// 2. Built-in Presets
|
||||
"presets": {
|
||||
"gemini-flagship": { "provider": "google", "model": "gemini-3.6-flash" },
|
||||
"openai-gpt4o": { "provider": "openai", "model": "gpt-4o" },
|
||||
"cost-saving": { "provider": "google", "model": "gemini-3.5-flash-lite" },
|
||||
"local-ollama": { "provider": "openai", "model": "llama3.2-vision" }
|
||||
},
|
||||
|
||||
// 3. Agent & Sub-Agent Node Customizations (Optional overrides)
|
||||
"nodes": {
|
||||
// 🧠 Planner: High-level goal decomposition and strategic step planning
|
||||
"planner": { "thinking_level": "medium" },
|
||||
|
||||
// 🎯 Operator: Screen perception, element interaction, and direct execution
|
||||
"operator": { "thinking_level": "medium", "include_thoughts": true },
|
||||
|
||||
// 🛡️ Validator & Failure Analyzer: Self-healing and auto-recovery
|
||||
"validator_failure_analyzer": { "thinking_level": "medium" }
|
||||
},
|
||||
|
||||
// 4. Agent Capabilities & Tool Filtering
|
||||
"agent": {
|
||||
"explorer_versions": { "operator": "flash", "validator": "flash" },
|
||||
"blacklisted_tools": {
|
||||
"explorer": ["ask_image_processor", "get_ocr_list", "inspect_region"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> **Zero-Friction Global Change**: To switch the entire platform from Gemini to GPT-4o or Claude, simply edit the `"default"` block once in `config/artemis.jsonc`. All 15 sub-agents automatically inherit the new model!
|
||||
|
||||
### Using Non-Google LLMs
|
||||
|
||||
ARTEMIS supports multi-provider LLM backends via standard LangChain integrations (Google Gemini, OpenAI, OpenRouter, xAI / Grok, or Google Vertex AI).
|
||||
|
||||
#### Supported Providers & API Key Setup
|
||||
|
||||
| Provider | Provider Key | Environment Variable | Base URL Override | Example Models |
|
||||
| :--- | :--- | :--- | :--- | :--- |
|
||||
| **Google Gemini** | `google` *(default)* | `GEMINI_API_KEY` | N/A | `gemini-3.6-flash`, `gemini-3.5-flash` |
|
||||
| **Google Vision OCR** | `ocr` | `OCR_API_KEY` | N/A | Google Cloud Vision Text Detection |
|
||||
| **Google Vertex AI** | `vertexai` | Google ADC (`gcloud auth application-default login`) | N/A | `gemini-3.6-flash` |
|
||||
| **OpenAI** | `openai` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | `gpt-4o`, `gpt-4o-mini`, `o3` |
|
||||
| **OpenRouter** | `openrouter` | `OPEN_ROUTER_API_KEY` | N/A | `anthropic/claude-3.5-sonnet`, `meta-llama/llama-3.3-70b-instruct` |
|
||||
| **xAI (Grok)** | `xai` | `XAI_API_KEY` | N/A | `grok-2-vision-1212`, `grok-beta` |
|
||||
|
||||
> [!NOTE]
|
||||
> ARTEMIS uses multimodal visual grounding (screenshots) for UI navigation. When assigning non-Google models to vision-dependent agents (such as `operator`), ensure the selected model supports image/vision inputs.
|
||||
|
||||
#### Switching Provider in `config/artemis.jsonc`
|
||||
|
||||
To use OpenAI or Claude globally, simply change the `default` section in [`config/artemis.jsonc`](config/artemis.jsonc):
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"default": {
|
||||
"provider": "openai",
|
||||
"model": "gpt-4o",
|
||||
"fallback": {
|
||||
"provider": "openai",
|
||||
"model": "gpt-4o-mini"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Custom / Local OpenAI-Compatible Endpoints (Ollama, vLLM, LocalAI)
|
||||
|
||||
To connect ARTEMIS to local or self-hosted LLM endpoints (e.g., Ollama or vLLM), set `OPENAI_BASE_URL` in `.env` and specify `openai` as the provider in your configuration:
|
||||
|
||||
```bash
|
||||
# Route OpenAI provider calls to local Ollama / vLLM endpoint
|
||||
OPENAI_API_KEY=ollama # dummy key if required by client
|
||||
OPENAI_BASE_URL=http://localhost:11434/v1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Trace Inspection & Observability
|
||||
|
||||
Every task execution can record rich execution artifacts stored in `./traces/<trace_id>/`:
|
||||
- **`notes/`**: High-level planner notes, step summaries, and generated reports (`output.md`).
|
||||
- **`screenshots/`**:
|
||||
- `before_screenshot.jpg`: Initial screen observed by the agent.
|
||||
- `action_overlay_screenshot.jpg`: Screen with visual tap markers (red circles) and swipe vectors (red arrows).
|
||||
- **`logs/`**: Raw framework, operator, and failure analyzer diagnostic logs.
|
||||
|
||||
---
|
||||
|
||||
## 🛠 Development & Testing
|
||||
|
||||
ARTEMIS provides standard `make` targets for development:
|
||||
|
||||
```bash
|
||||
# Setup environment and pre-commit hooks
|
||||
make setup
|
||||
|
||||
# Run test suite
|
||||
make test
|
||||
|
||||
# Run code formatters (ruff)
|
||||
make format
|
||||
|
||||
# Run linter checks
|
||||
make lint
|
||||
|
||||
# Run static type checker (pyright)
|
||||
make typecheck
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📄 License
|
||||
|
||||
This project is licensed under the [Apache License 2.0](LICENSE).
|
||||
@@ -0,0 +1,26 @@
|
||||
# 📱 Artemis Applications Ecosystem (`apps/`)
|
||||
|
||||
This directory houses the two primary applications and user interfaces of the Artemis ecosystem.
|
||||
|
||||
---
|
||||
|
||||
## 📂 Subsystems Overview
|
||||
|
||||
```
|
||||
apps/
|
||||
├── 🛠️ admin_console/ # [Interface 1] Full-fidelity Trace Inspector & Task Control Console (FastAPI + Web)
|
||||
└── ✨ showcase_ui/ # [Interface 2] Aesthetic User Presentation & Interaction Workspace (Angular 19)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Detailed Comparison of the Two UIs
|
||||
|
||||
| Dimension | 🛠️ Admin & Trace Console (`admin_console`) | ✨ Showcase & Workspace UI (`showcase_ui`) |
|
||||
| :--- | :--- | :--- |
|
||||
| **Primary Audience** | Developers, Test Engineers, System Admins | End Users, Customers, Product Demo Audiences |
|
||||
| **UI Paradigm** | High-density, full-fidelity tree logs, step-by-step inspector | Dynamic Aurora Glow, Glassmorphism, dual-pane stream |
|
||||
| **Core Capabilities** | • Full hierarchical Call Tree & LLM thinking inspection<br>• Pre/post action screenshots & visual overlay verification<br>• Video playback & step replay engine (`/api/replay`)<br>• Queue tasks & live state toggle | • Real-time Agent thought & action progress streaming<br>• Interactive natural language chat sidebar<br>• Aesthetic visual representation of mobile autonomy |
|
||||
| **Tech Stack** | Python (FastAPI) + Vanilla Responsive Web | TypeScript, Angular 19, SCSS |
|
||||
| **Start Command** | `uv run uvicorn apps.admin_console.server:app --port 8000 --reload` | `cd apps/showcase_ui && npm start` (Port 4200) |
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
"""Artemis / Latte Applications Package."""
|
||||
@@ -0,0 +1,41 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""☕ ARTEMIS: Autonomous Multimodal Android Agent & Testing Framework."""
|
||||
|
||||
__version__ = "0.2.0"
|
||||
|
||||
from artemis.interfaces.sdk.client import ArtemisClient
|
||||
from artemis.interfaces.sdk.task import StreamEvent, StreamEventType, Task, TaskResult
|
||||
from artemis.sdk.agent import Agent
|
||||
from artemis.sdk.builders import Builders
|
||||
from artemis.agents.base import BaseAgent, AgentRegistry
|
||||
from artemis.drivers.base import BaseDeviceDriver
|
||||
from artemis.tools.base import artemis_tool, ToolRegistry
|
||||
|
||||
__all__ = [
|
||||
"__version__",
|
||||
"ArtemisClient",
|
||||
"StreamEvent",
|
||||
"StreamEventType",
|
||||
"Agent",
|
||||
"Builders",
|
||||
"Task",
|
||||
"TaskResult",
|
||||
"BaseAgent",
|
||||
"AgentRegistry",
|
||||
"BaseDeviceDriver",
|
||||
"artemis_tool",
|
||||
"ToolRegistry",
|
||||
]
|
||||
@@ -0,0 +1,18 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from artemis.interfaces.cli.main import app
|
||||
|
||||
if __name__ == "__main__":
|
||||
app()
|
||||
@@ -0,0 +1,37 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Modular Agent Framework for ARTEMIS."""
|
||||
|
||||
from artemis.agents.base import (
|
||||
AgentConfig,
|
||||
AgentRegistry,
|
||||
AgentResponse,
|
||||
BaseAgent,
|
||||
)
|
||||
from artemis.agents.operator.operator_agent import OperatorAgent
|
||||
from artemis.agents.planner.planner_agent import PlannerAgent
|
||||
from artemis.agents.summarizer.summarizer_agent import SummarizerAgent
|
||||
from artemis.agents.validator.failure_analyzer_agent import FailureAnalyzerAgent
|
||||
|
||||
__all__ = [
|
||||
"BaseAgent",
|
||||
"AgentConfig",
|
||||
"AgentResponse",
|
||||
"AgentRegistry",
|
||||
"PlannerAgent",
|
||||
"OperatorAgent",
|
||||
"SummarizerAgent",
|
||||
"FailureAnalyzerAgent",
|
||||
]
|
||||
@@ -0,0 +1,146 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Base Agent Protocol & Lifecycle Framework for ARTEMIS.
|
||||
|
||||
Provides a unified interface and lifecycle management for all specialized
|
||||
multimodal agents (Planner, Operator, FailureAnalyzer, Explorer, Summarizer, etc.).
|
||||
"""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from artemis.drivers.factory import get_driver
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
pass
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class AgentConfig(BaseModel):
|
||||
"""Configuration options for an individual agent."""
|
||||
|
||||
model_name: str | None = Field(default=None, description="LLM/VLM model override")
|
||||
temperature: float = Field(default=0.0, description="Sampling temperature")
|
||||
max_tokens: int = Field(default=4096, description="Maximum output tokens")
|
||||
system_prompt_override: str | None = Field(
|
||||
default=None, description="Custom system prompt override"
|
||||
)
|
||||
enabled_tools: list[str] = Field(
|
||||
default_factory=list, description="Tool names enabled for this agent"
|
||||
)
|
||||
|
||||
|
||||
class AgentResponse(BaseModel):
|
||||
"""Standardized structured response from an agent invocation."""
|
||||
|
||||
status: str = Field(
|
||||
default="success", description="Execution status ('success', 'retry', 'fail', 'done')"
|
||||
)
|
||||
thought: str | None = Field(default=None, description="Reasoning and internal monologue")
|
||||
action_taken: str | None = Field(default=None, description="Action or tool called")
|
||||
output: Any = Field(default=None, description="Output payload or state updates")
|
||||
metadata: dict[str, Any] = Field(
|
||||
default_factory=dict, description="Diagnostic and profiling metadata"
|
||||
)
|
||||
|
||||
|
||||
class BaseAgent(ABC):
|
||||
"""Abstract Base Class defining the standard lifecycle for all ARTEMIS agents."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
name: str,
|
||||
role: str,
|
||||
description: str,
|
||||
config: AgentConfig | None = None,
|
||||
ctx: Any | None = None,
|
||||
):
|
||||
self.name = name
|
||||
self.role = role
|
||||
self.description = description
|
||||
self.config = config or AgentConfig()
|
||||
self.ctx: Any | None = ctx
|
||||
self._is_initialized = False
|
||||
|
||||
@property
|
||||
def driver(self) -> Any | None:
|
||||
"""Helper to get the active device driver from the context."""
|
||||
if self.ctx:
|
||||
return get_driver(self.ctx)
|
||||
return None
|
||||
|
||||
async def initialize(self, ctx: Any) -> None:
|
||||
"""Lifecycle hook: Called once when the agent is loaded into the workflow."""
|
||||
self.ctx = ctx
|
||||
self._is_initialized = True
|
||||
logger.info(f"Agent '{self.name}' ({self.role}) initialized.")
|
||||
|
||||
@abstractmethod
|
||||
async def process(self, state: Any) -> dict[str, Any]:
|
||||
"""Main execution hook: Receives state, reasons, interacts with tools, and returns state updates."""
|
||||
...
|
||||
|
||||
async def cleanup(self) -> None:
|
||||
"""Lifecycle hook: Called upon session or workflow termination."""
|
||||
self._is_initialized = False
|
||||
logger.debug(f"Agent '{self.name}' cleanup complete.")
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"<{self.__class__.__name__} name={self.name!r} role={self.role!r}>"
|
||||
|
||||
|
||||
class AgentRegistry:
|
||||
"""Central registry and factory for ARTEMIS agents."""
|
||||
|
||||
_registry: dict[str, type[BaseAgent]] = {}
|
||||
|
||||
@classmethod
|
||||
def register(cls, name: str | None = None):
|
||||
"""Decorator to register an agent class with a unique identifier."""
|
||||
|
||||
def decorator(subclass: type[BaseAgent]) -> type[BaseAgent]:
|
||||
reg_name = name or subclass.__name__.lower()
|
||||
cls._registry[reg_name] = subclass
|
||||
return subclass
|
||||
|
||||
return decorator
|
||||
|
||||
@classmethod
|
||||
def get_agent_class(cls, name: str) -> type[BaseAgent] | None:
|
||||
return cls._registry.get(name.lower())
|
||||
|
||||
@classmethod
|
||||
def create_agent(
|
||||
cls,
|
||||
name: str,
|
||||
ctx: Any | None = None,
|
||||
config: AgentConfig | None = None,
|
||||
**kwargs: Any,
|
||||
) -> BaseAgent:
|
||||
"""Instantiates an agent by registered name."""
|
||||
agent_cls = cls.get_agent_class(name)
|
||||
if not agent_cls:
|
||||
raise KeyError(
|
||||
f"Agent '{name}' is not registered. Available: {list(cls._registry.keys())}"
|
||||
)
|
||||
return agent_cls(ctx=ctx, config=config, **kwargs)
|
||||
|
||||
@classmethod
|
||||
def list_agents(cls) -> list[str]:
|
||||
return list(cls._registry.keys())
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"checker_system": "You are a precise and reliable verification expert (Checker). Your task is to determine whether the Operator has successfully completed the active subgoal by combining visual evidence from the current screenshot with data stored in persistent notes.",
|
||||
"checker_prompt": "# Inputs\n- Active Subgoal: {{ subgoal }}\n- Execution History:\n{{ plan_and_history }}\n- Dialogue History: {{ dialogue_history }}\n- Visual State: The provided screenshot.\n\n# Guidelines\n- **Visual + Data Verification**: You must combine the screenshot with note contents. If the subgoal involves extracting, saving, or recording information (e.g., \"save contacts\", \"record address\"), you **MUST** verify that the content actually exists in the note. Do not assume success just because the information is visible on the screen. If the signs of success are not obvious, you can check the history, especially the most recent steps, to see if the operator performed their own check and whether that check was reasonable, to determine if the subgoal was truly successful.\n- **Visual Priority (Anti-History-Bias)**: Your primary ground truth is the visual evidence on the **Current Screenshot**. The `Execution History` lists previous successes or failures, which may be outdated due to dynamic background recovery or self-healing. If the screenshot demonstrably shows the active subgoal's target state is achieved, you **MUST** mark `success: true`, regardless of previous failed steps. Trust what you see, not past FAILED texts.\n- **Constraint Verification**: For subgoals containing specific parameters (such as names, brands, topics, events, numbers, or dates), confirm that the visible text precisely reflects these specified details. A successful state is defined by a clear, direct match with the subgoal's constraints.\n- **Realistic Expectations**: You cannot scroll the screen. If a subgoal requires saving all items but only some are visible on the screen, verify that at least the **visible items** are recorded in the note. As long as the visible part is recorded and there are no obvious errors, consider it a success. Do not fail the task simply because off-screen content cannot be verified.\n- **Success Criteria**: Set `success` to `true` if the core objective of the subgoal is met and it does not block subsequent steps. Set `success` to `false` if the state is incorrect, required data was not saved, or the operator is stuck/looping.\n\n# Output Protocol\nInvoke the `submit_verification` tool with `success` (boolean) and `reason` (string) to submit your final decision. If you need to inspect note data first, use `read_note`."
|
||||
}
|
||||
@@ -0,0 +1,460 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from jinja2 import Template
|
||||
from langchain_core.messages import HumanMessage, SystemMessage, ToolMessage
|
||||
from langchain_core.tools import tool
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from artemis.constants import CHECKER_MAX_ITERATIONS
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.controllers.unified_controller import UnifiedMobileController
|
||||
from artemis.data_engine.trace import (
|
||||
TraceSpan,
|
||||
trace,
|
||||
trace_langchain_tool,
|
||||
)
|
||||
from artemis.services.llm import get_llm, invoke_llm_with_timeout_message
|
||||
from artemis.tools.index import get_tool_by_name
|
||||
from artemis.tools.scratchpad import get_read_note_tool_pure
|
||||
from artemis.tools.tool_wrapper import (
|
||||
get_tool_result_content,
|
||||
invoke_tool_with_injection,
|
||||
)
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.notes import get_note_file_path, get_notes_dir
|
||||
from artemis.utils.ocr_api import perform_ocr
|
||||
from artemis.utils.ocr_xml_fusion import (
|
||||
_crop_image_remove_status_bar,
|
||||
_detect_status_bar_height,
|
||||
_map_coordinates_back,
|
||||
fuse_ocr_with_xml,
|
||||
)
|
||||
from artemis.utils.task_tree import (
|
||||
build_plan_and_history,
|
||||
get_recent_subgoal_hashes,
|
||||
)
|
||||
from artemis.utils.verification import (
|
||||
append_verification_chat,
|
||||
get_verification_chat_path,
|
||||
get_verification_chat_rounds,
|
||||
read_verification_chat,
|
||||
)
|
||||
from artemis.utils.text import safe_extract_text
|
||||
from artemis.utils.visualization import format_minimal_list_with_elements
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class CheckerResult(BaseModel):
|
||||
"""Structured output contract for Checker verification."""
|
||||
|
||||
success: bool = Field(
|
||||
description="Whether the active subgoal was successfully completed based on visual and note evidence."
|
||||
)
|
||||
reason: str = Field(
|
||||
description="Detailed explanation of the result. If failed, specify what is missing or wrong and give actionable hints for the Operator."
|
||||
)
|
||||
|
||||
|
||||
@trace(type="agent", name="checker")
|
||||
async def run_async_check(
|
||||
ctx: ArtemisContext,
|
||||
subgoal_text: str,
|
||||
subgoal_hash: str | None = None,
|
||||
raw_perception_data: dict[str, Any] | None = None,
|
||||
latest_ui_hierarchy: list[Any] | None = None,
|
||||
):
|
||||
"""Runs the checker verification as a background task."""
|
||||
logger.info(f"Starting async Checker for: {subgoal_text}")
|
||||
|
||||
if not ctx.data_engine:
|
||||
logger.warning("DataEngine not available, skipping checker.")
|
||||
return {"status": "success", "reason": "DataEngine unavailable."}
|
||||
|
||||
notes_dir = get_notes_dir(ctx.data_engine.base_dir)
|
||||
status_path = notes_dir / "checker_status.json"
|
||||
|
||||
if ctx.execution_setup and ctx.execution_setup.disable_checker:
|
||||
logger.info("Checker is disabled via flag. Skipping.")
|
||||
try:
|
||||
status_path.write_text(
|
||||
json.dumps({"status": "success", "reason": "Checker disabled."}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
return {"status": "success", "reason": "Checker disabled."}
|
||||
|
||||
# 1. Set status to running
|
||||
try:
|
||||
status_path.write_text(
|
||||
json.dumps({"status": "running", "subgoal": subgoal_text}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
logger.info(f"Wrote checker status 'running' to {status_path}")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to write checker status: {e}")
|
||||
return {"status": "success", "reason": f"Status write error: {e}"}
|
||||
|
||||
# 2. Run Checker logic
|
||||
try:
|
||||
# Retrieve perception data: directly reuse upstream perception if available
|
||||
if isinstance(raw_perception_data, dict) and raw_perception_data.get("screenshot_b64"):
|
||||
screenshot_b64 = raw_perception_data["screenshot_b64"]
|
||||
width = raw_perception_data.get("width", 1080)
|
||||
height = raw_perception_data.get("height", 2400)
|
||||
fused_xml = latest_ui_hierarchy or raw_perception_data.get("xml_hierarchy") or []
|
||||
elif (
|
||||
hasattr(ctx, "latest_perception_data")
|
||||
and isinstance(ctx.latest_perception_data, dict)
|
||||
and ctx.latest_perception_data
|
||||
):
|
||||
screenshot_b64 = ctx.latest_perception_data.get("screenshot_b64")
|
||||
width = ctx.latest_perception_data.get("width", 1080)
|
||||
height = ctx.latest_perception_data.get("height", 2400)
|
||||
fused_xml = ctx.latest_perception_data.get("latest_ui_hierarchy") or []
|
||||
else:
|
||||
# Fallback to direct controller query
|
||||
controller = UnifiedMobileController(ctx)
|
||||
device_data = await controller.get_screen_data()
|
||||
screenshot_b64 = device_data.base64
|
||||
xml_hierarchy = device_data.elements
|
||||
width = device_data.width
|
||||
height = device_data.height
|
||||
|
||||
ocr_results = []
|
||||
try:
|
||||
status_bar_height = _detect_status_bar_height(xml_hierarchy, height)
|
||||
if status_bar_height > 0:
|
||||
cropped_b64, _, _ = _crop_image_remove_status_bar(
|
||||
screenshot_b64, status_bar_height
|
||||
)
|
||||
raw_ocr_results = await perform_ocr(cropped_b64)
|
||||
ocr_results = _map_coordinates_back(raw_ocr_results, status_bar_height)
|
||||
else:
|
||||
ocr_results = await perform_ocr(screenshot_b64)
|
||||
except Exception as e:
|
||||
logger.error(f"Checker OCR failed, proceeding without OCR: {e}")
|
||||
|
||||
fused_xml = fuse_ocr_with_xml(xml_hierarchy, ocr_results)
|
||||
|
||||
minimal_list, elements, labels = format_minimal_list_with_elements(fused_xml, width, height)
|
||||
|
||||
# Build task tree
|
||||
task_plan = "No task plan yet."
|
||||
task_plan_path = get_note_file_path(ctx.data_engine.base_dir, "task_plan")
|
||||
if task_plan_path.exists():
|
||||
try:
|
||||
task_plan = task_plan_path.read_text(encoding="utf-8")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to read task plan for Checker: {e}")
|
||||
|
||||
steps = ctx.data_engine.get_agent_friendly_steps() if ctx.data_engine else []
|
||||
|
||||
# Calculate exact subgoal hash from subgoal_text directly to avoid 'default' placeholder
|
||||
if not subgoal_hash or subgoal_hash == "default":
|
||||
clean_subgoal = safe_extract_text(subgoal_text).strip()
|
||||
subgoal_hash = hashlib.md5(clean_subgoal.encode("utf-8")).hexdigest()
|
||||
|
||||
keep_hashes = get_recent_subgoal_hashes(steps, subgoal_hash, ctx.data_engine.base_dir)
|
||||
|
||||
plan_and_history = build_plan_and_history(
|
||||
task_plan, steps, subgoal_hash, keep_subgoal_hashes=keep_hashes
|
||||
)
|
||||
|
||||
chat_path = get_verification_chat_path(ctx.data_engine.base_dir, subgoal_hash)
|
||||
turns = read_verification_chat(chat_path)
|
||||
|
||||
# Calculate round number
|
||||
max_op, max_chk = get_verification_chat_rounds(turns)
|
||||
round_num = max(max_op, max_chk + 1)
|
||||
if round_num == 0:
|
||||
round_num = 1
|
||||
|
||||
# Render dialogue history for prompt
|
||||
dialogue_lines = []
|
||||
for t in turns:
|
||||
role = "Operator" if t["role"] == "operator" else "Checker"
|
||||
dialogue_lines.append(f"**{role} (Round {t['round']})**:\n{t['content']}")
|
||||
dialogue_history = (
|
||||
"\n\n".join(dialogue_lines) if dialogue_lines else "No previous dialogue."
|
||||
)
|
||||
|
||||
prompts = {}
|
||||
prompts_path = Path(__file__).parent / "checker.json"
|
||||
if prompts_path.exists():
|
||||
try:
|
||||
prompts = json.loads(prompts_path.read_text(encoding="utf-8"))
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load checker prompts: {e}")
|
||||
|
||||
prompt_template = prompts.get(
|
||||
"checker_prompt",
|
||||
"You are the Checker. Your job is to verify if the current subtask"
|
||||
" was successfully completed.",
|
||||
)
|
||||
|
||||
full_prompt = Template(prompt_template).render(
|
||||
subgoal=subgoal_text,
|
||||
plan_and_history=plan_and_history,
|
||||
dialogue_history=dialogue_history if dialogue_history else "No previous dialogue.",
|
||||
minimal_list=minimal_list,
|
||||
)
|
||||
|
||||
llm = get_llm(ctx=ctx, name="checker")
|
||||
read_note = get_read_note_tool_pure(ctx)
|
||||
|
||||
@tool
|
||||
def submit_verification(
|
||||
success: bool,
|
||||
reason: str,
|
||||
) -> str:
|
||||
"""[TERMINAL] Submit the final verification outcome (success or failure with reason). This completes verification."""
|
||||
return f"Verification decision: success={success}, reason={reason}"
|
||||
|
||||
tools_to_bind = [
|
||||
trace_langchain_tool(read_note, ctx),
|
||||
submit_verification,
|
||||
]
|
||||
|
||||
content = [
|
||||
{"type": "text", "text": full_prompt},
|
||||
{"type": "text", "text": "--- Current Screenshot ---"},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/jpeg;base64,{screenshot_b64}"},
|
||||
},
|
||||
]
|
||||
|
||||
messages = [
|
||||
SystemMessage(
|
||||
content=prompts.get(
|
||||
"checker_system",
|
||||
"You are the Checker agent. Your objective is to accurately"
|
||||
" assess whether the active subgoal was successfully"
|
||||
" completed.",
|
||||
)
|
||||
),
|
||||
HumanMessage(content=content),
|
||||
]
|
||||
|
||||
max_iterations = (
|
||||
getattr(
|
||||
ctx.execution_setup,
|
||||
"checker_max_iterations",
|
||||
CHECKER_MAX_ITERATIONS,
|
||||
)
|
||||
if ctx.execution_setup
|
||||
else CHECKER_MAX_ITERATIONS
|
||||
)
|
||||
|
||||
success = True
|
||||
reason = "Checker completed."
|
||||
|
||||
for i in range(max_iterations):
|
||||
is_final_round = i == max_iterations - 1
|
||||
|
||||
if is_final_round:
|
||||
messages.append(
|
||||
HumanMessage(
|
||||
content=(
|
||||
"This is your final iteration; all tools are"
|
||||
" stripped, and you must provide your final structured verification result."
|
||||
)
|
||||
)
|
||||
)
|
||||
structured_llm = llm.with_structured_output(CheckerResult)
|
||||
structured_res: CheckerResult = await invoke_llm_with_timeout_message(
|
||||
structured_llm.ainvoke(messages)
|
||||
)
|
||||
if isinstance(structured_res, CheckerResult):
|
||||
success = structured_res.success
|
||||
reason = structured_res.reason
|
||||
elif isinstance(structured_res, dict):
|
||||
success = structured_res.get("success", True)
|
||||
reason = structured_res.get("reason", "Verification completed.")
|
||||
break
|
||||
else:
|
||||
if i > 0:
|
||||
messages.append(
|
||||
HumanMessage(
|
||||
content=(
|
||||
"You have not completed verification yet"
|
||||
f" (iteration {i + 1} of {max_iterations})."
|
||||
)
|
||||
)
|
||||
)
|
||||
active_llm = llm.bind_tools(tools=tools_to_bind)
|
||||
|
||||
async def run_stream():
|
||||
full_response = None
|
||||
async for chunk in active_llm.astream(messages):
|
||||
if full_response is None:
|
||||
full_response = chunk
|
||||
else:
|
||||
full_response += chunk
|
||||
return full_response
|
||||
|
||||
response = await invoke_llm_with_timeout_message(run_stream())
|
||||
|
||||
# Check if submit_verification was invoked
|
||||
submit_tc = next(
|
||||
(
|
||||
tc
|
||||
for tc in response.tool_calls
|
||||
if tc["name"].endswith("submit_verification")
|
||||
),
|
||||
None,
|
||||
)
|
||||
if submit_tc:
|
||||
tc_args = dict(submit_tc["args"])
|
||||
success = bool(tc_args.get("success", True))
|
||||
reason = str(tc_args.get("reason", "Verification submitted."))
|
||||
break
|
||||
|
||||
if not response.tool_calls:
|
||||
# Model answered in plain text; use structured_llm directly
|
||||
structured_llm = llm.with_structured_output(CheckerResult)
|
||||
structured_res: CheckerResult = await invoke_llm_with_timeout_message(
|
||||
structured_llm.ainvoke(messages + [response])
|
||||
)
|
||||
if isinstance(structured_res, CheckerResult):
|
||||
success = structured_res.success
|
||||
reason = structured_res.reason
|
||||
elif isinstance(structured_res, dict):
|
||||
success = structured_res.get("success", True)
|
||||
reason = structured_res.get("reason", "Verification completed.")
|
||||
break
|
||||
|
||||
messages.append(response)
|
||||
|
||||
async def run_tool(tc):
|
||||
tool_name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
|
||||
logger.info(f"Checker requested tool: {tool_name}")
|
||||
|
||||
result_str = ""
|
||||
tool_status = "success"
|
||||
try:
|
||||
if ":" in tool_name:
|
||||
tool_to_run = get_tool_by_name(tool_name, traced_tools)
|
||||
else:
|
||||
tool_to_run = next(
|
||||
(t for t in traced_tools if t.name == tool_name),
|
||||
None,
|
||||
)
|
||||
if tool_to_run:
|
||||
args = dict(tc["args"])
|
||||
with TraceSpan(name=tool_name, ctx=ctx) as span:
|
||||
span.payload = {"args": args}
|
||||
try:
|
||||
result_obj = await invoke_tool_with_injection(
|
||||
tool=tool_to_run,
|
||||
args=args,
|
||||
tool_call_id=tc["id"],
|
||||
)
|
||||
result_str = get_tool_result_content(result_obj)
|
||||
span.result = result_str
|
||||
if result_str.startswith("Error"):
|
||||
tool_status = "error"
|
||||
span.status = "failed"
|
||||
span.error = result_str
|
||||
except Exception as err:
|
||||
tool_status = "error"
|
||||
span.status = "failed"
|
||||
span.error = str(err)
|
||||
raise err
|
||||
else:
|
||||
result_str = f"Error: Tool {tool_name} not supported"
|
||||
tool_status = "error"
|
||||
except Exception as err:
|
||||
logger.error(f"Error running tool {tool_name}: {err}")
|
||||
result_str = f"Error running tool {tool_name}: {err}"
|
||||
tool_status = "error"
|
||||
|
||||
return ToolMessage(
|
||||
tool_call_id=tc["id"],
|
||||
content=result_str,
|
||||
status=tool_status,
|
||||
)
|
||||
|
||||
tool_outputs = await asyncio.gather(*(run_tool(tc) for tc in response.tool_calls))
|
||||
for tm in tool_outputs:
|
||||
messages.append(tm)
|
||||
|
||||
logger.info(
|
||||
f"Async Checker result for '{subgoal_text}': Success={success}, Reason={reason}"
|
||||
)
|
||||
|
||||
# Update status file
|
||||
status_path.write_text(
|
||||
json.dumps({"status": "success" if success else "failed", "reason": reason}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
# Clean up dialogue file if successful
|
||||
if success:
|
||||
if chat_path.exists():
|
||||
logger.info(f"Checker succeeded. Cleaning up dialogue file: {chat_path}")
|
||||
chat_path.unlink()
|
||||
|
||||
# If failed, revert status in task_plan.md and write feedback
|
||||
if not success:
|
||||
task_plan_path = get_note_file_path(ctx.data_engine.base_dir, "task_plan")
|
||||
if task_plan_path.exists():
|
||||
task_plan = task_plan_path.read_text(encoding="utf-8")
|
||||
lines = task_plan.split("\n")
|
||||
|
||||
for i in range(len(lines) - 1, -1, -1):
|
||||
if lines[i].startswith("- ["):
|
||||
task_content = safe_extract_text(lines[i][5:]).strip()
|
||||
safe_subgoal = safe_extract_text(subgoal_text).strip()
|
||||
if task_content == safe_subgoal:
|
||||
lines[i] = re.sub(r"^(- \[)[x/ ](\])", r"\g<1>/\g<2>", lines[i])
|
||||
logger.info(
|
||||
"Matched top-level subgoal by content."
|
||||
f" Reverted status to [/]: {subgoal_text}"
|
||||
)
|
||||
break
|
||||
task_plan_path.write_text("\n".join(lines), encoding="utf-8")
|
||||
|
||||
append_verification_chat(
|
||||
chat_path,
|
||||
"checker",
|
||||
f"Status: Failed.\nReason: {reason}",
|
||||
round_num,
|
||||
)
|
||||
|
||||
return {"status": "success" if success else "failed", "reason": reason}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Async Checker encountered exception (fail-open): {e}")
|
||||
status_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"status": "success",
|
||||
"reason": f"Checker Fail-Open: {e}",
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
return {"status": "success", "reason": f"Checker Fail-Open: {e}"}
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"system_prompt": [
|
||||
"# Diagnoser",
|
||||
"",
|
||||
"## Objective",
|
||||
"You are an expert AI diagnoser. Your objective is to pinpoint the root cause of mobile execution failures by cross-referencing visual evidence (videos/screenshots) with system telemetry (logs). Precisely answer the `Diagnostic Query` based on the provided context.",
|
||||
"",
|
||||
"## Core Guidelines",
|
||||
"- **Evidence-Based**: Base all conclusions strictly on facts from tools and context. Do not assume or hallucinate.",
|
||||
"- **Laser Focus**: Focus exclusively on the `Diagnostic Query`. Do not investigate unrelated issues.",
|
||||
"- **No Direct Operations**: Do not perform user actions or modify device state (e.g., tap, swipe, text input, BACK/HOME keys, or force-starting activities). Focus solely on analysis.",
|
||||
"- **Tool Constraints**: Video analysis is expensive. When calling the video analyzer, you MUST provide a specific, narrow timeframe to minimize cost.",
|
||||
"- **Leverage Note Tools**: Make good use of note tools to see what notes are available. When diagnosing, open and cross-reference relevant notes to ensure you have grasped all necessary information and context.",
|
||||
"- **Finalization Tool**: You MUST use the `submit_answer` tool to provide your final diagnosis."
|
||||
],
|
||||
"checker_feedback_template": [
|
||||
"--- Checker Feedback ---",
|
||||
"The Checker agent has provided feedback regarding the verification of the active subgoal:",
|
||||
"{checker_feedback}",
|
||||
""
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,575 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
import uuid
|
||||
|
||||
from langchain_core.messages import HumanMessage, SystemMessage, ToolMessage
|
||||
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.data_engine.trace import (
|
||||
CURRENT_TRACE_ID,
|
||||
TraceSpan,
|
||||
trace,
|
||||
trace_langchain_tool,
|
||||
)
|
||||
from artemis.graph.state import State
|
||||
from artemis.services.llm import get_llm, invoke_llm_with_timeout_message
|
||||
from artemis.tools.command_tool import get_run_short_adb_command_tool
|
||||
from artemis.tools.diagnoser_submit_answer_tool import get_submit_answer_tool
|
||||
from artemis.tools.index import get_tool_by_name
|
||||
from artemis.tools.log_tool import get_analyze_logs_tool
|
||||
from artemis.tools.mobile.read_hierarchy import get_ui_hierarchy_tool
|
||||
from artemis.tools.scratchpad import get_list_notes_tool, get_read_note_tool
|
||||
from artemis.tools.tool_wrapper import (
|
||||
get_tool_result_content,
|
||||
invoke_tool_with_injection,
|
||||
)
|
||||
from artemis.tools.video_tool import get_video_analyzer_tool
|
||||
from artemis.tools.wait_tool import get_wait_tool
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.task_tree import (
|
||||
build_plan_and_history,
|
||||
get_active_subgoal_hashes,
|
||||
get_recent_subgoal_hashes,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class Diagnoser:
|
||||
def __init__(self, ctx: ArtemisContext):
|
||||
self.ctx = ctx
|
||||
self.is_device_online = self._is_device_online()
|
||||
|
||||
def _is_device_online(self) -> bool:
|
||||
try:
|
||||
if self.ctx.adb_client is None:
|
||||
return False
|
||||
if getattr(self.ctx.adb_client, "is_online", True) is False:
|
||||
return False
|
||||
self.ctx.get_adb_client()
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
@trace(type="agent", name="diagnoser")
|
||||
async def run(self, prompt: str, state: State) -> str:
|
||||
"""Runs the Diagnoser and returns its summary."""
|
||||
|
||||
# 1. Load prompt
|
||||
prompt_path = Path(__file__).parent.joinpath("diagnoser.json")
|
||||
if not prompt_path.exists():
|
||||
raise FileNotFoundError(f"Prompt file not found at {prompt_path}")
|
||||
|
||||
with open(prompt_path, encoding="utf-8") as f:
|
||||
prompt_data = json.load(f)
|
||||
|
||||
system_prompt = prompt_data.get("system_prompt", "")
|
||||
if isinstance(system_prompt, list):
|
||||
system_prompt = "\n".join(system_prompt)
|
||||
checker_feedback_template = prompt_data.get("checker_feedback_template", "")
|
||||
if isinstance(checker_feedback_template, list):
|
||||
checker_feedback_template = "\n".join(checker_feedback_template)
|
||||
|
||||
# 2. Prepare LLM
|
||||
llm = get_llm(ctx=self.ctx, name="diagnoser")
|
||||
|
||||
log_tool = get_analyze_logs_tool(self.ctx)
|
||||
|
||||
read_note_tool = get_read_note_tool(self.ctx)
|
||||
list_notes_tool = get_list_notes_tool(self.ctx)
|
||||
adb_command_tool = get_run_short_adb_command_tool(self.ctx)
|
||||
wait_tool = get_wait_tool(self.ctx)
|
||||
ui_hierarchy_tool = get_ui_hierarchy_tool(self.ctx)
|
||||
submit_answer_tool = get_submit_answer_tool(self.ctx)
|
||||
|
||||
all_tools = [
|
||||
log_tool,
|
||||
read_note_tool,
|
||||
list_notes_tool,
|
||||
adb_command_tool,
|
||||
wait_tool,
|
||||
ui_hierarchy_tool,
|
||||
submit_answer_tool,
|
||||
]
|
||||
if not self.is_device_online:
|
||||
logger.info(
|
||||
"Diagnoser running in OFFLINE environment: stripping adb short"
|
||||
" command and wait tools."
|
||||
)
|
||||
all_tools = [
|
||||
t
|
||||
for t in all_tools
|
||||
if t.name not in ("run_adb_command", "wait")
|
||||
and not t.name.endswith(":run_adb_command")
|
||||
and not t.name.endswith(":wait")
|
||||
]
|
||||
|
||||
if self.ctx.execution_setup and self.ctx.execution_setup.video_recording_tools_enabled:
|
||||
video_tool = get_video_analyzer_tool(self.ctx, role="diagnoser")
|
||||
all_tools.append(video_tool)
|
||||
|
||||
traced_tools = [trace_langchain_tool(t, self.ctx) for t in all_tools]
|
||||
|
||||
# Bind custom tools
|
||||
base_llm = llm
|
||||
|
||||
# 4. Get context from DataEngine and State
|
||||
initial_goal = state.initial_goal
|
||||
|
||||
task_plan = "No task plan yet."
|
||||
steps = []
|
||||
subgoal_hash = "default"
|
||||
|
||||
if self.ctx.data_engine:
|
||||
notes_dir = Path(self.ctx.data_engine.base_dir) / "notes"
|
||||
task_plan_path = notes_dir / "task_plan.md"
|
||||
|
||||
if task_plan_path.exists():
|
||||
try:
|
||||
task_plan = task_plan_path.read_text(encoding="utf-8")
|
||||
|
||||
subgoal_hash, _ = get_active_subgoal_hashes(task_plan)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to parse active subgoal in Diagnoser: {e}")
|
||||
task_plan = f"Error reading task plan: {e}"
|
||||
|
||||
steps = self.ctx.data_engine.get_agent_friendly_steps()
|
||||
|
||||
# Build task tree
|
||||
plan_and_history = "No plan or history available."
|
||||
if self.ctx.data_engine and task_plan != "No task plan yet.":
|
||||
try:
|
||||
keep_hashes = get_recent_subgoal_hashes(
|
||||
steps,
|
||||
subgoal_hash,
|
||||
window_steps=self.ctx.agent_config.history_window_steps,
|
||||
)
|
||||
|
||||
plan_and_history = build_plan_and_history(
|
||||
task_plan,
|
||||
steps,
|
||||
subgoal_hash,
|
||||
keep_subgoal_hashes=keep_hashes,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to build plan and history in Diagnoser: {e}")
|
||||
plan_and_history = f"Error building plan and history: {e}"
|
||||
|
||||
# Get Checker Feedback
|
||||
checker_feedback = ""
|
||||
if self.ctx.data_engine:
|
||||
notes_dir = Path(self.ctx.data_engine.base_dir) / "notes"
|
||||
verification_chat_path = notes_dir / f"verification_chat_{subgoal_hash}.json"
|
||||
if verification_chat_path.exists():
|
||||
try:
|
||||
turns = json.loads(verification_chat_path.read_text(encoding="utf-8"))
|
||||
dialogue_lines = []
|
||||
for t in turns:
|
||||
role = "Operator" if t["role"] == "operator" else "Checker"
|
||||
dialogue_lines.append(f"**{role} (Round {t['round']})**:\n{t['content']}")
|
||||
checker_feedback = "\n\n".join(dialogue_lines)
|
||||
except Exception as e:
|
||||
logger.error(f"Error reading verification chat in Diagnoser: {e}")
|
||||
|
||||
# Get Current Screenshot and UI Hierarchy
|
||||
latest_screenshot_b64 = None
|
||||
minimal_list = ""
|
||||
|
||||
if state.latest_screenshot and os.path.exists(state.latest_screenshot):
|
||||
try:
|
||||
with open(state.latest_screenshot, "rb") as f:
|
||||
latest_screenshot_b64 = base64.b64encode(f.read()).decode("utf-8")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to read latest screenshot in Diagnoser: {e}")
|
||||
|
||||
if state.latest_ui_hierarchy:
|
||||
minimal_list = self._format_minimal_list(state.latest_ui_hierarchy)
|
||||
|
||||
# Construct rich human message
|
||||
content = []
|
||||
content.append({"type": "text", "text": f"Initial Goal: {initial_goal}\n"})
|
||||
content.append(
|
||||
{
|
||||
"type": "text",
|
||||
"text": f"--- Plan & History ---\n{plan_and_history}\n",
|
||||
}
|
||||
)
|
||||
|
||||
if checker_feedback and checker_feedback_template:
|
||||
formatted_feedback = checker_feedback_template.format(checker_feedback=checker_feedback)
|
||||
content.append({"type": "text", "text": formatted_feedback})
|
||||
|
||||
if minimal_list:
|
||||
content.append(
|
||||
{
|
||||
"type": "text",
|
||||
"text": f"--- Visible UI Elements ---\n{minimal_list}\n",
|
||||
}
|
||||
)
|
||||
|
||||
if latest_screenshot_b64:
|
||||
content.append({"type": "text", "text": "--- Current Screenshot ---"})
|
||||
content.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/jpeg;base64,{latest_screenshot_b64}"},
|
||||
}
|
||||
)
|
||||
|
||||
content.append({"type": "text", "text": f"--- Diagnostic Query ---\n{prompt}"})
|
||||
|
||||
current_messages = [
|
||||
SystemMessage(content=system_prompt),
|
||||
HumanMessage(content=content),
|
||||
]
|
||||
|
||||
self.ctx.background_jobs = {}
|
||||
|
||||
max_iterations = 30
|
||||
outcome = "No result"
|
||||
|
||||
try:
|
||||
for i in range(max_iterations):
|
||||
# 5. Agent Loop Injection & Final Turn Guard
|
||||
if i == max_iterations - 1:
|
||||
# Identify all jobs in the registry with status "running"
|
||||
running_jobs = [
|
||||
job
|
||||
for job in self.ctx.background_jobs.values()
|
||||
if job.get("status") == "running"
|
||||
]
|
||||
if running_jobs:
|
||||
# Await all associated task objects
|
||||
await asyncio.gather(
|
||||
*(job["task"] for job in running_jobs if job.get("task") is not None),
|
||||
return_exceptions=True,
|
||||
)
|
||||
|
||||
# Iterate through the background_jobs registry for all jobs where status is "completed" or "failed" and consumed is false
|
||||
new_tool_messages = []
|
||||
for job in self.ctx.background_jobs.values():
|
||||
if job.get("status") in (
|
||||
"completed",
|
||||
"failed",
|
||||
) and not job.get("consumed", False):
|
||||
tm_status = "success" if job["status"] == "completed" else "error"
|
||||
tm = ToolMessage(
|
||||
tool_call_id=job["tool_call_id"],
|
||||
content=job["result"] or "",
|
||||
status=tm_status,
|
||||
)
|
||||
new_tool_messages.append(tm)
|
||||
job["consumed"] = True
|
||||
|
||||
if new_tool_messages:
|
||||
current_messages.extend(new_tool_messages)
|
||||
|
||||
if i == max_iterations - 1:
|
||||
current_messages.append(
|
||||
HumanMessage(
|
||||
content=(
|
||||
"[WARNING] This is your final iteration. Call"
|
||||
" 'submit_answer' to provide your final"
|
||||
" diagnosis."
|
||||
)
|
||||
)
|
||||
)
|
||||
submit_tool = next(
|
||||
(t for t in traced_tools if t.name == "submit_answer"),
|
||||
None,
|
||||
)
|
||||
llm = base_llm.bind_tools(tools=[submit_tool] if submit_tool else [])
|
||||
else:
|
||||
if i > 0:
|
||||
current_messages.append(
|
||||
HumanMessage(
|
||||
content=(
|
||||
"You have not completed the diagnosis yet"
|
||||
f" (iteration {i + 1} of {max_iterations})."
|
||||
)
|
||||
)
|
||||
)
|
||||
llm = base_llm.bind_tools(tools=traced_tools)
|
||||
|
||||
async def run_stream():
|
||||
full_response = None
|
||||
trace_id = CURRENT_TRACE_ID.get()
|
||||
async for chunk in llm.astream(current_messages):
|
||||
if full_response is None:
|
||||
full_response = chunk
|
||||
else:
|
||||
full_response += chunk
|
||||
|
||||
pass
|
||||
return full_response
|
||||
|
||||
with TraceSpan(name="gemini_diagnoser_call", ctx=self.ctx) as span:
|
||||
span.payload = {
|
||||
"context_components": {
|
||||
"system_prompt": system_prompt,
|
||||
"plan_and_history": plan_and_history,
|
||||
"checker_feedback": checker_feedback,
|
||||
"visible_ui_elements": minimal_list,
|
||||
"diagnostic_query": prompt,
|
||||
}
|
||||
}
|
||||
response = await invoke_llm_with_timeout_message(run_stream())
|
||||
span.result = (
|
||||
response.content if hasattr(response, "content") else str(response)
|
||||
)
|
||||
|
||||
if hasattr(response, "usage_metadata") and response.usage_metadata is not None:
|
||||
usage = response.usage_metadata
|
||||
if isinstance(usage, dict):
|
||||
prompt_token_count = usage.get("prompt_token_count")
|
||||
candidates_token_count = usage.get("candidates_token_count")
|
||||
cached_content_token_count = usage.get("cached_content_token_count")
|
||||
else:
|
||||
prompt_token_count = getattr(usage, "prompt_token_count", None)
|
||||
candidates_token_count = getattr(usage, "candidates_token_count", None)
|
||||
cached_content_token_count = getattr(
|
||||
usage, "cached_content_token_count", None
|
||||
)
|
||||
|
||||
if span.payload is None:
|
||||
span.payload = {}
|
||||
|
||||
span.payload["token_usage"] = {
|
||||
"prompt_token_count": prompt_token_count,
|
||||
"candidates_token_count": candidates_token_count,
|
||||
"cached_content_token_count": (cached_content_token_count),
|
||||
}
|
||||
|
||||
if response.content:
|
||||
outcome = response.content
|
||||
|
||||
if not response.tool_calls:
|
||||
break
|
||||
|
||||
submit_call = next(
|
||||
(
|
||||
tc
|
||||
for tc in response.tool_calls
|
||||
if tc["name"] == "submit_answer" or tc["name"].endswith(":submit_answer")
|
||||
),
|
||||
None,
|
||||
)
|
||||
if submit_call:
|
||||
args = submit_call["args"]
|
||||
analysis = args.get("analysis", "No analysis provided.")
|
||||
steps = args.get("actionable_steps", [])
|
||||
|
||||
submit_tool = next(
|
||||
(
|
||||
t
|
||||
for t in traced_tools
|
||||
if t.name == "submit_answer" or t.name.endswith(":submit_answer")
|
||||
),
|
||||
None,
|
||||
)
|
||||
if submit_tool:
|
||||
submit_tool.invoke(args)
|
||||
|
||||
outcome = f"Analysis: {analysis} | Actionalble Steps: "
|
||||
for step in steps:
|
||||
outcome += f"{step} "
|
||||
break
|
||||
|
||||
if i == max_iterations - 1:
|
||||
tool_names = ", ".join(tc["name"] for tc in response.tool_calls)
|
||||
warning = (
|
||||
"\n\n[Warning: Diagnoser reached the maximum iteration"
|
||||
f" limit of {max_iterations} and could not execute"
|
||||
f" tool(s): {tool_names}]"
|
||||
)
|
||||
if outcome == "No result":
|
||||
outcome = (
|
||||
"Diagnoser reached the maximum iteration limit of"
|
||||
f" {max_iterations} while attempting to call tools:"
|
||||
f" {tool_names}"
|
||||
)
|
||||
else:
|
||||
outcome += warning
|
||||
break
|
||||
|
||||
current_messages.append(response)
|
||||
|
||||
async def run_tool(tc):
|
||||
tool_name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
|
||||
logger.info(f"Diagnoser requested tool: {tool_name}")
|
||||
|
||||
result = ""
|
||||
status = "success"
|
||||
try:
|
||||
if tool_name == "video_analyzer":
|
||||
job_id = f"job_{uuid.uuid4().hex}"
|
||||
self.ctx.background_jobs[job_id] = {
|
||||
"tool_call_id": tc["id"],
|
||||
"status": "running",
|
||||
"result": None,
|
||||
"consumed": False,
|
||||
"task": None,
|
||||
}
|
||||
if self.ctx.data_engine:
|
||||
self.ctx.data_engine.register_background_task(
|
||||
task_id=job_id,
|
||||
summary=(f"Diagnoser Video Analyzer Tool Call: {tc['id']}"),
|
||||
)
|
||||
|
||||
async def run_bg_job():
|
||||
try:
|
||||
tool_to_run = next(
|
||||
(t for t in traced_tools if t.name == "video_analyzer"),
|
||||
None,
|
||||
)
|
||||
if tool_to_run:
|
||||
args = dict(tc["args"])
|
||||
result_obj = await invoke_tool_with_injection(
|
||||
tool=tool_to_run,
|
||||
args=args,
|
||||
tool_call_id=tc["id"],
|
||||
state=state,
|
||||
)
|
||||
result_content = get_tool_result_content(result_obj)
|
||||
self.ctx.background_jobs[job_id]["result"] = result_content
|
||||
self.ctx.background_jobs[job_id]["status"] = "completed"
|
||||
if self.ctx.data_engine:
|
||||
self.ctx.data_engine.unregister_background_task(
|
||||
task_id=job_id,
|
||||
status="completed",
|
||||
)
|
||||
else:
|
||||
self.ctx.background_jobs[job_id]["result"] = (
|
||||
"Error: video_analyzer tool not found"
|
||||
)
|
||||
self.ctx.background_jobs[job_id]["status"] = "failed"
|
||||
if self.ctx.data_engine:
|
||||
self.ctx.data_engine.unregister_background_task(
|
||||
task_id=job_id, status="failed"
|
||||
)
|
||||
except asyncio.CancelledError:
|
||||
logger.info(f"Background job {job_id} cancelled.")
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Background job {job_id} failed: {e}")
|
||||
self.ctx.background_jobs[job_id]["result"] = str(e)
|
||||
self.ctx.background_jobs[job_id]["status"] = "failed"
|
||||
if self.ctx.data_engine:
|
||||
self.ctx.data_engine.unregister_background_task(
|
||||
task_id=job_id, status="failed"
|
||||
)
|
||||
|
||||
task = asyncio.create_task(run_bg_job())
|
||||
self.ctx.background_jobs[job_id]["task"] = task
|
||||
|
||||
return ToolMessage(
|
||||
tool_call_id=tc["id"],
|
||||
content=(
|
||||
f"Video analyzer started. Job ID: {job_id}."
|
||||
" [Warning] No background logs, status"
|
||||
" updates, or errors will be logged."
|
||||
),
|
||||
status="success",
|
||||
)
|
||||
|
||||
if ":" in tool_name:
|
||||
tool_to_run = get_tool_by_name(tool_name, traced_tools)
|
||||
else:
|
||||
tool_to_run = next(
|
||||
(t for t in traced_tools if t.name == tool_name),
|
||||
None,
|
||||
)
|
||||
if tool_to_run:
|
||||
args = dict(tc["args"])
|
||||
with TraceSpan(name=tool_name, ctx=self.ctx) as span:
|
||||
span.payload = {"args": args}
|
||||
try:
|
||||
result_obj = await invoke_tool_with_injection(
|
||||
tool=tool_to_run,
|
||||
args=args,
|
||||
tool_call_id=tc["id"],
|
||||
state=state,
|
||||
)
|
||||
result = get_tool_result_content(result_obj)
|
||||
if isinstance(result, list):
|
||||
result_str = "\n".join(map(str, result))
|
||||
elif not isinstance(result, str):
|
||||
result_str = str(result)
|
||||
else:
|
||||
result_str = result
|
||||
span.result = result
|
||||
if result_str.startswith("Error"):
|
||||
status = "error"
|
||||
span.status = "failed"
|
||||
span.error = result
|
||||
except Exception as e:
|
||||
status = "error"
|
||||
span.status = "failed"
|
||||
span.error = str(e)
|
||||
raise e
|
||||
else:
|
||||
result = f"Error: Tool {tool_name} not supported"
|
||||
status = "error"
|
||||
except Exception as e:
|
||||
logger.error(f"Error running tool {tool_name}: {e}")
|
||||
result = f"Error running tool {tool_name}: {e}"
|
||||
status = "error"
|
||||
|
||||
return ToolMessage(
|
||||
tool_call_id=tc["id"],
|
||||
content=result,
|
||||
status=status,
|
||||
)
|
||||
|
||||
# Execute all requested tools in parallel
|
||||
active_tool_calls = [
|
||||
tc
|
||||
for tc in response.tool_calls
|
||||
if (tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"])
|
||||
!= "google_search"
|
||||
]
|
||||
tool_outputs = await asyncio.gather(*(run_tool(tc) for tc in active_tool_calls))
|
||||
|
||||
for tm in tool_outputs:
|
||||
current_messages.append(tm)
|
||||
finally:
|
||||
# 6. Cleanup on Exit
|
||||
for job in self.ctx.background_jobs.values():
|
||||
if job.get("status") == "running" and job.get("task") is not None:
|
||||
job["task"].cancel()
|
||||
|
||||
return outcome
|
||||
|
||||
def _format_minimal_list(self, fused_xml: list[dict[str, Any]]) -> str:
|
||||
"""Formats fused XML into a minimal structured list."""
|
||||
lines = []
|
||||
for i, node in enumerate(fused_xml):
|
||||
text = node.get("text") or node.get("content-desc") or ""
|
||||
bounds = node.get("bounds")
|
||||
|
||||
ocr_elements = node.get("ocr_elements")
|
||||
if ocr_elements:
|
||||
for ocr in ocr_elements:
|
||||
lines.append(f"[{i}] OCR Text: '{ocr['text']}' | Bounds: {ocr['bounds']}")
|
||||
elif text.strip():
|
||||
lines.append(f"[{i}] Text: '{text.strip()}' | Bounds: {bounds}")
|
||||
|
||||
return "\n".join(lines)
|
||||
@@ -0,0 +1,72 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Constants and prompt templates for the UI Explorer agent."""
|
||||
|
||||
EXPLORE_DESCRIPTIONS = {
|
||||
"flash": {
|
||||
"description": (
|
||||
"[EXPLORER] Call this to ask the UI Explorer agent to "
|
||||
"locate coordinates on the screen layout by object detection."
|
||||
),
|
||||
"query_description": (
|
||||
"Pipe-separated list of target element descriptions, semantic"
|
||||
" action terms, or color descriptors to search for concurrently"
|
||||
" (e.g., 'red settings button | gear icon')."
|
||||
),
|
||||
"rule_prompt": (
|
||||
"Use the explorer tool to quickly check for visual icons, buttons, or text by name."
|
||||
),
|
||||
},
|
||||
"pro": {
|
||||
"description": (
|
||||
"[EXPLORER] Call this to ask the UI Explorer agent to locate"
|
||||
" coordinates on the screen layout by executing XML, OCR,"
|
||||
" coordinate search, and object detection."
|
||||
),
|
||||
"query_description": (
|
||||
"The target element or information to search for, including descriptions."
|
||||
),
|
||||
"rule_prompt": (
|
||||
"Call the explorer tool to search, OCR, or inspect it (maximum {max_iterations} tries)."
|
||||
),
|
||||
"version_prompt": (
|
||||
"You are running with a maximum of {max_iterations} turns: on your"
|
||||
" final turn, you will receive a warning reminding you to call"
|
||||
" submit_answer."
|
||||
),
|
||||
},
|
||||
"ultra": {
|
||||
"description": (
|
||||
"[EXPLORER] Full visual reasoning agent. Deeply inspects screen"
|
||||
" regions and utilizes all perception/pixel-level tools with a high"
|
||||
" execution limit (up to {max_iterations} turns) for"
|
||||
" high-difficulty or layout-critical tasks."
|
||||
),
|
||||
"query_description": (
|
||||
"The target element or information to search for, including descriptions."
|
||||
),
|
||||
"rule_prompt": (
|
||||
"For highly complex visual searches, details validation, or layout"
|
||||
" verification, invoke the explorer tool to perform multi-turn"
|
||||
" visual reasoning and detailed coordinate checks (maximum"
|
||||
" {max_iterations} tries)."
|
||||
),
|
||||
"version_prompt": (
|
||||
"You are running with a maximum of {max_iterations} turns: on your"
|
||||
" final turn, you will receive a warning reminding you to call"
|
||||
" submit_answer."
|
||||
),
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"IDENTITY": "You are the agentic UI Explorer designed to identify objects, scan text, and extract UI attributes on a phone screen.",
|
||||
"OPERATING PRINCIPLES": [
|
||||
"**Confidence-Driven Cost-Benefit Strategy**: Maximize localization accuracy while minimizing execution latency and tool calls:",
|
||||
" - **Direct Submit (High Confidence)**: If you are highly confident in the target coordinates from the initial annotated screenshot and minimal list, call `submit_answer` immediately in Turn 1 to save time.",
|
||||
" - **Cost-Benefit Turn Planning (Low Confidence / Ambiguity)**: If you are not highly confident. Consider structuring your exploration turns by weighing the latency/cost of each tool against the expected accuracy payoff:",
|
||||
" - **Parallel Context Gathering**: MUST parallel calling all perception tools in the first turn to gather broad screen context.",
|
||||
" - **Targeted Verification**: If micro-details or coordinates are still ambiguous, zoom in on promising candidate areas (parallel tool calls). Note that a single zoom-in or multi-zoom-in operation takes one turn, plan your verification steps accordingly.",
|
||||
" - **Finalization**: Submit coordinates using `submit_answer`. Stop exploring as soon as further tool calls would yield negligible accuracy gains relative to their latency.",
|
||||
"**Tool Capabilities**: You have three classes of tools: `Perception` tools (to extract and locate elements, coordinates, or inspect specific regions), `Transformation` tools (for pixel-level image editing), and the `Finalization` tool (to submit final answers and terminate). Note that in your final turn, all other tools are stripped away, leaving only the finalization tool, forcing you to submit your final answer.",
|
||||
"**Parallel vs Sequential Invocations**: You are fully supported to call perception tools in parallel (within a single turn) to expedite your search. However, transformation tools are expected to be called sequentially (since they transform the image for subsequent perception steps).",
|
||||
"**Multimodal Perception**: You have access to images marked with bounding boxes and labels. Use both the visual information in the images and the structured text lists to make decisions.",
|
||||
"**Image Pool Tracking**: Images are managed in an Image Pool (starting with 'img_0'). Transformations via transformation tools will register new image IDs (e.g., 'img_1').",
|
||||
"**Prefix System**: Elements in the screenshots are labeled with prefixes returned by your tools (e.g., Numbers for initial UI, letters for tool outputs). Always reference these precise labels (e.g., \"I choose S2\") when making decisions.",
|
||||
"**Contextual Awareness**: Always prioritize checking the context feedback to learn from previous mistakes.",
|
||||
"**Ask Perception Tool**: Call the unified `ask_perception_tool` specifying `search_query`, `nx`, `ny`, and `detect_queries` to perform text lookups, coordinate queries, and visual detection concurrently.",
|
||||
"**Locating Components**: Locate target components on screen by matching the X-labels (from text/coordinate searches) and O-labels (from object detection) in the text response with their corresponding green, blue, or red indicator dots on the output images."
|
||||
],
|
||||
"BEHAVIORAL GUIDELINES & SOFT CONSTRAINTS": [
|
||||
"**Avoid Hallucination**: Avoid guessing coordinates. If you are unable to locate the target element on the screen after trying all perception options, submit `[]` in `candidates` and document your observations as specified in the `submit_answer` schema.",
|
||||
"**Ambiguity Resolution**: If the screen contains multiple identical elements or several candidates that match the query (e.g., three \"Add\" buttons), you are encouraged to return a ranked list of all viable candidates (up to 10 candidates) via `submit_answer`, ordered by your confidence, with brief descriptions to distinguish them.",
|
||||
"**Semantic & Icon Synonym Fallback**: Literal exact matches are ideal, but semantic equivalents and synonyms are fully acceptable. If a text element is requested but only a corresponding icon is available, you should attempt to map the synonym and locate the icon (e.g., map \"Search\" to a magnifying glass icon, map \"Add\" to \"+\" symbol)."
|
||||
],
|
||||
"STRICT CONSTRAINTS": [
|
||||
"**Validation Standards**: All submitted coordinates `[nx, ny]` for candidates in `submit_answer` must strictly be integers in the `[0-1000]` range inclusive. If your parameters fail business validation rules, you will receive dynamic error responses in your execution loop, allowing you to self-correct and re-submit."
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,118 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Universal Context Compressor for Artemis Flash profile.
|
||||
|
||||
Replaces pruned historical screenshots with high-density visual step summaries
|
||||
when ready, while seamlessly falling back to standard pruned action text if
|
||||
the background summary is still in flight.
|
||||
"""
|
||||
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.messages import BaseMessage
|
||||
|
||||
from artemis.agents.flash.summarizer import VisualStepSummarizer
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def compress_flash_messages(
|
||||
messages: list[BaseMessage],
|
||||
summarizer: VisualStepSummarizer | None = None,
|
||||
prune_history_xml: bool = True,
|
||||
) -> None:
|
||||
"""Compresses multi-turn conversation messages in-place.
|
||||
|
||||
Rules:
|
||||
1. Keeps the latest screenshot and live UI hierarchy completely intact.
|
||||
2. For earlier steps, if a VisualStepSummarizer has completed the step's summary,
|
||||
replaces the intermediate image block with the objective summary text.
|
||||
3. If the background summary is not ready yet, gracefully falls back to the existing
|
||||
behavior (omitting the image block and retaining the original action text).
|
||||
4. If prune_history_xml is True, removes heavy outdated UI Element lists from past turns.
|
||||
"""
|
||||
if not messages:
|
||||
return
|
||||
|
||||
# 1. Identify the last message containing an active screenshot
|
||||
last_img_msg_idx = -1
|
||||
for idx in range(len(messages) - 1, -1, -1):
|
||||
msg = messages[idx]
|
||||
msg_content = getattr(msg, "content", None)
|
||||
if isinstance(msg_content, list):
|
||||
has_image = any(
|
||||
isinstance(block, dict) and block.get("type") in ("image_url", "image")
|
||||
for block in msg_content
|
||||
)
|
||||
if has_image:
|
||||
last_img_msg_idx = idx
|
||||
break
|
||||
|
||||
# 2. Iterate through historical messages and apply compression/fallback
|
||||
tool_step_counter = 0
|
||||
for idx in range(len(messages)):
|
||||
msg = messages[idx]
|
||||
msg_content = getattr(msg, "content", None)
|
||||
if not isinstance(msg_content, list):
|
||||
continue
|
||||
|
||||
# Keep the latest live screen intact
|
||||
if idx == last_img_msg_idx:
|
||||
continue
|
||||
|
||||
# Count tool steps strictly corresponding to FlashRunner action turns
|
||||
is_tool_msg = getattr(msg, "tool_call_id", None) is not None
|
||||
if is_tool_msg:
|
||||
tool_step_counter += 1
|
||||
current_step_num = tool_step_counter
|
||||
else:
|
||||
current_step_num = None
|
||||
|
||||
new_blocks: list[dict[str, Any]] = []
|
||||
summary_injected = False
|
||||
|
||||
for b in msg_content:
|
||||
if not isinstance(b, dict):
|
||||
continue
|
||||
b_type = b.get("type", "")
|
||||
b_text = b.get("text", "")
|
||||
|
||||
# Filter heavy outdated XML hierarchy from past steps to save tokens
|
||||
if prune_history_xml and b_type == "text" and "--- UI Element List ---" in b_text:
|
||||
continue
|
||||
|
||||
if b_type in ("image_url", "image"):
|
||||
# Check if this is a tool execution step and background summarizer has a ready summary
|
||||
if (
|
||||
current_step_num
|
||||
and summarizer
|
||||
and summarizer.has_summary(current_step_num)
|
||||
and not summary_injected
|
||||
):
|
||||
summary_text = summarizer.get_summary(current_step_num)
|
||||
if summary_text:
|
||||
new_blocks.append({"type": "text", "text": summary_text})
|
||||
summary_injected = True
|
||||
# If summarizer is not ready or initial screen, gracefully omit the image block
|
||||
continue
|
||||
|
||||
new_blocks.append(b)
|
||||
|
||||
# Defensive fallback: never leave ToolMessage or HumanMessage content empty
|
||||
if not new_blocks:
|
||||
new_blocks = [{"type": "text", "text": "Action completed." if is_tool_msg else ""}]
|
||||
|
||||
msg.content = new_blocks
|
||||
@@ -0,0 +1,41 @@
|
||||
# ROLE & OBJECTIVE
|
||||
You are **ARTEMIS Flash Runner**, an autonomous and highly efficient Android Device Execution Agent. Your goal is to accomplish the user's objective on the device.
|
||||
|
||||
**Objective: {{ goal }}**
|
||||
|
||||
---
|
||||
|
||||
# 1. COGNITIVE PROTOCOL
|
||||
You operate in a dynamic observation-reasoning-execution loop. **CRITICAL RULE: YOU MUST NEVER INVOKE A TOOL SILENTLY!** In every single turn, you **MUST FIRST output a natural language reasoning paragraph BEFORE generating any tool call.**
|
||||
|
||||
- **Turn 1 Only (Initial Planning)**: Briefly decompose the overall `{{ goal }}` into high-level visual **Milestones** in your text output. Do not repeat this macro breakdown in subsequent turns.
|
||||
|
||||
---
|
||||
|
||||
# 2. DYNAMIC ENVIRONMENT & TIME AWARENESS
|
||||
Keep in mind that the mobile device is an asynchronous, constantly evolving environment. However, your actions entail an inherent "turn latency" (a 3–7 second delay caused by model inference and network transmission between normal consecutive actions).
|
||||
- **State Drift & Progression Tolerance**: While you think, video playback advances, banners scroll, and web pages finish loading. If the latest observation has progressed further than expected (e.g., an ad automatically skipped or a redirect landed), **do not treat it as an error**. Adapt smoothly to the live screen in front of you.
|
||||
- **Timed Waiting & Transitions**: When facing mid-transition animations, loading spinners, required watch durations, or ad countdown timers, use the precise `wait_for_delay` command to allow time to pass or the UI state to stabilize. Some transition states are subtle and may even resemble a phone bug or system freeze; it is always advisable to wait first, and only consider retrying or choosing an alternative path if the state remains broken.
|
||||
- **Time-Sensitive Tasks & Transient UI**: First, you must understand that each of your decision turns takes several seconds of model inference and network delay, while the mobile device state is continuously changing. A target element that you see on the current screenshot (e.g., auto-dismissing overlays, temporary message bars, menus or control bars with short timeouts) may have already vanished by the time your physical click command reaches the device! This is one of the most common reasons why the UI does not change as expected after a single step. We call these **Time-Sensitive Tasks**.
|
||||
- *Mental Model*: When facing these tasks, any attempt to interact step-by-step (such as tapping once to wake up a menu/bar and waiting for the next turn to tap the target button) WILL fail because the element will auto-dismiss during your decision delay. You MUST combine the summon action on the trigger element (`Trigger`) and the target action (`Target`) into a single atomic sequence.
|
||||
- *Chained Execution*: Use `click_sequence(sequence=[[trigger_x, trigger_y], [target_x, target_y]])` in `0-1000` normalized scale to execute both taps consecutively with a 50ms delay, catching the transient element before the timeout closes it.
|
||||
|
||||
---
|
||||
|
||||
# 3. STAGNATION & LOOP PREVENTION
|
||||
- **Retrying is not the only option.**: If you perform the same action on the same target for two consecutive turns and the screen or XML shows **no substantial change**, further retries are unlikely to be useful; the probability of the execution tool itself malfunctioning is very low, so you should pause and consider the underlying reason.
|
||||
- **Flexibly Adjust Strategies:** Always prioritize resolving the immediate issue. If you encounter overlays or pop-ups blocking input, or if an element is unresponsive, actively consider alternative ways to complete the task. If an action fails to proceed as expected, systematically consider whether the issue lies with the device itself, the operational path, or a change in state occurring between your actual interaction and the screenshot (Time-Sensitive Tasks).
|
||||
|
||||
---
|
||||
|
||||
# 4. ACTION & MEMORY RULES
|
||||
1. **Normalized Coordinates (`0-1000` Scale)**: Always use normalized `[x, y]` coordinates in `0-1000` scale for any location-based target (e.g., `target=[500, 600]` or `click_sequence=[[300, 400], [500, 600]]`).
|
||||
2. **Do not determine element coordinates based on guesswork.** If the element is visible in the screenshot but its coordinates are missing from the provided XML tree, please use the `ask_explorer` tool to locate the element first.
|
||||
3. **Screenshot Pruning Awareness**: To optimize speed, the system **prunes old intermediate screenshots from your history**—you only ever see the *latest* live screenshot and UI tree. Therefore, if you spot critical text in an earlier step that you will need later (e.g., verification codes, specific titles, search lists), **you must explicitly verbalize and write those details into your text thinking/notes** so they persist across turns.
|
||||
4. **App Launching**: To start or reset an application, always use `manage_app(action="launch", app_name="...")` or `action="stop"`. Do not manually swipe across home screens.
|
||||
|
||||
---
|
||||
|
||||
# 5. TERMINATION PROTOCOL (`report_task_status`)
|
||||
- **Completion (`status="completed"`)**: ONLY call `report_task_status(status="completed", ...)` after you have **visually verified** from the latest screen that the user's objective (`{{ goal }}`) is fully achieved.
|
||||
- **Failure (`status="failed"`)**: If the task is genuinely impossible due to paywalls, unrecoverable crashes, or hitting an absolute dead end after sensible fallbacks, call `report_task_status(status="failed", ...)` with the exact blocking reason.
|
||||
@@ -0,0 +1,41 @@
|
||||
# ROLE & OBJECTIVE
|
||||
You are the Step Summarizer for an Android UI automation agent. Your task is to synthesize the Before/After screenshots and physical action into a concise, high-information-density, strictly objective first-person historical memory.
|
||||
|
||||
---
|
||||
|
||||
# PERSPECTIVE & FORMAT CONSTRAINTS
|
||||
1. **First-Person Perspective**: Write strictly from the agent's first-person perspective using **"I"** (e.g., "In Step {{ step_number }}, I tapped...", "I swiped up on... and observed..."). NEVER use third-person terms like "The agent", "The operator", or "The system".
|
||||
2. **Single Continuous Paragraph**: Your output MUST be exactly **one compact, continuous paragraph** (1–3 sentences, 35–65 words).
|
||||
3. **No Lists or Formatting**: Do NOT use bullet points, numbered lists, markdown headers, bold labels, or line breaks in your output.
|
||||
|
||||
---
|
||||
|
||||
# CORE OBSERVATION RULES
|
||||
1. **Zero Subjective Validation (STRICT)**:
|
||||
- Absolutely avoid declaring semantic goal achievement, subjective success, or final failure. In historical compression, subjective assumptions cause future turns to falsely believe a subgoal was definitively accomplished or permanently blocked.
|
||||
- **STRICTLY BANNED WORDS**: `successfully`, `completed`, `achieved`, `entered`, `navigated to`, `arrived at`, `failed`, `unsuccessful`, `could not`, `impossible`.
|
||||
2. **Action & Visual Delta Structure**:
|
||||
- **Target & Action**: Identify what control/button was targeted (referencing the red visual indicator on the BEFORE screen) and the physical action performed.
|
||||
- **Objective Transition**: Describe the physical screen change (e.g., page transitioned, modal dialog opened, list scrolled revealing new items, checkbox/toggle toggled, keyboard appeared).
|
||||
- **Preserve Critical Data & Verifications**: Faithfully transcribe visible text, toast alerts, error banners, verification codes, prices, account names, or tracking numbers that appeared on the AFTER screen.
|
||||
|
||||
---
|
||||
|
||||
# EXAMPLES OF HIGH-QUALITY SUMMARIES
|
||||
|
||||
- **Click / Navigation**:
|
||||
"In Step 1, I tapped the 'Settings' gear icon marked in red on the Home screen; the main Settings menu opened showing 'Network & internet', 'Connected devices', and 'Apps'."
|
||||
|
||||
- **Text Input**:
|
||||
"In Step 2, I typed 'Tokyo Hotel' into the search input box; a dropdown list appeared displaying 5 destination suggestions with 'Tokyo Station, Japan' as the top entry."
|
||||
|
||||
- **Swipe / Scroll**:
|
||||
"In Step 3, I swiped up along the flight results list; the screen scrolled downward approximately one page, revealing 3 additional evening flights starting from '$420'."
|
||||
|
||||
- **Modal / Alert State**:
|
||||
"In Step 4, I tapped the 'Confirm Booking' button; an alert dialog titled 'Payment Method Required' appeared over the view with an 'Add Card' option."
|
||||
|
||||
---
|
||||
|
||||
# TASK
|
||||
Generate the single-paragraph first-person summary for Step {{ step_number }} following all constraints above:
|
||||
@@ -0,0 +1,524 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Universal Multi-Model FlashRunner for Artemis.
|
||||
|
||||
Executes autonomous reactive mobile workflows across Google Gemini,
|
||||
OpenAI GPT-4o/o3, Anthropic Claude 3.5/3.7, and OpenRouter endpoints.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
from pathlib import Path
|
||||
import uuid
|
||||
|
||||
from jinja2 import Template
|
||||
from langchain_core.messages import (
|
||||
BaseMessage,
|
||||
HumanMessage,
|
||||
SystemMessage,
|
||||
ToolMessage,
|
||||
)
|
||||
|
||||
from artemis.agents.flash.context_compressor import compress_flash_messages
|
||||
from artemis.agents.flash.summarizer import VisualStepSummarizer
|
||||
from artemis.agents.validator.tool_declarations import (
|
||||
ACTION_TOOL_NAMES,
|
||||
ASK_EXPLORER_TOOL,
|
||||
CLICK_SEQUENCE_TOOL,
|
||||
MobileActionExecutor,
|
||||
REPORT_TASK_STATUS_TOOL,
|
||||
VALIDATOR_TOOLS_DECLARATION,
|
||||
capture_screenshot_and_parse_ui,
|
||||
prune_intermediate_screenshots,
|
||||
)
|
||||
from artemis.config import StepSummarizerConfig, load_agent_config
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.controllers.unified_controller import UnifiedMobileController
|
||||
from artemis.data_engine.trace import trace
|
||||
from artemis.graph.perception import _check_injected_instruction_file
|
||||
from artemis.graph.state import State
|
||||
from artemis.services.llm import (
|
||||
RobustChatModelWrapper,
|
||||
get_google_llm,
|
||||
get_llm,
|
||||
invoke_llm_with_timeout_message,
|
||||
)
|
||||
from artemis.utils.coordinates import parse_swipe_parameters
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class FlashRunner:
|
||||
"""Reactive, ultra-fast agent execution loop supporting all VLM providers."""
|
||||
|
||||
def __init__(self, ctx: ArtemisContext, goal: str, max_turns: int | None = None):
|
||||
self.ctx = ctx
|
||||
self.goal = goal
|
||||
try:
|
||||
cfg = load_agent_config()
|
||||
self.max_turns = max_turns if max_turns is not None else cfg.flash.max_turns
|
||||
self.step_summarizer_cfg = cfg.flash.step_summarizer
|
||||
except Exception:
|
||||
self.max_turns = max_turns if max_turns is not None else 30
|
||||
self.step_summarizer_cfg = StepSummarizerConfig()
|
||||
|
||||
self.controller = UnifiedMobileController(ctx)
|
||||
self.executor = MobileActionExecutor(ctx, self.controller)
|
||||
self.summarizer = (
|
||||
VisualStepSummarizer(ctx, model_name=self.step_summarizer_cfg.model)
|
||||
if self.step_summarizer_cfg.enabled
|
||||
else None
|
||||
)
|
||||
|
||||
def _get_tools(self) -> list:
|
||||
tools = [t for t in VALIDATOR_TOOLS_DECLARATION if t.name != "report_failure_analysis"]
|
||||
tools.insert(1, CLICK_SEQUENCE_TOOL)
|
||||
tools.append(ASK_EXPLORER_TOOL)
|
||||
tools.append(REPORT_TASK_STATUS_TOOL)
|
||||
return tools
|
||||
|
||||
def _prune_intermediate_screenshots(self, messages: list[BaseMessage]) -> None:
|
||||
"""Prunes binary screenshot blocks from intermediate observation messages."""
|
||||
prune_intermediate_screenshots(messages)
|
||||
|
||||
@trace(type="agent", name="FlashRunner")
|
||||
async def run(self, state: State) -> dict:
|
||||
logger.info(f"Starting Artemis Flash reactive loop for goal: {self.goal}")
|
||||
|
||||
# 1. Initialize Universal LLM via Service Layer
|
||||
try:
|
||||
llm = get_llm(self.ctx, name="operator")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to get operator LLM from config, using default: {e}")
|
||||
|
||||
llm = RobustChatModelWrapper(get_google_llm(model_name="gemini-2.5-flash"), self.ctx)
|
||||
|
||||
tools_declaration = self._get_tools()
|
||||
|
||||
# 2. Capture Initial State (Screenshot + UI Tree)
|
||||
shot_path, img_bytes, xml_list = await capture_screenshot_and_parse_ui(
|
||||
self.ctx, state, self.controller, skip_settling=False
|
||||
)
|
||||
state.latest_screenshot = shot_path
|
||||
|
||||
user_content: list[dict] = [
|
||||
{"type": "text", "text": f"Your objective is: {self.goal}"},
|
||||
]
|
||||
if img_bytes:
|
||||
img_b64 = base64.b64encode(img_bytes).decode("utf-8")
|
||||
user_content.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/jpeg;base64,{img_b64}"},
|
||||
}
|
||||
)
|
||||
if xml_list:
|
||||
user_content.append(
|
||||
{
|
||||
"type": "text",
|
||||
"text": f"--- UI Element List ---\n{xml_list}",
|
||||
}
|
||||
)
|
||||
user_content.append(
|
||||
{
|
||||
"type": "text",
|
||||
"text": (
|
||||
"CRITICAL RULE: In every single turn, you MUST FIRST output a natural"
|
||||
" language reasoning/explanation paragraph BEFORE invoking any tool call."
|
||||
),
|
||||
}
|
||||
)
|
||||
|
||||
# Render System Prompt
|
||||
prompt_path = Path(__file__).parent / "flash_runner.md"
|
||||
prompt_template = prompt_path.read_text(encoding="utf-8")
|
||||
system_prompt = Template(prompt_template).render(goal=self.goal)
|
||||
|
||||
messages: list[BaseMessage] = [
|
||||
SystemMessage(content=system_prompt),
|
||||
HumanMessage(content=user_content),
|
||||
]
|
||||
|
||||
turns = 0
|
||||
final_report = None
|
||||
current_pre_screenshot_bytes = img_bytes
|
||||
current_xml_list = xml_list
|
||||
|
||||
while turns < self.max_turns:
|
||||
turns += 1
|
||||
logger.info(f"--- Artemis Flash Turn {turns}/{self.max_turns} ---")
|
||||
|
||||
if self.ctx.data_engine:
|
||||
self.ctx.data_engine.allocate_step_id()
|
||||
|
||||
# Check for real-time injected instructions
|
||||
if self.ctx.data_engine and self.ctx.data_engine.base_dir:
|
||||
try:
|
||||
injected_instruction = await asyncio.to_thread(
|
||||
_check_injected_instruction_file,
|
||||
str(self.ctx.data_engine.base_dir),
|
||||
)
|
||||
if injected_instruction:
|
||||
messages.append(
|
||||
HumanMessage(
|
||||
content=(
|
||||
"[REAL-TIME INJECTED INSTRUCTION from user]:"
|
||||
f" {injected_instruction}\nYou MUST immediately follow"
|
||||
" this instruction and adjust your plan/actions."
|
||||
)
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to check injected instruction in FlashRunner: {e}")
|
||||
|
||||
# Compress previous history before LLM call to save context and inject objective summaries
|
||||
if self.summarizer:
|
||||
compress_flash_messages(
|
||||
messages,
|
||||
summarizer=self.summarizer,
|
||||
prune_history_xml=self.step_summarizer_cfg.prune_history_xml,
|
||||
)
|
||||
else:
|
||||
prune_intermediate_screenshots(messages)
|
||||
|
||||
# Tool restriction on the final turn
|
||||
if turns == self.max_turns:
|
||||
messages.append(
|
||||
HumanMessage(
|
||||
content=(
|
||||
"[WARNING] This is your final turn; only"
|
||||
" 'report_task_status' is available."
|
||||
)
|
||||
)
|
||||
)
|
||||
current_tools = [t for t in tools_declaration if t.name == "report_task_status"]
|
||||
else:
|
||||
current_tools = tools_declaration
|
||||
|
||||
# Bind active tools
|
||||
bound_llm = llm.bind_tools(current_tools)
|
||||
|
||||
# Invoke Model with Streaming & Retry
|
||||
response = None
|
||||
max_retries = 3
|
||||
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
|
||||
async def run_stream():
|
||||
full_response = None
|
||||
async for chunk in bound_llm.astream(messages):
|
||||
if full_response is None:
|
||||
full_response = chunk
|
||||
else:
|
||||
full_response += chunk
|
||||
return full_response
|
||||
|
||||
response = await invoke_llm_with_timeout_message(
|
||||
run_stream(), timeout_seconds=10, hard_timeout=180
|
||||
)
|
||||
if response:
|
||||
break
|
||||
except Exception as e:
|
||||
logger.warning(f"FlashRunner LLM turn failed on attempt {attempt + 1}: {e}")
|
||||
if attempt == max_retries - 1:
|
||||
raise e
|
||||
await asyncio.sleep(2.0**attempt)
|
||||
|
||||
if response is None:
|
||||
break
|
||||
|
||||
messages.append(response)
|
||||
|
||||
# Extract thought and tool calls
|
||||
raw_text = response.content if isinstance(response.content, str) else ""
|
||||
if isinstance(response.content, list):
|
||||
raw_text = "".join(
|
||||
b.get("text", "")
|
||||
for b in response.content
|
||||
if isinstance(b, dict) and "text" in b
|
||||
)
|
||||
|
||||
# Extract token usage metadata from response
|
||||
step_token_usage = None
|
||||
if hasattr(response, "usage_metadata") and response.usage_metadata:
|
||||
u = response.usage_metadata
|
||||
pr = u.get("input_tokens") or u.get("prompt_tokens") or 0
|
||||
co = u.get("output_tokens") or u.get("completion_tokens") or 0
|
||||
to = u.get("total_tokens") or (pr + co)
|
||||
if to > 0:
|
||||
step_token_usage = {
|
||||
"prompt_tokens": int(pr),
|
||||
"completion_tokens": int(co),
|
||||
"total_tokens": int(to),
|
||||
}
|
||||
elif hasattr(response, "response_metadata") and isinstance(
|
||||
response.response_metadata, dict
|
||||
):
|
||||
u = response.response_metadata.get(
|
||||
"usage_metadata"
|
||||
) or response.response_metadata.get("token_usage")
|
||||
if isinstance(u, dict):
|
||||
pr = (
|
||||
u.get("input_tokens")
|
||||
or u.get("prompt_tokens")
|
||||
or u.get("prompt_token_count")
|
||||
or 0
|
||||
)
|
||||
co = (
|
||||
u.get("output_tokens")
|
||||
or u.get("completion_tokens")
|
||||
or u.get("candidates_token_count")
|
||||
or 0
|
||||
)
|
||||
to = u.get("total_tokens") or u.get("total_token_count") or (pr + co)
|
||||
if to > 0:
|
||||
step_token_usage = {
|
||||
"prompt_tokens": int(pr),
|
||||
"completion_tokens": int(co),
|
||||
"total_tokens": int(to),
|
||||
}
|
||||
|
||||
if not step_token_usage or step_token_usage.get("total_tokens", 0) <= 0:
|
||||
calc_prompt_chars = 0
|
||||
calc_images = 0
|
||||
for msg in messages:
|
||||
c = getattr(msg, "content", "")
|
||||
if isinstance(c, str):
|
||||
calc_prompt_chars += len(c)
|
||||
elif isinstance(c, list):
|
||||
for block in c:
|
||||
if isinstance(block, dict):
|
||||
if block.get("type") == "image_url" or "image_url" in block:
|
||||
calc_images += 1
|
||||
else:
|
||||
calc_prompt_chars += len(str(block.get("text", "")))
|
||||
else:
|
||||
calc_prompt_chars += len(str(block))
|
||||
prompt_tokens = (calc_prompt_chars // 4) + (calc_images * 258)
|
||||
completion_tokens = len(raw_text) // 4
|
||||
step_token_usage = {
|
||||
"prompt_tokens": max(1, prompt_tokens),
|
||||
"completion_tokens": max(1, completion_tokens),
|
||||
"total_tokens": max(1, prompt_tokens + completion_tokens),
|
||||
}
|
||||
|
||||
if self.ctx.data_engine:
|
||||
current_step_id = getattr(self.ctx.data_engine, "current_step_id", None)
|
||||
self.ctx.data_engine.record_trace(
|
||||
type="llm_call",
|
||||
name="FlashRunner",
|
||||
payload={"token_usage": step_token_usage, "response": raw_text},
|
||||
step_id=current_step_id,
|
||||
status="success",
|
||||
)
|
||||
|
||||
tool_calls = response.tool_calls or []
|
||||
|
||||
# Fallback text parsing if tool_calls not parsed natively
|
||||
if not tool_calls and "```json" in raw_text:
|
||||
try:
|
||||
snippet = raw_text.split("```json")[1].split("```")[0].strip()
|
||||
parsed = json.loads(snippet)
|
||||
if isinstance(parsed, dict) and "name" in parsed:
|
||||
tool_calls = [
|
||||
{
|
||||
"name": parsed["name"],
|
||||
"args": parsed.get("args", {}),
|
||||
"id": str(uuid.uuid4()),
|
||||
}
|
||||
]
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if not tool_calls:
|
||||
logger.info(
|
||||
f"FlashRunner received response without tool calls at turn {turns}:"
|
||||
f" {raw_text[:100]}..."
|
||||
)
|
||||
if turns == self.max_turns:
|
||||
final_report = {"status": "failed", "explanation": raw_text}
|
||||
break
|
||||
messages.append(
|
||||
HumanMessage(
|
||||
content=(
|
||||
"You did not call any tools. Please make progress"
|
||||
" by calling an action tool or 'report_task_status'."
|
||||
)
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
# Process tool calls
|
||||
for tc in tool_calls:
|
||||
name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
|
||||
args = tc.get("args") or {}
|
||||
tc_id = tc.get("id") or str(uuid.uuid4())
|
||||
logger.info(f"Executing Flash tool: {name}({args})")
|
||||
|
||||
if name == "report_task_status":
|
||||
final_report = args
|
||||
if self.ctx.data_engine:
|
||||
try:
|
||||
if self.ctx.data_engine.current_step_id is None:
|
||||
self.ctx.data_engine.allocate_step_id()
|
||||
self.ctx.data_engine.record_step(
|
||||
pre_screenshot_bytes=current_pre_screenshot_bytes,
|
||||
ui_tree=current_xml_list,
|
||||
action_taken={"action": "report_task_status", "args": args},
|
||||
operator_raw_thinking=raw_text,
|
||||
last_execution_result={
|
||||
"result": "Task completed with final report."
|
||||
},
|
||||
extra_metadata={"token_usage": step_token_usage},
|
||||
)
|
||||
except Exception as step_err:
|
||||
logger.warning(f"Error recording final step in FlashRunner: {step_err}")
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc_id,
|
||||
name=name,
|
||||
content=json.dumps({"status": "acknowledged"}),
|
||||
status="success",
|
||||
)
|
||||
)
|
||||
if self.summarizer:
|
||||
await self.summarizer.flush()
|
||||
return final_report
|
||||
|
||||
try:
|
||||
exec_result = await self.executor.execute(name, args, tc_id, state)
|
||||
|
||||
post_img_bytes = exec_result.screenshot_bytes
|
||||
if not post_img_bytes and name in ACTION_TOOL_NAMES:
|
||||
try:
|
||||
controller = UnifiedMobileController(self.ctx)
|
||||
screen_data = await controller.get_screen_data()
|
||||
post_img_bytes = base64.b64decode(screen_data.base64)
|
||||
if not exec_result.ui_elements_text:
|
||||
exec_result.ui_elements_text = screen_data.elements
|
||||
except Exception as shot_err:
|
||||
logger.warning(
|
||||
f"Failed to capture fallback screenshot in FlashRunner: {shot_err}"
|
||||
)
|
||||
|
||||
# Record telemetry / step in DataEngine
|
||||
if self.ctx.data_engine and name in ACTION_TOOL_NAMES:
|
||||
try:
|
||||
if self.ctx.data_engine.current_step_id is None:
|
||||
self.ctx.data_engine.allocate_step_id()
|
||||
|
||||
# Extract and enrich coordinate metadata
|
||||
norm_coords = None
|
||||
norm_start = None
|
||||
norm_end = None
|
||||
if name == "swipe":
|
||||
kind, target_val, _ = parse_swipe_parameters(args)
|
||||
if (
|
||||
kind == "coords"
|
||||
and isinstance(target_val, list)
|
||||
and len(target_val) == 4
|
||||
):
|
||||
norm_coords = target_val
|
||||
norm_start = target_val[:2]
|
||||
norm_end = target_val[2:]
|
||||
elif kind == "direction" and isinstance(target_val, str):
|
||||
g_lower = target_val.lower()
|
||||
if "up" in g_lower:
|
||||
norm_coords = [600, 800, 600, 200]
|
||||
elif "down" in g_lower:
|
||||
norm_coords = [600, 200, 600, 800]
|
||||
elif "left" in g_lower:
|
||||
norm_coords = [800, 500, 200, 500]
|
||||
elif "right" in g_lower:
|
||||
norm_coords = [200, 500, 800, 500]
|
||||
if norm_coords:
|
||||
norm_start = norm_coords[:2]
|
||||
norm_end = norm_coords[2:]
|
||||
elif name in ("click", "tap", "long_press", "input_text"):
|
||||
target = args.get("target") or args.get("coordinates")
|
||||
if isinstance(target, (list, tuple)) and len(target) == 2:
|
||||
norm_coords = list(target)
|
||||
elif isinstance(target, str):
|
||||
nums = re.findall(r"-?\d+(?:\.\d+)?", target)
|
||||
if len(nums) == 2:
|
||||
norm_coords = [int(float(nums[0])), int(float(nums[1]))]
|
||||
|
||||
action_dict = {
|
||||
"action": name,
|
||||
"coordinates": (
|
||||
args.get("target")
|
||||
or args.get("coordinates")
|
||||
or args.get("sequence")
|
||||
or norm_coords
|
||||
),
|
||||
"args": args,
|
||||
}
|
||||
if norm_coords:
|
||||
action_dict["normalized_coordinates"] = norm_coords
|
||||
if norm_start and norm_end:
|
||||
action_dict["normalized_start_coordinates"] = norm_start
|
||||
action_dict["normalized_end_coordinates"] = norm_end
|
||||
|
||||
self.ctx.data_engine.record_step(
|
||||
pre_screenshot_bytes=current_pre_screenshot_bytes,
|
||||
post_screenshot_bytes=post_img_bytes,
|
||||
ui_tree=(exec_result.ui_elements_text or current_xml_list),
|
||||
action_taken=action_dict,
|
||||
operator_raw_thinking=raw_text,
|
||||
last_execution_result={"result": exec_result.text_summary},
|
||||
extra_metadata={"token_usage": step_token_usage},
|
||||
)
|
||||
except Exception as step_err:
|
||||
logger.warning(f"Error recording step in FlashRunner: {step_err}")
|
||||
|
||||
# ⚡ Non-blocking dispatch of objective visual transition summarizer
|
||||
if self.summarizer and name in ACTION_TOOL_NAMES:
|
||||
self.summarizer.dispatch(
|
||||
step_number=turns,
|
||||
action_name=name,
|
||||
action_args=args,
|
||||
pre_img_bytes=current_pre_screenshot_bytes,
|
||||
post_img_bytes=post_img_bytes,
|
||||
exec_outcome=exec_result.text_summary,
|
||||
)
|
||||
|
||||
if post_img_bytes:
|
||||
current_pre_screenshot_bytes = post_img_bytes
|
||||
if exec_result.ui_elements_text:
|
||||
current_xml_list = exec_result.ui_elements_text
|
||||
|
||||
messages.append(exec_result.to_langchain_tool_message())
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error executing tool {name}: {e}")
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc_id,
|
||||
name=name,
|
||||
content=f"Error executing tool {name}: {e}",
|
||||
status="error",
|
||||
)
|
||||
)
|
||||
|
||||
if self.summarizer:
|
||||
await self.summarizer.flush()
|
||||
|
||||
return final_report or {
|
||||
"status": "failed",
|
||||
"explanation": "Max turns reached without final status report.",
|
||||
}
|
||||
@@ -0,0 +1,226 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Asynchronous Visual Step Summarizer for Artemis Flash profile.
|
||||
|
||||
Processes Before/After screenshot pairs and action metadata in the background
|
||||
to generate high-density, strictly objective visual state transition summaries.
|
||||
Ensures zero blocking latency for the main runner and includes out-of-order self-healing retries.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from jinja2 import Template
|
||||
from langchain_core.messages import BaseMessage, HumanMessage, SystemMessage
|
||||
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.services.llm import get_google_llm, get_llm
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.visualization import draw_action_overlay_on_image
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class VisualStepSummarizer:
|
||||
"""Non-blocking background worker generating objective visual step summaries.
|
||||
|
||||
Key design properties:
|
||||
1. Zero-blocking dispatch: The main Flash runner dispatches and proceeds immediately.
|
||||
2. Graceful fallback: If a step summary is not ready, it cleanly falls back to the original action text.
|
||||
3. Self-healing re-trigger: If a previous step's summary is stalled or lost while subsequent steps
|
||||
complete, it automatically re-dispatches the stalled step.
|
||||
"""
|
||||
|
||||
def __init__(self, ctx: ArtemisContext, model_name: str | None = None):
|
||||
self.ctx = ctx
|
||||
self._summaries: dict[int, str] = {}
|
||||
self._pending_tasks: dict[int, asyncio.Task] = {}
|
||||
self._step_inputs: dict[int, dict[str, Any]] = {}
|
||||
self._retry_counts: dict[int, int] = {}
|
||||
self._max_retries = 2
|
||||
|
||||
# Initialize lightweight VLM: prioritize explicit model_name
|
||||
target_model = model_name or "gemini-2.5-flash-lite"
|
||||
try:
|
||||
if model_name:
|
||||
self._llm = get_google_llm(model_name=target_model, temperature=0.0)
|
||||
else:
|
||||
self._llm = get_llm(ctx, name="summarizer", is_utils=True)
|
||||
except Exception:
|
||||
self._llm = get_google_llm(model_name=target_model, temperature=0.0)
|
||||
|
||||
# Load system prompt template
|
||||
prompt_path = Path(__file__).parent / "flash_summarizer.md"
|
||||
if prompt_path.exists():
|
||||
self._prompt_template = prompt_path.read_text(encoding="utf-8")
|
||||
else:
|
||||
self._prompt_template = (
|
||||
"You are the Step Summarizer for an Android UI automation agent.\n"
|
||||
"Synthesize the physical action and visual delta between BEFORE and AFTER screens in exactly ONE "
|
||||
"continuous first-person paragraph using 'I' (e.g., 'In Step {{ step_number }}, I tapped... and observed...').\n"
|
||||
"Strictly avoid subjective validation words: successfully, completed, failed, achieved, navigated to."
|
||||
)
|
||||
|
||||
def dispatch(
|
||||
self,
|
||||
step_number: int,
|
||||
action_name: str,
|
||||
action_args: dict[str, Any],
|
||||
pre_img_bytes: bytes | None,
|
||||
post_img_bytes: bytes | None,
|
||||
exec_outcome: str,
|
||||
) -> None:
|
||||
"""Dispatches an asynchronous summarization task without blocking the caller."""
|
||||
self._step_inputs[step_number] = {
|
||||
"action_name": action_name,
|
||||
"action_args": action_args,
|
||||
"pre_img_bytes": pre_img_bytes,
|
||||
"post_img_bytes": post_img_bytes,
|
||||
"exec_outcome": exec_outcome,
|
||||
}
|
||||
|
||||
task = asyncio.create_task(self._run_summary(step_number))
|
||||
self._pending_tasks[step_number] = task
|
||||
|
||||
# Check for and re-trigger any previous stalled steps
|
||||
self._check_and_retrigger_stalled_steps(current_step=step_number)
|
||||
|
||||
def _check_and_retrigger_stalled_steps(self, current_step: int) -> None:
|
||||
"""Re-dispatches any previous steps that remain unsummarized while later steps arrive."""
|
||||
for past_step in range(1, current_step):
|
||||
if past_step not in self._summaries and past_step in self._step_inputs:
|
||||
task = self._pending_tasks.get(past_step)
|
||||
retries = self._retry_counts.get(past_step, 0)
|
||||
if (task is None or task.done()) and retries < self._max_retries:
|
||||
self._retry_counts[past_step] = retries + 1
|
||||
logger.info(
|
||||
f"VisualStepSummarizer: Retriggering stalled summary for Step {past_step} "
|
||||
f"(attempt {self._retry_counts[past_step] + 1})"
|
||||
)
|
||||
self._pending_tasks[past_step] = asyncio.create_task(
|
||||
self._run_summary(past_step)
|
||||
)
|
||||
|
||||
async def _run_summary(self, step_number: int) -> None:
|
||||
"""Executes the lightweight VLM call to summarize the step transition."""
|
||||
input_data = self._step_inputs.get(step_number)
|
||||
if not input_data:
|
||||
return
|
||||
|
||||
action_name = input_data["action_name"]
|
||||
action_args = input_data["action_args"]
|
||||
pre_bytes = input_data["pre_img_bytes"]
|
||||
post_bytes = input_data["post_img_bytes"]
|
||||
exec_outcome = input_data["exec_outcome"]
|
||||
|
||||
try:
|
||||
rendered_prompt = Template(self._prompt_template).render(step_number=step_number)
|
||||
content_blocks: list[dict[str, Any]] = [
|
||||
{
|
||||
"type": "text",
|
||||
"text": (
|
||||
f"Step {step_number} Physical Action: {action_name}({action_args})\n"
|
||||
f"Controller Outcome: {exec_outcome}"
|
||||
),
|
||||
}
|
||||
]
|
||||
|
||||
if pre_bytes:
|
||||
# 🎨 Visually mark the exact action (tap ripple, sequence numbers, swipe arrow) on the BEFORE screenshot
|
||||
annotated_pre_bytes = draw_action_overlay_on_image(
|
||||
image_bytes=pre_bytes,
|
||||
action_name=action_name,
|
||||
action_args=action_args,
|
||||
)
|
||||
b64_pre = base64.b64encode(annotated_pre_bytes).decode("utf-8")
|
||||
content_blocks.append(
|
||||
{
|
||||
"type": "text",
|
||||
"text": "--- [1] BEFORE ACTION SCREEN (Action Marked Visually in Red) ---",
|
||||
}
|
||||
)
|
||||
content_blocks.append(
|
||||
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64_pre}"}}
|
||||
)
|
||||
|
||||
if post_bytes:
|
||||
b64_post = base64.b64encode(post_bytes).decode("utf-8")
|
||||
content_blocks.append({"type": "text", "text": "--- [2] AFTER ACTION SCREEN ---"})
|
||||
content_blocks.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/jpeg;base64,{b64_post}"},
|
||||
}
|
||||
)
|
||||
|
||||
messages: list[BaseMessage] = [
|
||||
SystemMessage(content=rendered_prompt),
|
||||
HumanMessage(content=content_blocks),
|
||||
]
|
||||
|
||||
response = await asyncio.wait_for(self._llm.ainvoke(messages), timeout=25.0)
|
||||
summary_raw = response.content if isinstance(response.content, str) else ""
|
||||
if isinstance(response.content, list):
|
||||
summary_raw = "".join(
|
||||
b.get("text", "")
|
||||
for b in response.content
|
||||
if isinstance(b, dict) and "text" in b
|
||||
)
|
||||
|
||||
summary_text = summary_raw.strip()
|
||||
if summary_text:
|
||||
self._summaries[step_number] = summary_text
|
||||
logger.info(
|
||||
f"VisualStepSummarizer: Generated summary for Step {step_number}: {summary_text[:80]}..."
|
||||
)
|
||||
|
||||
# Free binary image buffers from memory once summary is secured
|
||||
if step_number in self._step_inputs:
|
||||
self._step_inputs[step_number]["pre_img_bytes"] = None
|
||||
self._step_inputs[step_number]["post_img_bytes"] = None
|
||||
|
||||
# Update DataEngine telemetry if active
|
||||
if self.ctx.data_engine:
|
||||
try:
|
||||
self.ctx.data_engine.update_step_summary(step_number, summary_text)
|
||||
except Exception as de_err:
|
||||
logger.debug(f"DataEngine step summary update skipped: {de_err}")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
f"VisualStepSummarizer: Error generating summary for step {step_number}: {e}"
|
||||
)
|
||||
if self._retry_counts.get(step_number, 0) >= self._max_retries:
|
||||
# Max retries reached, free memory buffer
|
||||
if step_number in self._step_inputs:
|
||||
self._step_inputs[step_number]["pre_img_bytes"] = None
|
||||
self._step_inputs[step_number]["post_img_bytes"] = None
|
||||
|
||||
def get_summary(self, step_number: int, fallback_text: str | None = None) -> str | None:
|
||||
"""Retrieves summarized state for step_number, or fallback_text if not ready yet."""
|
||||
return self._summaries.get(step_number, fallback_text)
|
||||
|
||||
def has_summary(self, step_number: int) -> bool:
|
||||
"""Checks if a step summary has completed."""
|
||||
return step_number in self._summaries
|
||||
|
||||
async def flush(self) -> None:
|
||||
"""Flushes and awaits all pending background summary tasks upon completion."""
|
||||
tasks = [t for t in self._pending_tasks.values() if not t.done()]
|
||||
if tasks:
|
||||
logger.info(f"VisualStepSummarizer: Flushing {len(tasks)} pending summary tasks...")
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
@@ -0,0 +1,8 @@
|
||||
# IDENTITY
|
||||
You are a **History Analyzer**, expert in analyzing multi-agent session execution history to answer user queries.
|
||||
|
||||
# MANDATE
|
||||
1. You are provided with the **Task Plan and Execution History** summarizing the high-level plan and history steps.
|
||||
2. For queries about specific step details (such as precise actions, coordinates, `operator_raw_thinking`, or `last_execution_result`), you should call the `get_step_details` tool.
|
||||
3. Never assume, guess, or invent details not visible in the Task Plan and Execution History. Always query `get_step_details` for specifics. If details are unavailable, state it clearly.
|
||||
4. Make good use of note tools to see what notes are available. When analyzing history records, open and cross-reference relevant notes to ensure you have grasped all necessary information.
|
||||
@@ -0,0 +1,250 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.messages import (
|
||||
BaseMessage,
|
||||
HumanMessage,
|
||||
SystemMessage,
|
||||
ToolMessage,
|
||||
)
|
||||
from langchain_core.tools import BaseTool, tool
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.data_engine.trace import CURRENT_TRACE_ID, TraceSpan, trace
|
||||
from artemis.services.llm import get_llm, invoke_llm_with_timeout_message
|
||||
from artemis.tools.scratchpad import (
|
||||
get_list_notes_tool_pure,
|
||||
get_read_note_tool_pure,
|
||||
)
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.notes import (
|
||||
get_note_file_path,
|
||||
get_notes_dir,
|
||||
)
|
||||
from artemis.utils.task_tree import build_plan_and_history, get_active_subgoal_hashes
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class HistoryAnalyzer:
|
||||
def __init__(self, ctx: ArtemisContext):
|
||||
self.ctx = ctx
|
||||
|
||||
def exec_get_step_details(self, start_step: int, end_step: int) -> str:
|
||||
"""Retrieve the detailed, full-granularity information for a range of steps (inclusive).
|
||||
|
||||
Use this when you need to inspect specific actions taken, operator
|
||||
thinking, or execution results.
|
||||
"""
|
||||
try:
|
||||
s_step = int(start_step)
|
||||
e_step = int(end_step)
|
||||
except (ValueError, TypeError):
|
||||
return (
|
||||
f"Error: start_step and end_step must be integers, got {start_step} and {end_step}."
|
||||
)
|
||||
|
||||
matched_steps = []
|
||||
for s in getattr(self, "history_steps", []):
|
||||
step_num = s.get("step_number")
|
||||
if step_num is not None and s_step <= step_num <= e_step:
|
||||
details = {
|
||||
"step_number": s.get("step_number"),
|
||||
"relative_time": s.get("relative_time"),
|
||||
"summary": s.get("summary"),
|
||||
"action_taken": s.get("action_taken"),
|
||||
"operator_raw_thinking": s.get("operator_raw_thinking"),
|
||||
"last_execution_result": s.get("last_execution_result"),
|
||||
}
|
||||
matched_steps.append(details)
|
||||
if not matched_steps:
|
||||
return f"No steps found in range [{s_step}, {e_step}]."
|
||||
return json.dumps(matched_steps, indent=2, ensure_ascii=False)
|
||||
|
||||
def _get_step_details_tool(self, history_steps: list[dict[str, Any]]) -> BaseTool:
|
||||
self.history_steps = history_steps
|
||||
|
||||
@tool
|
||||
def get_step_details(start_step: int, end_step: int) -> str:
|
||||
"""Retrieve the detailed, full-granularity information for a range of steps (inclusive).
|
||||
|
||||
Use this when you need to inspect specific actions taken, operator
|
||||
thinking, or execution results.
|
||||
"""
|
||||
return self.exec_get_step_details(start_step, end_step)
|
||||
|
||||
return get_step_details
|
||||
|
||||
@trace(type="agent", name="history_analyzer")
|
||||
async def run(self, query: str) -> str:
|
||||
if not self.ctx.data_engine:
|
||||
return "Error: DataEngine is not available to retrieve history."
|
||||
|
||||
# 1. Fetch all steps from Data Engine
|
||||
self.history_steps = self.ctx.data_engine.get_agent_friendly_steps()
|
||||
if not self.history_steps:
|
||||
return "No history recorded for this session yet."
|
||||
|
||||
# 2. Build the plan and history or step list
|
||||
plan_and_history = ""
|
||||
try:
|
||||
notes_dir = get_notes_dir(self.ctx.data_engine.base_dir)
|
||||
current_plan = ""
|
||||
current_path = get_note_file_path(self.ctx.data_engine.base_dir, "task_plan")
|
||||
if current_path.exists():
|
||||
current_plan = current_path.read_text(encoding="utf-8")
|
||||
|
||||
if current_plan:
|
||||
active_subgoal_hash, _ = get_active_subgoal_hashes(current_plan)
|
||||
plan_and_history = build_plan_and_history(
|
||||
current_plan,
|
||||
self.history_steps,
|
||||
active_subgoal_hash,
|
||||
last_n_detailed=1,
|
||||
min_summaries=len(self.history_steps),
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to build plan and history in HistoryAnalyzer: {e}")
|
||||
|
||||
if not plan_and_history:
|
||||
# Fallback to a simple list
|
||||
plan_and_history = "\n".join(
|
||||
[
|
||||
f"- Step {s.get('step_number')} ({s.get('relative_time')}):"
|
||||
f" {s.get('summary') or 'No summary'}"
|
||||
for s in self.history_steps
|
||||
]
|
||||
)
|
||||
|
||||
# 3. Load system prompt
|
||||
prompt_path = Path(__file__).parent.joinpath("history_analyzer.md")
|
||||
if prompt_path.exists():
|
||||
system_prompt_template = prompt_path.read_text(encoding="utf-8")
|
||||
else:
|
||||
system_prompt_template = (
|
||||
"You are a History Analyzer. Your role is to analyze the"
|
||||
" execution history of a session and answer user queries in"
|
||||
" natural language.\nIf you need specific details for a range"
|
||||
" of steps, use the `get_step_details` tool."
|
||||
)
|
||||
|
||||
system_message_content = (
|
||||
system_prompt_template + f"\n\n### Task Plan and Execution History:\n{plan_and_history}"
|
||||
)
|
||||
|
||||
# 4. Prepare LLM and bind tools
|
||||
llm = get_llm(ctx=self.ctx, name="history_analyzer")
|
||||
get_step_details = self._get_step_details_tool(self.history_steps)
|
||||
|
||||
list_notes_tool = get_list_notes_tool_pure(self.ctx)
|
||||
read_note_tool = get_read_note_tool_pure(self.ctx)
|
||||
llm = llm.bind_tools(tools=[get_step_details, list_notes_tool, read_note_tool])
|
||||
|
||||
# 5. ReAct Loop
|
||||
messages: list[BaseMessage] = [
|
||||
SystemMessage(content=system_message_content),
|
||||
HumanMessage(content=query),
|
||||
]
|
||||
|
||||
max_turns = 5
|
||||
for turn in range(max_turns):
|
||||
logger.info(f"HistoryAnalyzer ReAct turn {turn + 1}")
|
||||
|
||||
async def run_stream():
|
||||
full_response = None
|
||||
trace_id = CURRENT_TRACE_ID.get()
|
||||
async for chunk in llm.astream(messages):
|
||||
if full_response is None:
|
||||
full_response = chunk
|
||||
else:
|
||||
full_response += chunk
|
||||
|
||||
pass
|
||||
return full_response
|
||||
|
||||
response = await invoke_llm_with_timeout_message(run_stream())
|
||||
messages.append(response)
|
||||
|
||||
if not response.tool_calls:
|
||||
return (
|
||||
response.content.strip()
|
||||
if isinstance(response.content, str)
|
||||
else str(response.content)
|
||||
)
|
||||
|
||||
# Process tool calls
|
||||
for tc in response.tool_calls:
|
||||
tool_name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
|
||||
args = tc["args"]
|
||||
|
||||
with TraceSpan(name=tool_name, ctx=self.ctx) as span:
|
||||
span.payload = {"args": args}
|
||||
if tool_name == "get_step_details":
|
||||
logger.info(
|
||||
f"HistoryAnalyzer executing tool get_step_details with args: {args}"
|
||||
)
|
||||
try:
|
||||
start_step = int(args.get("start_step", 0))
|
||||
end_step = int(args.get("end_step", 0))
|
||||
result = get_step_details.invoke(
|
||||
{"start_step": start_step, "end_step": end_step}
|
||||
)
|
||||
span.result = result
|
||||
status = "success"
|
||||
except Exception as e:
|
||||
span.status = "failed"
|
||||
span.error = str(e)
|
||||
result = f"Error running get_step_details: {e}"
|
||||
status = "error"
|
||||
elif tool_name == "list_notes":
|
||||
logger.info("HistoryAnalyzer executing tool list_notes")
|
||||
try:
|
||||
result = list_notes_tool.invoke({})
|
||||
span.result = result
|
||||
status = "success"
|
||||
except Exception as e:
|
||||
span.status = "failed"
|
||||
span.error = str(e)
|
||||
result = f"Error running list_notes: {e}"
|
||||
status = "error"
|
||||
elif tool_name == "read_note":
|
||||
logger.info(f"HistoryAnalyzer executing tool read_note with args: {args}")
|
||||
try:
|
||||
key = args.get("key", "")
|
||||
result = read_note_tool.invoke({"key": key})
|
||||
span.result = result
|
||||
status = "success"
|
||||
except Exception as e:
|
||||
span.status = "failed"
|
||||
span.error = str(e)
|
||||
result = f"Error running read_note: {e}"
|
||||
status = "error"
|
||||
else:
|
||||
result = f"Error: Tool {tool_name} is not supported."
|
||||
status = "error"
|
||||
span.status = "failed"
|
||||
span.error = result
|
||||
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc["id"],
|
||||
content=result,
|
||||
status=status,
|
||||
)
|
||||
)
|
||||
|
||||
return "Error: HistoryAnalyzer failed to resolve the query within maximum turns."
|
||||
@@ -0,0 +1,15 @@
|
||||
## Hopper
|
||||
|
||||
Extract relevant information from batch data. **Keep extracted data exactly as-is** - no reformatting.
|
||||
|
||||
## Output
|
||||
- **found**: `true` if data was found, `false` otherwise
|
||||
- **output**: Extracted information if found, `null` otherwise
|
||||
- **reason**: Brief explanation of search logic
|
||||
|
||||
## Rules
|
||||
1. **Search entire input** - may contain hundreds of entries
|
||||
2. **For app package lookup**: Match app name (or variations) in package identifier
|
||||
- Common patterns: lowercase app name, company+app, brand name, codenames
|
||||
3. **Prefer direct matches** over partial matches
|
||||
4. **Return `null`** if not found or if multiple ambiguous matches exist - don't guess
|
||||
@@ -0,0 +1,93 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from jinja2 import Template
|
||||
from langchain_core.messages import HumanMessage, SystemMessage
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.data_engine.trace import trace
|
||||
from artemis.services.llm import get_llm, invoke_llm_with_timeout_message, with_fallback
|
||||
from artemis.utils.logger import get_logger
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class _CyFunctionDetectorMeta(type):
|
||||
def __instancecheck__(self, instance):
|
||||
name = type(instance).__name__
|
||||
return (
|
||||
name
|
||||
in (
|
||||
"cyfunction",
|
||||
"cython_function_or_method",
|
||||
"builtin_function_or_method",
|
||||
)
|
||||
or "cyfunction" in name.lower()
|
||||
)
|
||||
|
||||
|
||||
class CyFunctionDetector(metaclass=_CyFunctionDetectorMeta):
|
||||
pass
|
||||
|
||||
|
||||
class HopperOutput(BaseModel):
|
||||
model_config = {"ignored_types": (CyFunctionDetector,)}
|
||||
found: bool = Field(description="True if the requested data was found, False otherwise.")
|
||||
output: str | None = Field(description="The extracted data if found, null otherwise.")
|
||||
reason: str = Field(
|
||||
description="A short explanation of what you looked for"
|
||||
+ " and how you decided what to extract."
|
||||
)
|
||||
|
||||
|
||||
@trace(type="agent", name="hopper")
|
||||
async def hopper(
|
||||
ctx: ArtemisContext,
|
||||
request: str,
|
||||
data: str,
|
||||
use_fallback: bool = True,
|
||||
) -> HopperOutput:
|
||||
logger.info(f"Starting Hopper Agent (use_fallback={use_fallback})")
|
||||
system_message = Template(
|
||||
Path(__file__).parent.joinpath("hopper.md").read_text(encoding="utf-8")
|
||||
).render()
|
||||
messages = [
|
||||
SystemMessage(content=system_message),
|
||||
HumanMessage(content=f"{request}\nHere is the data you must dig:\n{data}"),
|
||||
]
|
||||
|
||||
llm = get_llm(ctx=ctx, name="hopper", is_utils=True).with_structured_output(HopperOutput)
|
||||
try:
|
||||
if use_fallback:
|
||||
llm_fallback = get_llm(
|
||||
ctx=ctx, name="hopper", is_utils=True, use_fallback=True
|
||||
).with_structured_output(HopperOutput)
|
||||
response: HopperOutput = await with_fallback(
|
||||
main_call=lambda: invoke_llm_with_timeout_message(llm.ainvoke(messages)),
|
||||
fallback_call=lambda: invoke_llm_with_timeout_message(
|
||||
llm_fallback.ainvoke(messages)
|
||||
),
|
||||
) # type: ignore
|
||||
else:
|
||||
response: HopperOutput = await invoke_llm_with_timeout_message(llm.ainvoke(messages))
|
||||
return response
|
||||
except Exception as e:
|
||||
logger.error(f"Hopper LLM invocation failed: {e}")
|
||||
return HopperOutput(
|
||||
found=False,
|
||||
output=None,
|
||||
reason=f"Failed due to LLM error: {str(e)[:500]}",
|
||||
)
|
||||
@@ -0,0 +1,74 @@
|
||||
# IDENTITY
|
||||
You are the expert UI Image Processor, a coding agent that can write Python scripts to analyze and modify images (like color filtering, cropping, highlighting, coloring, or performing custom image processing).
|
||||
|
||||
# CONTEXT
|
||||
Target Image ID: `img_0` (This is the original input screenshot)
|
||||
Instruction: {instruction}
|
||||
Intermediate Artifact Save Path: `{intermediate_artifact_save_path}`
|
||||
If you want to save intermediate steps to inspect them or load them later, you must call `canvas.save(final=False)`. Any intermediate images saved this way will be automatically loaded and fed back to you as visual context at the beginning of your next turn, and can be retrieved in subsequent runs by instantiating a new canvas: `ImageCanvas(image_id, '{intermediate_artifact_save_path}')`.
|
||||
|
||||
|
||||
# GOAL
|
||||
Your goal is to perform image analysis, filter, crop, draw annotations/markers, or modify the image to visually highlight or isolate specific features as requested. Write a Python script using OpenCV (`cv2`), `numpy`, or other standard libraries to analyze and modify the target image `img_0` according to the `Instruction`. The final output should be the modified image. Call `submit_result` tool after you have successfully completed the task.
|
||||
|
||||
# CRITICAL BOUNDARIES FOR GEOMETRIC TRANSFORMATIONS
|
||||
1. **DO NOT** use numpy array slicing (e.g. `img[y:y+h, x:x+w]`) to crop the image.
|
||||
2. **DO NOT** use `cv2.resize()` to scale the image.
|
||||
Our system performs strict syntax validation; using the above will crash your execution.
|
||||
|
||||
If you MUST crop or scale the image to isolate small details or improve OCR/detection downstream, you MUST use the pre-imported `ImageCanvas` tool exactly as follows:
|
||||
|
||||
### Snippet 1: Saving & Resuming Intermediate Steps (Iterative Workflow)
|
||||
Use this to crop/zoom into specific areas and save them so you can visually inspect them in your next turn, or load them back to create new branches.
|
||||
```python
|
||||
from artemis.utils.cv_canvas import ImageCanvas
|
||||
|
||||
# 1. Start from the original image:
|
||||
canvas = ImageCanvas("img_0", "{intermediate_artifact_save_path}")
|
||||
|
||||
# 2. Crop and scale:
|
||||
canvas.crop(x=100, y=200, w=300, h=300)
|
||||
canvas.resize_by_factor(2.0)
|
||||
|
||||
# 3. Save as an intermediate step for verification:
|
||||
# This will auto-generate 'img_1' and save it in the intermediate directory.
|
||||
# In the next turn, you will receive 'img_1' as visual confirmation.
|
||||
canvas.save(final=False)
|
||||
|
||||
# --- In a subsequent turn, if you want to resume from 'img_1' ---
|
||||
# canvas = ImageCanvas('img_1', '{intermediate_artifact_save_path}')
|
||||
```
|
||||
|
||||
### Snippet 2: Finalizing and Saving the Output
|
||||
Use this to draw your final annotations, highlights, or markers, and save the final result to be returned.
|
||||
```python
|
||||
from artemis.utils.cv_canvas import ImageCanvas
|
||||
|
||||
# 1. Load the original or a cropped intermediate step to finalize:
|
||||
canvas = ImageCanvas("img_1", "{intermediate_artifact_save_path}")
|
||||
|
||||
# 2. Draw annotations, highlight elements, or apply filters:
|
||||
# # Convert normalized coordinates (0-1000) to pixel coordinates:
|
||||
# x, y = canvas.normalized_to_pixel_coords(nx, ny)
|
||||
# Draw specific verification dots and register their labels with unique label_idx (e.g. V1):
|
||||
# canvas.draw_dot(100, 200, 1)
|
||||
|
||||
# 3. Save the final result (this is required to complete the task):
|
||||
# This will auto-generate the next sequential image ID (e.g., 'img_2') and mark it as a final output.
|
||||
canvas.save(final=True)
|
||||
```
|
||||
|
||||
### Snippet 3: Simple Modification Without Cropping or Scaling
|
||||
If you do NOT need to crop or scale, but just want to draw annotations or apply filters, you can still use `ImageCanvas` for consistency:
|
||||
```python
|
||||
from artemis.utils.cv_canvas import ImageCanvas
|
||||
|
||||
canvas = ImageCanvas("img_0", "{intermediate_artifact_save_path}")
|
||||
# ... modify canvas.img ...
|
||||
canvas.save(final=True)
|
||||
```
|
||||
|
||||
# EXECUTION LOOP
|
||||
1. Write and execute your script using the `execute_python` tool.
|
||||
2. Read the stdout/stderr. If there are errors, fix your code and execute again.
|
||||
3. Once you have successfully called `canvas.save(final=True)` on your final output image(s), call the `submit_result` tool with a summary to finish your job.
|
||||
@@ -0,0 +1,322 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Universal Multi-Model Image Processor for Artemis."""
|
||||
|
||||
import base64
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import time
|
||||
import uuid
|
||||
|
||||
from langchain_core.messages import (
|
||||
BaseMessage,
|
||||
HumanMessage,
|
||||
SystemMessage,
|
||||
ToolMessage,
|
||||
)
|
||||
|
||||
from artemis.config import settings
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.core.tool_declaration import ToolDeclaration
|
||||
from artemis.data_engine.trace import TraceSpan, trace
|
||||
from artemis.services.llm import get_llm
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.python_executor import PythonExecutor
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
EXECUTE_PYTHON_TOOL = ToolDeclaration(
|
||||
name="execute_python",
|
||||
description=(
|
||||
"Executes the provided Python code within an isolated"
|
||||
" Jupyter environment and returns the resulting stdout,"
|
||||
" stderr, and traceback outputs."
|
||||
),
|
||||
parameters={
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "string",
|
||||
"description": "The Python code to execute",
|
||||
}
|
||||
},
|
||||
"required": ["code"],
|
||||
},
|
||||
)
|
||||
|
||||
SUBMIT_RESULT_TOOL = ToolDeclaration(
|
||||
name="submit_result",
|
||||
description=("Submits a summary of the modifications to successfully complete the task."),
|
||||
parameters={
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"summary": {
|
||||
"type": "string",
|
||||
"description": "Summary of what you found or changed.",
|
||||
}
|
||||
},
|
||||
"required": ["summary"],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
class ImageProcessor:
|
||||
def __init__(self, ctx: ArtemisContext):
|
||||
self.ctx = ctx
|
||||
|
||||
@trace(type="agent", name="image_processor")
|
||||
async def run(self, instruction: str, target_image_path: str) -> dict:
|
||||
try:
|
||||
llm = get_llm(self.ctx, name="image_processor")
|
||||
except Exception:
|
||||
llm = get_llm(self.ctx, name="operator")
|
||||
|
||||
base_dir = settings.TRACES_PATH
|
||||
image_processor_dir = base_dir / "images" / "image_processor"
|
||||
image_processor_dir.mkdir(parents=True, exist_ok=True)
|
||||
intermediate_artifact_save_path = image_processor_dir / f"intermediates_{int(time.time())}"
|
||||
intermediate_artifact_save_path.mkdir(parents=True, exist_ok=True)
|
||||
intermediate_artifact_save_path_str = str(intermediate_artifact_save_path.resolve())
|
||||
|
||||
max_iterations = 5
|
||||
|
||||
prompt_path = Path(__file__).parent / "image_processor.md"
|
||||
prompt_template = prompt_path.read_text(encoding="utf-8")
|
||||
prompt_template = (
|
||||
prompt_template.replace("{target_image_path}", target_image_path)
|
||||
.replace("{instruction}", instruction)
|
||||
.replace(
|
||||
"{intermediate_artifact_save_path}",
|
||||
intermediate_artifact_save_path_str,
|
||||
)
|
||||
.replace("{max_iterations}", str(max_iterations))
|
||||
)
|
||||
|
||||
tools_declaration = [EXECUTE_PYTHON_TOOL, SUBMIT_RESULT_TOOL]
|
||||
|
||||
with open(target_image_path, "rb") as f:
|
||||
img_bytes = f.read()
|
||||
img_b64 = base64.b64encode(img_bytes).decode("utf-8")
|
||||
|
||||
messages: list[BaseMessage] = [
|
||||
SystemMessage(content=prompt_template),
|
||||
HumanMessage(
|
||||
content=[
|
||||
{"type": "text", "text": "Here is the target image. Begin writing your code."},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/jpeg;base64,{img_b64}"},
|
||||
},
|
||||
]
|
||||
),
|
||||
]
|
||||
|
||||
img_0_path = intermediate_artifact_save_path / "img_0.jpg"
|
||||
try:
|
||||
shutil.copy2(target_image_path, img_0_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to copy original screenshot to intermediate pool: {e}")
|
||||
img_0_path = Path(target_image_path)
|
||||
|
||||
self.intermediate_image_pool = {
|
||||
"img_0": {
|
||||
"image_id": "img_0",
|
||||
"path": str(img_0_path.resolve()),
|
||||
"turn_id": 0,
|
||||
"transform": {
|
||||
"offset_x": 0.0,
|
||||
"offset_y": 0.0,
|
||||
"scale_x": 1.0,
|
||||
"scale_y": 1.0,
|
||||
},
|
||||
"annotations": {},
|
||||
}
|
||||
}
|
||||
|
||||
pool_json_path = intermediate_artifact_save_path / "intermediate_transforms.json"
|
||||
sync_pool = {
|
||||
"img_0": {
|
||||
"image_id": "img_0",
|
||||
"path": self.intermediate_image_pool["img_0"]["path"],
|
||||
"transform": self.intermediate_image_pool["img_0"]["transform"],
|
||||
"annotations": {},
|
||||
}
|
||||
}
|
||||
with open(pool_json_path, "w") as f:
|
||||
json.dump(sync_pool, f)
|
||||
|
||||
executor = PythonExecutor(session_dir=image_processor_dir)
|
||||
iterations = 0
|
||||
final_result = None
|
||||
|
||||
try:
|
||||
while iterations < max_iterations:
|
||||
iterations += 1
|
||||
logger.info(f"ImageProcessor Iteration {iterations}")
|
||||
|
||||
# Inject newly created intermediate images from last turn
|
||||
if self.intermediate_image_pool:
|
||||
image_blocks = []
|
||||
last_turn_id = iterations - 1
|
||||
for img_id, entry in self.intermediate_image_pool.items():
|
||||
if entry["turn_id"] == last_turn_id:
|
||||
try:
|
||||
path = Path(entry["path"])
|
||||
if path.exists():
|
||||
b64_data = base64.b64encode(path.read_bytes()).decode("utf-8")
|
||||
is_output = entry.get("is_output", False)
|
||||
label = (
|
||||
f"img_id: {img_id} (Output)"
|
||||
if is_output
|
||||
else f"img_id: {img_id}"
|
||||
)
|
||||
image_blocks.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": f"data:image/jpeg;base64,{b64_data}"
|
||||
},
|
||||
}
|
||||
)
|
||||
image_blocks.append({"type": "text", "text": f"\n{label}\n"})
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to read intermediate image {entry['path']}: {e}"
|
||||
)
|
||||
|
||||
if image_blocks:
|
||||
if (
|
||||
messages
|
||||
and isinstance(messages[-1], HumanMessage)
|
||||
and isinstance(messages[-1].content, list)
|
||||
):
|
||||
messages[-1].content.extend(image_blocks)
|
||||
else:
|
||||
messages.append(HumanMessage(content=image_blocks))
|
||||
|
||||
if iterations == max_iterations - 1:
|
||||
warning_msg = (
|
||||
"\n[WARNING] This is your next-to-last iteration. You"
|
||||
" must complete your task and call `submit_result` now"
|
||||
" or in the next turn, as no further code execution"
|
||||
" will be allowed after that."
|
||||
)
|
||||
messages.append(HumanMessage(content=warning_msg))
|
||||
elif iterations == max_iterations:
|
||||
warning_msg = (
|
||||
"\n[WARNING] This is your final iteration. You MUST"
|
||||
" call `submit_result` now to complete the task. No"
|
||||
" other tools are available."
|
||||
)
|
||||
messages.append(HumanMessage(content=warning_msg))
|
||||
|
||||
current_tools = tools_declaration
|
||||
if iterations == max_iterations:
|
||||
current_tools = [t for t in tools_declaration if t.name == "submit_result"]
|
||||
|
||||
bound_llm = llm.bind_tools(current_tools)
|
||||
|
||||
with TraceSpan(name="image_processor_call", ctx=self.ctx):
|
||||
response = await bound_llm.ainvoke(messages)
|
||||
|
||||
tool_calls = response.tool_calls or []
|
||||
if not tool_calls:
|
||||
break
|
||||
|
||||
messages.append(response)
|
||||
|
||||
submit_called = False
|
||||
for tc in tool_calls:
|
||||
name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
|
||||
args = tc.get("args") or {}
|
||||
tc_id = tc.get("id") or str(uuid.uuid4())
|
||||
|
||||
if name == "submit_result":
|
||||
final_result = args
|
||||
submit_called = True
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc_id,
|
||||
name=name,
|
||||
content=json.dumps({"status": "submitted"}),
|
||||
status="success",
|
||||
)
|
||||
)
|
||||
break
|
||||
|
||||
elif name == "execute_python":
|
||||
code = args.get("code", "")
|
||||
logger.info("Executing Python code via PythonExecutor")
|
||||
exec_output = executor.execute(code)
|
||||
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc_id,
|
||||
name=name,
|
||||
content=exec_output,
|
||||
status="success",
|
||||
)
|
||||
)
|
||||
|
||||
if pool_json_path.exists():
|
||||
try:
|
||||
with open(pool_json_path) as f:
|
||||
loaded_pool = json.load(f)
|
||||
for img_id, entry in loaded_pool.items():
|
||||
if img_id not in self.intermediate_image_pool:
|
||||
self.intermediate_image_pool[img_id] = {
|
||||
"image_id": img_id,
|
||||
"path": entry["path"],
|
||||
"turn_id": iterations,
|
||||
"transform": entry["transform"],
|
||||
"is_output": entry.get("is_output", False),
|
||||
"annotations": entry.get("annotations", {}),
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to read pool_json_path: {e}")
|
||||
|
||||
if submit_called:
|
||||
break
|
||||
|
||||
finally:
|
||||
executor.close()
|
||||
|
||||
outputs = [
|
||||
entry
|
||||
for entry in self.intermediate_image_pool.values()
|
||||
if entry.get("is_output") is True
|
||||
]
|
||||
|
||||
if not outputs:
|
||||
sorted_keys = sorted(
|
||||
self.intermediate_image_pool.keys(),
|
||||
key=lambda k: (
|
||||
int(k.split("_")[1])
|
||||
if k.startswith("img_") and k.split("_")[1].isdigit()
|
||||
else -1
|
||||
),
|
||||
)
|
||||
if sorted_keys:
|
||||
last_key = sorted_keys[-1]
|
||||
outputs = [self.intermediate_image_pool[last_key]]
|
||||
|
||||
summary = (
|
||||
final_result.get("summary", "")
|
||||
if final_result
|
||||
else "Failed to submit result within iterations"
|
||||
)
|
||||
|
||||
return {"outputs": outputs, "summary": summary}
|
||||
@@ -0,0 +1,15 @@
|
||||
# Log Analysis Instructions
|
||||
|
||||
You are the Log Analyzer Agent, responsible for diagnosing issues by querying Android device logs.
|
||||
Your primary goal is to find errors or specific events efficiently without exhausting system resources.
|
||||
|
||||
## Core Strategy: Separation of Search and Read (Grep-like Behavior)
|
||||
You MUST avoid reading massive chunks of logs. Instead, use a "Grep-then-Read" approach:
|
||||
1. **Precision Search (Grep)**: Always start by using `search_logs` with `context_lines=0`. Search for specific keywords, package names (e.g., `net.osmand`), or Exception tags. This will return the exact line numbers and timestamps of the events.
|
||||
2. **Targeted Reading (Read)**: Once you find a highly relevant match from your search (e.g., an error stack trace start), use `read_logs` and specify the `since_time` and `until_time` to fetch ONLY the small 20-50 line window around that exact event.
|
||||
|
||||
## Guidelines
|
||||
1. **Efficiency**: Never call `search_logs` with `context_lines` > 5 unless you are absolutely sure there are very few matches. Large context windows will cause the system to truncate your output and you will lose data.
|
||||
2. **Unknown Errors**: Use `google_search` for unfamiliar error codes or system tags.
|
||||
3. **Handling Missing Data**: If you have executed several searches and cannot find any matching logs, DO NOT loop indefinitely. Return a final summary stating that no logs matching the criteria were found.
|
||||
4. **Response Format**: You should provide your final log analysis in one paragraph. You can organize it into clear, concise parts (e.g., Search Results, Analysis, Key Events, and more). You should include all helpful technical details (e.g. original logs, timestamps, and more).
|
||||
@@ -0,0 +1,431 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Annotated, Any
|
||||
from langchain_core.messages import HumanMessage, SystemMessage, ToolMessage
|
||||
from langchain_core.tools import BaseTool, StructuredTool
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class _CyFunctionDetectorMeta(type):
|
||||
def __instancecheck__(self, instance):
|
||||
name = type(instance).__name__
|
||||
return (
|
||||
name
|
||||
in (
|
||||
"cyfunction",
|
||||
"cython_function_or_method",
|
||||
"builtin_function_or_method",
|
||||
)
|
||||
or "cyfunction" in name.lower()
|
||||
)
|
||||
|
||||
|
||||
class CyFunctionDetector(metaclass=_CyFunctionDetectorMeta):
|
||||
pass
|
||||
|
||||
|
||||
class SpawnLogReaderArgs(BaseModel):
|
||||
model_config = {"ignored_types": (CyFunctionDetector,)}
|
||||
specific_query: str = Field(
|
||||
...,
|
||||
description=(
|
||||
"Specific instructions for the sub-agent, including what to look for and analyze."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
from langgraph.prebuilt import InjectedState
|
||||
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.data_engine.trace import (
|
||||
CURRENT_TRACE_ID,
|
||||
TraceSpan,
|
||||
trace,
|
||||
)
|
||||
from artemis.graph.state import State
|
||||
from artemis.services.llm import get_llm, invoke_llm_with_timeout_message
|
||||
from artemis.tools.index import get_tool_by_name
|
||||
from artemis.tools.mobile.log_utils import fetch_and_filter_logs
|
||||
from artemis.tools.mobile.read_logs import get_read_logs_tool
|
||||
from artemis.tools.mobile.search_logs import (
|
||||
get_search_logs_tool,
|
||||
search_and_merge_logs,
|
||||
)
|
||||
from artemis.tools.tool_wrapper import (
|
||||
get_tool_result_content,
|
||||
invoke_tool_with_injection,
|
||||
)
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class LogAnalyzerNode:
|
||||
def __init__(self, ctx: ArtemisContext):
|
||||
self.ctx = ctx
|
||||
self._log_cache = {}
|
||||
self._raw_log_cache = {}
|
||||
|
||||
def _wrap_with_caching(self, tool: BaseTool) -> BaseTool:
|
||||
"""Wraps a tool with caching logic based on tool name and arguments."""
|
||||
original_coroutine = tool.coroutine
|
||||
|
||||
async def cached_coroutine(**kwargs):
|
||||
if tool.name == "read_logs":
|
||||
# Intercept read_logs to cache raw logs
|
||||
lines = kwargs.get("lines", 200)
|
||||
# Since we might have since_time/until_time, let's keep it simple or key by all.
|
||||
# The instruction says: "cache the returned string based on the lines parameter."
|
||||
# We can also include since_time and until_time in the cache key to be safe, but let's follow the lines-based requirement.
|
||||
cache_key = (
|
||||
lines,
|
||||
kwargs.get("since_time"),
|
||||
kwargs.get("until_time"),
|
||||
)
|
||||
if cache_key in self._raw_log_cache:
|
||||
logger.info(f"Raw log cache hit for read_logs with key {cache_key}")
|
||||
return self._raw_log_cache[cache_key]
|
||||
|
||||
result = await original_coroutine(**kwargs)
|
||||
if isinstance(result, str) and not result.startswith("Failed to read logs"):
|
||||
self._raw_log_cache[cache_key] = result
|
||||
return result
|
||||
|
||||
elif tool.name == "search_logs":
|
||||
lines = kwargs.get("lines", 10000)
|
||||
since_time = kwargs.get("since_time")
|
||||
until_time = kwargs.get("until_time")
|
||||
raw_cache_key = (lines, since_time, until_time)
|
||||
|
||||
# Check if we have the raw logs cached
|
||||
if raw_cache_key in self._raw_log_cache:
|
||||
logger.info(
|
||||
"Raw log cache hit for search_logs with key"
|
||||
f" {raw_cache_key}. Running local search."
|
||||
)
|
||||
raw_logs = self._raw_log_cache[raw_cache_key]
|
||||
# Run search_and_merge_logs locally
|
||||
|
||||
try:
|
||||
return search_and_merge_logs(
|
||||
logs=raw_logs,
|
||||
keyword=kwargs.get("keyword"),
|
||||
context_lines=kwargs.get("context_lines", 0),
|
||||
is_regex=kwargs.get("is_regex", False),
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to search logs locally: {e}")
|
||||
return f"Error searching logs: {e}"
|
||||
|
||||
# If not cached, fetch them once via read_logs (or the original tool, which will fetch them)
|
||||
# Note: search_logs itself fetches and filters. We can let the original_coroutine run,
|
||||
# but to cache the raw logs, we need the raw logs.
|
||||
# Wait, search_logs returns the SEARCHED results, not the raw logs.
|
||||
# So if we just run original_coroutine, we don't get the raw logs to cache them!
|
||||
# To cache them, we should fetch them using get_read_logs_tool or fetch_and_filter_logs,
|
||||
# cache them, and then run search_and_merge_logs.
|
||||
logger.info(
|
||||
"Raw log cache miss for search_logs with key"
|
||||
f" {raw_cache_key}. Fetching raw logs first."
|
||||
)
|
||||
|
||||
try:
|
||||
raw_logs = fetch_and_filter_logs(
|
||||
ctx=self.ctx,
|
||||
lines=lines,
|
||||
since_time=since_time,
|
||||
until_time=until_time,
|
||||
)
|
||||
self._raw_log_cache[raw_cache_key] = raw_logs
|
||||
|
||||
return search_and_merge_logs(
|
||||
logs=raw_logs,
|
||||
keyword=kwargs.get("keyword"),
|
||||
context_lines=kwargs.get("context_lines", 0),
|
||||
is_regex=kwargs.get("is_regex", False),
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch and search logs: {e}")
|
||||
return f"Error searching logs: {e}"
|
||||
|
||||
# Fallback to standard tool caching for other tools
|
||||
serialized_args = json.dumps(kwargs, sort_keys=True)
|
||||
cache_key = f"{tool.name}:{serialized_args}"
|
||||
if cache_key in self._log_cache:
|
||||
logger.info(f"Cache hit for tool {tool.name} with args {kwargs}")
|
||||
return self._log_cache[cache_key]
|
||||
|
||||
logger.info(
|
||||
f"Cache miss for tool {tool.name} with args {kwargs}. Invoking original tool."
|
||||
)
|
||||
result = await original_coroutine(**kwargs)
|
||||
self._log_cache[cache_key] = result
|
||||
return result
|
||||
|
||||
return StructuredTool.from_function(
|
||||
coroutine=cached_coroutine,
|
||||
name=tool.name,
|
||||
description=tool.description,
|
||||
args_schema=tool.args_schema,
|
||||
)
|
||||
|
||||
async def _run_agent_loop(
|
||||
self,
|
||||
base_llm: Any,
|
||||
tools_for_binding: list[Any],
|
||||
tools: list[BaseTool],
|
||||
current_messages: list[Any],
|
||||
state: State,
|
||||
max_iterations: int = 10,
|
||||
agent_name: str = "Agent",
|
||||
) -> str:
|
||||
outcome = "No result"
|
||||
|
||||
for i in range(max_iterations):
|
||||
if i == max_iterations - 1:
|
||||
current_messages.append(
|
||||
HumanMessage(
|
||||
content=(
|
||||
"[WARNING] This is your final iteration; all tools"
|
||||
" are stripped, and you must provide your final"
|
||||
" answer/summary directly."
|
||||
)
|
||||
)
|
||||
)
|
||||
llm = base_llm.bind_tools(tools=[])
|
||||
else:
|
||||
if i > 0:
|
||||
current_messages.append(
|
||||
HumanMessage(
|
||||
content=(
|
||||
"[WARNING] You have not completed the analysis"
|
||||
f" yet (iteration {i + 1} of {max_iterations})."
|
||||
)
|
||||
)
|
||||
)
|
||||
llm = base_llm.bind_tools(tools=tools_for_binding)
|
||||
|
||||
async def run_stream():
|
||||
full_response = None
|
||||
trace_id = CURRENT_TRACE_ID.get()
|
||||
async for chunk in llm.astream(current_messages):
|
||||
if full_response is None:
|
||||
full_response = chunk
|
||||
else:
|
||||
full_response += chunk
|
||||
|
||||
pass
|
||||
return full_response
|
||||
|
||||
response = await invoke_llm_with_timeout_message(run_stream())
|
||||
|
||||
if response.content:
|
||||
outcome = response.content
|
||||
|
||||
if not response.tool_calls:
|
||||
break
|
||||
|
||||
if i == max_iterations - 1:
|
||||
tool_names = ", ".join(tc["name"] for tc in response.tool_calls)
|
||||
warning = (
|
||||
f"\n\n[Warning: {agent_name} reached the maximum iteration"
|
||||
f" limit of {max_iterations} and could not execute tool(s):"
|
||||
f" {tool_names}]"
|
||||
)
|
||||
if outcome == "No result":
|
||||
outcome = (
|
||||
f"{agent_name} reached the maximum iteration limit of"
|
||||
f" {max_iterations} while attempting to call tools:"
|
||||
f" {tool_names}"
|
||||
)
|
||||
else:
|
||||
outcome += warning
|
||||
break
|
||||
|
||||
current_messages.append(response)
|
||||
|
||||
# Handle tool calls in parallel
|
||||
async def run_tool(tc) -> ToolMessage:
|
||||
tool_name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
|
||||
logger.info(f"{agent_name} requested tool: {tool_name}")
|
||||
|
||||
result = ""
|
||||
status = "success"
|
||||
try:
|
||||
if ":" in tool_name:
|
||||
tool_to_run = get_tool_by_name(tool_name, tools)
|
||||
else:
|
||||
tool_to_run = next((t for t in tools if t.name == tool_name), None)
|
||||
if tool_to_run:
|
||||
args = dict(tc["args"])
|
||||
with TraceSpan(name=tool_name, ctx=self.ctx) as span:
|
||||
span.payload = {"args": args}
|
||||
try:
|
||||
result_obj = await invoke_tool_with_injection(
|
||||
tool=tool_to_run,
|
||||
args=args,
|
||||
tool_call_id=tc["id"],
|
||||
state=state,
|
||||
)
|
||||
result = get_tool_result_content(result_obj)
|
||||
if isinstance(result, list):
|
||||
result = "\n".join(map(str, result))
|
||||
elif not isinstance(result, str):
|
||||
result = str(result)
|
||||
span.result = result
|
||||
if result.startswith("Error"):
|
||||
status = "error"
|
||||
span.status = "failed"
|
||||
span.error = result
|
||||
except Exception as e:
|
||||
status = "error"
|
||||
span.status = "failed"
|
||||
span.error = str(e)
|
||||
raise e
|
||||
else:
|
||||
result = f"Error: Tool {tool_name} not supported"
|
||||
status = "error"
|
||||
except Exception as e:
|
||||
logger.error(f"Error running tool {tool_name}: {e}")
|
||||
result = f"Error running tool {tool_name}: {e}"
|
||||
status = "error"
|
||||
|
||||
return ToolMessage(
|
||||
tool_call_id=tc["id"],
|
||||
content=result,
|
||||
status=status,
|
||||
)
|
||||
|
||||
active_tool_calls = [
|
||||
tc
|
||||
for tc in response.tool_calls
|
||||
if (tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"])
|
||||
!= "google_search"
|
||||
]
|
||||
tool_outputs = await asyncio.gather(*(run_tool(tc) for tc in active_tool_calls))
|
||||
|
||||
for tm in tool_outputs:
|
||||
current_messages.append(tm)
|
||||
|
||||
return outcome
|
||||
|
||||
@trace(type="agent", name="log_analyzer")
|
||||
async def run(self, prompt: str, state: State) -> str:
|
||||
"""Runs the main Log Analyzer and returns its summary."""
|
||||
prompt_path = Path(__file__).parent.joinpath("log_analyzer.md")
|
||||
if not prompt_path.exists():
|
||||
raise FileNotFoundError(f"System prompt file not found at {prompt_path}")
|
||||
|
||||
system_prompt = prompt_path.read_text(encoding="utf-8")
|
||||
|
||||
llm = get_llm(ctx=self.ctx, name="log_analyzer")
|
||||
|
||||
all_tools = [
|
||||
self._wrap_with_caching(get_read_logs_tool(self.ctx)),
|
||||
self._wrap_with_caching(get_search_logs_tool(self.ctx)),
|
||||
self._get_spawn_log_reader_tool(),
|
||||
]
|
||||
|
||||
current_messages = [
|
||||
SystemMessage(content=system_prompt),
|
||||
HumanMessage(content=prompt),
|
||||
]
|
||||
|
||||
return await self._run_agent_loop(
|
||||
base_llm=llm,
|
||||
tools_for_binding=all_tools,
|
||||
tools=all_tools,
|
||||
current_messages=current_messages,
|
||||
state=state,
|
||||
max_iterations=10,
|
||||
agent_name="Log Analyzer",
|
||||
)
|
||||
|
||||
@trace(type="agent", name="log_reader_sub_agent")
|
||||
async def _run_log_reader(self, specific_query: str, state: State) -> str:
|
||||
"""Runs the Log Reader Sub-Agent and returns its summary."""
|
||||
sub_prompt_path = Path(__file__).parent.joinpath("log_reader_sub_agent.md")
|
||||
if not sub_prompt_path.exists():
|
||||
sub_prompt_path.write_text(
|
||||
"# Log Reader Sub-Agent\nYou are a specialized agent for"
|
||||
" reading and analyzing Android logs.\nYour job is to fulfill"
|
||||
" the specific prompt given by the Main Agent.\nYou have access"
|
||||
" to tools: `read_logs`.\nAnalyze the logs carefully and"
|
||||
" provide a clear, structured summary of your findings.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
sub_system_prompt = sub_prompt_path.read_text(encoding="utf-8")
|
||||
|
||||
llm = get_llm(ctx=self.ctx, name="log_reader_sub_agent")
|
||||
|
||||
sub_tools = [self._wrap_with_caching(get_read_logs_tool(self.ctx))]
|
||||
|
||||
# Extract the list of keys (metadata) currently stored in _log_cache.
|
||||
cache_keys = list(self._log_cache.keys())
|
||||
|
||||
current_messages = [
|
||||
SystemMessage(content=sub_system_prompt),
|
||||
]
|
||||
|
||||
# Append a system message listing the available cache keys so the sub-agent knows what data can be retrieved instantly.
|
||||
if cache_keys:
|
||||
current_messages.append(
|
||||
SystemMessage(
|
||||
content=(
|
||||
"The following log reads are already cached and can be"
|
||||
" retrieved instantly without cost:\n"
|
||||
)
|
||||
+ "\n".join(f"- {k}" for k in cache_keys)
|
||||
)
|
||||
)
|
||||
|
||||
current_messages.append(HumanMessage(content=specific_query))
|
||||
|
||||
return await self._run_agent_loop(
|
||||
base_llm=llm,
|
||||
tools_for_binding=sub_tools,
|
||||
tools=sub_tools,
|
||||
current_messages=current_messages,
|
||||
state=state,
|
||||
max_iterations=10,
|
||||
agent_name="Log Reader Sub-Agent",
|
||||
)
|
||||
|
||||
def _get_spawn_log_reader_tool(self) -> BaseTool:
|
||||
|
||||
async def spawn_log_reader(
|
||||
specific_query: str,
|
||||
state: Annotated[State, InjectedState] = None,
|
||||
) -> str:
|
||||
"""[SPAWN SUB-AGENT] Delegates complex log analysis to a sub-agent.
|
||||
|
||||
- Use when: Task requires deep analysis, cross-referencing, or
|
||||
reading huge log volumes. - Note: The sub-agent has no search tools
|
||||
and can only read raw logs. - Input: specific_query (str):
|
||||
Instructions for the sub-agent. - Output: A structured summary
|
||||
report from the sub-agent.
|
||||
"""
|
||||
logger.info(f"Spawning Log Reader Sub-Agent with query: '{specific_query}'")
|
||||
return await self._run_log_reader(specific_query, state)
|
||||
|
||||
return StructuredTool.from_function(
|
||||
coroutine=spawn_log_reader,
|
||||
name="spawn_log_reader",
|
||||
args_schema=SpawnLogReaderArgs,
|
||||
)
|
||||
@@ -0,0 +1,8 @@
|
||||
# Log Deep Analysis Instructions
|
||||
|
||||
Fulfill the specific log analysis prompt by reading and synthesizing Android logs.
|
||||
|
||||
## Guidelines
|
||||
1. **Direct Answer**: Address the exact questions or patterns requested in the prompt.
|
||||
2. **Synthesis**: Do not output raw, unsorted logs. Provide a structured summary of findings.
|
||||
3. **Precision**: Include relevant timestamps, line numbers, and error codes to support your conclusions.
|
||||
@@ -0,0 +1,11 @@
|
||||
# Task Output Analyzer
|
||||
|
||||
You are a specialized agent for analyzing the stdout/stderr output of a command executed on an Android device.
|
||||
The user (another agent) will ask you a question about the command's execution output.
|
||||
Analyze the output carefully to answer their question.
|
||||
|
||||
You have access to the following tools to read and search the full output:
|
||||
- `read_task_output`: Reads a range of lines from the output.
|
||||
- `search_task_output`: Searches the output for a keyword or regex.
|
||||
|
||||
Be concise, accurate, and direct in your answer.
|
||||
@@ -0,0 +1,162 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Annotated
|
||||
from langchain_core.messages import HumanMessage, SystemMessage, ToolMessage
|
||||
from langchain_core.tools import tool
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.services.llm import get_llm, invoke_llm_with_timeout_message
|
||||
from artemis.tools.mobile.search_logs import search_and_merge_logs
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class TaskOutputAnalyzerNode:
|
||||
"""Agent that analyzes large command/task execution outputs using LLM and tools."""
|
||||
|
||||
def __init__(self, ctx: ArtemisContext):
|
||||
self.ctx = ctx
|
||||
|
||||
async def run(
|
||||
self,
|
||||
command: str,
|
||||
output_text: str,
|
||||
query: str,
|
||||
) -> str:
|
||||
prompt_path = Path(__file__).parent.joinpath("output_analyzer.md")
|
||||
if not prompt_path.exists():
|
||||
raise FileNotFoundError(f"System prompt file not found at {prompt_path}")
|
||||
|
||||
system_prompt = prompt_path.read_text(encoding="utf-8")
|
||||
|
||||
# Get the LLM for output_analyzer (falls back to log_analyzer)
|
||||
llm = get_llm(ctx=self.ctx, name="output_analyzer")
|
||||
|
||||
# Local tools created with closure on output_text
|
||||
@tool
|
||||
def read_task_output(
|
||||
start_line: Annotated[int, "The 1-indexed line number to start reading from."],
|
||||
end_line: Annotated[int, "The 1-indexed line number to stop reading at (inclusive)."],
|
||||
) -> str:
|
||||
"""Reads a specific range of lines from the task output.
|
||||
|
||||
Use this tool when you know the line number range to read a specific
|
||||
portion of the output.
|
||||
"""
|
||||
lines = output_text.splitlines()
|
||||
if not lines:
|
||||
return "Task output is empty."
|
||||
start = max(1, start_line)
|
||||
end = min(len(lines), end_line)
|
||||
if start > end:
|
||||
return (
|
||||
f"Error: start_line {start_line} is greater than end_line"
|
||||
f" {end_line} or task output length {len(lines)}."
|
||||
)
|
||||
segment = lines[start - 1 : end]
|
||||
return "\n".join(segment)
|
||||
|
||||
@tool
|
||||
def search_task_output(
|
||||
keyword: Annotated[
|
||||
str,
|
||||
"The keyword or regular expression to search for in the output.",
|
||||
],
|
||||
is_regex: Annotated[
|
||||
bool,
|
||||
"Whether to treat the keyword as a regular expression. Default is False.",
|
||||
] = False,
|
||||
context_lines: Annotated[
|
||||
int,
|
||||
"Number of context lines to include before and after the"
|
||||
" matched line. Default is 5.",
|
||||
] = 5,
|
||||
) -> str:
|
||||
"""Searches the task output for a keyword or regular expression, returning matching lines and context.
|
||||
|
||||
Use this tool to quickly locate error messages, warnings, or
|
||||
specific key-value pairs in large outputs.
|
||||
"""
|
||||
return search_and_merge_logs(
|
||||
logs=output_text,
|
||||
keyword=keyword,
|
||||
context_lines=context_lines,
|
||||
is_regex=is_regex,
|
||||
)
|
||||
|
||||
tools = [read_task_output, search_task_output]
|
||||
llm = llm.bind_tools(tools=tools)
|
||||
|
||||
# Optimize prompt context with preview if output is large
|
||||
lines = output_text.splitlines()
|
||||
if len(lines) <= 400:
|
||||
log_preview = f"Full Output:\n{output_text}"
|
||||
else:
|
||||
first_part = "\n".join(lines[:200])
|
||||
last_part = "\n".join(lines[-200:])
|
||||
log_preview = (
|
||||
f"Note: Output is very large ({len(lines)} lines). Here is a"
|
||||
" preview of the start and end of the output:\n\n--- START OF"
|
||||
f" OUTPUT ---\n{first_part}\n--- TRUNCATED {len(lines) - 400}"
|
||||
f" LINES ---\n--- END OF OUTPUT ---\n{last_part}\n\nYou can use"
|
||||
" `read_task_output` or `search_task_output` to fetch any"
|
||||
" lines in the middle if needed."
|
||||
)
|
||||
|
||||
user_content = f"Command Executed: {command}\n\n{log_preview}\n\nQuestion: {query}"
|
||||
|
||||
current_messages = [
|
||||
SystemMessage(content=system_prompt),
|
||||
HumanMessage(content=user_content),
|
||||
]
|
||||
|
||||
max_iterations = 5
|
||||
outcome = "Failed to analyze output."
|
||||
|
||||
for _ in range(max_iterations):
|
||||
response = await invoke_llm_with_timeout_message(llm.ainvoke(current_messages))
|
||||
|
||||
if response.content:
|
||||
outcome = response.content
|
||||
|
||||
if not response.tool_calls:
|
||||
break
|
||||
|
||||
current_messages.append(response)
|
||||
|
||||
for tc in response.tool_calls:
|
||||
tool_name = tc["name"]
|
||||
args = tc["args"]
|
||||
logger.info(f"Output Analyzer requested tool: {tool_name}")
|
||||
|
||||
tool_to_run = next((t for t in tools if t.name == tool_name), None)
|
||||
if tool_to_run:
|
||||
try:
|
||||
result = tool_to_run.invoke(args)
|
||||
except Exception as e:
|
||||
result = f"Failed to run tool {tool_name}: {e}"
|
||||
else:
|
||||
result = f"Error: Tool {tool_name} not found."
|
||||
|
||||
current_messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc["id"],
|
||||
content=str(result),
|
||||
status="success" if not str(result).startswith("Error") else "error",
|
||||
)
|
||||
)
|
||||
|
||||
return outcome
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"templates": [
|
||||
"Point to the following objects in the provided image: {labels_str}.",
|
||||
"Detect the location of the following objects in the image: {labels_str}.",
|
||||
"Locate the following UI components on the screen: {labels_str}."
|
||||
],
|
||||
"global_timeout": 30.0,
|
||||
"instructions": "Return a JSON list containing the detected objects. Each object must have a 'point' array [y, x] (normalized to 0-1000) and a 'label' string.\n\nExample output:\n[\n {\"point\": [500, 250], \"label\": \"search icon\"}\n]\n\nIf no objects are found, return an empty list []."
|
||||
}
|
||||
@@ -0,0 +1,228 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Universal Multi-Model Object Detector for Artemis."""
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
|
||||
from langchain_core.messages import HumanMessage, ToolMessage
|
||||
from langgraph.types import Command
|
||||
|
||||
from artemis.constants import VALIDATOR_MESSAGES_KEY
|
||||
from artemis.services.llm import get_llm
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
_DETECTOR_SEMAPHORE = asyncio.Semaphore(6)
|
||||
|
||||
|
||||
async def _detect_single_label(
|
||||
llm,
|
||||
ctx,
|
||||
image_bytes,
|
||||
mime_type,
|
||||
label,
|
||||
templates,
|
||||
timeout_val=None,
|
||||
) -> list[dict]:
|
||||
if timeout_val is None:
|
||||
timeout_val = float(os.environ.get("OBJECT_DETECTOR_TIMEOUT", "10.0"))
|
||||
|
||||
img_b64 = base64.b64encode(image_bytes).decode("utf-8")
|
||||
|
||||
for i, template in enumerate(templates):
|
||||
prompt = template.replace("{labels_str}", label)
|
||||
messages = [
|
||||
HumanMessage(
|
||||
content=[
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:{mime_type};base64,{img_b64}"},
|
||||
},
|
||||
{"type": "text", "text": prompt},
|
||||
]
|
||||
)
|
||||
]
|
||||
|
||||
try:
|
||||
logger.info(f"Calling Object Detector for '{label}' (Attempt {i + 1})...")
|
||||
async with _DETECTOR_SEMAPHORE:
|
||||
response = await asyncio.wait_for(
|
||||
llm.ainvoke(messages),
|
||||
timeout=timeout_val,
|
||||
)
|
||||
|
||||
output = response.content if isinstance(response.content, str) else ""
|
||||
if isinstance(response.content, list):
|
||||
output = "".join(
|
||||
b.get("text", "")
|
||||
for b in response.content
|
||||
if isinstance(b, dict) and "text" in b
|
||||
)
|
||||
logger.info(f"Received response for '{label}' (Attempt {i + 1})")
|
||||
|
||||
cleaned_initial = output.strip()
|
||||
code_block_match = re.search(r"```(?:json)?\s*([\s\S]*?)\s*```", cleaned_initial)
|
||||
if code_block_match:
|
||||
cleaned_initial = code_block_match.group(1).strip()
|
||||
|
||||
try:
|
||||
res_json = json.loads(cleaned_initial)
|
||||
if isinstance(res_json, list):
|
||||
valid_results = []
|
||||
for item in res_json:
|
||||
if isinstance(item, dict):
|
||||
item["label"] = label
|
||||
valid_results.append(item)
|
||||
return valid_results
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Detection attempt {i + 1} failed for '{label}': {e}")
|
||||
|
||||
return []
|
||||
|
||||
|
||||
async def _run_object_detection(
|
||||
ctx,
|
||||
image_bytes: bytes | str | Path | None = None,
|
||||
queries: list[str] | None = None,
|
||||
templates: list[str] | None = None,
|
||||
mime_type: str = "image/jpeg",
|
||||
global_timeout: float = 30.0,
|
||||
image_path: str | Path | None = None,
|
||||
) -> dict:
|
||||
target_img = image_bytes if image_bytes is not None else image_path
|
||||
if target_img is None:
|
||||
raise ValueError(
|
||||
"Either image_bytes or image_path must be provided to _run_object_detection"
|
||||
)
|
||||
if isinstance(target_img, (str, Path)):
|
||||
image_data = Path(target_img).read_bytes()
|
||||
else:
|
||||
image_data = target_img
|
||||
|
||||
queries = queries or []
|
||||
templates = templates or ["Point to the following objects: {labels_str}"]
|
||||
try:
|
||||
llm = get_llm(ctx, name="object_detector")
|
||||
except Exception:
|
||||
llm = get_llm(ctx, name="operator")
|
||||
|
||||
raw_timeout = getattr(getattr(ctx, "llm_config", None), "timeout", None)
|
||||
if isinstance(raw_timeout, (int, float)):
|
||||
timeout_val = float(raw_timeout)
|
||||
else:
|
||||
timeout_val = float(os.environ.get("OBJECT_DETECTOR_TIMEOUT", "10.0"))
|
||||
|
||||
tasks = [
|
||||
asyncio.create_task(
|
||||
_detect_single_label(
|
||||
llm,
|
||||
ctx,
|
||||
image_data,
|
||||
mime_type,
|
||||
label=query,
|
||||
templates=templates,
|
||||
timeout_val=timeout_val,
|
||||
)
|
||||
)
|
||||
for query in queries
|
||||
]
|
||||
|
||||
done, pending = await asyncio.wait(tasks, timeout=global_timeout)
|
||||
|
||||
for task in pending:
|
||||
task.cancel()
|
||||
if pending:
|
||||
await asyncio.gather(*pending, return_exceptions=True)
|
||||
|
||||
fused_results = []
|
||||
for task in done:
|
||||
try:
|
||||
res = task.result()
|
||||
if res:
|
||||
fused_results.extend(res)
|
||||
except Exception as e:
|
||||
logger.error(f"Task raised exception: {e}")
|
||||
|
||||
# Swap coordinates from [y, x] normalized to [x, y] normalized if needed
|
||||
for item in fused_results:
|
||||
if (
|
||||
isinstance(item, dict)
|
||||
and "point" in item
|
||||
and isinstance(item["point"], list)
|
||||
and len(item["point"]) == 2
|
||||
):
|
||||
y_norm, x_norm = item["point"]
|
||||
item["point"] = [x_norm, y_norm]
|
||||
|
||||
detected_labels = set(
|
||||
item["label"] for item in fused_results if isinstance(item, dict) and "label" in item
|
||||
)
|
||||
failed_queries = [query for query in queries if query not in detected_labels]
|
||||
|
||||
result_dict = {"detected": fused_results, "failed": failed_queries}
|
||||
|
||||
if failed_queries:
|
||||
result_dict["message"] = (
|
||||
"Detection failed for some targets. Please try changing to a more"
|
||||
" detailed description (e.g., describe its shape, color, or nearby"
|
||||
" text)."
|
||||
)
|
||||
|
||||
logger.info("Fused results from parallel inference.")
|
||||
return result_dict
|
||||
|
||||
|
||||
async def _create_error_command(ctx, state, tool_call_id, error_message, wrapper):
|
||||
tool_message = ToolMessage(
|
||||
tool_call_id=tool_call_id or "default_tool_call",
|
||||
content=wrapper.on_failure_fn(error_message),
|
||||
additional_kwargs={"error": error_message},
|
||||
status="error",
|
||||
)
|
||||
return Command(
|
||||
update=await state.asanitize_update(
|
||||
ctx=ctx,
|
||||
update={
|
||||
VALIDATOR_MESSAGES_KEY: [tool_message],
|
||||
},
|
||||
agent="validator",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
async def _create_success_command(ctx, state, tool_call_id, output, wrapper):
|
||||
tool_message = ToolMessage(
|
||||
tool_call_id=tool_call_id or "default_tool_call",
|
||||
content=wrapper.on_success_fn(output),
|
||||
status="success",
|
||||
)
|
||||
return Command(
|
||||
update=await state.asanitize_update(
|
||||
ctx=ctx,
|
||||
update={
|
||||
VALIDATOR_MESSAGES_KEY: [tool_message],
|
||||
},
|
||||
agent="validator",
|
||||
),
|
||||
)
|
||||
@@ -0,0 +1,42 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Loop, deadlock, and action repetition detector for Operator Agent."""
|
||||
|
||||
from typing import Any
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class LoopDetector:
|
||||
"""Monitors recent actions to detect and break infinite interaction loops."""
|
||||
|
||||
def __init__(self, repetition_threshold: int = 3):
|
||||
self.repetition_threshold = repetition_threshold
|
||||
self._action_signatures: list[str] = []
|
||||
|
||||
def record_action(self, action_name: str, params: dict[str, Any]) -> None:
|
||||
sig = f"{action_name}:{sorted(params.items())}"
|
||||
self._action_signatures.append(sig)
|
||||
|
||||
def is_loop_detected(self) -> bool:
|
||||
"""Returns True if the identical action has been repeated >= threshold times."""
|
||||
if len(self._action_signatures) < self.repetition_threshold:
|
||||
return False
|
||||
last_sig = self._action_signatures[-1]
|
||||
return all(s == last_sig for s in self._action_signatures[-self.repetition_threshold :])
|
||||
|
||||
def reset(self) -> None:
|
||||
self._action_signatures.clear()
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List/Text/Complex UI Exploration Strategy (Zero Omission, Zero Duplication, Single-Pass Rule)**:\n When traversing large lists, long text bodies, or continuous content streams, adhere strictly to the **Single-Pass Exploration Standard**:\n 1. **Unidirectional Anchor-Based Traversal (Zero Omission)**: Maintain a consistent, unidirectional scan (e.g., scrolling steadily from top to bottom). Before each scroll, use the bottom-most visible item as a visual anchor. Ensure your scrolling distance allows this anchor to remain visible near the top of the subsequent screen, providing seamless visual overlap so no items are jumped over or omitted. Conclude exploration when reaching the definite boundary (e.g., reaching the bottom of the list or when further scrolling reveals no new content).\n 2. **Memory & Plan-Driven Deduplication (Zero Duplication)**: Systematically log extracted item names, identifiers, or processed statuses into notes or memory as you progress. Upon every screen update, cross-reference visible items with your recorded data and strictly process only *newly surfaced, unrecorded* items to avoid duplicate actions on overlapping elements. Furthermore, actively maintaining and updating subtasks in your task plan (e.g., segmenting exploration by batches, categories, or ranges like a-d) explicitly documents the exact paths and progress already completed, maintaining clear self-awareness of your exploration trajectory and preventing backtracking.\n 3. **Single-Pass Trust & Anti-Oscillation (No Excessive Re-verification)**: Trust your recorded observations as the ground truth. Once a section has been traversed and logged, consider it permanently verified. Never scroll back anLine truncated
|
||||
"troubleshooter_template": "# ROLE\nYou are the Expert Android Device Troubleshooter. A previous attempt to complete the task was rejected by the verification agent (Checker). Your priority is diagnosis and execution accuracy to resolve the rejection.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List/Text/Complex UI Exploration Strategy (Zero Omission, Zero Duplication, Single-Pass Rule)**:\n When traversing large lists, long text bodies, or continuous content streams, adhere strictly to the **Single-Pass Exploration Standard**:\n 1. **Unidirectional Anchor-Based Traversal (Zero Omission)**: Maintain a consistent, unidirectional scan (e.g., scrolling steadily from top to bottom). Before each scroll, use the bottom-most visible item as a visual anchor. Ensure your scrolling distance allows this anchor to remain visible near the top of the subsequent screen, providing seamless visual overlap so no items are jumped over or omitted. Conclude exploration when reaching the definite boundary (e.g., reaching the bottom of the list or when further scrolling reveals no new content).\n 2. **Memory & Plan-Driven Deduplication (Zero Duplication)**: Systematically log extracted item names, identifiers, or processed statuses into notes or memory as you progress. Upon every screen update, cross-reference visible items with your recorded data and strictly process only *newly surfaced, unrecorded* items to avoid duplicate actions on overlapping elements. Furthermore, actively maintaining and updating subtasks in your task plan (e.g., segmenting exploration by batches, categories, or ranges like a-d) explicitly documents the exact paths and progress already completed, maintaining clear self-awareness of your exploration trajectory and preventing backtracking.\n 3. **Single-Pass Trust & Anti-Oscillation (No Excessive Re-verification)**: Trust your recorded observations as the ground truth. Once a section has been traversed and loggedLine truncated
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,72 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Modular Operator Agent implementation adhering to BaseAgent lifecycle."""
|
||||
|
||||
from typing import Any
|
||||
from artemis.agents.base import AgentConfig, AgentRegistry, BaseAgent
|
||||
from artemis.agents.operator.loop_detector import LoopDetector
|
||||
from artemis.agents.operator.prompt_builder import OperatorPromptBuilder
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@AgentRegistry.register("operator")
|
||||
class OperatorAgent(BaseAgent):
|
||||
"""Specialized agent responsible for real-time screen analysis and UI interaction decisions."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config: AgentConfig | None = None,
|
||||
ctx: Any | None = None,
|
||||
**kwargs: Any,
|
||||
):
|
||||
super().__init__(
|
||||
name="operator",
|
||||
role="Mobile UI Interaction Operator",
|
||||
description="Perceives mobile UI and executes touch/key gestures to progress tasks.",
|
||||
config=config,
|
||||
ctx=ctx,
|
||||
)
|
||||
self.loop_detector = LoopDetector(repetition_threshold=3)
|
||||
|
||||
async def process(self, state: Any) -> dict[str, Any]:
|
||||
"""Observes screen state, reasons next action, and dispatches tool call."""
|
||||
if not self.driver:
|
||||
raise RuntimeError("OperatorAgent requires an active device driver in context.")
|
||||
|
||||
# 1. Capture screen data
|
||||
screen_data = await self.driver.get_screen_data()
|
||||
|
||||
# 2. Check for loop deadlock
|
||||
if self.loop_detector.is_loop_detected():
|
||||
logger.warning(
|
||||
"Operator detected repetitive interaction loop! Injecting recovery back-press."
|
||||
)
|
||||
await self.driver.press_key("back")
|
||||
self.loop_detector.reset()
|
||||
return {"action_taken": "press_key:back", "status": "recovering"}
|
||||
|
||||
# 3. Formulate prompt & messages
|
||||
goal = getattr(state, "task_goal", "")
|
||||
human_text = OperatorPromptBuilder.build_human_message(goal=goal)
|
||||
|
||||
return {
|
||||
"thought": "Observed screen and deciding next interaction.",
|
||||
"action_taken": "click",
|
||||
"screen_width": screen_data.width,
|
||||
"screen_height": screen_data.height,
|
||||
"status": "success",
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Dynamic prompt template builder for Operator Agent."""
|
||||
|
||||
from jinja2 import Template
|
||||
|
||||
OPERATOR_SYSTEM_PROMPT = """You are ARTEMIS Operator, an expert autonomous mobile device interaction agent.
|
||||
Your objective is to observe the current Android mobile screen and take precise actions to achieve the user's goal.
|
||||
|
||||
Available action commands:
|
||||
- click(target=[x, y]): Normalized coordinates 0-1000 scale.
|
||||
- swipe(action="up"|"down"|"left"|"right"|[x1, y1, x2, y2]): Drag across screen.
|
||||
- input_text(text="...", target=[x, y]): Click field and type.
|
||||
- press_key(key="home"|"back"|"enter"|"delete"): Press hardware key.
|
||||
- wait_for_delay(seconds=...): Pause execution.
|
||||
"""
|
||||
|
||||
OPERATOR_HUMAN_TEMPLATE = """Goal: {{ goal }}
|
||||
Active Sub-Goal: {{ sub_goal }}
|
||||
Current Turn: {{ current_turn }}
|
||||
Execution History:
|
||||
{{ history }}
|
||||
"""
|
||||
|
||||
|
||||
class OperatorPromptBuilder:
|
||||
"""Builds prompt messages and multimodal contents for the Operator Agent."""
|
||||
|
||||
@classmethod
|
||||
def build_system_message(cls) -> str:
|
||||
return OPERATOR_SYSTEM_PROMPT
|
||||
|
||||
@classmethod
|
||||
def build_human_message(
|
||||
cls,
|
||||
goal: str,
|
||||
sub_goal: str = "",
|
||||
current_turn: int = 1,
|
||||
history: str = "",
|
||||
) -> str:
|
||||
template = Template(OPERATOR_HUMAN_TEMPLATE)
|
||||
return template.render(
|
||||
goal=goal,
|
||||
sub_goal=sub_goal,
|
||||
current_turn=current_turn,
|
||||
history=history,
|
||||
)
|
||||
@@ -0,0 +1,376 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import base64
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from jinja2 import Template
|
||||
from langchain_core.messages import HumanMessage, SystemMessage
|
||||
from PIL import Image
|
||||
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.graph.state import State
|
||||
from artemis.tools.command_tool import (
|
||||
_format_long_output_response,
|
||||
_is_output_long,
|
||||
)
|
||||
from artemis.agents.explorer.constants import EXPLORE_DESCRIPTIONS
|
||||
from artemis.config import resolve_explorer_version
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.task_tree import get_active_subgoal_hashes
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class PromptBuilder:
|
||||
def __init__(self):
|
||||
self.system_parts = []
|
||||
self.human_parts = []
|
||||
self.human_footer = None
|
||||
|
||||
def add_system_text(self, text: str):
|
||||
self.system_parts.append(text)
|
||||
|
||||
def add_human_content(self, content: str | dict):
|
||||
self.human_parts.append(content)
|
||||
|
||||
def set_human_footer(self, content: str):
|
||||
self.human_footer = content
|
||||
|
||||
def build(self) -> list[SystemMessage | HumanMessage]:
|
||||
system_content = "".join(self.system_parts)
|
||||
human_content = []
|
||||
for p in self.human_parts:
|
||||
if isinstance(p, str):
|
||||
human_content.append({"type": "text", "text": p})
|
||||
else:
|
||||
human_content.append(p)
|
||||
|
||||
if self.human_footer:
|
||||
human_content.append({"type": "text", "text": self.human_footer})
|
||||
|
||||
return [
|
||||
SystemMessage(content=system_content),
|
||||
HumanMessage(content=human_content),
|
||||
]
|
||||
|
||||
|
||||
class PromptComponent:
|
||||
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class TemplatePromptComponent(PromptComponent):
|
||||
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
||||
prompts = kwargs.get("prompts", {})
|
||||
template_name = kwargs.get("template_name", "main_template")
|
||||
prompt_template = prompts.get(template_name)
|
||||
if not prompt_template:
|
||||
raise KeyError(
|
||||
"Failed to format prompt, template not found in operator prompts config."
|
||||
)
|
||||
|
||||
op_version = resolve_explorer_version(ctx, agent_or_profile_name="operator")
|
||||
rule_prompt_info = EXPLORE_DESCRIPTIONS.get(op_version, EXPLORE_DESCRIPTIONS["pro"])
|
||||
rule_prompt = rule_prompt_info["rule_prompt"]
|
||||
if "{max_iterations}" in rule_prompt:
|
||||
max_iterations = 8 if op_version == "ultra" else 3
|
||||
rule_prompt = rule_prompt.format(max_iterations=max_iterations)
|
||||
|
||||
prompt_template = prompt_template.replace(
|
||||
"# OPERATIONAL PRINCIPLES\n",
|
||||
f"# OPERATIONAL PRINCIPLES\n- **Visual Explorer Rule**: {rule_prompt}\n",
|
||||
)
|
||||
|
||||
plan_and_history = kwargs.get("plan_and_history", "No plan or history yet.")
|
||||
|
||||
full_prompt = Template(prompt_template).render(
|
||||
initial_goal=state.initial_goal,
|
||||
subgoals_status="",
|
||||
plan_and_history=plan_and_history,
|
||||
unified_history="",
|
||||
)
|
||||
|
||||
parts = full_prompt.split("# CURRENT OBSERVATION")
|
||||
builder.add_system_text(parts[0] + "# CURRENT OBSERVATION\n")
|
||||
|
||||
if len(parts) > 1:
|
||||
builder.set_human_footer(parts[1])
|
||||
|
||||
|
||||
class ObservationPromptComponent(PromptComponent):
|
||||
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
||||
latest_screenshot_b64 = kwargs.get("latest_screenshot_b64")
|
||||
minimal_list = kwargs.get("minimal_list")
|
||||
|
||||
builder.add_human_content("--- Current Screenshot ---")
|
||||
builder.add_human_content(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/jpeg;base64,{latest_screenshot_b64}"},
|
||||
}
|
||||
)
|
||||
builder.add_human_content(f"--- Visible UI Elements ---\n{minimal_list}")
|
||||
|
||||
|
||||
class CheckerFeedbackPromptComponent(PromptComponent):
|
||||
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
||||
if not ctx.data_engine:
|
||||
return
|
||||
|
||||
notes_dir = Path(ctx.data_engine.base_dir) / "notes"
|
||||
|
||||
subgoal_hash = "default"
|
||||
task_plan_path = notes_dir / "task_plan.md"
|
||||
if task_plan_path.exists():
|
||||
try:
|
||||
content = task_plan_path.read_text(encoding="utf-8")
|
||||
|
||||
parent_hash, _ = get_active_subgoal_hashes(content)
|
||||
subgoal_hash = parent_hash
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to parse active subgoal in component: {e}")
|
||||
|
||||
verification_chat_path = notes_dir / f"verification_chat_{subgoal_hash}.json"
|
||||
turns = []
|
||||
if verification_chat_path.exists():
|
||||
try:
|
||||
turns = json.loads(verification_chat_path.read_text(encoding="utf-8"))
|
||||
logger.info(f"Read verification chat for {subgoal_hash}: {len(turns)} turns")
|
||||
except Exception as e:
|
||||
logger.error(f"Error reading verification chat: {e}")
|
||||
|
||||
if turns:
|
||||
dialogue_lines = []
|
||||
for t in turns:
|
||||
role = "Operator" if t["role"] == "operator" else "Checker"
|
||||
dialogue_lines.append(f"**{role} (Round {t['round']})**:\n{t['content']}")
|
||||
checker_feedback = "\n\n".join(dialogue_lines)
|
||||
|
||||
feedback_prompt = f"--- Checker Feedback ---\n{checker_feedback}"
|
||||
|
||||
builder.add_human_content(feedback_prompt)
|
||||
|
||||
|
||||
class BackgroundTasksPromptComponent(PromptComponent):
|
||||
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
||||
active_tasks = kwargs.get("active_background_tasks", [])
|
||||
if active_tasks:
|
||||
lines = [
|
||||
"--- Active Background ADB Tasks ---",
|
||||
]
|
||||
for task in active_tasks:
|
||||
lines.append(
|
||||
f"- TaskId: {task['task_id']}\n Command:"
|
||||
f" `{task['command']}`\n Cwd: `{task['cwd']}`\n "
|
||||
f" TerminalID: `{task['terminal_id']}`\n Accumulated"
|
||||
f" Output: {task['output_line_count']} lines of logs"
|
||||
)
|
||||
builder.add_human_content("\n".join(lines) + "\n")
|
||||
|
||||
newly_finished_tasks = kwargs.get("newly_finished_tasks", [])
|
||||
if newly_finished_tasks:
|
||||
lines = [
|
||||
"--- NEWLY FINISHED ADB TASKS (Since last step) ---",
|
||||
]
|
||||
for task in newly_finished_tasks:
|
||||
task_id = task["task_id"]
|
||||
command = task["command"]
|
||||
status = task["status"]
|
||||
output_text = task.get("output_text", "")
|
||||
|
||||
intro = f"- TaskId: {task_id}\n Command: `{command[:60]}...`\n Status: {status}"
|
||||
|
||||
if _is_output_long(output_text):
|
||||
formatted = _format_long_output_response(task_id, output_text, intro)
|
||||
# format_long_output_response is multi-line, let's indent it nicely
|
||||
indented = "\n".join(f" {line}" for line in formatted.splitlines())
|
||||
lines.append(indented)
|
||||
else:
|
||||
lines.append(f"{intro}\n Final Output:\n {output_text.strip()}")
|
||||
builder.add_human_content("\n".join(lines) + "\n")
|
||||
|
||||
|
||||
class ShortTermMemoryPromptComponent(PromptComponent):
|
||||
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
||||
if state.short_term_memory:
|
||||
builder.add_human_content(
|
||||
f"--- Short-Term Memory (Scratchpad) ---\n{state.short_term_memory}\n"
|
||||
)
|
||||
|
||||
|
||||
class TaskPlanWarningPromptComponent(PromptComponent):
|
||||
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
||||
steps = kwargs.get("steps", [])
|
||||
if len(steps) >= 2:
|
||||
last_two_steps = steps[-2:]
|
||||
modified_in_last_two = False
|
||||
for step in last_two_steps:
|
||||
tool_calls = step.get("tool_calls", [])
|
||||
for tc in tool_calls:
|
||||
if tc.get("name") in [
|
||||
"update_note",
|
||||
"save_note",
|
||||
"append_note",
|
||||
]:
|
||||
args = tc.get("args", {})
|
||||
note_key = args.get("key") or args.get("name")
|
||||
if note_key == "task_plan":
|
||||
modified_in_last_two = True
|
||||
break
|
||||
if modified_in_last_two:
|
||||
break
|
||||
|
||||
if not modified_in_last_two:
|
||||
builder.add_human_content(
|
||||
"\nReminder: You have not updated the task_plan for two"
|
||||
" consecutive turns. Please check the latest progress,"
|
||||
" reflect on whether the task planning is detailed enough,"
|
||||
" and whether every pending item is listed as a subtask."
|
||||
" Please update the completed tasks and pending tasks in"
|
||||
" detail."
|
||||
)
|
||||
|
||||
|
||||
class ToolLimitWarningPromptComponent(PromptComponent):
|
||||
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
||||
if getattr(state, "operator_tool_limit_exceeded", False):
|
||||
builder.add_human_content(
|
||||
"\n Warning: You did not execute any screen interaction actions"
|
||||
" in your last turn. Please re-examine the task goal and revise"
|
||||
" your plan; your current approach may not be the correct path."
|
||||
" Actively calling ask_diagnoser can help you diagnose the"
|
||||
" issue."
|
||||
)
|
||||
|
||||
|
||||
class InjectedInstructionPromptComponent(PromptComponent):
|
||||
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
||||
injected = getattr(state, "injected_instruction", None)
|
||||
if injected:
|
||||
builder.add_human_content(
|
||||
f"\n--- User Guidance ---\n"
|
||||
f"The user observing your progress has provided the following"
|
||||
f" feedback or correction:\n"
|
||||
f'"{injected}"\n\n'
|
||||
f"Please review this guidance, evaluate it against your current"
|
||||
f" screen state and recent history, "
|
||||
f"and integrate it into your reasoning. Use it to refine your"
|
||||
f" task plan and determine the "
|
||||
f"most appropriate next action."
|
||||
)
|
||||
|
||||
|
||||
class ScreenshotSimilarityPromptComponent(PromptComponent):
|
||||
"""Background check: compare current screen against post-action screenshots of the last few steps.
|
||||
|
||||
Inject a note if any are identical. This helps detect if the same screen
|
||||
keeps reappearing unexpectedly.
|
||||
"""
|
||||
|
||||
NUM_STEPS_BACK: int = 3
|
||||
MAX_ALLOWED_DIFF_PIXELS: int = 3
|
||||
COLOR_TOLERANCE: int = 8
|
||||
|
||||
async def __call__(self, builder: PromptBuilder, state: State, ctx: ArtemisContext, **kwargs):
|
||||
if not ctx or not ctx.data_engine:
|
||||
return
|
||||
latest_screenshot_b64 = kwargs.get("latest_screenshot_b64")
|
||||
steps = kwargs.get("steps") or []
|
||||
if not latest_screenshot_b64 or not steps:
|
||||
return
|
||||
# 1. Load current live image and re-encode to JPEG for symmetric compression matching
|
||||
try:
|
||||
# Strip data URI header if present
|
||||
if "," in latest_screenshot_b64:
|
||||
latest_screenshot_b64 = latest_screenshot_b64.split(",", 1)[1]
|
||||
curr_img_bytes = base64.b64decode(latest_screenshot_b64)
|
||||
raw_img = Image.open(io.BytesIO(curr_img_bytes)).convert("RGB")
|
||||
# Symmetric in-memory re-encoding to match JPEG compression artifacts
|
||||
jpeg_buf = io.BytesIO()
|
||||
raw_img.save(jpeg_buf, format="JPEG", quality=75)
|
||||
jpeg_buf.seek(0)
|
||||
curr_img = Image.open(jpeg_buf).convert("RGB")
|
||||
except Exception:
|
||||
return
|
||||
# 2. Get last steps with a recorded post_image_name or pre_image_name
|
||||
history_steps = [s for s in steps if s.get("post_image_name") or s.get("pre_image_name")]
|
||||
recent_steps = history_steps[-self.NUM_STEPS_BACK :]
|
||||
matched_step_nums = []
|
||||
images_dir = Path(ctx.data_engine.global_base_dir) / "images"
|
||||
# 3. Compare current image with each past step's post-action screenshot
|
||||
for step_rec in recent_steps:
|
||||
step_num = step_rec.get("step_number")
|
||||
if step_num is None:
|
||||
continue
|
||||
image_name = step_rec.get("post_image_name") or step_rec.get("pre_image_name")
|
||||
if not image_name:
|
||||
image_name = step_rec.get("post_image_name")
|
||||
if not image_name:
|
||||
continue
|
||||
image_path = images_dir / f"{image_name}.jpg"
|
||||
if not image_path.exists():
|
||||
continue
|
||||
try:
|
||||
past_img = Image.open(image_path).convert("RGB")
|
||||
|
||||
# Must be same dimensions to compare pixel-by-pixel
|
||||
if curr_img.size != past_img.size:
|
||||
continue
|
||||
# Count differing pixels
|
||||
diff_count = self._count_differing_pixels(
|
||||
curr_img, past_img, max_allowed=self.MAX_ALLOWED_DIFF_PIXELS
|
||||
)
|
||||
if diff_count <= self.MAX_ALLOWED_DIFF_PIXELS:
|
||||
matched_step_nums.append(str(step_num))
|
||||
except Exception:
|
||||
continue
|
||||
# 4. Inject note if any identical screenshots are found
|
||||
if matched_step_nums:
|
||||
steps_str = ", ".join(matched_step_nums)
|
||||
note_text = (
|
||||
f"Note: Screenshot in step {steps_str} seem to be identical to"
|
||||
" current screen. This could be intended since not all actions"
|
||||
" alter screens."
|
||||
)
|
||||
builder.add_human_content(note_text)
|
||||
|
||||
def _count_differing_pixels(
|
||||
self,
|
||||
img1: Image.Image,
|
||||
img2: Image.Image,
|
||||
max_allowed: int | None = None,
|
||||
) -> int:
|
||||
"""Count pixels that differ between two RGB images.
|
||||
|
||||
Early-exits if diff_count exceeds max_allowed for performance.
|
||||
"""
|
||||
if max_allowed is None:
|
||||
max_allowed = self.MAX_ALLOWED_DIFF_PIXELS
|
||||
data1 = img1.getdata()
|
||||
data2 = img2.getdata()
|
||||
diff_count = 0
|
||||
tolerance = self.COLOR_TOLERANCE
|
||||
for p1, p2 in zip(data1, data2):
|
||||
if (
|
||||
abs(p1[0] - p2[0]) > tolerance
|
||||
or abs(p1[1] - p2[1]) > tolerance
|
||||
or abs(p1[2] - p2[2]) > tolerance
|
||||
):
|
||||
diff_count += 1
|
||||
if diff_count > max_allowed:
|
||||
break
|
||||
return diff_count
|
||||
@@ -0,0 +1,28 @@
|
||||
# Context & Inputs
|
||||
|
||||
## User's Initial Goal
|
||||
> {{ initial_goal }}
|
||||
|
||||
{% if plan_and_history %}
|
||||
## Execution History & Plan
|
||||
```markdown
|
||||
{{ plan_and_history }}
|
||||
```
|
||||
{% endif %}
|
||||
|
||||
{% if output_description and not structured_output %}
|
||||
## Target Information to Extract
|
||||
The user explicitly requested to extract the following information:
|
||||
> "{{ output_description }}"
|
||||
{% endif %}
|
||||
|
||||
---
|
||||
|
||||
# Instruction
|
||||
|
||||
1. **Assess Confidence & Verify**: Based on the execution history, final state, and the target information to extract, determine if you can confidently formulate the final answer. If the current evidence is insufficient or you require more granular proof, you **MUST** actively gather information and details using your tools first. If your tools fail (e.g., return errors or time out), or if the steps were skipped/not observed, you **MUST** report this lack of evidence honestly. Do not assume, guess, or fabricate any unobserved steps.
|
||||
2. **Write Persistent Report**: Once your investigation is complete, you **MUST** synthesize your findings into a beautifully formatted Markdown summary report (detailing Status [SUCCESS/FAILED/PARTIAL], Evidence, and a clear explanation). You must write this report to persistent memory under the key "output" by calling:
|
||||
`save_note(key="output", content="...your markdown report...")`
|
||||
3. **Final Response**: Only after saving the report, write a direct, helpful, and jargon-free final summary response to the user. Do not use any XML tags, code block wrappers, or special delimiters—just state the conclusion clearly and naturally, ensuring any extracted target information is clearly presented.
|
||||
|
||||
|
||||
@@ -0,0 +1,317 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from jinja2 import Template
|
||||
from langchain_core.messages import BaseMessage, HumanMessage, SystemMessage, ToolMessage
|
||||
from artemis.agents.outputter.tools import (
|
||||
get_search_history_tool,
|
||||
get_step_details_tool,
|
||||
get_step_screenshot_tool,
|
||||
)
|
||||
from artemis.config import OutputConfig
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.data_engine.trace import trace
|
||||
from artemis.graph.state import State
|
||||
from artemis.services.llm import get_llm, invoke_llm_with_timeout_message, with_fallback
|
||||
from artemis.tools.scratchpad import (
|
||||
get_append_note_tool_pure,
|
||||
get_list_notes_tool_pure,
|
||||
get_read_note_tool_pure,
|
||||
get_save_note_tool_pure,
|
||||
get_update_note_tool_pure,
|
||||
)
|
||||
from artemis.tools.video_tool import get_video_analyzer_tool_pure
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.task_tree import build_plan_and_history, get_active_subgoal_hashes
|
||||
from pydantic import BaseModel
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@trace(type="agent", name="outputter")
|
||||
async def outputter(
|
||||
ctx: ArtemisContext,
|
||||
output_config: OutputConfig,
|
||||
graph_output: State,
|
||||
plan_and_history: str | None = None,
|
||||
) -> dict:
|
||||
logger.info("Starting Outputter Agent")
|
||||
|
||||
# Fetch plan and history if not provided
|
||||
if plan_and_history is None and ctx.data_engine:
|
||||
try:
|
||||
history = ctx.data_engine.get_agent_friendly_steps()
|
||||
|
||||
# Read current plan if it exists and base_dir is valid
|
||||
current_plan = ""
|
||||
base_dir = getattr(ctx.data_engine, "base_dir", None)
|
||||
if base_dir and isinstance(base_dir, (str, Path)):
|
||||
notes_dir = Path(base_dir) / "notes"
|
||||
current_path = notes_dir / "task_plan.md"
|
||||
if current_path.exists():
|
||||
try:
|
||||
current_plan = current_path.read_text(encoding="utf-8")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to read current plan for outputter: {e}")
|
||||
|
||||
if history:
|
||||
active_subgoal_hash = "default"
|
||||
if current_plan:
|
||||
try:
|
||||
active_subgoal_hash, _ = get_active_subgoal_hashes(current_plan)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to parse active subgoal in outputter: {e}")
|
||||
|
||||
plan_and_history = build_plan_and_history(
|
||||
current_plan,
|
||||
history,
|
||||
active_subgoal_hash,
|
||||
last_n_detailed=0,
|
||||
min_summaries=len(history),
|
||||
strict_milestone_pruning=False,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to resolve plan and history in outputter: {e}")
|
||||
|
||||
system_message = (
|
||||
"You are the Output Synthesis Agent for an Android UI automation"
|
||||
" system. Your sole objective is to verify whether the user's initial"
|
||||
" goal was achieved and synthesize the final report.\n\n## Core"
|
||||
" Principles\n1. STRICTLY EVIDENCE-BASED & NO HALLUCINATION: Rely ONLY"
|
||||
" on the actual observed execution history and successful tool outputs."
|
||||
" Do not assume, reconstruct, or hallucinate any actions, elements, or"
|
||||
" steps that did not actually occur during this specific run. If a step"
|
||||
" was skipped or not observed (e.g., because the device was already in"
|
||||
" the target state), you must explicitly state that it was 'Not"
|
||||
" Observed / Already in State' rather than fabricating the interaction."
|
||||
" If a tool call (like `video_analyzer`) fails or returns an error, you"
|
||||
" must report the failure and the resulting lack of evidence, rather"
|
||||
" than assuming success or guessing the details. If the user requests a"
|
||||
" step-by-step guide, you must only include steps that were actually"
|
||||
" executed and observed; do not invent 'standard' steps to fill in"
|
||||
" gaps.\n2. NO JARGON IN FINAL ANSWER: The 'Final Answer' must be"
|
||||
" written in clear, user-friendly language, completely free of"
|
||||
" agent-specific jargon (e.g., do not mention 'nodes', 'XPath',"
|
||||
" 'selectors', 'ReAct', or 'tools').\n\n## Tool Usage Guidelines\n-"
|
||||
" LAZY VERIFICATION: If the execution history and the final screenshot"
|
||||
" already in your context provide undeniable proof of the outcome, do"
|
||||
" not call tools. Directly output your conclusion.\n- ACTIVE RETRIEVAL:"
|
||||
" You should only call tools (e.g., `get_step_details`,"
|
||||
" `get_step_screenshot`, `search_history_for_text`) if you need to"
|
||||
" extract specific hidden data (like verification codes, tracking"
|
||||
" numbers) or if the final state is ambiguous.\n- VIDEO ANALYSIS COST:"
|
||||
" The `video_analyzer` tool is expensive. Use it only when necessary"
|
||||
" (e.g., to verify video playback or motion).\n- PERSISTENT WRITE: If"
|
||||
" you extract important information requested by the user (such as a"
|
||||
" verification code, tracking number, or a text summary), you should"
|
||||
" use `save_note` or `update_note` to write it directly into persistent"
|
||||
" notes so it can be retrieved by other agents or the system."
|
||||
)
|
||||
|
||||
render_kwargs = {
|
||||
"initial_goal": graph_output.initial_goal,
|
||||
"structured_output": output_config.structured_output,
|
||||
"output_description": output_config.output_description,
|
||||
"plan_and_history": plan_and_history,
|
||||
}
|
||||
|
||||
human_message = Template(
|
||||
Path(__file__).parent.joinpath("outputter.md").read_text(encoding="utf-8")
|
||||
).render(**render_kwargs)
|
||||
|
||||
screenshot_b64 = (
|
||||
graph_output.operator_raw_data.get("screenshot_b64")
|
||||
if graph_output.operator_raw_data
|
||||
else None
|
||||
)
|
||||
|
||||
human_message_content = [{"type": "text", "text": human_message}]
|
||||
if screenshot_b64:
|
||||
human_message_content.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/jpeg;base64,{screenshot_b64}"},
|
||||
}
|
||||
)
|
||||
|
||||
messages: list[BaseMessage] = [
|
||||
SystemMessage(content=system_message),
|
||||
HumanMessage(content=human_message_content),
|
||||
]
|
||||
|
||||
# Setup tools
|
||||
history_steps = []
|
||||
if ctx.data_engine:
|
||||
history_steps = ctx.data_engine.get_agent_friendly_steps()
|
||||
|
||||
details_tool = get_step_details_tool(history_steps)
|
||||
screenshot_tool = get_step_screenshot_tool(ctx, history_steps)
|
||||
search_tool = get_search_history_tool(ctx)
|
||||
list_notes_tool = get_list_notes_tool_pure(ctx)
|
||||
read_note_tool = get_read_note_tool_pure(ctx)
|
||||
save_note_tool = get_save_note_tool_pure(ctx)
|
||||
update_note_tool = get_update_note_tool_pure(ctx)
|
||||
append_note_tool = get_append_note_tool_pure(ctx)
|
||||
video_analyzer_tool = get_video_analyzer_tool_pure(ctx)
|
||||
|
||||
tools = [
|
||||
details_tool,
|
||||
screenshot_tool,
|
||||
search_tool,
|
||||
list_notes_tool,
|
||||
read_note_tool,
|
||||
save_note_tool,
|
||||
update_note_tool,
|
||||
append_note_tool,
|
||||
video_analyzer_tool,
|
||||
]
|
||||
|
||||
llm = get_llm(ctx=ctx, name="outputter", is_utils=True)
|
||||
llm_fallback = get_llm(ctx=ctx, name="outputter", is_utils=True, use_fallback=True)
|
||||
|
||||
llm_with_tools = llm.bind_tools(tools=tools)
|
||||
llm_fallback_with_tools = llm_fallback.bind_tools(tools=tools)
|
||||
|
||||
# ReAct Loop
|
||||
max_turns = 20
|
||||
raw_answer = None
|
||||
for turn in range(max_turns):
|
||||
logger.info(f"Outputter ReAct turn {turn + 1}")
|
||||
|
||||
response = await with_fallback(
|
||||
main_call=lambda: invoke_llm_with_timeout_message(llm_with_tools.ainvoke(messages)),
|
||||
fallback_call=lambda: invoke_llm_with_timeout_message(
|
||||
llm_fallback_with_tools.ainvoke(messages)
|
||||
),
|
||||
)
|
||||
|
||||
messages.append(response)
|
||||
|
||||
if not response.tool_calls:
|
||||
raw_answer = response.content
|
||||
break
|
||||
|
||||
for tc in response.tool_calls:
|
||||
tool_name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
|
||||
args = tc["args"]
|
||||
|
||||
logger.info(f"Outputter executing tool {tool_name} with args: {args}")
|
||||
try:
|
||||
if tool_name == "get_step_details":
|
||||
result = await details_tool.ainvoke(args)
|
||||
elif tool_name == "get_step_screenshot":
|
||||
result = await screenshot_tool.ainvoke(args)
|
||||
elif tool_name == "search_history_for_text":
|
||||
result = await search_tool.ainvoke(args)
|
||||
elif tool_name == "list_notes":
|
||||
result = await list_notes_tool.ainvoke(args)
|
||||
elif tool_name == "read_note":
|
||||
result = await read_note_tool.ainvoke(args)
|
||||
elif tool_name == "save_note":
|
||||
result = await save_note_tool.ainvoke(args)
|
||||
elif tool_name == "update_note":
|
||||
result = await update_note_tool.ainvoke(args)
|
||||
elif tool_name == "append_note":
|
||||
result = await append_note_tool.ainvoke(args)
|
||||
elif tool_name == "video_analyzer":
|
||||
result = await video_analyzer_tool.ainvoke(args)
|
||||
else:
|
||||
result = f"Error: Tool {tool_name} is not supported."
|
||||
status = (
|
||||
"success"
|
||||
if not (isinstance(result, str) and result.startswith("Error"))
|
||||
else "error"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error running tool {tool_name}: {e}")
|
||||
result = f"Error running tool {tool_name}: {e}"
|
||||
status = "error"
|
||||
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc["id"],
|
||||
content=result if isinstance(result, (str, list)) else str(result),
|
||||
status=status,
|
||||
)
|
||||
)
|
||||
|
||||
if raw_answer is None:
|
||||
raw_answer = "Error: Outputter failed to resolve the query within maximum turns."
|
||||
logger.error(raw_answer)
|
||||
|
||||
# Final Formatting if structured output is requested
|
||||
if output_config.structured_output:
|
||||
logger.info("Formatting raw output to structured format...")
|
||||
schema = None
|
||||
so = output_config.structured_output
|
||||
if isinstance(so, dict):
|
||||
schema = so
|
||||
elif isinstance(so, BaseModel):
|
||||
schema = type(so)
|
||||
elif isinstance(so, type) and issubclass(so, BaseModel):
|
||||
schema = so
|
||||
|
||||
if schema is not None:
|
||||
structured_llm = llm.with_structured_output(schema)
|
||||
structured_llm_fallback = llm_fallback.with_structured_output(schema)
|
||||
|
||||
format_messages = [
|
||||
SystemMessage(
|
||||
content=(
|
||||
"You are a helpful assistant. Your task is to take the"
|
||||
" raw execution summary and format it into the"
|
||||
" requested structured output schema. Do not invent any"
|
||||
" information. If the raw summary does not contain the"
|
||||
" required info, leave those fields empty or null."
|
||||
)
|
||||
),
|
||||
HumanMessage(
|
||||
content=(
|
||||
f"Initial Goal: {graph_output.initial_goal}\nRaw Summary:\n{raw_answer}"
|
||||
)
|
||||
),
|
||||
]
|
||||
|
||||
response = await with_fallback(
|
||||
main_call=lambda: invoke_llm_with_timeout_message(
|
||||
structured_llm.ainvoke(format_messages)
|
||||
),
|
||||
fallback_call=lambda: invoke_llm_with_timeout_message(
|
||||
structured_llm_fallback.ainvoke(format_messages)
|
||||
),
|
||||
)
|
||||
|
||||
if isinstance(response, BaseModel):
|
||||
return response.model_dump()
|
||||
return response
|
||||
|
||||
# Fallback to old behavior if it didn't match the template
|
||||
if isinstance(raw_answer, str):
|
||||
raw_answer_stripped = raw_answer.strip()
|
||||
if raw_answer_stripped.startswith("{") and raw_answer_stripped.endswith("}"):
|
||||
try:
|
||||
return json.loads(raw_answer_stripped)
|
||||
except Exception:
|
||||
pass
|
||||
elif raw_answer_stripped.startswith("```json") and raw_answer_stripped.endswith("```"):
|
||||
try:
|
||||
content = raw_answer_stripped[7:-3].strip()
|
||||
return json.loads(content)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return raw_answer
|
||||
@@ -0,0 +1,170 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import base64
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.tools import BaseTool, tool
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def get_step_details_tool(history_steps: list[dict[str, Any]]) -> BaseTool:
|
||||
@tool
|
||||
def get_step_details(
|
||||
start_step: int,
|
||||
end_step: int,
|
||||
) -> str:
|
||||
"""Retrieve the detailed, full-granularity information for a range of steps (inclusive).
|
||||
|
||||
Use this when you need to inspect specific actions taken, operator
|
||||
thinking, or execution results.
|
||||
"""
|
||||
try:
|
||||
s_step = int(start_step)
|
||||
e_step = int(end_step)
|
||||
except (ValueError, TypeError):
|
||||
return (
|
||||
f"Error: start_step and end_step must be integers, got {start_step} and {end_step}."
|
||||
)
|
||||
|
||||
matched_steps = []
|
||||
for s in history_steps:
|
||||
step_num = s.get("step_number")
|
||||
if step_num is not None and s_step <= step_num <= e_step:
|
||||
details = {
|
||||
"step_number": s.get("step_number"),
|
||||
"relative_time": s.get("relative_time"),
|
||||
"summary": s.get("summary"),
|
||||
"action_taken": s.get("action_taken"),
|
||||
"operator_raw_thinking": s.get("operator_raw_thinking"),
|
||||
"operator_native_thinking": s.get("operator_native_thinking"),
|
||||
"last_execution_result": s.get("last_execution_result"),
|
||||
"interleaved_events": s.get("interleaved_events"),
|
||||
}
|
||||
matched_steps.append(details)
|
||||
if not matched_steps:
|
||||
return f"No steps found in range [{s_step}, {e_step}]."
|
||||
return json.dumps(matched_steps, indent=2, ensure_ascii=False)
|
||||
|
||||
return get_step_details
|
||||
|
||||
|
||||
def get_step_screenshot_tool(ctx: ArtemisContext, history_steps: list[dict[str, Any]]) -> BaseTool:
|
||||
@tool
|
||||
def get_step_screenshot(step_number: int) -> list[dict[str, Any]] | str:
|
||||
"""Retrieve the screenshot for a specific step.
|
||||
|
||||
Returns a multimodal content block containing the image.
|
||||
"""
|
||||
if not ctx.data_engine:
|
||||
return "Error: DataEngine not available."
|
||||
|
||||
try:
|
||||
step_num = int(step_number)
|
||||
except (ValueError, TypeError):
|
||||
return f"Error: step_number must be an integer, got {step_number}."
|
||||
|
||||
# Find the step to get the image name
|
||||
step_record = next((s for s in history_steps if s.get("step_number") == step_num), None)
|
||||
if not step_record:
|
||||
return f"Error: Step {step_num} not found in history."
|
||||
|
||||
image_name = step_record.get("pre_image_name") or step_record.get("post_image_name")
|
||||
if not image_name:
|
||||
return f"Error: No screenshot recorded for step {step_num}."
|
||||
|
||||
images_dir = Path(ctx.data_engine.global_base_dir) / "images"
|
||||
image_path = images_dir / f"{image_name}.jpg"
|
||||
|
||||
if not image_path.exists():
|
||||
return f"Error: Screenshot file for step {step_num} does not exist on disk."
|
||||
|
||||
try:
|
||||
with open(image_path, "rb") as f:
|
||||
img_base64 = base64.b64encode(f.read()).decode("utf-8")
|
||||
except Exception as e:
|
||||
return f"Error reading screenshot file: {e}"
|
||||
|
||||
# Return a LangChain multimodal content structure
|
||||
return [
|
||||
{"type": "text", "text": f"Screenshot for step {step_num}:"},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/jpeg;base64,{img_base64}"},
|
||||
},
|
||||
]
|
||||
|
||||
return get_step_screenshot
|
||||
|
||||
|
||||
def get_search_history_tool(ctx: ArtemisContext) -> BaseTool:
|
||||
@tool
|
||||
def search_history_for_text(query: str) -> str:
|
||||
"""Search for a specific text query within the OCR and UI hierarchy of all historical steps."""
|
||||
if not ctx.data_engine:
|
||||
return "Error: DataEngine not available."
|
||||
|
||||
session_id = ctx.data_engine.current_session_id
|
||||
if not session_id:
|
||||
return "Error: No active session."
|
||||
|
||||
try:
|
||||
steps = ctx.data_engine.storage.get_steps(session_id)
|
||||
except Exception as e:
|
||||
return f"Error retrieving steps from storage: {e}"
|
||||
|
||||
results = []
|
||||
for step in steps:
|
||||
step_matches = []
|
||||
# Search pre-action image
|
||||
if step.pre_image_name:
|
||||
try:
|
||||
matches = ctx.data_engine.storage.search_ui_by_hash(step.pre_image_name, query)
|
||||
if matches:
|
||||
step_matches.extend(matches)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to search UI by hash {step.pre_image_name}: {e}")
|
||||
|
||||
# Search post-action image
|
||||
if step.post_image_name:
|
||||
try:
|
||||
matches = ctx.data_engine.storage.search_ui_by_hash(step.post_image_name, query)
|
||||
if matches:
|
||||
step_matches.extend(matches)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to search UI by hash {step.post_image_name}: {e}")
|
||||
|
||||
if step_matches:
|
||||
# Clean up match details to not bloat context, keep top 3
|
||||
clean_matches = []
|
||||
for m in step_matches[:3]:
|
||||
clean_matches.append(
|
||||
{
|
||||
"type": m.get("type"),
|
||||
"matched_text": m.get("matched_text"),
|
||||
"score": m.get("score"),
|
||||
}
|
||||
)
|
||||
results.append({"step_number": step.step_number, "matches": clean_matches})
|
||||
|
||||
if not results:
|
||||
return f"No matches found for '{query}' in history."
|
||||
return json.dumps(results, indent=2, ensure_ascii=False)
|
||||
|
||||
return search_history_for_text
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"blocks": {
|
||||
"role": "## Role\nYou are the Planner agent. Your objective is to decompose high-level user goals into sequential milestones for mobile device execution.",
|
||||
"role_validator": "## Role\nYou are the Task Plan Validator. The Operator has attempted to modify the top-level task plan. Your job is to ensure the proposed new plan still aligns with the user's original intent and is viable.",
|
||||
"guidelines": "## Operating Principles\n\n### 1. Visual Grounding (Start from Current State)\nBase all planning decisions strictly on the provided screenshot of the current device state. Your plan must begin from the current context of the screen.\n\n### 2. Milestone Granularity\nDefine high-level milestones rather than granular UI operations. The milestones you establish are essential checkpoints that the task must pass through; your goal is to keep the task on the right track, not to think about how the task should be completed step-by-step.\n\n### 3. Flexibility and Possibility Retention (Avoid Over-Specification)\nAvoid unnecessarily restricting the execution path or specific tools/applications unless strictly mandated by the user. For example, if the user asks to perform a task that can be accomplished via multiple interfaces (such as a native app or a web browser), describe the milestone in terms of the desired outcome rather than forcing a specific application or interface (e.g. forcing the use of a specific native app when a web browser could also serve). Allow the downstream execution agent the flexibility to choose the most viable path based on the dynamic state of the device.\n\n### 4. Heuristics for Complex & Iterative Operations\nWhen a milestone requires the downstream agent to perform complex iterations, maintain state, or map computed data back to the UI (e.g., adding a pre-selected list of items to a system), you MUST provide heuristics via indented sub-tasks. These sub-tasks should instruct the agent on the exact iterative process (e.g., searching for each item one by one from a note, applying it, and verifying the final state).",
|
||||
"capabilities_operator": "## Execution Capabilities\n\nThe execution agent primarily interacts with the mobile device and analyzes screen recordings and logs. It can perform the following types of actions:\n- **UI Interaction**: Tapping, swiping, and entering text on the screen.\n- **App Management**: Launching and stopping specific applications.\n- **Information Retrieval & Analysis**: Using OCR to read text on screen, searching the web for external information, querying system logs, and analyzing screen recordings.\n- **Terminal & Background Operations**: Executing arbitrary ADB shell commands on the device for exploration. It is capable of running long-running ADB commands asynchronously in the background as tasks.\n\n### Capability Boundaries & Planning Rules\n- **Cross-App Operations**: When a task requires interacting with multiple applications, split the operations into distinct milestones for each app. Avoid mixing actions for different apps in a single milestone.",
|
||||
"output_protocol_initial": "## Task Plan & Scratchpad\nSubmit your initial plan by invoking the `save_note` tool with `key='task_plan'`. Avoid writing the plan in your text response, as it will only be recognized if submitted via the tool.\n\nFormat the plan as a Markdown task list starting with `- [ ]` for uncompleted items. You can use indented sub-tasks (e.g., ` - [ ]`) to break down complex exploratory steps and provide heuristics to guide the downstream agent. Ensure the content of the `task_plan` note contains only the task list, excluding any introductions, explanations, or headers.\n\n### Creating Additional Notes\nYou are highly recommended to create additional empty notes (like answer sheets) for the downstream executor to fill out with findings, lists, or analytical information relevant to the task. For example, if the user asks you to analyze specific YouTube videos, you should pre-create an empty note with a descriptive name (e.g., `analysis_of_xxx_videos`), and make sure to put the note name directly inside your milestones to guide the downstream executor in filling out this answer sheet.",
|
||||
"examples_initial": "## Examples\n\n### Example 1: Basic Task\n**Goal**: \"Send 'I will be late' to Alice on WhatsApp\"\n\n**Incorrect (Too granular)**:\n- [ ] Tap search icon\n- [ ] Type Alice\n- [ ] Tap Alice's avatar\n\n**Correct (Milestones)**:\n- [ ] Open the chat window with Alice in WhatsApp\n- [ ] Successfully send the message \"I will be late\" in Alice's chat window\n\n### Example 2: Information Retrieval & Scratchpad\n**Goal**: \"Find 10 nature videos on YouTube\"\n\n**Incorrect (Vague or directly jumping to end)**:\n- [ ] Open YouTube\n- [ ] Search nature videos and pick 10\n\n**Correct (Leveraging Scratchpad & Sub-tasks Heuristics)**:\n*(Note: As the Planner, pre-create the required empty notes `nature_videos_candidates` and `nature_videos` using the note-taking tools before completing the plan)*\n- [ ] Search for \"nature videos\" on YouTube\n- [ ] Save a list of 15 candidate videos into a temporary note called `nature_videos_candidates`\n - [ ] Add candidate 1\n - [ ] Add candidate 2\n - [ ] (Continue until 15 candidates are found)\n- [ ] Evaluate candidate videos from `nature_videos_candidates` one by one, select exactly 10 correct ones, and save them into `nature_videos`\n - [ ] Evaluate candidate 1: watch, filter out if irrelevant, save if correct\n - [ ] Evaluate candidate 2\n - [ ] (Continue evaluating one by one until 10 are selected)",
|
||||
"validator_guidelines": "## Validation Principles\n- **Goal Alignment**: Ensure the new plan ultimately accomplishes the `Initial Goal` requested by the user. If the Operator has abandoned the core objective or fundamentally altered it, you must reject the plan.\n- **Logical Coherence**: Verify that the proposed milestones form a logical, achievable sequence to reach the goal. Reject the plan if it introduces contradictory steps, infinite loops, or obvious logical fallacies.\n- **Adaptive Acceptance**: The Operator operates in a dynamic environment and may encounter unexpected UI states. You should approve alternative strategies, extra intermediate steps, or necessary workarounds as long as they serve the primary goal effectively.\n- **Constructive Feedback**: If you must reject the plan, provide clear, objective, and actionable feedback explaining the logical flaw or goal deviation, and suggest how the Operator should correct the plan."
|
||||
},
|
||||
"modes": {
|
||||
"initial_plan": {
|
||||
"system": [
|
||||
"role",
|
||||
"guidelines",
|
||||
"capabilities_operator",
|
||||
"output_protocol_initial",
|
||||
"examples_initial"
|
||||
],
|
||||
"human": "Goal: {{ initial_goal }}"
|
||||
},
|
||||
"validator": {
|
||||
"system": [
|
||||
"role_validator",
|
||||
"validator_guidelines"
|
||||
],
|
||||
"human": "Initial Goal: {{ initial_goal }}\n\n### Execution History\n{{ history_str }}\n\n### Operator's Thoughts Before Modifying\n{% if operator_native_thinking %}Native Thinking:\n{{ operator_native_thinking }}\n\n{% endif %}{% if operator_raw_thinking %}Raw Thinking:\n{{ operator_raw_thinking }}\n\n{% endif %}### Old Plan\n{{ content_before }}\n\n### Proposed New Plan\n{{ content_after }}\n\nPlease evaluate if the proposed new plan should be approved."
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,422 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import base64
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from jinja2 import Template
|
||||
from langchain_core.messages import HumanMessage, SystemMessage, ToolMessage
|
||||
from langchain_core.tools import BaseTool
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.controllers.unified_controller import UnifiedMobileController
|
||||
from artemis.data_engine.trace import CURRENT_TRACE_ID, TraceSpan, trace
|
||||
from artemis.graph.state import State
|
||||
from artemis.services.llm import (
|
||||
get_llm,
|
||||
invoke_llm_with_timeout_message,
|
||||
with_fallback,
|
||||
)
|
||||
from artemis.tools.index import get_tool_by_name, get_tools_from_wrappers
|
||||
from artemis.tools.scratchpad import (
|
||||
list_notes_wrapper,
|
||||
read_note_wrapper,
|
||||
save_note_wrapper,
|
||||
update_note_wrapper,
|
||||
)
|
||||
from artemis.tools.tool_wrapper import (
|
||||
get_tool_result_content,
|
||||
invoke_tool_with_injection,
|
||||
)
|
||||
from artemis.utils.decorators import wrap_with_callbacks
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.notes import get_note_file_path
|
||||
from artemis.utils.task_tree import build_plan_and_history
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class _CyFunctionDetectorMeta(type):
|
||||
def __instancecheck__(self, instance):
|
||||
name = type(instance).__name__
|
||||
return (
|
||||
name
|
||||
in (
|
||||
"cyfunction",
|
||||
"cython_function_or_method",
|
||||
"builtin_function_or_method",
|
||||
)
|
||||
or "cyfunction" in name.lower()
|
||||
)
|
||||
|
||||
|
||||
class CyFunctionDetector(metaclass=_CyFunctionDetectorMeta):
|
||||
pass
|
||||
|
||||
|
||||
class ValidationResult(BaseModel):
|
||||
model_config = {"ignored_types": (CyFunctionDetector,)}
|
||||
is_approved: bool = Field(
|
||||
description=(
|
||||
"True if the new plan is acceptable and aligns with the initial goal, False otherwise."
|
||||
)
|
||||
)
|
||||
feedback: str = Field(
|
||||
description=(
|
||||
"If rejected, provide constructive feedback explaining why and what"
|
||||
" to do instead. If approved, can be empty."
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
async def run_async_planner_validation(
|
||||
ctx: ArtemisContext,
|
||||
initial_goal: str,
|
||||
content_before: str,
|
||||
content_after: str,
|
||||
operator_raw_thinking: str | None = None,
|
||||
operator_native_thinking: str | None = None,
|
||||
) -> dict:
|
||||
"""Asynchronously validates a top-level plan change proposed by the Operator."""
|
||||
logger.info("Starting Async Planner Validation...")
|
||||
try:
|
||||
prompts_path = Path(__file__).parent / "planner.json"
|
||||
with open(prompts_path, encoding="utf-8") as f:
|
||||
prompts_data = json.load(f)
|
||||
|
||||
mode_config = prompts_data["modes"]["validator"]
|
||||
system_blocks = mode_config["system"]
|
||||
system_content = "\n\n".join(prompts_data["blocks"][block] for block in system_blocks)
|
||||
system_message = Template(system_content).render()
|
||||
|
||||
history_str = "No history available."
|
||||
if ctx.data_engine:
|
||||
try:
|
||||
steps = ctx.data_engine.get_agent_friendly_steps()
|
||||
history_str = build_plan_and_history(
|
||||
"", # Pass empty string to avoid rendering the task plan in the history
|
||||
steps,
|
||||
"default",
|
||||
last_n_detailed=2,
|
||||
strict_milestone_pruning=True,
|
||||
recent_window_size=5,
|
||||
chronological_last_step=True,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch execution history for validator: {e}")
|
||||
|
||||
human_content = mode_config["human"]
|
||||
human_message = Template(human_content).render(
|
||||
initial_goal=initial_goal,
|
||||
content_before=content_before,
|
||||
content_after=content_after,
|
||||
operator_raw_thinking=operator_raw_thinking,
|
||||
operator_native_thinking=operator_native_thinking,
|
||||
history_str=history_str,
|
||||
)
|
||||
|
||||
messages = [
|
||||
SystemMessage(content=system_message),
|
||||
HumanMessage(content=human_message),
|
||||
]
|
||||
|
||||
llm = get_llm(ctx=ctx, name="planner").with_structured_output(ValidationResult)
|
||||
|
||||
result: ValidationResult = await invoke_llm_with_timeout_message(llm.ainvoke(messages))
|
||||
|
||||
return {
|
||||
"status": "success" if result.is_approved else "failed",
|
||||
"feedback": result.feedback,
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error(f"Planner async validation failed: {e}")
|
||||
# In case of error, we fallback to approving so we don't block the agent unnecessarily
|
||||
return {
|
||||
"status": "success",
|
||||
"feedback": f"Validation failed with error: {e}",
|
||||
}
|
||||
|
||||
|
||||
class PlannerNode:
|
||||
def __init__(self, ctx: ArtemisContext, tools: list[BaseTool] | None = None):
|
||||
self.ctx = ctx
|
||||
self.tools = tools or []
|
||||
|
||||
@wrap_with_callbacks(
|
||||
before=lambda: logger.info("Starting Planner Agent..."),
|
||||
on_success=lambda _: logger.success("Planner Agent"),
|
||||
on_failure=lambda _: logger.error("Planner Agent"),
|
||||
)
|
||||
@trace(type="agent", name="planner")
|
||||
async def __call__(self, state: State):
|
||||
prompts_path = Path(__file__).parent / "planner.json"
|
||||
with open(prompts_path, encoding="utf-8") as f:
|
||||
prompts_data = json.load(f)
|
||||
|
||||
mode = "initial_plan"
|
||||
mode_config = prompts_data["modes"][mode]
|
||||
|
||||
system_blocks = mode_config["system"]
|
||||
system_content = "\n\n".join(prompts_data["blocks"][block] for block in system_blocks)
|
||||
|
||||
system_message = Template(system_content).render()
|
||||
|
||||
# Get screenshot
|
||||
screenshot_b64 = None
|
||||
screenshot_path = state.latest_screenshot if hasattr(state, "latest_screenshot") else None
|
||||
|
||||
if screenshot_path and Path(screenshot_path).exists():
|
||||
try:
|
||||
with open(screenshot_path, "rb") as f:
|
||||
screenshot_b64 = base64.b64encode(f.read()).decode("utf-8")
|
||||
logger.info(f"Planner loaded screenshot from file: {screenshot_path}")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to read screenshot from {screenshot_path}: {e}")
|
||||
|
||||
# Fallback for Initial Plan: if no screenshot in state, take one on the spot
|
||||
if not screenshot_b64:
|
||||
logger.info("No screenshot found in state. Taking screenshot for Initial Plan...")
|
||||
try:
|
||||
controller = UnifiedMobileController(self.ctx)
|
||||
screenshot_b64 = await controller.take_screenshot()
|
||||
|
||||
# Save to Data Engine if available to keep trace
|
||||
if self.ctx.data_engine:
|
||||
img_bytes = base64.b64decode(screenshot_b64)
|
||||
image_name = self.ctx.data_engine.get_or_create_image(img_bytes)
|
||||
saved_path = str(self.ctx.data_engine.get_image_path(image_name))
|
||||
# Update local state object so it can be used later if needed
|
||||
state.latest_screenshot = saved_path
|
||||
logger.info(f"Saved initial plan screenshot to {saved_path}")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to take screenshot for Initial Plan: {e}")
|
||||
|
||||
if not screenshot_b64:
|
||||
logger.error("Planner failed to obtain a screenshot.")
|
||||
raise ValueError("Planner failed to obtain a screenshot.")
|
||||
|
||||
human_content = mode_config["human"]
|
||||
render_kwargs = {
|
||||
"initial_goal": state.initial_goal,
|
||||
}
|
||||
|
||||
human_message = Template(human_content).render(**render_kwargs)
|
||||
|
||||
human_message_content = [{"type": "text", "text": human_message}]
|
||||
if screenshot_b64:
|
||||
human_message_content.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/jpeg;base64,{screenshot_b64}"},
|
||||
}
|
||||
)
|
||||
messages = [
|
||||
SystemMessage(content=system_message),
|
||||
HumanMessage(content=human_message_content),
|
||||
]
|
||||
|
||||
llm = get_llm(ctx=self.ctx, name="planner")
|
||||
llm_fallback = get_llm(ctx=self.ctx, name="planner", use_fallback=True)
|
||||
|
||||
# Ensure tools are populated (fallback if not passed in __init__, e.g. in tests)
|
||||
if not self.tools:
|
||||
note_wrappers = [
|
||||
save_note_wrapper,
|
||||
read_note_wrapper,
|
||||
update_note_wrapper,
|
||||
list_notes_wrapper,
|
||||
]
|
||||
self.tools = get_tools_from_wrappers(self.ctx, note_wrappers)
|
||||
|
||||
all_tools = list(self.tools)
|
||||
|
||||
if all_tools:
|
||||
llm = llm.bind_tools(all_tools)
|
||||
llm_fallback = llm_fallback.bind_tools(all_tools)
|
||||
|
||||
def validate_plan_format(content: str) -> tuple[bool, str]:
|
||||
lines = content.splitlines()
|
||||
has_subgoal = False
|
||||
for line in lines:
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("- ["):
|
||||
has_subgoal = True
|
||||
if not (
|
||||
stripped.startswith("- [ ]")
|
||||
or stripped.startswith("- [x]")
|
||||
or stripped.startswith("- [!]")
|
||||
):
|
||||
return (
|
||||
False,
|
||||
(
|
||||
f"Invalid task status in line: {line}. Must be"
|
||||
" '[ ]', '[x]', or '[!]'."
|
||||
),
|
||||
)
|
||||
elif stripped.startswith("#") or not stripped:
|
||||
continue
|
||||
else:
|
||||
continue
|
||||
|
||||
if not has_subgoal:
|
||||
return (
|
||||
False,
|
||||
("Plan must contain at least one subgoal starting with '- [ ]'."),
|
||||
)
|
||||
|
||||
return True, ""
|
||||
|
||||
current_messages = messages
|
||||
max_validation_attempts = 3
|
||||
max_iterations = 10
|
||||
|
||||
for validation_attempt in range(max_validation_attempts):
|
||||
tool_called = False
|
||||
|
||||
async def run_stream(llm_model):
|
||||
full_response = None
|
||||
trace_id = CURRENT_TRACE_ID.get()
|
||||
async for chunk in llm_model.astream(current_messages):
|
||||
if full_response is None:
|
||||
full_response = chunk
|
||||
else:
|
||||
full_response += chunk
|
||||
|
||||
pass
|
||||
return full_response
|
||||
|
||||
for _ in range(max_iterations):
|
||||
response = await with_fallback(
|
||||
main_call=lambda: invoke_llm_with_timeout_message(run_stream(llm)),
|
||||
fallback_call=lambda: invoke_llm_with_timeout_message(run_stream(llm_fallback)),
|
||||
)
|
||||
|
||||
if response is None or not response.tool_calls:
|
||||
break
|
||||
|
||||
current_messages.append(response)
|
||||
|
||||
for tc in response.tool_calls:
|
||||
tool_name = tc["name"]
|
||||
logger.info(f"Planner called tool: {tool_name} with args: {tc['args']}")
|
||||
if ":" in tool_name:
|
||||
tool_to_run = get_tool_by_name(tool_name, all_tools)
|
||||
else:
|
||||
tool_to_run = next((t for t in all_tools if t.name == tool_name), None)
|
||||
|
||||
if tool_to_run and tool_to_run.name in [
|
||||
"save_note",
|
||||
"update_note",
|
||||
]:
|
||||
tool_called = True
|
||||
if tool_to_run:
|
||||
args = dict(tc["args"])
|
||||
with TraceSpan(name=tool_name, ctx=self.ctx) as span:
|
||||
span.payload = {"args": args}
|
||||
try:
|
||||
logger.info(
|
||||
f"Tool {tool_name} expected args:"
|
||||
f" {list(tool_to_run.args.keys())}"
|
||||
)
|
||||
|
||||
logger.info(f"Invoking {tool_name} with args: {list(args.keys())}")
|
||||
result_obj = await invoke_tool_with_injection(
|
||||
tool=tool_to_run,
|
||||
args=args,
|
||||
tool_call_id=tc["id"],
|
||||
state=state,
|
||||
)
|
||||
result_content = get_tool_result_content(result_obj)
|
||||
span.result = result_content
|
||||
|
||||
current_messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc["id"],
|
||||
content=result_content,
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
span.status = "failed"
|
||||
span.error = str(e)
|
||||
current_messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc["id"],
|
||||
content=f"Error: {e}",
|
||||
status="error",
|
||||
)
|
||||
)
|
||||
else:
|
||||
current_messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc["id"],
|
||||
content=f"Error: Tool {tool_name} not found",
|
||||
status="error",
|
||||
)
|
||||
)
|
||||
|
||||
# Post-execution validation
|
||||
plan_exists = False
|
||||
plan_valid = False
|
||||
error_message = ""
|
||||
|
||||
if not tool_called:
|
||||
error_message = (
|
||||
"You did not call `save_note` or `update_note` to submit/update the plan."
|
||||
)
|
||||
elif self.ctx.data_engine:
|
||||
task_plan_path = get_note_file_path(self.ctx.data_engine.base_dir, "task_plan")
|
||||
if task_plan_path.exists():
|
||||
plan_exists = True
|
||||
try:
|
||||
content = task_plan_path.read_text(encoding="utf-8")
|
||||
plan_valid, error_message = validate_plan_format(content)
|
||||
except Exception as e:
|
||||
error_message = f"Failed to read task plan for validation: {e}"
|
||||
else:
|
||||
error_message = (
|
||||
"Task plan under key 'task_plan' does not exist despite tool call."
|
||||
)
|
||||
|
||||
if tool_called and plan_exists and plan_valid:
|
||||
logger.info("Plan submitted and validated successfully.")
|
||||
break
|
||||
|
||||
logger.warning(
|
||||
f"Plan validation failed (attempt {validation_attempt + 1}): {error_message}"
|
||||
)
|
||||
|
||||
# Feedback to model
|
||||
current_messages.append(
|
||||
HumanMessage(
|
||||
content=(
|
||||
f"Plan validation failed: {error_message}\nPlease"
|
||||
" correct the plan and try again by calling the"
|
||||
" appropriate tool."
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
else:
|
||||
logger.error(
|
||||
f"Failed to generate a valid plan after {max_validation_attempts} attempts."
|
||||
)
|
||||
|
||||
return await state.asanitize_update(
|
||||
ctx=self.ctx,
|
||||
update={},
|
||||
agent="planner",
|
||||
)
|
||||
@@ -0,0 +1,53 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Planner Agent implementation adhering to BaseAgent lifecycle."""
|
||||
|
||||
from typing import Any
|
||||
from artemis.agents.base import AgentConfig, AgentRegistry, BaseAgent
|
||||
from artemis.agents.planner.planner import PlannerNode
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@AgentRegistry.register("planner")
|
||||
class PlannerAgent(BaseAgent):
|
||||
"""Specialized agent responsible for high-level goal decomposition and strategic task planning."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config: AgentConfig | None = None,
|
||||
ctx: Any | None = None,
|
||||
**kwargs: Any,
|
||||
):
|
||||
super().__init__(
|
||||
name="planner",
|
||||
role="Strategic Task Planner",
|
||||
description="Decomposes user objectives into structured, observable milestone steps.",
|
||||
config=config,
|
||||
ctx=ctx,
|
||||
)
|
||||
self._node = PlannerNode(ctx=ctx, **kwargs) if ctx else None
|
||||
|
||||
async def initialize(self, ctx: Any) -> None:
|
||||
await super().initialize(ctx)
|
||||
self._node = PlannerNode(ctx=ctx)
|
||||
|
||||
async def process(self, state: Any) -> dict[str, Any]:
|
||||
if not self._node and self.ctx:
|
||||
self._node = PlannerNode(ctx=self.ctx)
|
||||
if not self._node:
|
||||
raise RuntimeError("PlannerAgent has not been initialized with a ArtemisContext.")
|
||||
return await self._node(state)
|
||||
@@ -0,0 +1,85 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from typing import Annotated
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class _CyFunctionDetectorMeta(type):
|
||||
def __instancecheck__(self, instance):
|
||||
name = type(instance).__name__
|
||||
return (
|
||||
name
|
||||
in (
|
||||
"cyfunction",
|
||||
"cython_function_or_method",
|
||||
"builtin_function_or_method",
|
||||
)
|
||||
or "cyfunction" in name.lower()
|
||||
)
|
||||
|
||||
|
||||
class CyFunctionDetector(metaclass=_CyFunctionDetectorMeta):
|
||||
pass
|
||||
|
||||
|
||||
class PlannerSubgoalOutput(BaseModel):
|
||||
model_config = {"ignored_types": (CyFunctionDetector,)}
|
||||
description: str
|
||||
|
||||
|
||||
class PlannerOutput(BaseModel):
|
||||
model_config = {"ignored_types": (CyFunctionDetector,)}
|
||||
subgoals: list[PlannerSubgoalOutput]
|
||||
|
||||
|
||||
class SubgoalStatus(Enum):
|
||||
NOT_STARTED = "NOT_STARTED"
|
||||
PENDING = "PENDING"
|
||||
SUCCESS = "SUCCESS"
|
||||
FAILURE = "FAILURE"
|
||||
|
||||
|
||||
class Subgoal(BaseModel):
|
||||
model_config = {"ignored_types": (CyFunctionDetector,)}
|
||||
id: Annotated[str, "Unique identifier of the subgoal"]
|
||||
description: Annotated[str, "Description of the subgoal"]
|
||||
completion_reason: Annotated[
|
||||
str | None, "Reason why the subgoal was completed (failure or success)"
|
||||
] = None
|
||||
status: SubgoalStatus
|
||||
started_at: Annotated[datetime | None, "When the subgoal started"] = None
|
||||
ended_at: Annotated[datetime | None, "When the subgoal ended"] = None
|
||||
|
||||
def __str__(self):
|
||||
status_emoji = "❓"
|
||||
if self.status == SubgoalStatus.SUCCESS:
|
||||
status_emoji = "✅"
|
||||
elif self.status == SubgoalStatus.FAILURE:
|
||||
status_emoji = "❌"
|
||||
elif self.status == SubgoalStatus.PENDING:
|
||||
status_emoji = "⏳"
|
||||
elif self.status == SubgoalStatus.NOT_STARTED:
|
||||
status_emoji = "(not started yet)"
|
||||
|
||||
output = f"- [ID:{self.id}]: {self.description} : {status_emoji}."
|
||||
if self.completion_reason:
|
||||
output += f" Completion reason: {self.completion_reason}"
|
||||
return output
|
||||
|
||||
def __repr__(self):
|
||||
return str(self)
|
||||
@@ -0,0 +1,89 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
try:
|
||||
from datetime import UTC, datetime
|
||||
except ImportError:
|
||||
from datetime import datetime
|
||||
|
||||
UTC = UTC
|
||||
import random
|
||||
import string
|
||||
|
||||
from artemis.agents.planner.types import Subgoal, SubgoalStatus
|
||||
|
||||
|
||||
def get_current_subgoal(subgoals: list[Subgoal]) -> Subgoal | None:
|
||||
return next((s for s in subgoals if s.status == SubgoalStatus.PENDING), None)
|
||||
|
||||
|
||||
def get_subgoals_by_ids(subgoals: list[Subgoal], ids: list[str]) -> list[Subgoal]:
|
||||
return [s for s in subgoals if s.id in ids]
|
||||
|
||||
|
||||
def get_next_subgoal(subgoals: list[Subgoal]) -> Subgoal | None:
|
||||
return next((s for s in subgoals if s.status == SubgoalStatus.NOT_STARTED), None)
|
||||
|
||||
|
||||
def nothing_started(subgoals: list[Subgoal]) -> bool:
|
||||
return all(s.status == SubgoalStatus.NOT_STARTED for s in subgoals)
|
||||
|
||||
|
||||
def complete_current_subgoal(subgoals: list[Subgoal]) -> list[Subgoal]:
|
||||
current_subgoal = get_current_subgoal(subgoals)
|
||||
if not current_subgoal:
|
||||
return subgoals
|
||||
current_subgoal.status = SubgoalStatus.SUCCESS
|
||||
current_subgoal.ended_at = datetime.now(UTC)
|
||||
return subgoals
|
||||
|
||||
|
||||
def complete_subgoals_by_ids(subgoals: list[Subgoal], ids: list[str]) -> list[Subgoal]:
|
||||
for subgoal in subgoals:
|
||||
if subgoal.id in ids:
|
||||
subgoal.status = SubgoalStatus.SUCCESS
|
||||
subgoal.ended_at = datetime.now(UTC)
|
||||
return subgoals
|
||||
|
||||
|
||||
def fail_current_subgoal(subgoals: list[Subgoal]) -> list[Subgoal]:
|
||||
current_subgoal = get_current_subgoal(subgoals)
|
||||
if not current_subgoal:
|
||||
return subgoals
|
||||
current_subgoal.status = SubgoalStatus.FAILURE
|
||||
current_subgoal.ended_at = datetime.now(UTC)
|
||||
return subgoals
|
||||
|
||||
|
||||
def all_completed(subgoals: list[Subgoal]) -> bool:
|
||||
return all(s.status == SubgoalStatus.SUCCESS for s in subgoals)
|
||||
|
||||
|
||||
def one_of_them_is_failure(subgoals: list[Subgoal]) -> bool:
|
||||
return any(s.status == SubgoalStatus.FAILURE for s in subgoals)
|
||||
|
||||
|
||||
def start_next_subgoal(subgoals: list[Subgoal]) -> list[Subgoal]:
|
||||
next_subgoal = get_next_subgoal(subgoals)
|
||||
if not next_subgoal:
|
||||
return subgoals
|
||||
next_subgoal.status = SubgoalStatus.PENDING
|
||||
next_subgoal.started_at = datetime.now(UTC)
|
||||
return subgoals
|
||||
|
||||
|
||||
def generate_id(length: int = 6) -> str:
|
||||
"""Generates a small and distinct random string ID."""
|
||||
chars = string.ascii_lowercase + string.digits
|
||||
return "".join(random.choice(chars) for _ in range(length))
|
||||
@@ -0,0 +1,31 @@
|
||||
# ROLE
|
||||
You are the Summarizer Agent for a mobile automation system. Your task is to summarize the latest execution step, providing a high-information-density and succinct history summary of the current step without losing any information.
|
||||
|
||||
# OUTPUT FORMAT
|
||||
- **Format Constraint**: You MUST output the summary as a **single, continuous paragraph**.
|
||||
- **Prohibited Formats**: Do NOT use bullet points, numbered lists, markdown formatting, or multiple paragraphs.
|
||||
- **Goal**: Maintain a consistent, easily parsable narrative for the Operator's historical record.
|
||||
|
||||
# CORE PHILOSOPHY & RULES
|
||||
|
||||
1. **Perspective**: You MUST write the summary from the Operator's first-person perspective using "I" (e.g., "I intended to...", "I clicked..."). NEVER use third-person terms like "The operator" or "The agent", nor second-person terms like "You" when referring to the Operator. For other agents (e.g., Failure Analyzer, Explorer), you can use their names.
|
||||
2. **Zero Subjective Validation**: Absolutely avoid declaring semantic success, completion, or failure. In historical compression, any subjective validation will cause future planners to falsely believe a goal was definitively met or permanently broken.
|
||||
- **BANNED WORDS**: "successfully", "completed", "entered", "navigated to", "failed", "unsuccessful", "could not", "achieved".
|
||||
3. **Focus on Intent & Strategy**: The specific physical action (exact coordinates, swipes) is now recorded automatically. Focus primarily on the high-level Intent and reasoning. If you realize the Operator is on the wrong path and needs to backtrack or pivot, describe this as a shift in strategy.
|
||||
4. **Intended Context Transitions**: Describe the intended navigation path ("From Where -> Intended Where"). Do not declare that the transition actually happened. (e.g., "From the Home screen, I intended to open the Network Settings view").
|
||||
5. **Preserve Verifications & Missing Items**: If the Operator verified any state in this step, or explicitly mentioned what prerequisites or items are still missing to accomplish the goal, you MUST state these contents in detail in the summary.
|
||||
6. **Objective Intervention Logs**: If the Failure Analyzer intervened, faithfully summarize what triggered the intervention and what the analyzer executed, without judging the outcome.
|
||||
|
||||
# LOOP & STAGNATION DETECTION
|
||||
Use the `RECENT 10 STEPS HISTORY` to intelligently detect loops or stagnation:
|
||||
1. **Physical Action Loops**: Be vigilant for loops spanning multiple steps based on physical interactions. If you detect a valid action loop, explicitly mention the pattern and the current iteration count (e.g., "This is the 3rd cycle of attempting to access the video group folder").
|
||||
2. **Cognitive Stagnation**: While you generally ignore internal background tool calls (e.g., reading logs, checking states) when counting physical action loops, there is an exception: if you detect continuous multiple steps of purely internal tool calls without any physical actions, you MUST objectively record this strategic "stagnation" or "analysis loop".
|
||||
|
||||
# INPUT STRUCTURE
|
||||
You will receive:
|
||||
- **RECENT 10 STEPS HISTORY**: A brief history of the last 10 steps to help you detect loops and stagnation patterns.
|
||||
- **CHRONOLOGICAL STEP TRACE (CURRENT STEP)**: The step-by-step chronological history of the current execution turn, including:
|
||||
- `[Operator Monologue] / [Operator Native Thought]`: The operator's planning, intentions, progress counting, reasoning, verified states, and explicit mentions of missing prerequisites.
|
||||
- `[Operator Tool Call] / [Operator Final Action]`: The actual action(s) planned and executed.
|
||||
- `[Pre-Execution Safety Net]`: Validation checks prior to action execution.
|
||||
- `[Failure Analyzer Recovery Loop]`: Interventions, thoughts, auxiliary tool calls, and outcomes of the Failure Analyzer.
|
||||
@@ -0,0 +1,231 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from pathlib import Path
|
||||
from uuid import UUID
|
||||
|
||||
from langchain_core.messages import (
|
||||
HumanMessage,
|
||||
SystemMessage,
|
||||
)
|
||||
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.data_engine.trace import CURRENT_TRACE_ID, trace
|
||||
from artemis.graph.state import State
|
||||
from artemis.services.llm import get_llm
|
||||
from artemis.utils.coordinates import normalize_any_structure
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.task_tree import (
|
||||
build_plan_and_history,
|
||||
get_active_subgoal_hashes,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class SummarizerNode:
|
||||
def __init__(self, ctx: ArtemisContext):
|
||||
self.ctx = ctx
|
||||
|
||||
@trace(type="agent", name="summarizer")
|
||||
async def __call__(self, state: State):
|
||||
# 1. Trigger background task for summary and data engine update
|
||||
if self.ctx.data_engine and state.current_step_id:
|
||||
step_id = UUID(state.current_step_id)
|
||||
decisions = state.structured_decisions
|
||||
operator_raw_thinking = getattr(state, "operator_raw_thinking", None)
|
||||
operator_native_thinking = getattr(state, "operator_native_thinking", None)
|
||||
last_execution_result = getattr(state, "last_execution_result", None)
|
||||
|
||||
# Spawn background task
|
||||
trace_id = CURRENT_TRACE_ID.get()
|
||||
task = asyncio.create_task(
|
||||
self._generate_summary_and_update_engine(
|
||||
step_id,
|
||||
decisions,
|
||||
operator_raw_thinking,
|
||||
operator_native_thinking,
|
||||
last_execution_result,
|
||||
trace_id,
|
||||
),
|
||||
name=f"summarizer_step_{step_id}",
|
||||
)
|
||||
if hasattr(self.ctx, "background_tasks") and isinstance(
|
||||
self.ctx.background_tasks, list
|
||||
):
|
||||
self.ctx.background_tasks.append(task)
|
||||
|
||||
return {}
|
||||
|
||||
async def _generate_summary_and_update_engine(
|
||||
self,
|
||||
step_id,
|
||||
decisions,
|
||||
operator_raw_thinking,
|
||||
operator_native_thinking,
|
||||
last_execution_result,
|
||||
trace_id,
|
||||
):
|
||||
"""Background task to generate summary and update Data Engine with retries."""
|
||||
|
||||
max_retries = 3
|
||||
retry_delay = 2
|
||||
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
logger.info(f"Generating summary for step {step_id} (attempt {attempt + 1})")
|
||||
|
||||
# Prepare prompt for summary
|
||||
llm = get_llm(ctx=self.ctx, name="summarizer")
|
||||
|
||||
prompt_path = Path(__file__).parent.joinpath("summarizer.md")
|
||||
if not prompt_path.exists():
|
||||
prompt = (
|
||||
"You are a summarizer. Summarize the latest action and"
|
||||
" its result in 1-2 concise sentences. Focus on: Action"
|
||||
" taken -> Result -> Observation."
|
||||
)
|
||||
else:
|
||||
prompt = prompt_path.read_text(encoding="utf-8")
|
||||
|
||||
# Normalize coordinates before presenting to the summarizer LLM
|
||||
width = (
|
||||
getattr(self.ctx.device, "device_width", 1080)
|
||||
if self.ctx and self.ctx.device
|
||||
else 1080
|
||||
)
|
||||
height = (
|
||||
getattr(self.ctx.device, "device_height", 2400)
|
||||
if self.ctx and self.ctx.device
|
||||
else 2400
|
||||
)
|
||||
|
||||
content = []
|
||||
|
||||
# Consistently use optimal build_plan_and_history to construct accurate context
|
||||
plan_and_history_str = ""
|
||||
if self.ctx.data_engine:
|
||||
try:
|
||||
friendly_steps = self.ctx.data_engine.get_agent_friendly_steps()
|
||||
if isinstance(friendly_steps, list):
|
||||
# 1. Trim history to only include steps up to the current step being summarized (inclusive)
|
||||
current_steps = []
|
||||
for s in friendly_steps:
|
||||
current_steps.append(s)
|
||||
if str(s.get("step_id")) == str(step_id):
|
||||
break
|
||||
|
||||
# 2. Read task plan to provide goal orientation to the summarizer
|
||||
task_plan = ""
|
||||
base_dir = getattr(self.ctx.data_engine, "base_dir", None)
|
||||
if base_dir and isinstance(base_dir, (str, Path)):
|
||||
notes_dir = Path(base_dir) / "notes"
|
||||
task_plan_path = notes_dir / "task_plan.md"
|
||||
if task_plan_path.exists():
|
||||
task_plan = task_plan_path.read_text(encoding="utf-8")
|
||||
|
||||
# 3. Unified rendering: concise for first N steps (with self-healing facts), detailed for current step (without redundant summary header)
|
||||
|
||||
subgoal_hash = "default"
|
||||
if task_plan:
|
||||
subgoal_hash, _ = get_active_subgoal_hashes(task_plan)
|
||||
|
||||
plan_and_history_str = build_plan_and_history(
|
||||
task_plan=task_plan,
|
||||
steps=current_steps,
|
||||
current_subgoal_hash=subgoal_hash,
|
||||
last_n_detailed=1,
|
||||
min_summaries=10,
|
||||
strict_milestone_pruning=False,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to build plan and history for Summarizer: {e}")
|
||||
|
||||
if plan_and_history_str:
|
||||
content.append(plan_and_history_str)
|
||||
else:
|
||||
# Fallback to standard raw fields if chronological trace couldn't be generated
|
||||
if decisions:
|
||||
try:
|
||||
dec_obj = json.loads(decisions)
|
||||
dec_obj = normalize_any_structure(dec_obj, width, height)
|
||||
content.append(
|
||||
f"Decisions made: {json.dumps(dec_obj, ensure_ascii=False)}"
|
||||
)
|
||||
except Exception:
|
||||
content.append(f"Decisions made: {decisions}")
|
||||
|
||||
if operator_raw_thinking:
|
||||
content.append(f"Operator explicit thoughts:\n{operator_raw_thinking}")
|
||||
|
||||
if operator_native_thinking:
|
||||
content.append(f"Operator native thoughts:\n{operator_native_thinking}")
|
||||
|
||||
messages = [
|
||||
SystemMessage(content=prompt),
|
||||
HumanMessage(content="\n\n".join(content)),
|
||||
]
|
||||
|
||||
generated_text = ""
|
||||
aggregated_chunk = None
|
||||
async for chunk in llm.astream(messages):
|
||||
if aggregated_chunk is None:
|
||||
aggregated_chunk = chunk
|
||||
else:
|
||||
aggregated_chunk += chunk
|
||||
|
||||
if chunk.content:
|
||||
text_to_stream = ""
|
||||
if isinstance(chunk.content, str):
|
||||
text_to_stream = chunk.content
|
||||
elif isinstance(chunk.content, list):
|
||||
for item in chunk.content:
|
||||
if isinstance(item, dict) and item.get("type") == "text":
|
||||
text_to_stream += item.get("text", "")
|
||||
|
||||
if text_to_stream:
|
||||
generated_text += text_to_stream
|
||||
pass
|
||||
|
||||
summary = generated_text.strip()
|
||||
logger.info(f"Summary generated: {summary}")
|
||||
|
||||
if aggregated_chunk and getattr(aggregated_chunk, "usage_metadata", None):
|
||||
usage = aggregated_chunk.usage_metadata
|
||||
cached_tokens = usage.get("input_token_details", {}).get("cache_read", 0)
|
||||
logger.info(
|
||||
"Summarizer LLM usage:"
|
||||
f" prompt_tokens={usage.get('input_tokens')}"
|
||||
f" (cached={cached_tokens}),"
|
||||
f" completion_tokens={usage.get('output_tokens')}"
|
||||
)
|
||||
|
||||
# Update Data Engine
|
||||
self.ctx.data_engine.update_step_summary(step_id, summary)
|
||||
logger.info(f"Data Engine updated with summary for step {step_id}")
|
||||
return # Success
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to generate summary or update engine (attempt {attempt + 1}): {e}"
|
||||
)
|
||||
if attempt < max_retries - 1:
|
||||
await asyncio.sleep(retry_delay)
|
||||
retry_delay *= 2 # Exponential backoff
|
||||
else:
|
||||
logger.error(
|
||||
f"Max retries reached for step {step_id}. Summary generation failed."
|
||||
)
|
||||
@@ -0,0 +1,53 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Summarizer Agent implementation adhering to BaseAgent lifecycle."""
|
||||
|
||||
from typing import Any
|
||||
from artemis.agents.base import AgentConfig, AgentRegistry, BaseAgent
|
||||
from artemis.agents.summarizer.summarizer import SummarizerNode
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@AgentRegistry.register("summarizer")
|
||||
class SummarizerAgent(BaseAgent):
|
||||
"""Specialized agent responsible for step execution summarization and trajectory logging."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config: AgentConfig | None = None,
|
||||
ctx: Any | None = None,
|
||||
**kwargs: Any,
|
||||
):
|
||||
super().__init__(
|
||||
name="summarizer",
|
||||
role="Execution History Summarizer",
|
||||
description="Synthesizes actions and observations into concise milestone summaries.",
|
||||
config=config,
|
||||
ctx=ctx,
|
||||
)
|
||||
self._node = SummarizerNode(ctx=ctx) if ctx else None
|
||||
|
||||
async def initialize(self, ctx: Any) -> None:
|
||||
await super().initialize(ctx)
|
||||
self._node = SummarizerNode(ctx=ctx)
|
||||
|
||||
async def process(self, state: Any) -> dict[str, Any]:
|
||||
if not self._node and self.ctx:
|
||||
self._node = SummarizerNode(ctx=self.ctx)
|
||||
if not self._node:
|
||||
raise RuntimeError("SummarizerAgent has not been initialized with a ArtemisContext.")
|
||||
return await self._node(state)
|
||||
@@ -0,0 +1,25 @@
|
||||
# Identity & Goal
|
||||
You are a specialized error recovery agent. You will be provided with a failed action. Your ONLY task is to repair this specific failure and restore the system state so the main system can resume.
|
||||
|
||||
# Context
|
||||
- **Initial Goal**: {{ initial_goal }}
|
||||
- **History & Plan**:
|
||||
{{ plan_and_history }}
|
||||
|
||||
# Core Workflow
|
||||
1. **Analyze**: Compare the two states provided to you:
|
||||
- **Decision State (`Screenshot Seen During System Decision`)**: This is the screen the system saw when it originally planned the action. It shows the target element successfully located.
|
||||
- **Failed State (`Latest Screenshot (Failed State)`)**: This is the live screen captured at the moment you were summoned. It shows the current state where the target is missing or unresponsive.
|
||||
Your goal in this comparison is to:
|
||||
- Read the Operator's thought process in the decision loop of the failed step to understand what they were originally trying to accomplish with this failed action.
|
||||
- Understand the expected target layout.
|
||||
- Identify the cause of the failure based on the error message and the visual mismatch.
|
||||
2. **Recover & Execute**: Restore the target state and execute the failed action.
|
||||
- **Flexibility**: Focus on outcome over form. Take alternative/adaptive steps (e.g., scroll, navigate, different click paths) if original coordinates are invalid.
|
||||
3. **Report**: Call `report_failure_analysis` to conclude:
|
||||
- `status="fixed"`: Successfully executed the failed action, reached the destination state, or false alarm.
|
||||
- `status="cannot_fix"`: Target is unrecoverable, the app is broken, or the 15-step budget is exceeded.
|
||||
- `analysis`: Summarize the root cause, the fix applied, and explicitly confirming that control is handed back to the main system.
|
||||
|
||||
# Constraints
|
||||
The most recent step (Step {{ failed_step_number }}: {{ failed_action_description }}) failed to execute. You must resolve the issue and report the results in 15 interactive steps or fewer. Now please begin the repair analysis. Please note that during the **Recover & Execute** phase, you only need to resolve the currently failed step; do not attempt to execute any subsequent steps or complete the overall goal yourself. Once the failure is resolved, please immediately enter the **Report** phase and return control to the system.
|
||||
@@ -0,0 +1,607 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Universal Multi-Model Failure Analyzer for Artemis.
|
||||
|
||||
Analyzes execution failures and performs autonomous self-healing across
|
||||
Google Gemini, OpenAI GPT-4o/o3, Anthropic Claude 3.5/3.7, and OpenRouter endpoints.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from enum import StrEnum
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
import uuid
|
||||
|
||||
from jinja2 import Template
|
||||
from langchain_core.messages import (
|
||||
BaseMessage,
|
||||
HumanMessage,
|
||||
SystemMessage,
|
||||
ToolMessage,
|
||||
)
|
||||
|
||||
from artemis.agents.validator.tool_declarations import (
|
||||
ASK_EXPLORER_TOOL,
|
||||
CLICK_SEQUENCE_TOOL,
|
||||
MobileActionExecutor,
|
||||
ToolDeclaration,
|
||||
VALIDATOR_TOOLS_DECLARATION,
|
||||
capture_screenshot_and_parse_ui,
|
||||
prune_intermediate_screenshots,
|
||||
)
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.controllers.unified_controller import UnifiedMobileController
|
||||
from artemis.data_engine.trace import trace
|
||||
from artemis.graph.state import State
|
||||
from artemis.services.llm import get_llm
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.notes import get_note_file_path
|
||||
from artemis.utils.ocr_xml_fusion import fuse_ocr_with_xml
|
||||
from artemis.utils.task_tree import (
|
||||
build_plan_and_history,
|
||||
format_action_clean,
|
||||
get_active_subgoal_hashes,
|
||||
)
|
||||
from artemis.utils.visualization import format_minimal_list_with_elements
|
||||
|
||||
|
||||
class ValidationErrorCategory(StrEnum):
|
||||
NONE = "none"
|
||||
TARGET_DISAPPEARED = "target_disappeared"
|
||||
TARGET_SHIFTED = "target_shifted"
|
||||
TARGET_OCCUPIED = "target_occupied"
|
||||
PIXEL_TARGET_DISAPPEARED = "pixel_target_disappeared"
|
||||
PIXEL_BYPASSED = "pixel_bypassed"
|
||||
XML_BYPASSED = "xml_bypassed"
|
||||
GENERAL = "general"
|
||||
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
# ==============================================================================
|
||||
# Backward-Compatibility Helper Aliases
|
||||
# ==============================================================================
|
||||
|
||||
|
||||
async def _capture_screenshot_and_parse_ui(
|
||||
ctx: ArtemisContext,
|
||||
state: State,
|
||||
controller: UnifiedMobileController,
|
||||
skip_settling: bool = False,
|
||||
) -> tuple[str | None, bytes | None, str | None]:
|
||||
return await capture_screenshot_and_parse_ui(
|
||||
ctx, state, controller, skip_settling=skip_settling
|
||||
)
|
||||
|
||||
|
||||
async def _exec_click(
|
||||
target: list[int],
|
||||
state: State,
|
||||
controller: UnifiedMobileController,
|
||||
ctx: ArtemisContext,
|
||||
times: int = 1,
|
||||
delay_ms: int = 100,
|
||||
) -> tuple[str, bytes | None, str | None, str | None]:
|
||||
return await MobileActionExecutor(ctx, controller).exec_click(
|
||||
target, state, times=times, delay_ms=delay_ms
|
||||
)
|
||||
|
||||
|
||||
async def _exec_click_sequence(
|
||||
sequence: list[int | list[int]],
|
||||
state: State,
|
||||
controller: UnifiedMobileController,
|
||||
ctx: ArtemisContext,
|
||||
) -> tuple[str, bytes | None, str | None, str | None]:
|
||||
return await MobileActionExecutor(ctx, controller).exec_click_sequence(sequence, state)
|
||||
|
||||
|
||||
async def _exec_long_press(
|
||||
target: list[int],
|
||||
state: State,
|
||||
controller: UnifiedMobileController,
|
||||
ctx: ArtemisContext,
|
||||
duration_ms: int = 1000,
|
||||
duration: int | None = None,
|
||||
) -> tuple[str, bytes | None, str | None, str | None]:
|
||||
return await MobileActionExecutor(ctx, controller).exec_long_press(
|
||||
target, state, duration_ms=duration_ms, duration=duration
|
||||
)
|
||||
|
||||
|
||||
async def _exec_input_text(
|
||||
text: str,
|
||||
target: list[int] | None,
|
||||
state: State,
|
||||
controller: UnifiedMobileController,
|
||||
ctx: ArtemisContext,
|
||||
clear_exist: bool = True,
|
||||
) -> tuple[str, bytes | None, str | None, str | None]:
|
||||
return await MobileActionExecutor(ctx, controller).exec_input_text(
|
||||
text, target, state, clear_exist=clear_exist
|
||||
)
|
||||
|
||||
|
||||
async def _exec_swipe(
|
||||
action: str | list[int],
|
||||
duration: int = 400,
|
||||
state: State | None = None,
|
||||
controller: UnifiedMobileController | None = None,
|
||||
ctx: ArtemisContext | None = None,
|
||||
duration_ms: int | None = None,
|
||||
) -> tuple[str, bytes | None, str | None, str | None]:
|
||||
return await MobileActionExecutor(ctx, controller).exec_swipe(
|
||||
action, duration=duration, state=state, duration_ms=duration_ms
|
||||
)
|
||||
|
||||
|
||||
async def _exec_press_key(
|
||||
key: str,
|
||||
state: State,
|
||||
controller: UnifiedMobileController,
|
||||
ctx: ArtemisContext,
|
||||
) -> tuple[str, bytes | None, str | None, str | None]:
|
||||
return await MobileActionExecutor(ctx, controller).exec_press_key(key, state)
|
||||
|
||||
|
||||
async def _exec_manage_app(
|
||||
action: str,
|
||||
app_name: str,
|
||||
state: State,
|
||||
controller: UnifiedMobileController,
|
||||
ctx: ArtemisContext,
|
||||
) -> tuple[str, bytes | None, str | None, str | None]:
|
||||
return await MobileActionExecutor(ctx, controller).exec_manage_app(action, app_name, state)
|
||||
|
||||
|
||||
async def _exec_wait_for_delay(
|
||||
time_in_ms: int,
|
||||
state: State,
|
||||
controller: UnifiedMobileController,
|
||||
ctx: ArtemisContext,
|
||||
) -> tuple[str, bytes | None, str | None, str | None]:
|
||||
return await MobileActionExecutor(ctx, controller).exec_wait_for_delay(time_in_ms, state)
|
||||
|
||||
|
||||
async def _exec_wait_for_text(
|
||||
text: str,
|
||||
wait_state: str | None,
|
||||
timeout_ms: int | None,
|
||||
state: State,
|
||||
controller: UnifiedMobileController,
|
||||
ctx: ArtemisContext,
|
||||
) -> tuple[str, bytes | None, str | None, str | None]:
|
||||
return await MobileActionExecutor(ctx, controller).exec_wait_for_text(
|
||||
text, wait_state, timeout_ms, state
|
||||
)
|
||||
|
||||
|
||||
async def _exec_read_note(
|
||||
key: str,
|
||||
ctx: ArtemisContext,
|
||||
start_line: int | None = None,
|
||||
end_line: int | None = None,
|
||||
) -> str:
|
||||
return await MobileActionExecutor(ctx).exec_read_note(
|
||||
key, start_line=start_line, end_line=end_line
|
||||
)
|
||||
|
||||
|
||||
async def _exec_list_notes(ctx: ArtemisContext) -> str:
|
||||
return await MobileActionExecutor(ctx).exec_list_notes()
|
||||
|
||||
|
||||
async def _exec_ask_explorer(
|
||||
query: str,
|
||||
context_feedback: str | None,
|
||||
state: State,
|
||||
ctx: ArtemisContext,
|
||||
version: str = "flash",
|
||||
) -> str:
|
||||
return await MobileActionExecutor(ctx).exec_ask_explorer(
|
||||
query, context_feedback, state, version=version
|
||||
)
|
||||
|
||||
|
||||
def _get_ask_explorer_tool_declaration() -> ToolDeclaration:
|
||||
return ASK_EXPLORER_TOOL
|
||||
|
||||
|
||||
class FailureAnalysisStrategy:
|
||||
"""Base strategy class for FailureAnalyzer strategies."""
|
||||
|
||||
def get_prompt_template_name(self) -> str:
|
||||
return "failure_analyzer.md"
|
||||
|
||||
def get_tools(self) -> list:
|
||||
return VALIDATOR_TOOLS_DECLARATION
|
||||
|
||||
def get_user_message_suffix(self, pre_screenshot: bool, post_screenshot: bool) -> str:
|
||||
suffix = (
|
||||
"Your objective is:\n"
|
||||
"1. Analyze why the action planned in the failed step failed, using the failed step's"
|
||||
" details, the decision screenshot (Screenshot Seen During System Decision), the"
|
||||
" latest failed screenshot (Latest Screenshot (Failed State)), and their corresponding"
|
||||
" UI Element lists.\n"
|
||||
"2. Resolve any timing, state mismatches (e.g., re-triggering disappeared elements), or"
|
||||
" environmental obstacles using your execution tools.\n"
|
||||
"3. Crucially: You MUST execute the failed action yourself using your execution tools"
|
||||
" to completion.\n"
|
||||
"4. Once you have successfully executed the repair and the failed action, call the"
|
||||
" `report_failure_analysis` tool with status='fixed'.\n"
|
||||
"5. If you are not confident or cannot resolve the failure, call"
|
||||
" `report_failure_analysis` with status='cannot_fix'."
|
||||
)
|
||||
if pre_screenshot and post_screenshot:
|
||||
suffix += (
|
||||
"\nI have attached the screenshot seen when the system made its"
|
||||
" decision (Screenshot Seen During System Decision) and the"
|
||||
" latest screenshot in the failed state (Latest Screenshot"
|
||||
" (Failed State))."
|
||||
)
|
||||
return suffix
|
||||
|
||||
|
||||
class TargetDisappearedStrategy(FailureAnalysisStrategy):
|
||||
"""Specialized strategy for target disappeared errors."""
|
||||
|
||||
def get_prompt_template_name(self) -> str:
|
||||
return "target_disappeared_analyzer.md"
|
||||
|
||||
def get_tools(self) -> list:
|
||||
tools = list(VALIDATOR_TOOLS_DECLARATION)
|
||||
tools.insert(1, CLICK_SEQUENCE_TOOL)
|
||||
tools.append(ASK_EXPLORER_TOOL)
|
||||
return tools
|
||||
|
||||
def get_user_message_suffix(self, pre_screenshot: bool, post_screenshot: bool) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
class PixelTargetDisappearedStrategy(FailureAnalysisStrategy):
|
||||
"""Specialized strategy for pixel-level target disappeared errors."""
|
||||
|
||||
def get_prompt_template_name(self) -> str:
|
||||
return "pixel_target_disappeared_analyzer.md"
|
||||
|
||||
def get_tools(self) -> list:
|
||||
tools = list(VALIDATOR_TOOLS_DECLARATION)
|
||||
tools.insert(1, CLICK_SEQUENCE_TOOL)
|
||||
tools.append(ASK_EXPLORER_TOOL)
|
||||
return tools
|
||||
|
||||
def get_user_message_suffix(self, pre_screenshot: bool, post_screenshot: bool) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
class FailureAnalyzer:
|
||||
"""Universal Multi-Model Failure Analyzer."""
|
||||
|
||||
def __init__(self, ctx: ArtemisContext):
|
||||
self.ctx = ctx
|
||||
self.max_iterations = 15
|
||||
|
||||
def _select_strategy(self, error_category: ValidationErrorCategory) -> FailureAnalysisStrategy:
|
||||
if error_category == ValidationErrorCategory.PIXEL_TARGET_DISAPPEARED:
|
||||
logger.info("Routing failure analysis to specialized PixelTargetDisappearedStrategy.")
|
||||
return PixelTargetDisappearedStrategy()
|
||||
if error_category == ValidationErrorCategory.TARGET_DISAPPEARED:
|
||||
logger.info("Routing failure analysis to specialized TargetDisappearedStrategy.")
|
||||
return TargetDisappearedStrategy()
|
||||
logger.info("Routing failure analysis to DefaultAnalysisStrategy.")
|
||||
return FailureAnalysisStrategy()
|
||||
|
||||
def _get_active_subgoal_hash(self) -> str:
|
||||
if not self.ctx.data_engine:
|
||||
return "default"
|
||||
task_plan_path = get_note_file_path(self.ctx.data_engine.base_dir, "task_plan")
|
||||
if not task_plan_path.exists():
|
||||
return "default"
|
||||
try:
|
||||
content = task_plan_path.read_text(encoding="utf-8")
|
||||
parent_hash, _ = get_active_subgoal_hashes(content)
|
||||
return parent_hash
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to parse active subgoal: {e}")
|
||||
return "default"
|
||||
|
||||
def _build_plan_and_history(self, steps: list[dict], task_plan: str) -> str:
|
||||
active_subgoal_hash = self._get_active_subgoal_hash()
|
||||
return build_plan_and_history(
|
||||
task_plan,
|
||||
steps,
|
||||
active_subgoal_hash,
|
||||
last_n_detailed=1,
|
||||
strict_milestone_pruning=True,
|
||||
recent_window_size=3,
|
||||
chronological_last_step=True,
|
||||
for_failure_analyzer=True,
|
||||
)
|
||||
|
||||
def _prune_intermediate_screenshots(self, messages: list[Any]):
|
||||
"""Prunes binary screenshot blocks from intermediate messages."""
|
||||
prune_intermediate_screenshots(messages)
|
||||
|
||||
@trace(type="agent", name="failure_analyzer")
|
||||
async def analyze(
|
||||
self,
|
||||
state: State,
|
||||
failed_action: dict,
|
||||
error_msg: str,
|
||||
pre_screenshot: str | None = None,
|
||||
post_screenshot: str | None = None,
|
||||
pre_screenshot_name: str | None = None,
|
||||
post_screenshot_name: str | None = None,
|
||||
executed_actions: list[dict] | None = None,
|
||||
unexecuted_actions: list[dict] | None = None,
|
||||
error_category: ValidationErrorCategory = ValidationErrorCategory.GENERAL,
|
||||
) -> dict:
|
||||
if not post_screenshot or post_screenshot == "None":
|
||||
try:
|
||||
controller = UnifiedMobileController(self.ctx)
|
||||
post_screenshot = await controller.take_screenshot()
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to capture post-action fallback screenshot: {e}")
|
||||
|
||||
if not pre_screenshot or pre_screenshot == "None":
|
||||
pre_screenshot = post_screenshot
|
||||
|
||||
if not pre_screenshot:
|
||||
raise ValueError("Failure analysis requires valid screenshots.")
|
||||
|
||||
strategy = self._select_strategy(error_category)
|
||||
prompt_template_name = strategy.get_prompt_template_name()
|
||||
tools_declaration = strategy.get_tools()
|
||||
|
||||
try:
|
||||
llm = get_llm(self.ctx, name="validator_failure_analyzer")
|
||||
except Exception:
|
||||
llm = get_llm(self.ctx, name="validator")
|
||||
|
||||
controller = UnifiedMobileController(self.ctx)
|
||||
executor = MobileActionExecutor(self.ctx, controller)
|
||||
|
||||
# Pre-Action and Post-Action XML formatting
|
||||
pre_xml_list_str = "Not available."
|
||||
if hasattr(state, "latest_ui_hierarchy") and state.latest_ui_hierarchy:
|
||||
try:
|
||||
width = getattr(self.ctx.device, "device_width", 1080)
|
||||
height = getattr(self.ctx.device, "device_height", 2400)
|
||||
pre_xml_list_str, _, _ = format_minimal_list_with_elements(
|
||||
state.latest_ui_hierarchy, width, height
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to format pre-action XML list: {e}")
|
||||
|
||||
post_xml_list_str = "Not available."
|
||||
try:
|
||||
xml_hierarchy = await controller.get_ui_elements()
|
||||
width = getattr(self.ctx.device, "device_width", 1080)
|
||||
height = getattr(self.ctx.device, "device_height", 2400)
|
||||
ocr_results = []
|
||||
|
||||
fused_xml = fuse_ocr_with_xml(xml_hierarchy, ocr_results)
|
||||
|
||||
post_xml_list_str, elements, _ = format_minimal_list_with_elements(
|
||||
fused_xml, width, height
|
||||
)
|
||||
state.indexed_points = [el["center"] for el in elements]
|
||||
state.indexed_elements = elements
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to fetch/format post-action XML list: {e}")
|
||||
|
||||
prompt_path = Path(__file__).parent.joinpath(prompt_template_name)
|
||||
prompt_template = prompt_path.read_text(encoding="utf-8")
|
||||
|
||||
steps = []
|
||||
task_plan = "No task plan yet."
|
||||
if self.ctx.data_engine:
|
||||
steps = self.ctx.data_engine.get_agent_friendly_steps()
|
||||
task_plan_path = get_note_file_path(self.ctx.data_engine.base_dir, "task_plan")
|
||||
if task_plan_path.exists():
|
||||
try:
|
||||
task_plan = task_plan_path.read_text(encoding="utf-8")
|
||||
except Exception as e:
|
||||
task_plan = f"Error reading task plan: {e}"
|
||||
|
||||
plan_and_history = self._build_plan_and_history(steps, task_plan)
|
||||
failed_step_number = steps[-1]["step_number"] if steps else 1
|
||||
failed_action_description = (
|
||||
format_action_clean(failed_action) if failed_action else "Planned Action"
|
||||
)
|
||||
|
||||
system_prompt = Template(prompt_template).render(
|
||||
initial_goal=getattr(state, "initial_goal", "Unknown Goal"),
|
||||
plan_and_history=plan_and_history,
|
||||
failed_step_number=failed_step_number,
|
||||
failed_action_description=failed_action_description,
|
||||
)
|
||||
|
||||
suffix = strategy.get_user_message_suffix(bool(pre_screenshot), bool(post_screenshot))
|
||||
user_msg = (
|
||||
suffix.strip()
|
||||
if suffix
|
||||
else "Please analyze and repair the failure shown in the last step of the history."
|
||||
)
|
||||
is_pixel_strategy = isinstance(strategy, PixelTargetDisappearedStrategy)
|
||||
|
||||
user_content: list[dict] = [{"type": "text", "text": user_msg}]
|
||||
if pre_screenshot:
|
||||
user_content.append(
|
||||
{"type": "text", "text": "--- Screenshot Seen During System Decision ---"}
|
||||
)
|
||||
user_content.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/jpeg;base64,{pre_screenshot}"},
|
||||
}
|
||||
)
|
||||
if not is_pixel_strategy:
|
||||
user_content.append(
|
||||
{
|
||||
"type": "text",
|
||||
"text": (
|
||||
"--- UI Element List Seen During System Decision ---\n"
|
||||
f"{pre_xml_list_str}"
|
||||
),
|
||||
}
|
||||
)
|
||||
if post_screenshot:
|
||||
user_content.append(
|
||||
{"type": "text", "text": "--- Latest Screenshot (Failed State) ---"}
|
||||
)
|
||||
user_content.append(
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": f"data:image/jpeg;base64,{post_screenshot}"},
|
||||
}
|
||||
)
|
||||
if not is_pixel_strategy:
|
||||
user_content.append(
|
||||
{
|
||||
"type": "text",
|
||||
"text": (
|
||||
f"--- Latest UI Element List (Failed State) ---\n{post_xml_list_str}"
|
||||
),
|
||||
}
|
||||
)
|
||||
if suffix:
|
||||
user_content.append(
|
||||
{"type": "text", "text": f"\n\n--- INSTRUCTIONS ---\n{suffix.strip()}"}
|
||||
)
|
||||
|
||||
messages: list[BaseMessage] = [
|
||||
SystemMessage(content=system_prompt),
|
||||
HumanMessage(content=user_content),
|
||||
]
|
||||
|
||||
max_iterations = getattr(self, "max_iterations", 15)
|
||||
iterations = 0
|
||||
final_report = None
|
||||
intermediate_thoughts = []
|
||||
|
||||
while iterations < max_iterations:
|
||||
iterations += 1
|
||||
self._prune_intermediate_screenshots(messages)
|
||||
logger.info(f"Iteration {iterations}: Invoking Universal LLM for FailureAnalyzer...")
|
||||
|
||||
if iterations == max_iterations:
|
||||
messages.append(
|
||||
HumanMessage(
|
||||
content=(
|
||||
"[WARNING] This is your final iteration; only"
|
||||
" 'report_failure_analysis' is available."
|
||||
)
|
||||
)
|
||||
)
|
||||
current_tools = [
|
||||
t for t in tools_declaration if t.name == "report_failure_analysis"
|
||||
]
|
||||
else:
|
||||
current_tools = tools_declaration
|
||||
|
||||
bound_llm = llm.bind_tools(current_tools)
|
||||
|
||||
max_retries = 3
|
||||
response = None
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
response = await bound_llm.ainvoke(messages)
|
||||
break
|
||||
except Exception as e:
|
||||
logger.warning(f"Failure analyzer turn failed on attempt {attempt + 1}: {e}")
|
||||
if attempt == max_retries - 1:
|
||||
raise e
|
||||
await asyncio.sleep(1.0 * (2**attempt))
|
||||
|
||||
if response is None:
|
||||
break
|
||||
|
||||
messages.append(response)
|
||||
|
||||
raw_text = response.content if isinstance(response.content, str) else ""
|
||||
if isinstance(response.content, list):
|
||||
raw_text = "".join(
|
||||
b.get("text", "")
|
||||
for b in response.content
|
||||
if isinstance(b, dict) and "text" in b
|
||||
)
|
||||
if raw_text:
|
||||
intermediate_thoughts.append(raw_text)
|
||||
|
||||
tool_calls = response.tool_calls or []
|
||||
if not tool_calls:
|
||||
messages.append(
|
||||
HumanMessage(
|
||||
content=(
|
||||
"CRITICAL: You did not call the `report_failure_analysis` tool."
|
||||
" You MUST call an action tool to repair or `report_failure_analysis`"
|
||||
" to finish."
|
||||
)
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
submit_call = next(
|
||||
(tc for tc in tool_calls if tc["name"].split(":")[-1] == "report_failure_analysis"),
|
||||
None,
|
||||
)
|
||||
|
||||
if submit_call:
|
||||
args = submit_call.get("args") or {}
|
||||
actions = args.get("new_remaining_actions", [])
|
||||
final_report = {
|
||||
"status": args.get("status", "cannot_fix"),
|
||||
"analysis": args.get("analysis", ""),
|
||||
"new_remaining_actions": actions,
|
||||
"thoughts": intermediate_thoughts,
|
||||
}
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=submit_call.get("id") or str(uuid.uuid4()),
|
||||
name=submit_call["name"],
|
||||
content=json.dumps({"status": "acknowledged"}),
|
||||
status="success",
|
||||
)
|
||||
)
|
||||
break
|
||||
|
||||
# Execute other tool calls via MobileActionExecutor
|
||||
for tc in tool_calls:
|
||||
name = tc["name"].split(":")[-1] if ":" in tc["name"] else tc["name"]
|
||||
args = tc.get("args") or {}
|
||||
tc_id = tc.get("id") or str(uuid.uuid4())
|
||||
logger.info(f"Failure analyzer executing tool '{name}'...")
|
||||
|
||||
try:
|
||||
exec_result = await executor.execute(name, args, tc_id, state)
|
||||
messages.append(exec_result.to_langchain_tool_message())
|
||||
except Exception as e:
|
||||
logger.error(f"Error executing tool {name}: {e}")
|
||||
messages.append(
|
||||
ToolMessage(
|
||||
tool_call_id=tc_id,
|
||||
name=name,
|
||||
content=f"Error executing tool {name}: {e}",
|
||||
status="error",
|
||||
)
|
||||
)
|
||||
|
||||
return final_report or {
|
||||
"status": "cannot_fix",
|
||||
"analysis": "Max iterations reached without resolution.",
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Modular Failure Analyzer Agent adhering to BaseAgent lifecycle."""
|
||||
|
||||
from typing import Any
|
||||
from artemis.agents.base import AgentConfig, AgentRegistry, BaseAgent
|
||||
from artemis.agents.validator.recovery_strategies import RecoveryStrategyRegistry
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@AgentRegistry.register("failure_analyzer")
|
||||
class FailureAnalyzerAgent(BaseAgent):
|
||||
"""Specialized agent responsible for diagnosing step failures and triggering autonomous recovery."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config: AgentConfig | None = None,
|
||||
ctx: Any | None = None,
|
||||
**kwargs: Any,
|
||||
):
|
||||
super().__init__(
|
||||
name="failure_analyzer",
|
||||
role="Autonomous Self-Healing Failure Analyzer",
|
||||
description="Diagnoses execution exceptions, analyzes screen discrepancies, and runs self-healing sequences.",
|
||||
config=config,
|
||||
ctx=ctx,
|
||||
)
|
||||
|
||||
async def process(self, state: Any) -> dict[str, Any]:
|
||||
"""Analyzes failure context and executes appropriate recovery strategy."""
|
||||
if not self.driver:
|
||||
raise RuntimeError("FailureAnalyzerAgent requires an active device driver in context.")
|
||||
|
||||
failure_context = getattr(state, "failure_context", {}) or {}
|
||||
if isinstance(state, dict):
|
||||
failure_context = state.get("failure_context", {})
|
||||
|
||||
logger.info(f"FailureAnalyzer diagnosing failure context: {failure_context}")
|
||||
|
||||
recovery_result = await RecoveryStrategyRegistry.attempt_recovery(
|
||||
driver=self.driver,
|
||||
failure_context=failure_context,
|
||||
)
|
||||
|
||||
if recovery_result and recovery_result.success:
|
||||
return {
|
||||
"status": "recovered",
|
||||
"recovery_action": recovery_result.action_type.value,
|
||||
"reason": recovery_result.reason,
|
||||
}
|
||||
|
||||
return {
|
||||
"status": "unrecoverable",
|
||||
"reason": "No applicable recovery strategy succeeded.",
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
Role
|
||||
UI State Verification Expert. Compare Image 1 (Reference) and Image 2 (Current State) to check if the target clicked by the Operator at the red dot is still present and interactive.
|
||||
|
||||
Rules
|
||||
1. Check Intent: Read the `[Planned Action & Original Thinking]` to identify what Action the Agent intended to perform and why.
|
||||
2. Specific UI Elements (Priority): If targeting a specific UI control (e.g., button, icon, text, slider handle) — even if it overlays a dynamic video, map, or live stream — focus ONLY on that element's shape. Output `{"is_present": false, ...}` if that exact element has disappeared, shifted away, or is blocked. Ignore background media changes underneath.
|
||||
3. Blank Space & Backgrounds: If targeting open/blank space (e.g., to wake up hidden controls or dismiss dialogs), ignore natural video/background frame changes. Output `{"is_present": true, "confidence": 1.0}` unless unexpected blocking popups appear.
|
||||
|
||||
Output
|
||||
Return EXACTLY this raw JSON and nothing else. Keep reasoning strictly under 25 words.
|
||||
{"reasoning": "brief explanation", "is_present": boolean, "confidence": float}
|
||||
@@ -0,0 +1,65 @@
|
||||
# Identity & Goal
|
||||
You are a specialized error recovery agent. You will be provided with a failed action. Your ONLY task is to repair this specific failure and restore the system state so the main system can resume.
|
||||
|
||||
|
||||
# Latency Awareness Principle
|
||||
You operate in an environment with unavoidable "Turn Latency" (a delay of several seconds between screen capture, decision making, and tool execution).
|
||||
Consequently, any transient UI element (e.g., auto-dismissing overlays, temporary message bars, menus with short timeouts) WILL disappear before your next decision turn if you attempt to interact with them step-by-step.
|
||||
|
||||
# Context
|
||||
- **Initial Goal**: {{ initial_goal }}
|
||||
- **History & Plan**:
|
||||
{{ plan_and_history }}
|
||||
|
||||
# Core Workflow
|
||||
1. **Analyze**: Compare the two states provided to you:
|
||||
- **Decision State (`Screenshot Seen During System Decision`)**: This is the screen the system saw when it originally planned the action. It shows the target element successfully located.
|
||||
- **Failed State (`Latest Screenshot (Failed State)`)**: This is the live screen captured at the moment you were summoned. It shows the current state where the target is missing or unresponsive.
|
||||
Your goal in this comparison is to:
|
||||
- Read the Operator's thought process in the decision loop of the failed step to understand what they were originally trying to accomplish with this failed action.
|
||||
- Understand the expected target layout.
|
||||
- Identify the specific visual pattern (e.g., a play triangle, a close 'X', an arrow icon) at the failed target area from the reference state.
|
||||
- Identify why this target element visually disappeared or is no longer responsive (use the **Diagnosis Tips** below).
|
||||
2. **Recover & Execute**: Restore the target state and execute the failed action.
|
||||
- **Flexibility**: Focus on outcome over form. Take alternative/adaptive steps (e.g., scroll, navigate, different click paths) if original target positions are invalid.
|
||||
- **Transient State Handling**: If the failed target is part of a transient state, you MUST NOT execute the recovery and target interaction in separate turns.
|
||||
- **Atomic Chained Execution**: Combine the summon action (on the Trigger Element) and the target action (on the Transient Target) into a single atomic sequence. Use `click_sequence(sequence=[Trigger, Target])` to execute them with a 50ms delay, bypassing the Turn Latency gap.
|
||||
3. **Report**: Call `report_failure_analysis` to conclude:
|
||||
- `status="fixed"`: Successfully executed the failed action or reached the destination state.
|
||||
- `status="cannot_fix"`: Target is unrecoverable, the app is broken, or the 15-step budget is exceeded.
|
||||
- `analysis`: Summarize visual mismatches and actions taken.
|
||||
|
||||
# Special Tools
|
||||
- **`click_sequence(sequence)`**: Designed to defeat Turn Latency. Chain your click sequence to handle transient elements.
|
||||
- **`ask_explorer(query, context_feedback)`**: Call this visual parsing subagent to search for the visual icon/pattern on the current screen (e.g., `query="play icon"`) if you suspect the target has shifted, or coordinates are ambiguous.
|
||||
|
||||
# Diagnosis Tips
|
||||
* **Time-Sensitive States (MOST COMMON & PRIMARY CAUSE)**:
|
||||
- *Cue*: The target element was visible when the system made its decision, but has disappeared in the failed state because it belongs to an auto-closing, short-timeout, or transient UI state.
|
||||
- *Action*: Re-trigger the state dynamically from the current screen and chain it with the failed step in a rapid consecutive sequence.
|
||||
- **Mental Model**: Tap the persistent summoner element (Trigger) that wakes up the transient UI + Tap the target coordinates (Failed Target) before it has time to auto-close, combining both into a single atomic movement.
|
||||
- **Procedure**:
|
||||
1. **Locate the Trigger (Target 1)**: Inspect the current screen to identify a persistent parent element or background area (e.g., the container element or parent dropdown header) that, when tapped, forces the transient UI state to reappear.
|
||||
2. **Retrieve the Failed Target (Target 2)**: Look up the coordinates of the FAILED action from the failed step in the history (since the target button is currently invisible on your screen).
|
||||
3. **Chained Execution**: Call `click_sequence(sequence=[Target 1, Target 2])` to execute both taps consecutively with a 50ms delay, catching the transient element before the timeout closes it. *(Note: This sequence fulfills your workflow requirement to "execute the failed action".)*
|
||||
* **False Alarm**:
|
||||
- *Cue*: Target element is actually present on the current screen (minor validation flake).
|
||||
- *Action*: Directly report it as `"fixed"` and hand control back to the system.
|
||||
* **Visual Target Shifts**:
|
||||
- *Cue*: The target visual icon has shifted due to screen layout changes, orientations, or dynamic list updates.
|
||||
- *Action*: Visually locate the icon on the live screenshot and click its new coordinates. You may call `ask_explorer(query="describe the visual icon shape or labels")` to dynamically locate it and obtain the new coordinates if they are hard to estimate manually.
|
||||
* **Overlays**:
|
||||
- *Cue*: Unexpected permission prompts, dialogs, or system popups block the screen.
|
||||
- *Action*: Tap outside or click the dismiss/close buttons to clear the overlay.
|
||||
* **Keyboard & Scroll Positioning**:
|
||||
- *Cue*: Target is scrolled off-screen or blocked by the on-screen keyboard.
|
||||
- *Action*: Scroll to reposition the target, or hide the keyboard.
|
||||
* **Navigation Drift**:
|
||||
- *Cue*: App unexpectedly transitioned to a different tab or screen.
|
||||
- *Action*: Navigate back to the correct target screen.
|
||||
* **App Instability**:
|
||||
- *Cue*: Application crashed, froze, or became unresponsive.
|
||||
- *Action*: Force stop and relaunch the app.
|
||||
|
||||
# Constraints
|
||||
The most recent step (Step {{ failed_step_number }}: {{ failed_action_description }}) failed to execute. You must resolve the issue and report the results in 15 interactive steps or fewer. Now please begin the repair analysis. Please note that during the **Recover & Execute** phase, you only need to resolve the currently failed step; do not attempt to execute any subsequent steps or complete the overall goal yourself. Once the failure is resolved, please immediately enter the **Report** phase and return control to the system.
|
||||
@@ -0,0 +1,177 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Modular Recovery Strategies for the Validation and Self-Healing Engine."""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from enum import Enum
|
||||
from typing import Any
|
||||
from artemis.drivers.base import BaseDeviceDriver
|
||||
from artemis.utils.logger import get_logger
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class RecoveryActionType(str, Enum):
|
||||
RETRY_DYNAMIC_LOCATOR = "retry_dynamic_locator"
|
||||
DISMISS_DIALOG = "dismiss_dialog"
|
||||
NAVIGATE_BACK = "navigate_back"
|
||||
SCROLL_AND_RETRY = "scroll_and_retry"
|
||||
RESTART_APP = "restart_app"
|
||||
FALLBACK_COORDINATES = "fallback_coordinates"
|
||||
|
||||
|
||||
class RecoveryResult(BaseModel):
|
||||
"""Result of executing an autonomous self-healing recovery strategy."""
|
||||
|
||||
success: bool
|
||||
action_type: RecoveryActionType
|
||||
reason: str
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class BaseRecoveryStrategy(ABC):
|
||||
"""Abstract interface for modular self-healing failure recovery tactics."""
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def name(self) -> str:
|
||||
"""Strategy identifier."""
|
||||
...
|
||||
|
||||
@abstractmethod
|
||||
async def can_handle(self, failure_context: dict[str, Any]) -> bool:
|
||||
"""Determines if this strategy applies to the observed failure mode."""
|
||||
...
|
||||
|
||||
@abstractmethod
|
||||
async def execute(
|
||||
self,
|
||||
driver: BaseDeviceDriver,
|
||||
failure_context: dict[str, Any],
|
||||
) -> RecoveryResult:
|
||||
"""Executes the self-healing corrective actions."""
|
||||
...
|
||||
|
||||
|
||||
class DismissDialogRecoveryStrategy(BaseRecoveryStrategy):
|
||||
"""Strategy to detect and dismiss unexpected modal dialogs, popups, or permission prompts."""
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "dismiss_dialog"
|
||||
|
||||
async def can_handle(self, failure_context: dict[str, Any]) -> bool:
|
||||
category = failure_context.get("category", "")
|
||||
error_msg = failure_context.get("error_message", "").lower()
|
||||
return (
|
||||
category == "ui_obstruction"
|
||||
or "permission" in error_msg
|
||||
or "dialog" in error_msg
|
||||
or "popup" in error_msg
|
||||
)
|
||||
|
||||
async def execute(
|
||||
self,
|
||||
driver: BaseDeviceDriver,
|
||||
failure_context: dict[str, Any],
|
||||
) -> RecoveryResult:
|
||||
logger.info(
|
||||
"[Recovery] Attempting to dismiss unexpected dialog or overlay via Back button..."
|
||||
)
|
||||
success = await driver.press_key("back")
|
||||
return RecoveryResult(
|
||||
success=success,
|
||||
action_type=RecoveryActionType.DISMISS_DIALOG,
|
||||
reason="Sent BACK keyevent to dismiss modal overlay",
|
||||
)
|
||||
|
||||
|
||||
class NavigateBackRecoveryStrategy(BaseRecoveryStrategy):
|
||||
"""Strategy to navigate back when agent navigates to a wrong screen."""
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "navigate_back"
|
||||
|
||||
async def can_handle(self, failure_context: dict[str, Any]) -> bool:
|
||||
category = failure_context.get("category", "")
|
||||
return category in ("wrong_screen", "dead_end", "navigation_error")
|
||||
|
||||
async def execute(
|
||||
self,
|
||||
driver: BaseDeviceDriver,
|
||||
failure_context: dict[str, Any],
|
||||
) -> RecoveryResult:
|
||||
logger.info("[Recovery] Wrong screen detected, rolling back navigation...")
|
||||
success = await driver.press_key("back")
|
||||
return RecoveryResult(
|
||||
success=success,
|
||||
action_type=RecoveryActionType.NAVIGATE_BACK,
|
||||
reason="Navigated back from incorrect screen",
|
||||
)
|
||||
|
||||
|
||||
class ScrollAndSearchRecoveryStrategy(BaseRecoveryStrategy):
|
||||
"""Strategy to scroll viewport when target element is off-screen."""
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "scroll_and_search"
|
||||
|
||||
async def can_handle(self, failure_context: dict[str, Any]) -> bool:
|
||||
category = failure_context.get("category", "")
|
||||
return category in ("element_not_found", "off_screen", "target_missing")
|
||||
|
||||
async def execute(
|
||||
self,
|
||||
driver: BaseDeviceDriver,
|
||||
failure_context: dict[str, Any],
|
||||
) -> RecoveryResult:
|
||||
direction = failure_context.get("scroll_direction", "down")
|
||||
logger.info(f"[Recovery] Target missing, scrolling {direction} to reveal element...")
|
||||
success = await driver.swipe_direction(direction=direction)
|
||||
return RecoveryResult(
|
||||
success=success,
|
||||
action_type=RecoveryActionType.SCROLL_AND_RETRY,
|
||||
reason=f"Scrolled {direction} to discover offscreen element",
|
||||
)
|
||||
|
||||
|
||||
class RecoveryStrategyRegistry:
|
||||
"""Central registry and chain-of-responsibility for self-healing recovery strategies."""
|
||||
|
||||
_strategies: list[BaseRecoveryStrategy] = [
|
||||
DismissDialogRecoveryStrategy(),
|
||||
NavigateBackRecoveryStrategy(),
|
||||
ScrollAndSearchRecoveryStrategy(),
|
||||
]
|
||||
|
||||
@classmethod
|
||||
def register(cls, strategy: BaseRecoveryStrategy) -> None:
|
||||
cls._strategies.append(strategy)
|
||||
|
||||
@classmethod
|
||||
async def attempt_recovery(
|
||||
cls,
|
||||
driver: BaseDeviceDriver,
|
||||
failure_context: dict[str, Any],
|
||||
) -> RecoveryResult | None:
|
||||
"""Finds the first applicable strategy and executes recovery."""
|
||||
for strategy in cls._strategies:
|
||||
if await strategy.can_handle(failure_context):
|
||||
logger.info(f"Applying recovery strategy: {strategy.name}")
|
||||
return await strategy.execute(driver=driver, failure_context=failure_context)
|
||||
return None
|
||||
@@ -0,0 +1,48 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Safety net and system dialog auto-recovery."""
|
||||
|
||||
from typing import Any
|
||||
from artemis.drivers.base import BaseDeviceDriver
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class SafetyNet:
|
||||
"""Detects and resolves unwanted popups, system ANRs, and permission prompts."""
|
||||
|
||||
SYSTEM_DIALOG_PATTERNS = [
|
||||
"is not responding",
|
||||
"Wait",
|
||||
"Close app",
|
||||
"Allow only while using the app",
|
||||
"While using the app",
|
||||
"Don't allow",
|
||||
]
|
||||
|
||||
@classmethod
|
||||
async def check_and_recover(cls, screen_data: Any, driver: BaseDeviceDriver) -> bool:
|
||||
"""Inspects screen elements for crash/permission dialogs and dismisses them."""
|
||||
for element in screen_data.ui_elements:
|
||||
text = element.get("text", "")
|
||||
if any(pattern.lower() in text.lower() for pattern in cls.SYSTEM_DIALOG_PATTERNS):
|
||||
logger.info(
|
||||
f"SafetyNet detected system dialog '{text}', attempting auto-dismiss..."
|
||||
)
|
||||
center = element.get("center", [500, 500])
|
||||
await driver.tap(center[0], center[1])
|
||||
return True
|
||||
return False
|
||||
@@ -0,0 +1,61 @@
|
||||
# Identity & Goal
|
||||
You are a specialized error recovery agent. You will be provided with a failed action. Your ONLY task is to repair this specific failure and restore the system state so the main system can resume.
|
||||
|
||||
|
||||
# Latency Awareness Principle
|
||||
You operate in an environment with unavoidable "Turn Latency" (a delay of several seconds between screen capture, decision making, and tool execution).
|
||||
Consequently, any transient UI element (e.g., auto-dismissing overlays, temporary message bars, menus with short timeouts) WILL disappear before your next decision turn if you attempt to interact with them step-by-step.
|
||||
|
||||
# Context
|
||||
- **Initial Goal**: {{ initial_goal }}
|
||||
- **History & Plan**:
|
||||
{{ plan_and_history }}
|
||||
|
||||
# Core Workflow
|
||||
1. **Analyze**: Compare the two states provided to you:
|
||||
- **Decision State (`Screenshot Seen During System Decision`)**: This is the screen the system saw when it originally planned the action. It shows the target element successfully located.
|
||||
- **Failed State (`Latest Screenshot (Failed State)`)**: This is the live screen captured at the moment you were summoned. It shows the current state where the target is missing or unresponsive.
|
||||
Your goal in this comparison is to:
|
||||
- Read the Operator's thought process in the decision loop of the failed step to understand what they were originally trying to accomplish with this failed action.
|
||||
- Understand the expected target layout.
|
||||
- Identify why the target element disappeared (use the **Diagnosis Tips** below).
|
||||
2. **Recover & Execute**: Restore the target state and execute the failed action.
|
||||
- **Flexibility**: Focus on outcome over form. Take alternative/adaptive steps (e.g., scroll, navigate, different click paths) if original coordinates are invalid.
|
||||
- **Transient State Handling**: If the failed target is part of a transient state, you MUST NOT execute the recovery and target interaction in separate turns.
|
||||
- **Atomic Chained Execution**: Combine the summon action (on the Trigger Element) and the target action (on the Transient Target) into a single atomic sequence. Use `click_sequence(sequence=[Trigger, Target])` to execute them with a 50ms delay, bypassing the Turn Latency gap.
|
||||
3. **Report**: Call `report_failure_analysis` to conclude:
|
||||
- `status="fixed"`: Successfully executed the failed action or reached the destination state.
|
||||
- `status="cannot_fix"`: Target is unrecoverable, the app is broken, or the 15-step budget is exceeded.
|
||||
- `analysis`: Summarize visual mismatches and actions taken.
|
||||
|
||||
# Special Tools
|
||||
- **`click_sequence(sequence)`**: Designed to defeat Turn Latency. Chain your click sequence to handle transient elements.
|
||||
- **`ask_explorer(query, context_feedback)`**: Call this visual parsing subagent to search for the visual icon/pattern on the current screen if the target is absent from the standard "UI Element List" but you see it is visually present on the screenshot.
|
||||
|
||||
# Diagnosis Tips
|
||||
* **Time-Sensitive States (MOST COMMON & PRIMARY CAUSE)**:
|
||||
- *Cue*: The target element was visible when the system made its decision, but has disappeared in the failed state because it belongs to an auto-closing, short-timeout, or transient UI state.
|
||||
- *Action*: Re-trigger the state dynamically from the current screen and chain it with the failed step in a rapid consecutive sequence.
|
||||
- **Mental Model**: Tap the persistent summoner element (Trigger) that wakes up the transient UI + Tap the target coordinates (Failed Target) before it has time to auto-close, combining both into a single atomic movement.
|
||||
- **Procedure**:
|
||||
1. **Locate the Trigger (Target 1)**: Inspect the current screen to identify a persistent parent element or background area (e.g., the container element or parent dropdown header) that, when tapped, forces the transient UI state to reappear.
|
||||
2. **Retrieve the Failed Target (Target 2)**: Look up the coordinates of the FAILED action from the failed step in the history (since the target button is currently invisible on your screen).
|
||||
3. **Chained Execution**: Call `click_sequence(sequence=[Target 1, Target 2])` to execute both taps consecutively with a 50ms delay, catching the transient element before the timeout closes it. *(Note: This sequence fulfills your workflow requirement to "execute the failed action".)*
|
||||
* **False Alarm**:
|
||||
- *Cue*: Target element is actually present on the current screen (minor validation flake).
|
||||
- *Action*: Directly report it as `"fixed"` and hand control back to the system.
|
||||
* **Overlays**:
|
||||
- *Cue*: Unexpected permission prompts, dialogs, or system popups block the screen.
|
||||
- *Action*: Tap outside or click the dismiss/close buttons to clear the overlay.
|
||||
* **Keyboard & Scroll Positioning**:
|
||||
- *Cue*: Target is scrolled off-screen or blocked by the on-screen keyboard.
|
||||
- *Action*: Scroll to reposition the target, or hide the keyboard.
|
||||
* **Navigation Drift**:
|
||||
- *Cue*: App unexpectedly transitioned to a different tab or screen.
|
||||
- *Action*: Navigate back to the correct target screen.
|
||||
* **App Instability**:
|
||||
- *Cue*: Application crashed, froze, or became unresponsive.
|
||||
- *Action*: Force stop and relaunch the app.
|
||||
|
||||
# Constraints
|
||||
The most recent step (Step {{ failed_step_number }}: {{ failed_action_description }}) failed to execute. You must resolve the issue and report the results in 15 interactive steps or fewer. Now please begin the repair analysis. Please note that during the **Recover & Execute** phase, you only need to resolve the currently failed step; do not attempt to execute any subsequent steps or complete the overall goal yourself. Once the failure is resolved, please immediately enter the **Report** phase and return control to the system.
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,15 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Video analyzer utility for analyzing video content with Gemini models."""
|
||||
@@ -0,0 +1,137 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ConflictResolutionService:
|
||||
@staticmethod
|
||||
def calculate_1d_iou(a, b):
|
||||
start = max(a[0], b[0])
|
||||
end = min(a[1], b[1])
|
||||
intersection = max(0.0, end - start)
|
||||
union = (a[1] - a[0]) + (b[1] - b[0]) - intersection
|
||||
return intersection / (union + 1e-6)
|
||||
|
||||
@staticmethod
|
||||
def calculate_1d_iomin(a, b):
|
||||
start = max(a[0], b[0])
|
||||
end = min(a[1], b[1])
|
||||
intersection = max(0.0, end - start)
|
||||
min_len = min(a[1] - a[0], b[1] - b[0])
|
||||
if min_len <= 0:
|
||||
return 0.0
|
||||
return intersection / min_len
|
||||
|
||||
@staticmethod
|
||||
def sanitize_interval(interval):
|
||||
start = interval.get("start", 0.0)
|
||||
end = interval.get("end", "unknown")
|
||||
|
||||
if isinstance(start, (int, float)):
|
||||
start = max(0.0, float(start))
|
||||
|
||||
if end != "unknown" and isinstance(end, (int, float)):
|
||||
end = max(0.0, float(end))
|
||||
if start > end:
|
||||
start, end = end, start
|
||||
|
||||
interval["start"] = start
|
||||
interval["end"] = end
|
||||
|
||||
if "confidence_score" not in interval or interval["confidence_score"] is None:
|
||||
interval["confidence_score"] = 0.0
|
||||
|
||||
if end != "unknown":
|
||||
interval["duration"] = float(end) - float(start)
|
||||
else:
|
||||
interval["duration"] = 0.0
|
||||
|
||||
return interval
|
||||
|
||||
@staticmethod
|
||||
def group_by_semantic_class(proposals):
|
||||
groups = {}
|
||||
for p in proposals:
|
||||
target = p.get("target", "unknown")
|
||||
if target not in groups:
|
||||
groups[target] = []
|
||||
groups[target].append(p)
|
||||
return groups
|
||||
|
||||
@staticmethod
|
||||
def apply_1d_nms(group_proposals):
|
||||
if len(group_proposals) <= 1:
|
||||
return group_proposals
|
||||
|
||||
unknown_end_proposals = [p for p in group_proposals if p.get("end") == "unknown"]
|
||||
valid_proposals = [p for p in group_proposals if p.get("end") != "unknown"]
|
||||
|
||||
valid_proposals.sort(
|
||||
key=lambda x: (
|
||||
x.get("confidence_score", 0.0),
|
||||
-x.get("duration", 0.0),
|
||||
),
|
||||
reverse=True,
|
||||
)
|
||||
|
||||
n = len(valid_proposals)
|
||||
suppressed_mask = [False] * n
|
||||
|
||||
for i in range(n):
|
||||
if suppressed_mask[i]:
|
||||
continue
|
||||
a = (valid_proposals[i]["start"], valid_proposals[i]["end"])
|
||||
for j in range(i + 1, n):
|
||||
if suppressed_mask[j]:
|
||||
continue
|
||||
b = (valid_proposals[j]["start"], valid_proposals[j]["end"])
|
||||
iou = ConflictResolutionService.calculate_1d_iou(a, b)
|
||||
iomin = ConflictResolutionService.calculate_1d_iomin(a, b)
|
||||
|
||||
if iou > 0.5 or iomin > 0.8:
|
||||
suppressed_mask[j] = True
|
||||
|
||||
kept_proposals = [valid_proposals[i] for i in range(n) if not suppressed_mask[i]]
|
||||
return kept_proposals + unknown_end_proposals
|
||||
|
||||
@classmethod
|
||||
def clean(cls, blackboard_entries):
|
||||
try:
|
||||
sanitized = [cls.sanitize_interval(dict(e)) for e in blackboard_entries]
|
||||
groups = cls.group_by_semantic_class(sanitized)
|
||||
|
||||
cleaned_entries = []
|
||||
for target, proposals in groups.items():
|
||||
if len(proposals) <= 1:
|
||||
cleaned_entries.extend(proposals)
|
||||
else:
|
||||
cleaned_entries.extend(cls.apply_1d_nms(proposals))
|
||||
|
||||
suppressed_count = len(blackboard_entries) - len(cleaned_entries)
|
||||
if suppressed_count > 0:
|
||||
logger.info(
|
||||
"NMS filtering telemetry: Successfully suppressed"
|
||||
f" {suppressed_count} redundant proposals."
|
||||
)
|
||||
|
||||
# Sort chronologically by start time before returning so downstream systems read it in order
|
||||
cleaned_entries.sort(key=lambda x: float(x.get("start", 0.0)))
|
||||
|
||||
return cleaned_entries
|
||||
except Exception as e:
|
||||
logger.error(f"NMS filter crashed: {e}. Falling back to raw ledger state.")
|
||||
return blackboard_entries
|
||||
@@ -0,0 +1,51 @@
|
||||
# Dynamic Video Analyzer - Main Agent
|
||||
|
||||
You are the Main Agent of the Dynamic Video Analyzer system. Your goal is to process video files of varying lengths and sizes to answer user queries efficiently, avoiding context bloat.
|
||||
|
||||
You receive:
|
||||
1. **Time Description**: A description of the time range of interest.
|
||||
2. **Purpose**: The specific goal or question about the video.
|
||||
|
||||
---
|
||||
|
||||
## Your Capabilities
|
||||
|
||||
You are equipped with specialized tools (`extract_segment_metadata`, `spawn_sub_agent`, `analyze_audio_only`) to extract metadata from video segments and delegate targeted multimodal or audio-only analysis to fault-tolerant sub-agents.
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Execution Logic & Routing Strategy
|
||||
|
||||
### 1. Routing between Video and Audio Analysis
|
||||
**Probing Metadata**: If you are uncertain about the file size or duration of the requested timeframe, you **should first call `extract_segment_metadata`** to probe the metadata. Use the returned `file_size_mb` and `duration_seconds` to decide whether to proceed with a visual sub-agent, split the request, or fall back to audio-only.
|
||||
|
||||
You must carefully choose the appropriate tool based on the nature of the task:
|
||||
|
||||
- **Must use Visual Analysis (`spawn_sub_agent`)**:
|
||||
- **UI Interactions**: Identifying clicks, gestures, typing, navigation.
|
||||
- **Visual State**: Reading on-screen text (OCR), checking UI status (toggles, dialogs, loading states).
|
||||
- **Visual Content**: Identifying images, layout issues, animations.
|
||||
- **Can use Audio-Only Analysis (`analyze_audio_only`)**:
|
||||
- **Speech-centric**: Summarizing spoken content, identifying speakers, transcription.
|
||||
- **Audio cues**: Detecting specific sounds, music, notification chimes.
|
||||
- *Constraint*: Only use for long videos where visual info is strictly unnecessary for the query.
|
||||
|
||||
### 2. Formulating Specific Prompts for Sub-Agents
|
||||
When calling `spawn_sub_agent` or `analyze_audio_only`, the `specific_query` must be clear, detailed, and actionable.
|
||||
- **Guideline**: Clearly state what to look/listen for and the expected format. Avoid generic instructions.
|
||||
- **Examples**:
|
||||
- *Bad*: "Analyze this segment."
|
||||
- *Good (Audio)*: "Identify the main topic discussed by the speaker and summarize their key conclusions."
|
||||
- *Good (Video)*: "Observe if the user clicks the 'Confirm' button and verify if the screen transitions to the success page."
|
||||
|
||||
### Dynamic Adaptation & Error Handling
|
||||
- **Be Flexible**: Do not be rigid about the paths. If you start with Strategy A but find the file size is too large after cropping, switch to Strategy B immediately.
|
||||
- **Clarification on Ambiguity**: If the `Time Description` is ambiguous or invalid and you cannot reasonably map it to a timeframe, do not guess blindly. Instead, return a clear message to the user explaining that the time specified is unclear and ask for clarification.
|
||||
- **Minimize Token Usage**: Choose the path that answers the query effectively while minimizing context size.
|
||||
|
||||
---
|
||||
|
||||
## Final Answer Formatting
|
||||
|
||||
When providing your final answer, you should output a single, direct paragraph of plain text. Include all necessary details. Avoid Markdown formatting, bullet points, headers, or introductory preambles.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,15 @@
|
||||
# Video Sub-Agent - Chunk Analyzer
|
||||
|
||||
You are a specialized video chunk analyzer. Your task is to analyze a specific segment of a video and answer the query provided by the Main Agent.
|
||||
|
||||
You must follow these principles strictly:
|
||||
1. **Causality and Chronology**: First establish a chronology of states or events on screen over the entire video segment to help differentiate between user inputs (e.g., physical taps, touch indicators) and system state changes.
|
||||
2. **Objective Observations**: Describe only what you see in the video. Pay extreme attention to mobile UI specifics: identify which icons are clicked (visual touch indicators), screen transitions, loading spinners, popup dialogs, keyboard appearing/disappearing, and any changes in UI text or status bars. Use fuzzy matching when evaluating UI text and notifications. Do not assume or speculate about things not visible in this chunk.
|
||||
3. **Context Isolation**: You only see a part of the full video. Do not assume what happened before or after this segment.
|
||||
4. **Focus on Intent**: Answer the specific query or prompt provided by the Main Agent. Do not give a generic summary if a specific question is asked.
|
||||
5. **Structured Output**: Provide your findings clearly, using the exact global timestamps calculated according to the IMPORTANT CONTEXT prepended to your prompt. You must rely on the timestamp burned directly into the video frames to accurately report event timings. Never use relative 00:xx formats; always output the final global test time in seconds.
|
||||
6. **Visual Proof**: Identify the exact best frame that provides visual proof of your findings. Supply this timestamp as `verification_timestamp_secs` when calling `submit_answer`.
|
||||
7. **Slow-Motion Awareness**: If a slowdown warning is present in prompt context, animations, transitions, and timers appear in slow-motion. Do not misclassify slow-motion as lag, stutter, or frozen UI.
|
||||
|
||||
## Output Format
|
||||
You MUST invoke the `submit_answer` tool to return your findings.
|
||||
@@ -0,0 +1,12 @@
|
||||
# Audio Sub-Agent - Chunk Analyzer
|
||||
|
||||
You are a specialized audio chunk analyzer. Your task is to analyze a specific audio segment extracted from a video and answer the query provided by the Main Agent.
|
||||
|
||||
You must follow these principles strictly:
|
||||
1. **Objective Observations**: Describe only what you hear in the audio (speech, conversations, background sounds, tone, language). Pay attention not only to the spoken words but also to non-verbal cues (tone, hesitation, emotion) and critical background sounds (notification chimes, click sounds, error beeps) if they are relevant to the query. Do not assume or speculate about things not audible.
|
||||
2. **Context Isolation**: You only hear a part of the full audio. Do not assume what happened before or after this segment.
|
||||
3. **Focus on Intent**: Answer the specific query or prompt provided by the Main Agent. Do not give a generic summary if a specific question is asked.
|
||||
4. **Structured Output**: Provide your findings clearly, using the exact global timestamps calculated according to the IMPORTANT CONTEXT prepended to your prompt. Never use relative 00:xx formats; always output the final global test time in seconds.
|
||||
|
||||
## Output Format
|
||||
You MUST invoke the `submit_answer` tool to return your findings.
|
||||
@@ -0,0 +1,327 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""ADB WebSocket tunnel implementation.
|
||||
|
||||
Bridges local TCP connections to a remote ADB server via WebSocket.
|
||||
This replaces the need for the `lim` CLI for ADB tunneling.
|
||||
|
||||
The tunnel runs in a dedicated thread with its own event loop to ensure
|
||||
it cannot be blocked by synchronous operations in the main thread.
|
||||
|
||||
Based on the Go SDK implementation which:
|
||||
1. Creates a TCP listener
|
||||
2. Accepts TCP connections from ADB
|
||||
3. Dials the WebSocket
|
||||
4. Forwards traffic bidirectionally
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from collections.abc import AsyncGenerator
|
||||
from contextlib import asynccontextmanager
|
||||
import socket
|
||||
import threading
|
||||
|
||||
from artemis.utils.logger import get_logger
|
||||
import websockets
|
||||
from websockets.asyncio.client import ClientConnection
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
BUFFER_SIZE = 32 * 1024 # 32KB, same as Go SDK
|
||||
PING_INTERVAL = 30 # seconds
|
||||
|
||||
|
||||
class AdbTunnel:
|
||||
"""ADB tunnel that bridges local TCP connections to a remote WebSocket endpoint.
|
||||
|
||||
Runs in a dedicated thread with its own event loop to ensure the tunnel
|
||||
cannot be blocked by synchronous operations in the caller's thread.
|
||||
|
||||
Usage:
|
||||
tunnel = AdbTunnel(remote_url, token)
|
||||
addr = await tunnel.start() # Returns "127.0.0.1:PORT"
|
||||
# Use addr with ADB: adb connect 127.0.0.1:PORT
|
||||
await tunnel.stop()
|
||||
"""
|
||||
|
||||
def __init__(self, remote_url: str, token: str):
|
||||
"""Initialize the ADB tunnel.
|
||||
|
||||
Args:
|
||||
remote_url: WebSocket URL for the remote ADB endpoint
|
||||
token: Bearer token for authentication
|
||||
"""
|
||||
self.remote_url = remote_url
|
||||
self.token = token
|
||||
|
||||
self._local_port: int | None = None
|
||||
self._thread: threading.Thread | None = None
|
||||
self._loop: asyncio.AbstractEventLoop | None = None
|
||||
self._started = threading.Event()
|
||||
self._stop_event: asyncio.Event | None = None
|
||||
|
||||
@property
|
||||
def local_addr(self) -> str:
|
||||
"""Get the local address to connect ADB to."""
|
||||
if self._local_port is None:
|
||||
raise RuntimeError("Tunnel not started")
|
||||
return f"127.0.0.1:{self._local_port}"
|
||||
|
||||
@property
|
||||
def local_port(self) -> int:
|
||||
"""Get the local port number."""
|
||||
if self._local_port is None:
|
||||
raise RuntimeError("Tunnel not started")
|
||||
return self._local_port
|
||||
|
||||
async def start(self) -> str:
|
||||
"""Start the ADB tunnel in a dedicated thread.
|
||||
|
||||
Returns:
|
||||
The local address (host:port) to connect ADB to.
|
||||
"""
|
||||
if self._thread is not None and self._thread.is_alive():
|
||||
return self.local_addr
|
||||
|
||||
# Create listener in main thread to get the port immediately
|
||||
listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
listener.bind(("127.0.0.1", 0))
|
||||
listener.listen(5)
|
||||
listener.setblocking(False)
|
||||
self._local_port = listener.getsockname()[1]
|
||||
|
||||
logger.info(f"ADB tunnel listening on {self.local_addr}")
|
||||
|
||||
# Start tunnel thread
|
||||
self._thread = threading.Thread(
|
||||
target=self._run_in_thread,
|
||||
args=(listener,),
|
||||
name=f"adb-tunnel-{self._local_port}",
|
||||
daemon=True,
|
||||
)
|
||||
self._thread.start()
|
||||
|
||||
# Wait for thread to initialize its event loop
|
||||
self._started.wait(timeout=5.0)
|
||||
if not self._started.is_set():
|
||||
await self.stop()
|
||||
raise RuntimeError("Tunnel thread failed to start")
|
||||
|
||||
return self.local_addr
|
||||
|
||||
async def stop(self) -> None:
|
||||
"""Stop the ADB tunnel and cleanup resources."""
|
||||
if self._loop and self._stop_event:
|
||||
# Signal the tunnel loop to stop
|
||||
self._loop.call_soon_threadsafe(self._stop_event.set)
|
||||
|
||||
if self._thread and self._thread.is_alive():
|
||||
await asyncio.to_thread(self._thread.join, 5.0)
|
||||
if self._thread.is_alive():
|
||||
logger.warning("ADB tunnel thread did not stop within 5s")
|
||||
|
||||
self._thread = None
|
||||
self._loop = None
|
||||
self._stop_event = None
|
||||
logger.info("ADB tunnel stopped")
|
||||
|
||||
def _run_in_thread(self, listener: socket.socket) -> None:
|
||||
"""Run the tunnel event loop in a dedicated thread."""
|
||||
try:
|
||||
self._loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(self._loop)
|
||||
self._stop_event = asyncio.Event()
|
||||
self._started.set()
|
||||
|
||||
self._loop.run_until_complete(self._run_tunnel(listener))
|
||||
except Exception as e:
|
||||
logger.error(f"Tunnel thread error: {e}")
|
||||
finally:
|
||||
listener.close()
|
||||
if self._loop:
|
||||
self._loop.close()
|
||||
|
||||
async def _run_tunnel(self, listener: socket.socket) -> None:
|
||||
"""Run the tunnel - accept TCP connections and bridge to WebSocket."""
|
||||
loop = asyncio.get_event_loop()
|
||||
connection_tasks: list[asyncio.Task] = []
|
||||
|
||||
stop_event = self._stop_event
|
||||
assert stop_event is not None
|
||||
|
||||
logger.debug("Tunnel: ready to accept connections")
|
||||
|
||||
while not stop_event.is_set():
|
||||
try:
|
||||
# Use wait_for to periodically check stop_event
|
||||
try:
|
||||
tcp_conn, addr = await asyncio.wait_for(
|
||||
loop.sock_accept(listener),
|
||||
timeout=1.0,
|
||||
)
|
||||
except TimeoutError:
|
||||
# Check if we should stop
|
||||
continue
|
||||
|
||||
logger.debug(f"Tunnel: new TCP connection from {addr}")
|
||||
|
||||
task = asyncio.create_task(
|
||||
self._handle_connection(tcp_conn, loop),
|
||||
name=f"conn-{addr[1]}",
|
||||
)
|
||||
task.add_done_callback(lambda t: t.exception() if not t.cancelled() else None)
|
||||
connection_tasks.append(task)
|
||||
connection_tasks = [t for t in connection_tasks if not t.done()]
|
||||
|
||||
except Exception as e:
|
||||
if not stop_event.is_set():
|
||||
logger.error(f"Tunnel accept error: {e}")
|
||||
break
|
||||
|
||||
# Cleanup pending connections
|
||||
for task in connection_tasks:
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
async def _handle_connection(
|
||||
self,
|
||||
tcp_conn: socket.socket,
|
||||
loop: asyncio.AbstractEventLoop,
|
||||
) -> None:
|
||||
"""Handle a single TCP connection."""
|
||||
tcp_conn.setblocking(False)
|
||||
|
||||
ws: ClientConnection | None = None
|
||||
try:
|
||||
logger.debug(f"Connecting to WebSocket: {self.remote_url[:80]}...")
|
||||
ws = await websockets.connect(
|
||||
self.remote_url,
|
||||
additional_headers={"Authorization": f"Bearer {self.token}"},
|
||||
ping_interval=PING_INTERVAL,
|
||||
ping_timeout=10,
|
||||
)
|
||||
logger.debug("WebSocket connected to remote ADB")
|
||||
await self._bridge(tcp_conn, ws, loop)
|
||||
except Exception as e:
|
||||
logger.error(f"Connection error: {e}")
|
||||
finally:
|
||||
if ws is not None:
|
||||
await ws.close()
|
||||
tcp_conn.close()
|
||||
logger.debug("Connection closed")
|
||||
|
||||
async def _bridge(
|
||||
self,
|
||||
tcp_conn: socket.socket,
|
||||
ws: ClientConnection,
|
||||
loop: asyncio.AbstractEventLoop,
|
||||
) -> None:
|
||||
"""Bridge TCP and WebSocket bidirectionally."""
|
||||
|
||||
stop_event = self._stop_event
|
||||
assert stop_event is not None
|
||||
|
||||
async def tcp_to_ws() -> None:
|
||||
"""Forward data from TCP to WebSocket."""
|
||||
msg_count = 0
|
||||
try:
|
||||
while not stop_event.is_set():
|
||||
try:
|
||||
data = await asyncio.wait_for(
|
||||
loop.sock_recv(tcp_conn, BUFFER_SIZE),
|
||||
timeout=1.0,
|
||||
)
|
||||
except TimeoutError:
|
||||
continue
|
||||
except (OSError, ConnectionError) as e:
|
||||
logger.debug(f"tcp->ws: recv error after {msg_count} msgs: {e}")
|
||||
break
|
||||
if not data:
|
||||
logger.debug(f"tcp->ws: closed by client after {msg_count} msgs")
|
||||
break
|
||||
msg_count += 1
|
||||
logger.debug(f"tcp->ws: [{msg_count}] sending {len(data)} bytes")
|
||||
await ws.send(data)
|
||||
logger.debug(f"tcp->ws: loop ended after {msg_count} messages")
|
||||
except asyncio.CancelledError:
|
||||
logger.debug(f"tcp->ws: cancelled after {msg_count} messages")
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"tcp->ws: error after {msg_count} messages: {e}")
|
||||
|
||||
async def ws_to_tcp() -> None:
|
||||
"""Forward data from WebSocket to TCP."""
|
||||
msg_count = 0
|
||||
try:
|
||||
logger.debug("ws->tcp: starting message loop")
|
||||
async for message in ws:
|
||||
if stop_event.is_set():
|
||||
break
|
||||
if isinstance(message, bytes) and message:
|
||||
msg_count += 1
|
||||
logger.debug(f"ws->tcp: [{msg_count}] received {len(message)} bytes")
|
||||
await loop.sock_sendall(tcp_conn, message)
|
||||
elif isinstance(message, str):
|
||||
logger.debug(f"ws->tcp: received string: {message[:100]}")
|
||||
logger.debug(f"ws->tcp: loop ended after {msg_count} messages")
|
||||
except asyncio.CancelledError:
|
||||
logger.debug(f"ws->tcp: cancelled after {msg_count} messages")
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"ws->tcp: error after {msg_count} messages: {e}")
|
||||
|
||||
tcp_task = asyncio.create_task(tcp_to_ws())
|
||||
ws_task = asyncio.create_task(ws_to_tcp())
|
||||
|
||||
done, pending = await asyncio.wait(
|
||||
[tcp_task, ws_task],
|
||||
return_when=asyncio.FIRST_COMPLETED,
|
||||
)
|
||||
|
||||
for task in pending:
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def adb_tunnel(remote_url: str, token: str) -> AsyncGenerator[AdbTunnel, None]:
|
||||
"""Context manager for creating and managing an ADB tunnel.
|
||||
|
||||
Args:
|
||||
remote_url: WebSocket URL for the remote ADB endpoint
|
||||
token: Bearer token for authentication
|
||||
|
||||
Yields:
|
||||
AdbTunnel instance with the tunnel running
|
||||
|
||||
Example:
|
||||
async with adb_tunnel(ws_url, token) as tunnel:
|
||||
# Connect ADB to tunnel.local_addr
|
||||
adb_client = AdbClient()
|
||||
adb_client.connect(tunnel.local_addr)
|
||||
"""
|
||||
tunnel = AdbTunnel(remote_url, token)
|
||||
try:
|
||||
await tunnel.start()
|
||||
yield tunnel
|
||||
finally:
|
||||
await tunnel.stop()
|
||||
@@ -0,0 +1,416 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""UIAutomator2 client for Android device screen data retrieval.
|
||||
|
||||
Provides an alternative to the Maestro-based screen API with direct device
|
||||
access.
|
||||
Handles Maestro blocker detection and removal before connecting.
|
||||
"""
|
||||
|
||||
import base64
|
||||
from io import BytesIO
|
||||
import re as _re
|
||||
import subprocess
|
||||
import time
|
||||
from typing import TYPE_CHECKING
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
from PIL import Image
|
||||
from pydantic import BaseModel
|
||||
import uiautomator2 as u2
|
||||
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from uiautomator2 import Device
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
MAESTRO_PACKAGE = "dev.mobile.maestro"
|
||||
|
||||
|
||||
class _CyFunctionDetectorMeta(type):
|
||||
def __instancecheck__(self, instance):
|
||||
name = type(instance).__name__
|
||||
return (
|
||||
name
|
||||
in (
|
||||
"cyfunction",
|
||||
"cython_function_or_method",
|
||||
"builtin_function_or_method",
|
||||
)
|
||||
or "cyfunction" in name.lower()
|
||||
)
|
||||
|
||||
|
||||
class CyFunctionDetector(metaclass=_CyFunctionDetectorMeta):
|
||||
pass
|
||||
|
||||
|
||||
class UIAutomatorScreenData(BaseModel):
|
||||
"""Screen data response from UIAutomator2."""
|
||||
|
||||
model_config = {"ignored_types": (CyFunctionDetector,)}
|
||||
|
||||
base64: str
|
||||
hierarchy_xml: str
|
||||
elements: list[dict]
|
||||
width: int
|
||||
height: int
|
||||
|
||||
|
||||
def _parse_hierarchy_xml_to_elements(hierarchy_xml: str) -> list[dict]:
|
||||
"""Parse uiautomator2 XML hierarchy into a flat list of element dictionaries.
|
||||
|
||||
The output format matches the existing screen API format with attributes
|
||||
like:
|
||||
- resource-id
|
||||
- text
|
||||
- content-desc
|
||||
- bounds (as string "[x1,y1][x2,y2]")
|
||||
- class
|
||||
- package
|
||||
- checkable, checked, clickable, enabled, focusable, focused
|
||||
- scrollable, long-clickable, password, selected
|
||||
|
||||
Args:
|
||||
hierarchy_xml: XML string from uiautomator2.dump_hierarchy()
|
||||
|
||||
Returns:
|
||||
Flat list of element dictionaries
|
||||
"""
|
||||
elements: list[dict] = []
|
||||
|
||||
try:
|
||||
root = ET.fromstring(hierarchy_xml)
|
||||
except ET.ParseError as e:
|
||||
logger.error(f"Failed to parse hierarchy XML: {e}")
|
||||
return elements
|
||||
|
||||
def _extract_element(node: ET.Element) -> None:
|
||||
"""Recursively extract elements from XML nodes."""
|
||||
element: dict = {}
|
||||
|
||||
for attr_name, attr_value in node.attrib.items():
|
||||
if attr_name == "resource-id":
|
||||
element["resource-id"] = attr_value
|
||||
elif attr_name == "text":
|
||||
element["text"] = attr_value
|
||||
elif attr_name == "content-desc":
|
||||
element["content-desc"] = attr_value
|
||||
element["accessibilityText"] = attr_value
|
||||
elif attr_name == "bounds":
|
||||
element["bounds"] = attr_value
|
||||
|
||||
match = _re.match(r"\[(\d+),(\d+)\]\[(\d+),(\d+)\]", attr_value)
|
||||
if match:
|
||||
x1, y1, x2, y2 = map(int, match.groups())
|
||||
element["parsed_bounds"] = {
|
||||
"left": x1,
|
||||
"top": y1,
|
||||
"right": x2,
|
||||
"bottom": y2,
|
||||
}
|
||||
elif attr_name == "class":
|
||||
element["class"] = attr_value
|
||||
elif attr_name == "package":
|
||||
element["package"] = attr_value
|
||||
elif attr_name in (
|
||||
"checkable",
|
||||
"checked",
|
||||
"clickable",
|
||||
"enabled",
|
||||
"focusable",
|
||||
"focused",
|
||||
"scrollable",
|
||||
"long-clickable",
|
||||
"password",
|
||||
"selected",
|
||||
):
|
||||
element[attr_name] = attr_value
|
||||
else:
|
||||
element[attr_name] = attr_value
|
||||
|
||||
if element:
|
||||
elements.append(element)
|
||||
|
||||
for child in node:
|
||||
_extract_element(child)
|
||||
|
||||
_extract_element(root)
|
||||
|
||||
return elements
|
||||
|
||||
|
||||
def _pil_to_base64(img: Image.Image, format: str = "JPEG", quality: int = 80) -> str:
|
||||
"""Convert PIL Image to base64 string with high-quality compression."""
|
||||
buffer = BytesIO()
|
||||
if format.upper() in ("JPEG", "JPG"):
|
||||
format = "JPEG"
|
||||
if img.mode in ("RGBA", "P"):
|
||||
img = img.convert("RGB")
|
||||
img.save(buffer, format=format, quality=quality, optimize=True)
|
||||
else:
|
||||
img.save(buffer, format=format)
|
||||
return base64.b64encode(buffer.getvalue()).decode("utf-8")
|
||||
|
||||
|
||||
def _is_package_installed(device_id: str, pkg: str) -> bool:
|
||||
"""Check if a package is installed on the device."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["adb", "-s", device_id, "shell", "pm", "list", "packages"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
logger.warning(f"Failed to list packages: {result.stderr}")
|
||||
return False
|
||||
|
||||
lines = result.stdout.splitlines()
|
||||
target = f"package:{pkg}"
|
||||
return target in lines
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("Timeout checking installed packages")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.warning(f"Error checking installed packages: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def _uninstall_package(device_id: str, pkg: str) -> bool:
|
||||
"""Uninstall a package from the device for the current user."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[
|
||||
"adb",
|
||||
"-s",
|
||||
device_id,
|
||||
"shell",
|
||||
"pm",
|
||||
"uninstall",
|
||||
"--user",
|
||||
"0",
|
||||
pkg,
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
logger.info(f"Successfully uninstalled {pkg}")
|
||||
return True
|
||||
else:
|
||||
logger.warning(f"Failed to uninstall {pkg}: {result.stderr}")
|
||||
return False
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning(f"Timeout uninstalling {pkg}")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.warning(f"Error uninstalling {pkg}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def _ensure_maestro_not_installed(device_id: str) -> None:
|
||||
"""Check if Maestro is installed and uninstall it.
|
||||
|
||||
Maestro conflicts with uiautomator2 - if Maestro's package is installed,
|
||||
u2.connect() will fail. This function ensures Maestro is removed before
|
||||
attempting to connect.
|
||||
"""
|
||||
if _is_package_installed(device_id, MAESTRO_PACKAGE):
|
||||
logger.warning(
|
||||
f"Maestro ({MAESTRO_PACKAGE}) detected - uninstalling to enable UIAutomator2..."
|
||||
)
|
||||
_uninstall_package(device_id, MAESTRO_PACKAGE)
|
||||
|
||||
|
||||
class UIAutomatorClient:
|
||||
"""UIAutomator2 client for Android screen data retrieval.
|
||||
|
||||
This client uses uiautomator2 library for direct device communication,
|
||||
providing faster hierarchy and screenshot retrieval compared to Maestro.
|
||||
|
||||
Important: Maestro must not be installed on the device as it conflicts
|
||||
with uiautomator2. This client automatically handles Maestro removal.
|
||||
"""
|
||||
|
||||
def __init__(self, device_id: str):
|
||||
"""Initialize the UIAutomator client.
|
||||
|
||||
Args:
|
||||
device_id: The Android device serial number (e.g., "emulator-5554")
|
||||
"""
|
||||
self._device_id = device_id
|
||||
self._device: Device | None = None
|
||||
|
||||
def _ensure_connected(self) -> "Device":
|
||||
"""Ensure connection to the device, handling Maestro blocker.
|
||||
|
||||
Returns:
|
||||
Connected uiautomator2 Device instance
|
||||
"""
|
||||
if self._device is not None:
|
||||
try:
|
||||
# Quick check if connection is still alive
|
||||
self._device.info
|
||||
return self._device
|
||||
except Exception:
|
||||
logger.warning("UIAutomator2 connection lost, reconnecting...")
|
||||
self._device = None
|
||||
|
||||
# Ensure Maestro is not blocking us
|
||||
_ensure_maestro_not_installed(self._device_id)
|
||||
|
||||
# Connect to device
|
||||
logger.info(f"Connecting UIAutomator2 to device: {self._device_id}")
|
||||
self._device = u2.connect(self._device_id)
|
||||
logger.info("UIAutomator2 connected successfully")
|
||||
return self._device
|
||||
|
||||
def press_key(self, key: str):
|
||||
"""Press a key on the device.
|
||||
|
||||
Args:
|
||||
key: Key to press (e.g., "home", "back", "enter"...)
|
||||
"""
|
||||
device = self._ensure_connected()
|
||||
return device.press(key=key)
|
||||
|
||||
def send_text(self, text: str) -> None:
|
||||
"""Send text input to the device using FastInputIME.
|
||||
|
||||
This method supports special characters (e.g., 'ö') that ADB shell
|
||||
input text cannot handle. The original IME is restored after input.
|
||||
|
||||
Args:
|
||||
text: The text to input
|
||||
"""
|
||||
|
||||
device = self._ensure_connected()
|
||||
device.set_fastinput_ime(True)
|
||||
# Allow IME to initialize
|
||||
time.sleep(0.3)
|
||||
try:
|
||||
device.send_keys(text)
|
||||
# Give FastInputIME time to process the broadcast and commit text
|
||||
# before switching it off and killing it.
|
||||
time.sleep(0.5)
|
||||
finally:
|
||||
device.set_fastinput_ime(False)
|
||||
|
||||
def clear_text(self) -> bool:
|
||||
"""Clear text of the currently focused element using UIAutomator2.
|
||||
|
||||
Returns True on success, False on failure.
|
||||
"""
|
||||
try:
|
||||
device = self._ensure_connected()
|
||||
device(focused=True).clear_text()
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"UIAutomator2 clear_text failed: {e}")
|
||||
return False
|
||||
|
||||
def get_hierarchy(self) -> str:
|
||||
"""Get the UI hierarchy XML from the device.
|
||||
|
||||
Returns:
|
||||
Compressed UI hierarchy XML string
|
||||
"""
|
||||
device = self._ensure_connected()
|
||||
return device.dump_hierarchy(compressed=True)
|
||||
|
||||
def get_screenshot(self) -> Image.Image | None:
|
||||
"""Capture a screenshot from the device.
|
||||
|
||||
Uses adb screencap for more reliable capture of all layers (including
|
||||
popups), falling back to UIAutomator2 if it fails.
|
||||
|
||||
Returns:
|
||||
PIL Image or None if capture failed
|
||||
"""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["adb", "-s", self._device_id, "exec-out", "screencap", "-p"],
|
||||
capture_output=True,
|
||||
timeout=10,
|
||||
check=True,
|
||||
)
|
||||
return Image.open(BytesIO(result.stdout))
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
f"Failed to capture screenshot via adb: {e}, falling back to uiautomator2"
|
||||
)
|
||||
device = self._ensure_connected()
|
||||
return device.screenshot()
|
||||
|
||||
def get_screenshot_base64(self) -> str | None:
|
||||
"""Capture a screenshot and return as base64 string.
|
||||
|
||||
Returns:
|
||||
Base64 encoded PNG screenshot or None if capture failed
|
||||
"""
|
||||
screenshot = self.get_screenshot()
|
||||
if screenshot is None:
|
||||
return None
|
||||
return _pil_to_base64(screenshot, format="JPEG")
|
||||
|
||||
def get_screen_data(self) -> UIAutomatorScreenData:
|
||||
"""Get complete screen data including screenshot and hierarchy.
|
||||
|
||||
Returns:
|
||||
UIAutomatorScreenData with screenshot, hierarchy, elements, and
|
||||
dimensions
|
||||
"""
|
||||
device = self._ensure_connected()
|
||||
|
||||
# Get screenshot
|
||||
screenshot = self.get_screenshot()
|
||||
if screenshot is None:
|
||||
raise RuntimeError("Failed to capture screenshot")
|
||||
|
||||
# Get hierarchy
|
||||
hierarchy_xml = device.dump_hierarchy(compressed=True)
|
||||
|
||||
# Parse XML to flat elements list
|
||||
elements = _parse_hierarchy_xml_to_elements(hierarchy_xml)
|
||||
|
||||
return UIAutomatorScreenData(
|
||||
base64=_pil_to_base64(screenshot, format="JPEG"),
|
||||
hierarchy_xml=hierarchy_xml,
|
||||
elements=elements,
|
||||
width=screenshot.width,
|
||||
height=screenshot.height,
|
||||
)
|
||||
|
||||
def disconnect(self) -> None:
|
||||
"""Disconnect from the device."""
|
||||
self._device = None
|
||||
logger.info("UIAutomator2 client disconnected")
|
||||
|
||||
|
||||
def get_client(device_id: str) -> UIAutomatorClient:
|
||||
"""Factory function to create a UIAutomatorClient.
|
||||
|
||||
Args:
|
||||
device_id: The Android device serial number
|
||||
|
||||
Returns:
|
||||
UIAutomatorClient instance
|
||||
"""
|
||||
return UIAutomatorClient(device_id=device_id)
|
||||
@@ -0,0 +1,253 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Unified configuration package and single entry point for ARTEMIS settings, paths, and models."""
|
||||
|
||||
from artemis.config.agent import (
|
||||
AgentGlobalConfig,
|
||||
CheckerConfig,
|
||||
CommitteeConfig,
|
||||
ExplorerConfig,
|
||||
FlashProfileConfig,
|
||||
OutputterConfig,
|
||||
PlannerValidationConfig,
|
||||
ProProfileConfig,
|
||||
StepSummarizerConfig,
|
||||
VideoAnalyzerConfig,
|
||||
load_agent_config,
|
||||
resolve_explorer_version,
|
||||
)
|
||||
from artemis.config.constants import (
|
||||
AGENT_CONFIG_FILENAME,
|
||||
DATA_ENGINE_DB_FILENAME,
|
||||
DEFAULT_ACTION_TIMEOUT,
|
||||
DEFAULT_ADB_HOST,
|
||||
DEFAULT_ADB_PORT,
|
||||
DEFAULT_EDGE_MONITOR_PORT,
|
||||
DEFAULT_EXPLORER_VERSION,
|
||||
DEFAULT_GATEWAY_PORT,
|
||||
DEFAULT_MODEL,
|
||||
DEFAULT_PROFILE,
|
||||
DEFAULT_STREAM_PING_INTERVAL,
|
||||
DOTENV_FILENAME,
|
||||
ENV_ADB_DEVICE_SERIAL,
|
||||
ENV_ADB_HOST,
|
||||
ENV_ADB_PORT,
|
||||
ENV_ADB_SERVER_SOCKET,
|
||||
ENV_ANTHROPIC_API_KEY,
|
||||
ENV_ANTIGRAVITY_APP_DIR,
|
||||
ENV_ANTIGRAVITY_LS_ADDRESS,
|
||||
ENV_DATA_ENGINE_DB_PATH,
|
||||
ENV_EVENTS_OUTPUT_PATH,
|
||||
ENV_GCP_API_KEY,
|
||||
ENV_GEMINI_API_KEY,
|
||||
ENV_GOOGLE_API_KEY,
|
||||
ENV_KEEP_VIDEOS,
|
||||
ENV_ARTEMIS_APP_DIR,
|
||||
ENV_ARTEMIS_CLOUD_GATEWAY_URL,
|
||||
ENV_ARTEMIS_CLOUD_MODE,
|
||||
ENV_ARTEMIS_CLOUD_SESSION_ID,
|
||||
ENV_ARTEMIS_DEBUG,
|
||||
ENV_ARTEMIS_DEFAULT_MODEL,
|
||||
ENV_ARTEMIS_DEFAULT_PROFILE,
|
||||
ENV_ARTEMIS_DEVICE_ID,
|
||||
ENV_ARTEMIS_EDGE_PORT,
|
||||
ENV_ARTEMIS_EXPLORER_VERSION,
|
||||
ENV_ARTEMIS_IPC_PORT,
|
||||
ENV_ARTEMIS_MCP_SERVER,
|
||||
ENV_ARTEMIS_TENANT_ID,
|
||||
ENV_ARTEMIS_TENANT_TOKEN,
|
||||
ENV_ARTEMIS_TRACES_DIR,
|
||||
ENV_ARTEMIS_USE_FILE_API,
|
||||
ENV_ARTEMIS_USE_USER_DIR,
|
||||
ENV_OPEN_ROUTER_API_KEY,
|
||||
ENV_OPENAI_API_KEY,
|
||||
ENV_OPENAI_BASE_URL,
|
||||
ENV_RESULTS_OUTPUT_PATH,
|
||||
ENV_XAI_API_KEY,
|
||||
IPC_PORT_FILENAME,
|
||||
LLM_CONFIG_FILENAME,
|
||||
LLM_CONFIG_OVERRIDE_FILENAME,
|
||||
LS_ADDRESS_FILENAME,
|
||||
AgentNode,
|
||||
AgentNodeWithFallback,
|
||||
ExplorerVersion,
|
||||
LLMProvider,
|
||||
LLMUtilsNode,
|
||||
LLMUtilsNodeWithFallback,
|
||||
)
|
||||
from artemis.config.llm import (
|
||||
CyFunctionDetector,
|
||||
LLM,
|
||||
LLMConfig,
|
||||
LLMConfigUtils,
|
||||
LLMWithFallback,
|
||||
deep_merge_llm_config,
|
||||
get_default_llm_config,
|
||||
initialize_llm_config,
|
||||
load_llm_config_override,
|
||||
parse_llm_config,
|
||||
validate_vertex_ai_credentials,
|
||||
)
|
||||
from artemis.config.output import (
|
||||
OutputConfig,
|
||||
prepare_output_files,
|
||||
record_events,
|
||||
)
|
||||
from artemis.config.paths import (
|
||||
CONFIG_DIR,
|
||||
GLOBAL_APP_DIR,
|
||||
GLOBAL_JETSKI_DIR,
|
||||
ROOT_DIR,
|
||||
get_app_dir,
|
||||
get_config_path,
|
||||
get_data_engine_db_path,
|
||||
get_default_traces_path,
|
||||
get_ipc_port_file,
|
||||
get_ls_address_file,
|
||||
get_temp_dir,
|
||||
get_traces_dir,
|
||||
is_frozen_bundle,
|
||||
)
|
||||
from artemis.config.runtime import (
|
||||
cleanup_temp_dir,
|
||||
clear_ipc_port,
|
||||
clear_ls_address,
|
||||
read_ipc_port,
|
||||
read_ls_address,
|
||||
write_ipc_port,
|
||||
write_ls_address,
|
||||
)
|
||||
from artemis.config.settings import (
|
||||
Settings,
|
||||
settings,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
# Settings & Singleton
|
||||
"Settings",
|
||||
"settings",
|
||||
# Paths & Dirs
|
||||
"ROOT_DIR",
|
||||
"CONFIG_DIR",
|
||||
"GLOBAL_APP_DIR",
|
||||
"GLOBAL_JETSKI_DIR",
|
||||
"get_app_dir",
|
||||
"get_traces_dir",
|
||||
"get_default_traces_path",
|
||||
"get_temp_dir",
|
||||
"get_config_path",
|
||||
"get_data_engine_db_path",
|
||||
"get_ipc_port_file",
|
||||
"get_ls_address_file",
|
||||
"is_frozen_bundle",
|
||||
# LLM Config & Models
|
||||
"CyFunctionDetector",
|
||||
"LLM",
|
||||
"LLMWithFallback",
|
||||
"LLMConfigUtils",
|
||||
"LLMConfig",
|
||||
"parse_llm_config",
|
||||
"initialize_llm_config",
|
||||
"get_default_llm_config",
|
||||
"deep_merge_llm_config",
|
||||
"load_llm_config_override",
|
||||
"validate_vertex_ai_credentials",
|
||||
# Agent & Tool Config
|
||||
"AgentGlobalConfig",
|
||||
"CheckerConfig",
|
||||
"CommitteeConfig",
|
||||
"ExplorerConfig",
|
||||
"FlashProfileConfig",
|
||||
"OutputterConfig",
|
||||
"PlannerValidationConfig",
|
||||
"ProProfileConfig",
|
||||
"StepSummarizerConfig",
|
||||
"VideoAnalyzerConfig",
|
||||
"load_agent_config",
|
||||
"resolve_explorer_version",
|
||||
# Output Config
|
||||
"OutputConfig",
|
||||
"prepare_output_files",
|
||||
"record_events",
|
||||
# Runtime & IPC State
|
||||
"read_ipc_port",
|
||||
"write_ipc_port",
|
||||
"clear_ipc_port",
|
||||
"read_ls_address",
|
||||
"write_ls_address",
|
||||
"clear_ls_address",
|
||||
"cleanup_temp_dir",
|
||||
# Types & Literals
|
||||
"LLMProvider",
|
||||
"ExplorerVersion",
|
||||
"LLMUtilsNode",
|
||||
"LLMUtilsNodeWithFallback",
|
||||
"AgentNode",
|
||||
"AgentNodeWithFallback",
|
||||
# Filenames
|
||||
"LLM_CONFIG_FILENAME",
|
||||
"LLM_CONFIG_OVERRIDE_FILENAME",
|
||||
"AGENT_CONFIG_FILENAME",
|
||||
"DATA_ENGINE_DB_FILENAME",
|
||||
"IPC_PORT_FILENAME",
|
||||
"LS_ADDRESS_FILENAME",
|
||||
"DOTENV_FILENAME",
|
||||
# Defaults
|
||||
"DEFAULT_ADB_HOST",
|
||||
"DEFAULT_ADB_PORT",
|
||||
"DEFAULT_GATEWAY_PORT",
|
||||
"DEFAULT_EDGE_MONITOR_PORT",
|
||||
"DEFAULT_ACTION_TIMEOUT",
|
||||
"DEFAULT_STREAM_PING_INTERVAL",
|
||||
"DEFAULT_PROFILE",
|
||||
"DEFAULT_MODEL",
|
||||
"DEFAULT_EXPLORER_VERSION",
|
||||
# Environment Variables
|
||||
"ENV_ARTEMIS_APP_DIR",
|
||||
"ENV_ANTIGRAVITY_APP_DIR",
|
||||
"ENV_ARTEMIS_USE_USER_DIR",
|
||||
"ENV_ARTEMIS_TRACES_DIR",
|
||||
"ENV_DATA_ENGINE_DB_PATH",
|
||||
"ENV_ARTEMIS_CLOUD_MODE",
|
||||
"ENV_ARTEMIS_CLOUD_SESSION_ID",
|
||||
"ENV_ARTEMIS_CLOUD_GATEWAY_URL",
|
||||
"ENV_ARTEMIS_TENANT_ID",
|
||||
"ENV_ARTEMIS_TENANT_TOKEN",
|
||||
"ENV_ARTEMIS_EDGE_PORT",
|
||||
"ENV_ARTEMIS_IPC_PORT",
|
||||
"ENV_ANTIGRAVITY_LS_ADDRESS",
|
||||
"ENV_ARTEMIS_MCP_SERVER",
|
||||
"ENV_ADB_DEVICE_SERIAL",
|
||||
"ENV_ARTEMIS_DEVICE_ID",
|
||||
"ENV_ADB_HOST",
|
||||
"ENV_ADB_PORT",
|
||||
"ENV_ADB_SERVER_SOCKET",
|
||||
"ENV_GOOGLE_API_KEY",
|
||||
"ENV_GEMINI_API_KEY",
|
||||
"ENV_GCP_API_KEY",
|
||||
"ENV_OPENAI_API_KEY",
|
||||
"ENV_OPENAI_BASE_URL",
|
||||
"ENV_ANTHROPIC_API_KEY",
|
||||
"ENV_OPEN_ROUTER_API_KEY",
|
||||
"ENV_XAI_API_KEY",
|
||||
"ENV_ARTEMIS_EXPLORER_VERSION",
|
||||
"ENV_ARTEMIS_DEFAULT_PROFILE",
|
||||
"ENV_ARTEMIS_DEFAULT_MODEL",
|
||||
"ENV_ARTEMIS_USE_FILE_API",
|
||||
"ENV_KEEP_VIDEOS",
|
||||
"ENV_ARTEMIS_DEBUG",
|
||||
"ENV_EVENTS_OUTPUT_PATH",
|
||||
"ENV_RESULTS_OUTPUT_PATH",
|
||||
]
|
||||
@@ -0,0 +1,490 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Agent tools, sub-agent capabilities, and explorer settings configuration."""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from artemis.config.constants import (
|
||||
AGENT_CONFIG_FILENAME,
|
||||
DEFAULT_EXPLORER_VERSION,
|
||||
ExplorerVersion,
|
||||
)
|
||||
from artemis.config.paths import ROOT_DIR, get_config_path
|
||||
from artemis.utils.file import load_jsonc
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class VideoAnalyzerConfig(BaseModel):
|
||||
"""Configuration specific to VideoAnalyzer and screen recording."""
|
||||
|
||||
enabled: bool | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Whether continuous screen video recording and video analyzer tools "
|
||||
"are enabled. If None, auto-detected based on scrcpy & ffmpeg availability."
|
||||
),
|
||||
)
|
||||
enable_ledger: bool = Field(
|
||||
default=True,
|
||||
description="Whether to enable video action ledger state tracking.",
|
||||
)
|
||||
model_config = {"extra": "allow"}
|
||||
|
||||
|
||||
class PlannerValidationConfig(BaseModel):
|
||||
"""Configuration specific to Planner validation of milestone changes."""
|
||||
|
||||
enabled: bool = Field(
|
||||
default=False,
|
||||
description="Whether to enable async planner validation when task plan milestones change.",
|
||||
)
|
||||
similarity_threshold: float = Field(
|
||||
default=0.85,
|
||||
description="Difflib similarity threshold below which milestone text changes trigger re-validation.",
|
||||
)
|
||||
model_config = {"extra": "allow"}
|
||||
|
||||
|
||||
class CommitteeConfig(BaseModel):
|
||||
"""Configuration specific to Multi-Agent Committee / Council."""
|
||||
|
||||
enabled: bool = Field(
|
||||
default=False,
|
||||
description="Whether to equip Operator with the ask_committee tool.",
|
||||
)
|
||||
debate_rounds: int = Field(
|
||||
default=2,
|
||||
ge=1,
|
||||
le=5,
|
||||
description="Number of debate rounds among council members (1-5).",
|
||||
)
|
||||
model_config = {"extra": "allow"}
|
||||
|
||||
|
||||
class CheckerConfig(BaseModel):
|
||||
"""Configuration specific to Checker subgoal verification and rollback."""
|
||||
|
||||
enabled: bool = Field(
|
||||
default=False,
|
||||
description="Whether to enable visual verification and snapshot rollback by the Checker agent.",
|
||||
)
|
||||
max_iterations: int = Field(
|
||||
default=20,
|
||||
ge=1,
|
||||
le=50,
|
||||
description="Maximum verification iterations for Checker LLM reasoning and inspection.",
|
||||
)
|
||||
max_chat_rounds: int = Field(
|
||||
default=4,
|
||||
ge=1,
|
||||
le=10,
|
||||
description="Maximum debate/rebuttal rounds between Operator and Checker.",
|
||||
)
|
||||
model_config = {"extra": "allow"}
|
||||
|
||||
|
||||
class OutputterConfig(BaseModel):
|
||||
"""Configuration specific to Outputter post-execution report synthesis agent."""
|
||||
|
||||
enabled: bool = Field(
|
||||
default=True,
|
||||
description="Whether Outputter agent is mounted for post-execution report/structured output synthesis.",
|
||||
)
|
||||
force_synthesis: bool = Field(
|
||||
default=False,
|
||||
description="Whether to force Outputter synthesis even when no structured schema is explicitly requested.",
|
||||
)
|
||||
model_config = {"extra": "allow"}
|
||||
|
||||
|
||||
class ExplorerConfig(BaseModel):
|
||||
"""Configuration specific to UI Explorer visual sub-agent (Flash / Pro / Ultra)."""
|
||||
|
||||
default_version: ExplorerVersion = Field(
|
||||
default=DEFAULT_EXPLORER_VERSION,
|
||||
description="Default Explorer version mode ('flash', 'pro', or 'ultra').",
|
||||
)
|
||||
flash_mode: ExplorerVersion = Field(
|
||||
default="flash",
|
||||
description="Explorer version mode when main system runs under Flash profile (FlashRunner).",
|
||||
)
|
||||
pro_mode: ExplorerVersion = Field(
|
||||
default="flash",
|
||||
description="Explorer version mode when main system runs under Pro profile (Graph / Operator / Validator).",
|
||||
)
|
||||
caching: bool = Field(
|
||||
default=True,
|
||||
description="Whether to enable context caching for multi-turn pro/ultra Explorer.",
|
||||
)
|
||||
model_config = {"extra": "allow"}
|
||||
|
||||
def resolve(
|
||||
self,
|
||||
explicit_version: str | None = None,
|
||||
agent_name: str | None = None,
|
||||
profile: str | None = None,
|
||||
per_agent_overrides: dict[str, str] | None = None,
|
||||
) -> ExplorerVersion:
|
||||
"""Resolves active Explorer version with clean precedence rules."""
|
||||
if explicit_version:
|
||||
v = str(explicit_version).strip().lower()
|
||||
if v in ("flash", "pro", "ultra"):
|
||||
return v # type: ignore
|
||||
|
||||
env_v = os.getenv("ARTEMIS_EXPLORER_VERSION", "").strip().lower()
|
||||
if env_v in ("flash", "pro", "ultra"):
|
||||
return env_v # type: ignore
|
||||
|
||||
if per_agent_overrides and agent_name and agent_name in per_agent_overrides:
|
||||
v = str(per_agent_overrides[agent_name]).strip().lower()
|
||||
if v in ("flash", "pro", "ultra"):
|
||||
return v # type: ignore
|
||||
|
||||
if profile in ("flash", "flash_runner") or agent_name in ("flash", "flash_runner"):
|
||||
return self.flash_mode
|
||||
if profile in ("pro", "operator", "validator") or agent_name in (
|
||||
"pro",
|
||||
"operator",
|
||||
"validator",
|
||||
):
|
||||
return self.pro_mode
|
||||
|
||||
return self.default_version
|
||||
|
||||
|
||||
class StepSummarizerConfig(BaseModel):
|
||||
"""Configuration specific to Flash asynchronous step visual summarization."""
|
||||
|
||||
enabled: bool = Field(
|
||||
default=True,
|
||||
description="Whether to asynchronously summarize historical steps to replace pruned images.",
|
||||
)
|
||||
model: str = Field(
|
||||
default="gemini-2.5-flash-lite",
|
||||
description="Lightweight model used for background step state summarization.",
|
||||
)
|
||||
prune_history_xml: bool = Field(
|
||||
default=True,
|
||||
description="Whether to prune outdated heavy UI XML trees from historical turns.",
|
||||
)
|
||||
model_config = {"extra": "allow"}
|
||||
|
||||
|
||||
class FlashProfileConfig(BaseModel):
|
||||
"""Configuration options specific to the ⚡ Flash execution profile (FlashRunner/ReactiveRunner)."""
|
||||
|
||||
max_turns: int = Field(
|
||||
default=30,
|
||||
ge=1,
|
||||
le=100,
|
||||
description="Maximum reactive turns for Flash execution before timeout.",
|
||||
)
|
||||
explorer_mode: ExplorerVersion = Field(
|
||||
default="flash",
|
||||
description="Perception mode for Flash runner ('flash' for 1-shot detection).",
|
||||
)
|
||||
step_summarizer: StepSummarizerConfig = Field(
|
||||
default_factory=StepSummarizerConfig,
|
||||
description="Asynchronous visual context compressor settings.",
|
||||
)
|
||||
model_config = {"extra": "allow"}
|
||||
|
||||
|
||||
class ProExplorerConfig(BaseModel):
|
||||
"""Explorer perception settings under Pro profile."""
|
||||
|
||||
mode: ExplorerVersion = Field(
|
||||
default="flash",
|
||||
description="Explorer perception mode under Pro profile ('flash', 'pro', or 'ultra').",
|
||||
)
|
||||
caching: bool = Field(
|
||||
default=True,
|
||||
description="Whether to enable context caching for multi-turn Pro/Ultra exploration.",
|
||||
)
|
||||
model_config = {"extra": "allow"}
|
||||
|
||||
|
||||
class ProProfileConfig(BaseModel):
|
||||
"""Configuration options specific to the 🚀 Pro execution profile (LangGraph / Multi-Agent Closed-Loop)."""
|
||||
|
||||
explorer: ProExplorerConfig = Field(
|
||||
default_factory=ProExplorerConfig,
|
||||
description="Explorer perception settings for Pro profile.",
|
||||
)
|
||||
planner_validation: PlannerValidationConfig = Field(
|
||||
default_factory=PlannerValidationConfig,
|
||||
description="Planner milestone consistency validation settings.",
|
||||
)
|
||||
committee: CommitteeConfig = Field(
|
||||
default_factory=CommitteeConfig,
|
||||
description="Multi-Agent Committee council debate settings.",
|
||||
)
|
||||
checker: CheckerConfig = Field(
|
||||
default_factory=CheckerConfig,
|
||||
description="Checker visual verification and rollback settings.",
|
||||
)
|
||||
video_analyzer: VideoAnalyzerConfig = Field(
|
||||
default_factory=VideoAnalyzerConfig,
|
||||
description="Screen video action ledger tracking settings.",
|
||||
)
|
||||
model_config = {"extra": "allow"}
|
||||
|
||||
|
||||
class AgentGlobalConfig(BaseModel):
|
||||
"""Global configuration parsed from artemis.jsonc / agent_config.json."""
|
||||
|
||||
# ⚡ Profile-categorized configurations
|
||||
flash: FlashProfileConfig = Field(
|
||||
default_factory=FlashProfileConfig,
|
||||
description="Options for ⚡ Flash reactive runner execution.",
|
||||
)
|
||||
pro: ProProfileConfig = Field(
|
||||
default_factory=ProProfileConfig,
|
||||
description="Options for 🚀 Pro multi-agent graph execution.",
|
||||
)
|
||||
|
||||
# 🌐 Core Component Configurations
|
||||
explorer: ExplorerConfig = Field(
|
||||
default_factory=ExplorerConfig,
|
||||
description="UI Explorer sub-agent runtime options.",
|
||||
)
|
||||
explorer_versions: dict[str, str] = Field(
|
||||
default_factory=lambda: {"operator": "flash", "validator": "flash"},
|
||||
description="Per-agent mapping of UI explorer version (e.g. flash, pro, ultra).",
|
||||
)
|
||||
planner_validation: PlannerValidationConfig = Field(
|
||||
default_factory=PlannerValidationConfig,
|
||||
description="Planner milestone validation runtime options.",
|
||||
)
|
||||
committee: CommitteeConfig = Field(
|
||||
default_factory=CommitteeConfig,
|
||||
description="Committee council runtime options.",
|
||||
)
|
||||
checker: CheckerConfig = Field(
|
||||
default_factory=CheckerConfig,
|
||||
description="Checker subgoal verification and rollback runtime options.",
|
||||
)
|
||||
outputter: OutputterConfig = Field(
|
||||
default_factory=OutputterConfig,
|
||||
description="Outputter synthesis agent runtime options.",
|
||||
)
|
||||
blacklisted_tools: dict[str, list[str]] = Field(
|
||||
default_factory=dict,
|
||||
description="List of blacklisted tools per agent.",
|
||||
)
|
||||
video_analyzer: VideoAnalyzerConfig = Field(
|
||||
default_factory=VideoAnalyzerConfig,
|
||||
description="Video analyzer runtime options.",
|
||||
)
|
||||
|
||||
model_config = {"extra": "allow"}
|
||||
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def synchronize_profiles_and_components(cls, data: Any) -> Any:
|
||||
"""Bidirectionally synchronize between categorized 'flash'/'pro' blocks and component fields."""
|
||||
if not isinstance(data, dict):
|
||||
return data
|
||||
|
||||
# 1. Sync from pro block to root components if provided
|
||||
if "pro" in data and isinstance(data["pro"], dict):
|
||||
pro_data = data["pro"]
|
||||
if "checker" in pro_data and "checker" not in data:
|
||||
data["checker"] = pro_data["checker"]
|
||||
if "committee" in pro_data and "committee" not in data:
|
||||
data["committee"] = pro_data["committee"]
|
||||
if "planner_validation" in pro_data and "planner_validation" not in data:
|
||||
data["planner_validation"] = pro_data["planner_validation"]
|
||||
if "video_analyzer" in pro_data and "video_analyzer" not in data:
|
||||
data["video_analyzer"] = pro_data["video_analyzer"]
|
||||
if "explorer" in pro_data and isinstance(pro_data["explorer"], dict):
|
||||
exp_pro = pro_data["explorer"]
|
||||
if "explorer" not in data:
|
||||
data["explorer"] = {}
|
||||
if isinstance(data["explorer"], dict):
|
||||
if "mode" in exp_pro and "pro_mode" not in data["explorer"]:
|
||||
data["explorer"]["pro_mode"] = exp_pro["mode"]
|
||||
if "caching" in exp_pro and "caching" not in data["explorer"]:
|
||||
data["explorer"]["caching"] = exp_pro["caching"]
|
||||
|
||||
# 2. Sync from flash block to explorer.flash_mode
|
||||
if "flash" in data and isinstance(data["flash"], dict):
|
||||
flash_data = data["flash"]
|
||||
if "explorer_mode" in flash_data:
|
||||
if "explorer" not in data:
|
||||
data["explorer"] = {}
|
||||
if isinstance(data["explorer"], dict) and "flash_mode" not in data["explorer"]:
|
||||
data["explorer"]["flash_mode"] = flash_data["explorer_mode"]
|
||||
|
||||
# 3. Sync from root components to pro/flash blocks if pro/flash block is omitted
|
||||
if "checker" in data and ("pro" not in data or "checker" not in data.get("pro", {})):
|
||||
data.setdefault("pro", {})["checker"] = data["checker"]
|
||||
if "committee" in data and ("pro" not in data or "committee" not in data.get("pro", {})):
|
||||
data.setdefault("pro", {})["committee"] = data["committee"]
|
||||
if "planner_validation" in data and (
|
||||
"pro" not in data or "planner_validation" not in data.get("pro", {})
|
||||
):
|
||||
data.setdefault("pro", {})["planner_validation"] = data["planner_validation"]
|
||||
if "video_analyzer" in data and (
|
||||
"pro" not in data or "video_analyzer" not in data.get("pro", {})
|
||||
):
|
||||
data.setdefault("pro", {})["video_analyzer"] = data["video_analyzer"]
|
||||
|
||||
# 4. Environment variable overrides (highest precedence for runtime switches)
|
||||
def _parse_bool(var_name: str) -> bool | None:
|
||||
v = os.getenv(var_name)
|
||||
if v is None:
|
||||
return None
|
||||
v_clean = v.strip().lower()
|
||||
if v_clean in ("1", "true", "yes", "on"):
|
||||
return True
|
||||
if v_clean in ("0", "false", "no", "off"):
|
||||
return False
|
||||
return None
|
||||
|
||||
chk_env = _parse_bool("ARTEMIS_CHECKER_ENABLED")
|
||||
if chk_env is not None:
|
||||
data.setdefault("checker", {})["enabled"] = chk_env
|
||||
data.setdefault("pro", {}).setdefault("checker", {})["enabled"] = chk_env
|
||||
|
||||
comm_env = _parse_bool("ARTEMIS_COMMITTEE_ENABLED")
|
||||
if comm_env is not None:
|
||||
data.setdefault("committee", {})["enabled"] = comm_env
|
||||
data.setdefault("pro", {}).setdefault("committee", {})["enabled"] = comm_env
|
||||
|
||||
pv_env = _parse_bool("ARTEMIS_PLANNER_VALIDATION_ENABLED")
|
||||
if pv_env is not None:
|
||||
data.setdefault("planner_validation", {})["enabled"] = pv_env
|
||||
data.setdefault("pro", {}).setdefault("planner_validation", {})["enabled"] = pv_env
|
||||
|
||||
out_env = _parse_bool("ARTEMIS_OUTPUTTER_ENABLED")
|
||||
if out_env is not None:
|
||||
data.setdefault("outputter", {})["enabled"] = out_env
|
||||
|
||||
vid_rec_env = _parse_bool("ARTEMIS_VIDEO_RECORDING_ENABLED") or _parse_bool(
|
||||
"ARTEMIS_WITH_VIDEO_RECORDING_TOOLS"
|
||||
)
|
||||
if vid_rec_env is not None:
|
||||
data.setdefault("video_analyzer", {})["enabled"] = vid_rec_env
|
||||
data.setdefault("pro", {}).setdefault("video_analyzer", {})["enabled"] = vid_rec_env
|
||||
|
||||
vid_env = _parse_bool("ARTEMIS_VIDEO_LEDGER_ENABLED")
|
||||
if vid_env is not None:
|
||||
data.setdefault("video_analyzer", {})["enable_ledger"] = vid_env
|
||||
data.setdefault("pro", {}).setdefault("video_analyzer", {})["enable_ledger"] = vid_env
|
||||
|
||||
return data
|
||||
|
||||
def get_explorer_version(
|
||||
self,
|
||||
explicit_version: str | None = None,
|
||||
agent_name: str | None = "operator",
|
||||
profile: str | None = None,
|
||||
) -> ExplorerVersion:
|
||||
"""Resolve Explorer version from this configuration."""
|
||||
return self.explorer.resolve(
|
||||
explicit_version=explicit_version,
|
||||
agent_name=agent_name,
|
||||
profile=profile,
|
||||
per_agent_overrides=self.explorer_versions,
|
||||
)
|
||||
|
||||
|
||||
def resolve_explorer_version(
|
||||
ctx: Any | None = None,
|
||||
explicit_version: str | None = None,
|
||||
agent_or_profile_name: str | None = "operator",
|
||||
) -> ExplorerVersion:
|
||||
"""Resolves active Explorer version based on explicit args, env vars, agent_config, and execution setup."""
|
||||
if explicit_version:
|
||||
v = str(explicit_version).strip().lower()
|
||||
if v in ("flash", "pro", "ultra"):
|
||||
return v # type: ignore
|
||||
|
||||
env_v = os.getenv("ARTEMIS_EXPLORER_VERSION", "").strip().lower()
|
||||
if env_v in ("flash", "pro", "ultra"):
|
||||
return env_v # type: ignore
|
||||
|
||||
if ctx:
|
||||
agent_cfg = getattr(ctx, "agent_config", None)
|
||||
exec_setup = getattr(ctx, "execution_setup", None)
|
||||
|
||||
if agent_cfg:
|
||||
if hasattr(agent_cfg, "get_explorer_version"):
|
||||
return agent_cfg.get_explorer_version(
|
||||
explicit_version=explicit_version,
|
||||
agent_name=agent_or_profile_name,
|
||||
)
|
||||
if hasattr(agent_cfg, "explorer") and hasattr(agent_cfg.explorer, "resolve"):
|
||||
overrides = getattr(agent_cfg, "explorer_versions", None)
|
||||
return agent_cfg.explorer.resolve(
|
||||
explicit_version=explicit_version,
|
||||
agent_name=agent_or_profile_name,
|
||||
per_agent_overrides=overrides,
|
||||
)
|
||||
|
||||
if (
|
||||
exec_setup
|
||||
and hasattr(exec_setup, "explorer")
|
||||
and hasattr(exec_setup.explorer, "resolve")
|
||||
):
|
||||
overrides = getattr(exec_setup, "explorer_versions", None)
|
||||
return exec_setup.explorer.resolve(
|
||||
explicit_version=explicit_version,
|
||||
agent_name=agent_or_profile_name,
|
||||
per_agent_overrides=overrides,
|
||||
)
|
||||
|
||||
return "flash"
|
||||
|
||||
|
||||
def load_agent_config(
|
||||
config_path: Path | None = None,
|
||||
) -> AgentGlobalConfig:
|
||||
"""Load and validate AgentGlobalConfig from artemis.jsonc or agent_config.json.
|
||||
|
||||
Args:
|
||||
config_path: Optional explicit file path to configuration file.
|
||||
|
||||
Returns:
|
||||
Validated AgentGlobalConfig instance.
|
||||
"""
|
||||
resolved_path = config_path
|
||||
if not resolved_path:
|
||||
for candidate in ("artemis.jsonc", "artemis.json", AGENT_CONFIG_FILENAME):
|
||||
try:
|
||||
resolved_path = get_config_path(candidate)
|
||||
break
|
||||
except FileNotFoundError:
|
||||
continue
|
||||
|
||||
if not resolved_path:
|
||||
resolved_path = get_config_path(AGENT_CONFIG_FILENAME, ROOT_DIR / AGENT_CONFIG_FILENAME)
|
||||
|
||||
try:
|
||||
with open(resolved_path, encoding="utf-8") as f:
|
||||
data = load_jsonc(f)
|
||||
if isinstance(data, dict) and "agent" in data:
|
||||
return AgentGlobalConfig.model_validate(data["agent"])
|
||||
return AgentGlobalConfig.model_validate(data)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load agent config from '{resolved_path}': {e}")
|
||||
raise ValueError(f"Failed to parse '{resolved_path}': {e}") from e
|
||||
@@ -0,0 +1,142 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Centralized constants, environment variable names, and type definitions for ARTEMIS."""
|
||||
|
||||
from typing import Literal
|
||||
|
||||
# ==============================================================================
|
||||
# Environment Variable Keys
|
||||
# ==============================================================================
|
||||
|
||||
# Application & Workspace Paths
|
||||
ENV_ARTEMIS_APP_DIR = "ARTEMIS_APP_DIR"
|
||||
ENV_ANTIGRAVITY_APP_DIR = "ANTIGRAVITY_APP_DIR"
|
||||
ENV_ARTEMIS_USE_USER_DIR = "ARTEMIS_USE_USER_DIR"
|
||||
ENV_ARTEMIS_TRACES_DIR = "ARTEMIS_TRACES_DIR"
|
||||
ENV_DATA_ENGINE_DB_PATH = "DATA_ENGINE_DB_PATH"
|
||||
|
||||
# Cloud Brain & Distributed Execution
|
||||
ENV_ARTEMIS_CLOUD_MODE = "ARTEMIS_CLOUD_MODE"
|
||||
ENV_ARTEMIS_CLOUD_SESSION_ID = "ARTEMIS_CLOUD_SESSION_ID"
|
||||
ENV_ARTEMIS_CLOUD_GATEWAY_URL = "ARTEMIS_CLOUD_GATEWAY_URL"
|
||||
ENV_ARTEMIS_TENANT_ID = "ARTEMIS_TENANT_ID"
|
||||
ENV_ARTEMIS_TENANT_TOKEN = "ARTEMIS_TENANT_TOKEN"
|
||||
ENV_ARTEMIS_EDGE_PORT = "ARTEMIS_EDGE_PORT"
|
||||
|
||||
# Runtime State & IPC
|
||||
ENV_ARTEMIS_IPC_PORT = "ARTEMIS_IPC_PORT"
|
||||
ENV_ANTIGRAVITY_LS_ADDRESS = "ANTIGRAVITY_LS_ADDRESS"
|
||||
ENV_ARTEMIS_MCP_SERVER = "ARTEMIS_MCP_SERVER"
|
||||
|
||||
# Device & ADB
|
||||
ENV_ADB_DEVICE_SERIAL = "ADB_DEVICE_SERIAL"
|
||||
ENV_ARTEMIS_DEVICE_ID = "ARTEMIS_DEVICE_ID"
|
||||
ENV_ADB_HOST = "ADB_HOST"
|
||||
ENV_ADB_PORT = "ADB_PORT"
|
||||
ENV_ADB_SERVER_SOCKET = "ADB_SERVER_SOCKET"
|
||||
|
||||
# LLM & Vision OCR API Keys
|
||||
ENV_GOOGLE_API_KEY = "GOOGLE_API_KEY"
|
||||
ENV_GEMINI_API_KEY = "GEMINI_API_KEY"
|
||||
ENV_GCP_API_KEY = "GCP_API_KEY"
|
||||
ENV_OCR_API_KEY = "OCR_API_KEY"
|
||||
ENV_VISION_API_KEY = "VISION_API_KEY"
|
||||
ENV_API_KEY = "API_KEY"
|
||||
ENV_OPENAI_API_KEY = "OPENAI_API_KEY"
|
||||
ENV_OPENAI_BASE_URL = "OPENAI_BASE_URL"
|
||||
ENV_ANTHROPIC_API_KEY = "ANTHROPIC_API_KEY"
|
||||
ENV_OPEN_ROUTER_API_KEY = "OPEN_ROUTER_API_KEY"
|
||||
ENV_XAI_API_KEY = "XAI_API_KEY"
|
||||
|
||||
# Explorer & Agent Defaults
|
||||
ENV_ARTEMIS_EXPLORER_VERSION = "ARTEMIS_EXPLORER_VERSION"
|
||||
ENV_ARTEMIS_DEFAULT_PROFILE = "ARTEMIS_DEFAULT_PROFILE"
|
||||
ENV_ARTEMIS_DEFAULT_MODEL = "ARTEMIS_DEFAULT_MODEL"
|
||||
ENV_ARTEMIS_USE_FILE_API = "ARTEMIS_USE_FILE_API"
|
||||
|
||||
# Debugging & Output Paths
|
||||
ENV_KEEP_VIDEOS = "KEEP_VIDEOS"
|
||||
ENV_ARTEMIS_DEBUG = "ARTEMIS_DEBUG"
|
||||
ENV_EVENTS_OUTPUT_PATH = "EVENTS_OUTPUT_PATH"
|
||||
ENV_RESULTS_OUTPUT_PATH = "RESULTS_OUTPUT_PATH"
|
||||
|
||||
|
||||
# ==============================================================================
|
||||
# Configuration Filenames
|
||||
# ==============================================================================
|
||||
|
||||
ARTEMIS_CONFIG_FILENAME = "artemis.jsonc"
|
||||
LLM_CONFIG_FILENAME = "llm-config.json"
|
||||
LLM_CONFIG_OVERRIDE_FILENAME = "llm-config.override.jsonc"
|
||||
AGENT_CONFIG_FILENAME = "agent_config.json"
|
||||
DATA_ENGINE_DB_FILENAME = "data_engine.db"
|
||||
IPC_PORT_FILENAME = ".artemis_ipc_port"
|
||||
LS_ADDRESS_FILENAME = ".jetski_ls_address"
|
||||
DOTENV_FILENAME = ".env"
|
||||
|
||||
|
||||
# ==============================================================================
|
||||
# Default Values & Network Ports
|
||||
# ==============================================================================
|
||||
|
||||
DEFAULT_ADB_HOST = "127.0.0.1"
|
||||
DEFAULT_ADB_PORT = 5037
|
||||
DEFAULT_GATEWAY_PORT = 8000
|
||||
DEFAULT_EDGE_MONITOR_PORT = 8000
|
||||
DEFAULT_ACTION_TIMEOUT = 30.0
|
||||
DEFAULT_STREAM_PING_INTERVAL = 15.0
|
||||
|
||||
DEFAULT_PROFILE = "pro"
|
||||
DEFAULT_MODEL = "gemini-2.5-flash"
|
||||
DEFAULT_EXPLORER_VERSION: Literal["flash", "pro", "ultra"] = "flash"
|
||||
|
||||
|
||||
# ==============================================================================
|
||||
# Literals and Type Aliases
|
||||
# ==============================================================================
|
||||
|
||||
LLMProvider = Literal[
|
||||
"openai", "google", "openrouter", "xai", "vertexai", "anthropic", "ollama", "vllm", "custom"
|
||||
]
|
||||
ExplorerVersion = Literal["flash", "pro", "ultra"]
|
||||
|
||||
LLMUtilsNode = Literal[
|
||||
"outputter",
|
||||
"hopper",
|
||||
"video_analyzer",
|
||||
"object_detector",
|
||||
]
|
||||
LLMUtilsNodeWithFallback = LLMUtilsNode
|
||||
|
||||
AgentNode = Literal[
|
||||
"planner",
|
||||
"summarizer",
|
||||
"operator",
|
||||
"operator_summarizer",
|
||||
"log_reader_sub_agent",
|
||||
"log_analyzer",
|
||||
"diagnoser",
|
||||
"validator_failure_analyzer",
|
||||
"checker",
|
||||
"planner_avatar",
|
||||
"history_analyzer_expert",
|
||||
"diagnoser_expert",
|
||||
"explorer",
|
||||
"history_analyzer",
|
||||
"validator_pixel_safety_net",
|
||||
"validator",
|
||||
"output_analyzer",
|
||||
]
|
||||
AgentNodeWithFallback = AgentNode
|
||||
@@ -0,0 +1,364 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""LLM provider, model hierarchy, fallback chaining, and configuration loaders."""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal
|
||||
|
||||
import google.auth
|
||||
from google.auth.exceptions import DefaultCredentialsError
|
||||
from pydantic import BaseModel, ValidationError
|
||||
|
||||
from artemis.config.constants import (
|
||||
LLM_CONFIG_FILENAME,
|
||||
AgentNode,
|
||||
LLMProvider,
|
||||
LLMUtilsNode,
|
||||
)
|
||||
from artemis.config.paths import ROOT_DIR, get_config_path
|
||||
from artemis.config.settings import settings
|
||||
from artemis.utils.file import load_jsonc
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def validate_vertex_ai_credentials() -> None:
|
||||
"""Validate Google Application Default Credentials for VertexAI provider."""
|
||||
try:
|
||||
_, project = google.auth.default()
|
||||
if not project:
|
||||
raise Exception("VertexAI requires a Google Cloud project to be set.")
|
||||
except DefaultCredentialsError as e:
|
||||
raise Exception(
|
||||
f"VertexAI requires valid Google Application Default Credentials (ADC): {e}"
|
||||
)
|
||||
|
||||
|
||||
class _CyFunctionDetectorMeta(type):
|
||||
def __instancecheck__(self, instance):
|
||||
name = type(instance).__name__
|
||||
return (
|
||||
name
|
||||
in (
|
||||
"cyfunction",
|
||||
"cython_function_or_method",
|
||||
"builtin_function_or_method",
|
||||
)
|
||||
or "cyfunction" in name.lower()
|
||||
)
|
||||
|
||||
|
||||
class CyFunctionDetector(metaclass=_CyFunctionDetectorMeta):
|
||||
"""Utility type detector for Cython and C-extension functions in Pydantic."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class LLM(BaseModel):
|
||||
"""Base model representing an LLM model provider and runtime parameters."""
|
||||
|
||||
model_config = {"ignored_types": (CyFunctionDetector,)}
|
||||
provider: LLMProvider
|
||||
model: str
|
||||
temperature: float | None = None
|
||||
thinking_budget: int | None = None
|
||||
thinking_level: Literal["minimal", "low", "medium", "high"] | None = None
|
||||
reasoning_effort: Literal["none", "low", "medium", "high"] | None = None
|
||||
include_thoughts: bool | None = None
|
||||
enable_grounding: bool | None = None
|
||||
|
||||
def validate_provider(self, name: str) -> None:
|
||||
"""Ensure the required API key or credentials exist in settings for this provider."""
|
||||
if self.provider == "openai":
|
||||
if not settings.OPENAI_API_KEY:
|
||||
raise Exception(f"{name} requires OPENAI_API_KEY in .env")
|
||||
elif self.provider == "google":
|
||||
if not settings.GOOGLE_API_KEY:
|
||||
raise Exception(f"{name} requires GOOGLE_API_KEY in .env")
|
||||
elif self.provider == "vertexai":
|
||||
validate_vertex_ai_credentials()
|
||||
elif self.provider == "anthropic":
|
||||
if not (settings.ANTHROPIC_API_KEY or os.environ.get("ANTHROPIC_API_KEY")):
|
||||
raise Exception(f"{name} requires ANTHROPIC_API_KEY in .env")
|
||||
elif self.provider == "openrouter":
|
||||
if not settings.OPEN_ROUTER_API_KEY:
|
||||
raise Exception(f"{name} requires OPEN_ROUTER_API_KEY in .env")
|
||||
elif self.provider == "xai":
|
||||
if not settings.XAI_API_KEY:
|
||||
raise Exception(f"{name} requires XAI_API_KEY in .env")
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f"{self.provider}/{self.model}"
|
||||
|
||||
|
||||
class LLMWithFallback(LLM):
|
||||
"""LLM configuration with automatic secondary fallback and timeout specs."""
|
||||
|
||||
fallback: LLM
|
||||
fix_model: str | None = None
|
||||
timeout: float | None = None
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f"{self.provider}/{self.model} (fallback: {self.fallback})"
|
||||
|
||||
|
||||
class LLMConfigUtils(BaseModel):
|
||||
"""Configuration container for auxiliary utility agents/nodes."""
|
||||
|
||||
model_config = {"ignored_types": (CyFunctionDetector,)}
|
||||
outputter: LLMWithFallback
|
||||
hopper: LLMWithFallback
|
||||
video_analyzer: LLMWithFallback | None = None
|
||||
object_detector: LLMWithFallback | None = None
|
||||
|
||||
|
||||
class LLMConfig(BaseModel):
|
||||
"""Comprehensive LLM configuration mapping every node to primary/fallback models."""
|
||||
|
||||
model_config = {"ignored_types": (CyFunctionDetector,)}
|
||||
planner: LLMWithFallback
|
||||
utils: LLMConfigUtils
|
||||
summarizer: LLMWithFallback
|
||||
operator: LLMWithFallback
|
||||
operator_summarizer: LLMWithFallback
|
||||
log_reader_sub_agent: LLMWithFallback
|
||||
log_analyzer: LLMWithFallback
|
||||
diagnoser: LLMWithFallback
|
||||
validator_failure_analyzer: LLMWithFallback
|
||||
checker: LLMWithFallback
|
||||
planner_avatar: LLMWithFallback
|
||||
history_analyzer_expert: LLMWithFallback
|
||||
diagnoser_expert: LLMWithFallback
|
||||
explorer: LLMWithFallback
|
||||
history_analyzer: LLMWithFallback | None = None
|
||||
validator_pixel_safety_net: LLMWithFallback | None = None
|
||||
output_analyzer: LLMWithFallback | None = None
|
||||
|
||||
def validate_providers(self) -> None:
|
||||
"""Validate credentials across all configured agent nodes."""
|
||||
self.planner.validate_provider("Planner")
|
||||
self.utils.outputter.validate_provider("Outputter")
|
||||
self.utils.hopper.validate_provider("Hopper")
|
||||
if self.utils.video_analyzer:
|
||||
self.utils.video_analyzer.validate_provider("VideoAnalyzer")
|
||||
if self.utils.object_detector:
|
||||
self.utils.object_detector.validate_provider("ObjectDetector")
|
||||
self.summarizer.validate_provider("Summarizer")
|
||||
self.operator.validate_provider("Operator")
|
||||
self.operator_summarizer.validate_provider("OperatorSummarizer")
|
||||
self.log_reader_sub_agent.validate_provider("LogReaderSubAgent")
|
||||
self.log_analyzer.validate_provider("LogAnalyzer")
|
||||
self.diagnoser.validate_provider("Diagnoser")
|
||||
self.validator_failure_analyzer.validate_provider("ValidatorFailureAnalyzer")
|
||||
self.checker.validate_provider("Checker")
|
||||
self.planner_avatar.validate_provider("PlannerAvatar")
|
||||
self.history_analyzer_expert.validate_provider("HistoryAnalyzerExpert")
|
||||
self.diagnoser_expert.validate_provider("DiagnoserExpert")
|
||||
self.explorer.validate_provider("Explorer")
|
||||
if self.history_analyzer:
|
||||
self.history_analyzer.validate_provider("HistoryAnalyzer")
|
||||
if self.validator_pixel_safety_net:
|
||||
self.validator_pixel_safety_net.validate_provider("ValidatorPixelSafetyNet")
|
||||
if self.output_analyzer:
|
||||
self.output_analyzer.validate_provider("OutputAnalyzer")
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f"""
|
||||
📃 Planner: {self.planner}
|
||||
🧩 Utils:
|
||||
🔽 Hopper: {self.utils.hopper}
|
||||
📝 Outputter: {self.utils.outputter}
|
||||
🎬 Video Analyzer: {self.utils.video_analyzer or "Not configured"}
|
||||
👁️ Object Detector: {self.utils.object_detector or "Not configured"}
|
||||
"""
|
||||
|
||||
def get_agent(self, item: AgentNode) -> LLMWithFallback:
|
||||
"""Retrieve model configuration for a specific agent node with sensible defaults."""
|
||||
val = getattr(self, item)
|
||||
if val is None:
|
||||
if item == "history_analyzer":
|
||||
return self.operator
|
||||
elif item == "validator_pixel_safety_net":
|
||||
return LLMWithFallback(
|
||||
provider="google",
|
||||
model="gemini-3.5-flash-lite",
|
||||
temperature=0.0,
|
||||
fallback=LLM(
|
||||
provider="google",
|
||||
model="gemini-3.1-flash-lite",
|
||||
temperature=0.0,
|
||||
),
|
||||
)
|
||||
elif item == "output_analyzer":
|
||||
return self.log_analyzer
|
||||
return val
|
||||
|
||||
def get_utils(self, item: LLMUtilsNode) -> LLMWithFallback:
|
||||
"""Retrieve model configuration for a specific utility node."""
|
||||
value = getattr(self.utils, item)
|
||||
if value is None:
|
||||
raise ValueError(
|
||||
f"Utils '{item}' is not configured. Please add it to your LLM "
|
||||
"config or enable it via AgentConfigBuilder."
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def _expand_default_into_nodes(config_dict: dict) -> dict:
|
||||
"""Expand unified config format with 'default' and 'nodes' into full LLMConfig schema."""
|
||||
if "planner" in config_dict and "utils" in config_dict:
|
||||
return config_dict
|
||||
|
||||
default_model_cfg = config_dict.get(
|
||||
"default",
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-3.6-flash",
|
||||
"fallback": {
|
||||
"provider": "google",
|
||||
"model": "gemini-3.5-flash",
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
nodes_override = config_dict.get("nodes", {})
|
||||
|
||||
all_agent_nodes = [
|
||||
"planner",
|
||||
"summarizer",
|
||||
"operator",
|
||||
"operator_summarizer",
|
||||
"log_reader_sub_agent",
|
||||
"log_analyzer",
|
||||
"diagnoser",
|
||||
"validator_failure_analyzer",
|
||||
"checker",
|
||||
"planner_avatar",
|
||||
"history_analyzer_expert",
|
||||
"diagnoser_expert",
|
||||
"explorer",
|
||||
]
|
||||
|
||||
all_utils_nodes = [
|
||||
"outputter",
|
||||
"hopper",
|
||||
"video_analyzer",
|
||||
"object_detector",
|
||||
]
|
||||
|
||||
result: dict[str, Any] = {}
|
||||
for node in all_agent_nodes:
|
||||
node_cfg = dict(default_model_cfg)
|
||||
if node in nodes_override:
|
||||
for k, v in nodes_override[node].items():
|
||||
if isinstance(v, dict) and isinstance(node_cfg.get(k), dict):
|
||||
node_cfg[k] = {**node_cfg[k], **v}
|
||||
else:
|
||||
node_cfg[k] = v
|
||||
result[node] = node_cfg
|
||||
|
||||
utils_dict: dict[str, Any] = {}
|
||||
for util in all_utils_nodes:
|
||||
util_cfg = dict(default_model_cfg)
|
||||
if util in nodes_override:
|
||||
for k, v in nodes_override[util].items():
|
||||
if isinstance(v, dict) and isinstance(util_cfg.get(k), dict):
|
||||
util_cfg[k] = {**util_cfg[k], **v}
|
||||
else:
|
||||
util_cfg[k] = v
|
||||
utils_dict[util] = util_cfg
|
||||
result["utils"] = utils_dict
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def parse_llm_config() -> LLMConfig:
|
||||
"""Parse and instantiate LLMConfig from artemis.jsonc or llm-config.json."""
|
||||
config_path = None
|
||||
for candidate in ("artemis.jsonc", "artemis.json", LLM_CONFIG_FILENAME):
|
||||
try:
|
||||
config_path = get_config_path(candidate)
|
||||
break
|
||||
except FileNotFoundError:
|
||||
continue
|
||||
|
||||
if not config_path:
|
||||
config_path = get_config_path(LLM_CONFIG_FILENAME, ROOT_DIR / LLM_CONFIG_FILENAME)
|
||||
|
||||
try:
|
||||
with open(config_path, encoding="utf-8") as f:
|
||||
config_dict = load_jsonc(f)
|
||||
expanded_dict = _expand_default_into_nodes(config_dict)
|
||||
return LLMConfig.model_validate(expanded_dict)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load or parse llm config: {config_path}. Error: {e}")
|
||||
raise
|
||||
|
||||
|
||||
def initialize_llm_config() -> LLMConfig:
|
||||
"""Parse and validate credentials for LLMConfig."""
|
||||
llm_config = parse_llm_config()
|
||||
llm_config.validate_providers()
|
||||
logger.success("LLM config initialized")
|
||||
return llm_config
|
||||
|
||||
|
||||
def get_default_llm_config() -> LLMConfig:
|
||||
"""Returns default LLMConfig parsed from standard configuration file."""
|
||||
return parse_llm_config()
|
||||
|
||||
|
||||
def deep_merge_llm_config(base: LLMConfig, overrides: dict) -> LLMConfig:
|
||||
"""Recursively merge dictionary overrides into an existing LLMConfig object."""
|
||||
base_dict = base.model_dump()
|
||||
|
||||
def merge(d1: dict, d2: dict) -> None:
|
||||
for k, v in d2.items():
|
||||
if k in d1 and isinstance(d1[k], dict) and isinstance(v, dict):
|
||||
merge(d1[k], v)
|
||||
else:
|
||||
d1[k] = v
|
||||
|
||||
merge(base_dict, overrides)
|
||||
return LLMConfig.model_validate(base_dict)
|
||||
|
||||
|
||||
def load_llm_config_override(path: Path | str) -> LLMConfig:
|
||||
"""Load custom LLM configuration JSON/JSONC overrides on top of default configuration."""
|
||||
default_config = get_default_llm_config()
|
||||
|
||||
resolved_path = Path(path)
|
||||
if not resolved_path.exists():
|
||||
try:
|
||||
resolved_path = get_config_path(str(path))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
override_config_dict = {}
|
||||
if resolved_path.exists():
|
||||
logger.info(f"Loading custom LLM config from {resolved_path.resolve()}...")
|
||||
with open(resolved_path, encoding="utf-8") as f:
|
||||
override_config_dict = load_jsonc(f)
|
||||
else:
|
||||
logger.warning(f"Custom LLM config not found at {path} - using default config")
|
||||
|
||||
try:
|
||||
return deep_merge_llm_config(default_config, override_config_dict)
|
||||
except ValidationError as e:
|
||||
logger.error(f"Invalid LLM config: {e}")
|
||||
logger.info("Falling back to default config")
|
||||
return default_config
|
||||
@@ -0,0 +1,159 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Structured output schemas, event recording, and output file management."""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Annotated, Any
|
||||
import warnings
|
||||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from artemis.config.constants import (
|
||||
ENV_EVENTS_OUTPUT_PATH,
|
||||
ENV_RESULTS_OUTPUT_PATH,
|
||||
)
|
||||
from artemis.config.llm import CyFunctionDetector
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class OutputConfig(BaseModel):
|
||||
"""Configuration for structured output formatting and schemas."""
|
||||
|
||||
model_config = {"ignored_types": (CyFunctionDetector,)}
|
||||
structured_output: Annotated[
|
||||
type[BaseModel] | dict | None,
|
||||
Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Optional structured schema (as a BaseModel or dict) to shape the output. "
|
||||
"If provided, it takes precedence over 'output_description'."
|
||||
),
|
||||
),
|
||||
]
|
||||
output_description: Annotated[
|
||||
str | None,
|
||||
Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Optional natural language description of the expected output format. "
|
||||
"Used only if 'structured_output' is not provided. "
|
||||
"Example: 'Output a JSON with 3 keys: color, price, websiteUrl'."
|
||||
),
|
||||
),
|
||||
]
|
||||
enable_outputter: Annotated[
|
||||
bool | None,
|
||||
Field(
|
||||
default=None,
|
||||
description="Explicitly enable or disable Outputter post-execution synthesis.",
|
||||
),
|
||||
] = None
|
||||
force_synthesis: Annotated[
|
||||
bool,
|
||||
Field(
|
||||
default=False,
|
||||
description="Whether to run Outputter synthesis even without explicit output_description or schema.",
|
||||
),
|
||||
] = False
|
||||
|
||||
def __str__(self) -> str:
|
||||
s_builder = ""
|
||||
if self.structured_output:
|
||||
s_builder += f"Structured Output: {self.structured_output}\n"
|
||||
if self.output_description:
|
||||
s_builder += f"Output Description: {self.output_description}\n"
|
||||
if self.output_description and self.structured_output:
|
||||
s_builder += (
|
||||
"Both 'structured_output' and 'output_description' are provided. "
|
||||
"'structured_output' will take precedence.\n"
|
||||
)
|
||||
return s_builder
|
||||
|
||||
@model_validator(mode="after")
|
||||
def warn_if_both_outputs_provided(self) -> "OutputConfig":
|
||||
if self.structured_output and self.output_description:
|
||||
warnings.warn(
|
||||
"Both 'structured_output' and 'output_description' are provided. "
|
||||
"'structured_output' will take precedence.",
|
||||
stacklevel=2,
|
||||
)
|
||||
return self
|
||||
|
||||
def needs_structured_format(self, default_enabled: bool = True) -> bool:
|
||||
"""Returns True if Outputter should run to synthesize/structure the output."""
|
||||
if self.enable_outputter is False:
|
||||
return False
|
||||
if self.enable_outputter is True or self.force_synthesis:
|
||||
return True
|
||||
if not default_enabled:
|
||||
return False
|
||||
return bool(self.structured_output or self.output_description)
|
||||
|
||||
|
||||
def prepare_output_files() -> tuple[str | None, str | None]:
|
||||
"""Validate and prepare events and results output files from environment variables."""
|
||||
events_output_path = os.getenv(ENV_EVENTS_OUTPUT_PATH)
|
||||
results_output_path = os.getenv(ENV_RESULTS_OUTPUT_PATH)
|
||||
|
||||
def validate_and_prepare_file(file_path: str) -> str | None:
|
||||
if not file_path:
|
||||
return None
|
||||
|
||||
path_obj = Path(file_path)
|
||||
|
||||
if path_obj.exists() and path_obj.is_dir():
|
||||
logger.error(f"Error: Path '{file_path}' points to an existing directory, not a file.")
|
||||
return None
|
||||
|
||||
if not path_obj.suffix or file_path.endswith(("/", "\\")):
|
||||
logger.error(f"Error: Path '{file_path}' appears to be a directory path, not a file.")
|
||||
return None
|
||||
|
||||
try:
|
||||
path_obj.parent.mkdir(parents=True, exist_ok=True)
|
||||
path_obj.touch(exist_ok=True)
|
||||
return file_path
|
||||
except OSError as e:
|
||||
logger.error(f"Error creating file '{file_path}': {e}")
|
||||
return None
|
||||
|
||||
validated_events_path = (
|
||||
validate_and_prepare_file(events_output_path) if events_output_path else None
|
||||
)
|
||||
validated_results_path = (
|
||||
validate_and_prepare_file(results_output_path) if results_output_path else None
|
||||
)
|
||||
|
||||
return validated_events_path, validated_results_path
|
||||
|
||||
|
||||
def record_events(output_path: Path | None, events: list[str] | BaseModel | Any) -> None:
|
||||
"""Serialize and write execution events to disk."""
|
||||
if not output_path:
|
||||
return
|
||||
|
||||
if isinstance(events, str):
|
||||
events_content = events
|
||||
elif isinstance(events, BaseModel):
|
||||
events_content = events.model_dump_json(indent=2)
|
||||
else:
|
||||
events_content = json.dumps(events, indent=2)
|
||||
|
||||
with open(output_path, "w", encoding="utf-8") as f:
|
||||
f.write(events_content)
|
||||
@@ -0,0 +1,170 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Centralized workspace, configuration, temporary files, and directory management."""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
from artemis.config.constants import (
|
||||
DATA_ENGINE_DB_FILENAME,
|
||||
ENV_ANTIGRAVITY_APP_DIR,
|
||||
ENV_ARTEMIS_APP_DIR,
|
||||
ENV_ARTEMIS_TRACES_DIR,
|
||||
ENV_ARTEMIS_USE_USER_DIR,
|
||||
IPC_PORT_FILENAME,
|
||||
LS_ADDRESS_FILENAME,
|
||||
)
|
||||
|
||||
# Project root directory (3 levels up from artemis/config/paths.py)
|
||||
ROOT_DIR = Path(__file__).resolve().parent.parent.parent
|
||||
CONFIG_DIR = ROOT_DIR / "config"
|
||||
|
||||
|
||||
def is_frozen_bundle() -> bool:
|
||||
"""Check if running inside a PyInstaller / Nuitka standalone compiled binary."""
|
||||
return getattr(sys, "frozen", False) and hasattr(sys, "_MEIPASS")
|
||||
|
||||
|
||||
def get_app_dir() -> Path:
|
||||
"""Returns the central application directory for ARTEMIS configurations, DBs, and traces.
|
||||
|
||||
Order of precedence:
|
||||
1. ARTEMIS_APP_DIR or ANTIGRAVITY_APP_DIR environment variable.
|
||||
2. Legacy ~/.gemini/jetski directory if it exists.
|
||||
3. Default ~/.artemis directory.
|
||||
"""
|
||||
env_dir = os.getenv(ENV_ARTEMIS_APP_DIR) or os.getenv(ENV_ANTIGRAVITY_APP_DIR)
|
||||
if env_dir:
|
||||
path = Path(env_dir)
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return path
|
||||
|
||||
legacy_dir = Path(os.path.expanduser("~/.gemini/jetski"))
|
||||
if legacy_dir.exists():
|
||||
return legacy_dir
|
||||
|
||||
default_dir = Path(os.path.expanduser("~/.artemis"))
|
||||
default_dir.mkdir(parents=True, exist_ok=True)
|
||||
return default_dir
|
||||
|
||||
|
||||
def get_default_traces_path() -> Path:
|
||||
"""Returns default traces directory.
|
||||
|
||||
Uses user app dir when running as a frozen binary or when app dir env is set.
|
||||
"""
|
||||
env_traces = os.getenv(ENV_ARTEMIS_TRACES_DIR)
|
||||
if env_traces:
|
||||
traces_dir = Path(env_traces)
|
||||
elif (
|
||||
is_frozen_bundle()
|
||||
or os.getenv(ENV_ARTEMIS_USE_USER_DIR) == "true"
|
||||
or os.getenv(ENV_ARTEMIS_APP_DIR)
|
||||
or os.getenv(ENV_ANTIGRAVITY_APP_DIR)
|
||||
):
|
||||
traces_dir = get_app_dir() / "traces"
|
||||
else:
|
||||
traces_dir = ROOT_DIR / "traces"
|
||||
|
||||
traces_dir.mkdir(parents=True, exist_ok=True)
|
||||
return traces_dir
|
||||
|
||||
|
||||
def get_traces_dir() -> Path:
|
||||
"""Alias for get_default_traces_path."""
|
||||
return get_default_traces_path()
|
||||
|
||||
|
||||
def get_temp_dir(subfolder: str | None = None) -> Path:
|
||||
"""Returns a unified temporary directory for ARTEMIS runtime operations (e.g., screenshots, recordings).
|
||||
|
||||
Args:
|
||||
subfolder: Optional subfolder name under the temp directory.
|
||||
|
||||
Returns:
|
||||
Path object pointing to the initialized temporary directory.
|
||||
"""
|
||||
base_temp = get_app_dir() / "temp"
|
||||
target_dir = base_temp / subfolder if subfolder else base_temp
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
return target_dir
|
||||
|
||||
|
||||
def get_data_engine_db_path() -> Path:
|
||||
"""Returns the unified SQLite database path for ARTEMIS DataEngine."""
|
||||
return get_default_traces_path() / DATA_ENGINE_DB_FILENAME
|
||||
|
||||
|
||||
def get_ipc_port_file() -> Path:
|
||||
"""Returns the location of the IPC port synchronization file."""
|
||||
app_dir_file = get_app_dir() / IPC_PORT_FILENAME
|
||||
if app_dir_file.exists():
|
||||
return app_dir_file
|
||||
return ROOT_DIR / IPC_PORT_FILENAME
|
||||
|
||||
|
||||
def get_ls_address_file() -> Path:
|
||||
"""Returns the location of the Language Server address synchronization file."""
|
||||
app_dir_file = get_app_dir() / LS_ADDRESS_FILENAME
|
||||
if app_dir_file.exists():
|
||||
return app_dir_file
|
||||
return ROOT_DIR / LS_ADDRESS_FILENAME
|
||||
|
||||
|
||||
def get_config_path(filename: str, default_bundled_path: Path | None = None) -> Path:
|
||||
"""Resolve a configuration file path across env vars, config dir, project dir, app dir, and bundles.
|
||||
|
||||
Resolution Order:
|
||||
1. Environment variable override: `ARTEMIS_<FILENAME_UPPER>`
|
||||
2. Central `config/` directory: `<ROOT_DIR>/config/<filename>`
|
||||
3. Project root directory: `<ROOT_DIR>/<filename>`
|
||||
4. Custom default bundled path (if provided and exists)
|
||||
5. User application directory: `<APP_DIR>/<filename>`
|
||||
"""
|
||||
env_var = f"ARTEMIS_{filename.upper().replace('.', '_').replace('-', '_')}"
|
||||
env_path_str = os.getenv(env_var)
|
||||
if env_path_str:
|
||||
env_path = Path(env_path_str)
|
||||
if env_path.exists():
|
||||
return env_path
|
||||
|
||||
# Check config/ directory
|
||||
config_dir_path = CONFIG_DIR / filename
|
||||
if config_dir_path.exists():
|
||||
return config_dir_path
|
||||
|
||||
# Check project root directory
|
||||
project_path = ROOT_DIR / filename
|
||||
if project_path.exists():
|
||||
return project_path
|
||||
|
||||
# Check bundled fallback
|
||||
if default_bundled_path and default_bundled_path.exists():
|
||||
return default_bundled_path
|
||||
|
||||
# Check user app directory
|
||||
app_dir_path = get_app_dir() / filename
|
||||
if app_dir_path.exists():
|
||||
return app_dir_path
|
||||
|
||||
raise FileNotFoundError(
|
||||
f"Configuration file '{filename}' not found in config dir ({config_dir_path}), "
|
||||
f"project root ({project_path}), or app dir ({app_dir_path})."
|
||||
)
|
||||
|
||||
|
||||
GLOBAL_APP_DIR = get_app_dir()
|
||||
GLOBAL_JETSKI_DIR = GLOBAL_APP_DIR # Backward-compatibility alias
|
||||
@@ -0,0 +1,142 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Runtime state coordination, temporary file lifecycles, and IPC synchronization."""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
import time
|
||||
|
||||
from artemis.config.constants import (
|
||||
ENV_ANTIGRAVITY_LS_ADDRESS,
|
||||
ENV_ARTEMIS_IPC_PORT,
|
||||
)
|
||||
from artemis.config.paths import (
|
||||
get_ipc_port_file,
|
||||
get_ls_address_file,
|
||||
get_temp_dir,
|
||||
)
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def read_ipc_port() -> int | None:
|
||||
"""Read the active ARTEMIS IPC port from environment or temporary state file."""
|
||||
env_port = os.getenv(ENV_ARTEMIS_IPC_PORT)
|
||||
if env_port and env_port.strip().isdigit():
|
||||
return int(env_port.strip())
|
||||
|
||||
port_file = get_ipc_port_file()
|
||||
if port_file.exists():
|
||||
try:
|
||||
content = port_file.read_text(encoding="utf-8").strip()
|
||||
if content.isdigit():
|
||||
return int(content)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to read IPC port from {port_file}: {e}")
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def write_ipc_port(port: int) -> Path:
|
||||
"""Save active IPC port to environment and temporary synchronization file."""
|
||||
os.environ[ENV_ARTEMIS_IPC_PORT] = str(port)
|
||||
port_file = get_ipc_port_file()
|
||||
try:
|
||||
port_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
port_file.write_text(str(port), encoding="utf-8")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to write IPC port file to {port_file}: {e}")
|
||||
return port_file
|
||||
|
||||
|
||||
def clear_ipc_port() -> None:
|
||||
"""Remove IPC port synchronization state file."""
|
||||
port_file = get_ipc_port_file()
|
||||
if port_file.exists():
|
||||
try:
|
||||
port_file.unlink(missing_ok=True)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to remove IPC port file {port_file}: {e}")
|
||||
|
||||
|
||||
def read_ls_address() -> str | None:
|
||||
"""Read the active Language Server address from environment or state file."""
|
||||
env_addr = os.getenv(ENV_ANTIGRAVITY_LS_ADDRESS)
|
||||
if env_addr and env_addr.strip():
|
||||
return env_addr.strip()
|
||||
|
||||
addr_file = get_ls_address_file()
|
||||
if addr_file.exists():
|
||||
try:
|
||||
content = addr_file.read_text(encoding="utf-8").strip()
|
||||
if content:
|
||||
return content
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to read LS address from {addr_file}: {e}")
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def write_ls_address(address: str) -> Path:
|
||||
"""Save Language Server address to environment and temporary synchronization file."""
|
||||
clean_addr = address.strip()
|
||||
os.environ[ENV_ANTIGRAVITY_LS_ADDRESS] = clean_addr
|
||||
addr_file = get_ls_address_file()
|
||||
try:
|
||||
addr_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
addr_file.write_text(clean_addr, encoding="utf-8")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to write LS address file to {addr_file}: {e}")
|
||||
return addr_file
|
||||
|
||||
|
||||
def clear_ls_address() -> None:
|
||||
"""Remove Language Server address synchronization state file."""
|
||||
addr_file = get_ls_address_file()
|
||||
if addr_file.exists():
|
||||
try:
|
||||
addr_file.unlink(missing_ok=True)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to remove LS address file {addr_file}: {e}")
|
||||
|
||||
|
||||
def cleanup_temp_dir(subfolder: str | None = None, max_age_seconds: float | None = None) -> int:
|
||||
"""Clean up expired temporary runtime files in the central temp directory.
|
||||
|
||||
Args:
|
||||
subfolder: Optional subfolder within the temp directory.
|
||||
max_age_seconds: Maximum age in seconds before a file is deleted. If None, all files are purged.
|
||||
|
||||
Returns:
|
||||
Number of files successfully deleted.
|
||||
"""
|
||||
temp_dir = get_temp_dir(subfolder)
|
||||
if not temp_dir.exists():
|
||||
return 0
|
||||
|
||||
now = time.time()
|
||||
deleted_count = 0
|
||||
|
||||
for file_path in temp_dir.glob("*"):
|
||||
if file_path.is_file():
|
||||
try:
|
||||
if max_age_seconds is None or (now - file_path.stat().st_mtime) > max_age_seconds:
|
||||
file_path.unlink(missing_ok=True)
|
||||
deleted_count += 1
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not delete temporary file {file_path}: {e}")
|
||||
|
||||
return deleted_count
|
||||
@@ -0,0 +1,199 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Application settings and environment variable configuration management."""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from pydantic import Field, SecretStr, model_validator
|
||||
from pydantic_settings import BaseSettings
|
||||
|
||||
from artemis.config.constants import (
|
||||
DEFAULT_ADB_HOST,
|
||||
DEFAULT_ADB_PORT,
|
||||
DEFAULT_EXPLORER_VERSION,
|
||||
DEFAULT_MODEL,
|
||||
DEFAULT_PROFILE,
|
||||
ENV_DATA_ENGINE_DB_PATH,
|
||||
ENV_GCP_API_KEY,
|
||||
ENV_GEMINI_API_KEY,
|
||||
ENV_GOOGLE_API_KEY,
|
||||
ENV_OPEN_ROUTER_API_KEY,
|
||||
ENV_OPENAI_API_KEY,
|
||||
ENV_XAI_API_KEY,
|
||||
ExplorerVersion,
|
||||
)
|
||||
from artemis.config.paths import (
|
||||
GLOBAL_APP_DIR,
|
||||
get_app_dir,
|
||||
get_data_engine_db_path,
|
||||
get_default_traces_path,
|
||||
get_temp_dir,
|
||||
)
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
# Load environment configuration from app dir if present, else current working dir
|
||||
_global_env = GLOBAL_APP_DIR / ".env"
|
||||
if _global_env.exists():
|
||||
load_dotenv(dotenv_path=_global_env, verbose=True)
|
||||
else:
|
||||
load_dotenv(verbose=True)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
"""Centralized ARTEMIS runtime settings loaded from environment and .env files."""
|
||||
|
||||
# LLM Provider Authentication
|
||||
OPENAI_API_KEY: SecretStr | None = None
|
||||
GOOGLE_API_KEY: SecretStr | None = None
|
||||
GEMINI_API_KEY: SecretStr | None = None
|
||||
GCP_API_KEY: SecretStr | None = None
|
||||
ANTHROPIC_API_KEY: SecretStr | None = None
|
||||
XAI_API_KEY: SecretStr | None = None
|
||||
OPEN_ROUTER_API_KEY: SecretStr | None = None
|
||||
|
||||
# Google Cloud Vision OCR Authentication
|
||||
OCR_API_KEY: SecretStr | None = None
|
||||
VISION_API_KEY: SecretStr | None = None
|
||||
API_KEY: SecretStr | None = None
|
||||
|
||||
# Custom Provider Endpoints
|
||||
OPENAI_BASE_URL: str | None = None
|
||||
|
||||
# Android ADB Connectivity
|
||||
ADB_HOST: str | None = Field(default=DEFAULT_ADB_HOST)
|
||||
ADB_PORT: int | None = Field(default=DEFAULT_ADB_PORT)
|
||||
ADB_DEVICE_SERIAL: str | None = None
|
||||
|
||||
# Execution Defaults
|
||||
PROJECT_NAME: str | None = None
|
||||
ARTEMIS_DEFAULT_PROFILE: str = Field(default=DEFAULT_PROFILE)
|
||||
ARTEMIS_DEFAULT_MODEL: str = Field(default=DEFAULT_MODEL)
|
||||
|
||||
# Explorer Tool Settings
|
||||
EXPLORER_VERSION: ExplorerVersion = Field(
|
||||
default=DEFAULT_EXPLORER_VERSION,
|
||||
description="Active version mode for the UI Explorer tool ('flash', 'pro', or 'ultra')",
|
||||
)
|
||||
EXPLORER_CACHING: bool = Field(
|
||||
default=True,
|
||||
description="Enable context caching when running multi-turn pro/ultra Explorer",
|
||||
)
|
||||
|
||||
# Paths & Storage
|
||||
TRACES_PATH: Path = Field(default_factory=get_default_traces_path)
|
||||
DATA_ENGINE_DB_PATH: Path = Field(default_factory=get_data_engine_db_path)
|
||||
TEMP_PATH: Path = Field(default_factory=get_temp_dir)
|
||||
|
||||
model_config = {"env_file": ".env", "extra": "ignore"}
|
||||
|
||||
@model_validator(mode="after")
|
||||
def fallback_api_keys(self) -> "Settings":
|
||||
"""Normalize Google / Gemini / GCP and OCR API keys for seamless interoperability."""
|
||||
if not self.GOOGLE_API_KEY:
|
||||
if self.GEMINI_API_KEY:
|
||||
self.GOOGLE_API_KEY = self.GEMINI_API_KEY
|
||||
elif self.GCP_API_KEY:
|
||||
self.GOOGLE_API_KEY = self.GCP_API_KEY
|
||||
|
||||
# Fallback for OCR & Vision API keys
|
||||
if not self.OCR_API_KEY:
|
||||
if self.VISION_API_KEY:
|
||||
self.OCR_API_KEY = self.VISION_API_KEY
|
||||
elif self.API_KEY:
|
||||
self.OCR_API_KEY = self.API_KEY
|
||||
if not self.API_KEY and self.OCR_API_KEY:
|
||||
self.API_KEY = self.OCR_API_KEY
|
||||
return self
|
||||
|
||||
def get_api_key(self, provider: str) -> SecretStr | None:
|
||||
"""Get API key for a specified provider.
|
||||
|
||||
Args:
|
||||
provider: Provider name (e.g., 'google', 'gemini', 'openai', 'ocr', 'vision').
|
||||
|
||||
Returns:
|
||||
SecretStr containing the API key or None if not configured.
|
||||
"""
|
||||
provider_lower = provider.lower()
|
||||
if provider_lower in ("google", "gemini", "vertexai"):
|
||||
return self.GOOGLE_API_KEY or self.GEMINI_API_KEY or self.GCP_API_KEY
|
||||
elif provider_lower in ("ocr", "vision", "google_vision"):
|
||||
return self.OCR_API_KEY or self.VISION_API_KEY or self.API_KEY or self.GOOGLE_API_KEY
|
||||
elif provider_lower == "openai":
|
||||
return self.OPENAI_API_KEY
|
||||
elif provider_lower == "anthropic":
|
||||
return self.ANTHROPIC_API_KEY
|
||||
elif provider_lower == "openrouter":
|
||||
return self.OPEN_ROUTER_API_KEY
|
||||
elif provider_lower == "xai":
|
||||
return self.XAI_API_KEY
|
||||
return None
|
||||
|
||||
def set_api_key(self, provider: str, key: str, persist_to_env: bool = False) -> None:
|
||||
"""Dynamically set an API key at runtime, optionally persisting to .env in the app dir.
|
||||
|
||||
Args:
|
||||
provider: Target provider name.
|
||||
key: Secret API key string.
|
||||
persist_to_env: Whether to save the key to the app directory's .env file.
|
||||
"""
|
||||
secret = SecretStr(key)
|
||||
provider_lower = provider.lower()
|
||||
env_key_name = None
|
||||
|
||||
if provider_lower in ("google", "gemini", "vertexai"):
|
||||
self.GOOGLE_API_KEY = secret
|
||||
self.GEMINI_API_KEY = secret
|
||||
self.GCP_API_KEY = secret
|
||||
env_key_name = ENV_GOOGLE_API_KEY
|
||||
os.environ[ENV_GOOGLE_API_KEY] = key
|
||||
os.environ[ENV_GEMINI_API_KEY] = key
|
||||
os.environ[ENV_GCP_API_KEY] = key
|
||||
elif provider_lower == "openai":
|
||||
self.OPENAI_API_KEY = secret
|
||||
env_key_name = ENV_OPENAI_API_KEY
|
||||
os.environ[ENV_OPENAI_API_KEY] = key
|
||||
elif provider_lower == "openrouter":
|
||||
self.OPEN_ROUTER_API_KEY = secret
|
||||
env_key_name = ENV_OPEN_ROUTER_API_KEY
|
||||
os.environ[ENV_OPEN_ROUTER_API_KEY] = key
|
||||
elif provider_lower == "xai":
|
||||
self.XAI_API_KEY = secret
|
||||
env_key_name = ENV_XAI_API_KEY
|
||||
os.environ[ENV_XAI_API_KEY] = key
|
||||
|
||||
if persist_to_env and env_key_name:
|
||||
try:
|
||||
env_file = get_app_dir() / ".env"
|
||||
lines = []
|
||||
if env_file.exists():
|
||||
lines = env_file.read_text(encoding="utf-8").splitlines()
|
||||
lines = [line for line in lines if not line.startswith(f"{env_key_name}=")]
|
||||
lines.append(f"{env_key_name}={key}")
|
||||
env_file.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not persist {env_key_name} to .env: {e}")
|
||||
|
||||
|
||||
# Singleton instance
|
||||
settings = Settings()
|
||||
|
||||
# Synchronize DATA_ENGINE_DB_PATH in environment for external sub-processes/tools
|
||||
if settings.DATA_ENGINE_DB_PATH:
|
||||
os.environ[ENV_DATA_ENGINE_DB_PATH] = str(settings.DATA_ENGINE_DB_PATH)
|
||||
@@ -0,0 +1,50 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
RECURSION_LIMIT = 400
|
||||
MAX_MESSAGES_IN_HISTORY = 25
|
||||
VALIDATOR_MESSAGES_KEY = "validator_messages"
|
||||
|
||||
# Optimization Constants
|
||||
CHECKER_MAX_ITERATIONS = 20
|
||||
OPERATOR_MAX_CHAT_ROUNDS = 4
|
||||
VALIDATOR_POLL_TIMEOUT = 2.0
|
||||
VALIDATOR_POLL_INTERVAL = 0.2
|
||||
VALIDATOR_OCR_INTERVAL = 1.0
|
||||
VALIDATOR_UI_HIERARCHY_TIMEOUT = 1.0
|
||||
|
||||
from google.genai import types as _genai_types
|
||||
|
||||
SAFETY_SETTINGS_BLOCK_NONE = [
|
||||
_genai_types.SafetySetting(
|
||||
category=_genai_types.HarmCategory.HARM_CATEGORY_HATE_SPEECH,
|
||||
threshold=_genai_types.HarmBlockThreshold.BLOCK_NONE,
|
||||
),
|
||||
_genai_types.SafetySetting(
|
||||
category=_genai_types.HarmCategory.HARM_CATEGORY_HARASSMENT,
|
||||
threshold=_genai_types.HarmBlockThreshold.BLOCK_NONE,
|
||||
),
|
||||
_genai_types.SafetySetting(
|
||||
category=_genai_types.HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT,
|
||||
threshold=_genai_types.HarmBlockThreshold.BLOCK_NONE,
|
||||
),
|
||||
_genai_types.SafetySetting(
|
||||
category=_genai_types.HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT,
|
||||
threshold=_genai_types.HarmBlockThreshold.BLOCK_NONE,
|
||||
),
|
||||
_genai_types.SafetySetting(
|
||||
category=_genai_types.HarmCategory.HARM_CATEGORY_CIVIC_INTEGRITY,
|
||||
threshold=_genai_types.HarmBlockThreshold.BLOCK_NONE,
|
||||
),
|
||||
]
|
||||
@@ -0,0 +1,219 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Context variables for global state management.
|
||||
|
||||
Uses ContextVar to avoid prop drilling and maintain clean function signatures.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
try:
|
||||
from enum import StrEnum
|
||||
except ImportError:
|
||||
from enum import Enum
|
||||
|
||||
class StrEnum(str, Enum):
|
||||
pass
|
||||
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal, TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from artemis.data_engine.engine import DataEngine
|
||||
|
||||
try:
|
||||
from adbutils import AdbClient
|
||||
except ImportError:
|
||||
AdbClient = Any
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, PrivateAttr
|
||||
from artemis.utils.video import detect_video_tools_enabled
|
||||
|
||||
|
||||
from artemis.config import ExplorerConfig, LLMConfig, OutputterConfig
|
||||
|
||||
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class AppLaunchResult(BaseModel):
|
||||
"""Result of initial app launch attempt."""
|
||||
|
||||
model_config = ConfigDict(extra="allow", arbitrary_types_allowed=True)
|
||||
|
||||
locked_app_package: str
|
||||
locked_app_initial_launch_success: bool | None
|
||||
locked_app_initial_launch_error: str | None
|
||||
|
||||
|
||||
class DevicePlatform(StrEnum):
|
||||
"""Mobile device platform enumeration."""
|
||||
|
||||
ANDROID = "android"
|
||||
|
||||
|
||||
class DeviceContext(BaseModel):
|
||||
model_config = ConfigDict(
|
||||
arbitrary_types_allowed=True,
|
||||
extra="allow",
|
||||
)
|
||||
|
||||
host_platform: Literal["WINDOWS", "LINUX"]
|
||||
mobile_platform: DevicePlatform
|
||||
device_id: str
|
||||
|
||||
device_width: int
|
||||
device_height: int
|
||||
|
||||
def to_str(self):
|
||||
return (
|
||||
f"Host platform: {self.host_platform}\n"
|
||||
f"Mobile platform: {self.mobile_platform.value}\n"
|
||||
f"Device ID: {self.device_id}\n"
|
||||
f"Device width: {self.device_width}\n"
|
||||
f"Device height: {self.device_height}\n"
|
||||
)
|
||||
|
||||
|
||||
class ExecutionSetup(BaseModel):
|
||||
"""Execution setup for a task."""
|
||||
|
||||
model_config = ConfigDict(
|
||||
arbitrary_types_allowed=True,
|
||||
extra="allow",
|
||||
)
|
||||
|
||||
traces_path: Path | None = None
|
||||
trace_name: str | None = None
|
||||
enable_remote_tracing: bool = False
|
||||
app_lock_status: AppLaunchResult | None = None
|
||||
video_recording_tools_enabled: bool = Field(default_factory=detect_video_tools_enabled)
|
||||
disable_checker: bool = True
|
||||
checker_max_iterations: int = 20
|
||||
checker_max_chat_rounds: int = 4
|
||||
disable_planner_validation: bool = True
|
||||
planner_validation_threshold: float = 0.85
|
||||
enable_committee: bool = False
|
||||
committee_debate_rounds: int = 2
|
||||
disable_outputter: bool = False
|
||||
outputter: OutputterConfig = Field(default_factory=OutputterConfig)
|
||||
explorer: ExplorerConfig = Field(default_factory=ExplorerConfig)
|
||||
explorer_versions: dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
@property
|
||||
def explorer_version(self) -> str:
|
||||
return self.explorer.default_version
|
||||
|
||||
@property
|
||||
def explorer_flash_mode(self) -> str:
|
||||
return self.explorer.flash_mode
|
||||
|
||||
@property
|
||||
def explorer_pro_mode(self) -> str:
|
||||
return self.explorer.pro_mode
|
||||
|
||||
@property
|
||||
def explorer_caching(self) -> bool:
|
||||
return self.explorer.caching
|
||||
|
||||
def get_locked_app_package(self) -> str | None:
|
||||
"""Get the locked app package name if app locking is enabled.
|
||||
|
||||
Returns:
|
||||
The locked app package name, or None if app locking is not enabled.
|
||||
"""
|
||||
if self.app_lock_status:
|
||||
return self.app_lock_status.locked_app_package
|
||||
return None
|
||||
|
||||
|
||||
class ArtemisContext(BaseModel):
|
||||
model_config = ConfigDict(
|
||||
arbitrary_types_allowed=True,
|
||||
extra="allow",
|
||||
)
|
||||
|
||||
device: DeviceContext
|
||||
llm_config: LLMConfig | None = None
|
||||
model_router: Any | None = None
|
||||
agent_config: Any = None
|
||||
adb_client: Any | None = None
|
||||
ui_adb_client: Any | None = None
|
||||
|
||||
execution_setup: ExecutionSetup | None = None
|
||||
|
||||
data_engine: DataEngine | None = None
|
||||
checker_task: asyncio.Task[Any] | None = None
|
||||
planner_task: asyncio.Task[Any] | None = None
|
||||
background_tasks: list[asyncio.Task[Any]] = Field(default_factory=list)
|
||||
package_cache: dict[str, str | None] = Field(default_factory=dict)
|
||||
background_jobs: dict[str, dict[str, Any]] = Field(default_factory=dict)
|
||||
|
||||
task_plan_snapshot: Path | None = None
|
||||
task_plan_content_before: str | None = None
|
||||
|
||||
_genai_client: Any | None = PrivateAttr(default=None)
|
||||
mcp_client_ctx: Any | None = None
|
||||
mcp_session: Any | None = None
|
||||
|
||||
async def __aenter__(self) -> ArtemisContext:
|
||||
return self
|
||||
|
||||
async def __aexit__(self, exc_type, exc_val, exc_tb):
|
||||
if self.background_tasks:
|
||||
logger.info(f"Waiting for {len(self.background_tasks)} background tasks to complete...")
|
||||
results = await asyncio.gather(*self.background_tasks, return_exceptions=True)
|
||||
for task, result in zip(self.background_tasks, results):
|
||||
if isinstance(result, Exception):
|
||||
logger.error(
|
||||
f"Background task {task.get_name()} failed: {result}",
|
||||
exc_info=result,
|
||||
)
|
||||
self.background_tasks.clear()
|
||||
|
||||
if self.mcp_client_ctx:
|
||||
try:
|
||||
await self.mcp_client_ctx.__aexit__(exc_type, exc_val, exc_tb)
|
||||
except Exception:
|
||||
pass
|
||||
finally:
|
||||
self.mcp_client_ctx = None
|
||||
self.mcp_session = None
|
||||
|
||||
if self.data_engine:
|
||||
try:
|
||||
await self.data_engine.shutdown()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def get_adb_client(self) -> Any:
|
||||
if self.adb_client is None:
|
||||
raise ValueError("No ADB client in context.")
|
||||
return self.adb_client
|
||||
|
||||
def get_ui_adb_client(self) -> Any:
|
||||
if self.ui_adb_client is None:
|
||||
raise ValueError("No UIAutomator client in context.")
|
||||
return self.ui_adb_client
|
||||
|
||||
|
||||
from artemis.data_engine.engine import DataEngine
|
||||
|
||||
ArtemisContext.model_rebuild()
|
||||
@@ -0,0 +1,41 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Backward compatibility layer for legacy artemis.controllers module.
|
||||
All new development should use artemis.drivers instead.
|
||||
"""
|
||||
|
||||
from artemis.drivers.base import BaseDeviceDriver, KeyCode, ScreenData, SwipeDirection
|
||||
from artemis.drivers.factory import create_driver, get_driver
|
||||
from artemis.drivers.types import (
|
||||
Bounds,
|
||||
CoordinatesSelectorRequest,
|
||||
PercentagesSelectorRequest,
|
||||
SwipeRequest,
|
||||
TapOutput,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"BaseDeviceDriver",
|
||||
"KeyCode",
|
||||
"ScreenData",
|
||||
"SwipeDirection",
|
||||
"Bounds",
|
||||
"CoordinatesSelectorRequest",
|
||||
"PercentagesSelectorRequest",
|
||||
"SwipeRequest",
|
||||
"TapOutput",
|
||||
"create_driver",
|
||||
"get_driver",
|
||||
]
|
||||
@@ -0,0 +1,912 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
import re
|
||||
import tempfile
|
||||
import time
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from adbutils import AdbClient, AdbDevice
|
||||
from PIL import Image
|
||||
|
||||
from artemis.clients.ui_automator_client import (
|
||||
UIAutomatorClient,
|
||||
_parse_hierarchy_xml_to_elements,
|
||||
)
|
||||
from artemis.config import get_temp_dir
|
||||
from artemis.controllers.device_controller import (
|
||||
MobileDeviceController,
|
||||
ScreenDataResponse,
|
||||
)
|
||||
from artemis.controllers.types import (
|
||||
Bounds,
|
||||
CoordinatesSelectorRequest,
|
||||
TapOutput,
|
||||
)
|
||||
from artemis.data_engine.models import VideoRecordingRecord
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.video import (
|
||||
DEFAULT_MAX_DURATION_SECONDS,
|
||||
RecordingSession,
|
||||
VideoRecordingResult,
|
||||
cleanup_video_segments,
|
||||
concatenate_videos,
|
||||
get_active_session,
|
||||
has_active_session,
|
||||
remove_active_session,
|
||||
set_active_session,
|
||||
trim_video,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class AndroidDeviceController(MobileDeviceController):
|
||||
def __init__(
|
||||
self,
|
||||
device_id: str,
|
||||
adb_client: AdbClient,
|
||||
ui_adb_client: UIAutomatorClient,
|
||||
device_width: int,
|
||||
device_height: int,
|
||||
data_engine_start_time: float | None = None,
|
||||
data_engine: Any | None = None,
|
||||
):
|
||||
self.device_id = device_id
|
||||
self.adb_client = adb_client
|
||||
self.ui_adb_client = ui_adb_client
|
||||
self.device_width = device_width
|
||||
self.device_height = device_height
|
||||
self.data_engine_start_time = data_engine_start_time
|
||||
self.data_engine = data_engine
|
||||
self._device: AdbDevice | None = None
|
||||
self._segment_cache: dict[tuple[float, float | None], VideoRecordingResult] = {}
|
||||
|
||||
@property
|
||||
def device(self) -> AdbDevice:
|
||||
if self._device is None:
|
||||
self._device = self.adb_client.device(serial=self.device_id)
|
||||
return self._device
|
||||
|
||||
async def tap(
|
||||
self,
|
||||
coords: CoordinatesSelectorRequest,
|
||||
long_press: bool = False,
|
||||
long_press_duration: int = 1000,
|
||||
times: int = 1,
|
||||
delay_ms: int = 100,
|
||||
) -> TapOutput:
|
||||
try:
|
||||
if long_press:
|
||||
cmd = (
|
||||
f"input swipe {coords.x} {coords.y} {coords.x} {coords.y} {long_press_duration}"
|
||||
)
|
||||
else:
|
||||
if times <= 1:
|
||||
cmd = f"input tap {coords.x} {coords.y}"
|
||||
else:
|
||||
# Chain multiple taps in a single shell command to eliminate shell spawning overhead
|
||||
# between rapid consecutive clicks (e.g. 7 taps to enter developer mode)
|
||||
taps = [f"input tap {coords.x} {coords.y}"] * times
|
||||
cmd = f" && sleep {delay_ms / 1000.0:.3f} && ".join(taps)
|
||||
|
||||
logger.info(f"Executing ADB shell: {cmd}")
|
||||
self.device.shell(cmd)
|
||||
return TapOutput(error=None)
|
||||
except Exception as e:
|
||||
return TapOutput(error=f"ADB tap failed: {str(e)}")
|
||||
|
||||
async def swipe(
|
||||
self,
|
||||
start: CoordinatesSelectorRequest,
|
||||
end: CoordinatesSelectorRequest,
|
||||
duration: int = 400,
|
||||
) -> str | None:
|
||||
try:
|
||||
cmd = f"input touchscreen swipe {start.x} {start.y} {end.x} {end.y} {duration}"
|
||||
logger.info(f"Executing ADB command: {cmd}")
|
||||
self.device.shell(cmd)
|
||||
return None
|
||||
except Exception as e:
|
||||
return f"ADB swipe failed: {str(e)}"
|
||||
|
||||
async def get_screen_data(self) -> ScreenDataResponse:
|
||||
"""Get screen data using the UIAutomator2 client"""
|
||||
try:
|
||||
logger.info("Using UIAutomator2 for screen data retrieval")
|
||||
ui_data = self.ui_adb_client.get_screen_data()
|
||||
return ScreenDataResponse(
|
||||
base64=ui_data.base64,
|
||||
elements=ui_data.elements,
|
||||
width=ui_data.width,
|
||||
height=ui_data.height,
|
||||
platform="android",
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get screen data: {e}")
|
||||
raise
|
||||
|
||||
async def screenshot(self) -> str:
|
||||
try:
|
||||
b64 = self.ui_adb_client.get_screenshot_base64()
|
||||
if b64 is None:
|
||||
raise RuntimeError("Failed to capture screenshot")
|
||||
return b64
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to take screenshot: {e}")
|
||||
raise
|
||||
|
||||
async def input_text(self, text: str) -> bool:
|
||||
try:
|
||||
# Fast path for ASCII text: Use ADB directly.
|
||||
# This avoids toggling FastInputIME, which hides the keyboard and can cause focus loss.
|
||||
is_ascii = all(ord(c) < 128 for c in text)
|
||||
if is_ascii:
|
||||
return self._input_text_adb_fallback(text)
|
||||
|
||||
self.ui_adb_client.send_text(text)
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.warning(f"UIAutomator2 send_text failed: {e}, falling back to ADB shell")
|
||||
return self._input_text_adb_fallback(text)
|
||||
|
||||
def _input_text_adb_fallback(self, text: str) -> bool:
|
||||
"""Fallback method using ADB shell input text command."""
|
||||
try:
|
||||
lines = text.split("\n")
|
||||
for line_idx, line in enumerate(lines):
|
||||
if line_idx > 0:
|
||||
self.device.shell("input keyevent 66")
|
||||
if not line:
|
||||
continue
|
||||
parts = line.split("%s")
|
||||
for i, part in enumerate(parts):
|
||||
to_write = ""
|
||||
for char in part:
|
||||
if char == " ":
|
||||
to_write += "%s"
|
||||
elif char in [
|
||||
"&",
|
||||
"<",
|
||||
">",
|
||||
"|",
|
||||
";",
|
||||
"(",
|
||||
")",
|
||||
"$",
|
||||
"`",
|
||||
"\\",
|
||||
'"',
|
||||
"'",
|
||||
]:
|
||||
to_write += f"\\{char}"
|
||||
else:
|
||||
to_write += char
|
||||
|
||||
if to_write:
|
||||
self.device.shell(f"input text '{to_write}'")
|
||||
|
||||
if i < len(parts) - 1:
|
||||
self.device.shell("input keyevent 62")
|
||||
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to input text via ADB fallback: {e}")
|
||||
return False
|
||||
|
||||
async def launch_app(self, package_or_bundle_id: str) -> bool:
|
||||
try:
|
||||
# Clear foreground obstructions before launching
|
||||
logger.info(
|
||||
"Clearing foreground obstructions (status bar, system dialogs,"
|
||||
" keyboard) before launching app..."
|
||||
)
|
||||
try:
|
||||
# 1. Collapse status bar / notification shade
|
||||
self.device.shell("cmd statusbar collapse")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to collapse status bar: {e}")
|
||||
|
||||
try:
|
||||
# 2. Close system dialogs
|
||||
self.device.shell("am broadcast -a android.intent.action.CLOSE_SYSTEM_DIALOGS")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to close system dialogs: {e}")
|
||||
|
||||
# 3. Dismiss keyboard if visible
|
||||
if self.is_keyboard_visible():
|
||||
await self.dismiss_keyboard()
|
||||
|
||||
self.device.app_start(package_or_bundle_id)
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to launch app {package_or_bundle_id}: {e}")
|
||||
return False
|
||||
|
||||
async def terminate_app(self, package_or_bundle_id: str | None) -> bool:
|
||||
try:
|
||||
if package_or_bundle_id is None:
|
||||
current_app = self._get_current_foreground_package()
|
||||
if current_app:
|
||||
logger.info(f"Stopping currently running app: {current_app}")
|
||||
self.device.app_stop(current_app)
|
||||
else:
|
||||
logger.warning("No foreground app detected")
|
||||
return False
|
||||
else:
|
||||
self.device.app_stop(package_or_bundle_id)
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to terminate app {package_or_bundle_id}: {e}")
|
||||
return False
|
||||
|
||||
async def open_url(self, url: str) -> bool:
|
||||
try:
|
||||
cmd = f"am start -a android.intent.action.VIEW -d {url}"
|
||||
logger.info(f"Executing ADB shell: {cmd}")
|
||||
self.device.shell(cmd)
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to open URL {url}: {e}")
|
||||
return False
|
||||
|
||||
def is_keyboard_visible(self) -> bool:
|
||||
try:
|
||||
res = self.device.shell("dumpsys input_method")
|
||||
if isinstance(res, bytes):
|
||||
res = res.decode("utf-8")
|
||||
|
||||
# Common patterns for open keyboard in dumpsys
|
||||
patterns = [
|
||||
"mInputShown=true",
|
||||
"mInputViewShowing=true",
|
||||
"mInputViewShowing=VISIBLE",
|
||||
]
|
||||
for p in patterns:
|
||||
if p in res:
|
||||
return True
|
||||
|
||||
# Alternate check using dumpsys window
|
||||
res_win = self.device.shell(
|
||||
"dumpsys window | grep -E 'mInputMethodWindow|mShowingInputMethod'"
|
||||
)
|
||||
if isinstance(res_win, bytes):
|
||||
res_win = res_win.decode("utf-8")
|
||||
if "mInputMethodWindow" in res_win and "visible=true" in res_win.lower():
|
||||
return True
|
||||
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to check keyboard visibility: {e}")
|
||||
return False
|
||||
|
||||
async def dismiss_keyboard(self) -> bool:
|
||||
try:
|
||||
if not self.is_keyboard_visible():
|
||||
return True
|
||||
|
||||
logger.info("Keyboard is visible, attempting to dismiss via Keyevent 111 (Escape)")
|
||||
self.device.shell("input keyevent 111")
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
if not self.is_keyboard_visible():
|
||||
logger.info("Keyboard successfully dismissed via Keyevent 111")
|
||||
return True
|
||||
|
||||
logger.info("Keyboard still visible, trying Keyevent 4 (BACK) as fallback to dismiss")
|
||||
self.device.shell("input keyevent 4")
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
if not self.is_keyboard_visible():
|
||||
logger.info("Keyboard successfully dismissed via Keyevent 4")
|
||||
return True
|
||||
|
||||
logger.warning("Failed to dismiss keyboard after Escape and BACK")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"Error in dismiss_keyboard: {e}")
|
||||
return False
|
||||
|
||||
async def press_back(self) -> bool:
|
||||
try:
|
||||
if self.is_keyboard_visible():
|
||||
logger.info(
|
||||
"Intercepted press_back: Keyboard is visible. Attempting to"
|
||||
" dismiss keyboard first."
|
||||
)
|
||||
return await self.dismiss_keyboard()
|
||||
|
||||
logger.info("Executing normal press_back (keyboard not visible)")
|
||||
self.device.shell("input keyevent 4")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to press back: {e}")
|
||||
return False
|
||||
|
||||
async def press_home(self) -> bool:
|
||||
try:
|
||||
self.device.shell("input keyevent 3")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to press home: {e}")
|
||||
return False
|
||||
|
||||
async def press_enter(self) -> bool:
|
||||
try:
|
||||
self.device.shell("input keyevent 66")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to press enter: {e}")
|
||||
return False
|
||||
|
||||
async def press_key(self, keycode: str) -> bool:
|
||||
try:
|
||||
# Intercept BACK keycodes to handle keyboard dismissal first
|
||||
clean_keycode = str(keycode).strip().upper()
|
||||
if clean_keycode in ("4", "KEYCODE_BACK", "BACK"):
|
||||
if self.is_keyboard_visible():
|
||||
logger.info(
|
||||
f"Intercepted press_key({keycode}): Keyboard is"
|
||||
" visible. Dismissing keyboard first."
|
||||
)
|
||||
return await self.dismiss_keyboard()
|
||||
|
||||
self.device.shell(f"input keyevent {keycode}")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to press key {keycode}: {e}")
|
||||
return False
|
||||
|
||||
async def get_ui_hierarchy(self) -> list[dict]:
|
||||
|
||||
max_attempts = 4
|
||||
retry_delay = 0.5
|
||||
|
||||
for attempt in range(1, max_attempts + 1):
|
||||
try:
|
||||
hierarchy_xml = self.ui_adb_client.get_hierarchy()
|
||||
elements = _parse_hierarchy_xml_to_elements(hierarchy_xml)
|
||||
|
||||
# Check if hierarchy contains actual UI elements besides the root node
|
||||
if len(elements) > 1:
|
||||
if attempt > 1:
|
||||
logger.info(
|
||||
f"Successfully retrieved non-empty UI hierarchy on attempt {attempt}"
|
||||
)
|
||||
return elements
|
||||
|
||||
logger.warning(
|
||||
"Retrieved empty or root-only UI hierarchy (length"
|
||||
f" {len(elements)}) on attempt {attempt}/{max_attempts}."
|
||||
f" Retrying in {retry_delay}s..."
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
f"Failed to get UI hierarchy on attempt {attempt}/{max_attempts}: {e}"
|
||||
)
|
||||
if attempt == max_attempts:
|
||||
logger.error(f"Failed to get UI hierarchy after {max_attempts} attempts: {e}")
|
||||
return []
|
||||
|
||||
if attempt < max_attempts:
|
||||
await asyncio.sleep(retry_delay)
|
||||
|
||||
return []
|
||||
|
||||
def find_element(
|
||||
self,
|
||||
ui_hierarchy: list[dict],
|
||||
resource_id: str | None = None,
|
||||
text: str | None = None,
|
||||
index: int = 0,
|
||||
) -> tuple[dict | None, Bounds | None, str | None]:
|
||||
if not resource_id and not text:
|
||||
return None, None, "No resource_id or text provided"
|
||||
|
||||
matches = []
|
||||
for element in ui_hierarchy:
|
||||
if resource_id and element.get("resource-id") == resource_id:
|
||||
matches.append(element)
|
||||
elif text and (element.get("text") == text or element.get("accessibilityText") == text):
|
||||
matches.append(element)
|
||||
|
||||
if not matches:
|
||||
criteria = f"resource_id='{resource_id}'" if resource_id else f"text='{text}'"
|
||||
return None, None, f"No element found with {criteria}"
|
||||
|
||||
if index >= len(matches):
|
||||
criteria = f"resource_id='{resource_id}'" if resource_id else f"text='{text}'"
|
||||
return (
|
||||
None,
|
||||
None,
|
||||
(f"Index {index} out of range for {criteria} (found {len(matches)} matches)"),
|
||||
)
|
||||
|
||||
element = matches[index]
|
||||
bounds = self._extract_bounds(element)
|
||||
|
||||
return element, bounds, None
|
||||
|
||||
def _get_current_foreground_package(self) -> str | None:
|
||||
try:
|
||||
app_info = self.device.current_app()
|
||||
if app_info and app_info.package:
|
||||
return app_info.package
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to get current foreground package using adbutils: {e}."
|
||||
" Falling back to dumpsys parsing."
|
||||
)
|
||||
try:
|
||||
result = self.device.shell("dumpsys window | grep mCurrentFocus")
|
||||
|
||||
# Convert to string if bytes
|
||||
if isinstance(result, bytes):
|
||||
result_str = result.decode("utf-8")
|
||||
elif isinstance(result, str):
|
||||
result_str = result
|
||||
else:
|
||||
return None
|
||||
|
||||
if result_str and "=" in result_str:
|
||||
parts = result_str.split("/")
|
||||
if len(parts) > 0:
|
||||
package = parts[0].split()[-1]
|
||||
return package if package else None
|
||||
except Exception as fallback_err:
|
||||
logger.error(f"Fallback dumpsys parsing also failed: {fallback_err}")
|
||||
return None
|
||||
|
||||
def _extract_bounds(self, element: dict) -> Bounds | None:
|
||||
bounds_str = element.get("bounds")
|
||||
if not bounds_str or not isinstance(bounds_str, str):
|
||||
return None
|
||||
|
||||
try:
|
||||
match = re.match(r"\[(\d+),(\d+)\]\[(\d+),(\d+)\]", bounds_str)
|
||||
if match:
|
||||
return Bounds(
|
||||
x1=int(match.group(1)),
|
||||
y1=int(match.group(2)),
|
||||
x2=int(match.group(3)),
|
||||
y2=int(match.group(4)),
|
||||
)
|
||||
except (ValueError, IndexError):
|
||||
return None
|
||||
|
||||
return None
|
||||
|
||||
async def erase_text(self, nb_chars: int | None = None) -> bool:
|
||||
# 1. If clearing the whole field, try fast UIAutomator2 clear first
|
||||
if nb_chars is None:
|
||||
logger.info("Attempting fast UIAutomator2 clear_text...")
|
||||
if self.ui_adb_client.clear_text():
|
||||
return True
|
||||
logger.warning("UIAutomator2 clear_text failed, falling back to optimized ADB shell")
|
||||
|
||||
# 2. Optimized ADB fallback: combine multiple KEYCODE_DEL keyevents into a single command
|
||||
try:
|
||||
chars_to_delete = nb_chars if nb_chars is not None else 50
|
||||
if chars_to_delete > 0:
|
||||
# KEYCODE_DEL is 67. Build arguments list "67 67 67 ..."
|
||||
keycode_args = " ".join(["67"] * chars_to_delete)
|
||||
cmd = f"input keyevent {keycode_args}"
|
||||
logger.info(f"Executing combined ADB shell delete: {cmd}")
|
||||
self.device.shell(cmd)
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to erase text via ADB fallback: {e}")
|
||||
return False
|
||||
|
||||
async def cleanup(self) -> None:
|
||||
pass
|
||||
|
||||
def get_compressed_b64_screenshot(self, image_base64: str, quality: int = 50) -> str:
|
||||
if image_base64.startswith("data:image"):
|
||||
image_base64 = image_base64.split(",")[1]
|
||||
|
||||
image_data = base64.b64decode(image_base64)
|
||||
image = Image.open(BytesIO(image_data))
|
||||
|
||||
compressed_io = BytesIO()
|
||||
image.save(compressed_io, format="JPEG", quality=quality, optimize=True)
|
||||
|
||||
compressed_base64 = base64.b64encode(compressed_io.getvalue()).decode("utf-8")
|
||||
return compressed_base64
|
||||
|
||||
async def start_video_recording(
|
||||
self,
|
||||
max_duration_seconds: int = DEFAULT_MAX_DURATION_SECONDS,
|
||||
output_dir: Path | None = None,
|
||||
) -> VideoRecordingResult:
|
||||
"""Start screen recording on Android device using scrcpy."""
|
||||
self._segment_cache.clear()
|
||||
if has_active_session(self.device_id):
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=(f"Recording already in progress for device {self.device_id}"),
|
||||
)
|
||||
|
||||
try:
|
||||
# Create a temp file for the MKV recording if output_dir not provided
|
||||
if not output_dir:
|
||||
output_dir = Path(
|
||||
tempfile.mkdtemp(prefix="scrcpy_", dir=get_temp_dir("recordings"))
|
||||
)
|
||||
else:
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
local_video_path = output_dir / "recording.mkv"
|
||||
|
||||
video_id = uuid4()
|
||||
session = RecordingSession(
|
||||
video_id=video_id,
|
||||
device_id=self.device_id,
|
||||
start_time=time.time(),
|
||||
data_engine_start_time=self.data_engine_start_time,
|
||||
local_video_path=local_video_path,
|
||||
)
|
||||
|
||||
# Persist to local database if Data Engine is active
|
||||
if self.data_engine and self.data_engine.storage:
|
||||
try:
|
||||
record = VideoRecordingRecord(
|
||||
video_id=video_id,
|
||||
session_id=self.data_engine.current_session_id,
|
||||
device_id=self.device_id,
|
||||
start_time=session.start_time,
|
||||
local_video_path=str(local_video_path),
|
||||
)
|
||||
self.data_engine.storage.create_video_recording(record)
|
||||
except Exception as db_err:
|
||||
logger.error(f"Failed to persist video recording start to DB: {db_err}")
|
||||
|
||||
# Start scrcpy in background
|
||||
cmd = [
|
||||
"scrcpy",
|
||||
"--serial",
|
||||
self.device_id,
|
||||
"--no-window",
|
||||
"--record",
|
||||
str(local_video_path),
|
||||
"--record-format",
|
||||
"mkv",
|
||||
"--video-bit-rate",
|
||||
"2M",
|
||||
]
|
||||
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
*cmd,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
|
||||
session.process = process
|
||||
set_active_session(self.device_id, session)
|
||||
|
||||
logger.info(
|
||||
f"Started scrcpy recording on {self.device_id}, saving to {local_video_path}"
|
||||
)
|
||||
|
||||
# Wait a bit and check if it's still running.
|
||||
await asyncio.sleep(1.0)
|
||||
if process.returncode is not None:
|
||||
stderr = await process.stderr.read()
|
||||
logger.error(f"scrcpy failed to start: {stderr.decode()}")
|
||||
remove_active_session(self.device_id)
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=f"scrcpy failed to start: {stderr.decode()}",
|
||||
)
|
||||
|
||||
return VideoRecordingResult(
|
||||
success=True,
|
||||
message=f"Recording started on {self.device_id}",
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start scrcpy recording: {e}")
|
||||
remove_active_session(self.device_id)
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=f"Failed to start recording: {e}",
|
||||
)
|
||||
|
||||
async def stop_video_recording(self) -> VideoRecordingResult:
|
||||
"""Stop scrcpy recording and return the video file."""
|
||||
self._segment_cache.clear()
|
||||
session = get_active_session(self.device_id)
|
||||
if not session:
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=f"No active recording for device {self.device_id}",
|
||||
)
|
||||
|
||||
try:
|
||||
process = session.process
|
||||
if process is not None:
|
||||
process.terminate()
|
||||
await asyncio.wait_for(process.wait(), timeout=5.0)
|
||||
|
||||
output_path = session.local_video_path
|
||||
|
||||
# Fix: Concatenate previous segments if scrcpy crashed and restarted
|
||||
if session.android_video_segments:
|
||||
all_segments = session.android_video_segments + [output_path]
|
||||
concatenated_mkv = output_path.parent / "full_recording.mkv"
|
||||
|
||||
if await concatenate_videos(all_segments, concatenated_mkv):
|
||||
cleanup_video_segments(all_segments)
|
||||
output_path = concatenated_mkv
|
||||
|
||||
# Convert MKV to MP4
|
||||
mp4_path = (
|
||||
output_path.parent / "recording.mp4"
|
||||
if output_path.name == "full_recording.mkv"
|
||||
else output_path.with_suffix(".mp4")
|
||||
)
|
||||
success = await self._convert_mkv_to_mp4(output_path, mp4_path)
|
||||
|
||||
remove_active_session(self.device_id)
|
||||
|
||||
final_video_path = mp4_path if (success and mp4_path.exists()) else output_path
|
||||
|
||||
# Persist update to local database if Data Engine is active
|
||||
if self.data_engine and self.data_engine.storage:
|
||||
try:
|
||||
record = VideoRecordingRecord(
|
||||
video_id=session.video_id,
|
||||
session_id=self.data_engine.current_session_id,
|
||||
device_id=self.device_id,
|
||||
start_time=session.start_time,
|
||||
end_time=time.time(),
|
||||
local_video_path=str(final_video_path),
|
||||
)
|
||||
self.data_engine.storage.update_video_recording(record)
|
||||
except Exception as db_err:
|
||||
logger.error(f"Failed to persist video recording stop to DB: {db_err}")
|
||||
|
||||
if success and mp4_path.exists():
|
||||
try:
|
||||
output_path.unlink()
|
||||
except Exception:
|
||||
pass
|
||||
return VideoRecordingResult(
|
||||
success=True,
|
||||
message=f"Recording stopped, saved to {mp4_path}",
|
||||
video_path=mp4_path,
|
||||
)
|
||||
else:
|
||||
return VideoRecordingResult(
|
||||
success=True,
|
||||
message=(
|
||||
f"Recording stopped, saved as MKV to {output_path} (conversion failed)"
|
||||
),
|
||||
video_path=output_path,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to stop scrcpy recording: {e}")
|
||||
remove_active_session(self.device_id)
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=f"Failed to stop recording: {e}",
|
||||
)
|
||||
|
||||
async def extract_segment_metadata(
|
||||
self,
|
||||
start_relative_time: float,
|
||||
end_relative_time: float | None = None,
|
||||
) -> VideoRecordingResult:
|
||||
"""Get a video segment for a specific time range (relative to video start)."""
|
||||
cache_key = (
|
||||
round(start_relative_time, 1),
|
||||
round(end_relative_time, 1) if end_relative_time is not None else None,
|
||||
)
|
||||
if cache_key in self._segment_cache:
|
||||
cached_res = self._segment_cache[cache_key]
|
||||
if cached_res.success and cached_res.video_path and cached_res.video_path.exists():
|
||||
logger.info(
|
||||
"Reusing cached trimmed video segment for range"
|
||||
f" {cache_key[0]}s to {cache_key[1]}s"
|
||||
)
|
||||
return cached_res
|
||||
|
||||
session = get_active_session(self.device_id)
|
||||
if not session:
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=f"No active recording for device {self.device_id}",
|
||||
)
|
||||
|
||||
try:
|
||||
mkv_path = session.local_video_path
|
||||
if not mkv_path or not mkv_path.exists():
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message="Recording file not found",
|
||||
)
|
||||
|
||||
# Check if process is still running (best effort for disconnection check)
|
||||
crashed = False
|
||||
restart_message = ""
|
||||
|
||||
if session.process and session.process.returncode is not None:
|
||||
logger.warning(f"scrcpy process for {self.device_id} has terminated unexpectedly")
|
||||
crashed = True
|
||||
|
||||
# Move current file to segments list (save it before overwriting session state)
|
||||
session.android_video_segments.append(mkv_path)
|
||||
|
||||
# Try to restart scrcpy
|
||||
try:
|
||||
session.android_segment_index += 1
|
||||
output_dir = mkv_path.parent
|
||||
new_video_path = output_dir / f"recording_{session.android_segment_index}.mkv"
|
||||
|
||||
cmd = [
|
||||
"scrcpy",
|
||||
"--serial",
|
||||
self.device_id,
|
||||
"--no-window",
|
||||
"--record",
|
||||
str(new_video_path),
|
||||
"--record-format",
|
||||
"mkv",
|
||||
"--video-bit-rate",
|
||||
"2M",
|
||||
]
|
||||
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
*cmd,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
|
||||
# Wait a bit and check if it's still running.
|
||||
await asyncio.sleep(1.0)
|
||||
if process.returncode is None:
|
||||
session.process = process
|
||||
session.local_video_path = new_video_path
|
||||
restart_message = "Successfully restarted scrcpy."
|
||||
else:
|
||||
stderr = await process.stderr.read()
|
||||
restart_message = f"Failed to restart scrcpy: {stderr.decode()}"
|
||||
except Exception as e:
|
||||
restart_message = f"Failed to restart scrcpy: {e}"
|
||||
|
||||
current_time = time.time()
|
||||
video_duration = current_time - session.start_time
|
||||
|
||||
# Calculate offset
|
||||
offset = 0.0
|
||||
if session.data_engine_start_time is not None:
|
||||
offset = session.start_time - session.data_engine_start_time
|
||||
if offset < 0:
|
||||
offset = 0.0
|
||||
|
||||
video_start_relative_time = start_relative_time - offset
|
||||
video_end_relative_time = (
|
||||
end_relative_time - offset if end_relative_time is not None else None
|
||||
)
|
||||
|
||||
# Safety buffer: avoid reading the very last second being written
|
||||
safe_duration = max(0.0, video_duration - 1.0)
|
||||
|
||||
truncation_warning = None
|
||||
if video_end_relative_time is not None and video_end_relative_time > safe_duration:
|
||||
truncation_warning = (
|
||||
"Video segment truncated. Requested end time"
|
||||
f" {end_relative_time:.1f}s exceeded available safe"
|
||||
f" duration. Truncated to {safe_duration + offset:.1f}s."
|
||||
)
|
||||
|
||||
if video_end_relative_time is None or video_end_relative_time > safe_duration:
|
||||
video_end_relative_time = safe_duration
|
||||
|
||||
if video_start_relative_time < 0:
|
||||
video_start_relative_time = 0.0
|
||||
|
||||
if video_start_relative_time >= video_end_relative_time:
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=(
|
||||
"Invalid time range or requested too close to current"
|
||||
f" time. Available duration: {safe_duration:.1f}s"
|
||||
),
|
||||
)
|
||||
|
||||
# Trim directly from MKV to MP4 (using mkv_path which points to the file before we potentially changed it for restart)
|
||||
|
||||
trim_output_dir = tempfile.mkdtemp(
|
||||
prefix="video_trimmed_",
|
||||
dir=get_temp_dir("trimmed_videos"),
|
||||
)
|
||||
trim_output_path = Path(trim_output_dir) / "segment.mp4"
|
||||
|
||||
# Retry mechanism for ffmpeg stability
|
||||
success = False
|
||||
for attempt in range(3):
|
||||
success = await trim_video(
|
||||
mkv_path,
|
||||
video_start_relative_time,
|
||||
video_end_relative_time,
|
||||
trim_output_path,
|
||||
)
|
||||
if success and trim_output_path.exists():
|
||||
break
|
||||
logger.warning(f"ffmpeg trim attempt {attempt + 1} failed, retrying in 0.5s...")
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
if not success or not trim_output_path.exists():
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message="Failed to trim video segment after retries",
|
||||
)
|
||||
|
||||
actual_start = video_start_relative_time + offset
|
||||
actual_end = video_end_relative_time + offset
|
||||
|
||||
message = f"Video segment retrieved for range {actual_start:.1f}s to {actual_end:.1f}s"
|
||||
if crashed:
|
||||
message += (
|
||||
f". Warning: scrcpy crashed. Data loss may have occurred. {restart_message}"
|
||||
)
|
||||
|
||||
file_size_mb = trim_output_path.stat().st_size / (1024 * 1024)
|
||||
duration = video_end_relative_time - video_start_relative_time
|
||||
|
||||
res = VideoRecordingResult(
|
||||
success=True,
|
||||
message=message,
|
||||
video_path=trim_output_path,
|
||||
file_size_mb=round(file_size_mb, 2),
|
||||
duration_seconds=round(duration, 2),
|
||||
actual_start_relative_time=actual_start,
|
||||
warning=truncation_warning,
|
||||
)
|
||||
self._segment_cache[cache_key] = res
|
||||
return res
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get video segment: {e}")
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=f"Failed to get video segment: {e}",
|
||||
)
|
||||
|
||||
async def _convert_mkv_to_mp4(self, mkv_path: Path, mp4_path: Path) -> bool:
|
||||
"""Convert MKV to MP4 using ffmpeg."""
|
||||
try:
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
"ffmpeg",
|
||||
"-y",
|
||||
"-i",
|
||||
str(mkv_path),
|
||||
"-c",
|
||||
"copy",
|
||||
str(mp4_path),
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
await process.wait()
|
||||
return mp4_path.exists()
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to convert MKV to MP4: {e}")
|
||||
return False
|
||||
@@ -0,0 +1,29 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.controllers.unified_controller import UnifiedMobileController
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def create_device_controller(ctx: ArtemisContext) -> UnifiedMobileController:
|
||||
"""Factory creating the unified mobile controller delegating to device drivers."""
|
||||
return UnifiedMobileController(ctx)
|
||||
|
||||
|
||||
def get_controller(ctx: ArtemisContext) -> UnifiedMobileController:
|
||||
"""Backward-compatible alias for create_device_controller."""
|
||||
return create_device_controller(ctx)
|
||||
@@ -0,0 +1,245 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any
|
||||
|
||||
from artemis.controllers.types import Bounds, CoordinatesSelectorRequest, TapOutput
|
||||
from artemis.utils.video import VideoRecordingResult
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class _CyFunctionDetectorMeta(type):
|
||||
def __instancecheck__(self, instance):
|
||||
name = type(instance).__name__
|
||||
return (
|
||||
name
|
||||
in (
|
||||
"cyfunction",
|
||||
"cython_function_or_method",
|
||||
"builtin_function_or_method",
|
||||
)
|
||||
or "cyfunction" in name.lower()
|
||||
)
|
||||
|
||||
|
||||
class CyFunctionDetector(metaclass=_CyFunctionDetectorMeta):
|
||||
pass
|
||||
|
||||
|
||||
class ScreenDataResponse(BaseModel):
|
||||
model_config = {"ignored_types": (CyFunctionDetector,)}
|
||||
base64: str = Field(default="", description="Base64 encoded screenshot string")
|
||||
elements: list[dict[str, Any]] = Field(
|
||||
default_factory=list, description="Parsed UI hierarchy elements"
|
||||
)
|
||||
width: int = Field(default=1080, description="Device screen width in pixels")
|
||||
height: int = Field(default=2400, description="Device screen height in pixels")
|
||||
platform: str = Field(default="android", description="Platform identifier")
|
||||
|
||||
|
||||
class MobileDeviceController(ABC):
|
||||
@abstractmethod
|
||||
async def tap(
|
||||
self,
|
||||
coords: CoordinatesSelectorRequest,
|
||||
long_press: bool = False,
|
||||
long_press_duration: int = 1000,
|
||||
times: int = 1,
|
||||
delay_ms: int = 100,
|
||||
) -> TapOutput:
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def swipe(
|
||||
self,
|
||||
start: CoordinatesSelectorRequest,
|
||||
end: CoordinatesSelectorRequest,
|
||||
duration: int = 400,
|
||||
) -> str | None:
|
||||
"""Swipe from start to end coordinates.
|
||||
|
||||
Returns error message on failure, None on success.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def screenshot(self) -> str:
|
||||
"""Take a screenshot and return raw image data."""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def input_text(self, text: str) -> bool:
|
||||
"""Input text at the currently focused field.
|
||||
|
||||
Returns True on success, False on failure.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def launch_app(self, package_or_bundle_id: str) -> bool:
|
||||
"""Launch an application by package name (Android).
|
||||
|
||||
Returns True on success, False on failure.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def terminate_app(self, package_or_bundle_id: str | None) -> bool:
|
||||
"""Terminate an application.
|
||||
|
||||
Returns True on success, False on failure.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def open_url(self, url: str) -> bool:
|
||||
"""Open a URL.
|
||||
|
||||
Returns True on success, False on failure.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def press_back(self) -> bool:
|
||||
"""Press the back button (Android).
|
||||
|
||||
Returns True on success, False on failure.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def press_home(self) -> bool:
|
||||
"""Press the home button.
|
||||
|
||||
Returns True on success, False on failure.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def press_enter(self) -> bool:
|
||||
"""Press the enter/return key.
|
||||
|
||||
Returns True on success, False on failure.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def press_key(self, keycode: str) -> bool:
|
||||
"""Press a specific key event.
|
||||
|
||||
Returns True on success, False on failure.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def get_ui_hierarchy(self) -> list[dict]:
|
||||
"""Get the UI element hierarchy.
|
||||
|
||||
Returns a list of UI elements with their properties.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
def find_element(
|
||||
self,
|
||||
ui_hierarchy: list[dict],
|
||||
resource_id: str | None = None,
|
||||
text: str | None = None,
|
||||
index: int = 0,
|
||||
) -> tuple[dict | None, Bounds | None, str | None]:
|
||||
"""Find a UI element in the hierarchy.
|
||||
|
||||
Returns:
|
||||
Tuple of (element_dict, bounds, error_message)
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def cleanup(self) -> None:
|
||||
"""Clean up resources (e.g., stop companion processes)."""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def erase_text(self, nb_chars: int | None = None) -> bool:
|
||||
"""Erase the last nb_chars characters.
|
||||
|
||||
Returns True on success, False on failure.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def get_screen_data(self) -> "ScreenDataResponse":
|
||||
"""Get screen data including screenshot (base64), UI hierarchy elements,
|
||||
|
||||
screen dimensions, and platform.
|
||||
|
||||
Returns:
|
||||
ScreenDataResponse with base64 screenshot, elements, width, height,
|
||||
platform
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
def get_compressed_b64_screenshot(self, image_base64: str, quality: int = 50) -> str:
|
||||
"""Compress a base64 image.
|
||||
|
||||
Returns the compressed base64 image.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def start_video_recording(
|
||||
self,
|
||||
max_duration_seconds: int = 900,
|
||||
) -> VideoRecordingResult:
|
||||
"""Start screen recording on the device.
|
||||
|
||||
Args:
|
||||
max_duration_seconds: Maximum recording duration in seconds.
|
||||
|
||||
Returns:
|
||||
VideoRecordingResult with success status and message.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def stop_video_recording(self) -> VideoRecordingResult:
|
||||
"""Stop screen recording and retrieve the video file.
|
||||
|
||||
Returns:
|
||||
VideoRecordingResult with success status, message, and video_path if
|
||||
successful.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
|
||||
@abstractmethod
|
||||
async def extract_segment_metadata(
|
||||
self,
|
||||
start_relative_time: float,
|
||||
end_relative_time: float | None = None,
|
||||
) -> VideoRecordingResult:
|
||||
"""Get a video segment for a specific time range.
|
||||
|
||||
Args:
|
||||
start_relative_time: Start time in seconds relative to session
|
||||
start.
|
||||
end_relative_time: End time in seconds relative to session start, or
|
||||
None for latest.
|
||||
|
||||
Returns:
|
||||
VideoRecordingResult with success status, message, and video_path if
|
||||
successful.
|
||||
"""
|
||||
raise NotImplementedError("Subclasses must implement this method")
|
||||
@@ -0,0 +1,137 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import os
|
||||
from shutil import which
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from artemis.context import ArtemisContext, DevicePlatform
|
||||
from artemis.utils.logger import ArtemisLogger, get_logger
|
||||
from artemis.utils.shell_utils import run_shell_command_on_host
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def get_adb_device(ctx: ArtemisContext) -> Any:
|
||||
"""Retrieve the ADB device instance (AdbDevice locally or RemoteAdbDevice in Cloud mode)."""
|
||||
if ctx.adb_client is None:
|
||||
if os.environ.get("ARTEMIS_CLOUD_MODE") == "1":
|
||||
from cloud_service.virtualization import RemoteAdbClient
|
||||
|
||||
ctx.adb_client = RemoteAdbClient()
|
||||
return ctx.adb_client.device(serial=ctx.device.device_id)
|
||||
return None
|
||||
|
||||
if ctx.device.mobile_platform != DevicePlatform.ANDROID:
|
||||
return None
|
||||
|
||||
adb = ctx.get_adb_client()
|
||||
return adb.device(serial=ctx.device.device_id)
|
||||
|
||||
|
||||
def get_first_device(
|
||||
logger: ArtemisLogger | None = None,
|
||||
) -> tuple[str | None, DevicePlatform | None, None]:
|
||||
"""Gets the first available device."""
|
||||
if which("adb"):
|
||||
try:
|
||||
android_output = run_shell_command_on_host("adb devices")
|
||||
lines = android_output.strip().split("\n")
|
||||
for line in lines:
|
||||
if "device" in line and not line.startswith("List of devices"):
|
||||
return line.split()[0], DevicePlatform.ANDROID, None
|
||||
except RuntimeError as e:
|
||||
if logger:
|
||||
logger.error(f"ADB command failed: {e}")
|
||||
|
||||
return None, None, None
|
||||
|
||||
|
||||
def get_device_date(ctx: ArtemisContext) -> str:
|
||||
device = get_adb_device(ctx)
|
||||
if not device:
|
||||
return time.strftime("%Y-%m-%d %H:%M:%S")
|
||||
try:
|
||||
return str(device.shell("date")).strip()
|
||||
except Exception as e:
|
||||
logger.debug(f"Failed to get device date: {e}")
|
||||
return time.strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
|
||||
def list_packages(ctx: ArtemisContext) -> str:
|
||||
"""List installed packages dynamically from the physical/remote device."""
|
||||
device = get_adb_device(ctx)
|
||||
if not device:
|
||||
return ""
|
||||
|
||||
try:
|
||||
cmd = ["pm", "list", "packages", "-f"]
|
||||
raw_output = str(device.shell(" ".join(cmd)))
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to query package list from device: {e}")
|
||||
return ""
|
||||
|
||||
lines = raw_output.strip().split("\n")
|
||||
packages = []
|
||||
for line in lines:
|
||||
if "=" in line:
|
||||
package_name = line.split("=")[-1].strip()
|
||||
if package_name:
|
||||
packages.append(package_name)
|
||||
elif line.startswith("package:"):
|
||||
package_name = line.split("package:")[-1].strip()
|
||||
if package_name:
|
||||
packages.append(package_name)
|
||||
|
||||
return "\n".join(sorted(packages))
|
||||
|
||||
|
||||
async def list_packages_async(ctx: ArtemisContext) -> str:
|
||||
return list_packages(ctx)
|
||||
|
||||
|
||||
def get_current_foreground_package(ctx: ArtemisContext) -> str | None:
|
||||
"""Get the package name of the currently focused/foreground app dynamically."""
|
||||
device = get_adb_device(ctx)
|
||||
if not device:
|
||||
return None
|
||||
|
||||
try:
|
||||
app_info = device.current_app()
|
||||
if app_info and app_info.package:
|
||||
return app_info.package
|
||||
except Exception as e:
|
||||
logger.debug(f"device.current_app failed: {e}. Falling back to dumpsys.")
|
||||
|
||||
try:
|
||||
output = str(device.shell("dumpsys window | grep mCurrentFocus"))
|
||||
if "mCurrentFocus=" in output:
|
||||
segment = output.split("mCurrentFocus=")[-1]
|
||||
if "/" in segment:
|
||||
tokens = segment.split()
|
||||
for token in tokens:
|
||||
if "." in token and not token.startswith("Window"):
|
||||
package = token.split("/")[0]
|
||||
package = package.rstrip("}")
|
||||
if package and "." in package:
|
||||
return package
|
||||
except Exception as e:
|
||||
logger.debug(f"Fallback dumpsys parsing failed: {e}")
|
||||
|
||||
return None
|
||||
|
||||
|
||||
async def get_current_foreground_package_async(ctx: ArtemisContext) -> str | None:
|
||||
return get_current_foreground_package(ctx)
|
||||
@@ -0,0 +1,37 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Backward compatibility alias for artemis.drivers.types."""
|
||||
|
||||
from artemis.drivers.types import (
|
||||
Bounds,
|
||||
CoordinatesSelectorRequest,
|
||||
PercentagesSelectorRequest,
|
||||
SwipeRequest,
|
||||
SwipeStartEndPercentagesRequest,
|
||||
TapOutput,
|
||||
)
|
||||
|
||||
# Aliases for backward compatibility
|
||||
SwipeStartEndCoordinatesRequest = SwipeRequest
|
||||
|
||||
__all__ = [
|
||||
"Bounds",
|
||||
"CoordinatesSelectorRequest",
|
||||
"PercentagesSelectorRequest",
|
||||
"SwipeRequest",
|
||||
"SwipeStartEndCoordinatesRequest",
|
||||
"SwipeStartEndPercentagesRequest",
|
||||
"TapOutput",
|
||||
]
|
||||
@@ -0,0 +1,796 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import time
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
from artemis.config.paths import get_temp_dir
|
||||
from artemis.context import ArtemisContext
|
||||
from artemis.drivers.factory import get_driver
|
||||
from artemis.drivers.base import BaseDeviceDriver
|
||||
from artemis.controllers.device_controller import ScreenDataResponse
|
||||
from artemis.controllers.types import (
|
||||
SwipeRequest,
|
||||
SwipeStartEndCoordinatesRequest,
|
||||
SwipeStartEndPercentagesRequest,
|
||||
TapOutput,
|
||||
)
|
||||
from artemis.utils.logger import get_logger
|
||||
from artemis.utils.video import (
|
||||
DEFAULT_MAX_DURATION_SECONDS,
|
||||
RecordingSession,
|
||||
VideoRecordingResult,
|
||||
cleanup_video_segments,
|
||||
concatenate_videos,
|
||||
get_active_session,
|
||||
get_ffmpeg_path,
|
||||
has_active_session,
|
||||
remove_active_session,
|
||||
set_active_session,
|
||||
trim_video,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class UnifiedMobileController:
|
||||
def __init__(self, ctx: ArtemisContext):
|
||||
self.ctx = ctx
|
||||
self._driver: BaseDeviceDriver = get_driver(ctx)
|
||||
self._segment_cache: dict[tuple[float, float | None], VideoRecordingResult] = {}
|
||||
|
||||
@property
|
||||
def driver(self) -> BaseDeviceDriver:
|
||||
return self._driver
|
||||
|
||||
@property
|
||||
def controller(self) -> Any:
|
||||
return self._driver
|
||||
|
||||
async def tap_at(
|
||||
self,
|
||||
x: int,
|
||||
y: int,
|
||||
long_press: bool = False,
|
||||
long_press_duration: int = 1000,
|
||||
times: int = 1,
|
||||
delay_ms: int = 100,
|
||||
) -> TapOutput:
|
||||
try:
|
||||
if long_press:
|
||||
success = await self._driver.long_press(x, y, duration_ms=long_press_duration)
|
||||
else:
|
||||
success = await self._driver.tap(
|
||||
x, y, duration_ms=100, times=times, delay_ms=delay_ms
|
||||
)
|
||||
return TapOutput(error=None if success else f"Tap failed at ({x}, {y})")
|
||||
except Exception as e:
|
||||
return TapOutput(error=str(e))
|
||||
|
||||
async def tap_percentage(
|
||||
self,
|
||||
x_percent: int,
|
||||
y_percent: int,
|
||||
long_press: bool = False,
|
||||
long_press_duration: int = 1000,
|
||||
) -> TapOutput:
|
||||
"""Tap at percentage-based coordinates (0 to 100)."""
|
||||
norm_x = int(x_percent * 10)
|
||||
norm_y = int(y_percent * 10)
|
||||
success = await self._driver.tap_normalized(
|
||||
norm_x, norm_y, long_press=long_press, duration_ms=long_press_duration
|
||||
)
|
||||
return TapOutput(
|
||||
error=None if success else f"Tap percentage failed at ({x_percent}%, {y_percent}%)"
|
||||
)
|
||||
|
||||
async def tap_element(
|
||||
self,
|
||||
resource_id: str | None = None,
|
||||
text: str | None = None,
|
||||
index: int = 0,
|
||||
long_press: bool = False,
|
||||
long_press_duration: int = 1000,
|
||||
) -> TapOutput:
|
||||
"""Tap on a UI element by finding it in the hierarchy."""
|
||||
success = await self._driver.tap_element(
|
||||
resource_id=resource_id,
|
||||
text=text,
|
||||
index=index,
|
||||
long_press=long_press,
|
||||
duration_ms=long_press_duration,
|
||||
)
|
||||
return TapOutput(
|
||||
error=None
|
||||
if success
|
||||
else f"Failed to tap element (resource_id={resource_id}, text={text})"
|
||||
)
|
||||
|
||||
async def swipe_coords(
|
||||
self,
|
||||
start_x: int,
|
||||
start_y: int,
|
||||
end_x: int,
|
||||
end_y: int,
|
||||
duration: int = 400,
|
||||
) -> str | None:
|
||||
"""Swipe between two coordinate points."""
|
||||
success = await self._driver.swipe(start_x, start_y, end_x, end_y, duration_ms=duration)
|
||||
return None if success else f"Swipe failed from ({start_x},{start_y}) to ({end_x},{end_y})"
|
||||
|
||||
async def swipe_percentage(
|
||||
self,
|
||||
start_x_percent: int,
|
||||
start_y_percent: int,
|
||||
end_x_percent: int,
|
||||
end_y_percent: int,
|
||||
duration: int = 400,
|
||||
) -> str | None:
|
||||
"""Swipe using percentage-based coordinates (0 to 100)."""
|
||||
start_norm = [int(start_x_percent * 10), int(start_y_percent * 10)]
|
||||
end_norm = [int(end_x_percent * 10), int(end_y_percent * 10)]
|
||||
success = await self._driver.swipe_normalized(start_norm, end_norm, duration_ms=duration)
|
||||
return None if success else "Swipe percentage failed"
|
||||
|
||||
async def swipe_request(self, request: SwipeRequest) -> str | None:
|
||||
mode = request.swipe_mode
|
||||
|
||||
if isinstance(mode, SwipeStartEndCoordinatesRequest):
|
||||
return await self.swipe_coords(
|
||||
start_x=mode.start.x,
|
||||
start_y=mode.start.y,
|
||||
end_x=mode.end.x,
|
||||
end_y=mode.end.y,
|
||||
duration=request.duration or 400,
|
||||
)
|
||||
elif isinstance(mode, SwipeStartEndPercentagesRequest):
|
||||
return await self.swipe_percentage(
|
||||
start_x_percent=mode.start.x_percent,
|
||||
start_y_percent=mode.start.y_percent,
|
||||
end_x_percent=mode.end.x_percent,
|
||||
end_y_percent=mode.end.y_percent,
|
||||
duration=request.duration or 400,
|
||||
)
|
||||
else:
|
||||
return "Unsupported swipe mode"
|
||||
|
||||
async def type_text(self, text: str, clear_existing: bool = True) -> bool:
|
||||
return await self._driver.input_text(text, clear_existing=clear_existing)
|
||||
|
||||
async def take_screenshot(self) -> str:
|
||||
screen_data = await self._driver.get_screen_data()
|
||||
return screen_data.screenshot_base64
|
||||
|
||||
async def launch_app(self, package_or_bundle_id: str) -> bool:
|
||||
return await self._driver.launch_app(package_or_bundle_id)
|
||||
|
||||
async def terminate_app(self, package_or_bundle_id: str | None) -> bool:
|
||||
if not package_or_bundle_id:
|
||||
return False
|
||||
return await self._driver.stop_app(package_or_bundle_id)
|
||||
|
||||
async def open_url(self, url: str) -> bool:
|
||||
await self._driver.execute_shell(f"am start -a android.intent.action.VIEW -d '{url}'")
|
||||
return True
|
||||
|
||||
async def go_back(self) -> bool:
|
||||
return await self._driver.press_key("back")
|
||||
|
||||
async def go_home(self) -> bool:
|
||||
return await self._driver.press_key("home")
|
||||
|
||||
async def press_enter(self) -> bool:
|
||||
return await self._driver.press_key("enter")
|
||||
|
||||
async def press_key(self, keycode: str) -> bool:
|
||||
return await self._driver.press_key(keycode)
|
||||
|
||||
async def erase_text(self, nb_chars: int | None = None) -> bool:
|
||||
if nb_chars is not None and nb_chars > 0:
|
||||
for _ in range(nb_chars):
|
||||
await self._driver.press_key("delete")
|
||||
return True
|
||||
# Safe & robust full clear: Move to End -> Shift+Home selection -> Delete -> Fallback backspaces
|
||||
try:
|
||||
clear_cmd = (
|
||||
"input keyevent 123 && "
|
||||
"input keyevent --meta 1 122 && "
|
||||
"input keyevent 67 && "
|
||||
"input keyevent 67 67 67 67 67 67 67 67 67 67 67 67 67 67 67 67 67 67 67 67"
|
||||
)
|
||||
await self._driver.execute_shell(clear_cmd)
|
||||
except Exception:
|
||||
for _ in range(30):
|
||||
await self._driver.press_key("delete")
|
||||
return True
|
||||
|
||||
async def get_ui_elements(self) -> list[dict]:
|
||||
screen_data = await self._driver.get_screen_data()
|
||||
return screen_data.ui_elements or []
|
||||
|
||||
async def get_screen_data(self) -> "ScreenDataResponse":
|
||||
"""Get screen data including screenshot, UI hierarchy, dimensions, and platform."""
|
||||
data = await self._driver.get_screen_data()
|
||||
return ScreenDataResponse(
|
||||
base64=data.screenshot_base64,
|
||||
elements=data.ui_elements or [],
|
||||
width=data.width,
|
||||
height=data.height,
|
||||
platform=data.platform,
|
||||
)
|
||||
|
||||
async def find_element(
|
||||
self,
|
||||
resource_id: str | None = None,
|
||||
text: str | None = None,
|
||||
index: int = 0,
|
||||
) -> tuple[dict | None, str | None]:
|
||||
elem, _, error = await self._driver.find_element(
|
||||
resource_id=resource_id,
|
||||
text=text,
|
||||
index=index,
|
||||
)
|
||||
return elem, error
|
||||
|
||||
def _get_device_id(self) -> str:
|
||||
if self.ctx and self.ctx.device and self.ctx.device.device_id:
|
||||
return self.ctx.device.device_id
|
||||
return getattr(self._driver, "device_id", None) or "default"
|
||||
|
||||
async def extract_segment_metadata(
|
||||
self,
|
||||
start_time: float,
|
||||
end_time: float | None = None,
|
||||
output_path: Path | None = None,
|
||||
) -> VideoRecordingResult:
|
||||
"""Get a video segment for a specific time range (relative to video start)."""
|
||||
cache_key = (
|
||||
round(start_time, 1),
|
||||
round(end_time, 1) if end_time is not None else None,
|
||||
)
|
||||
if cache_key in self._segment_cache:
|
||||
cached_res = self._segment_cache[cache_key]
|
||||
if cached_res.success and cached_res.video_path and cached_res.video_path.exists():
|
||||
logger.info(
|
||||
f"Reusing cached trimmed video segment for range {cache_key[0]}s to {cache_key[1]}s"
|
||||
)
|
||||
return cached_res
|
||||
|
||||
device_id = self._get_device_id()
|
||||
|
||||
# Handle mock driver
|
||||
if (
|
||||
getattr(self._driver, "is_mock", False)
|
||||
or getattr(getattr(self.ctx, "device", None), "mobile_platform", None) == "mock"
|
||||
or os.environ.get("ARTEMIS_MOCK_DRIVER") == "1"
|
||||
):
|
||||
mock_video = Path("/tmp/mock_recording.mp4")
|
||||
return VideoRecordingResult(
|
||||
success=True,
|
||||
video_path=mock_video,
|
||||
message="Mock segment extracted",
|
||||
)
|
||||
|
||||
session = get_active_session(device_id)
|
||||
if not session:
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=f"No active recording for device {device_id}",
|
||||
)
|
||||
|
||||
try:
|
||||
mkv_path = session.local_video_path
|
||||
if not mkv_path:
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message="Recording file not found",
|
||||
)
|
||||
|
||||
# Check if scrcpy process terminated and restart if needed (synchronous fallback)
|
||||
crashed = False
|
||||
restart_message = ""
|
||||
if session.process and session.process.returncode is not None:
|
||||
logger.warning(f"scrcpy process for {device_id} has terminated unexpectedly")
|
||||
crashed = True
|
||||
if mkv_path.exists() and mkv_path not in session.android_video_segments:
|
||||
session.android_video_segments.append(mkv_path)
|
||||
try:
|
||||
session.android_segment_index += 1
|
||||
output_dir = mkv_path.parent
|
||||
new_video_path = output_dir / f"recording_{session.android_segment_index}.mkv"
|
||||
cmd = [
|
||||
"scrcpy",
|
||||
"--serial",
|
||||
device_id,
|
||||
"--no-window",
|
||||
"--record",
|
||||
str(new_video_path),
|
||||
"--record-format",
|
||||
"mkv",
|
||||
"--video-bit-rate",
|
||||
"2M",
|
||||
]
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
*cmd,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
await asyncio.sleep(0.8)
|
||||
if process.returncode is None:
|
||||
session.process = process
|
||||
session.local_video_path = new_video_path
|
||||
mkv_path = new_video_path
|
||||
restart_message = "Successfully restarted scrcpy."
|
||||
else:
|
||||
stderr = await process.stderr.read()
|
||||
restart_message = f"Failed to restart scrcpy: {stderr.decode()}"
|
||||
except Exception as e:
|
||||
restart_message = f"Failed to restart scrcpy: {e}"
|
||||
|
||||
# Determine source file to trim from (support multi-segment seamless trimming)
|
||||
all_segments = [p for p in session.android_video_segments if p.exists()]
|
||||
if mkv_path.exists() and mkv_path not in all_segments:
|
||||
all_segments.append(mkv_path)
|
||||
|
||||
if not all_segments:
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message="No recording segments found on disk",
|
||||
)
|
||||
|
||||
temp_combined_file = None
|
||||
if len(all_segments) > 1:
|
||||
temp_combined_file = mkv_path.parent / "temp_combined_active.mkv"
|
||||
concat_ok = await concatenate_videos(all_segments, temp_combined_file)
|
||||
trim_source = (
|
||||
temp_combined_file if (concat_ok and temp_combined_file.exists()) else mkv_path
|
||||
)
|
||||
else:
|
||||
trim_source = all_segments[0]
|
||||
|
||||
current_time = time.time()
|
||||
video_duration = current_time - session.start_time
|
||||
|
||||
# Strict timeline alignment:
|
||||
# DataEngine session start time is the reference T0 (0.0s).
|
||||
# Video recording start time is session.start_time.
|
||||
offset = 0.0
|
||||
if session.data_engine_start_time is not None:
|
||||
offset = max(0.0, session.start_time - session.data_engine_start_time)
|
||||
|
||||
video_start_relative_time = start_time - offset
|
||||
video_end_relative_time = end_time - offset if end_time is not None else None
|
||||
|
||||
safe_duration = max(0.0, video_duration - 0.5)
|
||||
|
||||
truncation_warning = None
|
||||
if video_end_relative_time is not None and video_end_relative_time > safe_duration:
|
||||
truncation_warning = (
|
||||
f"Video segment truncated. Requested end time {end_time:.1f}s "
|
||||
f"exceeded available safe duration. Truncated to {safe_duration + offset:.1f}s."
|
||||
)
|
||||
|
||||
if video_end_relative_time is None or video_end_relative_time > safe_duration:
|
||||
video_end_relative_time = safe_duration
|
||||
|
||||
if video_start_relative_time < 0:
|
||||
video_start_relative_time = 0.0
|
||||
|
||||
if video_start_relative_time >= video_end_relative_time:
|
||||
if safe_duration > 0:
|
||||
video_end_relative_time = safe_duration
|
||||
video_start_relative_time = max(
|
||||
0.0,
|
||||
safe_duration - ((end_time - start_time) if end_time else 2.0),
|
||||
)
|
||||
|
||||
if video_start_relative_time >= video_end_relative_time:
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=(
|
||||
f"Invalid time range or requested too close to current time. Available duration: {safe_duration:.1f}s"
|
||||
),
|
||||
)
|
||||
|
||||
trim_output_dir = tempfile.mkdtemp(
|
||||
prefix="video_trimmed_",
|
||||
dir=get_temp_dir("trimmed_videos"),
|
||||
)
|
||||
trim_output_path = Path(trim_output_dir) / "segment.mp4"
|
||||
|
||||
success = False
|
||||
for attempt in range(3):
|
||||
success = await trim_video(
|
||||
trim_source,
|
||||
video_start_relative_time,
|
||||
video_end_relative_time,
|
||||
trim_output_path,
|
||||
)
|
||||
if success and trim_output_path.exists():
|
||||
break
|
||||
logger.warning(f"ffmpeg trim attempt {attempt + 1} failed, retrying in 0.5s...")
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
# Cleanup temp combined MKV if created
|
||||
if temp_combined_file and temp_combined_file.exists():
|
||||
try:
|
||||
temp_combined_file.unlink()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if not success or not trim_output_path.exists():
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message="Failed to trim video segment after retries",
|
||||
)
|
||||
|
||||
actual_start = video_start_relative_time + offset
|
||||
actual_end = video_end_relative_time + offset
|
||||
file_size_mb = trim_output_path.stat().st_size / (1024 * 1024)
|
||||
duration = video_end_relative_time - video_start_relative_time
|
||||
|
||||
message = f"Video segment retrieved for range {actual_start:.1f}s to {actual_end:.1f}s"
|
||||
if crashed:
|
||||
message += (
|
||||
f". Warning: scrcpy crashed. Data loss may have occurred. {restart_message}"
|
||||
)
|
||||
|
||||
res = VideoRecordingResult(
|
||||
success=True,
|
||||
message=message,
|
||||
video_path=trim_output_path,
|
||||
file_size_mb=round(file_size_mb, 2),
|
||||
duration_seconds=round(duration, 2),
|
||||
actual_start_relative_time=actual_start,
|
||||
warning=truncation_warning,
|
||||
)
|
||||
self._segment_cache[cache_key] = res
|
||||
return res
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get video segment: {e}")
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=f"Failed to get video segment: {e}",
|
||||
)
|
||||
|
||||
async def _recording_watchdog(self, device_id: str) -> None:
|
||||
"""Background watchdog to automatically detect crashes and recover scrcpy recording."""
|
||||
while True:
|
||||
session = get_active_session(device_id)
|
||||
if not session or not session.is_active:
|
||||
break
|
||||
proc = session.process
|
||||
if proc is None:
|
||||
break
|
||||
try:
|
||||
ret = await proc.wait()
|
||||
session = get_active_session(device_id)
|
||||
if not session or not session.is_active:
|
||||
break
|
||||
|
||||
logger.warning(
|
||||
f"scrcpy recording process for {device_id} terminated unexpectedly (code {ret}). "
|
||||
f"Auto-restarting recording segment {session.android_segment_index + 1}..."
|
||||
)
|
||||
if session.local_video_path and session.local_video_path.exists():
|
||||
if session.local_video_path not in session.android_video_segments:
|
||||
session.android_video_segments.append(session.local_video_path)
|
||||
|
||||
session.android_segment_index += 1
|
||||
output_dir = session.local_video_path.parent
|
||||
new_video_path = output_dir / f"recording_{session.android_segment_index}.mkv"
|
||||
cmd = [
|
||||
"scrcpy",
|
||||
"--serial",
|
||||
device_id,
|
||||
"--no-window",
|
||||
"--record",
|
||||
str(new_video_path),
|
||||
"--record-format",
|
||||
"mkv",
|
||||
"--video-bit-rate",
|
||||
"2M",
|
||||
]
|
||||
new_proc = await asyncio.create_subprocess_exec(
|
||||
*cmd,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
session.process = new_proc
|
||||
session.local_video_path = new_video_path
|
||||
await asyncio.sleep(0.8)
|
||||
if new_proc.returncode is not None:
|
||||
stderr = await new_proc.stderr.read()
|
||||
logger.error(f"Failed to auto-restart scrcpy recording: {stderr.decode()}")
|
||||
break
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
except Exception as e:
|
||||
logger.error(f"Error in recording watchdog for {device_id}: {e}")
|
||||
break
|
||||
|
||||
async def start_video_recording(
|
||||
self,
|
||||
output_dir: Path | None = None,
|
||||
max_duration_seconds: int = DEFAULT_MAX_DURATION_SECONDS,
|
||||
) -> VideoRecordingResult:
|
||||
"""Start screen recording on Android device using scrcpy."""
|
||||
self._segment_cache.clear()
|
||||
device_id = self._get_device_id()
|
||||
|
||||
# Check mock driver first
|
||||
if (
|
||||
getattr(self._driver, "is_mock", False)
|
||||
or getattr(getattr(self.ctx, "device", None), "mobile_platform", None) == "mock"
|
||||
or os.environ.get("ARTEMIS_MOCK_DRIVER") == "1"
|
||||
):
|
||||
await self._driver.start_video_recording(output_dir)
|
||||
return VideoRecordingResult(success=True, message="Mock recording started")
|
||||
|
||||
if has_active_session(device_id):
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=f"Recording already in progress for device {device_id}",
|
||||
)
|
||||
|
||||
try:
|
||||
if not output_dir:
|
||||
output_dir = Path(
|
||||
tempfile.mkdtemp(prefix="scrcpy_", dir=get_temp_dir("recordings"))
|
||||
)
|
||||
else:
|
||||
output_dir = Path(output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
local_video_path = output_dir / "recording.mkv"
|
||||
video_id = uuid4()
|
||||
start_time = time.time()
|
||||
data_engine_start_time = (
|
||||
self.ctx.data_engine.session_start_time
|
||||
if (self.ctx and self.ctx.data_engine)
|
||||
else start_time
|
||||
)
|
||||
|
||||
session = RecordingSession(
|
||||
video_id=video_id,
|
||||
device_id=device_id,
|
||||
start_time=start_time,
|
||||
data_engine_start_time=data_engine_start_time,
|
||||
local_video_path=local_video_path,
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
# Persist to local database if Data Engine is active
|
||||
if self.ctx and self.ctx.data_engine:
|
||||
self.ctx.data_engine.record_video_start(
|
||||
video_id=video_id,
|
||||
device_id=device_id,
|
||||
local_video_path=local_video_path,
|
||||
start_time=session.start_time,
|
||||
)
|
||||
|
||||
# Start scrcpy in background
|
||||
cmd = [
|
||||
"scrcpy",
|
||||
"--serial",
|
||||
device_id,
|
||||
"--no-window",
|
||||
"--record",
|
||||
str(local_video_path),
|
||||
"--record-format",
|
||||
"mkv",
|
||||
"--video-bit-rate",
|
||||
"2M",
|
||||
]
|
||||
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
*cmd,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
|
||||
session.process = process
|
||||
set_active_session(device_id, session)
|
||||
|
||||
logger.info(f"Started scrcpy recording on {device_id}, saving to {local_video_path}")
|
||||
|
||||
# Check if process failed immediately
|
||||
await asyncio.sleep(0.8)
|
||||
if process.returncode is not None:
|
||||
stderr = await process.stderr.read()
|
||||
err_msg = stderr.decode()
|
||||
logger.error(f"scrcpy failed to start on {device_id}: {err_msg}")
|
||||
remove_active_session(device_id)
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=f"scrcpy failed to start: {err_msg}",
|
||||
)
|
||||
|
||||
# Start background watchdog to auto-recover if scrcpy terminates
|
||||
session.watchdog_task = asyncio.create_task(self._recording_watchdog(device_id))
|
||||
|
||||
return VideoRecordingResult(
|
||||
success=True,
|
||||
message=f"Recording started on {device_id}",
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start scrcpy recording: {e}")
|
||||
remove_active_session(device_id)
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=f"Failed to start recording: {e}",
|
||||
)
|
||||
|
||||
async def stop_video_recording(self) -> VideoRecordingResult:
|
||||
"""Stop scrcpy recording and return the converted MP4 video file."""
|
||||
self._segment_cache.clear()
|
||||
device_id = self._get_device_id()
|
||||
|
||||
# Check mock driver first
|
||||
if (
|
||||
getattr(self._driver, "is_mock", False)
|
||||
or getattr(getattr(self.ctx, "device", None), "mobile_platform", None) == "mock"
|
||||
or os.environ.get("ARTEMIS_MOCK_DRIVER") == "1"
|
||||
):
|
||||
p = await self._driver.stop_video_recording()
|
||||
return VideoRecordingResult(
|
||||
success=True,
|
||||
video_path=Path(p) if p else None,
|
||||
message="Mock recording stopped",
|
||||
)
|
||||
|
||||
session = get_active_session(device_id)
|
||||
if not session:
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=f"No active recording for device {device_id}",
|
||||
)
|
||||
|
||||
# Mark inactive and cancel watchdog task
|
||||
session.is_active = False
|
||||
if session.watchdog_task and not session.watchdog_task.done():
|
||||
session.watchdog_task.cancel()
|
||||
|
||||
try:
|
||||
process = session.process
|
||||
if process is not None:
|
||||
try:
|
||||
process.terminate()
|
||||
await asyncio.wait_for(process.wait(), timeout=5.0)
|
||||
except Exception as proc_e:
|
||||
logger.warning(f"Error terminating scrcpy process: {proc_e}")
|
||||
|
||||
output_path = session.local_video_path
|
||||
if not output_path or not output_path.exists():
|
||||
remove_active_session(device_id)
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message="Recording file not found on disk",
|
||||
)
|
||||
|
||||
# Concatenate previous segments if scrcpy crashed and restarted
|
||||
if session.android_video_segments:
|
||||
all_segments = session.android_video_segments
|
||||
if output_path.exists() and output_path not in all_segments:
|
||||
all_segments.append(output_path)
|
||||
concatenated_mkv = output_path.parent / "full_recording.mkv"
|
||||
if await concatenate_videos(all_segments, concatenated_mkv):
|
||||
cleanup_video_segments(all_segments)
|
||||
output_path = concatenated_mkv
|
||||
|
||||
# Convert MKV to web-streamable MP4
|
||||
mp4_path = (
|
||||
output_path.parent / "recording.mp4"
|
||||
if output_path.name == "full_recording.mkv"
|
||||
else output_path.with_suffix(".mp4")
|
||||
)
|
||||
success = await self._convert_mkv_to_mp4(output_path, mp4_path)
|
||||
|
||||
remove_active_session(device_id)
|
||||
|
||||
final_video_path = mp4_path if (success and mp4_path.exists()) else output_path
|
||||
|
||||
# Persist update to local database if Data Engine is active
|
||||
if self.ctx and self.ctx.data_engine:
|
||||
self.ctx.data_engine.record_video_stop(
|
||||
video_id=session.video_id,
|
||||
device_id=device_id,
|
||||
local_video_path=final_video_path,
|
||||
start_time=session.start_time,
|
||||
end_time=time.time(),
|
||||
)
|
||||
|
||||
if success and mp4_path.exists():
|
||||
try:
|
||||
if output_path != mp4_path and output_path.exists():
|
||||
output_path.unlink()
|
||||
except Exception:
|
||||
pass
|
||||
return VideoRecordingResult(
|
||||
success=True,
|
||||
message=f"Recording stopped, saved to {mp4_path}",
|
||||
video_path=mp4_path,
|
||||
)
|
||||
else:
|
||||
return VideoRecordingResult(
|
||||
success=True,
|
||||
message=f"Recording stopped, saved as MKV to {output_path}",
|
||||
video_path=output_path,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to stop scrcpy recording: {e}")
|
||||
remove_active_session(device_id)
|
||||
return VideoRecordingResult(
|
||||
success=False,
|
||||
message=f"Failed to stop recording: {e}",
|
||||
)
|
||||
|
||||
async def _convert_mkv_to_mp4(self, mkv_path: Path, mp4_path: Path) -> bool:
|
||||
"""Convert MKV to MP4 using ffmpeg with faststart for web streaming."""
|
||||
if not mkv_path.exists():
|
||||
return False
|
||||
try:
|
||||
# 1. Try fast stream copy with +faststart
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
get_ffmpeg_path(),
|
||||
"-y",
|
||||
"-i",
|
||||
str(mkv_path),
|
||||
"-c",
|
||||
"copy",
|
||||
"-movflags",
|
||||
"+faststart",
|
||||
str(mp4_path),
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
await process.wait()
|
||||
if mp4_path.exists() and mp4_path.stat().st_size > 0:
|
||||
return True
|
||||
|
||||
# 2. Fallback to re-encoding if stream copy fails
|
||||
logger.warning("ffmpeg fast copy failed, re-encoding MKV to MP4...")
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
get_ffmpeg_path(),
|
||||
"-y",
|
||||
"-i",
|
||||
str(mkv_path),
|
||||
"-c:v",
|
||||
"libx264",
|
||||
"-preset",
|
||||
"fast",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
"-movflags",
|
||||
"+faststart",
|
||||
str(mp4_path),
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
await process.wait()
|
||||
return mp4_path.exists() and mp4_path.stat().st_size > 0
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to convert MKV to MP4: {e}")
|
||||
return False
|
||||
|
||||
async def cleanup(self) -> None:
|
||||
await self._driver.disconnect()
|
||||
@@ -0,0 +1,51 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Core framework contracts and lifecycle primitives."""
|
||||
|
||||
from artemis.core.context import ExecutionContext, DeviceInfoContext
|
||||
from artemis.core.state import ExecutionContextState, ExecutionStatus, StepRecord
|
||||
from artemis.core.events import Event, EventBus, HookType, global_event_bus
|
||||
from artemis.core.exceptions import (
|
||||
ArtemisException,
|
||||
DeviceDriverException,
|
||||
DeviceNotFoundException,
|
||||
AgentExecutionException,
|
||||
ToolExecutionException,
|
||||
LLMProviderException,
|
||||
StateTransitionException,
|
||||
)
|
||||
from artemis.core.registry import AgentRegistry, ToolRegistry, DriverRegistry
|
||||
|
||||
__all__ = [
|
||||
"ExecutionContext",
|
||||
"DeviceInfoContext",
|
||||
"ExecutionContextState",
|
||||
"ExecutionStatus",
|
||||
"StepRecord",
|
||||
"Event",
|
||||
"EventBus",
|
||||
"HookType",
|
||||
"global_event_bus",
|
||||
"ArtemisException",
|
||||
"DeviceDriverException",
|
||||
"DeviceNotFoundException",
|
||||
"AgentExecutionException",
|
||||
"ToolExecutionException",
|
||||
"LLMProviderException",
|
||||
"StateTransitionException",
|
||||
"AgentRegistry",
|
||||
"ToolRegistry",
|
||||
"DriverRegistry",
|
||||
]
|
||||
@@ -0,0 +1,59 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Clean Execution Context for ARTEMIS."""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class DeviceInfoContext(BaseModel):
|
||||
"""Device metadata and configuration."""
|
||||
|
||||
device_id: str = Field(default="default-device", description="ADB serial or device identifier")
|
||||
platform: str = Field(
|
||||
default="android", description="Platform type ('android', 'cloud', 'mock')"
|
||||
)
|
||||
width: int = Field(default=1080, description="Screen width in pixels")
|
||||
height: int = Field(default=2400, description="Screen height in pixels")
|
||||
|
||||
|
||||
class ExecutionContext:
|
||||
"""Central context holding device driver, session identifiers, and telemetry tracer."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
task_goal: str,
|
||||
device_id: str = "default-device",
|
||||
trace_id: str | None = None,
|
||||
traces_path: Path | str | None = None,
|
||||
locked_package: str | None = None,
|
||||
platform: str = "android",
|
||||
):
|
||||
self.task_goal = task_goal
|
||||
self.trace_id = trace_id or str(uuid4())
|
||||
self.traces_path = Path(traces_path) if traces_path else Path("traces") / self.trace_id
|
||||
self.locked_package = locked_package
|
||||
self.device = DeviceInfoContext(device_id=device_id, platform=platform)
|
||||
self.driver: Any | None = None
|
||||
self.telemetry: Any | None = None
|
||||
self.custom_attributes: dict[str, Any] = {}
|
||||
|
||||
def set_attribute(self, key: str, value: Any) -> None:
|
||||
self.custom_attributes[key] = value
|
||||
|
||||
def get_attribute(self, key: str, default: Any = None) -> Any:
|
||||
return self.custom_attributes.get(key, default)
|
||||
@@ -0,0 +1,77 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Event Bus and Lifecycle Hooks System for ARTEMIS."""
|
||||
|
||||
from collections import defaultdict
|
||||
from collections.abc import Callable
|
||||
from enum import Enum
|
||||
import inspect
|
||||
import time
|
||||
from typing import Any
|
||||
from pydantic import BaseModel, Field
|
||||
from artemis.utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class HookType(str, Enum):
|
||||
BEFORE_TASK = "before_task"
|
||||
AFTER_TASK = "after_task"
|
||||
BEFORE_STEP = "before_step"
|
||||
AFTER_STEP = "after_step"
|
||||
ON_ACTION = "on_action"
|
||||
ON_ERROR = "on_error"
|
||||
ON_SCREEN_CHANGE = "on_screen_change"
|
||||
|
||||
|
||||
class Event(BaseModel):
|
||||
"""Encapsulates an event payload emitted through the event bus."""
|
||||
|
||||
event_type: HookType = Field(..., description="Type of the lifecycle event")
|
||||
timestamp: float = Field(default_factory=time.time, description="Event occurrence timestamp")
|
||||
trace_id: str = Field(..., description="Active session trace identifier")
|
||||
payload: dict[str, Any] = Field(default_factory=dict, description="Event data dictionary")
|
||||
|
||||
|
||||
class EventBus:
|
||||
"""Asynchronous event bus supporting subscription and broadcast of lifecycle events."""
|
||||
|
||||
def __init__(self):
|
||||
self._subscribers: dict[HookType, list[Callable[..., Any]]] = defaultdict(list)
|
||||
|
||||
def subscribe(self, hook: HookType, callback: Callable[..., Any]) -> None:
|
||||
"""Registers a listener callback for a specific lifecycle event."""
|
||||
self._subscribers[hook].append(callback)
|
||||
|
||||
def unsubscribe(self, hook: HookType, callback: Callable[..., Any]) -> None:
|
||||
"""Removes a registered callback."""
|
||||
if callback in self._subscribers[hook]:
|
||||
self._subscribers[hook].remove(callback)
|
||||
|
||||
async def emit(self, event: Event) -> None:
|
||||
"""Asynchronously dispatches an event to all subscribed listeners."""
|
||||
listeners = self._subscribers.get(event.event_type, [])
|
||||
for listener in listeners:
|
||||
try:
|
||||
if inspect.iscoroutinefunction(listener):
|
||||
await listener(event)
|
||||
else:
|
||||
listener(event)
|
||||
except Exception as e:
|
||||
logger.warning(f"Error executing event listener for {event.event_type}: {e}")
|
||||
|
||||
|
||||
# Global singleton event bus instance
|
||||
global_event_bus = EventBus()
|
||||
@@ -0,0 +1,57 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Standardized exception hierarchy for ARTEMIS."""
|
||||
|
||||
|
||||
class ArtemisException(Exception):
|
||||
"""Base exception for all ARTEMIS framework errors."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class DeviceDriverException(ArtemisException):
|
||||
"""Raised when hardware, emulator, or ADB interactions fail."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class DeviceNotFoundException(DeviceDriverException):
|
||||
"""Raised when the specified target device cannot be located or connected."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class AgentExecutionException(ArtemisException):
|
||||
"""Raised when an autonomous agent fails during reasoning or execution."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class ToolExecutionException(ArtemisException):
|
||||
"""Raised when a tool action fails or receives invalid arguments."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class LLMProviderException(ArtemisException):
|
||||
"""Raised when an upstream LLM/VLM API call fails or times out."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class StateTransitionException(ArtemisException):
|
||||
"""Raised when graph execution encounters an invalid state transition."""
|
||||
|
||||
pass
|
||||
@@ -0,0 +1,46 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Global Plugin and Component Registry for Agents, Tools, and Drivers."""
|
||||
|
||||
from typing import Any, TypeVar
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
class ComponentRegistry:
|
||||
"""Generic typed component registry supporting runtime dynamic plugin loading."""
|
||||
|
||||
def __init__(self, name: str):
|
||||
self.name = name
|
||||
self._items: dict[str, Any] = {}
|
||||
|
||||
def register(self, key: str, item: Any) -> Any:
|
||||
self._items[key.lower()] = item
|
||||
return item
|
||||
|
||||
def get(self, key: str) -> Any | None:
|
||||
return self._items.get(key.lower())
|
||||
|
||||
def list_all(self) -> dict[str, Any]:
|
||||
return dict(self._items)
|
||||
|
||||
def contains(self, key: str) -> bool:
|
||||
return key.lower() in self._items
|
||||
|
||||
|
||||
# Global Registries
|
||||
AgentRegistry = ComponentRegistry("Agents")
|
||||
ToolRegistry = ComponentRegistry("Tools")
|
||||
DriverRegistry = ComponentRegistry("Drivers")
|
||||
@@ -0,0 +1,70 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Core Execution State and Lifecycle Data Structures."""
|
||||
|
||||
from enum import Enum
|
||||
import time
|
||||
from typing import Any
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class ExecutionStatus(str, Enum):
|
||||
PENDING = "pending"
|
||||
RUNNING = "running"
|
||||
SUCCESS = "success"
|
||||
FAILED = "failed"
|
||||
PAUSED = "paused"
|
||||
INTERRUPTED = "interrupted"
|
||||
|
||||
|
||||
class StepRecord(BaseModel):
|
||||
"""Encapsulates a single step's observation, thought, and action."""
|
||||
|
||||
step_number: int = Field(..., description="1-indexed step turn counter")
|
||||
thought: str | None = Field(default=None, description="Reasoning text")
|
||||
action_name: str | None = Field(default=None, description="Triggered action name")
|
||||
action_params: dict[str, Any] = Field(default_factory=dict, description="Action arguments")
|
||||
result: str | None = Field(default=None, description="Action execution result output")
|
||||
screenshot_path: str | None = Field(default=None, description="Saved step screenshot file URI")
|
||||
timestamp: float = Field(default_factory=time.time, description="Timestamp of step start")
|
||||
duration_seconds: float = Field(default=0.0, description="Step processing duration in seconds")
|
||||
|
||||
|
||||
class ExecutionContextState(BaseModel):
|
||||
"""Full execution state passed across agents and runners."""
|
||||
|
||||
task_goal: str = Field(..., description="Initial user task objective")
|
||||
trace_id: str = Field(..., description="Unique task trace identifier")
|
||||
status: ExecutionStatus = Field(
|
||||
default=ExecutionStatus.RUNNING, description="Current execution lifecycle status"
|
||||
)
|
||||
current_turn: int = Field(default=0, description="Current step turn count")
|
||||
max_turns: int = Field(default=30, description="Maximum allowed step turns before timeout")
|
||||
steps: list[StepRecord] = Field(
|
||||
default_factory=list, description="Chronological execution step history"
|
||||
)
|
||||
active_plan: list[str] = Field(
|
||||
default_factory=list, description="Decomposed milestone plan items"
|
||||
)
|
||||
current_step_index: int = Field(default=0, description="Current index within active_plan")
|
||||
metadata: dict[str, Any] = Field(
|
||||
default_factory=dict, description="Custom metadata and state parameters"
|
||||
)
|
||||
error_message: str | None = Field(
|
||||
default=None, description="Failure reason if status is FAILED"
|
||||
)
|
||||
final_output: Any | None = Field(
|
||||
default=None, description="Final synthesized structured output from Outputter or runner"
|
||||
)
|
||||
@@ -0,0 +1,46 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Standardized tool declaration adhering to JSON Schema format."""
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
class ToolDeclaration(dict):
|
||||
"""Standardized tool declaration adhering to JSON Schema format with property accessors."""
|
||||
|
||||
def __init__(self, name: str, description: str, parameters: dict[str, Any]):
|
||||
super().__init__(
|
||||
type="function",
|
||||
function={
|
||||
"name": name,
|
||||
"description": description,
|
||||
"parameters": parameters,
|
||||
},
|
||||
name=name,
|
||||
description=description,
|
||||
parameters=parameters,
|
||||
)
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return self.get("name") or self.get("function", {}).get("name", "")
|
||||
|
||||
@property
|
||||
def description(self) -> str:
|
||||
return self.get("description") or self.get("function", {}).get("description", "")
|
||||
|
||||
@property
|
||||
def parameters(self) -> dict[str, Any]:
|
||||
return self.get("parameters") or self.get("function", {}).get("parameters", {})
|
||||
@@ -0,0 +1,24 @@
|
||||
# Copyright 2026 Google LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""Context variables for Artemis DataEngine runtime tracing."""
|
||||
|
||||
import contextvars
|
||||
|
||||
CURRENT_TRACE_ID: contextvars.ContextVar[str | None] = contextvars.ContextVar(
|
||||
"current_trace_id", default=None
|
||||
)
|
||||
CURRENT_NODE_NAME: contextvars.ContextVar[str | None] = contextvars.ContextVar(
|
||||
"current_node_name", default=None
|
||||
)
|
||||
Loaded 100 of 229 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user