feat: introduce MCP server, cross-platform runtime, and Showcase UI reorganization
- Showcase UI Reorganization: * Move web UI dashboard from root directory into apps/showcase_ui/ for cleaner project structure * Add 'artemis ui' CLI command (artemis/interfaces/cli/commands/ui.py) and launcher scripts for the interactive showcase console - Standalone MCP Server & Notifiers (mcp_server/): * Implement dedicated MCP server package with mobile automation tools (device_state, inspect_trace, task_manager, task_runner) * Add multi-channel notifiers (AgentAPI, Desktop notifications, Webhooks, File output, and Composite routing) * Introduce background task runner and trace store for autonomous mobile execution monitoring - Cross-Platform Runtime & Toolchain Support: * Add artemis/platform/ with OS-specific implementations for Linux, macOS (Darwin), and Windows * Add artemis/runtime/supervisor.py for resilient process and agent runtime supervision * Add artemis/toolchain/ with dependency resolution and SDK descriptors * Add cross-platform launcher scripts (start.sh, start.bat) and installer scripts (scripts/install_deps.*) - Core Enhancements & Diagnostics: * Enhance coordinate normalization, OCR API fallback, and perception pipeline * Add credential validation (credentials_validator.py) and system diagnostics * Update bilingual documentation (README.md, README_CN.md) and architecture diagrams - Test Suite Expansion: * Add integration tests for cloud gateway, remote ADB, and Artemis login * Add unit tests covering core diagnostics, platform detection, toolchain, credentials validator, optional OCR, and MCP tools
This commit is contained in:
somew1nd committed
1 parent
78e478f502
commit
9a5aaf99b7
249 files changed
+37226
-9339
No files matched your search
@@ -1,16 +0,0 @@
|
|||||||
.git
|
|
||||||
.venv
|
|
||||||
__pycache__
|
|
||||||
*.pyc
|
|
||||||
traces/
|
|
||||||
latte-traces/
|
|
||||||
debug/replay/
|
|
||||||
debug_pub/replay/
|
|
||||||
*.mp4
|
|
||||||
*.jpg
|
|
||||||
*.png
|
|
||||||
pytest_temp/
|
|
||||||
/scratch/
|
|
||||||
android_world*
|
|
||||||
/dist/
|
|
||||||
node_modules/
|
|
||||||
+18
-13
@@ -1,19 +1,24 @@
|
|||||||
# ==============================================================================
|
# ==============================================================================
|
||||||
# ☕ Latte Environment Configuration
|
# Artemis Environment Configuration Template
|
||||||
# ==============================================================================
|
# ==============================================================================
|
||||||
|
# Copy this file to .env and fill in your API credentials:
|
||||||
|
# cp .env.example .env
|
||||||
|
|
||||||
# 1. Primary LLM Provider API Key (Required for Agent Planning & Reasoning)
|
# LLM Providers (Configure at least one)
|
||||||
GEMINI_API_KEY=your_gemini_api_key_here
|
GEMINI_API_KEY=
|
||||||
|
GOOGLE_API_KEY=
|
||||||
|
OPENAI_API_KEY=
|
||||||
|
ANTHROPIC_API_KEY=
|
||||||
|
OPEN_ROUTER_API_KEY=
|
||||||
|
XAI_API_KEY=
|
||||||
|
|
||||||
# 2. Google Cloud Vision OCR API Key (Required for fast text perception & XML fusion)
|
# Google Cloud Vision OCR (Optional - for advanced OCR processing)
|
||||||
OCR_API_KEY=your_google_cloud_vision_api_key_here
|
OCR_API_KEY=
|
||||||
|
|
||||||
# Optional: Alternative LLM Providers
|
# Android ADB Connectivity (Optional - defaults to 127.0.0.1:5037)
|
||||||
# OPENAI_API_KEY=your_openai_api_key_here
|
ADB_HOST=127.0.0.1
|
||||||
# ANTHROPIC_API_KEY=your_anthropic_api_key_here
|
ADB_PORT=5037
|
||||||
# OPEN_ROUTER_API_KEY=your_openrouter_api_key_here
|
# ADB_DEVICE_SERIAL=
|
||||||
# XAI_API_KEY=your_xai_api_key_here
|
|
||||||
|
|
||||||
# Optional: Target Android Device Serial (Leave commented for auto-detection)
|
|
||||||
# ADB_DEVICE_SERIAL=emulator-5554
|
|
||||||
|
|
||||||
|
# Artemis Execution Defaults (flash or pro)
|
||||||
|
ARTEMIS_DEFAULT_PROFILE=flash
|
||||||
@@ -1,17 +0,0 @@
|
|||||||
.gcloudignore
|
|
||||||
.git
|
|
||||||
.venv/
|
|
||||||
__pycache__/
|
|
||||||
*.pyc
|
|
||||||
traces/
|
|
||||||
latte-traces/
|
|
||||||
debug/replay/
|
|
||||||
debug_pub/replay/
|
|
||||||
*.mp4
|
|
||||||
*.jpg
|
|
||||||
*.png
|
|
||||||
pytest_temp/
|
|
||||||
/scratch/
|
|
||||||
android_world*
|
|
||||||
/dist/
|
|
||||||
!frontend/dist/**
|
|
||||||
+73
-83
@@ -1,109 +1,99 @@
|
|||||||
# ==============================================================================
|
# Python
|
||||||
# ☕ Latte Framework - Master Git Ignore Configuration
|
__pycache__/
|
||||||
# ==============================================================================
|
*.py[cod]
|
||||||
|
*$py.class
|
||||||
# --- 1. Environment Variables & Virtual Environments (Recursive) ---
|
*.so
|
||||||
.env
|
.Python
|
||||||
.env.*
|
build/
|
||||||
!.env.example
|
|
||||||
uv.lock
|
|
||||||
**/.venv/
|
|
||||||
**/venv/
|
|
||||||
**/ENV/
|
|
||||||
**/env/
|
|
||||||
**/.pyenv/
|
|
||||||
.python-version
|
|
||||||
|
|
||||||
# --- 2. Node & Web Frontend Dependencies / Artifacts (Recursive) ---
|
|
||||||
**/node_modules/
|
|
||||||
**/node_modules
|
|
||||||
**/.angular/
|
|
||||||
**/.angular/cache/
|
|
||||||
npm-debug.log*
|
|
||||||
yarn-debug.log*
|
|
||||||
yarn-error.log*
|
|
||||||
|
|
||||||
# --- 3. Python Caches, Bytecode & Build Artifacts (Recursive) ---
|
|
||||||
**/__pycache__/
|
|
||||||
**/*.py[cod]
|
|
||||||
**/*$py.class
|
|
||||||
**/*.so
|
|
||||||
**/*.pyd
|
|
||||||
**/*.c
|
|
||||||
**/build/
|
|
||||||
**/dist/
|
|
||||||
**/*.egg-info/
|
|
||||||
**/.eggs/
|
|
||||||
develop-eggs/
|
develop-eggs/
|
||||||
|
dist/
|
||||||
downloads/
|
downloads/
|
||||||
eggs/
|
eggs/
|
||||||
|
.eggs/
|
||||||
lib/
|
lib/
|
||||||
lib64/
|
lib64/
|
||||||
parts/
|
parts/
|
||||||
sdist/
|
sdist/
|
||||||
var/
|
var/
|
||||||
wheels/
|
wheels/
|
||||||
|
share/python-wheels/
|
||||||
|
*.egg-info/
|
||||||
.installed.cfg
|
.installed.cfg
|
||||||
*.egg
|
*.egg
|
||||||
MANIFEST
|
MANIFEST
|
||||||
|
|
||||||
# --- 4. Testing, Coverage & Linters (Recursive) ---
|
# Virtual Environments
|
||||||
**/.pytest_cache/
|
.venv/
|
||||||
**/.ruff_cache/
|
env/
|
||||||
**/.mypy_cache/
|
venv/
|
||||||
**/.pytype/
|
ENV/
|
||||||
|
env.bak/
|
||||||
|
venv.bak/
|
||||||
|
|
||||||
|
# Testing & Coverage
|
||||||
|
.pytest_cache/
|
||||||
.coverage
|
.coverage
|
||||||
.coverage.*
|
.coverage.*
|
||||||
.cache
|
.cache
|
||||||
coverage.xml
|
|
||||||
nosetests.xml
|
nosetests.xml
|
||||||
htmlcov/
|
coverage.xml
|
||||||
pytest_temp/
|
*.cover
|
||||||
tests_tmp/
|
*.py,cover
|
||||||
|
.hypothesis/
|
||||||
|
.tox/
|
||||||
|
.nox/
|
||||||
tests/**/outputs/
|
tests/**/outputs/
|
||||||
|
**/outputs/
|
||||||
|
|
||||||
# --- 5. Runtime Logs, Spans, Replay & Traces (Recursive) ---
|
# Linters & Formatters
|
||||||
traces/
|
.ruff_cache/
|
||||||
traces_test/
|
.mypy_cache/
|
||||||
latte-traces/
|
.pyre/
|
||||||
data_engine.db*
|
.pytype/
|
||||||
**/replay/outputs/
|
.dmypy.json
|
||||||
**/replay/data/
|
dmypy.json
|
||||||
**/*.log
|
|
||||||
**/*.log.*
|
|
||||||
**/*.out
|
|
||||||
**/*.tmp
|
|
||||||
.latte_ipc_port
|
|
||||||
.latte_paused
|
|
||||||
.latte/
|
|
||||||
.artemis_ipc_port
|
|
||||||
.artemis_paused
|
|
||||||
.artemis/
|
|
||||||
*.bak
|
|
||||||
live_screenshot*
|
|
||||||
recording.mp4
|
|
||||||
recording.mkv
|
|
||||||
screenshot.jpg
|
|
||||||
screen.png
|
|
||||||
|
|
||||||
# --- 6. Workspace Scratch, Dumps & Temporary Experimentation ---
|
# IDEs & Editors
|
||||||
scratch/
|
|
||||||
artifacts/
|
|
||||||
ignores/
|
|
||||||
*.docx
|
|
||||||
*gdoc.html
|
|
||||||
*.bug.log
|
|
||||||
|
|
||||||
# --- 7. AI Agent Workspace & IDE Metadata ---
|
|
||||||
.agents/
|
|
||||||
_agents/
|
|
||||||
jetski/
|
|
||||||
.jetski*
|
|
||||||
.benchmarks/
|
|
||||||
.idea/
|
|
||||||
.vscode/
|
.vscode/
|
||||||
|
.idea/
|
||||||
*.swp
|
*.swp
|
||||||
*.swo
|
*.swo
|
||||||
*~
|
*~
|
||||||
|
.project
|
||||||
|
.pydevproject
|
||||||
|
.settings/
|
||||||
|
|
||||||
|
# Node & Frontend (Showcase UI)
|
||||||
|
apps/showcase_ui/node_modules/
|
||||||
|
apps/showcase_ui/.angular/
|
||||||
|
apps/showcase_ui/.angular/cache/
|
||||||
|
.npm/
|
||||||
|
npm-debug.log*
|
||||||
|
yarn-debug.log*
|
||||||
|
yarn-error.log*
|
||||||
|
|
||||||
|
# Artemis Runtime & Telemetry
|
||||||
|
traces/
|
||||||
|
scratch/
|
||||||
|
.artemis_paused
|
||||||
|
*.log
|
||||||
|
data_engine.db
|
||||||
|
data_engine.db-journal
|
||||||
|
.ipc_port
|
||||||
|
.ls_address
|
||||||
|
|
||||||
|
# Environment & Secrets
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
|
!.env.example
|
||||||
|
*.pem
|
||||||
|
*.key
|
||||||
|
|
||||||
|
# OS Artifacts
|
||||||
.DS_Store
|
.DS_Store
|
||||||
|
.DS_Store?
|
||||||
|
._*
|
||||||
|
.Spotlight-V100
|
||||||
|
.Trashes
|
||||||
|
ehthumbs.db
|
||||||
Thumbs.db
|
Thumbs.db
|
||||||
+10
-95
@@ -1,103 +1,18 @@
|
|||||||
# How to Contribute
|
# Contributing to ARTEMIS
|
||||||
|
|
||||||
We'd love to accept your patches and contributions to ARTEMIS! There are just a few small guidelines you need to follow.
|
Thank you for your interest in contributing to **ARTEMIS**! We welcome and appreciate all forms of contributions—whether it's reporting bugs, proposing new features, improving documentation, or submitting code.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Contributor License Agreement (CLA)
|
## 🤝 Ways to Contribute
|
||||||
|
|
||||||
Contributions to this project must be accompanied by a Contributor License Agreement (CLA). You (or your employer) retain the copyright to your contribution; this simply gives us permission to use and redistribute your contributions as part of the project.
|
- **🐛 Report Bugs & Suggest Features**: If you find an issue or have an idea to improve ARTEMIS, please open an [Issue](https://github.com/google/artemis/issues).
|
||||||
|
- **💡 Submit Pull Requests**:
|
||||||
- Head over to <https://cla.developers.google.com/> to check your current agreements on file or to sign a new one.
|
1. Fork the repository and create your feature/fix branch (`git checkout -b feat/my-feature`).
|
||||||
- If you are an individual contributing original source code, you must sign the **Individual CLA**.
|
2. Make your changes and verify with `make test` and `make lint`.
|
||||||
- If you are contributing on behalf of a company or organization, a **Corporate CLA** must be executed by your employer.
|
3. Open a Pull Request with a brief summary of your work.
|
||||||
- You generally only need to submit a CLA once, so if you have already submitted one for another Google project, you do not need to sign it again.
|
- **💬 Community & Discussion**: Join our [Discord Community](https://discord.gg/wF2FN4WHGY) to chat with maintainers, ask questions, or share feedback.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Community Guidelines
|
Every contribution—big or small—helps make ARTEMIS better. Thank you for building with us!
|
||||||
|
|
||||||
This project adheres to [Google's Open Source Community Guidelines](https://opensource.google/conduct/). By participating, all contributors and maintainers are expected to uphold these principles and maintain a welcoming, inclusive, and professional environment.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Code Reviews
|
|
||||||
|
|
||||||
All submissions, including submissions by project maintainers, require review. We use standard GitHub pull requests for this purpose:
|
|
||||||
|
|
||||||
1. Consult [GitHub Help](https://docs.github.com/articles/about-pull-requests/) for information on using pull requests.
|
|
||||||
2. Maintainers will review your changes, offer feedback, and approve when ready.
|
|
||||||
3. Once approved and all CI checks pass, a project maintainer will merge your PR.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Development & Contribution Workflow
|
|
||||||
|
|
||||||
### 1. Setting Up Your Development Environment
|
|
||||||
|
|
||||||
- **Prerequisites**:
|
|
||||||
- Python **3.12+**
|
|
||||||
- Android SDK Platform Tools (`adb` available in `PATH`)
|
|
||||||
- [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip`
|
|
||||||
|
|
||||||
- **Clone & Install**:
|
|
||||||
```bash
|
|
||||||
git clone https://github.com/<your-username>/artemis.git
|
|
||||||
cd artemis
|
|
||||||
|
|
||||||
# Install all dependencies and setup pre-commit hooks
|
|
||||||
make setup
|
|
||||||
```
|
|
||||||
*(Alternatively: `uv sync --dev` and `uv run pre-commit install`)*
|
|
||||||
|
|
||||||
### 2. Making Changes
|
|
||||||
|
|
||||||
- Create a descriptive topic branch off `main`:
|
|
||||||
```bash
|
|
||||||
git checkout -b feat/my-new-feature
|
|
||||||
# or
|
|
||||||
git checkout -b fix/issue-description
|
|
||||||
```
|
|
||||||
|
|
||||||
- Follow the project's coding and style conventions:
|
|
||||||
- **Absolute Imports**: Always use absolute imports (`from artemis.sdk import Agent`). Relative imports are disallowed.
|
|
||||||
- **Typing**: Use comprehensive type annotations on all public functions, classes, and methods.
|
|
||||||
- **Testing**: Add or update test cases under `tests/` for any modified or new behavior.
|
|
||||||
|
|
||||||
### 3. Local Verification
|
|
||||||
|
|
||||||
Before creating a pull request, ensure all linters, type checks, and tests pass:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Format code (Ruff)
|
|
||||||
make format
|
|
||||||
|
|
||||||
# Lint code (Ruff)
|
|
||||||
make lint
|
|
||||||
|
|
||||||
# Run static type checker (Pyright)
|
|
||||||
make typecheck
|
|
||||||
|
|
||||||
# Run test suite (Pytest)
|
|
||||||
make test
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. Submitting a Pull Request
|
|
||||||
|
|
||||||
1. Push your branch to your fork:
|
|
||||||
```bash
|
|
||||||
git push origin feat/my-new-feature
|
|
||||||
```
|
|
||||||
2. Open a Pull Request against the `main` branch of the ARTEMIS repository.
|
|
||||||
3. Complete the PR template, describing:
|
|
||||||
- The motivation behind the change.
|
|
||||||
- Any related issue numbers (e.g., `Fixes #42`).
|
|
||||||
- How the change was tested (device, emulator, unit tests).
|
|
||||||
4. Ensure the CLA check automatically passes on your PR.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Reporting Bugs and Feature Requests
|
|
||||||
|
|
||||||
- **Bug Reports**: Open an issue describing the bug, including steps to reproduce, device/emulator specifications, Android version, logs, and screenshots if applicable.
|
|
||||||
- **Feature Requests**: Open an issue detailing the use case, proposed API or workflow, and alternatives considered.
|
|
||||||
+18
-41
@@ -1,3 +1,17 @@
|
|||||||
|
# Copyright 2026 Google LLC
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
FROM python:3.12-slim
|
FROM python:3.12-slim
|
||||||
|
|
||||||
WORKDIR /app
|
WORKDIR /app
|
||||||
@@ -10,52 +24,15 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
|||||||
adb \
|
adb \
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
# Copy application source code and apps
|
# Copy application source code and configuration
|
||||||
COPY artemis/ /app/artemis/
|
COPY artemis/ /app/artemis/
|
||||||
COPY apps/ /app/apps/
|
COPY apps/ /app/apps/
|
||||||
|
COPY mcp_server/ /app/mcp_server/
|
||||||
COPY config/ /app/config/
|
COPY config/ /app/config/
|
||||||
COPY pyproject.toml setup.py README.md LICENSE /app/
|
COPY pyproject.toml setup.py README.md LICENSE /app/
|
||||||
|
|
||||||
# Install server and core dependencies
|
# Install dependencies and artemis package
|
||||||
RUN pip install --no-cache-dir \
|
RUN pip install --no-cache-dir -e /app
|
||||||
uvicorn \
|
|
||||||
fastapi \
|
|
||||||
pydantic \
|
|
||||||
pydantic-settings \
|
|
||||||
python-dotenv \
|
|
||||||
sseclient-py \
|
|
||||||
websockets \
|
|
||||||
adbutils \
|
|
||||||
jinja2 \
|
|
||||||
colorama \
|
|
||||||
psutil \
|
|
||||||
httpx \
|
|
||||||
retry \
|
|
||||||
google-auth \
|
|
||||||
langchain \
|
|
||||||
langchain-core \
|
|
||||||
langchain-community \
|
|
||||||
langchain-google-genai \
|
|
||||||
langchain-google-vertexai \
|
|
||||||
langchain-mcp-adapters \
|
|
||||||
google-genai \
|
|
||||||
langgraph \
|
|
||||||
typer \
|
|
||||||
pillow \
|
|
||||||
packaging \
|
|
||||||
uiautomator2 \
|
|
||||||
opencv-python-headless \
|
|
||||||
uuid-utils \
|
|
||||||
posthog \
|
|
||||||
mcp \
|
|
||||||
inquirer \
|
|
||||||
rich \
|
|
||||||
requests \
|
|
||||||
imageio-ffmpeg \
|
|
||||||
langchain-openai
|
|
||||||
|
|
||||||
# Install artemis package locally (no-deps to prevent pulling unnecessary heavy optional libraries)
|
|
||||||
RUN pip install --no-cache-dir --no-deps -e /app
|
|
||||||
|
|
||||||
ENV PYTHONUNBUFFERED=1
|
ENV PYTHONUNBUFFERED=1
|
||||||
ENV PYTHONPATH=/app
|
ENV PYTHONPATH=/app
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
|
|
||||||
Apache License
|
Apache License
|
||||||
Version 2.0, January 2004
|
Version 2.0, January 2004
|
||||||
http://www.apache.org/licenses/
|
https://www.apache.org/licenses/
|
||||||
|
|
||||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||||
|
|
||||||
@@ -187,16 +187,16 @@
|
|||||||
same "printed page" as the copyright notice for easier
|
same "printed page" as the copyright notice for easier
|
||||||
identification within third-party archives.
|
identification within third-party archives.
|
||||||
|
|
||||||
Copyright [yyyy] [name of copyright owner]
|
Copyright 2026 Google LLC.
|
||||||
|
|
||||||
Licensed under the Apache License, Version 2.0 (the "License");
|
Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
you may not use this file except in compliance with the License.
|
you may not use this file except in compliance with the License.
|
||||||
You may obtain a copy of the License at
|
You may obtain a copy of the License at
|
||||||
|
|
||||||
http://www.apache.org/licenses/LICENSE-2.0
|
https://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
Unless required by applicable law or agreed to in writing, software
|
Unless required by applicable law or agreed to in writing, software
|
||||||
distributed under the License is distributed on an "AS IS" BASIS,
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
See the License for the specific language governing permissions and
|
See the License for the specific language governing permissions and
|
||||||
limitations under the License.
|
limitations under the License.
|
||||||
@@ -1,4 +1,18 @@
|
|||||||
.PHONY: help test install setup clean precommit-install precommit lint format typecheck
|
# Copyright 2026 Google LLC
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
.PHONY: help test install install-deps setup start ui doctor clean precommit-install precommit lint format typecheck
|
||||||
|
|
||||||
help: ## Show this help message
|
help: ## Show this help message
|
||||||
@echo 'Usage: make [target]'
|
@echo 'Usage: make [target]'
|
||||||
@@ -6,14 +20,27 @@ help: ## Show this help message
|
|||||||
@echo 'Available targets:'
|
@echo 'Available targets:'
|
||||||
@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf " %-20s %s\n", $$1, $$2}' $(MAKEFILE_LIST)
|
@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf " %-20s %s\n", $$1, $$2}' $(MAKEFILE_LIST)
|
||||||
|
|
||||||
|
start: ## One-click start Artemis Showcase UI and auto-open browser
|
||||||
|
@bash start.sh
|
||||||
|
|
||||||
|
ui: ## Launch the unified Showcase UI & Admin Console in browser
|
||||||
|
@uv run artemis ui --open
|
||||||
|
|
||||||
|
doctor: ## Run system, device, and toolchain diagnostics
|
||||||
|
@uv run artemis doctor
|
||||||
|
|
||||||
test: ## Run all tests
|
test: ## Run all tests
|
||||||
@echo "🧪 Running all tests..."
|
@echo "🧪 Running all tests..."
|
||||||
@uv run pytest -v
|
@uv run pytest -v
|
||||||
|
|
||||||
install: ## Install dependencies
|
install: ## Install python dependencies via uv
|
||||||
@echo "📦 Installing dependencies..."
|
@echo "📦 Installing python dependencies..."
|
||||||
@uv sync --dev
|
@uv sync --dev
|
||||||
|
|
||||||
|
install-deps: ## One-click install all system dependencies (ADB, FFmpeg, scrcpy, Python, uv)
|
||||||
|
@echo "⚡ Running one-click dependency installer..."
|
||||||
|
@bash scripts/install_deps.sh
|
||||||
|
|
||||||
setup: ## Setup project (install dependencies + pre-commit hooks)
|
setup: ## Setup project (install dependencies + pre-commit hooks)
|
||||||
@echo "🚀 Setting up project..."
|
@echo "🚀 Setting up project..."
|
||||||
@$(MAKE) install
|
@$(MAKE) install
|
||||||
@@ -47,10 +74,12 @@ precommit: ## Run pre-commit hooks manually on all files
|
|||||||
@echo "🔍 Running pre-commit checks..."
|
@echo "🔍 Running pre-commit checks..."
|
||||||
@uv run pre-commit run --all-files
|
@uv run pre-commit run --all-files
|
||||||
|
|
||||||
clean: ## Clean up generated files and caches
|
clean: ## Clean up generated files, caches, and traces
|
||||||
@echo "🧹 Cleaning up..."
|
@echo "🧹 Cleaning up caches and temporary files..."
|
||||||
@rm -rf .pytest_cache
|
@rm -rf .pytest_cache .ruff_cache .artemis_paused traces scratch .venv apps/showcase_ui/.angular build dist
|
||||||
@rm -rf .ruff_cache
|
@find . -type d -name "outputs" -exec rm -rf {} + 2>/dev/null || true
|
||||||
@rm -rf **/__pycache__
|
@find . -type f -name ".artemis_paused" -delete 2>/dev/null || true
|
||||||
|
@find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
|
||||||
@find . -type d -name "*.egg-info" -exec rm -rf {} + 2>/dev/null || true
|
@find . -type d -name "*.egg-info" -exec rm -rf {} + 2>/dev/null || true
|
||||||
|
@find . -type f -name "*.py[cod]" -delete 2>/dev/null || true
|
||||||
@echo "✅ Cleanup complete"
|
@echo "✅ Cleanup complete"
|
||||||
@@ -1,398 +1,248 @@
|
|||||||
# ARTEMIS ☕📱
|
<p align="center">
|
||||||
|
<img src="./docs/assets/artemis-banner.png" alt="ARTEMIS Banner" width="100%" />
|
||||||
|
</p>
|
||||||
|
|
||||||
**Autonomous Multimodal Android Agent & Mobile UI Automation Platform**
|
<p align="center">
|
||||||
|
<strong>ARTEMIS: Autonomous AI Assistant for Mobile Automation</strong>
|
||||||
|
</p>
|
||||||
|
|
||||||
[](https://www.python.org/downloads/)
|
<p align="center">
|
||||||
[](LICENSE)
|
<em>⚡ Drive real phones from Cursor & Claude Code • Cross-App Automation • Zero-Maintenance UI Testing • Flash Mode as Fast as 3-5s</em>
|
||||||
[](https://github.com/langchain-ai/langgraph)
|
</p>
|
||||||
[](https://ai.google.dev/)
|
|
||||||
|
|
||||||
ARTEMIS is a next-generation, production-grade autonomous agent and testing framework engineered for Android devices and emulators. Combining state-of-the-art multimodal Large Vision-Language Models (VLMs) with hierarchical closed-loop planning, resilient self-healing, and deep Android subsystem integration (ADB, UIAutomator2, accessibility trees), ARTEMIS enables autonomous UI navigation, robust automated testing, and intelligent device diagnostics.
|
<p align="center">
|
||||||
|
<a href="./README.md"><b>English</b></a> •
|
||||||
|
<a href="./README_CN.md">中文文档</a> •
|
||||||
|
<a href="#quick-start">Quick Start</a> •
|
||||||
|
<a href="#mcp-setup">MCP for IDEs</a> •
|
||||||
|
<a href="#benchmarks">Benchmarks</a> •
|
||||||
|
<a href="https://discord.gg/wF2FN4WHGY">Discord Community</a>
|
||||||
|
</p>
|
||||||
|
|
||||||
---
|
<p align="center">
|
||||||
|
<a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/Python-3.12+-3776AB.svg?logo=python&logoColor=white" alt="Python 3.12+"></a>
|
||||||
|
<a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License: Apache-2.0"></a>
|
||||||
|
<a href="https://modelcontextprotocol.io/"><img src="https://img.shields.io/badge/MCP-Native%20Server-8A2BE2.svg" alt="MCP Native"></a>
|
||||||
|
<a href="https://ai.google.dev/"><img src="https://img.shields.io/badge/Multimodal-Gemini%20%7C%20Claude%20%7C%20GPT--4o%20%7C%20Qwen--VL-4285F4.svg" alt="Multi-Model"></a>
|
||||||
|
<a href="https://github.com/google-research/android_world"><img src="https://img.shields.io/badge/AndroidWorld-99%25%2B%20SOTA-success.svg" alt="AndroidWorld SOTA"></a>
|
||||||
|
</p>
|
||||||
|
|
||||||
## 📑 Table of Contents
|
<!-- Demo Showcase -->
|
||||||
|
<p align="center">
|
||||||
|
<img src="./docs/assets/demo.gif" alt="Artemis in Action" width="88%" />
|
||||||
|
<br>
|
||||||
|
<em><b>Live Demo</b>: Setup driving routes and calculate total durations in Google Maps, then open YouTube to play a Coldplay song.</em>
|
||||||
|
</p>
|
||||||
|
|
||||||
- [Key Features](#-key-features)
|
## ✨ Key Highlights
|
||||||
- [Architecture Overview](#-architecture-overview)
|
|
||||||
- [Prerequisites](#-prerequisites)
|
|
||||||
- [Installation & Setup](#-installation--setup)
|
|
||||||
- [Quick Start](#-quick-start)
|
|
||||||
- [Command Line Interface (CLI)](#command-line-interface-cli)
|
|
||||||
- [Python SDK](#python-sdk)
|
|
||||||
- [Execution Profiles: Flash vs. Pro](#-execution-profiles-flash-vs-pro)
|
|
||||||
- [Model Context Protocol (MCP) Integration](#-model-context-protocol-mcp-integration)
|
|
||||||
- [Configuration Guide](#-configuration-guide)
|
|
||||||
- [Using Non-Google LLMs](#using-non-google-llms)
|
|
||||||
- [Trace Inspection & Observability](#-trace-inspection--observability)
|
|
||||||
- [Development & Testing](#-development--testing)
|
|
||||||
- [License](#-license)
|
|
||||||
|
|
||||||
---
|
* ⚡ **Lightning Fast**: First-of-its-kind **Optimistic Asynchronous Pipeline** decouples the foreground execution stream from heavy LLM reasoning;
|
||||||
|
* 🔋 **Ultra-Long Autonomous Execution**: **Pro mode** runs stably for over 10 hours, while monitoring tasks support around-the-clock 24/7 execution;
|
||||||
|
* 🔌 **Native Model Context Protocol (MCP)**: Empowers **Cursor, Claude Code, OpenClaw, Windsurf** with direct mobile device manipulation from within your chat prompts;
|
||||||
|
* 💾 **Visual Console & Persistent Debugging**: Real-time Web UI screen projection, execution replay, and task management with natural language test generation and high-fidelity historical review;
|
||||||
|
* 🏆 **Industry-Leading SOTA**: Achieved **99%+ task completion** on Google Research's **AndroidWorld** benchmark (100+ complex multi-step tasks).
|
||||||
|
|
||||||
## ✨ Key Features
|
<a id="quick-start"></a>
|
||||||
|
## ⚡ Quick Start
|
||||||
|
|
||||||
- **Dual-Engine Execution**:
|
Ensure an Android device (with **USB Debugging** enabled) or emulator is connected.
|
||||||
- **ARTEMIS Flash**: High-speed, reactive single-agent loop for deterministic, low-latency tasks (< 35 steps).
|
|
||||||
- **ARTEMIS Pro**: Multi-agent closed-loop graph orchestration (Planner $\rightarrow$ Operator $\rightarrow$ Failure Analyzer $\rightarrow$ Summarizer) powered by LangGraph.
|
|
||||||
- **Multimodal Perception**:
|
|
||||||
- Combined visual screenshot grounding, real-time OCR, accessibility hierarchy (XML) parsing, and dynamic video analysis.
|
|
||||||
- **Resilient Locating Strategy ("Dynamic-First, Coordinate-Fallback")**:
|
|
||||||
- Prioritizes semantic IDs, XPath, and accessibility text matching to withstand UI layout drift; seamlessly falls back to calibrated absolute/relative coordinates.
|
|
||||||
- **Autonomous Self-Healing**:
|
|
||||||
- Built-in failure analyzer and safety net detect blocked flows, system dialogs, app crashes, or unexpected states and auto-recover in real time.
|
|
||||||
- **App Locking & Session Boundaries**:
|
|
||||||
- Restricts execution context to target application packages with automatic foreground recovery.
|
|
||||||
- **Model Context Protocol (MCP) Native**:
|
|
||||||
- Exposes Android automation, state inspection, and trace querying as MCP tools for IDEs and AI orchestrators.
|
|
||||||
- **Structured Pydantic Outputs**:
|
|
||||||
- Automatically parses agent observations and findings into validated Pydantic schemas.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🏛 Architecture Overview
|
|
||||||
|
|
||||||
ARTEMIS operates on a closed-loop multi-agent architecture:
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────┐
|
|
||||||
│ User Goal / Task │
|
|
||||||
└────────────────┬────────────────┘
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
┌───────────────────────────────┐
|
|
||||||
│ Planner Agent │
|
|
||||||
│ (Hierarchical Task Breakdown) │
|
|
||||||
└───────────────┬───────────────┘
|
|
||||||
│
|
|
||||||
┌──────────────┴──────────────┐
|
|
||||||
▼ ▼
|
|
||||||
┌─────────────────────────────────┐ ┌───────────────────────────────┐
|
|
||||||
│ Operator Agent │ │ Diagnostic Agents │
|
|
||||||
│ (Screen VLM + XML Grounding) │ │ (Log Reader, Video Analyzer, │
|
|
||||||
└────────────────┬────────────────┘ │ Object Detector, Explorer) │
|
|
||||||
│ └───────────────────────────────┘
|
|
||||||
▼
|
|
||||||
┌─────────────────────────────────┐
|
|
||||||
│ Execution & Device Control │
|
|
||||||
│ (ADB / UIAutomator2 / Taps) │
|
|
||||||
└────────────────┬────────────────┘
|
|
||||||
│
|
|
||||||
[Action Success / State Change?]
|
|
||||||
│ │
|
|
||||||
(Yes) (No / Blocked)
|
|
||||||
│ │
|
|
||||||
▼ ▼
|
|
||||||
┌──────────────────────┐ ┌─────────────────────────────────────────┐
|
|
||||||
│ Next Step / Complete │ │ Failure Analyzer & Safety Net │
|
|
||||||
└──────────┬───────────┘ │ (Popup dismissal, Retry, Self-healing) │
|
|
||||||
│ └────────────────────┬────────────────────┘
|
|
||||||
▼ │
|
|
||||||
┌──────────────────────┐ │
|
|
||||||
│ Summarizer / Output │◄──────────────────────┘
|
|
||||||
│ (Reports, Traces) │
|
|
||||||
└──────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🔧 Prerequisites
|
|
||||||
|
|
||||||
1. **Python**: Version `3.12` or higher.
|
|
||||||
2. **Android SDK / ADB**:
|
|
||||||
- `adb` must be installed and accessible in your system `PATH`.
|
|
||||||
- Verify with `adb devices`.
|
|
||||||
3. **Android Device or Emulator**:
|
|
||||||
- A physically connected Android device with **USB Debugging** enabled, or
|
|
||||||
- An active Android Emulator (AVD) running on your local machine.
|
|
||||||
4. **Screen Recording & Video Tools** *(optional for `--with-video-recording-tools`)*:
|
|
||||||
- **`scrcpy`**: Required for live Android device screen video recording (`apt install scrcpy` / `brew install scrcpy`).
|
|
||||||
- **`ffmpeg`**: Required for video transcoding, trimming, and audio extraction (`apt install ffmpeg` / `brew install ffmpeg`).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🚀 Installation & Setup
|
|
||||||
|
|
||||||
### 1. Clone the Repository
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone https://github.com/google/artemis.git
|
# 1. Clone repo & navigate to directory
|
||||||
cd artemis
|
git clone https://github.com/google/artemis.git && cd artemis
|
||||||
|
|
||||||
|
# 2. One-click launch (automatically installs ADB, scrcpy, FFmpeg, uv runtime, and opens console)
|
||||||
|
# 🍎 macOS & 🐧 Linux
|
||||||
|
./start.sh
|
||||||
|
|
||||||
|
# 🪟 Windows (CMD / PowerShell)
|
||||||
|
start.bat
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. Install Dependencies
|
> 💡 **Tip**: Opens `http://localhost:8000` in your default browser with a device connection wizard, live screen mirroring, prompt sandbox, and execution replays. You can also run directly from CLI: `artemis run "Open Settings, find Battery and tell me current level" --profile flash`.
|
||||||
|
|
||||||
Using [`uv`](https://docs.astral.sh/uv/) (recommended):
|
<a id="mcp-setup"></a>
|
||||||
|
<a id="mcp"></a>
|
||||||
|
<details>
|
||||||
|
<summary><b>🔌 MCP Setup for Cursor / Claude Code / Windsurf (Click to expand)</b></summary>
|
||||||
|
|
||||||
|
<br>
|
||||||
|
|
||||||
|
ARTEMIS includes a native **Model Context Protocol (MCP)** server. Connect your real phone directly into AI IDEs:
|
||||||
|
|
||||||
|
### 1. Generate MCP Config
|
||||||
|
|
||||||
|
Run the built-in generator to produce ready-to-use JSON configuration:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Install dependencies into a virtual environment
|
artemis mcp --generate-config cursor
|
||||||
uv sync --dev
|
# Or generate configs for all IDEs:
|
||||||
|
artemis mcp --generate-config all
|
||||||
```
|
```
|
||||||
|
|
||||||
Or using standard `pip`:
|
### 2. Copy Config to Your IDE
|
||||||
|
|
||||||
```bash
|
* **Cursor** (`.cursor/mcp.json` or Settings ➔ MCP Servers):
|
||||||
python3 -m venv .venv
|
```json
|
||||||
source .venv/bin/activate
|
{
|
||||||
pip install -e ".[dev]"
|
"mcpServers": {
|
||||||
|
"artemis": {
|
||||||
|
"command": "python",
|
||||||
|
"args": ["-m", "mcp_server"],
|
||||||
|
"cwd": "/path/to/artemis",
|
||||||
|
"env": {
|
||||||
|
"PYTHONUNBUFFERED": "1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. Quick Setup (10 Seconds)
|
* **Claude Desktop** (`claude_desktop_config.json`):
|
||||||
|
```json
|
||||||
Run the interactive setup wizard to configure your API key and connect your device:
|
{
|
||||||
|
"mcpServers": {
|
||||||
```bash
|
"artemis": {
|
||||||
artemis init
|
"command": "python",
|
||||||
|
"args": ["-m", "mcp_server"],
|
||||||
|
"cwd": "/path/to/artemis"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
Verify your environment and connected devices anytime with the doctor diagnostic tool:
|
### 3. Prompt Your Phone in the IDE Chat
|
||||||
|
In Cursor or Claude Code, simply prompt:
|
||||||
|
> 💬 *"Build the latest changes into an APK, install it on the connected device, open the login screen with a test account, verify if there are any unexpected popups after login, and return screenshots of the final page."*
|
||||||
|
|
||||||
```bash
|
</details>
|
||||||
artemis doctor
|
|
||||||
```
|
|
||||||
|
|
||||||
*(Alternatively, copy `.env.example` to `.env` and set `GEMINI_API_KEY` for LLM reasoning and `OCR_API_KEY` for Vision perception.)*
|
<a id="python-sdk"></a>
|
||||||
|
<details>
|
||||||
|
<summary><b>🐍 Python SDK Integration (Click to expand)</b></summary>
|
||||||
|
|
||||||
---
|
<br>
|
||||||
|
|
||||||
## 💡 Quick Start
|
Embed the mobile automation engine into your Python workflows in just a few lines:
|
||||||
|
|
||||||
### Command Line Interface (CLI)
|
|
||||||
|
|
||||||
ARTEMIS installs a unified command-line tool `artemis`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# 1. Environment & Device Diagnostics
|
|
||||||
artemis doctor
|
|
||||||
|
|
||||||
# 2. Interactive Setup Wizard
|
|
||||||
artemis init
|
|
||||||
|
|
||||||
# 3. Basic task execution (ARTEMIS Pro by default)
|
|
||||||
artemis run "Open YouTube and search for Lo-Fi Hip Hop"
|
|
||||||
|
|
||||||
# 4. Fast reactive profile (ARTEMIS Flash)
|
|
||||||
artemis run "Open Settings and verify Airplane Mode is turned off" --profile flash
|
|
||||||
|
|
||||||
# 5. Execute with trace recording and structured output
|
|
||||||
artemis run "Open Maps, search for coffee shops, and extract the top 3 results" \
|
|
||||||
--test-name "coffee_search_test" \
|
|
||||||
--traces-path "./traces" \
|
|
||||||
--output-description '{"places": "list of shop names", "ratings": "list of rating scores"}'
|
|
||||||
|
|
||||||
# 6. Enable dynamic video recording analysis
|
|
||||||
artemis run "Play the latest video in the feed and verify playback" --with-video-recording-tools
|
|
||||||
```
|
|
||||||
|
|
||||||
### Python SDK
|
|
||||||
|
|
||||||
Use the fluent SDK to integrate ARTEMIS into your test suites or automation workflows:
|
|
||||||
|
|
||||||
```python
|
```python
|
||||||
import asyncio
|
import asyncio
|
||||||
from pydantic import BaseModel, Field
|
from artemis.interfaces.sdk import ArtemisClient
|
||||||
from artemis.sdk import Agent
|
|
||||||
|
|
||||||
|
|
||||||
class VideoSearchResult(BaseModel):
|
|
||||||
query: str = Field(..., description="The search term used")
|
|
||||||
top_video_title: str = Field(..., description="Title of the first video result")
|
|
||||||
channel_name: str = Field(..., description="Channel name of the first video")
|
|
||||||
|
|
||||||
|
|
||||||
async def main():
|
async def main():
|
||||||
# 1. Initialize the agent
|
# Initialize client (choose "flash" for speed or "pro" for deep reasoning)
|
||||||
agent = Agent()
|
client = ArtemisClient(default_profile="flash")
|
||||||
await agent.init()
|
|
||||||
|
|
||||||
try:
|
# Run autonomous task
|
||||||
# 2. Build task specification
|
result = await client.run("Open calculator and compute (128 * 45) + 330")
|
||||||
task = (
|
|
||||||
agent.new_task("Search for 'Pixel 9 Pro' and get the first video details")
|
|
||||||
.with_name("youtube_search_verification")
|
|
||||||
.with_locked_app_package("com.google.android.youtube")
|
|
||||||
.with_output_format(VideoSearchResult)
|
|
||||||
.with_max_steps(50)
|
|
||||||
.build()
|
|
||||||
)
|
|
||||||
|
|
||||||
# 3. Execute task
|
print(f"Status: {result.status}")
|
||||||
result: VideoSearchResult = await agent.run_task(request=task)
|
print(f"Turns taken: {result.turns}")
|
||||||
|
print(f"Trace ID: {result.trace_id}")
|
||||||
if result:
|
|
||||||
print("Search Results:")
|
|
||||||
print(f"- Query: {result.query}")
|
|
||||||
print(f"- First Video: {result.top_video_title}")
|
|
||||||
print(f"- Channel: {result.channel_name}")
|
|
||||||
else:
|
|
||||||
print("Task did not complete successfully.")
|
|
||||||
|
|
||||||
finally:
|
|
||||||
# 4. Clean up agent session & device connections
|
|
||||||
await agent.clean()
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
asyncio.run(main())
|
asyncio.run(main())
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
</details>
|
||||||
|
|
||||||
|
## 🕹️ Usage Modes
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<img src="./docs/assets/artemis-ui-showcase-en.png" alt="Artemis Web Console" width="90%" />
|
||||||
|
<br />
|
||||||
|
<sub>💡 <b>Console Overview</b>: <b>① View Switcher</b> (Home / Workspace) · <b>② Model & Replay</b> (Flash/Pro status & video replay) · <b>③ Live Agent Stream</b> (Action perception, target coordinates & structured results) · <b>④ Prompt Dock</b> (Natural language dispatch) · <b>⑤ Task Queue & Dashboard</b> (Lifecycle & history)</sub>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
* 🖥️ **Web Visual Console (`artemis ui`)**: Real-time screen projection and interactive panel, supporting natural language task dispatch, live reasoning telemetry, action trajectories, and execution replay;
|
||||||
|
* 🔌 **Native MCP Protocol (IDE Integration)**: Operates as a standard MCP server seamlessly integrating with **Cursor, Claude Code, OpenClaw**, etc., directly invoking mobile device manipulation inside the IDE;
|
||||||
|
* 💻 **Developer CLI (`artemis run`)**: Direct terminal execution for automated tasks, test cases, or AndroidWorld benchmarks with high-fidelity structured terminal output;
|
||||||
|
* 🐍 **Python SDK**: Integrates as a standard Python library into existing automated testing frameworks or pipelines with strongly typed Pydantic structured outputs.
|
||||||
|
|
||||||
|
## 📊 Head-to-Head Comparison
|
||||||
|
|
||||||
|
| Scenario / Dimension | Traditional Automation | Generic Screen AI Agents | **ARTEMIS ☕** |
|
||||||
|
| :--- | :--- | :--- | :--- |
|
||||||
|
| **Natural Language Execution** | ❌ Rigid hard-coded scripts only | ⚠️ Sluggish (waits 20-30s per tap) | ⚡ **High Throughput**: As fast as 3–5s per step in Flash mode with fluid responses |
|
||||||
|
| **UI Redesign Resilience** | ❌ Layout changes break XPath selectors | ⚠️ Coordinate guessing leads to drift | 🎯 **Semantic Grounding**: Blends text, icons & UI tree structure to adapt to UI changes |
|
||||||
|
| **Popup Interception** | ❌ Blocked and crashed immediately | ❌ Loops endlessly on unfamiliar screens | 🩹 **Pre-Execution Self-Healing**: Verifies target state before clicking; dismisses popups |
|
||||||
|
| **Dynamic & Video Tasks** | ❌ Blind static sleep only | ❌ Blind to dynamic video playback | ⏱️ **Real-Time Multimedia**: Live video stream analysis, countdown timers & speculative chaining |
|
||||||
|
| **IDE & Tooling Integration** | ❌ Standalone test runner | ❌ Web-only with high integration friction | 🔌 **Native MCP Server**: Directly callable inside Cursor, Claude Desktop, etc. |
|
||||||
|
|
||||||
|
<a id="benchmarks"></a>
|
||||||
|
## 🏆 Benchmarks: AndroidWorld (SOTA 99%+)
|
||||||
|
|
||||||
|
Evaluated on [AndroidWorld](https://github.com/google-research/android_world) — Google Research's gold-standard benchmark spanning 20+ real apps and 100+ complex multi-step tasks: **Artemis demonstrated exceptional robustness across the entire benchmark suite, achieving a 99%+ completion rate.**
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<img src="./docs/assets/androidworld_benchmark_comparison.png" alt="AndroidWorld Benchmark Comparison" width="85%" />
|
||||||
|
</p>
|
||||||
|
|
||||||
|
## 🚀 How ARTEMIS is Architected
|
||||||
|
|
||||||
|
* ⚡ **Optimistic Asynchronous Pipeline**: The front-facing loop responds in milliseconds, while memory pruning and assertion verification run concurrently in the background without blocking execution;
|
||||||
|
* 🛡️ **Safety Net Pre-Execution Gate**: Dual-layer pre-check validates target availability milliseconds before action dispatch, instantly intercepting unexpected popups to eliminate blind clicks;
|
||||||
|
* ⏱️ **Time-Sensitive Speculative Chaining**: Overcomes LLM inference latency for transient UI elements (e.g. video fullscreen) by predicting target coordinates and executing rapid chained taps.
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><b>🔍 Click to expand: Architecture Deep Dive & Pipeline Diagram</b></summary>
|
||||||
|
|
||||||
|
<br>
|
||||||
|
|
||||||
|
### 1. ⚡ Optimistic Asynchronous Pipeline
|
||||||
|
* **Status Quo & Pain Points**: Conventional mobile agents rely on a **fully synchronous blocking model** — every single action must wait sequentially for the LLM to prune historical context, check milestone assertions, and audit long-term plans. This inflates per-step latency to 20–40 seconds, creating a sluggish user experience.
|
||||||
|
* **Artemis's Architectural Solution**: Inspired by **Optimistic Concurrency Control (OCC) and Snapshot Isolation** in database systems, Artemis completely decouples the main execution loop from heavy auxiliary computation:
|
||||||
|
* **High-Throughput Main Loop**: The front-facing execution path is strictly narrowed to a high-speed "Perception → Decision → Safety Gate → Execution" pipeline;
|
||||||
|
* **Background Concurrent Tasks**: Context token pruning, milestone checkers (`Checker`), and planner validations (`Planner`) run dynamically in parallel without halting device interaction;
|
||||||
|
* **Snapshot Isolation & Rollback**: The agent optimistically charges forward. If background verification detects a deviation, Artemis instantly **rolls back via state snapshots (Rollback)** and injects self-healing feedback.
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<img src="./docs/assets/artemis-architecture-pipeline.png" alt="Artemis Optimistic Async Pipeline Architecture" width="100%" />
|
||||||
|
</p>
|
||||||
|
|
||||||
|
### 2. 🛡️ Pre-Execution Safety Net & Time-Sensitive Speculative Chaining
|
||||||
|
* **Status Quo & The Transient UI Dilemma**:
|
||||||
|
* Mobile applications feature numerous **time-sensitive transient UI controls** (e.g. video fullscreen: tapping the screen wakes up the floating overlay, followed immediately by tapping the fullscreen icon).
|
||||||
|
* Traditional agents tap the screen to reveal controls, take a new screenshot, and wait 3–15 seconds for LLM reasoning. By the time the click is dispatched, **the player controls have already auto-faded away** — causing the click to strike the underlying video, triggering an endless loop of accidental pausing and waking.
|
||||||
|
* **Artemis's Architectural Solution**:
|
||||||
|
* **Speculative Chained Actions**: Upon recognizing time-sensitive dependencies, the agent dispatches compound chained actions (Wakeup → Millisecond Chained Tap) to hit the target within its transient visibility window;
|
||||||
|
* **Two Pillars Ensuring Reliable Chaining**:
|
||||||
|
1. **Historical UI Prior Prediction**: Predicts the target control's wake-up coordinates based on prior interaction history and app layout heuristics;
|
||||||
|
2. **Safety Net Pre-Execution Gate**: Milliseconds before the chained action lands, the Safety Net instantly verifies that the target control was successfully revealed at the expected coordinates. If the wakeup failed or an unexpected popup intercepted it, execution is **immediately blocked to prevent blind clicks**.
|
||||||
|
|
||||||
|
</details>
|
||||||
|
|
||||||
## ⚡ Execution Profiles: Flash vs. Pro
|
## ⚡ Execution Profiles: Flash vs. Pro
|
||||||
|
|
||||||
| Feature | ARTEMIS Flash (`--profile flash`) | ARTEMIS Pro (`--profile default`) |
|
| Feature / Dimension | ⚡ **ARTEMIS Flash** (`--profile flash`) | 🧠 **ARTEMIS Pro** (`--profile pro`) |
|
||||||
| :--- | :--- | :--- |
|
| :--- | :--- | :--- |
|
||||||
| **Model Loop** | Single reactive agent (Observe-Think-Act) | Multi-node LangGraph orchestration |
|
| **Design Purpose** | **Lightweight & Fast**: Direct deterministic UI actions | **Deep Reasoning**: Multi-step planning & complex self-healing |
|
||||||
| **Best For** | Short, deterministic tasks (< 35 steps) | Deep, complex, multi-stage workflows |
|
| **Step Latency** | **3–5 seconds** / step | **15–30 seconds** / turn (includes planning & verification) |
|
||||||
| **Step Latency** | ~3–5 seconds / step | ~15–30 seconds / turn (with planning & validation) |
|
| **Task Duration** | Minute-level short tasks (typically ≤35 steps) | Runs stably for **10+ hours**; monitoring tasks support **24/7 execution** |
|
||||||
| **Planning** | Direct goal execution | Hierarchical multi-step plan decomposition |
|
| **Best Suited For** | Well-defined standard UI tasks | Complex cross-app workflows, failure self-healing, continuous monitoring |
|
||||||
| **Diagnostics** | Basic error reporting | Deep diagnostics, log analysis & self-healing |
|
| **Self-Healing** | Local step retries | **Safety Net Gate** + dialog suppression + crash recovery + snapshot rollback |
|
||||||
|
| **Media Analysis** | **Basic visual perception** + High-Speed OCR | Full `scrcpy`/`ffmpeg` video stream analysis + Logcat logs |
|
||||||
|
|
||||||
---
|
## 🗺️ Roadmap
|
||||||
|
|
||||||
## 🔌 Model Context Protocol (MCP) Integration
|
- [x] **Optimistic Asynchronous Pipeline**: Ultra-lean main loop + background context compression & milestone checks.
|
||||||
|
- [x] **Pre-Execution Safety Net**: Millisecond pre-check gate & speculative chained actions.
|
||||||
|
- [x] **Time-Sensitive Media Tasks**: Fullscreen video and audio stream analysis with `scrcpy` & `ffmpeg`.
|
||||||
|
- [x] **Native MCP Server**: Seamless integration with tools like Cursor and Claude Desktop.
|
||||||
|
- [x] **Web Visual Console**: Live screen projection, interactive playground, and trajectory review.
|
||||||
|
- [x] **AndroidWorld SOTA**: Achieved 99%+ task completion rate.
|
||||||
|
- [ ] **Cross-Platform Extensions**: Exploring iOS and desktop Web perception and execution.
|
||||||
|
- [ ] **On-Device Lightweight VLMs**: Zero-cloud local execution with lightweight edge vision models.
|
||||||
|
- [ ] **Real-time Duplex Voice Mode**: Natural voice input with real-time interruption (barge-in) control.
|
||||||
|
|
||||||
ARTEMIS includes built-in MCP servers and tools to allow AI IDEs (such as Claude Desktop, Cursor, and Gemini Jetski) to control Android devices directly:
|
## 🤝 Community & Contributing
|
||||||
|
|
||||||
| Tool Name | Description |
|
Contributions are warmly welcomed!
|
||||||
| :--- | :--- |
|
* ⭐ **Star the repo** to follow updates and releases
|
||||||
| `mobile_run_task` | Launches an autonomous background automation task on the connected device. |
|
* 💬 Join the [Discord Community](https://discord.gg/wF2FN4WHGY) for technical discussions
|
||||||
| `mobile_manage_task` | Inspects status, terminates tasks, or injects real-time steering instructions. |
|
* 🐛 Open an [Issue](https://github.com/google/artemis/issues) or submit a [Pull Request](https://github.com/google/artemis/pulls)
|
||||||
| `mobile_get_device_state` | Retrieves real-time screenshots or simplified accessibility UI trees. |
|
|
||||||
| `mobile_inspect_trace` | Inspects step-by-step screenshots, visual action overlays, and execution logs. |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## ⚙️ Configuration Guide
|
|
||||||
|
|
||||||
All model configurations, thinking budgets, agent capabilities, and tool filtering are consolidated into a single self-explanatory configuration file: [`config/artemis.jsonc`](config/artemis.jsonc).
|
|
||||||
|
|
||||||
```jsonc
|
|
||||||
{
|
|
||||||
// 1. Global Default Model (Applied across all sub-agents unless overridden)
|
|
||||||
"default": {
|
|
||||||
"provider": "google",
|
|
||||||
"model": "gemini-3.6-flash",
|
|
||||||
"thinking_level": "medium",
|
|
||||||
"fallback": {
|
|
||||||
"provider": "google",
|
|
||||||
"model": "gemini-3.5-flash"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
|
|
||||||
// 2. Built-in Presets
|
|
||||||
"presets": {
|
|
||||||
"gemini-flagship": { "provider": "google", "model": "gemini-3.6-flash" },
|
|
||||||
"openai-gpt4o": { "provider": "openai", "model": "gpt-4o" },
|
|
||||||
"cost-saving": { "provider": "google", "model": "gemini-3.5-flash-lite" },
|
|
||||||
"local-ollama": { "provider": "openai", "model": "llama3.2-vision" }
|
|
||||||
},
|
|
||||||
|
|
||||||
// 3. Agent & Sub-Agent Node Customizations (Optional overrides)
|
|
||||||
"nodes": {
|
|
||||||
// 🧠 Planner: High-level goal decomposition and strategic step planning
|
|
||||||
"planner": { "thinking_level": "medium" },
|
|
||||||
|
|
||||||
// 🎯 Operator: Screen perception, element interaction, and direct execution
|
|
||||||
"operator": { "thinking_level": "medium", "include_thoughts": true },
|
|
||||||
|
|
||||||
// 🛡️ Validator & Failure Analyzer: Self-healing and auto-recovery
|
|
||||||
"validator_failure_analyzer": { "thinking_level": "medium" }
|
|
||||||
},
|
|
||||||
|
|
||||||
// 4. Agent Capabilities & Tool Filtering
|
|
||||||
"agent": {
|
|
||||||
"explorer_versions": { "operator": "flash", "validator": "flash" },
|
|
||||||
"denylisted_tools": {
|
|
||||||
"explorer": ["ask_image_processor", "get_ocr_list", "inspect_region"]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
> [!TIP]
|
|
||||||
> **Zero-Friction Global Change**: To switch the entire platform from Gemini to GPT-4o or Claude, simply edit the `"default"` block once in `config/artemis.jsonc`. All 15 sub-agents automatically inherit the new model!
|
|
||||||
|
|
||||||
### Using Non-Google LLMs
|
|
||||||
|
|
||||||
ARTEMIS supports multi-provider LLM backends via standard LangChain integrations (Google Gemini, OpenAI, OpenRouter, xAI / Grok, or Google Vertex AI).
|
|
||||||
|
|
||||||
#### Supported Providers & API Key Setup
|
|
||||||
|
|
||||||
| Provider | Provider Key | Environment Variable | Base URL Override | Example Models |
|
|
||||||
| :--- | :--- | :--- | :--- | :--- |
|
|
||||||
| **Google Gemini** | `google` *(default)* | `GEMINI_API_KEY` | N/A | `gemini-3.6-flash`, `gemini-3.5-flash` |
|
|
||||||
| **Google Vision OCR** | `ocr` | `OCR_API_KEY` | N/A | Google Cloud Vision Text Detection |
|
|
||||||
| **Google Vertex AI** | `vertexai` | Google ADC (`gcloud auth application-default login`) | N/A | `gemini-3.6-flash` |
|
|
||||||
| **OpenAI** | `openai` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | `gpt-4o`, `gpt-4o-mini`, `o3` |
|
|
||||||
| **OpenRouter** | `openrouter` | `OPEN_ROUTER_API_KEY` | N/A | `anthropic/claude-3.5-sonnet`, `meta-llama/llama-3.3-70b-instruct` |
|
|
||||||
| **xAI (Grok)** | `xai` | `XAI_API_KEY` | N/A | `grok-2-vision-1212`, `grok-beta` |
|
|
||||||
|
|
||||||
> [!NOTE]
|
|
||||||
> ARTEMIS uses multimodal visual grounding (screenshots) for UI navigation. When assigning non-Google models to vision-dependent agents (such as `operator`), ensure the selected model supports image/vision inputs.
|
|
||||||
|
|
||||||
#### Switching Provider in `config/artemis.jsonc`
|
|
||||||
|
|
||||||
To use OpenAI or Claude globally, simply change the `default` section in [`config/artemis.jsonc`](config/artemis.jsonc):
|
|
||||||
|
|
||||||
```jsonc
|
|
||||||
{
|
|
||||||
"default": {
|
|
||||||
"provider": "openai",
|
|
||||||
"model": "gpt-4o",
|
|
||||||
"fallback": {
|
|
||||||
"provider": "openai",
|
|
||||||
"model": "gpt-4o-mini"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Custom / Local OpenAI-Compatible Endpoints (Ollama, vLLM, LocalAI)
|
|
||||||
|
|
||||||
To connect ARTEMIS to local or self-hosted LLM endpoints (e.g., Ollama or vLLM), set `OPENAI_BASE_URL` in `.env` and specify `openai` as the provider in your configuration:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Route OpenAI provider calls to local Ollama / vLLM endpoint
|
|
||||||
OPENAI_API_KEY=ollama # dummy key if required by client
|
|
||||||
OPENAI_BASE_URL=http://localhost:11434/v1
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📊 Trace Inspection & Observability
|
|
||||||
|
|
||||||
Every task execution can record rich execution artifacts stored in `./traces/<trace_id>/`:
|
|
||||||
- **`notes/`**: High-level planner notes, step summaries, and generated reports (`output.md`).
|
|
||||||
- **`screenshots/`**:
|
|
||||||
- `before_screenshot.jpg`: Initial screen observed by the agent.
|
|
||||||
- `action_overlay_screenshot.jpg`: Screen with visual tap markers (red circles) and swipe vectors (red arrows).
|
|
||||||
- **`logs/`**: Raw framework, operator, and failure analyzer diagnostic logs.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🛠 Development & Testing
|
|
||||||
|
|
||||||
ARTEMIS provides standard `make` targets for development:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Setup environment and pre-commit hooks
|
|
||||||
make setup
|
|
||||||
|
|
||||||
# Run test suite
|
|
||||||
make test
|
|
||||||
|
|
||||||
# Run code formatters (ruff)
|
|
||||||
make format
|
|
||||||
|
|
||||||
# Run linter checks
|
|
||||||
make lint
|
|
||||||
|
|
||||||
# Run static type checker (pyright)
|
|
||||||
make typecheck
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📄 License
|
## 📄 License
|
||||||
|
|
||||||
|
|||||||
+251
@@ -0,0 +1,251 @@
|
|||||||
|
<p align="center">
|
||||||
|
<img src="./docs/assets/artemis-banner.png" alt="ARTEMIS Banner" width="100%" />
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<strong>ARTEMIS: 全自动操控手机的 AI 个人助理</strong>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<em>⚡ 让 Cursor / Claude Code 接管真机 • 跨 App 复杂自动化 • 零脚本维护 UI 测试 • Flash 模式单步最快 3-5 秒</em>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<a href="./README.md">English</a> •
|
||||||
|
<a href="./README_CN.md"><b>中文文档</b></a> •
|
||||||
|
<a href="#quick-start">快速上手</a> •
|
||||||
|
<a href="#mcp-setup">MCP 接入 IDE</a> •
|
||||||
|
<a href="#benchmarks">基准评测</a> •
|
||||||
|
<a href="https://discord.gg/wF2FN4WHGY">Discord 社区</a>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/Python-3.12+-3776AB.svg?logo=python&logoColor=white" alt="Python 3.12+"></a>
|
||||||
|
<a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License: Apache-2.0"></a>
|
||||||
|
<a href="https://modelcontextprotocol.io/"><img src="https://img.shields.io/badge/MCP-Native%20Server-8A2BE2.svg" alt="MCP Native"></a>
|
||||||
|
<a href="https://ai.google.dev/"><img src="https://img.shields.io/badge/Multimodal-Gemini%20%7C%20Claude%20%7C%20GPT--4o%20%7C%20Qwen--VL-4285F4.svg" alt="Multi-Model"></a>
|
||||||
|
<a href="https://github.com/google-research/android_world"><img src="https://img.shields.io/badge/AndroidWorld-99%25%2B%20SOTA-success.svg" alt="AndroidWorld SOTA"></a>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<!-- 演示效果图 -->
|
||||||
|
<p align="center">
|
||||||
|
<img src="./docs/assets/demo.gif" alt="Artemis 演示效果" width="88%" />
|
||||||
|
<br>
|
||||||
|
<em><b>实机演示</b>：在 Google Maps 中设置驾车路线并计算总耗时，随后打开 YouTube 播放 Coldplay 的歌曲。</em>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
## ✨ 核心亮点
|
||||||
|
|
||||||
|
* ⚡ **极致响应速度**：首创**全链路乐观异步流水线 (Optimistic Async Pipeline)**，主干执行流与后台重度推理完全解耦；
|
||||||
|
* 🔋 **超长程稳定运行**：**Pro 模式**可运行超 10 小时，监控任务甚至可以全天候运行；
|
||||||
|
* 🔌 **原生 MCP (Model Context Protocol) 赋能**：让 **Cursor、Claude Code、OpenClaw、Windsurf** 直接长出“操作真机的双手”，在聊天窗口一句话完成真机测试与 Bug 复现；
|
||||||
|
* 💾 **可视化控制台与持久化调试**：提供 Web UI 实时投屏、操作回放与任务持久化管理，支持通过自然语言生成测试用例，并对历史执行进行高保真可视化审查；
|
||||||
|
* 🏆 **业界顶尖 SOTA**：在 Google Research **AndroidWorld** 基准评测（100+ 复杂长程任务）中取得 **99%+ 任务完成率**。
|
||||||
|
|
||||||
|
<a id="quick-start"></a>
|
||||||
|
<a id="快速上手"></a>
|
||||||
|
## ⚡ 快速上手
|
||||||
|
|
||||||
|
确保电脑已连接 Android 实体机（已开启 **USB 调试**）或 Android 模拟器。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. 克隆代码仓库并进入目录
|
||||||
|
git clone https://github.com/google/artemis.git && cd artemis
|
||||||
|
|
||||||
|
# 2. 一键启动（自动检测并安装 ADB、scrcpy、FFmpeg、uv 运行时等全部依赖，并拉起控制台）
|
||||||
|
# 🍎 macOS & 🐧 Linux
|
||||||
|
./start.sh
|
||||||
|
|
||||||
|
# 🪟 Windows (CMD / PowerShell)
|
||||||
|
start.bat
|
||||||
|
```
|
||||||
|
|
||||||
|
> 💡 **提示**：启动后将自动在默认浏览器中打开 Web 控制台（`http://localhost:8000`），提供设备连接向导、实时投屏、任务演练与状态回放面板。你也可以通过命令行直接运行：`artemis run "打开系统设置，找到电池选项并告诉我当前电量" --profile flash`。
|
||||||
|
|
||||||
|
<a id="mcp-setup"></a>
|
||||||
|
<a id="mcp"></a>
|
||||||
|
<details>
|
||||||
|
<summary><b>🔌 接入 Cursor / Claude Code / Windsurf (MCP)（点击展开）</b></summary>
|
||||||
|
|
||||||
|
<br>
|
||||||
|
|
||||||
|
ARTEMIS 内置原生 **Model Context Protocol (MCP)** 服务。只需将以下配置加入你的 IDE 配置文件中，即可在编写代码时直接驱动真机：
|
||||||
|
|
||||||
|
### 1. 生成或查看配置
|
||||||
|
|
||||||
|
运行内置命令一键获取当前环境的完整配置 JSON：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
artemis mcp --generate-config cursor
|
||||||
|
# 或生成所有 IDE 配置：
|
||||||
|
artemis mcp --generate-config all
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. 复制配置到 IDE
|
||||||
|
|
||||||
|
* **Cursor** (`.cursor/mcp.json` 或 设置 ➔ MCP Servers)：
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"mcpServers": {
|
||||||
|
"artemis": {
|
||||||
|
"command": "python",
|
||||||
|
"args": ["-m", "mcp_server"],
|
||||||
|
"cwd": "/path/to/artemis",
|
||||||
|
"env": {
|
||||||
|
"PYTHONUNBUFFERED": "1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
* **Claude Desktop** (`claude_desktop_config.json`)：
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"mcpServers": {
|
||||||
|
"artemis": {
|
||||||
|
"command": "python",
|
||||||
|
"args": ["-m", "mcp_server"],
|
||||||
|
"cwd": "/path/to/artemis"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. 在 IDE 中体验真机协同
|
||||||
|
在 Cursor / Claude Code 对话框中直接输入：
|
||||||
|
> 💬 *"请帮我把刚刚修改的代码编译成 APK 并安装到手机上，打开登录页面输入测试账号，验证登录后是否有异常弹窗，并把最终页面截图回传。"*
|
||||||
|
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<a id="python-sdk"></a>
|
||||||
|
<details>
|
||||||
|
<summary><b>🐍 Python SDK 极简集成（点击展开）</b></summary>
|
||||||
|
|
||||||
|
<br>
|
||||||
|
|
||||||
|
只需几行代码，将移动端自动化智能体无缝嵌入你的 Python 脚本与 CI/CD 流水线中：
|
||||||
|
|
||||||
|
```python
|
||||||
|
import asyncio
|
||||||
|
from artemis.interfaces.sdk import ArtemisClient
|
||||||
|
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
# 初始化客户端（支持 "flash" 极速模式 或 "pro" 深度推理模式）
|
||||||
|
client = ArtemisClient(default_profile="flash")
|
||||||
|
|
||||||
|
# 执行自然语言任务
|
||||||
|
result = await client.run("打开计算器，计算 (128 * 45) + 330 的结果")
|
||||||
|
|
||||||
|
print(f"执行状态: {result.status}")
|
||||||
|
print(f"消耗步数: {result.turns}")
|
||||||
|
print(f"Trace ID: {result.trace_id}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(main())
|
||||||
|
```
|
||||||
|
|
||||||
|
</details>
|
||||||
|
|
||||||
|
## 🕹️ 使用方式
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<img src="./docs/assets/artemis-ui-showcase.png" alt="Artemis 可视化控制台" width="90%" />
|
||||||
|
<br />
|
||||||
|
<sub>💡 <b>控制台功能布局</b>：<b>① 顶栏视图切换</b>（主页与工作区） · <b>② 运行模式与录屏回放</b>（Flash/Pro 状态与视频回放） · <b>③ 实时感知推理流</b>（动作分解、点击坐标与结构化总结） · <b>④ 自然语言下发胶囊</b>（自然语言驱动真机） · <b>⑤ 任务队列看板</b>（状态流转与历史回溯）</sub>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
* 🖥️ **Web 可视化控制台 (`artemis ui`)**：集成设备实时投屏与交互面板，支持通过自然语言下发任务，实时观测推理过程、操作轨迹与状态回放；
|
||||||
|
* 🔌 **MCP 协议集成 (IDE 联动)**：作为标准 MCP 服务器无缝接入 **Cursor、Claude Code、OpenClaw** 等环境，直接在 IDE 中调用移动端真机操作能力；
|
||||||
|
* 💻 **命令行工具 (`artemis run`)**：支持通过 CLI 直接执行自动化任务、测试用例或 AndroidWorld 基准评测，提供高保真结构化终端输出；
|
||||||
|
* 🐍 **Python SDK**：作为标准 Python 库集成至现有自动化测试框架或流水线，提供基于 Pydantic 的强类型结构化输出。
|
||||||
|
|
||||||
|
## 📊 方案横向对比
|
||||||
|
|
||||||
|
| 场景 / 维度 | 传统自动化框架 | 常见屏幕 Agent | **ARTEMIS ☕** |
|
||||||
|
| :--- | :--- | :--- | :--- |
|
||||||
|
| **自然语言指令支持** | ❌ 仅支持硬编码脚本，无语义理解 | ⚠️ 响应较慢，单步交互通常需 20-30 秒 | ⚡ **高效执行**：Flash 模式下单步最快约 3-5 秒，交互流畅 |
|
||||||
|
| **UI 改版适应能力** | ❌ 元素变更易导致脚本执行中断 | ⚠️ 依赖绝对坐标，存在位置偏移风险 | 🎯 **多模态语义定位**：结合文本、图标与层次结构，自适应 UI 变化 |
|
||||||
|
| **异常弹窗处理** | ❌ 弹窗遮挡易导致定位失败 | ❌ 面对未知弹窗易陷入无效循环 | 🩹 **执行前校验与自愈**：操作前确认目标状态，自动识别并处理干扰弹窗 |
|
||||||
|
| **动态与视频场景** | ❌ 仅支持静态等待 (sleep)，无动态感知 | ❌ 无法识别视频与动态流媒体内容 | ⏱️ **流媒体分析**：支持视频流分析、倒计时识别与预测性连点 |
|
||||||
|
| **开发环境集成** | ❌ 独立运行，难以直接与 AI IDE 联动 | ❌ 通常仅提供独立网页，集成成本高 | 🔌 **原生 MCP 支持**：可在 Cursor、Claude Desktop 等环境中直接调用 |
|
||||||
|
|
||||||
|
<a id="benchmarks"></a>
|
||||||
|
<a id="基准评测"></a>
|
||||||
|
## 🏆 基准评测：AndroidWorld (SOTA 99%+)
|
||||||
|
|
||||||
|
在 Google Research 发布的业界基准评测 [AndroidWorld](https://github.com/google-research/android_world)（涵盖 20+ 款常用应用与 100+ 项复杂多步长程任务）中：**Artemis 在全套长程任务评测中展现了高鲁棒性，取得了超过 99% 的任务完成率。**
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<img src="./docs/assets/androidworld_benchmark_comparison.png" alt="AndroidWorld 评测基准对比" width="85%" />
|
||||||
|
</p>
|
||||||
|
|
||||||
|
## 🚀 ARTEMIS 是如何建构的
|
||||||
|
|
||||||
|
* ⚡ **全链路乐观异步 (Optimistic Async)**：主交互循环毫秒级响应下发，记忆修剪与阶段断言全部在后台并发静默运行，告别死等；
|
||||||
|
* 🛡️ **Safety Net 动作守门**：动作下发前毫秒级双重校验目标，突发弹窗即刻拦截自愈，绝不盲点空气；
|
||||||
|
* ⏱️ **时效敏感预测性连点 (Speculative Chaining)**：针对视频全屏等瞬态交互，结合历史先验预测坐标并极速连点，彻底解决大模型延迟导致控件消失的问题。
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><b>🔍 点击展开：核心架构设计细节与流水线流程图</b></summary>
|
||||||
|
|
||||||
|
<br>
|
||||||
|
|
||||||
|
### 1. ⚡ 全链路乐观异步流水线 (Optimistic Asynchronous Pipeline)
|
||||||
|
* **现有方案与问题**：传统移动端智能体普遍采用**全同步阻塞模型** —— 每一步交互都必须同步串行等待大模型完成历史上下文修剪、阶段性断言检查与长程规划审计。这导致主交互循环极度臃肿，单步延迟动辄 20–40 秒，交互体验迟缓；
|
||||||
|
* **Artemis 的架构解法**：借鉴数据库的**乐观并发控制 (OCC) 与快照隔离**思想，将主干执行流与重度计算彻底解耦：
|
||||||
|
* **高吞吐前台主循环**：前台严格收敛为「感知 → 决策 → 守门 → 执行」极简流水线，单步响应压缩至秒级；
|
||||||
|
* **后台并发异步处理**：上下文修剪、里程碑断言校验（Checker）、全局规划审计（Planner）全部在后台并发静默运行，零阻塞前台；
|
||||||
|
* **快照隔离与无损回滚**：前台以前瞻视角乐观执行；一旦后台异步断言检测到状态偏离，系统基于快照机制**一键无损回滚 (Rollback)** 并注入纠偏反馈，兼得极低延迟与可靠自愈。
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<img src="./docs/assets/artemis-architecture-pipeline.png" alt="Artemis 乐观异步流水线架构" width="100%" />
|
||||||
|
</p>
|
||||||
|
|
||||||
|
### 2. 🛡️ 动作守门机制 (Safety Net) 与时效敏感预测性连点 (Speculative Chaining)
|
||||||
|
* **现有方案与时效困境**：
|
||||||
|
* 移动端存在大量**时效敏感型交互**（如视频全屏播放：必须先轻点屏幕唤醒浮层控件，再点击全屏按钮）。
|
||||||
|
* 传统智能体在点击屏幕唤醒控件后，重新截屏并等待大模型推理决策耗时数秒；而此时**播放器控件早已自动淡出隐藏**，导致后续点击直接打在视频画面上，陷入“唤醒 → 控件消失 → 误触暂停 → 再次唤醒”的死循环。
|
||||||
|
* **Artemis 的架构解法**：
|
||||||
|
* **预测性链式连点 (Speculative Chaining)**：智能体在识别此类时序依赖后，直接下发复合连击指令（唤醒 → 毫秒级连击全屏），在控件有效窗口期内瞬时完成交互；
|
||||||
|
* **保障连点绝对可靠的两大支柱**：
|
||||||
|
1. **历史先验位置预测**：基于历史交互轨迹与应用 UI 先验知识，精准预判控件唤醒后的目标坐标；
|
||||||
|
2. **Safety Net 毫秒级拦截守门**：在连击动作落下的前一刻，瞬间核验目标控件是否已被成功唤醒并位于预判位置。若唤醒失败或遭遇意外弹窗遮挡，**立刻前置拦截熔断，杜绝盲点误触**。
|
||||||
|
|
||||||
|
</details>
|
||||||
|
|
||||||
|
## ⚡ 运行模式对比：Flash vs. Pro
|
||||||
|
|
||||||
|
| 特性对比 | ⚡ **ARTEMIS Flash** (`--profile flash`) | 🧠 **ARTEMIS Pro** (`--profile pro`) |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| **设计定位** | **轻量极速**：面向常规确定性操作 | **深度规划**：面向长流程与复杂自愈 |
|
||||||
|
| **单步延迟** | **3–5 秒** / 步 | **15–30 秒** / 轮 (包含深度推理与全局校验) |
|
||||||
|
| **任务时长** | 分钟级短程任务（通常 ≤35 步） | 可稳定运行**超 10 小时**，监控任务可**全天候运行** |
|
||||||
|
| **适用场景** | 目标明确的常规 UI 操作 | 复杂的跨 App 长流程、故障自愈、持续状态观察 |
|
||||||
|
| **自愈机制** | 基础局部重试 | **Safety Net 校验** + 弹窗处理 + 崩溃恢复 + 快照回滚 |
|
||||||
|
| **多媒体分析** | 基础视觉感知 + 高速 OCR | 完整 `scrcpy`/`ffmpeg` 视频流分析 + Logcat 日志采集 |
|
||||||
|
|
||||||
|
## 🗺️ 路线图
|
||||||
|
|
||||||
|
- [x] **全链路乐观异步流水线**：主流程轻量化，后台并发执行记忆压缩与断言校验。
|
||||||
|
- [x] **Safety Net 执行前校验**：动作下发前双层核验与预测性链式执行。
|
||||||
|
- [x] **时间敏感多媒体处理**：集成 `scrcpy` 与 `ffmpeg` 分析全屏视频与音频流。
|
||||||
|
- [x] **原生 MCP 服务支持**：支持与 Cursor、Claude Desktop 等工具无缝集成。
|
||||||
|
- [x] **Web 可视化控制台**：提供设备投屏、交互演练与轨迹回溯。
|
||||||
|
- [x] **AndroidWorld SOTA**：达到 99%+ 的任务完成率。
|
||||||
|
- [ ] **跨平台扩展**：探索 iOS 与 Web 端的定位与操作能力。
|
||||||
|
- [ ] **端侧轻量化模型**：支持在设备本地离线运行的轻量级 Edge VLM 模型。
|
||||||
|
- [ ] **实时语音双工交互**：支持自然语音输入与实时打断控制。
|
||||||
|
|
||||||
|
## 🤝 社区与贡献
|
||||||
|
|
||||||
|
欢迎通过以下方式参与项目建设：
|
||||||
|
* ⭐ **Star 本项目**以关注最新进展与更新
|
||||||
|
* 💬 加入 [Discord 社区](https://discord.gg/wF2FN4WHGY) 参与技术探讨与功能建议
|
||||||
|
* 🐛 提交 [Issue](https://github.com/google/artemis/issues) 反馈 Bug，欢迎发起 [Pull Request](https://github.com/google/artemis/pulls) 贡献代码
|
||||||
|
|
||||||
|
## 📄 开源许可证
|
||||||
|
|
||||||
|
本项目基于 [Apache License 2.0](LICENSE) 协议开源。
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
# 📱 Artemis Applications Ecosystem (`apps/`)
|
|
||||||
|
|
||||||
This directory houses the two primary applications and user interfaces of the Artemis ecosystem.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📂 Subsystems Overview
|
|
||||||
|
|
||||||
```
|
|
||||||
apps/
|
|
||||||
├── 🛠️ admin_console/ # [Interface 1] Full-fidelity Trace Inspector & Task Control Console (FastAPI + Web)
|
|
||||||
└── ✨ showcase_ui/ # [Interface 2] Aesthetic User Presentation & Interaction Workspace (Angular 19)
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🔍 Detailed Comparison of the Two UIs
|
|
||||||
|
|
||||||
| Dimension | 🛠️ Admin & Trace Console (`admin_console`) | ✨ Showcase & Workspace UI (`showcase_ui`) |
|
|
||||||
| :--- | :--- | :--- |
|
|
||||||
| **Primary Audience** | Developers, Test Engineers, System Admins | End Users, Customers, Product Demo Audiences |
|
|
||||||
| **UI Paradigm** | High-density, full-fidelity tree logs, step-by-step inspector | Dynamic Aurora Glow, Glassmorphism, dual-pane stream |
|
|
||||||
| **Core Capabilities** | • Full hierarchical Call Tree & LLM thinking inspection<br>• Pre/post action screenshots & visual overlay verification<br>• Video playback & step replay engine (`/api/replay`)<br>• Queue tasks & live state toggle | • Real-time Agent thought & action progress streaming<br>• Interactive natural language chat sidebar<br>• Aesthetic visual representation of mobile autonomy |
|
|
||||||
| **Tech Stack** | Python (FastAPI) + Vanilla Responsive Web | TypeScript, Angular 19, SCSS |
|
|
||||||
| **Start Command** | `uv run uvicorn apps.admin_console.server:app --port 8000 --reload` | `cd apps/showcase_ui && npm start` (Port 4200) |
|
|
||||||
|
|
||||||
+1
-2
@@ -12,5 +12,4 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
|
"""Artemis Applications Package."""
|
||||||
"""Artemis / Latte Applications Package."""
|
|
||||||
@@ -1,51 +0,0 @@
|
|||||||
# 🛠️ Artemis Admin & Trace Console
|
|
||||||
|
|
||||||
A comprehensive developer/administrator control plane and trace inspection console for the Latte/Artemis mobile automation system.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🎯 Key Features
|
|
||||||
|
|
||||||
1. **Full Trace Inspection (`/api/sessions`, `/api/steps`)**:
|
|
||||||
- Hierarchical Call Tree exploration (LLM thoughts, tool calls, agent motivations).
|
|
||||||
- Step-by-step pre/post screenshots and action overlays (tap circles, swipe paths).
|
|
||||||
- Streaming logs and video recording playback.
|
|
||||||
2. **Task Control Center (`/api/run`)**:
|
|
||||||
- Directly launch autonomous tasks with custom goals and model profiles (Flash / Pro).
|
|
||||||
- Pause, resume, and manage FIFO task execution queues.
|
|
||||||
3. **Step Replay & Debugging Engine (`/api/replay`)**:
|
|
||||||
- Replay individual steps with isolated inputs, state inspection, and live device execution.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🚀 How to Start
|
|
||||||
|
|
||||||
### Option 1: Using `uv` (Recommended)
|
|
||||||
From the repository root:
|
|
||||||
```bash
|
|
||||||
uv run uvicorn apps.admin_console.server:app --host 0.0.0.0 --port 8000 --reload
|
|
||||||
```
|
|
||||||
Or directly using Python module:
|
|
||||||
```bash
|
|
||||||
uv run python -m apps.admin_console.server
|
|
||||||
```
|
|
||||||
|
|
||||||
### Option 2: From the `apps/admin_console` directory
|
|
||||||
```bash
|
|
||||||
cd apps/admin_console
|
|
||||||
uv run uvicorn server:app --host 0.0.0.0 --port 8000 --reload
|
|
||||||
```
|
|
||||||
|
|
||||||
Once started, open your browser and navigate to:
|
|
||||||
👉 **`http://localhost:8000`**
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🎥 Video Debugging Mode
|
|
||||||
|
|
||||||
By default, `VideoAnalyzer` cleans up local temporary video files after execution. To retain video recordings for full playback in the console:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
KEEP_VIDEOS=1 artemis run "your task goal"
|
|
||||||
```
|
|
||||||
Or set `ARTEMIS_DEBUG=1` in your environment.
|
|
||||||
@@ -63,6 +63,12 @@ except ImportError:
|
|||||||
REPLAY_BASE_DIR = TRACES_PATH / "replay"
|
REPLAY_BASE_DIR = TRACES_PATH / "replay"
|
||||||
TEST_DATA_DIR = REPLAY_BASE_DIR / "data"
|
TEST_DATA_DIR = REPLAY_BASE_DIR / "data"
|
||||||
TEST_OUTPUTS_DIR = REPLAY_BASE_DIR / "outputs"
|
TEST_OUTPUTS_DIR = REPLAY_BASE_DIR / "outputs"
|
||||||
|
|
||||||
IMAGES_DIR = TRACES_PATH / "images"
|
IMAGES_DIR = TRACES_PATH / "images"
|
||||||
PAUSE_FILE = WORKSPACE_ROOT / ".artemis_paused"
|
PAUSE_FILE = WORKSPACE_ROOT / ".artemis_paused"
|
||||||
|
|
||||||
|
# Ensure all essential runtime directories exist automatically
|
||||||
|
for _d in (TRACES_PATH, REPLAY_BASE_DIR, TEST_DATA_DIR, TEST_OUTPUTS_DIR, IMAGES_DIR):
|
||||||
|
try:
|
||||||
|
_d.mkdir(parents=True, exist_ok=True)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
@@ -36,25 +36,96 @@ class ServerState:
|
|||||||
self.active_connections: dict[str, dict[str, Any]] = {}
|
self.active_connections: dict[str, dict[str, Any]] = {}
|
||||||
self.active_session_id: str | None = None
|
self.active_session_id: str | None = None
|
||||||
self.was_stopped_manually: bool = False
|
self.was_stopped_manually: bool = False
|
||||||
|
self.cancelled_session_ids: set[str] = set()
|
||||||
|
|
||||||
# FIFO Task queue
|
# Unified single source of truth for task queue
|
||||||
self.task_queue: asyncio.Queue = asyncio.Queue()
|
self.queue_items: list[dict[str, Any]] = []
|
||||||
self.queue_tasks: list[dict[str, Any]] = []
|
self._wake_event: asyncio.Event | None = None
|
||||||
self.worker_task: asyncio.Task | None = None
|
self.worker_task: asyncio.Task | None = None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def wake_event(self) -> asyncio.Event:
|
||||||
|
try:
|
||||||
|
loop = asyncio.get_running_loop()
|
||||||
|
except RuntimeError:
|
||||||
|
loop = None
|
||||||
|
|
||||||
|
if self._wake_event is None or (
|
||||||
|
loop
|
||||||
|
and getattr(self._wake_event, "_loop", None) is not None
|
||||||
|
and self._wake_event._loop != loop
|
||||||
|
):
|
||||||
|
self._wake_event = asyncio.Event()
|
||||||
|
return self._wake_event
|
||||||
|
|
||||||
|
@property
|
||||||
|
def task_queue(self) -> list[dict[str, Any]]:
|
||||||
|
"""Backward compatibility alias for queue_items."""
|
||||||
|
return self.queue_items
|
||||||
|
|
||||||
|
@property
|
||||||
|
def queue_tasks(self) -> list[dict[str, Any]]:
|
||||||
|
"""Returns all currently pending tasks in the queue."""
|
||||||
|
return [t for t in self.queue_items if isinstance(t, dict) and t.get("status") == "pending"]
|
||||||
|
|
||||||
|
@queue_tasks.setter
|
||||||
|
def queue_tasks(self, val: list[dict[str, Any]]):
|
||||||
|
# Compatibility setter
|
||||||
|
self.queue_items = list(val)
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def queue_goals(self) -> list[str]:
|
def queue_goals(self) -> list[str]:
|
||||||
"""Backward compatibility helper for queue goals."""
|
"""Backward compatibility helper for queue goals."""
|
||||||
return [t.get("goal", "") for t in self.queue_tasks if isinstance(t, dict)]
|
return [t.get("goal", "") for t in self.queue_tasks]
|
||||||
|
|
||||||
@queue_goals.setter
|
@queue_goals.setter
|
||||||
def queue_goals(self, val: list[Any]):
|
def queue_goals(self, val: list[Any]):
|
||||||
# Allow setting if legacy code updates it
|
|
||||||
pass
|
pass
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def is_running(self) -> bool:
|
def is_running(self) -> bool:
|
||||||
return self.current_process is not None and self.current_process.returncode is None
|
has_proc = False
|
||||||
|
if self.current_process is not None:
|
||||||
|
if self.current_process.returncode is not None:
|
||||||
|
has_proc = False
|
||||||
|
self.current_process = None
|
||||||
|
else:
|
||||||
|
pid = getattr(self.current_process, "pid", None)
|
||||||
|
if pid:
|
||||||
|
try:
|
||||||
|
import os
|
||||||
|
|
||||||
|
os.kill(pid, 0)
|
||||||
|
has_proc = True
|
||||||
|
except (ProcessLookupError, PermissionError, OSError):
|
||||||
|
has_proc = False
|
||||||
|
self.current_process = None
|
||||||
|
else:
|
||||||
|
has_proc = False
|
||||||
|
self.current_process = None
|
||||||
|
|
||||||
|
has_running_item = any(
|
||||||
|
isinstance(t, dict) and t.get("status") == "running" for t in self.queue_items
|
||||||
|
)
|
||||||
|
|
||||||
|
has_live_connection = False
|
||||||
|
for sid, conn in list(self.active_connections.items()):
|
||||||
|
c_pid = conn.get("pid")
|
||||||
|
if c_pid:
|
||||||
|
try:
|
||||||
|
import os
|
||||||
|
|
||||||
|
os.kill(c_pid, 0)
|
||||||
|
has_live_connection = True
|
||||||
|
except (ProcessLookupError, PermissionError, OSError):
|
||||||
|
self.active_connections.pop(sid, None)
|
||||||
|
|
||||||
|
if not has_proc and not has_running_item and not has_live_connection:
|
||||||
|
self.active_session_id = None
|
||||||
|
self.current_process = None
|
||||||
|
return False
|
||||||
|
|
||||||
|
return True
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def is_paused(self) -> bool:
|
def is_paused(self) -> bool:
|
||||||
@@ -69,13 +140,10 @@ class ServerState:
|
|||||||
self.ipc_subscribers.remove(callback)
|
self.ipc_subscribers.remove(callback)
|
||||||
|
|
||||||
def clear_queue(self):
|
def clear_queue(self):
|
||||||
while not self.task_queue.empty():
|
"""Clears all pending items from the task queue."""
|
||||||
try:
|
self.queue_items = [t for t in self.queue_items if t.get("status") == "running"]
|
||||||
self.task_queue.get_nowait()
|
if self._wake_event:
|
||||||
self.task_queue.task_done()
|
self._wake_event.set()
|
||||||
except (asyncio.QueueEmpty, ValueError):
|
|
||||||
break
|
|
||||||
self.queue_tasks.clear()
|
|
||||||
|
|
||||||
|
|
||||||
# Global shared instance
|
# Global shared instance
|
||||||
|
|||||||
@@ -15,7 +15,6 @@
|
|||||||
from contextlib import contextmanager
|
from contextlib import contextmanager
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
import sqlite3
|
import sqlite3
|
||||||
from fastapi import HTTPException
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
from admin_console.core.config import DB_PATH
|
from admin_console.core.config import DB_PATH
|
||||||
@@ -23,13 +22,29 @@ except ImportError:
|
|||||||
from apps.admin_console.core.config import DB_PATH
|
from apps.admin_console.core.config import DB_PATH
|
||||||
|
|
||||||
|
|
||||||
|
_initialized_dbs = set()
|
||||||
|
|
||||||
|
|
||||||
def get_db(db_path: Path | None = None) -> sqlite3.Connection:
|
def get_db(db_path: Path | None = None) -> sqlite3.Connection:
|
||||||
"""Returns a SQLite connection with timeout and Row row_factory."""
|
"""Returns a SQLite connection with timeout and Row row_factory."""
|
||||||
path = db_path or DB_PATH
|
path = db_path or DB_PATH
|
||||||
if not path.exists():
|
path_key = str(path)
|
||||||
raise HTTPException(status_code=500, detail=f"Database not found at {path}")
|
if path_key not in _initialized_dbs:
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
try:
|
||||||
|
from artemis.data_engine.storage import StorageManager
|
||||||
|
|
||||||
|
StorageManager(db_path=path, base_trace_dir=path.parent)
|
||||||
|
_initialized_dbs.add(path_key)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
conn = sqlite3.connect(path, timeout=10.0)
|
conn = sqlite3.connect(path, timeout=10.0)
|
||||||
conn.row_factory = sqlite3.Row
|
conn.row_factory = sqlite3.Row
|
||||||
|
try:
|
||||||
|
conn.execute("PRAGMA journal_mode=WAL")
|
||||||
|
conn.execute("PRAGMA busy_timeout=10000")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
return conn
|
return conn
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -10,4 +10,4 @@
|
|||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
@@ -33,13 +33,6 @@ class SessionRepository:
|
|||||||
cursor.execute("SELECT * FROM sessions ORDER BY start_time DESC")
|
cursor.execute("SELECT * FROM sessions ORDER BY start_time DESC")
|
||||||
return [dict(r) for r in cursor.fetchall()]
|
return [dict(r) for r in cursor.fetchall()]
|
||||||
|
|
||||||
def get_session_by_id(self, session_id: str) -> dict[str, Any] | None:
|
|
||||||
with db_session(self.db_path) as conn:
|
|
||||||
cursor = conn.cursor()
|
|
||||||
cursor.execute("SELECT * FROM sessions WHERE session_id = ?", (str(session_id),))
|
|
||||||
row = cursor.fetchone()
|
|
||||||
return dict(row) if row else None
|
|
||||||
|
|
||||||
def get_video_recordings_map(self) -> dict[str, str]:
|
def get_video_recordings_map(self) -> dict[str, str]:
|
||||||
with db_session(self.db_path) as conn:
|
with db_session(self.db_path) as conn:
|
||||||
cursor = conn.cursor()
|
cursor = conn.cursor()
|
||||||
|
|||||||
@@ -13,6 +13,7 @@
|
|||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
import json
|
import json
|
||||||
|
import re
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -183,9 +184,27 @@ class StepRepository:
|
|||||||
)
|
)
|
||||||
|
|
||||||
args = payload_obj.get("args", payload_obj)
|
args = payload_obj.get("args", payload_obj)
|
||||||
if isinstance(args, dict):
|
cleaned_args = (
|
||||||
return {"args": self._clean_value(args)}
|
self._clean_value(args) if isinstance(args, dict) else self._clean_value(args)
|
||||||
return {"args": self._clean_value(args)}
|
)
|
||||||
|
result_payload: dict[str, Any] = {"args": cleaned_args}
|
||||||
|
|
||||||
|
# Extract structured action execution results
|
||||||
|
res = payload_obj.get("result")
|
||||||
|
if isinstance(res, dict):
|
||||||
|
if res.get("post_image_name"):
|
||||||
|
result_payload["post_image_name"] = res["post_image_name"]
|
||||||
|
if res.get("pre_image_name"):
|
||||||
|
result_payload["pre_image_name"] = res["pre_image_name"]
|
||||||
|
if res.get("outcome"):
|
||||||
|
result_payload["outcome"] = res["outcome"]
|
||||||
|
elif res:
|
||||||
|
res_str = str(res)
|
||||||
|
match = re.search(r"([a-f0-9]{64})", res_str)
|
||||||
|
if match:
|
||||||
|
result_payload["post_image_name"] = match.group(1)
|
||||||
|
|
||||||
|
return result_payload
|
||||||
|
|
||||||
def get_session_steps(self, session_id: str, client: str | None = None) -> list[dict[str, Any]]:
|
def get_session_steps(self, session_id: str, client: str | None = None) -> list[dict[str, Any]]:
|
||||||
with db_session(self.db_path) as conn:
|
with db_session(self.db_path) as conn:
|
||||||
@@ -221,7 +240,7 @@ class StepRepository:
|
|||||||
if pre_img and step_dict.get("post_image_name") == pre_img:
|
if pre_img and step_dict.get("post_image_name") == pre_img:
|
||||||
step_dict["post_image_name"] = None
|
step_dict["post_image_name"] = None
|
||||||
|
|
||||||
# Fetch generic tool traces for this step
|
# Fetch action and generic tool traces for this step
|
||||||
step_id = step_dict.get("step_id")
|
step_id = step_dict.get("step_id")
|
||||||
if step_id:
|
if step_id:
|
||||||
cursor.execute(
|
cursor.execute(
|
||||||
@@ -229,7 +248,7 @@ class StepRepository:
|
|||||||
" t1.type, t1.name, t1.timestamp, t1.duration, t1.status,"
|
" t1.type, t1.name, t1.timestamp, t1.duration, t1.status,"
|
||||||
" t1.payload, t2.name as agent_name FROM traces t1 LEFT"
|
" t1.payload, t2.name as agent_name FROM traces t1 LEFT"
|
||||||
" JOIN traces t2 ON t1.parent_trace_id = t2.trace_id WHERE"
|
" JOIN traces t2 ON t1.parent_trace_id = t2.trace_id WHERE"
|
||||||
" t1.step_id = ? AND (t1.type = 'tool' OR (t1.type ="
|
" t1.step_id = ? AND (t1.type = 'tool' OR t1.type = 'action' OR (t1.type ="
|
||||||
" 'llm_call' AND t1.status = 'failed')) ORDER BY"
|
" 'llm_call' AND t1.status = 'failed')) ORDER BY"
|
||||||
" t1.timestamp ASC",
|
" t1.timestamp ASC",
|
||||||
(step_id,),
|
(step_id,),
|
||||||
|
|||||||
+123
-21
@@ -20,6 +20,11 @@
|
|||||||
<meta charset="UTF-8">
|
<meta charset="UTF-8">
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
<title>Artemis Debug UI</title>
|
<title>Artemis Debug UI</title>
|
||||||
|
<link rel="icon" type="image/png" sizes="192x192" href="/logo.png?v=2">
|
||||||
|
<link rel="icon" type="image/png" sizes="32x32" href="/logo.png?v=2">
|
||||||
|
<link rel="icon" type="image/svg+xml" href="/logo.svg?v=2">
|
||||||
|
<link rel="shortcut icon" href="/favicon.ico?v=2">
|
||||||
|
<link rel="apple-touch-icon" href="/logo.png?v=2">
|
||||||
<style>
|
<style>
|
||||||
pre.light-code {
|
pre.light-code {
|
||||||
background: transparent !important;
|
background: transparent !important;
|
||||||
@@ -385,21 +390,33 @@
|
|||||||
marker.style.top = `${pctY}%`;
|
marker.style.top = `${pctY}%`;
|
||||||
marker.style.transform = 'translate(-50%, -50%)';
|
marker.style.transform = 'translate(-50%, -50%)';
|
||||||
marker.style.zIndex = '10';
|
marker.style.zIndex = '10';
|
||||||
|
marker.style.pointerEvents = 'none';
|
||||||
|
|
||||||
|
const dotSize = sequenceIndex !== undefined ? 20 : 14;
|
||||||
const dot = document.createElement('div');
|
const dot = document.createElement('div');
|
||||||
dot.style.width = '14px';
|
dot.style.width = `${dotSize}px`;
|
||||||
dot.style.height = '14px';
|
dot.style.height = `${dotSize}px`;
|
||||||
dot.style.borderRadius = '50%';
|
dot.style.borderRadius = '50%';
|
||||||
dot.style.backgroundColor = '#ef4444';
|
dot.style.backgroundColor = '#ef4444';
|
||||||
dot.style.border = '2px solid white';
|
dot.style.border = '2px solid white';
|
||||||
dot.style.boxShadow = '0 0 6px rgba(0,0,0,0.4)';
|
dot.style.boxShadow = '0 0 6px rgba(0,0,0,0.4)';
|
||||||
|
if (sequenceIndex !== undefined) {
|
||||||
|
dot.style.display = 'flex';
|
||||||
|
dot.style.alignItems = 'center';
|
||||||
|
dot.style.justifyContent = 'center';
|
||||||
|
dot.style.fontSize = '11px';
|
||||||
|
dot.style.fontWeight = 'bold';
|
||||||
|
dot.style.color = 'white';
|
||||||
|
dot.innerText = String(sequenceIndex);
|
||||||
|
}
|
||||||
|
|
||||||
|
const pulseSize = dotSize * 2;
|
||||||
const pulse = document.createElement('div');
|
const pulse = document.createElement('div');
|
||||||
pulse.style.position = 'absolute';
|
pulse.style.position = 'absolute';
|
||||||
pulse.style.top = '-9px';
|
pulse.style.top = `-${dotSize / 2 + 2}px`;
|
||||||
pulse.style.left = '-9px';
|
pulse.style.left = `-${dotSize / 2 + 2}px`;
|
||||||
pulse.style.width = '28px';
|
pulse.style.width = `${pulseSize}px`;
|
||||||
pulse.style.height = '28px';
|
pulse.style.height = `${pulseSize}px`;
|
||||||
pulse.style.borderRadius = '50%';
|
pulse.style.borderRadius = '50%';
|
||||||
pulse.style.border = '2px solid #ef4444';
|
pulse.style.border = '2px solid #ef4444';
|
||||||
pulse.style.animation = 'pulse-animation 1.5s infinite ease-out';
|
pulse.style.animation = 'pulse-animation 1.5s infinite ease-out';
|
||||||
@@ -409,14 +426,18 @@
|
|||||||
marker.appendChild(dot);
|
marker.appendChild(dot);
|
||||||
|
|
||||||
const label = document.createElement('div');
|
const label = document.createElement('div');
|
||||||
label.innerText = actionType.toUpperCase();
|
if (sequenceIndex !== undefined) {
|
||||||
|
label.innerText = totalSequence && totalSequence > 1 ? `#${sequenceIndex} TAP` : actionType.toUpperCase();
|
||||||
|
} else {
|
||||||
|
label.innerText = actionType.toUpperCase();
|
||||||
|
}
|
||||||
label.style.position = 'absolute';
|
label.style.position = 'absolute';
|
||||||
label.style.top = '16px';
|
label.style.top = `${dotSize + 2}px`;
|
||||||
label.style.left = '50%';
|
label.style.left = '50%';
|
||||||
label.style.transform = 'translateX(-50%)';
|
label.style.transform = 'translateX(-50%)';
|
||||||
label.style.backgroundColor = 'rgba(15, 23, 42, 0.8)';
|
label.style.backgroundColor = 'rgba(15, 23, 42, 0.9)';
|
||||||
label.style.color = 'white';
|
label.style.color = 'white';
|
||||||
label.style.padding = '1px 5px';
|
label.style.padding = '1.5px 5px';
|
||||||
label.style.borderRadius = '3px';
|
label.style.borderRadius = '3px';
|
||||||
label.style.fontSize = '8px';
|
label.style.fontSize = '8px';
|
||||||
label.style.fontWeight = 'bold';
|
label.style.fontWeight = 'bold';
|
||||||
@@ -427,6 +448,66 @@
|
|||||||
overlay.appendChild(marker);
|
overlay.appendChild(marker);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function createSequenceConnector(x1, y1, x2, y2, overlay, isNormalized) {
|
||||||
|
let pctX1 = x1 / 10;
|
||||||
|
let pctY1 = y1 / 10;
|
||||||
|
let pctX2 = x2 / 10;
|
||||||
|
let pctY2 = y2 / 10;
|
||||||
|
|
||||||
|
const container = overlay.parentElement;
|
||||||
|
const img = container ? container.querySelector('img') : null;
|
||||||
|
if (!isNormalized && img && img.naturalWidth && img.naturalHeight) {
|
||||||
|
if (x1 > 1000 || y1 > 1000 || x2 > 1000 || y2 > 1000 || (x1 > 100 && y1 > 100 && img.naturalWidth > 500)) {
|
||||||
|
pctX1 = (x1 / img.naturalWidth) * 100;
|
||||||
|
pctY1 = (y1 / img.naturalHeight) * 100;
|
||||||
|
pctX2 = (x2 / img.naturalWidth) * 100;
|
||||||
|
pctY2 = (y2 / img.naturalHeight) * 100;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let svg = overlay.querySelector('svg');
|
||||||
|
if (!svg) {
|
||||||
|
svg = document.createElementNS('http://www.w3.org/2000/svg', 'svg');
|
||||||
|
svg.setAttribute('width', '100%');
|
||||||
|
svg.setAttribute('height', '100%');
|
||||||
|
svg.style.position = 'absolute';
|
||||||
|
svg.style.top = '0';
|
||||||
|
svg.style.left = '0';
|
||||||
|
svg.style.pointerEvents = 'none';
|
||||||
|
|
||||||
|
const defs = document.createElementNS('http://www.w3.org/2000/svg', 'defs');
|
||||||
|
const marker = document.createElementNS('http://www.w3.org/2000/svg', 'marker');
|
||||||
|
marker.setAttribute('id', 'seq-arrow');
|
||||||
|
marker.setAttribute('viewBox', '0 0 10 10');
|
||||||
|
marker.setAttribute('refX', '5');
|
||||||
|
marker.setAttribute('refY', '5');
|
||||||
|
marker.setAttribute('markerWidth', '6');
|
||||||
|
marker.setAttribute('markerHeight', '6');
|
||||||
|
marker.setAttribute('orient', 'auto');
|
||||||
|
|
||||||
|
const path = document.createElementNS('http://www.w3.org/2000/svg', 'path');
|
||||||
|
path.setAttribute('d', 'M 0 0 L 10 5 L 0 10 z');
|
||||||
|
path.setAttribute('fill', '#ef4444');
|
||||||
|
|
||||||
|
marker.appendChild(path);
|
||||||
|
defs.appendChild(marker);
|
||||||
|
svg.appendChild(defs);
|
||||||
|
overlay.appendChild(svg);
|
||||||
|
}
|
||||||
|
|
||||||
|
const line = document.createElementNS('http://www.w3.org/2000/svg', 'line');
|
||||||
|
line.setAttribute('x1', `${pctX1}%`);
|
||||||
|
line.setAttribute('y1', `${pctY1}%`);
|
||||||
|
line.setAttribute('x2', `${pctX2}%`);
|
||||||
|
line.setAttribute('y2', `${pctY2}%`);
|
||||||
|
line.setAttribute('stroke', '#ef4444');
|
||||||
|
line.setAttribute('stroke-width', '2.5');
|
||||||
|
line.setAttribute('marker-end', 'url(#seq-arrow)');
|
||||||
|
line.setAttribute('stroke-dasharray', '4,4');
|
||||||
|
|
||||||
|
svg.appendChild(line);
|
||||||
|
}
|
||||||
|
|
||||||
function createLineMarker(x1, y1, x2, y2, overlay, isNormalized) {
|
function createLineMarker(x1, y1, x2, y2, overlay, isNormalized) {
|
||||||
let pctX1 = x1 / 10;
|
let pctX1 = x1 / 10;
|
||||||
let pctY1 = y1 / 10;
|
let pctY1 = y1 / 10;
|
||||||
@@ -518,9 +599,37 @@
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 1. Multi-point click sequence
|
||||||
|
const isSequenceAction = actionName.includes('sequence') || Boolean(act.sequence || args.sequence || act.normalized_sequence || args.normalized_sequence);
|
||||||
|
let rawSeq = act.normalized_sequence || args.normalized_sequence || act.sequence || args.sequence || act.targets || args.targets;
|
||||||
|
if (typeof rawSeq === 'string') {
|
||||||
|
try { rawSeq = JSON.parse(rawSeq); } catch (e) {}
|
||||||
|
}
|
||||||
|
if (Array.isArray(rawSeq) && rawSeq.length > 0) {
|
||||||
|
const validPoints = [];
|
||||||
|
rawSeq.forEach(pt => {
|
||||||
|
if (typeof pt === 'string') {
|
||||||
|
try { pt = JSON.parse(pt); } catch (e) {}
|
||||||
|
}
|
||||||
|
if (Array.isArray(pt) && pt.length >= 2 && typeof pt[0] === 'number' && typeof pt[1] === 'number') {
|
||||||
|
validPoints.push([pt[0], pt[1]]);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
if (validPoints.length > 0 && (isSequenceAction || validPoints.length > 1)) {
|
||||||
|
const isNormSeq = Boolean(act.normalized_sequence || args.normalized_sequence) || validPoints.every(p => p[0] <= 1000 && p[1] <= 1000);
|
||||||
|
for (let i = 0; i < validPoints.length - 1; i++) {
|
||||||
|
createSequenceConnector(validPoints[i][0], validPoints[i][1], validPoints[i+1][0], validPoints[i+1][1], overlayElement, isNormSeq);
|
||||||
|
}
|
||||||
|
validPoints.forEach((pt, idx) => {
|
||||||
|
createPointMarker(pt[0], pt[1], actionName, overlayElement, isNormSeq, idx + 1, validPoints.length);
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
const isSwipeLike = actionName.includes('swipe') || actionName.includes('scroll') || actionName.includes('drag') || actionName.includes('slide');
|
const isSwipeLike = actionName.includes('swipe') || actionName.includes('scroll') || actionName.includes('drag') || actionName.includes('slide');
|
||||||
|
|
||||||
// 1. Separate start and end coordinates
|
// 2. Separate start and end coordinates
|
||||||
const normStart = act.normalized_start_coordinates || args.normalized_start_coordinates;
|
const normStart = act.normalized_start_coordinates || args.normalized_start_coordinates;
|
||||||
const normEnd = act.normalized_end_coordinates || args.normalized_end_coordinates;
|
const normEnd = act.normalized_end_coordinates || args.normalized_end_coordinates;
|
||||||
const startCoords = normStart || act.start_coordinates || act.start_point || act.start || act.from ||
|
const startCoords = normStart || act.start_coordinates || act.start_point || act.start || act.from ||
|
||||||
@@ -538,11 +647,11 @@
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
// 2. 4-coordinate array
|
// 3. 4-coordinate array
|
||||||
const normCoords = act.normalized_coordinates || args.normalized_coordinates;
|
const normCoords = act.normalized_coordinates || args.normalized_coordinates;
|
||||||
const isNorm = Boolean(normCoords);
|
const isNorm = Boolean(normCoords);
|
||||||
const rawCoords = normCoords || act.coordinates || act.coords || act.target ||
|
const rawCoords = normCoords || act.coordinates || act.coords || act.target ||
|
||||||
args.target || args.coordinates || args.coords || args.sequence ||
|
args.target || args.coordinates || args.coords ||
|
||||||
(isSwipeLike && Array.isArray(args.action) ? args.action : null) ||
|
(isSwipeLike && Array.isArray(args.action) ? args.action : null) ||
|
||||||
(isSwipeLike && Array.isArray(act.action) ? act.action : null) ||
|
(isSwipeLike && Array.isArray(act.action) ? act.action : null) ||
|
||||||
(isSwipeLike && Array.isArray(args.gesture) ? args.gesture : null) ||
|
(isSwipeLike && Array.isArray(args.gesture) ? args.gesture : null) ||
|
||||||
@@ -557,17 +666,10 @@
|
|||||||
createPointMarker(rawCoords[0], rawCoords[1], actionName, overlayElement, isNorm);
|
createPointMarker(rawCoords[0], rawCoords[1], actionName, overlayElement, isNorm);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
} else if (Array.isArray(rawCoords[0]) && rawCoords[0].length === 2) {
|
|
||||||
rawCoords.forEach(pt => {
|
|
||||||
if (Array.isArray(pt) && pt.length === 2 && typeof pt[0] === 'number') {
|
|
||||||
createPointMarker(pt[0], pt[1], actionName, overlayElement, isNorm);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
return;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// 3. Directional swipe / scroll ONLY for swipe-like actions
|
// 4. Directional swipe / scroll ONLY for swipe-like actions
|
||||||
if (isSwipeLike) {
|
if (isSwipeLike) {
|
||||||
const rawDir = (typeof args.action === 'string' ? args.action : null) ||
|
const rawDir = (typeof args.action === 'string' ? args.action : null) ||
|
||||||
(typeof act.action === 'string' && act.action !== actionName ? act.action : null) ||
|
(typeof act.action === 'string' && act.action !== actionName ? act.action : null) ||
|
||||||
|
|||||||
@@ -1493,7 +1493,7 @@ class ReplayManager:
|
|||||||
f" offset={start_offset:.3f}s, duration={duration:.3f}s"
|
f" offset={start_offset:.3f}s, duration={duration:.3f}s"
|
||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
res = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
res = subprocess.run(cmd, capture_output=True, text=True)
|
||||||
if res.returncode != 0:
|
if res.returncode != 0:
|
||||||
print(
|
print(
|
||||||
f"Warning: ffmpeg failed with exit code {res.returncode}. stderr: {res.stderr}"
|
f"Warning: ffmpeg failed with exit code {res.returncode}. stderr: {res.stderr}"
|
||||||
|
|||||||
@@ -26,10 +26,9 @@ except ImportError:
|
|||||||
router = APIRouter(tags=["media"])
|
router = APIRouter(tags=["media"])
|
||||||
|
|
||||||
|
|
||||||
@router.get("/", response_class=HTMLResponse)
|
|
||||||
@router.get("/admin", response_class=HTMLResponse)
|
@router.get("/admin", response_class=HTMLResponse)
|
||||||
@router.get("/debug", response_class=HTMLResponse)
|
@router.get("/debug", response_class=HTMLResponse)
|
||||||
async def get_index():
|
async def get_admin_index():
|
||||||
admin_index = Path(__file__).resolve().parent.parent / "index.html"
|
admin_index = Path(__file__).resolve().parent.parent / "index.html"
|
||||||
if admin_index.exists():
|
if admin_index.exists():
|
||||||
return admin_index.read_text(encoding="utf-8")
|
return admin_index.read_text(encoding="utf-8")
|
||||||
|
|||||||
@@ -0,0 +1,348 @@
|
|||||||
|
# Copyright 2026 Google LLC
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
"""System Readiness & Diagnostics Router for Artemis Admin Console."""
|
||||||
|
|
||||||
|
from fastapi import APIRouter, HTTPException
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
from artemis.core.diagnostics import readiness_engine
|
||||||
|
from artemis.core.diagnostics.schema import SystemReadinessReport
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/api/system", tags=["system"])
|
||||||
|
|
||||||
|
|
||||||
|
class SelectDeviceRequest(BaseModel):
|
||||||
|
"""Payload to select an active target Android device."""
|
||||||
|
|
||||||
|
serial: str = Field(description="Serial number or identifier of the Android device to select")
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/readiness", response_model=SystemReadinessReport)
|
||||||
|
async def get_system_readiness() -> SystemReadinessReport:
|
||||||
|
"""Execute all diagnostic probes and return a comprehensive system readiness report."""
|
||||||
|
return await readiness_engine.run_all()
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/devices/select")
|
||||||
|
async def select_active_device(request: SelectDeviceRequest):
|
||||||
|
"""Select the active Android device or emulator for subsequent automated tasks."""
|
||||||
|
serial = request.serial.strip()
|
||||||
|
if not serial:
|
||||||
|
raise HTTPException(status_code=400, detail="Device serial cannot be empty.")
|
||||||
|
|
||||||
|
readiness_engine.set_active_device_serial(serial)
|
||||||
|
# Return updated readiness
|
||||||
|
report = await readiness_engine.run_all()
|
||||||
|
return {
|
||||||
|
"status": "success",
|
||||||
|
"selected_serial": serial,
|
||||||
|
"report": report,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/adb/restart")
|
||||||
|
async def restart_adb_server():
|
||||||
|
"""Restart local ADB server and return an updated readiness check."""
|
||||||
|
restart_result = await readiness_engine.restart_adb_server()
|
||||||
|
updated_report = await readiness_engine.run_all()
|
||||||
|
return {
|
||||||
|
"restart_result": restart_result,
|
||||||
|
"report": updated_report,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectAdbRequest(BaseModel):
|
||||||
|
"""Payload to connect to an Android device over Wi-Fi."""
|
||||||
|
|
||||||
|
host: str = Field(description="IP address of Android device")
|
||||||
|
port: int = Field(default=5555, description="Port number")
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/adb/connect")
|
||||||
|
async def connect_wireless_adb(request: ConnectAdbRequest):
|
||||||
|
"""Connect to a device over Wi-Fi and return updated readiness."""
|
||||||
|
connect_result = await readiness_engine.connect_wireless_adb(request.host, request.port)
|
||||||
|
updated_report = await readiness_engine.run_all()
|
||||||
|
return {
|
||||||
|
"connect_result": connect_result,
|
||||||
|
"report": updated_report,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class LaunchEmulatorRequest(BaseModel):
|
||||||
|
"""Payload to launch a local Android Virtual Device (AVD)."""
|
||||||
|
|
||||||
|
avd_name: str = Field(
|
||||||
|
description="Name of the installed AVD emulator to launch (e.g. Android_2)"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/emulator/launch")
|
||||||
|
async def launch_emulator(request: LaunchEmulatorRequest):
|
||||||
|
"""Launch an Android emulator in the background and return initiation status."""
|
||||||
|
avd_name = request.avd_name.strip()
|
||||||
|
if not avd_name:
|
||||||
|
raise HTTPException(status_code=400, detail="AVD name cannot be empty.")
|
||||||
|
|
||||||
|
launch_res = await readiness_engine.launch_emulator(avd_name)
|
||||||
|
return launch_res
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/emulator/status")
|
||||||
|
async def get_emulator_status():
|
||||||
|
"""Query real-time progress and logs of background emulator launch."""
|
||||||
|
return readiness_engine.get_emulator_status()
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/emulator/stop")
|
||||||
|
async def stop_emulator():
|
||||||
|
"""Stop active emulator process."""
|
||||||
|
return await readiness_engine.stop_emulator()
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/emulator/dismiss")
|
||||||
|
async def dismiss_emulator():
|
||||||
|
"""Dismiss emulator launch tracking state."""
|
||||||
|
return readiness_engine.dismiss_emulator()
|
||||||
|
|
||||||
|
|
||||||
|
class UpdateCredentialsRequest(BaseModel):
|
||||||
|
"""Payload to update and configure LLM or Vision OCR API credentials."""
|
||||||
|
|
||||||
|
provider: str = Field(
|
||||||
|
default="google",
|
||||||
|
description="Provider identifier (e.g. google, gemini, openai, anthropic, openrouter, ocr)",
|
||||||
|
)
|
||||||
|
api_key: str = Field(description="The secret API key string to configure")
|
||||||
|
persist_to_env: bool = Field(
|
||||||
|
default=True, description="Whether to persist the key to .env file"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ValidateCredentialsRequest(BaseModel):
|
||||||
|
"""Payload to test and verify LLM or Vision OCR API credentials without saving."""
|
||||||
|
|
||||||
|
provider: str = Field(
|
||||||
|
default="google",
|
||||||
|
description="Provider identifier (e.g. google, gemini, openai, anthropic, openrouter, ocr)",
|
||||||
|
)
|
||||||
|
api_key: str = Field(description="The secret API key string to test")
|
||||||
|
base_url: str | None = Field(default=None, description="Optional custom base URL")
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/credentials")
|
||||||
|
async def get_credentials():
|
||||||
|
"""Retrieve current configured API keys for management."""
|
||||||
|
from artemis.config import settings
|
||||||
|
|
||||||
|
gemini_k = settings.get_api_key("google")
|
||||||
|
openai_k = settings.get_api_key("openai")
|
||||||
|
anthropic_k = settings.get_api_key("anthropic")
|
||||||
|
openrouter_k = settings.get_api_key("openrouter")
|
||||||
|
ocr_k = settings.get_api_key("ocr")
|
||||||
|
return {
|
||||||
|
"google": gemini_k.get_secret_value() if gemini_k else None,
|
||||||
|
"gemini": gemini_k.get_secret_value() if gemini_k else None,
|
||||||
|
"openai": openai_k.get_secret_value() if openai_k else None,
|
||||||
|
"anthropic": anthropic_k.get_secret_value() if anthropic_k else None,
|
||||||
|
"openrouter": openrouter_k.get_secret_value() if openrouter_k else None,
|
||||||
|
"ocr": ocr_k.get_secret_value() if ocr_k else None,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/credentials/test")
|
||||||
|
async def test_credentials(request: ValidateCredentialsRequest):
|
||||||
|
"""Test and verify whether an API key is valid and usable with the corresponding provider endpoint."""
|
||||||
|
from artemis.utils.credentials_validator import validate_api_key
|
||||||
|
|
||||||
|
provider = request.provider.strip().lower()
|
||||||
|
key = request.api_key.strip()
|
||||||
|
|
||||||
|
if not key:
|
||||||
|
raise HTTPException(status_code=400, detail="API key cannot be empty.")
|
||||||
|
|
||||||
|
is_valid, message = await validate_api_key(
|
||||||
|
provider=provider,
|
||||||
|
api_key=key,
|
||||||
|
base_url=request.base_url,
|
||||||
|
)
|
||||||
|
if not is_valid:
|
||||||
|
raise HTTPException(status_code=400, detail=message)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"valid": True,
|
||||||
|
"provider": provider,
|
||||||
|
"message": message,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/credentials")
|
||||||
|
async def update_credentials(request: UpdateCredentialsRequest):
|
||||||
|
"""Dynamically configure and persist LLM or Vision API key, returning updated readiness report."""
|
||||||
|
from artemis.utils.credentials_validator import validate_api_key
|
||||||
|
|
||||||
|
provider = request.provider.strip().lower()
|
||||||
|
key = request.api_key.strip()
|
||||||
|
|
||||||
|
# If a non-empty key is provided, verify it before saving
|
||||||
|
if key:
|
||||||
|
is_valid, validation_msg = await validate_api_key(provider=provider, api_key=key)
|
||||||
|
if not is_valid:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=f"API key verification failed: {validation_msg}",
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
from artemis.config import settings
|
||||||
|
|
||||||
|
settings.set_api_key(provider, key, persist_to_env=request.persist_to_env)
|
||||||
|
|
||||||
|
# Re-run all diagnostic probes to build updated report
|
||||||
|
updated_report = await readiness_engine.run_all()
|
||||||
|
action_desc = (
|
||||||
|
"successfully verified, updated, and applied" if key else "successfully cleared"
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"status": "success",
|
||||||
|
"message": f"API key for {provider} {action_desc}.",
|
||||||
|
"provider": provider,
|
||||||
|
"report": updated_report,
|
||||||
|
}
|
||||||
|
except HTTPException:
|
||||||
|
raise
|
||||||
|
except Exception as exc:
|
||||||
|
raise HTTPException(status_code=500, detail=f"Failed to update credentials: {exc}")
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/model-config-env")
|
||||||
|
async def get_model_config_and_env():
|
||||||
|
"""Retrieve the current active artemis.jsonc configuration and .env status for custom setup."""
|
||||||
|
import os
|
||||||
|
from artemis.config.paths import ROOT_DIR, get_config_path
|
||||||
|
from artemis.config import settings
|
||||||
|
from artemis.utils.file import load_jsonc
|
||||||
|
|
||||||
|
from artemis.config.settings import is_placeholder_key
|
||||||
|
|
||||||
|
# 1. Config file resolution
|
||||||
|
config_path = None
|
||||||
|
config_content = ""
|
||||||
|
parsed_config = {}
|
||||||
|
try:
|
||||||
|
config_path_obj = get_config_path("artemis.jsonc")
|
||||||
|
config_path = str(config_path_obj)
|
||||||
|
config_content = config_path_obj.read_text(encoding="utf-8")
|
||||||
|
with open(config_path_obj, encoding="utf-8") as f:
|
||||||
|
parsed_config = load_jsonc(f)
|
||||||
|
except Exception as e:
|
||||||
|
config_content = f"// Error reading config: {e}"
|
||||||
|
|
||||||
|
# 2. Env file resolution
|
||||||
|
env_path = ROOT_DIR / ".env"
|
||||||
|
|
||||||
|
# 3. Relevant Env Variables status
|
||||||
|
def get_real_env_value(key_name: str, provider: str) -> str | None:
|
||||||
|
k = settings.get_api_key(provider)
|
||||||
|
if k and not is_placeholder_key(k):
|
||||||
|
return k.get_secret_value()
|
||||||
|
val = os.environ.get(key_name)
|
||||||
|
if val and val.strip() and not is_placeholder_key(val):
|
||||||
|
return val.strip()
|
||||||
|
return None
|
||||||
|
|
||||||
|
def mask_key(k: str | None) -> str | None:
|
||||||
|
if not k:
|
||||||
|
return None
|
||||||
|
if len(k) <= 8:
|
||||||
|
return "****"
|
||||||
|
return f"{k[:4]}...{k[-4:]}"
|
||||||
|
|
||||||
|
gemini_real = get_real_env_value("GEMINI_API_KEY", "google") or get_real_env_value(
|
||||||
|
"GOOGLE_API_KEY", "google"
|
||||||
|
)
|
||||||
|
openai_real = get_real_env_value("OPENAI_API_KEY", "openai")
|
||||||
|
anthropic_real = get_real_env_value("ANTHROPIC_API_KEY", "anthropic")
|
||||||
|
openrouter_real = get_real_env_value("OPEN_ROUTER_API_KEY", "openrouter")
|
||||||
|
xai_real = get_real_env_value("XAI_API_KEY", "xai")
|
||||||
|
base_url_val = settings.OPENAI_BASE_URL or os.environ.get("OPENAI_BASE_URL")
|
||||||
|
if base_url_val and is_placeholder_key(base_url_val):
|
||||||
|
base_url_val = None
|
||||||
|
ocr_real = get_real_env_value("OCR_API_KEY", "ocr") or get_real_env_value(
|
||||||
|
"VISION_API_KEY", "ocr"
|
||||||
|
)
|
||||||
|
|
||||||
|
env_vars = [
|
||||||
|
{
|
||||||
|
"name": "GEMINI_API_KEY",
|
||||||
|
"provider": "google",
|
||||||
|
"is_set": bool(gemini_real),
|
||||||
|
"preview": mask_key(gemini_real),
|
||||||
|
"description": "Google Gemini multimodal vision API key (free tier available)",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "OPENAI_API_KEY",
|
||||||
|
"provider": "openai",
|
||||||
|
"is_set": bool(openai_real),
|
||||||
|
"preview": mask_key(openai_real),
|
||||||
|
"description": "OpenAI API key (GPT-4o, GPT-4o-mini)",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "ANTHROPIC_API_KEY",
|
||||||
|
"provider": "anthropic",
|
||||||
|
"is_set": bool(anthropic_real),
|
||||||
|
"preview": mask_key(anthropic_real),
|
||||||
|
"description": "Anthropic Claude API key (Claude 3.5 Sonnet, Claude 3.7)",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "OPEN_ROUTER_API_KEY",
|
||||||
|
"provider": "openrouter",
|
||||||
|
"is_set": bool(openrouter_real),
|
||||||
|
"preview": mask_key(openrouter_real),
|
||||||
|
"description": "OpenRouter unified API gateway key",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "XAI_API_KEY",
|
||||||
|
"provider": "xai",
|
||||||
|
"is_set": bool(xai_real),
|
||||||
|
"preview": mask_key(xai_real),
|
||||||
|
"description": "xAI Grok vision API key",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "OPENAI_BASE_URL",
|
||||||
|
"provider": "custom",
|
||||||
|
"is_set": bool(base_url_val),
|
||||||
|
"preview": base_url_val,
|
||||||
|
"description": "Custom API endpoint (for local Ollama, vLLM, DeepSeek, or proxies)",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "VISION_API_KEY",
|
||||||
|
"provider": "ocr",
|
||||||
|
"is_set": bool(ocr_real),
|
||||||
|
"preview": mask_key(ocr_real),
|
||||||
|
"description": "Google Cloud Vision OCR key for screen text detection (optional)",
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
return {
|
||||||
|
"config_path": config_path or "config/artemis.jsonc",
|
||||||
|
"config_filename": "artemis.jsonc",
|
||||||
|
"config_content": config_content,
|
||||||
|
"default_model": parsed_config.get("default", {}),
|
||||||
|
"presets": parsed_config.get("presets", {}),
|
||||||
|
"env_path": str(env_path),
|
||||||
|
"env_filename": ".env",
|
||||||
|
"env_vars": env_vars,
|
||||||
|
}
|
||||||
@@ -23,6 +23,7 @@ try:
|
|||||||
from admin_console.schemas.task_schema import RunRequest
|
from admin_console.schemas.task_schema import RunRequest
|
||||||
from admin_console.services.ipc_service import ipc_service
|
from admin_console.services.ipc_service import ipc_service
|
||||||
from admin_console.services.model_service import model_service
|
from admin_console.services.model_service import model_service
|
||||||
|
from admin_console.services.task_preset_catalog import task_recommendation_engine
|
||||||
from admin_console.services.task_queue_service import task_queue_service
|
from admin_console.services.task_queue_service import task_queue_service
|
||||||
except ImportError:
|
except ImportError:
|
||||||
from apps.admin_console.core.state import state
|
from apps.admin_console.core.state import state
|
||||||
@@ -30,12 +31,37 @@ except ImportError:
|
|||||||
from apps.admin_console.schemas.task_schema import RunRequest
|
from apps.admin_console.schemas.task_schema import RunRequest
|
||||||
from apps.admin_console.services.ipc_service import ipc_service
|
from apps.admin_console.services.ipc_service import ipc_service
|
||||||
from apps.admin_console.services.model_service import model_service
|
from apps.admin_console.services.model_service import model_service
|
||||||
|
from apps.admin_console.services.task_preset_catalog import task_recommendation_engine
|
||||||
from apps.admin_console.services.task_queue_service import task_queue_service
|
from apps.admin_console.services.task_queue_service import task_queue_service
|
||||||
|
|
||||||
|
|
||||||
router = APIRouter(tags=["tasks"])
|
router = APIRouter(tags=["tasks"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/api/tasks/presets")
|
||||||
|
async def get_task_presets(
|
||||||
|
category: str = "recommended",
|
||||||
|
packages: str | None = None,
|
||||||
|
limit: int = 24,
|
||||||
|
):
|
||||||
|
"""Retrieve smart task recommendations optionally tailored to detected device packages."""
|
||||||
|
pkg_list = [p.strip() for p in packages.split(",") if p.strip()] if packages else None
|
||||||
|
return task_recommendation_engine.recommend_tasks(
|
||||||
|
installed_packages=pkg_list,
|
||||||
|
category=category,
|
||||||
|
limit=limit,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/api/tasks/catalog")
|
||||||
|
async def get_task_catalog():
|
||||||
|
"""Retrieve full catalog of predefined tasks and app package registry."""
|
||||||
|
return {
|
||||||
|
"tasks": [t.model_dump() for t in task_recommendation_engine.get_all_tasks()],
|
||||||
|
"app_registry": task_recommendation_engine.get_app_registry(),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
@router.post("/api/run")
|
@router.post("/api/run")
|
||||||
async def run_task(request: RunRequest):
|
async def run_task(request: RunRequest):
|
||||||
incoming_goals = []
|
incoming_goals = []
|
||||||
@@ -51,7 +77,12 @@ async def run_task(request: RunRequest):
|
|||||||
)
|
)
|
||||||
|
|
||||||
return await task_queue_service.enqueue_tasks(
|
return await task_queue_service.enqueue_tasks(
|
||||||
incoming_goals, profile=request.profile or "flash"
|
incoming_goals,
|
||||||
|
profile=request.profile or "flash",
|
||||||
|
expected_output=request.expected_output,
|
||||||
|
enable_outputter=request.enable_outputter,
|
||||||
|
locked_app_package=request.locked_app_package,
|
||||||
|
app_path=request.app_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -73,15 +104,33 @@ async def resume_task():
|
|||||||
|
|
||||||
@router.get("/api/status")
|
@router.get("/api/status")
|
||||||
async def get_status():
|
async def get_status():
|
||||||
|
# Watchdog check to ensure background worker is alive
|
||||||
|
task_queue_service.ensure_worker_running()
|
||||||
|
|
||||||
latest_session = session_repo.get_latest_session()
|
latest_session = session_repo.get_latest_session()
|
||||||
latest_session_id = latest_session.get("session_id") if latest_session else None
|
latest_session_id = latest_session.get("session_id") if latest_session else None
|
||||||
bg_tasks = session_repo.get_background_tasks(latest_session_id) if latest_session_id else []
|
bg_tasks = session_repo.get_background_tasks(latest_session_id) if latest_session_id else []
|
||||||
|
|
||||||
is_running = state.is_running
|
is_running = state.is_running
|
||||||
running_sid = state.active_session_id or (
|
running_task = next(
|
||||||
session_repo.get_running_session_id() if is_running else None
|
(t for t in state.queue_items if isinstance(t, dict) and t.get("status") == "running"), None
|
||||||
|
)
|
||||||
|
if not running_task and is_running:
|
||||||
|
running_task = next(
|
||||||
|
(t for t in state.queue_items if isinstance(t, dict) and t.get("status") == "pending"),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
|
||||||
|
running_sid = (
|
||||||
|
state.active_session_id
|
||||||
|
or (running_task.get("session_id") if running_task else None)
|
||||||
|
or (session_repo.get_running_session_id() if is_running else None)
|
||||||
|
)
|
||||||
|
running_goal = state.current_goal or (running_task.get("goal") if running_task else None)
|
||||||
|
|
||||||
|
active_profile = state.current_profile or (
|
||||||
|
running_task.get("profile") if running_task else None
|
||||||
)
|
)
|
||||||
active_profile = state.current_profile
|
|
||||||
if not active_profile and (running_sid or latest_session_id):
|
if not active_profile and (running_sid or latest_session_id):
|
||||||
check_sid = running_sid or latest_session_id
|
check_sid = running_sid or latest_session_id
|
||||||
sess_row = session_repo.get_session_by_id(check_sid)
|
sess_row = session_repo.get_session_by_id(check_sid)
|
||||||
@@ -95,30 +144,17 @@ async def get_status():
|
|||||||
model_info = model_service.get_active_model_info(active_profile)
|
model_info = model_service.get_active_model_info(active_profile)
|
||||||
queue_data = state.queue_tasks
|
queue_data = state.queue_tasks
|
||||||
|
|
||||||
if state.current_process:
|
if is_running:
|
||||||
if state.current_process.returncode is None:
|
is_paused = state.is_paused
|
||||||
is_paused = state.is_paused
|
return {
|
||||||
return {
|
"status": "paused" if is_paused else "running",
|
||||||
"status": "paused" if is_paused else "running",
|
"goal": running_goal,
|
||||||
"goal": state.current_goal,
|
"pid": state.current_process.pid if state.current_process else None,
|
||||||
"pid": state.current_process.pid,
|
"session_id": running_sid,
|
||||||
"session_id": running_sid,
|
"background_tasks": bg_tasks,
|
||||||
"background_tasks": bg_tasks,
|
"queue": queue_data,
|
||||||
"queue": queue_data,
|
"model_info": model_info,
|
||||||
"model_info": model_info,
|
}
|
||||||
}
|
|
||||||
else:
|
|
||||||
completed_status = {
|
|
||||||
"status": "completed",
|
|
||||||
"goal": state.current_goal,
|
|
||||||
"returncode": state.current_process.returncode,
|
|
||||||
"session_id": latest_session_id,
|
|
||||||
"background_tasks": bg_tasks,
|
|
||||||
"queue": queue_data,
|
|
||||||
"model_info": model_info,
|
|
||||||
}
|
|
||||||
state.current_process = None
|
|
||||||
return completed_status
|
|
||||||
|
|
||||||
if latest_session_id and str(latest_session_id) in state.active_connections:
|
if latest_session_id and str(latest_session_id) in state.active_connections:
|
||||||
conn_info = state.active_connections[str(latest_session_id)]
|
conn_info = state.active_connections[str(latest_session_id)]
|
||||||
@@ -151,21 +187,27 @@ async def stream_events(session_id: str, client: str | None = None):
|
|||||||
|
|
||||||
def callback(event_type, data):
|
def callback(event_type, data):
|
||||||
try:
|
try:
|
||||||
# Filter events by session_id when subscribed to a specific session
|
# Global queue lifecycle events should always be delivered
|
||||||
if session_id and session_id not in ("all", "active"):
|
if event_type not in (
|
||||||
evt_session_id = None
|
"session_started",
|
||||||
if isinstance(data, dict):
|
"session_ended",
|
||||||
evt_session_id = data.get("session_id")
|
"background_tasks_updated",
|
||||||
|
):
|
||||||
|
# Filter events by session_id when subscribed to a specific session
|
||||||
|
if session_id and session_id not in ("all", "active"):
|
||||||
|
evt_session_id = None
|
||||||
|
if isinstance(data, dict):
|
||||||
|
evt_session_id = data.get("session_id")
|
||||||
|
|
||||||
if evt_session_id and str(evt_session_id) != str(session_id):
|
if evt_session_id and str(evt_session_id) != str(session_id):
|
||||||
return
|
return
|
||||||
|
|
||||||
if (
|
if (
|
||||||
not evt_session_id
|
not evt_session_id
|
||||||
and state.active_session_id
|
and state.active_session_id
|
||||||
and str(state.active_session_id) != str(session_id)
|
and str(state.active_session_id) != str(session_id)
|
||||||
):
|
):
|
||||||
return
|
return
|
||||||
|
|
||||||
sanitized_data = ipc_service.sanitize_event_data(event_type, data)
|
sanitized_data = ipc_service.sanitize_event_data(event_type, data)
|
||||||
loop = asyncio.get_running_loop()
|
loop = asyncio.get_running_loop()
|
||||||
|
|||||||
@@ -19,6 +19,10 @@ class RunRequest(BaseModel):
|
|||||||
goal: str | None = None
|
goal: str | None = None
|
||||||
goals: list[str] | None = None
|
goals: list[str] | None = None
|
||||||
profile: str | None = "flash"
|
profile: str | None = "flash"
|
||||||
|
expected_output: str | None = None
|
||||||
|
enable_outputter: bool | None = None
|
||||||
|
locked_app_package: str | None = None
|
||||||
|
app_path: str | None = None
|
||||||
|
|
||||||
|
|
||||||
class ReplayRequest(BaseModel):
|
class ReplayRequest(BaseModel):
|
||||||
|
|||||||
@@ -42,6 +42,7 @@ for _p in (str(_workspace_root), str(_apps_dir), str(_admin_console_dir), str(_c
|
|||||||
|
|
||||||
from fastapi import FastAPI
|
from fastapi import FastAPI
|
||||||
from fastapi.middleware.cors import CORSMiddleware
|
from fastapi.middleware.cors import CORSMiddleware
|
||||||
|
from fastapi.responses import FileResponse, HTMLResponse
|
||||||
|
|
||||||
try:
|
try:
|
||||||
from admin_console.core.config import (
|
from admin_console.core.config import (
|
||||||
@@ -59,7 +60,7 @@ try:
|
|||||||
from admin_console.database.repositories.session_repository import session_repo
|
from admin_console.database.repositories.session_repository import session_repo
|
||||||
from admin_console.database.repositories.step_repository import step_repo
|
from admin_console.database.repositories.step_repository import step_repo
|
||||||
from admin_console.database.repositories.trace_repository import trace_repo
|
from admin_console.database.repositories.trace_repository import trace_repo
|
||||||
from admin_console.routers import media, replay, sessions, steps, stream, tasks
|
from admin_console.routers import media, replay, sessions, steps, stream, system, tasks
|
||||||
from admin_console.routers.replay import replay_manager
|
from admin_console.routers.replay import replay_manager
|
||||||
from admin_console.services.ipc_service import ipc_service
|
from admin_console.services.ipc_service import ipc_service
|
||||||
from admin_console.services.media_service import media_service
|
from admin_console.services.media_service import media_service
|
||||||
@@ -71,7 +72,7 @@ except ImportError:
|
|||||||
)
|
)
|
||||||
from apps.admin_console.core.state import state
|
from apps.admin_console.core.state import state
|
||||||
from apps.admin_console.database.repositories.session_repository import session_repo
|
from apps.admin_console.database.repositories.session_repository import session_repo
|
||||||
from apps.admin_console.routers import media, replay, sessions, steps, stream, tasks
|
from apps.admin_console.routers import media, replay, sessions, steps, stream, system, tasks
|
||||||
from apps.admin_console.routers.replay import replay_manager
|
from apps.admin_console.routers.replay import replay_manager
|
||||||
from apps.admin_console.services.ipc_service import ipc_service
|
from apps.admin_console.services.ipc_service import ipc_service
|
||||||
from apps.admin_console.services.model_service import model_service
|
from apps.admin_console.services.model_service import model_service
|
||||||
@@ -116,6 +117,7 @@ app.include_router(sessions.router)
|
|||||||
app.include_router(steps.router)
|
app.include_router(steps.router)
|
||||||
app.include_router(tasks.router)
|
app.include_router(tasks.router)
|
||||||
app.include_router(replay.router)
|
app.include_router(replay.router)
|
||||||
|
app.include_router(system.router)
|
||||||
|
|
||||||
# Mount cloud gateway router for Frappe / Cloud integration if present
|
# Mount cloud gateway router for Frappe / Cloud integration if present
|
||||||
try:
|
try:
|
||||||
@@ -131,6 +133,55 @@ except ImportError:
|
|||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------
|
||||||
|
# Showcase UI (Angular 19) Unified Single-Port SPA Static Hosting
|
||||||
|
# ------------------------------------------------------------------------------
|
||||||
|
_showcase_dist = _workspace_root / "apps" / "showcase_ui" / "dist" / "frontend" / "browser"
|
||||||
|
if not _showcase_dist.exists():
|
||||||
|
_showcase_dist = _workspace_root / "apps" / "showcase_ui" / "dist"
|
||||||
|
|
||||||
|
if _showcase_dist.is_dir() and (_showcase_dist / "index.html").exists():
|
||||||
|
|
||||||
|
@app.get("/{full_path:path}", include_in_schema=False)
|
||||||
|
async def serve_showcase_spa(full_path: str):
|
||||||
|
# Do not intercept API, media, or replay paths
|
||||||
|
if (
|
||||||
|
full_path.startswith("api/")
|
||||||
|
or full_path.startswith("images/")
|
||||||
|
or full_path.startswith("videos/")
|
||||||
|
or full_path.startswith("local_file")
|
||||||
|
or full_path == "docs"
|
||||||
|
or full_path == "openapi.json"
|
||||||
|
or full_path.startswith("redoc")
|
||||||
|
):
|
||||||
|
from fastapi import HTTPException
|
||||||
|
|
||||||
|
raise HTTPException(status_code=404, detail="Endpoint not found")
|
||||||
|
|
||||||
|
# Admin / Debug Console routes
|
||||||
|
if full_path in ("admin", "debug"):
|
||||||
|
admin_index = _admin_console_dir / "index.html"
|
||||||
|
if admin_index.exists():
|
||||||
|
return HTMLResponse(admin_index.read_text(encoding="utf-8"))
|
||||||
|
|
||||||
|
# Exact static file match (js, css, images, fonts, favicon, logo)
|
||||||
|
target_file = _showcase_dist / full_path
|
||||||
|
if full_path and target_file.is_file():
|
||||||
|
return FileResponse(target_file)
|
||||||
|
|
||||||
|
# Default fallback to Angular SPA index.html
|
||||||
|
index_file = _showcase_dist / "index.html"
|
||||||
|
return HTMLResponse(index_file.read_text(encoding="utf-8"))
|
||||||
|
else:
|
||||||
|
|
||||||
|
@app.get("/", response_class=HTMLResponse, include_in_schema=False)
|
||||||
|
async def fallback_root():
|
||||||
|
admin_index = _admin_console_dir / "index.html"
|
||||||
|
if admin_index.exists():
|
||||||
|
return HTMLResponse(admin_index.read_text(encoding="utf-8"))
|
||||||
|
return HTMLResponse("<h1>Artemis Console</h1>")
|
||||||
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------------------
|
# ------------------------------------------------------------------------------
|
||||||
# Backward compatibility exports
|
# Backward compatibility exports
|
||||||
# ------------------------------------------------------------------------------
|
# ------------------------------------------------------------------------------
|
||||||
|
|||||||
@@ -24,6 +24,8 @@ import subprocess
|
|||||||
import time
|
import time
|
||||||
from collections.abc import AsyncGenerator
|
from collections.abc import AsyncGenerator
|
||||||
|
|
||||||
|
from artemis.toolchain import find_adb
|
||||||
|
|
||||||
logger = logging.getLogger("artemis.stream_service")
|
logger = logging.getLogger("artemis.stream_service")
|
||||||
|
|
||||||
|
|
||||||
@@ -41,8 +43,9 @@ class DeviceStreamService:
|
|||||||
async def get_device_serial(self) -> str | None:
|
async def get_device_serial(self) -> str | None:
|
||||||
"""Find the currently connected active ADB device serial."""
|
"""Find the currently connected active ADB device serial."""
|
||||||
try:
|
try:
|
||||||
|
adb_bin = find_adb()
|
||||||
proc = await asyncio.create_subprocess_exec(
|
proc = await asyncio.create_subprocess_exec(
|
||||||
"adb",
|
adb_bin,
|
||||||
"devices",
|
"devices",
|
||||||
stdout=subprocess.PIPE,
|
stdout=subprocess.PIPE,
|
||||||
stderr=subprocess.PIPE,
|
stderr=subprocess.PIPE,
|
||||||
@@ -64,10 +67,11 @@ class DeviceStreamService:
|
|||||||
try:
|
try:
|
||||||
start_t = time.time()
|
start_t = time.time()
|
||||||
serial = await self.get_device_serial()
|
serial = await self.get_device_serial()
|
||||||
|
adb_bin = find_adb()
|
||||||
cmd = (
|
cmd = (
|
||||||
["adb", "-s", serial, "exec-out", "screencap", "-p"]
|
[adb_bin, "-s", serial, "exec-out", "screencap", "-p"]
|
||||||
if serial
|
if serial
|
||||||
else ["adb", "exec-out", "screencap", "-p"]
|
else [adb_bin, "exec-out", "screencap", "-p"]
|
||||||
)
|
)
|
||||||
proc = await asyncio.create_subprocess_exec(
|
proc = await asyncio.create_subprocess_exec(
|
||||||
*cmd,
|
*cmd,
|
||||||
|
|||||||
@@ -78,7 +78,7 @@ class IPCService:
|
|||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
if tr_type == "tool" and isinstance(payload, dict):
|
if tr_type in ("tool", "action") and isinstance(payload, dict):
|
||||||
args = payload.get("args", payload)
|
args = payload.get("args", payload)
|
||||||
data["payload"] = {"args": cls._clean_value(args)}
|
data["payload"] = {"args": cls._clean_value(args)}
|
||||||
elif tr_type == "llm_call" and isinstance(payload, dict):
|
elif tr_type == "llm_call" and isinstance(payload, dict):
|
||||||
@@ -144,8 +144,17 @@ class IPCService:
|
|||||||
data = payload.get("data")
|
data = payload.get("data")
|
||||||
|
|
||||||
if event_type == "session_started" and data:
|
if event_type == "session_started" and data:
|
||||||
state.active_session_id = data.get("session_id")
|
incoming_sid = data.get("session_id")
|
||||||
current_session_id = data.get("session_id")
|
if incoming_sid and str(incoming_sid) in getattr(
|
||||||
|
state, "cancelled_session_ids", set()
|
||||||
|
):
|
||||||
|
print(
|
||||||
|
f"[IPC] Ignoring session_started for cancelled session: {incoming_sid}"
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
|
state.active_session_id = incoming_sid
|
||||||
|
current_session_id = incoming_sid
|
||||||
pid = data.get("pid")
|
pid = data.get("pid")
|
||||||
goal = data.get("initial_goal")
|
goal = data.get("initial_goal")
|
||||||
device_info = data.get("device_info")
|
device_info = data.get("device_info")
|
||||||
|
|||||||
@@ -26,18 +26,18 @@ class ModelService:
|
|||||||
if "pro" in p_lower:
|
if "pro" in p_lower:
|
||||||
return {
|
return {
|
||||||
"name": "Pro",
|
"name": "Pro",
|
||||||
"id": "gemini-3.6-pro",
|
"id": "gemini-3.7-pro",
|
||||||
"provider": "google",
|
"provider": "google",
|
||||||
}
|
}
|
||||||
elif "flash" in p_lower:
|
elif "flash" in p_lower:
|
||||||
return {
|
return {
|
||||||
"name": "Flash",
|
"name": "Flash",
|
||||||
"id": "gemini-3.6-flash",
|
"id": "gemini-3.7-flash",
|
||||||
"provider": "google",
|
"provider": "google",
|
||||||
}
|
}
|
||||||
|
|
||||||
model_name = "Flash"
|
model_name = "Flash"
|
||||||
model_id = "gemini-3.6-flash"
|
model_id = "gemini-3.7-flash"
|
||||||
provider = "google"
|
provider = "google"
|
||||||
try:
|
try:
|
||||||
from artemis.config import parse_llm_config
|
from artemis.config import parse_llm_config
|
||||||
@@ -118,7 +118,8 @@ class ModelService:
|
|||||||
|
|
||||||
m_str = str(p_obj.get("model") or p_obj.get("model_name") or "").lower()
|
m_str = str(p_obj.get("model") or p_obj.get("model_name") or "").lower()
|
||||||
if (
|
if (
|
||||||
"gemini-3.6-pro" in m_str
|
"gemini-3.7-pro" in m_str
|
||||||
|
or "gemini-3.6-pro" in m_str
|
||||||
or "gemini-1.5-pro" in m_str
|
or "gemini-1.5-pro" in m_str
|
||||||
or "gpt-4" in m_str
|
or "gpt-4" in m_str
|
||||||
or "claude" in m_str
|
or "claude" in m_str
|
||||||
|
|||||||
@@ -0,0 +1,514 @@
|
|||||||
|
# Copyright 2026 Google LLC
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
"""Preset Task Catalog & Dynamic Recommendation Engine for Artemis."""
|
||||||
|
|
||||||
|
from typing import Any, Literal
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
|
||||||
|
class AppInfo(BaseModel):
|
||||||
|
"""Application metadata."""
|
||||||
|
|
||||||
|
name: str
|
||||||
|
icon: str
|
||||||
|
pkg: str
|
||||||
|
category: str = "general"
|
||||||
|
|
||||||
|
|
||||||
|
class TaskPreset(BaseModel):
|
||||||
|
"""Smart suggestion task definition."""
|
||||||
|
|
||||||
|
id: str
|
||||||
|
title: str
|
||||||
|
description: str
|
||||||
|
goal: str
|
||||||
|
profile: Literal["flash", "pro"]
|
||||||
|
category: Literal["flash", "pro", "cross_app", "monitor"]
|
||||||
|
tag: str
|
||||||
|
apps: list[AppInfo]
|
||||||
|
required_packages: list[str] = Field(default_factory=list)
|
||||||
|
match_mode: Literal["any", "all"] = "any"
|
||||||
|
priority: int = 50
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# APP PACKAGE REGISTRY
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
APP_REGISTRY: dict[str, dict[str, str]] = {
|
||||||
|
# Google Suite & System
|
||||||
|
"com.google.android.apps.maps": {"name": "Maps", "icon": "explore", "category": "navigation"},
|
||||||
|
"com.google.android.gm": {"name": "Gmail", "icon": "mail", "category": "productivity"},
|
||||||
|
"com.android.chrome": {"name": "Chrome", "icon": "public", "category": "browser"},
|
||||||
|
"com.google.android.youtube": {
|
||||||
|
"name": "YouTube",
|
||||||
|
"icon": "smart_display",
|
||||||
|
"category": "entertainment",
|
||||||
|
},
|
||||||
|
"com.android.settings": {"name": "Settings", "icon": "settings", "category": "system"},
|
||||||
|
"com.google.android.deskclock": {"name": "Clock", "icon": "timer", "category": "utility"},
|
||||||
|
"com.android.deskclock": {"name": "Clock", "icon": "timer", "category": "utility"},
|
||||||
|
"com.google.android.calculator": {
|
||||||
|
"name": "Calculator",
|
||||||
|
"icon": "calculate",
|
||||||
|
"category": "utility",
|
||||||
|
},
|
||||||
|
"com.android.calculator2": {"name": "Calculator", "icon": "calculate", "category": "utility"},
|
||||||
|
"com.google.android.apps.photos": {
|
||||||
|
"name": "Photos",
|
||||||
|
"icon": "photo_library",
|
||||||
|
"category": "media",
|
||||||
|
},
|
||||||
|
"com.google.android.calendar": {
|
||||||
|
"name": "Calendar",
|
||||||
|
"icon": "calendar_month",
|
||||||
|
"category": "productivity",
|
||||||
|
},
|
||||||
|
"com.google.android.keep": {
|
||||||
|
"name": "Keep Notes",
|
||||||
|
"icon": "note_alt",
|
||||||
|
"category": "productivity",
|
||||||
|
},
|
||||||
|
"com.android.vending": {"name": "Play Store", "icon": "storefront", "category": "tools"},
|
||||||
|
"com.google.android.apps.messaging": {
|
||||||
|
"name": "Messages",
|
||||||
|
"icon": "chat",
|
||||||
|
"category": "communication",
|
||||||
|
},
|
||||||
|
# Popular Ecosystem Apps
|
||||||
|
"com.tencent.mm": {"name": "WeChat", "icon": "forum", "category": "social"},
|
||||||
|
"com.xingin.xhs": {"name": "Xiaohongshu", "icon": "auto_stories", "category": "social"},
|
||||||
|
"com.sankuai.meituan": {"name": "Meituan", "icon": "restaurant", "category": "lifestyle"},
|
||||||
|
"com.dianping.v1": {"name": "Dianping", "icon": "star", "category": "lifestyle"},
|
||||||
|
"tv.danmaku.bili": {"name": "Bilibili", "icon": "video_library", "category": "entertainment"},
|
||||||
|
"com.eg.android.AlipayGphone": {
|
||||||
|
"name": "Alipay",
|
||||||
|
"icon": "account_balance_wallet",
|
||||||
|
"category": "finance",
|
||||||
|
},
|
||||||
|
"com.netease.cloudmusic": {
|
||||||
|
"name": "NetEase Music",
|
||||||
|
"icon": "headphones",
|
||||||
|
"category": "entertainment",
|
||||||
|
},
|
||||||
|
"com.spotify.music": {"name": "Spotify", "icon": "music_note", "category": "entertainment"},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# CURATED TASK PRESET LIBRARY (1-2 curated tasks per common app)
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
PRESET_TASK_CATALOG: list[TaskPreset] = [
|
||||||
|
# 1. Google Maps
|
||||||
|
TaskPreset(
|
||||||
|
id="maps_coffee",
|
||||||
|
title="Find Specialty Coffee",
|
||||||
|
description="Search nearby top-rated cafes in Google Maps",
|
||||||
|
goal="Open Google Maps, search for top-rated specialty coffee shops nearby, and view the top result details.",
|
||||||
|
profile="flash",
|
||||||
|
category="flash",
|
||||||
|
tag="Maps",
|
||||||
|
apps=[
|
||||||
|
AppInfo(
|
||||||
|
name="Maps",
|
||||||
|
icon="explore",
|
||||||
|
pkg="com.google.android.apps.maps",
|
||||||
|
category="navigation",
|
||||||
|
)
|
||||||
|
],
|
||||||
|
required_packages=["com.google.android.apps.maps"],
|
||||||
|
priority=95,
|
||||||
|
),
|
||||||
|
TaskPreset(
|
||||||
|
id="pro_commute_share",
|
||||||
|
title="Commute ETA & Message Draft",
|
||||||
|
description="Check transit time on Maps and draft arrival ETA in Messages",
|
||||||
|
goal="Open Google Maps to check commute time to the International Airport, calculate arrival time, then open Messages and draft an ETA text message.",
|
||||||
|
profile="pro",
|
||||||
|
category="cross_app",
|
||||||
|
tag="Maps + Messages",
|
||||||
|
apps=[
|
||||||
|
AppInfo(
|
||||||
|
name="Maps",
|
||||||
|
icon="explore",
|
||||||
|
pkg="com.google.android.apps.maps",
|
||||||
|
category="navigation",
|
||||||
|
),
|
||||||
|
AppInfo(
|
||||||
|
name="Messages",
|
||||||
|
icon="chat",
|
||||||
|
pkg="com.google.android.apps.messaging",
|
||||||
|
category="communication",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
required_packages=["com.google.android.apps.maps", "com.google.android.apps.messaging"],
|
||||||
|
match_mode="all",
|
||||||
|
priority=92,
|
||||||
|
),
|
||||||
|
# 2. Gmail
|
||||||
|
TaskPreset(
|
||||||
|
id="gmail_receipts",
|
||||||
|
title="Search Order Receipts",
|
||||||
|
description="Find recent flight or delivery confirmation emails in Gmail",
|
||||||
|
goal="Open Gmail and search for recent flight or package delivery confirmation emails.",
|
||||||
|
profile="flash",
|
||||||
|
category="flash",
|
||||||
|
tag="Gmail",
|
||||||
|
apps=[
|
||||||
|
AppInfo(name="Gmail", icon="mail", pkg="com.google.android.gm", category="productivity")
|
||||||
|
],
|
||||||
|
required_packages=["com.google.android.gm"],
|
||||||
|
priority=90,
|
||||||
|
),
|
||||||
|
TaskPreset(
|
||||||
|
id="pro_email_to_calendar",
|
||||||
|
title="Email Itinerary to Calendar",
|
||||||
|
description="Extract flight or event dates from Gmail and schedule in Calendar",
|
||||||
|
goal="Open Gmail to find the latest event invitation or itinerary, extract dates and location, then open Google Calendar and create a corresponding calendar event.",
|
||||||
|
profile="pro",
|
||||||
|
category="cross_app",
|
||||||
|
tag="Gmail + Calendar",
|
||||||
|
apps=[
|
||||||
|
AppInfo(
|
||||||
|
name="Gmail", icon="mail", pkg="com.google.android.gm", category="productivity"
|
||||||
|
),
|
||||||
|
AppInfo(
|
||||||
|
name="Calendar",
|
||||||
|
icon="calendar_month",
|
||||||
|
pkg="com.google.android.calendar",
|
||||||
|
category="productivity",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
required_packages=["com.google.android.gm"],
|
||||||
|
priority=94,
|
||||||
|
),
|
||||||
|
# 3. Chrome
|
||||||
|
TaskPreset(
|
||||||
|
id="chrome_research",
|
||||||
|
title="Search AI News Breakthroughs",
|
||||||
|
description="Search latest multimodal AI developments in Chrome browser",
|
||||||
|
goal="Open Chrome browser and search for latest breakthroughs in multimodal mobile AI agents.",
|
||||||
|
profile="flash",
|
||||||
|
category="flash",
|
||||||
|
tag="Chrome",
|
||||||
|
apps=[AppInfo(name="Chrome", icon="public", pkg="com.android.chrome", category="browser")],
|
||||||
|
required_packages=["com.android.chrome"],
|
||||||
|
priority=88,
|
||||||
|
),
|
||||||
|
TaskPreset(
|
||||||
|
id="pro_research_keep",
|
||||||
|
title="Product Research & Notes Note",
|
||||||
|
description="Compare top 3 headphones on Chrome and record comparison in Keep",
|
||||||
|
goal="Open Chrome, research top 3 noise-cancelling headphones comparing price and battery life, then write a structured comparison summary note in Keep Notes.",
|
||||||
|
profile="pro",
|
||||||
|
category="pro",
|
||||||
|
tag="Chrome + Keep",
|
||||||
|
apps=[
|
||||||
|
AppInfo(name="Chrome", icon="public", pkg="com.android.chrome", category="browser"),
|
||||||
|
AppInfo(
|
||||||
|
name="Keep Notes",
|
||||||
|
icon="note_alt",
|
||||||
|
pkg="com.google.android.keep",
|
||||||
|
category="productivity",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
required_packages=["com.android.chrome"],
|
||||||
|
priority=91,
|
||||||
|
),
|
||||||
|
# 4. YouTube
|
||||||
|
TaskPreset(
|
||||||
|
id="youtube_lofi",
|
||||||
|
title="Play Lo-Fi Music Radio",
|
||||||
|
description="Search and play a Lo-Fi hip hop live stream on YouTube",
|
||||||
|
goal='Open YouTube, search for "Lofi hip hop beats relaxing radio" and tap on the live stream.',
|
||||||
|
profile="flash",
|
||||||
|
category="flash",
|
||||||
|
tag="YouTube",
|
||||||
|
apps=[
|
||||||
|
AppInfo(
|
||||||
|
name="YouTube",
|
||||||
|
icon="smart_display",
|
||||||
|
pkg="com.google.android.youtube",
|
||||||
|
category="entertainment",
|
||||||
|
)
|
||||||
|
],
|
||||||
|
required_packages=["com.google.android.youtube"],
|
||||||
|
priority=85,
|
||||||
|
),
|
||||||
|
# 5. Settings
|
||||||
|
TaskPreset(
|
||||||
|
id="settings_display_wifi",
|
||||||
|
title="Dark Mode & Wi-Fi Check",
|
||||||
|
description="Toggle dark theme and verify network connection in Settings",
|
||||||
|
goal="Open Settings app, navigate to Display settings, ensure Dark theme is enabled, and check Wi-Fi connection status.",
|
||||||
|
profile="flash",
|
||||||
|
category="flash",
|
||||||
|
tag="Settings",
|
||||||
|
apps=[
|
||||||
|
AppInfo(name="Settings", icon="settings", pkg="com.android.settings", category="system")
|
||||||
|
],
|
||||||
|
required_packages=["com.android.settings"],
|
||||||
|
priority=87,
|
||||||
|
),
|
||||||
|
TaskPreset(
|
||||||
|
id="pro_settings_qa",
|
||||||
|
title="Subsystem Health & Crash Probe",
|
||||||
|
description="Traverse Settings submenus to verify screens and check for crash dialogs",
|
||||||
|
goal="Explore Settings submenus (Network, Connected devices, Apps, Battery, Storage), verify each screen loads properly without ANR or crash dialogs, and summarize results.",
|
||||||
|
profile="pro",
|
||||||
|
category="monitor",
|
||||||
|
tag="Settings QA",
|
||||||
|
apps=[
|
||||||
|
AppInfo(name="Settings", icon="settings", pkg="com.android.settings", category="system")
|
||||||
|
],
|
||||||
|
required_packages=["com.android.settings"],
|
||||||
|
priority=93,
|
||||||
|
),
|
||||||
|
# 6. Clock
|
||||||
|
TaskPreset(
|
||||||
|
id="clock_timer",
|
||||||
|
title="25-Min Pomodoro Timer",
|
||||||
|
description="Start a 25-minute focus countdown timer in Clock app",
|
||||||
|
goal="Open Clock app, switch to Timer tab, set 25 minutes and start the countdown timer.",
|
||||||
|
profile="flash",
|
||||||
|
category="flash",
|
||||||
|
tag="Clock",
|
||||||
|
apps=[
|
||||||
|
AppInfo(
|
||||||
|
name="Clock", icon="timer", pkg="com.google.android.deskclock", category="utility"
|
||||||
|
)
|
||||||
|
],
|
||||||
|
required_packages=["com.google.android.deskclock", "com.android.deskclock"],
|
||||||
|
priority=86,
|
||||||
|
),
|
||||||
|
# 7. Calculator
|
||||||
|
TaskPreset(
|
||||||
|
id="calc_gratuity",
|
||||||
|
title="Split Bill & Calculate Tip",
|
||||||
|
description="Calculate 18% gratuity on $186.40 for 3 people in Calculator",
|
||||||
|
goal="Open Calculator and calculate 18% tip on a bill of $186.40, then divide by 3 people.",
|
||||||
|
profile="flash",
|
||||||
|
category="flash",
|
||||||
|
tag="Calculator",
|
||||||
|
apps=[
|
||||||
|
AppInfo(
|
||||||
|
name="Calculator",
|
||||||
|
icon="calculate",
|
||||||
|
pkg="com.google.android.calculator",
|
||||||
|
category="utility",
|
||||||
|
)
|
||||||
|
],
|
||||||
|
required_packages=["com.google.android.calculator", "com.android.calculator2"],
|
||||||
|
priority=84,
|
||||||
|
),
|
||||||
|
# 8. Photos
|
||||||
|
TaskPreset(
|
||||||
|
id="photos_inspect",
|
||||||
|
title="Inspect Recent Screenshot",
|
||||||
|
description="Open Google Photos and review the latest screenshot taken",
|
||||||
|
goal="Open Google Photos and view the most recent screenshot in the screenshots album.",
|
||||||
|
profile="flash",
|
||||||
|
category="flash",
|
||||||
|
tag="Photos",
|
||||||
|
apps=[
|
||||||
|
AppInfo(
|
||||||
|
name="Photos",
|
||||||
|
icon="photo_library",
|
||||||
|
pkg="com.google.android.apps.photos",
|
||||||
|
category="media",
|
||||||
|
)
|
||||||
|
],
|
||||||
|
required_packages=["com.google.android.apps.photos"],
|
||||||
|
priority=82,
|
||||||
|
),
|
||||||
|
# 9. WeChat (微信)
|
||||||
|
TaskPreset(
|
||||||
|
id="wechat_browse",
|
||||||
|
title="Check WeChat Messages",
|
||||||
|
description="Open WeChat and view top recent chat conversations",
|
||||||
|
goal="Open WeChat (微信) and view the top recent chat messages.",
|
||||||
|
profile="flash",
|
||||||
|
category="flash",
|
||||||
|
tag="WeChat",
|
||||||
|
apps=[AppInfo(name="WeChat", icon="forum", pkg="com.tencent.mm", category="social")],
|
||||||
|
required_packages=["com.tencent.mm"],
|
||||||
|
priority=89,
|
||||||
|
),
|
||||||
|
TaskPreset(
|
||||||
|
id="pro_wechat_to_calendar",
|
||||||
|
title="WeChat Notice to Calendar",
|
||||||
|
description="Extract meeting notice from WeChat chat and add to Calendar",
|
||||||
|
goal="Open WeChat (微信), locate the latest meeting announcement or event message in the top chat, extract the time and topic, then open Calendar and schedule an event.",
|
||||||
|
profile="pro",
|
||||||
|
category="cross_app",
|
||||||
|
tag="WeChat + Calendar",
|
||||||
|
apps=[
|
||||||
|
AppInfo(name="WeChat", icon="forum", pkg="com.tencent.mm", category="social"),
|
||||||
|
AppInfo(
|
||||||
|
name="Calendar",
|
||||||
|
icon="calendar_month",
|
||||||
|
pkg="com.google.android.calendar",
|
||||||
|
category="productivity",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
required_packages=["com.tencent.mm"],
|
||||||
|
priority=93,
|
||||||
|
),
|
||||||
|
# 10. Xiaohongshu (小红书)
|
||||||
|
TaskPreset(
|
||||||
|
id="xhs_coffee_guide",
|
||||||
|
title="RED Cafe Guide Search",
|
||||||
|
description="Search trending specialty cafe reviews on Xiaohongshu",
|
||||||
|
goal="Open Xiaohongshu (小红书), search for top-rated specialty coffee shops, and view the top post.",
|
||||||
|
profile="flash",
|
||||||
|
category="flash",
|
||||||
|
tag="Xiaohongshu",
|
||||||
|
apps=[
|
||||||
|
AppInfo(
|
||||||
|
name="Xiaohongshu", icon="auto_stories", pkg="com.xingin.xhs", category="social"
|
||||||
|
)
|
||||||
|
],
|
||||||
|
required_packages=["com.xingin.xhs"],
|
||||||
|
priority=87,
|
||||||
|
),
|
||||||
|
# 11. Meituan / Dianping (美团 / 大众点评)
|
||||||
|
TaskPreset(
|
||||||
|
id="meituan_ramen_search",
|
||||||
|
title="Meituan Food Search",
|
||||||
|
description="Search top-rated Ramen nearby on Meituan or Dianping",
|
||||||
|
goal="Open Meituan (美团) or Dianping (大众点评), search for top-rated Ramen nearby, and view top restaurant rating.",
|
||||||
|
profile="flash",
|
||||||
|
category="flash",
|
||||||
|
tag="Meituan",
|
||||||
|
apps=[
|
||||||
|
AppInfo(
|
||||||
|
name="Meituan", icon="restaurant", pkg="com.sankuai.meituan", category="lifestyle"
|
||||||
|
)
|
||||||
|
],
|
||||||
|
required_packages=["com.sankuai.meituan", "com.dianping.v1"],
|
||||||
|
priority=86,
|
||||||
|
),
|
||||||
|
# 12. Bilibili (哔哩哔哩)
|
||||||
|
TaskPreset(
|
||||||
|
id="bilibili_stream",
|
||||||
|
title="Bilibili Tech Video",
|
||||||
|
description="Search and play an AI Agent tutorial video on Bilibili",
|
||||||
|
goal='Open Bilibili (哔哩哔哩), search for "AI Agent 架构实战", and play the top matching video.',
|
||||||
|
profile="flash",
|
||||||
|
category="flash",
|
||||||
|
tag="Bilibili",
|
||||||
|
apps=[
|
||||||
|
AppInfo(
|
||||||
|
name="Bilibili",
|
||||||
|
icon="video_library",
|
||||||
|
pkg="tv.danmaku.bili",
|
||||||
|
category="entertainment",
|
||||||
|
)
|
||||||
|
],
|
||||||
|
required_packages=["tv.danmaku.bili"],
|
||||||
|
priority=85,
|
||||||
|
),
|
||||||
|
# 13. Play Store
|
||||||
|
TaskPreset(
|
||||||
|
id="pro_playstore_review",
|
||||||
|
title="Play Store App Review Study",
|
||||||
|
description="Compare top task management apps and ratings on Google Play",
|
||||||
|
goal="Open Google Play Store, search for top rated task management apps, compare ratings and latest user reviews of the top 2 candidates, and record recommendations.",
|
||||||
|
profile="pro",
|
||||||
|
category="pro",
|
||||||
|
tag="Play Store",
|
||||||
|
apps=[
|
||||||
|
AppInfo(
|
||||||
|
name="Play Store", icon="storefront", pkg="com.android.vending", category="tools"
|
||||||
|
)
|
||||||
|
],
|
||||||
|
required_packages=["com.android.vending"],
|
||||||
|
priority=88,
|
||||||
|
),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# RECOMMENDATION ENGINE
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
|
||||||
|
class TaskRecommendationEngine:
|
||||||
|
"""Intelligent recommendation engine matching device capabilities."""
|
||||||
|
|
||||||
|
def __init__(self, catalog: list[TaskPreset] | None = None):
|
||||||
|
self.catalog = catalog or PRESET_TASK_CATALOG
|
||||||
|
|
||||||
|
def get_all_tasks(self) -> list[dict[str, Any]]:
|
||||||
|
return [task.model_dump() for task in self.catalog]
|
||||||
|
|
||||||
|
def recommend_tasks(
|
||||||
|
self, installed_packages: list[str] | set[str], category: str = "all", limit: int = 12
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
pkgs_set = (
|
||||||
|
set(installed_packages) if isinstance(installed_packages, list) else installed_packages
|
||||||
|
)
|
||||||
|
|
||||||
|
scored_tasks: list[tuple[int, TaskPreset, bool]] = []
|
||||||
|
|
||||||
|
for task in self.catalog:
|
||||||
|
# Check package presence
|
||||||
|
is_matched = False
|
||||||
|
if pkgs_set:
|
||||||
|
if not task.required_packages:
|
||||||
|
is_matched = True
|
||||||
|
elif task.match_mode == "all":
|
||||||
|
is_matched = all(pkg in pkgs_set for pkg in task.required_packages)
|
||||||
|
else:
|
||||||
|
is_matched = any(pkg in pkgs_set for pkg in task.required_packages)
|
||||||
|
else:
|
||||||
|
is_matched = False
|
||||||
|
|
||||||
|
# Calculate score
|
||||||
|
score = task.priority
|
||||||
|
if pkgs_set:
|
||||||
|
if is_matched:
|
||||||
|
score += 100
|
||||||
|
if len(task.required_packages) > 1 and task.match_mode == "all":
|
||||||
|
score += 30
|
||||||
|
else:
|
||||||
|
score -= 40
|
||||||
|
|
||||||
|
# Filter by category
|
||||||
|
if category == "flash" and task.profile != "flash":
|
||||||
|
continue
|
||||||
|
elif category == "pro" and task.profile != "pro":
|
||||||
|
continue
|
||||||
|
elif category == "cross_app" and task.category != "cross_app":
|
||||||
|
continue
|
||||||
|
elif category == "monitor" and task.category != "monitor":
|
||||||
|
continue
|
||||||
|
|
||||||
|
scored_tasks.append((score, task, is_matched))
|
||||||
|
|
||||||
|
# Sort descending by score
|
||||||
|
scored_tasks.sort(key=lambda item: item[0], reverse=True)
|
||||||
|
|
||||||
|
results: list[dict[str, Any]] = []
|
||||||
|
for _, task, matched in scored_tasks[:limit]:
|
||||||
|
d = task.model_dump()
|
||||||
|
d["is_device_matched"] = matched
|
||||||
|
results.append(d)
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
task_recommendation_engine = TaskRecommendationEngine()
|
||||||
@@ -16,7 +16,6 @@ import asyncio
|
|||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
import os
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import signal
|
|
||||||
import sys
|
import sys
|
||||||
import time
|
import time
|
||||||
from typing import Any
|
from typing import Any
|
||||||
@@ -31,6 +30,7 @@ try:
|
|||||||
)
|
)
|
||||||
from admin_console.core.state import state
|
from admin_console.core.state import state
|
||||||
from admin_console.database.repositories.session_repository import session_repo
|
from admin_console.database.repositories.session_repository import session_repo
|
||||||
|
from artemis.runtime import process_supervisor
|
||||||
except ImportError:
|
except ImportError:
|
||||||
from apps.admin_console.core.config import (
|
from apps.admin_console.core.config import (
|
||||||
PAUSE_FILE,
|
PAUSE_FILE,
|
||||||
@@ -40,6 +40,7 @@ except ImportError:
|
|||||||
)
|
)
|
||||||
from apps.admin_console.core.state import state
|
from apps.admin_console.core.state import state
|
||||||
from apps.admin_console.database.repositories.session_repository import session_repo
|
from apps.admin_console.database.repositories.session_repository import session_repo
|
||||||
|
from artemis.runtime import process_supervisor
|
||||||
|
|
||||||
|
|
||||||
class TaskQueueService:
|
class TaskQueueService:
|
||||||
@@ -48,51 +49,114 @@ class TaskQueueService:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
async def queue_worker(cls):
|
def _broadcast_event(cls, event_type: str, data: Any):
|
||||||
while True:
|
"""Broadcasts an event safely to all registered subscribers."""
|
||||||
goal = None
|
for cb in list(state.ipc_subscribers):
|
||||||
sess_id = None
|
|
||||||
try:
|
try:
|
||||||
item = await state.task_queue.get()
|
cb(event_type, data)
|
||||||
if isinstance(item, dict):
|
except Exception:
|
||||||
goal = item.get("goal")
|
pass
|
||||||
profile = item.get("profile", "flash")
|
|
||||||
sess_id = item.get("session_id")
|
@classmethod
|
||||||
else:
|
def _get_next_pending_task(cls) -> dict[str, Any] | None:
|
||||||
goal = item
|
"""Finds and returns the first pending task from queue_items."""
|
||||||
profile = "flash"
|
for item in state.queue_items:
|
||||||
sess_id = None
|
if isinstance(item, dict) and item.get("status") == "pending":
|
||||||
|
return item
|
||||||
|
return None
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def _remove_task(cls, session_id: str | None):
|
||||||
|
"""Removes a task from queue_items by session_id."""
|
||||||
|
if not session_id:
|
||||||
|
return
|
||||||
|
state.queue_items = [
|
||||||
|
t
|
||||||
|
for t in state.queue_items
|
||||||
|
if not (isinstance(t, dict) and t.get("session_id") == session_id)
|
||||||
|
]
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def ensure_worker_running(cls):
|
||||||
|
"""Guarantees that the background queue worker task is active and running."""
|
||||||
|
try:
|
||||||
|
loop = asyncio.get_running_loop()
|
||||||
|
if state.worker_task is None or state.worker_task.done():
|
||||||
|
state.worker_task = loop.create_task(cls.queue_worker())
|
||||||
|
try:
|
||||||
|
state.wake_event.set()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
except RuntimeError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
async def queue_worker(cls):
|
||||||
|
"""Persistent, self-healing background worker loop processing tasks in FIFO order."""
|
||||||
|
print("[QueueWorker] Background worker loop initialized and running.")
|
||||||
|
while True:
|
||||||
|
task_item = None
|
||||||
|
sess_id = None
|
||||||
|
goal = None
|
||||||
|
profile = "flash"
|
||||||
|
try:
|
||||||
|
if state.is_running:
|
||||||
|
await asyncio.sleep(0.2)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# 1. Fetch the next pending task
|
||||||
|
task_item = cls._get_next_pending_task()
|
||||||
|
if task_item is None:
|
||||||
|
await asyncio.sleep(0.3)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# 2. Extract and initialize task execution metadata
|
||||||
|
sess_id = task_item.get("session_id")
|
||||||
|
if sess_id and str(sess_id) in getattr(state, "cancelled_session_ids", set()):
|
||||||
|
cls._remove_task(sess_id)
|
||||||
|
continue
|
||||||
|
|
||||||
|
goal = task_item.get("goal")
|
||||||
|
profile = task_item.get("profile", "flash")
|
||||||
|
expected_output = task_item.get("expected_output")
|
||||||
|
enable_outputter = task_item.get("enable_outputter")
|
||||||
|
locked_app = task_item.get("locked_app_package") or task_item.get("locked_app")
|
||||||
|
app_path = task_item.get("app_path")
|
||||||
|
task_item["status"] = "running"
|
||||||
|
task_item["start_time"] = time.time()
|
||||||
|
|
||||||
state.current_goal = goal
|
state.current_goal = goal
|
||||||
state.current_profile = profile
|
state.current_profile = profile
|
||||||
if sess_id:
|
state.active_session_id = sess_id
|
||||||
state.active_session_id = sess_id
|
state.was_stopped_manually = False
|
||||||
|
|
||||||
# Remove from state.queue_tasks
|
# Broadcast session_started so all connected clients know the task has started
|
||||||
state.queue_tasks = [
|
cls._broadcast_event(
|
||||||
t
|
"session_started",
|
||||||
for t in state.queue_tasks
|
{
|
||||||
if not (
|
"session_id": sess_id,
|
||||||
isinstance(t, dict)
|
"initial_goal": goal,
|
||||||
and (t.get("session_id") == sess_id or t.get("goal") == goal)
|
"profile": profile,
|
||||||
)
|
},
|
||||||
]
|
)
|
||||||
|
|
||||||
test_name = f"web_{int(time.time())}"
|
test_name = f"web_{int(time.time())}"
|
||||||
env = os.environ.copy()
|
env = os.environ.copy()
|
||||||
pythonpath_parts = [
|
pythonpath_parts = [
|
||||||
str(WORKSPACE_ROOT),
|
str(WORKSPACE_ROOT),
|
||||||
str(WORKSPACE_ROOT / "apps/admin_console"),
|
str(WORKSPACE_ROOT / "apps" / "admin_console"),
|
||||||
str(WORKSPACE_ROOT / "apps/cloud_service"),
|
str(WORKSPACE_ROOT / "apps" / "cloud_service"),
|
||||||
env.get("PYTHONPATH", ""),
|
env.get("PYTHONPATH", ""),
|
||||||
]
|
]
|
||||||
|
env["PYTHONPATH"] = os.pathsep.join([p for p in pythonpath_parts if p])
|
||||||
|
env["PYTHONUTF8"] = "1"
|
||||||
|
env["PYTHONUNBUFFERED"] = "1"
|
||||||
if state.ipc_port is not None:
|
if state.ipc_port is not None:
|
||||||
env["ARTEMIS_IPC_PORT"] = str(state.ipc_port)
|
env["ARTEMIS_IPC_PORT"] = str(state.ipc_port)
|
||||||
if sess_id:
|
if sess_id:
|
||||||
env["ARTEMIS_SESSION_ID"] = str(sess_id)
|
env["ARTEMIS_SESSION_ID"] = str(sess_id)
|
||||||
|
|
||||||
print(f"[QueueWorker] Starting task: {goal} with profile: {profile}")
|
cmd = [
|
||||||
state.current_process = await asyncio.create_subprocess_exec(
|
|
||||||
sys.executable,
|
sys.executable,
|
||||||
"-m",
|
"-m",
|
||||||
"artemis.main",
|
"artemis.main",
|
||||||
@@ -101,48 +165,98 @@ class TaskQueueService:
|
|||||||
profile,
|
profile,
|
||||||
"--test-name",
|
"--test-name",
|
||||||
test_name,
|
test_name,
|
||||||
|
]
|
||||||
|
if expected_output:
|
||||||
|
cmd.extend(["--output-description", str(expected_output)])
|
||||||
|
if enable_outputter is not None:
|
||||||
|
cmd.append("--enable-outputter" if enable_outputter else "--disable-outputter")
|
||||||
|
if locked_app:
|
||||||
|
cmd.extend(["--locked-app", str(locked_app)])
|
||||||
|
if app_path:
|
||||||
|
cmd.extend(["--app-path", str(app_path)])
|
||||||
|
|
||||||
|
print(
|
||||||
|
f"[QueueWorker] Starting task [{sess_id}]: '{goal}' (profile: {profile}, outputter: {bool(expected_output or enable_outputter)})"
|
||||||
|
)
|
||||||
|
proc = await asyncio.create_subprocess_exec(
|
||||||
|
*cmd,
|
||||||
cwd=str(WORKSPACE_ROOT),
|
cwd=str(WORKSPACE_ROOT),
|
||||||
env=env,
|
env=env,
|
||||||
)
|
)
|
||||||
await state.current_process.wait()
|
state.current_process = proc
|
||||||
print(
|
task_item["pid"] = proc.pid
|
||||||
f"[QueueWorker] Task finished with code "
|
|
||||||
f"{state.current_process.returncode}: {goal}"
|
|
||||||
)
|
|
||||||
|
|
||||||
# Fallback status update
|
if sess_id and (
|
||||||
if state.active_session_id:
|
str(sess_id) in getattr(state, "cancelled_session_ids", set())
|
||||||
curr_status = session_repo.get_session_status(state.active_session_id)
|
or state.was_stopped_manually
|
||||||
if curr_status == "running":
|
):
|
||||||
if state.was_stopped_manually:
|
print(
|
||||||
new_status = "cancelled"
|
f"[QueueWorker] Task [{sess_id}] was cancelled during launch. Terminating."
|
||||||
else:
|
)
|
||||||
new_status = (
|
try:
|
||||||
"completed" if state.current_process.returncode == 0 else "failed"
|
proc.kill()
|
||||||
)
|
except Exception:
|
||||||
session_repo.update_session_status(
|
pass
|
||||||
state.active_session_id, new_status, time.time()
|
|
||||||
)
|
|
||||||
print(
|
|
||||||
f"[QueueWorker] Fallback: Updated session "
|
|
||||||
f"{state.active_session_id} status to {new_status}"
|
|
||||||
)
|
|
||||||
state.active_session_id = None
|
|
||||||
|
|
||||||
state.current_profile = None
|
# 3. Await subprocess completion
|
||||||
state.was_stopped_manually = False
|
returncode = await proc.wait()
|
||||||
|
print(f"[QueueWorker] Task [{sess_id}] exited with returncode {returncode}")
|
||||||
|
|
||||||
|
# 4. Perform fallback database status update and notification
|
||||||
|
if sess_id:
|
||||||
|
if state.was_stopped_manually:
|
||||||
|
new_status = "cancelled"
|
||||||
|
elif returncode == 0:
|
||||||
|
new_status = "completed"
|
||||||
|
else:
|
||||||
|
new_status = "failed"
|
||||||
|
session_repo.update_session_status(sess_id, new_status, time.time())
|
||||||
|
state.active_connections.pop(sess_id, None)
|
||||||
|
print(f"[QueueWorker] Updated session {sess_id} status to '{new_status}'")
|
||||||
|
cls._broadcast_event(
|
||||||
|
"session_ended",
|
||||||
|
{
|
||||||
|
"session_id": sess_id,
|
||||||
|
"status": new_status,
|
||||||
|
"was_stopped_manually": state.was_stopped_manually,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
except asyncio.CancelledError:
|
except asyncio.CancelledError:
|
||||||
print("[QueueWorker] Background worker task was cancelled.")
|
print("[QueueWorker] Task received cancellation signal.")
|
||||||
|
if state.current_process and state.current_process.returncode is None:
|
||||||
|
try:
|
||||||
|
state.current_process.kill()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
break
|
break
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"[QueueWorker] Error executing task: {e}")
|
print(f"[QueueWorker] Unexpected error executing task: {e}")
|
||||||
|
await asyncio.sleep(0.5)
|
||||||
finally:
|
finally:
|
||||||
if goal is not None:
|
# 5. Clean up completed task from queue and reset execution state
|
||||||
state.task_queue.task_done()
|
if sess_id:
|
||||||
|
cls._remove_task(sess_id)
|
||||||
|
state.current_process = None
|
||||||
|
state.active_session_id = None
|
||||||
|
state.current_goal = None
|
||||||
|
state.current_profile = None
|
||||||
|
state.was_stopped_manually = False
|
||||||
|
state.wake_event.set()
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
async def enqueue_tasks(cls, goals: list[str], profile: str = "flash") -> dict[str, Any]:
|
async def enqueue_tasks(
|
||||||
|
cls,
|
||||||
|
goals: list[str],
|
||||||
|
profile: str = "flash",
|
||||||
|
expected_output: str | None = None,
|
||||||
|
enable_outputter: bool | None = None,
|
||||||
|
locked_app_package: str | None = None,
|
||||||
|
app_path: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Enqueues one or more goals and wakes up the background worker."""
|
||||||
|
cls.ensure_worker_running()
|
||||||
|
|
||||||
enqueued_tasks = []
|
enqueued_tasks = []
|
||||||
now = time.time()
|
now = time.time()
|
||||||
for i, goal in enumerate(goals):
|
for i, goal in enumerate(goals):
|
||||||
@@ -151,14 +265,20 @@ class TaskQueueService:
|
|||||||
"session_id": sess_id,
|
"session_id": sess_id,
|
||||||
"goal": goal,
|
"goal": goal,
|
||||||
"profile": profile or "flash",
|
"profile": profile or "flash",
|
||||||
|
"expected_output": expected_output,
|
||||||
|
"enable_outputter": enable_outputter,
|
||||||
|
"locked_app_package": locked_app_package,
|
||||||
|
"app_path": app_path,
|
||||||
"status": "pending",
|
"status": "pending",
|
||||||
"created_at": now + i * 0.001,
|
"created_at": now + i * 0.001,
|
||||||
"start_time": now + i * 0.001,
|
"start_time": now + i * 0.001,
|
||||||
}
|
}
|
||||||
await state.task_queue.put(task_item)
|
state.queue_items.append(task_item)
|
||||||
state.queue_tasks.append(task_item)
|
|
||||||
enqueued_tasks.append(task_item)
|
enqueued_tasks.append(task_item)
|
||||||
|
|
||||||
|
# Wake worker immediately
|
||||||
|
state.wake_event.set()
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"status": "queued" if state.is_running else "started",
|
"status": "queued" if state.is_running else "started",
|
||||||
"tasks": enqueued_tasks,
|
"tasks": enqueued_tasks,
|
||||||
@@ -168,36 +288,105 @@ class TaskQueueService:
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def stop_tasks(cls, clear_all: bool = False) -> bool:
|
def stop_tasks(cls, clear_all: bool = False) -> bool:
|
||||||
|
"""Stops the currently executing task and optionally clears pending tasks."""
|
||||||
stopped = False
|
stopped = False
|
||||||
|
stopped_session_id = state.active_session_id
|
||||||
|
|
||||||
|
if not stopped_session_id:
|
||||||
|
running_item = next(
|
||||||
|
(
|
||||||
|
t
|
||||||
|
for t in state.queue_items
|
||||||
|
if isinstance(t, dict) and t.get("status") == "running"
|
||||||
|
),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if running_item:
|
||||||
|
stopped_session_id = running_item.get("session_id")
|
||||||
|
elif state.queue_items and not clear_all:
|
||||||
|
stopped_session_id = state.queue_items[0].get("session_id")
|
||||||
|
|
||||||
|
if stopped_session_id:
|
||||||
|
getattr(state, "cancelled_session_ids", set()).add(str(stopped_session_id))
|
||||||
|
for t in state.queue_items:
|
||||||
|
if isinstance(t, dict) and t.get("status") == "running" and t.get("session_id"):
|
||||||
|
getattr(state, "cancelled_session_ids", set()).add(str(t["session_id"]))
|
||||||
|
|
||||||
if clear_all:
|
if clear_all:
|
||||||
state.clear_queue()
|
state.clear_queue()
|
||||||
|
stopped = True
|
||||||
|
|
||||||
state.was_stopped_manually = True
|
state.was_stopped_manually = True
|
||||||
if state.is_running and state.current_process:
|
if state.current_process:
|
||||||
|
pid = getattr(state.current_process, "pid", None)
|
||||||
|
if pid:
|
||||||
|
try:
|
||||||
|
process_supervisor.terminate_tree(pid)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
try:
|
try:
|
||||||
state.current_process.kill()
|
state.current_process.kill()
|
||||||
stopped = True
|
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
stopped = True
|
||||||
|
|
||||||
for _, conn_info in list(state.active_connections.items()):
|
for _, conn_info in list(state.active_connections.items()):
|
||||||
pid = conn_info.get("pid")
|
pid = conn_info.get("pid")
|
||||||
if pid:
|
if pid:
|
||||||
try:
|
try:
|
||||||
os.kill(pid, signal.SIGINT)
|
process_supervisor.terminate_tree(pid)
|
||||||
stopped = True
|
stopped = True
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"Failed to kill external process {pid}: {e}")
|
print(f"Failed to kill external process {pid}: {e}")
|
||||||
|
|
||||||
|
# Always reset process and active states cleanly
|
||||||
|
state.current_process = None
|
||||||
|
state.active_connections.clear()
|
||||||
session_repo.mark_all_running_cancelled()
|
session_repo.mark_all_running_cancelled()
|
||||||
|
|
||||||
|
# Broadcast session_ended event so all SSE streams and subscribers update immediately
|
||||||
|
if stopped_session_id:
|
||||||
|
cls._broadcast_event(
|
||||||
|
"session_ended",
|
||||||
|
{
|
||||||
|
"session_id": stopped_session_id,
|
||||||
|
"status": "cancelled",
|
||||||
|
"was_stopped_manually": True,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
stopped = True
|
||||||
|
|
||||||
|
state.active_session_id = None
|
||||||
|
state.current_goal = None
|
||||||
|
state.current_profile = None
|
||||||
|
|
||||||
|
# Ensure any running or stopped task is removed from queue_items so worker_loop is unblocked
|
||||||
|
if clear_all:
|
||||||
|
state.queue_items = []
|
||||||
|
else:
|
||||||
|
state.queue_items = [
|
||||||
|
t
|
||||||
|
for t in state.queue_items
|
||||||
|
if not (
|
||||||
|
isinstance(t, dict)
|
||||||
|
and (
|
||||||
|
t.get("status") == "running"
|
||||||
|
or (stopped_session_id and t.get("session_id") == stopped_session_id)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
if PAUSE_FILE.exists():
|
if PAUSE_FILE.exists():
|
||||||
try:
|
try:
|
||||||
PAUSE_FILE.unlink()
|
PAUSE_FILE.unlink()
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
return stopped
|
# Wake up worker to process the next pending task or transition cleanly
|
||||||
|
cls.ensure_worker_running()
|
||||||
|
state.wake_event.set()
|
||||||
|
|
||||||
|
return stopped or True
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def resume_task(cls) -> bool:
|
def resume_task(cls) -> bool:
|
||||||
|
|||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# ✨ Artemis Showcase & User Workspace UI
|
||||||
|
|
||||||
|
A modern, highly aesthetic presentation and interaction frontend built with **Angular 19** and **SCSS** for showcasing the Artemis mobile autonomous agent.
|
||||||
|
|
||||||
|
## 🎨 Visual Design Highlights
|
||||||
|
|
||||||
|
* **Aurora Glow Dynamics**: Ambient background lighting with smooth gradients.
|
||||||
|
* **Glassmorphism Aesthetic**: Translucent frosted-glass panels with subtle borders.
|
||||||
|
* **Real-time Dual-Pane Workspace**:
|
||||||
|
- **Left Pane (`app-agent-stream`)**: Live streaming of Agent thought steps, plan breakdowns, tool actions, and status.
|
||||||
|
- **Right Pane (`app-chat-interface`)**: Natural language chat input sidebar with interactive feedback and status pills.
|
||||||
|
|
||||||
|
## 🚀 How to Run
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
- Node.js (>= 18)
|
||||||
|
- Backend API running on `http://localhost:8000` (via `apps/admin_console` or `artemis ui`)
|
||||||
|
|
||||||
|
### Development Server
|
||||||
|
```bash
|
||||||
|
cd apps/showcase_ui
|
||||||
|
npm install
|
||||||
|
npm start
|
||||||
|
```
|
||||||
|
This runs `ng serve --proxy-config proxy.conf.json` on **`http://localhost:4200/`**.
|
||||||
|
|
||||||
|
### Production Build
|
||||||
|
```bash
|
||||||
|
npm run build
|
||||||
|
```
|
||||||
|
Build artifacts will be emitted to `dist/`, which can be served statically by `apps.admin_console` or `artemis ui`.
|
||||||
@@ -45,13 +45,13 @@
|
|||||||
"budgets": [
|
"budgets": [
|
||||||
{
|
{
|
||||||
"type": "initial",
|
"type": "initial",
|
||||||
"maximumWarning": "500kB",
|
"maximumWarning": "2MB",
|
||||||
"maximumError": "1MB"
|
"maximumError": "4MB"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"type": "anyComponentStyle",
|
"type": "anyComponentStyle",
|
||||||
"maximumWarning": "50kB",
|
"maximumWarning": "200kB",
|
||||||
"maximumError": "100kB"
|
"maximumError": "400kB"
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"outputHashing": "all"
|
"outputHashing": "all"
|
||||||
+6215
-6560
File diff suppressed because it is too large.
Load diff
@@ -1,7 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "latte-showcase-ui",
|
"name": "artemis-showcase-ui",
|
||||||
"version": "1.0.0",
|
"version": "1.0.0",
|
||||||
|
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"ng": "ng",
|
"ng": "ng",
|
||||||
"start": "ng serve --proxy-config proxy.conf.json",
|
"start": "ng serve --proxy-config proxy.conf.json",
|
||||||
@@ -11,21 +10,20 @@
|
|||||||
},
|
},
|
||||||
"private": true,
|
"private": true,
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@angular/common": "^22.1.2",
|
"@angular/common": "^19.2.0",
|
||||||
"@angular/compiler": "^22.1.2",
|
"@angular/compiler": "^19.2.0",
|
||||||
"@angular/core": "^22.1.2",
|
"@angular/core": "^19.2.0",
|
||||||
"@angular/forms": "^19.2.0",
|
"@angular/forms": "^19.2.0",
|
||||||
"@angular/platform-browser": "^19.2.0",
|
"@angular/platform-browser": "^19.2.0",
|
||||||
"@angular/platform-browser-dynamic": "^19.2.0",
|
"@angular/platform-browser-dynamic": "^19.2.0",
|
||||||
"@angular/router": "^19.2.0",
|
"@angular/router": "^19.2.0",
|
||||||
"node": "^26.5.1",
|
|
||||||
"rxjs": "~7.8.0",
|
"rxjs": "~7.8.0",
|
||||||
"tslib": "^2.3.0",
|
"tslib": "^2.3.0",
|
||||||
"zone.js": "~0.15.0"
|
"zone.js": "~0.15.0"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"@angular-devkit/build-angular": "^22.1.4",
|
"@angular-devkit/build-angular": "^19.2.0",
|
||||||
"@angular/cli": "^22.1.4",
|
"@angular/cli": "^19.2.0",
|
||||||
"@angular/compiler-cli": "^19.2.0",
|
"@angular/compiler-cli": "^19.2.0",
|
||||||
"@types/jasmine": "~5.1.0",
|
"@types/jasmine": "~5.1.0",
|
||||||
"jasmine-core": "~5.6.0",
|
"jasmine-core": "~5.6.0",
|
||||||
@@ -35,5 +33,6 @@
|
|||||||
"karma-jasmine": "~5.1.0",
|
"karma-jasmine": "~5.1.0",
|
||||||
"karma-jasmine-html-reporter": "~2.1.0",
|
"karma-jasmine-html-reporter": "~2.1.0",
|
||||||
"typescript": "~5.7.2"
|
"typescript": "~5.7.2"
|
||||||
}
|
},
|
||||||
|
"packageManager": "pnpm@11.22.0+sha512.1ff870c4c6133dfd88fb2afc46dd13d47f09c9794b438c6fdb47ca98caf3bc16381ee0be93a091b8e3824cf01f889f46d7d9e20910fb0be1ab0fb5baa80dd621"
|
||||||
}
|
}
|
||||||
File renamed without changes.
Binary file not shown.
|
After Width: | Height: | Size: 51 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 221 KiB |
@@ -0,0 +1,56 @@
|
|||||||
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 240 240" width="240" height="240">
|
||||||
|
<defs>
|
||||||
|
<!-- Radial Glows for Mesh effect -->
|
||||||
|
<radialGradient id="redGlow" cx="50%" cy="50%" r="50%">
|
||||||
|
<stop offset="0%" stop-color="#EA4335" stop-opacity="1" />
|
||||||
|
<stop offset="40%" stop-color="#EA4335" stop-opacity="0.95" />
|
||||||
|
<stop offset="100%" stop-color="#EA4335" stop-opacity="0" />
|
||||||
|
</radialGradient>
|
||||||
|
<radialGradient id="yellowGlow" cx="50%" cy="50%" r="50%">
|
||||||
|
<stop offset="0%" stop-color="#FBBC05" stop-opacity="1" />
|
||||||
|
<stop offset="40%" stop-color="#FBBC05" stop-opacity="0.95" />
|
||||||
|
<stop offset="100%" stop-color="#FBBC05" stop-opacity="0" />
|
||||||
|
</radialGradient>
|
||||||
|
<radialGradient id="greenGlow" cx="50%" cy="50%" r="50%">
|
||||||
|
<stop offset="0%" stop-color="#34A853" stop-opacity="1" />
|
||||||
|
<stop offset="40%" stop-color="#34A853" stop-opacity="0.95" />
|
||||||
|
<stop offset="100%" stop-color="#34A853" stop-opacity="0" />
|
||||||
|
</radialGradient>
|
||||||
|
|
||||||
|
<!-- Robot Silhouette Mask -->
|
||||||
|
<mask id="robotMeshMask">
|
||||||
|
<rect width="240" height="240" fill="black" />
|
||||||
|
|
||||||
|
<!-- Antennae -->
|
||||||
|
<path d="M 90 60 Q 75 35 65 28" stroke="white" stroke-width="10" stroke-linecap="round" />
|
||||||
|
<circle cx="62" cy="24" r="10" fill="white" />
|
||||||
|
<path d="M 150 60 Q 165 35 175 28" stroke="white" stroke-width="10" stroke-linecap="round" />
|
||||||
|
<circle cx="178" cy="24" r="10" fill="white" />
|
||||||
|
|
||||||
|
<!-- Ears -->
|
||||||
|
<rect x="28" y="100" width="16" height="48" rx="8" fill="white" />
|
||||||
|
<rect x="196" y="100" width="16" height="48" rx="8" fill="white" />
|
||||||
|
|
||||||
|
<!-- Head -->
|
||||||
|
<rect x="44" y="58" width="152" height="132" rx="40" fill="white" />
|
||||||
|
|
||||||
|
<!-- Visor Cutout -->
|
||||||
|
<rect x="64" y="88" width="112" height="52" rx="22" fill="black" />
|
||||||
|
|
||||||
|
<!-- Eyes (Inside Visor) -->
|
||||||
|
<path d="M 82 118 Q 92 106 102 118" stroke="white" stroke-width="6" stroke-linecap="round" fill="none" />
|
||||||
|
<path d="M 138 118 Q 148 106 158 118" stroke="white" stroke-width="6" stroke-linecap="round" fill="none" />
|
||||||
|
|
||||||
|
<!-- Smile Cutout -->
|
||||||
|
<path d="M 104 158 Q 120 172 136 158" stroke="black" stroke-width="6" stroke-linecap="round" fill="none" />
|
||||||
|
</mask>
|
||||||
|
</defs>
|
||||||
|
|
||||||
|
<!-- Mesh Gradient applied to Mask -->
|
||||||
|
<g mask="url(#robotMeshMask)">
|
||||||
|
<rect width="240" height="240" fill="#4285F4" />
|
||||||
|
<circle cx="160" cy="50" r="100" fill="url(#redGlow)" />
|
||||||
|
<circle cx="60" cy="120" r="95" fill="url(#yellowGlow)" />
|
||||||
|
<circle cx="110" cy="200" r="95" fill="url(#greenGlow)" />
|
||||||
|
</g>
|
||||||
|
</svg>
|
||||||
|
After Width: | Height: | Size: 2.5 KiB |
@@ -14,4 +14,5 @@
|
|||||||
limitations under the License.
|
limitations under the License.
|
||||||
-->
|
-->
|
||||||
|
|
||||||
|
<app-nav-switcher></app-nav-switcher>
|
||||||
<router-outlet></router-outlet>
|
<router-outlet></router-outlet>
|
||||||
File renamed without changes.
@@ -15,12 +15,15 @@
|
|||||||
*/
|
*/
|
||||||
|
|
||||||
import { TestBed } from '@angular/core/testing';
|
import { TestBed } from '@angular/core/testing';
|
||||||
|
import { provideRouter } from '@angular/router';
|
||||||
|
import { provideHttpClient } from '@angular/common/http';
|
||||||
import { AppComponent } from './app.component';
|
import { AppComponent } from './app.component';
|
||||||
|
|
||||||
describe('AppComponent', () => {
|
describe('AppComponent', () => {
|
||||||
beforeEach(async () => {
|
beforeEach(async () => {
|
||||||
await TestBed.configureTestingModule({
|
await TestBed.configureTestingModule({
|
||||||
imports: [AppComponent],
|
imports: [AppComponent],
|
||||||
|
providers: [provideRouter([]), provideHttpClient()]
|
||||||
}).compileComponents();
|
}).compileComponents();
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -36,10 +39,4 @@ describe('AppComponent', () => {
|
|||||||
expect(app.title).toEqual('frontend');
|
expect(app.title).toEqual('frontend');
|
||||||
});
|
});
|
||||||
|
|
||||||
it('should render title', () => {
|
|
||||||
const fixture = TestBed.createComponent(AppComponent);
|
|
||||||
fixture.detectChanges();
|
|
||||||
const compiled = fixture.nativeElement as HTMLElement;
|
|
||||||
expect(compiled.querySelector('h1')?.textContent).toContain('Hello, frontend');
|
|
||||||
});
|
|
||||||
});
|
});
|
||||||
@@ -16,10 +16,11 @@
|
|||||||
|
|
||||||
import { Component } from '@angular/core';
|
import { Component } from '@angular/core';
|
||||||
import { RouterOutlet } from '@angular/router';
|
import { RouterOutlet } from '@angular/router';
|
||||||
|
import { NavSwitcherComponent } from './components/nav-switcher/nav-switcher.component';
|
||||||
|
|
||||||
@Component({
|
@Component({
|
||||||
selector: 'app-root',
|
selector: 'app-root',
|
||||||
imports: [RouterOutlet],
|
imports: [RouterOutlet, NavSwitcherComponent],
|
||||||
templateUrl: './app.component.html',
|
templateUrl: './app.component.html',
|
||||||
styleUrl: './app.component.scss'
|
styleUrl: './app.component.scss'
|
||||||
})
|
})
|
||||||
File renamed without changes.
@@ -15,10 +15,13 @@
|
|||||||
*/
|
*/
|
||||||
|
|
||||||
import { Routes } from '@angular/router';
|
import { Routes } from '@angular/router';
|
||||||
|
import { HomeComponent } from './pages/home/home.component';
|
||||||
import { WorkspaceComponent } from './pages/workspace/workspace.component';
|
import { WorkspaceComponent } from './pages/workspace/workspace.component';
|
||||||
import { LegacyWorkspaceComponent } from './pages/legacy-workspace/legacy-workspace.component';
|
import { LegacyWorkspaceComponent } from './pages/legacy-workspace/legacy-workspace.component';
|
||||||
|
|
||||||
export const routes: Routes = [
|
export const routes: Routes = [
|
||||||
{ path: '', component: WorkspaceComponent },
|
{ path: '', component: HomeComponent },
|
||||||
{ path: 'check', component: LegacyWorkspaceComponent }
|
{ path: 'workspace', component: WorkspaceComponent },
|
||||||
|
{ path: 'check', component: LegacyWorkspaceComponent },
|
||||||
|
{ path: '**', redirectTo: '' }
|
||||||
];
|
];
|
||||||
+177
-39
@@ -1,11 +1,29 @@
|
|||||||
|
<!--
|
||||||
|
Copyright 2026 Google LLC
|
||||||
|
|
||||||
|
Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
you may not use this file except in compliance with the License.
|
||||||
|
You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License.
|
||||||
|
-->
|
||||||
|
|
||||||
<div class="agent-stream-container">
|
<div class="agent-stream-container">
|
||||||
<!-- VS Code Style Navigation Header -->
|
<!-- VS Code Style Navigation Header (for responsive small screens) -->
|
||||||
<header class="vscode-nav-header">
|
<header class="vscode-nav-header">
|
||||||
<div class="vscode-tab-container">
|
<div class="vscode-tab-container">
|
||||||
<button
|
<button
|
||||||
class="vscode-tab-btn"
|
class="vscode-tab-btn"
|
||||||
[class.active]="isTaskDropdownOpen()"
|
[class.active]="isTaskDropdownOpen()"
|
||||||
(click)="toggleTaskDropdown($event)"
|
(click)="toggleTaskDropdown($event)"
|
||||||
|
title="Task Queue"
|
||||||
|
aria-label="Task Queue"
|
||||||
>
|
>
|
||||||
<span class="material-symbols-outlined tab-icon">list_alt</span>
|
<span class="material-symbols-outlined tab-icon">list_alt</span>
|
||||||
<span class="tab-label">Task Queue</span>
|
<span class="tab-label">Task Queue</span>
|
||||||
@@ -14,6 +32,8 @@
|
|||||||
class="vscode-tab-btn"
|
class="vscode-tab-btn"
|
||||||
[class.active]="isNotesDropdownOpen()"
|
[class.active]="isNotesDropdownOpen()"
|
||||||
(click)="toggleNotesDropdown($event)"
|
(click)="toggleNotesDropdown($event)"
|
||||||
|
title="Notes & Plans"
|
||||||
|
aria-label="Notes & Plans"
|
||||||
>
|
>
|
||||||
<span class="material-symbols-outlined tab-icon">description</span>
|
<span class="material-symbols-outlined tab-icon">description</span>
|
||||||
<span class="tab-label">Notes & Plans</span>
|
<span class="tab-label">Notes & Plans</span>
|
||||||
@@ -22,6 +42,10 @@
|
|||||||
|
|
||||||
<!-- Dropdown Popover Panel for Task Queue -->
|
<!-- Dropdown Popover Panel for Task Queue -->
|
||||||
<div class="vscode-dropdown-menu task-dropdown" *ngIf="isTaskDropdownOpen()">
|
<div class="vscode-dropdown-menu task-dropdown" *ngIf="isTaskDropdownOpen()">
|
||||||
|
<div class="popover-title-header">
|
||||||
|
<span class="material-symbols-outlined">list_alt</span>
|
||||||
|
<span>Task Queue</span>
|
||||||
|
</div>
|
||||||
<div class="dropdown-section-label">
|
<div class="dropdown-section-label">
|
||||||
<span class="material-symbols-outlined icon-bolt">bolt</span>
|
<span class="material-symbols-outlined icon-bolt">bolt</span>
|
||||||
<span>Active Queue ({{ activeQueue().length }})</span>
|
<span>Active Queue ({{ activeQueue().length }})</span>
|
||||||
@@ -43,7 +67,19 @@
|
|||||||
<span class="status-dot" *ngIf="getTaskStatus(session) === 'running'"></span>
|
<span class="status-dot" *ngIf="getTaskStatus(session) === 'running'"></span>
|
||||||
{{ getTaskStatus(session) | uppercase }}
|
{{ getTaskStatus(session) | uppercase }}
|
||||||
</span>
|
</span>
|
||||||
<span class="item-time">{{ session.start_time * 1000 | date:'shortTime' }}</span>
|
<div class="item-header-actions">
|
||||||
|
<span class="item-time">{{ session.start_time * 1000 | date:'shortTime' }}</span>
|
||||||
|
<button
|
||||||
|
*ngIf="getTaskStatus(session) !== 'running'"
|
||||||
|
type="button"
|
||||||
|
class="btn-delete-card"
|
||||||
|
(click)="deleteTask(session.session_id, $event)"
|
||||||
|
title="Delete this task"
|
||||||
|
aria-label="Delete this task"
|
||||||
|
>
|
||||||
|
<span class="material-symbols-outlined">delete</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="item-goal" title="{{ session.initial_goal }}">
|
<div class="item-goal" title="{{ session.initial_goal }}">
|
||||||
{{ session.initial_goal || 'Unnamed Task' }}
|
{{ session.initial_goal || 'Unnamed Task' }}
|
||||||
@@ -76,7 +112,19 @@
|
|||||||
<span class="status-dot" *ngIf="getTaskStatus(session) === 'running'"></span>
|
<span class="status-dot" *ngIf="getTaskStatus(session) === 'running'"></span>
|
||||||
{{ getTaskStatus(session) | uppercase }}
|
{{ getTaskStatus(session) | uppercase }}
|
||||||
</span>
|
</span>
|
||||||
<span class="item-time">{{ session.start_time * 1000 | date:'shortTime' }}</span>
|
<div class="item-header-actions">
|
||||||
|
<span class="item-time">{{ session.start_time * 1000 | date:'shortTime' }}</span>
|
||||||
|
<button
|
||||||
|
*ngIf="getTaskStatus(session) !== 'running'"
|
||||||
|
type="button"
|
||||||
|
class="btn-delete-card"
|
||||||
|
(click)="deleteTask(session.session_id, $event)"
|
||||||
|
title="Delete this task"
|
||||||
|
aria-label="Delete this task"
|
||||||
|
>
|
||||||
|
<span class="material-symbols-outlined">delete</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="item-goal" title="{{ session.initial_goal }}">
|
<div class="item-goal" title="{{ session.initial_goal }}">
|
||||||
{{ session.initial_goal || 'Unnamed Task' }}
|
{{ session.initial_goal || 'Unnamed Task' }}
|
||||||
@@ -91,6 +139,10 @@
|
|||||||
|
|
||||||
<!-- Dropdown Popover Panel for Notes & Plans -->
|
<!-- Dropdown Popover Panel for Notes & Plans -->
|
||||||
<div class="vscode-dropdown-menu notes-dropdown" *ngIf="isNotesDropdownOpen()">
|
<div class="vscode-dropdown-menu notes-dropdown" *ngIf="isNotesDropdownOpen()">
|
||||||
|
<div class="popover-title-header">
|
||||||
|
<span class="material-symbols-outlined">description</span>
|
||||||
|
<span>Notes & Plans</span>
|
||||||
|
</div>
|
||||||
<!-- Notes Selector / Pills (Shown if there are note files) -->
|
<!-- Notes Selector / Pills (Shown if there are note files) -->
|
||||||
<div class="notes-selector-container" *ngIf="noteKeys().length > 1">
|
<div class="notes-selector-container" *ngIf="noteKeys().length > 1">
|
||||||
<div class="notes-selector-list">
|
<div class="notes-selector-list">
|
||||||
@@ -232,7 +284,7 @@
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
<!-- Event Logs Section -->
|
<!-- Event Logs Section -->
|
||||||
<main class="stream-logs-content">
|
<main class="stream-logs-content" (scroll)="onStreamScroll($event)">
|
||||||
|
|
||||||
<div *ngIf="consolidatedBlocks().length === 0 && !showPlanningLoader()" class="empty-state">
|
<div *ngIf="consolidatedBlocks().length === 0 && !showPlanningLoader()" class="empty-state">
|
||||||
<div class="empty-icon"><span class="material-symbols-outlined">sensors</span></div>
|
<div class="empty-icon"><span class="material-symbols-outlined">sensors</span></div>
|
||||||
@@ -300,6 +352,69 @@
|
|||||||
<!-- Tool Call / Action section (Unified Chronological Loop matching Debug UI execution order) -->
|
<!-- Tool Call / Action section (Unified Chronological Loop matching Debug UI execution order) -->
|
||||||
<div class="step-actions-container" *ngIf="(block.data.generic_tools && block.data.generic_tools.length > 0) || block.data.action_taken">
|
<div class="step-actions-container" *ngIf="(block.data.generic_tools && block.data.generic_tools.length > 0) || block.data.action_taken">
|
||||||
<ng-container *ngFor="let item of getSortedStepEvents(block.data); let itemIdx = index; trackBy: trackStepEvent">
|
<ng-container *ngFor="let item of getSortedStepEvents(block.data); let itemIdx = index; trackBy: trackStepEvent">
|
||||||
|
<!-- Task Submission / Report Card (when item is report_task_status action or tool) -->
|
||||||
|
<div class="step-android-action-card task-report-card"
|
||||||
|
[class.is-completed]="getReportStatusValue(item.data) === 'completed' || getReportStatusValue(item.data) === 'success'"
|
||||||
|
[class.is-failed]="getReportStatusValue(item.data) === 'failed'"
|
||||||
|
[class.is-expanded]="isActionExpanded('action-' + block.id + '-' + itemIdx)"
|
||||||
|
*ngIf="isReportStatusAction(item.data)">
|
||||||
|
<div class="action-card-header" (click)="toggleActionExpanded('action-' + block.id + '-' + itemIdx, $event)">
|
||||||
|
<span class="material-symbols-outlined action-icon report-icon" [class.is-failed]="getReportStatusValue(item.data) === 'failed'">
|
||||||
|
{{ getReportStatusValue(item.data) === 'failed' ? 'cancel' : 'task_alt' }}
|
||||||
|
</span>
|
||||||
|
<span class="action-title">
|
||||||
|
{{ getReportStatusValue(item.data) === 'failed' ? 'Task Execution Failed' : 'Task Completed' }}
|
||||||
|
</span>
|
||||||
|
<span class="action-status-badge" [class.is-failed]="getReportStatusValue(item.data) === 'failed'" [class.is-completed]="getReportStatusValue(item.data) !== 'failed'">
|
||||||
|
<span class="material-symbols-outlined status-icon">
|
||||||
|
{{ getReportStatusValue(item.data) === 'failed' ? 'error' : 'check_circle' }}
|
||||||
|
</span>
|
||||||
|
{{ getReportStatusValue(item.data) === 'failed' ? 'Failed' : 'Completed' }}
|
||||||
|
</span>
|
||||||
|
<span class="action-expand-icon material-symbols-outlined">
|
||||||
|
{{ isActionExpanded('action-' + block.id + '-' + itemIdx) ? 'expand_less' : 'expand_more' }}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Compact preview body (when collapsed, matches standard action card) -->
|
||||||
|
<div class="action-card-body" *ngIf="!isActionExpanded('action-' + block.id + '-' + itemIdx) && getReportStatusExplanation(item.data)">
|
||||||
|
<div class="action-detail-row">
|
||||||
|
<span class="detail-label">Summary:</span>
|
||||||
|
<span class="detail-value">"{{ getReportStatusExplanation(item.data) }}"</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Detailed expanded view (when clicked, matches standard action card structure) -->
|
||||||
|
<div class="action-card-expanded-details" *ngIf="isActionExpanded('action-' + block.id + '-' + itemIdx)" (click)="$event.stopPropagation()">
|
||||||
|
<div class="expanded-grid">
|
||||||
|
<div class="grid-item full-width" *ngIf="getReportStatusExplanation(item.data) as expText">
|
||||||
|
<span class="grid-label">Report & Findings</span>
|
||||||
|
<span class="grid-value report-markdown-body" [innerHTML]="renderMarkdown(expText)"></span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Final Verified Screenshot -->
|
||||||
|
<div class="expanded-screenshots-container" *ngIf="getStepPreImageUrl(block.data, item.data) || getStepPostImageUrl(block.data, item.data)">
|
||||||
|
<div class="screenshot-column" *ngIf="getStepPreImageUrl(block.data, item.data)">
|
||||||
|
<span class="screenshot-label">Final Verified State</span>
|
||||||
|
<div class="image-wrapper" (click)="openImageModal(getStepPreImageUrl(block.data, item.data)!, item.data, $event)">
|
||||||
|
<div class="img-relative-box">
|
||||||
|
<img [src]="getStepPreImageUrl(block.data, item.data)" alt="Final Screen Screenshot" />
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="screenshot-column" *ngIf="getStepPostImageUrl(block.data, item.data)">
|
||||||
|
<span class="screenshot-label">Post Verification</span>
|
||||||
|
<div class="image-wrapper" (click)="openImageModal(getStepPostImageUrl(block.data, item.data)!, null, $event)">
|
||||||
|
<div class="img-relative-box">
|
||||||
|
<img [src]="getStepPostImageUrl(block.data, item.data)" alt="Post Screenshot" />
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
<!-- Android Action Card (when item is an action) -->
|
<!-- Android Action Card (when item is an action) -->
|
||||||
<div class="step-android-action-card"
|
<div class="step-android-action-card"
|
||||||
[class.is-running]="isItemRunning(item, block)"
|
[class.is-running]="isItemRunning(item, block)"
|
||||||
@@ -311,7 +426,7 @@
|
|||||||
<span class="action-title">{{ getActionTitle(item.data) }}</span>
|
<span class="action-title">{{ getActionTitle(item.data) }}</span>
|
||||||
<span class="action-status-badge is-failed" *ngIf="isActionFailed(item.data, block.data)">
|
<span class="action-status-badge is-failed" *ngIf="isActionFailed(item.data, block.data)">
|
||||||
<span class="material-symbols-outlined status-icon">error</span>
|
<span class="material-symbols-outlined status-icon">error</span>
|
||||||
{{ getActionErrorMessage(item.data, block.data) }}
|
Failed
|
||||||
</span>
|
</span>
|
||||||
<span class="action-expand-icon material-symbols-outlined">
|
<span class="action-expand-icon material-symbols-outlined">
|
||||||
{{ isActionExpanded('action-' + block.id + '-' + itemIdx) ? 'expand_less' : 'expand_more' }}
|
{{ isActionExpanded('action-' + block.id + '-' + itemIdx) ? 'expand_less' : 'expand_more' }}
|
||||||
@@ -330,7 +445,7 @@
|
|||||||
</div>
|
</div>
|
||||||
<div class="action-detail-row" *ngIf="getActionCoords(item.data)">
|
<div class="action-detail-row" *ngIf="getActionCoords(item.data)">
|
||||||
<span class="detail-label">Coordinates:</span>
|
<span class="detail-label">Coordinates:</span>
|
||||||
<span class="detail-value coordinate-badge">[{{ getActionCoords(item.data) }}]</span>
|
<span class="detail-value coordinate-badge">{{ getActionCoords(item.data) }}</span>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
@@ -347,7 +462,7 @@
|
|||||||
</div>
|
</div>
|
||||||
<div class="grid-item" *ngIf="getActionCoords(item.data)">
|
<div class="grid-item" *ngIf="getActionCoords(item.data)">
|
||||||
<span class="grid-label">Coordinates</span>
|
<span class="grid-label">Coordinates</span>
|
||||||
<span class="grid-value code-pill">[{{ getActionCoords(item.data) }}]</span>
|
<span class="grid-value code-pill">{{ getActionCoords(item.data) }}</span>
|
||||||
</div>
|
</div>
|
||||||
<div class="grid-item" *ngIf="getActionBounds(item.data)">
|
<div class="grid-item" *ngIf="getActionBounds(item.data)">
|
||||||
<span class="grid-label">Target Bounds</span>
|
<span class="grid-label">Target Bounds</span>
|
||||||
@@ -403,7 +518,7 @@
|
|||||||
<div class="step-android-action-card tool-action-card is-failed" *ngIf="item.data.type === 'llm_call' && item.data.status === 'failed'">
|
<div class="step-android-action-card tool-action-card is-failed" *ngIf="item.data.type === 'llm_call' && item.data.status === 'failed'">
|
||||||
<div class="action-card-header">
|
<div class="action-card-header">
|
||||||
<span class="material-symbols-outlined action-icon is-failed">error</span>
|
<span class="material-symbols-outlined action-icon is-failed">error</span>
|
||||||
<span class="action-title">Gemini AI Failed</span>
|
<span class="action-title">LLM Request Failed</span>
|
||||||
<span class="action-status-badge is-failed">
|
<span class="action-status-badge is-failed">
|
||||||
<span class="material-symbols-outlined status-icon">warning</span>
|
<span class="material-symbols-outlined status-icon">warning</span>
|
||||||
Failed
|
Failed
|
||||||
@@ -422,7 +537,7 @@
|
|||||||
[class.is-running]="isItemRunning(item, block)"
|
[class.is-running]="isItemRunning(item, block)"
|
||||||
[class.is-note-tool]="isNoteTool(item.data)"
|
[class.is-note-tool]="isNoteTool(item.data)"
|
||||||
[class.is-video-tool]="isVideoTool(item.data)"
|
[class.is-video-tool]="isVideoTool(item.data)"
|
||||||
*ngIf="shouldShowTool(item.data, block.data) && !isFailureAnalyzerActionTool(item.data)">
|
*ngIf="shouldShowTool(item.data, block.data) && !isDeviceActionTool(item.data)">
|
||||||
<div class="tool-call-label">
|
<div class="tool-call-label">
|
||||||
<span class="stream-status-dot" [class.streaming]="isItemRunning(item, block)"></span>
|
<span class="stream-status-dot" [class.streaming]="isItemRunning(item, block)"></span>
|
||||||
<div class="tool-call-text">
|
<div class="tool-call-text">
|
||||||
@@ -440,14 +555,14 @@
|
|||||||
</span>
|
</span>
|
||||||
</ng-container>
|
</ng-container>
|
||||||
|
|
||||||
<!-- Video Analyzer Pill -->
|
<!-- Video Tool Pill -->
|
||||||
<ng-container *ngIf="isVideoTool(item.data) && getVideoToolTarget(item.data) as videoTarget">
|
<ng-container *ngIf="isVideoTool(item.data) && getVideoToolTarget(item.data) as videoTarget">
|
||||||
<span
|
<span
|
||||||
class="tool-key-pill video-pill"
|
class="tool-key-pill video-pill"
|
||||||
(click)="onVideoToolClick(item.data)"
|
(click)="onVideoToolClick(item.data)"
|
||||||
title="Click to open video player"
|
title="Click to open video player"
|
||||||
>
|
>
|
||||||
<span class="material-symbols-outlined pill-icon">video_camera_back</span>
|
<span class="material-symbols-outlined pill-icon">videocam</span>
|
||||||
{{ videoTarget }}
|
{{ videoTarget }}
|
||||||
</span>
|
</span>
|
||||||
</ng-container>
|
</ng-container>
|
||||||
@@ -455,19 +570,20 @@
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<!-- Action Card format (ONLY for FailureAnalyzer non-note tools) -->
|
<!-- Action Card format (For Device Actions & ADB Commands) -->
|
||||||
<div class="step-android-action-card tool-action-card"
|
<div class="step-android-action-card tool-action-card"
|
||||||
[class.is-running]="isItemRunning(item, block)"
|
[class.is-running]="isItemRunning(item, block)"
|
||||||
[class.has-no-details]="isAdbCommandTool(item.data) ? !getAdbCommandLine(item.data) : (!getToolTargetText(item.data) && !getToolInputText(item.data) && !getToolCoords(item.data) && !getToolAnalysisText(item.data) && !getToolGenericDetails(item.data) && getToolExtraParams(item.data).length === 0)"
|
[class.has-no-details]="isAdbCommandTool(item.data) ? !getAdbCommandLine(item.data) : (!getToolTargetText(item.data) && !getToolInputText(item.data) && !getToolCoords(item.data) && !getToolAnalysisText(item.data) && !getToolGenericDetails(item.data) && getToolExtraParams(item.data).length === 0)"
|
||||||
[class.is-failed]="isToolFailed(item.data)"
|
[class.is-failed]="isToolFailed(item.data)"
|
||||||
[class.is-expanded]="isActionExpanded('tool-' + block.id + '-' + itemIdx)"
|
[class.is-expanded]="isActionExpanded('tool-' + block.id + '-' + itemIdx)"
|
||||||
*ngIf="shouldShowTool(item.data, block.data) && isFailureAnalyzerActionTool(item.data)">
|
*ngIf="shouldShowTool(item.data, block.data) && isDeviceActionTool(item.data)">
|
||||||
<div class="action-card-header" (click)="toggleActionExpanded('tool-' + block.id + '-' + itemIdx, $event)">
|
<div class="action-card-header" (click)="toggleActionExpanded('tool-' + block.id + '-' + itemIdx, $event)">
|
||||||
<span class="material-symbols-outlined action-icon" [class.is-failed]="isToolFailed(item.data)">{{ getToolIcon(item.data) }}</span>
|
<span class="material-symbols-outlined action-icon" [class.is-failed]="isToolFailed(item.data)">{{ getToolIcon(item.data) }}</span>
|
||||||
<span class="action-title">{{ getToolTitle(item.data) }}</span>
|
<span class="action-title">{{ getToolTitle(item.data) }}</span>
|
||||||
|
<span class="action-agent-badge" [class.is-outputter]="getToolAgentName(item.data) === 'Outputter'" *ngIf="getToolAgentName(item.data)">{{ getToolAgentName(item.data) }}</span>
|
||||||
<span class="action-status-badge is-failed" *ngIf="isToolFailed(item.data)">
|
<span class="action-status-badge is-failed" *ngIf="isToolFailed(item.data)">
|
||||||
<span class="material-symbols-outlined status-icon">error</span>
|
<span class="material-symbols-outlined status-icon">error</span>
|
||||||
{{ getToolErrorMessage(item.data) }}
|
Failed
|
||||||
</span>
|
</span>
|
||||||
<span class="action-expand-icon material-symbols-outlined">
|
<span class="action-expand-icon material-symbols-outlined">
|
||||||
{{ isActionExpanded('tool-' + block.id + '-' + itemIdx) ? 'expand_less' : 'expand_more' }}
|
{{ isActionExpanded('tool-' + block.id + '-' + itemIdx) ? 'expand_less' : 'expand_more' }}
|
||||||
@@ -504,7 +620,7 @@
|
|||||||
</div>
|
</div>
|
||||||
<div class="action-detail-row" *ngIf="getToolCoords(item.data)">
|
<div class="action-detail-row" *ngIf="getToolCoords(item.data)">
|
||||||
<span class="detail-label">Coordinates:</span>
|
<span class="detail-label">Coordinates:</span>
|
||||||
<span class="detail-value coordinate-badge">[{{ getToolCoords(item.data) }}]</span>
|
<span class="detail-value coordinate-badge">{{ getToolCoords(item.data) }}</span>
|
||||||
</div>
|
</div>
|
||||||
<div class="action-detail-row" *ngIf="getToolAnalysisText(item.data)">
|
<div class="action-detail-row" *ngIf="getToolAnalysisText(item.data)">
|
||||||
<span class="detail-label">Analysis:</span>
|
<span class="detail-label">Analysis:</span>
|
||||||
@@ -546,7 +662,7 @@
|
|||||||
</div>
|
</div>
|
||||||
<div class="grid-item" *ngIf="getToolCoords(item.data)">
|
<div class="grid-item" *ngIf="getToolCoords(item.data)">
|
||||||
<span class="grid-label">Coordinates</span>
|
<span class="grid-label">Coordinates</span>
|
||||||
<span class="grid-value code-pill">[{{ getToolCoords(item.data) }}]</span>
|
<span class="grid-value code-pill">{{ getToolCoords(item.data) }}</span>
|
||||||
</div>
|
</div>
|
||||||
<div class="grid-item" *ngIf="getToolAnalysisText(item.data)">
|
<div class="grid-item" *ngIf="getToolAnalysisText(item.data)">
|
||||||
<span class="grid-label">Analysis</span>
|
<span class="grid-label">Analysis</span>
|
||||||
@@ -600,38 +716,60 @@
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<!-- Outputter Post-Execution Synthesis Card (Rendered ONCE at the end of stream) -->
|
||||||
|
<div class="outputter-synthesis-stream-card" *ngIf="outputterReport() as outReport">
|
||||||
|
<div class="synthesis-card-header">
|
||||||
|
<div class="header-left">
|
||||||
|
<span class="material-symbols-outlined icon-sparkle">assignment_turned_in</span>
|
||||||
|
<span class="main-title">Task Report</span>
|
||||||
|
</div>
|
||||||
|
<div class="header-right">
|
||||||
|
<button type="button" class="btn-open-note" (click)="openOutputterNote($event)">
|
||||||
|
<span class="material-symbols-outlined">visibility</span>
|
||||||
|
<span>View in Notes</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="synthesis-card-preview">
|
||||||
|
<div class="parsed-note-container" *ngIf="getParsedNote(outReport) as parsedOut">
|
||||||
|
<h3 class="parsed-note-title" *ngIf="parsedOut.title">{{ parsedOut.title }}</h3>
|
||||||
|
<div class="parsed-milestones" *ngIf="parsedOut.milestones.length > 0">
|
||||||
|
<div class="milestone-item" *ngFor="let m of parsedOut.milestones" [class]="'status-' + m.type">
|
||||||
|
<span class="material-symbols-outlined status-icon">
|
||||||
|
{{ m.type === 'checked' ? 'check_circle' : (m.type === 'progress' ? 'pending' : 'radio_button_unchecked') }}
|
||||||
|
</span>
|
||||||
|
<span class="milestone-text">
|
||||||
|
<ng-container *ngFor="let seg of m.segments">
|
||||||
|
<strong *ngIf="seg.bold">{{ seg.text }}</strong>
|
||||||
|
<code *ngIf="seg.code">{{ seg.text }}</code>
|
||||||
|
<span *ngIf="!seg.bold && !seg.code">{{ seg.text }}</span>
|
||||||
|
</ng-container>
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="parsed-other-lines" *ngIf="parsedOut.otherLines.length > 0">
|
||||||
|
<p class="other-line" *ngFor="let line of parsedOut.otherLines" [class]="'line-' + line.type">
|
||||||
|
<ng-container *ngFor="let seg of line.segments">
|
||||||
|
<strong *ngIf="seg.bold">{{ seg.text }}</strong>
|
||||||
|
<code *ngIf="seg.code">{{ seg.text }}</code>
|
||||||
|
<span *ngIf="!seg.bold && !seg.code">{{ seg.text }}</span>
|
||||||
|
</ng-container>
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
<!-- Planning Loader -->
|
<!-- Planning Loader -->
|
||||||
<div class="planning-loader-container" *ngIf="showPlanningLoader()">
|
<div class="planning-loader-container" *ngIf="showPlanningLoader()">
|
||||||
<div class="planning-loader">
|
<div class="planning-loader">
|
||||||
<span class="material-symbols-outlined spin-animation">progress_activity</span>
|
<span class="material-symbols-outlined spin-animation">progress_activity</span>
|
||||||
<span class="loader-text">Planning next steps...</span>
|
<span class="loader-text">{{ currentPlanningText() }}</span>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</main>
|
</main>
|
||||||
|
|
||||||
<!-- Floating Action Buttons for Task Execution (Icon-only) -->
|
|
||||||
<div class="fab-actions-container">
|
|
||||||
<button
|
|
||||||
*ngIf="agentService.isPaused()"
|
|
||||||
class="fab-btn fab-resume-task"
|
|
||||||
(click)="agentService.resumeTask()"
|
|
||||||
title="Resume task"
|
|
||||||
aria-label="Resume Task"
|
|
||||||
>
|
|
||||||
<span class="material-symbols-outlined">play_arrow</span>
|
|
||||||
</button>
|
|
||||||
<button
|
|
||||||
*ngIf="agentService.isRunningTask() && !agentService.isPaused()"
|
|
||||||
class="fab-btn fab-stop-task"
|
|
||||||
(click)="agentService.stopTask()"
|
|
||||||
title="Stop running task"
|
|
||||||
aria-label="Stop Task"
|
|
||||||
>
|
|
||||||
<span class="material-symbols-outlined">stop</span>
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- Fullscreen Image Zoom Lightbox Modal -->
|
<!-- Fullscreen Image Zoom Lightbox Modal -->
|
||||||
<div class="image-modal-backdrop" *ngIf="selectedZoomImage()" (click)="closeImageModal($event)">
|
<div class="image-modal-backdrop" *ngIf="selectedZoomImage()" (click)="closeImageModal($event)">
|
||||||
<div class="image-modal-content" (click)="$event.stopPropagation()">
|
<div class="image-modal-content" (click)="$event.stopPropagation()">
|
||||||
+599
-231
File diff suppressed because it is too large.
Load diff
+193
-49
@@ -14,15 +14,35 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { Component, signal, computed, effect, inject, untracked } from '@angular/core';
|
import { Component, signal, computed, effect, inject, untracked, DestroyRef, AfterViewInit } from '@angular/core';
|
||||||
import { CommonModule } from '@angular/common';
|
import { CommonModule } from '@angular/common';
|
||||||
import { FormsModule } from '@angular/forms';
|
import { FormsModule } from '@angular/forms';
|
||||||
|
import { RouterLink } from '@angular/router';
|
||||||
import { HttpClient } from '@angular/common/http';
|
import { HttpClient } from '@angular/common/http';
|
||||||
import { AgentService } from '../../services/agent.service';
|
import { AgentService } from '../../services/agent.service';
|
||||||
import { Session } from '../../core/models/session.model';
|
import { Session } from '../../core/models/session.model';
|
||||||
import { MarkdownSegment, MarkdownLine, NoteMilestone, ParsedNote } from '../../core/models/markdown.model';
|
import { MarkdownSegment, MarkdownLine, NoteMilestone, ParsedNote } from '../../core/models/markdown.model';
|
||||||
import { StepBlock, PhaseBlock, StepEvent, ActionParam, CheckerResult } from '../../core/models/stream.model';
|
import { StepBlock, PhaseBlock, StepEvent, ActionParam, CheckerResult } from '../../core/models/stream.model';
|
||||||
|
|
||||||
|
export const PLANNING_LOADER_PHRASES: string[] = [
|
||||||
|
'Planning next step...',
|
||||||
|
'Analyzing screen coordinates...',
|
||||||
|
'Consulting neural network...',
|
||||||
|
'Formulating tactical action plan...',
|
||||||
|
'Deciphering UI state & elements...',
|
||||||
|
'Synthesizing decision pathways...',
|
||||||
|
'Calibrating next move...',
|
||||||
|
'Aligning logical vectors...',
|
||||||
|
'Evaluating optimal sub-goals...',
|
||||||
|
'Gathering sensory inputs...',
|
||||||
|
'Computing next interaction...',
|
||||||
|
'Optimizing execution strategy...',
|
||||||
|
'Simulating probable outcomes...',
|
||||||
|
'Summoning AI intuition...',
|
||||||
|
'Brewing the next command...',
|
||||||
|
'Strategizing tactical moves...'
|
||||||
|
];
|
||||||
|
|
||||||
import {
|
import {
|
||||||
parseNote,
|
parseNote,
|
||||||
parseNoteLines,
|
parseNoteLines,
|
||||||
@@ -46,7 +66,10 @@ import {
|
|||||||
getActionErrorMessage,
|
getActionErrorMessage,
|
||||||
extractActionExtraParams,
|
extractActionExtraParams,
|
||||||
getStepPreImageUrl,
|
getStepPreImageUrl,
|
||||||
getStepPostImageUrl
|
getStepPostImageUrl,
|
||||||
|
isReportStatusAction,
|
||||||
|
getReportStatusValue,
|
||||||
|
getReportStatusExplanation
|
||||||
} from '../../utils/action-formatter.util';
|
} from '../../utils/action-formatter.util';
|
||||||
|
|
||||||
import {
|
import {
|
||||||
@@ -54,7 +77,9 @@ import {
|
|||||||
isNoteTool,
|
isNoteTool,
|
||||||
isVideoTool,
|
isVideoTool,
|
||||||
getVideoToolTarget,
|
getVideoToolTarget,
|
||||||
|
isDeviceActionTool,
|
||||||
isFailureAnalyzerActionTool,
|
isFailureAnalyzerActionTool,
|
||||||
|
getToolAgentName,
|
||||||
shouldShowTool,
|
shouldShowTool,
|
||||||
getUniqueGenericTools,
|
getUniqueGenericTools,
|
||||||
getToolKey,
|
getToolKey,
|
||||||
@@ -100,9 +125,19 @@ export type { MarkdownSegment, MarkdownLine, NoteMilestone, ParsedNote };
|
|||||||
templateUrl: './agent-stream.component.html',
|
templateUrl: './agent-stream.component.html',
|
||||||
styleUrl: './agent-stream.component.scss'
|
styleUrl: './agent-stream.component.scss'
|
||||||
})
|
})
|
||||||
export class AgentStreamComponent {
|
export class AgentStreamComponent implements AfterViewInit {
|
||||||
public agentService = inject(AgentService);
|
public agentService = inject(AgentService);
|
||||||
private http = inject(HttpClient);
|
private http = inject(HttpClient);
|
||||||
|
private destroyRef = inject(DestroyRef);
|
||||||
|
|
||||||
|
// Auto-scroll state tracking
|
||||||
|
public isUserAtBottom = true;
|
||||||
|
private resizeObserver: ResizeObserver | null = null;
|
||||||
|
|
||||||
|
// Dynamic Planning Loader Game-style Phrases
|
||||||
|
public currentPlanningText = signal<string>(PLANNING_LOADER_PHRASES[0]);
|
||||||
|
private planningIntervalId: any = null;
|
||||||
|
private currentPhraseIndex = 0;
|
||||||
|
|
||||||
// Top Nav Dropdown States
|
// Top Nav Dropdown States
|
||||||
public isTaskDropdownOpen = signal<boolean>(false);
|
public isTaskDropdownOpen = signal<boolean>(false);
|
||||||
@@ -120,8 +155,9 @@ export class AgentStreamComponent {
|
|||||||
public typedTextsSignal = signal<Record<string, string>>({});
|
public typedTextsSignal = signal<Record<string, string>>({});
|
||||||
private typingTimers = new Map<string, any>();
|
private typingTimers = new Map<string, any>();
|
||||||
|
|
||||||
// Set to track expanded state of action cards
|
// Set to track expanded and collapsed state of action cards
|
||||||
public expandedActionCards = signal<Set<string>>(new Set<string>());
|
public expandedActionCards = signal<Set<string>>(new Set<string>());
|
||||||
|
public collapsedActionCards = signal<Set<string>>(new Set<string>());
|
||||||
|
|
||||||
// Performance caches for parameter and event extractions
|
// Performance caches for parameter and event extractions
|
||||||
private actionParamsCache = new WeakMap<any, ActionParam[]>();
|
private actionParamsCache = new WeakMap<any, ActionParam[]>();
|
||||||
@@ -146,7 +182,7 @@ export class AgentStreamComponent {
|
|||||||
public historyTasks = computed(() => {
|
public historyTasks = computed(() => {
|
||||||
return this.agentService.sessions().filter((s) => {
|
return this.agentService.sessions().filter((s) => {
|
||||||
const status = this.getTaskStatus(s);
|
const status = this.getTaskStatus(s);
|
||||||
return status === 'completed' || status === 'failed';
|
return status === 'completed' || status === 'failed' || status === 'cancelled';
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -161,6 +197,12 @@ export class AgentStreamComponent {
|
|||||||
return Object.keys(this.agentService.currentNotes()).filter(key => key.toLowerCase().endsWith('.md'));
|
return Object.keys(this.agentService.currentNotes()).filter(key => key.toLowerCase().endsWith('.md'));
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|
||||||
|
public outputterReport = computed(() => {
|
||||||
|
const notes = this.agentService.currentNotes();
|
||||||
|
return notes['output.md'] || null;
|
||||||
|
});
|
||||||
|
|
||||||
public parsedNote = computed<ParsedNote>(() => {
|
public parsedNote = computed<ParsedNote>(() => {
|
||||||
return this.getParsedNote(this.currentNoteContent());
|
return this.getParsedNote(this.currentNoteContent());
|
||||||
});
|
});
|
||||||
@@ -230,12 +272,8 @@ export class AgentStreamComponent {
|
|||||||
|
|
||||||
setTimeout(() => {
|
setTimeout(() => {
|
||||||
const container = document.querySelector('.stream-logs-content');
|
const container = document.querySelector('.stream-logs-content');
|
||||||
if (container) {
|
if (container && this.isUserAtBottom) {
|
||||||
const threshold = 150;
|
container.scrollTop = container.scrollHeight;
|
||||||
const isNearBottom = container.scrollHeight - container.scrollTop - container.clientHeight <= threshold;
|
|
||||||
if (isNearBottom) {
|
|
||||||
container.scrollTop = container.scrollHeight;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}, 50);
|
}, 50);
|
||||||
});
|
});
|
||||||
@@ -316,6 +354,7 @@ export class AgentStreamComponent {
|
|||||||
this.typedTextsSignal.set({});
|
this.typedTextsSignal.set({});
|
||||||
this.scheduledCollapses.clear();
|
this.scheduledCollapses.clear();
|
||||||
this.collapsedStreams.set(new Set<string>());
|
this.collapsedStreams.set(new Set<string>());
|
||||||
|
this.isUserAtBottom = true;
|
||||||
|
|
||||||
setTimeout(() => {
|
setTimeout(() => {
|
||||||
const container = document.querySelector('.stream-logs-content');
|
const container = document.querySelector('.stream-logs-content');
|
||||||
@@ -325,6 +364,58 @@ export class AgentStreamComponent {
|
|||||||
}, 100);
|
}, 100);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Periodically rotate loading text like game loading screen hints
|
||||||
|
effect(() => {
|
||||||
|
const isLoading = this.showPlanningLoader();
|
||||||
|
if (isLoading) {
|
||||||
|
if (!this.planningIntervalId) {
|
||||||
|
this.currentPhraseIndex = Math.floor(Math.random() * PLANNING_LOADER_PHRASES.length);
|
||||||
|
this.currentPlanningText.set(PLANNING_LOADER_PHRASES[this.currentPhraseIndex]);
|
||||||
|
|
||||||
|
this.planningIntervalId = setInterval(() => {
|
||||||
|
this.currentPhraseIndex = (this.currentPhraseIndex + 1) % PLANNING_LOADER_PHRASES.length;
|
||||||
|
this.currentPlanningText.set(PLANNING_LOADER_PHRASES[this.currentPhraseIndex]);
|
||||||
|
}, 2800);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
if (this.planningIntervalId) {
|
||||||
|
clearInterval(this.planningIntervalId);
|
||||||
|
this.planningIntervalId = null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
this.destroyRef.onDestroy(() => {
|
||||||
|
if (this.planningIntervalId) {
|
||||||
|
clearInterval(this.planningIntervalId);
|
||||||
|
this.planningIntervalId = null;
|
||||||
|
}
|
||||||
|
if (this.resizeObserver) {
|
||||||
|
this.resizeObserver.disconnect();
|
||||||
|
this.resizeObserver = null;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
public ngAfterViewInit(): void {
|
||||||
|
const container = document.querySelector('.stream-logs-content');
|
||||||
|
if (container && typeof ResizeObserver !== 'undefined') {
|
||||||
|
this.resizeObserver = new ResizeObserver(() => {
|
||||||
|
if (this.isUserAtBottom) {
|
||||||
|
container.scrollTop = container.scrollHeight;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
this.resizeObserver.observe(container);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public onStreamScroll(event: Event): void {
|
||||||
|
const container = event.target as HTMLElement;
|
||||||
|
if (!container) return;
|
||||||
|
const threshold = 150;
|
||||||
|
const distanceToBottom = container.scrollHeight - container.scrollTop - container.clientHeight;
|
||||||
|
this.isUserAtBottom = distanceToBottom <= threshold;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Top Nav Action Methods
|
// Top Nav Action Methods
|
||||||
@@ -338,24 +429,31 @@ export class AgentStreamComponent {
|
|||||||
|
|
||||||
public toggleNotesDropdown(event: Event): void {
|
public toggleNotesDropdown(event: Event): void {
|
||||||
event.stopPropagation();
|
event.stopPropagation();
|
||||||
this.isNotesDropdownOpen.update(v => !v);
|
this.isNotesDropdownOpen.update((v) => !v);
|
||||||
if (this.isNotesDropdownOpen()) {
|
if (this.isNotesDropdownOpen()) {
|
||||||
this.isTaskDropdownOpen.set(false);
|
this.isTaskDropdownOpen.set(false);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public openOutputterNote(event?: Event): void {
|
||||||
|
if (event) event.stopPropagation();
|
||||||
|
this.agentService.selectedNoteKey.set('output.md');
|
||||||
|
this.isNotesDropdownOpen.set(true);
|
||||||
|
this.isTaskDropdownOpen.set(false);
|
||||||
|
}
|
||||||
|
|
||||||
public closeAllDropdowns(): void {
|
public closeAllDropdowns(): void {
|
||||||
this.isTaskDropdownOpen.set(false);
|
this.isTaskDropdownOpen.set(false);
|
||||||
this.isNotesDropdownOpen.set(false);
|
this.isNotesDropdownOpen.set(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
public getTaskStatus(session: Session): 'running' | 'completed' | 'pending' | 'failed' {
|
public getTaskStatus(session: Session): 'running' | 'completed' | 'pending' | 'failed' | 'cancelled' {
|
||||||
if (session.session_id === this.agentService.runningSessionId()) {
|
if (session.session_id === this.agentService.runningSessionId() && this.agentService.agentStatus() === 'running') {
|
||||||
return 'running';
|
return 'running';
|
||||||
}
|
}
|
||||||
if (session.status) {
|
if (session.status) {
|
||||||
const s = session.status.toLowerCase();
|
const s = session.status.toLowerCase();
|
||||||
if (s === 'running' || s === 'completed' || s === 'pending' || s === 'failed') {
|
if (s === 'running' || s === 'completed' || s === 'pending' || s === 'failed' || s === 'cancelled') {
|
||||||
return s as any;
|
return s as any;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -364,10 +462,22 @@ export class AgentStreamComponent {
|
|||||||
|
|
||||||
public selectTask(sessionId: string, event?: Event): void {
|
public selectTask(sessionId: string, event?: Event): void {
|
||||||
if (event) event.stopPropagation();
|
if (event) event.stopPropagation();
|
||||||
this.agentService.selectSession(sessionId);
|
this.agentService.selectSession(sessionId, true);
|
||||||
this.isTaskDropdownOpen.set(false);
|
this.isTaskDropdownOpen.set(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public deleteTask(sessionId: string, event?: Event): void {
|
||||||
|
if (event) event.stopPropagation();
|
||||||
|
if (!confirm(`Are you sure you want to delete this task? This cannot be undone.`)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.agentService.deleteSession(sessionId).subscribe({
|
||||||
|
error: (err) => {
|
||||||
|
console.error(`Failed to delete session ${sessionId}:`, err);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
public selectNote(key: string, event?: Event): void {
|
public selectNote(key: string, event?: Event): void {
|
||||||
if (event) event.stopPropagation();
|
if (event) event.stopPropagation();
|
||||||
this.agentService.selectedNoteKey.set(key);
|
this.agentService.selectedNoteKey.set(key);
|
||||||
@@ -499,23 +609,40 @@ export class AgentStreamComponent {
|
|||||||
return getActionErrorMessage(action, stepData);
|
return getActionErrorMessage(action, stepData);
|
||||||
}
|
}
|
||||||
|
|
||||||
public isActionExpanded(cardId: string): boolean {
|
public isActionExpanded(cardId: string, defaultExpanded: boolean = false): boolean {
|
||||||
return this.expandedActionCards().has(cardId);
|
if (this.collapsedActionCards().has(cardId)) return false;
|
||||||
|
if (this.expandedActionCards().has(cardId)) return true;
|
||||||
|
return defaultExpanded;
|
||||||
}
|
}
|
||||||
|
|
||||||
public toggleActionExpanded(cardId: string, event?: Event): void {
|
public toggleActionExpanded(cardId: string, event?: Event, defaultExpanded: boolean = false): void {
|
||||||
if (event) {
|
if (event) {
|
||||||
event.stopPropagation();
|
event.stopPropagation();
|
||||||
}
|
}
|
||||||
this.expandedActionCards.update(set => {
|
const isCurrentlyExpanded = this.isActionExpanded(cardId, defaultExpanded);
|
||||||
const newSet = new Set(set);
|
if (isCurrentlyExpanded) {
|
||||||
if (newSet.has(cardId)) {
|
this.expandedActionCards.update(set => {
|
||||||
|
const newSet = new Set(set);
|
||||||
newSet.delete(cardId);
|
newSet.delete(cardId);
|
||||||
} else {
|
return newSet;
|
||||||
|
});
|
||||||
|
this.collapsedActionCards.update(set => {
|
||||||
|
const newSet = new Set(set);
|
||||||
newSet.add(cardId);
|
newSet.add(cardId);
|
||||||
}
|
return newSet;
|
||||||
return newSet;
|
});
|
||||||
});
|
} else {
|
||||||
|
this.collapsedActionCards.update(set => {
|
||||||
|
const newSet = new Set(set);
|
||||||
|
newSet.delete(cardId);
|
||||||
|
return newSet;
|
||||||
|
});
|
||||||
|
this.expandedActionCards.update(set => {
|
||||||
|
const newSet = new Set(set);
|
||||||
|
newSet.add(cardId);
|
||||||
|
return newSet;
|
||||||
|
});
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public getActionBounds(action: any): string {
|
public getActionBounds(action: any): string {
|
||||||
@@ -629,8 +756,16 @@ export class AgentStreamComponent {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public isDeviceActionTool(tool: any): boolean {
|
||||||
|
return isDeviceActionTool(tool);
|
||||||
|
}
|
||||||
|
|
||||||
public isFailureAnalyzerActionTool(tool: any): boolean {
|
public isFailureAnalyzerActionTool(tool: any): boolean {
|
||||||
return isFailureAnalyzerActionTool(tool);
|
return isDeviceActionTool(tool);
|
||||||
|
}
|
||||||
|
|
||||||
|
public getToolAgentName(tool: any): string | null {
|
||||||
|
return getToolAgentName(tool);
|
||||||
}
|
}
|
||||||
|
|
||||||
public shouldShowTool(tool: any, stepData?: any): boolean {
|
public shouldShowTool(tool: any, stepData?: any): boolean {
|
||||||
@@ -662,11 +797,23 @@ export class AgentStreamComponent {
|
|||||||
if (!block) return false;
|
if (!block) return false;
|
||||||
const hasNative = Boolean(this.getNativeThinking(block));
|
const hasNative = Boolean(this.getNativeThinking(block));
|
||||||
const hasRaw = Boolean(this.getRawThinking(block));
|
const hasRaw = Boolean(this.getRawThinking(block));
|
||||||
const hasVisibleTools = Boolean(block.data?.generic_tools) && block.data.generic_tools.some((t: any) => this.shouldShowTool(t, block.data) || (t.type === 'llm_call' && t.status === 'failed'));
|
const hasVisibleTools = Boolean(block.data?.generic_tools) && block.data.generic_tools.some((t: any) => this.shouldShowTool(t, block.data) || (t.type === 'llm_call' && t.status === 'failed') || this.isReportStatusAction(t));
|
||||||
const hasAndroidActions = Boolean(block.data?.action_taken) && this.isAndroidAction(block.data.action_taken);
|
const hasAndroidActions = Boolean(block.data?.action_taken) && (this.isAndroidAction(block.data.action_taken) || this.isReportStatusAction(block.data.action_taken));
|
||||||
return hasNative || hasRaw || hasVisibleTools || hasAndroidActions;
|
return hasNative || hasRaw || hasVisibleTools || hasAndroidActions;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public isReportStatusAction(action: any): boolean {
|
||||||
|
return isReportStatusAction(action);
|
||||||
|
}
|
||||||
|
|
||||||
|
public getReportStatusValue(action: any): string {
|
||||||
|
return getReportStatusValue(action);
|
||||||
|
}
|
||||||
|
|
||||||
|
public getReportStatusExplanation(action: any): string {
|
||||||
|
return getReportStatusExplanation(action);
|
||||||
|
}
|
||||||
|
|
||||||
public isHumanThinking(text: string | null): boolean {
|
public isHumanThinking(text: string | null): boolean {
|
||||||
return isHumanThinking(text);
|
return isHumanThinking(text);
|
||||||
}
|
}
|
||||||
@@ -779,7 +926,7 @@ export class AgentStreamComponent {
|
|||||||
const selectElement = event.target as HTMLSelectElement;
|
const selectElement = event.target as HTMLSelectElement;
|
||||||
const sessionId = selectElement.value;
|
const sessionId = selectElement.value;
|
||||||
if (sessionId) {
|
if (sessionId) {
|
||||||
this.agentService.selectSession(sessionId);
|
this.agentService.selectSession(sessionId, true);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -927,20 +1074,19 @@ export class AgentStreamComponent {
|
|||||||
|
|
||||||
public isItemRunning(item: any, block: any): boolean {
|
public isItemRunning(item: any, block: any): boolean {
|
||||||
if (!item || !item.data) return false;
|
if (!item || !item.data) return false;
|
||||||
|
if (!this.isCurrentSessionActive()) return false;
|
||||||
|
if (item.data.status === 'success' || item.data.status === 'failed' || item.data.status === 'cancelled') return false;
|
||||||
|
if (this.isReportStatusAction(item.data) || this.isReportStatusAction(item.data?.action_taken)) return false;
|
||||||
if (item.data.status === 'running') return true;
|
if (item.data.status === 'running') return true;
|
||||||
if (item.data.status === 'success' || item.data.status === 'failed') return false;
|
|
||||||
if (item.data.action === 'report_task_status' || item.data?.action_taken?.action === 'report_task_status') return false;
|
|
||||||
|
|
||||||
// Check if session is currently running and this is the active trailing item
|
// Check if session is currently running and this is the active trailing item
|
||||||
if (this.isCurrentSessionActive()) {
|
const blocks = this.consolidatedBlocks();
|
||||||
const blocks = this.consolidatedBlocks();
|
const lastBlock = blocks.length > 0 ? blocks[blocks.length - 1] : null;
|
||||||
const lastBlock = blocks.length > 0 ? blocks[blocks.length - 1] : null;
|
if (lastBlock && (lastBlock.id === block?.id || !block?.data?.isCompleted)) {
|
||||||
if (lastBlock && (lastBlock.id === block?.id || !block?.data?.isCompleted)) {
|
const events = this.getSortedStepEvents(block.data);
|
||||||
const events = this.getSortedStepEvents(block.data);
|
const lastEvent = events.length > 0 ? events[events.length - 1] : null;
|
||||||
const lastEvent = events.length > 0 ? events[events.length - 1] : null;
|
if (lastEvent && (lastEvent.data?.trace_id === item.data?.trace_id || lastEvent === item)) {
|
||||||
if (lastEvent && (lastEvent.data?.trace_id === item.data?.trace_id || lastEvent === item)) {
|
return true;
|
||||||
return true;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return false;
|
return false;
|
||||||
@@ -948,13 +1094,11 @@ export class AgentStreamComponent {
|
|||||||
|
|
||||||
public isBlockRunning(block: any): boolean {
|
public isBlockRunning(block: any): boolean {
|
||||||
if (!block || !block.data) return false;
|
if (!block || !block.data) return false;
|
||||||
if (block.data.action_taken?.action === 'report_task_status' || block.data?.status === 'completed' || block.data?.status === 'failed') return false;
|
if (!this.isCurrentSessionActive()) return false;
|
||||||
if (block.data.isCompleted === false && this.isCurrentSessionActive()) return true;
|
if (this.isReportStatusAction(block.data.action_taken) || block.data?.status === 'completed' || block.data?.status === 'failed' || block.data?.status === 'cancelled') return false;
|
||||||
if (this.isCurrentSessionActive()) {
|
if (block.data.isCompleted === false) return true;
|
||||||
const blocks = this.consolidatedBlocks();
|
const blocks = this.consolidatedBlocks();
|
||||||
const lastBlock = blocks.length > 0 ? blocks[blocks.length - 1] : null;
|
const lastBlock = blocks.length > 0 ? blocks[blocks.length - 1] : null;
|
||||||
return lastBlock?.id === block.id;
|
return lastBlock?.id === block.id;
|
||||||
}
|
|
||||||
return false;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
+29
-48
@@ -17,8 +17,6 @@
|
|||||||
<div class="chat-interface-container">
|
<div class="chat-interface-container">
|
||||||
<!-- Task Queue & History Header -->
|
<!-- Task Queue & History Header -->
|
||||||
<header class="chat-header">
|
<header class="chat-header">
|
||||||
<div class="header-left"></div>
|
|
||||||
|
|
||||||
<!-- Tab Selector Header inside row -->
|
<!-- Tab Selector Header inside row -->
|
||||||
<div class="tab-selector-bar">
|
<div class="tab-selector-bar">
|
||||||
<button
|
<button
|
||||||
@@ -78,7 +76,20 @@
|
|||||||
<span class="status-dot" *ngIf="getTaskStatus(session) === 'running'"></span>
|
<span class="status-dot" *ngIf="getTaskStatus(session) === 'running'"></span>
|
||||||
{{ getTaskStatus(session) | uppercase }}
|
{{ getTaskStatus(session) | uppercase }}
|
||||||
</span>
|
</span>
|
||||||
<span class="task-time">{{ session.start_time * 1000 | date:'shortTime' }}</span>
|
<div class="task-card-header-actions">
|
||||||
|
<span class="task-time">{{ session.start_time * 1000 | date:'shortTime' }}</span>
|
||||||
|
<button
|
||||||
|
*ngIf="getTaskStatus(session) !== 'running'"
|
||||||
|
type="button"
|
||||||
|
class="btn-delete-card"
|
||||||
|
(click)="deleteTask(session.session_id, $event)"
|
||||||
|
title="Delete this task"
|
||||||
|
aria-label="Delete this task"
|
||||||
|
[disabled]="isSubmitting"
|
||||||
|
>
|
||||||
|
<span class="material-symbols-outlined">delete</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="task-goal" title="{{ session.initial_goal }}">
|
<div class="task-goal" title="{{ session.initial_goal }}">
|
||||||
@@ -114,7 +125,20 @@
|
|||||||
<span class="status-dot" *ngIf="getTaskStatus(session) === 'running'"></span>
|
<span class="status-dot" *ngIf="getTaskStatus(session) === 'running'"></span>
|
||||||
{{ getTaskStatus(session) | uppercase }}
|
{{ getTaskStatus(session) | uppercase }}
|
||||||
</span>
|
</span>
|
||||||
<span class="task-time">{{ session.start_time * 1000 | date:'shortTime' }}</span>
|
<div class="task-card-header-actions">
|
||||||
|
<span class="task-time">{{ session.start_time * 1000 | date:'shortTime' }}</span>
|
||||||
|
<button
|
||||||
|
*ngIf="getTaskStatus(session) !== 'running'"
|
||||||
|
type="button"
|
||||||
|
class="btn-delete-card"
|
||||||
|
(click)="deleteTask(session.session_id, $event)"
|
||||||
|
title="Delete this task"
|
||||||
|
aria-label="Delete this task"
|
||||||
|
[disabled]="isSubmitting"
|
||||||
|
>
|
||||||
|
<span class="material-symbols-outlined">delete</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="task-goal" title="{{ session.initial_goal }}">
|
<div class="task-goal" title="{{ session.initial_goal }}">
|
||||||
@@ -237,48 +261,5 @@
|
|||||||
</div>
|
</div>
|
||||||
</ng-container>
|
</ng-container>
|
||||||
</main>
|
</main>
|
||||||
|
|
||||||
<!-- Task Submission Area (Only shown in Tasks tab) - Temporarily disabled/hidden
|
|
||||||
<footer class="chat-input-panel" *ngIf="agentService.activeTab() === 'tasks'">
|
|
||||||
<div class="error-banner" *ngIf="errorMessage">
|
|
||||||
<span class="material-symbols-outlined error-icon">warning</span> {{ errorMessage }}
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="input-row">
|
|
||||||
<textarea
|
|
||||||
[(ngModel)]="taskInput"
|
|
||||||
(keydown.enter)="$event.preventDefault(); submitTask()"
|
|
||||||
placeholder="Enter goal (e.g. add contact name Jerry)..."
|
|
||||||
class="chat-input"
|
|
||||||
rows="1"
|
|
||||||
[disabled]="isSubmitting"
|
|
||||||
></textarea>
|
|
||||||
<button
|
|
||||||
*ngIf="agentService.agentStatus() === 'running' || activeQueue().length > 0"
|
|
||||||
(click)="stopTasks(false)"
|
|
||||||
[disabled]="isSubmitting"
|
|
||||||
class="btn-stop"
|
|
||||||
[title]="activeQueue().length > 1 ? 'Stop currently running task only' : 'Stop currently running task'"
|
|
||||||
>
|
|
||||||
{{ activeQueue().length > 1 ? 'Stop Current' : 'Stop' }}
|
|
||||||
</button>
|
|
||||||
<button
|
|
||||||
*ngIf="activeQueue().length > 1"
|
|
||||||
(click)="stopTasks(true)"
|
|
||||||
[disabled]="isSubmitting"
|
|
||||||
class="btn-stop-secondary"
|
|
||||||
title="Stop currently running task and clear queue"
|
|
||||||
>
|
|
||||||
Stop All
|
|
||||||
</button>
|
|
||||||
<button
|
|
||||||
(click)="submitTask()"
|
|
||||||
[disabled]="isSubmitting || !taskInput.trim()"
|
|
||||||
class="btn-send"
|
|
||||||
>
|
|
||||||
{{ isSubmitting ? '...' : 'Run' }}
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
</footer>
|
|
||||||
-->
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
+138
-224
@@ -46,36 +46,59 @@ $text-muted: #a1a1aa;
|
|||||||
background: #ffffff;
|
background: #ffffff;
|
||||||
box-sizing: border-box;
|
box-sizing: border-box;
|
||||||
|
|
||||||
.header-left {
|
.tab-selector-bar {
|
||||||
display: flex;
|
display: flex;
|
||||||
|
height: 36px;
|
||||||
|
box-sizing: border-box;
|
||||||
|
background: rgba(241, 245, 249, 0.72);
|
||||||
|
backdrop-filter: blur(16px) saturate(180%);
|
||||||
|
-webkit-backdrop-filter: blur(16px) saturate(180%);
|
||||||
|
border: 1px solid rgba(226, 232, 240, 0.85);
|
||||||
|
border-radius: 18px;
|
||||||
|
padding: 3px;
|
||||||
|
gap: 3px;
|
||||||
align-items: center;
|
align-items: center;
|
||||||
gap: 12px;
|
flex-shrink: 0;
|
||||||
flex: 1 1 0;
|
box-shadow: 0 2px 8px -1px rgba(15, 23, 42, 0.04), inset 0 1px 1px rgba(255, 255, 255, 0.8);
|
||||||
min-width: 0;
|
|
||||||
|
|
||||||
.avatar {
|
.tab-selector-btn {
|
||||||
width: 32px;
|
display: inline-flex;
|
||||||
height: 32px;
|
|
||||||
background: #f3f4f6;
|
|
||||||
border-radius: 50%;
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
align-items: center;
|
||||||
justify-content: center;
|
justify-content: center;
|
||||||
box-shadow: 0 2px 4px rgba(0,0,0,0.03);
|
gap: 6px;
|
||||||
|
height: 30px;
|
||||||
|
padding: 0 14px;
|
||||||
|
box-sizing: border-box;
|
||||||
|
border: none;
|
||||||
|
background: transparent;
|
||||||
|
border-radius: 15px;
|
||||||
|
font-size: 13px;
|
||||||
|
font-weight: 600;
|
||||||
|
color: #64748b;
|
||||||
|
cursor: pointer;
|
||||||
|
line-height: 1;
|
||||||
|
transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1);
|
||||||
|
white-space: nowrap;
|
||||||
|
|
||||||
.material-symbols-outlined {
|
.tab-icon {
|
||||||
font-size: 18px;
|
|
||||||
color: #4b5563;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
.chat-title {
|
|
||||||
h3 {
|
|
||||||
margin: 0;
|
|
||||||
font-size: 16px;
|
font-size: 16px;
|
||||||
|
width: 16px;
|
||||||
|
height: 16px;
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
&:hover:not(.active) {
|
||||||
|
color: #0f172a;
|
||||||
|
background: rgba(255, 255, 255, 0.7);
|
||||||
|
}
|
||||||
|
|
||||||
|
&.active {
|
||||||
|
background: #ffffff;
|
||||||
|
color: #2563eb;
|
||||||
font-weight: 700;
|
font-weight: 700;
|
||||||
color: $text-primary;
|
box-shadow: 0 2px 6px rgba(0, 0, 0, 0.06), 0 1px 2px rgba(0, 0, 0, 0.02);
|
||||||
white-space: nowrap;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -89,20 +112,22 @@ $text-muted: #a1a1aa;
|
|||||||
flex-shrink: 0;
|
flex-shrink: 0;
|
||||||
|
|
||||||
.btn-resume-task {
|
.btn-resume-task {
|
||||||
padding: 6px 12px;
|
padding: 6px 14px;
|
||||||
background: #2563eb;
|
background: linear-gradient(135deg, rgba(37, 99, 235, 0.92), rgba(26, 115, 232, 0.85));
|
||||||
border: 1px solid #2563eb;
|
backdrop-filter: blur(14px) saturate(180%);
|
||||||
|
-webkit-backdrop-filter: blur(14px) saturate(180%);
|
||||||
|
border: 1px solid rgba(255, 255, 255, 0.35);
|
||||||
border-radius: 16px;
|
border-radius: 16px;
|
||||||
font-size: 14px;
|
font-size: 13.5px;
|
||||||
font-weight: 600;
|
font-weight: 600;
|
||||||
color: #ffffff;
|
color: #ffffff;
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
display: flex;
|
display: flex;
|
||||||
align-items: center;
|
align-items: center;
|
||||||
gap: 4px;
|
gap: 5px;
|
||||||
transition: all 0.2s;
|
transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1);
|
||||||
margin-right: 8px;
|
margin-right: 8px;
|
||||||
box-shadow: 0 2px 4px rgba(37, 99, 235, 0.2);
|
box-shadow: 0 4px 14px rgba(37, 99, 235, 0.3), inset 0 1px 1px rgba(255, 255, 255, 0.45);
|
||||||
|
|
||||||
.material-symbols-outlined {
|
.material-symbols-outlined {
|
||||||
font-size: 16px;
|
font-size: 16px;
|
||||||
@@ -110,9 +135,14 @@ $text-muted: #a1a1aa;
|
|||||||
}
|
}
|
||||||
|
|
||||||
&:hover {
|
&:hover {
|
||||||
background: #1d4ed8;
|
background: linear-gradient(135deg, rgba(37, 99, 235, 1), rgba(26, 115, 232, 0.95));
|
||||||
border-color: #1d4ed8;
|
border-color: rgba(255, 255, 255, 0.6);
|
||||||
box-shadow: 0 4px 8px rgba(37, 99, 235, 0.3);
|
box-shadow: 0 6px 18px rgba(37, 99, 235, 0.42), inset 0 1px 1px rgba(255, 255, 255, 0.6);
|
||||||
|
transform: translateY(-0.5px);
|
||||||
|
}
|
||||||
|
|
||||||
|
&:active {
|
||||||
|
transform: scale(0.97);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -120,18 +150,22 @@ $text-muted: #a1a1aa;
|
|||||||
width: 32px;
|
width: 32px;
|
||||||
height: 32px;
|
height: 32px;
|
||||||
padding: 0;
|
padding: 0;
|
||||||
background: transparent;
|
box-sizing: border-box;
|
||||||
border: 1px solid #cbd5e1;
|
background: rgba(255, 255, 255, 0.72);
|
||||||
|
backdrop-filter: blur(16px) saturate(180%);
|
||||||
|
-webkit-backdrop-filter: blur(16px) saturate(180%);
|
||||||
|
border: 1px solid rgba(203, 213, 225, 0.85);
|
||||||
border-radius: 50%;
|
border-radius: 50%;
|
||||||
color: #64748b;
|
color: #64748b;
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
display: flex;
|
display: flex;
|
||||||
align-items: center;
|
align-items: center;
|
||||||
justify-content: center;
|
justify-content: center;
|
||||||
transition: all 0.2s ease;
|
box-shadow: 0 2px 6px -1px rgba(15, 23, 42, 0.04), inset 0 1px 1px rgba(255, 255, 255, 0.9);
|
||||||
|
transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1);
|
||||||
|
|
||||||
.material-symbols-outlined {
|
.material-symbols-outlined {
|
||||||
font-size: 18px;
|
font-size: 16px;
|
||||||
color: inherit;
|
color: inherit;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -139,6 +173,7 @@ $text-muted: #a1a1aa;
|
|||||||
background: #fef2f2;
|
background: #fef2f2;
|
||||||
color: #ef4444;
|
color: #ef4444;
|
||||||
border-color: #fca5a5;
|
border-color: #fca5a5;
|
||||||
|
box-shadow: 0 3px 10px rgba(239, 68, 68, 0.15);
|
||||||
transform: scale(1.05);
|
transform: scale(1.05);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -153,7 +188,7 @@ $text-muted: #a1a1aa;
|
|||||||
.chat-messages {
|
.chat-messages {
|
||||||
flex: 1;
|
flex: 1;
|
||||||
overflow-y: auto;
|
overflow-y: auto;
|
||||||
padding: 12px;
|
padding: 12px 12px 100px 12px;
|
||||||
display: flex;
|
display: flex;
|
||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
gap: 8px;
|
gap: 8px;
|
||||||
@@ -259,6 +294,17 @@ $text-muted: #a1a1aa;
|
|||||||
transform: translateY(-1px);
|
transform: translateY(-1px);
|
||||||
border-color: rgba(203, 213, 225, 1);
|
border-color: rgba(203, 213, 225, 1);
|
||||||
box-shadow: 0 3px 8px rgba(148, 163, 184, 0.08);
|
box-shadow: 0 3px 8px rgba(148, 163, 184, 0.08);
|
||||||
|
|
||||||
|
.task-card-header .task-card-header-actions {
|
||||||
|
&:has(.btn-delete-card) .task-time {
|
||||||
|
opacity: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.btn-delete-card {
|
||||||
|
opacity: 1;
|
||||||
|
pointer-events: auto;
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
&.selected {
|
&.selected {
|
||||||
@@ -336,15 +382,66 @@ $text-muted: #a1a1aa;
|
|||||||
background-color: #ef4444;
|
background-color: #ef4444;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
&.cancelled {
|
||||||
|
background: #fafafa;
|
||||||
|
color: #d97706;
|
||||||
|
border-color: #fef3c7;
|
||||||
|
&::before {
|
||||||
|
background-color: #f59e0b;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
.status-dot {
|
.status-dot {
|
||||||
display: none;
|
display: none;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
.task-time {
|
.task-card-header-actions {
|
||||||
font-size: 11.5px;
|
display: flex;
|
||||||
color: #94a3b8;
|
align-items: center;
|
||||||
|
position: relative;
|
||||||
|
|
||||||
|
.task-time {
|
||||||
|
font-size: 11.5px;
|
||||||
|
color: #94a3b8;
|
||||||
|
transition: opacity 0.15s ease;
|
||||||
|
}
|
||||||
|
|
||||||
|
.btn-delete-card {
|
||||||
|
position: absolute;
|
||||||
|
right: 0;
|
||||||
|
top: 50%;
|
||||||
|
transform: translateY(-50%);
|
||||||
|
width: 20px;
|
||||||
|
height: 20px;
|
||||||
|
padding: 0;
|
||||||
|
border: none;
|
||||||
|
background: transparent;
|
||||||
|
border-radius: 4px;
|
||||||
|
color: #94a3b8;
|
||||||
|
cursor: pointer;
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
opacity: 0;
|
||||||
|
pointer-events: none;
|
||||||
|
transition: all 0.15s ease;
|
||||||
|
|
||||||
|
.material-symbols-outlined {
|
||||||
|
font-size: 14px;
|
||||||
|
line-height: 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
&:hover:not(:disabled) {
|
||||||
|
color: #ef4444;
|
||||||
|
background: #fee2e2;
|
||||||
|
}
|
||||||
|
|
||||||
|
&:disabled {
|
||||||
|
opacity: 0.3;
|
||||||
|
cursor: not-allowed;
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -396,192 +493,8 @@ $text-muted: #a1a1aa;
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
.chat-input-panel {
|
|
||||||
display: flex;
|
|
||||||
flex-direction: column;
|
|
||||||
gap: 8px;
|
|
||||||
padding: 16px;
|
|
||||||
border-top: 1px solid $border-color;
|
|
||||||
background: #ffffff;
|
|
||||||
|
|
||||||
.error-banner {
|
|
||||||
background: #fef2f2;
|
|
||||||
border: 1px solid #fca5a5;
|
|
||||||
color: #991b1b;
|
|
||||||
font-size: 12px;
|
|
||||||
padding: 8px 12px;
|
|
||||||
border-radius: 8px;
|
|
||||||
line-height: 1.4;
|
|
||||||
animation: bannerSlideIn 0.2s ease-out;
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
gap: 6px;
|
|
||||||
|
|
||||||
.error-icon {
|
|
||||||
font-size: 16px;
|
|
||||||
color: #991b1b;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
.input-row {
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
gap: 10px;
|
|
||||||
width: 100%;
|
|
||||||
|
|
||||||
.chat-input {
|
|
||||||
flex: 1;
|
|
||||||
border: 1px solid #e2e8f0;
|
|
||||||
border-radius: 12px;
|
|
||||||
padding: 10px 14px;
|
|
||||||
font-size: 13.5px;
|
|
||||||
line-height: 1.4;
|
|
||||||
outline: none;
|
|
||||||
resize: none;
|
|
||||||
box-sizing: border-box;
|
|
||||||
max-height: 80px;
|
|
||||||
font-family: inherit;
|
|
||||||
color: #0f172a;
|
|
||||||
transition: all 0.2s;
|
|
||||||
|
|
||||||
&:focus {
|
|
||||||
border-color: #3b82f6;
|
|
||||||
box-shadow: 0 0 0 2px rgba(59, 130, 246, 0.1);
|
|
||||||
}
|
|
||||||
|
|
||||||
&:disabled {
|
|
||||||
background: #f8fafc;
|
|
||||||
color: #94a3b8;
|
|
||||||
cursor: not-allowed;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
.btn-stop {
|
|
||||||
padding: 10px 16px;
|
|
||||||
background: #ef4444;
|
|
||||||
border: none;
|
|
||||||
border-radius: 12px;
|
|
||||||
font-size: 17px;
|
|
||||||
font-weight: 600;
|
|
||||||
color: #ffffff;
|
|
||||||
cursor: pointer;
|
|
||||||
transition: background 0.2s, transform 0.1s;
|
|
||||||
min-width: 65px;
|
|
||||||
|
|
||||||
&:hover:not(:disabled) {
|
|
||||||
background: #dc2626;
|
|
||||||
}
|
|
||||||
|
|
||||||
&:active:not(:disabled) {
|
|
||||||
transform: scale(0.97);
|
|
||||||
}
|
|
||||||
|
|
||||||
&:disabled {
|
|
||||||
background: #fca5a5;
|
|
||||||
color: #ffffff;
|
|
||||||
cursor: not-allowed;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
.btn-stop-secondary {
|
|
||||||
padding: 10px 16px;
|
|
||||||
background: transparent;
|
|
||||||
border: 1.5px solid #ef4444;
|
|
||||||
border-radius: 12px;
|
|
||||||
font-size: 17px;
|
|
||||||
font-weight: 600;
|
|
||||||
color: #ef4444;
|
|
||||||
cursor: pointer;
|
|
||||||
transition: background 0.2s, transform 0.1s;
|
|
||||||
min-width: 65px;
|
|
||||||
|
|
||||||
&:hover:not(:disabled) {
|
|
||||||
background: #fef2f2;
|
|
||||||
}
|
|
||||||
|
|
||||||
&:active:not(:disabled) {
|
|
||||||
transform: scale(0.97);
|
|
||||||
}
|
|
||||||
|
|
||||||
&:disabled {
|
|
||||||
border-color: #fca5a5;
|
|
||||||
color: #fca5a5;
|
|
||||||
cursor: not-allowed;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
.btn-send {
|
|
||||||
padding: 10px 16px;
|
|
||||||
background: #0f172a;
|
|
||||||
border: none;
|
|
||||||
border-radius: 12px;
|
|
||||||
font-size: 17px;
|
|
||||||
font-weight: 600;
|
|
||||||
color: #ffffff;
|
|
||||||
cursor: pointer;
|
|
||||||
transition: background 0.2s, transform 0.1s;
|
|
||||||
min-width: 60px;
|
|
||||||
|
|
||||||
&:hover:not(:disabled) {
|
|
||||||
background: #1e293b;
|
|
||||||
}
|
|
||||||
|
|
||||||
&:active:not(:disabled) {
|
|
||||||
transform: scale(0.97);
|
|
||||||
}
|
|
||||||
|
|
||||||
&:disabled {
|
|
||||||
background: #e2e8f0;
|
|
||||||
color: #94a3b8;
|
|
||||||
cursor: not-allowed;
|
|
||||||
box-shadow: none;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Tab selector bar inline capsule format inside header
|
|
||||||
.tab-selector-bar {
|
|
||||||
display: flex;
|
|
||||||
background: #f1f5f9;
|
|
||||||
border-radius: 20px;
|
|
||||||
padding: 3px;
|
|
||||||
gap: 2px;
|
|
||||||
align-items: center;
|
|
||||||
box-sizing: border-box;
|
|
||||||
flex-shrink: 0;
|
|
||||||
|
|
||||||
.tab-selector-btn {
|
|
||||||
display: inline-flex;
|
|
||||||
align-items: center;
|
|
||||||
justify-content: center;
|
|
||||||
gap: 6px;
|
|
||||||
padding: 6px 14px;
|
|
||||||
border: none;
|
|
||||||
background: transparent;
|
|
||||||
border-radius: 16px;
|
|
||||||
font-size: 13.5px;
|
|
||||||
font-weight: 600;
|
|
||||||
color: #64748b;
|
|
||||||
cursor: pointer;
|
|
||||||
transition: all 0.2s ease;
|
|
||||||
white-space: nowrap;
|
|
||||||
|
|
||||||
.tab-icon {
|
|
||||||
font-size: 16px;
|
|
||||||
}
|
|
||||||
|
|
||||||
&:hover {
|
|
||||||
color: #334155;
|
|
||||||
}
|
|
||||||
|
|
||||||
&.active {
|
|
||||||
background: #ffffff;
|
|
||||||
color: #2563eb;
|
|
||||||
box-shadow: 0 1px 3px rgba(0, 0, 0, 0.08);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Notes tab content
|
// Notes tab content
|
||||||
.notes-tab-content {
|
.notes-tab-content {
|
||||||
@@ -795,6 +708,7 @@ $text-muted: #a1a1aa;
|
|||||||
display: flex;
|
display: flex;
|
||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
gap: 10px;
|
gap: 10px;
|
||||||
|
padding-left: 28px;
|
||||||
|
|
||||||
.substep-row {
|
.substep-row {
|
||||||
display: flex;
|
display: flex;
|
||||||
+42
-27
@@ -58,12 +58,12 @@ export class ChatInterfaceComponent {
|
|||||||
});
|
});
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Filtered computed list of historical/completed tasks (completed or failed)
|
* Filtered computed list of historical/completed tasks (completed, failed, or cancelled)
|
||||||
*/
|
*/
|
||||||
public historyTasks = computed(() => {
|
public historyTasks = computed(() => {
|
||||||
return this.agentService.sessions().filter((s) => {
|
return this.agentService.sessions().filter((s) => {
|
||||||
const status = this.getTaskStatus(s);
|
const status = this.getTaskStatus(s);
|
||||||
return status === 'completed' || status === 'failed';
|
return status === 'completed' || status === 'failed' || status === 'cancelled';
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -79,6 +79,7 @@ export class ChatInterfaceComponent {
|
|||||||
this.isSubmitting = true;
|
this.isSubmitting = true;
|
||||||
this.errorMessage = null;
|
this.errorMessage = null;
|
||||||
|
|
||||||
|
this.agentService.clearUserPinnedSession();
|
||||||
this.http.post<any>('/api/run', { goal }).subscribe({
|
this.http.post<any>('/api/run', { goal }).subscribe({
|
||||||
next: (res) => {
|
next: (res) => {
|
||||||
this.taskInput = '';
|
this.taskInput = '';
|
||||||
@@ -86,7 +87,13 @@ export class ChatInterfaceComponent {
|
|||||||
if (res && res.tasks && res.tasks.length > 0) {
|
if (res && res.tasks && res.tasks.length > 0) {
|
||||||
const firstTask = res.tasks[0];
|
const firstTask = res.tasks[0];
|
||||||
if (firstTask && firstTask.session_id) {
|
if (firstTask && firstTask.session_id) {
|
||||||
this.agentService.selectSession(firstTask.session_id);
|
// Only auto-select immediately if no task is currently executing/running
|
||||||
|
const isCurrentlyRunning =
|
||||||
|
this.agentService.agentStatus() === 'running' ||
|
||||||
|
this.agentService.sessions().some((s) => s.status === 'running');
|
||||||
|
if (!isCurrentlyRunning) {
|
||||||
|
this.agentService.selectSession(firstTask.session_id, false);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Fetch status to refresh sessions list and select new session
|
// Fetch status to refresh sessions list and select new session
|
||||||
@@ -110,18 +117,10 @@ export class ChatInterfaceComponent {
|
|||||||
public stopTasks(stopAll: boolean = false): void {
|
public stopTasks(stopAll: boolean = false): void {
|
||||||
this.isSubmitting = true;
|
this.isSubmitting = true;
|
||||||
this.errorMessage = null;
|
this.errorMessage = null;
|
||||||
this.http.post<any>(`/api/stop?all=${stopAll}`, {}).subscribe({
|
this.agentService.stopTask(stopAll);
|
||||||
next: () => {
|
setTimeout(() => {
|
||||||
this.isSubmitting = false;
|
this.isSubmitting = false;
|
||||||
// Refresh status and sessions list
|
}, 400);
|
||||||
this.agentService.fetchStatus();
|
|
||||||
},
|
|
||||||
error: (err) => {
|
|
||||||
console.error('Failed to stop tasks:', err);
|
|
||||||
this.isSubmitting = false;
|
|
||||||
this.errorMessage = err.error?.detail || 'Failed to stop tasks.';
|
|
||||||
}
|
|
||||||
});
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -133,16 +132,11 @@ export class ChatInterfaceComponent {
|
|||||||
}
|
}
|
||||||
this.isSubmitting = true;
|
this.isSubmitting = true;
|
||||||
this.errorMessage = null;
|
this.errorMessage = null;
|
||||||
this.http.post<any>('/api/cleanup', {}).subscribe({
|
this.agentService.clearAllHistory().subscribe({
|
||||||
next: () => {
|
next: () => {
|
||||||
this.isSubmitting = false;
|
this.isSubmitting = false;
|
||||||
// Reset current session selection
|
|
||||||
this.agentService.selectSession('');
|
|
||||||
// Refresh status and sessions
|
|
||||||
this.agentService.fetchStatus();
|
|
||||||
this.agentService.fetchSessions();
|
|
||||||
},
|
},
|
||||||
error: (err) => {
|
error: (err: any) => {
|
||||||
console.error('Failed to clear history:', err);
|
console.error('Failed to clear history:', err);
|
||||||
this.isSubmitting = false;
|
this.isSubmitting = false;
|
||||||
this.errorMessage = err.error?.detail || 'Failed to clear history.';
|
this.errorMessage = err.error?.detail || 'Failed to clear history.';
|
||||||
@@ -150,20 +144,41 @@ export class ChatInterfaceComponent {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Delete an individual task / session
|
||||||
|
*/
|
||||||
|
public deleteTask(sessionId: string, event: MouseEvent): void {
|
||||||
|
event.stopPropagation();
|
||||||
|
if (!confirm(`Are you sure you want to delete this task? This cannot be undone.`)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.isSubmitting = true;
|
||||||
|
this.errorMessage = null;
|
||||||
|
this.agentService.deleteSession(sessionId).subscribe({
|
||||||
|
next: () => {
|
||||||
|
this.isSubmitting = false;
|
||||||
|
},
|
||||||
|
error: (err: any) => {
|
||||||
|
console.error(`Failed to delete task ${sessionId}:`, err);
|
||||||
|
this.isSubmitting = false;
|
||||||
|
this.errorMessage = err.error?.detail || 'Failed to delete task.';
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Determine the current task execution status
|
* Determine the current task execution status
|
||||||
*/
|
*/
|
||||||
public getTaskStatus(session: Session): 'running' | 'completed' | 'pending' | 'failed' {
|
public getTaskStatus(session: Session): 'running' | 'completed' | 'pending' | 'failed' | 'cancelled' {
|
||||||
if (session.session_id === this.agentService.runningSessionId()) {
|
if (session.session_id === this.agentService.runningSessionId() && this.agentService.agentStatus() === 'running') {
|
||||||
return 'running';
|
return 'running';
|
||||||
}
|
}
|
||||||
if (session.status) {
|
if (session.status) {
|
||||||
const s = session.status.toLowerCase();
|
const s = session.status.toLowerCase();
|
||||||
if (s === 'running' || s === 'completed' || s === 'pending' || s === 'failed') {
|
if (s === 'running' || s === 'completed' || s === 'pending' || s === 'failed' || s === 'cancelled') {
|
||||||
return s as any;
|
return s as any;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Fallback: If it's a historical session and not currently running, it's completed
|
|
||||||
return 'completed';
|
return 'completed';
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -171,7 +186,7 @@ export class ChatInterfaceComponent {
|
|||||||
* Select a session in the UI to monitor its steps
|
* Select a session in the UI to monitor its steps
|
||||||
*/
|
*/
|
||||||
public selectTask(sessionId: string): void {
|
public selectTask(sessionId: string): void {
|
||||||
this.agentService.selectSession(sessionId);
|
this.agentService.selectSession(sessionId, true);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Notes Computed Properties
|
// Notes Computed Properties
|
||||||
File renamed without changes.
+80
-45
@@ -31,10 +31,12 @@
|
|||||||
position: fixed;
|
position: fixed;
|
||||||
width: 360px;
|
width: 360px;
|
||||||
max-width: calc(100vw - 32px);
|
max-width: calc(100vw - 32px);
|
||||||
background: #ffffff;
|
background: rgba(255, 255, 255, 0.84);
|
||||||
border: 1px solid #cbd5e1;
|
backdrop-filter: blur(24px) saturate(180%);
|
||||||
border-radius: 12px;
|
-webkit-backdrop-filter: blur(24px) saturate(180%);
|
||||||
box-shadow: 0 16px 40px -8px rgba(15, 23, 42, 0.22), 0 0 0 1px rgba(0, 0, 0, 0.05);
|
border: 1px solid rgba(255, 255, 255, 0.85);
|
||||||
|
border-radius: 16px;
|
||||||
|
box-shadow: 0 20px 50px -10px rgba(15, 23, 42, 0.22), 0 0 0 1px rgba(226, 232, 240, 0.6), inset 0 1px 1px rgba(255, 255, 255, 0.9);
|
||||||
z-index: 1050;
|
z-index: 1050;
|
||||||
overflow: hidden;
|
overflow: hidden;
|
||||||
display: flex;
|
display: flex;
|
||||||
@@ -77,8 +79,9 @@
|
|||||||
/* --- Window Header (Draggable Handle) --- */
|
/* --- Window Header (Draggable Handle) --- */
|
||||||
.window-header {
|
.window-header {
|
||||||
height: 44px;
|
height: 44px;
|
||||||
background: #ffffff;
|
background: rgba(255, 255, 255, 0.75);
|
||||||
border-bottom: 1px solid #e2e8f0;
|
backdrop-filter: blur(16px);
|
||||||
|
border-bottom: 1px solid rgba(226, 232, 240, 0.75);
|
||||||
display: flex;
|
display: flex;
|
||||||
align-items: center;
|
align-items: center;
|
||||||
justify-content: space-between;
|
justify-content: space-between;
|
||||||
@@ -117,10 +120,11 @@
|
|||||||
.mode-switcher-group {
|
.mode-switcher-group {
|
||||||
display: flex;
|
display: flex;
|
||||||
align-items: center;
|
align-items: center;
|
||||||
background: #f1f5f9;
|
background: rgba(241, 245, 249, 0.65);
|
||||||
border: 1px solid #cbd5e1;
|
backdrop-filter: blur(12px);
|
||||||
border-radius: 6px;
|
border: 1px solid rgba(226, 232, 240, 0.8);
|
||||||
padding: 1.5px;
|
border-radius: 8px;
|
||||||
|
padding: 2px;
|
||||||
margin-left: 4px;
|
margin-left: 4px;
|
||||||
flex-shrink: 0;
|
flex-shrink: 0;
|
||||||
|
|
||||||
@@ -131,7 +135,7 @@
|
|||||||
font-size: 10px;
|
font-size: 10px;
|
||||||
font-weight: 700;
|
font-weight: 700;
|
||||||
padding: 2px 7px;
|
padding: 2px 7px;
|
||||||
border-radius: 4px;
|
border-radius: 6px;
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
transition: all 0.15s ease;
|
transition: all 0.15s ease;
|
||||||
display: flex;
|
display: flex;
|
||||||
@@ -148,7 +152,7 @@
|
|||||||
|
|
||||||
&:hover:not(:disabled) {
|
&:hover:not(:disabled) {
|
||||||
color: #0f172a;
|
color: #0f172a;
|
||||||
background: #e2e8f0;
|
background: rgba(255, 255, 255, 0.6);
|
||||||
}
|
}
|
||||||
|
|
||||||
&.active {
|
&.active {
|
||||||
@@ -172,12 +176,15 @@
|
|||||||
gap: 4px;
|
gap: 4px;
|
||||||
flex-shrink: 0;
|
flex-shrink: 0;
|
||||||
|
|
||||||
.speed-pill-group {
|
.speed-pill-group {
|
||||||
display: flex;
|
display: flex;
|
||||||
align-items: center;
|
align-items: center;
|
||||||
background: #e2e8f0;
|
background: rgba(241, 245, 249, 0.65);
|
||||||
border: 1px solid #cbd5e1;
|
backdrop-filter: blur(12px);
|
||||||
border-radius: 6px;
|
border: 1px solid rgba(226, 232, 240, 0.8);
|
||||||
|
border-radius: 8px;
|
||||||
|
padding: 2px;
|
||||||
|
margin-right: 4px;
|
||||||
padding: 2px;
|
padding: 2px;
|
||||||
margin-right: 4px;
|
margin-right: 4px;
|
||||||
|
|
||||||
@@ -207,8 +214,10 @@
|
|||||||
}
|
}
|
||||||
|
|
||||||
.header-btn {
|
.header-btn {
|
||||||
background: #f1f5f9;
|
background: rgba(241, 245, 249, 0.75);
|
||||||
border: 1px solid #cbd5e1;
|
backdrop-filter: blur(12px);
|
||||||
|
-webkit-backdrop-filter: blur(12px);
|
||||||
|
border: 1px solid rgba(226, 232, 240, 0.85);
|
||||||
color: #1e293b;
|
color: #1e293b;
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
display: flex;
|
display: flex;
|
||||||
@@ -216,19 +225,27 @@
|
|||||||
justify-content: center;
|
justify-content: center;
|
||||||
width: 28px;
|
width: 28px;
|
||||||
height: 28px;
|
height: 28px;
|
||||||
border-radius: 6px;
|
border-radius: 7px;
|
||||||
transition: all 0.15s ease;
|
box-shadow: 0 1px 3px rgba(15, 23, 42, 0.04), inset 0 1px 1px rgba(255, 255, 255, 0.8);
|
||||||
|
transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1);
|
||||||
|
|
||||||
&:hover {
|
&:hover {
|
||||||
background: #e2e8f0;
|
background: rgba(255, 255, 255, 0.95);
|
||||||
color: #0f172a;
|
color: #0f172a;
|
||||||
border-color: #94a3b8;
|
border-color: rgba(203, 213, 225, 0.95);
|
||||||
|
transform: translateY(-0.5px);
|
||||||
|
box-shadow: 0 2px 6px rgba(15, 23, 42, 0.08);
|
||||||
}
|
}
|
||||||
|
|
||||||
&.close-btn:hover {
|
&.close-btn:hover {
|
||||||
background: #fee2e2;
|
background: rgba(254, 226, 226, 0.95);
|
||||||
color: #dc2626;
|
color: #dc2626;
|
||||||
border-color: #fca5a5;
|
border-color: rgba(252, 165, 165, 0.9);
|
||||||
|
box-shadow: 0 2px 6px rgba(220, 38, 38, 0.15);
|
||||||
|
}
|
||||||
|
|
||||||
|
&:active {
|
||||||
|
transform: scale(0.96);
|
||||||
}
|
}
|
||||||
|
|
||||||
.material-symbols-outlined {
|
.material-symbols-outlined {
|
||||||
@@ -375,14 +392,16 @@
|
|||||||
}
|
}
|
||||||
|
|
||||||
.video-controls-footer {
|
.video-controls-footer {
|
||||||
background: #ffffff;
|
background: rgba(255, 255, 255, 0.84);
|
||||||
border-top: 1px solid #f1f5f9;
|
backdrop-filter: blur(20px) saturate(180%);
|
||||||
|
-webkit-backdrop-filter: blur(20px) saturate(180%);
|
||||||
|
border-top: 1px solid rgba(226, 232, 240, 0.75);
|
||||||
padding: 12px 14px;
|
padding: 12px 14px;
|
||||||
display: flex;
|
display: flex;
|
||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
gap: 10px;
|
gap: 10px;
|
||||||
flex-shrink: 0;
|
flex-shrink: 0;
|
||||||
box-shadow: 0 -2px 8px rgba(0, 0, 0, 0.04);
|
box-shadow: 0 -4px 16px rgba(15, 23, 42, 0.04), inset 0 1px 1px rgba(255, 255, 255, 0.9);
|
||||||
|
|
||||||
.scrubber-bar-container {
|
.scrubber-bar-container {
|
||||||
position: relative;
|
position: relative;
|
||||||
@@ -408,7 +427,7 @@
|
|||||||
left: 0;
|
left: 0;
|
||||||
right: 0;
|
right: 0;
|
||||||
height: 5px;
|
height: 5px;
|
||||||
background: #e2e8f0;
|
background: rgba(226, 232, 240, 0.85);
|
||||||
border-radius: 3px;
|
border-radius: 3px;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -416,10 +435,10 @@
|
|||||||
position: absolute;
|
position: absolute;
|
||||||
left: 0;
|
left: 0;
|
||||||
height: 5px;
|
height: 5px;
|
||||||
background: #0284c7;
|
background: linear-gradient(90deg, #0284c7, #2563eb);
|
||||||
border-radius: 3px;
|
border-radius: 3px;
|
||||||
pointer-events: none;
|
pointer-events: none;
|
||||||
box-shadow: 0 0 6px rgba(2, 132, 199, 0.4);
|
box-shadow: 0 0 8px rgba(2, 132, 199, 0.4);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -435,43 +454,59 @@
|
|||||||
}
|
}
|
||||||
|
|
||||||
.ctrl-btn {
|
.ctrl-btn {
|
||||||
background: #f8fafc;
|
background: rgba(241, 245, 249, 0.75);
|
||||||
border: 1px solid #cbd5e1;
|
backdrop-filter: blur(12px);
|
||||||
|
-webkit-backdrop-filter: blur(12px);
|
||||||
|
border: 1px solid rgba(226, 232, 240, 0.85);
|
||||||
|
box-shadow: 0 1px 3px rgba(15, 23, 42, 0.04), inset 0 1px 1px rgba(255, 255, 255, 0.85);
|
||||||
color: #0f172a;
|
color: #0f172a;
|
||||||
width: 32px;
|
width: 32px;
|
||||||
height: 32px;
|
height: 32px;
|
||||||
border-radius: 6px;
|
border-radius: 8px;
|
||||||
display: flex;
|
display: flex;
|
||||||
align-items: center;
|
align-items: center;
|
||||||
justify-content: center;
|
justify-content: center;
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
transition: all 0.15s ease;
|
transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1);
|
||||||
|
|
||||||
&:hover {
|
&:hover {
|
||||||
background: #e2e8f0;
|
background: rgba(255, 255, 255, 0.95);
|
||||||
border-color: #94a3b8;
|
border-color: rgba(191, 219, 254, 0.9);
|
||||||
color: #0f172a;
|
color: #0284c7;
|
||||||
transform: scale(1.05);
|
transform: translateY(-0.5px);
|
||||||
|
box-shadow: 0 3px 8px rgba(15, 23, 42, 0.08);
|
||||||
|
}
|
||||||
|
|
||||||
|
&:active {
|
||||||
|
transform: scale(0.96);
|
||||||
}
|
}
|
||||||
|
|
||||||
&.active {
|
&.active {
|
||||||
color: #0284c7;
|
color: #0284c7;
|
||||||
background: #e0f2fe;
|
background: rgba(224, 242, 254, 0.85);
|
||||||
border-color: #38bdf8;
|
border-color: rgba(56, 189, 248, 0.8);
|
||||||
|
box-shadow: 0 2px 6px rgba(2, 132, 199, 0.15);
|
||||||
}
|
}
|
||||||
|
|
||||||
&.play-btn {
|
&.play-btn {
|
||||||
background: #0284c7;
|
background: linear-gradient(135deg, rgba(2, 132, 199, 0.92), rgba(37, 99, 235, 0.88));
|
||||||
border: 1px solid #0284c7;
|
backdrop-filter: blur(14px) saturate(180%);
|
||||||
|
-webkit-backdrop-filter: blur(14px) saturate(180%);
|
||||||
|
border: 1px solid rgba(255, 255, 255, 0.35);
|
||||||
color: #ffffff;
|
color: #ffffff;
|
||||||
width: 36px;
|
width: 36px;
|
||||||
height: 36px;
|
height: 36px;
|
||||||
border-radius: 50%;
|
border-radius: 50%;
|
||||||
box-shadow: 0 2px 8px rgba(2, 132, 199, 0.35);
|
box-shadow: 0 4px 14px rgba(2, 132, 199, 0.35), inset 0 1px 1px rgba(255, 255, 255, 0.5);
|
||||||
|
|
||||||
&:hover {
|
&:hover {
|
||||||
background: #0369a1;
|
background: linear-gradient(135deg, rgba(2, 132, 199, 1), rgba(37, 99, 235, 0.95));
|
||||||
box-shadow: 0 3px 12px rgba(2, 132, 199, 0.5);
|
box-shadow: 0 6px 18px rgba(2, 132, 199, 0.45), inset 0 1px 1px rgba(255, 255, 255, 0.6);
|
||||||
|
transform: translateY(-1px) scale(1.04);
|
||||||
|
}
|
||||||
|
|
||||||
|
&:active {
|
||||||
|
transform: scale(0.96);
|
||||||
}
|
}
|
||||||
|
|
||||||
.material-symbols-outlined {
|
.material-symbols-outlined {
|
||||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
+1
-1
@@ -36,7 +36,7 @@ export class LegacyAgentStreamComponent {
|
|||||||
const selectElement = event.target as HTMLSelectElement;
|
const selectElement = event.target as HTMLSelectElement;
|
||||||
const sessionId = selectElement.value;
|
const sessionId = selectElement.value;
|
||||||
if (sessionId) {
|
if (sessionId) {
|
||||||
this.agentService.selectSession(sessionId);
|
this.agentService.selectSession(sessionId, true);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -0,0 +1,110 @@
|
|||||||
|
/**
|
||||||
|
* Copyright 2026 Google LLC
|
||||||
|
*
|
||||||
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
* you may not use this file except in compliance with the License.
|
||||||
|
* You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
|
.floating-nav-switcher {
|
||||||
|
position: fixed;
|
||||||
|
top: 12px;
|
||||||
|
left: 24px;
|
||||||
|
z-index: 1100;
|
||||||
|
height: 36px;
|
||||||
|
box-sizing: border-box;
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
background: rgba(255, 255, 255, 0.72);
|
||||||
|
backdrop-filter: blur(16px) saturate(180%);
|
||||||
|
-webkit-backdrop-filter: blur(16px) saturate(180%);
|
||||||
|
border: 1px solid rgba(226, 232, 240, 0.85);
|
||||||
|
border-radius: 18px;
|
||||||
|
padding: 3px;
|
||||||
|
gap: 3px;
|
||||||
|
box-shadow: 0 2px 8px -1px rgba(15, 23, 42, 0.06), 0 1px 2px rgba(0, 0, 0, 0.02), inset 0 1px 1px rgba(255, 255, 255, 0.9);
|
||||||
|
user-select: none;
|
||||||
|
font-family: 'Google Sans', 'Google Sans Text', 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||||
|
transition: all 0.25s cubic-bezier(0.4, 0, 0.2, 1);
|
||||||
|
|
||||||
|
&:hover {
|
||||||
|
background: rgba(255, 255, 255, 0.85);
|
||||||
|
box-shadow: 0 4px 14px -2px rgba(15, 23, 42, 0.1), 0 2px 4px rgba(0, 0, 0, 0.03), inset 0 1px 1px rgba(255, 255, 255, 1);
|
||||||
|
border-color: rgba(203, 213, 225, 0.9);
|
||||||
|
}
|
||||||
|
|
||||||
|
.nav-tab-btn {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
gap: 6px;
|
||||||
|
height: 30px;
|
||||||
|
padding: 0 14px;
|
||||||
|
box-sizing: border-box;
|
||||||
|
border-radius: 15px;
|
||||||
|
font-size: 13px;
|
||||||
|
font-weight: 600;
|
||||||
|
color: #475569;
|
||||||
|
text-decoration: none;
|
||||||
|
cursor: pointer;
|
||||||
|
line-height: 1;
|
||||||
|
transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1);
|
||||||
|
outline: none;
|
||||||
|
white-space: nowrap;
|
||||||
|
|
||||||
|
.tab-icon {
|
||||||
|
font-size: 16px;
|
||||||
|
width: 16px;
|
||||||
|
height: 16px;
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
color: #64748b;
|
||||||
|
transition: color 0.15s ease, transform 0.15s ease;
|
||||||
|
}
|
||||||
|
|
||||||
|
.tab-label {
|
||||||
|
color: #475569;
|
||||||
|
transition: color 0.15s ease;
|
||||||
|
}
|
||||||
|
|
||||||
|
&:hover:not(.active) {
|
||||||
|
background: rgba(255, 255, 255, 0.7);
|
||||||
|
color: #0f172a;
|
||||||
|
|
||||||
|
.tab-icon {
|
||||||
|
color: #1a73e8;
|
||||||
|
}
|
||||||
|
.tab-label {
|
||||||
|
color: #0f172a;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
&.active {
|
||||||
|
background: rgba(26, 115, 232, 0.95) !important;
|
||||||
|
backdrop-filter: blur(12px);
|
||||||
|
-webkit-backdrop-filter: blur(12px);
|
||||||
|
color: #ffffff !important;
|
||||||
|
font-weight: 700;
|
||||||
|
box-shadow: 0 2px 8px rgba(26, 115, 232, 0.35), inset 0 1px 1px rgba(255, 255, 255, 0.35);
|
||||||
|
|
||||||
|
.tab-icon,
|
||||||
|
.tab-label,
|
||||||
|
span {
|
||||||
|
color: #ffffff !important;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
&:active {
|
||||||
|
transform: scale(0.97);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
/**
|
||||||
|
* Copyright 2026 Google LLC
|
||||||
|
*
|
||||||
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
* you may not use this file except in compliance with the License.
|
||||||
|
* You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { Component } from '@angular/core';
|
||||||
|
import { CommonModule } from '@angular/common';
|
||||||
|
import { RouterLink, RouterLinkActive } from '@angular/router';
|
||||||
|
|
||||||
|
@Component({
|
||||||
|
selector: 'app-nav-switcher',
|
||||||
|
standalone: true,
|
||||||
|
imports: [CommonModule, RouterLink, RouterLinkActive],
|
||||||
|
template: `
|
||||||
|
<nav class="floating-nav-switcher" aria-label="Main Navigation">
|
||||||
|
<a
|
||||||
|
routerLink="/"
|
||||||
|
routerLinkActive="active"
|
||||||
|
[routerLinkActiveOptions]="{exact: true}"
|
||||||
|
class="nav-tab-btn"
|
||||||
|
title="Return to Home Launcher to start a new task"
|
||||||
|
>
|
||||||
|
<span class="material-symbols-outlined tab-icon">add_task</span>
|
||||||
|
<span class="tab-label">New / Home</span>
|
||||||
|
</a>
|
||||||
|
<a
|
||||||
|
routerLink="/workspace"
|
||||||
|
routerLinkActive="active"
|
||||||
|
class="nav-tab-btn"
|
||||||
|
title="Open Workspace"
|
||||||
|
>
|
||||||
|
<span class="material-symbols-outlined tab-icon">space_dashboard</span>
|
||||||
|
<span class="tab-label">Workspace</span>
|
||||||
|
</a>
|
||||||
|
</nav>
|
||||||
|
`,
|
||||||
|
styleUrls: ['./nav-switcher.component.scss']
|
||||||
|
})
|
||||||
|
export class NavSwitcherComponent {}
|
||||||
@@ -0,0 +1,334 @@
|
|||||||
|
/**
|
||||||
|
* Copyright 2026 Google LLC
|
||||||
|
*
|
||||||
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
* you may not use this file except in compliance with the License.
|
||||||
|
* You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export interface AppReference {
|
||||||
|
name: string;
|
||||||
|
icon: string;
|
||||||
|
pkg?: string;
|
||||||
|
category?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type SuggestionCategory =
|
||||||
|
| 'all'
|
||||||
|
| 'flash'
|
||||||
|
| 'pro'
|
||||||
|
| 'cross_app'
|
||||||
|
| 'monitor';
|
||||||
|
|
||||||
|
export interface SmartSuggestion {
|
||||||
|
id: string;
|
||||||
|
title: string;
|
||||||
|
description: string;
|
||||||
|
goal: string;
|
||||||
|
profile: 'flash' | 'pro';
|
||||||
|
category: 'flash' | 'pro' | 'cross_app' | 'monitor';
|
||||||
|
tag: string;
|
||||||
|
apps: AppReference[];
|
||||||
|
requiredPackages?: string[];
|
||||||
|
matchMode?: 'any' | 'all';
|
||||||
|
priority?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Recognized Android App Package Registry
|
||||||
|
*/
|
||||||
|
export const APP_REGISTRY: Record<string, AppReference> = {
|
||||||
|
// Google Suite & System
|
||||||
|
'com.google.android.apps.maps': { name: 'Maps', icon: 'explore', pkg: 'com.google.android.apps.maps', category: 'navigation' },
|
||||||
|
'com.google.android.gm': { name: 'Gmail', icon: 'mail', pkg: 'com.google.android.gm', category: 'productivity' },
|
||||||
|
'com.android.chrome': { name: 'Chrome', icon: 'public', pkg: 'com.android.chrome', category: 'browser' },
|
||||||
|
'com.google.android.youtube': { name: 'YouTube', icon: 'smart_display', pkg: 'com.google.android.youtube', category: 'entertainment' },
|
||||||
|
'com.android.settings': { name: 'Settings', icon: 'settings', pkg: 'com.android.settings', category: 'system' },
|
||||||
|
'com.google.android.deskclock': { name: 'Clock', icon: 'timer', pkg: 'com.google.android.deskclock', category: 'utility' },
|
||||||
|
'com.android.deskclock': { name: 'Clock', icon: 'timer', pkg: 'com.android.deskclock', category: 'utility' },
|
||||||
|
'com.google.android.calculator': { name: 'Calculator', icon: 'calculate', pkg: 'com.google.android.calculator', category: 'utility' },
|
||||||
|
'com.android.calculator2': { name: 'Calculator', icon: 'calculate', pkg: 'com.android.calculator2', category: 'utility' },
|
||||||
|
'com.google.android.apps.photos': { name: 'Photos', icon: 'photo_library', pkg: 'com.google.android.apps.photos', category: 'media' },
|
||||||
|
'com.google.android.calendar': { name: 'Calendar', icon: 'calendar_month', pkg: 'com.google.android.calendar', category: 'productivity' },
|
||||||
|
'com.google.android.keep': { name: 'Keep Notes', icon: 'note_alt', pkg: 'com.google.android.keep', category: 'productivity' },
|
||||||
|
'com.android.vending': { name: 'Play Store', icon: 'storefront', pkg: 'com.android.vending', category: 'tools' },
|
||||||
|
'com.google.android.apps.messaging': { name: 'Messages', icon: 'chat', pkg: 'com.google.android.apps.messaging', category: 'communication' },
|
||||||
|
|
||||||
|
// Popular Ecosystem Apps
|
||||||
|
'com.tencent.mm': { name: 'WeChat', icon: 'forum', pkg: 'com.tencent.mm', category: 'social' },
|
||||||
|
'com.xingin.xhs': { name: 'Xiaohongshu', icon: 'auto_stories', pkg: 'com.xingin.xhs', category: 'social' },
|
||||||
|
'com.sankuai.meituan': { name: 'Meituan', icon: 'restaurant', pkg: 'com.sankuai.meituan', category: 'lifestyle' },
|
||||||
|
'com.dianping.v1': { name: 'Dianping', icon: 'star', pkg: 'com.dianping.v1', category: 'lifestyle' },
|
||||||
|
'tv.danmaku.bili': { name: 'Bilibili', icon: 'video_library', pkg: 'tv.danmaku.bili', category: 'entertainment' },
|
||||||
|
'com.eg.android.AlipayGphone': { name: 'Alipay', icon: 'account_balance_wallet', pkg: 'com.eg.android.AlipayGphone', category: 'finance' },
|
||||||
|
'com.netease.cloudmusic': { name: 'NetEase Music', icon: 'headphones', pkg: 'com.netease.cloudmusic', category: 'entertainment' },
|
||||||
|
'com.spotify.music': { name: 'Spotify', icon: 'music_note', pkg: 'com.spotify.music', category: 'entertainment' }
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Curated, clean task preset library (1~2 representative tasks per common app)
|
||||||
|
*/
|
||||||
|
export const SMART_TASK_LIBRARY: SmartSuggestion[] = [
|
||||||
|
// 1. Google Maps
|
||||||
|
{
|
||||||
|
id: 'maps_coffee',
|
||||||
|
title: 'Find Specialty Coffee',
|
||||||
|
description: 'Search nearby top-rated cafes in Google Maps',
|
||||||
|
goal: 'Open Google Maps, search for top-rated specialty coffee shops nearby, and view the top result details.',
|
||||||
|
profile: 'flash',
|
||||||
|
category: 'flash',
|
||||||
|
tag: 'Maps',
|
||||||
|
apps: [{ name: 'Maps', icon: 'explore', pkg: 'com.google.android.apps.maps' }],
|
||||||
|
requiredPackages: ['com.google.android.apps.maps'],
|
||||||
|
priority: 95
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'pro_commute_share',
|
||||||
|
title: 'Commute ETA & Message Draft',
|
||||||
|
description: 'Check transit time on Maps and draft arrival ETA in Messages',
|
||||||
|
goal: 'Open Google Maps to check commute time to the International Airport, calculate arrival time, then open Messages and draft an ETA text message.',
|
||||||
|
profile: 'pro',
|
||||||
|
category: 'cross_app',
|
||||||
|
tag: 'Maps + Messages',
|
||||||
|
apps: [
|
||||||
|
{ name: 'Maps', icon: 'explore', pkg: 'com.google.android.apps.maps' },
|
||||||
|
{ name: 'Messages', icon: 'chat', pkg: 'com.google.android.apps.messaging' }
|
||||||
|
],
|
||||||
|
requiredPackages: ['com.google.android.apps.maps', 'com.google.android.apps.messaging'],
|
||||||
|
matchMode: 'all',
|
||||||
|
priority: 92
|
||||||
|
},
|
||||||
|
|
||||||
|
// 2. Gmail
|
||||||
|
{
|
||||||
|
id: 'gmail_receipts',
|
||||||
|
title: 'Search Order Receipts',
|
||||||
|
description: 'Find recent flight or delivery confirmation emails in Gmail',
|
||||||
|
goal: 'Open Gmail and search for recent flight or package delivery confirmation emails.',
|
||||||
|
profile: 'flash',
|
||||||
|
category: 'flash',
|
||||||
|
tag: 'Gmail',
|
||||||
|
apps: [{ name: 'Gmail', icon: 'mail', pkg: 'com.google.android.gm' }],
|
||||||
|
requiredPackages: ['com.google.android.gm'],
|
||||||
|
priority: 90
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'pro_email_to_calendar',
|
||||||
|
title: 'Email Itinerary to Calendar',
|
||||||
|
description: 'Extract flight or event dates from Gmail and schedule in Calendar',
|
||||||
|
goal: 'Open Gmail to find the latest event invitation or itinerary, extract dates and location, then open Google Calendar and create a corresponding calendar event.',
|
||||||
|
profile: 'pro',
|
||||||
|
category: 'cross_app',
|
||||||
|
tag: 'Gmail + Calendar',
|
||||||
|
apps: [
|
||||||
|
{ name: 'Gmail', icon: 'mail', pkg: 'com.google.android.gm' },
|
||||||
|
{ name: 'Calendar', icon: 'calendar_month', pkg: 'com.google.android.calendar' }
|
||||||
|
],
|
||||||
|
requiredPackages: ['com.google.android.gm'],
|
||||||
|
priority: 94
|
||||||
|
},
|
||||||
|
|
||||||
|
// 3. Chrome
|
||||||
|
{
|
||||||
|
id: 'chrome_research',
|
||||||
|
title: 'Search AI News Breakthroughs',
|
||||||
|
description: 'Search latest multimodal AI developments in Chrome browser',
|
||||||
|
goal: 'Open Chrome browser and search for latest breakthroughs in multimodal mobile AI agents.',
|
||||||
|
profile: 'flash',
|
||||||
|
category: 'flash',
|
||||||
|
tag: 'Chrome',
|
||||||
|
apps: [{ name: 'Chrome', icon: 'public', pkg: 'com.android.chrome' }],
|
||||||
|
requiredPackages: ['com.android.chrome'],
|
||||||
|
priority: 88
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'pro_research_keep',
|
||||||
|
title: 'Product Research & Notes Note',
|
||||||
|
description: 'Compare top 3 headphones on Chrome and record comparison in Keep',
|
||||||
|
goal: 'Open Chrome, research top 3 noise-cancelling headphones comparing price and battery life, then write a structured comparison summary note in Keep Notes.',
|
||||||
|
profile: 'pro',
|
||||||
|
category: 'pro',
|
||||||
|
tag: 'Chrome + Keep',
|
||||||
|
apps: [
|
||||||
|
{ name: 'Chrome', icon: 'public', pkg: 'com.android.chrome' },
|
||||||
|
{ name: 'Keep Notes', icon: 'note_alt', pkg: 'com.google.android.keep' }
|
||||||
|
],
|
||||||
|
requiredPackages: ['com.android.chrome'],
|
||||||
|
priority: 91
|
||||||
|
},
|
||||||
|
|
||||||
|
// 4. YouTube
|
||||||
|
{
|
||||||
|
id: 'youtube_lofi',
|
||||||
|
title: 'Play Lo-Fi Music Radio',
|
||||||
|
description: 'Search and play a Lo-Fi hip hop live stream on YouTube',
|
||||||
|
goal: 'Open YouTube, search for "Lofi hip hop beats relaxing radio" and tap on the live stream.',
|
||||||
|
profile: 'flash',
|
||||||
|
category: 'flash',
|
||||||
|
tag: 'YouTube',
|
||||||
|
apps: [{ name: 'YouTube', icon: 'smart_display', pkg: 'com.google.android.youtube' }],
|
||||||
|
requiredPackages: ['com.google.android.youtube'],
|
||||||
|
priority: 85
|
||||||
|
},
|
||||||
|
|
||||||
|
// 5. Settings
|
||||||
|
{
|
||||||
|
id: 'settings_display_wifi',
|
||||||
|
title: 'Dark Mode & Wi-Fi Check',
|
||||||
|
description: 'Toggle dark theme and verify network connection in Settings',
|
||||||
|
goal: 'Open Settings app, navigate to Display settings, ensure Dark theme is enabled, and check Wi-Fi connection status.',
|
||||||
|
profile: 'flash',
|
||||||
|
category: 'flash',
|
||||||
|
tag: 'Settings',
|
||||||
|
apps: [{ name: 'Settings', icon: 'settings', pkg: 'com.android.settings' }],
|
||||||
|
requiredPackages: ['com.android.settings'],
|
||||||
|
priority: 87
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'pro_settings_qa',
|
||||||
|
title: 'Subsystem Health & Crash Probe',
|
||||||
|
description: 'Traverse Settings submenus to verify screens and check for crash dialogs',
|
||||||
|
goal: 'Explore Settings submenus (Network, Connected devices, Apps, Battery, Storage), verify each screen loads properly without ANR or crash dialogs, and summarize results.',
|
||||||
|
profile: 'pro',
|
||||||
|
category: 'monitor',
|
||||||
|
tag: 'Settings QA',
|
||||||
|
apps: [{ name: 'Settings', icon: 'settings', pkg: 'com.android.settings' }],
|
||||||
|
requiredPackages: ['com.android.settings'],
|
||||||
|
priority: 93
|
||||||
|
},
|
||||||
|
|
||||||
|
// 6. Clock
|
||||||
|
{
|
||||||
|
id: 'clock_timer',
|
||||||
|
title: '25-Min Pomodoro Timer',
|
||||||
|
description: 'Start a 25-minute focus countdown timer in Clock app',
|
||||||
|
goal: 'Open Clock app, switch to Timer tab, set 25 minutes and start the countdown timer.',
|
||||||
|
profile: 'flash',
|
||||||
|
category: 'flash',
|
||||||
|
tag: 'Clock',
|
||||||
|
apps: [{ name: 'Clock', icon: 'timer', pkg: 'com.google.android.deskclock' }],
|
||||||
|
requiredPackages: ['com.google.android.deskclock', 'com.android.deskclock'],
|
||||||
|
priority: 86
|
||||||
|
},
|
||||||
|
|
||||||
|
// 7. Calculator
|
||||||
|
{
|
||||||
|
id: 'calc_gratuity',
|
||||||
|
title: 'Split Bill & Calculate Tip',
|
||||||
|
description: 'Calculate 18% gratuity on $186.40 for 3 people in Calculator',
|
||||||
|
goal: 'Open Calculator and calculate 18% tip on a bill of $186.40, then divide by 3 people.',
|
||||||
|
profile: 'flash',
|
||||||
|
category: 'flash',
|
||||||
|
tag: 'Calculator',
|
||||||
|
apps: [{ name: 'Calculator', icon: 'calculate', pkg: 'com.google.android.calculator' }],
|
||||||
|
requiredPackages: ['com.google.android.calculator', 'com.android.calculator2'],
|
||||||
|
priority: 84
|
||||||
|
},
|
||||||
|
|
||||||
|
// 8. Photos
|
||||||
|
{
|
||||||
|
id: 'photos_inspect',
|
||||||
|
title: 'Inspect Recent Screenshot',
|
||||||
|
description: 'Open Google Photos and review the latest screenshot taken',
|
||||||
|
goal: 'Open Google Photos and view the most recent screenshot in the screenshots album.',
|
||||||
|
profile: 'flash',
|
||||||
|
category: 'flash',
|
||||||
|
tag: 'Photos',
|
||||||
|
apps: [{ name: 'Photos', icon: 'photo_library', pkg: 'com.google.android.apps.photos' }],
|
||||||
|
requiredPackages: ['com.google.android.apps.photos'],
|
||||||
|
priority: 82
|
||||||
|
},
|
||||||
|
|
||||||
|
// 9. WeChat (微信)
|
||||||
|
{
|
||||||
|
id: 'wechat_browse',
|
||||||
|
title: 'Check WeChat Messages',
|
||||||
|
description: 'Open WeChat and view top recent chat conversations',
|
||||||
|
goal: 'Open WeChat (微信) and view the top recent chat messages.',
|
||||||
|
profile: 'flash',
|
||||||
|
category: 'flash',
|
||||||
|
tag: 'WeChat',
|
||||||
|
apps: [{ name: 'WeChat', icon: 'forum', pkg: 'com.tencent.mm' }],
|
||||||
|
requiredPackages: ['com.tencent.mm'],
|
||||||
|
priority: 89
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'pro_wechat_to_calendar',
|
||||||
|
title: 'WeChat Notice to Calendar',
|
||||||
|
description: 'Extract meeting notice from WeChat chat and add to Calendar',
|
||||||
|
goal: 'Open WeChat (微信), locate the latest meeting announcement or event message in the top chat, extract the time and topic, then open Calendar and schedule an event.',
|
||||||
|
profile: 'pro',
|
||||||
|
category: 'cross_app',
|
||||||
|
tag: 'WeChat + Calendar',
|
||||||
|
apps: [
|
||||||
|
{ name: 'WeChat', icon: 'forum', pkg: 'com.tencent.mm' },
|
||||||
|
{ name: 'Calendar', icon: 'calendar_month', pkg: 'com.google.android.calendar' }
|
||||||
|
],
|
||||||
|
requiredPackages: ['com.tencent.mm'],
|
||||||
|
priority: 93
|
||||||
|
},
|
||||||
|
|
||||||
|
// 10. Xiaohongshu (小红书)
|
||||||
|
{
|
||||||
|
id: 'xhs_coffee_guide',
|
||||||
|
title: 'RED Cafe Guide Search',
|
||||||
|
description: 'Search trending specialty cafe reviews on Xiaohongshu',
|
||||||
|
goal: 'Open Xiaohongshu (小红书), search for top-rated specialty coffee shops, and view the top post.',
|
||||||
|
profile: 'flash',
|
||||||
|
category: 'flash',
|
||||||
|
tag: 'Xiaohongshu',
|
||||||
|
apps: [{ name: 'Xiaohongshu', icon: 'auto_stories', pkg: 'com.xingin.xhs' }],
|
||||||
|
requiredPackages: ['com.xingin.xhs'],
|
||||||
|
priority: 87
|
||||||
|
},
|
||||||
|
|
||||||
|
// 11. Meituan / Dianping (美团 / 大众点评)
|
||||||
|
{
|
||||||
|
id: 'meituan_ramen_search',
|
||||||
|
title: 'Meituan Food Search',
|
||||||
|
description: 'Search top-rated Ramen nearby on Meituan or Dianping',
|
||||||
|
goal: 'Open Meituan (美团) or Dianping (大众点评), search for top-rated Ramen nearby, and view top restaurant rating.',
|
||||||
|
profile: 'flash',
|
||||||
|
category: 'flash',
|
||||||
|
tag: 'Meituan',
|
||||||
|
apps: [{ name: 'Meituan', icon: 'restaurant', pkg: 'com.sankuai.meituan' }],
|
||||||
|
requiredPackages: ['com.sankuai.meituan', 'com.dianping.v1'],
|
||||||
|
priority: 86
|
||||||
|
},
|
||||||
|
|
||||||
|
// 12. Bilibili (哔哩哔哩)
|
||||||
|
{
|
||||||
|
id: 'bilibili_stream',
|
||||||
|
title: 'Bilibili Tech Video',
|
||||||
|
description: 'Search and play an AI Agent tutorial video on Bilibili',
|
||||||
|
goal: 'Open Bilibili (哔哩哔哩), search for "AI Agent 架构实战", and play the top matching video.',
|
||||||
|
profile: 'flash',
|
||||||
|
category: 'flash',
|
||||||
|
tag: 'Bilibili',
|
||||||
|
apps: [{ name: 'Bilibili', icon: 'video_library', pkg: 'tv.danmaku.bili' }],
|
||||||
|
requiredPackages: ['tv.danmaku.bili'],
|
||||||
|
priority: 85
|
||||||
|
},
|
||||||
|
|
||||||
|
// 13. Play Store
|
||||||
|
{
|
||||||
|
id: 'pro_playstore_review',
|
||||||
|
title: 'Play Store App Review Study',
|
||||||
|
description: 'Compare top task management apps and ratings on Google Play',
|
||||||
|
goal: 'Open Google Play Store, search for top rated task management apps, compare ratings and latest user reviews of the top 2 candidates, and record recommendations.',
|
||||||
|
profile: 'pro',
|
||||||
|
category: 'pro',
|
||||||
|
tag: 'Play Store',
|
||||||
|
apps: [{ name: 'Play Store', icon: 'storefront', pkg: 'com.android.vending' }],
|
||||||
|
requiredPackages: ['com.android.vending'],
|
||||||
|
priority: 88
|
||||||
|
}
|
||||||
|
];
|
||||||
File renamed without changes.
+1
-1
@@ -51,5 +51,5 @@ export interface AgentStatusResponse {
|
|||||||
|
|
||||||
export type AgentStatus = 'idle' | 'running' | 'completed' | 'offline' | string;
|
export type AgentStatus = 'idle' | 'running' | 'completed' | 'offline' | string;
|
||||||
|
|
||||||
export type TaskStatus = 'running' | 'completed' | 'pending' | 'failed';
|
export type TaskStatus = 'running' | 'completed' | 'pending' | 'failed' | 'cancelled';
|
||||||
|
|
||||||
+19
@@ -19,6 +19,25 @@ export interface ActionParam {
|
|||||||
value: string;
|
value: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface ActionExecution {
|
||||||
|
id?: string;
|
||||||
|
trace_id?: string;
|
||||||
|
agent?: string;
|
||||||
|
agent_name?: string;
|
||||||
|
action: string;
|
||||||
|
name?: string;
|
||||||
|
args?: Record<string, any>;
|
||||||
|
params?: Record<string, any>;
|
||||||
|
payload?: any;
|
||||||
|
pre_image_name?: string | null;
|
||||||
|
post_image_name?: string | null;
|
||||||
|
status?: 'success' | 'failed' | 'error' | 'running' | string;
|
||||||
|
duration?: number;
|
||||||
|
duration_ms?: number;
|
||||||
|
error?: string | null;
|
||||||
|
timestamp?: number;
|
||||||
|
}
|
||||||
|
|
||||||
export interface StepEvent {
|
export interface StepEvent {
|
||||||
type: 'action' | 'tool';
|
type: 'action' | 'tool';
|
||||||
data: any;
|
data: any;
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
/**
|
||||||
|
* Copyright 2026 Google LLC
|
||||||
|
*
|
||||||
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
* you may not use this file except in compliance with the License.
|
||||||
|
* You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export type ProbeStatus = 'pass' | 'warn' | 'fail' | 'skipped';
|
||||||
|
|
||||||
|
export type ProbeCategory = 'device' | 'auth' | 'toolchain' | 'runtime';
|
||||||
|
|
||||||
|
export interface ProbeAction {
|
||||||
|
action_type: 'command' | 'hint' | 'link';
|
||||||
|
label: string;
|
||||||
|
payload: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DeviceInfo {
|
||||||
|
serial: string;
|
||||||
|
state: string;
|
||||||
|
model: string | null;
|
||||||
|
product: string | null;
|
||||||
|
android_version: string | null;
|
||||||
|
screen_resolution: string | null;
|
||||||
|
is_emulator: boolean;
|
||||||
|
installed_packages?: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ProbeResult {
|
||||||
|
id: string;
|
||||||
|
category: ProbeCategory;
|
||||||
|
title: string;
|
||||||
|
status: ProbeStatus;
|
||||||
|
is_blocker: boolean;
|
||||||
|
summary: string;
|
||||||
|
description: string;
|
||||||
|
metadata: Record<string, any>;
|
||||||
|
actions: ProbeAction[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface SystemReadinessReport {
|
||||||
|
overall_ready: boolean;
|
||||||
|
blocker_count: number;
|
||||||
|
passed_blocker_count: number;
|
||||||
|
probes: ProbeResult[];
|
||||||
|
active_device: DeviceInfo | null;
|
||||||
|
os_type?: 'linux' | 'darwin' | 'windows' | string;
|
||||||
|
timestamp: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type EmulatorLaunchStage =
|
||||||
|
| 'idle'
|
||||||
|
| 'starting'
|
||||||
|
| 'waiting_for_adb'
|
||||||
|
| 'booting'
|
||||||
|
| 'ready'
|
||||||
|
| 'failed'
|
||||||
|
| 'stopped';
|
||||||
|
|
||||||
|
export interface EmulatorLaunchState {
|
||||||
|
avd_name: string | null;
|
||||||
|
status: EmulatorLaunchStage;
|
||||||
|
pid: number | null;
|
||||||
|
serial: string | null;
|
||||||
|
stage_message: string;
|
||||||
|
progress_percent: number;
|
||||||
|
started_at: number | null;
|
||||||
|
elapsed_seconds: number;
|
||||||
|
error: string | null;
|
||||||
|
logs: string[];
|
||||||
|
can_retry: boolean;
|
||||||
|
}
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
/**
|
||||||
|
* Copyright 2026 Google LLC
|
||||||
|
*
|
||||||
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
* you may not use this file except in compliance with the License.
|
||||||
|
* You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { Injectable } from '@angular/core';
|
||||||
|
import {
|
||||||
|
AppReference,
|
||||||
|
SmartSuggestion,
|
||||||
|
SuggestionCategory,
|
||||||
|
APP_REGISTRY,
|
||||||
|
SMART_TASK_LIBRARY
|
||||||
|
} from '../data/smart-tasks.data';
|
||||||
|
|
||||||
|
@Injectable({
|
||||||
|
providedIn: 'root'
|
||||||
|
})
|
||||||
|
export class TaskRecommendationService {
|
||||||
|
public readonly allTasks: SmartSuggestion[] = SMART_TASK_LIBRARY;
|
||||||
|
public readonly appRegistry = APP_REGISTRY;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if task matches any package installed on device
|
||||||
|
*/
|
||||||
|
public isSuggestionOnDevice(
|
||||||
|
suggestion: SmartSuggestion,
|
||||||
|
installedPackages: Set<string>
|
||||||
|
): boolean {
|
||||||
|
if (installedPackages.size === 0) return false;
|
||||||
|
if (!suggestion.requiredPackages || suggestion.requiredPackages.length === 0) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (suggestion.matchMode === 'all') {
|
||||||
|
return suggestion.requiredPackages.every(pkg => installedPackages.has(pkg));
|
||||||
|
}
|
||||||
|
return suggestion.requiredPackages.some(pkg => installedPackages.has(pkg));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Intelligently rank and filter tasks:
|
||||||
|
* Prioritizes tasks matching apps detected on user's phone without exposing clutter.
|
||||||
|
*/
|
||||||
|
public filterAndRankTasks(
|
||||||
|
installedPackages: Set<string>,
|
||||||
|
category: SuggestionCategory,
|
||||||
|
shuffleOffset: number = 0
|
||||||
|
): SmartSuggestion[] {
|
||||||
|
const hasDevice = installedPackages.size > 0;
|
||||||
|
|
||||||
|
// 1. Score tasks: prioritize tasks whose apps are on the user's phone
|
||||||
|
const scored = this.allTasks.map(task => {
|
||||||
|
const isMatched = this.isSuggestionOnDevice(task, installedPackages);
|
||||||
|
let score = task.priority ?? 50;
|
||||||
|
|
||||||
|
if (hasDevice) {
|
||||||
|
if (isMatched) {
|
||||||
|
score += 100;
|
||||||
|
if (task.requiredPackages && task.requiredPackages.length > 1 && task.matchMode === 'all') {
|
||||||
|
score += 30; // bonus for full multi-app synergy match
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
score -= 40;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return { task, isMatched, score };
|
||||||
|
});
|
||||||
|
|
||||||
|
// 2. Filter by category
|
||||||
|
let filtered = scored.filter(item => {
|
||||||
|
if (category === 'flash') {
|
||||||
|
return item.task.profile === 'flash';
|
||||||
|
}
|
||||||
|
if (category === 'pro') {
|
||||||
|
return item.task.profile === 'pro';
|
||||||
|
}
|
||||||
|
if (category === 'cross_app') {
|
||||||
|
return item.task.category === 'cross_app';
|
||||||
|
}
|
||||||
|
if (category === 'monitor') {
|
||||||
|
return item.task.category === 'monitor';
|
||||||
|
}
|
||||||
|
return true; // 'all'
|
||||||
|
});
|
||||||
|
|
||||||
|
// 3. Sort by score descending (matched apps first)
|
||||||
|
filtered.sort((a, b) => b.score - a.score);
|
||||||
|
|
||||||
|
let result = filtered.map(f => f.task);
|
||||||
|
|
||||||
|
// 4. Apply shuffle rotation
|
||||||
|
if (shuffleOffset > 0 && result.length > 0) {
|
||||||
|
const shift = shuffleOffset % result.length;
|
||||||
|
result = [...result.slice(shift), ...result.slice(0, shift)];
|
||||||
|
}
|
||||||
|
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
}
|
||||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,766 @@
|
|||||||
|
/**
|
||||||
|
* Copyright 2026 Google LLC
|
||||||
|
*
|
||||||
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
* you may not use this file except in compliance with the License.
|
||||||
|
* You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { Component, signal, computed, effect, inject, OnInit, OnDestroy } from '@angular/core';
|
||||||
|
import { CommonModule } from '@angular/common';
|
||||||
|
import { FormsModule } from '@angular/forms';
|
||||||
|
import { Router, RouterLink } from '@angular/router';
|
||||||
|
import { AgentService } from '../../services/agent.service';
|
||||||
|
import { SystemService } from '../../services/system.service';
|
||||||
|
import { DeviceInfo, ProbeResult } from '../../core/models/system.model';
|
||||||
|
import {
|
||||||
|
AppReference,
|
||||||
|
SmartSuggestion,
|
||||||
|
SuggestionCategory
|
||||||
|
} from '../../core/data/smart-tasks.data';
|
||||||
|
import { TaskRecommendationService } from '../../core/services/task-recommendation.service';
|
||||||
|
|
||||||
|
export type { AppReference, SmartSuggestion, SuggestionCategory };
|
||||||
|
|
||||||
|
|
||||||
|
@Component({
|
||||||
|
selector: 'app-home',
|
||||||
|
standalone: true,
|
||||||
|
imports: [CommonModule, FormsModule],
|
||||||
|
templateUrl: './home.component.html',
|
||||||
|
styleUrl: './home.component.scss'
|
||||||
|
})
|
||||||
|
export class HomeComponent implements OnInit, OnDestroy {
|
||||||
|
public agentService = inject(AgentService);
|
||||||
|
public systemService = inject(SystemService);
|
||||||
|
public taskRecService = inject(TaskRecommendationService);
|
||||||
|
private router = inject(Router);
|
||||||
|
|
||||||
|
// High-level navigation mode: 'diagnostics' (System Setup Guide) vs 'launcher' (Task Execution)
|
||||||
|
public activeTab = signal<'diagnostics' | 'launcher'>('launcher');
|
||||||
|
|
||||||
|
// Interactive guide sub-tab inside ADB section: 'emulator' | 'usb' | 'wifi'
|
||||||
|
public activeAdbGuideTab = signal<'emulator' | 'usb' | 'wifi'>('emulator');
|
||||||
|
public emulatorSetupMode = signal<'studio' | 'cli'>('studio');
|
||||||
|
|
||||||
|
// Interactive guide tab for LLM / OCR credentials: 'gemini' | 'ocr'
|
||||||
|
public modelSetupMode = signal<'gemini' | 'custom'>('gemini');
|
||||||
|
public showOcrConfig = signal<boolean>(false);
|
||||||
|
public showFullConfigFile = signal<boolean>(false);
|
||||||
|
|
||||||
|
// Model & Environment configuration from backend
|
||||||
|
public modelConfigEnv = computed(() => this.systemService.modelConfigEnv());
|
||||||
|
|
||||||
|
// Google Gemini API Key State
|
||||||
|
public geminiKeyInput = signal<string>('');
|
||||||
|
public showGeminiKey = signal<boolean>(false);
|
||||||
|
public isSavingGeminiKey = signal<boolean>(false);
|
||||||
|
public isTestingGeminiKey = signal<boolean>(false);
|
||||||
|
public geminiSaveMessage = signal<string | null>(null);
|
||||||
|
public geminiSaveError = signal<string | null>(null);
|
||||||
|
public isGeminiKeyEdited = signal<boolean>(false);
|
||||||
|
|
||||||
|
// Vision OCR API Key State
|
||||||
|
public ocrKeyInput = signal<string>('');
|
||||||
|
public showOcrKey = signal<boolean>(false);
|
||||||
|
public isSavingOcrKey = signal<boolean>(false);
|
||||||
|
public isTestingOcrKey = signal<boolean>(false);
|
||||||
|
public ocrSaveMessage = signal<string | null>(null);
|
||||||
|
public ocrSaveError = signal<string | null>(null);
|
||||||
|
public isOcrKeyEdited = signal<boolean>(false);
|
||||||
|
|
||||||
|
// Clipboard copy state tracker for interactive feedback
|
||||||
|
public copiedId = signal<string | null>(null);
|
||||||
|
|
||||||
|
// Diagnostic re-check state
|
||||||
|
public isRefreshingDiagnostics = signal<boolean>(false);
|
||||||
|
|
||||||
|
// Wireless ADB Interactive connection signals
|
||||||
|
public wifiHost = signal<string>('192.168.1.100');
|
||||||
|
public wifiPort = signal<string>('5555');
|
||||||
|
public isConnectingWifi = signal<boolean>(false);
|
||||||
|
public wifiConnectMessage = signal<string | null>(null);
|
||||||
|
public wifiConnectError = signal<string | null>(null);
|
||||||
|
public adbRestartFeedback = signal<string | null>(null);
|
||||||
|
|
||||||
|
public wifiCommand = computed(() => {
|
||||||
|
const h = this.wifiHost().trim() || '<phone-ip>';
|
||||||
|
const p = this.wifiPort().trim() || '5555';
|
||||||
|
return `adb connect ${h}:${p}`;
|
||||||
|
});
|
||||||
|
|
||||||
|
// Task execution parameters
|
||||||
|
public selectedProfile = signal<'flash' | 'pro'>('flash');
|
||||||
|
public taskGoal = signal<string>('');
|
||||||
|
public isSubmitting = signal<boolean>(false);
|
||||||
|
public errorMessage = signal<string | null>(null);
|
||||||
|
|
||||||
|
// Pro Mode Outputter & Structured Output Configuration
|
||||||
|
public expectedOutput = signal<string>('');
|
||||||
|
public enableOutputter = signal<boolean>(true);
|
||||||
|
public showOutputterDrawer = signal<boolean>(false);
|
||||||
|
|
||||||
|
public toggleOutputterDrawer(): void {
|
||||||
|
this.showOutputterDrawer.update((v) => !v);
|
||||||
|
}
|
||||||
|
|
||||||
|
public applyOutputPreset(preset: string): void {
|
||||||
|
if (this.expectedOutput() === preset) {
|
||||||
|
this.expectedOutput.set('');
|
||||||
|
} else {
|
||||||
|
this.expectedOutput.set(preset);
|
||||||
|
this.showOutputterDrawer.set(true);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Smart intent detection for model recommendation
|
||||||
|
public isIntentSuggestingPro = computed<boolean>(() => {
|
||||||
|
const text = this.taskGoal().toLowerCase();
|
||||||
|
if (!text.trim()) return false;
|
||||||
|
const keywords = [
|
||||||
|
'monitor', 'polling', 'poll', 'wait until', 'loop', 'keep watching',
|
||||||
|
'crash', 'logcat', 'troubleshoot', 'diagnose', 'debug', 'investigate',
|
||||||
|
'compare', 'extract', 'summarize', 'report',
|
||||||
|
'监控', '轮询', '等待', '一直', '直到', '崩溃', '闪退', '排查', '分析日志', '对比', '总结'
|
||||||
|
];
|
||||||
|
return keywords.some(k => text.includes(k));
|
||||||
|
});
|
||||||
|
|
||||||
|
public showIntentSuggestion = computed<boolean>(() => {
|
||||||
|
return this.isIntentSuggestingPro() && this.selectedProfile() === 'flash';
|
||||||
|
});
|
||||||
|
|
||||||
|
// Computed helper states delegating to SystemService
|
||||||
|
public isReady = computed(() => this.systemService.isReady());
|
||||||
|
public isLoading = computed(() => this.systemService.isLoading());
|
||||||
|
public isRestartingAdb = computed(() => this.systemService.isRestartingAdb());
|
||||||
|
public launchingAvd = computed(() => this.systemService.launchingAvd());
|
||||||
|
public emulatorLaunchState = computed(() => this.systemService.emulatorLaunchState());
|
||||||
|
public isEmulatorLaunching = computed(() => this.systemService.isEmulatorLaunching());
|
||||||
|
public showLaunchLogs = signal<boolean>(false);
|
||||||
|
|
||||||
|
// Probes
|
||||||
|
public pythonProbe = computed(() => this.systemService.pythonProbe());
|
||||||
|
public configProbe = computed(() => this.systemService.configProbe());
|
||||||
|
public adbProbe = computed(() => this.systemService.adbProbe());
|
||||||
|
public llmProbe = computed(() => this.systemService.llmProbe());
|
||||||
|
public geminiProbe = computed(() => this.systemService.geminiProbe());
|
||||||
|
public ocrProbe = computed(() => this.systemService.ocrProbe());
|
||||||
|
public toolchainProbe = computed(() => this.systemService.toolchainProbe());
|
||||||
|
|
||||||
|
// Step-level readiness
|
||||||
|
public isEnvironmentReady = computed(() => this.systemService.isEnvironmentReady());
|
||||||
|
public isCredentialsReady = computed(() => this.systemService.isCredentialsReady());
|
||||||
|
public isSkipCredentialsCheck = computed(() => this.systemService.isSkipCredentialsCheck());
|
||||||
|
public isDeviceReady = computed(() => this.systemService.isDeviceReady());
|
||||||
|
|
||||||
|
// Device information
|
||||||
|
public activeDevice = computed(() => this.systemService.activeDevice());
|
||||||
|
public connectedDevices = computed(() => this.systemService.connectedDevices());
|
||||||
|
public installedAvds = computed(() => this.systemService.installedAvds());
|
||||||
|
public emulatorPath = computed(() => this.systemService.emulatorPath());
|
||||||
|
public isEmulatorInPath = computed(() => this.systemService.isEmulatorInPath());
|
||||||
|
public totalStepCount = computed(() => this.systemService.totalStepCount());
|
||||||
|
public passedStepCount = computed(() => this.systemService.passedStepCount());
|
||||||
|
public blockerCount = computed(() => this.systemService.blockerCount());
|
||||||
|
public passedBlockerCount = computed(() => this.systemService.passedBlockerCount());
|
||||||
|
|
||||||
|
// Configured LLM providers from probe metadata
|
||||||
|
public configuredLlmProviders = computed<any[]>(() => {
|
||||||
|
const meta = this.llmProbe()?.metadata;
|
||||||
|
if (meta && Array.isArray(meta['providers'])) {
|
||||||
|
return meta['providers'];
|
||||||
|
}
|
||||||
|
return [];
|
||||||
|
});
|
||||||
|
|
||||||
|
// Multi-OS detection & active OS selection
|
||||||
|
public selectedOs = signal<'linux' | 'darwin' | 'windows' | null>(null);
|
||||||
|
public effectiveOs = computed<'linux' | 'darwin' | 'windows'>(() => {
|
||||||
|
return this.selectedOs() || this.systemService.osType();
|
||||||
|
});
|
||||||
|
|
||||||
|
public oneClickSetupCmd = computed(() => {
|
||||||
|
const os = this.effectiveOs();
|
||||||
|
if (os === 'windows') {
|
||||||
|
return 'powershell -ExecutionPolicy Bypass -File scripts/install_deps.ps1';
|
||||||
|
}
|
||||||
|
return 'bash scripts/install_deps.sh';
|
||||||
|
});
|
||||||
|
|
||||||
|
public adbInstallCmd = computed(() => {
|
||||||
|
const os = this.effectiveOs();
|
||||||
|
if (os === 'windows') {
|
||||||
|
return 'winget install Google.PlatformTools';
|
||||||
|
}
|
||||||
|
if (os === 'darwin') {
|
||||||
|
return 'brew install android-platform-tools';
|
||||||
|
}
|
||||||
|
return 'sudo apt-get install -y adb';
|
||||||
|
});
|
||||||
|
|
||||||
|
public toolchainInstallCmd = computed(() => {
|
||||||
|
const os = this.effectiveOs();
|
||||||
|
if (os === 'windows') {
|
||||||
|
return 'winget install Gyan.FFmpeg Genymobile.scrcpy';
|
||||||
|
}
|
||||||
|
if (os === 'darwin') {
|
||||||
|
return 'brew install ffmpeg scrcpy';
|
||||||
|
}
|
||||||
|
return 'sudo apt-get install -y ffmpeg scrcpy';
|
||||||
|
});
|
||||||
|
|
||||||
|
public emuHypervisorTitle = computed(() => {
|
||||||
|
const os = this.effectiveOs();
|
||||||
|
if (os === 'windows') return 'Enable Windows Hypervisor Platform (WHPX)';
|
||||||
|
if (os === 'darwin') return 'Verify macOS Hypervisor / Install Tools';
|
||||||
|
return 'Enable KVM Hardware Acceleration (Linux)';
|
||||||
|
});
|
||||||
|
|
||||||
|
public emuHypervisorDesc = computed(() => {
|
||||||
|
const os = this.effectiveOs();
|
||||||
|
if (os === 'windows') return 'Enable Windows Hypervisor Platform in PowerShell (Run as Administrator):';
|
||||||
|
if (os === 'darwin') return 'macOS uses native Hypervisor.framework. Install SDK tools via Homebrew (or Studio):';
|
||||||
|
return 'Ensure virtualization permissions are granted to your user account:';
|
||||||
|
});
|
||||||
|
|
||||||
|
public emuHypervisorCmd = computed(() => {
|
||||||
|
const os = this.effectiveOs();
|
||||||
|
if (os === 'windows') return 'Enable-WindowsOptionalFeature -Online -FeatureName HypervisorPlatform';
|
||||||
|
if (os === 'darwin') return 'brew install --cask android-commandlinetools';
|
||||||
|
return 'sudo apt-get install -y qemu-kvm libvirt-daemon-system && sudo adduser $USER kvm';
|
||||||
|
});
|
||||||
|
|
||||||
|
public emuSdkInstallCmd = computed(() => {
|
||||||
|
const os = this.effectiveOs();
|
||||||
|
if (os === 'darwin') {
|
||||||
|
return 'sdkmanager --install "system-images;android-34;google_apis;arm64-v8a" "emulator" "platform-tools"';
|
||||||
|
}
|
||||||
|
return 'sdkmanager --install "system-images;android-34;google_apis;x86_64" "emulator" "platform-tools"';
|
||||||
|
});
|
||||||
|
|
||||||
|
public emuCreateAvdCmd = computed(() => {
|
||||||
|
const os = this.effectiveOs();
|
||||||
|
if (os === 'darwin') {
|
||||||
|
return 'avdmanager create avd -n Pixel_8_API_34 -k "system-images;android-34;google_apis;arm64-v8a" --device "pixel_8"';
|
||||||
|
}
|
||||||
|
return 'avdmanager create avd -n Pixel_8_API_34 -k "system-images;android-34;google_apis;x86_64" --device "pixel_8"';
|
||||||
|
});
|
||||||
|
|
||||||
|
// Flag indicating whether Google Cloud Vision OCR is configured
|
||||||
|
public isOcrConfigured = computed<boolean>(() => {
|
||||||
|
const meta = this.ocrProbe()?.metadata;
|
||||||
|
return meta?.['configured'] === true;
|
||||||
|
});
|
||||||
|
|
||||||
|
public currentApiKey = computed<string>(() => this.systemService.currentApiKey());
|
||||||
|
public apiKeysMap = computed<Record<string, string>>(() => this.systemService.apiKeysMap());
|
||||||
|
|
||||||
|
public savedGeminiKey = computed<string>(() => {
|
||||||
|
const keys = this.apiKeysMap();
|
||||||
|
return keys['google'] || this.currentApiKey() || '';
|
||||||
|
});
|
||||||
|
|
||||||
|
public isGeminiModified = computed<boolean>(() => {
|
||||||
|
return this.geminiKeyInput().trim() !== this.savedGeminiKey().trim();
|
||||||
|
});
|
||||||
|
|
||||||
|
public savedOcrKey = computed<string>(() => {
|
||||||
|
const keys = this.apiKeysMap();
|
||||||
|
return keys['ocr'] || '';
|
||||||
|
});
|
||||||
|
|
||||||
|
public isOcrModified = computed<boolean>(() => {
|
||||||
|
return this.ocrKeyInput().trim() !== this.savedOcrKey().trim();
|
||||||
|
});
|
||||||
|
|
||||||
|
// Rich Smart Suggestions Library (Device-Aware, Flash vs Pro Tailored)
|
||||||
|
public readonly allSuggestions = this.taskRecService.allTasks;
|
||||||
|
|
||||||
|
// Suggestion category filter & shuffle state
|
||||||
|
public selectedCategory = signal<SuggestionCategory>('all');
|
||||||
|
public shuffleOffset = signal<number>(0);
|
||||||
|
|
||||||
|
// Set of installed package strings from active device
|
||||||
|
public installedPackages = computed<Set<string>>(() => {
|
||||||
|
const pkgs = this.activeDevice()?.installed_packages;
|
||||||
|
if (pkgs && Array.isArray(pkgs)) {
|
||||||
|
return new Set(pkgs);
|
||||||
|
}
|
||||||
|
return new Set();
|
||||||
|
});
|
||||||
|
|
||||||
|
public filteredSuggestions = computed<SmartSuggestion[]>(() => {
|
||||||
|
return this.taskRecService.filterAndRankTasks(
|
||||||
|
this.installedPackages(),
|
||||||
|
this.selectedCategory(),
|
||||||
|
this.shuffleOffset()
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
private focusListener = () => {
|
||||||
|
// Silently re-check environment when user returns to the browser tab
|
||||||
|
this.systemService.fetchReadiness().subscribe();
|
||||||
|
};
|
||||||
|
|
||||||
|
constructor() {
|
||||||
|
effect(() => {
|
||||||
|
const keys = this.systemService.apiKeysMap();
|
||||||
|
const current = this.systemService.currentApiKey();
|
||||||
|
const googleKey = keys['google'] || current || '';
|
||||||
|
const ocrKey = keys['ocr'] || '';
|
||||||
|
if (!this.isGeminiKeyEdited()) {
|
||||||
|
this.geminiKeyInput.set(googleKey);
|
||||||
|
}
|
||||||
|
if (!this.isOcrKeyEdited()) {
|
||||||
|
this.ocrKeyInput.set(ocrKey);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
ngOnInit(): void {
|
||||||
|
// Initial fetch of system readiness & model configuration
|
||||||
|
this.systemService.fetchReadiness().subscribe();
|
||||||
|
this.systemService.fetchModelConfigEnv().subscribe();
|
||||||
|
|
||||||
|
window.addEventListener('focus', this.focusListener);
|
||||||
|
}
|
||||||
|
|
||||||
|
ngOnDestroy(): void {
|
||||||
|
window.removeEventListener('focus', this.focusListener);
|
||||||
|
}
|
||||||
|
|
||||||
|
public setTab(tab: 'diagnostics' | 'launcher'): void {
|
||||||
|
this.activeTab.set(tab);
|
||||||
|
if (tab === 'diagnostics') {
|
||||||
|
this.systemService.fetchReadiness().subscribe();
|
||||||
|
this.systemService.fetchModelConfigEnv().subscribe();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public setSelectedOs(os: 'linux' | 'darwin' | 'windows'): void {
|
||||||
|
this.selectedOs.set(os);
|
||||||
|
}
|
||||||
|
|
||||||
|
public setAdbGuideTab(tab: 'emulator' | 'usb' | 'wifi'): void {
|
||||||
|
this.activeAdbGuideTab.set(tab);
|
||||||
|
}
|
||||||
|
|
||||||
|
public setEmulatorSetupMode(mode: 'studio' | 'cli'): void {
|
||||||
|
this.emulatorSetupMode.set(mode);
|
||||||
|
}
|
||||||
|
|
||||||
|
public setModelSetupMode(mode: 'gemini' | 'custom'): void {
|
||||||
|
this.modelSetupMode.set(mode);
|
||||||
|
if (mode === 'custom') {
|
||||||
|
this.systemService.setSkipCredentialsCheck(true);
|
||||||
|
this.systemService.fetchModelConfigEnv().subscribe();
|
||||||
|
} else {
|
||||||
|
this.systemService.setSkipCredentialsCheck(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public toggleFullConfigFile(): void {
|
||||||
|
this.showFullConfigFile.update(v => !v);
|
||||||
|
}
|
||||||
|
|
||||||
|
public toggleGeminiKeyVisibility(): void {
|
||||||
|
this.showGeminiKey.update(v => !v);
|
||||||
|
}
|
||||||
|
|
||||||
|
public toggleOcrKeyVisibility(): void {
|
||||||
|
this.showOcrKey.update(v => !v);
|
||||||
|
}
|
||||||
|
|
||||||
|
public toggleOcrConfig(): void {
|
||||||
|
this.showOcrConfig.update(v => !v);
|
||||||
|
}
|
||||||
|
|
||||||
|
public onGeminiKeyChange(val: string): void {
|
||||||
|
this.geminiKeyInput.set(val);
|
||||||
|
this.isGeminiKeyEdited.set(true);
|
||||||
|
this.geminiSaveError.set(null);
|
||||||
|
this.geminiSaveMessage.set(null);
|
||||||
|
}
|
||||||
|
|
||||||
|
public onOcrKeyChange(val: string): void {
|
||||||
|
this.ocrKeyInput.set(val);
|
||||||
|
this.isOcrKeyEdited.set(true);
|
||||||
|
this.ocrSaveError.set(null);
|
||||||
|
this.ocrSaveMessage.set(null);
|
||||||
|
}
|
||||||
|
|
||||||
|
public saveGeminiKey(): void {
|
||||||
|
const key = this.geminiKeyInput().trim();
|
||||||
|
if (!key) return;
|
||||||
|
this.isSavingGeminiKey.set(true);
|
||||||
|
this.geminiSaveError.set(null);
|
||||||
|
this.geminiSaveMessage.set(null);
|
||||||
|
|
||||||
|
this.systemService.updateApiKey('google', key, true).subscribe({
|
||||||
|
next: (res) => {
|
||||||
|
this.isSavingGeminiKey.set(false);
|
||||||
|
this.isGeminiKeyEdited.set(false);
|
||||||
|
this.geminiSaveMessage.set(res?.message || '✓ Gemini API key verified & saved successfully.');
|
||||||
|
setTimeout(() => this.geminiSaveMessage.set(null), 5000);
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
this.isSavingGeminiKey.set(false);
|
||||||
|
this.geminiSaveError.set(err?.error?.detail || err?.message || 'Failed to update Gemini API key.');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
public clearGeminiKey(): void {
|
||||||
|
this.geminiKeyInput.set('');
|
||||||
|
this.isGeminiKeyEdited.set(false);
|
||||||
|
this.geminiSaveError.set(null);
|
||||||
|
this.geminiSaveMessage.set(null);
|
||||||
|
|
||||||
|
if (this.savedGeminiKey().trim()) {
|
||||||
|
this.isSavingGeminiKey.set(true);
|
||||||
|
this.systemService.updateApiKey('google', '', true).subscribe({
|
||||||
|
next: (res) => {
|
||||||
|
this.isSavingGeminiKey.set(false);
|
||||||
|
this.geminiSaveMessage.set(res?.message || '✓ Gemini API key cleared.');
|
||||||
|
setTimeout(() => this.geminiSaveMessage.set(null), 5000);
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
this.isSavingGeminiKey.set(false);
|
||||||
|
this.geminiSaveError.set(err?.error?.detail || err?.message || 'Failed to clear Gemini API key.');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
} else {
|
||||||
|
this.geminiSaveMessage.set('✓ Gemini API key cleared.');
|
||||||
|
setTimeout(() => this.geminiSaveMessage.set(null), 3000);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public saveOcrKey(): void {
|
||||||
|
const key = this.ocrKeyInput().trim();
|
||||||
|
if (!key) return;
|
||||||
|
this.isSavingOcrKey.set(true);
|
||||||
|
this.ocrSaveError.set(null);
|
||||||
|
this.ocrSaveMessage.set(null);
|
||||||
|
|
||||||
|
this.systemService.updateApiKey('ocr', key, true).subscribe({
|
||||||
|
next: (res) => {
|
||||||
|
this.isSavingOcrKey.set(false);
|
||||||
|
this.isOcrKeyEdited.set(false);
|
||||||
|
this.ocrSaveMessage.set(res?.message || '✓ Vision OCR API key verified & saved.');
|
||||||
|
setTimeout(() => this.ocrSaveMessage.set(null), 5000);
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
this.isSavingOcrKey.set(false);
|
||||||
|
this.ocrSaveError.set(err?.error?.detail || err?.message || 'Failed to update Vision OCR key.');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
public clearOcrKey(): void {
|
||||||
|
this.ocrKeyInput.set('');
|
||||||
|
this.isOcrKeyEdited.set(false);
|
||||||
|
this.ocrSaveError.set(null);
|
||||||
|
this.ocrSaveMessage.set(null);
|
||||||
|
|
||||||
|
if (this.savedOcrKey().trim()) {
|
||||||
|
this.isSavingOcrKey.set(true);
|
||||||
|
this.systemService.updateApiKey('ocr', '', true).subscribe({
|
||||||
|
next: (res) => {
|
||||||
|
this.isSavingOcrKey.set(false);
|
||||||
|
this.ocrSaveMessage.set(res?.message || '✓ Vision OCR API key cleared.');
|
||||||
|
setTimeout(() => this.ocrSaveMessage.set(null), 5000);
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
this.isSavingOcrKey.set(false);
|
||||||
|
this.ocrSaveError.set(err?.error?.detail || err?.message || 'Failed to clear Vision OCR key.');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
} else {
|
||||||
|
this.ocrSaveMessage.set('✓ Vision OCR API key cleared.');
|
||||||
|
setTimeout(() => this.ocrSaveMessage.set(null), 3000);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public testGeminiKey(): void {
|
||||||
|
const key = this.geminiKeyInput().trim();
|
||||||
|
if (!key) return;
|
||||||
|
this.isTestingGeminiKey.set(true);
|
||||||
|
this.geminiSaveError.set(null);
|
||||||
|
this.geminiSaveMessage.set(null);
|
||||||
|
|
||||||
|
this.systemService.testApiKey('google', key).subscribe({
|
||||||
|
next: (res) => {
|
||||||
|
this.isTestingGeminiKey.set(false);
|
||||||
|
if (res?.valid) {
|
||||||
|
this.geminiSaveMessage.set(res?.message || '✓ Gemini API key is valid!');
|
||||||
|
} else {
|
||||||
|
this.geminiSaveError.set(res?.message || 'Gemini API key verification failed.');
|
||||||
|
}
|
||||||
|
setTimeout(() => this.geminiSaveMessage.set(null), 5000);
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
this.isTestingGeminiKey.set(false);
|
||||||
|
this.geminiSaveError.set(err?.error?.detail || err?.message || 'Gemini API key test failed.');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
public testOcrKey(): void {
|
||||||
|
const key = this.ocrKeyInput().trim();
|
||||||
|
if (!key) return;
|
||||||
|
this.isTestingOcrKey.set(true);
|
||||||
|
this.ocrSaveError.set(null);
|
||||||
|
this.ocrSaveMessage.set(null);
|
||||||
|
|
||||||
|
this.systemService.testApiKey('ocr', key).subscribe({
|
||||||
|
next: (res) => {
|
||||||
|
this.isTestingOcrKey.set(false);
|
||||||
|
if (res?.valid) {
|
||||||
|
this.ocrSaveMessage.set(res?.message || '✓ Vision OCR API key is valid!');
|
||||||
|
} else {
|
||||||
|
this.ocrSaveError.set(res?.message || 'Vision OCR API key verification failed.');
|
||||||
|
}
|
||||||
|
setTimeout(() => this.ocrSaveMessage.set(null), 5000);
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
this.isTestingOcrKey.set(false);
|
||||||
|
this.ocrSaveError.set(err?.error?.detail || err?.message || 'Vision OCR API key test failed.');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
public getProviderDisplayName(tab: string): string {
|
||||||
|
switch (tab) {
|
||||||
|
case 'gemini': return 'Gemini';
|
||||||
|
case 'ocr': return 'Vision OCR';
|
||||||
|
default: return tab;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public getProviderEnvVar(tab: string): string {
|
||||||
|
switch (tab) {
|
||||||
|
case 'gemini': return 'GEMINI_API_KEY';
|
||||||
|
case 'ocr': return 'GOOGLE_VISION_API_KEY';
|
||||||
|
default: return 'API_KEY';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public getProviderHint(tab: string): string {
|
||||||
|
switch (tab) {
|
||||||
|
case 'gemini':
|
||||||
|
return 'For a quick start, Google Gemini provides a free API key. Artemis also supports other models (OpenAI, Claude, OpenRouter, etc.)—you can configure your own API keys directly in .env or your environment.';
|
||||||
|
case 'ocr':
|
||||||
|
return 'Google Cloud Vision API key for on-screen OCR text detection and UI grounding.';
|
||||||
|
default:
|
||||||
|
return 'Configure your API key or use environment definitions.';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public skipCredentialsCheck(): void {
|
||||||
|
this.systemService.skipCredentialsCheck();
|
||||||
|
}
|
||||||
|
|
||||||
|
public getApiKeyPlaceholder(tab: string): string {
|
||||||
|
switch (tab) {
|
||||||
|
case 'gemini': return 'Enter Gemini API Key (e.g. AIzaSy...)';
|
||||||
|
case 'ocr': return 'Enter Vision OCR API Key (e.g. AIzaSy...)';
|
||||||
|
default: return 'Enter API Key...';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public isTabProviderActive(tab: string): boolean {
|
||||||
|
if (tab === 'ocr') {
|
||||||
|
return this.isOcrConfigured();
|
||||||
|
}
|
||||||
|
const targetProvider = tab === 'gemini' ? 'google' : tab;
|
||||||
|
const providers = this.configuredLlmProviders();
|
||||||
|
return providers.some(p => p.provider === targetProvider);
|
||||||
|
}
|
||||||
|
|
||||||
|
public refreshReadiness(): void {
|
||||||
|
this.isRefreshingDiagnostics.set(true);
|
||||||
|
this.systemService.fetchReadiness().subscribe({
|
||||||
|
next: () => {
|
||||||
|
this.systemService.fetchModelConfigEnv().subscribe({
|
||||||
|
next: () => {
|
||||||
|
setTimeout(() => this.isRefreshingDiagnostics.set(false), 450);
|
||||||
|
},
|
||||||
|
error: () => this.isRefreshingDiagnostics.set(false)
|
||||||
|
});
|
||||||
|
},
|
||||||
|
error: () => this.isRefreshingDiagnostics.set(false)
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
public restartAdbServer(): void {
|
||||||
|
this.adbRestartFeedback.set(null);
|
||||||
|
this.systemService.restartAdb().subscribe({
|
||||||
|
next: (res) => {
|
||||||
|
this.adbRestartFeedback.set('ADB Refreshed ✓');
|
||||||
|
setTimeout(() => this.adbRestartFeedback.set(null), 2500);
|
||||||
|
},
|
||||||
|
error: () => {
|
||||||
|
this.adbRestartFeedback.set('Restart Failed');
|
||||||
|
setTimeout(() => this.adbRestartFeedback.set(null), 3000);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
public connectWifiDevice(): void {
|
||||||
|
const host = this.wifiHost().trim();
|
||||||
|
const portStr = this.wifiPort().trim() || '5555';
|
||||||
|
const port = parseInt(portStr, 10) || 5555;
|
||||||
|
|
||||||
|
if (!host) {
|
||||||
|
this.wifiConnectError.set('Please enter a valid IP address.');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
this.isConnectingWifi.set(true);
|
||||||
|
this.wifiConnectError.set(null);
|
||||||
|
this.wifiConnectMessage.set(null);
|
||||||
|
|
||||||
|
this.systemService.connectWirelessAdb(host, port).subscribe({
|
||||||
|
next: (res) => {
|
||||||
|
this.isConnectingWifi.set(false);
|
||||||
|
const cr = res?.connect_result;
|
||||||
|
if (cr?.success) {
|
||||||
|
this.wifiConnectMessage.set(`Connected to ${host}:${port}!`);
|
||||||
|
setTimeout(() => this.wifiConnectMessage.set(null), 4000);
|
||||||
|
} else {
|
||||||
|
this.wifiConnectError.set(cr?.message || 'Connection failed. Please check phone IP & Wi-Fi.');
|
||||||
|
}
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
this.isConnectingWifi.set(false);
|
||||||
|
this.wifiConnectError.set(err?.error?.detail || 'Failed to connect. Please check adb connection.');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
public launchAvdEmulator(avdName: string): void {
|
||||||
|
this.systemService.launchEmulator(avdName).subscribe();
|
||||||
|
}
|
||||||
|
|
||||||
|
public toggleLaunchLogs(): void {
|
||||||
|
this.showLaunchLogs.update(v => !v);
|
||||||
|
}
|
||||||
|
|
||||||
|
public stopEmulator(): void {
|
||||||
|
this.systemService.stopEmulator().subscribe();
|
||||||
|
}
|
||||||
|
|
||||||
|
public dismissEmulatorStatus(): void {
|
||||||
|
this.systemService.dismissEmulatorStatus().subscribe();
|
||||||
|
}
|
||||||
|
|
||||||
|
public selectTargetDevice(serial: string): void {
|
||||||
|
this.systemService.selectDevice(serial).subscribe();
|
||||||
|
}
|
||||||
|
|
||||||
|
public getEmulatorCommand(avdName: string): string {
|
||||||
|
const p = this.emulatorPath();
|
||||||
|
const cmd = this.isEmulatorInPath() ? 'emulator' : (p || '~/Android/Sdk/emulator/emulator');
|
||||||
|
return `${cmd} -avd ${avdName}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
public copyToClipboard(text: string, id: string): void {
|
||||||
|
if (navigator.clipboard) {
|
||||||
|
navigator.clipboard.writeText(text).then(() => {
|
||||||
|
this.copiedId.set(id);
|
||||||
|
setTimeout(() => {
|
||||||
|
if (this.copiedId() === id) {
|
||||||
|
this.copiedId.set(null);
|
||||||
|
}
|
||||||
|
}, 2000);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public setProfile(profile: 'flash' | 'pro'): void {
|
||||||
|
this.selectedProfile.set(profile);
|
||||||
|
}
|
||||||
|
|
||||||
|
public setCategory(cat: SuggestionCategory): void {
|
||||||
|
this.selectedCategory.set(cat);
|
||||||
|
}
|
||||||
|
|
||||||
|
public shuffleSuggestions(): void {
|
||||||
|
this.shuffleOffset.update(v => v + 3);
|
||||||
|
}
|
||||||
|
|
||||||
|
public getAppNamesDisplay(apps: AppReference[]): string {
|
||||||
|
return apps.map(a => a.name).join(' + ');
|
||||||
|
}
|
||||||
|
|
||||||
|
public applySuggestion(item: SmartSuggestion): void {
|
||||||
|
this.taskGoal.set(item.goal);
|
||||||
|
this.selectedProfile.set(item.profile);
|
||||||
|
this.errorMessage.set(null);
|
||||||
|
}
|
||||||
|
|
||||||
|
public applyQuickPrompt(promptGoal: string, profile?: 'flash' | 'pro'): void {
|
||||||
|
this.taskGoal.set(promptGoal);
|
||||||
|
if (profile) {
|
||||||
|
this.selectedProfile.set(profile);
|
||||||
|
}
|
||||||
|
this.errorMessage.set(null);
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
public proceedToLauncher(): void {
|
||||||
|
this.activeTab.set('launcher');
|
||||||
|
}
|
||||||
|
|
||||||
|
public runTask(): void {
|
||||||
|
const goal = this.taskGoal().trim();
|
||||||
|
if (!goal) {
|
||||||
|
this.errorMessage.set('Please enter a task goal before running.');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!this.isReady()) {
|
||||||
|
this.errorMessage.set('System prerequisites are not satisfied. Please review System Setup first.');
|
||||||
|
this.activeTab.set('diagnostics');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
this.isSubmitting.set(true);
|
||||||
|
this.errorMessage.set(null);
|
||||||
|
|
||||||
|
this.agentService
|
||||||
|
.runTask(
|
||||||
|
goal,
|
||||||
|
this.selectedProfile(),
|
||||||
|
this.selectedProfile() === 'pro' && this.expectedOutput().trim()
|
||||||
|
? this.expectedOutput().trim()
|
||||||
|
: undefined,
|
||||||
|
this.selectedProfile() === 'pro' ? this.enableOutputter() : undefined
|
||||||
|
)
|
||||||
|
.subscribe({
|
||||||
|
next: () => {
|
||||||
|
this.isSubmitting.set(false);
|
||||||
|
this.router.navigate(['/workspace']);
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
console.error('Failed to submit task from home page:', err);
|
||||||
|
this.isSubmitting.set(false);
|
||||||
|
this.errorMessage.set(
|
||||||
|
err?.error?.detail || 'Failed to submit task. Please check server connection.'
|
||||||
|
);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -0,0 +1,189 @@
|
|||||||
|
<!--
|
||||||
|
Copyright 2026 Google LLC
|
||||||
|
|
||||||
|
Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
you may not use this file except in compliance with the License.
|
||||||
|
You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License.
|
||||||
|
-->
|
||||||
|
|
||||||
|
<div class="workspace-container" [class.is-resizing]="isDragging">
|
||||||
|
<!-- Left Main Panel: Raw Agent Log Stream -->
|
||||||
|
<main class="left-panel glass-panel">
|
||||||
|
<app-agent-stream></app-agent-stream>
|
||||||
|
|
||||||
|
<!-- Bottom Floating Command Bar Dock -->
|
||||||
|
<div class="workspace-floating-bar-wrapper" [class.is-expanded]="isBarExpanded()" [class.is-dormant]="!isBarExpanded()"
|
||||||
|
[class.is-focused]="isInputFocused()" [class.has-text]="!!taskInput.trim()" [class.is-running]="isTaskRunning()">
|
||||||
|
<!-- Floating Error / Notification Toast -->
|
||||||
|
<div class="floating-dock-toast error" *ngIf="errorMessage">
|
||||||
|
<span class="material-symbols-outlined toast-icon">error</span>
|
||||||
|
<span class="toast-text">{{ errorMessage }}</span>
|
||||||
|
<button type="button" class="btn-toast-close" (click)="errorMessage = null" aria-label="Close error message">
|
||||||
|
<span class="material-symbols-outlined">close</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Dock Row (Contains the Main Card and the Separate Stop Button when running) -->
|
||||||
|
<div class="floating-dock-row">
|
||||||
|
<!-- Floating Dock Main Card -->
|
||||||
|
<div class="floating-dock-card" (click)="onCardClick($event)"
|
||||||
|
(mouseenter)="isHoveringCard.set(true)" (mouseleave)="isHoveringCard.set(false)">
|
||||||
|
<!-- Ambient Liquid Wave / Aurora Background Layer (Active in both Dormant & Expanded) -->
|
||||||
|
<div class="dock-wave-container" [class.is-pro]="selectedProfile() === 'pro'">
|
||||||
|
<div class="wave-glow-ambient"></div>
|
||||||
|
<div class="wave-tracks-wrapper">
|
||||||
|
<!-- Back Fluid Wave -->
|
||||||
|
<svg class="liquid-wave wave-back" viewBox="0 0 2400 80" preserveAspectRatio="none">
|
||||||
|
<defs>
|
||||||
|
<linearGradient id="waveGradBack" x1="0%" y1="0%" x2="100%" y2="0%">
|
||||||
|
<stop offset="0%" class="wave-stop-1" />
|
||||||
|
<stop offset="25%" class="wave-stop-2" />
|
||||||
|
<stop offset="50%" class="wave-stop-3" />
|
||||||
|
<stop offset="75%" class="wave-stop-4" />
|
||||||
|
<stop offset="100%" class="wave-stop-1" />
|
||||||
|
</linearGradient>
|
||||||
|
</defs>
|
||||||
|
<path d="M 0,25 Q 300,55 600,20 T 1200,25 Q 1500,55 1800,20 T 2400,25 L 2400,80 L 0,80 Z"
|
||||||
|
fill="url(#waveGradBack)"></path>
|
||||||
|
</svg>
|
||||||
|
|
||||||
|
<!-- Mid Fluid Wave -->
|
||||||
|
<svg class="liquid-wave wave-mid" viewBox="0 0 2400 80" preserveAspectRatio="none">
|
||||||
|
<defs>
|
||||||
|
<linearGradient id="waveGradMid" x1="0%" y1="0%" x2="100%" y2="0%">
|
||||||
|
<stop offset="0%" class="wave-stop-3" />
|
||||||
|
<stop offset="35%" class="wave-stop-1" />
|
||||||
|
<stop offset="70%" class="wave-stop-4" />
|
||||||
|
<stop offset="100%" class="wave-stop-3" />
|
||||||
|
</linearGradient>
|
||||||
|
</defs>
|
||||||
|
<path d="M 0,35 Q 300,5 600,45 T 1200,35 Q 1500,5 1800,45 T 2400,35 L 2400,80 L 0,80 Z"
|
||||||
|
fill="url(#waveGradMid)"></path>
|
||||||
|
</svg>
|
||||||
|
|
||||||
|
<!-- Front Fluid Wave -->
|
||||||
|
<svg class="liquid-wave wave-front" viewBox="0 0 2400 80" preserveAspectRatio="none">
|
||||||
|
<defs>
|
||||||
|
<linearGradient id="waveGradFront" x1="0%" y1="0%" x2="100%" y2="0%">
|
||||||
|
<stop offset="0%" class="wave-stop-2" />
|
||||||
|
<stop offset="30%" class="wave-stop-4" />
|
||||||
|
<stop offset="65%" class="wave-stop-1" />
|
||||||
|
<stop offset="100%" class="wave-stop-2" />
|
||||||
|
</linearGradient>
|
||||||
|
</defs>
|
||||||
|
<path d="M 0,18 Q 300,50 600,15 T 1200,18 Q 1500,50 1800,15 T 2400,18 L 2400,80 L 0,80 Z"
|
||||||
|
fill="url(#waveGradFront)"></path>
|
||||||
|
</svg>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Dormant / Capsule Peek View (Visible when mouse is away and input idle) -->
|
||||||
|
<div class="capsule-peek-view">
|
||||||
|
<span class="capsule-placeholder">Start a new task...</span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Expanded Full View Content (Visible on hover, focus, or text) -->
|
||||||
|
<div class="expanded-card-content">
|
||||||
|
<div class="card-main-input-row">
|
||||||
|
<!-- Textarea Container with Dynamic English Placeholder -->
|
||||||
|
<div class="input-content-area">
|
||||||
|
<textarea #dockInput [(ngModel)]="taskInput"
|
||||||
|
(input)="onTextareaInput($event)"
|
||||||
|
(keydown)="onKeyDown($event)"
|
||||||
|
(focus)="onInputFocus()"
|
||||||
|
(blur)="onInputBlur()"
|
||||||
|
rows="1"
|
||||||
|
[disabled]="isSubmitting"
|
||||||
|
[placeholder]="selectedProfile() === 'pro'
|
||||||
|
? 'Pro: Best for complex long-running tasks, deep reasoning & video analysis'
|
||||||
|
: 'Flash: Fast execution for direct actions and tasks within 25 steps'"
|
||||||
|
class="dock-textarea"
|
||||||
|
spellcheck="false"
|
||||||
|
></textarea>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Clear Button (visible when text is entered) -->
|
||||||
|
<button *ngIf="taskInput.trim() && !isSubmitting" type="button" class="dock-circle-btn btn-clear"
|
||||||
|
(click)="clearInput($event)" title="Clear input text" aria-label="Clear input text">
|
||||||
|
<span class="material-symbols-outlined">close</span>
|
||||||
|
</button>
|
||||||
|
|
||||||
|
<!-- Stop Button (Visible when task is running) -->
|
||||||
|
<button *ngIf="isTaskRunning()" type="button" class="dock-circle-btn btn-stop" [disabled]="isSubmitting"
|
||||||
|
(click)="stopTask($event)"
|
||||||
|
title="Stop running task"
|
||||||
|
aria-label="Stop running task"
|
||||||
|
>
|
||||||
|
<span class="material-symbols-outlined">stop</span>
|
||||||
|
</button>
|
||||||
|
|
||||||
|
<!-- Send Button -->
|
||||||
|
<button type="button" class="dock-circle-btn btn-send"
|
||||||
|
[class.has-content]="!!taskInput.trim()"
|
||||||
|
[disabled]="!taskInput.trim() || isSubmitting"
|
||||||
|
(click)="submitTask()"
|
||||||
|
[title]="isSubmitting ? 'Starting task...' : 'Start task (Enter)'"
|
||||||
|
aria-label="Start task"
|
||||||
|
>
|
||||||
|
<span class="material-symbols-outlined spinning-icon" *ngIf="isSubmitting">sync</span>
|
||||||
|
<span class="material-symbols-outlined" *ngIf="!isSubmitting">arrow_upward</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Bottom Segmented Controls Row -->
|
||||||
|
<div class="card-footer-row">
|
||||||
|
<div class="footer-left-group">
|
||||||
|
<!-- Segmented Model Toggle (Flash / Pro) -->
|
||||||
|
<div class="model-profile-toggle" role="group" aria-label="Select AI Model">
|
||||||
|
<button type="button" class="profile-toggle-pill" [class.active]="selectedProfile() === 'flash'"
|
||||||
|
(mousedown)="$event.preventDefault()" (click)="setProfile('flash', $event)"
|
||||||
|
title="Flash: Fast execution for standard app workflows">
|
||||||
|
<span class="material-symbols-outlined pill-icon icon-flash">bolt</span>
|
||||||
|
<span class="pill-label">Flash</span>
|
||||||
|
</button>
|
||||||
|
<button type="button" class="profile-toggle-pill" [class.active]="selectedProfile() === 'pro'"
|
||||||
|
(mousedown)="$event.preventDefault()" (click)="setProfile('pro', $event)"
|
||||||
|
title="Pro: Deep multimodal reasoning for complex multi-step tasks">
|
||||||
|
<span class="material-symbols-outlined pill-icon icon-pro">psychology</span>
|
||||||
|
<span class="pill-label">Pro</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Separate Standalone Stop Button Beside the Capsule (Visible when task is running) -->
|
||||||
|
<button *ngIf="isTaskRunning()" type="button" class="standalone-stop-btn"
|
||||||
|
[disabled]="isSubmitting"
|
||||||
|
(click)="stopTask($event)"
|
||||||
|
title="Stop running task"
|
||||||
|
aria-label="Stop running task"
|
||||||
|
>
|
||||||
|
<span class="material-symbols-outlined">stop</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</main>
|
||||||
|
|
||||||
|
<!-- Split Pane Drag Resizer -->
|
||||||
|
<div class="resizer" (mousedown)="onDragStart($event)">
|
||||||
|
<div class="resizer-handle"></div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Right Sidebar Panel: Chat Interface Mock -->
|
||||||
|
<aside class="right-panel" [style.width.px]="rightPanelWidth">
|
||||||
|
<app-chat-interface></app-chat-interface>
|
||||||
|
</aside>
|
||||||
|
|
||||||
|
<!-- Floating Video Replay Player Modal / Window -->
|
||||||
|
<app-floating-video-player></app-floating-video-player>
|
||||||
|
</div>
|
||||||
@@ -0,0 +1,865 @@
|
|||||||
|
/**
|
||||||
|
* Copyright 2026 Google LLC
|
||||||
|
*
|
||||||
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
* you may not use this file except in compliance with the License.
|
||||||
|
* You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
|
$bg-base: #e2e5ea;
|
||||||
|
$font-sans: 'Google Sans Text', 'Google Sans', 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif;
|
||||||
|
$primary: #0f172a;
|
||||||
|
$primary-hover: #1e293b;
|
||||||
|
$rose: #ef4444;
|
||||||
|
$rose-dark: #dc2626;
|
||||||
|
|
||||||
|
:host {
|
||||||
|
display: block;
|
||||||
|
height: 100vh;
|
||||||
|
width: 100vw;
|
||||||
|
overflow: hidden;
|
||||||
|
font-family: $font-sans;
|
||||||
|
}
|
||||||
|
|
||||||
|
.workspace-container {
|
||||||
|
position: relative;
|
||||||
|
display: flex;
|
||||||
|
height: 100vh;
|
||||||
|
width: 100vw;
|
||||||
|
background-color: $bg-base;
|
||||||
|
overflow: hidden;
|
||||||
|
|
||||||
|
&.is-resizing {
|
||||||
|
cursor: col-resize;
|
||||||
|
user-select: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* --- Glassmorphism Panel --- */
|
||||||
|
.glass-panel {
|
||||||
|
position: relative;
|
||||||
|
z-index: 1;
|
||||||
|
background: #f4fafb;
|
||||||
|
border: none;
|
||||||
|
box-shadow: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.left-panel {
|
||||||
|
flex: 1 0 500px;
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
overflow: hidden;
|
||||||
|
background: #f4fafb;
|
||||||
|
border-right: 1px solid #e2e8f0;
|
||||||
|
border-left: none;
|
||||||
|
border-top: none;
|
||||||
|
border-bottom: none;
|
||||||
|
min-width: 450px;
|
||||||
|
height: 100%;
|
||||||
|
}
|
||||||
|
|
||||||
|
.right-panel {
|
||||||
|
flex: 0 1 auto;
|
||||||
|
min-width: 250px;
|
||||||
|
max-width: 100%;
|
||||||
|
padding: 0;
|
||||||
|
border: none;
|
||||||
|
background-color: #ffffff;
|
||||||
|
box-shadow: -6px 0 24px rgba(0, 0, 0, 0.03);
|
||||||
|
height: 100%;
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
box-sizing: border-box;
|
||||||
|
z-index: 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
.resizer {
|
||||||
|
width: 12px;
|
||||||
|
cursor: col-resize;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
z-index: 10;
|
||||||
|
transition: background 0.2s ease;
|
||||||
|
margin: 0 -6px;
|
||||||
|
|
||||||
|
&:hover, .workspace-container.is-resizing & {
|
||||||
|
background: rgba(255, 255, 255, 0.15);
|
||||||
|
|
||||||
|
.resizer-handle {
|
||||||
|
background: rgba(94, 114, 228, 0.8);
|
||||||
|
height: 60px;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
.resizer-handle {
|
||||||
|
width: 4px;
|
||||||
|
height: 40px;
|
||||||
|
background: rgba(0, 0, 0, 0.15);
|
||||||
|
border-radius: 4px;
|
||||||
|
transition: all 0.2s ease;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========================================================================= */
|
||||||
|
/* BOTTOM FLOATING COMMAND BAR (Proximity Reveal & Minimalist Glassmorphism) */
|
||||||
|
/* ========================================================================= */
|
||||||
|
.workspace-floating-bar-wrapper {
|
||||||
|
position: absolute;
|
||||||
|
bottom: 18px;
|
||||||
|
left: 50%;
|
||||||
|
transform: translateX(-50%);
|
||||||
|
width: 100%;
|
||||||
|
max-width: min(680px, calc(100% - 36px));
|
||||||
|
z-index: 50;
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
align-items: center;
|
||||||
|
pointer-events: none;
|
||||||
|
transition: all 0.25s cubic-bezier(0.16, 1, 0.3, 1);
|
||||||
|
|
||||||
|
/* Floating Toast / Error Notification */
|
||||||
|
.floating-dock-toast {
|
||||||
|
pointer-events: auto;
|
||||||
|
margin-bottom: 8px;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 8px;
|
||||||
|
padding: 7px 14px;
|
||||||
|
border-radius: 12px;
|
||||||
|
font-size: 12.5px;
|
||||||
|
font-weight: 500;
|
||||||
|
box-shadow: 0 8px 24px -4px rgba(239, 68, 68, 0.15);
|
||||||
|
animation: slideUpFade 0.2s ease-out;
|
||||||
|
max-width: 90%;
|
||||||
|
|
||||||
|
&.error {
|
||||||
|
background: rgba(254, 242, 242, 0.88);
|
||||||
|
backdrop-filter: blur(20px) saturate(180%);
|
||||||
|
-webkit-backdrop-filter: blur(20px) saturate(180%);
|
||||||
|
border: 1px solid rgba(254, 202, 202, 0.9);
|
||||||
|
color: #991b1b;
|
||||||
|
|
||||||
|
.toast-icon {
|
||||||
|
font-size: 16px;
|
||||||
|
color: #ef4444;
|
||||||
|
flex-shrink: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.toast-text {
|
||||||
|
flex: 1;
|
||||||
|
line-height: 1.4;
|
||||||
|
}
|
||||||
|
|
||||||
|
.btn-toast-close {
|
||||||
|
background: transparent;
|
||||||
|
border: none;
|
||||||
|
padding: 2px;
|
||||||
|
cursor: pointer;
|
||||||
|
color: #991b1b;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
border-radius: 4px;
|
||||||
|
opacity: 0.7;
|
||||||
|
transition: opacity 0.15s ease;
|
||||||
|
|
||||||
|
&:hover {
|
||||||
|
opacity: 1;
|
||||||
|
background: rgba(0, 0, 0, 0.05);
|
||||||
|
}
|
||||||
|
|
||||||
|
.material-symbols-outlined {
|
||||||
|
font-size: 15px;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Dock Row: Flex container holding Main Card + Standalone Stop Button */
|
||||||
|
.floating-dock-row {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
gap: 10px;
|
||||||
|
width: 100%;
|
||||||
|
pointer-events: none;
|
||||||
|
transition: gap 0.32s cubic-bezier(0.16, 1, 0.3, 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Standalone Floating Stop Button (Separated from Capsule) */
|
||||||
|
.standalone-stop-btn {
|
||||||
|
pointer-events: auto;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
width: 34px;
|
||||||
|
height: 34px;
|
||||||
|
min-width: 34px;
|
||||||
|
border-radius: 50%;
|
||||||
|
border: 1px solid rgba(255, 255, 255, 0.85);
|
||||||
|
background: rgba(239, 68, 68, 0.9);
|
||||||
|
backdrop-filter: blur(24px) saturate(190%) brightness(105%);
|
||||||
|
-webkit-backdrop-filter: blur(24px) saturate(190%) brightness(105%);
|
||||||
|
color: #ffffff;
|
||||||
|
box-shadow: 0 6px 18px -2px rgba(239, 68, 68, 0.4), inset 0 1px 1.5px rgba(255, 255, 255, 0.7);
|
||||||
|
cursor: pointer;
|
||||||
|
outline: none;
|
||||||
|
flex-shrink: 0;
|
||||||
|
transition: opacity 0.28s cubic-bezier(0.16, 1, 0.3, 1),
|
||||||
|
transform 0.28s cubic-bezier(0.16, 1, 0.3, 1),
|
||||||
|
width 0.28s cubic-bezier(0.16, 1, 0.3, 1),
|
||||||
|
min-width 0.28s cubic-bezier(0.16, 1, 0.3, 1),
|
||||||
|
margin 0.28s cubic-bezier(0.16, 1, 0.3, 1),
|
||||||
|
background 0.2s ease,
|
||||||
|
box-shadow 0.2s ease;
|
||||||
|
animation: popIn 0.25s cubic-bezier(0.16, 1, 0.3, 1);
|
||||||
|
|
||||||
|
.material-symbols-outlined {
|
||||||
|
font-size: 17px;
|
||||||
|
line-height: 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
&:hover:not(:disabled) {
|
||||||
|
background: #dc2626;
|
||||||
|
transform: scale(1.1);
|
||||||
|
box-shadow: 0 8px 22px rgba(239, 68, 68, 0.55), inset 0 1px 1.5px rgba(255, 255, 255, 0.9);
|
||||||
|
}
|
||||||
|
|
||||||
|
&:active:not(:disabled) {
|
||||||
|
transform: scale(0.93);
|
||||||
|
}
|
||||||
|
|
||||||
|
&:disabled {
|
||||||
|
opacity: 0.6;
|
||||||
|
cursor: not-allowed;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Floating Dock Card Container */
|
||||||
|
.floating-dock-card {
|
||||||
|
pointer-events: auto;
|
||||||
|
position: relative;
|
||||||
|
width: 100%;
|
||||||
|
border-radius: 18px;
|
||||||
|
box-sizing: border-box;
|
||||||
|
overflow: hidden;
|
||||||
|
transition: max-width 0.32s cubic-bezier(0.16, 1, 0.3, 1),
|
||||||
|
width 0.32s cubic-bezier(0.16, 1, 0.3, 1),
|
||||||
|
border-radius 0.28s cubic-bezier(0.16, 1, 0.3, 1),
|
||||||
|
box-shadow 0.28s ease,
|
||||||
|
background 0.28s ease,
|
||||||
|
transform 0.28s cubic-bezier(0.16, 1, 0.3, 1),
|
||||||
|
border-color 0.28s ease;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========================================================================= */
|
||||||
|
/* AMBIENT LIQUID WAVE COLOR BLOCKS (Soft Diffused Aurora Glow Waves) */
|
||||||
|
/* ========================================================================= */
|
||||||
|
.dock-wave-container {
|
||||||
|
position: absolute;
|
||||||
|
bottom: 0;
|
||||||
|
left: 0;
|
||||||
|
right: 0;
|
||||||
|
height: 100%;
|
||||||
|
pointer-events: none;
|
||||||
|
border-radius: inherit;
|
||||||
|
overflow: hidden;
|
||||||
|
z-index: 0;
|
||||||
|
opacity: 0.78;
|
||||||
|
transition: opacity 0.35s ease, filter 0.35s ease;
|
||||||
|
|
||||||
|
/* Flash Mode Color Stops (Soft Electric Blue, Violet, Amber, Cyan) */
|
||||||
|
.wave-stop-1 { stop-color: #3b82f6; stop-opacity: 0.65; }
|
||||||
|
.wave-stop-2 { stop-color: #8b5cf6; stop-opacity: 0.7; }
|
||||||
|
.wave-stop-3 { stop-color: #f59e0b; stop-opacity: 0.65; }
|
||||||
|
.wave-stop-4 { stop-color: #06b6d4; stop-opacity: 0.6; }
|
||||||
|
|
||||||
|
/* Pro Mode Color Stops (Soft Cosmic Nebula / Amethyst / Fuchsia) */
|
||||||
|
&.is-pro {
|
||||||
|
.wave-stop-1 { stop-color: #7c3aed; stop-opacity: 0.7; }
|
||||||
|
.wave-stop-2 { stop-color: #ec4899; stop-opacity: 0.75; }
|
||||||
|
.wave-stop-3 { stop-color: #6366f1; stop-opacity: 0.65; }
|
||||||
|
.wave-stop-4 { stop-color: #a855f7; stop-opacity: 0.7; }
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Deep Ambient Aurora Glow Diffusion */
|
||||||
|
.wave-glow-ambient {
|
||||||
|
position: absolute;
|
||||||
|
bottom: -8px;
|
||||||
|
left: 5%;
|
||||||
|
right: 5%;
|
||||||
|
height: 30px;
|
||||||
|
border-radius: 50%;
|
||||||
|
background: radial-gradient(ellipse at center, rgba(99, 102, 241, 0.4) 0%, rgba(245, 158, 11, 0.2) 50%, transparent 80%);
|
||||||
|
filter: blur(16px);
|
||||||
|
opacity: 0.7;
|
||||||
|
animation: glowBreathe 5s ease-in-out infinite alternate;
|
||||||
|
}
|
||||||
|
|
||||||
|
&.is-pro .wave-glow-ambient {
|
||||||
|
background: radial-gradient(ellipse at center, rgba(168, 85, 247, 0.45) 0%, rgba(236, 72, 153, 0.25) 50%, transparent 80%);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Wave Tracks Wrapper with Soft Gaussian Blur */
|
||||||
|
.wave-tracks-wrapper {
|
||||||
|
position: absolute;
|
||||||
|
bottom: 0;
|
||||||
|
left: 0;
|
||||||
|
width: 100%;
|
||||||
|
height: 22px;
|
||||||
|
overflow: hidden;
|
||||||
|
filter: blur(7px); /* Soften sharp vector wave edges into silky liquid light */
|
||||||
|
mask-image: linear-gradient(to top, rgba(0,0,0,0.9) 0%, rgba(0,0,0,0.5) 60%, transparent 100%);
|
||||||
|
-webkit-mask-image: linear-gradient(to top, rgba(0,0,0,0.9) 0%, rgba(0,0,0,0.5) 60%, transparent 100%);
|
||||||
|
|
||||||
|
.liquid-wave {
|
||||||
|
position: absolute;
|
||||||
|
bottom: 0;
|
||||||
|
left: 0;
|
||||||
|
width: 200%;
|
||||||
|
height: 100%;
|
||||||
|
will-change: transform;
|
||||||
|
filter: blur(3px); /* Secondary soft layer blur */
|
||||||
|
|
||||||
|
&.wave-back {
|
||||||
|
opacity: 0.45;
|
||||||
|
animation: waveSlide 12s ease-in-out infinite alternate;
|
||||||
|
}
|
||||||
|
|
||||||
|
&.wave-mid {
|
||||||
|
opacity: 0.6;
|
||||||
|
animation: waveSlideReverse 9s ease-in-out infinite alternate;
|
||||||
|
}
|
||||||
|
|
||||||
|
&.wave-front {
|
||||||
|
opacity: 0.75;
|
||||||
|
animation: waveSlide 6.5s ease-in-out infinite alternate;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ======================================================================= */
|
||||||
|
/* 1. DORMANT / RESTING CAPSULE STATE (Subtle Frosted Peek) */
|
||||||
|
/* ======================================================================= */
|
||||||
|
&.is-dormant {
|
||||||
|
.floating-dock-card {
|
||||||
|
max-width: 160px;
|
||||||
|
width: 160px;
|
||||||
|
margin: 0;
|
||||||
|
background: rgba(255, 255, 255, 0.65);
|
||||||
|
backdrop-filter: blur(26px) saturate(190%) brightness(104%);
|
||||||
|
-webkit-backdrop-filter: blur(26px) saturate(190%) brightness(104%);
|
||||||
|
border: 1px solid rgba(255, 255, 255, 0.85);
|
||||||
|
border-radius: 17px;
|
||||||
|
box-shadow: 0 6px 18px -2px rgba(15, 23, 42, 0.07), inset 0 1px 1.5px rgba(255, 255, 255, 0.95), inset 0 -1px 1px rgba(255, 255, 255, 0.3);
|
||||||
|
cursor: pointer;
|
||||||
|
opacity: 0.92;
|
||||||
|
transform: translateY(2px);
|
||||||
|
|
||||||
|
.dock-wave-container {
|
||||||
|
opacity: 0.78;
|
||||||
|
.wave-tracks-wrapper {
|
||||||
|
height: 12px;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
&:hover {
|
||||||
|
opacity: 1;
|
||||||
|
transform: translateY(0);
|
||||||
|
background: rgba(255, 255, 255, 0.82);
|
||||||
|
border-color: rgba(255, 255, 255, 1);
|
||||||
|
box-shadow: 0 8px 24px -2px rgba(15, 23, 42, 0.12), inset 0 1px 1.5px rgba(255, 255, 255, 1);
|
||||||
|
|
||||||
|
.dock-wave-container {
|
||||||
|
opacity: 0.95;
|
||||||
|
filter: drop-shadow(0 0 6px rgba(99, 102, 241, 0.35));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
.capsule-peek-view {
|
||||||
|
display: flex;
|
||||||
|
opacity: 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
.expanded-card-content {
|
||||||
|
display: none;
|
||||||
|
opacity: 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Capsule Peek View Layout */
|
||||||
|
.capsule-peek-view {
|
||||||
|
position: relative;
|
||||||
|
z-index: 1;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
padding: 6px 14px;
|
||||||
|
min-height: 34px;
|
||||||
|
box-sizing: border-box;
|
||||||
|
user-select: none;
|
||||||
|
|
||||||
|
.capsule-placeholder {
|
||||||
|
color: #475569;
|
||||||
|
font-size: 12px;
|
||||||
|
font-weight: 500;
|
||||||
|
letter-spacing: 0.1px;
|
||||||
|
white-space: nowrap;
|
||||||
|
text-align: center;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ======================================================================= */
|
||||||
|
/* 2. EXPANDED / ACTIVE STATE (Rich Frosted Command Box) */
|
||||||
|
/* ======================================================================= */
|
||||||
|
&.is-expanded {
|
||||||
|
.floating-dock-row {
|
||||||
|
gap: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.standalone-stop-btn {
|
||||||
|
opacity: 0;
|
||||||
|
width: 0;
|
||||||
|
min-width: 0;
|
||||||
|
margin: 0;
|
||||||
|
padding: 0;
|
||||||
|
border-width: 0;
|
||||||
|
transform: scale(0.5);
|
||||||
|
pointer-events: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.floating-dock-card {
|
||||||
|
max-width: 100%;
|
||||||
|
width: 100%;
|
||||||
|
background: rgba(255, 255, 255, 0.68);
|
||||||
|
backdrop-filter: blur(32px) saturate(210%) brightness(105%);
|
||||||
|
-webkit-backdrop-filter: blur(32px) saturate(210%) brightness(105%);
|
||||||
|
border: 1px solid rgba(255, 255, 255, 0.82);
|
||||||
|
box-shadow: 0 16px 44px -8px rgba(15, 23, 42, 0.12), 0 2px 8px rgba(0, 0, 0, 0.02), inset 0 1.5px 2px rgba(255, 255, 255, 0.95), inset 0 -1px 2px rgba(255, 255, 255, 0.3);
|
||||||
|
opacity: 1;
|
||||||
|
transform: translateY(0);
|
||||||
|
|
||||||
|
.dock-wave-container {
|
||||||
|
opacity: 0.88;
|
||||||
|
.wave-tracks-wrapper {
|
||||||
|
height: 22px;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
.capsule-peek-view {
|
||||||
|
display: none;
|
||||||
|
opacity: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.expanded-card-content {
|
||||||
|
display: block;
|
||||||
|
opacity: 1;
|
||||||
|
animation: cardFadeIn 0.25s cubic-bezier(0.16, 1, 0.3, 1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Focused State Glow */
|
||||||
|
&.is-focused .floating-dock-card {
|
||||||
|
background: rgba(255, 255, 255, 0.78);
|
||||||
|
backdrop-filter: blur(36px) saturate(220%) brightness(106%);
|
||||||
|
-webkit-backdrop-filter: blur(36px) saturate(220%) brightness(106%);
|
||||||
|
border-color: rgba(59, 130, 246, 0.55);
|
||||||
|
box-shadow: 0 20px 48px -6px rgba(37, 99, 235, 0.16), 0 0 0 2.5px rgba(59, 130, 246, 0.18), inset 0 1.5px 2px rgba(255, 255, 255, 1);
|
||||||
|
|
||||||
|
.dock-wave-container {
|
||||||
|
opacity: 1;
|
||||||
|
filter: drop-shadow(0 0 8px rgba(59, 130, 246, 0.35));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Expanded Content Rows */
|
||||||
|
.expanded-card-content {
|
||||||
|
position: relative;
|
||||||
|
z-index: 1;
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
width: 100%;
|
||||||
|
|
||||||
|
.card-main-input-row {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 10px;
|
||||||
|
padding: 8px 10px 3px 14px;
|
||||||
|
min-height: 48px;
|
||||||
|
box-sizing: border-box;
|
||||||
|
|
||||||
|
.leading-status-icon {
|
||||||
|
width: 14px;
|
||||||
|
height: 14px;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
flex-shrink: 0;
|
||||||
|
|
||||||
|
.pulse-indicator-dot {
|
||||||
|
width: 8px;
|
||||||
|
height: 8px;
|
||||||
|
border-radius: 50%;
|
||||||
|
background: #ef4444;
|
||||||
|
box-shadow: 0 0 0 0 rgba(239, 68, 68, 0.6);
|
||||||
|
animation: pulseDot 1.4s infinite;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
.input-content-area {
|
||||||
|
flex: 1;
|
||||||
|
min-width: 0;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
|
||||||
|
.dock-textarea {
|
||||||
|
width: 100%;
|
||||||
|
border: none;
|
||||||
|
outline: none;
|
||||||
|
background: transparent;
|
||||||
|
resize: none;
|
||||||
|
font-family: inherit;
|
||||||
|
font-size: 13.5px;
|
||||||
|
font-weight: 450;
|
||||||
|
line-height: 1.5;
|
||||||
|
color: #0f172a;
|
||||||
|
padding: 3px 0;
|
||||||
|
max-height: 120px;
|
||||||
|
box-sizing: border-box;
|
||||||
|
display: block;
|
||||||
|
|
||||||
|
&::placeholder {
|
||||||
|
color: #64748b;
|
||||||
|
font-size: 12.5px;
|
||||||
|
font-weight: 400;
|
||||||
|
transition: color 0.15s ease;
|
||||||
|
}
|
||||||
|
|
||||||
|
&:disabled {
|
||||||
|
opacity: 0.6;
|
||||||
|
cursor: not-allowed;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
.dock-circle-btn {
|
||||||
|
border-radius: 50%;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
border: none;
|
||||||
|
cursor: pointer;
|
||||||
|
outline: none;
|
||||||
|
flex-shrink: 0;
|
||||||
|
transition: all 0.18s cubic-bezier(0.16, 1, 0.3, 1);
|
||||||
|
|
||||||
|
.material-symbols-outlined {
|
||||||
|
font-size: 18px;
|
||||||
|
line-height: 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
.spinning-icon {
|
||||||
|
font-size: 17px;
|
||||||
|
animation: spin 1s linear infinite;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Clear Button */
|
||||||
|
&.btn-clear {
|
||||||
|
width: 26px;
|
||||||
|
height: 26px;
|
||||||
|
background: rgba(255, 255, 255, 0.45);
|
||||||
|
backdrop-filter: blur(8px);
|
||||||
|
border: 1px solid rgba(255, 255, 255, 0.6);
|
||||||
|
color: #64748b;
|
||||||
|
|
||||||
|
&:hover {
|
||||||
|
background: rgba(255, 255, 255, 0.75);
|
||||||
|
color: #0f172a;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Stop Button (Frosted Red) */
|
||||||
|
&.btn-stop {
|
||||||
|
width: 36px;
|
||||||
|
height: 36px;
|
||||||
|
background: rgba(239, 68, 68, 0.9);
|
||||||
|
backdrop-filter: blur(8px);
|
||||||
|
color: #ffffff;
|
||||||
|
border: 1px solid rgba(255, 255, 255, 0.35);
|
||||||
|
box-shadow: 0 4px 12px rgba(239, 68, 68, 0.3);
|
||||||
|
animation: popIn 0.2s cubic-bezier(0.16, 1, 0.3, 1);
|
||||||
|
|
||||||
|
&:hover:not(:disabled) {
|
||||||
|
background: #dc2626;
|
||||||
|
transform: scale(1.05);
|
||||||
|
box-shadow: 0 6px 16px rgba(239, 68, 68, 0.4);
|
||||||
|
}
|
||||||
|
|
||||||
|
&:active:not(:disabled) {
|
||||||
|
transform: scale(0.95);
|
||||||
|
}
|
||||||
|
|
||||||
|
&:disabled {
|
||||||
|
opacity: 0.6;
|
||||||
|
cursor: not-allowed;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Send Button */
|
||||||
|
&.btn-send {
|
||||||
|
width: 36px;
|
||||||
|
height: 36px;
|
||||||
|
background: rgba(15, 23, 42, 0.88);
|
||||||
|
backdrop-filter: blur(12px);
|
||||||
|
border: 1px solid rgba(255, 255, 255, 0.25);
|
||||||
|
color: #ffffff;
|
||||||
|
box-shadow: 0 4px 14px rgba(15, 23, 42, 0.2), inset 0 1px 1px rgba(255, 255, 255, 0.25);
|
||||||
|
|
||||||
|
&:disabled {
|
||||||
|
background: rgba(226, 232, 240, 0.6);
|
||||||
|
color: #94a3b8;
|
||||||
|
cursor: not-allowed;
|
||||||
|
box-shadow: none;
|
||||||
|
border-color: rgba(255, 255, 255, 0.5);
|
||||||
|
transform: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
&.has-content:not(:disabled) {
|
||||||
|
background: linear-gradient(135deg, rgba(30, 41, 59, 0.95), rgba(15, 23, 42, 0.98));
|
||||||
|
box-shadow: 0 4px 14px rgba(15, 23, 42, 0.25), inset 0 1px 1px rgba(255, 255, 255, 0.3);
|
||||||
|
|
||||||
|
&:hover {
|
||||||
|
background: linear-gradient(135deg, #2563eb, #1d4ed8);
|
||||||
|
transform: scale(1.05);
|
||||||
|
box-shadow: 0 6px 18px rgba(37, 99, 235, 0.35);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
&:active:not(:disabled) {
|
||||||
|
transform: scale(0.95);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Model Profile Selector & Status Footer Row */
|
||||||
|
.card-footer-row {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: space-between;
|
||||||
|
padding: 3px 14px 8px 14px;
|
||||||
|
box-sizing: border-box;
|
||||||
|
|
||||||
|
.footer-left-group {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 8px;
|
||||||
|
|
||||||
|
/* Segmented Model Toggle with Frosted Glass */
|
||||||
|
.model-profile-toggle {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
background: rgba(255, 255, 255, 0.45);
|
||||||
|
backdrop-filter: blur(14px);
|
||||||
|
-webkit-backdrop-filter: blur(14px);
|
||||||
|
border: 1px solid rgba(255, 255, 255, 0.65);
|
||||||
|
box-shadow: inset 0 1px 1px rgba(255, 255, 255, 0.8), 0 1px 3px rgba(0, 0, 0, 0.03);
|
||||||
|
border-radius: 8px;
|
||||||
|
padding: 2px;
|
||||||
|
gap: 2px;
|
||||||
|
|
||||||
|
.profile-toggle-pill {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
gap: 3.5px;
|
||||||
|
width: 62px;
|
||||||
|
min-width: 62px;
|
||||||
|
height: 22px;
|
||||||
|
border-radius: 6px;
|
||||||
|
border: none;
|
||||||
|
background: transparent;
|
||||||
|
color: #64748b;
|
||||||
|
font-family: inherit;
|
||||||
|
font-size: 11px;
|
||||||
|
font-weight: 550;
|
||||||
|
cursor: pointer;
|
||||||
|
transition: all 0.16s cubic-bezier(0.16, 1, 0.3, 1);
|
||||||
|
outline: none;
|
||||||
|
user-select: none;
|
||||||
|
box-sizing: border-box;
|
||||||
|
|
||||||
|
.pill-icon {
|
||||||
|
font-size: 13px;
|
||||||
|
line-height: 1;
|
||||||
|
flex-shrink: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.pill-label {
|
||||||
|
line-height: 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
&:hover:not(.active) {
|
||||||
|
color: #1e293b;
|
||||||
|
background: rgba(255, 255, 255, 0.65);
|
||||||
|
}
|
||||||
|
|
||||||
|
&.active {
|
||||||
|
background: rgba(255, 255, 255, 0.92);
|
||||||
|
backdrop-filter: blur(8px);
|
||||||
|
color: #0f172a;
|
||||||
|
font-weight: 700;
|
||||||
|
box-shadow: 0 1.5px 4px rgba(0, 0, 0, 0.08), inset 0 1px 1.5px rgba(255, 255, 255, 1);
|
||||||
|
|
||||||
|
.icon-flash {
|
||||||
|
color: #d97706;
|
||||||
|
}
|
||||||
|
|
||||||
|
.icon-pro {
|
||||||
|
color: #7c3aed;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
.footer-right-status {
|
||||||
|
.running-status-pill {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 4.5px;
|
||||||
|
font-size: 10.5px;
|
||||||
|
font-weight: 600;
|
||||||
|
color: #dc2626;
|
||||||
|
background: rgba(254, 226, 226, 0.7);
|
||||||
|
padding: 2px 7px;
|
||||||
|
border-radius: 10px;
|
||||||
|
|
||||||
|
.running-pulse-dot {
|
||||||
|
width: 5.5px;
|
||||||
|
height: 5.5px;
|
||||||
|
border-radius: 50%;
|
||||||
|
background: #ef4444;
|
||||||
|
animation: pulseDot 1.2s infinite;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* When viewport shrinks to approximately 2/3 of standard desktop width (<= 1150px), hide right panel and leave only agent-stream */
|
||||||
|
@media (max-width: 1150px) {
|
||||||
|
.right-panel, .resizer {
|
||||||
|
display: none !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.left-panel {
|
||||||
|
border-right: none !important;
|
||||||
|
width: 100% !important;
|
||||||
|
flex: 1 1 100% !important;
|
||||||
|
min-width: 0 !important;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Animations */
|
||||||
|
@keyframes popIn {
|
||||||
|
0% {
|
||||||
|
opacity: 0;
|
||||||
|
transform: scale(0.8);
|
||||||
|
}
|
||||||
|
100% {
|
||||||
|
opacity: 1;
|
||||||
|
transform: scale(1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@keyframes spin {
|
||||||
|
from { transform: rotate(0deg); }
|
||||||
|
to { transform: rotate(360deg); }
|
||||||
|
}
|
||||||
|
|
||||||
|
@keyframes slideUpFade {
|
||||||
|
from {
|
||||||
|
opacity: 0;
|
||||||
|
transform: translateY(8px);
|
||||||
|
}
|
||||||
|
to {
|
||||||
|
opacity: 1;
|
||||||
|
transform: translateY(0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@keyframes cardFadeIn {
|
||||||
|
from {
|
||||||
|
opacity: 0;
|
||||||
|
transform: translateY(4px);
|
||||||
|
}
|
||||||
|
to {
|
||||||
|
opacity: 1;
|
||||||
|
transform: translateY(0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@keyframes fadeIn {
|
||||||
|
from { opacity: 0; }
|
||||||
|
to { opacity: 1; }
|
||||||
|
}
|
||||||
|
|
||||||
|
@keyframes pulseDot {
|
||||||
|
0% {
|
||||||
|
transform: scale(0.95);
|
||||||
|
box-shadow: 0 0 0 0 rgba(239, 68, 68, 0.7);
|
||||||
|
}
|
||||||
|
70% {
|
||||||
|
transform: scale(1);
|
||||||
|
box-shadow: 0 0 0 6px rgba(239, 68, 68, 0);
|
||||||
|
}
|
||||||
|
100% {
|
||||||
|
transform: scale(0.95);
|
||||||
|
box-shadow: 0 0 0 0 rgba(239, 68, 68, 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Fluid Wave Horizontal Slide (Seamless loop) */
|
||||||
|
@keyframes waveSlide {
|
||||||
|
0% {
|
||||||
|
transform: translate3d(0, 0, 0);
|
||||||
|
}
|
||||||
|
100% {
|
||||||
|
transform: translate3d(-50%, 0, 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@keyframes waveSlideReverse {
|
||||||
|
0% {
|
||||||
|
transform: translate3d(-50%, 0, 0);
|
||||||
|
}
|
||||||
|
100% {
|
||||||
|
transform: translate3d(0, 0, 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Ambient Radial Glow Breathing */
|
||||||
|
@keyframes glowBreathe {
|
||||||
|
0% {
|
||||||
|
opacity: 0.55;
|
||||||
|
transform: scaleX(0.92);
|
||||||
|
}
|
||||||
|
100% {
|
||||||
|
opacity: 0.95;
|
||||||
|
transform: scaleX(1.08);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,284 @@
|
|||||||
|
/**
|
||||||
|
* Copyright 2026 Google LLC
|
||||||
|
*
|
||||||
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
* you may not use this file except in compliance with the License.
|
||||||
|
* You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { Component, HostListener, inject, computed, signal, ViewChild, ElementRef, OnInit } from '@angular/core';
|
||||||
|
import { CommonModule } from '@angular/common';
|
||||||
|
import { FormsModule } from '@angular/forms';
|
||||||
|
import { HttpClient } from '@angular/common/http';
|
||||||
|
import { AgentStreamComponent } from '../../components/agent-stream/agent-stream.component';
|
||||||
|
import { ChatInterfaceComponent } from '../../components/chat-interface/chat-interface.component';
|
||||||
|
import { FloatingVideoPlayerComponent } from '../../components/floating-video-player/floating-video-player.component';
|
||||||
|
import { AgentService } from '../../services/agent.service';
|
||||||
|
|
||||||
|
@Component({
|
||||||
|
selector: 'app-workspace',
|
||||||
|
standalone: true,
|
||||||
|
imports: [
|
||||||
|
CommonModule,
|
||||||
|
FormsModule,
|
||||||
|
AgentStreamComponent,
|
||||||
|
ChatInterfaceComponent,
|
||||||
|
FloatingVideoPlayerComponent
|
||||||
|
],
|
||||||
|
templateUrl: './workspace.component.html',
|
||||||
|
styleUrl: './workspace.component.scss'
|
||||||
|
})
|
||||||
|
export class WorkspaceComponent implements OnInit {
|
||||||
|
public agentService = inject(AgentService);
|
||||||
|
private http = inject(HttpClient);
|
||||||
|
|
||||||
|
// Default right panel width to 1/3 of the screen (or 450px as fallback)
|
||||||
|
public rightPanelWidth = typeof window !== 'undefined' ? Math.round(window.innerWidth / 3) : 450;
|
||||||
|
public isDragging = false;
|
||||||
|
|
||||||
|
// Floating Command Bar State
|
||||||
|
public taskInput: string = '';
|
||||||
|
public isSubmitting: boolean = false;
|
||||||
|
public errorMessage: string | null = null;
|
||||||
|
public selectedProfile = signal<'flash' | 'pro'>('flash');
|
||||||
|
|
||||||
|
// Expand States (Signals for 0-latency reactivity)
|
||||||
|
public isHoveringCard = signal<boolean>(false);
|
||||||
|
public isInputFocused = signal<boolean>(false);
|
||||||
|
|
||||||
|
@ViewChild('dockInput') public dockInputRef?: ElementRef<HTMLTextAreaElement>;
|
||||||
|
|
||||||
|
ngOnInit(): void {
|
||||||
|
if (typeof localStorage !== 'undefined') {
|
||||||
|
const saved = localStorage.getItem('artemis_selected_profile');
|
||||||
|
if (saved === 'flash' || saved === 'pro') {
|
||||||
|
this.selectedProfile.set(saved);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Set model profile ('flash' vs 'pro')
|
||||||
|
*/
|
||||||
|
public setProfile(profile: 'flash' | 'pro', event?: MouseEvent): void {
|
||||||
|
if (event) {
|
||||||
|
event.stopPropagation();
|
||||||
|
}
|
||||||
|
this.selectedProfile.set(profile);
|
||||||
|
if (typeof localStorage !== 'undefined') {
|
||||||
|
localStorage.setItem('artemis_selected_profile', profile);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Computed boolean whether a task is currently executing or active
|
||||||
|
*/
|
||||||
|
public isTaskRunning = computed(() => {
|
||||||
|
return this.agentService.isRunningTask();
|
||||||
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Computed boolean whether the command bar should be in its expanded state:
|
||||||
|
* - Mouse is hovering directly on the command card
|
||||||
|
* - Input textarea is focused
|
||||||
|
* - User has entered task text (drafting)
|
||||||
|
*/
|
||||||
|
public isBarExpanded = computed(() => {
|
||||||
|
return (
|
||||||
|
this.isHoveringCard() ||
|
||||||
|
this.isInputFocused() ||
|
||||||
|
this.taskInput.trim().length > 0
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Focus input handler
|
||||||
|
*/
|
||||||
|
public onInputFocus(): void {
|
||||||
|
this.isInputFocused.set(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Blur input handler
|
||||||
|
*/
|
||||||
|
public onInputBlur(): void {
|
||||||
|
this.isInputFocused.set(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Click on the resting capsule or dock card to focus textarea
|
||||||
|
*/
|
||||||
|
public onCardClick(event: MouseEvent): void {
|
||||||
|
const target = event.target as HTMLElement;
|
||||||
|
// Don't steal focus if clicking action buttons or textarea directly
|
||||||
|
if (target.closest('button') || target.tagName.toLowerCase() === 'textarea') {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.focusInput();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Focus textarea programmatically
|
||||||
|
*/
|
||||||
|
public focusInput(): void {
|
||||||
|
setTimeout(() => {
|
||||||
|
this.dockInputRef?.nativeElement?.focus();
|
||||||
|
}, 30);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Clear typed task input
|
||||||
|
*/
|
||||||
|
public clearInput(event?: MouseEvent): void {
|
||||||
|
if (event) {
|
||||||
|
event.stopPropagation();
|
||||||
|
}
|
||||||
|
this.taskInput = '';
|
||||||
|
if (this.dockInputRef?.nativeElement) {
|
||||||
|
this.dockInputRef.nativeElement.style.height = 'auto';
|
||||||
|
this.dockInputRef.nativeElement.focus();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Auto-resize textarea as user types multi-line tasks
|
||||||
|
*/
|
||||||
|
public onTextareaInput(event: Event): void {
|
||||||
|
const textarea = event.target as HTMLTextAreaElement;
|
||||||
|
if (!textarea) return;
|
||||||
|
textarea.style.height = 'auto';
|
||||||
|
const newHeight = Math.min(Math.max(textarea.scrollHeight, 24), 120);
|
||||||
|
textarea.style.height = `${newHeight}px`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Global keyboard shortcut (⌘K or Ctrl+K) to focus input bar from anywhere
|
||||||
|
*/
|
||||||
|
@HostListener('window:keydown', ['$event'])
|
||||||
|
public onGlobalKeyDown(event: KeyboardEvent): void {
|
||||||
|
if ((event.metaKey || event.ctrlKey) && event.key.toLowerCase() === 'k') {
|
||||||
|
event.preventDefault();
|
||||||
|
this.focusInput();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Handle enter key in floating dock textarea
|
||||||
|
*/
|
||||||
|
public onKeyDown(event: KeyboardEvent): void {
|
||||||
|
if (event.key === 'Enter' && !event.shiftKey) {
|
||||||
|
event.preventDefault();
|
||||||
|
this.submitTask();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Submit a new task instruction
|
||||||
|
*/
|
||||||
|
public submitTask(): void {
|
||||||
|
const goal = this.taskInput.trim();
|
||||||
|
if (!goal || this.isSubmitting) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
this.isSubmitting = true;
|
||||||
|
this.errorMessage = null;
|
||||||
|
|
||||||
|
if (this.dockInputRef?.nativeElement) {
|
||||||
|
this.dockInputRef.nativeElement.blur();
|
||||||
|
}
|
||||||
|
this.isInputFocused.set(false);
|
||||||
|
|
||||||
|
this.agentService.clearUserPinnedSession();
|
||||||
|
this.http.post<any>('/api/run', { goal, profile: this.selectedProfile() }).subscribe({
|
||||||
|
next: (res) => {
|
||||||
|
this.taskInput = '';
|
||||||
|
if (this.dockInputRef?.nativeElement) {
|
||||||
|
this.dockInputRef.nativeElement.style.height = 'auto';
|
||||||
|
}
|
||||||
|
this.isSubmitting = false;
|
||||||
|
if (res && res.tasks && res.tasks.length > 0) {
|
||||||
|
const firstTask = res.tasks[0];
|
||||||
|
if (firstTask && firstTask.session_id) {
|
||||||
|
const isCurrentlyRunning =
|
||||||
|
this.agentService.agentStatus() === 'running' ||
|
||||||
|
this.agentService.sessions().some((s) => s.status === 'running');
|
||||||
|
if (!isCurrentlyRunning) {
|
||||||
|
this.agentService.selectSession(firstTask.session_id, false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
this.agentService.fetchStatus();
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
console.error('Failed to submit task:', err);
|
||||||
|
this.isSubmitting = false;
|
||||||
|
this.errorMessage = err.error?.detail || 'The runner is busy. Please wait for current task to finish.';
|
||||||
|
setTimeout(() => {
|
||||||
|
this.errorMessage = null;
|
||||||
|
}, 5000);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Stop currently running task
|
||||||
|
*/
|
||||||
|
public stopTask(event?: MouseEvent): void {
|
||||||
|
if (event) {
|
||||||
|
event.stopPropagation();
|
||||||
|
}
|
||||||
|
if (!this.isTaskRunning()) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.isSubmitting = true;
|
||||||
|
this.errorMessage = null;
|
||||||
|
this.agentService.stopTask(false);
|
||||||
|
setTimeout(() => {
|
||||||
|
this.isSubmitting = false;
|
||||||
|
}, 400);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Handle mouse down on resizer bar to start dragging
|
||||||
|
*/
|
||||||
|
public onDragStart(event: MouseEvent): void {
|
||||||
|
this.isDragging = true;
|
||||||
|
event.preventDefault();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Listen to global mousemove events to adjust the width dynamically
|
||||||
|
*/
|
||||||
|
@HostListener('document:mousemove', ['$event'])
|
||||||
|
public onMouseMove(event: MouseEvent): void {
|
||||||
|
if (!this.isDragging) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const newWidth = window.innerWidth - event.clientX;
|
||||||
|
const minWidth = 250;
|
||||||
|
const maxWidth = window.innerWidth - 300;
|
||||||
|
|
||||||
|
// Apply boundary limits to prevent panels from shrinking too much
|
||||||
|
if (newWidth >= minWidth && newWidth <= maxWidth) {
|
||||||
|
this.rightPanelWidth = newWidth;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Listen to global mouseup events to stop resizing
|
||||||
|
*/
|
||||||
|
@HostListener('document:mouseup')
|
||||||
|
public onMouseUp(): void {
|
||||||
|
this.isDragging = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
+231
-25
@@ -16,6 +16,7 @@
|
|||||||
|
|
||||||
import { Injectable, signal, inject, computed } from '@angular/core';
|
import { Injectable, signal, inject, computed } from '@angular/core';
|
||||||
import { HttpClient } from '@angular/common/http';
|
import { HttpClient } from '@angular/common/http';
|
||||||
|
import { Observable } from 'rxjs';
|
||||||
|
|
||||||
import { Session, ModelInfo, TaskQueueItem, AgentStatusResponse } from '../core/models/session.model';
|
import { Session, ModelInfo, TaskQueueItem, AgentStatusResponse } from '../core/models/session.model';
|
||||||
import { StepItemData } from '../core/models/stream.model';
|
import { StepItemData } from '../core/models/stream.model';
|
||||||
@@ -38,20 +39,47 @@ export class AgentService {
|
|||||||
const goal = this.runningGoal();
|
const goal = this.runningGoal();
|
||||||
const runId = this.runningSessionId();
|
const runId = this.runningSessionId();
|
||||||
|
|
||||||
const hasRunningInRaw = raw.some(s => s.status === 'running' || (runId && s.session_id === runId));
|
const sessionMap = new Map<string, Session>();
|
||||||
|
|
||||||
if (status === 'running' && goal && runId && !hasRunningInRaw) {
|
// 1. First add raw sessions from DB
|
||||||
const activeRunningSession: Session = {
|
raw.forEach((s) => {
|
||||||
|
const isCurrentRunning = status === 'running' && runId === s.session_id;
|
||||||
|
const isPending = this.pendingQueue().some(p => p.session_id === s.session_id) && !isCurrentRunning;
|
||||||
|
let sStatus = isCurrentRunning ? 'running' : (isPending ? 'pending' : s.status);
|
||||||
|
if (sStatus === 'running' && !isCurrentRunning) {
|
||||||
|
const activeRun = raw.find(r => r.session_id === runId);
|
||||||
|
sStatus = (activeRun && (s.start_time || 0) > (activeRun.start_time || 0)) ? 'pending' : 'completed';
|
||||||
|
}
|
||||||
|
sessionMap.set(s.session_id, {
|
||||||
|
...s,
|
||||||
|
status: sStatus,
|
||||||
|
model_info: isCurrentRunning && this.activeModel() ? this.activeModel()! : s.model_info
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// 2. Add pending queue sessions if not yet in raw sessions
|
||||||
|
pending.forEach((p) => {
|
||||||
|
if (!sessionMap.has(p.session_id)) {
|
||||||
|
const isCurrentRunning = status === 'running' && runId === p.session_id;
|
||||||
|
sessionMap.set(p.session_id, {
|
||||||
|
...p,
|
||||||
|
status: isCurrentRunning ? 'running' : 'pending'
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// 3. Ensure currently active running session is present
|
||||||
|
if (status === 'running' && runId && !sessionMap.has(runId)) {
|
||||||
|
sessionMap.set(runId, {
|
||||||
session_id: runId,
|
session_id: runId,
|
||||||
initial_goal: goal,
|
initial_goal: goal || '',
|
||||||
start_time: Date.now() / 1000,
|
start_time: Date.now() / 1000,
|
||||||
status: 'running',
|
status: 'running',
|
||||||
model_info: this.activeModel() || undefined
|
model_info: this.activeModel() || undefined
|
||||||
};
|
});
|
||||||
return [activeRunningSession, ...pending, ...raw];
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return [...pending, ...raw];
|
return Array.from(sessionMap.values()).sort((a, b) => b.start_time - a.start_time);
|
||||||
});
|
});
|
||||||
|
|
||||||
public currentSessionId = signal<string | null>(null);
|
public currentSessionId = signal<string | null>(null);
|
||||||
@@ -64,6 +92,7 @@ export class AgentService {
|
|||||||
public isRetrying = signal<boolean>(false);
|
public isRetrying = signal<boolean>(false);
|
||||||
public retryMessage = signal<string | null>(null);
|
public retryMessage = signal<string | null>(null);
|
||||||
public activeModel = signal<{ name: string; id: string; provider: string } | null>(null);
|
public activeModel = signal<{ name: string; id: string; provider: string } | null>(null);
|
||||||
|
public userPinnedSessionId = signal<string | null>(null); // Pinned session if user explicitly selected a non-running task
|
||||||
|
|
||||||
// Notes and Tab States
|
// Notes and Tab States
|
||||||
public currentNotes = signal<Record<string, string>>({});
|
public currentNotes = signal<Record<string, string>>({});
|
||||||
@@ -77,6 +106,13 @@ export class AgentService {
|
|||||||
public activeVideoTitle = signal<string>('');
|
public activeVideoTitle = signal<string>('');
|
||||||
public isVideoLoading = signal<boolean>(false);
|
public isVideoLoading = signal<boolean>(false);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Clear user-pinned selection so subsequent runs automatically follow active runner
|
||||||
|
*/
|
||||||
|
public clearUserPinnedSession(): void {
|
||||||
|
this.userPinnedSessionId.set(null);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Computed active/viewed session
|
* Computed active/viewed session
|
||||||
*/
|
*/
|
||||||
@@ -122,18 +158,82 @@ export class AgentService {
|
|||||||
});
|
});
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Stop the currently running task
|
* Run a new task by submitting to backend queue
|
||||||
*/
|
*/
|
||||||
public stopTask(): void {
|
public runTask(
|
||||||
this.http.post('/api/stop', {}).subscribe({
|
goal: string,
|
||||||
|
profile: string = 'flash',
|
||||||
|
expectedOutput?: string,
|
||||||
|
enableOutputter?: boolean
|
||||||
|
): Observable<any> {
|
||||||
|
return new Observable((obs) => {
|
||||||
|
const payload: any = { goal, profile };
|
||||||
|
if (expectedOutput && expectedOutput.trim()) {
|
||||||
|
payload.expected_output = expectedOutput.trim();
|
||||||
|
}
|
||||||
|
if (enableOutputter !== undefined) {
|
||||||
|
payload.enable_outputter = enableOutputter;
|
||||||
|
}
|
||||||
|
this.clearUserPinnedSession();
|
||||||
|
this.http.post<any>('/api/run', payload).subscribe({
|
||||||
|
next: (res) => {
|
||||||
|
if (res && res.tasks && res.tasks.length > 0) {
|
||||||
|
const newSessionId = res.tasks[0].session_id;
|
||||||
|
if (newSessionId) {
|
||||||
|
const isCurrentlyRunning =
|
||||||
|
this.agentStatus() === 'running' ||
|
||||||
|
this.sessions().some((s) => s.status === 'running');
|
||||||
|
if (!isCurrentlyRunning) {
|
||||||
|
this.selectSession(newSessionId, false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
obs.next(res);
|
||||||
|
obs.complete();
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
obs.error(err);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Stop the currently running task (and optionally clear queue)
|
||||||
|
*/
|
||||||
|
public stopTask(stopAll: boolean = false): void {
|
||||||
|
// Apply optimistic updates: only set idle if stopAll is true or pending queue is empty
|
||||||
|
if (stopAll || this.pendingQueue().length === 0) {
|
||||||
|
this.agentStatus.set('idle');
|
||||||
|
this.runningSessionId.set(null);
|
||||||
|
this.runningGoal.set(null);
|
||||||
|
}
|
||||||
|
this.isPaused.set(false);
|
||||||
|
this.pausedError.set(null);
|
||||||
|
this.isRetrying.set(false);
|
||||||
|
if (stopAll) {
|
||||||
|
this.pendingQueue.set([]);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Mark active streaming / pending logs in current session as finished
|
||||||
|
this.sessionLogs.update((logs) =>
|
||||||
|
logs.map((l) => {
|
||||||
|
if (l.type === 'llm_stream' && !l.data?.isCompleted) {
|
||||||
|
return { ...l, data: { ...l.data, isCompleted: true } };
|
||||||
|
}
|
||||||
|
return l;
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
|
this.http.post<any>(`/api/stop?all=${stopAll}`, {}).subscribe({
|
||||||
next: () => {
|
next: () => {
|
||||||
this.agentStatus.set('idle');
|
this.fetchStatus();
|
||||||
this.isPaused.set(false);
|
|
||||||
this.pausedError.set(null);
|
|
||||||
this.fetchSessions();
|
this.fetchSessions();
|
||||||
},
|
},
|
||||||
error: (err) => {
|
error: (err) => {
|
||||||
console.error('Failed to stop task:', err);
|
console.error('Failed to stop task:', err);
|
||||||
|
this.fetchSessions();
|
||||||
|
this.fetchStatus();
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
@@ -161,6 +261,10 @@ export class AgentService {
|
|||||||
next: (data) => {
|
next: (data) => {
|
||||||
this.rawSessions.set(data);
|
this.rawSessions.set(data);
|
||||||
this.updateModelForCurrentView(data);
|
this.updateModelForCurrentView(data);
|
||||||
|
// On initial load, if nothing is selected, not pinned, not running, and sessions exist, select latest
|
||||||
|
if (!this.currentSessionId() && !this.userPinnedSessionId() && this.agentStatus() !== 'running' && data.length > 0) {
|
||||||
|
this.selectSession(data[0].session_id, false);
|
||||||
|
}
|
||||||
},
|
},
|
||||||
error: (err) => {
|
error: (err) => {
|
||||||
console.error('Failed to fetch sessions from backend:', err);
|
console.error('Failed to fetch sessions from backend:', err);
|
||||||
@@ -169,9 +273,97 @@ export class AgentService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Connect to the Server-Sent Events stream for a specific session
|
* Delete an individual session / task
|
||||||
*/
|
*/
|
||||||
public selectSession(sessionId: string): void {
|
public deleteSession(sessionId: string): Observable<any> {
|
||||||
|
// 1. Optimistically update local session state immediately
|
||||||
|
this.rawSessions.update((list) => list.filter((s) => s.session_id !== sessionId));
|
||||||
|
this.pendingQueue.update((list) => list.filter((s) => s.session_id !== sessionId));
|
||||||
|
|
||||||
|
if (this.userPinnedSessionId() === sessionId) {
|
||||||
|
this.userPinnedSessionId.set(null);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (this.currentSessionId() === sessionId) {
|
||||||
|
this.selectSession('', false);
|
||||||
|
if (this.isVideoWindowOpen()) {
|
||||||
|
this.closeVideoPlayer();
|
||||||
|
}
|
||||||
|
const runningId = this.runningSessionId();
|
||||||
|
if (runningId && this.agentStatus() === 'running') {
|
||||||
|
this.selectSession(runningId, false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. Send API request
|
||||||
|
return new Observable((obs) => {
|
||||||
|
this.http.post<any>(`/api/sessions/${sessionId}/delete`, {}).subscribe({
|
||||||
|
next: (res) => {
|
||||||
|
this.fetchSessions();
|
||||||
|
this.fetchStatus();
|
||||||
|
obs.next(res);
|
||||||
|
obs.complete();
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
console.error(`Failed to delete session ${sessionId}:`, err);
|
||||||
|
this.fetchSessions();
|
||||||
|
this.fetchStatus();
|
||||||
|
obs.error(err);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Clear all sessions, tasks, and history
|
||||||
|
*/
|
||||||
|
public clearAllHistory(): Observable<any> {
|
||||||
|
// 1. Optimistically clear all local session state immediately
|
||||||
|
this.userPinnedSessionId.set(null);
|
||||||
|
this.rawSessions.set([]);
|
||||||
|
this.pendingQueue.set([]);
|
||||||
|
this.selectSession('', false);
|
||||||
|
if (this.isVideoWindowOpen()) {
|
||||||
|
this.closeVideoPlayer();
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. Send API request
|
||||||
|
return new Observable((obs) => {
|
||||||
|
this.http.post<any>('/api/cleanup', {}).subscribe({
|
||||||
|
next: (res) => {
|
||||||
|
this.fetchSessions();
|
||||||
|
this.fetchStatus();
|
||||||
|
obs.next(res);
|
||||||
|
obs.complete();
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
console.error('Failed to cleanup history:', err);
|
||||||
|
this.fetchSessions();
|
||||||
|
this.fetchStatus();
|
||||||
|
obs.error(err);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Connect to the Server-Sent Events stream for a specific session
|
||||||
|
* @param sessionId The ID of the session to select
|
||||||
|
* @param isUserAction Whether this selection was explicitly triggered by user interaction
|
||||||
|
*/
|
||||||
|
public selectSession(sessionId: string, isUserAction: boolean = false): void {
|
||||||
|
if (isUserAction) {
|
||||||
|
if (sessionId && sessionId === this.runningSessionId() && this.agentStatus() === 'running') {
|
||||||
|
// User explicitly clicked the currently running task -> follow live runner
|
||||||
|
this.userPinnedSessionId.set(null);
|
||||||
|
} else if (sessionId) {
|
||||||
|
// User explicitly clicked a non-running or historical task -> pin to it
|
||||||
|
this.userPinnedSessionId.set(sessionId);
|
||||||
|
} else {
|
||||||
|
this.userPinnedSessionId.set(null);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
if (!sessionId) {
|
if (!sessionId) {
|
||||||
this.currentSessionId.set(null);
|
this.currentSessionId.set(null);
|
||||||
this.sessionLogs.set([]);
|
this.sessionLogs.set([]);
|
||||||
@@ -262,12 +454,13 @@ export class AgentService {
|
|||||||
this.runningGoal.set(parsedData.initial_goal);
|
this.runningGoal.set(parsedData.initial_goal);
|
||||||
}
|
}
|
||||||
this.fetchSessions();
|
this.fetchSessions();
|
||||||
|
// Only auto-select if user hasn't explicitly chosen to inspect another task
|
||||||
|
const pinnedId = this.userPinnedSessionId();
|
||||||
|
if (!pinnedId && parsedData?.session_id) {
|
||||||
|
this.selectSession(parsedData.session_id, false);
|
||||||
|
}
|
||||||
} else if (eventType === 'session_ended') {
|
} else if (eventType === 'session_ended') {
|
||||||
this.agentStatus.set('idle');
|
this.fetchStatus();
|
||||||
this.runningSessionId.set(null);
|
|
||||||
this.runningGoal.set(null);
|
|
||||||
this.isRetrying.set(false);
|
|
||||||
this.isPaused.set(false);
|
|
||||||
this.fetchSessions();
|
this.fetchSessions();
|
||||||
this.updateModelForCurrentView();
|
this.updateModelForCurrentView();
|
||||||
}
|
}
|
||||||
@@ -298,6 +491,7 @@ export class AgentService {
|
|||||||
this.isRetrying.set(false);
|
this.isRetrying.set(false);
|
||||||
this.isPaused.set(false);
|
this.isPaused.set(false);
|
||||||
this.fetchSessions();
|
this.fetchSessions();
|
||||||
|
this.fetchNotes(sessionId);
|
||||||
this.updateModelForCurrentView();
|
this.updateModelForCurrentView();
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
@@ -416,7 +610,7 @@ export class AgentService {
|
|||||||
|
|
||||||
if (eventType === 'trace_recorded' && parsedData) {
|
if (eventType === 'trace_recorded' && parsedData) {
|
||||||
const trName = parsedData.name || '';
|
const trName = parsedData.name || '';
|
||||||
if (['save_note', 'read_note', 'update_note', 'append_note', 'list_notes'].includes(trName.toLowerCase())) {
|
if (['save_note', 'read_note', 'update_note', 'append_note', 'list_notes', 'outputter'].includes(trName.toLowerCase())) {
|
||||||
const curSessionId = this.currentSessionId();
|
const curSessionId = this.currentSessionId();
|
||||||
if (curSessionId === sessionId) {
|
if (curSessionId === sessionId) {
|
||||||
this.fetchNotes(curSessionId);
|
this.fetchNotes(curSessionId);
|
||||||
@@ -443,6 +637,8 @@ export class AgentService {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private pollCounter: number = 0;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Start periodic status polling from the backend
|
* Start periodic status polling from the backend
|
||||||
*/
|
*/
|
||||||
@@ -450,6 +646,11 @@ export class AgentService {
|
|||||||
this.fetchStatus();
|
this.fetchStatus();
|
||||||
this.statusInterval = setInterval(() => {
|
this.statusInterval = setInterval(() => {
|
||||||
this.fetchStatus();
|
this.fetchStatus();
|
||||||
|
this.pollCounter++;
|
||||||
|
// Periodically refresh sessions every 6 seconds (every 3 polling intervals of 2s) to stay in sync with external DB modifications
|
||||||
|
if (this.pollCounter % 3 === 0) {
|
||||||
|
this.fetchSessions();
|
||||||
|
}
|
||||||
}, 2000);
|
}, 2000);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -511,11 +712,16 @@ export class AgentService {
|
|||||||
this.fetchSessions();
|
this.fetchSessions();
|
||||||
}
|
}
|
||||||
|
|
||||||
// Auto-select the session if a new active session starts and user is not browsing an old historical session
|
// Auto-select the session if running and user hasn't explicitly chosen to inspect another task
|
||||||
if (data.status === 'running' && data.session_id && data.session_id !== oldRunningSessionId) {
|
if (data.status === 'running' && data.session_id) {
|
||||||
const currentId = this.currentSessionId();
|
const currentId = this.currentSessionId();
|
||||||
if (!currentId || currentId === oldRunningSessionId) {
|
const pinnedId = this.userPinnedSessionId();
|
||||||
this.selectSession(data.session_id);
|
if (!currentId) {
|
||||||
|
this.selectSession(data.session_id, false);
|
||||||
|
} else if (currentId !== data.session_id) {
|
||||||
|
if (!pinnedId) {
|
||||||
|
this.selectSession(data.session_id, false);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,482 @@
|
|||||||
|
/**
|
||||||
|
* Copyright 2026 Google LLC
|
||||||
|
*
|
||||||
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
* you may not use this file except in compliance with the License.
|
||||||
|
* You may obtain a copy of the License at
|
||||||
|
*
|
||||||
|
* http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
*
|
||||||
|
* Unless required by applicable law or agreed to in writing, software
|
||||||
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
* See the License for the specific language governing permissions and
|
||||||
|
* limitations under the License.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { Injectable, signal, computed, inject } from '@angular/core';
|
||||||
|
import { HttpClient } from '@angular/common/http';
|
||||||
|
import { Observable, tap } from 'rxjs';
|
||||||
|
import { SystemReadinessReport, DeviceInfo, ProbeResult, EmulatorLaunchState, EmulatorLaunchStage } from '../core/models/system.model';
|
||||||
|
|
||||||
|
@Injectable({
|
||||||
|
providedIn: 'root'
|
||||||
|
})
|
||||||
|
export class SystemService {
|
||||||
|
private http = inject(HttpClient);
|
||||||
|
|
||||||
|
// Core reactive signals
|
||||||
|
public readinessReport = signal<SystemReadinessReport | null>(null);
|
||||||
|
public isLoading = signal<boolean>(false);
|
||||||
|
public isRestartingAdb = signal<boolean>(false);
|
||||||
|
public launchingAvd = signal<string | null>(null);
|
||||||
|
public emulatorLaunchState = signal<EmulatorLaunchState | null>(null);
|
||||||
|
public isEmulatorLaunching = computed(() => {
|
||||||
|
const s = this.emulatorLaunchState()?.status;
|
||||||
|
return s === 'starting' || s === 'waiting_for_adb' || s === 'booting';
|
||||||
|
});
|
||||||
|
public lastCheckedTime = signal<Date | null>(null);
|
||||||
|
private emulatorPollTimer: any = null;
|
||||||
|
|
||||||
|
// Probe lookups
|
||||||
|
public pythonProbe = computed(() => this.probes().find(p => p.id === 'python_runtime') || null);
|
||||||
|
public configProbe = computed(() => this.probes().find(p => p.id === 'system_config') || null);
|
||||||
|
public toolchainProbe = computed(() => this.probes().find(p => p.id === 'toolchain') || null);
|
||||||
|
public adbProbe = computed(() => this.probes().find(p => p.id === 'android_adb') || null);
|
||||||
|
public llmProbe = computed(() => this.probes().find(p => p.id === 'gemini_api_key' || p.id === 'llm_api_key') || null);
|
||||||
|
public geminiProbe = computed(() => this.llmProbe());
|
||||||
|
public ocrProbe = computed(() => this.probes().find(p => p.id === 'vision_ocr_key' || p.id === 'ocr_api_key') || null);
|
||||||
|
|
||||||
|
// Grouped readiness helpers for the 3-step onboarding flow
|
||||||
|
public isEnvironmentReady = computed(() => {
|
||||||
|
const py = this.pythonProbe();
|
||||||
|
const cfg = this.configProbe();
|
||||||
|
const adb = this.adbProbe();
|
||||||
|
const tc = this.toolchainProbe();
|
||||||
|
const pyOk = py?.status === 'pass';
|
||||||
|
const cfgOk = cfg?.status === 'pass';
|
||||||
|
const adbInstalled = adb?.metadata?.['installed'] ?? (adb?.status !== 'fail');
|
||||||
|
const tcOk = tc?.status === 'pass' || tc?.is_blocker === false;
|
||||||
|
return pyOk && cfgOk && adbInstalled && tcOk;
|
||||||
|
});
|
||||||
|
|
||||||
|
public isSkipCredentialsCheck = signal<boolean>(false);
|
||||||
|
|
||||||
|
public isCredentialsReady = computed(() => {
|
||||||
|
if (this.isSkipCredentialsCheck()) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
const llm = this.llmProbe();
|
||||||
|
return llm?.status === 'pass';
|
||||||
|
});
|
||||||
|
|
||||||
|
public isDeviceReady = computed(() => {
|
||||||
|
const adb = this.adbProbe();
|
||||||
|
return adb?.status === 'pass';
|
||||||
|
});
|
||||||
|
|
||||||
|
// Step-level counting matching the 3-step onboarding guide
|
||||||
|
public totalStepCount = computed(() => 3);
|
||||||
|
public passedStepCount = computed(() => {
|
||||||
|
let count = 0;
|
||||||
|
if (this.isEnvironmentReady()) count++;
|
||||||
|
if (this.isCredentialsReady()) count++;
|
||||||
|
if (this.isDeviceReady()) count++;
|
||||||
|
return count;
|
||||||
|
});
|
||||||
|
|
||||||
|
// Computed helper signals
|
||||||
|
public isReady = computed(() => {
|
||||||
|
return this.isEnvironmentReady() && this.isCredentialsReady() && this.isDeviceReady();
|
||||||
|
});
|
||||||
|
public blockerCount = computed(() => this.totalStepCount());
|
||||||
|
public passedBlockerCount = computed(() => this.passedStepCount());
|
||||||
|
public probes = computed(() => this.readinessReport()?.probes ?? []);
|
||||||
|
public activeDevice = computed(() => this.readinessReport()?.active_device ?? null);
|
||||||
|
public osType = computed<'linux' | 'darwin' | 'windows'>(() => {
|
||||||
|
const raw = this.readinessReport()?.os_type;
|
||||||
|
if (raw === 'windows' || raw === 'win32') return 'windows';
|
||||||
|
if (raw === 'darwin' || raw === 'macos' || raw === 'mac') return 'darwin';
|
||||||
|
return 'linux';
|
||||||
|
});
|
||||||
|
|
||||||
|
public skipCredentialsCheck(): void {
|
||||||
|
this.isSkipCredentialsCheck.set(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
public setSkipCredentialsCheck(skip: boolean): void {
|
||||||
|
this.isSkipCredentialsCheck.set(skip);
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
// Device list from metadata
|
||||||
|
public connectedDevices = computed<DeviceInfo[]>(() => {
|
||||||
|
const meta = this.adbProbe()?.metadata;
|
||||||
|
if (meta && Array.isArray(meta['devices'])) {
|
||||||
|
return meta['devices'] as DeviceInfo[];
|
||||||
|
}
|
||||||
|
return [];
|
||||||
|
});
|
||||||
|
|
||||||
|
// Installed local AVD emulators from metadata
|
||||||
|
public installedAvds = computed<string[]>(() => {
|
||||||
|
const meta = this.adbProbe()?.metadata;
|
||||||
|
if (meta && Array.isArray(meta['installed_avds'])) {
|
||||||
|
return meta['installed_avds'] as string[];
|
||||||
|
}
|
||||||
|
return [];
|
||||||
|
});
|
||||||
|
|
||||||
|
public emulatorPath = computed<string | null>(() => {
|
||||||
|
return (this.adbProbe()?.metadata?.['emulator_path'] as string) || null;
|
||||||
|
});
|
||||||
|
|
||||||
|
public isEmulatorInPath = computed<boolean>(() => {
|
||||||
|
return (this.adbProbe()?.metadata?.['is_emulator_in_path'] as boolean) ?? true;
|
||||||
|
});
|
||||||
|
|
||||||
|
private pollingTimer: any = null;
|
||||||
|
|
||||||
|
constructor() {
|
||||||
|
this.fetchReadiness();
|
||||||
|
this.startAutoPolling(3000);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Start periodic auto-polling for system readiness changes
|
||||||
|
*/
|
||||||
|
public startAutoPolling(intervalMs: number = 3000): void {
|
||||||
|
if (this.pollingTimer) {
|
||||||
|
clearInterval(this.pollingTimer);
|
||||||
|
}
|
||||||
|
this.pollingTimer = setInterval(() => {
|
||||||
|
// Perform silent background check without disturbing UI loading state
|
||||||
|
this.fetchReadiness(true).subscribe({
|
||||||
|
error: () => {} // Silent catch
|
||||||
|
});
|
||||||
|
}, intervalMs);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Stop auto-polling
|
||||||
|
*/
|
||||||
|
public stopAutoPolling(): void {
|
||||||
|
if (this.pollingTimer) {
|
||||||
|
clearInterval(this.pollingTimer);
|
||||||
|
this.pollingTimer = null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fetch latest system readiness report from backend
|
||||||
|
* @param silent If true, updates signals silently without triggering global isLoading spinner
|
||||||
|
*/
|
||||||
|
public fetchReadiness(silent: boolean = false): Observable<SystemReadinessReport> {
|
||||||
|
if (!silent) {
|
||||||
|
this.isLoading.set(true);
|
||||||
|
}
|
||||||
|
return this.http.get<SystemReadinessReport>('/api/system/readiness').pipe(
|
||||||
|
tap({
|
||||||
|
next: (report) => {
|
||||||
|
this.readinessReport.set(report);
|
||||||
|
this.lastCheckedTime.set(new Date());
|
||||||
|
if (!silent) {
|
||||||
|
this.isLoading.set(false);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
console.error('Failed to fetch system readiness:', err);
|
||||||
|
if (!silent) {
|
||||||
|
this.isLoading.set(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fetch current emulator launch progress state snapshot
|
||||||
|
*/
|
||||||
|
public fetchEmulatorStatus(): Observable<EmulatorLaunchState> {
|
||||||
|
return this.http.get<EmulatorLaunchState>('/api/system/emulator/status').pipe(
|
||||||
|
tap({
|
||||||
|
next: (state) => {
|
||||||
|
this.emulatorLaunchState.set(state);
|
||||||
|
if (state.status === 'ready') {
|
||||||
|
this.launchingAvd.set(null);
|
||||||
|
this.stopEmulatorStatusPolling();
|
||||||
|
this.fetchReadiness().subscribe();
|
||||||
|
} else if (state.status === 'failed' || state.status === 'stopped' || state.status === 'idle') {
|
||||||
|
this.launchingAvd.set(null);
|
||||||
|
this.stopEmulatorStatusPolling();
|
||||||
|
} else {
|
||||||
|
if (state.avd_name) {
|
||||||
|
this.launchingAvd.set(state.avd_name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
console.error('Failed to fetch emulator status:', err);
|
||||||
|
}
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Start 1s interval polling for emulator boot progression
|
||||||
|
*/
|
||||||
|
public startEmulatorStatusPolling(): void {
|
||||||
|
if (this.emulatorPollTimer) {
|
||||||
|
clearInterval(this.emulatorPollTimer);
|
||||||
|
}
|
||||||
|
this.emulatorPollTimer = setInterval(() => {
|
||||||
|
this.fetchEmulatorStatus().subscribe({
|
||||||
|
error: () => {}
|
||||||
|
});
|
||||||
|
}, 1000);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Stop polling for emulator boot status
|
||||||
|
*/
|
||||||
|
public stopEmulatorStatusPolling(): void {
|
||||||
|
if (this.emulatorPollTimer) {
|
||||||
|
clearInterval(this.emulatorPollTimer);
|
||||||
|
this.emulatorPollTimer = null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Launch a local AVD emulator in the background and continuously track boot progress
|
||||||
|
*/
|
||||||
|
public launchEmulator(avdName: string): Observable<EmulatorLaunchState> {
|
||||||
|
this.launchingAvd.set(avdName);
|
||||||
|
this.emulatorLaunchState.set({
|
||||||
|
avd_name: avdName,
|
||||||
|
status: 'starting',
|
||||||
|
pid: null,
|
||||||
|
serial: null,
|
||||||
|
stage_message: 'Spawning emulator process...',
|
||||||
|
progress_percent: 15,
|
||||||
|
started_at: Date.now() / 1000,
|
||||||
|
elapsed_seconds: 0,
|
||||||
|
error: null,
|
||||||
|
logs: [`Initiating launch for AVD: ${avdName}...`],
|
||||||
|
can_retry: true,
|
||||||
|
});
|
||||||
|
|
||||||
|
return this.http.post<EmulatorLaunchState>('/api/system/emulator/launch', { avd_name: avdName }).pipe(
|
||||||
|
tap({
|
||||||
|
next: (state) => {
|
||||||
|
this.emulatorLaunchState.set(state);
|
||||||
|
if (state.status === 'failed') {
|
||||||
|
this.launchingAvd.set(null);
|
||||||
|
} else {
|
||||||
|
this.startEmulatorStatusPolling();
|
||||||
|
}
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
console.error('Failed to launch emulator:', err);
|
||||||
|
this.launchingAvd.set(null);
|
||||||
|
const errorMsg = err?.error?.detail || err?.message || 'Failed to start emulator process.';
|
||||||
|
this.emulatorLaunchState.set({
|
||||||
|
avd_name: avdName,
|
||||||
|
status: 'failed',
|
||||||
|
pid: null,
|
||||||
|
serial: null,
|
||||||
|
stage_message: 'Failed to initiate launch.',
|
||||||
|
progress_percent: 0,
|
||||||
|
started_at: null,
|
||||||
|
elapsed_seconds: 0,
|
||||||
|
error: errorMsg,
|
||||||
|
logs: [errorMsg],
|
||||||
|
can_retry: true,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Terminate running emulator
|
||||||
|
*/
|
||||||
|
public stopEmulator(): Observable<any> {
|
||||||
|
return this.http.post<any>('/api/system/emulator/stop', {}).pipe(
|
||||||
|
tap({
|
||||||
|
next: () => {
|
||||||
|
this.launchingAvd.set(null);
|
||||||
|
this.stopEmulatorStatusPolling();
|
||||||
|
this.fetchEmulatorStatus().subscribe();
|
||||||
|
this.fetchReadiness().subscribe();
|
||||||
|
}
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Clear and dismiss emulator launch tracker state
|
||||||
|
*/
|
||||||
|
public dismissEmulatorStatus(): Observable<any> {
|
||||||
|
return this.http.post<any>('/api/system/emulator/dismiss', {}).pipe(
|
||||||
|
tap({
|
||||||
|
next: () => {
|
||||||
|
this.emulatorLaunchState.set(null);
|
||||||
|
this.launchingAvd.set(null);
|
||||||
|
this.stopEmulatorStatusPolling();
|
||||||
|
}
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Restart local ADB server and refresh readiness state
|
||||||
|
*/
|
||||||
|
public restartAdb(): Observable<any> {
|
||||||
|
this.isRestartingAdb.set(true);
|
||||||
|
return this.http.post<any>('/api/system/adb/restart', {}).pipe(
|
||||||
|
tap({
|
||||||
|
next: (res) => {
|
||||||
|
if (res?.report) {
|
||||||
|
this.readinessReport.set(res.report);
|
||||||
|
this.lastCheckedTime.set(new Date());
|
||||||
|
}
|
||||||
|
this.isRestartingAdb.set(false);
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
console.error('Failed to restart ADB server:', err);
|
||||||
|
this.isRestartingAdb.set(false);
|
||||||
|
}
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Connect to an Android device over Wi-Fi
|
||||||
|
*/
|
||||||
|
public connectWirelessAdb(host: string, port: number = 5555): Observable<any> {
|
||||||
|
return this.http.post<any>('/api/system/adb/connect', { host, port }).pipe(
|
||||||
|
tap({
|
||||||
|
next: (res) => {
|
||||||
|
if (res?.report) {
|
||||||
|
this.readinessReport.set(res.report);
|
||||||
|
this.lastCheckedTime.set(new Date());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Select a specific connected device serial as active target
|
||||||
|
*/
|
||||||
|
public selectDevice(serial: string): Observable<any> {
|
||||||
|
this.isLoading.set(true);
|
||||||
|
return this.http.post<any>('/api/system/devices/select', { serial }).pipe(
|
||||||
|
tap({
|
||||||
|
next: (res) => {
|
||||||
|
if (res?.report) {
|
||||||
|
this.readinessReport.set(res.report);
|
||||||
|
}
|
||||||
|
this.isLoading.set(false);
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
console.error('Failed to select active device:', err);
|
||||||
|
this.isLoading.set(false);
|
||||||
|
}
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
public currentApiKey = computed<string>(() => {
|
||||||
|
return (this.llmProbe()?.metadata?.['current_key'] as string) || '';
|
||||||
|
});
|
||||||
|
|
||||||
|
public apiKeysMap = computed<Record<string, string>>(() => {
|
||||||
|
return (this.llmProbe()?.metadata?.['api_keys'] as Record<string, string>) || {};
|
||||||
|
});
|
||||||
|
|
||||||
|
public modelConfigEnv = signal<ModelConfigEnvResponse | null>(null);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fetch current model configuration (artemis.jsonc) and environment (.env) status
|
||||||
|
*/
|
||||||
|
public fetchModelConfigEnv(): Observable<ModelConfigEnvResponse> {
|
||||||
|
return this.http.get<ModelConfigEnvResponse>('/api/system/model-config-env').pipe(
|
||||||
|
tap({
|
||||||
|
next: (data) => {
|
||||||
|
this.modelConfigEnv.set(data);
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
console.error('Failed to fetch model config and env:', err);
|
||||||
|
}
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Test and verify an API key against live provider endpoint without persisting
|
||||||
|
*/
|
||||||
|
public testApiKey(provider: string, apiKey: string, baseUrl?: string): Observable<{ valid: boolean; provider: string; message: string }> {
|
||||||
|
return this.http.post<{ valid: boolean; provider: string; message: string }>('/api/system/credentials/test', {
|
||||||
|
provider,
|
||||||
|
api_key: apiKey,
|
||||||
|
base_url: baseUrl
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Update, verify, and configure API key for an LLM provider or Vision OCR
|
||||||
|
*/
|
||||||
|
public updateApiKey(provider: string, apiKey: string, persistToEnv: boolean = true): Observable<any> {
|
||||||
|
return this.http.post<any>('/api/system/credentials', {
|
||||||
|
provider,
|
||||||
|
api_key: apiKey,
|
||||||
|
persist_to_env: persistToEnv
|
||||||
|
}).pipe(
|
||||||
|
tap({
|
||||||
|
next: (res) => {
|
||||||
|
if (res?.report) {
|
||||||
|
this.readinessReport.set(res.report);
|
||||||
|
this.lastCheckedTime.set(new Date());
|
||||||
|
}
|
||||||
|
// Refresh model config & env after updating key
|
||||||
|
this.fetchModelConfigEnv().subscribe();
|
||||||
|
},
|
||||||
|
error: (err) => {
|
||||||
|
console.error(`Failed to update credentials for ${provider}:`, err);
|
||||||
|
}
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ModelConfigEnvResponse {
|
||||||
|
config_path: string;
|
||||||
|
config_filename: string;
|
||||||
|
config_content: string;
|
||||||
|
default_model: {
|
||||||
|
provider?: string;
|
||||||
|
model?: string;
|
||||||
|
thinking_level?: string;
|
||||||
|
fallback?: {
|
||||||
|
provider?: string;
|
||||||
|
model?: string;
|
||||||
|
thinking_level?: string;
|
||||||
|
};
|
||||||
|
};
|
||||||
|
presets: Record<string, {
|
||||||
|
provider: string;
|
||||||
|
model: string;
|
||||||
|
fallback?: { provider: string; model: string };
|
||||||
|
}>;
|
||||||
|
env_path: string;
|
||||||
|
env_filename: string;
|
||||||
|
env_vars: Array<{
|
||||||
|
name: string;
|
||||||
|
provider: string;
|
||||||
|
is_set: boolean;
|
||||||
|
preview: string | null;
|
||||||
|
description: string;
|
||||||
|
}>;
|
||||||
|
}
|
||||||
|
|
||||||
@@ -0,0 +1,112 @@
|
|||||||
|
import {
|
||||||
|
getStepPreImageUrl,
|
||||||
|
getStepPostImageUrl
|
||||||
|
} from './action-formatter.util';
|
||||||
|
|
||||||
|
describe('action-formatter.util screenshot chaining', () => {
|
||||||
|
it('should chain pre/post images correctly in FailureAnalyzer multi-step recovery', () => {
|
||||||
|
const stepData = {
|
||||||
|
step_id: 'step-06',
|
||||||
|
pre_image_name: '01a0e8d8cddbeb6a352e3bef1afbee0f0476a05b0fc62abb78e59c7b6aa16079',
|
||||||
|
post_image_name: '484361b1ab4b340415e01042e3496f0edc0549c837c897ece242835d12d7d71d',
|
||||||
|
action_taken: {
|
||||||
|
action: 'tap',
|
||||||
|
coordinates: [540, 720],
|
||||||
|
timestamp: 1787077538.0
|
||||||
|
},
|
||||||
|
last_execution_result: {
|
||||||
|
status: 'success',
|
||||||
|
repair_status: 'fixed',
|
||||||
|
execution: [
|
||||||
|
{
|
||||||
|
attempts: ['Pre-execution validation failed']
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
generic_tools: [
|
||||||
|
{
|
||||||
|
trace_id: 'tool-save-note',
|
||||||
|
name: 'save_note',
|
||||||
|
type: 'tool',
|
||||||
|
timestamp: 1787077537.0,
|
||||||
|
payload: { args: { title: 'Note 1' } }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
trace_id: 'agent-failure-analyzer',
|
||||||
|
name: 'failure_analyzer',
|
||||||
|
agent_name: 'failure_analyzer',
|
||||||
|
type: 'agent',
|
||||||
|
timestamp: 1787077540.0,
|
||||||
|
payload: {
|
||||||
|
args: {
|
||||||
|
pre_screenshot_name: '01a0e8d8cddbeb6a352e3bef1afbee0f0476a05b0fc62abb78e59c7b6aa16079',
|
||||||
|
post_screenshot_name: 'fbdf17fa7a970d2c37aa52ae4336119e76a7d13ea457811cf8aaab19b3f53843'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
trace_id: 'action-click-1',
|
||||||
|
name: 'click',
|
||||||
|
type: 'action',
|
||||||
|
agent_name: 'failure_analyzer',
|
||||||
|
timestamp: 1787077607.0,
|
||||||
|
payload: {
|
||||||
|
args: { target: '[500, 300]' },
|
||||||
|
result: {
|
||||||
|
outcome: 'Clicked successfully',
|
||||||
|
post_image_name: '18bd2ac49e64976f1e74e828ddae76e54da1587044b1d3fb01317dd202ff7c67'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
trace_id: 'action-click-seq-2',
|
||||||
|
name: 'click_sequence',
|
||||||
|
type: 'action',
|
||||||
|
agent_name: 'failure_analyzer',
|
||||||
|
timestamp: 1787077631.0,
|
||||||
|
payload: {
|
||||||
|
args: { sequence: '[[500, 300], [876, 360]]' },
|
||||||
|
result: {
|
||||||
|
outcome: 'Sequence clicked successfully',
|
||||||
|
post_image_name: '484361b1ab4b340415e01042e3496f0edc0549c837c897ece242835d12d7d71d'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
};
|
||||||
|
|
||||||
|
const action1 = stepData.generic_tools[2]; // click
|
||||||
|
const action2 = stepData.generic_tools[3]; // click_sequence
|
||||||
|
|
||||||
|
// Action 1: Pre should be the failed state screenshot (fbdf17fa...), Post should be Action 1's post (18bd2ac4...)
|
||||||
|
const act1Pre = getStepPreImageUrl(stepData, action1);
|
||||||
|
const act1Post = getStepPostImageUrl(stepData, action1);
|
||||||
|
expect(act1Pre).toBe('/images/fbdf17fa7a970d2c37aa52ae4336119e76a7d13ea457811cf8aaab19b3f53843');
|
||||||
|
expect(act1Post).toBe('/images/18bd2ac49e64976f1e74e828ddae76e54da1587044b1d3fb01317dd202ff7c67');
|
||||||
|
|
||||||
|
// Action 2: "下一次的决策前是上一次的动作后" -> Pre MUST be Action 1's post (18bd2ac4...), Post should be Action 2's post (484361b1...)
|
||||||
|
const act2Pre = getStepPreImageUrl(stepData, action2);
|
||||||
|
const act2Post = getStepPostImageUrl(stepData, action2);
|
||||||
|
expect(act2Pre).toBe('/images/18bd2ac49e64976f1e74e828ddae76e54da1587044b1d3fb01317dd202ff7c67');
|
||||||
|
expect(act2Post).toBe('/images/484361b1ab4b340415e01042e3496f0edc0549c837c897ece242835d12d7d71d');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('should handle standard single-action step correctly', () => {
|
||||||
|
const stepData = {
|
||||||
|
step_id: 'step-01',
|
||||||
|
pre_image_name: 'pre123',
|
||||||
|
post_image_name: 'post456',
|
||||||
|
action_taken: {
|
||||||
|
action: 'tap',
|
||||||
|
coordinates: [100, 200],
|
||||||
|
timestamp: 1000
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const preUrl = getStepPreImageUrl(stepData, stepData.action_taken);
|
||||||
|
const postUrl = getStepPostImageUrl(stepData, stepData.action_taken);
|
||||||
|
|
||||||
|
expect(preUrl).toBe('/images/pre123');
|
||||||
|
expect(postUrl).toBe('/images/post456');
|
||||||
|
});
|
||||||
|
});
|
||||||
+324
-18
@@ -15,7 +15,8 @@
|
|||||||
*/
|
*/
|
||||||
|
|
||||||
import { ActionParam } from '../core/models/stream.model';
|
import { ActionParam } from '../core/models/stream.model';
|
||||||
import { extractNumbersFromCoordinateValue, isPureDirectionString } from './image-overlay.util';
|
import { extractNumbersFromCoordinateValue, isPureDirectionString, parseSequenceCoordinates } from './image-overlay.util';
|
||||||
|
import { cleanErrorMessage } from './tool-formatter.util';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Safely extract action object from possible array or nested structure
|
* Safely extract action object from possible array or nested structure
|
||||||
@@ -43,6 +44,40 @@ export function isAndroidAction(action: any): boolean {
|
|||||||
].includes(name);
|
].includes(name);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if the action or tool is a task completion / submission report
|
||||||
|
*/
|
||||||
|
export function isReportStatusAction(action: any): boolean {
|
||||||
|
const act = getActionObject(action);
|
||||||
|
if (!act) return false;
|
||||||
|
const name = (act.name || act.action || '').toLowerCase().replace(/^(_)?(self\.)?exec_/, '');
|
||||||
|
return name === 'report_task_status' || name === 'report_status' || name === 'submit_task_status';
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get status of report_task_status (completed / failed)
|
||||||
|
*/
|
||||||
|
export function getReportStatusValue(action: any): string {
|
||||||
|
const act = getActionObject(action);
|
||||||
|
if (!act) return 'completed';
|
||||||
|
const args = act.args && typeof act.args === 'object' ? act.args : (act.payload?.args || act.payload || act);
|
||||||
|
const status = args.status || act.status;
|
||||||
|
if (typeof status === 'string') {
|
||||||
|
return status.toLowerCase();
|
||||||
|
}
|
||||||
|
return 'completed';
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get explanation of report_task_status
|
||||||
|
*/
|
||||||
|
export function getReportStatusExplanation(action: any): string {
|
||||||
|
const act = getActionObject(action);
|
||||||
|
if (!act) return '';
|
||||||
|
const args = act.args && typeof act.args === 'object' ? act.args : (act.payload?.args || act.payload || act);
|
||||||
|
return args.explanation || args.reason || args.summary || act.explanation || act.reason || act.summary || '';
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Get Material Symbol icon name for Android action
|
* Get Material Symbol icon name for Android action
|
||||||
*/
|
*/
|
||||||
@@ -239,6 +274,15 @@ export function getActionCoords(action: any): string {
|
|||||||
const act = getActionObject(action);
|
const act = getActionObject(action);
|
||||||
if (!act) return '';
|
if (!act) return '';
|
||||||
const args = act.args && typeof act.args === 'object' ? act.args : {};
|
const args = act.args && typeof act.args === 'object' ? act.args : {};
|
||||||
|
const name = (act.name || act.action || '').toLowerCase();
|
||||||
|
|
||||||
|
// Check sequence first if action is sequence or sequence arg is present
|
||||||
|
const isSequenceAction = name === 'click_sequence' || name === 'tap_sequence' || Boolean(act.sequence || args.sequence || act.normalized_sequence || args.normalized_sequence);
|
||||||
|
const rawSeq = act.normalized_sequence || args.normalized_sequence || act.sequence || args.sequence || act.targets || args.targets || (Array.isArray(act.coordinates) && Array.isArray(act.coordinates[0]) ? act.coordinates : null) || (Array.isArray(args.coordinates) && Array.isArray(args.coordinates[0]) ? args.coordinates : null);
|
||||||
|
const seqPoints = parseSequenceCoordinates(rawSeq);
|
||||||
|
if (seqPoints && seqPoints.length > 0 && (isSequenceAction || seqPoints.length > 1)) {
|
||||||
|
return seqPoints.map(pt => `[${pt[0]}, ${pt[1]}]`).join(' → ');
|
||||||
|
}
|
||||||
|
|
||||||
// Check normalized start and end first
|
// Check normalized start and end first
|
||||||
const normStart = extractNumbersFromCoordinateValue(act.normalized_start_coordinates || args.normalized_start_coordinates);
|
const normStart = extractNumbersFromCoordinateValue(act.normalized_start_coordinates || args.normalized_start_coordinates);
|
||||||
@@ -280,30 +324,67 @@ export function getActionCoords(action: any): string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Check if the action execution failed
|
* Check if the action execution failed or encountered an execution failure/interception
|
||||||
*/
|
*/
|
||||||
export function isActionFailed(action: any, stepData?: any): boolean {
|
export function isActionFailed(action: any, stepData?: any): boolean {
|
||||||
const act = getActionObject(action);
|
const act = getActionObject(action);
|
||||||
if (act) {
|
if (act) {
|
||||||
if (act.status === 'failed' || act.status === 'error' || act.success === false) return true;
|
if (act.status === 'failed' || act.status === 'error' || act.success === false) return true;
|
||||||
}
|
}
|
||||||
if (stepData && stepData.last_execution_result) {
|
if (stepData) {
|
||||||
const res = stepData.last_execution_result;
|
if (stepData.status === 'failed' || stepData.status === 'error') return true;
|
||||||
if (res.status === 'failed' || res.status === 'error' || res.success === false || res.is_successful === false) return true;
|
if (stepData.last_execution_result) {
|
||||||
|
const res = typeof stepData.last_execution_result === 'string'
|
||||||
|
? (() => { try { return JSON.parse(stepData.last_execution_result); } catch { return null; } })()
|
||||||
|
: stepData.last_execution_result;
|
||||||
|
|
||||||
|
if (res && typeof res === 'object') {
|
||||||
|
if (res.status === 'failed' || res.status === 'error' || res.success === false || res.is_successful === false) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (res.repair_status === 'fixed' || res.repair_status === 'failed' || res.repair_status === 'cannot_fix') {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (Array.isArray(res.execution) && res.execution.length > 0) {
|
||||||
|
const firstExec = res.execution[0];
|
||||||
|
if (firstExec && (firstExec.status === 'failed' || firstExec.status === 'error' || firstExec.error || (Array.isArray(firstExec.attempts) && firstExec.attempts.length > 0))) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Get failure message for an action
|
* Get failure message or reason for an action
|
||||||
*/
|
*/
|
||||||
export function getActionErrorMessage(action: any, stepData?: any): string {
|
export function getActionErrorMessage(action: any, stepData?: any): string {
|
||||||
const act = getActionObject(action);
|
const act = getActionObject(action);
|
||||||
if (act && (act.error || act.message)) return act.error || act.message;
|
if (act && (act.error || act.message || act.failure_reason)) {
|
||||||
|
return cleanErrorMessage(act.error || act.message || act.failure_reason);
|
||||||
|
}
|
||||||
if (stepData && stepData.last_execution_result) {
|
if (stepData && stepData.last_execution_result) {
|
||||||
const res = stepData.last_execution_result;
|
const res = typeof stepData.last_execution_result === 'string'
|
||||||
if (res.error || res.message || res.failure_reason) {
|
? (() => { try { return JSON.parse(stepData.last_execution_result); } catch { return null; } })()
|
||||||
return res.error || res.message || res.failure_reason;
|
: stepData.last_execution_result;
|
||||||
|
|
||||||
|
if (res && typeof res === 'object') {
|
||||||
|
if (Array.isArray(res.execution) && res.execution.length > 0) {
|
||||||
|
const firstExec = res.execution[0];
|
||||||
|
if (firstExec) {
|
||||||
|
if (Array.isArray(firstExec.attempts) && firstExec.attempts.length > 0) {
|
||||||
|
return cleanErrorMessage(firstExec.attempts[0]);
|
||||||
|
}
|
||||||
|
if (firstExec.error || firstExec.failure_reason || firstExec.message) {
|
||||||
|
return cleanErrorMessage(firstExec.error || firstExec.failure_reason || firstExec.message);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (res.error || res.message || res.failure_reason) {
|
||||||
|
return cleanErrorMessage(res.error || res.message || res.failure_reason);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return 'Action Failed';
|
return 'Action Failed';
|
||||||
@@ -437,24 +518,249 @@ export function formatImageUrl(candidate: any): string | null {
|
|||||||
return `/images/${trimmed}`;
|
return `/images/${trimmed}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Helper to check if an item is a generic tool trace rather than the step's primary action
|
||||||
|
*/
|
||||||
|
export function isGenericToolTrace(actionData: any, stepData?: any): boolean {
|
||||||
|
if (!actionData) return false;
|
||||||
|
if (actionData === stepData?.action_taken) return false;
|
||||||
|
if (Array.isArray(stepData?.action_taken)) {
|
||||||
|
if (stepData.action_taken === actionData || stepData.action_taken.includes(actionData)) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (actionData.is_primary_step_action) return false;
|
||||||
|
return Boolean(actionData.trace_id || actionData.payload || actionData.type === 'tool' || (actionData.type === 'action' && actionData.trace_id));
|
||||||
|
}
|
||||||
|
|
||||||
|
interface ResolvedImageEntry {
|
||||||
|
pre: string | null;
|
||||||
|
post: string | null;
|
||||||
|
hasExplicitPost: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
const stepImageMapCache = new WeakMap<any, { key: string; map: Map<any, ResolvedImageEntry> }>();
|
||||||
|
|
||||||
|
function safeParseJson(val: any): any {
|
||||||
|
if (typeof val === 'string' && (val.trim().startsWith('{') || val.trim().startsWith('['))) {
|
||||||
|
try {
|
||||||
|
return JSON.parse(val);
|
||||||
|
} catch {
|
||||||
|
return val;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return val;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function extractItemPreImage(item: any): string | null {
|
||||||
|
const act = getActionObject(item);
|
||||||
|
if (!act) return null;
|
||||||
|
const payload = safeParseJson(act.payload);
|
||||||
|
const result = safeParseJson(act.result || payload?.result);
|
||||||
|
const args = safeParseJson(act.args || payload?.args);
|
||||||
|
|
||||||
|
const candidate =
|
||||||
|
result?.pre_image_name || result?.pre_screenshot_name || result?.pre_screenshot ||
|
||||||
|
payload?.pre_image_name || payload?.pre_screenshot_name || payload?.pre_screenshot ||
|
||||||
|
args?.pre_image_name || args?.pre_screenshot_name || args?.pre_screenshot ||
|
||||||
|
act.pre_image_name || act.pre_screenshot_name || act.pre_screenshot || act.before_screenshot || act.screenshot;
|
||||||
|
|
||||||
|
return candidate ? String(candidate) : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function extractItemPostImage(item: any): string | null {
|
||||||
|
const act = getActionObject(item);
|
||||||
|
if (!act) return null;
|
||||||
|
const payload = safeParseJson(act.payload);
|
||||||
|
const result = safeParseJson(act.result || payload?.result);
|
||||||
|
const args = safeParseJson(act.args || payload?.args);
|
||||||
|
|
||||||
|
const candidate =
|
||||||
|
result?.post_image_name || result?.post_screenshot_name || result?.post_screenshot ||
|
||||||
|
payload?.post_image_name || payload?.post_screenshot_name || payload?.post_screenshot ||
|
||||||
|
args?.post_image_name || args?.post_screenshot_name || args?.post_screenshot ||
|
||||||
|
act.post_image_name || act.post_screenshot_name || act.post_screenshot || act.after_screenshot;
|
||||||
|
|
||||||
|
return candidate ? String(candidate) : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function getStepFailureScreenshot(stepData: any): string | null {
|
||||||
|
if (!stepData) return null;
|
||||||
|
if (stepData.failed_screenshot) return String(stepData.failed_screenshot);
|
||||||
|
if (stepData.failure_screenshot) return String(stepData.failure_screenshot);
|
||||||
|
if (stepData.failed_post_image_name) return String(stepData.failed_post_image_name);
|
||||||
|
|
||||||
|
if (Array.isArray(stepData.generic_tools)) {
|
||||||
|
for (const tool of stepData.generic_tools) {
|
||||||
|
const act = getActionObject(tool);
|
||||||
|
const payload = safeParseJson(act?.payload);
|
||||||
|
const args = safeParseJson(act?.args || payload?.args);
|
||||||
|
const name = String(act?.name || act?.agent_name || '').toLowerCase();
|
||||||
|
if (args?.post_screenshot_name) return String(args.post_screenshot_name);
|
||||||
|
if (args?.post_screenshot && (name.includes('failure') || name.includes('validator') || name.includes('diagnos'))) {
|
||||||
|
return String(args.post_screenshot);
|
||||||
|
}
|
||||||
|
if (act?.post_screenshot_name && (name.includes('failure') || name.includes('validator'))) {
|
||||||
|
return String(act.post_screenshot_name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function getEventTimestamp(obj: any): number {
|
||||||
|
if (!obj) return 0;
|
||||||
|
const ts = obj.timestamp ?? obj.start_time ?? obj.created_at;
|
||||||
|
if (typeof ts === 'number') {
|
||||||
|
return ts < 1e11 ? ts * 1000 : ts;
|
||||||
|
}
|
||||||
|
if (typeof ts === 'string') {
|
||||||
|
const parsed = new Date(ts).getTime();
|
||||||
|
if (!isNaN(parsed) && parsed > 0) return parsed;
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function resolveStepImageMap(stepData: any): Map<any, ResolvedImageEntry> {
|
||||||
|
const emptyMap = new Map<any, ResolvedImageEntry>();
|
||||||
|
if (!stepData || typeof stepData !== 'object') return emptyMap;
|
||||||
|
|
||||||
|
const toolsLen = Array.isArray(stepData.generic_tools) ? stepData.generic_tools.length : 0;
|
||||||
|
const cacheKey = `${stepData.step_id || ''}_${toolsLen}_${stepData.post_image_name || ''}_${stepData.pre_image_name || ''}`;
|
||||||
|
|
||||||
|
const cached = stepImageMapCache.get(stepData);
|
||||||
|
if (cached && cached.key === cacheKey) {
|
||||||
|
return cached.map;
|
||||||
|
}
|
||||||
|
|
||||||
|
const map = new Map<any, ResolvedImageEntry>();
|
||||||
|
const events: Array<{ item: any; act: any; timestamp: number; isPrimary: boolean }> = [];
|
||||||
|
|
||||||
|
if (stepData.action_taken) {
|
||||||
|
events.push({
|
||||||
|
item: stepData.action_taken,
|
||||||
|
act: getActionObject(stepData.action_taken),
|
||||||
|
timestamp: getEventTimestamp(stepData.action_taken),
|
||||||
|
isPrimary: true
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
if (Array.isArray(stepData.generic_tools)) {
|
||||||
|
stepData.generic_tools.forEach((t: any) => {
|
||||||
|
if (t) {
|
||||||
|
events.push({
|
||||||
|
item: t,
|
||||||
|
act: getActionObject(t),
|
||||||
|
timestamp: getEventTimestamp(t),
|
||||||
|
isPrimary: false
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
events.sort((a, b) => a.timestamp - b.timestamp);
|
||||||
|
|
||||||
|
const failureShot = getStepFailureScreenshot(stepData);
|
||||||
|
const hadPrimaryFailure = Boolean(failureShot) || (stepData.action_taken && isActionFailed(stepData.action_taken, stepData));
|
||||||
|
const hasSubsequentActions = events.some(e => !e.isPrimary && (isAndroidAction(e.act) || Boolean(extractItemPostImage(e.act))));
|
||||||
|
|
||||||
|
let lastActionItem: any = null;
|
||||||
|
for (let i = events.length - 1; i >= 0; i--) {
|
||||||
|
const e = events[i];
|
||||||
|
if (isAndroidAction(e.act) || extractItemPostImage(e.act) || e.isPrimary) {
|
||||||
|
lastActionItem = e.item;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let currentPreImage = stepData.pre_image_name || stepData.pre_screenshot || null;
|
||||||
|
let switchedToFailureShot = false;
|
||||||
|
|
||||||
|
for (const e of events) {
|
||||||
|
const act = e.act;
|
||||||
|
const explicitPre = extractItemPreImage(act);
|
||||||
|
const explicitPost = extractItemPostImage(act);
|
||||||
|
|
||||||
|
const agentName = String(
|
||||||
|
act?.agent_name || act?.agent || safeParseJson(act?.payload)?.agent_name || act?.name || ''
|
||||||
|
).toLowerCase();
|
||||||
|
const isRecoveryOrValidator = agentName.includes('failure') || agentName.includes('analyzer') || agentName.includes('validator') || agentName.includes('diagnos');
|
||||||
|
|
||||||
|
if (failureShot && !switchedToFailureShot && (isRecoveryOrValidator || (!e.isPrimary && hadPrimaryFailure))) {
|
||||||
|
currentPreImage = failureShot;
|
||||||
|
switchedToFailureShot = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
const resolvedPre = explicitPre || currentPreImage;
|
||||||
|
let resolvedPost: string | null = null;
|
||||||
|
let hasExplicitPost = false;
|
||||||
|
|
||||||
|
if (explicitPost) {
|
||||||
|
resolvedPost = explicitPost;
|
||||||
|
hasExplicitPost = true;
|
||||||
|
} else if (e.isPrimary && !hadPrimaryFailure && !hasSubsequentActions) {
|
||||||
|
resolvedPost = stepData.post_image_name || stepData.post_screenshot || null;
|
||||||
|
} else if (e.item === lastActionItem && !hadPrimaryFailure) {
|
||||||
|
resolvedPost = stepData.post_image_name || stepData.post_screenshot || null;
|
||||||
|
} else if (e.item === lastActionItem && isRecoveryOrValidator) {
|
||||||
|
resolvedPost = stepData.post_image_name || stepData.post_screenshot || null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// "下一次的决策前是上一次的动作后"
|
||||||
|
if (resolvedPost) {
|
||||||
|
currentPreImage = resolvedPost;
|
||||||
|
}
|
||||||
|
|
||||||
|
const entry: ResolvedImageEntry = { pre: resolvedPre, post: resolvedPost, hasExplicitPost };
|
||||||
|
map.set(e.item, entry);
|
||||||
|
if (act && act !== e.item) map.set(act, entry);
|
||||||
|
if (act?.trace_id) map.set(act.trace_id, entry);
|
||||||
|
}
|
||||||
|
|
||||||
|
stepImageMapCache.set(stepData, { key: cacheKey, map });
|
||||||
|
return map;
|
||||||
|
}
|
||||||
|
|
||||||
|
function lookupResolvedImages(stepData: any, actionData?: any): ResolvedImageEntry {
|
||||||
|
const map = resolveStepImageMap(stepData);
|
||||||
|
if (!actionData) {
|
||||||
|
return {
|
||||||
|
pre: stepData?.pre_image_name || stepData?.pre_screenshot || null,
|
||||||
|
post: stepData?.post_image_name || stepData?.post_screenshot || null,
|
||||||
|
hasExplicitPost: Boolean(stepData?.post_image_name || stepData?.post_screenshot)
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const act = getActionObject(actionData);
|
||||||
|
|
||||||
|
if (map.has(actionData)) return map.get(actionData)!;
|
||||||
|
if (act && map.has(act)) return map.get(act)!;
|
||||||
|
if (act?.trace_id && map.has(act.trace_id)) return map.get(act.trace_id)!;
|
||||||
|
|
||||||
|
const explicitPre = extractItemPreImage(act);
|
||||||
|
const explicitPost = extractItemPostImage(act);
|
||||||
|
const pre = explicitPre || stepData?.pre_image_name || stepData?.pre_screenshot || null;
|
||||||
|
const post = explicitPost || (isGenericToolTrace(actionData, stepData) ? null : (stepData?.post_image_name || stepData?.post_screenshot || null));
|
||||||
|
return { pre, post, hasExplicitPost: Boolean(explicitPost) };
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Get pre-action screenshot URL
|
* Get pre-action screenshot URL
|
||||||
*/
|
*/
|
||||||
export function getStepPreImageUrl(stepData: any, actionData?: any): string | null {
|
export function getStepPreImageUrl(stepData: any, actionData?: any): string | null {
|
||||||
const act = getActionObject(actionData);
|
const resolved = lookupResolvedImages(stepData, actionData);
|
||||||
const candidate = act?.pre_image_name || act?.pre_screenshot || act?.before_screenshot || act?.screenshot || stepData?.pre_image_name || stepData?.pre_screenshot;
|
return formatImageUrl(resolved.pre);
|
||||||
return formatImageUrl(candidate);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Get post-action screenshot URL
|
* Get post-action screenshot URL
|
||||||
*/
|
*/
|
||||||
export function getStepPostImageUrl(stepData: any, actionData?: any): string | null {
|
export function getStepPostImageUrl(stepData: any, actionData?: any): string | null {
|
||||||
const act = getActionObject(actionData);
|
const resolved = lookupResolvedImages(stepData, actionData);
|
||||||
const candidate = act?.post_image_name || act?.post_screenshot || act?.after_screenshot || stepData?.post_image_name || stepData?.post_screenshot;
|
const postUrl = formatImageUrl(resolved.post);
|
||||||
const postUrl = formatImageUrl(candidate);
|
const preUrl = formatImageUrl(resolved.pre);
|
||||||
const preUrl = getStepPreImageUrl(stepData, actionData);
|
|
||||||
if (postUrl && preUrl && postUrl === preUrl) {
|
if (postUrl && preUrl && postUrl === preUrl && !resolved.hasExplicitPost) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
return postUrl;
|
return postUrl;
|
||||||
+190
-30
@@ -77,6 +77,43 @@ export function extractNumbersFromCoordinateValue(val: any): number[] | null {
|
|||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Parse sequence of coordinates (e.g. [[x1, y1], [x2, y2]] or stringified format)
|
||||||
|
*/
|
||||||
|
export function parseSequenceCoordinates(seq: any): number[][] | null {
|
||||||
|
if (!seq) return null;
|
||||||
|
if (typeof seq === 'string') {
|
||||||
|
const trimmed = seq.trim();
|
||||||
|
try {
|
||||||
|
const parsed = JSON.parse(trimmed);
|
||||||
|
const res = parseSequenceCoordinates(parsed);
|
||||||
|
if (res && res.length > 0) return res;
|
||||||
|
} catch {}
|
||||||
|
|
||||||
|
const pairs: number[][] = [];
|
||||||
|
const pairRegex = /\[\s*(-?\d+(?:\.\d+)?)\s*,\s*(-?\d+(?:\.\d+)?)\s*\]/g;
|
||||||
|
let m: RegExpExecArray | null;
|
||||||
|
while ((m = pairRegex.exec(trimmed)) !== null) {
|
||||||
|
pairs.push([Number(m[1]), Number(m[2])]);
|
||||||
|
}
|
||||||
|
if (pairs.length > 0) return pairs;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (Array.isArray(seq)) {
|
||||||
|
const points: number[][] = [];
|
||||||
|
for (const item of seq) {
|
||||||
|
const p = extractNumbersFromCoordinateValue(item);
|
||||||
|
if (p && p.length === 2) {
|
||||||
|
points.push([p[0], p[1]]);
|
||||||
|
} else if (Array.isArray(item) && item.length >= 2 && typeof item[0] === 'number' && typeof item[1] === 'number') {
|
||||||
|
points.push([item[0], item[1]]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (points.length > 0) return points;
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
export function isPureDirectionString(str: any): boolean {
|
export function isPureDirectionString(str: any): boolean {
|
||||||
if (typeof str !== 'string') return false;
|
if (typeof str !== 'string') return false;
|
||||||
const s = str.trim().toLowerCase();
|
const s = str.trim().toLowerCase();
|
||||||
@@ -121,10 +158,42 @@ export function drawActionCoordinatesOnOverlay(
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 1. Multi-point click sequence (High priority for click_sequence / tap_sequence)
|
||||||
|
const isSequenceAction = actionName.includes('sequence') || Boolean(act.sequence || args.sequence || act.normalized_sequence || args.normalized_sequence);
|
||||||
|
const rawSeq = act.normalized_sequence || args.normalized_sequence || act.sequence || args.sequence || act.targets || args.targets || (Array.isArray(act.coordinates) && Array.isArray(act.coordinates[0]) ? act.coordinates : null) || (Array.isArray(args.coordinates) && Array.isArray(args.coordinates[0]) ? args.coordinates : null);
|
||||||
|
const seqPoints = parseSequenceCoordinates(rawSeq);
|
||||||
|
|
||||||
|
if (seqPoints && seqPoints.length > 0 && (isSequenceAction || seqPoints.length > 1)) {
|
||||||
|
const isNormSeq = Boolean(act.normalized_sequence || args.normalized_sequence) || seqPoints.every(p => p[0] <= 1000 && p[1] <= 1000);
|
||||||
|
// Draw connecting lines between consecutive points
|
||||||
|
for (let i = 0; i < seqPoints.length - 1; i++) {
|
||||||
|
createSequenceConnector(
|
||||||
|
seqPoints[i][0], seqPoints[i][1],
|
||||||
|
seqPoints[i + 1][0], seqPoints[i + 1][1],
|
||||||
|
naturalWidth, naturalHeight,
|
||||||
|
overlay,
|
||||||
|
isNormSeq
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// Draw numbered markers for each point in the sequence
|
||||||
|
seqPoints.forEach((pt, idx) => {
|
||||||
|
createPointMarker(
|
||||||
|
pt[0], pt[1],
|
||||||
|
actionName,
|
||||||
|
naturalWidth, naturalHeight,
|
||||||
|
overlay,
|
||||||
|
isNormSeq,
|
||||||
|
idx + 1,
|
||||||
|
seqPoints.length
|
||||||
|
);
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
// Determine if this is a swipe/scroll/drag gesture
|
// Determine if this is a swipe/scroll/drag gesture
|
||||||
const isSwipeLike = actionName.includes('swipe') || actionName.includes('scroll') || actionName.includes('drag') || actionName.includes('slide');
|
const isSwipeLike = actionName.includes('swipe') || actionName.includes('scroll') || actionName.includes('drag') || actionName.includes('slide');
|
||||||
|
|
||||||
// 1. Check separate start and end coordinate objects
|
// 2. Check separate start and end coordinate objects
|
||||||
const normStart = extractNumbersFromCoordinateValue(act.normalized_start_coordinates || args.normalized_start_coordinates);
|
const normStart = extractNumbersFromCoordinateValue(act.normalized_start_coordinates || args.normalized_start_coordinates);
|
||||||
const normEnd = extractNumbersFromCoordinateValue(act.normalized_end_coordinates || args.normalized_end_coordinates);
|
const normEnd = extractNumbersFromCoordinateValue(act.normalized_end_coordinates || args.normalized_end_coordinates);
|
||||||
const startCoords = normStart || extractNumbersFromCoordinateValue(act.start_coordinates || act.start_point || act.start || act.from ||
|
const startCoords = normStart || extractNumbersFromCoordinateValue(act.start_coordinates || act.start_point || act.start || act.from ||
|
||||||
@@ -144,7 +213,7 @@ export function drawActionCoordinatesOnOverlay(
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
// 2. Check 4-element coordinate array in coordinates, target, action, gesture, etc.
|
// 3. Check 4-element coordinate array in coordinates, target, action, gesture, etc.
|
||||||
const normCoords = extractNumbersFromCoordinateValue(act.normalized_coordinates || args.normalized_coordinates);
|
const normCoords = extractNumbersFromCoordinateValue(act.normalized_coordinates || args.normalized_coordinates);
|
||||||
const isNorm = Boolean(normCoords);
|
const isNorm = Boolean(normCoords);
|
||||||
const rawCoords = normCoords ||
|
const rawCoords = normCoords ||
|
||||||
@@ -154,8 +223,6 @@ export function drawActionCoordinatesOnOverlay(
|
|||||||
extractNumbersFromCoordinateValue(args.coords) ||
|
extractNumbersFromCoordinateValue(args.coords) ||
|
||||||
extractNumbersFromCoordinateValue(act.target) ||
|
extractNumbersFromCoordinateValue(act.target) ||
|
||||||
extractNumbersFromCoordinateValue(args.target) ||
|
extractNumbersFromCoordinateValue(args.target) ||
|
||||||
extractNumbersFromCoordinateValue(act.sequence) ||
|
|
||||||
extractNumbersFromCoordinateValue(args.sequence) ||
|
|
||||||
(isSwipeLike ? extractNumbersFromCoordinateValue(args.action) : null) ||
|
(isSwipeLike ? extractNumbersFromCoordinateValue(args.action) : null) ||
|
||||||
(isSwipeLike ? extractNumbersFromCoordinateValue(act.action) : null) ||
|
(isSwipeLike ? extractNumbersFromCoordinateValue(act.action) : null) ||
|
||||||
(isSwipeLike ? extractNumbersFromCoordinateValue(args.gesture) : null) ||
|
(isSwipeLike ? extractNumbersFromCoordinateValue(args.gesture) : null) ||
|
||||||
@@ -177,21 +244,7 @@ export function drawActionCoordinatesOnOverlay(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Multi-point click sequence
|
// 4. Check for directional swipe/scroll/drag gestures ONLY when action is swipe-like
|
||||||
const seq = act.sequence || args.sequence || (Array.isArray(act.target) && Array.isArray(act.target[0]) ? act.target : null);
|
|
||||||
if (seq && Array.isArray(seq)) {
|
|
||||||
let drawnAny = false;
|
|
||||||
seq.forEach((pt: any) => {
|
|
||||||
const p = extractNumbersFromCoordinateValue(pt);
|
|
||||||
if (p && p.length === 2) {
|
|
||||||
createPointMarker(p[0], p[1], actionName, naturalWidth, naturalHeight, overlay, isNorm);
|
|
||||||
drawnAny = true;
|
|
||||||
}
|
|
||||||
});
|
|
||||||
if (drawnAny) return;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 3. Check for directional swipe/scroll/drag gestures ONLY when action is swipe-like
|
|
||||||
if (isSwipeLike) {
|
if (isSwipeLike) {
|
||||||
const rawDir = (typeof args.action === 'string' && isPureDirectionString(args.action) ? args.action : null) ||
|
const rawDir = (typeof args.action === 'string' && isPureDirectionString(args.action) ? args.action : null) ||
|
||||||
(typeof act.action === 'string' && act.action !== actionName && isPureDirectionString(act.action) ? act.action : null) ||
|
(typeof act.action === 'string' && act.action !== actionName && isPureDirectionString(act.action) ? act.action : null) ||
|
||||||
@@ -224,7 +277,7 @@ export function drawActionCoordinatesOnOverlay(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// 4. Default tap/click coordinate fallback
|
// 5. Default tap/click coordinate fallback
|
||||||
if (rawCoords && Array.isArray(rawCoords) && rawCoords.length === 2) {
|
if (rawCoords && Array.isArray(rawCoords) && rawCoords.length === 2) {
|
||||||
createPointMarker(rawCoords[0], rawCoords[1], actionName, naturalWidth, naturalHeight, overlay, isNorm);
|
createPointMarker(rawCoords[0], rawCoords[1], actionName, naturalWidth, naturalHeight, overlay, isNorm);
|
||||||
}
|
}
|
||||||
@@ -241,7 +294,9 @@ export function createPointMarker(
|
|||||||
naturalWidth: number,
|
naturalWidth: number,
|
||||||
naturalHeight: number,
|
naturalHeight: number,
|
||||||
overlay: HTMLElement,
|
overlay: HTMLElement,
|
||||||
isNormalized: boolean = false
|
isNormalized: boolean = false,
|
||||||
|
sequenceIndex?: number,
|
||||||
|
totalSequence?: number
|
||||||
): void {
|
): void {
|
||||||
let pctX: number;
|
let pctX: number;
|
||||||
let pctY: number;
|
let pctY: number;
|
||||||
@@ -268,27 +323,42 @@ export function createPointMarker(
|
|||||||
marker.style.pointerEvents = 'none';
|
marker.style.pointerEvents = 'none';
|
||||||
marker.style.zIndex = '10';
|
marker.style.zIndex = '10';
|
||||||
|
|
||||||
|
const dotSize = sequenceIndex !== undefined ? 20 : 14;
|
||||||
const dot = document.createElement('div');
|
const dot = document.createElement('div');
|
||||||
dot.style.width = '14px';
|
dot.style.width = `${dotSize}px`;
|
||||||
dot.style.height = '14px';
|
dot.style.height = `${dotSize}px`;
|
||||||
dot.style.borderRadius = '50%';
|
dot.style.borderRadius = '50%';
|
||||||
dot.style.backgroundColor = '#ef4444';
|
dot.style.backgroundColor = '#ef4444';
|
||||||
dot.style.border = '2px solid white';
|
dot.style.border = '2px solid white';
|
||||||
dot.style.boxShadow = '0 0 6px rgba(0,0,0,0.5)';
|
dot.style.boxShadow = '0 0 6px rgba(0,0,0,0.5)';
|
||||||
|
if (sequenceIndex !== undefined) {
|
||||||
|
dot.style.display = 'flex';
|
||||||
|
dot.style.alignItems = 'center';
|
||||||
|
dot.style.justifyContent = 'center';
|
||||||
|
dot.style.fontSize = '11px';
|
||||||
|
dot.style.fontWeight = 'bold';
|
||||||
|
dot.style.color = 'white';
|
||||||
|
dot.innerText = String(sequenceIndex);
|
||||||
|
}
|
||||||
|
|
||||||
|
const pulseSize = dotSize * 2;
|
||||||
const pulse = document.createElement('div');
|
const pulse = document.createElement('div');
|
||||||
pulse.style.position = 'absolute';
|
pulse.style.position = 'absolute';
|
||||||
pulse.style.top = '-9px';
|
pulse.style.top = `-${dotSize / 2 + 2}px`;
|
||||||
pulse.style.left = '-9px';
|
pulse.style.left = `-${dotSize / 2 + 2}px`;
|
||||||
pulse.style.width = '28px';
|
pulse.style.width = `${pulseSize}px`;
|
||||||
pulse.style.height = '28px';
|
pulse.style.height = `${pulseSize}px`;
|
||||||
pulse.style.borderRadius = '50%';
|
pulse.style.borderRadius = '50%';
|
||||||
pulse.style.border = '2px solid #ef4444';
|
pulse.style.border = '2px solid #ef4444';
|
||||||
pulse.style.animation = 'pulse-animation 1.5s infinite ease-out';
|
pulse.style.animation = 'pulse-animation 1.5s infinite ease-out';
|
||||||
pulse.style.pointerEvents = 'none';
|
pulse.style.pointerEvents = 'none';
|
||||||
|
|
||||||
const label = document.createElement('div');
|
const label = document.createElement('div');
|
||||||
label.innerText = actionType.toUpperCase();
|
if (sequenceIndex !== undefined) {
|
||||||
|
label.innerText = totalSequence && totalSequence > 1 ? `#${sequenceIndex} TAP` : actionType.toUpperCase();
|
||||||
|
} else {
|
||||||
|
label.innerText = actionType.toUpperCase();
|
||||||
|
}
|
||||||
label.style.position = 'absolute';
|
label.style.position = 'absolute';
|
||||||
label.style.backgroundColor = 'rgba(15, 23, 42, 0.9)';
|
label.style.backgroundColor = 'rgba(15, 23, 42, 0.9)';
|
||||||
label.style.color = 'white';
|
label.style.color = 'white';
|
||||||
@@ -303,10 +373,10 @@ export function createPointMarker(
|
|||||||
|
|
||||||
// Smart edge-aware vertical positioning
|
// Smart edge-aware vertical positioning
|
||||||
if (clampedPctY > 85) {
|
if (clampedPctY > 85) {
|
||||||
label.style.bottom = '16px';
|
label.style.bottom = `${dotSize + 2}px`;
|
||||||
label.style.top = 'auto';
|
label.style.top = 'auto';
|
||||||
} else {
|
} else {
|
||||||
label.style.top = '16px';
|
label.style.top = `${dotSize + 2}px`;
|
||||||
label.style.bottom = 'auto';
|
label.style.bottom = 'auto';
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -331,6 +401,96 @@ export function createPointMarker(
|
|||||||
overlay.appendChild(marker);
|
overlay.appendChild(marker);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Helper to draw connecting line/arrow between points in a click sequence
|
||||||
|
*/
|
||||||
|
export function createSequenceConnector(
|
||||||
|
x1: number,
|
||||||
|
y1: number,
|
||||||
|
x2: number,
|
||||||
|
y2: number,
|
||||||
|
naturalWidth: number,
|
||||||
|
naturalHeight: number,
|
||||||
|
overlay: HTMLElement,
|
||||||
|
isNormalized: boolean = false
|
||||||
|
): void {
|
||||||
|
const isNormCoord = isNormalized || (x1 <= 1000 && y1 <= 1000 && x2 <= 1000 && y2 <= 1000);
|
||||||
|
const pctX1 = isNormCoord ? x1 / 10 : (x1 / (naturalWidth || 1080)) * 100;
|
||||||
|
const pctY1 = isNormCoord ? y1 / 10 : (y1 / (naturalHeight || 2400)) * 100;
|
||||||
|
const pctX2 = isNormCoord ? x2 / 10 : (x2 / (naturalWidth || 1080)) * 100;
|
||||||
|
const pctY2 = isNormCoord ? y2 / 10 : (y2 / (naturalHeight || 2400)) * 100;
|
||||||
|
|
||||||
|
const clampedX1 = Math.max(1.5, Math.min(98.5, pctX1));
|
||||||
|
const clampedY1 = Math.max(1.5, Math.min(98.5, pctY1));
|
||||||
|
const clampedX2 = Math.max(1.5, Math.min(98.5, pctX2));
|
||||||
|
const clampedY2 = Math.max(1.5, Math.min(98.5, pctY2));
|
||||||
|
|
||||||
|
let svg = overlay.querySelector('svg');
|
||||||
|
if (!svg) {
|
||||||
|
svg = document.createElementNS('http://www.w3.org/2000/svg', 'svg');
|
||||||
|
svg.setAttribute('width', '100%');
|
||||||
|
svg.setAttribute('height', '100%');
|
||||||
|
svg.style.position = 'absolute';
|
||||||
|
svg.style.top = '0';
|
||||||
|
svg.style.left = '0';
|
||||||
|
svg.style.pointerEvents = 'none';
|
||||||
|
svg.style.overflow = 'visible';
|
||||||
|
svg.style.zIndex = '5';
|
||||||
|
overlay.appendChild(svg);
|
||||||
|
}
|
||||||
|
|
||||||
|
let defs = svg.querySelector('defs');
|
||||||
|
if (!defs) {
|
||||||
|
defs = document.createElementNS('http://www.w3.org/2000/svg', 'defs');
|
||||||
|
svg.insertBefore(defs, svg.firstChild);
|
||||||
|
}
|
||||||
|
|
||||||
|
const markerId = 'seq-arrow-' + Math.random().toString(36).substring(2, 9);
|
||||||
|
const marker = document.createElementNS('http://www.w3.org/2000/svg', 'marker');
|
||||||
|
marker.setAttribute('id', markerId);
|
||||||
|
marker.setAttribute('viewBox', '0 0 12 12');
|
||||||
|
marker.setAttribute('refX', '8');
|
||||||
|
marker.setAttribute('refY', '6');
|
||||||
|
marker.setAttribute('markerWidth', '6');
|
||||||
|
marker.setAttribute('markerHeight', '6');
|
||||||
|
marker.setAttribute('orient', 'auto');
|
||||||
|
|
||||||
|
const path = document.createElementNS('http://www.w3.org/2000/svg', 'path');
|
||||||
|
path.setAttribute('d', 'M 1 2 L 10 6 L 1 10 z');
|
||||||
|
path.setAttribute('fill', '#ef4444');
|
||||||
|
path.setAttribute('stroke', '#ffffff');
|
||||||
|
path.setAttribute('stroke-width', '1');
|
||||||
|
|
||||||
|
marker.appendChild(path);
|
||||||
|
defs.appendChild(marker);
|
||||||
|
|
||||||
|
// Background white contrast line
|
||||||
|
const bgLine = document.createElementNS('http://www.w3.org/2000/svg', 'line');
|
||||||
|
bgLine.setAttribute('x1', `${clampedX1}%`);
|
||||||
|
bgLine.setAttribute('y1', `${clampedY1}%`);
|
||||||
|
bgLine.setAttribute('x2', `${clampedX2}%`);
|
||||||
|
bgLine.setAttribute('y2', `${clampedY2}%`);
|
||||||
|
bgLine.setAttribute('stroke', '#ffffff');
|
||||||
|
bgLine.setAttribute('stroke-width', '4');
|
||||||
|
bgLine.setAttribute('stroke-linecap', 'round');
|
||||||
|
bgLine.setAttribute('opacity', '0.8');
|
||||||
|
|
||||||
|
// Main red dashed sequence line
|
||||||
|
const line = document.createElementNS('http://www.w3.org/2000/svg', 'line');
|
||||||
|
line.setAttribute('x1', `${clampedX1}%`);
|
||||||
|
line.setAttribute('y1', `${clampedY1}%`);
|
||||||
|
line.setAttribute('x2', `${clampedX2}%`);
|
||||||
|
line.setAttribute('y2', `${clampedY2}%`);
|
||||||
|
line.setAttribute('stroke', '#ef4444');
|
||||||
|
line.setAttribute('stroke-width', '2.5');
|
||||||
|
line.setAttribute('stroke-linecap', 'round');
|
||||||
|
line.setAttribute('marker-end', `url(#${markerId})`);
|
||||||
|
line.setAttribute('stroke-dasharray', '4,4');
|
||||||
|
|
||||||
|
svg.appendChild(bgLine);
|
||||||
|
svg.appendChild(line);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Helper to create and position an SVG arrow, high-contrast dashed line, and badge for swipes/drags
|
* Helper to create and position an SVG arrow, high-contrast dashed line, and badge for swipes/drags
|
||||||
*/
|
*/
|
||||||
+118
-34
@@ -178,6 +178,27 @@ export function extractCheckerResult(text: string): CheckerResult | null {
|
|||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Helper to format inline markdown segments (bold, italic, code, strikethrough)
|
||||||
|
*/
|
||||||
|
function formatInlineMarkdown(text: string): string {
|
||||||
|
if (!text) return '';
|
||||||
|
return text
|
||||||
|
// Inline code first (so asterisks inside code aren't formatted)
|
||||||
|
.replace(/`([^`]+)`/g, '<code class="inline-code">$1</code>')
|
||||||
|
// Bold + Italic
|
||||||
|
.replace(/\*\*\*([^*]+)\*\*\*/g, '<strong><em>$1</em></strong>')
|
||||||
|
.replace(/___([^_]+)___/g, '<strong><em>$1</em></strong>')
|
||||||
|
// Bold
|
||||||
|
.replace(/\*\*([^*]+)\*\*/g, '<strong>$1</strong>')
|
||||||
|
.replace(/__([^_]+)__/g, '<strong>$1</strong>')
|
||||||
|
// Italic
|
||||||
|
.replace(/\*([^*]+)\*/g, '<em>$1</em>')
|
||||||
|
.replace(/\b_([^_]+)_\b/g, '<em>$1</em>')
|
||||||
|
// Strikethrough
|
||||||
|
.replace(/~~([^~]+)~~/g, '<del>$1</del>');
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Convert basic markdown text to safe HTML for agent logs/thinking
|
* Convert basic markdown text to safe HTML for agent logs/thinking
|
||||||
*/
|
*/
|
||||||
@@ -187,69 +208,132 @@ export function renderMarkdownToHtml(text: string): string {
|
|||||||
// Clean up internal agent XML-like tags (e.g. <thought>, <short_term_memory>, etc.)
|
// Clean up internal agent XML-like tags (e.g. <thought>, <short_term_memory>, etc.)
|
||||||
const cleanedText = text
|
const cleanedText = text
|
||||||
.replace(/<\/?(thought|short_term_memory|strong_term_memory|reasoning|plan|task_plan|call_tool)[^\n>]*>?/gi, '')
|
.replace(/<\/?(thought|short_term_memory|strong_term_memory|reasoning|plan|task_plan|call_tool)[^\n>]*>?/gi, '')
|
||||||
|
.replace(/\r\n/g, '\n')
|
||||||
|
.replace(/\r/g, '\n')
|
||||||
.replace(/\n{3,}/g, '\n\n')
|
.replace(/\n{3,}/g, '\n\n')
|
||||||
.trim();
|
.trim();
|
||||||
|
|
||||||
// 1. Escape HTML to prevent XSS
|
// Escape HTML first to prevent XSS
|
||||||
let html = cleanedText
|
const escaped = cleanedText
|
||||||
.replace(/&/g, '&')
|
.replace(/&/g, '&')
|
||||||
.replace(/</g, '<')
|
.replace(/</g, '<')
|
||||||
.replace(/>/g, '>');
|
.replace(/>/g, '>');
|
||||||
|
|
||||||
// 2. Bold: **text** -> <strong>text</strong>
|
const rawLines = escaped.split('\n');
|
||||||
html = html.replace(/\*\*(.*?)\*\*/g, '<strong>$1</strong>');
|
const result: string[] = [];
|
||||||
|
|
||||||
// 3. Inline code: `code` -> <code class="inline-code">code</code>
|
|
||||||
html = html.replace(/`(.*?)`/g, '<code class="inline-code">$1</code>');
|
|
||||||
|
|
||||||
// 4. Lines processing for lists
|
|
||||||
const lines = html.split('\n');
|
|
||||||
let output = '';
|
|
||||||
let currentListType: 'ul' | 'ol' | null = null;
|
let currentListType: 'ul' | 'ol' | null = null;
|
||||||
|
let inCodeBlock = false;
|
||||||
|
let codeBlockLang = '';
|
||||||
|
let codeBlockLines: string[] = [];
|
||||||
|
|
||||||
for (const line of lines) {
|
const closeList = () => {
|
||||||
const trimmed = line.trim();
|
if (currentListType) {
|
||||||
const ulMatch = line.match(/^(\s*)([\*\-])\s+(.*)$/);
|
result.push(`</${currentListType}>`);
|
||||||
const olMatch = line.match(/^(\s*)(\d+)\.\s+(.*)$/);
|
currentListType = null;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
for (let i = 0; i < rawLines.length; i++) {
|
||||||
|
const rawLine = rawLines[i];
|
||||||
|
const trimmed = rawLine.trim();
|
||||||
|
|
||||||
|
// 1. Code blocks: ```lang ... ```
|
||||||
|
if (trimmed.startsWith('```')) {
|
||||||
|
if (inCodeBlock) {
|
||||||
|
// End code block
|
||||||
|
const codeContent = codeBlockLines.join('\n');
|
||||||
|
result.push(`<pre class="code-block"><code class="lang-${codeBlockLang}">${codeContent}</code></pre>`);
|
||||||
|
inCodeBlock = false;
|
||||||
|
codeBlockLines = [];
|
||||||
|
codeBlockLang = '';
|
||||||
|
} else {
|
||||||
|
closeList();
|
||||||
|
inCodeBlock = true;
|
||||||
|
codeBlockLang = trimmed.substring(3).trim();
|
||||||
|
codeBlockLines = [];
|
||||||
|
}
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (inCodeBlock) {
|
||||||
|
codeBlockLines.push(rawLine);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. Headings: # to ######
|
||||||
|
const headingMatch = trimmed.match(/^(#{1,6})\s+(.*)$/);
|
||||||
|
if (headingMatch) {
|
||||||
|
closeList();
|
||||||
|
const level = headingMatch[1].length;
|
||||||
|
const headingContent = formatInlineMarkdown(headingMatch[2]);
|
||||||
|
result.push(`<h${level}>${headingContent}</h${level}>`);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3. Horizontal Rule: ---, ***, ___
|
||||||
|
if (/^(\-{3,}|\*{3,}|_{3,})$/.test(trimmed)) {
|
||||||
|
closeList();
|
||||||
|
result.push('<hr class="md-hr" />');
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 4. Blockquotes: > quote
|
||||||
|
if (trimmed.startsWith('> ') || trimmed.startsWith('>')) {
|
||||||
|
closeList();
|
||||||
|
const quoteContent = formatInlineMarkdown(trimmed.replace(/^>\s?/, ''));
|
||||||
|
result.push(`<blockquote>${quoteContent}</blockquote>`);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 5. Unordered List: - item or * item
|
||||||
|
const ulMatch = rawLine.match(/^(\s*)([\*\-])\s+(.*)$/);
|
||||||
if (ulMatch) {
|
if (ulMatch) {
|
||||||
if (currentListType !== 'ul') {
|
if (currentListType !== 'ul') {
|
||||||
if (currentListType) output += `</${currentListType}>\n`;
|
closeList();
|
||||||
currentListType = 'ul';
|
currentListType = 'ul';
|
||||||
output += '<ul>\n';
|
result.push('<ul>');
|
||||||
}
|
}
|
||||||
const spaces = ulMatch[1].length;
|
const spaces = ulMatch[1].length;
|
||||||
const subLevel = Math.max(0, Math.floor((spaces - 2) / 2));
|
const subLevel = Math.max(0, Math.floor((spaces - 2) / 2));
|
||||||
const indent = subLevel * 12;
|
const indent = subLevel * 12;
|
||||||
const style = indent ? ` style="margin-left: ${indent}px"` : '';
|
const style = indent ? ` style="margin-left: ${indent}px"` : '';
|
||||||
output += `<li${style}>${ulMatch[3]}</li>\n`;
|
const itemContent = formatInlineMarkdown(ulMatch[3]);
|
||||||
} else if (olMatch) {
|
result.push(`<li${style}>${itemContent}</li>`);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 6. Ordered List: 1. item
|
||||||
|
const olMatch = rawLine.match(/^(\s*)(\d+)\.\s+(.*)$/);
|
||||||
|
if (olMatch) {
|
||||||
if (currentListType !== 'ol') {
|
if (currentListType !== 'ol') {
|
||||||
if (currentListType) output += `</${currentListType}>\n`;
|
closeList();
|
||||||
currentListType = 'ol';
|
currentListType = 'ol';
|
||||||
output += '<ol>\n';
|
result.push('<ol>');
|
||||||
}
|
}
|
||||||
const spaces = olMatch[1].length;
|
const spaces = olMatch[1].length;
|
||||||
const subLevel = Math.max(0, Math.floor((spaces - 2) / 2));
|
const subLevel = Math.max(0, Math.floor((spaces - 2) / 2));
|
||||||
const indent = subLevel * 12;
|
const indent = subLevel * 12;
|
||||||
const style = indent ? ` style="margin-left: ${indent}px"` : '';
|
const style = indent ? ` style="margin-left: ${indent}px"` : '';
|
||||||
output += `<li${style}>${olMatch[3]}</li>\n`;
|
const itemContent = formatInlineMarkdown(olMatch[3]);
|
||||||
|
result.push(`<li${style}>${itemContent}</li>`);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 7. Regular paragraph / empty line
|
||||||
|
closeList();
|
||||||
|
if (trimmed === '') {
|
||||||
|
result.push('<div class="md-para-gap"></div>');
|
||||||
} else {
|
} else {
|
||||||
if (currentListType) {
|
const lineContent = formatInlineMarkdown(rawLine);
|
||||||
output += `</${currentListType}>\n`;
|
result.push(`<div>${lineContent}</div>`);
|
||||||
currentListType = null;
|
|
||||||
}
|
|
||||||
if (trimmed === '') {
|
|
||||||
output += '<div class="md-para-gap"></div>';
|
|
||||||
} else {
|
|
||||||
output += `<div>${line}</div>\n`;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (currentListType) {
|
if (inCodeBlock) {
|
||||||
output += `</${currentListType}>\n`;
|
const codeContent = codeBlockLines.join('\n');
|
||||||
|
result.push(`<pre class="code-block"><code class="lang-${codeBlockLang}">${codeContent}</code></pre>`);
|
||||||
}
|
}
|
||||||
|
|
||||||
return output;
|
closeList();
|
||||||
|
return result.join('\n');
|
||||||
}
|
}
|
||||||
+63
-24
@@ -15,7 +15,7 @@
|
|||||||
*/
|
*/
|
||||||
|
|
||||||
import { StepBlock, PhaseBlock, StepEvent } from '../core/models/stream.model';
|
import { StepBlock, PhaseBlock, StepEvent } from '../core/models/stream.model';
|
||||||
import { isAndroidAction } from './action-formatter.util';
|
import { isAndroidAction, isReportStatusAction, getReportStatusExplanation, getReportStatusValue } from './action-formatter.util';
|
||||||
import { getUniqueGenericTools } from './tool-formatter.util';
|
import { getUniqueGenericTools } from './tool-formatter.util';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -121,7 +121,10 @@ export function consolidateLogsToBlocks(rawLogs: any[]): StepBlock[] {
|
|||||||
const newTools = log.data.generic_tools || [];
|
const newTools = log.data.generic_tools || [];
|
||||||
const mergedTools = [...existingTools];
|
const mergedTools = [...existingTools];
|
||||||
newTools.forEach((nt: any) => {
|
newTools.forEach((nt: any) => {
|
||||||
if (!mergedTools.some(et => et.trace_id && nt.trace_id && et.trace_id === nt.trace_id)) {
|
const matchIdx = mergedTools.findIndex(et => et.trace_id && nt.trace_id && et.trace_id === nt.trace_id);
|
||||||
|
if (matchIdx > -1) {
|
||||||
|
mergedTools[matchIdx] = { ...mergedTools[matchIdx], ...nt };
|
||||||
|
} else {
|
||||||
mergedTools.push(nt);
|
mergedTools.push(nt);
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
@@ -157,44 +160,68 @@ export function consolidateLogsToBlocks(rawLogs: any[]): StepBlock[] {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
} else if (log.type === 'trace_recorded') {
|
} else if (log.type === 'trace_recorded') {
|
||||||
const isTool = log.data.type === 'tool';
|
const isAction = log.data.type === 'action';
|
||||||
|
const isTool = log.data.type === 'tool' || isAction;
|
||||||
const isFailedLLM = log.data.type === 'llm_call' && log.data.status === 'failed';
|
const isFailedLLM = log.data.type === 'llm_call' && log.data.status === 'failed';
|
||||||
if (!isTool && !isFailedLLM) return;
|
if (!isTool && !isFailedLLM) return;
|
||||||
|
|
||||||
let stepId = log.data.step_id;
|
let stepId = log.data.step_id;
|
||||||
let existingIndex = -1;
|
let existingIndex = -1;
|
||||||
if (stepId) {
|
|
||||||
|
// 1. If this trace_id already exists in some block, update that block directly
|
||||||
|
if (log.data.trace_id) {
|
||||||
|
existingIndex = blocks.findIndex(b =>
|
||||||
|
b.data?.generic_tools?.some((t: any) => t.trace_id === log.data.trace_id)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. If not found by trace_id and stepId is present, find by step_id
|
||||||
|
if (existingIndex === -1 && stepId) {
|
||||||
existingIndex = blocks.findIndex(b => b.id === `step-${stepId}` || b.data?.step_id === stepId);
|
existingIndex = blocks.findIndex(b => b.id === `step-${stepId}` || b.data?.step_id === stepId);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 3. If still not found and no stepId (e.g. pre-planning / initial planning phase)
|
||||||
|
if (existingIndex === -1 && !stepId) {
|
||||||
|
existingIndex = blocks.findIndex(b => b.id === 'step-pre-planning' || b.data?.step_id === 'pre-planning');
|
||||||
|
}
|
||||||
|
|
||||||
|
// 4. Otherwise find by timestamp among existing step blocks
|
||||||
if (existingIndex === -1) {
|
if (existingIndex === -1) {
|
||||||
const logTime = log.timestamp ? new Date(log.timestamp).getTime() : 0;
|
const logTime = log.timestamp ? new Date(log.timestamp).getTime() : 0;
|
||||||
let maxStepTime = -1;
|
let maxStepTime = -1;
|
||||||
for (let i = 0; i < blocks.length; i++) {
|
for (let i = 0; i < blocks.length; i++) {
|
||||||
const b = blocks[i];
|
const b = blocks[i];
|
||||||
if (b.id !== 'step-pre-planning') {
|
const bTime = b.timestamp ? new Date(b.timestamp).getTime() : 0;
|
||||||
const bTime = b.timestamp ? new Date(b.timestamp).getTime() : 0;
|
if (bTime <= logTime && bTime >= maxStepTime) {
|
||||||
if (bTime <= logTime && bTime >= maxStepTime) {
|
maxStepTime = bTime;
|
||||||
maxStepTime = bTime;
|
existingIndex = i;
|
||||||
existingIndex = i;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (existingIndex > -1) {
|
if (existingIndex > -1) {
|
||||||
const stepData = blocks[existingIndex].data;
|
const stepData = blocks[existingIndex].data;
|
||||||
const genericTools = stepData.generic_tools || [];
|
const genericTools = stepData.generic_tools ? [...stepData.generic_tools] : [];
|
||||||
if (!genericTools.some((t: any) => t.trace_id === log.data.trace_id)) {
|
const existingToolIdx = log.data.trace_id
|
||||||
const updatedTools = [...genericTools, log.data];
|
? genericTools.findIndex((t: any) => t.trace_id === log.data.trace_id)
|
||||||
blocks[existingIndex] = {
|
: -1;
|
||||||
...blocks[existingIndex],
|
|
||||||
data: {
|
if (existingToolIdx > -1) {
|
||||||
...stepData,
|
genericTools[existingToolIdx] = {
|
||||||
generic_tools: updatedTools
|
...genericTools[existingToolIdx],
|
||||||
}
|
...log.data
|
||||||
};
|
};
|
||||||
|
} else {
|
||||||
|
genericTools.push(log.data);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
blocks[existingIndex] = {
|
||||||
|
...blocks[existingIndex],
|
||||||
|
data: {
|
||||||
|
...stepData,
|
||||||
|
generic_tools: genericTools
|
||||||
|
}
|
||||||
|
};
|
||||||
} else {
|
} else {
|
||||||
const fallbackStepId = stepId || 'pre-planning';
|
const fallbackStepId = stepId || 'pre-planning';
|
||||||
blocks.push({
|
blocks.push({
|
||||||
@@ -399,7 +426,7 @@ export function getSortedStepEvents(
|
|||||||
const fallbackTime = getItemTimestamp(stepData.timestamp, 0);
|
const fallbackTime = getItemTimestamp(stepData.timestamp, 0);
|
||||||
|
|
||||||
// 1. Add action_taken if present and valid
|
// 1. Add action_taken if present and valid
|
||||||
if (stepData.action_taken && isAndroidAction(stepData.action_taken)) {
|
if (stepData.action_taken && (isAndroidAction(stepData.action_taken) || isReportStatusAction(stepData.action_taken))) {
|
||||||
const actTime = getItemTimestamp(
|
const actTime = getItemTimestamp(
|
||||||
stepData.action_taken.timestamp ?? stepData.action_taken.start_time ?? stepData.action_taken.created_at,
|
stepData.action_taken.timestamp ?? stepData.action_taken.start_time ?? stepData.action_taken.created_at,
|
||||||
fallbackTime
|
fallbackTime
|
||||||
@@ -441,17 +468,29 @@ export function getSortedStepEvents(
|
|||||||
export function compileSessionSummary(logs: any[]): string {
|
export function compileSessionSummary(logs: any[]): string {
|
||||||
if (!logs || logs.length === 0) return 'No logs available.';
|
if (!logs || logs.length === 0) return 'No logs available.';
|
||||||
|
|
||||||
// Check for goal completed / checker response
|
// Check for goal completed / checker response / cancellation / report_task_status
|
||||||
for (let i = logs.length - 1; i >= 0; i--) {
|
for (let i = logs.length - 1; i >= 0; i--) {
|
||||||
const log = logs[i];
|
const log = logs[i];
|
||||||
|
if (log.type === 'session_ended') {
|
||||||
|
if (log.data?.status === 'cancelled' || log.data?.was_stopped_manually) {
|
||||||
|
return 'Task stopped manually.';
|
||||||
|
}
|
||||||
|
}
|
||||||
if (log.type === 'llm_stream' && log.data?.text) {
|
if (log.type === 'llm_stream' && log.data?.text) {
|
||||||
const text = log.data.text;
|
const text = log.data.text;
|
||||||
if (text.includes('```json') && text.includes('"success"')) {
|
if (text.includes('```json') && text.includes('"success"')) {
|
||||||
return 'Execution completed and verified by checker.';
|
return 'Execution completed and verified by checker.';
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if (log.type === 'step_recorded' && log.data?.status === 'completed') {
|
if (log.type === 'step_recorded' || log.type === 'step_updated') {
|
||||||
return log.data.message || 'Task completed successfully.';
|
const act = log.data?.action_taken;
|
||||||
|
if (act && isReportStatusAction(act)) {
|
||||||
|
const exp = getReportStatusExplanation(act);
|
||||||
|
if (exp) return exp;
|
||||||
|
}
|
||||||
|
if (log.data?.status === 'completed') {
|
||||||
|
return log.data.message || 'Task completed successfully.';
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -486,7 +525,7 @@ export function checkPlanningLoader(
|
|||||||
if (log.type === 'session_ended') return false;
|
if (log.type === 'session_ended') return false;
|
||||||
if (log.type === 'step_recorded' || log.type === 'step_updated') {
|
if (log.type === 'step_recorded' || log.type === 'step_updated') {
|
||||||
const act = log.data?.action_taken;
|
const act = log.data?.action_taken;
|
||||||
if (act && (act.action === 'report_task_status' || act === 'report_task_status')) {
|
if (act && (act.action === 'report_task_status' || act === 'report_task_status' || isReportStatusAction(act))) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
if (log.data?.status === 'completed' || log.data?.status === 'failed') {
|
if (log.data?.status === 'completed' || log.data?.status === 'failed') {
|
||||||
+148
-37
@@ -15,8 +15,8 @@
|
|||||||
*/
|
*/
|
||||||
|
|
||||||
import { ActionParam } from '../core/models/stream.model';
|
import { ActionParam } from '../core/models/stream.model';
|
||||||
import { isAndroidAction } from './action-formatter.util';
|
import { isAndroidAction, isReportStatusAction } from './action-formatter.util';
|
||||||
import { extractNumbersFromCoordinateValue, isPureDirectionString } from './image-overlay.util';
|
import { extractNumbersFromCoordinateValue, isPureDirectionString, parseSequenceCoordinates } from './image-overlay.util';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Extract arguments from tool payload or direct args field
|
* Extract arguments from tool payload or direct args field
|
||||||
@@ -64,7 +64,7 @@ export function isVideoTool(tool: any): boolean {
|
|||||||
if (!tool || !tool.name) return false;
|
if (!tool || !tool.name) return false;
|
||||||
const cleanName = tool.name.replace(/^(_)?(self\.)?exec_/, '');
|
const cleanName = tool.name.replace(/^(_)?(self\.)?exec_/, '');
|
||||||
const nameLower = cleanName.toLowerCase();
|
const nameLower = cleanName.toLowerCase();
|
||||||
return ['video_analyzer', 'video_analyzer_pure', 'extract_segment_metadata', 'spawn_sub_agent', 'analyze_audio_only'].includes(nameLower);
|
return ['video_analyzer', 'video_analyzer_pure'].includes(nameLower);
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -92,45 +92,93 @@ export function getVideoToolTarget(tool: any): string | null {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Check if a tool should be rendered in Action Card format (ONLY real Android device operations and ADB commands)
|
* Canonical list of mobile device automation and ADB command tool names.
|
||||||
*/
|
*/
|
||||||
export function isFailureAnalyzerActionTool(tool: any): boolean {
|
export const DEVICE_ACTION_TOOL_NAMES = new Set([
|
||||||
|
'click', 'click_sequence', 'long_press', 'long_press_on', 'input', 'input_text',
|
||||||
|
'swipe', 'scroll', 'press_key', 'press_home', 'press_back', 'launch_app', 'manage_app', 'open_app',
|
||||||
|
'focus_and_input_text', 'focus_and_clear_text', 'tap', 'wait_for_delay', 'wait',
|
||||||
|
'run_adb_command', 'run_short_adb_command'
|
||||||
|
]);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if a tool represents a real mobile device action or ADB command.
|
||||||
|
*/
|
||||||
|
export function isDeviceActionTool(tool: any): boolean {
|
||||||
if (!tool || !tool.name) return false;
|
if (!tool || !tool.name) return false;
|
||||||
if (tool.type === 'llm_call') return false;
|
if (tool.type === 'llm_call') return false;
|
||||||
const cleanName = tool.name.replace(/^(_)?(self\.)?exec_/, '').toLowerCase();
|
const cleanName = tool.name.replace(/^(_)?(self\.)?exec_/, '').toLowerCase();
|
||||||
|
return DEVICE_ACTION_TOOL_NAMES.has(cleanName);
|
||||||
const realActionTools = [
|
|
||||||
'click', 'click_sequence', 'long_press', 'long_press_on', 'input', 'input_text',
|
|
||||||
'swipe', 'scroll', 'press_key', 'press_home', 'press_back', 'launch_app', 'manage_app', 'open_app',
|
|
||||||
'focus_and_input_text', 'focus_and_clear_text', 'tap', 'wait_for_delay', 'wait',
|
|
||||||
'run_adb_command', 'run_short_adb_command'
|
|
||||||
];
|
|
||||||
return realActionTools.includes(cleanName);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Check if a generic tool should be displayed in the UI
|
* Backward-compatible alias for isDeviceActionTool.
|
||||||
|
*/
|
||||||
|
export function isFailureAnalyzerActionTool(tool: any): boolean {
|
||||||
|
return isDeviceActionTool(tool);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Framework-internal infrastructure tools that are not intended for end-user timeline display.
|
||||||
|
*/
|
||||||
|
export function isInternalPlumbingTool(tool: any): boolean {
|
||||||
|
if (!tool || !tool.name) return true;
|
||||||
|
const cleanName = tool.name.replace(/^(_)?(self\.)?exec_/, '').toLowerCase();
|
||||||
|
if (cleanName.includes('safety_net')) return true;
|
||||||
|
if (cleanName === 'report_task_status' || cleanName === 'report_status' || cleanName === 'submit_task_status') {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Determines whether a tool invocation should be displayed in the timeline.
|
||||||
*/
|
*/
|
||||||
export function shouldShowTool(tool: any, stepData?: any): boolean {
|
export function shouldShowTool(tool: any, stepData?: any): boolean {
|
||||||
if (!tool || !tool.name) return false;
|
if (!tool || !tool.name) return false;
|
||||||
if (tool.type === 'llm_call') return false;
|
if (tool.type === 'llm_call') return false;
|
||||||
const cleanName = tool.name.replace(/^(_)?(self\.)?exec_/, '');
|
if (tool.type === 'agent') return false;
|
||||||
const nameLower = cleanName.toLowerCase();
|
if (isInternalPlumbingTool(tool)) return false;
|
||||||
if (nameLower.includes('safety_net')) return false;
|
|
||||||
|
|
||||||
// If step has an android action card rendered for action_taken, hide redundant generic tool text for atomic actions
|
// If this tool is an operator/validator atomic action that duplicates stepData.action_taken, skip it
|
||||||
const atomicActions = [
|
if (stepData && stepData.action_taken && isAndroidAction(stepData.action_taken) && isDeviceActionTool(tool)) {
|
||||||
'click', 'click_sequence', 'long_press', 'long_press_on', 'input', 'input_text',
|
const agentName = (tool.agent_name || tool.agent || '').toLowerCase();
|
||||||
'swipe', 'scroll', 'press_key', 'press_home', 'press_back', 'launch_app', 'manage_app', 'open_app',
|
const isSelfHealingOrSubAgent = agentName.includes('failure') ||
|
||||||
'focus_and_input_text', 'focus_and_clear_text', 'tap', 'wait_for_delay', 'wait'
|
agentName.includes('analyzer') ||
|
||||||
];
|
agentName.includes('diagnos') ||
|
||||||
if (stepData && stepData.action_taken && isAndroidAction(stepData.action_taken) && atomicActions.includes(nameLower)) {
|
(agentName && agentName !== 'operator' && agentName !== 'flashrunner' && agentName !== 'validator');
|
||||||
return false;
|
if (!isSelfHealingOrSubAgent) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Extracts human-readable initiating agent name / role for a tool.
|
||||||
|
*/
|
||||||
|
export function getToolAgentName(tool: any): string | null {
|
||||||
|
if (!tool) return null;
|
||||||
|
const name = (tool.agent_name || tool.agent || '').toLowerCase();
|
||||||
|
if (name.includes('failure') || name.includes('analyzer')) {
|
||||||
|
return null; // Omit self-healing label per user instruction
|
||||||
|
}
|
||||||
|
if (name.includes('outputter')) {
|
||||||
|
return 'Outputter';
|
||||||
|
}
|
||||||
|
if (name.includes('validator')) {
|
||||||
|
return 'Validator';
|
||||||
|
}
|
||||||
|
if (name.includes('diagnos')) {
|
||||||
|
return 'Diagnoser';
|
||||||
|
}
|
||||||
|
if (name.includes('explorer')) {
|
||||||
|
return 'Explorer';
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Filter out nested child wrapper duplicates while preserving independent executions at the same level
|
* Filter out nested child wrapper duplicates while preserving independent executions at the same level
|
||||||
*/
|
*/
|
||||||
@@ -145,11 +193,13 @@ export function getUniqueGenericTools(tools: any[] | undefined): any[] {
|
|||||||
}
|
}
|
||||||
|
|
||||||
const cleanName = (nameStr: string) => (nameStr || '').replace(/^(_)?(self\.)?exec_/, '').toLowerCase().trim();
|
const cleanName = (nameStr: string) => (nameStr || '').replace(/^(_)?(self\.)?exec_/, '').toLowerCase().trim();
|
||||||
const hasMainVideoAnalyzer = tools.some(t => t && (cleanName(t.name) === 'video_analyzer' || cleanName(t.name) === 'video_analyzer_pure'));
|
|
||||||
|
|
||||||
const result: any[] = [];
|
const result: any[] = [];
|
||||||
|
const seenTraces = new Set<string>();
|
||||||
|
|
||||||
for (const tool of tools) {
|
for (const tool of tools) {
|
||||||
if (!tool || !tool.name) continue;
|
if (!tool || !tool.name) continue;
|
||||||
|
if (tool.type === 'agent') continue;
|
||||||
|
|
||||||
if (tool.type === 'llm_call' && tool.status === 'failed') {
|
if (tool.type === 'llm_call' && tool.status === 'failed') {
|
||||||
result.push(tool);
|
result.push(tool);
|
||||||
@@ -158,11 +208,6 @@ export function getUniqueGenericTools(tools: any[] | undefined): any[] {
|
|||||||
|
|
||||||
const currClean = cleanName(tool.name);
|
const currClean = cleanName(tool.name);
|
||||||
|
|
||||||
// Hide internal video sub-agent worker tools if main video analyzer trace is already present
|
|
||||||
if (hasMainVideoAnalyzer && ['extract_segment_metadata', 'spawn_sub_agent', 'analyze_audio_only'].includes(currClean)) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Check if this tool is a nested child execution of a parent tool with the exact same tool name
|
// Check if this tool is a nested child execution of a parent tool with the exact same tool name
|
||||||
let isNestedChildDuplicate = false;
|
let isNestedChildDuplicate = false;
|
||||||
if (tool.parent_trace_id) {
|
if (tool.parent_trace_id) {
|
||||||
@@ -173,7 +218,15 @@ export function getUniqueGenericTools(tools: any[] | undefined): any[] {
|
|||||||
}
|
}
|
||||||
|
|
||||||
if (!isNestedChildDuplicate) {
|
if (!isNestedChildDuplicate) {
|
||||||
result.push(tool);
|
if (tool.trace_id) {
|
||||||
|
if (!seenTraces.has(tool.trace_id)) {
|
||||||
|
seenTraces.add(tool.trace_id);
|
||||||
|
// Always use the latest state from toolMap for this trace_id
|
||||||
|
result.push(toolMap.get(tool.trace_id) || tool);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
result.push(tool);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return result;
|
return result;
|
||||||
@@ -294,6 +347,31 @@ export function getToolDisplayLabel(tool: any, isFirstSaveNote: boolean = false)
|
|||||||
case 'video_analyzer':
|
case 'video_analyzer':
|
||||||
case 'video_analyzer_pure':
|
case 'video_analyzer_pure':
|
||||||
return 'Analyzing screen recording';
|
return 'Analyzing screen recording';
|
||||||
|
case 'extract_segment_metadata': {
|
||||||
|
const start = args.start_time !== undefined ? `${args.start_time}s` : '';
|
||||||
|
const end = args.end_time !== undefined ? `${args.end_time}s` : '';
|
||||||
|
const range = (start && end) ? ` (${start} - ${end})` : (start ? ` (from ${start})` : '');
|
||||||
|
return `Cropping screen recording segment${range}`;
|
||||||
|
}
|
||||||
|
case 'spawn_sub_agent': {
|
||||||
|
const q = args.specific_query || args.query || args.prompt || '';
|
||||||
|
return q ? `Analyzing recording with sub-agent: "${q}"` : 'Analyzing recording with sub-agent';
|
||||||
|
}
|
||||||
|
case 'analyze_audio_only': {
|
||||||
|
const q = args.specific_query || args.query || '';
|
||||||
|
return q ? `Analyzing audio track: "${q}"` : 'Analyzing recording audio track';
|
||||||
|
}
|
||||||
|
case 'get_step_details':
|
||||||
|
return 'Retrieving step execution details';
|
||||||
|
case 'get_step_screenshot':
|
||||||
|
return 'Inspecting screenshot evidence';
|
||||||
|
case 'search_history_for_text': {
|
||||||
|
const q = args.text || args.query || '';
|
||||||
|
return q ? `Searching execution trace for "${q}"` : 'Searching execution trace';
|
||||||
|
}
|
||||||
|
case 'outputter':
|
||||||
|
case 'output_synthesis':
|
||||||
|
return 'Synthesizing output report';
|
||||||
case 'web_search': {
|
case 'web_search': {
|
||||||
const q = args.query || '';
|
const q = args.query || '';
|
||||||
return q ? `Searching web for "${q}"` : 'Searching the web';
|
return q ? `Searching web for "${q}"` : 'Searching the web';
|
||||||
@@ -356,6 +434,21 @@ export function getToolIcon(tool: any): string {
|
|||||||
case 'video_analyzer':
|
case 'video_analyzer':
|
||||||
case 'video_analyzer_pure':
|
case 'video_analyzer_pure':
|
||||||
return 'video_camera_back';
|
return 'video_camera_back';
|
||||||
|
case 'extract_segment_metadata':
|
||||||
|
return 'crop';
|
||||||
|
case 'spawn_sub_agent':
|
||||||
|
return 'smart_toy';
|
||||||
|
case 'analyze_audio_only':
|
||||||
|
return 'graphic_eq';
|
||||||
|
case 'get_step_details':
|
||||||
|
return 'manage_search';
|
||||||
|
case 'get_step_screenshot':
|
||||||
|
return 'image_search';
|
||||||
|
case 'search_history_for_text':
|
||||||
|
return 'find_in_page';
|
||||||
|
case 'outputter':
|
||||||
|
case 'output_synthesis':
|
||||||
|
return 'assignment_turned_in';
|
||||||
default:
|
default:
|
||||||
return 'build';
|
return 'build';
|
||||||
}
|
}
|
||||||
@@ -429,6 +522,12 @@ export function getToolTitle(tool: any): string {
|
|||||||
case 'video_analyzer':
|
case 'video_analyzer':
|
||||||
case 'video_analyzer_pure':
|
case 'video_analyzer_pure':
|
||||||
return 'Analyzing Screen Recording';
|
return 'Analyzing Screen Recording';
|
||||||
|
case 'extract_segment_metadata':
|
||||||
|
return 'Cropping Screen Recording';
|
||||||
|
case 'spawn_sub_agent':
|
||||||
|
return 'Delegating Video Analysis';
|
||||||
|
case 'analyze_audio_only':
|
||||||
|
return 'Analyzing Audio Track';
|
||||||
default:
|
default:
|
||||||
return cleanName.replace(/_/g, ' ').replace(/\b\w/g, (c: string) => c.toUpperCase());
|
return cleanName.replace(/_/g, ' ').replace(/\b\w/g, (c: string) => c.toUpperCase());
|
||||||
}
|
}
|
||||||
@@ -546,6 +645,16 @@ export function getToolInputText(tool: any): string {
|
|||||||
export function getToolCoords(tool: any): string {
|
export function getToolCoords(tool: any): string {
|
||||||
if (!tool || !tool.name) return '';
|
if (!tool || !tool.name) return '';
|
||||||
const args = getToolArgs(tool);
|
const args = getToolArgs(tool);
|
||||||
|
const cleanName = tool.name.replace(/^(_)?exec_/, '');
|
||||||
|
const name = cleanName.toLowerCase();
|
||||||
|
|
||||||
|
// Check sequence first if action is sequence or sequence arg is present
|
||||||
|
const isSequenceAction = name === 'click_sequence' || name === 'tap_sequence' || Boolean(args.sequence || args.normalized_sequence);
|
||||||
|
const rawSeq = args.normalized_sequence || args.sequence || args.targets || (Array.isArray(args.coordinates) && Array.isArray(args.coordinates[0]) ? args.coordinates : null);
|
||||||
|
const seqPoints = parseSequenceCoordinates(rawSeq);
|
||||||
|
if (seqPoints && seqPoints.length > 0 && (isSequenceAction || seqPoints.length > 1)) {
|
||||||
|
return seqPoints.map(pt => `[${pt[0]}, ${pt[1]}]`).join(' → ');
|
||||||
|
}
|
||||||
|
|
||||||
// Check normalized start and end first
|
// Check normalized start and end first
|
||||||
const normStart = extractNumbersFromCoordinateValue(args.normalized_start_coordinates || args.start_coordinates || args.start);
|
const normStart = extractNumbersFromCoordinateValue(args.normalized_start_coordinates || args.start_coordinates || args.start);
|
||||||
@@ -572,10 +681,7 @@ export function getToolCoords(tool: any): string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
if (args.x !== undefined && args.y !== undefined) {
|
if (args.x !== undefined && args.y !== undefined) {
|
||||||
return `${args.x}, ${args.y}`;
|
return `[${args.x}, ${args.y}]`;
|
||||||
}
|
|
||||||
if (args.sequence && Array.isArray(args.sequence)) {
|
|
||||||
return JSON.stringify(args.sequence);
|
|
||||||
}
|
}
|
||||||
return '';
|
return '';
|
||||||
}
|
}
|
||||||
@@ -839,7 +945,12 @@ export function cleanErrorMessage(rawError: any): string {
|
|||||||
if (fallback.includes('[type.googleapis.com')) {
|
if (fallback.includes('[type.googleapis.com')) {
|
||||||
fallback = fallback.split('[type.googleapis.com')[0];
|
fallback = fallback.split('[type.googleapis.com')[0];
|
||||||
}
|
}
|
||||||
fallback = fallback.replace(/^ServerError:\s*/i, '').trim();
|
fallback = fallback
|
||||||
|
.replace(/^Pre-execution validation failed:\s*/i, '')
|
||||||
|
.replace(/^Pixel-level validation failed:\s*/i, '')
|
||||||
|
.replace(/^Execution error:\s*/i, '')
|
||||||
|
.replace(/^ServerError:\s*/i, '')
|
||||||
|
.trim();
|
||||||
|
|
||||||
return fallback || errorStr;
|
return fallback || errorStr;
|
||||||
}
|
}
|
||||||
@@ -21,7 +21,15 @@
|
|||||||
<title>artemis</title>
|
<title>artemis</title>
|
||||||
<base href="/">
|
<base href="/">
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
<link rel="icon" type="image/svg+xml" href="logo.svg">
|
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||||||
|
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||||
|
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Material+Symbols+Outlined:opsz,wght,FILL,GRAD@20..48,100..700,0..1,-50..200" />
|
||||||
|
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700;800&family=Outfit:wght@300;400;500;600;700&family=Roboto:wght@400;500;700&display=swap" />
|
||||||
|
<link rel="icon" type="image/png" sizes="192x192" href="/logo.png?v=2">
|
||||||
|
<link rel="icon" type="image/png" sizes="32x32" href="/logo.png?v=2">
|
||||||
|
<link rel="icon" type="image/svg+xml" href="/logo.svg?v=2">
|
||||||
|
<link rel="shortcut icon" href="/favicon.ico?v=2">
|
||||||
|
<link rel="apple-touch-icon" href="/logo.png?v=2">
|
||||||
</head>
|
</head>
|
||||||
<body>
|
<body>
|
||||||
<app-root></app-root>
|
<app-root></app-root>
|
||||||
File renamed without changes.
@@ -14,11 +14,16 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@import url('https://fonts.googleapis.com/css2?family=Outfit:wght@300;400;500;600;700&display=swap');
|
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700;800&family=Outfit:wght@300;400;500;600;700&family=Roboto:wght@400;500;700&display=swap');
|
||||||
@import url('https://fonts.googleapis.com/css2?family=Material+Symbols+Outlined:opsz,wght,FILL,GRAD@20..48,100..700,0..1,-50..200');
|
@import url('https://fonts.googleapis.com/css2?family=Material+Symbols+Outlined:opsz,wght,FILL,GRAD@20..48,100..700,0..1,-50..200');
|
||||||
|
|
||||||
|
*, *::before, *::after {
|
||||||
|
box-sizing: border-box;
|
||||||
|
font-family: 'Google Sans', 'Google Sans Text', 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif;
|
||||||
|
}
|
||||||
|
|
||||||
.material-symbols-outlined {
|
.material-symbols-outlined {
|
||||||
font-family: 'Material Symbols Outlined';
|
font-family: 'Material Symbols Outlined' !important;
|
||||||
font-weight: normal;
|
font-weight: normal;
|
||||||
font-style: normal;
|
font-style: normal;
|
||||||
font-size: 20px;
|
font-size: 20px;
|
||||||
@@ -36,13 +41,15 @@
|
|||||||
font-variation-settings: 'FILL' 0, 'wght' 300, 'GRAD' 0, 'opsz' 20;
|
font-variation-settings: 'FILL' 0, 'wght' 300, 'GRAD' 0, 'opsz' 20;
|
||||||
}
|
}
|
||||||
|
|
||||||
body, input, button, select, textarea, h1, h2, h3, h4, h5, h6, span, p, div, summary, details, li, a {
|
body, input, button, select, textarea {
|
||||||
font-family: 'Google Sans', 'Google Sans Text', 'Inter', Arial, "Liberation Sans", sans-serif;
|
font-family: 'Google Sans', 'Google Sans Text', 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif;
|
||||||
color: #1f1f1f;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
body {
|
body {
|
||||||
margin: 0;
|
margin: 0;
|
||||||
padding: 0;
|
padding: 0;
|
||||||
background-color: #fafafa;
|
color: #1e293b;
|
||||||
|
background-color: #f8f9fa;
|
||||||
|
-webkit-font-smoothing: antialiased;
|
||||||
|
-moz-osx-font-smoothing: grayscale;
|
||||||
}
|
}
|
||||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -1,4 +1,4 @@
|
|||||||
{
|
{
|
||||||
"checker_system": "You are a precise and reliable verification expert (Checker). Your task is to determine whether the Operator has successfully completed the active subgoal by combining visual evidence from the current screenshot with data stored in persistent notes.",
|
"checker_system": "You are a precise and reliable verification expert (Checker). Your task is to determine whether the Operator has successfully completed the active subgoal by combining visual evidence from the current screenshot with data stored in persistent notes.",
|
||||||
"checker_prompt": "# Inputs\n- Active Subgoal: {{ subgoal }}\n- Execution History:\n{{ plan_and_history }}\n- Dialogue History: {{ dialogue_history }}\n- Visual State: The provided screenshot.\n\n# Guidelines\n- **Visual + Data Verification**: You must combine the screenshot with note contents. If the subgoal involves extracting, saving, or recording information (e.g., \"save contacts\", \"record address\"), you **MUST** verify that the content actually exists in the note. Do not assume success just because the information is visible on the screen. If the signs of success are not obvious, you can check the history, especially the most recent steps, to see if the operator performed their own check and whether that check was reasonable, to determine if the subgoal was truly successful.\n- **Visual Priority (Anti-History-Bias)**: Your primary ground truth is the visual evidence on the **Current Screenshot**. The `Execution History` lists previous successes or failures, which may be outdated due to dynamic background recovery or self-healing. If the screenshot demonstrably shows the active subgoal's target state is achieved, you **MUST** mark `success: true`, regardless of previous failed steps. Trust what you see, not past FAILED texts.\n- **Constraint Verification**: For subgoals containing specific parameters (such as names, brands, topics, events, numbers, or dates), confirm that the visible text precisely reflects these specified details. A successful state is defined by a clear, direct match with the subgoal's constraints.\n- **Realistic Expectations**: You cannot scroll the screen. If a subgoal requires saving all items but only some are visible on the screen, verify that at least the **visible items** are recorded in the note. As long as the visible part is recorded and there are no obvious errors, consider it a success. Do not fail the task simply because off-screen content cannot be verified.\n- **Success Criteria**: Set `success` to `true` if the core objective of the subgoal is met and it does not block subsequent steps. Set `success` to `false` if the state is incorrect, required data was not saved, or the operator is stuck/looping.\n\n# Output Protocol\nInvoke the `submit_verification` tool with `success` (boolean) and `reason` (string) to submit your final decision. If you need to inspect note data first, use `read_note`."
|
"checker_prompt": "# Inputs\n- Active Subgoal: {{ subgoal }}\n- Execution History:\n{{ plan_and_history }}\n- Dialogue History: {{ dialogue_history }}\n- Visual State: The provided screenshot.\n\n# Guidelines\n- **Visual + Data Verification**: You must combine the screenshot with note contents. If the subgoal involves extracting, saving, or recording information (e.g., \"save contacts\", \"record address\"), you **MUST** verify that the content actually exists in the note. Do not assume success just because the information is visible on the screen. If the signs of success are not obvious, you can check the history, especially the most recent steps, to see if the operator performed their own check and whether that check was reasonable, to determine if the subgoal was truly successful.\n- **Visual Priority (Anti-History-Bias)**: Your primary ground truth is the visual evidence on the **Current Screenshot**. The `Execution History` lists previous successes or failures, which may be outdated due to dynamic background recovery or self-healing. If the screenshot demonstrably shows the active subgoal's target state is achieved, you **MUST** mark `success: true`, regardless of previous failed steps. Trust what you see, not past FAILED texts.\n- **Constraint Verification**: For subgoals containing specific parameters (such as names, brands, topics, events, numbers, or dates), confirm that the visible text precisely reflects these specified details. A successful state is defined by a clear, direct match with the subgoal's constraints.\n- **Iteration & Monitoring Subgoals Validation**: For concrete iteration sub-tasks (e.g., inspecting Candidate #1) or periodic polling subgoals (e.g., Polling Round #1 check), verify whether the Operator correctly executed the step and logged the progress/findings into notes or the task plan. Do NOT mark a monitoring or iteration step as failed simply because the overall monitoring session is ongoing or no new incoming events were detected in that specific polling cycle. As long as the discrete check was performed and recorded, mark `success: true`.\n- **Realistic Expectations**: You cannot scroll the screen. If a subgoal requires saving all items but only some are visible on the screen, verify that at least the **visible items** are recorded in the note. As long as the visible part is recorded and there are no obvious errors, consider it a success. Do not fail the task simply because off-screen content cannot be verified.\n- **Success Criteria**: Set `success` to `true` if the core objective of the subgoal is met and it does not block subsequent steps. Set `success` to `false` if the state is incorrect, required data was not saved, or the operator is stuck/looping.\n\n# Output Protocol\nInvoke the `submit_verification` tool with `success` (boolean) and `reason` (string) to submit your final decision. If you need to inspect note data first, use `read_note`."
|
||||||
}
|
}
|
||||||
@@ -42,7 +42,7 @@ from artemis.tools.tool_wrapper import (
|
|||||||
)
|
)
|
||||||
from artemis.utils.logger import get_logger
|
from artemis.utils.logger import get_logger
|
||||||
from artemis.utils.notes import get_note_file_path, get_notes_dir
|
from artemis.utils.notes import get_note_file_path, get_notes_dir
|
||||||
from artemis.utils.ocr_api import perform_ocr
|
from artemis.utils.ocr_api import is_ocr_configured, perform_ocr
|
||||||
from artemis.utils.ocr_xml_fusion import (
|
from artemis.utils.ocr_xml_fusion import (
|
||||||
_crop_image_remove_status_bar,
|
_crop_image_remove_status_bar,
|
||||||
_detect_status_bar_height,
|
_detect_status_bar_height,
|
||||||
@@ -143,18 +143,19 @@ async def run_async_check(
|
|||||||
height = device_data.height
|
height = device_data.height
|
||||||
|
|
||||||
ocr_results = []
|
ocr_results = []
|
||||||
try:
|
if is_ocr_configured():
|
||||||
status_bar_height = _detect_status_bar_height(xml_hierarchy, height)
|
try:
|
||||||
if status_bar_height > 0:
|
status_bar_height = _detect_status_bar_height(xml_hierarchy, height)
|
||||||
cropped_b64, _, _ = _crop_image_remove_status_bar(
|
if status_bar_height > 0:
|
||||||
screenshot_b64, status_bar_height
|
cropped_b64, _, _ = _crop_image_remove_status_bar(
|
||||||
)
|
screenshot_b64, status_bar_height
|
||||||
raw_ocr_results = await perform_ocr(cropped_b64)
|
)
|
||||||
ocr_results = _map_coordinates_back(raw_ocr_results, status_bar_height)
|
raw_ocr_results = await perform_ocr(cropped_b64)
|
||||||
else:
|
ocr_results = _map_coordinates_back(raw_ocr_results, status_bar_height)
|
||||||
ocr_results = await perform_ocr(screenshot_b64)
|
else:
|
||||||
except Exception as e:
|
ocr_results = await perform_ocr(screenshot_b64)
|
||||||
logger.error(f"Checker OCR failed, proceeding without OCR: {e}")
|
except Exception as e:
|
||||||
|
logger.warning(f"Checker OCR failed, proceeding without OCR: {e}")
|
||||||
|
|
||||||
fused_xml = fuse_ocr_with_xml(xml_hierarchy, ocr_results)
|
fused_xml = fuse_ocr_with_xml(xml_hierarchy, ocr_results)
|
||||||
|
|
||||||
@@ -356,10 +357,10 @@ async def run_async_check(
|
|||||||
tool_status = "success"
|
tool_status = "success"
|
||||||
try:
|
try:
|
||||||
if ":" in tool_name:
|
if ":" in tool_name:
|
||||||
tool_to_run = get_tool_by_name(tool_name, traced_tools)
|
tool_to_run = get_tool_by_name(tool_name, tools_to_bind)
|
||||||
else:
|
else:
|
||||||
tool_to_run = next(
|
tool_to_run = next(
|
||||||
(t for t in traced_tools if t.name == tool_name),
|
(t for t in tools_to_bind if t.name == tool_name),
|
||||||
None,
|
None,
|
||||||
)
|
)
|
||||||
if tool_to_run:
|
if tool_to_run:
|
||||||
|
|||||||
@@ -11,6 +11,7 @@
|
|||||||
"- **No Direct Operations**: Do not perform user actions or modify device state (e.g., tap, swipe, text input, BACK/HOME keys, or force-starting activities). Focus solely on analysis.",
|
"- **No Direct Operations**: Do not perform user actions or modify device state (e.g., tap, swipe, text input, BACK/HOME keys, or force-starting activities). Focus solely on analysis.",
|
||||||
"- **Tool Constraints**: Video analysis is expensive. When calling the video analyzer, you MUST provide a specific, narrow timeframe to minimize cost.",
|
"- **Tool Constraints**: Video analysis is expensive. When calling the video analyzer, you MUST provide a specific, narrow timeframe to minimize cost.",
|
||||||
"- **Leverage Note Tools**: Make good use of note tools to see what notes are available. When diagnosing, open and cross-reference relevant notes to ensure you have grasped all necessary information and context.",
|
"- **Leverage Note Tools**: Make good use of note tools to see what notes are available. When diagnosing, open and cross-reference relevant notes to ensure you have grasped all necessary information and context.",
|
||||||
|
"- **Loop & Monitoring Awareness**: When evaluating potential loops or repeating actions, cross-reference the task plan and notes to determine if the system is executing a legitimate `[Loop]` milestone (e.g., candidate iteration, periodic list polling with wait intervals, or continuous swipes for battery discharge). Do NOT misdiagnose planned iterations or paced polling checks as bugs or screen freezes.",
|
||||||
"- **Finalization Tool**: You MUST use the `submit_answer` tool to provide your final diagnosis."
|
"- **Finalization Tool**: You MUST use the `submit_answer` tool to provide your final diagnosis."
|
||||||
],
|
],
|
||||||
"checker_feedback_template": [
|
"checker_feedback_template": [
|
||||||
|
|||||||
@@ -32,14 +32,14 @@ EXPLORE_DESCRIPTIONS = {
|
|||||||
"pro": {
|
"pro": {
|
||||||
"description": (
|
"description": (
|
||||||
"[EXPLORER] Call this to ask the UI Explorer agent to locate"
|
"[EXPLORER] Call this to ask the UI Explorer agent to locate"
|
||||||
" coordinates on the screen layout by executing XML, OCR,"
|
" coordinates on the screen layout by executing UI search,"
|
||||||
" coordinate search, and object detection."
|
" coordinate search, and object detection."
|
||||||
),
|
),
|
||||||
"query_description": (
|
"query_description": (
|
||||||
"The target element or information to search for, including descriptions."
|
"The target element or information to search for, including descriptions."
|
||||||
),
|
),
|
||||||
"rule_prompt": (
|
"rule_prompt": (
|
||||||
"Call the explorer tool to search, OCR, or inspect it (maximum {max_iterations} tries)."
|
"Call the explorer tool to search or inspect it (maximum {max_iterations} tries)."
|
||||||
),
|
),
|
||||||
"version_prompt": (
|
"version_prompt": (
|
||||||
"You are running with a maximum of {max_iterations} turns: on your"
|
"You are running with a maximum of {max_iterations} turns: on your"
|
||||||
|
|||||||
@@ -53,7 +53,7 @@ from artemis.mcp.xml_search_server import (
|
|||||||
search_ui as search_ui_func,
|
search_ui as search_ui_func,
|
||||||
)
|
)
|
||||||
from artemis.utils.logger import get_logger
|
from artemis.utils.logger import get_logger
|
||||||
from artemis.utils.ocr_api import perform_ocr
|
from artemis.utils.ocr_api import is_ocr_configured, perform_ocr
|
||||||
from artemis.utils.ocr_xml_fusion import fuse_ocr_with_xml
|
from artemis.utils.ocr_xml_fusion import fuse_ocr_with_xml
|
||||||
from artemis.utils.visualization import draw_dots, format_minimal_list_with_points
|
from artemis.utils.visualization import draw_dots, format_minimal_list_with_points
|
||||||
|
|
||||||
@@ -73,7 +73,7 @@ UNIVERSAL_EXPLORER_TOOLS = [
|
|||||||
"properties": {
|
"properties": {
|
||||||
"search_query": {
|
"search_query": {
|
||||||
"type": "string",
|
"type": "string",
|
||||||
"description": "Required. Text query to search for in XML/OCR elements.",
|
"description": "Required. Text query to search for in UI elements.",
|
||||||
},
|
},
|
||||||
"nx": {
|
"nx": {
|
||||||
"type": "integer",
|
"type": "integer",
|
||||||
@@ -234,8 +234,8 @@ class Explorer:
|
|||||||
" in parallel, including database search, coordinate search,"
|
" in parallel, including database search, coordinate search,"
|
||||||
" and visual object detection on the screen.\n\nOutput"
|
" and visual object detection on the screen.\n\nOutput"
|
||||||
" Format:\nReturns:'text' (str): A single string consolidating"
|
" Format:\nReturns:'text' (str): A single string consolidating"
|
||||||
" results with 0-1000 normalized coordinates, including xml"
|
" results with 0-1000 normalized coordinates, including UI"
|
||||||
" search matches labeled [X], ocr search matches labeled [O],"
|
" search matches labeled [X],"
|
||||||
" coordinate search matches labeled [X], and visually detected"
|
" coordinate search matches labeled [X], and visually detected"
|
||||||
" object targets labeled [D].\n(Annotated screenshots are also"
|
" object targets labeled [D].\n(Annotated screenshots are also"
|
||||||
" added to the context containing corresponding colored label"
|
" added to the context containing corresponding colored label"
|
||||||
@@ -246,7 +246,7 @@ class Explorer:
|
|||||||
properties={
|
properties={
|
||||||
"search_query": types.Schema(
|
"search_query": types.Schema(
|
||||||
type=types.Type.STRING,
|
type=types.Type.STRING,
|
||||||
description=("Required. Text query to search for in XML/OCR elements."),
|
description=("Required. Text query to search for in UI elements."),
|
||||||
),
|
),
|
||||||
"nx": types.Schema(
|
"nx": types.Schema(
|
||||||
type=types.Type.INTEGER,
|
type=types.Type.INTEGER,
|
||||||
@@ -463,10 +463,10 @@ class Explorer:
|
|||||||
llm_config = getattr(ctx, "llm_config", None)
|
llm_config = getattr(ctx, "llm_config", None)
|
||||||
llm_cfg = getattr(llm_config, "explorer", None) if llm_config else None
|
llm_cfg = getattr(llm_config, "explorer", None) if llm_config else None
|
||||||
model_str = (
|
model_str = (
|
||||||
llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.6-flash"
|
llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.7-flash"
|
||||||
).lower()
|
).lower()
|
||||||
self.model_name = (
|
self.model_name = (
|
||||||
llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.6-flash"
|
llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.7-flash"
|
||||||
)
|
)
|
||||||
if "/" in self.model_name:
|
if "/" in self.model_name:
|
||||||
self.model_name = self.model_name.split("/")[-1]
|
self.model_name = self.model_name.split("/")[-1]
|
||||||
@@ -514,8 +514,11 @@ class Explorer:
|
|||||||
]
|
]
|
||||||
|
|
||||||
def get_exposed_tools(self, only_submit: bool = False) -> list[types.FunctionDeclaration]:
|
def get_exposed_tools(self, only_submit: bool = False) -> list[types.FunctionDeclaration]:
|
||||||
"""Returns the list of tools, filtering out denylisted ones."""
|
"""Returns the list of tools, filtering out denylisted and unconfigured ones."""
|
||||||
tools = [tool for tool in self.TOOLS if tool.name not in self.denylisted_tools]
|
denylisted = set(self.denylisted_tools)
|
||||||
|
if not is_ocr_configured():
|
||||||
|
denylisted.add("get_ocr_list")
|
||||||
|
tools = [tool for tool in self.TOOLS if tool.name not in denylisted]
|
||||||
if only_submit:
|
if only_submit:
|
||||||
tools = [tool for tool in tools if tool.name == "submit_answer"]
|
tools = [tool for tool in tools if tool.name == "submit_answer"]
|
||||||
return tools
|
return tools
|
||||||
@@ -855,7 +858,7 @@ class Explorer:
|
|||||||
|
|
||||||
xml_text = results[0].get("text", "No matches found.")
|
xml_text = results[0].get("text", "No matches found.")
|
||||||
if xml_text and xml_text != "No matches found.":
|
if xml_text and xml_text != "No matches found.":
|
||||||
text_parts.append(f"XML/OCR Text Search Results are: {xml_text}")
|
text_parts.append(f"Text Search Results are: {xml_text}")
|
||||||
|
|
||||||
coord_text = results[1].get("text", "No elements found.")
|
coord_text = results[1].get("text", "No elements found.")
|
||||||
if coord_text and coord_text != "No elements found.":
|
if coord_text and coord_text != "No elements found.":
|
||||||
@@ -1012,7 +1015,7 @@ class Explorer:
|
|||||||
record = storage.get_ui_record(self.image_name)
|
record = storage.get_ui_record(self.image_name)
|
||||||
if not record or not record.ocr_result:
|
if not record or not record.ocr_result:
|
||||||
return {
|
return {
|
||||||
"text": ("No OCR results found in Data Engine for the current screen."),
|
"text": "No text elements detected on the screen.",
|
||||||
"image_path": None,
|
"image_path": None,
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1143,11 +1146,12 @@ class Explorer:
|
|||||||
"""Executes Explorer reasoning loop via Universal LangChain ChatModel."""
|
"""Executes Explorer reasoning loop via Universal LangChain ChatModel."""
|
||||||
llm = get_llm(self.ctx, name="explorer")
|
llm = get_llm(self.ctx, name="explorer")
|
||||||
|
|
||||||
# Filter universal tools based on denylist
|
# Filter universal tools based on denylist and OCR configuration
|
||||||
|
denylisted = set(self.denylisted_tools)
|
||||||
|
if not is_ocr_configured():
|
||||||
|
denylisted.add("get_ocr_list")
|
||||||
exposed_tools = [
|
exposed_tools = [
|
||||||
t
|
t for t in UNIVERSAL_EXPLORER_TOOLS if t["function"]["name"] not in denylisted
|
||||||
for t in UNIVERSAL_EXPLORER_TOOLS
|
|
||||||
if t["function"]["name"] not in self.denylisted_tools
|
|
||||||
]
|
]
|
||||||
bound_llm = llm.bind_tools(exposed_tools)
|
bound_llm = llm.bind_tools(exposed_tools)
|
||||||
|
|
||||||
@@ -1447,6 +1451,9 @@ class Explorer:
|
|||||||
"inspect_region",
|
"inspect_region",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if not is_ocr_configured():
|
||||||
|
version_denylist.add("get_ocr_list")
|
||||||
|
|
||||||
try:
|
try:
|
||||||
denylisted_config = (
|
denylisted_config = (
|
||||||
self.ctx.agent_config.denylisted_tools.get("explorer", [])
|
self.ctx.agent_config.denylisted_tools.get("explorer", [])
|
||||||
@@ -1517,7 +1524,7 @@ class Explorer:
|
|||||||
if "/" in fallback_model:
|
if "/" in fallback_model:
|
||||||
fallback_model = fallback_model.split("/")[-1]
|
fallback_model = fallback_model.split("/")[-1]
|
||||||
else:
|
else:
|
||||||
model_name = "gemini-3.6-flash"
|
model_name = "gemini-3.7-flash"
|
||||||
|
|
||||||
# 5. Construct Prompt & Initial Message List
|
# 5. Construct Prompt & Initial Message List
|
||||||
prompt_path = Path(__file__).parent / "explorer.json"
|
prompt_path = Path(__file__).parent / "explorer.json"
|
||||||
@@ -1567,13 +1574,20 @@ class Explorer:
|
|||||||
ui_tree = record.ui_tree
|
ui_tree = record.ui_tree
|
||||||
ocr_results = getattr(record, "ocr_result", None)
|
ocr_results = getattr(record, "ocr_result", None)
|
||||||
if ocr_results is None:
|
if ocr_results is None:
|
||||||
try:
|
if is_ocr_configured():
|
||||||
logger.info("Previous screenshot OCR is missing. Running OCR on-the-fly...")
|
try:
|
||||||
with open(screenshot_path, "rb") as img_file:
|
logger.info(
|
||||||
img_b64 = base64.b64encode(img_file.read()).decode("utf-8")
|
"Previous screenshot OCR is missing. Running OCR on-the-fly..."
|
||||||
ocr_results = await perform_ocr(img_b64, client=self.http_client)
|
)
|
||||||
except Exception as ocr_err:
|
with open(screenshot_path, "rb") as img_file:
|
||||||
logger.error(f"On-the-fly OCR failed for previous screenshot: {ocr_err}")
|
img_b64 = base64.b64encode(img_file.read()).decode("utf-8")
|
||||||
|
ocr_results = await perform_ocr(img_b64, client=self.http_client)
|
||||||
|
except Exception as ocr_err:
|
||||||
|
logger.error(
|
||||||
|
f"On-the-fly OCR failed for previous screenshot: {ocr_err}"
|
||||||
|
)
|
||||||
|
ocr_results = []
|
||||||
|
else:
|
||||||
ocr_results = []
|
ocr_results = []
|
||||||
|
|
||||||
fused_xml = fuse_ocr_with_xml(ui_tree, ocr_results or [])
|
fused_xml = fuse_ocr_with_xml(ui_tree, ocr_results or [])
|
||||||
|
|||||||
@@ -22,6 +22,7 @@ import asyncio
|
|||||||
import base64
|
import base64
|
||||||
import json
|
import json
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
import re
|
||||||
import uuid
|
import uuid
|
||||||
|
|
||||||
from jinja2 import Template
|
from jinja2 import Template
|
||||||
@@ -439,13 +440,13 @@ class FlashRunner:
|
|||||||
elif kind == "direction" and isinstance(target_val, str):
|
elif kind == "direction" and isinstance(target_val, str):
|
||||||
g_lower = target_val.lower()
|
g_lower = target_val.lower()
|
||||||
if "up" in g_lower:
|
if "up" in g_lower:
|
||||||
norm_coords = [600, 800, 600, 200]
|
norm_coords = [600, 700, 600, 300]
|
||||||
elif "down" in g_lower:
|
elif "down" in g_lower:
|
||||||
norm_coords = [600, 200, 600, 800]
|
norm_coords = [600, 300, 600, 700]
|
||||||
elif "left" in g_lower:
|
elif "left" in g_lower:
|
||||||
norm_coords = [800, 500, 200, 500]
|
norm_coords = [750, 500, 250, 500]
|
||||||
elif "right" in g_lower:
|
elif "right" in g_lower:
|
||||||
norm_coords = [200, 500, 800, 500]
|
norm_coords = [250, 500, 750, 500]
|
||||||
if norm_coords:
|
if norm_coords:
|
||||||
norm_start = norm_coords[:2]
|
norm_start = norm_coords[:2]
|
||||||
norm_end = norm_coords[2:]
|
norm_end = norm_coords[2:]
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ Your goal is to perform image analysis, filter, crop, draw annotations/markers,
|
|||||||
2. **DO NOT** use `cv2.resize()` to scale the image.
|
2. **DO NOT** use `cv2.resize()` to scale the image.
|
||||||
Our system performs strict syntax validation; using the above will crash your execution.
|
Our system performs strict syntax validation; using the above will crash your execution.
|
||||||
|
|
||||||
If you MUST crop or scale the image to isolate small details or improve OCR/detection downstream, you MUST use the pre-imported `ImageCanvas` tool exactly as follows:
|
If you MUST crop or scale the image to isolate small details or improve reading/detection downstream, you MUST use the pre-imported `ImageCanvas` tool exactly as follows:
|
||||||
|
|
||||||
### Snippet 1: Saving & Resuming Intermediate Steps (Iterative Workflow)
|
### Snippet 1: Saving & Resuming Intermediate Steps (Iterative Workflow)
|
||||||
Use this to crop/zoom into specific areas and save them so you can visually inspect them in your next turn, or load them back to create new branches.
|
Use this to crop/zoom into specific areas and save them so you can visually inspect them in your next turn, or load them back to create new branches.
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
{
|
{
|
||||||
"main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List/Text/Complex UI Exploration Strategy (Zero Omission, Zero Duplication, Single-Pass Rule)**:\n When traversing large lists, long text bodies, or continuous content streams, adhere strictly to the **Single-Pass Exploration Standard**:\n 1. **Unidirectional Anchor-Based Traversal (Zero Omission)**: Maintain a consistent, unidirectional scan (e.g., scrolling steadily from top to bottom). Before each scroll, use the bottom-most visible item as a visual anchor. Ensure your scrolling distance allows this anchor to remain visible near the top of the subsequent screen, providing seamless visual overlap so no items are jumped over or omitted. Conclude exploration when reaching the definite boundary (e.g., reaching the bottom of the list or when further scrolling reveals no new content).\n 2. **Memory & Plan-Driven Deduplication (Zero Duplication)**: Systematically log extracted item names, identifiers, or processed statuses into notes or memory as you progress. Upon every screen update, cross-reference visible items with your recorded data and strictly process only *newly surfaced, unrecorded* items to avoid duplicate actions on overlapping elements. Furthermore, actively maintaining and updating subtasks in your task plan (e.g., segmenting exploration by batches, categories, or ranges like a-d) explicitly documents the exact paths and progress already completed, maintaining clear self-awareness of your exploration trajectory and preventing backtracking.\n 3. **Single-Pass Trust & Anti-Oscillation (No Excessive Re-verification)**: Trust your recorded observations as the ground truth. Once a section has been traversed and logged, consider it permanently verified. Never scroll back anLine truncated
|
"main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor. Calibrate your scroll distance so that this anchor remains visible near the top of the subsequent screen, creating a seamless visual overlap that guarantees zero missed items. Terminate exploration when a definitive boundary is reached (e.g., list bottom reached or no new content appears after scrolling).\n 2. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n 3. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\". Only perform incremental re-verification if the list has been refreshed or contentLine truncated
|
||||||
"troubleshooter_template": "# ROLE\nYou are the Expert Android Device Troubleshooter. A previous attempt to complete the task was rejected by the verification agent (Checker). Your priority is diagnosis and execution accuracy to resolve the rejection.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List/Text/Complex UI Exploration Strategy (Zero Omission, Zero Duplication, Single-Pass Rule)**:\n When traversing large lists, long text bodies, or continuous content streams, adhere strictly to the **Single-Pass Exploration Standard**:\n 1. **Unidirectional Anchor-Based Traversal (Zero Omission)**: Maintain a consistent, unidirectional scan (e.g., scrolling steadily from top to bottom). Before each scroll, use the bottom-most visible item as a visual anchor. Ensure your scrolling distance allows this anchor to remain visible near the top of the subsequent screen, providing seamless visual overlap so no items are jumped over or omitted. Conclude exploration when reaching the definite boundary (e.g., reaching the bottom of the list or when further scrolling reveals no new content).\n 2. **Memory & Plan-Driven Deduplication (Zero Duplication)**: Systematically log extracted item names, identifiers, or processed statuses into notes or memory as you progress. Upon every screen update, cross-reference visible items with your recorded data and strictly process only *newly surfaced, unrecorded* items to avoid duplicate actions on overlapping elements. Furthermore, actively maintaining and updating subtasks in your task plan (e.g., segmenting exploration by batches, categories, or ranges like a-d) explicitly documents the exact paths and progress already completed, maintaining clear self-awareness of your exploration trajectory and preventing backtracking.\n 3. **Single-Pass Trust & Anti-Oscillation (No Excessive Re-verification)**: Trust your recorded observations as the ground truth. Once a section has been traversed and loggedLine truncated
|
"troubleshooter_template": "# ROLE\nYou are the Expert Android Device Troubleshooter. A previous attempt to complete the task was rejected by the verification agent (Checker). Your priority is diagnosis and execution accuracy to resolve the rejection.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor. Calibrate your scroll distance so that this anchor remains visible near the top of the subsequent screen, creating a seamless visual overlap that guarantees zero missed items. Terminate exploration when a definitive boundary is reached (e.g., list bottom reached or no new content appears after scrolling).\n 2. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n 3. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\". Only perform incremental rLine truncated
|
||||||
}
|
}
|
||||||
@@ -46,7 +46,10 @@ from artemis.tools.tool_wrapper import (
|
|||||||
get_tool_result_content,
|
get_tool_result_content,
|
||||||
invoke_tool_with_injection,
|
invoke_tool_with_injection,
|
||||||
)
|
)
|
||||||
from artemis.utils.coordinates import parse_swipe_parameters
|
from artemis.utils.coordinates import (
|
||||||
|
compute_smart_swipe_coordinates,
|
||||||
|
parse_swipe_parameters,
|
||||||
|
)
|
||||||
from artemis.utils.decorators import wrap_with_callbacks
|
from artemis.utils.decorators import wrap_with_callbacks
|
||||||
from artemis.utils.file import create_snapshot
|
from artemis.utils.file import create_snapshot
|
||||||
from artemis.utils.logger import get_logger
|
from artemis.utils.logger import get_logger
|
||||||
@@ -1056,23 +1059,25 @@ class OperatorNode:
|
|||||||
def swipe(
|
def swipe(
|
||||||
direction: Annotated[
|
direction: Annotated[
|
||||||
Literal["up", "down", "left", "right"] | None,
|
Literal["up", "down", "left", "right"] | None,
|
||||||
"Smart directional swipe ('up', 'down', 'left', 'right') for general page"
|
"Direction for scrolling and swiping: 'up' (drags bottom-to-top, scrolling down to reveal content below),"
|
||||||
" scrolling/browsing. Automatically computes safe swipe vectors (locking axes"
|
" 'down' (drags top-to-bottom, scrolling up to reveal content above),"
|
||||||
" to prevent sidebar/edge mis-clicks), retains a ~40% visual overlap anchor,"
|
" 'left' (drags right-to-left, scrolling right),"
|
||||||
" and prevents inertial flings: 'up' (drags bottom-to-top, scrolls down to reveal content below),"
|
" 'right' (drags left-to-right, scrolling left).",
|
||||||
" 'down' (drags top-to-bottom, scrolls up to reveal content above),"
|
|
||||||
" 'left' (drags right-to-left, scrolls right), 'right' (drags left-to-right, scrolls left).",
|
|
||||||
] = None,
|
] = None,
|
||||||
start: Annotated[
|
start: Annotated[
|
||||||
list[int] | None,
|
list[int] | None,
|
||||||
"Start point normalized coordinates [start_x, start_y] in 0-1000 scale for precise,"
|
"Start normalized coordinates [start_x, start_y] in 0-1000 scale for precise,"
|
||||||
" local interactions (e.g. adjusting sliders, SeekBars, fine range selection, or drag-and-drop).",
|
" local interactions (e.g. adjusting sliders, SeekBars, fine range selection, or drag-and-drop).",
|
||||||
] = None,
|
] = None,
|
||||||
end: Annotated[
|
end: Annotated[
|
||||||
list[int] | None,
|
list[int] | None,
|
||||||
"End point normalized coordinates [end_x, end_y] in 0-1000 scale for precise,"
|
"End normalized coordinates [end_x, end_y] in 0-1000 scale for precise,"
|
||||||
" local interactions (e.g. adjusting sliders, SeekBars, fine range selection, or drag-and-drop).",
|
" local interactions (e.g. adjusting sliders, SeekBars, fine range selection, or drag-and-drop).",
|
||||||
] = None,
|
] = None,
|
||||||
|
target: Annotated[
|
||||||
|
int | list[int] | str | None,
|
||||||
|
"Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within.",
|
||||||
|
] = None,
|
||||||
gesture: Annotated[
|
gesture: Annotated[
|
||||||
Literal["up", "down", "left", "right"] | list[int] | None,
|
Literal["up", "down", "left", "right"] | list[int] | None,
|
||||||
"Backward-compatible swipe gesture: smart direction string ('up', 'down', 'left', 'right')"
|
"Backward-compatible swipe gesture: smart direction string ('up', 'down', 'left', 'right')"
|
||||||
@@ -1080,23 +1085,23 @@ class OperatorNode:
|
|||||||
] = None,
|
] = None,
|
||||||
duration: Annotated[
|
duration: Annotated[
|
||||||
int | None,
|
int | None,
|
||||||
"Optional swipe/drag duration in milliseconds. If specified, the gesture will run at this exact speed."
|
"Optional swipe/drag duration in milliseconds (default 800). For drag-and-drop,"
|
||||||
" For drag-and-drop, list reordering, or sliding/adjusting sliders (e.g., volume, brightness, SeekBars),"
|
" list reordering, or sliding/adjusting sliders (e.g., volume, brightness, SeekBars),"
|
||||||
" set duration >= 1000 (e.g. 1500). If omitted for directional swipe, the system automatically"
|
" set duration >= 1000 (e.g. 1500). If omitted for directional swipe, duration is computed automatically.",
|
||||||
" calculates the best duration based on the swipe distance to prevent unintended inertial flings.",
|
|
||||||
] = None,
|
] = None,
|
||||||
):
|
):
|
||||||
"""[ACTION] Perform a swipe, drag, or slider-adjustment gesture on the screen.
|
"""[ACTION] Perform a swipe, drag, or slider-adjustment gesture on the screen.
|
||||||
|
|
||||||
• Smart Directional Scrolling ('direction'): Best for general browsing and page scrolling. Automatically computes safe swipe vectors (locking axes to prevent edge/sidebar mis-clicks), retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings.
|
• Directional Scrolling ('direction'): Recommended for general browsing and standard page scrolling in most scenarios. Automatically computes safe swipe vectors and adaptive duration, retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings. Supports scoping to a sub-container via 'target'. If it fails on certain custom layouts, fall back to specifying exact coordinates ('start' and 'end') directly.
|
||||||
• Precise Coordinate Gestures ('start', 'end'): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars) or drag-and-drop / list reordering. Drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.
|
• Precise Coordinate Gestures ('start', 'end'): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars), drag-and-drop / list reordering, or as a reliable fallback when directional scrolling fails on specific containers. Always drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), swipe fully to the extreme boundary.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
direction: Smart directional scrolling ('up', 'down', 'left', 'right'). Automatically computes safe swipe vectors, retaining 40% of the previous screen's visible content as an overlapping anchor: 'up' (drags bottom-to-top, scrolling down to reveal content below), 'down' (drags top-to-bottom, scrolling up to reveal content above), 'left' (drags right-to-left), 'right' (drags left-to-right).
|
direction: Smart directional scrolling ('up', 'down', 'left', 'right'). Automatically computes safe swipe vectors, retaining 40% visual overlap: 'up' (reveal content below), 'down' (reveal content above), 'left', 'right'.
|
||||||
start: Start normalized coordinates [start_x, start_y] in 0-1000 scale for precise/local interactions.
|
start: Start normalized coordinates [start_x, start_y] in 0-1000 scale.
|
||||||
end: End normalized coordinates [end_x, end_y] in 0-1000 scale for precise/local interactions.
|
end: End normalized coordinates [end_x, end_y] in 0-1000 scale.
|
||||||
|
target: Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within.
|
||||||
gesture: Backward-compatible parameter: direction string OR custom coordinates list [start_x, start_y, end_x, end_y] in 0-1000 scale.
|
gesture: Backward-compatible parameter: direction string OR custom coordinates list [start_x, start_y, end_x, end_y] in 0-1000 scale.
|
||||||
duration: Optional gesture duration in milliseconds. If omitted for directional swipe, the system automatically computes a reliable duration based on the swipe distance to prevent unintended inertial flings.
|
duration: Optional gesture duration in milliseconds (default 800).
|
||||||
"""
|
"""
|
||||||
return "Action Recorded"
|
return "Action Recorded"
|
||||||
|
|
||||||
@@ -1134,16 +1139,16 @@ class OperatorNode:
|
|||||||
def wait_for_delay(
|
def wait_for_delay(
|
||||||
time_in_ms: Annotated[
|
time_in_ms: Annotated[
|
||||||
int,
|
int,
|
||||||
"The duration to wait in milliseconds (e.g., 2000 for 2"
|
"The exact duration to wait in milliseconds. Accurately convert the"
|
||||||
" seconds, 5000 for 5 seconds). Use a reasonable delay to allow"
|
" required time duration into milliseconds based on your objective"
|
||||||
" pages to load, apps to start, or screen animations to"
|
" or plan (e.g., 2000 for 2s, 5000 for 5s, 60000 for 1 minute,"
|
||||||
" finish.",
|
" 180000 for 3 minutes, 300000 for 5 minutes).",
|
||||||
],
|
],
|
||||||
):
|
):
|
||||||
"""[ACTION] Pause execution and wait for a fixed duration.
|
"""[ACTION] Pause execution and wait for a specified duration in milliseconds.
|
||||||
|
|
||||||
Use this to handle loading states, app launches, network buffers, or
|
Use this whenever you need time to elapse—whether for UI loading, animations,
|
||||||
simple timing gaps between actions when the UI is transitioning.
|
screen transitions, or longer scheduled delays and intervals specified in the task.
|
||||||
"""
|
"""
|
||||||
return "Action Recorded"
|
return "Action Recorded"
|
||||||
|
|
||||||
@@ -1356,42 +1361,17 @@ class OperatorNode:
|
|||||||
duration = parsed_duration
|
duration = parsed_duration
|
||||||
|
|
||||||
if kind == "direction":
|
if kind == "direction":
|
||||||
gesture = target
|
x1, y1, x2, y2, smart_dur = compute_smart_swipe_coordinates(
|
||||||
# Align with FinalRun directional swipe vector rules:
|
direction=target,
|
||||||
# Locks X axis to 60% of width for up/down to prevent mis-clicks on alphabet sidebar.
|
target=args.get("target"),
|
||||||
# Horizontal Y axis is locked to 60% of height.
|
indexed_elements=getattr(state, "indexed_elements", None),
|
||||||
if gesture == "up":
|
ui_hierarchy=getattr(state, "ui_tree", None),
|
||||||
coords = [
|
width=width,
|
||||||
int(width * 0.6),
|
height=height,
|
||||||
int(height * 0.8),
|
duration=duration,
|
||||||
int(width * 0.6),
|
)
|
||||||
int(height * 0.2),
|
if duration is None:
|
||||||
]
|
duration = smart_dur
|
||||||
elif gesture == "down":
|
|
||||||
coords = [
|
|
||||||
int(width * 0.6),
|
|
||||||
int(height * 0.2),
|
|
||||||
int(width * 0.6),
|
|
||||||
int(height * 0.8),
|
|
||||||
]
|
|
||||||
elif gesture == "left":
|
|
||||||
coords = [
|
|
||||||
int(width * 0.8),
|
|
||||||
int(height * 0.6),
|
|
||||||
int(width * 0.2),
|
|
||||||
int(height * 0.6),
|
|
||||||
]
|
|
||||||
elif gesture == "right":
|
|
||||||
coords = [
|
|
||||||
int(width * 0.2),
|
|
||||||
int(height * 0.6),
|
|
||||||
int(width * 0.8),
|
|
||||||
int(height * 0.6),
|
|
||||||
]
|
|
||||||
else:
|
|
||||||
return [], f"Error: Invalid swipe direction '{gesture}'."
|
|
||||||
|
|
||||||
x1, y1, x2, y2 = coords
|
|
||||||
elif kind == "coords" and isinstance(target, list) and len(target) == 4:
|
elif kind == "coords" and isinstance(target, list) and len(target) == 4:
|
||||||
try:
|
try:
|
||||||
nx1, ny1, nx2, ny2 = map(float, target)
|
nx1, ny1, nx2, ny2 = map(float, target)
|
||||||
@@ -1413,19 +1393,17 @@ class OperatorNode:
|
|||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
# Calculate duration automatically if not provided (safe range 280-400ms to avoid long-press and fling)
|
# Calculate duration automatically if not provided (safe range 350-900ms to eliminate Android inertial fling)
|
||||||
if duration is None:
|
if duration is None:
|
||||||
dist = ((x2 - x1) ** 2 + (y2 - y1) ** 2) ** 0.5
|
dist = ((x2 - x1) ** 2 + (y2 - y1) ** 2) ** 0.5
|
||||||
if dist <= 400:
|
if dist <= 300:
|
||||||
duration = 280
|
|
||||||
elif dist <= 800:
|
|
||||||
duration = 320
|
|
||||||
elif dist <= 1200:
|
|
||||||
duration = 350
|
duration = 350
|
||||||
elif dist <= 1600:
|
elif dist <= 600:
|
||||||
duration = 380
|
duration = 550
|
||||||
|
elif dist <= 1000:
|
||||||
|
duration = 800
|
||||||
else:
|
else:
|
||||||
duration = 400
|
duration = 900
|
||||||
|
|
||||||
nx1, ny1 = int(round(x1 * 1000.0 / width)), int(round(y1 * 1000.0 / height))
|
nx1, ny1 = int(round(x1 * 1000.0 / width)), int(round(y1 * 1000.0 / height))
|
||||||
nx2, ny2 = int(round(x2 * 1000.0 / width)), int(round(y2 * 1000.0 / height))
|
nx2, ny2 = int(round(x2 * 1000.0 / width)), int(round(y2 * 1000.0 / height))
|
||||||
|
|||||||
@@ -35,6 +35,7 @@ from artemis.tools.scratchpad import (
|
|||||||
get_update_note_tool_pure,
|
get_update_note_tool_pure,
|
||||||
)
|
)
|
||||||
from artemis.tools.video_tool import get_video_analyzer_tool_pure
|
from artemis.tools.video_tool import get_video_analyzer_tool_pure
|
||||||
|
from artemis.tools.tool_wrapper import invoke_tool_with_injection
|
||||||
from artemis.utils.logger import get_logger
|
from artemis.utils.logger import get_logger
|
||||||
from artemis.utils.task_tree import build_plan_and_history, get_active_subgoal_hashes
|
from artemis.utils.task_tree import build_plan_and_history, get_active_subgoal_hashes
|
||||||
from pydantic import BaseModel
|
from pydantic import BaseModel
|
||||||
@@ -210,25 +211,28 @@ async def outputter(
|
|||||||
args = tc["args"]
|
args = tc["args"]
|
||||||
|
|
||||||
logger.info(f"Outputter executing tool {tool_name} with args: {args}")
|
logger.info(f"Outputter executing tool {tool_name} with args: {args}")
|
||||||
|
tool_map = {
|
||||||
|
"get_step_details": details_tool,
|
||||||
|
"get_step_screenshot": screenshot_tool,
|
||||||
|
"search_history_for_text": search_tool,
|
||||||
|
"list_notes": list_notes_tool,
|
||||||
|
"read_note": read_note_tool,
|
||||||
|
"save_note": save_note_tool,
|
||||||
|
"update_note": update_note_tool,
|
||||||
|
"append_note": append_note_tool,
|
||||||
|
"video_analyzer": video_analyzer_tool,
|
||||||
|
}
|
||||||
|
|
||||||
try:
|
try:
|
||||||
if tool_name == "get_step_details":
|
selected_tool = tool_map.get(tool_name)
|
||||||
result = await details_tool.ainvoke(args)
|
if selected_tool:
|
||||||
elif tool_name == "get_step_screenshot":
|
result = await invoke_tool_with_injection(
|
||||||
result = await screenshot_tool.ainvoke(args)
|
tool=selected_tool,
|
||||||
elif tool_name == "search_history_for_text":
|
args=args,
|
||||||
result = await search_tool.ainvoke(args)
|
tool_call_id=tc["id"],
|
||||||
elif tool_name == "list_notes":
|
state=graph_output,
|
||||||
result = await list_notes_tool.ainvoke(args)
|
record_trace=True,
|
||||||
elif tool_name == "read_note":
|
)
|
||||||
result = await read_note_tool.ainvoke(args)
|
|
||||||
elif tool_name == "save_note":
|
|
||||||
result = await save_note_tool.ainvoke(args)
|
|
||||||
elif tool_name == "update_note":
|
|
||||||
result = await update_note_tool.ainvoke(args)
|
|
||||||
elif tool_name == "append_note":
|
|
||||||
result = await append_note_tool.ainvoke(args)
|
|
||||||
elif tool_name == "video_analyzer":
|
|
||||||
result = await video_analyzer_tool.ainvoke(args)
|
|
||||||
else:
|
else:
|
||||||
result = f"Error: Tool {tool_name} is not supported."
|
result = f"Error: Tool {tool_name} is not supported."
|
||||||
status = (
|
status = (
|
||||||
|
|||||||
@@ -1,18 +1,18 @@
|
|||||||
{
|
{
|
||||||
"blocks": {
|
"blocks": {
|
||||||
"role": "## Role\nYou are the Planner agent. Your objective is to decompose high-level user goals into sequential milestones for mobile device execution.",
|
"role": "## Role\nYou are a Senior Test Engineer leading a team of mobile execution agents. Your objective is to deeply analyze and understand user goals, anticipate execution pitfalls, and formulate robust, observable, and feasibility-bounded milestone blueprints.",
|
||||||
|
"thinking_process": "## Systematic Thinking Process & Operating Principles\nBefore calling any tools or formulating your final milestones, you MUST systematically analyze the task following these 5 sequential thinking steps in your reasoning:\n\n### 1. Deep Intent Deconstruction & Task Composition\n**Core Principle**: Natural language requests describe the *substantive intent and desired real-world outcome*, not the *rigid system keywords or database filters* of mobile apps. Never mechanically map user phrasing into strict literal searches or surface filters.\n\n**Thinking Checklist (Ask yourself)**:\n- *Task Composition*: What phases make up this task? Is it a sequence of one-off linear steps (e.g., open app, configure settings), an iterative loop over multiple items (e.g., inspecting a list of search results), or a periodic polling loop (e.g., refreshing to monitor new messages, waiting for battery level)?\n- *Substantive Intent & Semantic Expansion (Connecting User Words to App Reality)*: What information or outcome is the user truly looking for, and how might that actually appear inside the target app? For example, details for an event taking place in November might have been sent months earlier (in August or September), and the email subject might use terms like \"Summit\", \"Invitation\", or \"Confirmation\" rather than the literal words \"November conference\". Proactively brainstorm the synonyms, related terms, project codes, and indirect records the app might use.\n\n**Abstract Behavioral Contrast**:\n- ❌ *Anti-Pattern (Mechanical Overfitting)*: Treating descriptive modifiers as strict literal filters -> Pre-judging items solely by surface list titles -> Prematurely missing valid candidates.\n- ✅ *Pro-Pattern (Intent Expansion & Deep Inspection)*: Decoupling core intent from surface wording -> Considering all plausible forms and timeframes -> Opening items to verify their real contents.\n\n### 2. Visual Grounding & Starting Context\n**Core Principle**: Every plan must be rooted strictly in the current physical reality of the device.\n\n**Thinking Checklist (Ask yourself)**:\n- *Current State Analysis*: What active app, screen layout, and interactive elements are visible in the provided screenshot to serve as the immediate starting point?\n\n### 3. Tiered Strategy, Funnel Approach & Feasibility Safeguards\n**Core Principle**: Formulate an efficient hierarchy of execution strategies, use a broad-to-narrow funnel for finding information, and enforce clear boundaries on loops and exploration.\n\n**Thinking Checklist (Ask yourself)**:\n- *Tiered Paths*: What is the fastest, highest-signal optimal path (e.g., targeted in-app search, direct shortcuts)? If that fails or yields no results, what is the pragmatic fallback path (e.g., browsing categories, chronological scanning)?\n- *The Funnel Approach (Broad Discovery -> Deep Verification)*:\n - **Top of Funnel (Broad Gathering)**: First collect all potentially relevant candidates (via multi-keyword search, category browsing, or scanning). Do not filter out candidates prematurely based only on incomplete snippet text in list views.\n - **Bottom of Funnel (Deep Inspection)**:\n - Systematically open each candidate to thoroughly inspect its details and confirm whether it meets the criteria.\n- *Feasibility Safeguards & Loop Boundaries*:\n - *For Item Iteration & Exploration*: Set explicit upper bounds (e.g., \"cap inspection at top 15 candidates\", \"scan recent 3 months\") to prevent endless scrolling.\n - *For Periodic Monitoring / Polling*: Specify the check action (e.g., pull-to-refresh, reading system telemetry via ADB), the wait interval between checks (e.g., check every 5 minutes to conserve resources), and the exit criteria (e.g., stop when condition is met, or continue until manually stopped).\n\n### 4. Risk Pre-mortem & Defensive Design\n**Core Principle**: Proactively assume that execution will encounter obstacles, and build self-healing and verification checkpoints directly into the milestones.\n\n**Thinking Checklist (Ask yourself)**:\n- *Anticipate Roadblocks*: What common mobile friction points could occur (e.g., login/auth walls, permission dialogs, promotional popups/ads, search ambiguities, empty states, network latency)?\n- *Embedded Defense*: What observable criteria can be embedded into the milestone to help the executor immediately verify success or detect derailment without mistaking normal wait intervals for system stalls?\n\n### 5. Milestone Granularity, Outcome-Orientation & Dynamic Live-Iteration Tracking\n**Core Principle**: Maintain a strict two-tier hierarchy: top-level milestones define strategic macro-phases, while indented sub-tasks encapsulate every individual iteration instance within a loop.\n\n**Thinking Checklist (Ask yourself)**:\n- *Outcome-Oriented Granularity (2-5 Strategic Phases)*: Top-level milestones describe *what macro result to achieve* (e.g., \"Search for candidate emails and extract confirmed itiLine truncated
|
||||||
|
"capabilities_operator": "## Execution Capabilities\n\nThe execution agent primarily interacts with the mobile device and analyzes screen recordings and logs. It can perform the following types of actions:\n- **UI Interaction**: Tapping, swiping, and entering text on the screen.\n- **App Management**: Launching and stopping specific applications.\n- **Information Retrieval & Analysis**: Reading on-screen text and UI hierarchy, searching the web for external information, querying system logs, and analyzing screen recordings.\n- **Terminal & Background Operations**: Executing arbitrary ADB shell commands on the device for exploration. It is capable of running long-running ADB commands asynchronously in the background as tasks.\n\n### Capability Boundaries & Planning Rules\n- **Cross-App Operations**: When a task requires interacting with multiple applications, split the operations into distinct milestones for each app. Avoid mixing actions for different apps in a single milestone.",
|
||||||
|
"output_protocol_initial": "## Task Plan & Scratchpad Protocol\n\n1. **Articulate Thinking**: Begin by reasoning through the 5-step **Systematic Thinking Process** to establish your strategy, funnel approach, bounds, and pre-mortem checks.\n2. **Designate Notes in Milestones & Pre-generate Answer Cards**: If the task involves collecting findings, logging periodic monitoring events, recording candidates, or analytical summaries, designate clear, descriptive note names directly in the milestone descriptions (e.g., `candidates_list`, `monitoring_log`, `schedule_summary`). You can and should invoke the `save_note` tool to pre-generate these structured template files (answer card / scratchpad skeletons with Markdown headers, sections, or tables) before submitting `task_plan`. This provides downstream executors with a clear, ready-to-fill schema.\n3. **Submit Plan**: Submit your initial plan by invoking the `save_note` tool with `key='task_plan'`. Avoid writing the plan only in your text response, as it will only be recognized by the system if submitted via the tool.\n\n### Plan Formatting Rules\n- Format the plan strictly as a Markdown task list starting with `- [ ]` for uncompleted items.\n- For linear macro-milestones: `- [ ] <Phase Description>`\n- For repetitive, iterative, or periodic polling milestones, tag the item with `[Loop]` and declare its Exit Boundary and Interval (e.g., `- [ ] [Loop] <Overall Loop / Monitoring Goal> (Exit: <Condition or Continuous until manually stopped>; Interval: <e.g., every 5 minutes>)`).\n- **Strict Two-Tier Hierarchy & Loop Subtask Contract**:\n 1. **Subtask Nesting Rule**: ALL individual iteration instances, polling checks, candidate inspections, or baseline setups MUST be formatted as 2-space indented sub-tasks (` - [ ]`) directly nested under their parent `[Loop]` milestone.\n 2. **Mandatory First-Instance Initialization**: Every `[Loop]` milestone MUST be initialized with its first concrete child sub-task (e.g., ` - [ ] Inspect Candidate #1: ...` or ` - [ ] Polling Check #1 (Baseline & Initial Sync): ...`).\n 3. **No Flattened Cycles**: NEVER elevate individual checks, recurring polling cycles, or candidate inspections to top-level milestones outside the `[Loop]`. Top-level items are reserved exclusively for strategic macro-phases.\n- Ensure the content of the `task_plan` note contains only the task list, excluding introductions, explanations, conversational text, or headers.",
|
||||||
|
"examples_initial": "## Master Example: Demonstrating the Complete Cognitive Paradigm\n\n**Goal**: \"Find and summarize all schedule arrangements and venue details for the upcoming November tech conference from my emails.\"\n\n### Naive / Incorrect Approaches:\n1. *Flawed Semantics & Too Granular*:\n- [ ] Tap search icon in Gmail\n- [ ] Type \"November\"\n- [ ] Open first email and tap back\n*(Fails: Micromanages atomic UI taps; assumes emails for November arrangements must have \"November\" in the title or be received in November)*\n2. *Vague & Unbounded Trap*:\n- [ ] Open email and check all emails for conference details\n*(Fails: Unbounded scan trap; causes infinite scrolling and step budget exhaustion on mobile)*\n\n---\n\n### Correct Approach (Full Cognitive Thinking & Robust Plan):\n\n#### Step-by-Step Thinking:\n1. **Intent & Semantic Expansion**:\n - *Core Objective*: Extract confirmed schedule dates, sessions, and venue info for the November tech conference.\n - *Semantic Insight*: Conference communications are often sent months in advance (e.g., ticket confirmations or registration emails in August/September). The subject line may say \"Global Developer Summit\", \"Tech Expo Agenda\", or \"Registration Confirmed\" rather than literally \"November conference\".\n - *Task Composition*: Linear setup (open email app and search) -> Iterative candidate inspection loop (examine each promising email) -> Final consolidation.\n2. **Visual Grounding**:\n - The device screenshot shows the Home screen with the Gmail app icon visible. The plan must start by opening the email client.\n3. **Tiered Strategy, Funnel Approach & Safeguards**:\n - *Tier 1 (Optimal)*: Use in-app search with rich semantic keywords (\"conference\", \"summit\", \"schedule\", \"agenda\", \"venue\", \"November\").\n - *Tier 2 (Fallback)*: If search is unavailable or yields zero results, fallback to chronological scanning of recent 3 months.\n - *Funnel (Broad Gathering -> Deep Inspection)*: Gather candidate emails from search results; do not discard candidates based solely on list previews. Open each candidate thread to inspect full body details.\n - *Feasibility Boundary*: Cap the candidate email inspection pool at top 15 emails to prevent infinite scrolling.\n4. **Risk Pre-mortem**:\n - *Risks*: Promotional marketing spam; conflicting drafts with preliminary dates; cancelled sessions.\n - *Defensive Check*: Instruct downstream executor to filter out promotional newsletters, verify session timestamps, and confirm final venue address.\n5. **Milestone Granularity & Scratchpad Blueprint**:\n - Focus milestones on business outcomes (search, inspect candidates, consolidate), avoiding micromanaged click-by-click steps.\n - Use indented sub-tasks to guide the loop of inspecting each candidate email, initializing with Item #1 and adding subsequent items as they are identified. This prevents the agent from losing state or re-inspecting the same items.\n - Designate note `november_conference_summary` as the destination for recording findings and pre-generate the structured template note.\n\n#### Tool Invocation:\n1. Pre-generate designated answer card note template:\n`save_note(key='november_conference_summary', content='# November Tech Conference Summary\\n\\n## Key Details\\n- Conference Name: \\n- Dates: \\n- Venue / Address: \\n\\n## Daily Schedule & Sessions\\n(To be populated during email inspection)\\n')`\n\n2. Submit milestone plan:\n`save_note(key='task_plan', content='- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\\n - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\\n- [ ] Review and consolidate all extracted information in `november_conference_summary` into a clear, finalized conference itinerary')`\n\n#### Content of `task_plan` Note:\n- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\n - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\n- [ ] ReviLine truncated
|
||||||
"role_validator": "## Role\nYou are the Task Plan Validator. The Operator has attempted to modify the top-level task plan. Your job is to ensure the proposed new plan still aligns with the user's original intent and is viable.",
|
"role_validator": "## Role\nYou are the Task Plan Validator. The Operator has attempted to modify the top-level task plan. Your job is to ensure the proposed new plan still aligns with the user's original intent and is viable.",
|
||||||
"guidelines": "## Operating Principles\n\n### 1. Visual Grounding (Start from Current State)\nBase all planning decisions strictly on the provided screenshot of the current device state. Your plan must begin from the current context of the screen.\n\n### 2. Milestone Granularity\nDefine high-level milestones rather than granular UI operations. The milestones you establish are essential checkpoints that the task must pass through; your goal is to keep the task on the right track, not to think about how the task should be completed step-by-step.\n\n### 3. Flexibility and Possibility Retention (Avoid Over-Specification)\nAvoid unnecessarily restricting the execution path or specific tools/applications unless strictly mandated by the user. For example, if the user asks to perform a task that can be accomplished via multiple interfaces (such as a native app or a web browser), describe the milestone in terms of the desired outcome rather than forcing a specific application or interface (e.g. forcing the use of a specific native app when a web browser could also serve). Allow the downstream execution agent the flexibility to choose the most viable path based on the dynamic state of the device.\n\n### 4. Heuristics for Complex & Iterative Operations\nWhen a milestone requires the downstream agent to perform complex iterations, maintain state, or map computed data back to the UI (e.g., adding a pre-selected list of items to a system), you MUST provide heuristics via indented sub-tasks. These sub-tasks should instruct the agent on the exact iterative process (e.g., searching for each item one by one from a note, applying it, and verifying the final state).",
|
"validator_guidelines": "## Validation Principles\n- **Goal Alignment**: Ensure the new plan ultimately accomplishes the `Initial Goal` requested by the user. If the Operator has abandoned the core objective or fundamentally altered it, you must reject the plan.\n- **Logical Coherence**: Verify that the proposed milestones form a logical, achievable sequence to reach the goal. Support valid ongoing iteration/polling subgoals and trigger branches without misconstruing periodic checks as infinite loops. Reject the plan if it introduces contradictory steps, unhandled deadlocks, or obvious logical fallacies.\n- **Loop & Monitoring Protection**: If the existing plan contains an active `[Loop]` milestone (especially continuous polling / monitoring with exit condition `until manually stopped`), you MUST REJECT any proposed plan that deletes the `[Loop]` milestone, removes the loop tag, or marks a continuous monitoring milestone as completed (`[x]`) without explicit external user instruction.\n- **Adaptive Acceptance**: The Operator operates in a dynamic environment and may encounter unexpected UI states or dynamic event triggers. You should approve alternative strategies, extra intermediate steps, loop status tracking, or necessary workarounds as long as they serve the primary goal effectively.\n- **Constructive Feedback**: If you must reject the plan, provide clear, objective, and actionable feedback explaining the logical flaw or goal deviation, and suggest how the Operator should correct the plan."
|
||||||
"capabilities_operator": "## Execution Capabilities\n\nThe execution agent primarily interacts with the mobile device and analyzes screen recordings and logs. It can perform the following types of actions:\n- **UI Interaction**: Tapping, swiping, and entering text on the screen.\n- **App Management**: Launching and stopping specific applications.\n- **Information Retrieval & Analysis**: Using OCR to read text on screen, searching the web for external information, querying system logs, and analyzing screen recordings.\n- **Terminal & Background Operations**: Executing arbitrary ADB shell commands on the device for exploration. It is capable of running long-running ADB commands asynchronously in the background as tasks.\n\n### Capability Boundaries & Planning Rules\n- **Cross-App Operations**: When a task requires interacting with multiple applications, split the operations into distinct milestones for each app. Avoid mixing actions for different apps in a single milestone.",
|
|
||||||
"output_protocol_initial": "## Task Plan & Scratchpad\nSubmit your initial plan by invoking the `save_note` tool with `key='task_plan'`. Avoid writing the plan in your text response, as it will only be recognized if submitted via the tool.\n\nFormat the plan as a Markdown task list starting with `- [ ]` for uncompleted items. You can use indented sub-tasks (e.g., ` - [ ]`) to break down complex exploratory steps and provide heuristics to guide the downstream agent. Ensure the content of the `task_plan` note contains only the task list, excluding any introductions, explanations, or headers.\n\n### Creating Additional Notes\nYou are highly recommended to create additional empty notes (like answer sheets) for the downstream executor to fill out with findings, lists, or analytical information relevant to the task. For example, if the user asks you to analyze specific YouTube videos, you should pre-create an empty note with a descriptive name (e.g., `analysis_of_xxx_videos`), and make sure to put the note name directly inside your milestones to guide the downstream executor in filling out this answer sheet.",
|
|
||||||
"examples_initial": "## Examples\n\n### Example 1: Basic Task\n**Goal**: \"Send 'I will be late' to Alice on WhatsApp\"\n\n**Incorrect (Too granular)**:\n- [ ] Tap search icon\n- [ ] Type Alice\n- [ ] Tap Alice's avatar\n\n**Correct (Milestones)**:\n- [ ] Open the chat window with Alice in WhatsApp\n- [ ] Successfully send the message \"I will be late\" in Alice's chat window\n\n### Example 2: Information Retrieval & Scratchpad\n**Goal**: \"Find 10 nature videos on YouTube\"\n\n**Incorrect (Vague or directly jumping to end)**:\n- [ ] Open YouTube\n- [ ] Search nature videos and pick 10\n\n**Correct (Leveraging Scratchpad & Sub-tasks Heuristics)**:\n*(Note: As the Planner, pre-create the required empty notes `nature_videos_candidates` and `nature_videos` using the note-taking tools before completing the plan)*\n- [ ] Search for \"nature videos\" on YouTube\n- [ ] Save a list of 15 candidate videos into a temporary note called `nature_videos_candidates`\n - [ ] Add candidate 1\n - [ ] Add candidate 2\n - [ ] (Continue until 15 candidates are found)\n- [ ] Evaluate candidate videos from `nature_videos_candidates` one by one, select exactly 10 correct ones, and save them into `nature_videos`\n - [ ] Evaluate candidate 1: watch, filter out if irrelevant, save if correct\n - [ ] Evaluate candidate 2\n - [ ] (Continue evaluating one by one until 10 are selected)",
|
|
||||||
"validator_guidelines": "## Validation Principles\n- **Goal Alignment**: Ensure the new plan ultimately accomplishes the `Initial Goal` requested by the user. If the Operator has abandoned the core objective or fundamentally altered it, you must reject the plan.\n- **Logical Coherence**: Verify that the proposed milestones form a logical, achievable sequence to reach the goal. Reject the plan if it introduces contradictory steps, infinite loops, or obvious logical fallacies.\n- **Adaptive Acceptance**: The Operator operates in a dynamic environment and may encounter unexpected UI states. You should approve alternative strategies, extra intermediate steps, or necessary workarounds as long as they serve the primary goal effectively.\n- **Constructive Feedback**: If you must reject the plan, provide clear, objective, and actionable feedback explaining the logical flaw or goal deviation, and suggest how the Operator should correct the plan."
|
|
||||||
},
|
},
|
||||||
"modes": {
|
"modes": {
|
||||||
"initial_plan": {
|
"initial_plan": {
|
||||||
"system": [
|
"system": [
|
||||||
"role",
|
"role",
|
||||||
"guidelines",
|
"thinking_process",
|
||||||
"capabilities_operator",
|
"capabilities_operator",
|
||||||
"output_protocol_initial",
|
"output_protocol_initial",
|
||||||
"examples_initial"
|
"examples_initial"
|
||||||
@@ -27,4 +27,4 @@
|
|||||||
"human": "Initial Goal: {{ initial_goal }}\n\n### Execution History\n{{ history_str }}\n\n### Operator's Thoughts Before Modifying\n{% if operator_native_thinking %}Native Thinking:\n{{ operator_native_thinking }}\n\n{% endif %}{% if operator_raw_thinking %}Raw Thinking:\n{{ operator_raw_thinking }}\n\n{% endif %}### Old Plan\n{{ content_before }}\n\n### Proposed New Plan\n{{ content_after }}\n\nPlease evaluate if the proposed new plan should be approved."
|
"human": "Initial Goal: {{ initial_goal }}\n\n### Execution History\n{{ history_str }}\n\n### Operator's Thoughts Before Modifying\n{% if operator_native_thinking %}Native Thinking:\n{{ operator_native_thinking }}\n\n{% endif %}{% if operator_raw_thinking %}Raw Thinking:\n{{ operator_raw_thinking }}\n\n{% endif %}### Old Plan\n{{ content_before }}\n\n### Proposed New Plan\n{{ content_after }}\n\nPlease evaluate if the proposed new plan should be approved."
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -13,16 +13,19 @@ You are the Summarizer Agent for a mobile automation system. Your task is to sum
|
|||||||
- **BANNED WORDS**: "successfully", "completed", "entered", "navigated to", "failed", "unsuccessful", "could not", "achieved".
|
- **BANNED WORDS**: "successfully", "completed", "entered", "navigated to", "failed", "unsuccessful", "could not", "achieved".
|
||||||
3. **Focus on Intent & Strategy**: The specific physical action (exact coordinates, swipes) is now recorded automatically. Focus primarily on the high-level Intent and reasoning. If you realize the Operator is on the wrong path and needs to backtrack or pivot, describe this as a shift in strategy.
|
3. **Focus on Intent & Strategy**: The specific physical action (exact coordinates, swipes) is now recorded automatically. Focus primarily on the high-level Intent and reasoning. If you realize the Operator is on the wrong path and needs to backtrack or pivot, describe this as a shift in strategy.
|
||||||
4. **Intended Context Transitions**: Describe the intended navigation path ("From Where -> Intended Where"). Do not declare that the transition actually happened. (e.g., "From the Home screen, I intended to open the Network Settings view").
|
4. **Intended Context Transitions**: Describe the intended navigation path ("From Where -> Intended Where"). Do not declare that the transition actually happened. (e.g., "From the Home screen, I intended to open the Network Settings view").
|
||||||
5. **Preserve Verifications & Missing Items**: If the Operator verified any state in this step, or explicitly mentioned what prerequisites or items are still missing to accomplish the goal, you MUST state these contents in detail in the summary.
|
5. **Preserve Verifications & Iteration Progress**: If the Operator verified any state in this step, processed a specific candidate item/polling round, or explicitly mentioned what prerequisites or items are still missing, you MUST succinctly preserve these concrete progress details and verified items in the summary.
|
||||||
6. **Objective Intervention Logs**: If the Failure Analyzer intervened, faithfully summarize what triggered the intervention and what the analyzer executed, without judging the outcome.
|
6. **Objective Intervention Logs**: If the Failure Analyzer intervened, faithfully summarize what triggered the intervention and what the analyzer executed, without judging the outcome.
|
||||||
|
|
||||||
# LOOP & STAGNATION DETECTION
|
# LOOP & STAGNATION DETECTION
|
||||||
Use the `RECENT 10 STEPS HISTORY` to intelligently detect loops or stagnation:
|
Use the `RECENT 10 STEPS HISTORY` to intelligently detect loops or stagnation:
|
||||||
1. **Physical Action Loops**: Be vigilant for loops spanning multiple steps based on physical interactions. If you detect a valid action loop, explicitly mention the pattern and the current iteration count (e.g., "This is the 3rd cycle of attempting to access the video group folder").
|
1. **Physical Action Loops vs. Planned Iterations**:
|
||||||
2. **Cognitive Stagnation**: While you generally ignore internal background tool calls (e.g., reading logs, checking states) when counting physical action loops, there is an exception: if you detect continuous multiple steps of purely internal tool calls without any physical actions, you MUST objectively record this strategic "stagnation" or "analysis loop".
|
- *Planned Iterations / Monitoring (Valid Progress)*: If the Operator is systematically processing different candidate items in a loop, or executing planned periodic polling cycles with wait intervals under a `[Loop]` milestone, record this objectively as planned progress (e.g., "Inspected Candidate #2", "Completed Polling Round #2 for new messages"). Do NOT flag planned iterations across different items or spaced polling as abnormal loops.
|
||||||
|
- *Unintended Action Loops (Anomalies)*: If the Operator is repeatedly clicking the exact same target, failing to progress on the same item, or oscillating blindly between identical states without updating progress notes, explicitly record the anomalous loop pattern and iteration count (e.g., "This is the 3rd cycle of failing to open the same folder").
|
||||||
|
2. **Cognitive Stagnation**: While you generally ignore internal background tool calls (e.g., reading logs, checking states) when counting physical action loops, if you detect continuous multiple steps of purely internal tool calls without any physical actions or note updates, objectively record this "stagnation" or "analysis loop".
|
||||||
|
|
||||||
# INPUT STRUCTURE
|
# INPUT STRUCTURE
|
||||||
You will receive:
|
You will receive:
|
||||||
|
- **CURRENT PLAN & TASK PLAN**: The current task plan including milestones, active subgoals, and live iteration checklists to help you understand the strategic context and loop phase.
|
||||||
- **RECENT 10 STEPS HISTORY**: A brief history of the last 10 steps to help you detect loops and stagnation patterns.
|
- **RECENT 10 STEPS HISTORY**: A brief history of the last 10 steps to help you detect loops and stagnation patterns.
|
||||||
- **CHRONOLOGICAL STEP TRACE (CURRENT STEP)**: The step-by-step chronological history of the current execution turn, including:
|
- **CHRONOLOGICAL STEP TRACE (CURRENT STEP)**: The step-by-step chronological history of the current execution turn, including:
|
||||||
- `[Operator Monologue] / [Operator Native Thought]`: The operator's planning, intentions, progress counting, reasoning, verified states, and explicit mentions of missing prerequisites.
|
- `[Operator Monologue] / [Operator Native Thought]`: The operator's planning, intentions, progress counting, reasoning, verified states, and explicit mentions of missing prerequisites.
|
||||||
|
|||||||
Loaded 100 of 249 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user