chenmingxuan/Artemis-Eapil
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 32 Wiki Activity

feat: introduce MCP server, cross-platform runtime, and Showcase UI reorganization

Browse Source 
- Showcase UI Reorganization:
  * Move web UI dashboard from root directory into apps/showcase_ui/ for cleaner project structure
  * Add 'artemis ui' CLI command (artemis/interfaces/cli/commands/ui.py) and launcher scripts for the interactive showcase console
- Standalone MCP Server & Notifiers (mcp_server/):
  * Implement dedicated MCP server package with mobile automation tools (device_state, inspect_trace, task_manager, task_runner)
  * Add multi-channel notifiers (AgentAPI, Desktop notifications, Webhooks, File output, and Composite routing)
  * Introduce background task runner and trace store for autonomous mobile execution monitoring
- Cross-Platform Runtime & Toolchain Support:
  * Add artemis/platform/ with OS-specific implementations for Linux, macOS (Darwin), and Windows
  * Add artemis/runtime/supervisor.py for resilient process and agent runtime supervision
  * Add artemis/toolchain/ with dependency resolution and SDK descriptors
  * Add cross-platform launcher scripts (start.sh, start.bat) and installer scripts (scripts/install_deps.*)
- Core Enhancements & Diagnostics:
  * Enhance coordinate normalization, OCR API fallback, and perception pipeline
  * Add credential validation (credentials_validator.py) and system diagnostics
  * Update bilingual documentation (README.md, README_CN.md) and architecture diagrams
- Test Suite Expansion:
  * Add integration tests for cloud gateway, remote ADB, and Artemis login
  * Add unit tests covering core diagnostics, platform detection, toolchain, credentials validator, optional OCR, and MCP tools
This commit is contained in:
somew1nd committed 2026-08-18 13:59:27 -07:00
1 parent 78e478f502
commit 9a5aaf99b7
248 files changed
+37058 -9171
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.dockerignore
-16
View File
@@ -1,16 +0,0 @@
.git
.venv
__pycache__
*.pyc
traces/
latte-traces/
debug/replay/
debug_pub/replay/
*.mp4
*.jpg
*.png
pytest_temp/
/scratch/
android_world*
/dist/
node_modules/
.env.example
+18 -13
View File
@@ -1,19 +1,24 @@
# ==============================================================================
# ☕ Latte Environment Configuration
# Artemis Environment Configuration Template
# ==============================================================================
# Copy this file to .env and fill in your API credentials:
# cp .env.example .env
# 1. Primary LLM Provider API Key (Required for Agent Planning & Reasoning)
GEMINI_API_KEY=your_gemini_api_key_here
# LLM Providers (Configure at least one)
GEMINI_API_KEY=
GOOGLE_API_KEY=
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
OPEN_ROUTER_API_KEY=
XAI_API_KEY=
# 2. Google Cloud Vision OCR API Key (Required for fast text perception & XML fusion)
OCR_API_KEY=your_google_cloud_vision_api_key_here
# Google Cloud Vision OCR (Optional - for advanced OCR processing)
OCR_API_KEY=
# Optional: Alternative LLM Providers
# OPENAI_API_KEY=your_openai_api_key_here
# ANTHROPIC_API_KEY=your_anthropic_api_key_here
# OPEN_ROUTER_API_KEY=your_openrouter_api_key_here
# XAI_API_KEY=your_xai_api_key_here
# Optional: Target Android Device Serial (Leave commented for auto-detection)
# ADB_DEVICE_SERIAL=emulator-5554
# Android ADB Connectivity (Optional - defaults to 127.0.0.1:5037)
ADB_HOST=127.0.0.1
ADB_PORT=5037
# ADB_DEVICE_SERIAL=
# Artemis Execution Defaults (flash or pro)
ARTEMIS_DEFAULT_PROFILE=flash
.gcloudignore
-17
View File
@@ -1,17 +0,0 @@
.gcloudignore
.git
.venv/
__pycache__/
*.pyc
traces/
latte-traces/
debug/replay/
debug_pub/replay/
*.mp4
*.jpg
*.png
pytest_temp/
/scratch/
android_world*
/dist/
!frontend/dist/**
.gitignore
+73 -83
View File
@@ -1,109 +1,99 @@
# ==============================================================================
# ☕ Latte Framework - Master Git Ignore Configuration
# ==============================================================================
# --- 1. Environment Variables & Virtual Environments (Recursive) ---
.env
.env.*
!.env.example
uv.lock
**/.venv/
**/venv/
**/ENV/
**/env/
**/.pyenv/
.python-version
# --- 2. Node & Web Frontend Dependencies / Artifacts (Recursive) ---
**/node_modules/
**/node_modules
**/.angular/
**/.angular/cache/
npm-debug.log*
yarn-debug.log*
yarn-error.log*
# --- 3. Python Caches, Bytecode & Build Artifacts (Recursive) ---
**/__pycache__/
**/*.py[cod]
**/*$py.class
**/*.so
**/*.pyd
**/*.c
**/build/
**/dist/
**/*.egg-info/
**/.eggs/
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# --- 4. Testing, Coverage & Linters (Recursive) ---
**/.pytest_cache/
**/.ruff_cache/
**/.mypy_cache/
**/.pytype/
# Virtual Environments
.venv/
env/
venv/
ENV/
env.bak/
venv.bak/
# Testing & Coverage
.pytest_cache/
.coverage
.coverage.*
.cache
coverage.xml
nosetests.xml
htmlcov/
pytest_temp/
tests_tmp/
coverage.xml
*.cover
*.py,cover
.hypothesis/
.tox/
.nox/
tests/**/outputs/
**/outputs/
# --- 5. Runtime Logs, Spans, Replay & Traces (Recursive) ---
traces/
traces_test/
latte-traces/
data_engine.db*
**/replay/outputs/
**/replay/data/
**/*.log
**/*.log.*
**/*.out
**/*.tmp
.latte_ipc_port
.latte_paused
.latte/
.artemis_ipc_port
.artemis_paused
.artemis/
*.bak
live_screenshot*
recording.mp4
recording.mkv
screenshot.jpg
screen.png
# Linters & Formatters
.ruff_cache/
.mypy_cache/
.pyre/
.pytype/
.dmypy.json
dmypy.json
# --- 6. Workspace Scratch, Dumps & Temporary Experimentation ---
scratch/
artifacts/
ignores/
*.docx
*gdoc.html
*.bug.log
# --- 7. AI Agent Workspace & IDE Metadata ---
.agents/
_agents/
jetski/
.jetski*
.benchmarks/
.idea/
# IDEs & Editors
.vscode/
.idea/
*.swp
*.swo
*~
.project
.pydevproject
.settings/
# Node & Frontend (Showcase UI)
apps/showcase_ui/node_modules/
apps/showcase_ui/.angular/
apps/showcase_ui/.angular/cache/
.npm/
npm-debug.log*
yarn-debug.log*
yarn-error.log*
# Artemis Runtime & Telemetry
traces/
scratch/
.artemis_paused
*.log
data_engine.db
data_engine.db-journal
.ipc_port
.ls_address
# Environment & Secrets
.env
.env.*
!.env.example
*.pem
*.key
# OS Artifacts
.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
ehthumbs.db
Thumbs.db
CONTRIBUTING.md
+10 -95
View File
@@ -1,103 +1,18 @@
# How to Contribute
# Contributing to ARTEMIS
We'd love to accept your patches and contributions to ARTEMIS! There are just a few small guidelines you need to follow.
Thank you for your interest in contributing to **ARTEMIS**! We welcome and appreciate all forms of contributions—whether it's reporting bugs, proposing new features, improving documentation, or submitting code.
---
## Contributor License Agreement (CLA)
## 🤝 Ways to Contribute
Contributions to this project must be accompanied by a Contributor License Agreement (CLA). You (or your employer) retain the copyright to your contribution; this simply gives us permission to use and redistribute your contributions as part of the project.
- Head over to <https://cla.developers.google.com/> to check your current agreements on file or to sign a new one.
- If you are an individual contributing original source code, you must sign the **Individual CLA**.
- If you are contributing on behalf of a company or organization, a **Corporate CLA** must be executed by your employer.
- You generally only need to submit a CLA once, so if you have already submitted one for another Google project, you do not need to sign it again.
- **🐛 Report Bugs & Suggest Features**: If you find an issue or have an idea to improve ARTEMIS, please open an [Issue](https://github.com/google/artemis/issues).
- **💡 Submit Pull Requests**:
1. Fork the repository and create your feature/fix branch (`git checkout -b feat/my-feature`).
2. Make your changes and verify with `make test` and `make lint`.
3. Open a Pull Request with a brief summary of your work.
- **💬 Community & Discussion**: Join our [Discord Community](https://discord.gg/wF2FN4WHGY) to chat with maintainers, ask questions, or share feedback.
---
## Community Guidelines
This project adheres to [Google's Open Source Community Guidelines](https://opensource.google/conduct/). By participating, all contributors and maintainers are expected to uphold these principles and maintain a welcoming, inclusive, and professional environment.
---
## Code Reviews
All submissions, including submissions by project maintainers, require review. We use standard GitHub pull requests for this purpose:
1. Consult [GitHub Help](https://docs.github.com/articles/about-pull-requests/) for information on using pull requests.
2. Maintainers will review your changes, offer feedback, and approve when ready.
3. Once approved and all CI checks pass, a project maintainer will merge your PR.
---
## Development & Contribution Workflow
### 1. Setting Up Your Development Environment
- **Prerequisites**:
- Python **3.12+**
- Android SDK Platform Tools (`adb` available in `PATH`)
- [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip`
- **Clone & Install**:
```bash
git clone https://github.com/<your-username>/artemis.git
cd artemis
# Install all dependencies and setup pre-commit hooks
make setup
```
*(Alternatively: `uv sync --dev` and `uv run pre-commit install`)*
### 2. Making Changes
- Create a descriptive topic branch off `main`:
```bash
git checkout -b feat/my-new-feature
# or
git checkout -b fix/issue-description
```
- Follow the project's coding and style conventions:
- **Absolute Imports**: Always use absolute imports (`from artemis.sdk import Agent`). Relative imports are disallowed.
- **Typing**: Use comprehensive type annotations on all public functions, classes, and methods.
- **Testing**: Add or update test cases under `tests/` for any modified or new behavior.
### 3. Local Verification
Before creating a pull request, ensure all linters, type checks, and tests pass:
```bash
# Format code (Ruff)
make format
# Lint code (Ruff)
make lint
# Run static type checker (Pyright)
make typecheck
# Run test suite (Pytest)
make test
```
### 4. Submitting a Pull Request
1. Push your branch to your fork:
```bash
git push origin feat/my-new-feature
```
2. Open a Pull Request against the `main` branch of the ARTEMIS repository.
3. Complete the PR template, describing:
- The motivation behind the change.
- Any related issue numbers (e.g., `Fixes #42`).
- How the change was tested (device, emulator, unit tests).
4. Ensure the CLA check automatically passes on your PR.
---
## Reporting Bugs and Feature Requests
- **Bug Reports**: Open an issue describing the bug, including steps to reproduce, device/emulator specifications, Android version, logs, and screenshots if applicable.
- **Feature Requests**: Open an issue detailing the use case, proposed API or workflow, and alternatives considered.
Every contribution—big or small—helps make ARTEMIS better. Thank you for building with us!
Dockerfile
+18 -41
View File
@@ -1,3 +1,17 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
FROM python:3.12-slim
WORKDIR /app
@@ -10,52 +24,15 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
adb \
&& rm -rf /var/lib/apt/lists/*
# Copy application source code and apps
# Copy application source code and configuration
COPY artemis/ /app/artemis/
COPY apps/ /app/apps/
COPY mcp_server/ /app/mcp_server/
COPY config/ /app/config/
COPY pyproject.toml setup.py README.md LICENSE /app/
# Install server and core dependencies
RUN pip install --no-cache-dir \
uvicorn \
fastapi \
pydantic \
pydantic-settings \
python-dotenv \
sseclient-py \
websockets \
adbutils \
jinja2 \
colorama \
psutil \
httpx \
retry \
google-auth \
langchain \
langchain-core \
langchain-community \
langchain-google-genai \
langchain-google-vertexai \
langchain-mcp-adapters \
google-genai \
langgraph \
typer \
pillow \
packaging \
uiautomator2 \
opencv-python-headless \
uuid-utils \
posthog \
mcp \
inquirer \
rich \
requests \
imageio-ffmpeg \
langchain-openai
# Install artemis package locally (no-deps to prevent pulling unnecessary heavy optional libraries)
RUN pip install --no-cache-dir --no-deps -e /app
# Install dependencies and artemis package
RUN pip install --no-cache-dir -e /app
ENV PYTHONUNBUFFERED=1
ENV PYTHONPATH=/app
LICENSE
+3 -3
View File
@@ -1,7 +1,7 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
https://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
@@ -187,13 +187,13 @@
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Copyright 2026 Google LLC.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
Makefile
+37 -8
View File
@@ -1,4 +1,18 @@
.PHONY: help test install setup clean precommit-install precommit lint format typecheck
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
.PHONY: help test install install-deps setup start ui doctor clean precommit-install precommit lint format typecheck
help: ## Show this help message
@echo 'Usage: make [target]'
@@ -6,14 +20,27 @@ help: ## Show this help message
@echo 'Available targets:'
@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z_-]+:.*?## / {printf " %-20s %s\n", $$1, $$2}' $(MAKEFILE_LIST)
start: ## One-click start Artemis Showcase UI and auto-open browser
@bash start.sh
ui: ## Launch the unified Showcase UI & Admin Console in browser
@uv run artemis ui --open
doctor: ## Run system, device, and toolchain diagnostics
@uv run artemis doctor
test: ## Run all tests
@echo "🧪 Running all tests..."
@uv run pytest -v
install: ## Install dependencies
@echo "📦 Installing dependencies..."
install: ## Install python dependencies via uv
@echo "📦 Installing python dependencies..."
@uv sync --dev
install-deps: ## One-click install all system dependencies (ADB, FFmpeg, scrcpy, Python, uv)
@echo "⚡ Running one-click dependency installer..."
@bash scripts/install_deps.sh
setup: ## Setup project (install dependencies + pre-commit hooks)
@echo "🚀 Setting up project..."
@$(MAKE) install
@@ -47,10 +74,12 @@ precommit: ## Run pre-commit hooks manually on all files
@echo "🔍 Running pre-commit checks..."
@uv run pre-commit run --all-files
clean: ## Clean up generated files and caches
@echo "🧹 Cleaning up..."
@rm -rf .pytest_cache
@rm -rf .ruff_cache
@rm -rf **/__pycache__
clean: ## Clean up generated files, caches, and traces
@echo "🧹 Cleaning up caches and temporary files..."
@rm -rf .pytest_cache .ruff_cache .artemis_paused traces scratch .venv apps/showcase_ui/.angular build dist
@find . -type d -name "outputs" -exec rm -rf {} + 2>/dev/null || true
@find . -type f -name ".artemis_paused" -delete 2>/dev/null || true
@find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
@find . -type d -name "*.egg-info" -exec rm -rf {} + 2>/dev/null || true
@find . -type f -name "*.py[cod]" -delete 2>/dev/null || true
@echo "✅ Cleanup complete"
README.md
+196 -346
View File
@@ -1,398 +1,248 @@
# ARTEMIS ☕📱
<p align="center">
<img src="./docs/assets/artemis-banner.png" alt="ARTEMIS Banner" width="100%" />
</p>
**Autonomous Multimodal Android Agent & Mobile UI Automation Platform**
<p align="center">
<strong>ARTEMIS: Autonomous AI Assistant for Mobile Automation</strong>
</p>
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
[![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
[![Framework: LangGraph](https://img.shields.io/badge/Orchestration-LangGraph-orange.svg)](https://github.com/langchain-ai/langgraph)
[![Model: Google Gemini](https://img.shields.io/badge/Powered%20by-Google%20Gemini-4285F4.svg)](https://ai.google.dev/)
<p align="center">
<em>⚡ Drive real phones from Cursor & Claude Code • Cross-App Automation • Zero-Maintenance UI Testing • Flash Mode as Fast as 3-5s</em>
</p>
ARTEMIS is a next-generation, production-grade autonomous agent and testing framework engineered for Android devices and emulators. Combining state-of-the-art multimodal Large Vision-Language Models (VLMs) with hierarchical closed-loop planning, resilient self-healing, and deep Android subsystem integration (ADB, UIAutomator2, accessibility trees), ARTEMIS enables autonomous UI navigation, robust automated testing, and intelligent device diagnostics.
<p align="center">
<a href="./README.md"><b>English</b></a> •
<a href="./README_CN.md">中文文档</a> •
<a href="#quick-start">Quick Start</a> •
<a href="#mcp-setup">MCP for IDEs</a> •
<a href="#benchmarks">Benchmarks</a> •
<a href="https://discord.gg/wF2FN4WHGY">Discord Community</a>
</p>
---
<p align="center">
<a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/Python-3.12+-3776AB.svg?logo=python&logoColor=white" alt="Python 3.12+"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License: Apache-2.0"></a>
<a href="https://modelcontextprotocol.io/"><img src="https://img.shields.io/badge/MCP-Native%20Server-8A2BE2.svg" alt="MCP Native"></a>
<a href="https://ai.google.dev/"><img src="https://img.shields.io/badge/Multimodal-Gemini%20%7C%20Claude%20%7C%20GPT--4o%20%7C%20Qwen--VL-4285F4.svg" alt="Multi-Model"></a>
<a href="https://github.com/google-research/android_world"><img src="https://img.shields.io/badge/AndroidWorld-99%25%2B%20SOTA-success.svg" alt="AndroidWorld SOTA"></a>
</p>
## 📑 Table of Contents
<!-- Demo Showcase -->
<p align="center">
<img src="./docs/assets/demo.gif" alt="Artemis in Action" width="88%" />
<br>
<em><b>Live Demo</b>: Setup driving routes and calculate total durations in Google Maps, then open YouTube to play a Coldplay song.</em>
</p>
- [Key Features](#-key-features)
- [Architecture Overview](#-architecture-overview)
- [Prerequisites](#-prerequisites)
- [Installation & Setup](#-installation--setup)
- [Quick Start](#-quick-start)
- [Command Line Interface (CLI)](#command-line-interface-cli)
- [Python SDK](#python-sdk)
- [Execution Profiles: Flash vs. Pro](#-execution-profiles-flash-vs-pro)
- [Model Context Protocol (MCP) Integration](#-model-context-protocol-mcp-integration)
- [Configuration Guide](#-configuration-guide)
- [Using Non-Google LLMs](#using-non-google-llms)
- [Trace Inspection & Observability](#-trace-inspection--observability)
- [Development & Testing](#-development--testing)
- [License](#-license)
## ✨ Key Highlights
---
* ⚡ **Lightning Fast**: First-of-its-kind **Optimistic Asynchronous Pipeline** decouples the foreground execution stream from heavy LLM reasoning;
* 🔋 **Ultra-Long Autonomous Execution**: **Pro mode** runs stably for over 10 hours, while monitoring tasks support around-the-clock 24/7 execution;
* 🔌 **Native Model Context Protocol (MCP)**: Empowers **Cursor, Claude Code, OpenClaw, Windsurf** with direct mobile device manipulation from within your chat prompts;
* 💾 **Visual Console & Persistent Debugging**: Real-time Web UI screen projection, execution replay, and task management with natural language test generation and high-fidelity historical review;
* 🏆 **Industry-Leading SOTA**: Achieved **99%+ task completion** on Google Research's **AndroidWorld** benchmark (100+ complex multi-step tasks).
## ✨ Key Features
<a id="quick-start"></a>
## ⚡ Quick Start
- **Dual-Engine Execution**:
- **ARTEMIS Flash**: High-speed, reactive single-agent loop for deterministic, low-latency tasks (< 35 steps).
- **ARTEMIS Pro**: Multi-agent closed-loop graph orchestration (Planner $\rightarrow$ Operator $\rightarrow$ Failure Analyzer $\rightarrow$ Summarizer) powered by LangGraph.
- **Multimodal Perception**:
- Combined visual screenshot grounding, real-time OCR, accessibility hierarchy (XML) parsing, and dynamic video analysis.
- **Resilient Locating Strategy ("Dynamic-First, Coordinate-Fallback")**:
- Prioritizes semantic IDs, XPath, and accessibility text matching to withstand UI layout drift; seamlessly falls back to calibrated absolute/relative coordinates.
- **Autonomous Self-Healing**:
- Built-in failure analyzer and safety net detect blocked flows, system dialogs, app crashes, or unexpected states and auto-recover in real time.
- **App Locking & Session Boundaries**:
- Restricts execution context to target application packages with automatic foreground recovery.
- **Model Context Protocol (MCP) Native**:
- Exposes Android automation, state inspection, and trace querying as MCP tools for IDEs and AI orchestrators.
- **Structured Pydantic Outputs**:
- Automatically parses agent observations and findings into validated Pydantic schemas.
---
## 🏛 Architecture Overview
ARTEMIS operates on a closed-loop multi-agent architecture:
```
┌─────────────────────────────────┐
│ User Goal / Task │
└────────────────┬────────────────┘
│
▼
┌───────────────────────────────┐
│ Planner Agent │
│ (Hierarchical Task Breakdown) │
└───────────────┬───────────────┘
│
┌──────────────┴──────────────┐
▼ ▼
┌─────────────────────────────────┐ ┌───────────────────────────────┐
│ Operator Agent │ │ Diagnostic Agents │
│ (Screen VLM + XML Grounding) │ │ (Log Reader, Video Analyzer, │
└────────────────┬────────────────┘ │ Object Detector, Explorer) │
│ └───────────────────────────────┘
▼
┌─────────────────────────────────┐
│ Execution & Device Control │
│ (ADB / UIAutomator2 / Taps) │
└────────────────┬────────────────┘
│
[Action Success / State Change?]
│ │
(Yes) (No / Blocked)
│ │
▼ ▼
┌──────────────────────┐ ┌─────────────────────────────────────────┐
│ Next Step / Complete │ │ Failure Analyzer & Safety Net │
└──────────┬───────────┘ │ (Popup dismissal, Retry, Self-healing) │
│ └────────────────────┬────────────────────┘
▼ │
┌──────────────────────┐ │
│ Summarizer / Output │◄──────────────────────┘
│ (Reports, Traces) │
└──────────────────────┘
```
---
## 🔧 Prerequisites
1. **Python**: Version `3.12` or higher.
2. **Android SDK / ADB**:
- `adb` must be installed and accessible in your system `PATH`.
- Verify with `adb devices`.
3. **Android Device or Emulator**:
- A physically connected Android device with **USB Debugging** enabled, or
- An active Android Emulator (AVD) running on your local machine.
4. **Screen Recording & Video Tools** *(optional for `--with-video-recording-tools`)*:
- **`scrcpy`**: Required for live Android device screen video recording (`apt install scrcpy` / `brew install scrcpy`).
- **`ffmpeg`**: Required for video transcoding, trimming, and audio extraction (`apt install ffmpeg` / `brew install ffmpeg`).
---
## 🚀 Installation & Setup
### 1. Clone the Repository
Ensure an Android device (with **USB Debugging** enabled) or emulator is connected.
```bash
git clone https://github.com/google/artemis.git
cd artemis
# 1. Clone repo & navigate to directory
git clone https://github.com/google/artemis.git && cd artemis
# 2. One-click launch (automatically installs ADB, scrcpy, FFmpeg, uv runtime, and opens console)
# 🍎 macOS & 🐧 Linux
./start.sh
# 🪟 Windows (CMD / PowerShell)
start.bat
```
### 2. Install Dependencies
> 💡 **Tip**: Opens `http://localhost:8000` in your default browser with a device connection wizard, live screen mirroring, prompt sandbox, and execution replays. You can also run directly from CLI: `artemis run "Open Settings, find Battery and tell me current level" --profile flash`.
Using [`uv`](https://docs.astral.sh/uv/) (recommended):
<a id="mcp-setup"></a>
<a id="mcp"></a>
<details>
<summary><b>🔌 MCP Setup for Cursor / Claude Code / Windsurf (Click to expand)</b></summary>
<br>
ARTEMIS includes a native **Model Context Protocol (MCP)** server. Connect your real phone directly into AI IDEs:
### 1. Generate MCP Config
Run the built-in generator to produce ready-to-use JSON configuration:
```bash
# Install dependencies into a virtual environment
uv sync --dev
artemis mcp --generate-config cursor
# Or generate configs for all IDEs:
artemis mcp --generate-config all
```
Or using standard `pip`:
### 2. Copy Config to Your IDE
```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
* **Cursor** (`.cursor/mcp.json` or Settings ➔ MCP Servers):
```json
{
"mcpServers": {
"artemis": {
"command": "python",
"args": ["-m", "mcp_server"],
"cwd": "/path/to/artemis",
"env": {
"PYTHONUNBUFFERED": "1"
}
}
}
}
```
### 3. Quick Setup (10 Seconds)
Run the interactive setup wizard to configure your API key and connect your device:
```bash
artemis init
* **Claude Desktop** (`claude_desktop_config.json`):
```json
{
"mcpServers": {
"artemis": {
"command": "python",
"args": ["-m", "mcp_server"],
"cwd": "/path/to/artemis"
}
}
}
```
Verify your environment and connected devices anytime with the doctor diagnostic tool:
### 3. Prompt Your Phone in the IDE Chat
In Cursor or Claude Code, simply prompt:
> 💬 *"Build the latest changes into an APK, install it on the connected device, open the login screen with a test account, verify if there are any unexpected popups after login, and return screenshots of the final page."*
```bash
artemis doctor
```
</details>
*(Alternatively, copy `.env.example` to `.env` and set `GEMINI_API_KEY` for LLM reasoning and `OCR_API_KEY` for Vision perception.)*
<a id="python-sdk"></a>
<details>
<summary><b>🐍 Python SDK Integration (Click to expand)</b></summary>
---
<br>
## 💡 Quick Start
### Command Line Interface (CLI)
ARTEMIS installs a unified command-line tool `artemis`:
```bash
# 1. Environment & Device Diagnostics
artemis doctor
# 2. Interactive Setup Wizard
artemis init
# 3. Basic task execution (ARTEMIS Pro by default)
artemis run "Open YouTube and search for Lo-Fi Hip Hop"
# 4. Fast reactive profile (ARTEMIS Flash)
artemis run "Open Settings and verify Airplane Mode is turned off" --profile flash
# 5. Execute with trace recording and structured output
artemis run "Open Maps, search for coffee shops, and extract the top 3 results" \
--test-name "coffee_search_test" \
--traces-path "./traces" \
--output-description '{"places": "list of shop names", "ratings": "list of rating scores"}'
# 6. Enable dynamic video recording analysis
artemis run "Play the latest video in the feed and verify playback" --with-video-recording-tools
```
### Python SDK
Use the fluent SDK to integrate ARTEMIS into your test suites or automation workflows:
Embed the mobile automation engine into your Python workflows in just a few lines:
```python
import asyncio
from pydantic import BaseModel, Field
from artemis.sdk import Agent
class VideoSearchResult(BaseModel):
query: str = Field(..., description="The search term used")
top_video_title: str = Field(..., description="Title of the first video result")
channel_name: str = Field(..., description="Channel name of the first video")
from artemis.interfaces.sdk import ArtemisClient
async def main():
# 1. Initialize the agent
agent = Agent()
await agent.init()
# Initialize client (choose "flash" for speed or "pro" for deep reasoning)
client = ArtemisClient(default_profile="flash")
try:
# 2. Build task specification
task = (
agent.new_task("Search for 'Pixel 9 Pro' and get the first video details")
.with_name("youtube_search_verification")
.with_locked_app_package("com.google.android.youtube")
.with_output_format(VideoSearchResult)
.with_max_steps(50)
.build()
)
# Run autonomous task
result = await client.run("Open calculator and compute (128 * 45) + 330")
# 3. Execute task
result: VideoSearchResult = await agent.run_task(request=task)
if result:
print("Search Results:")
print(f"- Query: {result.query}")
print(f"- First Video: {result.top_video_title}")
print(f"- Channel: {result.channel_name}")
else:
print("Task did not complete successfully.")
finally:
# 4. Clean up agent session & device connections
await agent.clean()
print(f"Status: {result.status}")
print(f"Turns taken: {result.turns}")
print(f"Trace ID: {result.trace_id}")
if __name__ == "__main__":
asyncio.run(main())
```
---
</details>
## 🕹️ Usage Modes
<p align="center">
<img src="./docs/assets/artemis-ui-showcase-en.png" alt="Artemis Web Console" width="90%" />
<br />
<sub>💡 <b>Console Overview</b>: <b>① View Switcher</b> (Home / Workspace) · <b>② Model & Replay</b> (Flash/Pro status & video replay) · <b>③ Live Agent Stream</b> (Action perception, target coordinates & structured results) · <b>④ Prompt Dock</b> (Natural language dispatch) · <b>⑤ Task Queue & Dashboard</b> (Lifecycle & history)</sub>
</p>
* 🖥️ **Web Visual Console (`artemis ui`)**: Real-time screen projection and interactive panel, supporting natural language task dispatch, live reasoning telemetry, action trajectories, and execution replay;
* 🔌 **Native MCP Protocol (IDE Integration)**: Operates as a standard MCP server seamlessly integrating with **Cursor, Claude Code, OpenClaw**, etc., directly invoking mobile device manipulation inside the IDE;
* 💻 **Developer CLI (`artemis run`)**: Direct terminal execution for automated tasks, test cases, or AndroidWorld benchmarks with high-fidelity structured terminal output;
* 🐍 **Python SDK**: Integrates as a standard Python library into existing automated testing frameworks or pipelines with strongly typed Pydantic structured outputs.
## 📊 Head-to-Head Comparison
| Scenario / Dimension | Traditional Automation | Generic Screen AI Agents | **ARTEMIS ☕** |
| :--- | :--- | :--- | :--- |
| **Natural Language Execution** | ❌ Rigid hard-coded scripts only | ⚠️ Sluggish (waits 20-30s per tap) | ⚡ **High Throughput**: As fast as 3–5s per step in Flash mode with fluid responses |
| **UI Redesign Resilience** | ❌ Layout changes break XPath selectors | ⚠️ Coordinate guessing leads to drift | 🎯 **Semantic Grounding**: Blends text, icons & UI tree structure to adapt to UI changes |
| **Popup Interception** | ❌ Blocked and crashed immediately | ❌ Loops endlessly on unfamiliar screens | 🩹 **Pre-Execution Self-Healing**: Verifies target state before clicking; dismisses popups |
| **Dynamic & Video Tasks** | ❌ Blind static sleep only | ❌ Blind to dynamic video playback | ⏱️ **Real-Time Multimedia**: Live video stream analysis, countdown timers & speculative chaining |
| **IDE & Tooling Integration** | ❌ Standalone test runner | ❌ Web-only with high integration friction | 🔌 **Native MCP Server**: Directly callable inside Cursor, Claude Desktop, etc. |
<a id="benchmarks"></a>
## 🏆 Benchmarks: AndroidWorld (SOTA 99%+)
Evaluated on [AndroidWorld](https://github.com/google-research/android_world) — Google Research's gold-standard benchmark spanning 20+ real apps and 100+ complex multi-step tasks: **Artemis demonstrated exceptional robustness across the entire benchmark suite, achieving a 99%+ completion rate.**
<p align="center">
<img src="./docs/assets/androidworld_benchmark_comparison.png" alt="AndroidWorld Benchmark Comparison" width="85%" />
</p>
## 🚀 How ARTEMIS is Architected
* ⚡ **Optimistic Asynchronous Pipeline**: The front-facing loop responds in milliseconds, while memory pruning and assertion verification run concurrently in the background without blocking execution;
* 🛡️ **Safety Net Pre-Execution Gate**: Dual-layer pre-check validates target availability milliseconds before action dispatch, instantly intercepting unexpected popups to eliminate blind clicks;
* ⏱️ **Time-Sensitive Speculative Chaining**: Overcomes LLM inference latency for transient UI elements (e.g. video fullscreen) by predicting target coordinates and executing rapid chained taps.
<details>
<summary><b>🔍 Click to expand: Architecture Deep Dive & Pipeline Diagram</b></summary>
<br>
### 1. ⚡ Optimistic Asynchronous Pipeline
* **Status Quo & Pain Points**: Conventional mobile agents rely on a **fully synchronous blocking model** — every single action must wait sequentially for the LLM to prune historical context, check milestone assertions, and audit long-term plans. This inflates per-step latency to 20–40 seconds, creating a sluggish user experience.
* **Artemis's Architectural Solution**: Inspired by **Optimistic Concurrency Control (OCC) and Snapshot Isolation** in database systems, Artemis completely decouples the main execution loop from heavy auxiliary computation:
* **High-Throughput Main Loop**: The front-facing execution path is strictly narrowed to a high-speed "Perception → Decision → Safety Gate → Execution" pipeline;
* **Background Concurrent Tasks**: Context token pruning, milestone checkers (`Checker`), and planner validations (`Planner`) run dynamically in parallel without halting device interaction;
* **Snapshot Isolation & Rollback**: The agent optimistically charges forward. If background verification detects a deviation, Artemis instantly **rolls back via state snapshots (Rollback)** and injects self-healing feedback.
<p align="center">
<img src="./docs/assets/artemis-architecture-pipeline.png" alt="Artemis Optimistic Async Pipeline Architecture" width="100%" />
</p>
### 2. 🛡️ Pre-Execution Safety Net & Time-Sensitive Speculative Chaining
* **Status Quo & The Transient UI Dilemma**:
* Mobile applications feature numerous **time-sensitive transient UI controls** (e.g. video fullscreen: tapping the screen wakes up the floating overlay, followed immediately by tapping the fullscreen icon).
* Traditional agents tap the screen to reveal controls, take a new screenshot, and wait 3–15 seconds for LLM reasoning. By the time the click is dispatched, **the player controls have already auto-faded away** — causing the click to strike the underlying video, triggering an endless loop of accidental pausing and waking.
* **Artemis's Architectural Solution**:
* **Speculative Chained Actions**: Upon recognizing time-sensitive dependencies, the agent dispatches compound chained actions (Wakeup → Millisecond Chained Tap) to hit the target within its transient visibility window;
* **Two Pillars Ensuring Reliable Chaining**:
1. **Historical UI Prior Prediction**: Predicts the target control's wake-up coordinates based on prior interaction history and app layout heuristics;
2. **Safety Net Pre-Execution Gate**: Milliseconds before the chained action lands, the Safety Net instantly verifies that the target control was successfully revealed at the expected coordinates. If the wakeup failed or an unexpected popup intercepted it, execution is **immediately blocked to prevent blind clicks**.
</details>
## ⚡ Execution Profiles: Flash vs. Pro
| Feature | ARTEMIS Flash (`--profile flash`) | ARTEMIS Pro (`--profile default`) |
| Feature / Dimension | ⚡ **ARTEMIS Flash** (`--profile flash`) | 🧠 **ARTEMIS Pro** (`--profile pro`) |
| :--- | :--- | :--- |
| **Model Loop** | Single reactive agent (Observe-Think-Act) | Multi-node LangGraph orchestration |
| **Best For** | Short, deterministic tasks (< 35 steps) | Deep, complex, multi-stage workflows |
| **Step Latency** | ~3–5 seconds / step | ~15–30 seconds / turn (with planning & validation) |
| **Planning** | Direct goal execution | Hierarchical multi-step plan decomposition |
| **Diagnostics** | Basic error reporting | Deep diagnostics, log analysis & self-healing |
| **Design Purpose** | **Lightweight & Fast**: Direct deterministic UI actions | **Deep Reasoning**: Multi-step planning & complex self-healing |
| **Step Latency** | **3–5 seconds** / step | **15–30 seconds** / turn (includes planning & verification) |
| **Task Duration** | Minute-level short tasks (typically ≤35 steps) | Runs stably for **10+ hours**; monitoring tasks support **24/7 execution** |
| **Best Suited For** | Well-defined standard UI tasks | Complex cross-app workflows, failure self-healing, continuous monitoring |
| **Self-Healing** | Local step retries | **Safety Net Gate** + dialog suppression + crash recovery + snapshot rollback |
| **Media Analysis** | **Basic visual perception** + High-Speed OCR | Full `scrcpy`/`ffmpeg` video stream analysis + Logcat logs |
---
## 🗺️ Roadmap
## 🔌 Model Context Protocol (MCP) Integration
- [x] **Optimistic Asynchronous Pipeline**: Ultra-lean main loop + background context compression & milestone checks.
- [x] **Pre-Execution Safety Net**: Millisecond pre-check gate & speculative chained actions.
- [x] **Time-Sensitive Media Tasks**: Fullscreen video and audio stream analysis with `scrcpy` & `ffmpeg`.
- [x] **Native MCP Server**: Seamless integration with tools like Cursor and Claude Desktop.
- [x] **Web Visual Console**: Live screen projection, interactive playground, and trajectory review.
- [x] **AndroidWorld SOTA**: Achieved 99%+ task completion rate.
- [ ] **Cross-Platform Extensions**: Exploring iOS and desktop Web perception and execution.
- [ ] **On-Device Lightweight VLMs**: Zero-cloud local execution with lightweight edge vision models.
- [ ] **Real-time Duplex Voice Mode**: Natural voice input with real-time interruption (barge-in) control.
ARTEMIS includes built-in MCP servers and tools to allow AI IDEs (such as Claude Desktop, Cursor, and Gemini Jetski) to control Android devices directly:
## 🤝 Community & Contributing
| Tool Name | Description |
| :--- | :--- |
| `mobile_run_task` | Launches an autonomous background automation task on the connected device. |
| `mobile_manage_task` | Inspects status, terminates tasks, or injects real-time steering instructions. |
| `mobile_get_device_state` | Retrieves real-time screenshots or simplified accessibility UI trees. |
| `mobile_inspect_trace` | Inspects step-by-step screenshots, visual action overlays, and execution logs. |
---
## ⚙️ Configuration Guide
All model configurations, thinking budgets, agent capabilities, and tool filtering are consolidated into a single self-explanatory configuration file: [`config/artemis.jsonc`](config/artemis.jsonc).
```jsonc
{
// 1. Global Default Model (Applied across all sub-agents unless overridden)
"default": {
"provider": "google",
"model": "gemini-3.6-flash",
"thinking_level": "medium",
"fallback": {
"provider": "google",
"model": "gemini-3.5-flash"
}
},
// 2. Built-in Presets
"presets": {
"gemini-flagship": { "provider": "google", "model": "gemini-3.6-flash" },
"openai-gpt4o": { "provider": "openai", "model": "gpt-4o" },
"cost-saving": { "provider": "google", "model": "gemini-3.5-flash-lite" },
"local-ollama": { "provider": "openai", "model": "llama3.2-vision" }
},
// 3. Agent & Sub-Agent Node Customizations (Optional overrides)
"nodes": {
// 🧠 Planner: High-level goal decomposition and strategic step planning
"planner": { "thinking_level": "medium" },
// 🎯 Operator: Screen perception, element interaction, and direct execution
"operator": { "thinking_level": "medium", "include_thoughts": true },
// 🛡️ Validator & Failure Analyzer: Self-healing and auto-recovery
"validator_failure_analyzer": { "thinking_level": "medium" }
},
// 4. Agent Capabilities & Tool Filtering
"agent": {
"explorer_versions": { "operator": "flash", "validator": "flash" },
"denylisted_tools": {
"explorer": ["ask_image_processor", "get_ocr_list", "inspect_region"]
}
}
}
```
> [!TIP]
> **Zero-Friction Global Change**: To switch the entire platform from Gemini to GPT-4o or Claude, simply edit the `"default"` block once in `config/artemis.jsonc`. All 15 sub-agents automatically inherit the new model!
### Using Non-Google LLMs
ARTEMIS supports multi-provider LLM backends via standard LangChain integrations (Google Gemini, OpenAI, OpenRouter, xAI / Grok, or Google Vertex AI).
#### Supported Providers & API Key Setup
| Provider | Provider Key | Environment Variable | Base URL Override | Example Models |
| :--- | :--- | :--- | :--- | :--- |
| **Google Gemini** | `google` *(default)* | `GEMINI_API_KEY` | N/A | `gemini-3.6-flash`, `gemini-3.5-flash` |
| **Google Vision OCR** | `ocr` | `OCR_API_KEY` | N/A | Google Cloud Vision Text Detection |
| **Google Vertex AI** | `vertexai` | Google ADC (`gcloud auth application-default login`) | N/A | `gemini-3.6-flash` |
| **OpenAI** | `openai` | `OPENAI_API_KEY` | `OPENAI_BASE_URL` | `gpt-4o`, `gpt-4o-mini`, `o3` |
| **OpenRouter** | `openrouter` | `OPEN_ROUTER_API_KEY` | N/A | `anthropic/claude-3.5-sonnet`, `meta-llama/llama-3.3-70b-instruct` |
| **xAI (Grok)** | `xai` | `XAI_API_KEY` | N/A | `grok-2-vision-1212`, `grok-beta` |
> [!NOTE]
> ARTEMIS uses multimodal visual grounding (screenshots) for UI navigation. When assigning non-Google models to vision-dependent agents (such as `operator`), ensure the selected model supports image/vision inputs.
#### Switching Provider in `config/artemis.jsonc`
To use OpenAI or Claude globally, simply change the `default` section in [`config/artemis.jsonc`](config/artemis.jsonc):
```jsonc
{
"default": {
"provider": "openai",
"model": "gpt-4o",
"fallback": {
"provider": "openai",
"model": "gpt-4o-mini"
}
}
}
```
#### Custom / Local OpenAI-Compatible Endpoints (Ollama, vLLM, LocalAI)
To connect ARTEMIS to local or self-hosted LLM endpoints (e.g., Ollama or vLLM), set `OPENAI_BASE_URL` in `.env` and specify `openai` as the provider in your configuration:
```bash
# Route OpenAI provider calls to local Ollama / vLLM endpoint
OPENAI_API_KEY=ollama # dummy key if required by client
OPENAI_BASE_URL=http://localhost:11434/v1
```
---
## 📊 Trace Inspection & Observability
Every task execution can record rich execution artifacts stored in `./traces/<trace_id>/`:
- **`notes/`**: High-level planner notes, step summaries, and generated reports (`output.md`).
- **`screenshots/`**:
- `before_screenshot.jpg`: Initial screen observed by the agent.
- `action_overlay_screenshot.jpg`: Screen with visual tap markers (red circles) and swipe vectors (red arrows).
- **`logs/`**: Raw framework, operator, and failure analyzer diagnostic logs.
---
## 🛠 Development & Testing
ARTEMIS provides standard `make` targets for development:
```bash
# Setup environment and pre-commit hooks
make setup
# Run test suite
make test
# Run code formatters (ruff)
make format
# Run linter checks
make lint
# Run static type checker (pyright)
make typecheck
```
---
Contributions are warmly welcomed!
* ⭐ **Star the repo** to follow updates and releases
* 💬 Join the [Discord Community](https://discord.gg/wF2FN4WHGY) for technical discussions
* 🐛 Open an [Issue](https://github.com/google/artemis/issues) or submit a [Pull Request](https://github.com/google/artemis/pulls)
## 📄 License
README_CN.md
+251
View File
@@ -0,0 +1,251 @@
<p align="center">
<img src="./docs/assets/artemis-banner.png" alt="ARTEMIS Banner" width="100%" />
</p>
<p align="center">
<strong>ARTEMIS: 全自动操控手机的 AI 个人助理</strong>
</p>
<p align="center">
<em>⚡ 让 Cursor / Claude Code 接管真机 • 跨 App 复杂自动化 • 零脚本维护 UI 测试 • Flash 模式单步最快 3-5 秒</em>
</p>
<p align="center">
<a href="./README.md">English</a> •
<a href="./README_CN.md"><b>中文文档</b></a> •
<a href="#quick-start">快速上手</a> •
<a href="#mcp-setup">MCP 接入 IDE</a> •
<a href="#benchmarks">基准评测</a> •
<a href="https://discord.gg/wF2FN4WHGY">Discord 社区</a>
</p>
<p align="center">
<a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/Python-3.12+-3776AB.svg?logo=python&logoColor=white" alt="Python 3.12+"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License: Apache-2.0"></a>
<a href="https://modelcontextprotocol.io/"><img src="https://img.shields.io/badge/MCP-Native%20Server-8A2BE2.svg" alt="MCP Native"></a>
<a href="https://ai.google.dev/"><img src="https://img.shields.io/badge/Multimodal-Gemini%20%7C%20Claude%20%7C%20GPT--4o%20%7C%20Qwen--VL-4285F4.svg" alt="Multi-Model"></a>
<a href="https://github.com/google-research/android_world"><img src="https://img.shields.io/badge/AndroidWorld-99%25%2B%20SOTA-success.svg" alt="AndroidWorld SOTA"></a>
</p>
<!-- 演示效果图 -->
<p align="center">
<img src="./docs/assets/demo.gif" alt="Artemis 演示效果" width="88%" />
<br>
<em><b>实机演示</b>：在 Google Maps 中设置驾车路线并计算总耗时，随后打开 YouTube 播放 Coldplay 的歌曲。</em>
</p>
## ✨ 核心亮点
* ⚡ **极致响应速度**：首创**全链路乐观异步流水线 (Optimistic Async Pipeline)**，主干执行流与后台重度推理完全解耦；
* 🔋 **超长程稳定运行**：**Pro 模式**可运行超 10 小时，监控任务甚至可以全天候运行；
* 🔌 **原生 MCP (Model Context Protocol) 赋能**：让 **Cursor、Claude Code、OpenClaw、Windsurf** 直接长出“操作真机的双手”，在聊天窗口一句话完成真机测试与 Bug 复现；
* 💾 **可视化控制台与持久化调试**：提供 Web UI 实时投屏、操作回放与任务持久化管理，支持通过自然语言生成测试用例，并对历史执行进行高保真可视化审查；
* 🏆 **业界顶尖 SOTA**：在 Google Research **AndroidWorld** 基准评测（100+ 复杂长程任务）中取得 **99%+ 任务完成率**。
<a id="quick-start"></a>
<a id="快速上手"></a>
## ⚡ 快速上手
确保电脑已连接 Android 实体机（已开启 **USB 调试**）或 Android 模拟器。
```bash
# 1. 克隆代码仓库并进入目录
git clone https://github.com/google/artemis.git && cd artemis
# 2. 一键启动（自动检测并安装 ADB、scrcpy、FFmpeg、uv 运行时等全部依赖，并拉起控制台）
# 🍎 macOS & 🐧 Linux
./start.sh
# 🪟 Windows (CMD / PowerShell)
start.bat
```
> 💡 **提示**：启动后将自动在默认浏览器中打开 Web 控制台（`http://localhost:8000`），提供设备连接向导、实时投屏、任务演练与状态回放面板。你也可以通过命令行直接运行：`artemis run "打开系统设置，找到电池选项并告诉我当前电量" --profile flash`。
<a id="mcp-setup"></a>
<a id="mcp"></a>
<details>
<summary><b>🔌 接入 Cursor / Claude Code / Windsurf (MCP)（点击展开）</b></summary>
<br>
ARTEMIS 内置原生 **Model Context Protocol (MCP)** 服务。只需将以下配置加入你的 IDE 配置文件中，即可在编写代码时直接驱动真机：
### 1. 生成或查看配置
运行内置命令一键获取当前环境的完整配置 JSON：
```bash
artemis mcp --generate-config cursor
# 或生成所有 IDE 配置：
artemis mcp --generate-config all
```
### 2. 复制配置到 IDE
* **Cursor** (`.cursor/mcp.json` 或 设置 ➔ MCP Servers)：
```json
{
"mcpServers": {
"artemis": {
"command": "python",
"args": ["-m", "mcp_server"],
"cwd": "/path/to/artemis",
"env": {
"PYTHONUNBUFFERED": "1"
}
}
}
}
```
* **Claude Desktop** (`claude_desktop_config.json`)：
```json
{
"mcpServers": {
"artemis": {
"command": "python",
"args": ["-m", "mcp_server"],
"cwd": "/path/to/artemis"
}
}
}
```
### 3. 在 IDE 中体验真机协同
在 Cursor / Claude Code 对话框中直接输入：
> 💬 *"请帮我把刚刚修改的代码编译成 APK 并安装到手机上，打开登录页面输入测试账号，验证登录后是否有异常弹窗，并把最终页面截图回传。"*
</details>
<a id="python-sdk"></a>
<details>
<summary><b>🐍 Python SDK 极简集成（点击展开）</b></summary>
<br>
只需几行代码，将移动端自动化智能体无缝嵌入你的 Python 脚本与 CI/CD 流水线中：
```python
import asyncio
from artemis.interfaces.sdk import ArtemisClient
async def main():
# 初始化客户端（支持 "flash" 极速模式 或 "pro" 深度推理模式）
client = ArtemisClient(default_profile="flash")
# 执行自然语言任务
result = await client.run("打开计算器，计算 (128 * 45) + 330 的结果")
print(f"执行状态: {result.status}")
print(f"消耗步数: {result.turns}")
print(f"Trace ID: {result.trace_id}")
if __name__ == "__main__":
asyncio.run(main())
```
</details>
## 🕹️ 使用方式
<p align="center">
<img src="./docs/assets/artemis-ui-showcase.png" alt="Artemis 可视化控制台" width="90%" />
<br />
<sub>💡 <b>控制台功能布局</b>：<b>① 顶栏视图切换</b>（主页与工作区） · <b>② 运行模式与录屏回放</b>（Flash/Pro 状态与视频回放） · <b>③ 实时感知推理流</b>（动作分解、点击坐标与结构化总结） · <b>④ 自然语言下发胶囊</b>（自然语言驱动真机） · <b>⑤ 任务队列看板</b>（状态流转与历史回溯）</sub>
</p>
* 🖥️ **Web 可视化控制台 (`artemis ui`)**：集成设备实时投屏与交互面板，支持通过自然语言下发任务，实时观测推理过程、操作轨迹与状态回放；
* 🔌 **MCP 协议集成 (IDE 联动)**：作为标准 MCP 服务器无缝接入 **Cursor、Claude Code、OpenClaw** 等环境，直接在 IDE 中调用移动端真机操作能力；
* 💻 **命令行工具 (`artemis run`)**：支持通过 CLI 直接执行自动化任务、测试用例或 AndroidWorld 基准评测，提供高保真结构化终端输出；
* 🐍 **Python SDK**：作为标准 Python 库集成至现有自动化测试框架或流水线，提供基于 Pydantic 的强类型结构化输出。
## 📊 方案横向对比
| 场景 / 维度 | 传统自动化框架 | 常见屏幕 Agent | **ARTEMIS ☕** |
| :--- | :--- | :--- | :--- |
| **自然语言指令支持** | ❌ 仅支持硬编码脚本，无语义理解 | ⚠️ 响应较慢，单步交互通常需 20-30 秒 | ⚡ **高效执行**：Flash 模式下单步最快约 3-5 秒，交互流畅 |
| **UI 改版适应能力** | ❌ 元素变更易导致脚本执行中断 | ⚠️ 依赖绝对坐标，存在位置偏移风险 | 🎯 **多模态语义定位**：结合文本、图标与层次结构，自适应 UI 变化 |
| **异常弹窗处理** | ❌ 弹窗遮挡易导致定位失败 | ❌ 面对未知弹窗易陷入无效循环 | 🩹 **执行前校验与自愈**：操作前确认目标状态，自动识别并处理干扰弹窗 |
| **动态与视频场景** | ❌ 仅支持静态等待 (sleep)，无动态感知 | ❌ 无法识别视频与动态流媒体内容 | ⏱️ **流媒体分析**：支持视频流分析、倒计时识别与预测性连点 |
| **开发环境集成** | ❌ 独立运行，难以直接与 AI IDE 联动 | ❌ 通常仅提供独立网页，集成成本高 | 🔌 **原生 MCP 支持**：可在 Cursor、Claude Desktop 等环境中直接调用 |
<a id="benchmarks"></a>
<a id="基准评测"></a>
## 🏆 基准评测：AndroidWorld (SOTA 99%+)
在 Google Research 发布的业界基准评测 [AndroidWorld](https://github.com/google-research/android_world)（涵盖 20+ 款常用应用与 100+ 项复杂多步长程任务）中：**Artemis 在全套长程任务评测中展现了高鲁棒性，取得了超过 99% 的任务完成率。**
<p align="center">
<img src="./docs/assets/androidworld_benchmark_comparison.png" alt="AndroidWorld 评测基准对比" width="85%" />
</p>
## 🚀 ARTEMIS 是如何建构的
* ⚡ **全链路乐观异步 (Optimistic Async)**：主交互循环毫秒级响应下发，记忆修剪与阶段断言全部在后台并发静默运行，告别死等；
* 🛡️ **Safety Net 动作守门**：动作下发前毫秒级双重校验目标，突发弹窗即刻拦截自愈，绝不盲点空气；
* ⏱️ **时效敏感预测性连点 (Speculative Chaining)**：针对视频全屏等瞬态交互，结合历史先验预测坐标并极速连点，彻底解决大模型延迟导致控件消失的问题。
<details>
<summary><b>🔍 点击展开：核心架构设计细节与流水线流程图</b></summary>
<br>
### 1. ⚡ 全链路乐观异步流水线 (Optimistic Asynchronous Pipeline)
* **现有方案与问题**：传统移动端智能体普遍采用**全同步阻塞模型** —— 每一步交互都必须同步串行等待大模型完成历史上下文修剪、阶段性断言检查与长程规划审计。这导致主交互循环极度臃肿，单步延迟动辄 20–40 秒，交互体验迟缓；
* **Artemis 的架构解法**：借鉴数据库的**乐观并发控制 (OCC) 与快照隔离**思想，将主干执行流与重度计算彻底解耦：
* **高吞吐前台主循环**：前台严格收敛为「感知 → 决策 → 守门 → 执行」极简流水线，单步响应压缩至秒级；
* **后台并发异步处理**：上下文修剪、里程碑断言校验（Checker）、全局规划审计（Planner）全部在后台并发静默运行，零阻塞前台；
* **快照隔离与无损回滚**：前台以前瞻视角乐观执行；一旦后台异步断言检测到状态偏离，系统基于快照机制**一键无损回滚 (Rollback)** 并注入纠偏反馈，兼得极低延迟与可靠自愈。
<p align="center">
<img src="./docs/assets/artemis-architecture-pipeline.png" alt="Artemis 乐观异步流水线架构" width="100%" />
</p>
### 2. 🛡️ 动作守门机制 (Safety Net) 与时效敏感预测性连点 (Speculative Chaining)
* **现有方案与时效困境**：
* 移动端存在大量**时效敏感型交互**（如视频全屏播放：必须先轻点屏幕唤醒浮层控件，再点击全屏按钮）。
* 传统智能体在点击屏幕唤醒控件后，重新截屏并等待大模型推理决策耗时数秒；而此时**播放器控件早已自动淡出隐藏**，导致后续点击直接打在视频画面上，陷入“唤醒 → 控件消失 → 误触暂停 → 再次唤醒”的死循环。
* **Artemis 的架构解法**：
* **预测性链式连点 (Speculative Chaining)**：智能体在识别此类时序依赖后，直接下发复合连击指令（唤醒 → 毫秒级连击全屏），在控件有效窗口期内瞬时完成交互；
* **保障连点绝对可靠的两大支柱**：
1. **历史先验位置预测**：基于历史交互轨迹与应用 UI 先验知识，精准预判控件唤醒后的目标坐标；
2. **Safety Net 毫秒级拦截守门**：在连击动作落下的前一刻，瞬间核验目标控件是否已被成功唤醒并位于预判位置。若唤醒失败或遭遇意外弹窗遮挡，**立刻前置拦截熔断，杜绝盲点误触**。
</details>
## ⚡ 运行模式对比：Flash vs. Pro
| 特性对比 | ⚡ **ARTEMIS Flash** (`--profile flash`) | 🧠 **ARTEMIS Pro** (`--profile pro`) |
| :--- | :--- | :--- |
| **设计定位** | **轻量极速**：面向常规确定性操作 | **深度规划**：面向长流程与复杂自愈 |
| **单步延迟** | **3–5 秒** / 步 | **15–30 秒** / 轮 (包含深度推理与全局校验) |
| **任务时长** | 分钟级短程任务（通常 ≤35 步） | 可稳定运行**超 10 小时**，监控任务可**全天候运行** |
| **适用场景** | 目标明确的常规 UI 操作 | 复杂的跨 App 长流程、故障自愈、持续状态观察 |
| **自愈机制** | 基础局部重试 | **Safety Net 校验** + 弹窗处理 + 崩溃恢复 + 快照回滚 |
| **多媒体分析** | 基础视觉感知 + 高速 OCR | 完整 `scrcpy`/`ffmpeg` 视频流分析 + Logcat 日志采集 |
## 🗺️ 路线图
- [x] **全链路乐观异步流水线**：主流程轻量化，后台并发执行记忆压缩与断言校验。
- [x] **Safety Net 执行前校验**：动作下发前双层核验与预测性链式执行。
- [x] **时间敏感多媒体处理**：集成 `scrcpy` 与 `ffmpeg` 分析全屏视频与音频流。
- [x] **原生 MCP 服务支持**：支持与 Cursor、Claude Desktop 等工具无缝集成。
- [x] **Web 可视化控制台**：提供设备投屏、交互演练与轨迹回溯。
- [x] **AndroidWorld SOTA**：达到 99%+ 的任务完成率。
- [ ] **跨平台扩展**：探索 iOS 与 Web 端的定位与操作能力。
- [ ] **端侧轻量化模型**：支持在设备本地离线运行的轻量级 Edge VLM 模型。
- [ ] **实时语音双工交互**：支持自然语音输入与实时打断控制。
## 🤝 社区与贡献
欢迎通过以下方式参与项目建设：
* ⭐ **Star 本项目**以关注最新进展与更新
* 💬 加入 [Discord 社区](https://discord.gg/wF2FN4WHGY) 参与技术探讨与功能建议
* 🐛 提交 [Issue](https://github.com/google/artemis/issues) 反馈 Bug，欢迎发起 [Pull Request](https://github.com/google/artemis/pulls) 贡献代码
## 📄 开源许可证
本项目基于 [Apache License 2.0](LICENSE) 协议开源。
apps/README.md
-26
View File
@@ -1,26 +0,0 @@
# 📱 Artemis Applications Ecosystem (`apps/`)
This directory houses the two primary applications and user interfaces of the Artemis ecosystem.
---
## 📂 Subsystems Overview
```
apps/
├── 🛠️ admin_console/ # [Interface 1] Full-fidelity Trace Inspector & Task Control Console (FastAPI + Web)
└── ✨ showcase_ui/ # [Interface 2] Aesthetic User Presentation & Interaction Workspace (Angular 19)
```
---
## 🔍 Detailed Comparison of the Two UIs
| Dimension | 🛠️ Admin & Trace Console (`admin_console`) | ✨ Showcase & Workspace UI (`showcase_ui`) |
| :--- | :--- | :--- |
| **Primary Audience** | Developers, Test Engineers, System Admins | End Users, Customers, Product Demo Audiences |
| **UI Paradigm** | High-density, full-fidelity tree logs, step-by-step inspector | Dynamic Aurora Glow, Glassmorphism, dual-pane stream |
| **Core Capabilities** | • Full hierarchical Call Tree & LLM thinking inspection<br>• Pre/post action screenshots & visual overlay verification<br>• Video playback & step replay engine (`/api/replay`)<br>• Queue tasks & live state toggle | • Real-time Agent thought & action progress streaming<br>• Interactive natural language chat sidebar<br>• Aesthetic visual representation of mobile autonomy |
| **Tech Stack** | Python (FastAPI) + Vanilla Responsive Web | TypeScript, Angular 19, SCSS |
| **Start Command** | `uv run uvicorn apps.admin_console.server:app --port 8000 --reload` | `cd apps/showcase_ui && npm start` (Port 4200) |
apps/__init__.py
+1 -2
View File
@@ -12,5 +12,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Artemis / Latte Applications Package."""
"""Artemis Applications Package."""
apps/admin_console/README.md
-51
View File
@@ -1,51 +0,0 @@
# 🛠️ Artemis Admin & Trace Console
A comprehensive developer/administrator control plane and trace inspection console for the Latte/Artemis mobile automation system.
---
## 🎯 Key Features
1. **Full Trace Inspection (`/api/sessions`, `/api/steps`)**:
- Hierarchical Call Tree exploration (LLM thoughts, tool calls, agent motivations).
- Step-by-step pre/post screenshots and action overlays (tap circles, swipe paths).
- Streaming logs and video recording playback.
2. **Task Control Center (`/api/run`)**:
- Directly launch autonomous tasks with custom goals and model profiles (Flash / Pro).
- Pause, resume, and manage FIFO task execution queues.
3. **Step Replay & Debugging Engine (`/api/replay`)**:
- Replay individual steps with isolated inputs, state inspection, and live device execution.
---
## 🚀 How to Start
### Option 1: Using `uv` (Recommended)
From the repository root:
```bash
uv run uvicorn apps.admin_console.server:app --host 0.0.0.0 --port 8000 --reload
```
Or directly using Python module:
```bash
uv run python -m apps.admin_console.server
```
### Option 2: From the `apps/admin_console` directory
```bash
cd apps/admin_console
uv run uvicorn server:app --host 0.0.0.0 --port 8000 --reload
```
Once started, open your browser and navigate to:
👉 **`http://localhost:8000`**
---
## 🎥 Video Debugging Mode
By default, `VideoAnalyzer` cleans up local temporary video files after execution. To retain video recordings for full playback in the console:
```bash
KEEP_VIDEOS=1 artemis run "your task goal"
```
Or set `ARTEMIS_DEBUG=1` in your environment.
apps/admin_console/core/config.py
+7 -1
View File
@@ -63,6 +63,12 @@ except ImportError:
REPLAY_BASE_DIR = TRACES_PATH / "replay"
TEST_DATA_DIR = REPLAY_BASE_DIR / "data"
TEST_OUTPUTS_DIR = REPLAY_BASE_DIR / "outputs"
IMAGES_DIR = TRACES_PATH / "images"
PAUSE_FILE = WORKSPACE_ROOT / ".artemis_paused"
# Ensure all essential runtime directories exist automatically
for _d in (TRACES_PATH, REPLAY_BASE_DIR, TEST_DATA_DIR, TEST_OUTPUTS_DIR, IMAGES_DIR):
try:
_d.mkdir(parents=True, exist_ok=True)
except Exception:
pass
apps/admin_console/core/state.py
+81 -13
View File
@@ -36,25 +36,96 @@ class ServerState:
self.active_connections: dict[str, dict[str, Any]] = {}
self.active_session_id: str | None = None
self.was_stopped_manually: bool = False
self.cancelled_session_ids: set[str] = set()
# FIFO Task queue
self.task_queue: asyncio.Queue = asyncio.Queue()
self.queue_tasks: list[dict[str, Any]] = []
# Unified single source of truth for task queue
self.queue_items: list[dict[str, Any]] = []
self._wake_event: asyncio.Event | None = None
self.worker_task: asyncio.Task | None = None
@property
def wake_event(self) -> asyncio.Event:
try:
loop = asyncio.get_running_loop()
except RuntimeError:
loop = None
if self._wake_event is None or (
loop
and getattr(self._wake_event, "_loop", None) is not None
and self._wake_event._loop != loop
):
self._wake_event = asyncio.Event()
return self._wake_event
@property
def task_queue(self) -> list[dict[str, Any]]:
"""Backward compatibility alias for queue_items."""
return self.queue_items
@property
def queue_tasks(self) -> list[dict[str, Any]]:
"""Returns all currently pending tasks in the queue."""
return [t for t in self.queue_items if isinstance(t, dict) and t.get("status") == "pending"]
@queue_tasks.setter
def queue_tasks(self, val: list[dict[str, Any]]):
# Compatibility setter
self.queue_items = list(val)
@property
def queue_goals(self) -> list[str]:
"""Backward compatibility helper for queue goals."""
return [t.get("goal", "") for t in self.queue_tasks if isinstance(t, dict)]
return [t.get("goal", "") for t in self.queue_tasks]
@queue_goals.setter
def queue_goals(self, val: list[Any]):
# Allow setting if legacy code updates it
pass
@property
def is_running(self) -> bool:
return self.current_process is not None and self.current_process.returncode is None
has_proc = False
if self.current_process is not None:
if self.current_process.returncode is not None:
has_proc = False
self.current_process = None
else:
pid = getattr(self.current_process, "pid", None)
if pid:
try:
import os
os.kill(pid, 0)
has_proc = True
except (ProcessLookupError, PermissionError, OSError):
has_proc = False
self.current_process = None
else:
has_proc = False
self.current_process = None
has_running_item = any(
isinstance(t, dict) and t.get("status") == "running" for t in self.queue_items
)
has_live_connection = False
for sid, conn in list(self.active_connections.items()):
c_pid = conn.get("pid")
if c_pid:
try:
import os
os.kill(c_pid, 0)
has_live_connection = True
except (ProcessLookupError, PermissionError, OSError):
self.active_connections.pop(sid, None)
if not has_proc and not has_running_item and not has_live_connection:
self.active_session_id = None
self.current_process = None
return False
return True
@property
def is_paused(self) -> bool:
@@ -69,13 +140,10 @@ class ServerState:
self.ipc_subscribers.remove(callback)
def clear_queue(self):
while not self.task_queue.empty():
try:
self.task_queue.get_nowait()
self.task_queue.task_done()
except (asyncio.QueueEmpty, ValueError):
break
self.queue_tasks.clear()
"""Clears all pending items from the task queue."""
self.queue_items = [t for t in self.queue_items if t.get("status") == "running"]
if self._wake_event:
self._wake_event.set()
# Global shared instance
apps/admin_console/database/connection.py
+18 -3
View File
@@ -15,7 +15,6 @@
from contextlib import contextmanager
from pathlib import Path
import sqlite3
from fastapi import HTTPException
try:
from admin_console.core.config import DB_PATH
@@ -23,13 +22,29 @@ except ImportError:
from apps.admin_console.core.config import DB_PATH
_initialized_dbs = set()
def get_db(db_path: Path | None = None) -> sqlite3.Connection:
"""Returns a SQLite connection with timeout and Row row_factory."""
path = db_path or DB_PATH
if not path.exists():
raise HTTPException(status_code=500, detail=f"Database not found at {path}")
path_key = str(path)
if path_key not in _initialized_dbs:
path.parent.mkdir(parents=True, exist_ok=True)
try:
from artemis.data_engine.storage import StorageManager
StorageManager(db_path=path, base_trace_dir=path.parent)
_initialized_dbs.add(path_key)
except Exception:
pass
conn = sqlite3.connect(path, timeout=10.0)
conn.row_factory = sqlite3.Row
try:
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA busy_timeout=10000")
except Exception:
pass
return conn
apps/admin_console/database/repositories/session_repository.py
-7
View File
@@ -33,13 +33,6 @@ class SessionRepository:
cursor.execute("SELECT * FROM sessions ORDER BY start_time DESC")
return [dict(r) for r in cursor.fetchall()]
def get_session_by_id(self, session_id: str) -> dict[str, Any] | None:
with db_session(self.db_path) as conn:
cursor = conn.cursor()
cursor.execute("SELECT * FROM sessions WHERE session_id = ?", (str(session_id),))
row = cursor.fetchone()
return dict(row) if row else None
def get_video_recordings_map(self) -> dict[str, str]:
with db_session(self.db_path) as conn:
cursor = conn.cursor()
apps/admin_console/database/repositories/step_repository.py
+24 -5
View File
@@ -13,6 +13,7 @@
# limitations under the License.
import json
import re
from typing import Any
try:
@@ -183,9 +184,27 @@ class StepRepository:
)
args = payload_obj.get("args", payload_obj)
if isinstance(args, dict):
return {"args": self._clean_value(args)}
return {"args": self._clean_value(args)}
cleaned_args = (
self._clean_value(args) if isinstance(args, dict) else self._clean_value(args)
)
result_payload: dict[str, Any] = {"args": cleaned_args}
# Extract structured action execution results
res = payload_obj.get("result")
if isinstance(res, dict):
if res.get("post_image_name"):
result_payload["post_image_name"] = res["post_image_name"]
if res.get("pre_image_name"):
result_payload["pre_image_name"] = res["pre_image_name"]
if res.get("outcome"):
result_payload["outcome"] = res["outcome"]
elif res:
res_str = str(res)
match = re.search(r"([a-f0-9]{64})", res_str)
if match:
result_payload["post_image_name"] = match.group(1)
return result_payload
def get_session_steps(self, session_id: str, client: str | None = None) -> list[dict[str, Any]]:
with db_session(self.db_path) as conn:
@@ -221,7 +240,7 @@ class StepRepository:
if pre_img and step_dict.get("post_image_name") == pre_img:
step_dict["post_image_name"] = None
# Fetch generic tool traces for this step
# Fetch action and generic tool traces for this step
step_id = step_dict.get("step_id")
if step_id:
cursor.execute(
@@ -229,7 +248,7 @@ class StepRepository:
" t1.type, t1.name, t1.timestamp, t1.duration, t1.status,"
" t1.payload, t2.name as agent_name FROM traces t1 LEFT"
" JOIN traces t2 ON t1.parent_trace_id = t2.trace_id WHERE"
" t1.step_id = ? AND (t1.type = 'tool' OR (t1.type ="
" t1.step_id = ? AND (t1.type = 'tool' OR t1.type = 'action' OR (t1.type ="
" 'llm_call' AND t1.status = 'failed')) ORDER BY"
" t1.timestamp ASC",
(step_id,),
apps/admin_console/index.html
+122 -20
View File
@@ -20,6 +20,11 @@
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Artemis Debug UI</title>
<link rel="icon" type="image/png" sizes="192x192" href="/logo.png?v=2">
<link rel="icon" type="image/png" sizes="32x32" href="/logo.png?v=2">
<link rel="icon" type="image/svg+xml" href="/logo.svg?v=2">
<link rel="shortcut icon" href="/favicon.ico?v=2">
<link rel="apple-touch-icon" href="/logo.png?v=2">
<style>
pre.light-code {
background: transparent !important;
@@ -385,21 +390,33 @@
marker.style.top = `${pctY}%`;
marker.style.transform = 'translate(-50%, -50%)';
marker.style.zIndex = '10';
marker.style.pointerEvents = 'none';
const dotSize = sequenceIndex !== undefined ? 20 : 14;
const dot = document.createElement('div');
dot.style.width = '14px';
dot.style.height = '14px';
dot.style.width = `${dotSize}px`;
dot.style.height = `${dotSize}px`;
dot.style.borderRadius = '50%';
dot.style.backgroundColor = '#ef4444';
dot.style.border = '2px solid white';
dot.style.boxShadow = '0 0 6px rgba(0,0,0,0.4)';
if (sequenceIndex !== undefined) {
dot.style.display = 'flex';
dot.style.alignItems = 'center';
dot.style.justifyContent = 'center';
dot.style.fontSize = '11px';
dot.style.fontWeight = 'bold';
dot.style.color = 'white';
dot.innerText = String(sequenceIndex);
}
const pulseSize = dotSize * 2;
const pulse = document.createElement('div');
pulse.style.position = 'absolute';
pulse.style.top = '-9px';
pulse.style.left = '-9px';
pulse.style.width = '28px';
pulse.style.height = '28px';
pulse.style.top = `-${dotSize / 2 + 2}px`;
pulse.style.left = `-${dotSize / 2 + 2}px`;
pulse.style.width = `${pulseSize}px`;
pulse.style.height = `${pulseSize}px`;
pulse.style.borderRadius = '50%';
pulse.style.border = '2px solid #ef4444';
pulse.style.animation = 'pulse-animation 1.5s infinite ease-out';
@@ -409,14 +426,18 @@
marker.appendChild(dot);
const label = document.createElement('div');
if (sequenceIndex !== undefined) {
label.innerText = totalSequence && totalSequence > 1 ? `#${sequenceIndex} TAP` : actionType.toUpperCase();
} else {
label.innerText = actionType.toUpperCase();
}
label.style.position = 'absolute';
label.style.top = '16px';
label.style.top = `${dotSize + 2}px`;
label.style.left = '50%';
label.style.transform = 'translateX(-50%)';
label.style.backgroundColor = 'rgba(15, 23, 42, 0.8)';
label.style.backgroundColor = 'rgba(15, 23, 42, 0.9)';
label.style.color = 'white';
label.style.padding = '1px 5px';
label.style.padding = '1.5px 5px';
label.style.borderRadius = '3px';
label.style.fontSize = '8px';
label.style.fontWeight = 'bold';
@@ -427,6 +448,66 @@
overlay.appendChild(marker);
}
function createSequenceConnector(x1, y1, x2, y2, overlay, isNormalized) {
let pctX1 = x1 / 10;
let pctY1 = y1 / 10;
let pctX2 = x2 / 10;
let pctY2 = y2 / 10;
const container = overlay.parentElement;
const img = container ? container.querySelector('img') : null;
if (!isNormalized && img && img.naturalWidth && img.naturalHeight) {
if (x1 > 1000 || y1 > 1000 || x2 > 1000 || y2 > 1000 || (x1 > 100 && y1 > 100 && img.naturalWidth > 500)) {
pctX1 = (x1 / img.naturalWidth) * 100;
pctY1 = (y1 / img.naturalHeight) * 100;
pctX2 = (x2 / img.naturalWidth) * 100;
pctY2 = (y2 / img.naturalHeight) * 100;
}
}
let svg = overlay.querySelector('svg');
if (!svg) {
svg = document.createElementNS('http://www.w3.org/2000/svg', 'svg');
svg.setAttribute('width', '100%');
svg.setAttribute('height', '100%');
svg.style.position = 'absolute';
svg.style.top = '0';
svg.style.left = '0';
svg.style.pointerEvents = 'none';
const defs = document.createElementNS('http://www.w3.org/2000/svg', 'defs');
const marker = document.createElementNS('http://www.w3.org/2000/svg', 'marker');
marker.setAttribute('id', 'seq-arrow');
marker.setAttribute('viewBox', '0 0 10 10');
marker.setAttribute('refX', '5');
marker.setAttribute('refY', '5');
marker.setAttribute('markerWidth', '6');
marker.setAttribute('markerHeight', '6');
marker.setAttribute('orient', 'auto');
const path = document.createElementNS('http://www.w3.org/2000/svg', 'path');
path.setAttribute('d', 'M 0 0 L 10 5 L 0 10 z');
path.setAttribute('fill', '#ef4444');
marker.appendChild(path);
defs.appendChild(marker);
svg.appendChild(defs);
overlay.appendChild(svg);
}
const line = document.createElementNS('http://www.w3.org/2000/svg', 'line');
line.setAttribute('x1', `${pctX1}%`);
line.setAttribute('y1', `${pctY1}%`);
line.setAttribute('x2', `${pctX2}%`);
line.setAttribute('y2', `${pctY2}%`);
line.setAttribute('stroke', '#ef4444');
line.setAttribute('stroke-width', '2.5');
line.setAttribute('marker-end', 'url(#seq-arrow)');
line.setAttribute('stroke-dasharray', '4,4');
svg.appendChild(line);
}
function createLineMarker(x1, y1, x2, y2, overlay, isNormalized) {
let pctX1 = x1 / 10;
let pctY1 = y1 / 10;
@@ -518,9 +599,37 @@
return;
}
// 1. Multi-point click sequence
const isSequenceAction = actionName.includes('sequence') || Boolean(act.sequence || args.sequence || act.normalized_sequence || args.normalized_sequence);
let rawSeq = act.normalized_sequence || args.normalized_sequence || act.sequence || args.sequence || act.targets || args.targets;
if (typeof rawSeq === 'string') {
try { rawSeq = JSON.parse(rawSeq); } catch (e) {}
}
if (Array.isArray(rawSeq) && rawSeq.length > 0) {
const validPoints = [];
rawSeq.forEach(pt => {
if (typeof pt === 'string') {
try { pt = JSON.parse(pt); } catch (e) {}
}
if (Array.isArray(pt) && pt.length >= 2 && typeof pt[0] === 'number' && typeof pt[1] === 'number') {
validPoints.push([pt[0], pt[1]]);
}
});
if (validPoints.length > 0 && (isSequenceAction || validPoints.length > 1)) {
const isNormSeq = Boolean(act.normalized_sequence || args.normalized_sequence) || validPoints.every(p => p[0] <= 1000 && p[1] <= 1000);
for (let i = 0; i < validPoints.length - 1; i++) {
createSequenceConnector(validPoints[i][0], validPoints[i][1], validPoints[i+1][0], validPoints[i+1][1], overlayElement, isNormSeq);
}
validPoints.forEach((pt, idx) => {
createPointMarker(pt[0], pt[1], actionName, overlayElement, isNormSeq, idx + 1, validPoints.length);
});
return;
}
}
const isSwipeLike = actionName.includes('swipe') || actionName.includes('scroll') || actionName.includes('drag') || actionName.includes('slide');
// 1. Separate start and end coordinates
// 2. Separate start and end coordinates
const normStart = act.normalized_start_coordinates || args.normalized_start_coordinates;
const normEnd = act.normalized_end_coordinates || args.normalized_end_coordinates;
const startCoords = normStart || act.start_coordinates || act.start_point || act.start || act.from ||
@@ -538,11 +647,11 @@
return;
}
// 2. 4-coordinate array
// 3. 4-coordinate array
const normCoords = act.normalized_coordinates || args.normalized_coordinates;
const isNorm = Boolean(normCoords);
const rawCoords = normCoords || act.coordinates || act.coords || act.target ||
args.target || args.coordinates || args.coords || args.sequence ||
args.target || args.coordinates || args.coords ||
(isSwipeLike && Array.isArray(args.action) ? args.action : null) ||
(isSwipeLike && Array.isArray(act.action) ? act.action : null) ||
(isSwipeLike && Array.isArray(args.gesture) ? args.gesture : null) ||
@@ -557,17 +666,10 @@
createPointMarker(rawCoords[0], rawCoords[1], actionName, overlayElement, isNorm);
return;
}
} else if (Array.isArray(rawCoords[0]) && rawCoords[0].length === 2) {
rawCoords.forEach(pt => {
if (Array.isArray(pt) && pt.length === 2 && typeof pt[0] === 'number') {
createPointMarker(pt[0], pt[1], actionName, overlayElement, isNorm);
}
});
return;
}
}
// 3. Directional swipe / scroll ONLY for swipe-like actions
// 4. Directional swipe / scroll ONLY for swipe-like actions
if (isSwipeLike) {
const rawDir = (typeof args.action === 'string' ? args.action : null) ||
(typeof act.action === 'string' && act.action !== actionName ? act.action : null) ||
apps/admin_console/replay_manager.py
+1 -1
View File
@@ -1493,7 +1493,7 @@ class ReplayManager:
f" offset={start_offset:.3f}s, duration={duration:.3f}s"
)
try:
res = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
res = subprocess.run(cmd, capture_output=True, text=True)
if res.returncode != 0:
print(
f"Warning: ffmpeg failed with exit code {res.returncode}. stderr: {res.stderr}"
apps/admin_console/routers/media.py
+1 -2
View File
@@ -26,10 +26,9 @@ except ImportError:
router = APIRouter(tags=["media"])
@router.get("/", response_class=HTMLResponse)
@router.get("/admin", response_class=HTMLResponse)
@router.get("/debug", response_class=HTMLResponse)
async def get_index():
async def get_admin_index():
admin_index = Path(__file__).resolve().parent.parent / "index.html"
if admin_index.exists():
return admin_index.read_text(encoding="utf-8")
apps/admin_console/routers/system.py
+348
View File
@@ -0,0 +1,348 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""System Readiness & Diagnostics Router for Artemis Admin Console."""
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel, Field
from artemis.core.diagnostics import readiness_engine
from artemis.core.diagnostics.schema import SystemReadinessReport
router = APIRouter(prefix="/api/system", tags=["system"])
class SelectDeviceRequest(BaseModel):
"""Payload to select an active target Android device."""
serial: str = Field(description="Serial number or identifier of the Android device to select")
@router.get("/readiness", response_model=SystemReadinessReport)
async def get_system_readiness() -> SystemReadinessReport:
"""Execute all diagnostic probes and return a comprehensive system readiness report."""
return await readiness_engine.run_all()
@router.post("/devices/select")
async def select_active_device(request: SelectDeviceRequest):
"""Select the active Android device or emulator for subsequent automated tasks."""
serial = request.serial.strip()
if not serial:
raise HTTPException(status_code=400, detail="Device serial cannot be empty.")
readiness_engine.set_active_device_serial(serial)
# Return updated readiness
report = await readiness_engine.run_all()
return {
"status": "success",
"selected_serial": serial,
"report": report,
}
@router.post("/adb/restart")
async def restart_adb_server():
"""Restart local ADB server and return an updated readiness check."""
restart_result = await readiness_engine.restart_adb_server()
updated_report = await readiness_engine.run_all()
return {
"restart_result": restart_result,
"report": updated_report,
}
class ConnectAdbRequest(BaseModel):
"""Payload to connect to an Android device over Wi-Fi."""
host: str = Field(description="IP address of Android device")
port: int = Field(default=5555, description="Port number")
@router.post("/adb/connect")
async def connect_wireless_adb(request: ConnectAdbRequest):
"""Connect to a device over Wi-Fi and return updated readiness."""
connect_result = await readiness_engine.connect_wireless_adb(request.host, request.port)
updated_report = await readiness_engine.run_all()
return {
"connect_result": connect_result,
"report": updated_report,
}
class LaunchEmulatorRequest(BaseModel):
"""Payload to launch a local Android Virtual Device (AVD)."""
avd_name: str = Field(
description="Name of the installed AVD emulator to launch (e.g. Android_2)"
)
@router.post("/emulator/launch")
async def launch_emulator(request: LaunchEmulatorRequest):
"""Launch an Android emulator in the background and return initiation status."""
avd_name = request.avd_name.strip()
if not avd_name:
raise HTTPException(status_code=400, detail="AVD name cannot be empty.")
launch_res = await readiness_engine.launch_emulator(avd_name)
return launch_res
@router.get("/emulator/status")
async def get_emulator_status():
"""Query real-time progress and logs of background emulator launch."""
return readiness_engine.get_emulator_status()
@router.post("/emulator/stop")
async def stop_emulator():
"""Stop active emulator process."""
return await readiness_engine.stop_emulator()
@router.post("/emulator/dismiss")
async def dismiss_emulator():
"""Dismiss emulator launch tracking state."""
return readiness_engine.dismiss_emulator()
class UpdateCredentialsRequest(BaseModel):
"""Payload to update and configure LLM or Vision OCR API credentials."""
provider: str = Field(
default="google",
description="Provider identifier (e.g. google, gemini, openai, anthropic, openrouter, ocr)",
)
api_key: str = Field(description="The secret API key string to configure")
persist_to_env: bool = Field(
default=True, description="Whether to persist the key to .env file"
)
class ValidateCredentialsRequest(BaseModel):
"""Payload to test and verify LLM or Vision OCR API credentials without saving."""
provider: str = Field(
default="google",
description="Provider identifier (e.g. google, gemini, openai, anthropic, openrouter, ocr)",
)
api_key: str = Field(description="The secret API key string to test")
base_url: str | None = Field(default=None, description="Optional custom base URL")
@router.get("/credentials")
async def get_credentials():
"""Retrieve current configured API keys for management."""
from artemis.config import settings
gemini_k = settings.get_api_key("google")
openai_k = settings.get_api_key("openai")
anthropic_k = settings.get_api_key("anthropic")
openrouter_k = settings.get_api_key("openrouter")
ocr_k = settings.get_api_key("ocr")
return {
"google": gemini_k.get_secret_value() if gemini_k else None,
"gemini": gemini_k.get_secret_value() if gemini_k else None,
"openai": openai_k.get_secret_value() if openai_k else None,
"anthropic": anthropic_k.get_secret_value() if anthropic_k else None,
"openrouter": openrouter_k.get_secret_value() if openrouter_k else None,
"ocr": ocr_k.get_secret_value() if ocr_k else None,
}
@router.post("/credentials/test")
async def test_credentials(request: ValidateCredentialsRequest):
"""Test and verify whether an API key is valid and usable with the corresponding provider endpoint."""
from artemis.utils.credentials_validator import validate_api_key
provider = request.provider.strip().lower()
key = request.api_key.strip()
if not key:
raise HTTPException(status_code=400, detail="API key cannot be empty.")
is_valid, message = await validate_api_key(
provider=provider,
api_key=key,
base_url=request.base_url,
)
if not is_valid:
raise HTTPException(status_code=400, detail=message)
return {
"valid": True,
"provider": provider,
"message": message,
}
@router.post("/credentials")
async def update_credentials(request: UpdateCredentialsRequest):
"""Dynamically configure and persist LLM or Vision API key, returning updated readiness report."""
from artemis.utils.credentials_validator import validate_api_key
provider = request.provider.strip().lower()
key = request.api_key.strip()
# If a non-empty key is provided, verify it before saving
if key:
is_valid, validation_msg = await validate_api_key(provider=provider, api_key=key)
if not is_valid:
raise HTTPException(
status_code=400,
detail=f"API key verification failed: {validation_msg}",
)
try:
from artemis.config import settings
settings.set_api_key(provider, key, persist_to_env=request.persist_to_env)
# Re-run all diagnostic probes to build updated report
updated_report = await readiness_engine.run_all()
action_desc = (
"successfully verified, updated, and applied" if key else "successfully cleared"
)
return {
"status": "success",
"message": f"API key for {provider} {action_desc}.",
"provider": provider,
"report": updated_report,
}
except HTTPException:
raise
except Exception as exc:
raise HTTPException(status_code=500, detail=f"Failed to update credentials: {exc}")
@router.get("/model-config-env")
async def get_model_config_and_env():
"""Retrieve the current active artemis.jsonc configuration and .env status for custom setup."""
import os
from artemis.config.paths import ROOT_DIR, get_config_path
from artemis.config import settings
from artemis.utils.file import load_jsonc
from artemis.config.settings import is_placeholder_key
# 1. Config file resolution
config_path = None
config_content = ""
parsed_config = {}
try:
config_path_obj = get_config_path("artemis.jsonc")
config_path = str(config_path_obj)
config_content = config_path_obj.read_text(encoding="utf-8")
with open(config_path_obj, encoding="utf-8") as f:
parsed_config = load_jsonc(f)
except Exception as e:
config_content = f"// Error reading config: {e}"
# 2. Env file resolution
env_path = ROOT_DIR / ".env"
# 3. Relevant Env Variables status
def get_real_env_value(key_name: str, provider: str) -> str | None:
k = settings.get_api_key(provider)
if k and not is_placeholder_key(k):
return k.get_secret_value()
val = os.environ.get(key_name)
if val and val.strip() and not is_placeholder_key(val):
return val.strip()
return None
def mask_key(k: str | None) -> str | None:
if not k:
return None
if len(k) <= 8:
return "****"
return f"{k[:4]}...{k[-4:]}"
gemini_real = get_real_env_value("GEMINI_API_KEY", "google") or get_real_env_value(
"GOOGLE_API_KEY", "google"
)
openai_real = get_real_env_value("OPENAI_API_KEY", "openai")
anthropic_real = get_real_env_value("ANTHROPIC_API_KEY", "anthropic")
openrouter_real = get_real_env_value("OPEN_ROUTER_API_KEY", "openrouter")
xai_real = get_real_env_value("XAI_API_KEY", "xai")
base_url_val = settings.OPENAI_BASE_URL or os.environ.get("OPENAI_BASE_URL")
if base_url_val and is_placeholder_key(base_url_val):
base_url_val = None
ocr_real = get_real_env_value("OCR_API_KEY", "ocr") or get_real_env_value(
"VISION_API_KEY", "ocr"
)
env_vars = [
{
"name": "GEMINI_API_KEY",
"provider": "google",
"is_set": bool(gemini_real),
"preview": mask_key(gemini_real),
"description": "Google Gemini multimodal vision API key (free tier available)",
},
{
"name": "OPENAI_API_KEY",
"provider": "openai",
"is_set": bool(openai_real),
"preview": mask_key(openai_real),
"description": "OpenAI API key (GPT-4o, GPT-4o-mini)",
},
{
"name": "ANTHROPIC_API_KEY",
"provider": "anthropic",
"is_set": bool(anthropic_real),
"preview": mask_key(anthropic_real),
"description": "Anthropic Claude API key (Claude 3.5 Sonnet, Claude 3.7)",
},
{
"name": "OPEN_ROUTER_API_KEY",
"provider": "openrouter",
"is_set": bool(openrouter_real),
"preview": mask_key(openrouter_real),
"description": "OpenRouter unified API gateway key",
},
{
"name": "XAI_API_KEY",
"provider": "xai",
"is_set": bool(xai_real),
"preview": mask_key(xai_real),
"description": "xAI Grok vision API key",
},
{
"name": "OPENAI_BASE_URL",
"provider": "custom",
"is_set": bool(base_url_val),
"preview": base_url_val,
"description": "Custom API endpoint (for local Ollama, vLLM, DeepSeek, or proxies)",
},
{
"name": "VISION_API_KEY",
"provider": "ocr",
"is_set": bool(ocr_real),
"preview": mask_key(ocr_real),
"description": "Google Cloud Vision OCR key for screen text detection (optional)",
},
]
return {
"config_path": config_path or "config/artemis.jsonc",
"config_filename": "artemis.jsonc",
"config_content": config_content,
"default_model": parsed_config.get("default", {}),
"presets": parsed_config.get("presets", {}),
"env_path": str(env_path),
"env_filename": ".env",
"env_vars": env_vars,
}
apps/admin_console/routers/tasks.py
+62 -20
View File
@@ -23,6 +23,7 @@ try:
from admin_console.schemas.task_schema import RunRequest
from admin_console.services.ipc_service import ipc_service
from admin_console.services.model_service import model_service
from admin_console.services.task_preset_catalog import task_recommendation_engine
from admin_console.services.task_queue_service import task_queue_service
except ImportError:
from apps.admin_console.core.state import state
@@ -30,12 +31,37 @@ except ImportError:
from apps.admin_console.schemas.task_schema import RunRequest
from apps.admin_console.services.ipc_service import ipc_service
from apps.admin_console.services.model_service import model_service
from apps.admin_console.services.task_preset_catalog import task_recommendation_engine
from apps.admin_console.services.task_queue_service import task_queue_service
router = APIRouter(tags=["tasks"])
@router.get("/api/tasks/presets")
async def get_task_presets(
category: str = "recommended",
packages: str | None = None,
limit: int = 24,
):
"""Retrieve smart task recommendations optionally tailored to detected device packages."""
pkg_list = [p.strip() for p in packages.split(",") if p.strip()] if packages else None
return task_recommendation_engine.recommend_tasks(
installed_packages=pkg_list,
category=category,
limit=limit,
)
@router.get("/api/tasks/catalog")
async def get_task_catalog():
"""Retrieve full catalog of predefined tasks and app package registry."""
return {
"tasks": [t.model_dump() for t in task_recommendation_engine.get_all_tasks()],
"app_registry": task_recommendation_engine.get_app_registry(),
}
@router.post("/api/run")
async def run_task(request: RunRequest):
incoming_goals = []
@@ -51,7 +77,12 @@ async def run_task(request: RunRequest):
)
return await task_queue_service.enqueue_tasks(
incoming_goals, profile=request.profile or "flash"
incoming_goals,
profile=request.profile or "flash",
expected_output=request.expected_output,
enable_outputter=request.enable_outputter,
locked_app_package=request.locked_app_package,
app_path=request.app_path,
)
@@ -73,15 +104,33 @@ async def resume_task():
@router.get("/api/status")
async def get_status():
# Watchdog check to ensure background worker is alive
task_queue_service.ensure_worker_running()
latest_session = session_repo.get_latest_session()
latest_session_id = latest_session.get("session_id") if latest_session else None
bg_tasks = session_repo.get_background_tasks(latest_session_id) if latest_session_id else []
is_running = state.is_running
running_sid = state.active_session_id or (
session_repo.get_running_session_id() if is_running else None
running_task = next(
(t for t in state.queue_items if isinstance(t, dict) and t.get("status") == "running"), None
)
if not running_task and is_running:
running_task = next(
(t for t in state.queue_items if isinstance(t, dict) and t.get("status") == "pending"),
None,
)
running_sid = (
state.active_session_id
or (running_task.get("session_id") if running_task else None)
or (session_repo.get_running_session_id() if is_running else None)
)
running_goal = state.current_goal or (running_task.get("goal") if running_task else None)
active_profile = state.current_profile or (
running_task.get("profile") if running_task else None
)
active_profile = state.current_profile
if not active_profile and (running_sid or latest_session_id):
check_sid = running_sid or latest_session_id
sess_row = session_repo.get_session_by_id(check_sid)
@@ -95,30 +144,17 @@ async def get_status():
model_info = model_service.get_active_model_info(active_profile)
queue_data = state.queue_tasks
if state.current_process:
if state.current_process.returncode is None:
if is_running:
is_paused = state.is_paused
return {
"status": "paused" if is_paused else "running",
"goal": state.current_goal,
"pid": state.current_process.pid,
"goal": running_goal,
"pid": state.current_process.pid if state.current_process else None,
"session_id": running_sid,
"background_tasks": bg_tasks,
"queue": queue_data,
"model_info": model_info,
}
else:
completed_status = {
"status": "completed",
"goal": state.current_goal,
"returncode": state.current_process.returncode,
"session_id": latest_session_id,
"background_tasks": bg_tasks,
"queue": queue_data,
"model_info": model_info,
}
state.current_process = None
return completed_status
if latest_session_id and str(latest_session_id) in state.active_connections:
conn_info = state.active_connections[str(latest_session_id)]
@@ -151,6 +187,12 @@ async def stream_events(session_id: str, client: str | None = None):
def callback(event_type, data):
try:
# Global queue lifecycle events should always be delivered
if event_type not in (
"session_started",
"session_ended",
"background_tasks_updated",
):
# Filter events by session_id when subscribed to a specific session
if session_id and session_id not in ("all", "active"):
evt_session_id = None
apps/admin_console/schemas/task_schema.py
+4
View File
@@ -19,6 +19,10 @@ class RunRequest(BaseModel):
goal: str | None = None
goals: list[str] | None = None
profile: str | None = "flash"
expected_output: str | None = None
enable_outputter: bool | None = None
locked_app_package: str | None = None
app_path: str | None = None
class ReplayRequest(BaseModel):
apps/admin_console/server.py
+53 -2
View File
@@ -42,6 +42,7 @@ for _p in (str(_workspace_root), str(_apps_dir), str(_admin_console_dir), str(_c
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import FileResponse, HTMLResponse
try:
from admin_console.core.config import (
@@ -59,7 +60,7 @@ try:
from admin_console.database.repositories.session_repository import session_repo
from admin_console.database.repositories.step_repository import step_repo
from admin_console.database.repositories.trace_repository import trace_repo
from admin_console.routers import media, replay, sessions, steps, stream, tasks
from admin_console.routers import media, replay, sessions, steps, stream, system, tasks
from admin_console.routers.replay import replay_manager
from admin_console.services.ipc_service import ipc_service
from admin_console.services.media_service import media_service
@@ -71,7 +72,7 @@ except ImportError:
)
from apps.admin_console.core.state import state
from apps.admin_console.database.repositories.session_repository import session_repo
from apps.admin_console.routers import media, replay, sessions, steps, stream, tasks
from apps.admin_console.routers import media, replay, sessions, steps, stream, system, tasks
from apps.admin_console.routers.replay import replay_manager
from apps.admin_console.services.ipc_service import ipc_service
from apps.admin_console.services.model_service import model_service
@@ -116,6 +117,7 @@ app.include_router(sessions.router)
app.include_router(steps.router)
app.include_router(tasks.router)
app.include_router(replay.router)
app.include_router(system.router)
# Mount cloud gateway router for Frappe / Cloud integration if present
try:
@@ -131,6 +133,55 @@ except ImportError:
pass
# ------------------------------------------------------------------------------
# Showcase UI (Angular 19) Unified Single-Port SPA Static Hosting
# ------------------------------------------------------------------------------
_showcase_dist = _workspace_root / "apps" / "showcase_ui" / "dist" / "frontend" / "browser"
if not _showcase_dist.exists():
_showcase_dist = _workspace_root / "apps" / "showcase_ui" / "dist"
if _showcase_dist.is_dir() and (_showcase_dist / "index.html").exists():
@app.get("/{full_path:path}", include_in_schema=False)
async def serve_showcase_spa(full_path: str):
# Do not intercept API, media, or replay paths
if (
full_path.startswith("api/")
or full_path.startswith("images/")
or full_path.startswith("videos/")
or full_path.startswith("local_file")
or full_path == "docs"
or full_path == "openapi.json"
or full_path.startswith("redoc")
):
from fastapi import HTTPException
raise HTTPException(status_code=404, detail="Endpoint not found")
# Admin / Debug Console routes
if full_path in ("admin", "debug"):
admin_index = _admin_console_dir / "index.html"
if admin_index.exists():
return HTMLResponse(admin_index.read_text(encoding="utf-8"))
# Exact static file match (js, css, images, fonts, favicon, logo)
target_file = _showcase_dist / full_path
if full_path and target_file.is_file():
return FileResponse(target_file)
# Default fallback to Angular SPA index.html
index_file = _showcase_dist / "index.html"
return HTMLResponse(index_file.read_text(encoding="utf-8"))
else:
@app.get("/", response_class=HTMLResponse, include_in_schema=False)
async def fallback_root():
admin_index = _admin_console_dir / "index.html"
if admin_index.exists():
return HTMLResponse(admin_index.read_text(encoding="utf-8"))
return HTMLResponse("<h1>Artemis Console</h1>")
# ------------------------------------------------------------------------------
# Backward compatibility exports
# ------------------------------------------------------------------------------
apps/admin_console/services/device_stream_service.py
+7 -3
View File
@@ -24,6 +24,8 @@ import subprocess
import time
from collections.abc import AsyncGenerator
from artemis.toolchain import find_adb
logger = logging.getLogger("artemis.stream_service")
@@ -41,8 +43,9 @@ class DeviceStreamService:
async def get_device_serial(self) -> str | None:
"""Find the currently connected active ADB device serial."""
try:
adb_bin = find_adb()
proc = await asyncio.create_subprocess_exec(
"adb",
adb_bin,
"devices",
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
@@ -64,10 +67,11 @@ class DeviceStreamService:
try:
start_t = time.time()
serial = await self.get_device_serial()
adb_bin = find_adb()
cmd = (
["adb", "-s", serial, "exec-out", "screencap", "-p"]
[adb_bin, "-s", serial, "exec-out", "screencap", "-p"]
if serial
else ["adb", "exec-out", "screencap", "-p"]
else [adb_bin, "exec-out", "screencap", "-p"]
)
proc = await asyncio.create_subprocess_exec(
*cmd,
apps/admin_console/services/ipc_service.py
+12 -3
View File
@@ -78,7 +78,7 @@ class IPCService:
except Exception:
pass
if tr_type == "tool" and isinstance(payload, dict):
if tr_type in ("tool", "action") and isinstance(payload, dict):
args = payload.get("args", payload)
data["payload"] = {"args": cls._clean_value(args)}
elif tr_type == "llm_call" and isinstance(payload, dict):
@@ -144,8 +144,17 @@ class IPCService:
data = payload.get("data")
if event_type == "session_started" and data:
state.active_session_id = data.get("session_id")
current_session_id = data.get("session_id")
incoming_sid = data.get("session_id")
if incoming_sid and str(incoming_sid) in getattr(
state, "cancelled_session_ids", set()
):
print(
f"[IPC] Ignoring session_started for cancelled session: {incoming_sid}"
)
continue
state.active_session_id = incoming_sid
current_session_id = incoming_sid
pid = data.get("pid")
goal = data.get("initial_goal")
device_info = data.get("device_info")
apps/admin_console/services/model_service.py
+5 -4
View File
@@ -26,18 +26,18 @@ class ModelService:
if "pro" in p_lower:
return {
"name": "Pro",
"id": "gemini-3.6-pro",
"id": "gemini-3.7-pro",
"provider": "google",
}
elif "flash" in p_lower:
return {
"name": "Flash",
"id": "gemini-3.6-flash",
"id": "gemini-3.7-flash",
"provider": "google",
}
model_name = "Flash"
model_id = "gemini-3.6-flash"
model_id = "gemini-3.7-flash"
provider = "google"
try:
from artemis.config import parse_llm_config
@@ -118,7 +118,8 @@ class ModelService:
m_str = str(p_obj.get("model") or p_obj.get("model_name") or "").lower()
if (
"gemini-3.6-pro" in m_str
"gemini-3.7-pro" in m_str
or "gemini-3.6-pro" in m_str
or "gemini-1.5-pro" in m_str
or "gpt-4" in m_str
or "claude" in m_str
apps/admin_console/services/task_preset_catalog.py
+514
View File
@@ -0,0 +1,514 @@
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Preset Task Catalog & Dynamic Recommendation Engine for Artemis."""
from typing import Any, Literal
from pydantic import BaseModel, Field
class AppInfo(BaseModel):
"""Application metadata."""
name: str
icon: str
pkg: str
category: str = "general"
class TaskPreset(BaseModel):
"""Smart suggestion task definition."""
id: str
title: str
description: str
goal: str
profile: Literal["flash", "pro"]
category: Literal["flash", "pro", "cross_app", "monitor"]
tag: str
apps: list[AppInfo]
required_packages: list[str] = Field(default_factory=list)
match_mode: Literal["any", "all"] = "any"
priority: int = 50
# ============================================================================
# APP PACKAGE REGISTRY
# ============================================================================
APP_REGISTRY: dict[str, dict[str, str]] = {
# Google Suite & System
"com.google.android.apps.maps": {"name": "Maps", "icon": "explore", "category": "navigation"},
"com.google.android.gm": {"name": "Gmail", "icon": "mail", "category": "productivity"},
"com.android.chrome": {"name": "Chrome", "icon": "public", "category": "browser"},
"com.google.android.youtube": {
"name": "YouTube",
"icon": "smart_display",
"category": "entertainment",
},
"com.android.settings": {"name": "Settings", "icon": "settings", "category": "system"},
"com.google.android.deskclock": {"name": "Clock", "icon": "timer", "category": "utility"},
"com.android.deskclock": {"name": "Clock", "icon": "timer", "category": "utility"},
"com.google.android.calculator": {
"name": "Calculator",
"icon": "calculate",
"category": "utility",
},
"com.android.calculator2": {"name": "Calculator", "icon": "calculate", "category": "utility"},
"com.google.android.apps.photos": {
"name": "Photos",
"icon": "photo_library",
"category": "media",
},
"com.google.android.calendar": {
"name": "Calendar",
"icon": "calendar_month",
"category": "productivity",
},
"com.google.android.keep": {
"name": "Keep Notes",
"icon": "note_alt",
"category": "productivity",
},
"com.android.vending": {"name": "Play Store", "icon": "storefront", "category": "tools"},
"com.google.android.apps.messaging": {
"name": "Messages",
"icon": "chat",
"category": "communication",
},
# Popular Ecosystem Apps
"com.tencent.mm": {"name": "WeChat", "icon": "forum", "category": "social"},
"com.xingin.xhs": {"name": "Xiaohongshu", "icon": "auto_stories", "category": "social"},
"com.sankuai.meituan": {"name": "Meituan", "icon": "restaurant", "category": "lifestyle"},
"com.dianping.v1": {"name": "Dianping", "icon": "star", "category": "lifestyle"},
"tv.danmaku.bili": {"name": "Bilibili", "icon": "video_library", "category": "entertainment"},
"com.eg.android.AlipayGphone": {
"name": "Alipay",
"icon": "account_balance_wallet",
"category": "finance",
},
"com.netease.cloudmusic": {
"name": "NetEase Music",
"icon": "headphones",
"category": "entertainment",
},
"com.spotify.music": {"name": "Spotify", "icon": "music_note", "category": "entertainment"},
}
# ============================================================================
# CURATED TASK PRESET LIBRARY (1-2 curated tasks per common app)
# ============================================================================
PRESET_TASK_CATALOG: list[TaskPreset] = [
# 1. Google Maps
TaskPreset(
id="maps_coffee",
title="Find Specialty Coffee",
description="Search nearby top-rated cafes in Google Maps",
goal="Open Google Maps, search for top-rated specialty coffee shops nearby, and view the top result details.",
profile="flash",
category="flash",
tag="Maps",
apps=[
AppInfo(
name="Maps",
icon="explore",
pkg="com.google.android.apps.maps",
category="navigation",
)
],
required_packages=["com.google.android.apps.maps"],
priority=95,
),
TaskPreset(
id="pro_commute_share",
title="Commute ETA & Message Draft",
description="Check transit time on Maps and draft arrival ETA in Messages",
goal="Open Google Maps to check commute time to the International Airport, calculate arrival time, then open Messages and draft an ETA text message.",
profile="pro",
category="cross_app",
tag="Maps + Messages",
apps=[
AppInfo(
name="Maps",
icon="explore",
pkg="com.google.android.apps.maps",
category="navigation",
),
AppInfo(
name="Messages",
icon="chat",
pkg="com.google.android.apps.messaging",
category="communication",
),
],
required_packages=["com.google.android.apps.maps", "com.google.android.apps.messaging"],
match_mode="all",
priority=92,
),
# 2. Gmail
TaskPreset(
id="gmail_receipts",
title="Search Order Receipts",
description="Find recent flight or delivery confirmation emails in Gmail",
goal="Open Gmail and search for recent flight or package delivery confirmation emails.",
profile="flash",
category="flash",
tag="Gmail",
apps=[
AppInfo(name="Gmail", icon="mail", pkg="com.google.android.gm", category="productivity")
],
required_packages=["com.google.android.gm"],
priority=90,
),
TaskPreset(
id="pro_email_to_calendar",
title="Email Itinerary to Calendar",
description="Extract flight or event dates from Gmail and schedule in Calendar",
goal="Open Gmail to find the latest event invitation or itinerary, extract dates and location, then open Google Calendar and create a corresponding calendar event.",
profile="pro",
category="cross_app",
tag="Gmail + Calendar",
apps=[
AppInfo(
name="Gmail", icon="mail", pkg="com.google.android.gm", category="productivity"
),
AppInfo(
name="Calendar",
icon="calendar_month",
pkg="com.google.android.calendar",
category="productivity",
),
],
required_packages=["com.google.android.gm"],
priority=94,
),
# 3. Chrome
TaskPreset(
id="chrome_research",
title="Search AI News Breakthroughs",
description="Search latest multimodal AI developments in Chrome browser",
goal="Open Chrome browser and search for latest breakthroughs in multimodal mobile AI agents.",
profile="flash",
category="flash",
tag="Chrome",
apps=[AppInfo(name="Chrome", icon="public", pkg="com.android.chrome", category="browser")],
required_packages=["com.android.chrome"],
priority=88,
),
TaskPreset(
id="pro_research_keep",
title="Product Research & Notes Note",
description="Compare top 3 headphones on Chrome and record comparison in Keep",
goal="Open Chrome, research top 3 noise-cancelling headphones comparing price and battery life, then write a structured comparison summary note in Keep Notes.",
profile="pro",
category="pro",
tag="Chrome + Keep",
apps=[
AppInfo(name="Chrome", icon="public", pkg="com.android.chrome", category="browser"),
AppInfo(
name="Keep Notes",
icon="note_alt",
pkg="com.google.android.keep",
category="productivity",
),
],
required_packages=["com.android.chrome"],
priority=91,
),
# 4. YouTube
TaskPreset(
id="youtube_lofi",
title="Play Lo-Fi Music Radio",
description="Search and play a Lo-Fi hip hop live stream on YouTube",
goal='Open YouTube, search for "Lofi hip hop beats relaxing radio" and tap on the live stream.',
profile="flash",
category="flash",
tag="YouTube",
apps=[
AppInfo(
name="YouTube",
icon="smart_display",
pkg="com.google.android.youtube",
category="entertainment",
)
],
required_packages=["com.google.android.youtube"],
priority=85,
),
# 5. Settings
TaskPreset(
id="settings_display_wifi",
title="Dark Mode & Wi-Fi Check",
description="Toggle dark theme and verify network connection in Settings",
goal="Open Settings app, navigate to Display settings, ensure Dark theme is enabled, and check Wi-Fi connection status.",
profile="flash",
category="flash",
tag="Settings",
apps=[
AppInfo(name="Settings", icon="settings", pkg="com.android.settings", category="system")
],
required_packages=["com.android.settings"],
priority=87,
),
TaskPreset(
id="pro_settings_qa",
title="Subsystem Health & Crash Probe",
description="Traverse Settings submenus to verify screens and check for crash dialogs",
goal="Explore Settings submenus (Network, Connected devices, Apps, Battery, Storage), verify each screen loads properly without ANR or crash dialogs, and summarize results.",
profile="pro",
category="monitor",
tag="Settings QA",
apps=[
AppInfo(name="Settings", icon="settings", pkg="com.android.settings", category="system")
],
required_packages=["com.android.settings"],
priority=93,
),
# 6. Clock
TaskPreset(
id="clock_timer",
title="25-Min Pomodoro Timer",
description="Start a 25-minute focus countdown timer in Clock app",
goal="Open Clock app, switch to Timer tab, set 25 minutes and start the countdown timer.",
profile="flash",
category="flash",
tag="Clock",
apps=[
AppInfo(
name="Clock", icon="timer", pkg="com.google.android.deskclock", category="utility"
)
],
required_packages=["com.google.android.deskclock", "com.android.deskclock"],
priority=86,
),
# 7. Calculator
TaskPreset(
id="calc_gratuity",
title="Split Bill & Calculate Tip",
description="Calculate 18% gratuity on $186.40 for 3 people in Calculator",
goal="Open Calculator and calculate 18% tip on a bill of $186.40, then divide by 3 people.",
profile="flash",
category="flash",
tag="Calculator",
apps=[
AppInfo(
name="Calculator",
icon="calculate",
pkg="com.google.android.calculator",
category="utility",
)
],
required_packages=["com.google.android.calculator", "com.android.calculator2"],
priority=84,
),
# 8. Photos
TaskPreset(
id="photos_inspect",
title="Inspect Recent Screenshot",
description="Open Google Photos and review the latest screenshot taken",
goal="Open Google Photos and view the most recent screenshot in the screenshots album.",
profile="flash",
category="flash",
tag="Photos",
apps=[
AppInfo(
name="Photos",
icon="photo_library",
pkg="com.google.android.apps.photos",
category="media",
)
],
required_packages=["com.google.android.apps.photos"],
priority=82,
),
# 9. WeChat (微信)
TaskPreset(
id="wechat_browse",
title="Check WeChat Messages",
description="Open WeChat and view top recent chat conversations",
goal="Open WeChat (微信) and view the top recent chat messages.",
profile="flash",
category="flash",
tag="WeChat",
apps=[AppInfo(name="WeChat", icon="forum", pkg="com.tencent.mm", category="social")],
required_packages=["com.tencent.mm"],
priority=89,
),
TaskPreset(
id="pro_wechat_to_calendar",
title="WeChat Notice to Calendar",
description="Extract meeting notice from WeChat chat and add to Calendar",
goal="Open WeChat (微信), locate the latest meeting announcement or event message in the top chat, extract the time and topic, then open Calendar and schedule an event.",
profile="pro",
category="cross_app",
tag="WeChat + Calendar",
apps=[
AppInfo(name="WeChat", icon="forum", pkg="com.tencent.mm", category="social"),
AppInfo(
name="Calendar",
icon="calendar_month",
pkg="com.google.android.calendar",
category="productivity",
),
],
required_packages=["com.tencent.mm"],
priority=93,
),
# 10. Xiaohongshu (小红书)
TaskPreset(
id="xhs_coffee_guide",
title="RED Cafe Guide Search",
description="Search trending specialty cafe reviews on Xiaohongshu",
goal="Open Xiaohongshu (小红书), search for top-rated specialty coffee shops, and view the top post.",
profile="flash",
category="flash",
tag="Xiaohongshu",
apps=[
AppInfo(
name="Xiaohongshu", icon="auto_stories", pkg="com.xingin.xhs", category="social"
)
],
required_packages=["com.xingin.xhs"],
priority=87,
),
# 11. Meituan / Dianping (美团 / 大众点评)
TaskPreset(
id="meituan_ramen_search",
title="Meituan Food Search",
description="Search top-rated Ramen nearby on Meituan or Dianping",
goal="Open Meituan (美团) or Dianping (大众点评), search for top-rated Ramen nearby, and view top restaurant rating.",
profile="flash",
category="flash",
tag="Meituan",
apps=[
AppInfo(
name="Meituan", icon="restaurant", pkg="com.sankuai.meituan", category="lifestyle"
)
],
required_packages=["com.sankuai.meituan", "com.dianping.v1"],
priority=86,
),
# 12. Bilibili (哔哩哔哩)
TaskPreset(
id="bilibili_stream",
title="Bilibili Tech Video",
description="Search and play an AI Agent tutorial video on Bilibili",
goal='Open Bilibili (哔哩哔哩), search for "AI Agent 架构实战", and play the top matching video.',
profile="flash",
category="flash",
tag="Bilibili",
apps=[
AppInfo(
name="Bilibili",
icon="video_library",
pkg="tv.danmaku.bili",
category="entertainment",
)
],
required_packages=["tv.danmaku.bili"],
priority=85,
),
# 13. Play Store
TaskPreset(
id="pro_playstore_review",
title="Play Store App Review Study",
description="Compare top task management apps and ratings on Google Play",
goal="Open Google Play Store, search for top rated task management apps, compare ratings and latest user reviews of the top 2 candidates, and record recommendations.",
profile="pro",
category="pro",
tag="Play Store",
apps=[
AppInfo(
name="Play Store", icon="storefront", pkg="com.android.vending", category="tools"
)
],
required_packages=["com.android.vending"],
priority=88,
),
]
# ============================================================================
# RECOMMENDATION ENGINE
# ============================================================================
class TaskRecommendationEngine:
"""Intelligent recommendation engine matching device capabilities."""
def __init__(self, catalog: list[TaskPreset] | None = None):
self.catalog = catalog or PRESET_TASK_CATALOG
def get_all_tasks(self) -> list[dict[str, Any]]:
return [task.model_dump() for task in self.catalog]
def recommend_tasks(
self, installed_packages: list[str] | set[str], category: str = "all", limit: int = 12
) -> list[dict[str, Any]]:
pkgs_set = (
set(installed_packages) if isinstance(installed_packages, list) else installed_packages
)
scored_tasks: list[tuple[int, TaskPreset, bool]] = []
for task in self.catalog:
# Check package presence
is_matched = False
if pkgs_set:
if not task.required_packages:
is_matched = True
elif task.match_mode == "all":
is_matched = all(pkg in pkgs_set for pkg in task.required_packages)
else:
is_matched = any(pkg in pkgs_set for pkg in task.required_packages)
else:
is_matched = False
# Calculate score
score = task.priority
if pkgs_set:
if is_matched:
score += 100
if len(task.required_packages) > 1 and task.match_mode == "all":
score += 30
else:
score -= 40
# Filter by category
if category == "flash" and task.profile != "flash":
continue
elif category == "pro" and task.profile != "pro":
continue
elif category == "cross_app" and task.category != "cross_app":
continue
elif category == "monitor" and task.category != "monitor":
continue
scored_tasks.append((score, task, is_matched))
# Sort descending by score
scored_tasks.sort(key=lambda item: item[0], reverse=True)
results: list[dict[str, Any]] = []
for _, task, matched in scored_tasks[:limit]:
d = task.model_dump()
d["is_device_matched"] = matched
results.append(d)
return results
task_recommendation_engine = TaskRecommendationEngine()
apps/admin_console/services/task_queue_service.py
+249 -60
View File
@@ -16,7 +16,6 @@ import asyncio
from datetime import datetime
import os
import shutil
import signal
import sys
import time
from typing import Any
@@ -31,6 +30,7 @@ try:
)
from admin_console.core.state import state
from admin_console.database.repositories.session_repository import session_repo
from artemis.runtime import process_supervisor
except ImportError:
from apps.admin_console.core.config import (
PAUSE_FILE,
@@ -40,6 +40,7 @@ except ImportError:
)
from apps.admin_console.core.state import state
from apps.admin_console.database.repositories.session_repository import session_repo
from artemis.runtime import process_supervisor
class TaskQueueService:
@@ -48,51 +49,114 @@ class TaskQueueService:
"""
@classmethod
async def queue_worker(cls):
while True:
goal = None
sess_id = None
def _broadcast_event(cls, event_type: str, data: Any):
"""Broadcasts an event safely to all registered subscribers."""
for cb in list(state.ipc_subscribers):
try:
item = await state.task_queue.get()
if isinstance(item, dict):
goal = item.get("goal")
profile = item.get("profile", "flash")
sess_id = item.get("session_id")
else:
goal = item
profile = "flash"
cb(event_type, data)
except Exception:
pass
@classmethod
def _get_next_pending_task(cls) -> dict[str, Any] | None:
"""Finds and returns the first pending task from queue_items."""
for item in state.queue_items:
if isinstance(item, dict) and item.get("status") == "pending":
return item
return None
@classmethod
def _remove_task(cls, session_id: str | None):
"""Removes a task from queue_items by session_id."""
if not session_id:
return
state.queue_items = [
t
for t in state.queue_items
if not (isinstance(t, dict) and t.get("session_id") == session_id)
]
@classmethod
def ensure_worker_running(cls):
"""Guarantees that the background queue worker task is active and running."""
try:
loop = asyncio.get_running_loop()
if state.worker_task is None or state.worker_task.done():
state.worker_task = loop.create_task(cls.queue_worker())
try:
state.wake_event.set()
except Exception:
pass
except RuntimeError:
pass
@classmethod
async def queue_worker(cls):
"""Persistent, self-healing background worker loop processing tasks in FIFO order."""
print("[QueueWorker] Background worker loop initialized and running.")
while True:
task_item = None
sess_id = None
goal = None
profile = "flash"
try:
if state.is_running:
await asyncio.sleep(0.2)
continue
# 1. Fetch the next pending task
task_item = cls._get_next_pending_task()
if task_item is None:
await asyncio.sleep(0.3)
continue
# 2. Extract and initialize task execution metadata
sess_id = task_item.get("session_id")
if sess_id and str(sess_id) in getattr(state, "cancelled_session_ids", set()):
cls._remove_task(sess_id)
continue
goal = task_item.get("goal")
profile = task_item.get("profile", "flash")
expected_output = task_item.get("expected_output")
enable_outputter = task_item.get("enable_outputter")
locked_app = task_item.get("locked_app_package") or task_item.get("locked_app")
app_path = task_item.get("app_path")
task_item["status"] = "running"
task_item["start_time"] = time.time()
state.current_goal = goal
state.current_profile = profile
if sess_id:
state.active_session_id = sess_id
state.was_stopped_manually = False
# Remove from state.queue_tasks
state.queue_tasks = [
t
for t in state.queue_tasks
if not (
isinstance(t, dict)
and (t.get("session_id") == sess_id or t.get("goal") == goal)
# Broadcast session_started so all connected clients know the task has started
cls._broadcast_event(
"session_started",
{
"session_id": sess_id,
"initial_goal": goal,
"profile": profile,
},
)
]
test_name = f"web_{int(time.time())}"
env = os.environ.copy()
pythonpath_parts = [
str(WORKSPACE_ROOT),
str(WORKSPACE_ROOT / "apps/admin_console"),
str(WORKSPACE_ROOT / "apps/cloud_service"),
str(WORKSPACE_ROOT / "apps" / "admin_console"),
str(WORKSPACE_ROOT / "apps" / "cloud_service"),
env.get("PYTHONPATH", ""),
]
env["PYTHONPATH"] = os.pathsep.join([p for p in pythonpath_parts if p])
env["PYTHONUTF8"] = "1"
env["PYTHONUNBUFFERED"] = "1"
if state.ipc_port is not None:
env["ARTEMIS_IPC_PORT"] = str(state.ipc_port)
if sess_id:
env["ARTEMIS_SESSION_ID"] = str(sess_id)
print(f"[QueueWorker] Starting task: {goal} with profile: {profile}")
state.current_process = await asyncio.create_subprocess_exec(
cmd = [
sys.executable,
"-m",
"artemis.main",
@@ -101,48 +165,98 @@ class TaskQueueService:
profile,
"--test-name",
test_name,
]
if expected_output:
cmd.extend(["--output-description", str(expected_output)])
if enable_outputter is not None:
cmd.append("--enable-outputter" if enable_outputter else "--disable-outputter")
if locked_app:
cmd.extend(["--locked-app", str(locked_app)])
if app_path:
cmd.extend(["--app-path", str(app_path)])
print(
f"[QueueWorker] Starting task [{sess_id}]: '{goal}' (profile: {profile}, outputter: {bool(expected_output or enable_outputter)})"
)
proc = await asyncio.create_subprocess_exec(
*cmd,
cwd=str(WORKSPACE_ROOT),
env=env,
)
await state.current_process.wait()
print(
f"[QueueWorker] Task finished with code "
f"{state.current_process.returncode}: {goal}"
)
state.current_process = proc
task_item["pid"] = proc.pid
# Fallback status update
if state.active_session_id:
curr_status = session_repo.get_session_status(state.active_session_id)
if curr_status == "running":
if sess_id and (
str(sess_id) in getattr(state, "cancelled_session_ids", set())
or state.was_stopped_manually
):
print(
f"[QueueWorker] Task [{sess_id}] was cancelled during launch. Terminating."
)
try:
proc.kill()
except Exception:
pass
# 3. Await subprocess completion
returncode = await proc.wait()
print(f"[QueueWorker] Task [{sess_id}] exited with returncode {returncode}")
# 4. Perform fallback database status update and notification
if sess_id:
if state.was_stopped_manually:
new_status = "cancelled"
elif returncode == 0:
new_status = "completed"
else:
new_status = (
"completed" if state.current_process.returncode == 0 else "failed"
new_status = "failed"
session_repo.update_session_status(sess_id, new_status, time.time())
state.active_connections.pop(sess_id, None)
print(f"[QueueWorker] Updated session {sess_id} status to '{new_status}'")
cls._broadcast_event(
"session_ended",
{
"session_id": sess_id,
"status": new_status,
"was_stopped_manually": state.was_stopped_manually,
},
)
session_repo.update_session_status(
state.active_session_id, new_status, time.time()
)
print(
f"[QueueWorker] Fallback: Updated session "
f"{state.active_session_id} status to {new_status}"
)
state.active_session_id = None
state.current_profile = None
state.was_stopped_manually = False
except asyncio.CancelledError:
print("[QueueWorker] Background worker task was cancelled.")
print("[QueueWorker] Task received cancellation signal.")
if state.current_process and state.current_process.returncode is None:
try:
state.current_process.kill()
except Exception:
pass
break
except Exception as e:
print(f"[QueueWorker] Error executing task: {e}")
print(f"[QueueWorker] Unexpected error executing task: {e}")
await asyncio.sleep(0.5)
finally:
if goal is not None:
state.task_queue.task_done()
# 5. Clean up completed task from queue and reset execution state
if sess_id:
cls._remove_task(sess_id)
state.current_process = None
state.active_session_id = None
state.current_goal = None
state.current_profile = None
state.was_stopped_manually = False
state.wake_event.set()
@classmethod
async def enqueue_tasks(cls, goals: list[str], profile: str = "flash") -> dict[str, Any]:
async def enqueue_tasks(
cls,
goals: list[str],
profile: str = "flash",
expected_output: str | None = None,
enable_outputter: bool | None = None,
locked_app_package: str | None = None,
app_path: str | None = None,
) -> dict[str, Any]:
"""Enqueues one or more goals and wakes up the background worker."""
cls.ensure_worker_running()
enqueued_tasks = []
now = time.time()
for i, goal in enumerate(goals):
@@ -151,14 +265,20 @@ class TaskQueueService:
"session_id": sess_id,
"goal": goal,
"profile": profile or "flash",
"expected_output": expected_output,
"enable_outputter": enable_outputter,
"locked_app_package": locked_app_package,
"app_path": app_path,
"status": "pending",
"created_at": now + i * 0.001,
"start_time": now + i * 0.001,
}
await state.task_queue.put(task_item)
state.queue_tasks.append(task_item)
state.queue_items.append(task_item)
enqueued_tasks.append(task_item)
# Wake worker immediately
state.wake_event.set()
return {
"status": "queued" if state.is_running else "started",
"tasks": enqueued_tasks,
@@ -168,36 +288,105 @@ class TaskQueueService:
@classmethod
def stop_tasks(cls, clear_all: bool = False) -> bool:
"""Stops the currently executing task and optionally clears pending tasks."""
stopped = False
stopped_session_id = state.active_session_id
if not stopped_session_id:
running_item = next(
(
t
for t in state.queue_items
if isinstance(t, dict) and t.get("status") == "running"
),
None,
)
if running_item:
stopped_session_id = running_item.get("session_id")
elif state.queue_items and not clear_all:
stopped_session_id = state.queue_items[0].get("session_id")
if stopped_session_id:
getattr(state, "cancelled_session_ids", set()).add(str(stopped_session_id))
for t in state.queue_items:
if isinstance(t, dict) and t.get("status") == "running" and t.get("session_id"):
getattr(state, "cancelled_session_ids", set()).add(str(t["session_id"]))
if clear_all:
state.clear_queue()
stopped = True
state.was_stopped_manually = True
if state.is_running and state.current_process:
if state.current_process:
pid = getattr(state.current_process, "pid", None)
if pid:
try:
state.current_process.kill()
stopped = True
process_supervisor.terminate_tree(pid)
except Exception:
pass
try:
state.current_process.kill()
except Exception:
pass
stopped = True
for _, conn_info in list(state.active_connections.items()):
pid = conn_info.get("pid")
if pid:
try:
os.kill(pid, signal.SIGINT)
process_supervisor.terminate_tree(pid)
stopped = True
except Exception as e:
print(f"Failed to kill external process {pid}: {e}")
# Always reset process and active states cleanly
state.current_process = None
state.active_connections.clear()
session_repo.mark_all_running_cancelled()
# Broadcast session_ended event so all SSE streams and subscribers update immediately
if stopped_session_id:
cls._broadcast_event(
"session_ended",
{
"session_id": stopped_session_id,
"status": "cancelled",
"was_stopped_manually": True,
},
)
stopped = True
state.active_session_id = None
state.current_goal = None
state.current_profile = None
# Ensure any running or stopped task is removed from queue_items so worker_loop is unblocked
if clear_all:
state.queue_items = []
else:
state.queue_items = [
t
for t in state.queue_items
if not (
isinstance(t, dict)
and (
t.get("status") == "running"
or (stopped_session_id and t.get("session_id") == stopped_session_id)
)
)
]
if PAUSE_FILE.exists():
try:
PAUSE_FILE.unlink()
except Exception:
pass
return stopped
# Wake up worker to process the next pending task or transition cleanly
cls.ensure_worker_running()
state.wake_event.set()
return stopped or True
@classmethod
def resume_task(cls) -> bool:
apps/showcase_ui/README.md
+31
View File
@@ -0,0 +1,31 @@
# ✨ Artemis Showcase & User Workspace UI
A modern, highly aesthetic presentation and interaction frontend built with **Angular 19** and **SCSS** for showcasing the Artemis mobile autonomous agent.
## 🎨 Visual Design Highlights
* **Aurora Glow Dynamics**: Ambient background lighting with smooth gradients.
* **Glassmorphism Aesthetic**: Translucent frosted-glass panels with subtle borders.
* **Real-time Dual-Pane Workspace**:
- **Left Pane (`app-agent-stream`)**: Live streaming of Agent thought steps, plan breakdowns, tool actions, and status.
- **Right Pane (`app-chat-interface`)**: Natural language chat input sidebar with interactive feedback and status pills.
## 🚀 How to Run
### Prerequisites
- Node.js (>= 18)
- Backend API running on `http://localhost:8000` (via `apps/admin_console` or `artemis ui`)
### Development Server
```bash
cd apps/showcase_ui
npm install
npm start
```
This runs `ng serve --proxy-config proxy.conf.json` on **`http://localhost:4200/`**.
### Production Build
```bash
npm run build
```
Build artifacts will be emitted to `dist/`, which can be served statically by `apps.admin_console` or `artemis ui`.
angular.json → apps/showcase_ui/angular.json
+4 -4
View File
@@ -45,13 +45,13 @@
"budgets": [
{
"type": "initial",
"maximumWarning": "500kB",
"maximumError": "1MB"
"maximumWarning": "2MB",
"maximumError": "4MB"
},
{
"type": "anyComponentStyle",
"maximumWarning": "50kB",
"maximumError": "100kB"
"maximumWarning": "200kB",
"maximumError": "400kB"
}
],
"outputHashing": "all"
package-lock.json → apps/showcase_ui/package-lock.json
Generated
+6215 -6560
View File
File diff suppressed because it is too large. Load diff
package.json → apps/showcase_ui/package.json
+8 -9
View File
@@ -1,7 +1,6 @@
{
"name": "latte-showcase-ui",
"name": "artemis-showcase-ui",
"version": "1.0.0",
"scripts": {
"ng": "ng",
"start": "ng serve --proxy-config proxy.conf.json",
@@ -11,21 +10,20 @@
},
"private": true,
"dependencies": {
"@angular/common": "^22.1.2",
"@angular/compiler": "^22.1.2",
"@angular/core": "^22.1.2",
"@angular/common": "^19.2.0",
"@angular/compiler": "^19.2.0",
"@angular/core": "^19.2.0",
"@angular/forms": "^19.2.0",
"@angular/platform-browser": "^19.2.0",
"@angular/platform-browser-dynamic": "^19.2.0",
"@angular/router": "^19.2.0",
"node": "^26.5.1",
"rxjs": "~7.8.0",
"tslib": "^2.3.0",
"zone.js": "~0.15.0"
},
"devDependencies": {
"@angular-devkit/build-angular": "^22.1.4",
"@angular/cli": "^22.1.4",
"@angular-devkit/build-angular": "^19.2.0",
"@angular/cli": "^19.2.0",
"@angular/compiler-cli": "^19.2.0",
"@types/jasmine": "~5.1.0",
"jasmine-core": "~5.6.0",
@@ -35,5 +33,6 @@
"karma-jasmine": "~5.1.0",
"karma-jasmine-html-reporter": "~2.1.0",
"typescript": "~5.7.2"
}
},
"packageManager": "pnpm@11.22.0+sha512.1ff870c4c6133dfd88fb2afc46dd13d47f09c9794b438c6fdb47ca98caf3bc16381ee0be93a091b8e3824cf01f889f46d7d9e20910fb0be1ab0fb5baa80dd621"
}
proxy.conf.json → apps/showcase_ui/proxy.conf.json
View File
File renamed without changes.
apps/showcase_ui/public/favicon.ico
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 51 KiB

apps/showcase_ui/public/logo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 221 KiB

apps/showcase_ui/public/logo.svg
+56
View File
@@ -0,0 +1,56 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 240 240" width="240" height="240">
<defs>
<!-- Radial Glows for Mesh effect -->
<radialGradient id="redGlow" cx="50%" cy="50%" r="50%">
<stop offset="0%" stop-color="#EA4335" stop-opacity="1" />
<stop offset="40%" stop-color="#EA4335" stop-opacity="0.95" />
<stop offset="100%" stop-color="#EA4335" stop-opacity="0" />
</radialGradient>
<radialGradient id="yellowGlow" cx="50%" cy="50%" r="50%">
<stop offset="0%" stop-color="#FBBC05" stop-opacity="1" />
<stop offset="40%" stop-color="#FBBC05" stop-opacity="0.95" />
<stop offset="100%" stop-color="#FBBC05" stop-opacity="0" />
</radialGradient>
<radialGradient id="greenGlow" cx="50%" cy="50%" r="50%">
<stop offset="0%" stop-color="#34A853" stop-opacity="1" />
<stop offset="40%" stop-color="#34A853" stop-opacity="0.95" />
<stop offset="100%" stop-color="#34A853" stop-opacity="0" />
</radialGradient>
<!-- Robot Silhouette Mask -->
<mask id="robotMeshMask">
<rect width="240" height="240" fill="black" />
<!-- Antennae -->
<path d="M 90 60 Q 75 35 65 28" stroke="white" stroke-width="10" stroke-linecap="round" />
<circle cx="62" cy="24" r="10" fill="white" />
<path d="M 150 60 Q 165 35 175 28" stroke="white" stroke-width="10" stroke-linecap="round" />
<circle cx="178" cy="24" r="10" fill="white" />
<!-- Ears -->
<rect x="28" y="100" width="16" height="48" rx="8" fill="white" />
<rect x="196" y="100" width="16" height="48" rx="8" fill="white" />
<!-- Head -->
<rect x="44" y="58" width="152" height="132" rx="40" fill="white" />
<!-- Visor Cutout -->
<rect x="64" y="88" width="112" height="52" rx="22" fill="black" />
<!-- Eyes (Inside Visor) -->
<path d="M 82 118 Q 92 106 102 118" stroke="white" stroke-width="6" stroke-linecap="round" fill="none" />
<path d="M 138 118 Q 148 106 158 118" stroke="white" stroke-width="6" stroke-linecap="round" fill="none" />
<!-- Smile Cutout -->
<path d="M 104 158 Q 120 172 136 158" stroke="black" stroke-width="6" stroke-linecap="round" fill="none" />
</mask>
</defs>
<!-- Mesh Gradient applied to Mask -->
<g mask="url(#robotMeshMask)">
<rect width="240" height="240" fill="#4285F4" />
<circle cx="160" cy="50" r="100" fill="url(#redGlow)" />
<circle cx="60" cy="120" r="95" fill="url(#yellowGlow)" />
<circle cx="110" cy="200" r="95" fill="url(#greenGlow)" />
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 2.5 KiB

src/app/app.component.html → apps/showcase_ui/src/app/app.component.html
+1
View File
@@ -14,4 +14,5 @@
limitations under the License.
-->
<app-nav-switcher></app-nav-switcher>
<router-outlet></router-outlet>
src/app/app.component.scss → apps/showcase_ui/src/app/app.component.scss
View File
File renamed without changes.
src/app/app.component.spec.ts → apps/showcase_ui/src/app/app.component.spec.ts
+3 -6
View File
@@ -15,12 +15,15 @@
*/
import { TestBed } from '@angular/core/testing';
import { provideRouter } from '@angular/router';
import { provideHttpClient } from '@angular/common/http';
import { AppComponent } from './app.component';
describe('AppComponent', () => {
beforeEach(async () => {
await TestBed.configureTestingModule({
imports: [AppComponent],
providers: [provideRouter([]), provideHttpClient()]
}).compileComponents();
});
@@ -36,10 +39,4 @@ describe('AppComponent', () => {
expect(app.title).toEqual('frontend');
});
it('should render title', () => {
const fixture = TestBed.createComponent(AppComponent);
fixture.detectChanges();
const compiled = fixture.nativeElement as HTMLElement;
expect(compiled.querySelector('h1')?.textContent).toContain('Hello, frontend');
});
});
src/app/app.component.ts → apps/showcase_ui/src/app/app.component.ts
+2 -1
View File
@@ -16,10 +16,11 @@
import { Component } from '@angular/core';
import { RouterOutlet } from '@angular/router';
import { NavSwitcherComponent } from './components/nav-switcher/nav-switcher.component';
@Component({
selector: 'app-root',
imports: [RouterOutlet],
imports: [RouterOutlet, NavSwitcherComponent],
templateUrl: './app.component.html',
styleUrl: './app.component.scss'
})
src/app/app.config.ts → apps/showcase_ui/src/app/app.config.ts
View File
File renamed without changes.
src/app/app.routes.ts → apps/showcase_ui/src/app/app.routes.ts
+5 -2
View File
@@ -15,10 +15,13 @@
*/
import { Routes } from '@angular/router';
import { HomeComponent } from './pages/home/home.component';
import { WorkspaceComponent } from './pages/workspace/workspace.component';
import { LegacyWorkspaceComponent } from './pages/legacy-workspace/legacy-workspace.component';
export const routes: Routes = [
{ path: '', component: WorkspaceComponent },
{ path: 'check', component: LegacyWorkspaceComponent }
{ path: '', component: HomeComponent },
{ path: 'workspace', component: WorkspaceComponent },
{ path: 'check', component: LegacyWorkspaceComponent },
{ path: '**', redirectTo: '' }
];
src/app/components/agent-stream/agent-stream.component.html → apps/showcase_ui/src/app/components/agent-stream/agent-stream.component.html
+175 -37
View File
@@ -1,11 +1,29 @@
<!--
Copyright 2026 Google LLC
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<div class="agent-stream-container">
<!-- VS Code Style Navigation Header -->
<!-- VS Code Style Navigation Header (for responsive small screens) -->
<header class="vscode-nav-header">
<div class="vscode-tab-container">
<button
class="vscode-tab-btn"
[class.active]="isTaskDropdownOpen()"
(click)="toggleTaskDropdown($event)"
title="Task Queue"
aria-label="Task Queue"
>
<span class="material-symbols-outlined tab-icon">list_alt</span>
<span class="tab-label">Task Queue</span>
@@ -14,6 +32,8 @@
class="vscode-tab-btn"
[class.active]="isNotesDropdownOpen()"
(click)="toggleNotesDropdown($event)"
title="Notes & Plans"
aria-label="Notes & Plans"
>
<span class="material-symbols-outlined tab-icon">description</span>
<span class="tab-label">Notes & Plans</span>
@@ -22,6 +42,10 @@
<!-- Dropdown Popover Panel for Task Queue -->
<div class="vscode-dropdown-menu task-dropdown" *ngIf="isTaskDropdownOpen()">
<div class="popover-title-header">
<span class="material-symbols-outlined">list_alt</span>
<span>Task Queue</span>
</div>
<div class="dropdown-section-label">
<span class="material-symbols-outlined icon-bolt">bolt</span>
<span>Active Queue ({{ activeQueue().length }})</span>
@@ -43,7 +67,19 @@
<span class="status-dot" *ngIf="getTaskStatus(session) === 'running'"></span>
{{ getTaskStatus(session) | uppercase }}
</span>
<div class="item-header-actions">
<span class="item-time">{{ session.start_time * 1000 | date:'shortTime' }}</span>
<button
*ngIf="getTaskStatus(session) !== 'running'"
type="button"
class="btn-delete-card"
(click)="deleteTask(session.session_id, $event)"
title="Delete this task"
aria-label="Delete this task"
>
<span class="material-symbols-outlined">delete</span>
</button>
</div>
</div>
<div class="item-goal" title="{{ session.initial_goal }}">
{{ session.initial_goal || 'Unnamed Task' }}
@@ -76,7 +112,19 @@
<span class="status-dot" *ngIf="getTaskStatus(session) === 'running'"></span>
{{ getTaskStatus(session) | uppercase }}
</span>
<div class="item-header-actions">
<span class="item-time">{{ session.start_time * 1000 | date:'shortTime' }}</span>
<button
*ngIf="getTaskStatus(session) !== 'running'"
type="button"
class="btn-delete-card"
(click)="deleteTask(session.session_id, $event)"
title="Delete this task"
aria-label="Delete this task"
>
<span class="material-symbols-outlined">delete</span>
</button>
</div>
</div>
<div class="item-goal" title="{{ session.initial_goal }}">
{{ session.initial_goal || 'Unnamed Task' }}
@@ -91,6 +139,10 @@
<!-- Dropdown Popover Panel for Notes & Plans -->
<div class="vscode-dropdown-menu notes-dropdown" *ngIf="isNotesDropdownOpen()">
<div class="popover-title-header">
<span class="material-symbols-outlined">description</span>
<span>Notes & Plans</span>
</div>
<!-- Notes Selector / Pills (Shown if there are note files) -->
<div class="notes-selector-container" *ngIf="noteKeys().length > 1">
<div class="notes-selector-list">
@@ -232,7 +284,7 @@
</div>
<!-- Event Logs Section -->
<main class="stream-logs-content">
<main class="stream-logs-content" (scroll)="onStreamScroll($event)">
<div *ngIf="consolidatedBlocks().length === 0 && !showPlanningLoader()" class="empty-state">
<div class="empty-icon"><span class="material-symbols-outlined">sensors</span></div>
@@ -300,6 +352,69 @@
<!-- Tool Call / Action section (Unified Chronological Loop matching Debug UI execution order) -->
<div class="step-actions-container" *ngIf="(block.data.generic_tools && block.data.generic_tools.length > 0) || block.data.action_taken">
<ng-container *ngFor="let item of getSortedStepEvents(block.data); let itemIdx = index; trackBy: trackStepEvent">
<!-- Task Submission / Report Card (when item is report_task_status action or tool) -->
<div class="step-android-action-card task-report-card"
[class.is-completed]="getReportStatusValue(item.data) === 'completed' || getReportStatusValue(item.data) === 'success'"
[class.is-failed]="getReportStatusValue(item.data) === 'failed'"
[class.is-expanded]="isActionExpanded('action-' + block.id + '-' + itemIdx)"
*ngIf="isReportStatusAction(item.data)">
<div class="action-card-header" (click)="toggleActionExpanded('action-' + block.id + '-' + itemIdx, $event)">
<span class="material-symbols-outlined action-icon report-icon" [class.is-failed]="getReportStatusValue(item.data) === 'failed'">
{{ getReportStatusValue(item.data) === 'failed' ? 'cancel' : 'task_alt' }}
</span>
<span class="action-title">
{{ getReportStatusValue(item.data) === 'failed' ? 'Task Execution Failed' : 'Task Completed' }}
</span>
<span class="action-status-badge" [class.is-failed]="getReportStatusValue(item.data) === 'failed'" [class.is-completed]="getReportStatusValue(item.data) !== 'failed'">
<span class="material-symbols-outlined status-icon">
{{ getReportStatusValue(item.data) === 'failed' ? 'error' : 'check_circle' }}
</span>
{{ getReportStatusValue(item.data) === 'failed' ? 'Failed' : 'Completed' }}
</span>
<span class="action-expand-icon material-symbols-outlined">
{{ isActionExpanded('action-' + block.id + '-' + itemIdx) ? 'expand_less' : 'expand_more' }}
</span>
</div>
<!-- Compact preview body (when collapsed, matches standard action card) -->
<div class="action-card-body" *ngIf="!isActionExpanded('action-' + block.id + '-' + itemIdx) && getReportStatusExplanation(item.data)">
<div class="action-detail-row">
<span class="detail-label">Summary:</span>
<span class="detail-value">"{{ getReportStatusExplanation(item.data) }}"</span>
</div>
</div>
<!-- Detailed expanded view (when clicked, matches standard action card structure) -->
<div class="action-card-expanded-details" *ngIf="isActionExpanded('action-' + block.id + '-' + itemIdx)" (click)="$event.stopPropagation()">
<div class="expanded-grid">
<div class="grid-item full-width" *ngIf="getReportStatusExplanation(item.data) as expText">
<span class="grid-label">Report & Findings</span>
<span class="grid-value report-markdown-body" [innerHTML]="renderMarkdown(expText)"></span>
</div>
</div>
<!-- Final Verified Screenshot -->
<div class="expanded-screenshots-container" *ngIf="getStepPreImageUrl(block.data, item.data) || getStepPostImageUrl(block.data, item.data)">
<div class="screenshot-column" *ngIf="getStepPreImageUrl(block.data, item.data)">
<span class="screenshot-label">Final Verified State</span>
<div class="image-wrapper" (click)="openImageModal(getStepPreImageUrl(block.data, item.data)!, item.data, $event)">
<div class="img-relative-box">
<img [src]="getStepPreImageUrl(block.data, item.data)" alt="Final Screen Screenshot" />
</div>
</div>
</div>
<div class="screenshot-column" *ngIf="getStepPostImageUrl(block.data, item.data)">
<span class="screenshot-label">Post Verification</span>
<div class="image-wrapper" (click)="openImageModal(getStepPostImageUrl(block.data, item.data)!, null, $event)">
<div class="img-relative-box">
<img [src]="getStepPostImageUrl(block.data, item.data)" alt="Post Screenshot" />
</div>
</div>
</div>
</div>
</div>
</div>
<!-- Android Action Card (when item is an action) -->
<div class="step-android-action-card"
[class.is-running]="isItemRunning(item, block)"
@@ -311,7 +426,7 @@
<span class="action-title">{{ getActionTitle(item.data) }}</span>
<span class="action-status-badge is-failed" *ngIf="isActionFailed(item.data, block.data)">
<span class="material-symbols-outlined status-icon">error</span>
{{ getActionErrorMessage(item.data, block.data) }}
Failed
</span>
<span class="action-expand-icon material-symbols-outlined">
{{ isActionExpanded('action-' + block.id + '-' + itemIdx) ? 'expand_less' : 'expand_more' }}
@@ -330,7 +445,7 @@
</div>
<div class="action-detail-row" *ngIf="getActionCoords(item.data)">
<span class="detail-label">Coordinates:</span>
<span class="detail-value coordinate-badge">[{{ getActionCoords(item.data) }}]</span>
<span class="detail-value coordinate-badge">{{ getActionCoords(item.data) }}</span>
</div>
</div>
@@ -347,7 +462,7 @@
</div>
<div class="grid-item" *ngIf="getActionCoords(item.data)">
<span class="grid-label">Coordinates</span>
<span class="grid-value code-pill">[{{ getActionCoords(item.data) }}]</span>
<span class="grid-value code-pill">{{ getActionCoords(item.data) }}</span>
</div>
<div class="grid-item" *ngIf="getActionBounds(item.data)">
<span class="grid-label">Target Bounds</span>
@@ -403,7 +518,7 @@
<div class="step-android-action-card tool-action-card is-failed" *ngIf="item.data.type === 'llm_call' && item.data.status === 'failed'">
<div class="action-card-header">
<span class="material-symbols-outlined action-icon is-failed">error</span>
<span class="action-title">Gemini AI Failed</span>
<span class="action-title">LLM Request Failed</span>
<span class="action-status-badge is-failed">
<span class="material-symbols-outlined status-icon">warning</span>
Failed
@@ -422,7 +537,7 @@
[class.is-running]="isItemRunning(item, block)"
[class.is-note-tool]="isNoteTool(item.data)"
[class.is-video-tool]="isVideoTool(item.data)"
*ngIf="shouldShowTool(item.data, block.data) && !isFailureAnalyzerActionTool(item.data)">
*ngIf="shouldShowTool(item.data, block.data) && !isDeviceActionTool(item.data)">
<div class="tool-call-label">
<span class="stream-status-dot" [class.streaming]="isItemRunning(item, block)"></span>
<div class="tool-call-text">
@@ -440,14 +555,14 @@
</span>
</ng-container>
<!-- Video Analyzer Pill -->
<!-- Video Tool Pill -->
<ng-container *ngIf="isVideoTool(item.data) && getVideoToolTarget(item.data) as videoTarget">
<span
class="tool-key-pill video-pill"
(click)="onVideoToolClick(item.data)"
title="Click to open video player"
>
<span class="material-symbols-outlined pill-icon">video_camera_back</span>
<span class="material-symbols-outlined pill-icon">videocam</span>
{{ videoTarget }}
</span>
</ng-container>
@@ -455,19 +570,20 @@
</div>
</div>
<!-- Action Card format (ONLY for FailureAnalyzer non-note tools) -->
<!-- Action Card format (For Device Actions & ADB Commands) -->
<div class="step-android-action-card tool-action-card"
[class.is-running]="isItemRunning(item, block)"
[class.has-no-details]="isAdbCommandTool(item.data) ? !getAdbCommandLine(item.data) : (!getToolTargetText(item.data) && !getToolInputText(item.data) && !getToolCoords(item.data) && !getToolAnalysisText(item.data) && !getToolGenericDetails(item.data) && getToolExtraParams(item.data).length === 0)"
[class.is-failed]="isToolFailed(item.data)"
[class.is-expanded]="isActionExpanded('tool-' + block.id + '-' + itemIdx)"
*ngIf="shouldShowTool(item.data, block.data) && isFailureAnalyzerActionTool(item.data)">
*ngIf="shouldShowTool(item.data, block.data) && isDeviceActionTool(item.data)">
<div class="action-card-header" (click)="toggleActionExpanded('tool-' + block.id + '-' + itemIdx, $event)">
<span class="material-symbols-outlined action-icon" [class.is-failed]="isToolFailed(item.data)">{{ getToolIcon(item.data) }}</span>
<span class="action-title">{{ getToolTitle(item.data) }}</span>
<span class="action-agent-badge" [class.is-outputter]="getToolAgentName(item.data) === 'Outputter'" *ngIf="getToolAgentName(item.data)">{{ getToolAgentName(item.data) }}</span>
<span class="action-status-badge is-failed" *ngIf="isToolFailed(item.data)">
<span class="material-symbols-outlined status-icon">error</span>
{{ getToolErrorMessage(item.data) }}
Failed
</span>
<span class="action-expand-icon material-symbols-outlined">
{{ isActionExpanded('tool-' + block.id + '-' + itemIdx) ? 'expand_less' : 'expand_more' }}
@@ -504,7 +620,7 @@
</div>
<div class="action-detail-row" *ngIf="getToolCoords(item.data)">
<span class="detail-label">Coordinates:</span>
<span class="detail-value coordinate-badge">[{{ getToolCoords(item.data) }}]</span>
<span class="detail-value coordinate-badge">{{ getToolCoords(item.data) }}</span>
</div>
<div class="action-detail-row" *ngIf="getToolAnalysisText(item.data)">
<span class="detail-label">Analysis:</span>
@@ -546,7 +662,7 @@
</div>
<div class="grid-item" *ngIf="getToolCoords(item.data)">
<span class="grid-label">Coordinates</span>
<span class="grid-value code-pill">[{{ getToolCoords(item.data) }}]</span>
<span class="grid-value code-pill">{{ getToolCoords(item.data) }}</span>
</div>
<div class="grid-item" *ngIf="getToolAnalysisText(item.data)">
<span class="grid-label">Analysis</span>
@@ -600,38 +716,60 @@
</div>
</div>
<!-- Outputter Post-Execution Synthesis Card (Rendered ONCE at the end of stream) -->
<div class="outputter-synthesis-stream-card" *ngIf="outputterReport() as outReport">
<div class="synthesis-card-header">
<div class="header-left">
<span class="material-symbols-outlined icon-sparkle">assignment_turned_in</span>
<span class="main-title">Task Report</span>
</div>
<div class="header-right">
<button type="button" class="btn-open-note" (click)="openOutputterNote($event)">
<span class="material-symbols-outlined">visibility</span>
<span>View in Notes</span>
</button>
</div>
</div>
<div class="synthesis-card-preview">
<div class="parsed-note-container" *ngIf="getParsedNote(outReport) as parsedOut">
<h3 class="parsed-note-title" *ngIf="parsedOut.title">{{ parsedOut.title }}</h3>
<div class="parsed-milestones" *ngIf="parsedOut.milestones.length > 0">
<div class="milestone-item" *ngFor="let m of parsedOut.milestones" [class]="'status-' + m.type">
<span class="material-symbols-outlined status-icon">
{{ m.type === 'checked' ? 'check_circle' : (m.type === 'progress' ? 'pending' : 'radio_button_unchecked') }}
</span>
<span class="milestone-text">
<ng-container *ngFor="let seg of m.segments">
<strong *ngIf="seg.bold">{{ seg.text }}</strong>
<code *ngIf="seg.code">{{ seg.text }}</code>
<span *ngIf="!seg.bold && !seg.code">{{ seg.text }}</span>
</ng-container>
</span>
</div>
</div>
<div class="parsed-other-lines" *ngIf="parsedOut.otherLines.length > 0">
<p class="other-line" *ngFor="let line of parsedOut.otherLines" [class]="'line-' + line.type">
<ng-container *ngFor="let seg of line.segments">
<strong *ngIf="seg.bold">{{ seg.text }}</strong>
<code *ngIf="seg.code">{{ seg.text }}</code>
<span *ngIf="!seg.bold && !seg.code">{{ seg.text }}</span>
</ng-container>
</p>
</div>
</div>
</div>
</div>
<!-- Planning Loader -->
<div class="planning-loader-container" *ngIf="showPlanningLoader()">
<div class="planning-loader">
<span class="material-symbols-outlined spin-animation">progress_activity</span>
<span class="loader-text">Planning next steps...</span>
<span class="loader-text">{{ currentPlanningText() }}</span>
</div>
</div>
</div>
</main>
<!-- Floating Action Buttons for Task Execution (Icon-only) -->
<div class="fab-actions-container">
<button
*ngIf="agentService.isPaused()"
class="fab-btn fab-resume-task"
(click)="agentService.resumeTask()"
title="Resume task"
aria-label="Resume Task"
>
<span class="material-symbols-outlined">play_arrow</span>
</button>
<button
*ngIf="agentService.isRunningTask() && !agentService.isPaused()"
class="fab-btn fab-stop-task"
(click)="agentService.stopTask()"
title="Stop running task"
aria-label="Stop Task"
>
<span class="material-symbols-outlined">stop</span>
</button>
</div>
<!-- Fullscreen Image Zoom Lightbox Modal -->
<div class="image-modal-backdrop" *ngIf="selectedZoomImage()" (click)="closeImageModal($event)">
<div class="image-modal-content" (click)="$event.stopPropagation()">
src/app/components/agent-stream/agent-stream.component.scss → apps/showcase_ui/src/app/components/agent-stream/agent-stream.component.scss
+585 -217
View File
File diff suppressed because it is too large. Load diff
src/app/components/agent-stream/agent-stream.component.ts → apps/showcase_ui/src/app/components/agent-stream/agent-stream.component.ts
+179 -35
View File
@@ -14,15 +14,35 @@
* limitations under the License.
*/
import { Component, signal, computed, effect, inject, untracked } from '@angular/core';
import { Component, signal, computed, effect, inject, untracked, DestroyRef, AfterViewInit } from '@angular/core';
import { CommonModule } from '@angular/common';
import { FormsModule } from '@angular/forms';
import { RouterLink } from '@angular/router';
import { HttpClient } from '@angular/common/http';
import { AgentService } from '../../services/agent.service';
import { Session } from '../../core/models/session.model';
import { MarkdownSegment, MarkdownLine, NoteMilestone, ParsedNote } from '../../core/models/markdown.model';
import { StepBlock, PhaseBlock, StepEvent, ActionParam, CheckerResult } from '../../core/models/stream.model';
export const PLANNING_LOADER_PHRASES: string[] = [
'Planning next step...',
'Analyzing screen coordinates...',
'Consulting neural network...',
'Formulating tactical action plan...',
'Deciphering UI state & elements...',
'Synthesizing decision pathways...',
'Calibrating next move...',
'Aligning logical vectors...',
'Evaluating optimal sub-goals...',
'Gathering sensory inputs...',
'Computing next interaction...',
'Optimizing execution strategy...',
'Simulating probable outcomes...',
'Summoning AI intuition...',
'Brewing the next command...',
'Strategizing tactical moves...'
];
import {
parseNote,
parseNoteLines,
@@ -46,7 +66,10 @@ import {
getActionErrorMessage,
extractActionExtraParams,
getStepPreImageUrl,
getStepPostImageUrl
getStepPostImageUrl,
isReportStatusAction,
getReportStatusValue,
getReportStatusExplanation
} from '../../utils/action-formatter.util';
import {
@@ -54,7 +77,9 @@ import {
isNoteTool,
isVideoTool,
getVideoToolTarget,
isDeviceActionTool,
isFailureAnalyzerActionTool,
getToolAgentName,
shouldShowTool,
getUniqueGenericTools,
getToolKey,
@@ -100,9 +125,19 @@ export type { MarkdownSegment, MarkdownLine, NoteMilestone, ParsedNote };
templateUrl: './agent-stream.component.html',
styleUrl: './agent-stream.component.scss'
})
export class AgentStreamComponent {
export class AgentStreamComponent implements AfterViewInit {
public agentService = inject(AgentService);
private http = inject(HttpClient);
private destroyRef = inject(DestroyRef);
// Auto-scroll state tracking
public isUserAtBottom = true;
private resizeObserver: ResizeObserver | null = null;
// Dynamic Planning Loader Game-style Phrases
public currentPlanningText = signal<string>(PLANNING_LOADER_PHRASES[0]);
private planningIntervalId: any = null;
private currentPhraseIndex = 0;
// Top Nav Dropdown States
public isTaskDropdownOpen = signal<boolean>(false);
@@ -120,8 +155,9 @@ export class AgentStreamComponent {
public typedTextsSignal = signal<Record<string, string>>({});
private typingTimers = new Map<string, any>();
// Set to track expanded state of action cards
// Set to track expanded and collapsed state of action cards
public expandedActionCards = signal<Set<string>>(new Set<string>());
public collapsedActionCards = signal<Set<string>>(new Set<string>());
// Performance caches for parameter and event extractions
private actionParamsCache = new WeakMap<any, ActionParam[]>();
@@ -146,7 +182,7 @@ export class AgentStreamComponent {
public historyTasks = computed(() => {
return this.agentService.sessions().filter((s) => {
const status = this.getTaskStatus(s);
return status === 'completed' || status === 'failed';
return status === 'completed' || status === 'failed' || status === 'cancelled';
});
});
@@ -161,6 +197,12 @@ export class AgentStreamComponent {
return Object.keys(this.agentService.currentNotes()).filter(key => key.toLowerCase().endsWith('.md'));
});
public outputterReport = computed(() => {
const notes = this.agentService.currentNotes();
return notes['output.md'] || null;
});
public parsedNote = computed<ParsedNote>(() => {
return this.getParsedNote(this.currentNoteContent());
});
@@ -230,13 +272,9 @@ export class AgentStreamComponent {
setTimeout(() => {
const container = document.querySelector('.stream-logs-content');
if (container) {
const threshold = 150;
const isNearBottom = container.scrollHeight - container.scrollTop - container.clientHeight <= threshold;
if (isNearBottom) {
if (container && this.isUserAtBottom) {
container.scrollTop = container.scrollHeight;
}
}
}, 50);
});
@@ -316,6 +354,7 @@ export class AgentStreamComponent {
this.typedTextsSignal.set({});
this.scheduledCollapses.clear();
this.collapsedStreams.set(new Set<string>());
this.isUserAtBottom = true;
setTimeout(() => {
const container = document.querySelector('.stream-logs-content');
@@ -325,6 +364,58 @@ export class AgentStreamComponent {
}, 100);
});
});
// Periodically rotate loading text like game loading screen hints
effect(() => {
const isLoading = this.showPlanningLoader();
if (isLoading) {
if (!this.planningIntervalId) {
this.currentPhraseIndex = Math.floor(Math.random() * PLANNING_LOADER_PHRASES.length);
this.currentPlanningText.set(PLANNING_LOADER_PHRASES[this.currentPhraseIndex]);
this.planningIntervalId = setInterval(() => {
this.currentPhraseIndex = (this.currentPhraseIndex + 1) % PLANNING_LOADER_PHRASES.length;
this.currentPlanningText.set(PLANNING_LOADER_PHRASES[this.currentPhraseIndex]);
}, 2800);
}
} else {
if (this.planningIntervalId) {
clearInterval(this.planningIntervalId);
this.planningIntervalId = null;
}
}
});
this.destroyRef.onDestroy(() => {
if (this.planningIntervalId) {
clearInterval(this.planningIntervalId);
this.planningIntervalId = null;
}
if (this.resizeObserver) {
this.resizeObserver.disconnect();
this.resizeObserver = null;
}
});
}
public ngAfterViewInit(): void {
const container = document.querySelector('.stream-logs-content');
if (container && typeof ResizeObserver !== 'undefined') {
this.resizeObserver = new ResizeObserver(() => {
if (this.isUserAtBottom) {
container.scrollTop = container.scrollHeight;
}
});
this.resizeObserver.observe(container);
}
}
public onStreamScroll(event: Event): void {
const container = event.target as HTMLElement;
if (!container) return;
const threshold = 150;
const distanceToBottom = container.scrollHeight - container.scrollTop - container.clientHeight;
this.isUserAtBottom = distanceToBottom <= threshold;
}
// Top Nav Action Methods
@@ -338,24 +429,31 @@ export class AgentStreamComponent {
public toggleNotesDropdown(event: Event): void {
event.stopPropagation();
this.isNotesDropdownOpen.update(v => !v);
this.isNotesDropdownOpen.update((v) => !v);
if (this.isNotesDropdownOpen()) {
this.isTaskDropdownOpen.set(false);
}
}
public openOutputterNote(event?: Event): void {
if (event) event.stopPropagation();
this.agentService.selectedNoteKey.set('output.md');
this.isNotesDropdownOpen.set(true);
this.isTaskDropdownOpen.set(false);
}
public closeAllDropdowns(): void {
this.isTaskDropdownOpen.set(false);
this.isNotesDropdownOpen.set(false);
}
public getTaskStatus(session: Session): 'running' | 'completed' | 'pending' | 'failed' {
if (session.session_id === this.agentService.runningSessionId()) {
public getTaskStatus(session: Session): 'running' | 'completed' | 'pending' | 'failed' | 'cancelled' {
if (session.session_id === this.agentService.runningSessionId() && this.agentService.agentStatus() === 'running') {
return 'running';
}
if (session.status) {
const s = session.status.toLowerCase();
if (s === 'running' || s === 'completed' || s === 'pending' || s === 'failed') {
if (s === 'running' || s === 'completed' || s === 'pending' || s === 'failed' || s === 'cancelled') {
return s as any;
}
}
@@ -364,10 +462,22 @@ export class AgentStreamComponent {
public selectTask(sessionId: string, event?: Event): void {
if (event) event.stopPropagation();
this.agentService.selectSession(sessionId);
this.agentService.selectSession(sessionId, true);
this.isTaskDropdownOpen.set(false);
}
public deleteTask(sessionId: string, event?: Event): void {
if (event) event.stopPropagation();
if (!confirm(`Are you sure you want to delete this task? This cannot be undone.`)) {
return;
}
this.agentService.deleteSession(sessionId).subscribe({
error: (err) => {
console.error(`Failed to delete session ${sessionId}:`, err);
}
});
}
public selectNote(key: string, event?: Event): void {
if (event) event.stopPropagation();
this.agentService.selectedNoteKey.set(key);
@@ -499,23 +609,40 @@ export class AgentStreamComponent {
return getActionErrorMessage(action, stepData);
}
public isActionExpanded(cardId: string): boolean {
return this.expandedActionCards().has(cardId);
public isActionExpanded(cardId: string, defaultExpanded: boolean = false): boolean {
if (this.collapsedActionCards().has(cardId)) return false;
if (this.expandedActionCards().has(cardId)) return true;
return defaultExpanded;
}
public toggleActionExpanded(cardId: string, event?: Event): void {
public toggleActionExpanded(cardId: string, event?: Event, defaultExpanded: boolean = false): void {
if (event) {
event.stopPropagation();
}
const isCurrentlyExpanded = this.isActionExpanded(cardId, defaultExpanded);
if (isCurrentlyExpanded) {
this.expandedActionCards.update(set => {
const newSet = new Set(set);
if (newSet.has(cardId)) {
newSet.delete(cardId);
} else {
newSet.add(cardId);
}
return newSet;
});
this.collapsedActionCards.update(set => {
const newSet = new Set(set);
newSet.add(cardId);
return newSet;
});
} else {
this.collapsedActionCards.update(set => {
const newSet = new Set(set);
newSet.delete(cardId);
return newSet;
});
this.expandedActionCards.update(set => {
const newSet = new Set(set);
newSet.add(cardId);
return newSet;
});
}
}
public getActionBounds(action: any): string {
@@ -629,8 +756,16 @@ export class AgentStreamComponent {
}
}
public isDeviceActionTool(tool: any): boolean {
return isDeviceActionTool(tool);
}
public isFailureAnalyzerActionTool(tool: any): boolean {
return isFailureAnalyzerActionTool(tool);
return isDeviceActionTool(tool);
}
public getToolAgentName(tool: any): string | null {
return getToolAgentName(tool);
}
public shouldShowTool(tool: any, stepData?: any): boolean {
@@ -662,11 +797,23 @@ export class AgentStreamComponent {
if (!block) return false;
const hasNative = Boolean(this.getNativeThinking(block));
const hasRaw = Boolean(this.getRawThinking(block));
const hasVisibleTools = Boolean(block.data?.generic_tools) && block.data.generic_tools.some((t: any) => this.shouldShowTool(t, block.data) || (t.type === 'llm_call' && t.status === 'failed'));
const hasAndroidActions = Boolean(block.data?.action_taken) && this.isAndroidAction(block.data.action_taken);
const hasVisibleTools = Boolean(block.data?.generic_tools) && block.data.generic_tools.some((t: any) => this.shouldShowTool(t, block.data) || (t.type === 'llm_call' && t.status === 'failed') || this.isReportStatusAction(t));
const hasAndroidActions = Boolean(block.data?.action_taken) && (this.isAndroidAction(block.data.action_taken) || this.isReportStatusAction(block.data.action_taken));
return hasNative || hasRaw || hasVisibleTools || hasAndroidActions;
}
public isReportStatusAction(action: any): boolean {
return isReportStatusAction(action);
}
public getReportStatusValue(action: any): string {
return getReportStatusValue(action);
}
public getReportStatusExplanation(action: any): string {
return getReportStatusExplanation(action);
}
public isHumanThinking(text: string | null): boolean {
return isHumanThinking(text);
}
@@ -779,7 +926,7 @@ export class AgentStreamComponent {
const selectElement = event.target as HTMLSelectElement;
const sessionId = selectElement.value;
if (sessionId) {
this.agentService.selectSession(sessionId);
this.agentService.selectSession(sessionId, true);
}
}
@@ -927,12 +1074,12 @@ export class AgentStreamComponent {
public isItemRunning(item: any, block: any): boolean {
if (!item || !item.data) return false;
if (!this.isCurrentSessionActive()) return false;
if (item.data.status === 'success' || item.data.status === 'failed' || item.data.status === 'cancelled') return false;
if (this.isReportStatusAction(item.data) || this.isReportStatusAction(item.data?.action_taken)) return false;
if (item.data.status === 'running') return true;
if (item.data.status === 'success' || item.data.status === 'failed') return false;
if (item.data.action === 'report_task_status' || item.data?.action_taken?.action === 'report_task_status') return false;
// Check if session is currently running and this is the active trailing item
if (this.isCurrentSessionActive()) {
const blocks = this.consolidatedBlocks();
const lastBlock = blocks.length > 0 ? blocks[blocks.length - 1] : null;
if (lastBlock && (lastBlock.id === block?.id || !block?.data?.isCompleted)) {
@@ -942,19 +1089,16 @@ export class AgentStreamComponent {
return true;
}
}
}
return false;
}
public isBlockRunning(block: any): boolean {
if (!block || !block.data) return false;
if (block.data.action_taken?.action === 'report_task_status' || block.data?.status === 'completed' || block.data?.status === 'failed') return false;
if (block.data.isCompleted === false && this.isCurrentSessionActive()) return true;
if (this.isCurrentSessionActive()) {
if (!this.isCurrentSessionActive()) return false;
if (this.isReportStatusAction(block.data.action_taken) || block.data?.status === 'completed' || block.data?.status === 'failed' || block.data?.status === 'cancelled') return false;
if (block.data.isCompleted === false) return true;
const blocks = this.consolidatedBlocks();
const lastBlock = blocks.length > 0 ? blocks[blocks.length - 1] : null;
return lastBlock?.id === block.id;
}
return false;
}
}
src/app/components/chat-interface/chat-interface.component.html → apps/showcase_ui/src/app/components/chat-interface/chat-interface.component.html
+27 -46
View File
@@ -17,8 +17,6 @@
<div class="chat-interface-container">
<!-- Task Queue & History Header -->
<header class="chat-header">
<div class="header-left"></div>
<!-- Tab Selector Header inside row -->
<div class="tab-selector-bar">
<button
@@ -78,7 +76,20 @@
<span class="status-dot" *ngIf="getTaskStatus(session) === 'running'"></span>
{{ getTaskStatus(session) | uppercase }}
</span>
<div class="task-card-header-actions">
<span class="task-time">{{ session.start_time * 1000 | date:'shortTime' }}</span>
<button
*ngIf="getTaskStatus(session) !== 'running'"
type="button"
class="btn-delete-card"
(click)="deleteTask(session.session_id, $event)"
title="Delete this task"
aria-label="Delete this task"
[disabled]="isSubmitting"
>
<span class="material-symbols-outlined">delete</span>
</button>
</div>
</div>
<div class="task-goal" title="{{ session.initial_goal }}">
@@ -114,7 +125,20 @@
<span class="status-dot" *ngIf="getTaskStatus(session) === 'running'"></span>
{{ getTaskStatus(session) | uppercase }}
</span>
<div class="task-card-header-actions">
<span class="task-time">{{ session.start_time * 1000 | date:'shortTime' }}</span>
<button
*ngIf="getTaskStatus(session) !== 'running'"
type="button"
class="btn-delete-card"
(click)="deleteTask(session.session_id, $event)"
title="Delete this task"
aria-label="Delete this task"
[disabled]="isSubmitting"
>
<span class="material-symbols-outlined">delete</span>
</button>
</div>
</div>
<div class="task-goal" title="{{ session.initial_goal }}">
@@ -237,48 +261,5 @@
</div>
</ng-container>
</main>
<!-- Task Submission Area (Only shown in Tasks tab) - Temporarily disabled/hidden
<footer class="chat-input-panel" *ngIf="agentService.activeTab() === 'tasks'">
<div class="error-banner" *ngIf="errorMessage">
<span class="material-symbols-outlined error-icon">warning</span> {{ errorMessage }}
</div>
<div class="input-row">
<textarea
[(ngModel)]="taskInput"
(keydown.enter)="$event.preventDefault(); submitTask()"
placeholder="Enter goal (e.g. add contact name Jerry)..."
class="chat-input"
rows="1"
[disabled]="isSubmitting"
></textarea>
<button
*ngIf="agentService.agentStatus() === 'running' || activeQueue().length > 0"
(click)="stopTasks(false)"
[disabled]="isSubmitting"
class="btn-stop"
[title]="activeQueue().length > 1 ? 'Stop currently running task only' : 'Stop currently running task'"
>
{{ activeQueue().length > 1 ? 'Stop Current' : 'Stop' }}
</button>
<button
*ngIf="activeQueue().length > 1"
(click)="stopTasks(true)"
[disabled]="isSubmitting"
class="btn-stop-secondary"
title="Stop currently running task and clear queue"
>
Stop All
</button>
<button
(click)="submitTask()"
[disabled]="isSubmitting || !taskInput.trim()"
class="btn-send"
>
{{ isSubmitting ? '...' : 'Run' }}
</button>
</div>
</footer>
-->
</div>
src/app/components/chat-interface/chat-interface.component.scss → apps/showcase_ui/src/app/components/chat-interface/chat-interface.component.scss
+137 -223
View File
@@ -46,36 +46,59 @@ $text-muted: #a1a1aa;
background: #ffffff;
box-sizing: border-box;
.header-left {
.tab-selector-bar {
display: flex;
height: 36px;
box-sizing: border-box;
background: rgba(241, 245, 249, 0.72);
backdrop-filter: blur(16px) saturate(180%);
-webkit-backdrop-filter: blur(16px) saturate(180%);
border: 1px solid rgba(226, 232, 240, 0.85);
border-radius: 18px;
padding: 3px;
gap: 3px;
align-items: center;
gap: 12px;
flex: 1 1 0;
min-width: 0;
flex-shrink: 0;
box-shadow: 0 2px 8px -1px rgba(15, 23, 42, 0.04), inset 0 1px 1px rgba(255, 255, 255, 0.8);
.avatar {
width: 32px;
height: 32px;
background: #f3f4f6;
border-radius: 50%;
display: flex;
.tab-selector-btn {
display: inline-flex;
align-items: center;
justify-content: center;
box-shadow: 0 2px 4px rgba(0,0,0,0.03);
.material-symbols-outlined {
font-size: 18px;
color: #4b5563;
}
}
.chat-title {
h3 {
margin: 0;
font-size: 16px;
font-weight: 700;
color: $text-primary;
gap: 6px;
height: 30px;
padding: 0 14px;
box-sizing: border-box;
border: none;
background: transparent;
border-radius: 15px;
font-size: 13px;
font-weight: 600;
color: #64748b;
cursor: pointer;
line-height: 1;
transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1);
white-space: nowrap;
.tab-icon {
font-size: 16px;
width: 16px;
height: 16px;
display: inline-flex;
align-items: center;
justify-content: center;
}
&:hover:not(.active) {
color: #0f172a;
background: rgba(255, 255, 255, 0.7);
}
&.active {
background: #ffffff;
color: #2563eb;
font-weight: 700;
box-shadow: 0 2px 6px rgba(0, 0, 0, 0.06), 0 1px 2px rgba(0, 0, 0, 0.02);
}
}
}
@@ -89,20 +112,22 @@ $text-muted: #a1a1aa;
flex-shrink: 0;
.btn-resume-task {
padding: 6px 12px;
background: #2563eb;
border: 1px solid #2563eb;
padding: 6px 14px;
background: linear-gradient(135deg, rgba(37, 99, 235, 0.92), rgba(26, 115, 232, 0.85));
backdrop-filter: blur(14px) saturate(180%);
-webkit-backdrop-filter: blur(14px) saturate(180%);
border: 1px solid rgba(255, 255, 255, 0.35);
border-radius: 16px;
font-size: 14px;
font-size: 13.5px;
font-weight: 600;
color: #ffffff;
cursor: pointer;
display: flex;
align-items: center;
gap: 4px;
transition: all 0.2s;
gap: 5px;
transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1);
margin-right: 8px;
box-shadow: 0 2px 4px rgba(37, 99, 235, 0.2);
box-shadow: 0 4px 14px rgba(37, 99, 235, 0.3), inset 0 1px 1px rgba(255, 255, 255, 0.45);
.material-symbols-outlined {
font-size: 16px;
@@ -110,9 +135,14 @@ $text-muted: #a1a1aa;
}
&:hover {
background: #1d4ed8;
border-color: #1d4ed8;
box-shadow: 0 4px 8px rgba(37, 99, 235, 0.3);
background: linear-gradient(135deg, rgba(37, 99, 235, 1), rgba(26, 115, 232, 0.95));
border-color: rgba(255, 255, 255, 0.6);
box-shadow: 0 6px 18px rgba(37, 99, 235, 0.42), inset 0 1px 1px rgba(255, 255, 255, 0.6);
transform: translateY(-0.5px);
}
&:active {
transform: scale(0.97);
}
}
@@ -120,18 +150,22 @@ $text-muted: #a1a1aa;
width: 32px;
height: 32px;
padding: 0;
background: transparent;
border: 1px solid #cbd5e1;
box-sizing: border-box;
background: rgba(255, 255, 255, 0.72);
backdrop-filter: blur(16px) saturate(180%);
-webkit-backdrop-filter: blur(16px) saturate(180%);
border: 1px solid rgba(203, 213, 225, 0.85);
border-radius: 50%;
color: #64748b;
cursor: pointer;
display: flex;
align-items: center;
justify-content: center;
transition: all 0.2s ease;
box-shadow: 0 2px 6px -1px rgba(15, 23, 42, 0.04), inset 0 1px 1px rgba(255, 255, 255, 0.9);
transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1);
.material-symbols-outlined {
font-size: 18px;
font-size: 16px;
color: inherit;
}
@@ -139,6 +173,7 @@ $text-muted: #a1a1aa;
background: #fef2f2;
color: #ef4444;
border-color: #fca5a5;
box-shadow: 0 3px 10px rgba(239, 68, 68, 0.15);
transform: scale(1.05);
}
@@ -153,7 +188,7 @@ $text-muted: #a1a1aa;
.chat-messages {
flex: 1;
overflow-y: auto;
padding: 12px;
padding: 12px 12px 100px 12px;
display: flex;
flex-direction: column;
gap: 8px;
@@ -259,6 +294,17 @@ $text-muted: #a1a1aa;
transform: translateY(-1px);
border-color: rgba(203, 213, 225, 1);
box-shadow: 0 3px 8px rgba(148, 163, 184, 0.08);
.task-card-header .task-card-header-actions {
&:has(.btn-delete-card) .task-time {
opacity: 0;
}
.btn-delete-card {
opacity: 1;
pointer-events: auto;
}
}
}
&.selected {
@@ -336,15 +382,66 @@ $text-muted: #a1a1aa;
background-color: #ef4444;
}
}
&.cancelled {
background: #fafafa;
color: #d97706;
border-color: #fef3c7;
&::before {
background-color: #f59e0b;
}
}
.status-dot {
display: none;
}
}
.task-card-header-actions {
display: flex;
align-items: center;
position: relative;
.task-time {
font-size: 11.5px;
color: #94a3b8;
transition: opacity 0.15s ease;
}
.btn-delete-card {
position: absolute;
right: 0;
top: 50%;
transform: translateY(-50%);
width: 20px;
height: 20px;
padding: 0;
border: none;
background: transparent;
border-radius: 4px;
color: #94a3b8;
cursor: pointer;
display: inline-flex;
align-items: center;
justify-content: center;
opacity: 0;
pointer-events: none;
transition: all 0.15s ease;
.material-symbols-outlined {
font-size: 14px;
line-height: 1;
}
&:hover:not(:disabled) {
color: #ef4444;
background: #fee2e2;
}
&:disabled {
opacity: 0.3;
cursor: not-allowed;
}
}
}
}
@@ -396,192 +493,8 @@ $text-muted: #a1a1aa;
}
}
.chat-input-panel {
display: flex;
flex-direction: column;
gap: 8px;
padding: 16px;
border-top: 1px solid $border-color;
background: #ffffff;
.error-banner {
background: #fef2f2;
border: 1px solid #fca5a5;
color: #991b1b;
font-size: 12px;
padding: 8px 12px;
border-radius: 8px;
line-height: 1.4;
animation: bannerSlideIn 0.2s ease-out;
display: flex;
align-items: center;
gap: 6px;
.error-icon {
font-size: 16px;
color: #991b1b;
}
}
.input-row {
display: flex;
align-items: center;
gap: 10px;
width: 100%;
.chat-input {
flex: 1;
border: 1px solid #e2e8f0;
border-radius: 12px;
padding: 10px 14px;
font-size: 13.5px;
line-height: 1.4;
outline: none;
resize: none;
box-sizing: border-box;
max-height: 80px;
font-family: inherit;
color: #0f172a;
transition: all 0.2s;
&:focus {
border-color: #3b82f6;
box-shadow: 0 0 0 2px rgba(59, 130, 246, 0.1);
}
&:disabled {
background: #f8fafc;
color: #94a3b8;
cursor: not-allowed;
}
}
.btn-stop {
padding: 10px 16px;
background: #ef4444;
border: none;
border-radius: 12px;
font-size: 17px;
font-weight: 600;
color: #ffffff;
cursor: pointer;
transition: background 0.2s, transform 0.1s;
min-width: 65px;
&:hover:not(:disabled) {
background: #dc2626;
}
&:active:not(:disabled) {
transform: scale(0.97);
}
&:disabled {
background: #fca5a5;
color: #ffffff;
cursor: not-allowed;
}
}
.btn-stop-secondary {
padding: 10px 16px;
background: transparent;
border: 1.5px solid #ef4444;
border-radius: 12px;
font-size: 17px;
font-weight: 600;
color: #ef4444;
cursor: pointer;
transition: background 0.2s, transform 0.1s;
min-width: 65px;
&:hover:not(:disabled) {
background: #fef2f2;
}
&:active:not(:disabled) {
transform: scale(0.97);
}
&:disabled {
border-color: #fca5a5;
color: #fca5a5;
cursor: not-allowed;
}
}
.btn-send {
padding: 10px 16px;
background: #0f172a;
border: none;
border-radius: 12px;
font-size: 17px;
font-weight: 600;
color: #ffffff;
cursor: pointer;
transition: background 0.2s, transform 0.1s;
min-width: 60px;
&:hover:not(:disabled) {
background: #1e293b;
}
&:active:not(:disabled) {
transform: scale(0.97);
}
&:disabled {
background: #e2e8f0;
color: #94a3b8;
cursor: not-allowed;
box-shadow: none;
}
}
}
}
// Tab selector bar inline capsule format inside header
.tab-selector-bar {
display: flex;
background: #f1f5f9;
border-radius: 20px;
padding: 3px;
gap: 2px;
align-items: center;
box-sizing: border-box;
flex-shrink: 0;
.tab-selector-btn {
display: inline-flex;
align-items: center;
justify-content: center;
gap: 6px;
padding: 6px 14px;
border: none;
background: transparent;
border-radius: 16px;
font-size: 13.5px;
font-weight: 600;
color: #64748b;
cursor: pointer;
transition: all 0.2s ease;
white-space: nowrap;
.tab-icon {
font-size: 16px;
}
&:hover {
color: #334155;
}
&.active {
background: #ffffff;
color: #2563eb;
box-shadow: 0 1px 3px rgba(0, 0, 0, 0.08);
}
}
}
// Notes tab content
.notes-tab-content {
@@ -795,6 +708,7 @@ $text-muted: #a1a1aa;
display: flex;
flex-direction: column;
gap: 10px;
padding-left: 28px;
.substep-row {
display: flex;
src/app/components/chat-interface/chat-interface.component.ts → apps/showcase_ui/src/app/components/chat-interface/chat-interface.component.ts
+41 -26
View File
@@ -58,12 +58,12 @@ export class ChatInterfaceComponent {
});
/**
* Filtered computed list of historical/completed tasks (completed or failed)
* Filtered computed list of historical/completed tasks (completed, failed, or cancelled)
*/
public historyTasks = computed(() => {
return this.agentService.sessions().filter((s) => {
const status = this.getTaskStatus(s);
return status === 'completed' || status === 'failed';
return status === 'completed' || status === 'failed' || status === 'cancelled';
});
});
@@ -79,6 +79,7 @@ export class ChatInterfaceComponent {
this.isSubmitting = true;
this.errorMessage = null;
this.agentService.clearUserPinnedSession();
this.http.post<any>('/api/run', { goal }).subscribe({
next: (res) => {
this.taskInput = '';
@@ -86,7 +87,13 @@ export class ChatInterfaceComponent {
if (res && res.tasks && res.tasks.length > 0) {
const firstTask = res.tasks[0];
if (firstTask && firstTask.session_id) {
this.agentService.selectSession(firstTask.session_id);
// Only auto-select immediately if no task is currently executing/running
const isCurrentlyRunning =
this.agentService.agentStatus() === 'running' ||
this.agentService.sessions().some((s) => s.status === 'running');
if (!isCurrentlyRunning) {
this.agentService.selectSession(firstTask.session_id, false);
}
}
}
// Fetch status to refresh sessions list and select new session
@@ -110,18 +117,10 @@ export class ChatInterfaceComponent {
public stopTasks(stopAll: boolean = false): void {
this.isSubmitting = true;
this.errorMessage = null;
this.http.post<any>(`/api/stop?all=${stopAll}`, {}).subscribe({
next: () => {
this.agentService.stopTask(stopAll);
setTimeout(() => {
this.isSubmitting = false;
// Refresh status and sessions list
this.agentService.fetchStatus();
},
error: (err) => {
console.error('Failed to stop tasks:', err);
this.isSubmitting = false;
this.errorMessage = err.error?.detail || 'Failed to stop tasks.';
}
});
}, 400);
}
/**
@@ -133,16 +132,11 @@ export class ChatInterfaceComponent {
}
this.isSubmitting = true;
this.errorMessage = null;
this.http.post<any>('/api/cleanup', {}).subscribe({
this.agentService.clearAllHistory().subscribe({
next: () => {
this.isSubmitting = false;
// Reset current session selection
this.agentService.selectSession('');
// Refresh status and sessions
this.agentService.fetchStatus();
this.agentService.fetchSessions();
},
error: (err) => {
error: (err: any) => {
console.error('Failed to clear history:', err);
this.isSubmitting = false;
this.errorMessage = err.error?.detail || 'Failed to clear history.';
@@ -150,20 +144,41 @@ export class ChatInterfaceComponent {
});
}
/**
* Delete an individual task / session
*/
public deleteTask(sessionId: string, event: MouseEvent): void {
event.stopPropagation();
if (!confirm(`Are you sure you want to delete this task? This cannot be undone.`)) {
return;
}
this.isSubmitting = true;
this.errorMessage = null;
this.agentService.deleteSession(sessionId).subscribe({
next: () => {
this.isSubmitting = false;
},
error: (err: any) => {
console.error(`Failed to delete task ${sessionId}:`, err);
this.isSubmitting = false;
this.errorMessage = err.error?.detail || 'Failed to delete task.';
}
});
}
/**
* Determine the current task execution status
*/
public getTaskStatus(session: Session): 'running' | 'completed' | 'pending' | 'failed' {
if (session.session_id === this.agentService.runningSessionId()) {
public getTaskStatus(session: Session): 'running' | 'completed' | 'pending' | 'failed' | 'cancelled' {
if (session.session_id === this.agentService.runningSessionId() && this.agentService.agentStatus() === 'running') {
return 'running';
}
if (session.status) {
const s = session.status.toLowerCase();
if (s === 'running' || s === 'completed' || s === 'pending' || s === 'failed') {
if (s === 'running' || s === 'completed' || s === 'pending' || s === 'failed' || s === 'cancelled') {
return s as any;
}
}
// Fallback: If it's a historical session and not currently running, it's completed
return 'completed';
}
@@ -171,7 +186,7 @@ export class ChatInterfaceComponent {
* Select a session in the UI to monitor its steps
*/
public selectTask(sessionId: string): void {
this.agentService.selectSession(sessionId);
this.agentService.selectSession(sessionId, true);
}
// Notes Computed Properties
src/app/components/floating-video-player/floating-video-player.component.html → apps/showcase_ui/src/app/components/floating-video-player/floating-video-player.component.html
View File
File renamed without changes.
src/app/components/floating-video-player/floating-video-player.component.scss → apps/showcase_ui/src/app/components/floating-video-player/floating-video-player.component.scss
+79 -44
View File
@@ -31,10 +31,12 @@
position: fixed;
width: 360px;
max-width: calc(100vw - 32px);
background: #ffffff;
border: 1px solid #cbd5e1;
border-radius: 12px;
box-shadow: 0 16px 40px -8px rgba(15, 23, 42, 0.22), 0 0 0 1px rgba(0, 0, 0, 0.05);
background: rgba(255, 255, 255, 0.84);
backdrop-filter: blur(24px) saturate(180%);
-webkit-backdrop-filter: blur(24px) saturate(180%);
border: 1px solid rgba(255, 255, 255, 0.85);
border-radius: 16px;
box-shadow: 0 20px 50px -10px rgba(15, 23, 42, 0.22), 0 0 0 1px rgba(226, 232, 240, 0.6), inset 0 1px 1px rgba(255, 255, 255, 0.9);
z-index: 1050;
overflow: hidden;
display: flex;
@@ -77,8 +79,9 @@
/* --- Window Header (Draggable Handle) --- */
.window-header {
height: 44px;
background: #ffffff;
border-bottom: 1px solid #e2e8f0;
background: rgba(255, 255, 255, 0.75);
backdrop-filter: blur(16px);
border-bottom: 1px solid rgba(226, 232, 240, 0.75);
display: flex;
align-items: center;
justify-content: space-between;
@@ -117,10 +120,11 @@
.mode-switcher-group {
display: flex;
align-items: center;
background: #f1f5f9;
border: 1px solid #cbd5e1;
border-radius: 6px;
padding: 1.5px;
background: rgba(241, 245, 249, 0.65);
backdrop-filter: blur(12px);
border: 1px solid rgba(226, 232, 240, 0.8);
border-radius: 8px;
padding: 2px;
margin-left: 4px;
flex-shrink: 0;
@@ -131,7 +135,7 @@
font-size: 10px;
font-weight: 700;
padding: 2px 7px;
border-radius: 4px;
border-radius: 6px;
cursor: pointer;
transition: all 0.15s ease;
display: flex;
@@ -148,7 +152,7 @@
&:hover:not(:disabled) {
color: #0f172a;
background: #e2e8f0;
background: rgba(255, 255, 255, 0.6);
}
&.active {
@@ -175,9 +179,12 @@
.speed-pill-group {
display: flex;
align-items: center;
background: #e2e8f0;
border: 1px solid #cbd5e1;
border-radius: 6px;
background: rgba(241, 245, 249, 0.65);
backdrop-filter: blur(12px);
border: 1px solid rgba(226, 232, 240, 0.8);
border-radius: 8px;
padding: 2px;
margin-right: 4px;
padding: 2px;
margin-right: 4px;
@@ -207,8 +214,10 @@
}
.header-btn {
background: #f1f5f9;
border: 1px solid #cbd5e1;
background: rgba(241, 245, 249, 0.75);
backdrop-filter: blur(12px);
-webkit-backdrop-filter: blur(12px);
border: 1px solid rgba(226, 232, 240, 0.85);
color: #1e293b;
cursor: pointer;
display: flex;
@@ -216,19 +225,27 @@
justify-content: center;
width: 28px;
height: 28px;
border-radius: 6px;
transition: all 0.15s ease;
border-radius: 7px;
box-shadow: 0 1px 3px rgba(15, 23, 42, 0.04), inset 0 1px 1px rgba(255, 255, 255, 0.8);
transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1);
&:hover {
background: #e2e8f0;
background: rgba(255, 255, 255, 0.95);
color: #0f172a;
border-color: #94a3b8;
border-color: rgba(203, 213, 225, 0.95);
transform: translateY(-0.5px);
box-shadow: 0 2px 6px rgba(15, 23, 42, 0.08);
}
&.close-btn:hover {
background: #fee2e2;
background: rgba(254, 226, 226, 0.95);
color: #dc2626;
border-color: #fca5a5;
border-color: rgba(252, 165, 165, 0.9);
box-shadow: 0 2px 6px rgba(220, 38, 38, 0.15);
}
&:active {
transform: scale(0.96);
}
.material-symbols-outlined {
@@ -375,14 +392,16 @@
}
.video-controls-footer {
background: #ffffff;
border-top: 1px solid #f1f5f9;
background: rgba(255, 255, 255, 0.84);
backdrop-filter: blur(20px) saturate(180%);
-webkit-backdrop-filter: blur(20px) saturate(180%);
border-top: 1px solid rgba(226, 232, 240, 0.75);
padding: 12px 14px;
display: flex;
flex-direction: column;
gap: 10px;
flex-shrink: 0;
box-shadow: 0 -2px 8px rgba(0, 0, 0, 0.04);
box-shadow: 0 -4px 16px rgba(15, 23, 42, 0.04), inset 0 1px 1px rgba(255, 255, 255, 0.9);
.scrubber-bar-container {
position: relative;
@@ -408,7 +427,7 @@
left: 0;
right: 0;
height: 5px;
background: #e2e8f0;
background: rgba(226, 232, 240, 0.85);
border-radius: 3px;
}
@@ -416,10 +435,10 @@
position: absolute;
left: 0;
height: 5px;
background: #0284c7;
background: linear-gradient(90deg, #0284c7, #2563eb);
border-radius: 3px;
pointer-events: none;
box-shadow: 0 0 6px rgba(2, 132, 199, 0.4);
box-shadow: 0 0 8px rgba(2, 132, 199, 0.4);
}
}
@@ -435,43 +454,59 @@
}
.ctrl-btn {
background: #f8fafc;
border: 1px solid #cbd5e1;
background: rgba(241, 245, 249, 0.75);
backdrop-filter: blur(12px);
-webkit-backdrop-filter: blur(12px);
border: 1px solid rgba(226, 232, 240, 0.85);
box-shadow: 0 1px 3px rgba(15, 23, 42, 0.04), inset 0 1px 1px rgba(255, 255, 255, 0.85);
color: #0f172a;
width: 32px;
height: 32px;
border-radius: 6px;
border-radius: 8px;
display: flex;
align-items: center;
justify-content: center;
cursor: pointer;
transition: all 0.15s ease;
transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1);
&:hover {
background: #e2e8f0;
border-color: #94a3b8;
color: #0f172a;
transform: scale(1.05);
background: rgba(255, 255, 255, 0.95);
border-color: rgba(191, 219, 254, 0.9);
color: #0284c7;
transform: translateY(-0.5px);
box-shadow: 0 3px 8px rgba(15, 23, 42, 0.08);
}
&:active {
transform: scale(0.96);
}
&.active {
color: #0284c7;
background: #e0f2fe;
border-color: #38bdf8;
background: rgba(224, 242, 254, 0.85);
border-color: rgba(56, 189, 248, 0.8);
box-shadow: 0 2px 6px rgba(2, 132, 199, 0.15);
}
&.play-btn {
background: #0284c7;
border: 1px solid #0284c7;
background: linear-gradient(135deg, rgba(2, 132, 199, 0.92), rgba(37, 99, 235, 0.88));
backdrop-filter: blur(14px) saturate(180%);
-webkit-backdrop-filter: blur(14px) saturate(180%);
border: 1px solid rgba(255, 255, 255, 0.35);
color: #ffffff;
width: 36px;
height: 36px;
border-radius: 50%;
box-shadow: 0 2px 8px rgba(2, 132, 199, 0.35);
box-shadow: 0 4px 14px rgba(2, 132, 199, 0.35), inset 0 1px 1px rgba(255, 255, 255, 0.5);
&:hover {
background: #0369a1;
box-shadow: 0 3px 12px rgba(2, 132, 199, 0.5);
background: linear-gradient(135deg, rgba(2, 132, 199, 1), rgba(37, 99, 235, 0.95));
box-shadow: 0 6px 18px rgba(2, 132, 199, 0.45), inset 0 1px 1px rgba(255, 255, 255, 0.6);
transform: translateY(-1px) scale(1.04);
}
&:active {
transform: scale(0.96);
}
.material-symbols-outlined {
src/app/components/floating-video-player/floating-video-player.component.ts → apps/showcase_ui/src/app/components/floating-video-player/floating-video-player.component.ts
View File
File renamed without changes.
src/app/components/legacy-agent-stream/legacy-agent-stream.component.html → apps/showcase_ui/src/app/components/legacy-agent-stream/legacy-agent-stream.component.html
View File
File renamed without changes.
src/app/components/legacy-agent-stream/legacy-agent-stream.component.scss → apps/showcase_ui/src/app/components/legacy-agent-stream/legacy-agent-stream.component.scss
View File
File renamed without changes.
src/app/components/legacy-agent-stream/legacy-agent-stream.component.ts → apps/showcase_ui/src/app/components/legacy-agent-stream/legacy-agent-stream.component.ts
+1 -1
View File
@@ -36,7 +36,7 @@ export class LegacyAgentStreamComponent {
const selectElement = event.target as HTMLSelectElement;
const sessionId = selectElement.value;
if (sessionId) {
this.agentService.selectSession(sessionId);
this.agentService.selectSession(sessionId, true);
}
}
apps/showcase_ui/src/app/components/nav-switcher/nav-switcher.component.scss
+110
View File
@@ -0,0 +1,110 @@
/**
* Copyright 2026 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
.floating-nav-switcher {
position: fixed;
top: 12px;
left: 24px;
z-index: 1100;
height: 36px;
box-sizing: border-box;
display: inline-flex;
align-items: center;
background: rgba(255, 255, 255, 0.72);
backdrop-filter: blur(16px) saturate(180%);
-webkit-backdrop-filter: blur(16px) saturate(180%);
border: 1px solid rgba(226, 232, 240, 0.85);
border-radius: 18px;
padding: 3px;
gap: 3px;
box-shadow: 0 2px 8px -1px rgba(15, 23, 42, 0.06), 0 1px 2px rgba(0, 0, 0, 0.02), inset 0 1px 1px rgba(255, 255, 255, 0.9);
user-select: none;
font-family: 'Google Sans', 'Google Sans Text', 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
transition: all 0.25s cubic-bezier(0.4, 0, 0.2, 1);
&:hover {
background: rgba(255, 255, 255, 0.85);
box-shadow: 0 4px 14px -2px rgba(15, 23, 42, 0.1), 0 2px 4px rgba(0, 0, 0, 0.03), inset 0 1px 1px rgba(255, 255, 255, 1);
border-color: rgba(203, 213, 225, 0.9);
}
.nav-tab-btn {
display: inline-flex;
align-items: center;
justify-content: center;
gap: 6px;
height: 30px;
padding: 0 14px;
box-sizing: border-box;
border-radius: 15px;
font-size: 13px;
font-weight: 600;
color: #475569;
text-decoration: none;
cursor: pointer;
line-height: 1;
transition: all 0.2s cubic-bezier(0.4, 0, 0.2, 1);
outline: none;
white-space: nowrap;
.tab-icon {
font-size: 16px;
width: 16px;
height: 16px;
display: inline-flex;
align-items: center;
justify-content: center;
color: #64748b;
transition: color 0.15s ease, transform 0.15s ease;
}
.tab-label {
color: #475569;
transition: color 0.15s ease;
}
&:hover:not(.active) {
background: rgba(255, 255, 255, 0.7);
color: #0f172a;
.tab-icon {
color: #1a73e8;
}
.tab-label {
color: #0f172a;
}
}
&.active {
background: rgba(26, 115, 232, 0.95) !important;
backdrop-filter: blur(12px);
-webkit-backdrop-filter: blur(12px);
color: #ffffff !important;
font-weight: 700;
box-shadow: 0 2px 8px rgba(26, 115, 232, 0.35), inset 0 1px 1px rgba(255, 255, 255, 0.35);
.tab-icon,
.tab-label,
span {
color: #ffffff !important;
}
}
&:active {
transform: scale(0.97);
}
}
}
apps/showcase_ui/src/app/components/nav-switcher/nav-switcher.component.ts
+50
View File
@@ -0,0 +1,50 @@
/**
* Copyright 2026 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { Component } from '@angular/core';
import { CommonModule } from '@angular/common';
import { RouterLink, RouterLinkActive } from '@angular/router';
@Component({
selector: 'app-nav-switcher',
standalone: true,
imports: [CommonModule, RouterLink, RouterLinkActive],
template: `
<nav class="floating-nav-switcher" aria-label="Main Navigation">
<a
routerLink="/"
routerLinkActive="active"
[routerLinkActiveOptions]="{exact: true}"
class="nav-tab-btn"
title="Return to Home Launcher to start a new task"
>
<span class="material-symbols-outlined tab-icon">add_task</span>
<span class="tab-label">New / Home</span>
</a>
<a
routerLink="/workspace"
routerLinkActive="active"
class="nav-tab-btn"
title="Open Workspace"
>
<span class="material-symbols-outlined tab-icon">space_dashboard</span>
<span class="tab-label">Workspace</span>
</a>
</nav>
`,
styleUrls: ['./nav-switcher.component.scss']
})
export class NavSwitcherComponent {}
apps/showcase_ui/src/app/core/data/smart-tasks.data.ts
+334
View File
@@ -0,0 +1,334 @@
/**
* Copyright 2026 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
export interface AppReference {
name: string;
icon: string;
pkg?: string;
category?: string;
}
export type SuggestionCategory =
| 'all'
| 'flash'
| 'pro'
| 'cross_app'
| 'monitor';
export interface SmartSuggestion {
id: string;
title: string;
description: string;
goal: string;
profile: 'flash' | 'pro';
category: 'flash' | 'pro' | 'cross_app' | 'monitor';
tag: string;
apps: AppReference[];
requiredPackages?: string[];
matchMode?: 'any' | 'all';
priority?: number;
}
/**
* Recognized Android App Package Registry
*/
export const APP_REGISTRY: Record<string, AppReference> = {
// Google Suite & System
'com.google.android.apps.maps': { name: 'Maps', icon: 'explore', pkg: 'com.google.android.apps.maps', category: 'navigation' },
'com.google.android.gm': { name: 'Gmail', icon: 'mail', pkg: 'com.google.android.gm', category: 'productivity' },
'com.android.chrome': { name: 'Chrome', icon: 'public', pkg: 'com.android.chrome', category: 'browser' },
'com.google.android.youtube': { name: 'YouTube', icon: 'smart_display', pkg: 'com.google.android.youtube', category: 'entertainment' },
'com.android.settings': { name: 'Settings', icon: 'settings', pkg: 'com.android.settings', category: 'system' },
'com.google.android.deskclock': { name: 'Clock', icon: 'timer', pkg: 'com.google.android.deskclock', category: 'utility' },
'com.android.deskclock': { name: 'Clock', icon: 'timer', pkg: 'com.android.deskclock', category: 'utility' },
'com.google.android.calculator': { name: 'Calculator', icon: 'calculate', pkg: 'com.google.android.calculator', category: 'utility' },
'com.android.calculator2': { name: 'Calculator', icon: 'calculate', pkg: 'com.android.calculator2', category: 'utility' },
'com.google.android.apps.photos': { name: 'Photos', icon: 'photo_library', pkg: 'com.google.android.apps.photos', category: 'media' },
'com.google.android.calendar': { name: 'Calendar', icon: 'calendar_month', pkg: 'com.google.android.calendar', category: 'productivity' },
'com.google.android.keep': { name: 'Keep Notes', icon: 'note_alt', pkg: 'com.google.android.keep', category: 'productivity' },
'com.android.vending': { name: 'Play Store', icon: 'storefront', pkg: 'com.android.vending', category: 'tools' },
'com.google.android.apps.messaging': { name: 'Messages', icon: 'chat', pkg: 'com.google.android.apps.messaging', category: 'communication' },
// Popular Ecosystem Apps
'com.tencent.mm': { name: 'WeChat', icon: 'forum', pkg: 'com.tencent.mm', category: 'social' },
'com.xingin.xhs': { name: 'Xiaohongshu', icon: 'auto_stories', pkg: 'com.xingin.xhs', category: 'social' },
'com.sankuai.meituan': { name: 'Meituan', icon: 'restaurant', pkg: 'com.sankuai.meituan', category: 'lifestyle' },
'com.dianping.v1': { name: 'Dianping', icon: 'star', pkg: 'com.dianping.v1', category: 'lifestyle' },
'tv.danmaku.bili': { name: 'Bilibili', icon: 'video_library', pkg: 'tv.danmaku.bili', category: 'entertainment' },
'com.eg.android.AlipayGphone': { name: 'Alipay', icon: 'account_balance_wallet', pkg: 'com.eg.android.AlipayGphone', category: 'finance' },
'com.netease.cloudmusic': { name: 'NetEase Music', icon: 'headphones', pkg: 'com.netease.cloudmusic', category: 'entertainment' },
'com.spotify.music': { name: 'Spotify', icon: 'music_note', pkg: 'com.spotify.music', category: 'entertainment' }
};
/**
* Curated, clean task preset library (1~2 representative tasks per common app)
*/
export const SMART_TASK_LIBRARY: SmartSuggestion[] = [
// 1. Google Maps
{
id: 'maps_coffee',
title: 'Find Specialty Coffee',
description: 'Search nearby top-rated cafes in Google Maps',
goal: 'Open Google Maps, search for top-rated specialty coffee shops nearby, and view the top result details.',
profile: 'flash',
category: 'flash',
tag: 'Maps',
apps: [{ name: 'Maps', icon: 'explore', pkg: 'com.google.android.apps.maps' }],
requiredPackages: ['com.google.android.apps.maps'],
priority: 95
},
{
id: 'pro_commute_share',
title: 'Commute ETA & Message Draft',
description: 'Check transit time on Maps and draft arrival ETA in Messages',
goal: 'Open Google Maps to check commute time to the International Airport, calculate arrival time, then open Messages and draft an ETA text message.',
profile: 'pro',
category: 'cross_app',
tag: 'Maps + Messages',
apps: [
{ name: 'Maps', icon: 'explore', pkg: 'com.google.android.apps.maps' },
{ name: 'Messages', icon: 'chat', pkg: 'com.google.android.apps.messaging' }
],
requiredPackages: ['com.google.android.apps.maps', 'com.google.android.apps.messaging'],
matchMode: 'all',
priority: 92
},
// 2. Gmail
{
id: 'gmail_receipts',
title: 'Search Order Receipts',
description: 'Find recent flight or delivery confirmation emails in Gmail',
goal: 'Open Gmail and search for recent flight or package delivery confirmation emails.',
profile: 'flash',
category: 'flash',
tag: 'Gmail',
apps: [{ name: 'Gmail', icon: 'mail', pkg: 'com.google.android.gm' }],
requiredPackages: ['com.google.android.gm'],
priority: 90
},
{
id: 'pro_email_to_calendar',
title: 'Email Itinerary to Calendar',
description: 'Extract flight or event dates from Gmail and schedule in Calendar',
goal: 'Open Gmail to find the latest event invitation or itinerary, extract dates and location, then open Google Calendar and create a corresponding calendar event.',
profile: 'pro',
category: 'cross_app',
tag: 'Gmail + Calendar',
apps: [
{ name: 'Gmail', icon: 'mail', pkg: 'com.google.android.gm' },
{ name: 'Calendar', icon: 'calendar_month', pkg: 'com.google.android.calendar' }
],
requiredPackages: ['com.google.android.gm'],
priority: 94
},
// 3. Chrome
{
id: 'chrome_research',
title: 'Search AI News Breakthroughs',
description: 'Search latest multimodal AI developments in Chrome browser',
goal: 'Open Chrome browser and search for latest breakthroughs in multimodal mobile AI agents.',
profile: 'flash',
category: 'flash',
tag: 'Chrome',
apps: [{ name: 'Chrome', icon: 'public', pkg: 'com.android.chrome' }],
requiredPackages: ['com.android.chrome'],
priority: 88
},
{
id: 'pro_research_keep',
title: 'Product Research & Notes Note',
description: 'Compare top 3 headphones on Chrome and record comparison in Keep',
goal: 'Open Chrome, research top 3 noise-cancelling headphones comparing price and battery life, then write a structured comparison summary note in Keep Notes.',
profile: 'pro',
category: 'pro',
tag: 'Chrome + Keep',
apps: [
{ name: 'Chrome', icon: 'public', pkg: 'com.android.chrome' },
{ name: 'Keep Notes', icon: 'note_alt', pkg: 'com.google.android.keep' }
],
requiredPackages: ['com.android.chrome'],
priority: 91
},
// 4. YouTube
{
id: 'youtube_lofi',
title: 'Play Lo-Fi Music Radio',
description: 'Search and play a Lo-Fi hip hop live stream on YouTube',
goal: 'Open YouTube, search for "Lofi hip hop beats relaxing radio" and tap on the live stream.',
profile: 'flash',
category: 'flash',
tag: 'YouTube',
apps: [{ name: 'YouTube', icon: 'smart_display', pkg: 'com.google.android.youtube' }],
requiredPackages: ['com.google.android.youtube'],
priority: 85
},
// 5. Settings
{
id: 'settings_display_wifi',
title: 'Dark Mode & Wi-Fi Check',
description: 'Toggle dark theme and verify network connection in Settings',
goal: 'Open Settings app, navigate to Display settings, ensure Dark theme is enabled, and check Wi-Fi connection status.',
profile: 'flash',
category: 'flash',
tag: 'Settings',
apps: [{ name: 'Settings', icon: 'settings', pkg: 'com.android.settings' }],
requiredPackages: ['com.android.settings'],
priority: 87
},
{
id: 'pro_settings_qa',
title: 'Subsystem Health & Crash Probe',
description: 'Traverse Settings submenus to verify screens and check for crash dialogs',
goal: 'Explore Settings submenus (Network, Connected devices, Apps, Battery, Storage), verify each screen loads properly without ANR or crash dialogs, and summarize results.',
profile: 'pro',
category: 'monitor',
tag: 'Settings QA',
apps: [{ name: 'Settings', icon: 'settings', pkg: 'com.android.settings' }],
requiredPackages: ['com.android.settings'],
priority: 93
},
// 6. Clock
{
id: 'clock_timer',
title: '25-Min Pomodoro Timer',
description: 'Start a 25-minute focus countdown timer in Clock app',
goal: 'Open Clock app, switch to Timer tab, set 25 minutes and start the countdown timer.',
profile: 'flash',
category: 'flash',
tag: 'Clock',
apps: [{ name: 'Clock', icon: 'timer', pkg: 'com.google.android.deskclock' }],
requiredPackages: ['com.google.android.deskclock', 'com.android.deskclock'],
priority: 86
},
// 7. Calculator
{
id: 'calc_gratuity',
title: 'Split Bill & Calculate Tip',
description: 'Calculate 18% gratuity on $186.40 for 3 people in Calculator',
goal: 'Open Calculator and calculate 18% tip on a bill of $186.40, then divide by 3 people.',
profile: 'flash',
category: 'flash',
tag: 'Calculator',
apps: [{ name: 'Calculator', icon: 'calculate', pkg: 'com.google.android.calculator' }],
requiredPackages: ['com.google.android.calculator', 'com.android.calculator2'],
priority: 84
},
// 8. Photos
{
id: 'photos_inspect',
title: 'Inspect Recent Screenshot',
description: 'Open Google Photos and review the latest screenshot taken',
goal: 'Open Google Photos and view the most recent screenshot in the screenshots album.',
profile: 'flash',
category: 'flash',
tag: 'Photos',
apps: [{ name: 'Photos', icon: 'photo_library', pkg: 'com.google.android.apps.photos' }],
requiredPackages: ['com.google.android.apps.photos'],
priority: 82
},
// 9. WeChat (微信)
{
id: 'wechat_browse',
title: 'Check WeChat Messages',
description: 'Open WeChat and view top recent chat conversations',
goal: 'Open WeChat (微信) and view the top recent chat messages.',
profile: 'flash',
category: 'flash',
tag: 'WeChat',
apps: [{ name: 'WeChat', icon: 'forum', pkg: 'com.tencent.mm' }],
requiredPackages: ['com.tencent.mm'],
priority: 89
},
{
id: 'pro_wechat_to_calendar',
title: 'WeChat Notice to Calendar',
description: 'Extract meeting notice from WeChat chat and add to Calendar',
goal: 'Open WeChat (微信), locate the latest meeting announcement or event message in the top chat, extract the time and topic, then open Calendar and schedule an event.',
profile: 'pro',
category: 'cross_app',
tag: 'WeChat + Calendar',
apps: [
{ name: 'WeChat', icon: 'forum', pkg: 'com.tencent.mm' },
{ name: 'Calendar', icon: 'calendar_month', pkg: 'com.google.android.calendar' }
],
requiredPackages: ['com.tencent.mm'],
priority: 93
},
// 10. Xiaohongshu (小红书)
{
id: 'xhs_coffee_guide',
title: 'RED Cafe Guide Search',
description: 'Search trending specialty cafe reviews on Xiaohongshu',
goal: 'Open Xiaohongshu (小红书), search for top-rated specialty coffee shops, and view the top post.',
profile: 'flash',
category: 'flash',
tag: 'Xiaohongshu',
apps: [{ name: 'Xiaohongshu', icon: 'auto_stories', pkg: 'com.xingin.xhs' }],
requiredPackages: ['com.xingin.xhs'],
priority: 87
},
// 11. Meituan / Dianping (美团 / 大众点评)
{
id: 'meituan_ramen_search',
title: 'Meituan Food Search',
description: 'Search top-rated Ramen nearby on Meituan or Dianping',
goal: 'Open Meituan (美团) or Dianping (大众点评), search for top-rated Ramen nearby, and view top restaurant rating.',
profile: 'flash',
category: 'flash',
tag: 'Meituan',
apps: [{ name: 'Meituan', icon: 'restaurant', pkg: 'com.sankuai.meituan' }],
requiredPackages: ['com.sankuai.meituan', 'com.dianping.v1'],
priority: 86
},
// 12. Bilibili (哔哩哔哩)
{
id: 'bilibili_stream',
title: 'Bilibili Tech Video',
description: 'Search and play an AI Agent tutorial video on Bilibili',
goal: 'Open Bilibili (哔哩哔哩), search for "AI Agent 架构实战", and play the top matching video.',
profile: 'flash',
category: 'flash',
tag: 'Bilibili',
apps: [{ name: 'Bilibili', icon: 'video_library', pkg: 'tv.danmaku.bili' }],
requiredPackages: ['tv.danmaku.bili'],
priority: 85
},
// 13. Play Store
{
id: 'pro_playstore_review',
title: 'Play Store App Review Study',
description: 'Compare top task management apps and ratings on Google Play',
goal: 'Open Google Play Store, search for top rated task management apps, compare ratings and latest user reviews of the top 2 candidates, and record recommendations.',
profile: 'pro',
category: 'pro',
tag: 'Play Store',
apps: [{ name: 'Play Store', icon: 'storefront', pkg: 'com.android.vending' }],
requiredPackages: ['com.android.vending'],
priority: 88
}
];
src/app/core/models/markdown.model.ts → apps/showcase_ui/src/app/core/models/markdown.model.ts
View File
File renamed without changes.
src/app/core/models/session.model.ts → apps/showcase_ui/src/app/core/models/session.model.ts
+1 -1
View File
@@ -51,5 +51,5 @@ export interface AgentStatusResponse {
export type AgentStatus = 'idle' | 'running' | 'completed' | 'offline' | string;
export type TaskStatus = 'running' | 'completed' | 'pending' | 'failed';
export type TaskStatus = 'running' | 'completed' | 'pending' | 'failed' | 'cancelled';
src/app/core/models/stream.model.ts → apps/showcase_ui/src/app/core/models/stream.model.ts
+19
View File
@@ -19,6 +19,25 @@ export interface ActionParam {
value: string;
}
export interface ActionExecution {
id?: string;
trace_id?: string;
agent?: string;
agent_name?: string;
action: string;
name?: string;
args?: Record<string, any>;
params?: Record<string, any>;
payload?: any;
pre_image_name?: string | null;
post_image_name?: string | null;
status?: 'success' | 'failed' | 'error' | 'running' | string;
duration?: number;
duration_ms?: number;
error?: string | null;
timestamp?: number;
}
export interface StepEvent {
type: 'action' | 'tool';
data: any;
apps/showcase_ui/src/app/core/models/system.model.ts
+81
View File
@@ -0,0 +1,81 @@
/**
* Copyright 2026 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
export type ProbeStatus = 'pass' | 'warn' | 'fail' | 'skipped';
export type ProbeCategory = 'device' | 'auth' | 'toolchain' | 'runtime';
export interface ProbeAction {
action_type: 'command' | 'hint' | 'link';
label: string;
payload: string;
}
export interface DeviceInfo {
serial: string;
state: string;
model: string | null;
product: string | null;
android_version: string | null;
screen_resolution: string | null;
is_emulator: boolean;
installed_packages?: string[];
}
export interface ProbeResult {
id: string;
category: ProbeCategory;
title: string;
status: ProbeStatus;
is_blocker: boolean;
summary: string;
description: string;
metadata: Record<string, any>;
actions: ProbeAction[];
}
export interface SystemReadinessReport {
overall_ready: boolean;
blocker_count: number;
passed_blocker_count: number;
probes: ProbeResult[];
active_device: DeviceInfo | null;
os_type?: 'linux' | 'darwin' | 'windows' | string;
timestamp: number;
}
export type EmulatorLaunchStage =
| 'idle'
| 'starting'
| 'waiting_for_adb'
| 'booting'
| 'ready'
| 'failed'
| 'stopped';
export interface EmulatorLaunchState {
avd_name: string | null;
status: EmulatorLaunchStage;
pid: number | null;
serial: string | null;
stage_message: string;
progress_percent: number;
started_at: number | null;
elapsed_seconds: number;
error: string | null;
logs: string[];
can_retry: boolean;
}
apps/showcase_ui/src/app/core/services/task-recommendation.service.ts
+111
View File
@@ -0,0 +1,111 @@
/**
* Copyright 2026 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { Injectable } from '@angular/core';
import {
AppReference,
SmartSuggestion,
SuggestionCategory,
APP_REGISTRY,
SMART_TASK_LIBRARY
} from '../data/smart-tasks.data';
@Injectable({
providedIn: 'root'
})
export class TaskRecommendationService {
public readonly allTasks: SmartSuggestion[] = SMART_TASK_LIBRARY;
public readonly appRegistry = APP_REGISTRY;
/**
* Check if task matches any package installed on device
*/
public isSuggestionOnDevice(
suggestion: SmartSuggestion,
installedPackages: Set<string>
): boolean {
if (installedPackages.size === 0) return false;
if (!suggestion.requiredPackages || suggestion.requiredPackages.length === 0) {
return true;
}
if (suggestion.matchMode === 'all') {
return suggestion.requiredPackages.every(pkg => installedPackages.has(pkg));
}
return suggestion.requiredPackages.some(pkg => installedPackages.has(pkg));
}
/**
* Intelligently rank and filter tasks:
* Prioritizes tasks matching apps detected on user's phone without exposing clutter.
*/
public filterAndRankTasks(
installedPackages: Set<string>,
category: SuggestionCategory,
shuffleOffset: number = 0
): SmartSuggestion[] {
const hasDevice = installedPackages.size > 0;
// 1. Score tasks: prioritize tasks whose apps are on the user's phone
const scored = this.allTasks.map(task => {
const isMatched = this.isSuggestionOnDevice(task, installedPackages);
let score = task.priority ?? 50;
if (hasDevice) {
if (isMatched) {
score += 100;
if (task.requiredPackages && task.requiredPackages.length > 1 && task.matchMode === 'all') {
score += 30; // bonus for full multi-app synergy match
}
} else {
score -= 40;
}
}
return { task, isMatched, score };
});
// 2. Filter by category
let filtered = scored.filter(item => {
if (category === 'flash') {
return item.task.profile === 'flash';
}
if (category === 'pro') {
return item.task.profile === 'pro';
}
if (category === 'cross_app') {
return item.task.category === 'cross_app';
}
if (category === 'monitor') {
return item.task.category === 'monitor';
}
return true; // 'all'
});
// 3. Sort by score descending (matched apps first)
filtered.sort((a, b) => b.score - a.score);
let result = filtered.map(f => f.task);
// 4. Apply shuffle rotation
if (shuffleOffset > 0 && result.length > 0) {
const shift = shuffleOffset % result.length;
result = [...result.slice(shift), ...result.slice(0, shift)];
}
return result;
}
}
apps/showcase_ui/src/app/pages/home/home.component.html
+1751
View File
File diff suppressed because it is too large. Load diff
apps/showcase_ui/src/app/pages/home/home.component.scss
+4357
View File
File diff suppressed because it is too large. Load diff
apps/showcase_ui/src/app/pages/home/home.component.ts
+766
View File
@@ -0,0 +1,766 @@
/**
* Copyright 2026 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { Component, signal, computed, effect, inject, OnInit, OnDestroy } from '@angular/core';
import { CommonModule } from '@angular/common';
import { FormsModule } from '@angular/forms';
import { Router, RouterLink } from '@angular/router';
import { AgentService } from '../../services/agent.service';
import { SystemService } from '../../services/system.service';
import { DeviceInfo, ProbeResult } from '../../core/models/system.model';
import {
AppReference,
SmartSuggestion,
SuggestionCategory
} from '../../core/data/smart-tasks.data';
import { TaskRecommendationService } from '../../core/services/task-recommendation.service';
export type { AppReference, SmartSuggestion, SuggestionCategory };
@Component({
selector: 'app-home',
standalone: true,
imports: [CommonModule, FormsModule],
templateUrl: './home.component.html',
styleUrl: './home.component.scss'
})
export class HomeComponent implements OnInit, OnDestroy {
public agentService = inject(AgentService);
public systemService = inject(SystemService);
public taskRecService = inject(TaskRecommendationService);
private router = inject(Router);
// High-level navigation mode: 'diagnostics' (System Setup Guide) vs 'launcher' (Task Execution)
public activeTab = signal<'diagnostics' | 'launcher'>('launcher');
// Interactive guide sub-tab inside ADB section: 'emulator' | 'usb' | 'wifi'
public activeAdbGuideTab = signal<'emulator' | 'usb' | 'wifi'>('emulator');
public emulatorSetupMode = signal<'studio' | 'cli'>('studio');
// Interactive guide tab for LLM / OCR credentials: 'gemini' | 'ocr'
public modelSetupMode = signal<'gemini' | 'custom'>('gemini');
public showOcrConfig = signal<boolean>(false);
public showFullConfigFile = signal<boolean>(false);
// Model & Environment configuration from backend
public modelConfigEnv = computed(() => this.systemService.modelConfigEnv());
// Google Gemini API Key State
public geminiKeyInput = signal<string>('');
public showGeminiKey = signal<boolean>(false);
public isSavingGeminiKey = signal<boolean>(false);
public isTestingGeminiKey = signal<boolean>(false);
public geminiSaveMessage = signal<string | null>(null);
public geminiSaveError = signal<string | null>(null);
public isGeminiKeyEdited = signal<boolean>(false);
// Vision OCR API Key State
public ocrKeyInput = signal<string>('');
public showOcrKey = signal<boolean>(false);
public isSavingOcrKey = signal<boolean>(false);
public isTestingOcrKey = signal<boolean>(false);
public ocrSaveMessage = signal<string | null>(null);
public ocrSaveError = signal<string | null>(null);
public isOcrKeyEdited = signal<boolean>(false);
// Clipboard copy state tracker for interactive feedback
public copiedId = signal<string | null>(null);
// Diagnostic re-check state
public isRefreshingDiagnostics = signal<boolean>(false);
// Wireless ADB Interactive connection signals
public wifiHost = signal<string>('192.168.1.100');
public wifiPort = signal<string>('5555');
public isConnectingWifi = signal<boolean>(false);
public wifiConnectMessage = signal<string | null>(null);
public wifiConnectError = signal<string | null>(null);
public adbRestartFeedback = signal<string | null>(null);
public wifiCommand = computed(() => {
const h = this.wifiHost().trim() || '<phone-ip>';
const p = this.wifiPort().trim() || '5555';
return `adb connect ${h}:${p}`;
});
// Task execution parameters
public selectedProfile = signal<'flash' | 'pro'>('flash');
public taskGoal = signal<string>('');
public isSubmitting = signal<boolean>(false);
public errorMessage = signal<string | null>(null);
// Pro Mode Outputter & Structured Output Configuration
public expectedOutput = signal<string>('');
public enableOutputter = signal<boolean>(true);
public showOutputterDrawer = signal<boolean>(false);
public toggleOutputterDrawer(): void {
this.showOutputterDrawer.update((v) => !v);
}
public applyOutputPreset(preset: string): void {
if (this.expectedOutput() === preset) {
this.expectedOutput.set('');
} else {
this.expectedOutput.set(preset);
this.showOutputterDrawer.set(true);
}
}
// Smart intent detection for model recommendation
public isIntentSuggestingPro = computed<boolean>(() => {
const text = this.taskGoal().toLowerCase();
if (!text.trim()) return false;
const keywords = [
'monitor', 'polling', 'poll', 'wait until', 'loop', 'keep watching',
'crash', 'logcat', 'troubleshoot', 'diagnose', 'debug', 'investigate',
'compare', 'extract', 'summarize', 'report',
'监控', '轮询', '等待', '一直', '直到', '崩溃', '闪退', '排查', '分析日志', '对比', '总结'
];
return keywords.some(k => text.includes(k));
});
public showIntentSuggestion = computed<boolean>(() => {
return this.isIntentSuggestingPro() && this.selectedProfile() === 'flash';
});
// Computed helper states delegating to SystemService
public isReady = computed(() => this.systemService.isReady());
public isLoading = computed(() => this.systemService.isLoading());
public isRestartingAdb = computed(() => this.systemService.isRestartingAdb());
public launchingAvd = computed(() => this.systemService.launchingAvd());
public emulatorLaunchState = computed(() => this.systemService.emulatorLaunchState());
public isEmulatorLaunching = computed(() => this.systemService.isEmulatorLaunching());
public showLaunchLogs = signal<boolean>(false);
// Probes
public pythonProbe = computed(() => this.systemService.pythonProbe());
public configProbe = computed(() => this.systemService.configProbe());
public adbProbe = computed(() => this.systemService.adbProbe());
public llmProbe = computed(() => this.systemService.llmProbe());
public geminiProbe = computed(() => this.systemService.geminiProbe());
public ocrProbe = computed(() => this.systemService.ocrProbe());
public toolchainProbe = computed(() => this.systemService.toolchainProbe());
// Step-level readiness
public isEnvironmentReady = computed(() => this.systemService.isEnvironmentReady());
public isCredentialsReady = computed(() => this.systemService.isCredentialsReady());
public isSkipCredentialsCheck = computed(() => this.systemService.isSkipCredentialsCheck());
public isDeviceReady = computed(() => this.systemService.isDeviceReady());
// Device information
public activeDevice = computed(() => this.systemService.activeDevice());
public connectedDevices = computed(() => this.systemService.connectedDevices());
public installedAvds = computed(() => this.systemService.installedAvds());
public emulatorPath = computed(() => this.systemService.emulatorPath());
public isEmulatorInPath = computed(() => this.systemService.isEmulatorInPath());
public totalStepCount = computed(() => this.systemService.totalStepCount());
public passedStepCount = computed(() => this.systemService.passedStepCount());
public blockerCount = computed(() => this.systemService.blockerCount());
public passedBlockerCount = computed(() => this.systemService.passedBlockerCount());
// Configured LLM providers from probe metadata
public configuredLlmProviders = computed<any[]>(() => {
const meta = this.llmProbe()?.metadata;
if (meta && Array.isArray(meta['providers'])) {
return meta['providers'];
}
return [];
});
// Multi-OS detection & active OS selection
public selectedOs = signal<'linux' | 'darwin' | 'windows' | null>(null);
public effectiveOs = computed<'linux' | 'darwin' | 'windows'>(() => {
return this.selectedOs() || this.systemService.osType();
});
public oneClickSetupCmd = computed(() => {
const os = this.effectiveOs();
if (os === 'windows') {
return 'powershell -ExecutionPolicy Bypass -File scripts/install_deps.ps1';
}
return 'bash scripts/install_deps.sh';
});
public adbInstallCmd = computed(() => {
const os = this.effectiveOs();
if (os === 'windows') {
return 'winget install Google.PlatformTools';
}
if (os === 'darwin') {
return 'brew install android-platform-tools';
}
return 'sudo apt-get install -y adb';
});
public toolchainInstallCmd = computed(() => {
const os = this.effectiveOs();
if (os === 'windows') {
return 'winget install Gyan.FFmpeg Genymobile.scrcpy';
}
if (os === 'darwin') {
return 'brew install ffmpeg scrcpy';
}
return 'sudo apt-get install -y ffmpeg scrcpy';
});
public emuHypervisorTitle = computed(() => {
const os = this.effectiveOs();
if (os === 'windows') return 'Enable Windows Hypervisor Platform (WHPX)';
if (os === 'darwin') return 'Verify macOS Hypervisor / Install Tools';
return 'Enable KVM Hardware Acceleration (Linux)';
});
public emuHypervisorDesc = computed(() => {
const os = this.effectiveOs();
if (os === 'windows') return 'Enable Windows Hypervisor Platform in PowerShell (Run as Administrator):';
if (os === 'darwin') return 'macOS uses native Hypervisor.framework. Install SDK tools via Homebrew (or Studio):';
return 'Ensure virtualization permissions are granted to your user account:';
});
public emuHypervisorCmd = computed(() => {
const os = this.effectiveOs();
if (os === 'windows') return 'Enable-WindowsOptionalFeature -Online -FeatureName HypervisorPlatform';
if (os === 'darwin') return 'brew install --cask android-commandlinetools';
return 'sudo apt-get install -y qemu-kvm libvirt-daemon-system && sudo adduser $USER kvm';
});
public emuSdkInstallCmd = computed(() => {
const os = this.effectiveOs();
if (os === 'darwin') {
return 'sdkmanager --install "system-images;android-34;google_apis;arm64-v8a" "emulator" "platform-tools"';
}
return 'sdkmanager --install "system-images;android-34;google_apis;x86_64" "emulator" "platform-tools"';
});
public emuCreateAvdCmd = computed(() => {
const os = this.effectiveOs();
if (os === 'darwin') {
return 'avdmanager create avd -n Pixel_8_API_34 -k "system-images;android-34;google_apis;arm64-v8a" --device "pixel_8"';
}
return 'avdmanager create avd -n Pixel_8_API_34 -k "system-images;android-34;google_apis;x86_64" --device "pixel_8"';
});
// Flag indicating whether Google Cloud Vision OCR is configured
public isOcrConfigured = computed<boolean>(() => {
const meta = this.ocrProbe()?.metadata;
return meta?.['configured'] === true;
});
public currentApiKey = computed<string>(() => this.systemService.currentApiKey());
public apiKeysMap = computed<Record<string, string>>(() => this.systemService.apiKeysMap());
public savedGeminiKey = computed<string>(() => {
const keys = this.apiKeysMap();
return keys['google'] || this.currentApiKey() || '';
});
public isGeminiModified = computed<boolean>(() => {
return this.geminiKeyInput().trim() !== this.savedGeminiKey().trim();
});
public savedOcrKey = computed<string>(() => {
const keys = this.apiKeysMap();
return keys['ocr'] || '';
});
public isOcrModified = computed<boolean>(() => {
return this.ocrKeyInput().trim() !== this.savedOcrKey().trim();
});
// Rich Smart Suggestions Library (Device-Aware, Flash vs Pro Tailored)
public readonly allSuggestions = this.taskRecService.allTasks;
// Suggestion category filter & shuffle state
public selectedCategory = signal<SuggestionCategory>('all');
public shuffleOffset = signal<number>(0);
// Set of installed package strings from active device
public installedPackages = computed<Set<string>>(() => {
const pkgs = this.activeDevice()?.installed_packages;
if (pkgs && Array.isArray(pkgs)) {
return new Set(pkgs);
}
return new Set();
});
public filteredSuggestions = computed<SmartSuggestion[]>(() => {
return this.taskRecService.filterAndRankTasks(
this.installedPackages(),
this.selectedCategory(),
this.shuffleOffset()
);
});
private focusListener = () => {
// Silently re-check environment when user returns to the browser tab
this.systemService.fetchReadiness().subscribe();
};
constructor() {
effect(() => {
const keys = this.systemService.apiKeysMap();
const current = this.systemService.currentApiKey();
const googleKey = keys['google'] || current || '';
const ocrKey = keys['ocr'] || '';
if (!this.isGeminiKeyEdited()) {
this.geminiKeyInput.set(googleKey);
}
if (!this.isOcrKeyEdited()) {
this.ocrKeyInput.set(ocrKey);
}
});
}
ngOnInit(): void {
// Initial fetch of system readiness & model configuration
this.systemService.fetchReadiness().subscribe();
this.systemService.fetchModelConfigEnv().subscribe();
window.addEventListener('focus', this.focusListener);
}
ngOnDestroy(): void {
window.removeEventListener('focus', this.focusListener);
}
public setTab(tab: 'diagnostics' | 'launcher'): void {
this.activeTab.set(tab);
if (tab === 'diagnostics') {
this.systemService.fetchReadiness().subscribe();
this.systemService.fetchModelConfigEnv().subscribe();
}
}
public setSelectedOs(os: 'linux' | 'darwin' | 'windows'): void {
this.selectedOs.set(os);
}
public setAdbGuideTab(tab: 'emulator' | 'usb' | 'wifi'): void {
this.activeAdbGuideTab.set(tab);
}
public setEmulatorSetupMode(mode: 'studio' | 'cli'): void {
this.emulatorSetupMode.set(mode);
}
public setModelSetupMode(mode: 'gemini' | 'custom'): void {
this.modelSetupMode.set(mode);
if (mode === 'custom') {
this.systemService.setSkipCredentialsCheck(true);
this.systemService.fetchModelConfigEnv().subscribe();
} else {
this.systemService.setSkipCredentialsCheck(false);
}
}
public toggleFullConfigFile(): void {
this.showFullConfigFile.update(v => !v);
}
public toggleGeminiKeyVisibility(): void {
this.showGeminiKey.update(v => !v);
}
public toggleOcrKeyVisibility(): void {
this.showOcrKey.update(v => !v);
}
public toggleOcrConfig(): void {
this.showOcrConfig.update(v => !v);
}
public onGeminiKeyChange(val: string): void {
this.geminiKeyInput.set(val);
this.isGeminiKeyEdited.set(true);
this.geminiSaveError.set(null);
this.geminiSaveMessage.set(null);
}
public onOcrKeyChange(val: string): void {
this.ocrKeyInput.set(val);
this.isOcrKeyEdited.set(true);
this.ocrSaveError.set(null);
this.ocrSaveMessage.set(null);
}
public saveGeminiKey(): void {
const key = this.geminiKeyInput().trim();
if (!key) return;
this.isSavingGeminiKey.set(true);
this.geminiSaveError.set(null);
this.geminiSaveMessage.set(null);
this.systemService.updateApiKey('google', key, true).subscribe({
next: (res) => {
this.isSavingGeminiKey.set(false);
this.isGeminiKeyEdited.set(false);
this.geminiSaveMessage.set(res?.message || '✓ Gemini API key verified & saved successfully.');
setTimeout(() => this.geminiSaveMessage.set(null), 5000);
},
error: (err) => {
this.isSavingGeminiKey.set(false);
this.geminiSaveError.set(err?.error?.detail || err?.message || 'Failed to update Gemini API key.');
}
});
}
public clearGeminiKey(): void {
this.geminiKeyInput.set('');
this.isGeminiKeyEdited.set(false);
this.geminiSaveError.set(null);
this.geminiSaveMessage.set(null);
if (this.savedGeminiKey().trim()) {
this.isSavingGeminiKey.set(true);
this.systemService.updateApiKey('google', '', true).subscribe({
next: (res) => {
this.isSavingGeminiKey.set(false);
this.geminiSaveMessage.set(res?.message || '✓ Gemini API key cleared.');
setTimeout(() => this.geminiSaveMessage.set(null), 5000);
},
error: (err) => {
this.isSavingGeminiKey.set(false);
this.geminiSaveError.set(err?.error?.detail || err?.message || 'Failed to clear Gemini API key.');
}
});
} else {
this.geminiSaveMessage.set('✓ Gemini API key cleared.');
setTimeout(() => this.geminiSaveMessage.set(null), 3000);
}
}
public saveOcrKey(): void {
const key = this.ocrKeyInput().trim();
if (!key) return;
this.isSavingOcrKey.set(true);
this.ocrSaveError.set(null);
this.ocrSaveMessage.set(null);
this.systemService.updateApiKey('ocr', key, true).subscribe({
next: (res) => {
this.isSavingOcrKey.set(false);
this.isOcrKeyEdited.set(false);
this.ocrSaveMessage.set(res?.message || '✓ Vision OCR API key verified & saved.');
setTimeout(() => this.ocrSaveMessage.set(null), 5000);
},
error: (err) => {
this.isSavingOcrKey.set(false);
this.ocrSaveError.set(err?.error?.detail || err?.message || 'Failed to update Vision OCR key.');
}
});
}
public clearOcrKey(): void {
this.ocrKeyInput.set('');
this.isOcrKeyEdited.set(false);
this.ocrSaveError.set(null);
this.ocrSaveMessage.set(null);
if (this.savedOcrKey().trim()) {
this.isSavingOcrKey.set(true);
this.systemService.updateApiKey('ocr', '', true).subscribe({
next: (res) => {
this.isSavingOcrKey.set(false);
this.ocrSaveMessage.set(res?.message || '✓ Vision OCR API key cleared.');
setTimeout(() => this.ocrSaveMessage.set(null), 5000);
},
error: (err) => {
this.isSavingOcrKey.set(false);
this.ocrSaveError.set(err?.error?.detail || err?.message || 'Failed to clear Vision OCR key.');
}
});
} else {
this.ocrSaveMessage.set('✓ Vision OCR API key cleared.');
setTimeout(() => this.ocrSaveMessage.set(null), 3000);
}
}
public testGeminiKey(): void {
const key = this.geminiKeyInput().trim();
if (!key) return;
this.isTestingGeminiKey.set(true);
this.geminiSaveError.set(null);
this.geminiSaveMessage.set(null);
this.systemService.testApiKey('google', key).subscribe({
next: (res) => {
this.isTestingGeminiKey.set(false);
if (res?.valid) {
this.geminiSaveMessage.set(res?.message || '✓ Gemini API key is valid!');
} else {
this.geminiSaveError.set(res?.message || 'Gemini API key verification failed.');
}
setTimeout(() => this.geminiSaveMessage.set(null), 5000);
},
error: (err) => {
this.isTestingGeminiKey.set(false);
this.geminiSaveError.set(err?.error?.detail || err?.message || 'Gemini API key test failed.');
}
});
}
public testOcrKey(): void {
const key = this.ocrKeyInput().trim();
if (!key) return;
this.isTestingOcrKey.set(true);
this.ocrSaveError.set(null);
this.ocrSaveMessage.set(null);
this.systemService.testApiKey('ocr', key).subscribe({
next: (res) => {
this.isTestingOcrKey.set(false);
if (res?.valid) {
this.ocrSaveMessage.set(res?.message || '✓ Vision OCR API key is valid!');
} else {
this.ocrSaveError.set(res?.message || 'Vision OCR API key verification failed.');
}
setTimeout(() => this.ocrSaveMessage.set(null), 5000);
},
error: (err) => {
this.isTestingOcrKey.set(false);
this.ocrSaveError.set(err?.error?.detail || err?.message || 'Vision OCR API key test failed.');
}
});
}
public getProviderDisplayName(tab: string): string {
switch (tab) {
case 'gemini': return 'Gemini';
case 'ocr': return 'Vision OCR';
default: return tab;
}
}
public getProviderEnvVar(tab: string): string {
switch (tab) {
case 'gemini': return 'GEMINI_API_KEY';
case 'ocr': return 'GOOGLE_VISION_API_KEY';
default: return 'API_KEY';
}
}
public getProviderHint(tab: string): string {
switch (tab) {
case 'gemini':
return 'For a quick start, Google Gemini provides a free API key. Artemis also supports other models (OpenAI, Claude, OpenRouter, etc.)—you can configure your own API keys directly in .env or your environment.';
case 'ocr':
return 'Google Cloud Vision API key for on-screen OCR text detection and UI grounding.';
default:
return 'Configure your API key or use environment definitions.';
}
}
public skipCredentialsCheck(): void {
this.systemService.skipCredentialsCheck();
}
public getApiKeyPlaceholder(tab: string): string {
switch (tab) {
case 'gemini': return 'Enter Gemini API Key (e.g. AIzaSy...)';
case 'ocr': return 'Enter Vision OCR API Key (e.g. AIzaSy...)';
default: return 'Enter API Key...';
}
}
public isTabProviderActive(tab: string): boolean {
if (tab === 'ocr') {
return this.isOcrConfigured();
}
const targetProvider = tab === 'gemini' ? 'google' : tab;
const providers = this.configuredLlmProviders();
return providers.some(p => p.provider === targetProvider);
}
public refreshReadiness(): void {
this.isRefreshingDiagnostics.set(true);
this.systemService.fetchReadiness().subscribe({
next: () => {
this.systemService.fetchModelConfigEnv().subscribe({
next: () => {
setTimeout(() => this.isRefreshingDiagnostics.set(false), 450);
},
error: () => this.isRefreshingDiagnostics.set(false)
});
},
error: () => this.isRefreshingDiagnostics.set(false)
});
}
public restartAdbServer(): void {
this.adbRestartFeedback.set(null);
this.systemService.restartAdb().subscribe({
next: (res) => {
this.adbRestartFeedback.set('ADB Refreshed ✓');
setTimeout(() => this.adbRestartFeedback.set(null), 2500);
},
error: () => {
this.adbRestartFeedback.set('Restart Failed');
setTimeout(() => this.adbRestartFeedback.set(null), 3000);
}
});
}
public connectWifiDevice(): void {
const host = this.wifiHost().trim();
const portStr = this.wifiPort().trim() || '5555';
const port = parseInt(portStr, 10) || 5555;
if (!host) {
this.wifiConnectError.set('Please enter a valid IP address.');
return;
}
this.isConnectingWifi.set(true);
this.wifiConnectError.set(null);
this.wifiConnectMessage.set(null);
this.systemService.connectWirelessAdb(host, port).subscribe({
next: (res) => {
this.isConnectingWifi.set(false);
const cr = res?.connect_result;
if (cr?.success) {
this.wifiConnectMessage.set(`Connected to ${host}:${port}!`);
setTimeout(() => this.wifiConnectMessage.set(null), 4000);
} else {
this.wifiConnectError.set(cr?.message || 'Connection failed. Please check phone IP & Wi-Fi.');
}
},
error: (err) => {
this.isConnectingWifi.set(false);
this.wifiConnectError.set(err?.error?.detail || 'Failed to connect. Please check adb connection.');
}
});
}
public launchAvdEmulator(avdName: string): void {
this.systemService.launchEmulator(avdName).subscribe();
}
public toggleLaunchLogs(): void {
this.showLaunchLogs.update(v => !v);
}
public stopEmulator(): void {
this.systemService.stopEmulator().subscribe();
}
public dismissEmulatorStatus(): void {
this.systemService.dismissEmulatorStatus().subscribe();
}
public selectTargetDevice(serial: string): void {
this.systemService.selectDevice(serial).subscribe();
}
public getEmulatorCommand(avdName: string): string {
const p = this.emulatorPath();
const cmd = this.isEmulatorInPath() ? 'emulator' : (p || '~/Android/Sdk/emulator/emulator');
return `${cmd} -avd ${avdName}`;
}
public copyToClipboard(text: string, id: string): void {
if (navigator.clipboard) {
navigator.clipboard.writeText(text).then(() => {
this.copiedId.set(id);
setTimeout(() => {
if (this.copiedId() === id) {
this.copiedId.set(null);
}
}, 2000);
});
}
}
public setProfile(profile: 'flash' | 'pro'): void {
this.selectedProfile.set(profile);
}
public setCategory(cat: SuggestionCategory): void {
this.selectedCategory.set(cat);
}
public shuffleSuggestions(): void {
this.shuffleOffset.update(v => v + 3);
}
public getAppNamesDisplay(apps: AppReference[]): string {
return apps.map(a => a.name).join(' + ');
}
public applySuggestion(item: SmartSuggestion): void {
this.taskGoal.set(item.goal);
this.selectedProfile.set(item.profile);
this.errorMessage.set(null);
}
public applyQuickPrompt(promptGoal: string, profile?: 'flash' | 'pro'): void {
this.taskGoal.set(promptGoal);
if (profile) {
this.selectedProfile.set(profile);
}
this.errorMessage.set(null);
}
public proceedToLauncher(): void {
this.activeTab.set('launcher');
}
public runTask(): void {
const goal = this.taskGoal().trim();
if (!goal) {
this.errorMessage.set('Please enter a task goal before running.');
return;
}
if (!this.isReady()) {
this.errorMessage.set('System prerequisites are not satisfied. Please review System Setup first.');
this.activeTab.set('diagnostics');
return;
}
this.isSubmitting.set(true);
this.errorMessage.set(null);
this.agentService
.runTask(
goal,
this.selectedProfile(),
this.selectedProfile() === 'pro' && this.expectedOutput().trim()
? this.expectedOutput().trim()
: undefined,
this.selectedProfile() === 'pro' ? this.enableOutputter() : undefined
)
.subscribe({
next: () => {
this.isSubmitting.set(false);
this.router.navigate(['/workspace']);
},
error: (err) => {
console.error('Failed to submit task from home page:', err);
this.isSubmitting.set(false);
this.errorMessage.set(
err?.error?.detail || 'Failed to submit task. Please check server connection.'
);
}
});
}
}
src/app/pages/legacy-workspace/legacy-workspace.component.html → apps/showcase_ui/src/app/pages/legacy-workspace/legacy-workspace.component.html
View File
File renamed without changes.
src/app/pages/legacy-workspace/legacy-workspace.component.scss → apps/showcase_ui/src/app/pages/legacy-workspace/legacy-workspace.component.scss
View File
File renamed without changes.
src/app/pages/legacy-workspace/legacy-workspace.component.ts → apps/showcase_ui/src/app/pages/legacy-workspace/legacy-workspace.component.ts
View File
File renamed without changes.
apps/showcase_ui/src/app/pages/workspace/workspace.component.html
+189
View File
@@ -0,0 +1,189 @@
<!--
Copyright 2026 Google LLC
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<div class="workspace-container" [class.is-resizing]="isDragging">
<!-- Left Main Panel: Raw Agent Log Stream -->
<main class="left-panel glass-panel">
<app-agent-stream></app-agent-stream>
<!-- Bottom Floating Command Bar Dock -->
<div class="workspace-floating-bar-wrapper" [class.is-expanded]="isBarExpanded()" [class.is-dormant]="!isBarExpanded()"
[class.is-focused]="isInputFocused()" [class.has-text]="!!taskInput.trim()" [class.is-running]="isTaskRunning()">
<!-- Floating Error / Notification Toast -->
<div class="floating-dock-toast error" *ngIf="errorMessage">
<span class="material-symbols-outlined toast-icon">error</span>
<span class="toast-text">{{ errorMessage }}</span>
<button type="button" class="btn-toast-close" (click)="errorMessage = null" aria-label="Close error message">
<span class="material-symbols-outlined">close</span>
</button>
</div>
<!-- Dock Row (Contains the Main Card and the Separate Stop Button when running) -->
<div class="floating-dock-row">
<!-- Floating Dock Main Card -->
<div class="floating-dock-card" (click)="onCardClick($event)"
(mouseenter)="isHoveringCard.set(true)" (mouseleave)="isHoveringCard.set(false)">
<!-- Ambient Liquid Wave / Aurora Background Layer (Active in both Dormant & Expanded) -->
<div class="dock-wave-container" [class.is-pro]="selectedProfile() === 'pro'">
<div class="wave-glow-ambient"></div>
<div class="wave-tracks-wrapper">
<!-- Back Fluid Wave -->
<svg class="liquid-wave wave-back" viewBox="0 0 2400 80" preserveAspectRatio="none">
<defs>
<linearGradient id="waveGradBack" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" class="wave-stop-1" />
<stop offset="25%" class="wave-stop-2" />
<stop offset="50%" class="wave-stop-3" />
<stop offset="75%" class="wave-stop-4" />
<stop offset="100%" class="wave-stop-1" />
</linearGradient>
</defs>
<path d="M 0,25 Q 300,55 600,20 T 1200,25 Q 1500,55 1800,20 T 2400,25 L 2400,80 L 0,80 Z"
fill="url(#waveGradBack)"></path>
</svg>
<!-- Mid Fluid Wave -->
<svg class="liquid-wave wave-mid" viewBox="0 0 2400 80" preserveAspectRatio="none">
<defs>
<linearGradient id="waveGradMid" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" class="wave-stop-3" />
<stop offset="35%" class="wave-stop-1" />
<stop offset="70%" class="wave-stop-4" />
<stop offset="100%" class="wave-stop-3" />
</linearGradient>
</defs>
<path d="M 0,35 Q 300,5 600,45 T 1200,35 Q 1500,5 1800,45 T 2400,35 L 2400,80 L 0,80 Z"
fill="url(#waveGradMid)"></path>
</svg>
<!-- Front Fluid Wave -->
<svg class="liquid-wave wave-front" viewBox="0 0 2400 80" preserveAspectRatio="none">
<defs>
<linearGradient id="waveGradFront" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" class="wave-stop-2" />
<stop offset="30%" class="wave-stop-4" />
<stop offset="65%" class="wave-stop-1" />
<stop offset="100%" class="wave-stop-2" />
</linearGradient>
</defs>
<path d="M 0,18 Q 300,50 600,15 T 1200,18 Q 1500,50 1800,15 T 2400,18 L 2400,80 L 0,80 Z"
fill="url(#waveGradFront)"></path>
</svg>
</div>
</div>
<!-- Dormant / Capsule Peek View (Visible when mouse is away and input idle) -->
<div class="capsule-peek-view">
<span class="capsule-placeholder">Start a new task...</span>
</div>
<!-- Expanded Full View Content (Visible on hover, focus, or text) -->
<div class="expanded-card-content">
<div class="card-main-input-row">
<!-- Textarea Container with Dynamic English Placeholder -->
<div class="input-content-area">
<textarea #dockInput [(ngModel)]="taskInput"
(input)="onTextareaInput($event)"
(keydown)="onKeyDown($event)"
(focus)="onInputFocus()"
(blur)="onInputBlur()"
rows="1"
[disabled]="isSubmitting"
[placeholder]="selectedProfile() === 'pro'
? 'Pro: Best for complex long-running tasks, deep reasoning & video analysis'
: 'Flash: Fast execution for direct actions and tasks within 25 steps'"
class="dock-textarea"
spellcheck="false"
></textarea>
</div>
<!-- Clear Button (visible when text is entered) -->
<button *ngIf="taskInput.trim() && !isSubmitting" type="button" class="dock-circle-btn btn-clear"
(click)="clearInput($event)" title="Clear input text" aria-label="Clear input text">
<span class="material-symbols-outlined">close</span>
</button>
<!-- Stop Button (Visible when task is running) -->
<button *ngIf="isTaskRunning()" type="button" class="dock-circle-btn btn-stop" [disabled]="isSubmitting"
(click)="stopTask($event)"
title="Stop running task"
aria-label="Stop running task"
>
<span class="material-symbols-outlined">stop</span>
</button>
<!-- Send Button -->
<button type="button" class="dock-circle-btn btn-send"
[class.has-content]="!!taskInput.trim()"
[disabled]="!taskInput.trim() || isSubmitting"
(click)="submitTask()"
[title]="isSubmitting ? 'Starting task...' : 'Start task (Enter)'"
aria-label="Start task"
>
<span class="material-symbols-outlined spinning-icon" *ngIf="isSubmitting">sync</span>
<span class="material-symbols-outlined" *ngIf="!isSubmitting">arrow_upward</span>
</button>
</div>
<!-- Bottom Segmented Controls Row -->
<div class="card-footer-row">
<div class="footer-left-group">
<!-- Segmented Model Toggle (Flash / Pro) -->
<div class="model-profile-toggle" role="group" aria-label="Select AI Model">
<button type="button" class="profile-toggle-pill" [class.active]="selectedProfile() === 'flash'"
(mousedown)="$event.preventDefault()" (click)="setProfile('flash', $event)"
title="Flash: Fast execution for standard app workflows">
<span class="material-symbols-outlined pill-icon icon-flash">bolt</span>
<span class="pill-label">Flash</span>
</button>
<button type="button" class="profile-toggle-pill" [class.active]="selectedProfile() === 'pro'"
(mousedown)="$event.preventDefault()" (click)="setProfile('pro', $event)"
title="Pro: Deep multimodal reasoning for complex multi-step tasks">
<span class="material-symbols-outlined pill-icon icon-pro">psychology</span>
<span class="pill-label">Pro</span>
</button>
</div>
</div>
</div>
</div>
</div>
<!-- Separate Standalone Stop Button Beside the Capsule (Visible when task is running) -->
<button *ngIf="isTaskRunning()" type="button" class="standalone-stop-btn"
[disabled]="isSubmitting"
(click)="stopTask($event)"
title="Stop running task"
aria-label="Stop running task"
>
<span class="material-symbols-outlined">stop</span>
</button>
</div>
</div>
</main>
<!-- Split Pane Drag Resizer -->
<div class="resizer" (mousedown)="onDragStart($event)">
<div class="resizer-handle"></div>
</div>
<!-- Right Sidebar Panel: Chat Interface Mock -->
<aside class="right-panel" [style.width.px]="rightPanelWidth">
<app-chat-interface></app-chat-interface>
</aside>
<!-- Floating Video Replay Player Modal / Window -->
<app-floating-video-player></app-floating-video-player>
</div>
apps/showcase_ui/src/app/pages/workspace/workspace.component.scss
+865
View File
@@ -0,0 +1,865 @@
/**
* Copyright 2026 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
$bg-base: #e2e5ea;
$font-sans: 'Google Sans Text', 'Google Sans', 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif;
$primary: #0f172a;
$primary-hover: #1e293b;
$rose: #ef4444;
$rose-dark: #dc2626;
:host {
display: block;
height: 100vh;
width: 100vw;
overflow: hidden;
font-family: $font-sans;
}
.workspace-container {
position: relative;
display: flex;
height: 100vh;
width: 100vw;
background-color: $bg-base;
overflow: hidden;
&.is-resizing {
cursor: col-resize;
user-select: none;
}
/* --- Glassmorphism Panel --- */
.glass-panel {
position: relative;
z-index: 1;
background: #f4fafb;
border: none;
box-shadow: none;
}
.left-panel {
flex: 1 0 500px;
display: flex;
flex-direction: column;
overflow: hidden;
background: #f4fafb;
border-right: 1px solid #e2e8f0;
border-left: none;
border-top: none;
border-bottom: none;
min-width: 450px;
height: 100%;
}
.right-panel {
flex: 0 1 auto;
min-width: 250px;
max-width: 100%;
padding: 0;
border: none;
background-color: #ffffff;
box-shadow: -6px 0 24px rgba(0, 0, 0, 0.03);
height: 100%;
display: flex;
flex-direction: column;
box-sizing: border-box;
z-index: 2;
}
.resizer {
width: 12px;
cursor: col-resize;
display: flex;
align-items: center;
justify-content: center;
z-index: 10;
transition: background 0.2s ease;
margin: 0 -6px;
&:hover, .workspace-container.is-resizing & {
background: rgba(255, 255, 255, 0.15);
.resizer-handle {
background: rgba(94, 114, 228, 0.8);
height: 60px;
}
}
.resizer-handle {
width: 4px;
height: 40px;
background: rgba(0, 0, 0, 0.15);
border-radius: 4px;
transition: all 0.2s ease;
}
}
/* ========================================================================= */
/* BOTTOM FLOATING COMMAND BAR (Proximity Reveal & Minimalist Glassmorphism) */
/* ========================================================================= */
.workspace-floating-bar-wrapper {
position: absolute;
bottom: 18px;
left: 50%;
transform: translateX(-50%);
width: 100%;
max-width: min(680px, calc(100% - 36px));
z-index: 50;
display: flex;
flex-direction: column;
align-items: center;
pointer-events: none;
transition: all 0.25s cubic-bezier(0.16, 1, 0.3, 1);
/* Floating Toast / Error Notification */
.floating-dock-toast {
pointer-events: auto;
margin-bottom: 8px;
display: flex;
align-items: center;
gap: 8px;
padding: 7px 14px;
border-radius: 12px;
font-size: 12.5px;
font-weight: 500;
box-shadow: 0 8px 24px -4px rgba(239, 68, 68, 0.15);
animation: slideUpFade 0.2s ease-out;
max-width: 90%;
&.error {
background: rgba(254, 242, 242, 0.88);
backdrop-filter: blur(20px) saturate(180%);
-webkit-backdrop-filter: blur(20px) saturate(180%);
border: 1px solid rgba(254, 202, 202, 0.9);
color: #991b1b;
.toast-icon {
font-size: 16px;
color: #ef4444;
flex-shrink: 0;
}
.toast-text {
flex: 1;
line-height: 1.4;
}
.btn-toast-close {
background: transparent;
border: none;
padding: 2px;
cursor: pointer;
color: #991b1b;
display: flex;
align-items: center;
justify-content: center;
border-radius: 4px;
opacity: 0.7;
transition: opacity 0.15s ease;
&:hover {
opacity: 1;
background: rgba(0, 0, 0, 0.05);
}
.material-symbols-outlined {
font-size: 15px;
}
}
}
}
/* Dock Row: Flex container holding Main Card + Standalone Stop Button */
.floating-dock-row {
display: flex;
align-items: center;
justify-content: center;
gap: 10px;
width: 100%;
pointer-events: none;
transition: gap 0.32s cubic-bezier(0.16, 1, 0.3, 1);
}
/* Standalone Floating Stop Button (Separated from Capsule) */
.standalone-stop-btn {
pointer-events: auto;
display: flex;
align-items: center;
justify-content: center;
width: 34px;
height: 34px;
min-width: 34px;
border-radius: 50%;
border: 1px solid rgba(255, 255, 255, 0.85);
background: rgba(239, 68, 68, 0.9);
backdrop-filter: blur(24px) saturate(190%) brightness(105%);
-webkit-backdrop-filter: blur(24px) saturate(190%) brightness(105%);
color: #ffffff;
box-shadow: 0 6px 18px -2px rgba(239, 68, 68, 0.4), inset 0 1px 1.5px rgba(255, 255, 255, 0.7);
cursor: pointer;
outline: none;
flex-shrink: 0;
transition: opacity 0.28s cubic-bezier(0.16, 1, 0.3, 1),
transform 0.28s cubic-bezier(0.16, 1, 0.3, 1),
width 0.28s cubic-bezier(0.16, 1, 0.3, 1),
min-width 0.28s cubic-bezier(0.16, 1, 0.3, 1),
margin 0.28s cubic-bezier(0.16, 1, 0.3, 1),
background 0.2s ease,
box-shadow 0.2s ease;
animation: popIn 0.25s cubic-bezier(0.16, 1, 0.3, 1);
.material-symbols-outlined {
font-size: 17px;
line-height: 1;
}
&:hover:not(:disabled) {
background: #dc2626;
transform: scale(1.1);
box-shadow: 0 8px 22px rgba(239, 68, 68, 0.55), inset 0 1px 1.5px rgba(255, 255, 255, 0.9);
}
&:active:not(:disabled) {
transform: scale(0.93);
}
&:disabled {
opacity: 0.6;
cursor: not-allowed;
}
}
/* Floating Dock Card Container */
.floating-dock-card {
pointer-events: auto;
position: relative;
width: 100%;
border-radius: 18px;
box-sizing: border-box;
overflow: hidden;
transition: max-width 0.32s cubic-bezier(0.16, 1, 0.3, 1),
width 0.32s cubic-bezier(0.16, 1, 0.3, 1),
border-radius 0.28s cubic-bezier(0.16, 1, 0.3, 1),
box-shadow 0.28s ease,
background 0.28s ease,
transform 0.28s cubic-bezier(0.16, 1, 0.3, 1),
border-color 0.28s ease;
}
/* ========================================================================= */
/* AMBIENT LIQUID WAVE COLOR BLOCKS (Soft Diffused Aurora Glow Waves) */
/* ========================================================================= */
.dock-wave-container {
position: absolute;
bottom: 0;
left: 0;
right: 0;
height: 100%;
pointer-events: none;
border-radius: inherit;
overflow: hidden;
z-index: 0;
opacity: 0.78;
transition: opacity 0.35s ease, filter 0.35s ease;
/* Flash Mode Color Stops (Soft Electric Blue, Violet, Amber, Cyan) */
.wave-stop-1 { stop-color: #3b82f6; stop-opacity: 0.65; }
.wave-stop-2 { stop-color: #8b5cf6; stop-opacity: 0.7; }
.wave-stop-3 { stop-color: #f59e0b; stop-opacity: 0.65; }
.wave-stop-4 { stop-color: #06b6d4; stop-opacity: 0.6; }
/* Pro Mode Color Stops (Soft Cosmic Nebula / Amethyst / Fuchsia) */
&.is-pro {
.wave-stop-1 { stop-color: #7c3aed; stop-opacity: 0.7; }
.wave-stop-2 { stop-color: #ec4899; stop-opacity: 0.75; }
.wave-stop-3 { stop-color: #6366f1; stop-opacity: 0.65; }
.wave-stop-4 { stop-color: #a855f7; stop-opacity: 0.7; }
}
/* Deep Ambient Aurora Glow Diffusion */
.wave-glow-ambient {
position: absolute;
bottom: -8px;
left: 5%;
right: 5%;
height: 30px;
border-radius: 50%;
background: radial-gradient(ellipse at center, rgba(99, 102, 241, 0.4) 0%, rgba(245, 158, 11, 0.2) 50%, transparent 80%);
filter: blur(16px);
opacity: 0.7;
animation: glowBreathe 5s ease-in-out infinite alternate;
}
&.is-pro .wave-glow-ambient {
background: radial-gradient(ellipse at center, rgba(168, 85, 247, 0.45) 0%, rgba(236, 72, 153, 0.25) 50%, transparent 80%);
}
/* Wave Tracks Wrapper with Soft Gaussian Blur */
.wave-tracks-wrapper {
position: absolute;
bottom: 0;
left: 0;
width: 100%;
height: 22px;
overflow: hidden;
filter: blur(7px); /* Soften sharp vector wave edges into silky liquid light */
mask-image: linear-gradient(to top, rgba(0,0,0,0.9) 0%, rgba(0,0,0,0.5) 60%, transparent 100%);
-webkit-mask-image: linear-gradient(to top, rgba(0,0,0,0.9) 0%, rgba(0,0,0,0.5) 60%, transparent 100%);
.liquid-wave {
position: absolute;
bottom: 0;
left: 0;
width: 200%;
height: 100%;
will-change: transform;
filter: blur(3px); /* Secondary soft layer blur */
&.wave-back {
opacity: 0.45;
animation: waveSlide 12s ease-in-out infinite alternate;
}
&.wave-mid {
opacity: 0.6;
animation: waveSlideReverse 9s ease-in-out infinite alternate;
}
&.wave-front {
opacity: 0.75;
animation: waveSlide 6.5s ease-in-out infinite alternate;
}
}
}
}
/* ======================================================================= */
/* 1. DORMANT / RESTING CAPSULE STATE (Subtle Frosted Peek) */
/* ======================================================================= */
&.is-dormant {
.floating-dock-card {
max-width: 160px;
width: 160px;
margin: 0;
background: rgba(255, 255, 255, 0.65);
backdrop-filter: blur(26px) saturate(190%) brightness(104%);
-webkit-backdrop-filter: blur(26px) saturate(190%) brightness(104%);
border: 1px solid rgba(255, 255, 255, 0.85);
border-radius: 17px;
box-shadow: 0 6px 18px -2px rgba(15, 23, 42, 0.07), inset 0 1px 1.5px rgba(255, 255, 255, 0.95), inset 0 -1px 1px rgba(255, 255, 255, 0.3);
cursor: pointer;
opacity: 0.92;
transform: translateY(2px);
.dock-wave-container {
opacity: 0.78;
.wave-tracks-wrapper {
height: 12px;
}
}
&:hover {
opacity: 1;
transform: translateY(0);
background: rgba(255, 255, 255, 0.82);
border-color: rgba(255, 255, 255, 1);
box-shadow: 0 8px 24px -2px rgba(15, 23, 42, 0.12), inset 0 1px 1.5px rgba(255, 255, 255, 1);
.dock-wave-container {
opacity: 0.95;
filter: drop-shadow(0 0 6px rgba(99, 102, 241, 0.35));
}
}
.capsule-peek-view {
display: flex;
opacity: 1;
}
.expanded-card-content {
display: none;
opacity: 0;
}
}
}
/* Capsule Peek View Layout */
.capsule-peek-view {
position: relative;
z-index: 1;
display: flex;
align-items: center;
justify-content: center;
padding: 6px 14px;
min-height: 34px;
box-sizing: border-box;
user-select: none;
.capsule-placeholder {
color: #475569;
font-size: 12px;
font-weight: 500;
letter-spacing: 0.1px;
white-space: nowrap;
text-align: center;
}
}
/* ======================================================================= */
/* 2. EXPANDED / ACTIVE STATE (Rich Frosted Command Box) */
/* ======================================================================= */
&.is-expanded {
.floating-dock-row {
gap: 0;
}
.standalone-stop-btn {
opacity: 0;
width: 0;
min-width: 0;
margin: 0;
padding: 0;
border-width: 0;
transform: scale(0.5);
pointer-events: none;
}
.floating-dock-card {
max-width: 100%;
width: 100%;
background: rgba(255, 255, 255, 0.68);
backdrop-filter: blur(32px) saturate(210%) brightness(105%);
-webkit-backdrop-filter: blur(32px) saturate(210%) brightness(105%);
border: 1px solid rgba(255, 255, 255, 0.82);
box-shadow: 0 16px 44px -8px rgba(15, 23, 42, 0.12), 0 2px 8px rgba(0, 0, 0, 0.02), inset 0 1.5px 2px rgba(255, 255, 255, 0.95), inset 0 -1px 2px rgba(255, 255, 255, 0.3);
opacity: 1;
transform: translateY(0);
.dock-wave-container {
opacity: 0.88;
.wave-tracks-wrapper {
height: 22px;
}
}
.capsule-peek-view {
display: none;
opacity: 0;
}
.expanded-card-content {
display: block;
opacity: 1;
animation: cardFadeIn 0.25s cubic-bezier(0.16, 1, 0.3, 1);
}
}
}
/* Focused State Glow */
&.is-focused .floating-dock-card {
background: rgba(255, 255, 255, 0.78);
backdrop-filter: blur(36px) saturate(220%) brightness(106%);
-webkit-backdrop-filter: blur(36px) saturate(220%) brightness(106%);
border-color: rgba(59, 130, 246, 0.55);
box-shadow: 0 20px 48px -6px rgba(37, 99, 235, 0.16), 0 0 0 2.5px rgba(59, 130, 246, 0.18), inset 0 1.5px 2px rgba(255, 255, 255, 1);
.dock-wave-container {
opacity: 1;
filter: drop-shadow(0 0 8px rgba(59, 130, 246, 0.35));
}
}
/* Expanded Content Rows */
.expanded-card-content {
position: relative;
z-index: 1;
display: flex;
flex-direction: column;
width: 100%;
.card-main-input-row {
display: flex;
align-items: center;
gap: 10px;
padding: 8px 10px 3px 14px;
min-height: 48px;
box-sizing: border-box;
.leading-status-icon {
width: 14px;
height: 14px;
display: flex;
align-items: center;
justify-content: center;
flex-shrink: 0;
.pulse-indicator-dot {
width: 8px;
height: 8px;
border-radius: 50%;
background: #ef4444;
box-shadow: 0 0 0 0 rgba(239, 68, 68, 0.6);
animation: pulseDot 1.4s infinite;
}
}
.input-content-area {
flex: 1;
min-width: 0;
display: flex;
align-items: center;
.dock-textarea {
width: 100%;
border: none;
outline: none;
background: transparent;
resize: none;
font-family: inherit;
font-size: 13.5px;
font-weight: 450;
line-height: 1.5;
color: #0f172a;
padding: 3px 0;
max-height: 120px;
box-sizing: border-box;
display: block;
&::placeholder {
color: #64748b;
font-size: 12.5px;
font-weight: 400;
transition: color 0.15s ease;
}
&:disabled {
opacity: 0.6;
cursor: not-allowed;
}
}
}
.dock-circle-btn {
border-radius: 50%;
display: flex;
align-items: center;
justify-content: center;
border: none;
cursor: pointer;
outline: none;
flex-shrink: 0;
transition: all 0.18s cubic-bezier(0.16, 1, 0.3, 1);
.material-symbols-outlined {
font-size: 18px;
line-height: 1;
}
.spinning-icon {
font-size: 17px;
animation: spin 1s linear infinite;
}
/* Clear Button */
&.btn-clear {
width: 26px;
height: 26px;
background: rgba(255, 255, 255, 0.45);
backdrop-filter: blur(8px);
border: 1px solid rgba(255, 255, 255, 0.6);
color: #64748b;
&:hover {
background: rgba(255, 255, 255, 0.75);
color: #0f172a;
}
}
/* Stop Button (Frosted Red) */
&.btn-stop {
width: 36px;
height: 36px;
background: rgba(239, 68, 68, 0.9);
backdrop-filter: blur(8px);
color: #ffffff;
border: 1px solid rgba(255, 255, 255, 0.35);
box-shadow: 0 4px 12px rgba(239, 68, 68, 0.3);
animation: popIn 0.2s cubic-bezier(0.16, 1, 0.3, 1);
&:hover:not(:disabled) {
background: #dc2626;
transform: scale(1.05);
box-shadow: 0 6px 16px rgba(239, 68, 68, 0.4);
}
&:active:not(:disabled) {
transform: scale(0.95);
}
&:disabled {
opacity: 0.6;
cursor: not-allowed;
}
}
/* Send Button */
&.btn-send {
width: 36px;
height: 36px;
background: rgba(15, 23, 42, 0.88);
backdrop-filter: blur(12px);
border: 1px solid rgba(255, 255, 255, 0.25);
color: #ffffff;
box-shadow: 0 4px 14px rgba(15, 23, 42, 0.2), inset 0 1px 1px rgba(255, 255, 255, 0.25);
&:disabled {
background: rgba(226, 232, 240, 0.6);
color: #94a3b8;
cursor: not-allowed;
box-shadow: none;
border-color: rgba(255, 255, 255, 0.5);
transform: none;
}
&.has-content:not(:disabled) {
background: linear-gradient(135deg, rgba(30, 41, 59, 0.95), rgba(15, 23, 42, 0.98));
box-shadow: 0 4px 14px rgba(15, 23, 42, 0.25), inset 0 1px 1px rgba(255, 255, 255, 0.3);
&:hover {
background: linear-gradient(135deg, #2563eb, #1d4ed8);
transform: scale(1.05);
box-shadow: 0 6px 18px rgba(37, 99, 235, 0.35);
}
}
&:active:not(:disabled) {
transform: scale(0.95);
}
}
}
}
/* Model Profile Selector & Status Footer Row */
.card-footer-row {
display: flex;
align-items: center;
justify-content: space-between;
padding: 3px 14px 8px 14px;
box-sizing: border-box;
.footer-left-group {
display: flex;
align-items: center;
gap: 8px;
/* Segmented Model Toggle with Frosted Glass */
.model-profile-toggle {
display: inline-flex;
align-items: center;
background: rgba(255, 255, 255, 0.45);
backdrop-filter: blur(14px);
-webkit-backdrop-filter: blur(14px);
border: 1px solid rgba(255, 255, 255, 0.65);
box-shadow: inset 0 1px 1px rgba(255, 255, 255, 0.8), 0 1px 3px rgba(0, 0, 0, 0.03);
border-radius: 8px;
padding: 2px;
gap: 2px;
.profile-toggle-pill {
display: inline-flex;
align-items: center;
justify-content: center;
gap: 3.5px;
width: 62px;
min-width: 62px;
height: 22px;
border-radius: 6px;
border: none;
background: transparent;
color: #64748b;
font-family: inherit;
font-size: 11px;
font-weight: 550;
cursor: pointer;
transition: all 0.16s cubic-bezier(0.16, 1, 0.3, 1);
outline: none;
user-select: none;
box-sizing: border-box;
.pill-icon {
font-size: 13px;
line-height: 1;
flex-shrink: 0;
}
.pill-label {
line-height: 1;
}
&:hover:not(.active) {
color: #1e293b;
background: rgba(255, 255, 255, 0.65);
}
&.active {
background: rgba(255, 255, 255, 0.92);
backdrop-filter: blur(8px);
color: #0f172a;
font-weight: 700;
box-shadow: 0 1.5px 4px rgba(0, 0, 0, 0.08), inset 0 1px 1.5px rgba(255, 255, 255, 1);
.icon-flash {
color: #d97706;
}
.icon-pro {
color: #7c3aed;
}
}
}
}
}
.footer-right-status {
.running-status-pill {
display: inline-flex;
align-items: center;
gap: 4.5px;
font-size: 10.5px;
font-weight: 600;
color: #dc2626;
background: rgba(254, 226, 226, 0.7);
padding: 2px 7px;
border-radius: 10px;
.running-pulse-dot {
width: 5.5px;
height: 5.5px;
border-radius: 50%;
background: #ef4444;
animation: pulseDot 1.2s infinite;
}
}
}
}
}
}
/* When viewport shrinks to approximately 2/3 of standard desktop width (<= 1150px), hide right panel and leave only agent-stream */
@media (max-width: 1150px) {
.right-panel, .resizer {
display: none !important;
}
.left-panel {
border-right: none !important;
width: 100% !important;
flex: 1 1 100% !important;
min-width: 0 !important;
}
}
}
/* Animations */
@keyframes popIn {
0% {
opacity: 0;
transform: scale(0.8);
}
100% {
opacity: 1;
transform: scale(1);
}
}
@keyframes spin {
from { transform: rotate(0deg); }
to { transform: rotate(360deg); }
}
@keyframes slideUpFade {
from {
opacity: 0;
transform: translateY(8px);
}
to {
opacity: 1;
transform: translateY(0);
}
}
@keyframes cardFadeIn {
from {
opacity: 0;
transform: translateY(4px);
}
to {
opacity: 1;
transform: translateY(0);
}
}
@keyframes fadeIn {
from { opacity: 0; }
to { opacity: 1; }
}
@keyframes pulseDot {
0% {
transform: scale(0.95);
box-shadow: 0 0 0 0 rgba(239, 68, 68, 0.7);
}
70% {
transform: scale(1);
box-shadow: 0 0 0 6px rgba(239, 68, 68, 0);
}
100% {
transform: scale(0.95);
box-shadow: 0 0 0 0 rgba(239, 68, 68, 0);
}
}
/* Fluid Wave Horizontal Slide (Seamless loop) */
@keyframes waveSlide {
0% {
transform: translate3d(0, 0, 0);
}
100% {
transform: translate3d(-50%, 0, 0);
}
}
@keyframes waveSlideReverse {
0% {
transform: translate3d(-50%, 0, 0);
}
100% {
transform: translate3d(0, 0, 0);
}
}
/* Ambient Radial Glow Breathing */
@keyframes glowBreathe {
0% {
opacity: 0.55;
transform: scaleX(0.92);
}
100% {
opacity: 0.95;
transform: scaleX(1.08);
}
}
apps/showcase_ui/src/app/pages/workspace/workspace.component.ts
+284
View File
@@ -0,0 +1,284 @@
/**
* Copyright 2026 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { Component, HostListener, inject, computed, signal, ViewChild, ElementRef, OnInit } from '@angular/core';
import { CommonModule } from '@angular/common';
import { FormsModule } from '@angular/forms';
import { HttpClient } from '@angular/common/http';
import { AgentStreamComponent } from '../../components/agent-stream/agent-stream.component';
import { ChatInterfaceComponent } from '../../components/chat-interface/chat-interface.component';
import { FloatingVideoPlayerComponent } from '../../components/floating-video-player/floating-video-player.component';
import { AgentService } from '../../services/agent.service';
@Component({
selector: 'app-workspace',
standalone: true,
imports: [
CommonModule,
FormsModule,
AgentStreamComponent,
ChatInterfaceComponent,
FloatingVideoPlayerComponent
],
templateUrl: './workspace.component.html',
styleUrl: './workspace.component.scss'
})
export class WorkspaceComponent implements OnInit {
public agentService = inject(AgentService);
private http = inject(HttpClient);
// Default right panel width to 1/3 of the screen (or 450px as fallback)
public rightPanelWidth = typeof window !== 'undefined' ? Math.round(window.innerWidth / 3) : 450;
public isDragging = false;
// Floating Command Bar State
public taskInput: string = '';
public isSubmitting: boolean = false;
public errorMessage: string | null = null;
public selectedProfile = signal<'flash' | 'pro'>('flash');
// Expand States (Signals for 0-latency reactivity)
public isHoveringCard = signal<boolean>(false);
public isInputFocused = signal<boolean>(false);
@ViewChild('dockInput') public dockInputRef?: ElementRef<HTMLTextAreaElement>;
ngOnInit(): void {
if (typeof localStorage !== 'undefined') {
const saved = localStorage.getItem('artemis_selected_profile');
if (saved === 'flash' || saved === 'pro') {
this.selectedProfile.set(saved);
}
}
}
/**
* Set model profile ('flash' vs 'pro')
*/
public setProfile(profile: 'flash' | 'pro', event?: MouseEvent): void {
if (event) {
event.stopPropagation();
}
this.selectedProfile.set(profile);
if (typeof localStorage !== 'undefined') {
localStorage.setItem('artemis_selected_profile', profile);
}
}
/**
* Computed boolean whether a task is currently executing or active
*/
public isTaskRunning = computed(() => {
return this.agentService.isRunningTask();
});
/**
* Computed boolean whether the command bar should be in its expanded state:
* - Mouse is hovering directly on the command card
* - Input textarea is focused
* - User has entered task text (drafting)
*/
public isBarExpanded = computed(() => {
return (
this.isHoveringCard() ||
this.isInputFocused() ||
this.taskInput.trim().length > 0
);
});
/**
* Focus input handler
*/
public onInputFocus(): void {
this.isInputFocused.set(true);
}
/**
* Blur input handler
*/
public onInputBlur(): void {
this.isInputFocused.set(false);
}
/**
* Click on the resting capsule or dock card to focus textarea
*/
public onCardClick(event: MouseEvent): void {
const target = event.target as HTMLElement;
// Don't steal focus if clicking action buttons or textarea directly
if (target.closest('button') || target.tagName.toLowerCase() === 'textarea') {
return;
}
this.focusInput();
}
/**
* Focus textarea programmatically
*/
public focusInput(): void {
setTimeout(() => {
this.dockInputRef?.nativeElement?.focus();
}, 30);
}
/**
* Clear typed task input
*/
public clearInput(event?: MouseEvent): void {
if (event) {
event.stopPropagation();
}
this.taskInput = '';
if (this.dockInputRef?.nativeElement) {
this.dockInputRef.nativeElement.style.height = 'auto';
this.dockInputRef.nativeElement.focus();
}
}
/**
* Auto-resize textarea as user types multi-line tasks
*/
public onTextareaInput(event: Event): void {
const textarea = event.target as HTMLTextAreaElement;
if (!textarea) return;
textarea.style.height = 'auto';
const newHeight = Math.min(Math.max(textarea.scrollHeight, 24), 120);
textarea.style.height = `${newHeight}px`;
}
/**
* Global keyboard shortcut (⌘K or Ctrl+K) to focus input bar from anywhere
*/
@HostListener('window:keydown', ['$event'])
public onGlobalKeyDown(event: KeyboardEvent): void {
if ((event.metaKey || event.ctrlKey) && event.key.toLowerCase() === 'k') {
event.preventDefault();
this.focusInput();
}
}
/**
* Handle enter key in floating dock textarea
*/
public onKeyDown(event: KeyboardEvent): void {
if (event.key === 'Enter' && !event.shiftKey) {
event.preventDefault();
this.submitTask();
}
}
/**
* Submit a new task instruction
*/
public submitTask(): void {
const goal = this.taskInput.trim();
if (!goal || this.isSubmitting) {
return;
}
this.isSubmitting = true;
this.errorMessage = null;
if (this.dockInputRef?.nativeElement) {
this.dockInputRef.nativeElement.blur();
}
this.isInputFocused.set(false);
this.agentService.clearUserPinnedSession();
this.http.post<any>('/api/run', { goal, profile: this.selectedProfile() }).subscribe({
next: (res) => {
this.taskInput = '';
if (this.dockInputRef?.nativeElement) {
this.dockInputRef.nativeElement.style.height = 'auto';
}
this.isSubmitting = false;
if (res && res.tasks && res.tasks.length > 0) {
const firstTask = res.tasks[0];
if (firstTask && firstTask.session_id) {
const isCurrentlyRunning =
this.agentService.agentStatus() === 'running' ||
this.agentService.sessions().some((s) => s.status === 'running');
if (!isCurrentlyRunning) {
this.agentService.selectSession(firstTask.session_id, false);
}
}
}
this.agentService.fetchStatus();
},
error: (err) => {
console.error('Failed to submit task:', err);
this.isSubmitting = false;
this.errorMessage = err.error?.detail || 'The runner is busy. Please wait for current task to finish.';
setTimeout(() => {
this.errorMessage = null;
}, 5000);
}
});
}
/**
* Stop currently running task
*/
public stopTask(event?: MouseEvent): void {
if (event) {
event.stopPropagation();
}
if (!this.isTaskRunning()) {
return;
}
this.isSubmitting = true;
this.errorMessage = null;
this.agentService.stopTask(false);
setTimeout(() => {
this.isSubmitting = false;
}, 400);
}
/**
* Handle mouse down on resizer bar to start dragging
*/
public onDragStart(event: MouseEvent): void {
this.isDragging = true;
event.preventDefault();
}
/**
* Listen to global mousemove events to adjust the width dynamically
*/
@HostListener('document:mousemove', ['$event'])
public onMouseMove(event: MouseEvent): void {
if (!this.isDragging) {
return;
}
const newWidth = window.innerWidth - event.clientX;
const minWidth = 250;
const maxWidth = window.innerWidth - 300;
// Apply boundary limits to prevent panels from shrinking too much
if (newWidth >= minWidth && newWidth <= maxWidth) {
this.rightPanelWidth = newWidth;
}
}
/**
* Listen to global mouseup events to stop resizing
*/
@HostListener('document:mouseup')
public onMouseUp(): void {
this.isDragging = false;
}
}
src/app/services/agent.service.ts → apps/showcase_ui/src/app/services/agent.service.ts
+229 -23
View File
@@ -16,6 +16,7 @@
import { Injectable, signal, inject, computed } from '@angular/core';
import { HttpClient } from '@angular/common/http';
import { Observable } from 'rxjs';
import { Session, ModelInfo, TaskQueueItem, AgentStatusResponse } from '../core/models/session.model';
import { StepItemData } from '../core/models/stream.model';
@@ -38,20 +39,47 @@ export class AgentService {
const goal = this.runningGoal();
const runId = this.runningSessionId();
const hasRunningInRaw = raw.some(s => s.status === 'running' || (runId && s.session_id === runId));
const sessionMap = new Map<string, Session>();
if (status === 'running' && goal && runId && !hasRunningInRaw) {
const activeRunningSession: Session = {
// 1. First add raw sessions from DB
raw.forEach((s) => {
const isCurrentRunning = status === 'running' && runId === s.session_id;
const isPending = this.pendingQueue().some(p => p.session_id === s.session_id) && !isCurrentRunning;
let sStatus = isCurrentRunning ? 'running' : (isPending ? 'pending' : s.status);
if (sStatus === 'running' && !isCurrentRunning) {
const activeRun = raw.find(r => r.session_id === runId);
sStatus = (activeRun && (s.start_time || 0) > (activeRun.start_time || 0)) ? 'pending' : 'completed';
}
sessionMap.set(s.session_id, {
...s,
status: sStatus,
model_info: isCurrentRunning && this.activeModel() ? this.activeModel()! : s.model_info
});
});
// 2. Add pending queue sessions if not yet in raw sessions
pending.forEach((p) => {
if (!sessionMap.has(p.session_id)) {
const isCurrentRunning = status === 'running' && runId === p.session_id;
sessionMap.set(p.session_id, {
...p,
status: isCurrentRunning ? 'running' : 'pending'
});
}
});
// 3. Ensure currently active running session is present
if (status === 'running' && runId && !sessionMap.has(runId)) {
sessionMap.set(runId, {
session_id: runId,
initial_goal: goal,
initial_goal: goal || '',
start_time: Date.now() / 1000,
status: 'running',
model_info: this.activeModel() || undefined
};
return [activeRunningSession, ...pending, ...raw];
});
}
return [...pending, ...raw];
return Array.from(sessionMap.values()).sort((a, b) => b.start_time - a.start_time);
});
public currentSessionId = signal<string | null>(null);
@@ -64,6 +92,7 @@ export class AgentService {
public isRetrying = signal<boolean>(false);
public retryMessage = signal<string | null>(null);
public activeModel = signal<{ name: string; id: string; provider: string } | null>(null);
public userPinnedSessionId = signal<string | null>(null); // Pinned session if user explicitly selected a non-running task
// Notes and Tab States
public currentNotes = signal<Record<string, string>>({});
@@ -77,6 +106,13 @@ export class AgentService {
public activeVideoTitle = signal<string>('');
public isVideoLoading = signal<boolean>(false);
/**
* Clear user-pinned selection so subsequent runs automatically follow active runner
*/
public clearUserPinnedSession(): void {
this.userPinnedSessionId.set(null);
}
/**
* Computed active/viewed session
*/
@@ -122,18 +158,82 @@ export class AgentService {
});
/**
* Stop the currently running task
* Run a new task by submitting to backend queue
*/
public stopTask(): void {
this.http.post('/api/stop', {}).subscribe({
next: () => {
public runTask(
goal: string,
profile: string = 'flash',
expectedOutput?: string,
enableOutputter?: boolean
): Observable<any> {
return new Observable((obs) => {
const payload: any = { goal, profile };
if (expectedOutput && expectedOutput.trim()) {
payload.expected_output = expectedOutput.trim();
}
if (enableOutputter !== undefined) {
payload.enable_outputter = enableOutputter;
}
this.clearUserPinnedSession();
this.http.post<any>('/api/run', payload).subscribe({
next: (res) => {
if (res && res.tasks && res.tasks.length > 0) {
const newSessionId = res.tasks[0].session_id;
if (newSessionId) {
const isCurrentlyRunning =
this.agentStatus() === 'running' ||
this.sessions().some((s) => s.status === 'running');
if (!isCurrentlyRunning) {
this.selectSession(newSessionId, false);
}
}
}
obs.next(res);
obs.complete();
},
error: (err) => {
obs.error(err);
}
});
});
}
/**
* Stop the currently running task (and optionally clear queue)
*/
public stopTask(stopAll: boolean = false): void {
// Apply optimistic updates: only set idle if stopAll is true or pending queue is empty
if (stopAll || this.pendingQueue().length === 0) {
this.agentStatus.set('idle');
this.runningSessionId.set(null);
this.runningGoal.set(null);
}
this.isPaused.set(false);
this.pausedError.set(null);
this.isRetrying.set(false);
if (stopAll) {
this.pendingQueue.set([]);
}
// Mark active streaming / pending logs in current session as finished
this.sessionLogs.update((logs) =>
logs.map((l) => {
if (l.type === 'llm_stream' && !l.data?.isCompleted) {
return { ...l, data: { ...l.data, isCompleted: true } };
}
return l;
})
);
this.http.post<any>(`/api/stop?all=${stopAll}`, {}).subscribe({
next: () => {
this.fetchStatus();
this.fetchSessions();
},
error: (err) => {
console.error('Failed to stop task:', err);
this.fetchSessions();
this.fetchStatus();
}
});
}
@@ -161,6 +261,10 @@ export class AgentService {
next: (data) => {
this.rawSessions.set(data);
this.updateModelForCurrentView(data);
// On initial load, if nothing is selected, not pinned, not running, and sessions exist, select latest
if (!this.currentSessionId() && !this.userPinnedSessionId() && this.agentStatus() !== 'running' && data.length > 0) {
this.selectSession(data[0].session_id, false);
}
},
error: (err) => {
console.error('Failed to fetch sessions from backend:', err);
@@ -169,9 +273,97 @@ export class AgentService {
}
/**
* Connect to the Server-Sent Events stream for a specific session
* Delete an individual session / task
*/
public selectSession(sessionId: string): void {
public deleteSession(sessionId: string): Observable<any> {
// 1. Optimistically update local session state immediately
this.rawSessions.update((list) => list.filter((s) => s.session_id !== sessionId));
this.pendingQueue.update((list) => list.filter((s) => s.session_id !== sessionId));
if (this.userPinnedSessionId() === sessionId) {
this.userPinnedSessionId.set(null);
}
if (this.currentSessionId() === sessionId) {
this.selectSession('', false);
if (this.isVideoWindowOpen()) {
this.closeVideoPlayer();
}
const runningId = this.runningSessionId();
if (runningId && this.agentStatus() === 'running') {
this.selectSession(runningId, false);
}
}
// 2. Send API request
return new Observable((obs) => {
this.http.post<any>(`/api/sessions/${sessionId}/delete`, {}).subscribe({
next: (res) => {
this.fetchSessions();
this.fetchStatus();
obs.next(res);
obs.complete();
},
error: (err) => {
console.error(`Failed to delete session ${sessionId}:`, err);
this.fetchSessions();
this.fetchStatus();
obs.error(err);
}
});
});
}
/**
* Clear all sessions, tasks, and history
*/
public clearAllHistory(): Observable<any> {
// 1. Optimistically clear all local session state immediately
this.userPinnedSessionId.set(null);
this.rawSessions.set([]);
this.pendingQueue.set([]);
this.selectSession('', false);
if (this.isVideoWindowOpen()) {
this.closeVideoPlayer();
}
// 2. Send API request
return new Observable((obs) => {
this.http.post<any>('/api/cleanup', {}).subscribe({
next: (res) => {
this.fetchSessions();
this.fetchStatus();
obs.next(res);
obs.complete();
},
error: (err) => {
console.error('Failed to cleanup history:', err);
this.fetchSessions();
this.fetchStatus();
obs.error(err);
}
});
});
}
/**
* Connect to the Server-Sent Events stream for a specific session
* @param sessionId The ID of the session to select
* @param isUserAction Whether this selection was explicitly triggered by user interaction
*/
public selectSession(sessionId: string, isUserAction: boolean = false): void {
if (isUserAction) {
if (sessionId && sessionId === this.runningSessionId() && this.agentStatus() === 'running') {
// User explicitly clicked the currently running task -> follow live runner
this.userPinnedSessionId.set(null);
} else if (sessionId) {
// User explicitly clicked a non-running or historical task -> pin to it
this.userPinnedSessionId.set(sessionId);
} else {
this.userPinnedSessionId.set(null);
}
}
if (!sessionId) {
this.currentSessionId.set(null);
this.sessionLogs.set([]);
@@ -262,12 +454,13 @@ export class AgentService {
this.runningGoal.set(parsedData.initial_goal);
}
this.fetchSessions();
// Only auto-select if user hasn't explicitly chosen to inspect another task
const pinnedId = this.userPinnedSessionId();
if (!pinnedId && parsedData?.session_id) {
this.selectSession(parsedData.session_id, false);
}
} else if (eventType === 'session_ended') {
this.agentStatus.set('idle');
this.runningSessionId.set(null);
this.runningGoal.set(null);
this.isRetrying.set(false);
this.isPaused.set(false);
this.fetchStatus();
this.fetchSessions();
this.updateModelForCurrentView();
}
@@ -298,6 +491,7 @@ export class AgentService {
this.isRetrying.set(false);
this.isPaused.set(false);
this.fetchSessions();
this.fetchNotes(sessionId);
this.updateModelForCurrentView();
return;
}
@@ -416,7 +610,7 @@ export class AgentService {
if (eventType === 'trace_recorded' && parsedData) {
const trName = parsedData.name || '';
if (['save_note', 'read_note', 'update_note', 'append_note', 'list_notes'].includes(trName.toLowerCase())) {
if (['save_note', 'read_note', 'update_note', 'append_note', 'list_notes', 'outputter'].includes(trName.toLowerCase())) {
const curSessionId = this.currentSessionId();
if (curSessionId === sessionId) {
this.fetchNotes(curSessionId);
@@ -443,6 +637,8 @@ export class AgentService {
};
}
private pollCounter: number = 0;
/**
* Start periodic status polling from the backend
*/
@@ -450,6 +646,11 @@ export class AgentService {
this.fetchStatus();
this.statusInterval = setInterval(() => {
this.fetchStatus();
this.pollCounter++;
// Periodically refresh sessions every 6 seconds (every 3 polling intervals of 2s) to stay in sync with external DB modifications
if (this.pollCounter % 3 === 0) {
this.fetchSessions();
}
}, 2000);
}
@@ -511,11 +712,16 @@ export class AgentService {
this.fetchSessions();
}
// Auto-select the session if a new active session starts and user is not browsing an old historical session
if (data.status === 'running' && data.session_id && data.session_id !== oldRunningSessionId) {
// Auto-select the session if running and user hasn't explicitly chosen to inspect another task
if (data.status === 'running' && data.session_id) {
const currentId = this.currentSessionId();
if (!currentId || currentId === oldRunningSessionId) {
this.selectSession(data.session_id);
const pinnedId = this.userPinnedSessionId();
if (!currentId) {
this.selectSession(data.session_id, false);
} else if (currentId !== data.session_id) {
if (!pinnedId) {
this.selectSession(data.session_id, false);
}
}
}
}
apps/showcase_ui/src/app/services/system.service.ts
+482
View File
@@ -0,0 +1,482 @@
/**
* Copyright 2026 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { Injectable, signal, computed, inject } from '@angular/core';
import { HttpClient } from '@angular/common/http';
import { Observable, tap } from 'rxjs';
import { SystemReadinessReport, DeviceInfo, ProbeResult, EmulatorLaunchState, EmulatorLaunchStage } from '../core/models/system.model';
@Injectable({
providedIn: 'root'
})
export class SystemService {
private http = inject(HttpClient);
// Core reactive signals
public readinessReport = signal<SystemReadinessReport | null>(null);
public isLoading = signal<boolean>(false);
public isRestartingAdb = signal<boolean>(false);
public launchingAvd = signal<string | null>(null);
public emulatorLaunchState = signal<EmulatorLaunchState | null>(null);
public isEmulatorLaunching = computed(() => {
const s = this.emulatorLaunchState()?.status;
return s === 'starting' || s === 'waiting_for_adb' || s === 'booting';
});
public lastCheckedTime = signal<Date | null>(null);
private emulatorPollTimer: any = null;
// Probe lookups
public pythonProbe = computed(() => this.probes().find(p => p.id === 'python_runtime') || null);
public configProbe = computed(() => this.probes().find(p => p.id === 'system_config') || null);
public toolchainProbe = computed(() => this.probes().find(p => p.id === 'toolchain') || null);
public adbProbe = computed(() => this.probes().find(p => p.id === 'android_adb') || null);
public llmProbe = computed(() => this.probes().find(p => p.id === 'gemini_api_key' || p.id === 'llm_api_key') || null);
public geminiProbe = computed(() => this.llmProbe());
public ocrProbe = computed(() => this.probes().find(p => p.id === 'vision_ocr_key' || p.id === 'ocr_api_key') || null);
// Grouped readiness helpers for the 3-step onboarding flow
public isEnvironmentReady = computed(() => {
const py = this.pythonProbe();
const cfg = this.configProbe();
const adb = this.adbProbe();
const tc = this.toolchainProbe();
const pyOk = py?.status === 'pass';
const cfgOk = cfg?.status === 'pass';
const adbInstalled = adb?.metadata?.['installed'] ?? (adb?.status !== 'fail');
const tcOk = tc?.status === 'pass' || tc?.is_blocker === false;
return pyOk && cfgOk && adbInstalled && tcOk;
});
public isSkipCredentialsCheck = signal<boolean>(false);
public isCredentialsReady = computed(() => {
if (this.isSkipCredentialsCheck()) {
return true;
}
const llm = this.llmProbe();
return llm?.status === 'pass';
});
public isDeviceReady = computed(() => {
const adb = this.adbProbe();
return adb?.status === 'pass';
});
// Step-level counting matching the 3-step onboarding guide
public totalStepCount = computed(() => 3);
public passedStepCount = computed(() => {
let count = 0;
if (this.isEnvironmentReady()) count++;
if (this.isCredentialsReady()) count++;
if (this.isDeviceReady()) count++;
return count;
});
// Computed helper signals
public isReady = computed(() => {
return this.isEnvironmentReady() && this.isCredentialsReady() && this.isDeviceReady();
});
public blockerCount = computed(() => this.totalStepCount());
public passedBlockerCount = computed(() => this.passedStepCount());
public probes = computed(() => this.readinessReport()?.probes ?? []);
public activeDevice = computed(() => this.readinessReport()?.active_device ?? null);
public osType = computed<'linux' | 'darwin' | 'windows'>(() => {
const raw = this.readinessReport()?.os_type;
if (raw === 'windows' || raw === 'win32') return 'windows';
if (raw === 'darwin' || raw === 'macos' || raw === 'mac') return 'darwin';
return 'linux';
});
public skipCredentialsCheck(): void {
this.isSkipCredentialsCheck.set(true);
}
public setSkipCredentialsCheck(skip: boolean): void {
this.isSkipCredentialsCheck.set(skip);
}
// Device list from metadata
public connectedDevices = computed<DeviceInfo[]>(() => {
const meta = this.adbProbe()?.metadata;
if (meta && Array.isArray(meta['devices'])) {
return meta['devices'] as DeviceInfo[];
}
return [];
});
// Installed local AVD emulators from metadata
public installedAvds = computed<string[]>(() => {
const meta = this.adbProbe()?.metadata;
if (meta && Array.isArray(meta['installed_avds'])) {
return meta['installed_avds'] as string[];
}
return [];
});
public emulatorPath = computed<string | null>(() => {
return (this.adbProbe()?.metadata?.['emulator_path'] as string) || null;
});
public isEmulatorInPath = computed<boolean>(() => {
return (this.adbProbe()?.metadata?.['is_emulator_in_path'] as boolean) ?? true;
});
private pollingTimer: any = null;
constructor() {
this.fetchReadiness();
this.startAutoPolling(3000);
}
/**
* Start periodic auto-polling for system readiness changes
*/
public startAutoPolling(intervalMs: number = 3000): void {
if (this.pollingTimer) {
clearInterval(this.pollingTimer);
}
this.pollingTimer = setInterval(() => {
// Perform silent background check without disturbing UI loading state
this.fetchReadiness(true).subscribe({
error: () => {} // Silent catch
});
}, intervalMs);
}
/**
* Stop auto-polling
*/
public stopAutoPolling(): void {
if (this.pollingTimer) {
clearInterval(this.pollingTimer);
this.pollingTimer = null;
}
}
/**
* Fetch latest system readiness report from backend
* @param silent If true, updates signals silently without triggering global isLoading spinner
*/
public fetchReadiness(silent: boolean = false): Observable<SystemReadinessReport> {
if (!silent) {
this.isLoading.set(true);
}
return this.http.get<SystemReadinessReport>('/api/system/readiness').pipe(
tap({
next: (report) => {
this.readinessReport.set(report);
this.lastCheckedTime.set(new Date());
if (!silent) {
this.isLoading.set(false);
}
},
error: (err) => {
console.error('Failed to fetch system readiness:', err);
if (!silent) {
this.isLoading.set(false);
}
}
})
);
}
/**
* Fetch current emulator launch progress state snapshot
*/
public fetchEmulatorStatus(): Observable<EmulatorLaunchState> {
return this.http.get<EmulatorLaunchState>('/api/system/emulator/status').pipe(
tap({
next: (state) => {
this.emulatorLaunchState.set(state);
if (state.status === 'ready') {
this.launchingAvd.set(null);
this.stopEmulatorStatusPolling();
this.fetchReadiness().subscribe();
} else if (state.status === 'failed' || state.status === 'stopped' || state.status === 'idle') {
this.launchingAvd.set(null);
this.stopEmulatorStatusPolling();
} else {
if (state.avd_name) {
this.launchingAvd.set(state.avd_name);
}
}
},
error: (err) => {
console.error('Failed to fetch emulator status:', err);
}
})
);
}
/**
* Start 1s interval polling for emulator boot progression
*/
public startEmulatorStatusPolling(): void {
if (this.emulatorPollTimer) {
clearInterval(this.emulatorPollTimer);
}
this.emulatorPollTimer = setInterval(() => {
this.fetchEmulatorStatus().subscribe({
error: () => {}
});
}, 1000);
}
/**
* Stop polling for emulator boot status
*/
public stopEmulatorStatusPolling(): void {
if (this.emulatorPollTimer) {
clearInterval(this.emulatorPollTimer);
this.emulatorPollTimer = null;
}
}
/**
* Launch a local AVD emulator in the background and continuously track boot progress
*/
public launchEmulator(avdName: string): Observable<EmulatorLaunchState> {
this.launchingAvd.set(avdName);
this.emulatorLaunchState.set({
avd_name: avdName,
status: 'starting',
pid: null,
serial: null,
stage_message: 'Spawning emulator process...',
progress_percent: 15,
started_at: Date.now() / 1000,
elapsed_seconds: 0,
error: null,
logs: [`Initiating launch for AVD: ${avdName}...`],
can_retry: true,
});
return this.http.post<EmulatorLaunchState>('/api/system/emulator/launch', { avd_name: avdName }).pipe(
tap({
next: (state) => {
this.emulatorLaunchState.set(state);
if (state.status === 'failed') {
this.launchingAvd.set(null);
} else {
this.startEmulatorStatusPolling();
}
},
error: (err) => {
console.error('Failed to launch emulator:', err);
this.launchingAvd.set(null);
const errorMsg = err?.error?.detail || err?.message || 'Failed to start emulator process.';
this.emulatorLaunchState.set({
avd_name: avdName,
status: 'failed',
pid: null,
serial: null,
stage_message: 'Failed to initiate launch.',
progress_percent: 0,
started_at: null,
elapsed_seconds: 0,
error: errorMsg,
logs: [errorMsg],
can_retry: true,
});
}
})
);
}
/**
* Terminate running emulator
*/
public stopEmulator(): Observable<any> {
return this.http.post<any>('/api/system/emulator/stop', {}).pipe(
tap({
next: () => {
this.launchingAvd.set(null);
this.stopEmulatorStatusPolling();
this.fetchEmulatorStatus().subscribe();
this.fetchReadiness().subscribe();
}
})
);
}
/**
* Clear and dismiss emulator launch tracker state
*/
public dismissEmulatorStatus(): Observable<any> {
return this.http.post<any>('/api/system/emulator/dismiss', {}).pipe(
tap({
next: () => {
this.emulatorLaunchState.set(null);
this.launchingAvd.set(null);
this.stopEmulatorStatusPolling();
}
})
);
}
/**
* Restart local ADB server and refresh readiness state
*/
public restartAdb(): Observable<any> {
this.isRestartingAdb.set(true);
return this.http.post<any>('/api/system/adb/restart', {}).pipe(
tap({
next: (res) => {
if (res?.report) {
this.readinessReport.set(res.report);
this.lastCheckedTime.set(new Date());
}
this.isRestartingAdb.set(false);
},
error: (err) => {
console.error('Failed to restart ADB server:', err);
this.isRestartingAdb.set(false);
}
})
);
}
/**
* Connect to an Android device over Wi-Fi
*/
public connectWirelessAdb(host: string, port: number = 5555): Observable<any> {
return this.http.post<any>('/api/system/adb/connect', { host, port }).pipe(
tap({
next: (res) => {
if (res?.report) {
this.readinessReport.set(res.report);
this.lastCheckedTime.set(new Date());
}
}
})
);
}
/**
* Select a specific connected device serial as active target
*/
public selectDevice(serial: string): Observable<any> {
this.isLoading.set(true);
return this.http.post<any>('/api/system/devices/select', { serial }).pipe(
tap({
next: (res) => {
if (res?.report) {
this.readinessReport.set(res.report);
}
this.isLoading.set(false);
},
error: (err) => {
console.error('Failed to select active device:', err);
this.isLoading.set(false);
}
})
);
}
public currentApiKey = computed<string>(() => {
return (this.llmProbe()?.metadata?.['current_key'] as string) || '';
});
public apiKeysMap = computed<Record<string, string>>(() => {
return (this.llmProbe()?.metadata?.['api_keys'] as Record<string, string>) || {};
});
public modelConfigEnv = signal<ModelConfigEnvResponse | null>(null);
/**
* Fetch current model configuration (artemis.jsonc) and environment (.env) status
*/
public fetchModelConfigEnv(): Observable<ModelConfigEnvResponse> {
return this.http.get<ModelConfigEnvResponse>('/api/system/model-config-env').pipe(
tap({
next: (data) => {
this.modelConfigEnv.set(data);
},
error: (err) => {
console.error('Failed to fetch model config and env:', err);
}
})
);
}
/**
* Test and verify an API key against live provider endpoint without persisting
*/
public testApiKey(provider: string, apiKey: string, baseUrl?: string): Observable<{ valid: boolean; provider: string; message: string }> {
return this.http.post<{ valid: boolean; provider: string; message: string }>('/api/system/credentials/test', {
provider,
api_key: apiKey,
base_url: baseUrl
});
}
/**
* Update, verify, and configure API key for an LLM provider or Vision OCR
*/
public updateApiKey(provider: string, apiKey: string, persistToEnv: boolean = true): Observable<any> {
return this.http.post<any>('/api/system/credentials', {
provider,
api_key: apiKey,
persist_to_env: persistToEnv
}).pipe(
tap({
next: (res) => {
if (res?.report) {
this.readinessReport.set(res.report);
this.lastCheckedTime.set(new Date());
}
// Refresh model config & env after updating key
this.fetchModelConfigEnv().subscribe();
},
error: (err) => {
console.error(`Failed to update credentials for ${provider}:`, err);
}
})
);
}
}
export interface ModelConfigEnvResponse {
config_path: string;
config_filename: string;
config_content: string;
default_model: {
provider?: string;
model?: string;
thinking_level?: string;
fallback?: {
provider?: string;
model?: string;
thinking_level?: string;
};
};
presets: Record<string, {
provider: string;
model: string;
fallback?: { provider: string; model: string };
}>;
env_path: string;
env_filename: string;
env_vars: Array<{
name: string;
provider: string;
is_set: boolean;
preview: string | null;
description: string;
}>;
}
apps/showcase_ui/src/app/utils/action-formatter.util.spec.ts
+112
View File
@@ -0,0 +1,112 @@
import {
getStepPreImageUrl,
getStepPostImageUrl
} from './action-formatter.util';
describe('action-formatter.util screenshot chaining', () => {
it('should chain pre/post images correctly in FailureAnalyzer multi-step recovery', () => {
const stepData = {
step_id: 'step-06',
pre_image_name: '01a0e8d8cddbeb6a352e3bef1afbee0f0476a05b0fc62abb78e59c7b6aa16079',
post_image_name: '484361b1ab4b340415e01042e3496f0edc0549c837c897ece242835d12d7d71d',
action_taken: {
action: 'tap',
coordinates: [540, 720],
timestamp: 1787077538.0
},
last_execution_result: {
status: 'success',
repair_status: 'fixed',
execution: [
{
attempts: ['Pre-execution validation failed']
}
]
},
generic_tools: [
{
trace_id: 'tool-save-note',
name: 'save_note',
type: 'tool',
timestamp: 1787077537.0,
payload: { args: { title: 'Note 1' } }
},
{
trace_id: 'agent-failure-analyzer',
name: 'failure_analyzer',
agent_name: 'failure_analyzer',
type: 'agent',
timestamp: 1787077540.0,
payload: {
args: {
pre_screenshot_name: '01a0e8d8cddbeb6a352e3bef1afbee0f0476a05b0fc62abb78e59c7b6aa16079',
post_screenshot_name: 'fbdf17fa7a970d2c37aa52ae4336119e76a7d13ea457811cf8aaab19b3f53843'
}
}
},
{
trace_id: 'action-click-1',
name: 'click',
type: 'action',
agent_name: 'failure_analyzer',
timestamp: 1787077607.0,
payload: {
args: { target: '[500, 300]' },
result: {
outcome: 'Clicked successfully',
post_image_name: '18bd2ac49e64976f1e74e828ddae76e54da1587044b1d3fb01317dd202ff7c67'
}
}
},
{
trace_id: 'action-click-seq-2',
name: 'click_sequence',
type: 'action',
agent_name: 'failure_analyzer',
timestamp: 1787077631.0,
payload: {
args: { sequence: '[[500, 300], [876, 360]]' },
result: {
outcome: 'Sequence clicked successfully',
post_image_name: '484361b1ab4b340415e01042e3496f0edc0549c837c897ece242835d12d7d71d'
}
}
}
]
};
const action1 = stepData.generic_tools[2]; // click
const action2 = stepData.generic_tools[3]; // click_sequence
// Action 1: Pre should be the failed state screenshot (fbdf17fa...), Post should be Action 1's post (18bd2ac4...)
const act1Pre = getStepPreImageUrl(stepData, action1);
const act1Post = getStepPostImageUrl(stepData, action1);
expect(act1Pre).toBe('/images/fbdf17fa7a970d2c37aa52ae4336119e76a7d13ea457811cf8aaab19b3f53843');
expect(act1Post).toBe('/images/18bd2ac49e64976f1e74e828ddae76e54da1587044b1d3fb01317dd202ff7c67');
// Action 2: "下一次的决策前是上一次的动作后" -> Pre MUST be Action 1's post (18bd2ac4...), Post should be Action 2's post (484361b1...)
const act2Pre = getStepPreImageUrl(stepData, action2);
const act2Post = getStepPostImageUrl(stepData, action2);
expect(act2Pre).toBe('/images/18bd2ac49e64976f1e74e828ddae76e54da1587044b1d3fb01317dd202ff7c67');
expect(act2Post).toBe('/images/484361b1ab4b340415e01042e3496f0edc0549c837c897ece242835d12d7d71d');
});
it('should handle standard single-action step correctly', () => {
const stepData = {
step_id: 'step-01',
pre_image_name: 'pre123',
post_image_name: 'post456',
action_taken: {
action: 'tap',
coordinates: [100, 200],
timestamp: 1000
}
};
const preUrl = getStepPreImageUrl(stepData, stepData.action_taken);
const postUrl = getStepPostImageUrl(stepData, stepData.action_taken);
expect(preUrl).toBe('/images/pre123');
expect(postUrl).toBe('/images/post456');
});
});
src/app/utils/action-formatter.util.ts → apps/showcase_ui/src/app/utils/action-formatter.util.ts
+323 -17
View File
@@ -15,7 +15,8 @@
*/
import { ActionParam } from '../core/models/stream.model';
import { extractNumbersFromCoordinateValue, isPureDirectionString } from './image-overlay.util';
import { extractNumbersFromCoordinateValue, isPureDirectionString, parseSequenceCoordinates } from './image-overlay.util';
import { cleanErrorMessage } from './tool-formatter.util';
/**
* Safely extract action object from possible array or nested structure
@@ -43,6 +44,40 @@ export function isAndroidAction(action: any): boolean {
].includes(name);
}
/**
* Check if the action or tool is a task completion / submission report
*/
export function isReportStatusAction(action: any): boolean {
const act = getActionObject(action);
if (!act) return false;
const name = (act.name || act.action || '').toLowerCase().replace(/^(_)?(self\.)?exec_/, '');
return name === 'report_task_status' || name === 'report_status' || name === 'submit_task_status';
}
/**
* Get status of report_task_status (completed / failed)
*/
export function getReportStatusValue(action: any): string {
const act = getActionObject(action);
if (!act) return 'completed';
const args = act.args && typeof act.args === 'object' ? act.args : (act.payload?.args || act.payload || act);
const status = args.status || act.status;
if (typeof status === 'string') {
return status.toLowerCase();
}
return 'completed';
}
/**
* Get explanation of report_task_status
*/
export function getReportStatusExplanation(action: any): string {
const act = getActionObject(action);
if (!act) return '';
const args = act.args && typeof act.args === 'object' ? act.args : (act.payload?.args || act.payload || act);
return args.explanation || args.reason || args.summary || act.explanation || act.reason || act.summary || '';
}
/**
* Get Material Symbol icon name for Android action
*/
@@ -239,6 +274,15 @@ export function getActionCoords(action: any): string {
const act = getActionObject(action);
if (!act) return '';
const args = act.args && typeof act.args === 'object' ? act.args : {};
const name = (act.name || act.action || '').toLowerCase();
// Check sequence first if action is sequence or sequence arg is present
const isSequenceAction = name === 'click_sequence' || name === 'tap_sequence' || Boolean(act.sequence || args.sequence || act.normalized_sequence || args.normalized_sequence);
const rawSeq = act.normalized_sequence || args.normalized_sequence || act.sequence || args.sequence || act.targets || args.targets || (Array.isArray(act.coordinates) && Array.isArray(act.coordinates[0]) ? act.coordinates : null) || (Array.isArray(args.coordinates) && Array.isArray(args.coordinates[0]) ? args.coordinates : null);
const seqPoints = parseSequenceCoordinates(rawSeq);
if (seqPoints && seqPoints.length > 0 && (isSequenceAction || seqPoints.length > 1)) {
return seqPoints.map(pt => `[${pt[0]}, ${pt[1]}]`).join(' → ');
}
// Check normalized start and end first
const normStart = extractNumbersFromCoordinateValue(act.normalized_start_coordinates || args.normalized_start_coordinates);
@@ -280,30 +324,67 @@ export function getActionCoords(action: any): string {
}
/**
* Check if the action execution failed
* Check if the action execution failed or encountered an execution failure/interception
*/
export function isActionFailed(action: any, stepData?: any): boolean {
const act = getActionObject(action);
if (act) {
if (act.status === 'failed' || act.status === 'error' || act.success === false) return true;
}
if (stepData && stepData.last_execution_result) {
const res = stepData.last_execution_result;
if (res.status === 'failed' || res.status === 'error' || res.success === false || res.is_successful === false) return true;
if (stepData) {
if (stepData.status === 'failed' || stepData.status === 'error') return true;
if (stepData.last_execution_result) {
const res = typeof stepData.last_execution_result === 'string'
? (() => { try { return JSON.parse(stepData.last_execution_result); } catch { return null; } })()
: stepData.last_execution_result;
if (res && typeof res === 'object') {
if (res.status === 'failed' || res.status === 'error' || res.success === false || res.is_successful === false) {
return true;
}
if (res.repair_status === 'fixed' || res.repair_status === 'failed' || res.repair_status === 'cannot_fix') {
return true;
}
if (Array.isArray(res.execution) && res.execution.length > 0) {
const firstExec = res.execution[0];
if (firstExec && (firstExec.status === 'failed' || firstExec.status === 'error' || firstExec.error || (Array.isArray(firstExec.attempts) && firstExec.attempts.length > 0))) {
return true;
}
}
}
}
}
return false;
}
/**
* Get failure message for an action
* Get failure message or reason for an action
*/
export function getActionErrorMessage(action: any, stepData?: any): string {
const act = getActionObject(action);
if (act && (act.error || act.message)) return act.error || act.message;
if (act && (act.error || act.message || act.failure_reason)) {
return cleanErrorMessage(act.error || act.message || act.failure_reason);
}
if (stepData && stepData.last_execution_result) {
const res = stepData.last_execution_result;
const res = typeof stepData.last_execution_result === 'string'
? (() => { try { return JSON.parse(stepData.last_execution_result); } catch { return null; } })()
: stepData.last_execution_result;
if (res && typeof res === 'object') {
if (Array.isArray(res.execution) && res.execution.length > 0) {
const firstExec = res.execution[0];
if (firstExec) {
if (Array.isArray(firstExec.attempts) && firstExec.attempts.length > 0) {
return cleanErrorMessage(firstExec.attempts[0]);
}
if (firstExec.error || firstExec.failure_reason || firstExec.message) {
return cleanErrorMessage(firstExec.error || firstExec.failure_reason || firstExec.message);
}
}
}
if (res.error || res.message || res.failure_reason) {
return res.error || res.message || res.failure_reason;
return cleanErrorMessage(res.error || res.message || res.failure_reason);
}
}
}
return 'Action Failed';
@@ -437,24 +518,249 @@ export function formatImageUrl(candidate: any): string | null {
return `/images/${trimmed}`;
}
/**
* Helper to check if an item is a generic tool trace rather than the step's primary action
*/
export function isGenericToolTrace(actionData: any, stepData?: any): boolean {
if (!actionData) return false;
if (actionData === stepData?.action_taken) return false;
if (Array.isArray(stepData?.action_taken)) {
if (stepData.action_taken === actionData || stepData.action_taken.includes(actionData)) {
return false;
}
}
if (actionData.is_primary_step_action) return false;
return Boolean(actionData.trace_id || actionData.payload || actionData.type === 'tool' || (actionData.type === 'action' && actionData.trace_id));
}
interface ResolvedImageEntry {
pre: string | null;
post: string | null;
hasExplicitPost: boolean;
}
const stepImageMapCache = new WeakMap<any, { key: string; map: Map<any, ResolvedImageEntry> }>();
function safeParseJson(val: any): any {
if (typeof val === 'string' && (val.trim().startsWith('{') || val.trim().startsWith('['))) {
try {
return JSON.parse(val);
} catch {
return val;
}
}
return val;
}
export function extractItemPreImage(item: any): string | null {
const act = getActionObject(item);
if (!act) return null;
const payload = safeParseJson(act.payload);
const result = safeParseJson(act.result || payload?.result);
const args = safeParseJson(act.args || payload?.args);
const candidate =
result?.pre_image_name || result?.pre_screenshot_name || result?.pre_screenshot ||
payload?.pre_image_name || payload?.pre_screenshot_name || payload?.pre_screenshot ||
args?.pre_image_name || args?.pre_screenshot_name || args?.pre_screenshot ||
act.pre_image_name || act.pre_screenshot_name || act.pre_screenshot || act.before_screenshot || act.screenshot;
return candidate ? String(candidate) : null;
}
export function extractItemPostImage(item: any): string | null {
const act = getActionObject(item);
if (!act) return null;
const payload = safeParseJson(act.payload);
const result = safeParseJson(act.result || payload?.result);
const args = safeParseJson(act.args || payload?.args);
const candidate =
result?.post_image_name || result?.post_screenshot_name || result?.post_screenshot ||
payload?.post_image_name || payload?.post_screenshot_name || payload?.post_screenshot ||
args?.post_image_name || args?.post_screenshot_name || args?.post_screenshot ||
act.post_image_name || act.post_screenshot_name || act.post_screenshot || act.after_screenshot;
return candidate ? String(candidate) : null;
}
function getStepFailureScreenshot(stepData: any): string | null {
if (!stepData) return null;
if (stepData.failed_screenshot) return String(stepData.failed_screenshot);
if (stepData.failure_screenshot) return String(stepData.failure_screenshot);
if (stepData.failed_post_image_name) return String(stepData.failed_post_image_name);
if (Array.isArray(stepData.generic_tools)) {
for (const tool of stepData.generic_tools) {
const act = getActionObject(tool);
const payload = safeParseJson(act?.payload);
const args = safeParseJson(act?.args || payload?.args);
const name = String(act?.name || act?.agent_name || '').toLowerCase();
if (args?.post_screenshot_name) return String(args.post_screenshot_name);
if (args?.post_screenshot && (name.includes('failure') || name.includes('validator') || name.includes('diagnos'))) {
return String(args.post_screenshot);
}
if (act?.post_screenshot_name && (name.includes('failure') || name.includes('validator'))) {
return String(act.post_screenshot_name);
}
}
}
return null;
}
function getEventTimestamp(obj: any): number {
if (!obj) return 0;
const ts = obj.timestamp ?? obj.start_time ?? obj.created_at;
if (typeof ts === 'number') {
return ts < 1e11 ? ts * 1000 : ts;
}
if (typeof ts === 'string') {
const parsed = new Date(ts).getTime();
if (!isNaN(parsed) && parsed > 0) return parsed;
}
return 0;
}
export function resolveStepImageMap(stepData: any): Map<any, ResolvedImageEntry> {
const emptyMap = new Map<any, ResolvedImageEntry>();
if (!stepData || typeof stepData !== 'object') return emptyMap;
const toolsLen = Array.isArray(stepData.generic_tools) ? stepData.generic_tools.length : 0;
const cacheKey = `${stepData.step_id || ''}_${toolsLen}_${stepData.post_image_name || ''}_${stepData.pre_image_name || ''}`;
const cached = stepImageMapCache.get(stepData);
if (cached && cached.key === cacheKey) {
return cached.map;
}
const map = new Map<any, ResolvedImageEntry>();
const events: Array<{ item: any; act: any; timestamp: number; isPrimary: boolean }> = [];
if (stepData.action_taken) {
events.push({
item: stepData.action_taken,
act: getActionObject(stepData.action_taken),
timestamp: getEventTimestamp(stepData.action_taken),
isPrimary: true
});
}
if (Array.isArray(stepData.generic_tools)) {
stepData.generic_tools.forEach((t: any) => {
if (t) {
events.push({
item: t,
act: getActionObject(t),
timestamp: getEventTimestamp(t),
isPrimary: false
});
}
});
}
events.sort((a, b) => a.timestamp - b.timestamp);
const failureShot = getStepFailureScreenshot(stepData);
const hadPrimaryFailure = Boolean(failureShot) || (stepData.action_taken && isActionFailed(stepData.action_taken, stepData));
const hasSubsequentActions = events.some(e => !e.isPrimary && (isAndroidAction(e.act) || Boolean(extractItemPostImage(e.act))));
let lastActionItem: any = null;
for (let i = events.length - 1; i >= 0; i--) {
const e = events[i];
if (isAndroidAction(e.act) || extractItemPostImage(e.act) || e.isPrimary) {
lastActionItem = e.item;
break;
}
}
let currentPreImage = stepData.pre_image_name || stepData.pre_screenshot || null;
let switchedToFailureShot = false;
for (const e of events) {
const act = e.act;
const explicitPre = extractItemPreImage(act);
const explicitPost = extractItemPostImage(act);
const agentName = String(
act?.agent_name || act?.agent || safeParseJson(act?.payload)?.agent_name || act?.name || ''
).toLowerCase();
const isRecoveryOrValidator = agentName.includes('failure') || agentName.includes('analyzer') || agentName.includes('validator') || agentName.includes('diagnos');
if (failureShot && !switchedToFailureShot && (isRecoveryOrValidator || (!e.isPrimary && hadPrimaryFailure))) {
currentPreImage = failureShot;
switchedToFailureShot = true;
}
const resolvedPre = explicitPre || currentPreImage;
let resolvedPost: string | null = null;
let hasExplicitPost = false;
if (explicitPost) {
resolvedPost = explicitPost;
hasExplicitPost = true;
} else if (e.isPrimary && !hadPrimaryFailure && !hasSubsequentActions) {
resolvedPost = stepData.post_image_name || stepData.post_screenshot || null;
} else if (e.item === lastActionItem && !hadPrimaryFailure) {
resolvedPost = stepData.post_image_name || stepData.post_screenshot || null;
} else if (e.item === lastActionItem && isRecoveryOrValidator) {
resolvedPost = stepData.post_image_name || stepData.post_screenshot || null;
}
// "下一次的决策前是上一次的动作后"
if (resolvedPost) {
currentPreImage = resolvedPost;
}
const entry: ResolvedImageEntry = { pre: resolvedPre, post: resolvedPost, hasExplicitPost };
map.set(e.item, entry);
if (act && act !== e.item) map.set(act, entry);
if (act?.trace_id) map.set(act.trace_id, entry);
}
stepImageMapCache.set(stepData, { key: cacheKey, map });
return map;
}
function lookupResolvedImages(stepData: any, actionData?: any): ResolvedImageEntry {
const map = resolveStepImageMap(stepData);
if (!actionData) {
return {
pre: stepData?.pre_image_name || stepData?.pre_screenshot || null,
post: stepData?.post_image_name || stepData?.post_screenshot || null,
hasExplicitPost: Boolean(stepData?.post_image_name || stepData?.post_screenshot)
};
}
const act = getActionObject(actionData);
if (map.has(actionData)) return map.get(actionData)!;
if (act && map.has(act)) return map.get(act)!;
if (act?.trace_id && map.has(act.trace_id)) return map.get(act.trace_id)!;
const explicitPre = extractItemPreImage(act);
const explicitPost = extractItemPostImage(act);
const pre = explicitPre || stepData?.pre_image_name || stepData?.pre_screenshot || null;
const post = explicitPost || (isGenericToolTrace(actionData, stepData) ? null : (stepData?.post_image_name || stepData?.post_screenshot || null));
return { pre, post, hasExplicitPost: Boolean(explicitPost) };
}
/**
* Get pre-action screenshot URL
*/
export function getStepPreImageUrl(stepData: any, actionData?: any): string | null {
const act = getActionObject(actionData);
const candidate = act?.pre_image_name || act?.pre_screenshot || act?.before_screenshot || act?.screenshot || stepData?.pre_image_name || stepData?.pre_screenshot;
return formatImageUrl(candidate);
const resolved = lookupResolvedImages(stepData, actionData);
return formatImageUrl(resolved.pre);
}
/**
* Get post-action screenshot URL
*/
export function getStepPostImageUrl(stepData: any, actionData?: any): string | null {
const act = getActionObject(actionData);
const candidate = act?.post_image_name || act?.post_screenshot || act?.after_screenshot || stepData?.post_image_name || stepData?.post_screenshot;
const postUrl = formatImageUrl(candidate);
const preUrl = getStepPreImageUrl(stepData, actionData);
if (postUrl && preUrl && postUrl === preUrl) {
const resolved = lookupResolvedImages(stepData, actionData);
const postUrl = formatImageUrl(resolved.post);
const preUrl = formatImageUrl(resolved.pre);
if (postUrl && preUrl && postUrl === preUrl && !resolved.hasExplicitPost) {
return null;
}
return postUrl;
src/app/utils/image-overlay.util.ts → apps/showcase_ui/src/app/utils/image-overlay.util.ts
+189 -29
View File
@@ -77,6 +77,43 @@ export function extractNumbersFromCoordinateValue(val: any): number[] | null {
return null;
}
/**
* Parse sequence of coordinates (e.g. [[x1, y1], [x2, y2]] or stringified format)
*/
export function parseSequenceCoordinates(seq: any): number[][] | null {
if (!seq) return null;
if (typeof seq === 'string') {
const trimmed = seq.trim();
try {
const parsed = JSON.parse(trimmed);
const res = parseSequenceCoordinates(parsed);
if (res && res.length > 0) return res;
} catch {}
const pairs: number[][] = [];
const pairRegex = /\[\s*(-?\d+(?:\.\d+)?)\s*,\s*(-?\d+(?:\.\d+)?)\s*\]/g;
let m: RegExpExecArray | null;
while ((m = pairRegex.exec(trimmed)) !== null) {
pairs.push([Number(m[1]), Number(m[2])]);
}
if (pairs.length > 0) return pairs;
}
if (Array.isArray(seq)) {
const points: number[][] = [];
for (const item of seq) {
const p = extractNumbersFromCoordinateValue(item);
if (p && p.length === 2) {
points.push([p[0], p[1]]);
} else if (Array.isArray(item) && item.length >= 2 && typeof item[0] === 'number' && typeof item[1] === 'number') {
points.push([item[0], item[1]]);
}
}
if (points.length > 0) return points;
}
return null;
}
export function isPureDirectionString(str: any): boolean {
if (typeof str !== 'string') return false;
const s = str.trim().toLowerCase();
@@ -121,10 +158,42 @@ export function drawActionCoordinatesOnOverlay(
return;
}
// 1. Multi-point click sequence (High priority for click_sequence / tap_sequence)
const isSequenceAction = actionName.includes('sequence') || Boolean(act.sequence || args.sequence || act.normalized_sequence || args.normalized_sequence);
const rawSeq = act.normalized_sequence || args.normalized_sequence || act.sequence || args.sequence || act.targets || args.targets || (Array.isArray(act.coordinates) && Array.isArray(act.coordinates[0]) ? act.coordinates : null) || (Array.isArray(args.coordinates) && Array.isArray(args.coordinates[0]) ? args.coordinates : null);
const seqPoints = parseSequenceCoordinates(rawSeq);
if (seqPoints && seqPoints.length > 0 && (isSequenceAction || seqPoints.length > 1)) {
const isNormSeq = Boolean(act.normalized_sequence || args.normalized_sequence) || seqPoints.every(p => p[0] <= 1000 && p[1] <= 1000);
// Draw connecting lines between consecutive points
for (let i = 0; i < seqPoints.length - 1; i++) {
createSequenceConnector(
seqPoints[i][0], seqPoints[i][1],
seqPoints[i + 1][0], seqPoints[i + 1][1],
naturalWidth, naturalHeight,
overlay,
isNormSeq
);
}
// Draw numbered markers for each point in the sequence
seqPoints.forEach((pt, idx) => {
createPointMarker(
pt[0], pt[1],
actionName,
naturalWidth, naturalHeight,
overlay,
isNormSeq,
idx + 1,
seqPoints.length
);
});
return;
}
// Determine if this is a swipe/scroll/drag gesture
const isSwipeLike = actionName.includes('swipe') || actionName.includes('scroll') || actionName.includes('drag') || actionName.includes('slide');
// 1. Check separate start and end coordinate objects
// 2. Check separate start and end coordinate objects
const normStart = extractNumbersFromCoordinateValue(act.normalized_start_coordinates || args.normalized_start_coordinates);
const normEnd = extractNumbersFromCoordinateValue(act.normalized_end_coordinates || args.normalized_end_coordinates);
const startCoords = normStart || extractNumbersFromCoordinateValue(act.start_coordinates || act.start_point || act.start || act.from ||
@@ -144,7 +213,7 @@ export function drawActionCoordinatesOnOverlay(
return;
}
// 2. Check 4-element coordinate array in coordinates, target, action, gesture, etc.
// 3. Check 4-element coordinate array in coordinates, target, action, gesture, etc.
const normCoords = extractNumbersFromCoordinateValue(act.normalized_coordinates || args.normalized_coordinates);
const isNorm = Boolean(normCoords);
const rawCoords = normCoords ||
@@ -154,8 +223,6 @@ export function drawActionCoordinatesOnOverlay(
extractNumbersFromCoordinateValue(args.coords) ||
extractNumbersFromCoordinateValue(act.target) ||
extractNumbersFromCoordinateValue(args.target) ||
extractNumbersFromCoordinateValue(act.sequence) ||
extractNumbersFromCoordinateValue(args.sequence) ||
(isSwipeLike ? extractNumbersFromCoordinateValue(args.action) : null) ||
(isSwipeLike ? extractNumbersFromCoordinateValue(act.action) : null) ||
(isSwipeLike ? extractNumbersFromCoordinateValue(args.gesture) : null) ||
@@ -177,21 +244,7 @@ export function drawActionCoordinatesOnOverlay(
}
}
// Multi-point click sequence
const seq = act.sequence || args.sequence || (Array.isArray(act.target) && Array.isArray(act.target[0]) ? act.target : null);
if (seq && Array.isArray(seq)) {
let drawnAny = false;
seq.forEach((pt: any) => {
const p = extractNumbersFromCoordinateValue(pt);
if (p && p.length === 2) {
createPointMarker(p[0], p[1], actionName, naturalWidth, naturalHeight, overlay, isNorm);
drawnAny = true;
}
});
if (drawnAny) return;
}
// 3. Check for directional swipe/scroll/drag gestures ONLY when action is swipe-like
// 4. Check for directional swipe/scroll/drag gestures ONLY when action is swipe-like
if (isSwipeLike) {
const rawDir = (typeof args.action === 'string' && isPureDirectionString(args.action) ? args.action : null) ||
(typeof act.action === 'string' && act.action !== actionName && isPureDirectionString(act.action) ? act.action : null) ||
@@ -224,7 +277,7 @@ export function drawActionCoordinatesOnOverlay(
}
}
// 4. Default tap/click coordinate fallback
// 5. Default tap/click coordinate fallback
if (rawCoords && Array.isArray(rawCoords) && rawCoords.length === 2) {
createPointMarker(rawCoords[0], rawCoords[1], actionName, naturalWidth, naturalHeight, overlay, isNorm);
}
@@ -241,7 +294,9 @@ export function createPointMarker(
naturalWidth: number,
naturalHeight: number,
overlay: HTMLElement,
isNormalized: boolean = false
isNormalized: boolean = false,
sequenceIndex?: number,
totalSequence?: number
): void {
let pctX: number;
let pctY: number;
@@ -268,27 +323,42 @@ export function createPointMarker(
marker.style.pointerEvents = 'none';
marker.style.zIndex = '10';
const dotSize = sequenceIndex !== undefined ? 20 : 14;
const dot = document.createElement('div');
dot.style.width = '14px';
dot.style.height = '14px';
dot.style.width = `${dotSize}px`;
dot.style.height = `${dotSize}px`;
dot.style.borderRadius = '50%';
dot.style.backgroundColor = '#ef4444';
dot.style.border = '2px solid white';
dot.style.boxShadow = '0 0 6px rgba(0,0,0,0.5)';
if (sequenceIndex !== undefined) {
dot.style.display = 'flex';
dot.style.alignItems = 'center';
dot.style.justifyContent = 'center';
dot.style.fontSize = '11px';
dot.style.fontWeight = 'bold';
dot.style.color = 'white';
dot.innerText = String(sequenceIndex);
}
const pulseSize = dotSize * 2;
const pulse = document.createElement('div');
pulse.style.position = 'absolute';
pulse.style.top = '-9px';
pulse.style.left = '-9px';
pulse.style.width = '28px';
pulse.style.height = '28px';
pulse.style.top = `-${dotSize / 2 + 2}px`;
pulse.style.left = `-${dotSize / 2 + 2}px`;
pulse.style.width = `${pulseSize}px`;
pulse.style.height = `${pulseSize}px`;
pulse.style.borderRadius = '50%';
pulse.style.border = '2px solid #ef4444';
pulse.style.animation = 'pulse-animation 1.5s infinite ease-out';
pulse.style.pointerEvents = 'none';
const label = document.createElement('div');
if (sequenceIndex !== undefined) {
label.innerText = totalSequence && totalSequence > 1 ? `#${sequenceIndex} TAP` : actionType.toUpperCase();
} else {
label.innerText = actionType.toUpperCase();
}
label.style.position = 'absolute';
label.style.backgroundColor = 'rgba(15, 23, 42, 0.9)';
label.style.color = 'white';
@@ -303,10 +373,10 @@ export function createPointMarker(
// Smart edge-aware vertical positioning
if (clampedPctY > 85) {
label.style.bottom = '16px';
label.style.bottom = `${dotSize + 2}px`;
label.style.top = 'auto';
} else {
label.style.top = '16px';
label.style.top = `${dotSize + 2}px`;
label.style.bottom = 'auto';
}
@@ -331,6 +401,96 @@ export function createPointMarker(
overlay.appendChild(marker);
}
/**
* Helper to draw connecting line/arrow between points in a click sequence
*/
export function createSequenceConnector(
x1: number,
y1: number,
x2: number,
y2: number,
naturalWidth: number,
naturalHeight: number,
overlay: HTMLElement,
isNormalized: boolean = false
): void {
const isNormCoord = isNormalized || (x1 <= 1000 && y1 <= 1000 && x2 <= 1000 && y2 <= 1000);
const pctX1 = isNormCoord ? x1 / 10 : (x1 / (naturalWidth || 1080)) * 100;
const pctY1 = isNormCoord ? y1 / 10 : (y1 / (naturalHeight || 2400)) * 100;
const pctX2 = isNormCoord ? x2 / 10 : (x2 / (naturalWidth || 1080)) * 100;
const pctY2 = isNormCoord ? y2 / 10 : (y2 / (naturalHeight || 2400)) * 100;
const clampedX1 = Math.max(1.5, Math.min(98.5, pctX1));
const clampedY1 = Math.max(1.5, Math.min(98.5, pctY1));
const clampedX2 = Math.max(1.5, Math.min(98.5, pctX2));
const clampedY2 = Math.max(1.5, Math.min(98.5, pctY2));
let svg = overlay.querySelector('svg');
if (!svg) {
svg = document.createElementNS('http://www.w3.org/2000/svg', 'svg');
svg.setAttribute('width', '100%');
svg.setAttribute('height', '100%');
svg.style.position = 'absolute';
svg.style.top = '0';
svg.style.left = '0';
svg.style.pointerEvents = 'none';
svg.style.overflow = 'visible';
svg.style.zIndex = '5';
overlay.appendChild(svg);
}
let defs = svg.querySelector('defs');
if (!defs) {
defs = document.createElementNS('http://www.w3.org/2000/svg', 'defs');
svg.insertBefore(defs, svg.firstChild);
}
const markerId = 'seq-arrow-' + Math.random().toString(36).substring(2, 9);
const marker = document.createElementNS('http://www.w3.org/2000/svg', 'marker');
marker.setAttribute('id', markerId);
marker.setAttribute('viewBox', '0 0 12 12');
marker.setAttribute('refX', '8');
marker.setAttribute('refY', '6');
marker.setAttribute('markerWidth', '6');
marker.setAttribute('markerHeight', '6');
marker.setAttribute('orient', 'auto');
const path = document.createElementNS('http://www.w3.org/2000/svg', 'path');
path.setAttribute('d', 'M 1 2 L 10 6 L 1 10 z');
path.setAttribute('fill', '#ef4444');
path.setAttribute('stroke', '#ffffff');
path.setAttribute('stroke-width', '1');
marker.appendChild(path);
defs.appendChild(marker);
// Background white contrast line
const bgLine = document.createElementNS('http://www.w3.org/2000/svg', 'line');
bgLine.setAttribute('x1', `${clampedX1}%`);
bgLine.setAttribute('y1', `${clampedY1}%`);
bgLine.setAttribute('x2', `${clampedX2}%`);
bgLine.setAttribute('y2', `${clampedY2}%`);
bgLine.setAttribute('stroke', '#ffffff');
bgLine.setAttribute('stroke-width', '4');
bgLine.setAttribute('stroke-linecap', 'round');
bgLine.setAttribute('opacity', '0.8');
// Main red dashed sequence line
const line = document.createElementNS('http://www.w3.org/2000/svg', 'line');
line.setAttribute('x1', `${clampedX1}%`);
line.setAttribute('y1', `${clampedY1}%`);
line.setAttribute('x2', `${clampedX2}%`);
line.setAttribute('y2', `${clampedY2}%`);
line.setAttribute('stroke', '#ef4444');
line.setAttribute('stroke-width', '2.5');
line.setAttribute('stroke-linecap', 'round');
line.setAttribute('marker-end', `url(#${markerId})`);
line.setAttribute('stroke-dasharray', '4,4');
svg.appendChild(bgLine);
svg.appendChild(line);
}
/**
* Helper to create and position an SVG arrow, high-contrast dashed line, and badge for swipes/drags
*/
src/app/utils/markdown-parser.util.ts → apps/showcase_ui/src/app/utils/markdown-parser.util.ts
+116 -32
View File
@@ -178,6 +178,27 @@ export function extractCheckerResult(text: string): CheckerResult | null {
return null;
}
/**
* Helper to format inline markdown segments (bold, italic, code, strikethrough)
*/
function formatInlineMarkdown(text: string): string {
if (!text) return '';
return text
// Inline code first (so asterisks inside code aren't formatted)
.replace(/`([^`]+)`/g, '<code class="inline-code">$1</code>')
// Bold + Italic
.replace(/\*\*\*([^*]+)\*\*\*/g, '<strong><em>$1</em></strong>')
.replace(/___([^_]+)___/g, '<strong><em>$1</em></strong>')
// Bold
.replace(/\*\*([^*]+)\*\*/g, '<strong>$1</strong>')
.replace(/__([^_]+)__/g, '<strong>$1</strong>')
// Italic
.replace(/\*([^*]+)\*/g, '<em>$1</em>')
.replace(/\b_([^_]+)_\b/g, '<em>$1</em>')
// Strikethrough
.replace(/~~([^~]+)~~/g, '<del>$1</del>');
}
/**
* Convert basic markdown text to safe HTML for agent logs/thinking
*/
@@ -187,69 +208,132 @@ export function renderMarkdownToHtml(text: string): string {
// Clean up internal agent XML-like tags (e.g. <thought>, <short_term_memory>, etc.)
const cleanedText = text
.replace(/<\/?(thought|short_term_memory|strong_term_memory|reasoning|plan|task_plan|call_tool)[^\n>]*>?/gi, '')
.replace(/\r\n/g, '\n')
.replace(/\r/g, '\n')
.replace(/\n{3,}/g, '\n\n')
.trim();
// 1. Escape HTML to prevent XSS
let html = cleanedText
// Escape HTML first to prevent XSS
const escaped = cleanedText
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
// 2. Bold: **text** -> <strong>text</strong>
html = html.replace(/\*\*(.*?)\*\*/g, '<strong>$1</strong>');
// 3. Inline code: `code` -> <code class="inline-code">code</code>
html = html.replace(/`(.*?)`/g, '<code class="inline-code">$1</code>');
// 4. Lines processing for lists
const lines = html.split('\n');
let output = '';
const rawLines = escaped.split('\n');
const result: string[] = [];
let currentListType: 'ul' | 'ol' | null = null;
let inCodeBlock = false;
let codeBlockLang = '';
let codeBlockLines: string[] = [];
for (const line of lines) {
const trimmed = line.trim();
const ulMatch = line.match(/^(\s*)([\*\-])\s+(.*)$/);
const olMatch = line.match(/^(\s*)(\d+)\.\s+(.*)$/);
const closeList = () => {
if (currentListType) {
result.push(`</${currentListType}>`);
currentListType = null;
}
};
for (let i = 0; i < rawLines.length; i++) {
const rawLine = rawLines[i];
const trimmed = rawLine.trim();
// 1. Code blocks: ```lang ... ```
if (trimmed.startsWith('```')) {
if (inCodeBlock) {
// End code block
const codeContent = codeBlockLines.join('\n');
result.push(`<pre class="code-block"><code class="lang-${codeBlockLang}">${codeContent}</code></pre>`);
inCodeBlock = false;
codeBlockLines = [];
codeBlockLang = '';
} else {
closeList();
inCodeBlock = true;
codeBlockLang = trimmed.substring(3).trim();
codeBlockLines = [];
}
continue;
}
if (inCodeBlock) {
codeBlockLines.push(rawLine);
continue;
}
// 2. Headings: # to ######
const headingMatch = trimmed.match(/^(#{1,6})\s+(.*)$/);
if (headingMatch) {
closeList();
const level = headingMatch[1].length;
const headingContent = formatInlineMarkdown(headingMatch[2]);
result.push(`<h${level}>${headingContent}</h${level}>`);
continue;
}
// 3. Horizontal Rule: ---, ***, ___
if (/^(\-{3,}|\*{3,}|_{3,})$/.test(trimmed)) {
closeList();
result.push('<hr class="md-hr" />');
continue;
}
// 4. Blockquotes: > quote
if (trimmed.startsWith('&gt; ') || trimmed.startsWith('&gt;')) {
closeList();
const quoteContent = formatInlineMarkdown(trimmed.replace(/^&gt;\s?/, ''));
result.push(`<blockquote>${quoteContent}</blockquote>`);
continue;
}
// 5. Unordered List: - item or * item
const ulMatch = rawLine.match(/^(\s*)([\*\-])\s+(.*)$/);
if (ulMatch) {
if (currentListType !== 'ul') {
if (currentListType) output += `</${currentListType}>\n`;
closeList();
currentListType = 'ul';
output += '<ul>\n';
result.push('<ul>');
}
const spaces = ulMatch[1].length;
const subLevel = Math.max(0, Math.floor((spaces - 2) / 2));
const indent = subLevel * 12;
const style = indent ? ` style="margin-left: ${indent}px"` : '';
output += `<li${style}>${ulMatch[3]}</li>\n`;
} else if (olMatch) {
const itemContent = formatInlineMarkdown(ulMatch[3]);
result.push(`<li${style}>${itemContent}</li>`);
continue;
}
// 6. Ordered List: 1. item
const olMatch = rawLine.match(/^(\s*)(\d+)\.\s+(.*)$/);
if (olMatch) {
if (currentListType !== 'ol') {
if (currentListType) output += `</${currentListType}>\n`;
closeList();
currentListType = 'ol';
output += '<ol>\n';
result.push('<ol>');
}
const spaces = olMatch[1].length;
const subLevel = Math.max(0, Math.floor((spaces - 2) / 2));
const indent = subLevel * 12;
const style = indent ? ` style="margin-left: ${indent}px"` : '';
output += `<li${style}>${olMatch[3]}</li>\n`;
} else {
if (currentListType) {
output += `</${currentListType}>\n`;
currentListType = null;
const itemContent = formatInlineMarkdown(olMatch[3]);
result.push(`<li${style}>${itemContent}</li>`);
continue;
}
// 7. Regular paragraph / empty line
closeList();
if (trimmed === '') {
output += '<div class="md-para-gap"></div>';
result.push('<div class="md-para-gap"></div>');
} else {
output += `<div>${line}</div>\n`;
}
const lineContent = formatInlineMarkdown(rawLine);
result.push(`<div>${lineContent}</div>`);
}
}
if (currentListType) {
output += `</${currentListType}>\n`;
if (inCodeBlock) {
const codeContent = codeBlockLines.join('\n');
result.push(`<pre class="code-block"><code class="lang-${codeBlockLang}">${codeContent}</code></pre>`);
}
return output;
closeList();
return result.join('\n');
}
src/app/utils/stream-aggregator.util.ts → apps/showcase_ui/src/app/utils/stream-aggregator.util.ts
+54 -15
View File
@@ -15,7 +15,7 @@
*/
import { StepBlock, PhaseBlock, StepEvent } from '../core/models/stream.model';
import { isAndroidAction } from './action-formatter.util';
import { isAndroidAction, isReportStatusAction, getReportStatusExplanation, getReportStatusValue } from './action-formatter.util';
import { getUniqueGenericTools } from './tool-formatter.util';
/**
@@ -121,7 +121,10 @@ export function consolidateLogsToBlocks(rawLogs: any[]): StepBlock[] {
const newTools = log.data.generic_tools || [];
const mergedTools = [...existingTools];
newTools.forEach((nt: any) => {
if (!mergedTools.some(et => et.trace_id && nt.trace_id && et.trace_id === nt.trace_id)) {
const matchIdx = mergedTools.findIndex(et => et.trace_id && nt.trace_id && et.trace_id === nt.trace_id);
if (matchIdx > -1) {
mergedTools[matchIdx] = { ...mergedTools[matchIdx], ...nt };
} else {
mergedTools.push(nt);
}
});
@@ -157,22 +160,37 @@ export function consolidateLogsToBlocks(rawLogs: any[]): StepBlock[] {
});
}
} else if (log.type === 'trace_recorded') {
const isTool = log.data.type === 'tool';
const isAction = log.data.type === 'action';
const isTool = log.data.type === 'tool' || isAction;
const isFailedLLM = log.data.type === 'llm_call' && log.data.status === 'failed';
if (!isTool && !isFailedLLM) return;
let stepId = log.data.step_id;
let existingIndex = -1;
if (stepId) {
// 1. If this trace_id already exists in some block, update that block directly
if (log.data.trace_id) {
existingIndex = blocks.findIndex(b =>
b.data?.generic_tools?.some((t: any) => t.trace_id === log.data.trace_id)
);
}
// 2. If not found by trace_id and stepId is present, find by step_id
if (existingIndex === -1 && stepId) {
existingIndex = blocks.findIndex(b => b.id === `step-${stepId}` || b.data?.step_id === stepId);
}
// 3. If still not found and no stepId (e.g. pre-planning / initial planning phase)
if (existingIndex === -1 && !stepId) {
existingIndex = blocks.findIndex(b => b.id === 'step-pre-planning' || b.data?.step_id === 'pre-planning');
}
// 4. Otherwise find by timestamp among existing step blocks
if (existingIndex === -1) {
const logTime = log.timestamp ? new Date(log.timestamp).getTime() : 0;
let maxStepTime = -1;
for (let i = 0; i < blocks.length; i++) {
const b = blocks[i];
if (b.id !== 'step-pre-planning') {
const bTime = b.timestamp ? new Date(b.timestamp).getTime() : 0;
if (bTime <= logTime && bTime >= maxStepTime) {
maxStepTime = bTime;
@@ -180,21 +198,30 @@ export function consolidateLogsToBlocks(rawLogs: any[]): StepBlock[] {
}
}
}
}
if (existingIndex > -1) {
const stepData = blocks[existingIndex].data;
const genericTools = stepData.generic_tools || [];
if (!genericTools.some((t: any) => t.trace_id === log.data.trace_id)) {
const updatedTools = [...genericTools, log.data];
const genericTools = stepData.generic_tools ? [...stepData.generic_tools] : [];
const existingToolIdx = log.data.trace_id
? genericTools.findIndex((t: any) => t.trace_id === log.data.trace_id)
: -1;
if (existingToolIdx > -1) {
genericTools[existingToolIdx] = {
...genericTools[existingToolIdx],
...log.data
};
} else {
genericTools.push(log.data);
}
blocks[existingIndex] = {
...blocks[existingIndex],
data: {
...stepData,
generic_tools: updatedTools
generic_tools: genericTools
}
};
}
} else {
const fallbackStepId = stepId || 'pre-planning';
blocks.push({
@@ -399,7 +426,7 @@ export function getSortedStepEvents(
const fallbackTime = getItemTimestamp(stepData.timestamp, 0);
// 1. Add action_taken if present and valid
if (stepData.action_taken && isAndroidAction(stepData.action_taken)) {
if (stepData.action_taken && (isAndroidAction(stepData.action_taken) || isReportStatusAction(stepData.action_taken))) {
const actTime = getItemTimestamp(
stepData.action_taken.timestamp ?? stepData.action_taken.start_time ?? stepData.action_taken.created_at,
fallbackTime
@@ -441,19 +468,31 @@ export function getSortedStepEvents(
export function compileSessionSummary(logs: any[]): string {
if (!logs || logs.length === 0) return 'No logs available.';
// Check for goal completed / checker response
// Check for goal completed / checker response / cancellation / report_task_status
for (let i = logs.length - 1; i >= 0; i--) {
const log = logs[i];
if (log.type === 'session_ended') {
if (log.data?.status === 'cancelled' || log.data?.was_stopped_manually) {
return 'Task stopped manually.';
}
}
if (log.type === 'llm_stream' && log.data?.text) {
const text = log.data.text;
if (text.includes('```json') && text.includes('"success"')) {
return 'Execution completed and verified by checker.';
}
}
if (log.type === 'step_recorded' && log.data?.status === 'completed') {
if (log.type === 'step_recorded' || log.type === 'step_updated') {
const act = log.data?.action_taken;
if (act && isReportStatusAction(act)) {
const exp = getReportStatusExplanation(act);
if (exp) return exp;
}
if (log.data?.status === 'completed') {
return log.data.message || 'Task completed successfully.';
}
}
}
return 'Execution session in progress or ended.';
}
@@ -486,7 +525,7 @@ export function checkPlanningLoader(
if (log.type === 'session_ended') return false;
if (log.type === 'step_recorded' || log.type === 'step_updated') {
const act = log.data?.action_taken;
if (act && (act.action === 'report_task_status' || act === 'report_task_status')) {
if (act && (act.action === 'report_task_status' || act === 'report_task_status' || isReportStatusAction(act))) {
return false;
}
if (log.data?.status === 'completed' || log.data?.status === 'failed') {
src/app/utils/tool-formatter.util.ts → apps/showcase_ui/src/app/utils/tool-formatter.util.ts
+145 -34
View File
@@ -15,8 +15,8 @@
*/
import { ActionParam } from '../core/models/stream.model';
import { isAndroidAction } from './action-formatter.util';
import { extractNumbersFromCoordinateValue, isPureDirectionString } from './image-overlay.util';
import { isAndroidAction, isReportStatusAction } from './action-formatter.util';
import { extractNumbersFromCoordinateValue, isPureDirectionString, parseSequenceCoordinates } from './image-overlay.util';
/**
* Extract arguments from tool payload or direct args field
@@ -64,7 +64,7 @@ export function isVideoTool(tool: any): boolean {
if (!tool || !tool.name) return false;
const cleanName = tool.name.replace(/^(_)?(self\.)?exec_/, '');
const nameLower = cleanName.toLowerCase();
return ['video_analyzer', 'video_analyzer_pure', 'extract_segment_metadata', 'spawn_sub_agent', 'analyze_audio_only'].includes(nameLower);
return ['video_analyzer', 'video_analyzer_pure'].includes(nameLower);
}
/**
@@ -92,45 +92,93 @@ export function getVideoToolTarget(tool: any): string | null {
}
/**
* Check if a tool should be rendered in Action Card format (ONLY real Android device operations and ADB commands)
* Canonical list of mobile device automation and ADB command tool names.
*/
export function isFailureAnalyzerActionTool(tool: any): boolean {
if (!tool || !tool.name) return false;
if (tool.type === 'llm_call') return false;
const cleanName = tool.name.replace(/^(_)?(self\.)?exec_/, '').toLowerCase();
const realActionTools = [
export const DEVICE_ACTION_TOOL_NAMES = new Set([
'click', 'click_sequence', 'long_press', 'long_press_on', 'input', 'input_text',
'swipe', 'scroll', 'press_key', 'press_home', 'press_back', 'launch_app', 'manage_app', 'open_app',
'focus_and_input_text', 'focus_and_clear_text', 'tap', 'wait_for_delay', 'wait',
'run_adb_command', 'run_short_adb_command'
];
return realActionTools.includes(cleanName);
]);
/**
* Check if a tool represents a real mobile device action or ADB command.
*/
export function isDeviceActionTool(tool: any): boolean {
if (!tool || !tool.name) return false;
if (tool.type === 'llm_call') return false;
const cleanName = tool.name.replace(/^(_)?(self\.)?exec_/, '').toLowerCase();
return DEVICE_ACTION_TOOL_NAMES.has(cleanName);
}
/**
* Check if a generic tool should be displayed in the UI
* Backward-compatible alias for isDeviceActionTool.
*/
export function isFailureAnalyzerActionTool(tool: any): boolean {
return isDeviceActionTool(tool);
}
/**
* Framework-internal infrastructure tools that are not intended for end-user timeline display.
*/
export function isInternalPlumbingTool(tool: any): boolean {
if (!tool || !tool.name) return true;
const cleanName = tool.name.replace(/^(_)?(self\.)?exec_/, '').toLowerCase();
if (cleanName.includes('safety_net')) return true;
if (cleanName === 'report_task_status' || cleanName === 'report_status' || cleanName === 'submit_task_status') {
return true;
}
return false;
}
/**
* Determines whether a tool invocation should be displayed in the timeline.
*/
export function shouldShowTool(tool: any, stepData?: any): boolean {
if (!tool || !tool.name) return false;
if (tool.type === 'llm_call') return false;
const cleanName = tool.name.replace(/^(_)?(self\.)?exec_/, '');
const nameLower = cleanName.toLowerCase();
if (nameLower.includes('safety_net')) return false;
if (tool.type === 'agent') return false;
if (isInternalPlumbingTool(tool)) return false;
// If step has an android action card rendered for action_taken, hide redundant generic tool text for atomic actions
const atomicActions = [
'click', 'click_sequence', 'long_press', 'long_press_on', 'input', 'input_text',
'swipe', 'scroll', 'press_key', 'press_home', 'press_back', 'launch_app', 'manage_app', 'open_app',
'focus_and_input_text', 'focus_and_clear_text', 'tap', 'wait_for_delay', 'wait'
];
if (stepData && stepData.action_taken && isAndroidAction(stepData.action_taken) && atomicActions.includes(nameLower)) {
// If this tool is an operator/validator atomic action that duplicates stepData.action_taken, skip it
if (stepData && stepData.action_taken && isAndroidAction(stepData.action_taken) && isDeviceActionTool(tool)) {
const agentName = (tool.agent_name || tool.agent || '').toLowerCase();
const isSelfHealingOrSubAgent = agentName.includes('failure') ||
agentName.includes('analyzer') ||
agentName.includes('diagnos') ||
(agentName && agentName !== 'operator' && agentName !== 'flashrunner' && agentName !== 'validator');
if (!isSelfHealingOrSubAgent) {
return false;
}
}
return true;
}
/**
* Extracts human-readable initiating agent name / role for a tool.
*/
export function getToolAgentName(tool: any): string | null {
if (!tool) return null;
const name = (tool.agent_name || tool.agent || '').toLowerCase();
if (name.includes('failure') || name.includes('analyzer')) {
return null; // Omit self-healing label per user instruction
}
if (name.includes('outputter')) {
return 'Outputter';
}
if (name.includes('validator')) {
return 'Validator';
}
if (name.includes('diagnos')) {
return 'Diagnoser';
}
if (name.includes('explorer')) {
return 'Explorer';
}
return null;
}
/**
* Filter out nested child wrapper duplicates while preserving independent executions at the same level
*/
@@ -145,11 +193,13 @@ export function getUniqueGenericTools(tools: any[] | undefined): any[] {
}
const cleanName = (nameStr: string) => (nameStr || '').replace(/^(_)?(self\.)?exec_/, '').toLowerCase().trim();
const hasMainVideoAnalyzer = tools.some(t => t && (cleanName(t.name) === 'video_analyzer' || cleanName(t.name) === 'video_analyzer_pure'));
const result: any[] = [];
const seenTraces = new Set<string>();
for (const tool of tools) {
if (!tool || !tool.name) continue;
if (tool.type === 'agent') continue;
if (tool.type === 'llm_call' && tool.status === 'failed') {
result.push(tool);
@@ -158,11 +208,6 @@ export function getUniqueGenericTools(tools: any[] | undefined): any[] {
const currClean = cleanName(tool.name);
// Hide internal video sub-agent worker tools if main video analyzer trace is already present
if (hasMainVideoAnalyzer && ['extract_segment_metadata', 'spawn_sub_agent', 'analyze_audio_only'].includes(currClean)) {
continue;
}
// Check if this tool is a nested child execution of a parent tool with the exact same tool name
let isNestedChildDuplicate = false;
if (tool.parent_trace_id) {
@@ -173,9 +218,17 @@ export function getUniqueGenericTools(tools: any[] | undefined): any[] {
}
if (!isNestedChildDuplicate) {
if (tool.trace_id) {
if (!seenTraces.has(tool.trace_id)) {
seenTraces.add(tool.trace_id);
// Always use the latest state from toolMap for this trace_id
result.push(toolMap.get(tool.trace_id) || tool);
}
} else {
result.push(tool);
}
}
}
return result;
}
@@ -294,6 +347,31 @@ export function getToolDisplayLabel(tool: any, isFirstSaveNote: boolean = false)
case 'video_analyzer':
case 'video_analyzer_pure':
return 'Analyzing screen recording';
case 'extract_segment_metadata': {
const start = args.start_time !== undefined ? `${args.start_time}s` : '';
const end = args.end_time !== undefined ? `${args.end_time}s` : '';
const range = (start && end) ? ` (${start} - ${end})` : (start ? ` (from ${start})` : '');
return `Cropping screen recording segment${range}`;
}
case 'spawn_sub_agent': {
const q = args.specific_query || args.query || args.prompt || '';
return q ? `Analyzing recording with sub-agent: "${q}"` : 'Analyzing recording with sub-agent';
}
case 'analyze_audio_only': {
const q = args.specific_query || args.query || '';
return q ? `Analyzing audio track: "${q}"` : 'Analyzing recording audio track';
}
case 'get_step_details':
return 'Retrieving step execution details';
case 'get_step_screenshot':
return 'Inspecting screenshot evidence';
case 'search_history_for_text': {
const q = args.text || args.query || '';
return q ? `Searching execution trace for "${q}"` : 'Searching execution trace';
}
case 'outputter':
case 'output_synthesis':
return 'Synthesizing output report';
case 'web_search': {
const q = args.query || '';
return q ? `Searching web for "${q}"` : 'Searching the web';
@@ -356,6 +434,21 @@ export function getToolIcon(tool: any): string {
case 'video_analyzer':
case 'video_analyzer_pure':
return 'video_camera_back';
case 'extract_segment_metadata':
return 'crop';
case 'spawn_sub_agent':
return 'smart_toy';
case 'analyze_audio_only':
return 'graphic_eq';
case 'get_step_details':
return 'manage_search';
case 'get_step_screenshot':
return 'image_search';
case 'search_history_for_text':
return 'find_in_page';
case 'outputter':
case 'output_synthesis':
return 'assignment_turned_in';
default:
return 'build';
}
@@ -429,6 +522,12 @@ export function getToolTitle(tool: any): string {
case 'video_analyzer':
case 'video_analyzer_pure':
return 'Analyzing Screen Recording';
case 'extract_segment_metadata':
return 'Cropping Screen Recording';
case 'spawn_sub_agent':
return 'Delegating Video Analysis';
case 'analyze_audio_only':
return 'Analyzing Audio Track';
default:
return cleanName.replace(/_/g, ' ').replace(/\b\w/g, (c: string) => c.toUpperCase());
}
@@ -546,6 +645,16 @@ export function getToolInputText(tool: any): string {
export function getToolCoords(tool: any): string {
if (!tool || !tool.name) return '';
const args = getToolArgs(tool);
const cleanName = tool.name.replace(/^(_)?exec_/, '');
const name = cleanName.toLowerCase();
// Check sequence first if action is sequence or sequence arg is present
const isSequenceAction = name === 'click_sequence' || name === 'tap_sequence' || Boolean(args.sequence || args.normalized_sequence);
const rawSeq = args.normalized_sequence || args.sequence || args.targets || (Array.isArray(args.coordinates) && Array.isArray(args.coordinates[0]) ? args.coordinates : null);
const seqPoints = parseSequenceCoordinates(rawSeq);
if (seqPoints && seqPoints.length > 0 && (isSequenceAction || seqPoints.length > 1)) {
return seqPoints.map(pt => `[${pt[0]}, ${pt[1]}]`).join(' → ');
}
// Check normalized start and end first
const normStart = extractNumbersFromCoordinateValue(args.normalized_start_coordinates || args.start_coordinates || args.start);
@@ -572,10 +681,7 @@ export function getToolCoords(tool: any): string {
}
if (args.x !== undefined && args.y !== undefined) {
return `${args.x}, ${args.y}`;
}
if (args.sequence && Array.isArray(args.sequence)) {
return JSON.stringify(args.sequence);
return `[${args.x}, ${args.y}]`;
}
return '';
}
@@ -839,7 +945,12 @@ export function cleanErrorMessage(rawError: any): string {
if (fallback.includes('[type.googleapis.com')) {
fallback = fallback.split('[type.googleapis.com')[0];
}
fallback = fallback.replace(/^ServerError:\s*/i, '').trim();
fallback = fallback
.replace(/^Pre-execution validation failed:\s*/i, '')
.replace(/^Pixel-level validation failed:\s*/i, '')
.replace(/^Execution error:\s*/i, '')
.replace(/^ServerError:\s*/i, '')
.trim();
return fallback || errorStr;
}
src/index.html → apps/showcase_ui/src/index.html
+9 -1
View File
@@ -21,7 +21,15 @@
<title>artemis</title>
<base href="/">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="icon" type="image/svg+xml" href="logo.svg">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Material+Symbols+Outlined:opsz,wght,FILL,GRAD@20..48,100..700,0..1,-50..200" />
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700;800&family=Outfit:wght@300;400;500;600;700&family=Roboto:wght@400;500;700&display=swap" />
<link rel="icon" type="image/png" sizes="192x192" href="/logo.png?v=2">
<link rel="icon" type="image/png" sizes="32x32" href="/logo.png?v=2">
<link rel="icon" type="image/svg+xml" href="/logo.svg?v=2">
<link rel="shortcut icon" href="/favicon.ico?v=2">
<link rel="apple-touch-icon" href="/logo.png?v=2">
</head>
<body>
<app-root></app-root>
src/main.ts → apps/showcase_ui/src/main.ts
View File
File renamed without changes.
src/styles.scss → apps/showcase_ui/src/styles.scss
+13 -6
View File
@@ -14,11 +14,16 @@
* limitations under the License.
*/
@import url('https://fonts.googleapis.com/css2?family=Outfit:wght@300;400;500;600;700&display=swap');
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700;800&family=Outfit:wght@300;400;500;600;700&family=Roboto:wght@400;500;700&display=swap');
@import url('https://fonts.googleapis.com/css2?family=Material+Symbols+Outlined:opsz,wght,FILL,GRAD@20..48,100..700,0..1,-50..200');
*, *::before, *::after {
box-sizing: border-box;
font-family: 'Google Sans', 'Google Sans Text', 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif;
}
.material-symbols-outlined {
font-family: 'Material Symbols Outlined';
font-family: 'Material Symbols Outlined' !important;
font-weight: normal;
font-style: normal;
font-size: 20px;
@@ -36,13 +41,15 @@
font-variation-settings: 'FILL' 0, 'wght' 300, 'GRAD' 0, 'opsz' 20;
}
body, input, button, select, textarea, h1, h2, h3, h4, h5, h6, span, p, div, summary, details, li, a {
font-family: 'Google Sans', 'Google Sans Text', 'Inter', Arial, "Liberation Sans", sans-serif;
color: #1f1f1f;
body, input, button, select, textarea {
font-family: 'Google Sans', 'Google Sans Text', 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif;
}
body {
margin: 0;
padding: 0;
background-color: #fafafa;
color: #1e293b;
background-color: #f8f9fa;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
}
tsconfig.app.json → apps/showcase_ui/tsconfig.app.json
View File
File renamed without changes.
tsconfig.json → apps/showcase_ui/tsconfig.json
View File
File renamed without changes.
tsconfig.spec.json → apps/showcase_ui/tsconfig.spec.json
View File
File renamed without changes.
artemis/agents/checker/checker.json
+1 -1
View File
@@ -1,4 +1,4 @@
{
"checker_system": "You are a precise and reliable verification expert (Checker). Your task is to determine whether the Operator has successfully completed the active subgoal by combining visual evidence from the current screenshot with data stored in persistent notes.",
"checker_prompt": "# Inputs\n- Active Subgoal: {{ subgoal }}\n- Execution History:\n{{ plan_and_history }}\n- Dialogue History: {{ dialogue_history }}\n- Visual State: The provided screenshot.\n\n# Guidelines\n- **Visual + Data Verification**: You must combine the screenshot with note contents. If the subgoal involves extracting, saving, or recording information (e.g., \"save contacts\", \"record address\"), you **MUST** verify that the content actually exists in the note. Do not assume success just because the information is visible on the screen. If the signs of success are not obvious, you can check the history, especially the most recent steps, to see if the operator performed their own check and whether that check was reasonable, to determine if the subgoal was truly successful.\n- **Visual Priority (Anti-History-Bias)**: Your primary ground truth is the visual evidence on the **Current Screenshot**. The `Execution History` lists previous successes or failures, which may be outdated due to dynamic background recovery or self-healing. If the screenshot demonstrably shows the active subgoal's target state is achieved, you **MUST** mark `success: true`, regardless of previous failed steps. Trust what you see, not past FAILED texts.\n- **Constraint Verification**: For subgoals containing specific parameters (such as names, brands, topics, events, numbers, or dates), confirm that the visible text precisely reflects these specified details. A successful state is defined by a clear, direct match with the subgoal's constraints.\n- **Realistic Expectations**: You cannot scroll the screen. If a subgoal requires saving all items but only some are visible on the screen, verify that at least the **visible items** are recorded in the note. As long as the visible part is recorded and there are no obvious errors, consider it a success. Do not fail the task simply because off-screen content cannot be verified.\n- **Success Criteria**: Set `success` to `true` if the core objective of the subgoal is met and it does not block subsequent steps. Set `success` to `false` if the state is incorrect, required data was not saved, or the operator is stuck/looping.\n\n# Output Protocol\nInvoke the `submit_verification` tool with `success` (boolean) and `reason` (string) to submit your final decision. If you need to inspect note data first, use `read_note`."
"checker_prompt": "# Inputs\n- Active Subgoal: {{ subgoal }}\n- Execution History:\n{{ plan_and_history }}\n- Dialogue History: {{ dialogue_history }}\n- Visual State: The provided screenshot.\n\n# Guidelines\n- **Visual + Data Verification**: You must combine the screenshot with note contents. If the subgoal involves extracting, saving, or recording information (e.g., \"save contacts\", \"record address\"), you **MUST** verify that the content actually exists in the note. Do not assume success just because the information is visible on the screen. If the signs of success are not obvious, you can check the history, especially the most recent steps, to see if the operator performed their own check and whether that check was reasonable, to determine if the subgoal was truly successful.\n- **Visual Priority (Anti-History-Bias)**: Your primary ground truth is the visual evidence on the **Current Screenshot**. The `Execution History` lists previous successes or failures, which may be outdated due to dynamic background recovery or self-healing. If the screenshot demonstrably shows the active subgoal's target state is achieved, you **MUST** mark `success: true`, regardless of previous failed steps. Trust what you see, not past FAILED texts.\n- **Constraint Verification**: For subgoals containing specific parameters (such as names, brands, topics, events, numbers, or dates), confirm that the visible text precisely reflects these specified details. A successful state is defined by a clear, direct match with the subgoal's constraints.\n- **Iteration & Monitoring Subgoals Validation**: For concrete iteration sub-tasks (e.g., inspecting Candidate #1) or periodic polling subgoals (e.g., Polling Round #1 check), verify whether the Operator correctly executed the step and logged the progress/findings into notes or the task plan. Do NOT mark a monitoring or iteration step as failed simply because the overall monitoring session is ongoing or no new incoming events were detected in that specific polling cycle. As long as the discrete check was performed and recorded, mark `success: true`.\n- **Realistic Expectations**: You cannot scroll the screen. If a subgoal requires saving all items but only some are visible on the screen, verify that at least the **visible items** are recorded in the note. As long as the visible part is recorded and there are no obvious errors, consider it a success. Do not fail the task simply because off-screen content cannot be verified.\n- **Success Criteria**: Set `success` to `true` if the core objective of the subgoal is met and it does not block subsequent steps. Set `success` to `false` if the state is incorrect, required data was not saved, or the operator is stuck/looping.\n\n# Output Protocol\nInvoke the `submit_verification` tool with `success` (boolean) and `reason` (string) to submit your final decision. If you need to inspect note data first, use `read_note`."
}
artemis/agents/checker/checker.py
+5 -4
View File
@@ -42,7 +42,7 @@ from artemis.tools.tool_wrapper import (
)
from artemis.utils.logger import get_logger
from artemis.utils.notes import get_note_file_path, get_notes_dir
from artemis.utils.ocr_api import perform_ocr
from artemis.utils.ocr_api import is_ocr_configured, perform_ocr
from artemis.utils.ocr_xml_fusion import (
_crop_image_remove_status_bar,
_detect_status_bar_height,
@@ -143,6 +143,7 @@ async def run_async_check(
height = device_data.height
ocr_results = []
if is_ocr_configured():
try:
status_bar_height = _detect_status_bar_height(xml_hierarchy, height)
if status_bar_height > 0:
@@ -154,7 +155,7 @@ async def run_async_check(
else:
ocr_results = await perform_ocr(screenshot_b64)
except Exception as e:
logger.error(f"Checker OCR failed, proceeding without OCR: {e}")
logger.warning(f"Checker OCR failed, proceeding without OCR: {e}")
fused_xml = fuse_ocr_with_xml(xml_hierarchy, ocr_results)
@@ -356,10 +357,10 @@ async def run_async_check(
tool_status = "success"
try:
if ":" in tool_name:
tool_to_run = get_tool_by_name(tool_name, traced_tools)
tool_to_run = get_tool_by_name(tool_name, tools_to_bind)
else:
tool_to_run = next(
(t for t in traced_tools if t.name == tool_name),
(t for t in tools_to_bind if t.name == tool_name),
None,
)
if tool_to_run:
artemis/agents/diagnoser/diagnoser.json
+1
View File
@@ -11,6 +11,7 @@
"- **No Direct Operations**: Do not perform user actions or modify device state (e.g., tap, swipe, text input, BACK/HOME keys, or force-starting activities). Focus solely on analysis.",
"- **Tool Constraints**: Video analysis is expensive. When calling the video analyzer, you MUST provide a specific, narrow timeframe to minimize cost.",
"- **Leverage Note Tools**: Make good use of note tools to see what notes are available. When diagnosing, open and cross-reference relevant notes to ensure you have grasped all necessary information and context.",
"- **Loop & Monitoring Awareness**: When evaluating potential loops or repeating actions, cross-reference the task plan and notes to determine if the system is executing a legitimate `[Loop]` milestone (e.g., candidate iteration, periodic list polling with wait intervals, or continuous swipes for battery discharge). Do NOT misdiagnose planned iterations or paced polling checks as bugs or screen freezes.",
"- **Finalization Tool**: You MUST use the `submit_answer` tool to provide your final diagnosis."
],
"checker_feedback_template": [
artemis/agents/explorer/constants.py
+2 -2
View File
@@ -32,14 +32,14 @@ EXPLORE_DESCRIPTIONS = {
"pro": {
"description": (
"[EXPLORER] Call this to ask the UI Explorer agent to locate"
" coordinates on the screen layout by executing XML, OCR,"
" coordinates on the screen layout by executing UI search,"
" coordinate search, and object detection."
),
"query_description": (
"The target element or information to search for, including descriptions."
),
"rule_prompt": (
"Call the explorer tool to search, OCR, or inspect it (maximum {max_iterations} tries)."
"Call the explorer tool to search or inspect it (maximum {max_iterations} tries)."
),
"version_prompt": (
"You are running with a maximum of {max_iterations} turns: on your"
artemis/agents/explorer/explorer.py
+32 -18
View File
@@ -53,7 +53,7 @@ from artemis.mcp.xml_search_server import (
search_ui as search_ui_func,
)
from artemis.utils.logger import get_logger
from artemis.utils.ocr_api import perform_ocr
from artemis.utils.ocr_api import is_ocr_configured, perform_ocr
from artemis.utils.ocr_xml_fusion import fuse_ocr_with_xml
from artemis.utils.visualization import draw_dots, format_minimal_list_with_points
@@ -73,7 +73,7 @@ UNIVERSAL_EXPLORER_TOOLS = [
"properties": {
"search_query": {
"type": "string",
"description": "Required. Text query to search for in XML/OCR elements.",
"description": "Required. Text query to search for in UI elements.",
},
"nx": {
"type": "integer",
@@ -234,8 +234,8 @@ class Explorer:
" in parallel, including database search, coordinate search,"
" and visual object detection on the screen.\n\nOutput"
" Format:\nReturns:'text' (str): A single string consolidating"
" results with 0-1000 normalized coordinates, including xml"
" search matches labeled [X], ocr search matches labeled [O],"
" results with 0-1000 normalized coordinates, including UI"
" search matches labeled [X],"
" coordinate search matches labeled [X], and visually detected"
" object targets labeled [D].\n(Annotated screenshots are also"
" added to the context containing corresponding colored label"
@@ -246,7 +246,7 @@ class Explorer:
properties={
"search_query": types.Schema(
type=types.Type.STRING,
description=("Required. Text query to search for in XML/OCR elements."),
description=("Required. Text query to search for in UI elements."),
),
"nx": types.Schema(
type=types.Type.INTEGER,
@@ -463,10 +463,10 @@ class Explorer:
llm_config = getattr(ctx, "llm_config", None)
llm_cfg = getattr(llm_config, "explorer", None) if llm_config else None
model_str = (
llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.6-flash"
llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.7-flash"
).lower()
self.model_name = (
llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.6-flash"
llm_cfg.model if (llm_cfg and hasattr(llm_cfg, "model")) else "gemini-3.7-flash"
)
if "/" in self.model_name:
self.model_name = self.model_name.split("/")[-1]
@@ -514,8 +514,11 @@ class Explorer:
]
def get_exposed_tools(self, only_submit: bool = False) -> list[types.FunctionDeclaration]:
"""Returns the list of tools, filtering out denylisted ones."""
tools = [tool for tool in self.TOOLS if tool.name not in self.denylisted_tools]
"""Returns the list of tools, filtering out denylisted and unconfigured ones."""
denylisted = set(self.denylisted_tools)
if not is_ocr_configured():
denylisted.add("get_ocr_list")
tools = [tool for tool in self.TOOLS if tool.name not in denylisted]
if only_submit:
tools = [tool for tool in tools if tool.name == "submit_answer"]
return tools
@@ -855,7 +858,7 @@ class Explorer:
xml_text = results[0].get("text", "No matches found.")
if xml_text and xml_text != "No matches found.":
text_parts.append(f"XML/OCR Text Search Results are: {xml_text}")
text_parts.append(f"Text Search Results are: {xml_text}")
coord_text = results[1].get("text", "No elements found.")
if coord_text and coord_text != "No elements found.":
@@ -1012,7 +1015,7 @@ class Explorer:
record = storage.get_ui_record(self.image_name)
if not record or not record.ocr_result:
return {
"text": ("No OCR results found in Data Engine for the current screen."),
"text": "No text elements detected on the screen.",
"image_path": None,
}
@@ -1143,11 +1146,12 @@ class Explorer:
"""Executes Explorer reasoning loop via Universal LangChain ChatModel."""
llm = get_llm(self.ctx, name="explorer")
# Filter universal tools based on denylist
# Filter universal tools based on denylist and OCR configuration
denylisted = set(self.denylisted_tools)
if not is_ocr_configured():
denylisted.add("get_ocr_list")
exposed_tools = [
t
for t in UNIVERSAL_EXPLORER_TOOLS
if t["function"]["name"] not in self.denylisted_tools
t for t in UNIVERSAL_EXPLORER_TOOLS if t["function"]["name"] not in denylisted
]
bound_llm = llm.bind_tools(exposed_tools)
@@ -1447,6 +1451,9 @@ class Explorer:
"inspect_region",
}
if not is_ocr_configured():
version_denylist.add("get_ocr_list")
try:
denylisted_config = (
self.ctx.agent_config.denylisted_tools.get("explorer", [])
@@ -1517,7 +1524,7 @@ class Explorer:
if "/" in fallback_model:
fallback_model = fallback_model.split("/")[-1]
else:
model_name = "gemini-3.6-flash"
model_name = "gemini-3.7-flash"
# 5. Construct Prompt & Initial Message List
prompt_path = Path(__file__).parent / "explorer.json"
@@ -1567,13 +1574,20 @@ class Explorer:
ui_tree = record.ui_tree
ocr_results = getattr(record, "ocr_result", None)
if ocr_results is None:
if is_ocr_configured():
try:
logger.info("Previous screenshot OCR is missing. Running OCR on-the-fly...")
logger.info(
"Previous screenshot OCR is missing. Running OCR on-the-fly..."
)
with open(screenshot_path, "rb") as img_file:
img_b64 = base64.b64encode(img_file.read()).decode("utf-8")
ocr_results = await perform_ocr(img_b64, client=self.http_client)
except Exception as ocr_err:
logger.error(f"On-the-fly OCR failed for previous screenshot: {ocr_err}")
logger.error(
f"On-the-fly OCR failed for previous screenshot: {ocr_err}"
)
ocr_results = []
else:
ocr_results = []
fused_xml = fuse_ocr_with_xml(ui_tree, ocr_results or [])
artemis/agents/flash/runner.py
+5 -4
View File
@@ -22,6 +22,7 @@ import asyncio
import base64
import json
from pathlib import Path
import re
import uuid
from jinja2 import Template
@@ -439,13 +440,13 @@ class FlashRunner:
elif kind == "direction" and isinstance(target_val, str):
g_lower = target_val.lower()
if "up" in g_lower:
norm_coords = [600, 800, 600, 200]
norm_coords = [600, 700, 600, 300]
elif "down" in g_lower:
norm_coords = [600, 200, 600, 800]
norm_coords = [600, 300, 600, 700]
elif "left" in g_lower:
norm_coords = [800, 500, 200, 500]
norm_coords = [750, 500, 250, 500]
elif "right" in g_lower:
norm_coords = [200, 500, 800, 500]
norm_coords = [250, 500, 750, 500]
if norm_coords:
norm_start = norm_coords[:2]
norm_end = norm_coords[2:]
artemis/agents/image_processor/image_processor.md
+1 -1
View File
@@ -16,7 +16,7 @@ Your goal is to perform image analysis, filter, crop, draw annotations/markers,
2. **DO NOT** use `cv2.resize()` to scale the image.
Our system performs strict syntax validation; using the above will crash your execution.
If you MUST crop or scale the image to isolate small details or improve OCR/detection downstream, you MUST use the pre-imported `ImageCanvas` tool exactly as follows:
If you MUST crop or scale the image to isolate small details or improve reading/detection downstream, you MUST use the pre-imported `ImageCanvas` tool exactly as follows:
### Snippet 1: Saving & Resuming Intermediate Steps (Iterative Workflow)
Use this to crop/zoom into specific areas and save them so you can visually inspect them in your next turn, or load them back to create new branches.
artemis/agents/operator/operator.json
+2 -2
View File
@@ -1,4 +1,4 @@
{
"main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List/Text/Complex UI Exploration Strategy (Zero Omission, Zero Duplication, Single-Pass Rule)**:\n When traversing large lists, long text bodies, or continuous content streams, adhere strictly to the **Single-Pass Exploration Standard**:\n 1. **Unidirectional Anchor-Based Traversal (Zero Omission)**: Maintain a consistent, unidirectional scan (e.g., scrolling steadily from top to bottom). Before each scroll, use the bottom-most visible item as a visual anchor. Ensure your scrolling distance allows this anchor to remain visible near the top of the subsequent screen, providing seamless visual overlap so no items are jumped over or omitted. Conclude exploration when reaching the definite boundary (e.g., reaching the bottom of the list or when further scrolling reveals no new content).\n 2. **Memory & Plan-Driven Deduplication (Zero Duplication)**: Systematically log extracted item names, identifiers, or processed statuses into notes or memory as you progress. Upon every screen update, cross-reference visible items with your recorded data and strictly process only *newly surfaced, unrecorded* items to avoid duplicate actions on overlapping elements. Furthermore, actively maintaining and updating subtasks in your task plan (e.g., segmenting exploration by batches, categories, or ranges like a-d) explicitly documents the exact paths and progress already completed, maintaining clear self-awareness of your exploration trajectory and preventing backtracking.\n 3. **Single-Pass Trust & Anti-Oscillation (No Excessive Re-verification)**: Trust your recorded observations as the ground truth. Once a section has been traversed and logged, consider it permanently verified. Never scroll back anLine truncated
"troubleshooter_template": "# ROLE\nYou are the Expert Android Device Troubleshooter. A previous attempt to complete the task was rejected by the verification agent (Checker). Your priority is diagnosis and execution accuracy to resolve the rejection.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List/Text/Complex UI Exploration Strategy (Zero Omission, Zero Duplication, Single-Pass Rule)**:\n When traversing large lists, long text bodies, or continuous content streams, adhere strictly to the **Single-Pass Exploration Standard**:\n 1. **Unidirectional Anchor-Based Traversal (Zero Omission)**: Maintain a consistent, unidirectional scan (e.g., scrolling steadily from top to bottom). Before each scroll, use the bottom-most visible item as a visual anchor. Ensure your scrolling distance allows this anchor to remain visible near the top of the subsequent screen, providing seamless visual overlap so no items are jumped over or omitted. Conclude exploration when reaching the definite boundary (e.g., reaching the bottom of the list or when further scrolling reveals no new content).\n 2. **Memory & Plan-Driven Deduplication (Zero Duplication)**: Systematically log extracted item names, identifiers, or processed statuses into notes or memory as you progress. Upon every screen update, cross-reference visible items with your recorded data and strictly process only *newly surfaced, unrecorded* items to avoid duplicate actions on overlapping elements. Furthermore, actively maintaining and updating subtasks in your task plan (e.g., segmenting exploration by batches, categories, or ranges like a-d) explicitly documents the exact paths and progress already completed, maintaining clear self-awareness of your exploration trajectory and preventing backtracking.\n 3. **Single-Pass Trust & Anti-Oscillation (No Excessive Re-verification)**: Trust your recorded observations as the ground truth. Once a section has been traversed and loggedLine truncated
"main_template": "# ROLE\nYou are the Android Device Operator. Your priority is execution accuracy. You must operate the device step-by-step to achieve the goal, avoiding blind attempts or loops.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor. Calibrate your scroll distance so that this anchor remains visible near the top of the subsequent screen, creating a seamless visual overlap that guarantees zero missed items. Terminate exploration when a definitive boundary is reached (e.g., list bottom reached or no new content appears after scrolling).\n 2. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n 3. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\". Only perform incremental re-verification if the list has been refreshed or contentLine truncated
"troubleshooter_template": "# ROLE\nYou are the Expert Android Device Troubleshooter. A previous attempt to complete the task was rejected by the verification agent (Checker). Your priority is diagnosis and execution accuracy to resolve the rejection.\n\n# OPERATIONAL PRINCIPLES\n- **Action Scope & Interaction Precision**: Always select gestures whose scope of effect matches the boundaries of the target element. Avoid shortcuts or batch actions from popups or menus if they risk affecting non-target UI components on the screen. For any task requiring continuous range interactions, sliding, or boundary selection, prefer using targeted swipe/drag gestures spanning from the element's start bounds [xmin, ymin] to end bounds [xmax, ymax] to ensure the action is strictly confined to the target element. For drag-and-drop, list reordering, or sliding sliders/SeekBars, set duration >= 1000 (e.g. 1500) and always drag slightly PAST the target position to overcome touch slop and reliably trigger the drop/snap/value update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.\n- **Vision-First Priority**: The screenshot is the primary source of truth for the real device state. Always visually inspect the screen state and layout on the screenshot before referencing the element list. Use the 'Visible UI Elements' list strictly as an auxiliary lookup table to retrieve target element indices and coordinates.\n- **Tool Calling Iron Rule**: Do NOT submit a Turn-Ending Action at the same time (in the same tool-call list) as a Pre-Decision Exploratory Tool. You must first gather information (e.g., finding coordinates or diagnosing a state), manage ADB background tasks, or update memory notes. You can use pre-decision exploratory tools to complete this task. You will be prompted again with the tool's result, at which point you can output your final physical/turn-ending action.\n- **Loading & Transitions**: When entering page transitions (e.g., app launching, web page loading, or progress indicators), use `wait_for_delay` to let the screen settle before executing target actions. Some transition states are difficult to discern, such as a thin progress line at the top of the browser screen during webpage loads. Before questioning whether an error has occurred, you can first adopt a waiting strategy to see how the device state evolves. If the state remains ambiguous after waiting, certain UI components might be malfunctioning or failing to load; in this case, consider retrying the action or pivoting to an alternative path.\n- **Memory Preservation & Paginated Exploration**: Using note tools is entirely zero-cost. Actively use this tool to maintain goals and logs. The note tool is the only reliable way to persist memory. Therefore, when you find information on the screen that might be relevant to the task, even if you are not certain, you should first save the information currently visible on the screen before exploring further. If you have already recorded some information, you can use `append_note` to append the new information to the existing content. The note tool can also help you organize lists or information across multiple screens; you can consolidate all information into notes for unified processing.\n- **Large List & Long Text Exploration Strategy (Zero-Miss, Zero-Duplication Single-Pass Principle)**: When traversing large lists, browsing long text bodies, or scanning continuous content feeds, strictly adhere to the single-pass exploration standard:\n 1. **Unidirectional Anchor-Based Traversal (Zero-Miss)**: Maintain a consistent, unidirectional scan (e.g., uniform downward scrolling). Before each scroll, identify the bottom-most visible item/text segment as your visual anchor. Calibrate your scroll distance so that this anchor remains visible near the top of the subsequent screen, creating a seamless visual overlap that guarantees zero missed items. Terminate exploration when a definitive boundary is reached (e.g., list bottom reached or no new content appears after scrolling).\n 2. **Memory & Plan-Driven Deduplication (Zero-Duplication)**: Systematically log extracted item names, identifiers, or processing states into notes or memory as you progress. After every screen transition, cross-reference visible items against your recorded ledger and strictly interact only with newly surfaced, unrecorded items to eliminate redundant actions on overlapping elements. Proactively maintain subtasks in `task_plan` (e.g., segmenting by batch, category, or alphabetical ranges like `a-d`), tracking exact paths and progress to prevent backtracking.\n 3. **Single-Pass Trust & Anti-Oscillation (No Over-Verification)**: Treat recorded notes as verified ground truth. Once a segment is traversed and logged, consider it permanently resolved. Do NOT scroll back-and-forth or perform redundant re-scans for \"double-checking\". Only perform incremental rLine truncated
}
artemis/agents/operator/operator.py
+49 -71
View File
@@ -46,7 +46,10 @@ from artemis.tools.tool_wrapper import (
get_tool_result_content,
invoke_tool_with_injection,
)
from artemis.utils.coordinates import parse_swipe_parameters
from artemis.utils.coordinates import (
compute_smart_swipe_coordinates,
parse_swipe_parameters,
)
from artemis.utils.decorators import wrap_with_callbacks
from artemis.utils.file import create_snapshot
from artemis.utils.logger import get_logger
@@ -1056,23 +1059,25 @@ class OperatorNode:
def swipe(
direction: Annotated[
Literal["up", "down", "left", "right"] | None,
"Smart directional swipe ('up', 'down', 'left', 'right') for general page"
" scrolling/browsing. Automatically computes safe swipe vectors (locking axes"
" to prevent sidebar/edge mis-clicks), retains a ~40% visual overlap anchor,"
" and prevents inertial flings: 'up' (drags bottom-to-top, scrolls down to reveal content below),"
" 'down' (drags top-to-bottom, scrolls up to reveal content above),"
" 'left' (drags right-to-left, scrolls right), 'right' (drags left-to-right, scrolls left).",
"Direction for scrolling and swiping: 'up' (drags bottom-to-top, scrolling down to reveal content below),"
" 'down' (drags top-to-bottom, scrolling up to reveal content above),"
" 'left' (drags right-to-left, scrolling right),"
" 'right' (drags left-to-right, scrolling left).",
] = None,
start: Annotated[
list[int] | None,
"Start point normalized coordinates [start_x, start_y] in 0-1000 scale for precise,"
"Start normalized coordinates [start_x, start_y] in 0-1000 scale for precise,"
" local interactions (e.g. adjusting sliders, SeekBars, fine range selection, or drag-and-drop).",
] = None,
end: Annotated[
list[int] | None,
"End point normalized coordinates [end_x, end_y] in 0-1000 scale for precise,"
"End normalized coordinates [end_x, end_y] in 0-1000 scale for precise,"
" local interactions (e.g. adjusting sliders, SeekBars, fine range selection, or drag-and-drop).",
] = None,
target: Annotated[
int | list[int] | str | None,
"Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within.",
] = None,
gesture: Annotated[
Literal["up", "down", "left", "right"] | list[int] | None,
"Backward-compatible swipe gesture: smart direction string ('up', 'down', 'left', 'right')"
@@ -1080,23 +1085,23 @@ class OperatorNode:
] = None,
duration: Annotated[
int | None,
"Optional swipe/drag duration in milliseconds. If specified, the gesture will run at this exact speed."
" For drag-and-drop, list reordering, or sliding/adjusting sliders (e.g., volume, brightness, SeekBars),"
" set duration >= 1000 (e.g. 1500). If omitted for directional swipe, the system automatically"
" calculates the best duration based on the swipe distance to prevent unintended inertial flings.",
"Optional swipe/drag duration in milliseconds (default 800). For drag-and-drop,"
" list reordering, or sliding/adjusting sliders (e.g., volume, brightness, SeekBars),"
" set duration >= 1000 (e.g. 1500). If omitted for directional swipe, duration is computed automatically.",
] = None,
):
"""[ACTION] Perform a swipe, drag, or slider-adjustment gesture on the screen.
• Smart Directional Scrolling ('direction'): Best for general browsing and page scrolling. Automatically computes safe swipe vectors (locking axes to prevent edge/sidebar mis-clicks), retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings.
• Precise Coordinate Gestures ('start', 'end'): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars) or drag-and-drop / list reordering. Drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), always swipe fully to the extreme boundary to guarantee reaching the absolute limits.
• Directional Scrolling ('direction'): Recommended for general browsing and standard page scrolling in most scenarios. Automatically computes safe swipe vectors and adaptive duration, retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings. Supports scoping to a sub-container via 'target'. If it fails on certain custom layouts, fall back to specifying exact coordinates ('start' and 'end') directly.
• Precise Coordinate Gestures ('start', 'end'): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars), drag-and-drop / list reordering, or as a reliable fallback when directional scrolling fails on specific containers. Always drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), swipe fully to the extreme boundary.
Args:
direction: Smart directional scrolling ('up', 'down', 'left', 'right'). Automatically computes safe swipe vectors, retaining 40% of the previous screen's visible content as an overlapping anchor: 'up' (drags bottom-to-top, scrolling down to reveal content below), 'down' (drags top-to-bottom, scrolling up to reveal content above), 'left' (drags right-to-left), 'right' (drags left-to-right).
start: Start normalized coordinates [start_x, start_y] in 0-1000 scale for precise/local interactions.
end: End normalized coordinates [end_x, end_y] in 0-1000 scale for precise/local interactions.
direction: Smart directional scrolling ('up', 'down', 'left', 'right'). Automatically computes safe swipe vectors, retaining 40% visual overlap: 'up' (reveal content below), 'down' (reveal content above), 'left', 'right'.
start: Start normalized coordinates [start_x, start_y] in 0-1000 scale.
end: End normalized coordinates [end_x, end_y] in 0-1000 scale.
target: Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within.
gesture: Backward-compatible parameter: direction string OR custom coordinates list [start_x, start_y, end_x, end_y] in 0-1000 scale.
duration: Optional gesture duration in milliseconds. If omitted for directional swipe, the system automatically computes a reliable duration based on the swipe distance to prevent unintended inertial flings.
duration: Optional gesture duration in milliseconds (default 800).
"""
return "Action Recorded"
@@ -1134,16 +1139,16 @@ class OperatorNode:
def wait_for_delay(
time_in_ms: Annotated[
int,
"The duration to wait in milliseconds (e.g., 2000 for 2"
" seconds, 5000 for 5 seconds). Use a reasonable delay to allow"
" pages to load, apps to start, or screen animations to"
" finish.",
"The exact duration to wait in milliseconds. Accurately convert the"
" required time duration into milliseconds based on your objective"
" or plan (e.g., 2000 for 2s, 5000 for 5s, 60000 for 1 minute,"
" 180000 for 3 minutes, 300000 for 5 minutes).",
],
):
"""[ACTION] Pause execution and wait for a fixed duration.
"""[ACTION] Pause execution and wait for a specified duration in milliseconds.
Use this to handle loading states, app launches, network buffers, or
simple timing gaps between actions when the UI is transitioning.
Use this whenever you need time to elapse—whether for UI loading, animations,
screen transitions, or longer scheduled delays and intervals specified in the task.
"""
return "Action Recorded"
@@ -1356,42 +1361,17 @@ class OperatorNode:
duration = parsed_duration
if kind == "direction":
gesture = target
# Align with FinalRun directional swipe vector rules:
# Locks X axis to 60% of width for up/down to prevent mis-clicks on alphabet sidebar.
# Horizontal Y axis is locked to 60% of height.
if gesture == "up":
coords = [
int(width * 0.6),
int(height * 0.8),
int(width * 0.6),
int(height * 0.2),
]
elif gesture == "down":
coords = [
int(width * 0.6),
int(height * 0.2),
int(width * 0.6),
int(height * 0.8),
]
elif gesture == "left":
coords = [
int(width * 0.8),
int(height * 0.6),
int(width * 0.2),
int(height * 0.6),
]
elif gesture == "right":
coords = [
int(width * 0.2),
int(height * 0.6),
int(width * 0.8),
int(height * 0.6),
]
else:
return [], f"Error: Invalid swipe direction '{gesture}'."
x1, y1, x2, y2 = coords
x1, y1, x2, y2, smart_dur = compute_smart_swipe_coordinates(
direction=target,
target=args.get("target"),
indexed_elements=getattr(state, "indexed_elements", None),
ui_hierarchy=getattr(state, "ui_tree", None),
width=width,
height=height,
duration=duration,
)
if duration is None:
duration = smart_dur
elif kind == "coords" and isinstance(target, list) and len(target) == 4:
try:
nx1, ny1, nx2, ny2 = map(float, target)
@@ -1413,19 +1393,17 @@ class OperatorNode:
),
)
# Calculate duration automatically if not provided (safe range 280-400ms to avoid long-press and fling)
# Calculate duration automatically if not provided (safe range 350-900ms to eliminate Android inertial fling)
if duration is None:
dist = ((x2 - x1) ** 2 + (y2 - y1) ** 2) ** 0.5
if dist <= 400:
duration = 280
elif dist <= 800:
duration = 320
elif dist <= 1200:
if dist <= 300:
duration = 350
elif dist <= 1600:
duration = 380
elif dist <= 600:
duration = 550
elif dist <= 1000:
duration = 800
else:
duration = 400
duration = 900
nx1, ny1 = int(round(x1 * 1000.0 / width)), int(round(y1 * 1000.0 / height))
nx2, ny2 = int(round(x2 * 1000.0 / width)), int(round(y2 * 1000.0 / height))
artemis/agents/outputter/outputter.py
+22 -18
View File
@@ -35,6 +35,7 @@ from artemis.tools.scratchpad import (
get_update_note_tool_pure,
)
from artemis.tools.video_tool import get_video_analyzer_tool_pure
from artemis.tools.tool_wrapper import invoke_tool_with_injection
from artemis.utils.logger import get_logger
from artemis.utils.task_tree import build_plan_and_history, get_active_subgoal_hashes
from pydantic import BaseModel
@@ -210,25 +211,28 @@ async def outputter(
args = tc["args"]
logger.info(f"Outputter executing tool {tool_name} with args: {args}")
tool_map = {
"get_step_details": details_tool,
"get_step_screenshot": screenshot_tool,
"search_history_for_text": search_tool,
"list_notes": list_notes_tool,
"read_note": read_note_tool,
"save_note": save_note_tool,
"update_note": update_note_tool,
"append_note": append_note_tool,
"video_analyzer": video_analyzer_tool,
}
try:
if tool_name == "get_step_details":
result = await details_tool.ainvoke(args)
elif tool_name == "get_step_screenshot":
result = await screenshot_tool.ainvoke(args)
elif tool_name == "search_history_for_text":
result = await search_tool.ainvoke(args)
elif tool_name == "list_notes":
result = await list_notes_tool.ainvoke(args)
elif tool_name == "read_note":
result = await read_note_tool.ainvoke(args)
elif tool_name == "save_note":
result = await save_note_tool.ainvoke(args)
elif tool_name == "update_note":
result = await update_note_tool.ainvoke(args)
elif tool_name == "append_note":
result = await append_note_tool.ainvoke(args)
elif tool_name == "video_analyzer":
result = await video_analyzer_tool.ainvoke(args)
selected_tool = tool_map.get(tool_name)
if selected_tool:
result = await invoke_tool_with_injection(
tool=selected_tool,
args=args,
tool_call_id=tc["id"],
state=graph_output,
record_trace=True,
)
else:
result = f"Error: Tool {tool_name} is not supported."
status = (
artemis/agents/planner/planner.json
+7 -7
View File
@@ -1,18 +1,18 @@
{
"blocks": {
"role": "## Role\nYou are the Planner agent. Your objective is to decompose high-level user goals into sequential milestones for mobile device execution.",
"role": "## Role\nYou are a Senior Test Engineer leading a team of mobile execution agents. Your objective is to deeply analyze and understand user goals, anticipate execution pitfalls, and formulate robust, observable, and feasibility-bounded milestone blueprints.",
"thinking_process": "## Systematic Thinking Process & Operating Principles\nBefore calling any tools or formulating your final milestones, you MUST systematically analyze the task following these 5 sequential thinking steps in your reasoning:\n\n### 1. Deep Intent Deconstruction & Task Composition\n**Core Principle**: Natural language requests describe the *substantive intent and desired real-world outcome*, not the *rigid system keywords or database filters* of mobile apps. Never mechanically map user phrasing into strict literal searches or surface filters.\n\n**Thinking Checklist (Ask yourself)**:\n- *Task Composition*: What phases make up this task? Is it a sequence of one-off linear steps (e.g., open app, configure settings), an iterative loop over multiple items (e.g., inspecting a list of search results), or a periodic polling loop (e.g., refreshing to monitor new messages, waiting for battery level)?\n- *Substantive Intent & Semantic Expansion (Connecting User Words to App Reality)*: What information or outcome is the user truly looking for, and how might that actually appear inside the target app? For example, details for an event taking place in November might have been sent months earlier (in August or September), and the email subject might use terms like \"Summit\", \"Invitation\", or \"Confirmation\" rather than the literal words \"November conference\". Proactively brainstorm the synonyms, related terms, project codes, and indirect records the app might use.\n\n**Abstract Behavioral Contrast**:\n- ❌ *Anti-Pattern (Mechanical Overfitting)*: Treating descriptive modifiers as strict literal filters -> Pre-judging items solely by surface list titles -> Prematurely missing valid candidates.\n- ✅ *Pro-Pattern (Intent Expansion & Deep Inspection)*: Decoupling core intent from surface wording -> Considering all plausible forms and timeframes -> Opening items to verify their real contents.\n\n### 2. Visual Grounding & Starting Context\n**Core Principle**: Every plan must be rooted strictly in the current physical reality of the device.\n\n**Thinking Checklist (Ask yourself)**:\n- *Current State Analysis*: What active app, screen layout, and interactive elements are visible in the provided screenshot to serve as the immediate starting point?\n\n### 3. Tiered Strategy, Funnel Approach & Feasibility Safeguards\n**Core Principle**: Formulate an efficient hierarchy of execution strategies, use a broad-to-narrow funnel for finding information, and enforce clear boundaries on loops and exploration.\n\n**Thinking Checklist (Ask yourself)**:\n- *Tiered Paths*: What is the fastest, highest-signal optimal path (e.g., targeted in-app search, direct shortcuts)? If that fails or yields no results, what is the pragmatic fallback path (e.g., browsing categories, chronological scanning)?\n- *The Funnel Approach (Broad Discovery -> Deep Verification)*:\n - **Top of Funnel (Broad Gathering)**: First collect all potentially relevant candidates (via multi-keyword search, category browsing, or scanning). Do not filter out candidates prematurely based only on incomplete snippet text in list views.\n - **Bottom of Funnel (Deep Inspection)**:\n - Systematically open each candidate to thoroughly inspect its details and confirm whether it meets the criteria.\n- *Feasibility Safeguards & Loop Boundaries*:\n - *For Item Iteration & Exploration*: Set explicit upper bounds (e.g., \"cap inspection at top 15 candidates\", \"scan recent 3 months\") to prevent endless scrolling.\n - *For Periodic Monitoring / Polling*: Specify the check action (e.g., pull-to-refresh, reading system telemetry via ADB), the wait interval between checks (e.g., check every 5 minutes to conserve resources), and the exit criteria (e.g., stop when condition is met, or continue until manually stopped).\n\n### 4. Risk Pre-mortem & Defensive Design\n**Core Principle**: Proactively assume that execution will encounter obstacles, and build self-healing and verification checkpoints directly into the milestones.\n\n**Thinking Checklist (Ask yourself)**:\n- *Anticipate Roadblocks*: What common mobile friction points could occur (e.g., login/auth walls, permission dialogs, promotional popups/ads, search ambiguities, empty states, network latency)?\n- *Embedded Defense*: What observable criteria can be embedded into the milestone to help the executor immediately verify success or detect derailment without mistaking normal wait intervals for system stalls?\n\n### 5. Milestone Granularity, Outcome-Orientation & Dynamic Live-Iteration Tracking\n**Core Principle**: Maintain a strict two-tier hierarchy: top-level milestones define strategic macro-phases, while indented sub-tasks encapsulate every individual iteration instance within a loop.\n\n**Thinking Checklist (Ask yourself)**:\n- *Outcome-Oriented Granularity (2-5 Strategic Phases)*: Top-level milestones describe *what macro result to achieve* (e.g., \"Search for candidate emails and extract confirmed itiLine truncated
"capabilities_operator": "## Execution Capabilities\n\nThe execution agent primarily interacts with the mobile device and analyzes screen recordings and logs. It can perform the following types of actions:\n- **UI Interaction**: Tapping, swiping, and entering text on the screen.\n- **App Management**: Launching and stopping specific applications.\n- **Information Retrieval & Analysis**: Reading on-screen text and UI hierarchy, searching the web for external information, querying system logs, and analyzing screen recordings.\n- **Terminal & Background Operations**: Executing arbitrary ADB shell commands on the device for exploration. It is capable of running long-running ADB commands asynchronously in the background as tasks.\n\n### Capability Boundaries & Planning Rules\n- **Cross-App Operations**: When a task requires interacting with multiple applications, split the operations into distinct milestones for each app. Avoid mixing actions for different apps in a single milestone.",
"output_protocol_initial": "## Task Plan & Scratchpad Protocol\n\n1. **Articulate Thinking**: Begin by reasoning through the 5-step **Systematic Thinking Process** to establish your strategy, funnel approach, bounds, and pre-mortem checks.\n2. **Designate Notes in Milestones & Pre-generate Answer Cards**: If the task involves collecting findings, logging periodic monitoring events, recording candidates, or analytical summaries, designate clear, descriptive note names directly in the milestone descriptions (e.g., `candidates_list`, `monitoring_log`, `schedule_summary`). You can and should invoke the `save_note` tool to pre-generate these structured template files (answer card / scratchpad skeletons with Markdown headers, sections, or tables) before submitting `task_plan`. This provides downstream executors with a clear, ready-to-fill schema.\n3. **Submit Plan**: Submit your initial plan by invoking the `save_note` tool with `key='task_plan'`. Avoid writing the plan only in your text response, as it will only be recognized by the system if submitted via the tool.\n\n### Plan Formatting Rules\n- Format the plan strictly as a Markdown task list starting with `- [ ]` for uncompleted items.\n- For linear macro-milestones: `- [ ] <Phase Description>`\n- For repetitive, iterative, or periodic polling milestones, tag the item with `[Loop]` and declare its Exit Boundary and Interval (e.g., `- [ ] [Loop] <Overall Loop / Monitoring Goal> (Exit: <Condition or Continuous until manually stopped>; Interval: <e.g., every 5 minutes>)`).\n- **Strict Two-Tier Hierarchy & Loop Subtask Contract**:\n 1. **Subtask Nesting Rule**: ALL individual iteration instances, polling checks, candidate inspections, or baseline setups MUST be formatted as 2-space indented sub-tasks (` - [ ]`) directly nested under their parent `[Loop]` milestone.\n 2. **Mandatory First-Instance Initialization**: Every `[Loop]` milestone MUST be initialized with its first concrete child sub-task (e.g., ` - [ ] Inspect Candidate #1: ...` or ` - [ ] Polling Check #1 (Baseline & Initial Sync): ...`).\n 3. **No Flattened Cycles**: NEVER elevate individual checks, recurring polling cycles, or candidate inspections to top-level milestones outside the `[Loop]`. Top-level items are reserved exclusively for strategic macro-phases.\n- Ensure the content of the `task_plan` note contains only the task list, excluding introductions, explanations, conversational text, or headers.",
"examples_initial": "## Master Example: Demonstrating the Complete Cognitive Paradigm\n\n**Goal**: \"Find and summarize all schedule arrangements and venue details for the upcoming November tech conference from my emails.\"\n\n### Naive / Incorrect Approaches:\n1. *Flawed Semantics & Too Granular*:\n- [ ] Tap search icon in Gmail\n- [ ] Type \"November\"\n- [ ] Open first email and tap back\n*(Fails: Micromanages atomic UI taps; assumes emails for November arrangements must have \"November\" in the title or be received in November)*\n2. *Vague & Unbounded Trap*:\n- [ ] Open email and check all emails for conference details\n*(Fails: Unbounded scan trap; causes infinite scrolling and step budget exhaustion on mobile)*\n\n---\n\n### Correct Approach (Full Cognitive Thinking & Robust Plan):\n\n#### Step-by-Step Thinking:\n1. **Intent & Semantic Expansion**:\n - *Core Objective*: Extract confirmed schedule dates, sessions, and venue info for the November tech conference.\n - *Semantic Insight*: Conference communications are often sent months in advance (e.g., ticket confirmations or registration emails in August/September). The subject line may say \"Global Developer Summit\", \"Tech Expo Agenda\", or \"Registration Confirmed\" rather than literally \"November conference\".\n - *Task Composition*: Linear setup (open email app and search) -> Iterative candidate inspection loop (examine each promising email) -> Final consolidation.\n2. **Visual Grounding**:\n - The device screenshot shows the Home screen with the Gmail app icon visible. The plan must start by opening the email client.\n3. **Tiered Strategy, Funnel Approach & Safeguards**:\n - *Tier 1 (Optimal)*: Use in-app search with rich semantic keywords (\"conference\", \"summit\", \"schedule\", \"agenda\", \"venue\", \"November\").\n - *Tier 2 (Fallback)*: If search is unavailable or yields zero results, fallback to chronological scanning of recent 3 months.\n - *Funnel (Broad Gathering -> Deep Inspection)*: Gather candidate emails from search results; do not discard candidates based solely on list previews. Open each candidate thread to inspect full body details.\n - *Feasibility Boundary*: Cap the candidate email inspection pool at top 15 emails to prevent infinite scrolling.\n4. **Risk Pre-mortem**:\n - *Risks*: Promotional marketing spam; conflicting drafts with preliminary dates; cancelled sessions.\n - *Defensive Check*: Instruct downstream executor to filter out promotional newsletters, verify session timestamps, and confirm final venue address.\n5. **Milestone Granularity & Scratchpad Blueprint**:\n - Focus milestones on business outcomes (search, inspect candidates, consolidate), avoiding micromanaged click-by-click steps.\n - Use indented sub-tasks to guide the loop of inspecting each candidate email, initializing with Item #1 and adding subsequent items as they are identified. This prevents the agent from losing state or re-inspecting the same items.\n - Designate note `november_conference_summary` as the destination for recording findings and pre-generate the structured template note.\n\n#### Tool Invocation:\n1. Pre-generate designated answer card note template:\n`save_note(key='november_conference_summary', content='# November Tech Conference Summary\\n\\n## Key Details\\n- Conference Name: \\n- Dates: \\n- Venue / Address: \\n\\n## Daily Schedule & Sessions\\n(To be populated during email inspection)\\n')`\n\n2. Submit milestone plan:\n`save_note(key='task_plan', content='- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\\n - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\\n- [ ] Review and consolidate all extracted information in `november_conference_summary` into a clear, finalized conference itinerary')`\n\n#### Content of `task_plan` Note:\n- [ ] Open email application and search for candidate emails regarding the November tech conference (Search targeted keywords like \"conference\", \"summit\", \"agenda\", \"venue\"; if search is unavailable, fallback to scanning recent 3 months, capped at top 15 emails)\n- [ ] [Loop] Inspect candidate emails and record verified schedule sessions and venue details into note `november_conference_summary` (Exit: All key conference details identified OR candidate pool exhausted / max 15 inspected)\n - [ ] Inspect Candidate #1: Open candidate, filter out non-relevant content, check body for scheduling/venue, and log findings if present\n- [ ] ReviLine truncated
"role_validator": "## Role\nYou are the Task Plan Validator. The Operator has attempted to modify the top-level task plan. Your job is to ensure the proposed new plan still aligns with the user's original intent and is viable.",
"guidelines": "## Operating Principles\n\n### 1. Visual Grounding (Start from Current State)\nBase all planning decisions strictly on the provided screenshot of the current device state. Your plan must begin from the current context of the screen.\n\n### 2. Milestone Granularity\nDefine high-level milestones rather than granular UI operations. The milestones you establish are essential checkpoints that the task must pass through; your goal is to keep the task on the right track, not to think about how the task should be completed step-by-step.\n\n### 3. Flexibility and Possibility Retention (Avoid Over-Specification)\nAvoid unnecessarily restricting the execution path or specific tools/applications unless strictly mandated by the user. For example, if the user asks to perform a task that can be accomplished via multiple interfaces (such as a native app or a web browser), describe the milestone in terms of the desired outcome rather than forcing a specific application or interface (e.g. forcing the use of a specific native app when a web browser could also serve). Allow the downstream execution agent the flexibility to choose the most viable path based on the dynamic state of the device.\n\n### 4. Heuristics for Complex & Iterative Operations\nWhen a milestone requires the downstream agent to perform complex iterations, maintain state, or map computed data back to the UI (e.g., adding a pre-selected list of items to a system), you MUST provide heuristics via indented sub-tasks. These sub-tasks should instruct the agent on the exact iterative process (e.g., searching for each item one by one from a note, applying it, and verifying the final state).",
"capabilities_operator": "## Execution Capabilities\n\nThe execution agent primarily interacts with the mobile device and analyzes screen recordings and logs. It can perform the following types of actions:\n- **UI Interaction**: Tapping, swiping, and entering text on the screen.\n- **App Management**: Launching and stopping specific applications.\n- **Information Retrieval & Analysis**: Using OCR to read text on screen, searching the web for external information, querying system logs, and analyzing screen recordings.\n- **Terminal & Background Operations**: Executing arbitrary ADB shell commands on the device for exploration. It is capable of running long-running ADB commands asynchronously in the background as tasks.\n\n### Capability Boundaries & Planning Rules\n- **Cross-App Operations**: When a task requires interacting with multiple applications, split the operations into distinct milestones for each app. Avoid mixing actions for different apps in a single milestone.",
"output_protocol_initial": "## Task Plan & Scratchpad\nSubmit your initial plan by invoking the `save_note` tool with `key='task_plan'`. Avoid writing the plan in your text response, as it will only be recognized if submitted via the tool.\n\nFormat the plan as a Markdown task list starting with `- [ ]` for uncompleted items. You can use indented sub-tasks (e.g., ` - [ ]`) to break down complex exploratory steps and provide heuristics to guide the downstream agent. Ensure the content of the `task_plan` note contains only the task list, excluding any introductions, explanations, or headers.\n\n### Creating Additional Notes\nYou are highly recommended to create additional empty notes (like answer sheets) for the downstream executor to fill out with findings, lists, or analytical information relevant to the task. For example, if the user asks you to analyze specific YouTube videos, you should pre-create an empty note with a descriptive name (e.g., `analysis_of_xxx_videos`), and make sure to put the note name directly inside your milestones to guide the downstream executor in filling out this answer sheet.",
"examples_initial": "## Examples\n\n### Example 1: Basic Task\n**Goal**: \"Send 'I will be late' to Alice on WhatsApp\"\n\n**Incorrect (Too granular)**:\n- [ ] Tap search icon\n- [ ] Type Alice\n- [ ] Tap Alice's avatar\n\n**Correct (Milestones)**:\n- [ ] Open the chat window with Alice in WhatsApp\n- [ ] Successfully send the message \"I will be late\" in Alice's chat window\n\n### Example 2: Information Retrieval & Scratchpad\n**Goal**: \"Find 10 nature videos on YouTube\"\n\n**Incorrect (Vague or directly jumping to end)**:\n- [ ] Open YouTube\n- [ ] Search nature videos and pick 10\n\n**Correct (Leveraging Scratchpad & Sub-tasks Heuristics)**:\n*(Note: As the Planner, pre-create the required empty notes `nature_videos_candidates` and `nature_videos` using the note-taking tools before completing the plan)*\n- [ ] Search for \"nature videos\" on YouTube\n- [ ] Save a list of 15 candidate videos into a temporary note called `nature_videos_candidates`\n - [ ] Add candidate 1\n - [ ] Add candidate 2\n - [ ] (Continue until 15 candidates are found)\n- [ ] Evaluate candidate videos from `nature_videos_candidates` one by one, select exactly 10 correct ones, and save them into `nature_videos`\n - [ ] Evaluate candidate 1: watch, filter out if irrelevant, save if correct\n - [ ] Evaluate candidate 2\n - [ ] (Continue evaluating one by one until 10 are selected)",
"validator_guidelines": "## Validation Principles\n- **Goal Alignment**: Ensure the new plan ultimately accomplishes the `Initial Goal` requested by the user. If the Operator has abandoned the core objective or fundamentally altered it, you must reject the plan.\n- **Logical Coherence**: Verify that the proposed milestones form a logical, achievable sequence to reach the goal. Reject the plan if it introduces contradictory steps, infinite loops, or obvious logical fallacies.\n- **Adaptive Acceptance**: The Operator operates in a dynamic environment and may encounter unexpected UI states. You should approve alternative strategies, extra intermediate steps, or necessary workarounds as long as they serve the primary goal effectively.\n- **Constructive Feedback**: If you must reject the plan, provide clear, objective, and actionable feedback explaining the logical flaw or goal deviation, and suggest how the Operator should correct the plan."
"validator_guidelines": "## Validation Principles\n- **Goal Alignment**: Ensure the new plan ultimately accomplishes the `Initial Goal` requested by the user. If the Operator has abandoned the core objective or fundamentally altered it, you must reject the plan.\n- **Logical Coherence**: Verify that the proposed milestones form a logical, achievable sequence to reach the goal. Support valid ongoing iteration/polling subgoals and trigger branches without misconstruing periodic checks as infinite loops. Reject the plan if it introduces contradictory steps, unhandled deadlocks, or obvious logical fallacies.\n- **Loop & Monitoring Protection**: If the existing plan contains an active `[Loop]` milestone (especially continuous polling / monitoring with exit condition `until manually stopped`), you MUST REJECT any proposed plan that deletes the `[Loop]` milestone, removes the loop tag, or marks a continuous monitoring milestone as completed (`[x]`) without explicit external user instruction.\n- **Adaptive Acceptance**: The Operator operates in a dynamic environment and may encounter unexpected UI states or dynamic event triggers. You should approve alternative strategies, extra intermediate steps, loop status tracking, or necessary workarounds as long as they serve the primary goal effectively.\n- **Constructive Feedback**: If you must reject the plan, provide clear, objective, and actionable feedback explaining the logical flaw or goal deviation, and suggest how the Operator should correct the plan."
},
"modes": {
"initial_plan": {
"system": [
"role",
"guidelines",
"thinking_process",
"capabilities_operator",
"output_protocol_initial",
"examples_initial"
artemis/agents/summarizer/summarizer.md
+6 -3
View File
@@ -13,16 +13,19 @@ You are the Summarizer Agent for a mobile automation system. Your task is to sum
- **BANNED WORDS**: "successfully", "completed", "entered", "navigated to", "failed", "unsuccessful", "could not", "achieved".
3. **Focus on Intent & Strategy**: The specific physical action (exact coordinates, swipes) is now recorded automatically. Focus primarily on the high-level Intent and reasoning. If you realize the Operator is on the wrong path and needs to backtrack or pivot, describe this as a shift in strategy.
4. **Intended Context Transitions**: Describe the intended navigation path ("From Where -> Intended Where"). Do not declare that the transition actually happened. (e.g., "From the Home screen, I intended to open the Network Settings view").
5. **Preserve Verifications & Missing Items**: If the Operator verified any state in this step, or explicitly mentioned what prerequisites or items are still missing to accomplish the goal, you MUST state these contents in detail in the summary.
5. **Preserve Verifications & Iteration Progress**: If the Operator verified any state in this step, processed a specific candidate item/polling round, or explicitly mentioned what prerequisites or items are still missing, you MUST succinctly preserve these concrete progress details and verified items in the summary.
6. **Objective Intervention Logs**: If the Failure Analyzer intervened, faithfully summarize what triggered the intervention and what the analyzer executed, without judging the outcome.
# LOOP & STAGNATION DETECTION
Use the `RECENT 10 STEPS HISTORY` to intelligently detect loops or stagnation:
1. **Physical Action Loops**: Be vigilant for loops spanning multiple steps based on physical interactions. If you detect a valid action loop, explicitly mention the pattern and the current iteration count (e.g., "This is the 3rd cycle of attempting to access the video group folder").
2. **Cognitive Stagnation**: While you generally ignore internal background tool calls (e.g., reading logs, checking states) when counting physical action loops, there is an exception: if you detect continuous multiple steps of purely internal tool calls without any physical actions, you MUST objectively record this strategic "stagnation" or "analysis loop".
1. **Physical Action Loops vs. Planned Iterations**:
- *Planned Iterations / Monitoring (Valid Progress)*: If the Operator is systematically processing different candidate items in a loop, or executing planned periodic polling cycles with wait intervals under a `[Loop]` milestone, record this objectively as planned progress (e.g., "Inspected Candidate #2", "Completed Polling Round #2 for new messages"). Do NOT flag planned iterations across different items or spaced polling as abnormal loops.
- *Unintended Action Loops (Anomalies)*: If the Operator is repeatedly clicking the exact same target, failing to progress on the same item, or oscillating blindly between identical states without updating progress notes, explicitly record the anomalous loop pattern and iteration count (e.g., "This is the 3rd cycle of failing to open the same folder").
2. **Cognitive Stagnation**: While you generally ignore internal background tool calls (e.g., reading logs, checking states) when counting physical action loops, if you detect continuous multiple steps of purely internal tool calls without any physical actions or note updates, objectively record this "stagnation" or "analysis loop".
# INPUT STRUCTURE
You will receive:
- **CURRENT PLAN & TASK PLAN**: The current task plan including milestones, active subgoals, and live iteration checklists to help you understand the strategic context and loop phase.
- **RECENT 10 STEPS HISTORY**: A brief history of the last 10 steps to help you detect loops and stagnation patterns.
- **CHRONOLOGICAL STEP TRACE (CURRENT STEP)**: The step-by-step chronological history of the current execution turn, including:
- `[Operator Monologue] / [Operator Native Thought]`: The operator's planning, intentions, progress counting, reasoning, verified states, and explicit mentions of missing prerequisites.
artemis/agents/validator/tool_declarations.py
+91 -47
View File
@@ -17,8 +17,10 @@
import ast
import asyncio
import base64
import hashlib
import inspect
import json
from pathlib import Path
import time
from typing import Any, Literal
@@ -35,7 +37,10 @@ from artemis.tools.explorer_tool import _run_explorer_logic
from artemis.graph.state import State
from artemis.tools.mobile.launch_app import find_package
from artemis.utils.app_launch_utils import launch_app_with_retries
from artemis.utils.coordinates import parse_swipe_parameters
from artemis.utils.coordinates import (
compute_smart_swipe_coordinates,
parse_swipe_parameters,
)
from artemis.utils.logger import get_logger
from artemis.utils.ocr_xml_fusion import fuse_ocr_with_xml
from artemis.utils.visualization import format_minimal_list_with_elements
@@ -282,6 +287,31 @@ def normalize_coordinate_target(target: Any) -> int | list[int] | Any:
return target
def serialize_mobile_action_result(result: Any) -> dict[str, Any]:
"""Serializes MobileActionExecutor execution results into a clean structured dict.
Extracts the post-action screenshot hash from the captured image to eliminate guesswork.
"""
if not isinstance(result, tuple) or len(result) < 3:
return {"outcome": str(result) if result is not None else ""}
outcome, img_bytes, shot_path, *rest = result
xml_list = rest[0] if rest else None
post_image_name = None
if shot_path:
post_image_name = Path(shot_path).stem
elif img_bytes:
post_image_name = hashlib.sha256(img_bytes).hexdigest()
is_err = bool(outcome and ("Error" in outcome or "Failed" in outcome))
return {
"outcome": outcome,
"post_image_name": post_image_name,
"has_xml": bool(xml_list),
"status": "error" if is_err else "success",
}
class MobileActionExecutor:
"""Encapsulates device action execution, observation capture, and result reporting."""
@@ -289,7 +319,7 @@ class MobileActionExecutor:
self.ctx = ctx
self.controller = controller or UnifiedMobileController(ctx)
@trace(type="tool", name="click")
@trace(type="action", name="click", serializer=serialize_mobile_action_result)
async def exec_click(
self,
target: list[int] | Any,
@@ -327,7 +357,7 @@ class MobileActionExecutor:
return outcome, img_bytes, shot_path, xml_list
@trace(type="tool", name="click_sequence")
@trace(type="action", name="click_sequence", serializer=serialize_mobile_action_result)
async def exec_click_sequence(
self,
sequence: list[int | list[int] | str] | str,
@@ -407,7 +437,7 @@ class MobileActionExecutor:
except Exception as e:
return f"Error executing click sequence: {e}", None, None, None
@trace(type="tool", name="long_press")
@trace(type="action", name="long_press", serializer=serialize_mobile_action_result)
async def exec_long_press(
self,
target: list[int] | Any,
@@ -449,7 +479,7 @@ class MobileActionExecutor:
return outcome, img_bytes, shot_path, xml_list
@trace(type="tool", name="input_text")
@trace(type="action", name="input_text", serializer=serialize_mobile_action_result)
async def exec_input_text(
self,
text: str,
@@ -484,11 +514,11 @@ class MobileActionExecutor:
return outcome, img_bytes, shot_path, xml_list
@trace(type="tool", name="swipe")
@trace(type="action", name="swipe", serializer=serialize_mobile_action_result)
async def exec_swipe(
self,
action: str | list[int] | Any = None,
duration: int = 400,
duration: int = 800,
state: State | None = None,
duration_ms: int | None = None,
**kwargs: Any,
@@ -510,33 +540,16 @@ class MobileActionExecutor:
try:
if kind == "direction":
dir_name = target
dir_map = {
"up": (
int(width * 0.6),
int(height * 0.8),
int(width * 0.6),
int(height * 0.2),
),
"down": (
int(width * 0.6),
int(height * 0.2),
int(width * 0.6),
int(height * 0.8),
),
"left": (
int(width * 0.8),
int(height * 0.5),
int(width * 0.2),
int(height * 0.5),
),
"right": (
int(width * 0.2),
int(height * 0.5),
int(width * 0.8),
int(height * 0.5),
),
}
x1, y1, x2, y2 = dir_map[dir_name]
x1, y1, x2, y2, smart_dur = compute_smart_swipe_coordinates(
direction=target,
target=swipe_input.get("target"),
indexed_elements=getattr(state, "indexed_elements", None) if state else None,
ui_hierarchy=getattr(state, "ui_tree", None) if state else None,
width=width,
height=height,
duration=final_duration,
)
final_duration = smart_dur
err = await self.controller.swipe_coords(x1, y1, x2, y2, final_duration)
if err:
return f"Error swiping {dir_name}: {err}", None, None, None
@@ -569,7 +582,7 @@ class MobileActionExecutor:
return outcome, img_bytes, shot_path, xml_list
@trace(type="tool", name="press_key")
@trace(type="action", name="press_key", serializer=serialize_mobile_action_result)
async def exec_press_key(
self,
key: str,
@@ -603,7 +616,7 @@ class MobileActionExecutor:
return outcome, img_bytes, shot_path, xml_list
@trace(type="tool", name="manage_app")
@trace(type="action", name="manage_app", serializer=serialize_mobile_action_result)
async def exec_manage_app(
self,
action: str,
@@ -649,7 +662,7 @@ class MobileActionExecutor:
return outcome, img_bytes, shot_path, xml_list
@trace(type="tool", name="wait_for_delay")
@trace(type="action", name="wait_for_delay", serializer=serialize_mobile_action_result)
async def exec_wait_for_delay(
self,
time_in_ms: int,
@@ -668,7 +681,7 @@ class MobileActionExecutor:
return outcome, img_bytes, shot_path, xml_list
@trace(type="tool", name="wait_for_text")
@trace(type="action", name="wait_for_text", serializer=serialize_mobile_action_result)
async def exec_wait_for_text(
self,
text: str,
@@ -979,24 +992,51 @@ INPUT_TEXT_TOOL = ToolDeclaration(
SWIPE_TOOL = ToolDeclaration(
name="swipe",
description=(
"[ACTION] Scroll or swipe on the screen. The screen after swipe will be"
" returned automatically. Set duration >= 1000 to drag-and-drop."
"[ACTION] Perform a swipe, drag, or slider-adjustment gesture on the screen. The screen and UI hierarchy after swipe will be returned automatically.\n\n"
"• Directional Scrolling ('direction' or 'action'): Recommended for general browsing and standard page scrolling in most scenarios. Automatically computes safe swipe vectors and adaptive duration, retains a ~40% visual overlap anchor for zero-omission traversal, and prevents inertial flings. Supports scoping to a sub-container via 'target'. If it fails on certain custom layouts, fall back to specifying exact coordinates ('start' and 'end') directly.\n"
"• Precise Coordinate Gestures ('start', 'end' or coordinates list): Best for local, fine-grained interactions such as adjusting sliders/SeekBars (e.g., volume, brightness, progress bars), drag-and-drop / list reordering, or as a reliable fallback when directional scrolling fails on specific containers. Always drag slightly PAST the target position to overcome touch slop and reliably trigger the update. When setting a slider to Maximum (100%) or Minimum (0%), swipe fully to the extreme boundary."
),
parameters={
"type": "object",
"properties": {
"direction": {
"type": "string",
"enum": ["up", "down", "left", "right"],
"description": (
"Direction for scrolling and swiping: 'up' (drags bottom-to-top, scrolling down to reveal content below),"
" 'down' (drags top-to-bottom, scrolling up to reveal content above),"
" 'left' (drags right-to-left, scrolling right),"
" 'right' (drags left-to-right, scrolling left)."
),
},
"start": {
"type": "array",
"items": {"type": "integer"},
"description": "Start normalized coordinates [start_x, start_y] in 0-1000 scale.",
},
"end": {
"type": "array",
"items": {"type": "integer"},
"description": "End normalized coordinates [end_x, end_y] in 0-1000 scale.",
},
"target": {
"description": "Optional target element index (e.g. 2) or container bounds [left, top, right, bottom] to scope the directional swipe within.",
},
"action": {
"description": (
"Swipe action direction string ('up', 'down', 'left', 'right') OR custom"
" coordinates [start_x, start_y, end_x, end_y] in 0-1000 scale."
)
"Backward-compatible swipe gesture: smart direction string ('up', 'down', 'left', 'right')"
" OR precise custom coordinates [start_x, start_y, end_x, end_y] in 0-1000 scale."
),
},
"duration": {
"type": "integer",
"description": "Swipe duration in milliseconds (default 400).",
"description": (
"Optional swipe/drag duration in milliseconds (default 800). For drag-and-drop,"
" list reordering, or sliding/adjusting sliders (e.g., volume, brightness, SeekBars),"
" set duration >= 1000 (e.g. 1500). If omitted for directional swipe, duration is computed automatically."
),
},
},
"required": ["action"],
},
)
@@ -1074,7 +1114,7 @@ MANAGE_APP_TOOL = ToolDeclaration(
WAIT_FOR_DELAY_TOOL = ToolDeclaration(
name="wait_for_delay",
description=(
"[ACTION] Pause execution and wait for a fixed duration in milliseconds."
"[ACTION] Pause execution and wait for a specified duration in milliseconds."
" The screen after pause will be returned automatically."
),
parameters={
@@ -1082,7 +1122,11 @@ WAIT_FOR_DELAY_TOOL = ToolDeclaration(
"properties": {
"time_in_ms": {
"type": "integer",
"description": "The exact duration to pause in milliseconds (e.g., 2000).",
"description": (
"The exact duration to pause in milliseconds (e.g., 2000 for 2s,"
" 60000 for 1 min, 180000 for 3 mins, 300000 for 5 mins)."
" Convert any required waiting duration into milliseconds."
),
}
},
"required": ["time_in_ms"],
Loaded 100 of 248 files, more files were not shown because too many files have changed in this diff. Show more